# Mix and Mask Actor-Critic Methods

Dom Huh
1Department of Computer Science,
University of California, Davis, CA, USA
<EMAIL_ADDRESS>
###### Abstract

Shared feature spaces for actor-critic methods aims to capture generalized
latent representations to be used by the policy and value function with the
hopes for a more stable and sample-efficient optimization. However, such a
paradigm present a number of challenges in practice, as parameters generating
a shared representation must learn off two distinct objectives, resulting in
competing updates and learning perturbations. In this paper, we present a
novel feature-sharing framework to address these difficulties by introducing
the mix and mask mechanisms and the distributional scalarization technique.
These mechanisms behaves dynamically to couple and decouple connected latent
features variably between the policy and value function, while the
distributional scalarization standardizes the two objectives using a
probabilistic standpoint. From our experimental results, we demonstrate
significant performance improvements compared to alternative methods using
separate networks and networks with a shared backbone. The code for this paper
can be found in the following link: https://github.com/domhuh/mixmaskac.

##  1 Introduction

Deep reinforcement learning (RL) has risen to accomplish impressive feats in
diverse applications: from playing Atari games [14] to solving classic control
tasks [8]. The success of this field can be attributed to breakthroughs in
novel optimization methods [11, 13, 24, 6, 20, 22, 9] and innovative
mechanisms [14, 19, 2, 32] to boost stability and sample-efficiency. In
particular, many of these algorithms have relied on the RL framework called
the actor-critic method. As demonstrated in past literature [28], this class
of algorithms introduce a bootstrapping value function working in conjunction
with policy gradient optimization as to reduce variance and accelerate
learning.

In this paper, we suppose and argue the hypothesis that sharing latent
features between a policy and value function will capture more generalized and
rich representations and that latent features encouraged by one function may
be highly useful for the other and vice-versa, enabling for more stable and
sample-efficient optimization. In turn, we assert that the representations
learned by both the policy and value function are complementary in some
respect, thereby should be related in some manner. However, we must address
two central problems that arises in this proposition: the definability of
function interdependence and the challenge of multi-objective optimization.

Let us consider the dynamics of the relationship between the policy and value
function. We focus on two important components that ensures a stable and
definable interdependence: the initialization and conditioning of the two
functions in question. Often for actor-critic algorithms, we treat the
initialization and conditioning process for the policy and value function
independently, and update, or condition, these functions disproportionately.
Consequently, their interdependence therefore begins ambiguously and remains
non-stationary throughout optimization. In fact, this connection can be
quickly redefined as the application at hand changes. Hence, having shared
features between these two functions can detriment learning if proper
precautions are not in place since their relationship is difficult to define
and can be easily distorted. Secondly, as the actor-critic methods use two
distinct objectives for the policy and value function, the presence of a
shared latent space forms a multi-objective optimization problem. In this
setting, we are confronted with two issues. First, it is possible for one
objective to dominate over the other, resulting in performance disparity
between objectives. Second, there may be interference between the two
objectives during optimization, causing competing changes in the parameters.
With dominance and perturbations, training without any safeguard can be highly
unstable and sample-inefficient.

Figure 1: Change in Similarity Measure: On the left, the figure shows the
separated network paradigm. On the right, the figure shows the shared backbone
paradigm. For both paradigms, a visualization of the hidden representations
and the expected difference in the similarity measure before and after
training are shown. We obtain this expectation from $100$ independent rollouts
on $100$ independent untrained/trained models, each defined on different
random seeds.

We address the two central challenges by introducing the mix-and-mask actor-
critic: a novel approach to mix and mask latent representations shared between
the policy and value function for actor-critic method. We aim to overcome
these difficulties by inducing certain desired behaviors within the shared
features and modeling a distribution to normalize the two objectives. These
behaviors can be seen in Figure 1 from the expected difference in similarity
$\mathbb{E}[S(h^{\prime}_{\pi},h^{\prime}_{v})-S(h_{\pi},h_{v})]$ between the
policy and value function’s hidden representations before and after training.
The hidden representations $h^{\prime}_{\pi},h^{\prime}_{v}$ were taken from
the trained policy and value function, whereas $h_{\pi},h_{v}$ were taken from
their untrained counterparts. From this, we see that the separated network
approach develops more similar features, especially in mid-level abstractions,
whereas the shared backbone network’s features diverge, particularly after the
shared network. Hence, we present two penalty terms to the objectives to guide
the optimization of these mechanisms’ parameters to replicate this behavior.
We extend these base ideas with auxiliary mechanism representations and
contrastive representation learning formulations.

The main contributions of this work is as follows:

  * 1

We present a novel approach to maintain shared features with actor-critic
methods by using mix and mask mechanisms that control the information flow
across shared feature spaces.

  * 2

We introduce new penalty terms to the objective to guide and help these
mechanisms induce particular behaviors within the shared feature space.

  * 3

We consider a novel approach to normalize the objectives using distributional
scalarization.

  * 4

We demonstrate more stable, sample-efficient, and generalized learning with
our method on classic control RL tasks.

##  2 Related Work

Actor-critic methods. Policy gradient methods define a class of model-free RL
optimization algorithms that directly updates the parameters of the policy
$\phi_{\pi}$ with the goal to maximize the expected cumulative reward [33].
However, on it’s own, policy gradient optimization is unstable due to their
high variance performing gradient estimation [27]. To stabilize the
optimization, actor-critic methods introduce a value function estimator to
assist with the policy’s learning by bootstrapping a baseline for policy
gradient calculation. In addition, having access to a value function estimator
allows greater flexibility to work with different degrees of bootstrapping
when using the Bellman Equations. However, these methods do have their
limitations. As a function approximation is used to proxy the value function,
the policy updates have an asymptotic dependence on the quality of the
approximation [27]. In general, a wide array of different solutions were
proposed [20, 22, 34, 13, 32, 9, 24, 6] to improve the actor-critic methods.
In this paper, we focus on developing a framework that allows for shared
features spaces to obtain more sample-efficient and stable learning.

Sharing latent representation. The idea of sharing latent representation has
commonly been practiced in other domains such as vision [18], natural language
[7], and multitask learning [4]. Often employed using transfer learning, or
knowledge transfer across different tasks [15, 17], models have often
demonstrated and relied on the benefits of this sharing to achieve new level
of performance. We aim to extend this idea to deep RL through actor-critic
methods.

Handling shared actor-critic parameters. While past works [31, 3, 16] have
used shared parameters in some limited fashion, mostly by using a shared
backbone, the literature focused on this idea is quite limited. The most
relevant work focused on this concept is [5], where they introduce the Phasic
Policy Gradient (PPG), which splits the policy and value function’s training
into two phases: the policy and auxiliary phase. While the policy phase
follows standard policy gradient optimization using disjoint policy and value
function, the auxiliary phase optimizes an auxiliary loss with a behavioral
cloning loss, updating on a set of parameters shared between an auxiliary
value function and the policy. In this work, we instead aim to regulate the
shared parameters between the policy and value function with the mix and mask
mechanisms.

##  3 Preliminaries

MDP Formalization for RL Tasks. We consider a MDP
$\mathcal{M}=<S,A,P,R,\gamma>$ to describe RL tasks, where $S$ is the set of
all states, $A$ is the set of all actions, $P(s_{t+1},r|s_{t},a_{t})$ is the
transition probability function, $r(s_{t},a_{t})$ is the reward function, and
$\gamma$ is the discount factor. Often, as the classic objective of RL is to
find the parameters $\phi_{\pi}$ of policy $\pi_{\phi_{\pi}}(S|A)$ that
maximizes the cumulative reward $G_{t}=\sum_{k=t+1}^{\infty}\gamma
r(s_{k},a_{k})$, otherwise denoted as the return, we introduce a state-value
function $V(s_{t})=\mathbb{E}[G_{t}|s_{t}]$, or an action-value function
$Q(s_{t},a_{t})=\mathbb{E}[G_{t}|s_{t},a_{t}]$ under a policy $\pi$ to assess
the state’s and state-action pair’s quality respectively. As to solve these
MDPs efficiently, the Bellman equations provides a recursive definition of
these value functions.

$\displaystyle V(s_{t})=\mathbb{E}[r(s_{t+1},a_{t+1})+\gamma
V(s_{t+1})|s_{t}]$ (1) $\displaystyle
Q(s_{t},a_{t})=\mathbb{E}[r(s_{t+1},a_{t+1})+\gamma\mathbb{E}_{a\sim\pi}Q(s_{t+1},a)|s_{t},a_{t}]$
(2)

We use these fundamental ideas of RL in this paper to formulate the mix and
mask mechanisms.

Cart and Pole (CP) Environment. The objective of this task is for an agent to
balance pole on an un-actuated joint by moving a cart along a friction-less
track. The agent is provided the cart’s position, cart’s velocity, pole’s
angle, and pole’s angle velocity. The action space is defined by a discrete
set of forces $\\{+1,-1\\}$ applied onto the cart, which applies a set
magnitude of force. A reward function is defined to provide $+1$ for every
timestep that the pole remains upright, and $-1$ once the pole is more than 15
degrees from vertical or the cart moves more than 2.4 units from the center.
We consider the task to be solved if the agent is able to achieve a return of
$\geq 495$ over $100$ independent trials. In this paper, we use this
environment to evaluate our methods against prior approaches. We also
introduce a new environment, called the non-stationary Cart-and-Pole (nCP),
which randomly changes the length of the pole, mass of the pole, and the mass
of the cart independently at every trial. The agent will only learn on CP
environment, but tested on the nCP environment, as to test generalization.

Representation for Actor-critic methods. With actor-critic methods, the policy
and value function are often represented as neural networks. We can consider
various neural architectures depending on the application at hand. In terms of
function structure, separate networks are most commonly employed for policy
and value function. Some works [31, 3, 16] have used shared backbone with
separate heads, often for RL tasks requiring vision. In this paper, we build
on these two structures by introducing mechanisms that couple and decouple
shared feature spaces.

Multi-Objective Optimization. We construct an RL-specific multi-objective
optimization tasks as such. Let a task be defined by a set of functions
$F=\\{f_{1},f_{2},\dots\\}$, otherwise denoted as the constraint set, and a
corresponding set of objectives $O=\\{o_{1},o_{2},\dots\\}$.

$F^{*}=\min\limits_{F}(O(F))$ (3)

We consider a solution $F^{*}$ to be Pareto optimal if $O(f^{*}_{i})\not\geq
O(f_{i})$,$\forall f_{i}\in F,f^{*}_{i}\in F^{*}$. Thus, a Pareto solution
must take into account the balance over all $o\in O$ as to achieve optimality.
A common practice is to use scalarization techniques, which attempt to
simplify and map $O$ to a singular objective. This reduction thereby produces
an auxiliary single-objective optimization. A classic scalarization method is
the weighted sum method, where the singular objective is obtain by some linear
combination of $O$. However, for complex tasks, it is usually very difficult
to find proper weights to achieve an appropriate balance. Thus, in this work,
we reduce the two objectives of the actor-critic by scalarization using
distributional weighting, described in Section 4.

A2C Optimization. The A2C algorithm update the parameters of the policy
$\phi_{\pi}$ and value function $\phi_{v}$ approximations with the following
objective.

$\nabla
J(\pi,V)=A(s_{t},a_{t})\nabla_{\pi}\log(\pi_{\phi_{\pi}}(a_{t}|s_{t-1}))-\delta_{t}\nabla_{v}V(s_{t})+\mathcal{H}_{\pi}$
(4)

where $A(s_{t},a_{t})$ is the advantage function and
$\delta_{t}=\mathbb{E}[G_{t}]-V(s_{t})$ is the correction term for the value
function. Intuitively, the update seeks to adjust the log likelihood based on
the value of the advantage function. To better approximate the advantage
function, we use the exponentially-weighted estimator introduced in [21],
where now $A(s_{t},a_{t})=\sum_{l}(\gamma\lambda)^{l}\delta_{t+l}^{v}$. We
also add the entropy term $\mathcal{H}_{\pi}=\log(\pi_{\phi_{\pi}}(\cdot))$ to
the objective to increase policy’s exploration [1, 9]. We refer to $\nabla
J_{\pi}=A(s_{t},a_{t})\nabla_{\pi}\log(\pi_{\phi_{\pi}}(a_{t}|s_{t-1}))$ as
the policy update and $\nabla J_{v}=\delta_{t}\nabla_{v}V(s_{t})$ as the value
function update. For our experiments, we used a Monte Carlo approach to
estimate the return.

##  4 Algorithm

In this section, we will discuss our mix-and-mask actor-critic algorithm,
which introduces two novel mechanisms: the mix and mask. We found in practice
that using them in union requires more temperature parameter tuning, and this
variant described later in this section. Thus, we will describe each mechanism
first separately, then elaborate on their operation in conjunction. We then
describe other alternative variants, such as skip connections, meta-learning
and contrastive learning formulations. Lastly, we end this section introducing
the distributional scalarization technique and extending the idea of
projection of conflicting gradients to this work. The description provided
here assumes the actor-critic method requires only a single policy and a
single state-value function.

Figure 2: Mix Actor Critic: On the top left, the figure shows the overall
architecture of the Mix Actor Critic. On the top right, the figure shows the
MLP-Mixer representation of the mix function. On the bottom left, the figure
shows the base representation. On the bottom middle, the figure shows the
convolution representation. On the bottom right, the figure shows the cross-
attention representation.

###  4.1 Mix Actor-critic

Generalized Mix Function. Let $\phi$ be the parameters for the actor-critic
method, and let the representation of the policy and value function be some
neural network. First, consider a separate network paradigm, where
$\phi_{\pi}$, $\phi_{v}$ represent the parameters for the policy and value
function, hence making up a partition for $\phi$. We introduce a new set of
shared parameters $\phi_{s}$ for a function $\textit{Mix}(\cdot)$. Suppose
$x_{\pi}$,$x_{v}$ are latent spaces that can be extracted from the forward
propagation of the policy and value function at some intermediate layer. For
simplicity, we represent the two latent spaces as matrices of the same rank.
Then, we let $\textit{Mix}(\cdot)$ be defined as

$\textit{Mix}(x_{\pi},x_{v})=(\hat{x}_{\pi},\hat{x}_{v})$ (5)

The output of this shared function $\hat{x}_{\pi}$, $\hat{x}_{v}$ will then be
passed back into policy and value function from where the latent spaces
$x_{\pi}$,$x_{Q}$ were extracted. Efficiently, we have replaced independent
latent spaces from the policy and value functions with new representations
that have been coupled. We denote this shared function $\textit{Mix}(\cdot)$
as the mix function. In this paper, we introduce and evaluate on four
different representations of the mix function.

Mix Function Representations. The base representation of the mix function uses
two independent multi-layer perceptron (MLP) models. The extracted latent
representations are concatenated and propagated through each MLP to obtain the
mixed representations for the policy and value function. We now extend the mix
function to variant representations. The second representation, motivated by
the MLPMixer model [29], treats each latent space as a channel and pair
features from the two latent spaces $x_{\pi}$,$x_{v}$ to form a new
representation $x=((x_{(\pi,1)},x_{(v,1)}),(x_{(\pi,2)},x_{(v,2)}),\dots)$. We
propagate this representation $x$ through a MLP to obtain a coupled
representation. The output is then transposed and passed into another MLP. We
can repeat this process, each time increasing the policy and value function’s
coupling. The third representation stacks the two latent spaces
$x_{\pi}$,$x_{v}$ together such that the new representation has two distinct
channels. We convolve the representation using two independent kernel as the
coupling operation. The filters are defined separately for the policy and
value function. Again, we repeat this operation to increase the receptive
field of the mix function. The final representation we will discuss uses the
cross-attention mechanism [30]. Given the two latent spaces $x_{\pi}$,$x_{v}$,
we perform cross-attention in respects to each other. In other words, the
mixed policy representation is obtained by passing $x_{\pi}$ as the key and
value, and $x_{v}$ as the query, while the mixed value representation is
obtained by passing $x_{v}$ as the key and value, and $x_{\pi}$ as the query.
Once each mix function is completed, the mixed representations are placed back
to the policy and value functions.

Stochastic Mix Function. We can consider to parameterize the mix function with
a distribution $\mathcal{D}$. Let $\mathcal{D}$ be a normal distribution
$\mathcal{N}(\mu,\Sigma)$. We decompose the mix function into two mappings
$\textit{Mix}_{\mu}(\cdot),\textit{Mix}_{\Sigma}(\cdot)$ for the mean $\mu$
and covariance matrix $\Sigma$ respectively. If we represent each feature
distribution independently, we let $\Sigma=\sigma^{2}I$. For its
implementation in practice, we map $\textit{Mix}_{\Sigma}(\cdot)$ to the log
standard deviation. On the other hand, if the feature distribution was
multivariate, we must now obtain the full covariance matrix, which is
significantly more computationally expensive. Instead of learning a direct
mapping, we use the Cholesky factorization by learning a vector $c$ such that
$L=(c^{T}c)\odot L_{I}+I$, where $L_{I}$ is the identity lower-triangular
matrix. Thus, we compute the covariance matrix by $\Sigma=L^{T}L$. We use this
approach to softly constrain $\Sigma$ to be positive-definite and non-
singular. In future works, we encourage better methods to learn the covariance
matrix.

Another consideration we propose is whether $\Sigma$ should be state-
independent or state-dependent. For state-independent $\Sigma$, we can store
explicit matrices to hold the covariance information. For state-dependent
$\Sigma$, we use a MLP to process the state to obtain $\Sigma$.

Mix Function Objective. To guide the mixed representations for the policy
$\hat{x}_{\pi}$, $\hat{x}_{v}$ towards the desired behavior, we attach a
similarity term $\mathcal{S}(\hat{x}_{\pi},\hat{x}_{v})$ to the objective,
such that it only affects the weights of the mixing function. This similarity
term provides an incentive to maximize similarity in coupled features.

$\nabla_{\phi}J=\nabla_{\phi_{\pi}}J_{\pi}+\nabla_{\phi_{v}}J_{v}+\alpha_{s}\nabla_{\phi_{s}}\mathcal{S}(\hat{x}_{\pi},\hat{x}_{v})$
(6)

We introduce the temperature parameter $\alpha_{s}$ to control the relative
importance of having similar mixed representations. We decay $\alpha_{s}$
through the optimization $\alpha_{(s,t+1)}=\tau*\alpha_{(s,t)}$, where
$\tau\in[0,1]$.

Similarity Term Representation. If we assume the mixed representations are
sampled from some distribution $\hat{x}_{\pi}\sim\mathcal{D}_{\pi}$ and
$\hat{x}_{v}\sim\mathcal{D}_{v}$, we define our similarity function with a
symmetric statistic metric. One choice is the information radius [23]
$IR(\cdot)$, a symmetric average of the relative Jensen-Shannon divergence
$F(P||Q)=K(P||\frac{P+Q}{2})$, where $K$ denotes the relative information of
$P$ and $Q$.

$IR(P||Q)=\frac{1}{2}(F(P||Q)+F(Q||P))$ (7)

Other similar measures, such as J-divergence $J(\cdot)$ and Arithmetic-
geometric divergence $AG(\cdot)$, the symmetric average of the relative
arithmetic-geometric divergence $G(P||Q)=K(\frac{P+Q}{2}||P)$, were evaluated
as well in our experimentation.

$\displaystyle J(P||Q)=\frac{1}{2}(K(P||Q)+K(Q||P))$ (8) $\displaystyle
AG(P||Q)=\frac{1}{2}(G(P||Q)+G(Q||P))$ (9)

On the other hand, if we have a deterministic mix function, we use a non-
statistical symmetrical distance measure such as cosine similarity.

Figure 3: Mask Actor Critic: On the top left, the figure shows the overall
architecture of the Mask Actor Critic. On the top right, the figure shows the
base representation of the mask function. On the bottom left, the figure shows
the self-attention representation. On the bottom middle, the figure shows the
latent query self-attention representation. On the bottom right, the figure
shows the shared self-attention representation.

###  4.2 Mask Actor-critic

Generalized Mask Function. We follow the same notation as Section 4.1.
However, now consider a shared backbone network paradigm. We define parameters
for the policy $\phi_{\pi}$ and value function $\phi_{v}$ such that the
parameters of the shared backbone function $S$ are defined as
$\phi_{s}=\phi_{\pi}\cap\phi_{v}$ and the output $x_{s}$ of $S$ is propagated
into disjoint parameters $\phi^{\prime}_{\pi}$, $\phi^{\prime}_{v}$. We
introduce a new set of parameters $\phi_{d}$ belonging to a mask function
$\textit{Mask}(\cdot)$ such that it serves as an intermediary between the
parameters $\phi_{s}$ and $\phi^{\prime}_{\pi}$, $\phi^{\prime}_{v}$.

$\textit{Mask}(x_{s})=(x_{\pi},x_{v})$ (10)

Efficiently, we dedicate a function after the shared function $S$ to decouple
the inputs of the policy and value functions before propagating to the
disjoint networks. In this paper, we represent the mask function
$\textit{Mask}(\cdot)$ in four different manners.

Mask Function Representations. The base representation uses two independent
MLP models and propagates the shared latent spaces through the MLPs to obtain
the masked representations for the policy and value function. We now extend
the mask function to variant representations. The second representation for
the mask function consists of two separate self-attention modules, and
propagates the shared features to obtain the decoupled representation. The
third representation shares the same architecture as the second
representation, however, we introduce a learnable latent query that is shared
between the self-attention modules as the query variable. The final
representation uses a shared self-attention module for both policy and value
functions. However, to decouple, we invert the policy attention mask $a_{\pi}$
to obtain the value attention mask $a_{v}$.

$a_{v}=h(1-a_{\pi})$ (11)

where $h$ is a normalizing constant.

Mask Function Objective. Similar to the mix actor-critic, we introduce a
divergence term $\mathcal{D}(x_{\pi},x_{v})$ to the objective such that its
optimization occurs only on the mask function. Unlike the similarity term, the
divergence term seeks to maximize dissimilarity in decoupled features.

$\nabla_{\phi}J=\nabla_{\phi_{\pi}}J_{\pi}+\nabla_{\phi_{v}}J_{v}+\alpha_{D}\nabla_{\phi_{d}}\mathcal{D}(x_{\pi},x_{v})$
(12)

The temperature parameter $\alpha_{D}$ will control the relative importance
having dissimilar masked representations. We follow the same procedure in
handling this parameter as the temperature parameter for mix actor-critic.

We apply the ideas of similarity term representation and stochastic mix
function from the mix actor-critic and extend them naturally to the stochastic
mask function and divergence term representation.

###  4.3 Design Choices

Mix-and-Mask Actor-critic. Now we formulate a method for utilizing both the
mix and mask mechanisms. We again consider a shared backbone network, and
follow the notation from Section 4.2. However, now we introduce the mix
mechanism to the disjoint parameters
$\phi^{\prime}_{\pi}$,$\phi^{\prime}_{v}$. Therefore, the objective function
now consists both the similarity and divergence term, with each term only
affecting their respective mechanism.

Shortcut Connections. From [26], shortcut connections are a popular approach
to ease optimization and avoid performance degradation from additional
parameters. We can merge this idea to our mechanisms, passing back the
unaltered representations with the altered. In our ablative experiments, we
compare dense [12] and residual [10] skip connections.

Auxiliary Mechanisms. We consider an similar approach to [5], where the
outputs of the mix and mask mechanisms are not pass back into the processing
network, but rather only used to evaluate the similarity and divergence terms
for the objective function. In our case, the optimization of these two penalty
terms will affects the preceding parameters of the policy and value function,
and the mechanisms themselves are independent of the policy or value function.
With this, we can view this formulation as attaching auxiliary mechanisms to
the networks.

Contrastive Representation Learning. We can adapt a contrastive representation
learning paradigm to induce the mechanisms’ desired behaviors for the mix and
mask actor-critic models, tuning these functions instead with a contrastive
loss $\mathcal{C}(\cdot)$. To clarify the difference between
$\mathcal{C}(\cdot)$ and the penalty terms $(\mathcal{S},\mathcal{D})$, the
penalty terms act as soft constraints to the overall policy and value function
objective whereas the contrastive loss focuses more on the representation
learned and behaves independently from the main objectives. We adopt ideas
such as momentum functions
$\textit{Mix'}(x_{\pi},x_{v}),\textit{Mask'}(x_{s})$ from [25], where their
respective parameters
$\phi_{\textit{Mix'}}=m\phi_{\textit{Mix}}+(1-m)\phi_{\textit{Mix'}}$, and
$\phi_{\textit{Mask'}}=m\phi_{\textit{Mask}}+(1-m)\phi_{\textit{Mask'}}$,
$m\in(0,1)$ are defined as such. For the mix function, we set our anchors
$q_{\pi},q_{k}$ to be $\hat{x_{\pi}},\hat{x_{v}}$, the positives $k_{+}$ to be
$\hat{x^{\prime}_{v}},\hat{x^{\prime}_{\pi}}$ respectively, and the negatives
$k_{-}$ to be $\hat{x^{\prime}_{\pi}},\hat{x^{\prime}_{v}}$ respectively to
maximize similarity with opposing positives and minimizing individuality with
the complementary negatives. For the mask function, we set our anchors to be
$\hat{x_{\pi}},\hat{x_{v}}$, the positives to be
$\hat{x^{\prime}_{\pi}},\hat{x^{\prime}_{v}}$ respectively, and the negatives
to be $\hat{x^{\prime}_{v}},\hat{x^{\prime}_{\pi}}$ respectively to accomplish
the opposite of the mask’s contrastive objective. We define the contrastive
loss as follows:

$\mathcal{C}(q,k_{+},k_{-})=\log\frac{\exp(qWk_{+})}{\exp(qWk_{+})+\exp(qWk_{-})}$
(13)

where $W$ is a learnable bilinear product. We follow the logit trick to
calculate the cross entropy loss.

Distributional Scalarization. Consider the cost functions for the policy
$J_{\pi}$ and value function $J_{v}$ to be represented by some distribution
$\mathcal{D}$. In our formulation, we let this distribution be a normally
distributed. Then, we obtain the variance from the cost of an untrained policy
$\pi_{0}$ and value function $v_{0}$. Then, we set these costs
$J_{\pi}(\pi_{0})$, $J_{v}(v_{0})$ to have a high z-score, and standardize the
objectives using the calculated mean and sampled variance.

Additionally, we can choose whether or not these distributions are stationary
or non-stationary. If they are stationary, we do not update them throughout
the optimization process. However, if they are non-stationary, the
distributions are updated in intervals. We can aggregate the cost over some
interval to readjust the mean and variance set at a lower z-score. However,
given the tasks we are evaluating on, we sufficiently demonstrate the
efficiency of the distributional scalarization without making the process non-
stationary.

Project Conflicting Gradients (PCGrad). We adopt ideas from past works [35] to
handle gradient conflicts in multi-task learning tasks. In short, the
gradients are projected onto the normal plane of the other objective if they
are conflicting. The method of projection from one gradient $g_{1}$ to another
gradient $g_{2}$ is given as such.

$g_{1}=g_{1}-\frac{g_{1}\cdot g_{2}}{||g_{2}||^{2}}g_{2}$ (14)

We only enforce this procedure on gradients that update the parameters
generating shared feature spaces. In our ablation studies, we compare the
distributional scalarization technique with the PCGrad approach.

##  5 Results

Here, we provide a comparative analysis between our novel framework and
alternative approaches as well as ablative results across different design
considerations discussed in Section 4. For all experiments, we evaluate on the
CP and nCP environment with the A2C optimization algorithm described in
Section 3and run each network below with a wide range of learning rates and
$\epsilon$ values for exponentially-decayed $\epsilon$-greedy action
selection.

###  5.1 Comparative Evaluation

In this section, we will compare our methods with separated networks and
shared backbone networks. In Figure 4, we show a significant improvement in
sample-efficiency and performance, with our methods solving the CP
environment. With the methods using shared feature spaces, we do not use
distributional scalarization nor PCGrad. The mechanisms learned on fixed
temperature values on the penalty terms. The base representations are used for
both the mix and mask mechanisms.

Figure 4: CP (left) and nCP (right) Results: The figure shows the expected
return over $100$ independent trials over training. The training is stopped
once the task is considered solved or until $100$ episodes were reached.

However, on the nCP environment, the separated networks and shared backbone
networks were not able to fully solve the task within $100$ episodes, whereas
our proposed approach were able to. As the figure shows, all of networks
exhibit similar trends in the beginning, but we see improvements in only our
models. Now, we use the CP and nCP environment to evaluate on the variants and
additions we previously introduced in our ablative results.

###  5.2 Ablative Results

Figure 5: Training curve of Mix (left) and Mask (right) representations: The
figure shows the training results on the CP (dotted) and nCP (solid)
environments. The training is stopped once the task is considered solved or
until $100$ episodes were reached.

In this section, we evaluate different design choices for the mix and mask
actor-critic method and provide empirical results on the CP and nCP
environments. First, we test different representations of the mix and mask
mechanisms, seen in Figure 5. We find that the MLPMixer representation does
not converge within $100$ episodes, and exhibits a similar learning curve as
the separated network, while the other representations either outperforms or
performs comparatively to the base representations. When we compare the
training curve between the CP and nCP environments, we conclude that all the
networks with the mix and mask mechanisms were able to generalize
sufficiently.

On the other hand, we found, through experimentation, that parameterizing the
mix or mask functions as a probability distribution fails to converge often
with small changes in training hyperparameters, and remains sample-inefficient
even when the learning does converge. We notice minimal difference between
different probabilistic similarity term representations.

Figure 6: Temperature values on penalty terms: The figure shows the expected
return over $100$ independent trials over different temperature values on the
penalty terms for mix (left) and mask (right) mechanisms on the CP
environment.

Figure 7: Further Ablations on Mix-Mask Variants: The figures show the
following results: skip connections (top,left), auxiliary mechanism
(top,right), contrastive learning (bottom,left), and multi-objective handling
(bottom,right).

In Figure 6, we survey different temperature value on the penalty terms
imposed on the base mechanisms representations. The experimental results show
a need for a sufficiently higher temperature value for the mix mechanisms,
which was expected, and a noticeably relaxed requirement for the mask
mechanism. This is since, if there is no penalty term imposed on the mask
mechanism, the model is equivalent to the shared backbone network. We still
see improvements in sample-efficiency and stability once the temperature value
increases. However, at a certain point, we begin to see some degradation in
performance. This is likely the point at which the penalty term start to
overpower the policy and value function objectives. We encourage future work
to devise generalized approaches to finding optimal temperature values for the
penalty terms.

In Figure 7, we explore various design choices and evaluate on the classic
control tasks. For each approach, we again use the base representations of the
mix and mask mechanisms. We find that skip connections can boost sample-
efficiency, as seen obviously by the mix mechanism paired with a dense
connection. The performance of the mix mechanism with a residual connection
exhibits an interesting behavior as the nCP task is solved before the CP task,
which we can only attribute to the experimental noise. The figure also shows
that auxiliary mechanisms improves sample-efficiency from the base
representations, as does contrastive representation learning. In specific, we
see a huge improvement in the mask representation when contrastive learning is
employed. Lastly, the figure demonstrates the improvements from using the
multi-objective handling procedures: distribution scalarization and PCGrad.
From our results, we can see that the distribution scalarization technique
allows the networks to converge faster, however the training curve is less
stable than PCGrad, which was expected.

##  6 Conclusion

In this paper, we presented the mix and mask actor-critic method and the
distributional scalarization technique. We explored promising variants of the
base representations of our proposed algorithm, and conducted comparative
analysis with ablative studies on various design choices. We encourage future
work to extend the use of derived approaches onto different RL tasks and
explore different variants of these mechanisms.

## References

  * [1] Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans.  Understanding the impact of entropy on policy optimization, 2019.
  * [2] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba.  Hindsight experience replay, 2018.
  * [3] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang.  Dota 2 with large scale deep reinforcement learning.  CoRR, abs/1912.06680, 2019.
  * [4] Jianhui Chen, Lei Tang, Jun Liu, and Jieping Ye.  A convex formulation for learning shared structures from multiple tasks.  In Proceedings of the 26th Annual International Conference on Machine Learning, ICML ’09, page 137–144, New York, NY, USA, 2009. Association for Computing Machinery.
  * [5] Karl Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman.  Phasic policy gradient, 2020.
  * [6] Stephen Dankwa and Wenfeng Zheng.  Twin-delayed ddpg: A deep reinforcement learning technique to model a continuous movement of an intelligent robot agent.  In Proceedings of the 3rd International Conference on Vision, Image and Signal Processing, ICVISP 2019, New York, NY, USA, 2019. Association for Computing Machinery.
  * [7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * [8] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel.  Benchmarking deep reinforcement learning for continuous control.  CoRR, abs/1604.06778, 2016.
  * [9] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, 2018.
  * [10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition, 2015.
  * [11] Nicolas Heess, Greg Wayne, David Silver, Timothy P. Lillicrap, Yuval Tassa, and Tom Erez.  Learning continuous control policies by stochastic value gradients.  CoRR, abs/1510.09142, 2015.
  * [12] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger.  Densely connected convolutional networks, 2018.
  * [13] Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu.  Asynchronous methods for deep reinforcement learning.  CoRR, abs/1602.01783, 2016.
  * [14] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller.  Playing atari with deep reinforcement learning, 2013.
  * [15] David Perkins and Gavriel Salomon.  Transfer of learning.  11, 07 1999.
  * [16] Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojciech Zaremba, and Pieter Abbeel.  Asymmetric actor critic for image-based robot learning, 2017.
  * [17] L. Y. Pratt.  Discriminability-based transfer between neural networks.  In S. Hanson, J. Cowan, and C. Giles, editors, Advances in Neural Information Processing Systems, volume 5. Morgan-Kaufmann, 1993.
  * [18] Edmar Rezende, Guilherme Ruppert, Tiago Carvalho, Fabio Ramos, and Paulo De Geus.  Malicious software classification using transfer learning of resnet-50 deep neural network.  12 2017.
  * [19] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver.  Prioritized experience replay, 2016.
  * [20] John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel.  Trust region policy optimization, 2017.
  * [21] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel.  High-dimensional continuous control using generalized advantage estimation, 2018.
  * [22] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  CoRR, abs/1707.06347, 2017.
  * [23] R. Sibson.  Information radius.  Zeitschrift für Wahrscheinlichkeitstheorie und Verwandte Gebiete, 14:149–160, 1969.
  * [24] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller.  Deterministic policy gradient algorithms.  In Eric P. Xing and Tony Jebara, editors, Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages 387–395, Bejing, China, 22–24 Jun 2014. PMLR.
  * [25] Aravind Srinivas, Michael Laskin, and Pieter Abbeel.  Curl: Contrastive unsupervised representations for reinforcement learning, 2020.
  * [26] Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber.  Training very deep networks, 2015.
  * [27] Richard S. Sutton and Andrew G. Barto.  Reinforcement Learning: An Introduction.  A Bradford Book, Cambridge, MA, USA, 2018.
  * [28] Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour.  Policy gradient methods for reinforcement learning with function approximation.  In Proceedings of the 12th International Conference on Neural Information Processing Systems, NIPS’99, page 1057–1063, Cambridge, MA, USA, 1999. MIT Press.
  * [29] Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy.  Mlp-mixer: An all-mlp architecture for vision, 2021.
  * [30] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  CoRR, abs/1706.03762, 2017.
  * [31] Oriol Vinyals, Igor Babuschkin, Wojciech Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John Agapiou, Max Jaderberg, and David Silver.  Grandmaster level in starcraft ii using multi-agent reinforcement learning.  Nature, 575, 11 2019.
  * [32] Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Rémi Munos, Koray Kavukcuoglu, and Nando de Freitas.  Sample efficient actor-critic with experience replay.  CoRR, abs/1611.01224, 2016.
  * [33] Ronald J. Williams.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  Mach. Learn., 8(3–4):229–256, May 1992.
  * [34] Yuhuai Wu, Elman Mansimov, Shun Liao, Roger B. Grosse, and Jimmy Ba.  Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation.  CoRR, abs/1708.05144, 2017.
  * [35] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn.  Gradient surgery for multi-task learning, 2020.


# Separating the effects of earthside and far side solar events. A case study

Silja Pohjolainen Nasrin Talebpour Sheshvan Christian Monstein Tuorla
Observatory, Department of Physics and Astronomy, University of Turku, 20014
Turun yliopisto, Finland  Department of Physics and Astronomy, University of
Turku, 20014 Turun yliopisto, Finland  Istituto ricerche solari Aldo e Cele
Daccò (IRSOL), Faculty of Informatics, Università della Svizzera italiana
(USI), CH-6605 Locarno, Switzerland

###### Abstract

On 8 November 2013 a halo-type coronal mass ejection (CME) was observed,
together with flares and type II radio bursts, but the association between the
flares, radio bursts, and the CME was not clear. Our aim is to identify the
origin of the CME and its direction of propagation, and to exclude features
that were not connected to it. On the Earth-facing side, a GOES C5.7 class
flare occurred close to the estimated CME launch time, followed by an X1.1
class flare. The latter flare was associated with an EUV wave and metric type
II bursts. On the far side of the Sun, a filament eruption, EUV dimmings, and
ejected CME loops were observed by imaging instruments onboard the Solar
TErrestrial RElations Observatory (STEREO) spacecraft that were viewing the
backside of the Sun. The STEREO radio instruments observed an interplanetary
(IP) type II radio burst at decameter-hectometric wavelengths, which was not
observed by the radio instrument onboard the Wind spacecraft located at L1
near Earth. We show that the halo CME originated from the eruption on the far
side of the Sun, and that the IP type II burst was created by a shock wave
ahead of the halo CME. The radio burst remained unobserved from the earthside,
even at heliocentric source heights larger than 9 solar radii. During the CME
propagation, the X-class flare eruption caused a small plasmoid ejection
earthward, the material of which was superposed on the earlier CME structures
observed in projection. The estimated heights of the metric type II burst
match well with the EUV wave launched by the X-class flare. As this radio
emission did not continue to lower frequencies, we conclude that the shock
wave did not propagate any further. Either the shock driver died out, as a
blast wave, or the driver speed no longer exceeded the local Alfven speed.

###### keywords:

Solar eruptions , Coronal mass ejections , Solar radio bursts

††journal: Advances in Space Research

##  1 Introduction

Solar flares and eruptions are sometimes associated with propagating shock
waves and coronal mass ejections (CMEs). The flare intensities, durations,
eruption energetics, and mass sizes vary, and they do not necessarily depend
on each other (Kawabata et al., 2018; Kahler and Ling, 2022). Many events are
observed in projection, which makes it difficult to estimate source heights
(Kwon, Zhang, and Vourlidas, 2015; Balmaceda et al., 2018; Vourlidas et al.,
2020). Also separating far side events from Earth view events can be difficult
for halo CMEs, as we cannot conclude the propagation direction if we only see
a symmetrical halo surrounding the solar disc.

Flare-accelerated particles create emissions that can be observed especially
in hard X-rays and microwaves. Microwave emission is typically due to trapped
electrons gyrating in the closed magnetic field (gyrosynchrotron emission),
and hard X-rays are created when some of these electron populations collide
with dense material (often at the footpoints of flare loops). Plasma emission,
on the other hand, is created when a transient propagates through a medium and
causes oscillations that turn into radio waves at the local plasma frequency.
Reviews on the various aspects of solar radio emission associated with solar
events have been published by, e.g., Bastian, Benz, and Gary (1998), Nindos et
al. (2008), and Pick and Vilmer (2008).

Observations at plasma frequencies include type III bursts which are created
by electron beams that follow open magnetic field lines out from the Sun and
into the interplanetary (IP) space. Propagating shock waves can create type II
radio bursts, but the shock origin cannot always be determined (Fulara and
Kwon, 2021; Chernov and Fomichev, 2021). Flare (blast) waves, driven waves,
and bow shocks are all possible in connection with solar eruptions (Warmuth,
2007). Shock drivers can be fast CMEs or plasmoids, or fast expansion at the
CME flanks. For bow shocks, the transient speed must exceed the local Alfven
speed.

Radio emission at plasma frequency is useful in a sense that it can be used to
determine source heights. As plasma frequency depends only on the local
electron density, we can use coronal density models to obtain estimates of
distance from the solar surface. The conversion of radio frequency to
atmospheric height has been reviewed in, e.g., Pohjolainen et al. (2007).

The research questions addressed in the present analysis are: Can we
distinguish which parts of a halo CME are caused by far side eruption and
which parts belong to frontside eruption? Can we exclude one or the other,
based on observed flares and shock waves? Can radio emission help in
separating source origins? We present a case study of multiple emissions and
features that originated from active regions located on the earthside and the
far side of the Sun, with possible associations to a halo CME.

##  2 Data analysis

We have analysed solar eruptions that occurred on 8 November 2013. Eruptions
were observed on the earthside and the far side of the Sun, see Table 1. A
halo CME was observed during the investigated time period of 00:00–06:00 UT.

For the analysis we used solar EUV images provided by the Atmospheric Imaging
Assembly (AIA) (Lemen et al., 2012) onboard the Solar Dynamics Observatory
(SDO), and the Extreme Ultraviolet Imager (EUVI) onboard the STEREO A and B
spacecrafts (Wuelser et al., 2004). Coronagraph images and associated data
products were obtained from the CDAW LASCO CME Catalog at
https://cdaw.gsfc.nasa.gov that includes SOHO LASCO and STEREO A and B
coronagraph images. Full-disc soft X-ray flux were provided by GOES
satellites.

Radio data at decimeter-metric wavelengths include dynamic spectra from
several e-CALLISTO Network stations (Benz, Monstein, and Meyer, 2005),
https://www.e-callisto.org/, and from the Radio Solar Telescope Network (RSTN)
Learmonth station, https://www.ngdc.noaa.gov/stp/space-weather/solar-
data/solar-features/solar-radio/rstn-spectral. For interplanetary radio
emission at decameter-hectometric waves we used dynamic spectra from WAVES
onboard the Wind satellite (Bougeret et al., 1995), and SWAVES spectra from
the STEREO A and B satellites (Bougeret et al., 2008). These can be accessed
in the CDAW LASCO CME Catalog. Microwave single frequency data were obtained
from the Nobeyama radio polarimeters (NoRP) at
http://solar.nro.nao.ac.jp/norp/.

Table 1: Flare/eruption times, locations, and intensities on 8 November 2013 for the time period of 00:00–06:00 UT. Two flares were listed to occur on the visible solar disc and a filament eruption was observed on the far side of the Sun. End times have been estimated from the GOES flux curves (not as in the NOAA flare listings). Only one CME was listed for this time period.  Start | End | Location | GOES class/
---|---|---|---
time | time |  | or type
02:25 UT |  | S20W180 | Filament eruption
|  |  | southward
02:33 UT | 03:10 UT | S18W25 | C5.7
04:20 UT | 04:45 UT | S14E15 | X1.1
Fig. 1: GOES soft X-ray flux curve shows the C5.7 and X1.1 class flares.

Fig. 2: Top: GOES X-rays from the X1.1 class flare. Middle: Estimated
heliocentric distances/heights for the EUV wave (circles), and the metric type
II burst (crosses) observed by e-CALLISTO MRT3 at 450–50 MHz and RSTN
Learmonth at 180–25 MHz. The calculated radio source heights are listed in
Table 2. Bottom three plots: Radio flux densities from the Nobeyama
polarimeters at 17, 35, and 80 GHz, in the flare impulsive phase.  Fig. 3:
SDO/AIA 193 Å difference images that show the evolution of the EUV wave that
followed the GOES X1.1 class flare located at S14E15.

###  2.1 Flares and eruptions

The first flare observed by GOES on 8 November 2013 is GOES class C5.7, start
time at 02:33 UT, and duration of approximately 37 minutes, see Fig. 1. The
flare is reported to have occurred at location S18W25 in NOAA active region
(AR) 11891, and a corresponding small-scale loop brightening can be observed
in the SDO/AIA images.

It is followed by a GOES X1.1 class flare in another region, in AR 11890 at
location S14E15. The flare is reported to have a start time at 04:20 UT, flux
peak time at 04:26 UT, and we estimated that the flare duration was
approximately 25 minutes (Fig. 1).

This X-class flare was well-observed at microwaves and mm-waves, the peak flux
reaching 4000 sfu at 17–35 GHz, and 2200 sfu at 80 GHz (Fig. 2). The flux
began to rise at 04:23:40 UT, and multiple flux peaks were visible until
04:25:12 UT. Unfortunately no hard X-ray observations are available for this
flare, but the high flux density at 80 GHz indicates particle acceleration to
high energies.

Starting at 04:24 UT, decimeter-metric radio emission was observed by several
ground-based stations. These emission structures are described in more detail
in Section 2.2

Fig. 4: Dimming regions and the launch of a CME on the far side of the Sun
(STEREO A and B observations). The X-class flare on the earthside originated
from an active region at E15, the C-class flare was a brightening at W25, and
the far side eruption originated from a region near W180. Cartoon shows the
locations of these eruptions, together with STEREO A and B positions.
SOHO/LASCO and SDO have the same observing direction as Wind. The earthside
flaring active regions at low heights were unobservable from STEREO A and B
EUV imagers.

The X-class flare was associated with a bubble-like EUV wave. The direction of
the wave was southward from the AR, see Fig. 3. We measured the wave distances
(dashed lines shown in Fig. 3) from the AR center at S14E15, assuming that the
distance on the disc compares to the wave height. The distance increased from
0.333 R⊙ at 04:26 UT to 0.489 R⊙ at 04:28 UT, indicating a speed of
approximately 900 km s-1. In Fig. 2 the distances are shown in heliocentric
heights, for comparison purposes.

On the far side of the Sun a filament eruption was observed at 02:25 UT. The
most visible part of the filament was already at some distance from the AR
location at S20W180, and it was directed toward south. STEREO A and B EUVI
images show several dimming regions in the same direction (Fig. 4). Later on,
dimmings appeared also in the location of the AR loops.

Potential field source surface (pfss) magnetic field lines were calculated
using synoptic maps, as on the far side we have no direct magnetic field
measurements. The field lines are shown both for Earth view and far side views
in Fig. 5. They show that open field lines exist on the south-east side of AR
11890, but all other eruption regions lack open field lines in their vicinity.
Flare-accelerated electrons would need open field lines to stream out from the
Sun. Lack of type III bursts, especially in the interplanetary space, would
also suggest that particles did not have access to any. For connectivity to
observing particle detectors, it is useful to know where the open field lines
are directed.

We note that the flaring regions AR 11890 and AR 11891 on the disc could not
have been observed from STEREO A and B spacecraft, as they were located behind
the limb in their field of views, see cartoon in Fig. 4.

Fig. 5: Magnetic field lines calculated with the potential field source
surface (pfss) model, shown in STEREO B, SDO, and STEREO A view. Active
regions 11890 and 11891 are marked in the SDO view map, and white arrows mark
the erupting region on the far side, near S20W180, in STEREO A and B views.
Closed field lines are shown in white and open field lines in red and blue
(opposite polarities).

###  2.2 Radio emission

Decimeter-metric radio emission started as narrow-width type III burst
emission at $\approx$ 600 MHz at 04:24 UT. At 04:25 UT a clear type II burst
emission lane became visible at 160 MHz, and it could be followed down to 30
MHz (e-CALLISTO and RSTN Learmonth dynamic spectra), see Fig. 7. The burst
showed fundamental and harmonic emission lanes that were also band-split. The
real start time of the burst is unclear, due to the overlap with other
emission structures.

The metric type II burst did not continue to IP space (note that there is a
data gap between 25 and 15 MHz, and therefore the end time and frequency
cannot be determined). The disappearance of the type II burst may indicate
that the shock wave did not propagate any further. For example, a blast wave
shock would eventually loose it’s energy and stop. For a bow shock a rise in
the local Alfven speed would make the shock die out if the transient speed no
longer exceeded the local magnetosonic speed (note that the transient could
still continue to move out at the same speed). To identify shock origin or
shock driver is difficult without knowing the local plasma conditions, and
without radio imaging at these wavelengths.

We estimated the metric type II burst heights from the e-CALLISTO MRT3
(Mauritius Radio Telescope, antenna number 3) and RSTN Learmonth dynamic
spectra using the ’hybrid’ coronal density model of Vrs̆nak, Magdalenic̆, and
Zlobec (2004). The derived instantaneous type II shock speeds were in the
range of 670–980 km s-1. As we do not know the local coronal densities but
rely on the density model, this can be taken as an approximation of the shock
speed. However, the radio emission heights and speeds are not far away from
those of the EUV wave observed during the X-class flare. The metric type II
burst heights are shown in Fig. 2 (cross symbols in the bottom plot), with the
EUV wave distance/heights (circles). Type II burst frequencies and
corresponding heights are listed in Table 2.

Fig. 6: Solar-MACH plot (prepared with the Solar MAgnetic Connection HAUS
tool) shows the field line connecting to the reference longitude of the X1.1
class flare located at S14E15 (indicated with the dashed line, top).
SOHO/LASCO C2 observation of the CME at 05:24 UT (middle). A separate plasmoid
is visible, indicated with the green arrow. SOHO/LASCO C3 observation of the
CME at 06:30 UT (bottom). During 06:00–07:00 UT an IP type II burst was
observed by STEREO A and B, but not by Wind.

Fig. 7: Tracing the type II lanes in the e-CALLISTO MRT3 dynamic spectrum in
the frequency range of 450–50 MHz (top) and in the RSTN Learmonth spectrum at
180–25 MHz (bottom). The fundamental (F) and harmonic (H) lanes are band-split
(indicated with dashed and dotted lines in the Learmonth spectrum,
respectively). We used the fundamental, lower band-split lane in the height
calculations. The times and heights are listed in Table 2 and they are shown
in Figs. 2 and 9. The type II emission was preceded by type III bursts, which
makes it difficult to define the exact type II burst start time.  Fig. 8:
STEREO A/SWAVES (location A in red), Wind/WAVES (location in green), and
STEREO B/SWAVES (location B in blue) dynamic radio spectra from $\approx$ 10
MHz down to 10 kHz. The visible IP type II burst lanes are indicated with
arrows.

###  2.3 Coronal mass ejections and particles

On the far side of the Sun both STEREO COR1 coronagraphs observed a loop-
shaped mass ejection at 03:05 UT (Fig. 4). The eruption that started at 02:25
UT, near location S20W180, and the following EUV dimmings match well with the
CME shape and direction of propagation.

On the earthside a CME was first observed by SOHO LASCO at 03:24 UT, at height
2.7 R⊙. The CME loop became fully visible in the next available LASCO C2 image
at 03:47 UT. The CME speed, determined from LASCO observations, increased from
350 km s-1 to 700 km s-1 during the first seven hours of propagation (CDAW
Catalog, 2nd order fit to the CME leading front heights).

A separate plasmoid structure was observed inside the CME loop at 05:12 UT,
and it’s front was fully visible in the difference image at 05:24 UT, see Fig.
6 (the front is indicated with a green arrow). The speed of this structure was
estimated to be 400 km s-1. After 05:48 UT it could not be distinguished from
the bright main loop any longer.

The CME and plasmoid heights are shown in Fig. 9, where they are presented
together with the radio source heights that are discussed in Section 2.2.

We checked in-situ particle measurements from both STEREO spacecraft and Wind.
A high-speed halo CME that was launched the day before, near 10 UT on 7
November and originating also from the far side, caused an increase in proton
and electron intensities in STEREO A and B particle detectors. The intensities
remained high also on 8 November, making it difficult to separate later
enhancements. The 7 November flare-CME-SEP event is described in detail in
Dresing et al. (2016).

Table 2: Estimated type II burst heights. We used meter-wave observations from e-CALLISTO MRT3 and RSTN Learmonth, and DH-wave observations are from STEREO A and B SWAVES. Heights are heliocentric and they are calculated from emission at the fundamental plasma frequency, see text for details.  Time | Frequency | Height
---|---|---
04:25:30 UT | 163 MHz | 1.16 R⊙
04:25:56 UT | 147 MHz | 1.18 R⊙
04:26:14 UT | 135 MHz | 1.21 R⊙
04:26:55 UT | 111 MHz | 1.26 R⊙
04:27:54 UT | 83 MHz | 1.37 R⊙
04:28:33 UT | 78.1 MHz | 1.40 R⊙
04:29:24 UT | 70.0 MHz | 1.45 R⊙
04:30:41 UT | 54.9 MHz | 1.58 R⊙
04:33:01 UT | 44.2 MHz | 1.72 R⊙
04:35:00 UT | 36.2 MHz | 1.86 R⊙
06:00 UT | 1.22 MHz | 9.20 R⊙
06:10 UT | 1.12 MHz | 9.65 R⊙
06:20 UT | 1.00 MHz | 10.36 R⊙
06:30 UT | 1.02 MHz | 10.16 R⊙
06:45 UT | 930 kHz | 10.71 R⊙
06:55 UT | 815 kHz | 11.56 R⊙
07:02 UT | 790 kHz | 11.77 R⊙

In the interplanetary space, below 15 MHz frequencies, very few type III
bursts are observed. Dynamic spectrum from STEREO A shows a type III burst at
03:04 UT, first appearing at 2.5 MHz (height $\approx$ 6 R⊙), and Wind/WAVES
shows a very faint type III burst appearing at 03:45 UT at 4 MHz. This
indicates that despite the flaring activity and CME propagation, few
accelerated particles had access to open field lines. The Solar-MACH plot in
Fig. 6 shows how the field line from the X-class flare location is not
connected to any of the spacecraft, see Gieseler et al. (2023) on how the
visualization of the spatial configuration and the solar magnetic connection
to different observers is done.

An IP type II burst was observed by STEREO A and B, appearing at 1.2 MHz
frequency at 06:00 UT. This type II burst was not observed by Wind (located in
L1 on the Earth-facing side), see Fig. 8. The type II burst emission lane
could be followed down to 700 kHz, where the emission ended around 07:00 UT.
The calculated source heights, using the fundamental emission lane and the
same ’hybrid’ density model, are listed in Table 2.

The radio source heights, with all other feature heights we have identified
and analysed, are plotted in Fig. 9 for comparison. From this plot it is
evident that the IP type II burst heights correspond to the CME leading front
heights observed from the far side, and that the earthside LASCO observations
of the CME fronts have lower heights. Far side CME heights in Fig. 9 are from
STEREO B, as STEREO A observations show a wider CME structure, almost a double
loop, with several fronts. However, the heights of the leading parts are
similar to the STEREO B observations. LASCO height measurements most probably
suffer from projection effects, as the halo CME originated from almost exactly
the opposite side of the Sun. This may well explain the lower heights.

The height-time evolution indicates that the metric type II burst and the IP
type II burst are not connected, but were created by separate propagating
shock waves.

Fig. 9: Measured and calculated heights of all propagating structures. For
flares and filament eruption we show their start times.

##  3 Summary and conclusions

We have analysed features and emissions that could have been associated with a
halo CME on 8 November 2013. We wanted to identify the origin of the CME, and
to exclude features that were not connected to it. As eruptions were observed
both on the earthside and on the far side of the Sun, detailed analysis was
needed. In halo CMEs the bright coronal emission is observed in projection
around the solar disk, and from coronagraph images only it is impossible to
conclude if the CME is launched toward Earth or in the opposite direction.

After investigating the eruptions and flares, some followed by EUV waves and
dimmings, and estimating the type II radio burst heights, we conclude that the
halo CME originated from the eruption on the far side of the Sun. The IP type
II burst was created by a shock wave (bow shock) ahead of the halo CME, but
the radio burst location was invisible to Earth. The most probable reason for
not observing it would be dense plasma structures blocking wave propagation.
An example of this kind of configuration is presented in Fig. 11 in Talebpour
Sheshvan and Pohjolainen (2018). If the radio source was located in a region
directed towards STEREO A and B, also the solar disc, or the CME itself, could
have blocked radio emission toward the Earth.

It is well known that type II bursts are typically formed in narrow regions
somewhere along the shock front, where plasma conditions are favourable. The
relatively short duration and narrow band of the IP type II burst, and late
appearance after the CME launch, suggest that the radio emission could have
been caused by interaction with a small-volume ambient plasma structure.

During the CME propagation, the X-class flare eruption on the earthside of the
Sun caused a small plasmoid ejection, the material of which was superposed on
the halo CME features. The estimated heights of the metric type II burst match
well with the EUV wave launched by the X-class flare, and the height-time
trajectory of the plasmoid suggests they could be associated. As the type II
burst emission did not continue to IP space, we conclude that either the shock
died out (as a blast wave), or an increase in the local Alfven speed made it
impossible to propagate further out.

##  4 Acknowledgments

We thank the e-CALLISTO community for free access to their data and software.
MRT3 is located and operated at University of Mauritius, Poste de Flacq,
Mauritius. The Radio Solar Telescope Network (RSTN) is a network of solar
observatories maintained and operated by the US Air Force. Learmonth is
located in Australia and their data is included in Space Weather databases.
Nobeyama Radio Polarimeters (NoRP) are operated by Solar Science Observatory,
a branch of National Astronomical Observatory of Japan. The CME Catalog is
generated and maintained at the CDAW Data Center by NASA and the Catholic
University of America in cooperation with the Naval Research Laboratory.

## References

  * Balmaceda et al. (2018) Balmaceda, L.A., Vourlidas, A., Stenborg, G., Dal Lago, A., 2018. How Reliable Are the Properties of Coronal Mass Ejections Measured from a Single Viewpoint? Astrophys. J. 863, id. 57. https://doi.org/10.3847/1538-4357/aacff8/
  * Bastian, Benz, and Gary (1998) Bastian, T.S., Benz, A.O., Gary, D.E., 1998. Radio Emission from Solar Flares. Ann. Rev. Astron. Astrophys, 36, 131-188. https://doi.org/10.1146/annurev.astro.36.1.131
  * Benz, Monstein, and Meyer (2005) Benz, A.O., Monstein, C., Meyer, H., 2005. Callisto A New Concept for Solar Radio Spectrometers. Solar Phys. 226, 143-151. https://doi.org/10.1007/s11207-005-5688-9/
  * Bougeret et al. (2008) Bougeret, J.-L., Goetz, K., Kaiser, M.L., Bale, S.D., Kellogg, P.J., Maksimovic, M., et al., 2008. S/WAVES: The Radio and Plasma Wave Investigation on the STEREO Mission. Space Sci. Rev. 136, 487-528. https://doi.org/10.1007/s11214-007-9298-8
  * Bougeret et al. (1995) Bougeret, J.-L., Kaiser, M.L., Kellogg, P.J., Manning, R., Goetz, K., Monson, S.J., Monge, N., Friel, L., Meetre, C. A., Perche, C., Sitruk, L., Hoang, S., 1995. WAVES: The radio and plasma wave investigation on the wind spacecraft. Space Sci. Rev. 71, 231-263. https://doi.org/10.1007/BF00751331
  * Chernov and Fomichev (2021) Chernov, G., Fomichev, V., 2021. On the Issue of the Origin of Type II Solar Radio Bursts. Astrophys. J. 922, id.82. https://doi.org/10.3847/1538-4357/ac1f32
  * Dresing et al. (2016) Dresing, N., Gómez-Herrero, R., Heber, B., Hidalgo, M.A., Klassen, A., Temmer, M., Veronig, A., 2016. Injection of solar energetic particles into both loop legs of a magnetic cloud. Astron. Astrophys. 586, id. A55. https://doi.org/10.1051/0004-6361/201527347
  * Gieseler et al. (2023) Gieseler, J., Dresing, N., Palmroos, C., Freiherr von Forstner, J.L., Price, D.J., Vainio, R., et al., 2022. Solar-MACH: An open-source tool to analyze solar magnetic connection configurations. Front. Astronomy Space Sci. 9, id. 384. https://doi:10.3389/fspas.2022.1058810
  * Fulara and Kwon (2021) Fulara, A., Kwon, R-Y., 2021. Global Nature of Solar Coronal Shock Waves Shown by Inconsistency between EUV Waves and Type II Radio Bursts. Aastrophys. J. Lett. 919, id.L7. https://doi.org/10.3847/2041-8213/ac230d
  * Kahler and Ling (2022) Kahler, S.W., Ling, A.G., 2022. A Comparison of Solar X-Ray Flare Timescales and Peak Temperatures with Associated Coronal Mass Ejections. Astrophys. J. 934, id.175. https://doi.org/10.3847/1538-4357/ac7e56
  * Kawabata et al. (2018) Kawabata, Y., Iida, Y., Doi, T., Akiyama, S., Yashiro, S., Shimizu, T., 2018. Statistical Relation between Solar Flares and Coronal Mass Ejections with Respect to Sigmoidal Structures in Active Regions. Astrophys. J. 869, id. 99. https://doi.org/10.3847/1538-4357/aaebfc
  * Kwon, Zhang, and Vourlidas (2015) Kwon, R-Y., Zhang, J., Vourlidas, A., 2015. Are Halo-like Solar Coronal Mass Ejections Merely a Matter of Geometric Projection Effects? Astrophys. J. Lett. 799, id. L29. https://doi.org/10.1088/2041-8205/799/2/L29
  * Lemen et al. (2012) Lemen, J.R., Title, A.M., Akin, D.J., Boerner, P.F., Chou, C., Drake, J.F., Duncan, D.W., Edwards, C.G., Friedlaender, F.M., Heyman, G. F., et al., 2012. The Atmospheric Imaging Assembly (AIA) on the Solar Dynamics Observatory (SDO). Solar Phys. 275, 17-40. https://doi.org/10.1007/s11207-011-9776-8
  * Nindos et al. (2008) Nindos, A., Aurass, H., Klein, K.-L., Trottet, G., 2008. Radio Emission of Flares and Coronal Mass Ejections. Invited Review. Solar Phys. 253, 3-41. https://doi.org/10.1007/s11207-008-9258-9
  * Pick and Vilmer (2008) Pick, M., Vilmer, N., 2008. Sixty-five years of solar radioastronomy: flares, coronal mass ejections and Sun Earth connection. Astron. Astrophys. Rev. 16, 1-153. https://doi.org/10.1007/s00159-008-0013-x
  * Pohjolainen et al. (2007) Pohjolainen, S., van Driel-Gesztelyi, L., Culhane, et al., 2007. CME Propagation Characteristics from Radio Observations. Solar Phys. 244, 167-188. https://doi.org/10.1007/s11207-007-9006-6
  * Talebpour Sheshvan and Pohjolainen (2018) Talebpour Sheshvan, N., Pohjolainen, S., 2018. Visibility and Origin of Compact Interplanetary Radio Type IV Bursts. Solar Phys. 293, id. 148. http://doi.org/10.1007/s11207-018-1371-9
  * Vourlidas et al. (2020) Vourlidas, A., Balmaceda, L.A., Xie, H., St. Cyr, O.C., 2020. The Coronal Mass Ejection Visibility Function of Modern Coronagraphs. Astrophys. J. 900, id. 161. http://doi.org/10.3847/1538-4357/abada5
  * Vrs̆nak, Magdalenic̆, and Zlobec (2004) Vrs̆nak, B., Magdalenic̆, J., Zlobec, P., 2004. Band-splitting of coronal and interplanetary type II bursts. III. Physical conditions in the upper corona and interplanetary space. Astron. Astrophys. 413, 753-763. http://doi.org/10.1051/0004-6361:20034060
  * Warmuth (2007) Warmuth, A., 2007. Large-scale Waves and Shocks in the Solar Corona. In KL. Klein and A.L. MacKinnon (eds.), The High Energy Solar Corona: Waves, Eruptions, Particles. Lect. Notes Phys. 725, 107-138. https://doi.org/10.1007/978-3-540-71570-2_6
  * Wuelser et al. (2004) Wuelser, J.-P., Lemen, J.R., Tarbell, T.D., Wolfson, C.J., Cannon, J.C., Carpenter, B.A., Duncan, D.W., Gradwohl, G.S., Meyer, S.B., Moore, A.S., et al., 2004. EUVI: the STEREO-SECCHI extreme ultraviolet imager, in S. Fineschi and M.A. Gummin (eds.), Telescopes and Instrumentation for Solar Astrophysics. Proc. SPIE 5171, 111-122. https://doi.org/10.1117/12.506877