# Adversarial Evasion Attacks Practicality in Networks: Testing the Impact of
Dynamic Learning
Authors’ draft for soliciting feedback: June 8, 2023

Mohamed el Shehaby  Systems and Computer Engineering
Carleton University
Ottawa, Canada
<EMAIL_ADDRESS>Ashraf Matrawy  School of Information
Technology
Carleton University
Ottawa, Canada
<EMAIL_ADDRESS>
###### Abstract

Machine Learning (ML) has become ubiquitous, and its deployment in Network
Intrusion Detection Systems (NIDS) is inevitable due to its automated nature
and high accuracy in processing and classifying large volumes of data.
However, ML has been found to have several flaws, on top of them are
adversarial attacks, which aim to trick ML models into producing faulty
predictions. While most adversarial attack research focuses on computer vision
datasets, recent studies have explored the practicality of such attacks
against ML-based network security entities, especially NIDS.

This paper presents two distinct contributions: a taxonomy of practicality
issues associated with adversarial attacks against ML-based NIDS and an
investigation of the impact of continuous training on adversarial attacks
against NIDS. Our experiments indicate that continuous re-training, even
without adversarial training, can reduce the effect of adversarial attacks.
While adversarial attacks can harm ML-based NIDSs, our aim is to highlight
that there is a significant gap between research and real-world practicality
in this domain which requires attention.

###### Index Terms:

machine learning, continuous machine learning, stream learning, network
security, intrusion detection systems, adversarial attacks

##  I Introduction

Machine Learning (ML) applications are present in every corner. ML found its
way to network security applications due to its (1) automated nature and (2)
ability to process and classify high volumes of data with high accuracy. ML
deployment in Network Intrusion Detection Systems (NIDS) is a foregone
conclusion. However, in recent years, researchers highlighted that ML has a
lot of drawbacks and flaws. To name a few, ML has privacy concerns [1] and
suffers from reproducibility crises (when the results of a model can not be
replicated using the exact code and dataset) [2]. Nevertheless, one ML
challenge shined the most in the research community; adversarial attacks. In a
nutshell, these adversarial attacks aim to trick ML models, making them
produce faulty predictions. There is no doubt that these attacks are a menace
to ML. However, it was observed that the majority of adversarial attack
research uses computer vision datasets to evaluate its severity [3]. Thus,
recently, a few researchers [4] [5] have explored the practicality of
adversarial attacks against ML-based network security entities, especially
Network Intrusion Detection Systems (NIDS), and implied that attacking them is
much more complicated than computer vision.

Contributions: In this paper, we make two distinct contributions. First, we
extend the research discussion of the practicality of adversarial attacks
against NIDS by introducing a taxonomy of the practicality issues of
adversarial attacks against ML-based NIDS. In this taxonomy, we include issues
reported in the literature and add our own questions and observations.

The second contribution emerged from our observation that the overwhelming
bulk of adversarial attack research deals with ML approaches as static models.
However, nowadays, numerous ML models are dynamic and constantly evolving.
Approaches like stream learning [6] or online machine learning [7] constantly
retrain their models to make them adaptive to new cases and data (prevent
concept drifts). Therefore, we explore and test the effect of continuous
training on adversarial attacks against NIDS. To the best of our knowledge,
these are the first experiments on the impact of continuous training on such
invasions against ML-based NIDS.

The rest of the paper is structured as follows: the second section explores
related background. In Section III, we review previous work and surveys. In
Section IV, we introduce the taxonomy of the practicality challenges and
issues of adversarial attacks against ML-NIDS. In Section V, we present our
investigation environment and methodology. In Section VI, we present our
results that show the effect of continuous training on adversarial attacks in
the NIDS space. Finally, in Section VII, we conclude our work and provide some
discussions and lessons learned.

##  II Background

###  II-A ML-based NIDS

Network intrusion detection systems are tools designed to identify threats
against a network. As seen in figure 1, NIDS sends alerts to network
management in case of breaches. Thus, it is classified as a passive detection
system. In other words, unlike Intrusion Prevention System (IPS), NIDS doesn’t
prevent intrusions, it simply warns security teams in the case of intrusion
detection. On the other hand, the IPS, which is normally located between the
firewall and the switches, can control and stop network hazards.

Figure 1: The Deployment of Network Intrusion Detection System

There are two main network intrusion detection techniques; Signature-based
intrusion detection (SIDS) and Anomaly detection-based intrusion detection
(AIDS). Most attacks follow well-defined patterns and signatures, thus, SIDS
catches the threats by detecting the predetermined attack patterns
(signatures). An example of a pattern is the count of zeros and ones in the
traffic bits. SIDS has a low false positive rate in an environment of well-
defined attacks. However, its main disadvantage is that it cannot detect newer
or unknown invasions [8]. On the other hand, AIDS catches attacks by detecting
deviations from normal activity. This approach makes AIDS able to detect
unknown attacks. Yet, because of the difficulty of finding the boundary
between normal and abnormal behavior, AIDS can have a high false positive rate
[9]. Therefore, to solve the accuracy issues in AIDS, researchers and security
experts have explored the use of machine learning approaches like Support
Vector Machine (SVM), Decision Tree (DT), and Artificial Neural Network (ANN)
[10]. Although ML can quickly classify a huge amount of data with high
accuracy, some researchers highlighted that ML can be vulnerable to
adversarial attacks [1], which will be explored later in this section.

###  II-B Dynamic Machine Learning

Concept drift in machine learning is the unforeseeable statistical changes in
the prediction target’s properties [11]. In other words, the norm changes over
time, so the ML model makes wrong predictions. In an adversary environment,
where attackers change their techniques all the time, like the
network/cybersecurity domain, concept drifts will more than likely occur [6].
Thus, to counter these drifts ML models must be dynamic, i.e, continuously
evolving. Numerous terms are used to describe dynamic ML models (e.g., Online
ML, Evolving ML, Continuous/Continual ML, and Stream ML). In the ML community,
the implementation and definition of these kinds of ML techniques could differ
from one study to another. However, the following concept is common; _"The
initial training set loses its validity as time passes due to changes of
conditions in the aimed task. Thus, a mechanism to update a given model in
order to adapt to new conditions is necessary"_ [7]. In simple terms, dynamic
ML models must be updated and re-trained all the time to resist the effect of
concept drift. There are different techniques for re-training (or continuous
training); for example, adaptive windowing, where the model is re-trained when
a concept drift is detected [12]. However, in practice, there are more
elementary solutions, with less overhead on the system, like periodically re-
training the model with recent data. In our tests, we use the latter approach.

###  II-C Adversarial Attacks Against Network Security and Defenses

In 2013, Szegedy et al.[13] discovered that several machine learning models
are vulnerable to adversarial examples. These examples are extremely close to
accurately classified samples. However, they are misclassified by ML models.
Soon after, Goodfellow et al. [14] managed to harness adversarial examples to
create the first adversarial attack, FGSM (Fast Gradient Sign Method), to
trick neural networks. FGSM creates adversarial examples by adding an
unnoticeably small vector (perturbations) to a correctly classified image. The
components of the perturbation are equal to the sign of the elements of the
gradient of the cost function to the input. FGSM’s adversarial examples were
able to make ML models make the wrong classifications with high confidence.
Since then, unfortunately, adversarial attacks became more effective and
complex (more targets, goals, and strategies) with less knowledge of the
target ML model needed by the attacker. In this section, we explore the
adversarial attacks in network security. Then we explore some general
defenses.

####  II-C1 Taxonomy of Adversarial Attacks in Network Security

Several researchers have taxonomized adversarial attacks in the network and
cybersecurity [15, 16]. Here are some terms used to classify adversarial
attacks in network security.

##### Attackers’ Knowledge

In White-box Attacks, the attackers have complete knowledge of the attacked ML
model (which ML algorithm was used, gradient, hyperparameters, etc.). The
complete opposite is Black-box Attacks, where the attackers have no knowledge
of the attacked ML model. However, in most cases, researchers assume that the
attackers crafting black-box attacks have access to the target ML system’s
output. Thus, the attackers can create a substitute model (pretty close to the
attacked model) by querying the targeted system. Then they use that substitute
system to get the knowledge they need to attack. Grey-box Attacks lie between
them (the attackers have "some" knowledge of the attacked ML system).

##### Attackers’ Approaches

In Evasion Attacks, the attackers aim to deceive the ML model at the decision
time. Like [17], where a perturbation is added to the features to make the IoT
ML intrusion detection model output the wrong classification. As for Poisoning
Attacks, the attackers add malicious samples to the training datasets (dataset
contamination). Like [18], where a data poisoning attack against a federated
learning-based IoT intrusion detection system allowed the adversaries to
implant a backdoor into the ML detection model to wrongly classify malicious
traffic as benign. In Model Stealing Attacks, an attacker can probe a black
box model to extract all the vital information to create a "clone" model very
close to the target model. It is worth mentioning that ML models in a
monumental system like NIDS are very expensive, and requires a lot of
resources to create. Thus, stealing them will be a massive loss to the
associations that created them. The main focus of this paper is evasion
attacks.

##### Attack’s Space

Okunle Ibitoye et al. [15] presented the space term in adversarial attacks
taxonomy in the network security domain. In Feature-space attacks, the
attacker only modifies (by adding perturbations) the ML model input feature
vector to craft the attack. As for the Problem-space attacks, the attacker
modifies the actual file (malware, pcap, PE (Portable Executable), etc.) to
trick the attacked model.

####  II-C2 Defenses Against Adversarial Attacks

There are multiple approaches to defend ML models against adversarial attacks.
The following are some of them.

Gradient Masking: Hiding the model’s gradients by minimizing them during the
learning phase. In other words, making the model less sensitive. As we
mentioned in the last section, attackers usually use the gradients to craft
the perturbations used to fool the ML models. Thus, masking the gradients
might make the ML systems less vulnerable to adversarial attacks. However,
researchers [19] have proved that this defense approach might be weak in some
cases.

Adversarial Training: Inserting adversarial examples in the training dataset
to make the model more robust against them [20] [21].

Feature Reduction: Reducing the feature sets might make the perturbations less
effective, making the ML classifier less vulnerable to evasion attacks. This
technique showed adequate results in ML spam and malware detection models
[22].

Detecting adversarial examples in evasion attacks: There are several
approaches used to detect evasion attacks. For example, Kathrin Grosse et al.
[23] showed that adversarial examples are not drawn from the exact
distribution of the original data. Thus, they can be detected using
statistical tests.

Detecting poisoning samples: The most logical way to defend against poisoning
attacks is to detect and discard these malicious samples in the training
dataset before learning, so they can’t poison the model. The adversarial
training examples can be considered as anomalies or outliers, and using
anomaly detection techniques could detect them [24].

Although these defenses are not specific to Network security, they are
applicable in network protection.

##  III Related Work

In this section, we review previous related work and classify them into two
groups; adversarial attacks against ML-NIDS, and the practicality of
adversarial attacks against ML-NIDS.

###  III-A Adversarial attacks against ML-NIDS

Numerous researchers have examined adversarial attacks against machine
learning-based network intrusion detection systems. Alatwi et al. [25] who
surveyed and categorized the studies of adversarial attacks against ML-based
NIDSs and briefly discussed their applicability in real-world scenarios.
Martins et al. [26] has also examined adversarial machine learning against
intrusion and malware detection systems. They concluded that a wide variety of
the reviewed attacks could be effective, however, their practicality was not
been tested enough in intrusion scenarios. Furthermore, They have stated that
some of the datasets in intrusion scenarios are outdated, and most of them are
similar and lack variety.

###  III-B Practicality of Adversarial attacks against ML-NIDS

Some studies have focused on the practicality of adversarial attacks in the
network intrusion detection domain; Apruzzese et al. [4] attempted to specify
and model the abilities and circumstances needed by attackers to carry out
practical adversarial attacks. Furthermore, in different work, Apruzzese et
al. [3] emphasized the presence of a gap between researchers and practitioners
in the adversarial attacks domain. Based on a collaboration between industry
and academia, they made the following position: "Real attackers don’t compute
gradients". In other words, they couldn’t find an attacker that calculates
gradients to craft an adversarial attack.

Merzouk et al. [5] attempted to inspect the practicality of adversarial
attacks to evade network intrusion detection systems. They highlighted some
limitations of adversarial attacks against ML-NIDS. However, they have only
concentrated on the feature space attacks.

To the best of our knowledge, none of the previous work has explored and
tested the practicality of adversarial attacks against dynamic ML models in
the intrusion detection domain.

for tree= draw, text centered [ Practicality Issues of Adversarial
Attacks Against ML-NIDS [ Black Box Feasibility [Basic Knowledge][Querying
NIDSs]] [ Feature Space
Perturbations] [ Problem Space
Perturbations [Network Functionality][Malicious Functionality]] [ Adaptivity
to
Dynamic ML]] ]

Figure 2: Taxonomy of Practicality Isues of Adversarial Attacks Against ML-
NIDS

##  IV Taxonomy of Practicality Issues of Adversarial Attacks Against ML-NIDS

The majority of the adversarial attacks were designed and tested in the
literature using computer vision datasets [3]. Because of the focus on the
computer vision datasets, it is not clear if these attacks will be applicable
in the same way to the network/cybersecurity domain which has more adversaries
and more security constraints and measures, making it much more complicated to
evade.

So, in this section, we explore the issues and challenges of adversarial
attacks against ML-NIDS and present them in a taxonomy illustrated in figure 2
to highlight the gap between the research and real-world scenarios.

###  IV-A Black-Box Feasibility

As mentioned in section II, attackers in black-box settings have no knowledge
of the target ML model. There is no doubt that black-box evasions are the more
practical type of adversarial attack [25] as obtaining all the needed
information about the target model is extremely difficult in the network
security domain. In the next sub-section, we will explore the feasibility of
black-box attacks against ML-based NIDS.

####  IV-A1 Basic Knowledge

In recent years, researchers successfully crafted adversarial examples without
previous knowledge of the target’s learning algorithm, gradients, parameters,
training data, or hyper-parameters [27]. However, only a minority of these
researches use intrusion, malware, or spam detection datasets. As seen in
figure 3a, to generate adversarial examples in the computer vision domain, the
attackers adds perturbation by slightly changing the pixels’ values. The
medium in the computer vision domain is well-known as the attackers are
perturbing a digital image that is visible to them. On the other hand, in
network intrusion, the attackers don’t know the medium they are trying to
perturb. As seen in figure 5, each ML-based network intrusion detection system
uses its unique pre-processing/feature extraction and selection techniques. If
the attackers don’t know the selected features and the pre-processing methods
of the target model, then they don’t know what to perturb. Kuppa et al. [28]
labeled their attack as black-box, however, they assumed that the adversaries
have previous knowledge of the target model’s feature set, feature extractor,
and feature transformers. We found that most of the “black-box” attacks in the
NIDS domain make similar assumptions. In our opinion, if the attackers know
all this information about the features management and selection of the target
model, their attack cannot be considered black-box.

####  IV-A2 Querying NIDS

As mentioned in section II, one of the main corners of black-box adversarial
attacks is querying the target model to obtain a substitute model. Then using
that alternative model, the attackers can get all the knowledge they need to
craft their adversarial attack. It would be reasonable to assume that only the
system management teams have access to the output of NIDS. Thus, it is not
clear how querying NIDSs in black-box attack settings is possible if the
attacker is not an insider.

(a) Adversarial Example Generation in the Computer Vision Domain

(b) Adversarial Example Generation in the Network Security Domain

Figure 3: Adversarial Examples Generation

###  IV-B Feature-Space Perturbations

In feature-space attacks, the attacker just changes the ML model input feature
vector to fool the target model by adding perturbations to the features.
Normally, attackers don’t have access to the feature vector after the feature
extraction phase in the ML-based NIDS pipeline. So how would feature-space
adversarial attacks against NIDS be considered practical? That is an important
question to answer considering that most of NIDS’s adversarial attack research
is in feature-space attacks [4].

Moreover, as previously mentioned, knowing which features, and feature
transformation techniques are used by the target model is a very hard task for
the attacker. However, even if the attackers know the selected features and
have access to them, adding perturbations to the ML-based NIDS features is a
complicated task [5]. Unlike the computer vision domain, numerous constraints
make crafting adversarial attacks in the NIDS space tough. A sizable portion
of features in the NIDS datasets are boolean (such as flags) and discrete
numbers (such as the protocol feature). The presence of these features doesn’t
give flexibility to the attackers to add perturbations that maximizes the loss
to fool the target model. For example, the attacker cannot add a perturbation
to a flag in the positive direction if its value is 1 (because the maximum
value is 1). The same goes for a discrete feature, for instance, the attacker
cannot add an arbitrary value to the protocol feature because some protocol
numbers are unassigned (e.g., unassigned IP address). Moreover, some numerical
values have similar limitations, for example, the attacker cannot change the
packet length feature to a negative value.

###  IV-C Problem-Space Perturbations

In problem-space adversarial attacks, the attacker modifies the actual packet
to deceive the target model. These attacks are much more practical than
feature-space attacks, as the attacker only has access to the file and not the
feature vector. Having said that, adding perturbations to network packets is
extremely difficult. First, the attacker must know which features, and feature
transformation techniques are utilized by the target model, so that the
modification of the packet produces perturbations to the features after the
feature extraction phase. Furthermore, even if the attackers succeeded to
modify the packet correctly to trick the model, these modifications will more
than likely alter the network and malicious functionality of the perturbed
packets [16]. For example, changing flags or length values in the header will
make it inaccurate, and packets with faulty headers are immediately discarded
at the destination (network functionality loss). Moreover, adding
perturbations to the payload of the packet could change its intended malicious
functionality. For example, if the attackers aim for an SQL injection attack,
modifying the packets’ content will make them lose their hostile purpose.
Therefore, as seen in Figure 3, crafting adversarial attacks against NIDS is
way more complicated compared to computer vision. While adding small
perturbations to pixels in computer vision may not change their functionality,
in the case of NIDS, such attacks can potentially alter both their malicious
behavior and network functionality, as shown in Figure 3b.

###  IV-D Adaptivity to Dynamic ML

The vast majority of adversarial attacks depend on the gradients of the target
models to maximize the loss and make the attacked model produce faulty
decisions. On the other hand, as previously mentioned, there is a trend to use
dynamic ML which is reasonable to assume that NIDSs would adopt. Because the
gradients in dynamic ML are always changing, that could add to the complexity
of creating an attack. To test this assumption, we investigated the effect of
dynamic learning on adversarial attacks against ML-NIDS. In the next sections,
we will discuss our experimental environment and results.

Figure 4: Attacking Scenario with Continuous Training: the impact of
adversarial attacks before (attacking in Day n) and after re-training
(attacking in Day n+1) Figure 5: Target ML-based NIDS

##  V Investigating of The Effect of Dynamic Learning on Adversarial Attacks
Against ML-NIDS

###  V-A Overview

As noted in Section IV-D, we believe that defence that adopt dynamic model
should reduce the effectiveness of adversarial attack. We set up and conduct
experiments to test the effect of continuous training on the effectiveness of
adversarial attacks against ML-NIDS. As illustrated in figure 4, our
experiments test the impact of adversarial attacks before (attacking day n)
and after re-training (attacking day n+1). In this section, we will explore
the ML model, dataset, metrics, and continuous training criteria used to test
the practicality of adversarial attacks against dynamic ML-based NIDS.

###  V-B Dataset

The CSE-CIC-IDS2018 [29] dataset was used in our experiments. Although only 12
percent of the network intrusion detection research utilizes this dataset, it
was found that it is currently the most up-to-date, and ML models have better
accuracy when it is used in training and testing [10]. Moreover, the CSE-CIC-
IDS2018 dataset has a low number of duplicate and uncertain data [30]. The
dataset was created by Communications Security Establishment (CSE) and the
Canadian Institute for Cybersecurity (CIC) in 2018/2019. In the CSV format
files, each tuple has 80 features. During the creation of the dataset,
numerous scenarios were constructed, and data were collected and edited daily.
The dataset contains six different classes (labels); Benign, Bot, Brute Force,
DoS, Infiltration, and SQL Injection.

###  V-C The Target ML-based NIDS

As seen in figure 5, the target ML-based NIDS consists of multiple phases. We
crafted our own ML NIDS for our experiments. However, we didn’t create a
feature extraction stage, as we used the features provided by the CSV files of
the CSE-CIC-IDS2018 dataset. We will explore our ML NIDS in the next
subsections.

####  V-C1 Pre-Processing

The pre-processing steps employed in our experiments contained several key
procedures. Initially, a process of data cleansing was implemented, whereby
features containing nulls, infinite values, or single values were removed.

Subsequently, a correlation analysis was conducted to identify highly
correlated features, which were then partitioned into mutually exclusive sets
to exclude dependent features. The correlation between features and labels was
also calculated, and the importance order of features was determined through
the use of Random Forest on separate decision trees. Finally, based on the
results obtained, the most important features were selected for further
analysis.

####  V-C2 Training and Testing

We used a fully connected Artificial Neural Network (ANN) model for our ML-
Based NIDS. The network consists of an input layer containing 32 nodes
representing the selected features. Then, two hidden layers with 25 (Relu
activation function) and 10 nodes (TANH activation function), which represent
the significant attributes describing the nature of the packet, are connected
to an output layer with a single node (malicious or benign). The ANN had a
learning rate of 0.001, a dropout of 0.5, and 20 epochs. We used 80 percent of
the selected records from the dataset for training and validation and 20
percent for testing.

###  V-D The Adversarial Attacks in Our Experiments

The Adversarial Robustness Toolbox (ART) library [31] was used to generate
FGSM [14], Projected Gradient Descent [32], and LowProFool [33] adversarial
attacks for our experiments.

####  V-D1 Continuous Training

The CSE-CIC-IDS2018 dataset is partitioned by days, so we assumed that the
target ML-NIDS model retrains daily. We did not forget the old knowledge, we
just added the information obtained from the new day to the model. We trained
the model using eighty percent of 1.05 million records (64% benign). With the
remaining 20 percent, we tested our model. Then, we applied the three
different attacks, seen in the previous subsection, using the gradients of the
model in that state by adding perturbations on all of the testing samples and
retested the model using the same metrics. Afterward, we added the knowledge
using eighty percent of the 1.05 million records of the next day and crafted
the attacks using the gradients of the model obtained the previous day, and
retested the model using the same metrics to check the effect of continuous
training on adversarial attacks in the NIDS space.

###  V-E Metrics

We used the following metrics to test the efficiency of the target ML-based
NIDS before and after the evasion. We also utilized them to experiment with
the effect of continuous training on adversarial attacks.

####  V-E1 Accuracy

It is the ratio of accurately classified samples to the total number of
instances.

$Accuracy=\frac{TP+TN}{TP+TN+FP+FN}$

####  V-E2 Precision

It is the ratio of accurately predicted malicious to all the samples predicted
as malicious.

$Precision=\frac{TP}{TP+FP}$

####  V-E3 Recall

It is a ratio of all instances accurately classified as malicious to all the
truly malicious samples.

$Recall=\frac{TP}{TP+FN}$

####  V-E4 F1-measure

It is the harmonic mean of Precision and Recall. So to obtain a high F1 score,
the model must have high Precision and Recall.

$F1=\frac{2*Precision*Recall}{Precision+Recall}$

Where TP is True Positive, TN is True Negative, FP is False Postive, and FN is
False Negative samples.

TABLE I: Effect of Continuous Training on FGSM Attacks, where Day n is
attacking before re-training and Day n+1 is attacking after re-training FGSM
---
| Accuracy | Precision | Recall | F1
Pre-attack | 0.999 | 0.997 | 0.998 | 0.997
Day n | 0.602 | 0.140 | 0.377 | 0.195
Day n+1 | 0.933 | 0.985 | 0.667 | 0.780
TABLE II: Effect of Continuous Training on Projected Gradient Descent Attacks,
where Day n is attacking before re-training and Day n+1 is attacking after re-
training Projected Gradient Descent
---
| Accuracy | Precision | Recall | F1
Pre-attack | 0.999 | 0.997 | 0.998 | 0.997
Day n | 0.620 | 0.272 | 0.040 | 0.067
Day n+1 | 0.793 | 0.331 | 0.086 | 0.130
TABLE III: Effect of Continuous Training on LowProFool Attacks, where Day n is
attacking before re-training and Day n+1 is attacking after re-training
LowProFool
---
| Accuracy | Precision | Recall | F1
Pre-attack | 0.999 | 0.997 | 0.998 | 0.997
Day n | 0.608 | 0.379 | 0.114 | 0.167
Day n+1 | 0.909 | 0.976 | 0.549 | 0.681

##  VI Dynamic ML Effect Findings

Section IV addressed numerous issues that pose significant obstacles to
adversarial attacks against Network Intrusion Detection Systems (NIDS).
However, even if attackers were to overcome these challenges, the
implementation of dynamic learning would reduce the effectiveness of the
attacks, as seen in this section.

###  VI-A Attack Scenario

In this section, we will show the results of our experiments that test the
effect of the continuous training aspect of the dynamic learning models. Our
testing scenario (figure 4) is; despite the challenges discussed in Section IV
regarding obtaining information about the target, we assume that attackers
have somehow acquired the knowledge needed to calculate the gradients and
craft their adversarial attacks. However, due to the complexity of formulating
adversarial attacks against NIDS, there is a time delay (at least one day in
our experiments) before the attackers initiate the evasion. Meanwhile, the
target dynamic ML NIDS undergoes retraining, rendering the knowledge obtained
by the attackers one day old. In our experiments, we aim to test the
efficiency of adversarial attacks against NIDS using up-to-date knowledge (Day
n) for evasion versus outdated knowledge (Day n+1) after retraining the NIDS.

###  VI-B Results

As seen in tables I, II, and III, the accuracy, precision, recall, and F1
score of the target ML-NIDS were all above 0.997 percent before the attacks.
However, after adding the perturbations to all of the testing samples used up-
to-date knowledge (Day n) the model’s performance has significantly dropped in
all of the three tested evasions. Using FGSM, Projected Gradient Descent, and
LowProFool attacks, the accuracy has fallen to 0.602, 0.620, and 0.608, and
the F1 measure has plunged to 0.195, 0.067, and 0.167, respectively. Thus, we
can conclude that adversarial attacks can successfully and easily evade ML-
based NIDS, which may be true in our opinion. However, it is not that
simple.However, in the case of re-training in dynamic ML (on Day n+1), the
attacker is likely to use the outdated knowledge of the previous day - due to
the complexity of creating adversarial attacks in the NIDS space - which leads
to the attacks’ efficiency dropping as shown in the last line of each table.
In FGSM, Projected Gradient Descent, and LowProFool attacks, the accuracy of
the target model has recovered to 0.933, 0.793, and 0.909, and the F1 measure
has rebounded to 0.780, 0.130, and 0.681, respectively. It is worth mentioning
that this recovery of the model’s measures is after just one re-training.
Moreover, these promising results are even without adversarial training (re-
learning data don’t contain adversarial samples [20]).

On another note, as can be seen in the results, different attacks have
dissimilar responses to adversarial attacks and dissimilar recovery rates
after re-training. For example, the recall in the LowProFool and Projected
Gradient Descent attacks was significantly reduced after attacking the model
using up-to-date data, unlike FGSM, whose precision was more affected by the
attacks. On the other hand, the Projected Gradient Descent attack had a low
recovery rate after re-training, unlike the other attacks, which had higher
recovery rates. These dissimilarities need further investigation in the
future.

##  VII Conclusion and Lessons Learned

In this paper, we expanded the research discussion of the practicality of
adversarial attacks against NIDS by presenting a taxonomy of the practicality
issues of adversarial attacks against ML-based NIDS. It is worth mentioning
that our work does not claim that adversarial attacks will not harm ML-based
NIDSs. However, we find that the gap between research and real-world
practicality is wide and deserves to be addressed. Moreover, we investigated
and tested the effect of continuous training on adversarial attacks against
NIDS. Our experiments indicate that continuous re-training, even without
adversarial training, can limit the effect of adversarial attacks.

We also note the following lessons we learned while doing this work:

###  VII-A Dynamic Learning

  * •

Dynamic ML research is scarce in the machine learning-based NIDS domain. This
scarcity needs attention because it is expected that concept drifts will
happen in the NIDS space similar to other research domains [6]

  * •

The dynamic learning effect on adversarial attacks depends on numerous factors
such as the model used, re-learning rate, re-learning data, etc. Our work just
shows that it could reduce the effect of adversarial attacks on attacked
models.

  * •

Systems that adapt to changes in legitimate behavior over time, such as
evolving machine learning approaches, may be vulnerable to manipulation by
intruders who gradually introduce their own malicious behavior into the
baseline. This is known as session creep [34].

###  VII-B Datasets

  * •

Obtaining Labeled datasets to train even static ML NIDS is very tough and
expensive [35]. Unfortunately, dynamic ML models need more labeled datasets
for continuous training, and acquiring these labeled datasets is even harder.
This topic needs more research from network security and ML communities.

  * •

Although CSE-CIC-IDS-2018 is widely regarded as a reliable/up-to-date NIDS
dataset [10], it has been discovered that it has several issues with feature
generation and labeling [36]. This re-emphasises the issue that the ML and
NIDS research communities should prioritize the development of more robust
datasets.

###  VII-C Adversarial attacks in network security vs computer vision

  * •

Several adversarial attacks studied in the network/cybersecurity field may
prove impractical when applied in real-world contexts. Compared with other
domains where ML is deployed, such as computer vision, crafting adversarial
attacks in the network/cybersecurity domain is more complicated [16]. As we
mentioned previously, well-crafted perturbations added to a digital medium can
trick ML models. For example, in image classification adding perturbations to
an image is slightly changing the color values of the image’s pixels, which
will make the perturbed image almost visually unaffected. However, adding
perturbations to the network packets or PE files, could alter their
functionality. For example, adding perturbation to an executable malware could
make it un-functional. Furthermore, in white-box attacks, the attackers should
know a considerable amount of information (especially the gradients) about the
attacked ML model, which is not very practical in many domains. In network
security, it’s even less practical because of the tighter security in the
network environment (in comparison to, for example, computer vision where
there are fewer attackers [16]). Moreover, in the majority of black-box
attacks, the researchers assume that attackers query the model to calculate
the gradients to craft their attacks. Nevertheless, querying entities in
network security (for example, Network Intrusion Detection Systems) might be
impossible. Apruzzese et al. [37] highlighted that in real-world scenarios,
attackers often employ tactics based on simple yet effective techniques to
evade ML systems that are barely associated with gradient computations.

  * •

On the other hand, the perturbed network elements and features are more
diverse and heterogeneous, making detecting and defending against adversarial
attacks in the network/cybersecurity domain more difficult than in other
domains such as computer vision [15].

##  VIII Acknowledgement

This work was supported by the Natural Sciences and Engineering Research
Council of Canada (NSERC) throut the Discovery Grant program.

## References

  * [1] N. Papernot, P. McDaniel, A. Sinha, and M. P. Wellman, “Sok: Security and privacy in machine learning,” in 2018 IEEE European Symposium on Security and Privacy (EuroS&P), pp. 399–414, IEEE, 2018.
  * [2] S. Kapoor and A. Narayanan, “Leakage and the reproducibility crisis in ml-based science,” 2022.
  * [3] G. Apruzzese, H. Anderson, S. Dambra, D. Freeman, F. Pierazzi, and K. Roundy, “Position:“real attackers don’t compute gradients”: Bridging the gap between adversarial ml research and practice,” in IEEE Conference on Secure and Trustworthy Machine Learning, IEEE, 2022.
  * [4] G. Apruzzese, M. Andreolini, L. Ferretti, M. Marchetti, and M. Colajanni, “Modeling realistic adversarial attacks against network intrusion detection systems,” Digital Threats: Research and Practice (DTRAP), vol. 3, no. 3, pp. 1–19, 2022.
  * [5] M. A. Merzouk, F. Cuppens, N. Boulahia-Cuppens, and R. Yaich, “Investigating the practicality of adversarial evasion attacks on network intrusion detection,” Annals of Telecommunications, pp. 1–13, 2022.
  * [6] P. Casas, P. Mulinka, and J. Vanerio, “Should i (re) learn or should i go (on)? stream machine learning for adaptive defense against network attacks,” in Proceedings of the 6th ACM Workshop on Moving Target Defense, pp. 79–88, 2019.
  * [7] Ó. Fontenla-Romero, B. Guijarro-Berdiñas, D. Martinez-Rego, B. Pérez-Sánchez, and D. Peteiro-Barral, “Online machine learning,” in Efficiency and Scalability Methods for Computational Intellect, pp. 27–54, IGI Global, 2013.
  * [8] V. Kumar and O. P. Sangwan, “Signature based intrusion detection system using snort,” International Journal of Computer Applications & Information Technology, vol. 1, no. 3, pp. 35–41, 2012.
  * [9] V. Chandola, A. Banerjee, and V. Kumar, “Anomaly detection: A survey,” ACM computing surveys (CSUR), vol. 41, no. 3, pp. 1–58, 2009.
  * [10] Z. Ahmad, A. Shahid Khan, C. Wai Shiang, J. Abdullah, and F. Ahmad, “Network intrusion detection system: A systematic study of machine learning and deep learning approaches,” Transactions on Emerging Telecommunications Technologies, vol. 32, no. 1, p. e4150, 2021.
  * [11] J. Lu, A. Liu, F. Dong, F. Gu, J. Gama, and G. Zhang, “Learning under concept drift: A review,” IEEE Transactions on Knowledge and Data Engineering, vol. 31, no. 12, pp. 2346–2363, 2018.
  * [12] A. Bifet and R. Gavalda, “Learning from time-changing data with adaptive windowing,” in Proceedings of the 2007 SIAM international conference on data mining, pp. 443–448, SIAM, 2007.
  * [13] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” arXiv preprint arXiv:1312.6199, 2013.
  * [14] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” arXiv preprint arXiv:1412.6572, 2014.
  * [15] O. Ibitoye, R. Abou-Khamis, A. Matrawy, and M. O. Shafiq, “The threat of adversarial attacks on machine learning in network security–a survey,” arXiv preprint arXiv:1911.02621, 2019.
  * [16] I. Rosenberg, A. Shabtai, Y. Elovici, and L. Rokach, “Adversarial machine learning attacks and defense methods in the cyber security domain,” ACM Computing Surveys (CSUR), vol. 54, no. 5, pp. 1–36, 2021.
  * [17] O. Ibitoye, O. Shafiq, and A. Matrawy, “Analyzing adversarial attacks against deep learning for intrusion detection in iot networks,” in 2019 IEEE global communications conference (GLOBECOM), pp. 1–6, IEEE, 2019.
  * [18] T. D. Nguyen, P. Rieger, M. Miettinen, and A.-R. Sadeghi, “Poisoning attacks on federated learning-based iot intrusion detection system,” in Proc. Workshop Decentralized IoT Syst. Secur.(DISS), pp. 1–7, 2020.
  * [19] A. Athalye, N. Carlini, and D. Wagner, “Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples,” in International conference on machine learning, pp. 274–283, PMLR, 2018.
  * [20] R. Abou Khamis and A. Matrawy, “Evaluation of adversarial training on different types of neural networks in deep learning-based idss,” in 2020 international symposium on networks, computers and communications (ISNCC), pp. 1–6, IEEE, 2020.
  * [21] R. Abou Khamis, M. O. Shafiq, and A. Matrawy, “Investigating resistance of deep learning-based ids against adversaries using min-max optimization,” in ICC 2020-2020 IEEE International Conference on Communications (ICC), pp. 1–7, IEEE, 2020.
  * [22] F. Zhang, P. P. Chan, B. Biggio, D. S. Yeung, and F. Roli, “Adversarial feature selection against evasion attacks,” IEEE transactions on cybernetics, vol. 46, no. 3, pp. 766–777, 2015.
  * [23] K. Grosse, P. Manoharan, N. Papernot, M. Backes, and P. McDaniel, “On the (statistical) detection of adversarial examples,” arXiv preprint arXiv:1702.06280, 2017.
  * [24] A. Paudice, L. Muñoz-González, A. Gyorgy, and E. C. Lupu, “Detection of adversarial training examples in poisoning attacks through anomaly detection,” arXiv preprint arXiv:1802.03041, 2018.
  * [25] H. A. Alatwi and C. Morisset, “Adversarial machine learning in network intrusion detection domain: A systematic review,” arXiv preprint arXiv:2112.03315, 2021.
  * [26] N. Martins, J. M. Cruz, T. Cruz, and P. H. Abreu, “Adversarial machine learning applied to intrusion and malware scenarios: a systematic review,” IEEE Access, vol. 8, pp. 35403–35419, 2020.
  * [27] N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami, “Practical black-box attacks against machine learning,” in Proceedings of the 2017 ACM on Asia conference on computer and communications security, pp. 506–519, 2017.
  * [28] A. Kuppa, S. Grzonkowski, M. R. Asghar, and N.-A. Le-Khac, “Black box attacks on deep anomaly detectors,” in Proceedings of the 14th International Conference on Availability, Reliability and Security, pp. 1–10, 2019.
  * [29] I. Sharafaldin, A. H. Lashkari, and A. A. Ghorbani, “Toward generating a new intrusion detection dataset and intrusion traffic characterization.,” ICISSp, vol. 1, pp. 108–116, 2018.
  * [30] G. Karatas, O. Demir, and O. K. Sahingoz, “Increasing the performance of machine learning-based idss on an imbalanced and up-to-date dataset,” IEEE Access, vol. 8, pp. 32150–32162, 2020.
  * [31] M.-I. Nicolae, M. Sinn, M. N. Tran, B. Buesser, A. Rawat, M. Wistuba, V. Zantedeschi, N. Baracaldo, B. Chen, H. Ludwig, et al., “Adversarial robustness toolbox v1. 0.0,” arXiv preprint arXiv:1807.01069, 2018.
  * [32] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” arXiv preprint arXiv:1706.06083, 2017.
  * [33] V. Ballet, X. Renard, J. Aigrain, T. Laugel, P. Frossard, and M. Detyniecki, “Imperceptible adversarial attacks on tabular data,” arXiv preprint arXiv:1911.03274, 2019.
  * [34] P. C. van Oorschot and P. C. van Oorschot, “Intrusion detection and network-based attacks,” Computer Security and the Internet: Tools and Jewels from Malware to Bitcoin, pp. 309–338, 2021.
  * [35] G. Apruzzese, P. Laskov, and A. Tastemirova, “Sok: The impact of unlabelled data in cyberthreat detection,” arXiv preprint arXiv:2205.08944, 2022.
  * [36] L. Liu, G. Engelen, T. Lynar, D. Essam, and W. Joosen, “Error prevalence in nids datasets: A case study on cic-ids-2017 and cse-cic-ids-2018,” in 2022 IEEE Conference on Communications and Network Security (CNS), pp. 254–262, IEEE, 2022.
  * [37] G. Apruzzese, H. S. Anderson, S. Dambra, D. Freeman, F. Pierazzi, and K. A. Roundy, “Position:“real attackers don’t compute gradients”: Bridging the gap between adversarial ml research and practice,” in First IEEE Conference on Secure and Trustworthy Machine Learning.


# Cross-speaker Emotion Transfer by Manipulating Speech Style Latents

###### Abstract

In recent years, emotional text-to-speech has shown considerable progress.
However, it requires a large amount of labeled data, which is not easily
accessible. Even if it is possible to acquire an emotional speech dataset,
there is still a limitation in controlling emotion intensity. In this work, we
propose a novel method for cross-speaker emotion transfer and manipulation
using vector arithmetic in latent style space. By leveraging only a few
labeled samples, we generate emotional speech from reading-style speech
without losing the speaker identity. Furthermore, emotion strength is readily
controllable using a scalar value, providing an intuitive way for users to
manipulate speech. Experimental results show the proposed method affords
superior performance in terms of expressiveness, naturalness, and
controllability, preserving speaker identity.

Index Terms—  Speech Synthesis, Emotion Transfer, Emotional Speech Synthesis,
Latent Space Manipulation

##  1 Introduction

As there are growing expectations for human-like TTS, subtle changes in
prosody or emotion should be reflected in the output of a TTS model. Humans
can speak with different emotions and this leads to rich and diverse
conversations. However, emotional speech data are not easy to acquire. It is
extremely hard to record multiple sentences for a long time while consistently
preserving emotion. Moreover, due to the ambiguity of emotion labels, samples
with inconsistent emotion labels are easily observed in open source emotional
speech datasets [1]. Even if it is possible to find a correct emotion label,
expressiveness of emotion is limited without controllability of emotion
intensity.

Most of the previous methods for emotional speech synthesis use additional
emotion encoding or a reference audio [2, 3, 4]. These models require an
emotion label for each sample, consuming a fair amount of emotion-tagged
speech data [2, 4]. Furthermore, inconsistency in emotion labels leads to
degraded performance of emotional speech synthesis. Also, cross-speaker
emotion transfer often does not work when an unseen emotion is transferred to
a speaker. In regard to emotion strength control, there have been several
attempts [5, 6] to generate emotion strength scores using an emotion
classifier or a ranking function. These approaches are still exposed to
mislabeling problems, however, as they use emotion labeled data to extract
emotion intensity.

To address the issues noted above, we propose transferable and controllable
emotion speech synthesis by leveraging rich latent representation. Domain
adversarial training and cycle-consistency loss disentangle the speaker from
style, making the latent style space rich and transferable. During training,
the entire model is trained without any emotion label. To transfer emotion, we
utilize a SVM hyperplane during inference to manipulate the style latent
towards a desirable emotion. Our method successfully transfers emotion to an
emotion-neutral reading-style speaker with only a handful of emotion labeled
samples. Furthermore, without external labels, emotion intensity can be
controlled by a scalar value, which is easy and intuitive. The generated audio
samples are available at http://emo-transfer.github.io

##  2 Related Works

###  2.1 Emotional Speech Synthesis

For emotional speech synthesis, it is common to use an emotion label as a
global condition [2, 3]. Otherwise, emotion information is extracted from a
reference audio [7, 4] or text [8]. Most of these methods not only require a
large quantity of emotion labeling, but also often fail to achieve good
quality in cross-speaker emotion transfer. Some approaches [9, 10] use Speech
Emotion Recognition (SER) to obtain emotion labels. However, SER is another
challenging task and it still requires emotion tagged data to train itself. To
reflect not only a type of emotion but also its intensity, methods for
controlling emotion strength have been suggested [5, 6, 11]. [12] uses an
external SER model to extract emotion intensity scores, whereas [6] uses a
ranking function to predict emotion intensity. However, for these models all
data should still be labeled for training.

###  2.2 Latent Space Manipulation

In the image synthesis domain, attribute editing using StyleGAN [13] has been
widely studied [14, 15, 16]. Such methods utilize latent space of StyleGAN to
manipulate attributes. Among these methods, [14] provides a simple yet
effective approach for editing. Using a hyperplane in latent space that
discriminates attribute-positive and attribute-negative samples, facial
attributes such as age, gender or pose can be edited. The hyperplane is
acquired from SVM training. In practice, any binary attributes can be
manipulated when latent vectors from both positive and negative sides are
given. In this paper, we adopt the method of latent space manipulation
suggested in [14] to synthesize emotional speech from emotion-neutral
speakers. Details of training the SVM and manipulating latents will be
described in Section 3.2

##  3 Method

To facilitate cross-speaker emotion transfer, we propose domain adversarial
training and cycle-consistency loss for the acoustic model to learn
disentangled style and speaker representation. Along with this model, we
suggest a method for controlling emotion and its intensity by utilizing a
hyperplane obtained from training a SVM.

(a) The overall architecture of the proposed acoustic model

(b) SVM hyperplane

Fig. 1: From the first stage (1(a)), speaker-independent style vectors are
extracted. In the second stage (1(b)), we use a unit vector $n$ to transfer
emotion. The unit vector is perpendicular to a SVM hyperplane that separates
emotional style vectors from emotion-neutral ones.

###  3.1 Disentangled Latent Style Vectors

We focus on extracting rich yet disentangled style representation from speech.
If style space is disentangled from speaker identity, cross-speaker emotion
transfer becomes easier. Otherwise, latent vector manipulation using a SVM
will not work well, as a direction vector acquired from the SVM will also
transform speaker information. For example, converting an emotion-neutral
sample to an angry one can result in a change of the speaker identity.
Therefore, we try to disentangle the speaker identity to generate better style
latent space for emotion transfer.

####  3.1.1 Tacotron2-based Acoustic Model

Our proposed method is based on Tacotron2 with the following modifications.
The main difference is that we adopt a style encoder and a speaker encoder.
The style encoder has a target mel spectrogram as an input and generates a
style vector. The architecture of the style encoder is based on [17]. The
style encoder generates the final output, called a style vector, which is
added to the output of a text encoder. The speaker encoder also receives a mel
spectrogram as an input and generates a speaker vector. The speaker encoder is
composed of a LSTM and a projection layer. The speaker vector is concatenated
to an input of a decoder.

####  3.1.2 Domain Adversarial Training

By adversarially classifying a style vector to a speaker class, the style
encoder learns speaker-independent style. In the speech domain, there have
been many applications of domain adversarial training, such as [11, 18]. This
has been shown to be largely effective in disentangling information. Our
adversarial speaker classifier consists of linear layers. Prior to the layers,
a gradient reversal layer was attached. The gradient reversal layer reverses
gradients from the classifier so that a style vector cannot discriminate
speakers. On the speaker encoder side, a speaker classifier follows a speaker
encoder to maintain speaker information. The speaker classifier shares the
same structure with the adversarial speaker classifier. The only difference is
that it does not have a gradient reversal layer.

####  3.1.3 Cycle-consistency Loss

For both the style encoder and speaker encoder, we adopt cycle-consistency
loss to preserve information. If we let the speaker encoder $E_{s}$ and the
style encoder $E_{w}$ and the target mel spectrogram as $x$, a style vector
$s$ and a speaker vector $w$ are the outputs of $E_{s}({x})$ and $E_{w}({x})$,
respectively. The generated output of the entire TTS model is $f(t,w,s)$,
where $t$, $w$, $s$ are the input text, style vector, and speaker vector,
respectively. After generating a predicted output $\hat{x}$, we use speaker
vectors from randomly sampled speakers to produce ${s}^{\prime}$. Then,
${x}^{\prime}$ is generated by $f(t,w,{s}^{\prime})$. ${x}^{\prime}$ is used
as an input of both the speaker encoder and the style encoder, which leads to
$E_{s}({x}^{\prime})$, $E_{w}({x}^{\prime})$. Cycle-consistency loss for both
the speaker and style can be written as below. $N$ is the batch size during
training. We use Mean Squared Error (MSE) for the loss term.

$L_{style}=\frac{1}{N}\sum_{i=1}^{N}(w_{i}-E_{w}({x_{i}}^{\prime})))^{2}$ (1)
$L_{speaker}=\frac{1}{N}\sum_{i=1}^{N}({s_{i}^{\prime}}-E_{s}({x_{i}}^{\prime})))^{2}$
(2)

###  3.2 Controllable Cross-speaker Emotion Transfer

Before inference, we extract speaker vectors from all utterances of a target
speaker. We then obtain the centroid of those vectors to use it as a speaker
vector for inference. Likewise, a style vector that is fed to the model during
inference is a centroid of all style vectors from the target speaker. A style
vector without any manipulation represents emotion-neutral style, as the
speakers we experimented on are emotion-neutral speakers. To transfer emotion,
we edit the style vector using a SVM.

We train a SVM with a positive group of style vectors from a certain emotion
and a negative group from emotion-neutral samples. As [19] showed gender and
accent transformation through vector operations, it can be assumed that there
exists a hyperplane for each attribute that separates attribute-positive and
attribute-negative latent vectors. To find a hyperplane, we use a linear SVM.
After training the SVM, we define a unit normal vector $n$ that is
perpendicular to the hyperplane with its distance to the hyperplane 0. By
using $n$, we can determine on which side of the hyperplane a style vector $w$
lies. We use a metric defined as $d(n,w)=n^{T}w$. If $d(n,w)$ is negative, $w$
is on the attribute-negative side. On the other hand, If $d(n,w)$ is positive,
$w$ is attribute-positive. In our case, $w$ is the centroid of all style
vectors from a target speaker, as described above. By adding $n$ multiplied by
a scaling factor $\alpha$ to $w$, as in (3), emotion can be adjusted.

$w_{edit}=w+\alpha n$ (3)

If $\alpha>0$, the emotion of the edited vector $w_{edit}$ is led towards the
direction of an intended emotion since $d(n,w_{edit})$ becomes
$d(n,w)+\alpha$. If $\alpha<0$, $w_{edit}$ moves towards the opposite
direction. By increasing $\alpha$ ranging from [0, 2], a gradual change in
emotion intensity is shown. One more interesting point is that even if
$\alpha$ is set to a negative value, it gives a meaningful result. For
example, if there exists an editing vector that manipulates a style vector
towards sad emotion, we can adjust $\alpha$ to a negative value to make it
sound happy.

To manipulate emotion-related style alone without affecting other speaking
style from a speaker, we use conditional manipulation. Given $n_{1}$ and
$n_{2}$ from two hyperplanes, we obtain a projected direction as
$n_{1}-(n_{1}^{T}n_{2})n_{2}$ that makes $n_{1}$ independent of the direction
$n_{2}$. As we are trying to manipulate style, $n_{1}$ is a perpendicular unit
normal vector from a hyperplane learned from an emotion classification and
$n_{2}$ will be from a speaker classification.

Style | Ratio [%] | Style | Ratio[%]
---|---|---|---
Reading | 60.4 | Fairytale | 1.3
Conversational | 1.2 | Angry | 2.1
Animation Dubbing | 12.1 | Sad | 2.0
Whisper | 1.0 | Happy | 2.3
Children | 7.6 | Not defined | 10.0

Table 1: Speech Style of the Training Data

##  4 Experiments and Results

###  4.1 Dataset

In this paper, we conducted experiments in both English and Korean. Our
English dataset consists of 25 female speakers and 42 male speakers, with 127
hours in total, whereas our Korean dataset consists of 58 female speakers and
42 male speakers, with 270 hours in total. While an open-source dataset was
also used, most of the samples were collected internally, recorded by
professional voice actors. For open-source dataset, we used DailyTalk [20] and
Korean Single Speaker (KSS) Speech Dataset [21]. Table 1 shows the training
data and its speech style used for the acoustic model.

Fig. 2: VAD (Valence-Arousal-Dominance) plot according to emotion and its
intensity.

###  4.2 Experimental Setup

First, for training the acoustic mode, we follow details of the style encoder
of [17]. The speaker encoder consists of one LSTM layer with hidden size of
258 and one projection layer. Both the style vector and speaker vector are 512
dimensional. Both the adversarial speaker classifier and speaker classifier
consist of three fully-connected layers, followed by ReLU activation and
dropout rate by 0.1 for each layer. All the classification losses are
multiplied by 0.02 and added to loss terms of the acoustic model. In the
training process, we train the model with batch size of 32 using the Adam
optimizer. Learning rate scheduling follows the Noam Scheduler, with initial
learning rate $10^{-3}$ and a warm-up threshold of 4000 steps. We trained the
model for each gender and language, resulting in four models. All the models
were trained for 60 epochs. Until the attention loss reaches 0.7, the
attention module is trained alone to stabilize the training process.

To train the SVM, style latent vectors for a negative set and a positive set
were collected. The negative set consists of emotion-neutral samples, spoken
in reading-style. The positive set consists of emotional samples of a
desirable emotion, among angry, happy, and sad. Normally, 100 samples were
randomly selected for each negative and positive set among training samples
that we could find labels of. Although using 100 samples showed stable
performance, results on a single sample were good as well. When we train a SVM
under a one-shot setting, we selected paired samples, which share the same
script and speaker to eliminate confounding variables other than emotion. In
the classification task, our SVM models achieved over 90% accuracy on a
validation set. Additionally, a speaker classification task was conducted for
conditional manipulation. We used all samples from the positive speaker and
randomly selected the same number of negative samples. The speaker
classification task achieved over 98% accuracy on a validation set.

Setting | Naturalness | Emo. Similarity | Spk. Similarity
---|---|---|---
Ground Truth | $4.55\pm 0.05$ | - | -
Baseline | $3.54\pm 0.08$ | $2.48\pm 0.08$ | $1.90\pm 0.07$
Conventional [5] | $3.86\pm 0.04$ | $2.91\pm 0.04$ | $2.46\pm 0.05$
Proposed | $\textbf{4.70}\pm\textbf{0.04}$ | $\textbf{4.24}\pm\textbf{0.06}$ | $\textbf{3.98}\pm\textbf{0.07}$

Table 2: MOS on naturalness, speaker similarity and emotion similarity

###  4.3 Evaluation

We conducted a subjective evaluation using mean opinion score (MOS) to
evaluate naturalness, speaker similarity, and emotion similarity. In the test,
15 subjects were asked to rate 180 sentences on a scale from 1 to 5. The
generated samples were randomly selected from each emotion category. The
number of samples per each category was balanced. For speaker similarity, a
ground truth sample was given for each entry, as a reference. The participants
compared a given sample with the ground truth on the basis of speaker
identity. For emotion similarity, an emotion tag, for example, ”happy”, was
given and the subjects were asked to judge whether a given speech sample
expresses the emotion. To ease their decision, we provided a corresponding
emotion-neutral sample that was generated from the sample model with the same
text but without emotion transfer. For the baseline, we used Tacotron2. The
baseline model uses speaker embedding to encode speaker information while
maintaining a style encoder which has identical structure with the proposed
model. The baseline generates output with an averaged style vector, which was
derived from 100 randomly selected emotion-labeled style vectors for each
emotion. We used a HiFi-GAN [22] as a vocoder to generate waveforms.

Table 2 shows the results of the evaluation of the proposed model along with
ground truth, the baseline and the conventional model [5]. With respect to the
MOS score, the proposed model outperforms the baseline model and the
conventional model in all evaluation categories. In terms of naturalness, the
proposed model was rated higher than the ground truth. As we collected speech
data with various styles regardless of the source, a few ground truth samples
contain unnatural prosody or noise. However, during training, this phenomenon
appears to be diluted. We also observed that the baseline model and the
conventional model fall behind the proposed model due to its vulnerability to
a wide range of variation in style. Naturally, emotion similarity and speaker
similarity are at variance with each other because speech features such as
pitch or timber vary greatly from original speech samples as emotion becomes
intense. Even with this contradiction, the proposed model shows high scores in
both emotion similarity and speaker similarity. Overall, the proposed model
maintains high fidelity in speaker identity and naturalness, expressing proper
emotion at the same time.

Fig. 3: AB preference test on emotion similarity and speaker similarity

####  4.3.1 Emotion Intensity Control

To test emotion intensity control, we used a pretrained speech emotion
recognition (SER) model [23]. This SER model was trained using MSP-Podcast
[24]. We used the pretrained model as it was without any fine-tuning. MSP-
Podcast was labeled with arousal, valence, and dominance ranging from 1 to 7.
[23] normalized the values into an interval of 0 to 1. Happiness and anger are
known to include high degrees of arousal and dominance whereas sadness has low
degrees of arousal and dominance. Happiness also has high degrees of valence
while anger and sadness are low in valence.

In Figure 2, the x axis represents $\alpha$, a scaling factor for emotion
intensity, and the y axis represents prediction for arousal, valence, and
dominance ranging from 0 to 1. Fifty randomly selected samples were used to
extract a mean value for each metric. As shown in Figure 2, both for angry and
happy, arousal increases as we raise the intensity of emotion by controlling
$\alpha$, whereas arousal decreases for sad. Valence increases for happy
whereas it slightly decreases for both angry and sad emotion. The overall
tendency shows that emotion intensity is well adjusted towards an intended
emotion.

####  4.3.2 Few-shot Emotion Transfer

To demonstrate few-shot emotion transfer, we compare one-shot emotion transfer
to 100-shot setting in an A/B preference test. In the preference test,
participants were asked to select which of the two samples is more similar to
a given speaker or emotion. As shown in Figure 3, one-shot setting shows
comparable results to 100-shot. Neither showed superior results in terms of
preference.

####  4.3.3 Ablation Study

To show the effectiveness of each module of the proposed model, we conducted
an ablation study. Table 3 represents MOS scores for each case in terms of
naturalness, speaker similarity and emotion similarity. Removing an
adversarial speaker classifier or cycle-consistency loss resulted in
degradation in emotion similarity. In particular, the score for emotion
similarity drops drastically without cycle-consistency loss. It can be
inferred that cycle-consistency loss plays an important role in generating
disentangled yet rich latent style space. Even though it was highly rated in
speaker similarity, this is because emotion was not transferred. In
comparison, the proposed model shows superior performance in terms of emotion
expression, maintaining sufficiently good scores in speaker similarity. This
indicates that both components are effective in disentangling and preserving
style and speaker information. Figure 4 also supports this argument, showing
that speaker vectors learned by the model successfully reserve speaker
identity.

Fig. 4: Visualization of speaker vectors learned by a speaker encoder.

Setting | Naturalness | Emo. Similarity | Spk. Similarity
---|---|---|---
Proposed | $\textbf{4.77}\pm\textbf{0.04}$ | $\textbf{4.32}\pm\textbf{0.07}$ | $\textbf{4.31}\pm\textbf{0.09}$
w/o Adv. Speaker Classifier | $4.72\pm 0.04$ | $4.13\pm 0.07$ | $4.34\pm 0.06$
w/o Cycle-consistency loss | $4.77\pm 0.04$ | $3.55\pm 0.08$ | $4.71\pm 0.05$

Table 3: Ablation Study

##  5 Conclusion

This paper suggests a novel method for cross-speaker emotion transfer and
manipulation by applying vector arithmetic on a disentangled latent style
space. To extract latent style without interference from speaker information,
we propose domain adversarial training and cycle-consistency loss. In
addition, we provide an intuitive way to transfer and to manipulate the style
latent vector by using a SVM hyperplane. Experimental results show that our
method greatly improves speaker similarity and emotion similarity while
keeping naturalness, without leveraging a large amount of emotion labeled
data. In future work, we will conduct experiments on other semantic attributes
of speech data, such as age or gender.

## References

  * [1] Chenye Cui, Yi Ren, Jinglin Liu, Feiyang Chen, Rongjie Huang, Ming Lei, and Zhou Zhao,  “Emovie: A mandarin emotion speech dataset with a simple emotional text-to-speech model,”  arXiv preprint arXiv:2106.09317, 2021.

  * [2] Younggun Lee and Taesu Kim,  “Robust and fine-grained prosody control of end-to-end speech synthesis,”  in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 5911–5915.
  * [3] Pengfei Wu, Zhenhua Ling, Lijuan Liu, Yuan Jiang, Hongchuan Wu, and Lirong Dai,  “End-to-end emotional speech synthesis using style tokens and semi-supervised training,”  in 2019 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC). IEEE, 2019, pp. 623–627.
  * [4] Yi Lei, Shan Yang, Xinsheng Wang, and Lei Xie,  “Msemotts: Multi-scale emotion transfer, prediction, and control for emotional speech synthesis,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2022.
  * [5] Tao Li, Xinsheng Wang, Qicong Xie, Zhichao Wang, and Lei Xie,  “Cross-speaker emotion disentangling and transfer for end-to-end speech synthesis,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 1448–1460, 2022.
  * [6] Yi Lei, Shan Yang, and Lei Xie,  “Fine-grained emotion strength transfer, control and prediction for emotional speech synthesis,”  in 2021 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2021, pp. 423–430.
  * [7] RJ Skerry-Ryan, Eric Battenberg, Ying Xiao, Yuxuan Wang, Daisy Stanton, Joel Shor, Ron Weiss, Rob Clark, and Rif A Saurous,  “Towards end-to-end prosody transfer for expressive speech synthesis with tacotron,”  in international conference on machine learning. PMLR, 2018, pp. 4693–4702.
  * [8] Yookyung Shin, Younggun Lee, Suhee Jo, Yeongtae Hwang, and Taesu Kim,  “Text-driven emotional style control and cross-speaker style transfer in neural tts,”  arXiv preprint arXiv:2207.06000, 2022.
  * [9] Xiong Cai, Dongyang Dai, Zhiyong Wu, Xiang Li, Jingbei Li, and Helen Meng,  “Emotion controllable speech synthesis using emotion-unlabeled dataset with the assistance of cross-domain speech emotion recognition,”  in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 5734–5738.
  * [10] Kun Zhou, Berrak Sisman, Rui Liu, and Haizhou Li,  “Seen and unseen emotional style transfer for voice conversion with a new emotional speech dataset,”  in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 920–924.
  * [11] Chae-Bin Im, Sang-Hoon Lee, Seung-Bin Kim, and Seong-Whan Lee,  “Emoq-tts: Emotion intensity quantization for fine-grained controllable emotional text-to-speech,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 6317–6321.
  * [12] Bastian Schnell and Philip N Garner,  “Improving emotional tts with an emotion intensity input from unsupervised extraction,”  in Proc. 11th ISCA Speech Synthesis Workshop (SSW 11), 2021, pp. 60–65.
  * [13] Tero Karras, Samuli Laine, and Timo Aila,  “A style-based generator architecture for generative adversarial networks,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 4401–4410.
  * [14] Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou,  “Interfacegan: Interpreting the disentangled face representation learned by gans,”  IEEE transactions on pattern analysis and machine intelligence, 2020\.
  * [15] Edo Collins, Raja Bala, Bob Price, and Sabine Susstrunk,  “Editing in style: Uncovering the local semantics of gans,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 5771–5780.
  * [16] Rameen Abdal, Peihao Zhu, Niloy J Mitra, and Peter Wonka,  “Styleflow: Attribute-conditioned exploration of stylegan-generated images using conditional continuous normalizing flows,”  ACM Transactions on Graphics (ToG), vol. 40, no. 3, pp. 1–21, 2021\.
  * [17] Dongchan Min, Dong Bok Lee, Eunho Yang, and Sung Ju Hwang,  “Meta-stylespeech: Multi-speaker adaptive text-to-speech generation,”  in International Conference on Machine Learning. PMLR, 2021, pp. 7748–7759.
  * [18] Yu Zhang, Ron J Weiss, Heiga Zen, Yonghui Wu, Zhifeng Chen, RJ Skerry-Ryan, Ye Jia, Andrew Rosenberg, and Bhuvana Ramabhadran,  “Learning to speak fluently in a foreign language: Multilingual speech synthesis and cross-language voice cloning,”  arXiv preprint arXiv:1907.04448, 2019.
  * [19] Sercan Arik, Jitong Chen, Kainan Peng, Wei Ping, and Yanqi Zhou,  “Neural voice cloning with a few samples,”  Advances in Neural Information Processing Systems, vol. 31, 2018\.
  * [20] Keon Lee, Kyumin Park, and Daeyoung Kim,  “Dailytalk: Spoken dialogue dataset for conversational text-to-speech,”  arXiv preprint arXiv:2207.01063, 2022.
  * [21] Kyubyong Park,  “Kss dataset: Korean single speaker speech dataset,” https://kaggle.com/bryanpark/korean-single-speaker-speech-dataset, 2018\.
  * [22] Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae,  “Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,”  Advances in Neural Information Processing Systems, vol. 33, pp. 17022–17033, 2020.
  * [23] Johannes Wagner, Andreas Triantafyllopoulos, Hagen Wierstorf, Maximilian Schmitt, Felix Burkhardt, Florian Eyben, and Björn W. Schuller,  “Model for Dimensional Speech Emotion Recognition based on Wav2vec 2.0,” Feb. 2022.
  * [24] Reza Lotfian and Carlos Busso,  “Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings,”  IEEE Transactions on Affective Computing, vol. 10, no. 4, pp. 471–483, 2017.


Vol.0 (20xx) No.0, 000–000

11institutetext:  Xinjiang Astronomical Observatory, Chinese Academy of
Sciences, Urumqi 830011, China<EMAIL_ADDRESS>
22institutetext: Key Laboratory of Radio Astronomy, Chinese Academy of
Sciences, Nanjing 210008, China
33institutetext: Shanghai Astronomical Observatory, Chinese Academy of
Sciences, Shanghai 200030, China<EMAIL_ADDRESS>
44institutetext: Max Planck Institute for Radio Astronomy, Auf dem hügel 69,
Bonn 53121, Germany
55institutetext: Instituto Nacional de Astrofísica, Óptica y Electrónica, Luis
Enrique Erro $\\#$ 1, Tonantzintla, Puebla 72840, México
66institutetext: University of Chinese Academy of Sciences, Beijing 100049,
China
Received 20xx month day; accepted 20xx month day

# Resolving the inner jet of PKS 1749+096 with super-resolution VLBA images at
7 mm

Lang Cui 1122 Ru-Sen Lu Corresponding author332244 Wei Yu 33 Jun Liu 44 Víctor
M. Patiño-Álvarez 5544 Qi Yuan 1166

###### Abstract

High resolution imaging of inner jets in Active Galactic Nuclei (AGNs) with
VLBI at millimeter wavelengths provides deep insight into the launching and
collimation mechanisms of relativistic jets. The BL Lac object, PKS 1749+096,
shows a core-dominated jet pointing toward the northeast on parsec-scales
revealed by various VLBI observations. In order to investigate the jet
kinematics, in particular, the orientation of the inner jet on the smallest
accessible scales and the basic physical conditions of the core, in this work
we adopted a super-resolution technique, the Bi-Spectrum Maximum Entropy
Method (BSMEM), to reanalyze VLBI images based on the Very Long Baseline Array
(VLBA) observations of PKS 1749+096 within the VLBA-BU-BLAZAR 7 mm monitoring
program. These observations include a total of 105 epochs covering the period
from 2009 to 2019. We found that the stacked image of the inner jet is limb-
brightened with an apparent opening angle of $50\fdg 0\pm 8\fdg 0$ and $42\fdg
0\pm 6\fdg 0$ at the distance of 0.2 and 0.3 mas (0.9 and 1.4 pc) from the
core, corresponding to an intrinsic jet opening angle of $5\fdg 2\pm 1\fdg 0$
and $4\fdg 3\pm 0\fdg 7$, respectively. In addition, our images show a clear
jet position angle swing in PKS 1749+096 within the last ten years. We discuss
the possible implications of jet limb brightening and the connection of the
position angle with jet peak flux density and gamma-ray brightness.

###### keywords:

galaxies: quasars: individual: PKS 1749+096 — galaxies: jets — radio
continuum: galaxies

##  1 Introduction

High resolution observations of jets in Active Galactic Nuclei (AGNs) with
Very Long Baseline Interferometry (VLBI) show that some objects have regular
or irregular swings in the position angle of the innermost structure (i.e.,
jet wobbling, Agudo et al. 2007; Lu et al. 2012). The physical origin of this
phenomena is currently still not well understood. Possible mechanisms for the
jet wobbling include accretion disk precession, orbital motion of accretion
systems (binary black hole), and jet instabilities (e.g., Agudo 2009). In some
sources, a correlation between the variations of the inner jet position angle
and of the flux density in radio, X-ray, or gamma-ray is observed (e.g., Rani
et al. 2014, and references therein), suggesting a connection between the
inner jet morphology and the corresponding emitting regions.

At a redshift of 0.322 (Stickel et al. 1988), the ultra-luminous BL Lac object
PKS 1749+096 showed strong variability from radio to X-ray regime. In 2016
July, unprecedented bright flaring activity in very high energy gamma-ray
emission, together with X-ray and optical flares were detected (Becerra
Gonzalez et al. 2016; Ciprini et al. 2016; Mirzoyan 2016; Balonek et al.
2016). The radio morphology of PKS 1749+096 is dominated by its mas-scale
structure. Multi-epoch VLBI observations show that position angle swing exists
in its jet (Lu et al. 2012), which is more pronounced towards the core.

To investigate the structure of the innermost region of the jet and its
possible variations, here we report the results of a detailed VLBA imaging
study of PKS 1749+096 at 7 mm with unprecedented resolution. Through this
paper, we adopt the following parameters: the luminosity distance to PKS
1749+096 is $D_{L}$=1674 Mpc and 1 mas of angular separation corresponds to
4.64 pc, with cosmological parameters $\Omega_{M}$=0.27,
$\Omega_{\kappa}$=0.73, and $H_{0}$=71 km$s^{-1}$.

##  2 Archival Data and VLBI Images

For our study, we made use of data from the VLBA-BU-BLAZAR 7 mm monitoring
program (e.g., Jorstad et al. 2017), spanning a time interval from 2009 April
to 2019 October, during which 105 epochs of observations were performed with
an approximately monthly cadence. The typical resolution of the CLEAN images
from these observations is $\sim$0.3 mas and $\sim$0.15 mas in the north-south
and east-west direction, respectively.

Recently, various super-resolution imaging techniques have been developed and
demonstrated for black hole shadow imaging in Sgr A* and M87 and reconstruct
of polarization structures with VLBI polarimetry observations (e.g., Coughlan
& Gabuzda 2016; Chael et al. 2016; Akiyama et al. 2017a, b; Kuramochi et al.
2018, and references therein). These methods have been demonstrated that
better resolution (by a factor of $\gtrsim$ 2) than that is normally achieved
with the traditional CLEAN methods can be obtained. With these techniques, the
obtained resolution at 7 mm is comparable to that obtained with the
traditional CLEAN method with 3mm-VLBI of a similar array.

In order to investigate the fine-scale structure in the core region, where the
jet position angle swing seems to be more pronounced, we adopted a super-
resolution interferometric imaging technique, the Bi-Spectrum Maximum Entropy
Method (BSMEM, Buscher 1994) and re-imaged the inner jet. Applications of this
method to simulated and observed data of M87 show that it produces high-
resolution images with reliable structures (Lu et al. 2014; Kim et al. 2016).
As examples, we show the BSMEM reconstructed images of PKS 1749+096 in 2011
August and 2014 July in Figure 1.

Figure 1: BSMEM images of PKS 1749+096 in 2011 August and 2014 July. For each
image, contours start at 0.5 % of the peak brightness in steps of two.

##  3 Results & Discussion

To show the overall extent of the jet at 7 mm, we produced a “stacked”
uniformly weighted image (Figure 2, left). We created the stacked image by
first restoring the image at each epoch with an identical CLEAN beam whose
dimensions corresponded to the median uniformly weighted beam of all epochs.
We then shifted the convolved image at each epoch to align the fitted
positions of the image peak, and then combined them all with equal weight to
produce an averaged (stacked) image. At 7 mm, the overall VLBI-scale jet is
dominated by the bright core, with an extended emission pointing toward north-
east, consistent with the jet morphology seen at other epochs (Lu et al.
2012).

Figure 2: (Left) Stacked CLEAN image of PKS 1749+096. Before stacking, each
epoch is restored with a common restoring beam of 0.33$\times$0.14 at -40,
corresponding to the median uniformly weighted beam of all epochs. Contours
start at 0.05 % of the peak in steps of two. The color scale is given in
Jy/beam. (Middle) Stacked BSMEM image of PKS 1749+096 at 7 mm. Contours start
at 0.1 % of the peak in steps of two. The color scale is given in Jy/pixel,
with a pixel size of 5$\mu$as. The blue and red lines denote the position of
the slices in the right panel. (Right) The transverse jet intensity profiles
measured along the blue and red lines in the middle panel at core separation
of 0.2 and 0.3 mas, respectively. The dashed line marks $5\sigma$ off-source
image RMS noise level.

###  3.1 Transverse Jet Structure and its Implications

The high-resolution BSMEM images revealed detailed structure of the inner jet
(within $\sim$0.5 mas of the core), which cannot be clearly seen in normal
CLEAN images. A visual inspection of the position angle of the innermost
portion of the jet in these images indicates a variation of the position angle
with time. This suggests that the jet at a given epoch may only occupy a
portion of the whole underlying jet. If this is the case, we can stack the
BSMEM images using all the available epochs to reconstruct the whole jet.
Pushkarev et al. (2017) found that a true jet geometry appears only after
stacking epochs over several years for a considerable fraction of their
studied sample of AGNs. Interestingly, the transverse jet structure of the
stacked BSMEM image of PKS 1749+096 is transversely resolved and shows a limb-
brightened morphology (Figure 2, middle). The limb brightening is most
pronounced in the inner part of the jet ($<$0.4 mas), while the jet farther
out is resolved out and invisible.

By determining the transverse jet width, we can measure the apparent opening
angle of the underlying jet with the caveat that such a stacked opening angle
is different from a single epoch jet opening angle. We found that the apparent
jet opening angle is $50\fdg 0\pm 8\fdg 0$ at the core distance of 0.2 mas
(0.9 pc) and $42\fdg 0\pm 6\fdg 0$ at the core distance of 0.3 mas (1.4 pc)
(Figure 2, right), suggesting that the jet still undergoes collimation on
these scales. We note that Finke (2019) adopted a smaller jet-opening angle of
168 for PKS 1749+096 in a recent study of physical properties for a sample of
blazar radio jets, but this result was obtained based on a single-epoch 15 GHz
observation with lower resolution (Pushkarev et al. 2009). With the measured
apparent jet opening angle ($\alpha_{\rm app}$), the intrinsic jet opening
angle ($\alpha_{\rm int}$) can be calculated according to the relation of
tan($\alpha_{\rm int}/2$)=tan($\alpha_{\rm app}/2$)sin$\theta$ (Pushkarev et
al. 2017), where $\theta$ is the jet viewing angle. Here, we find the
intrinsic jet opening angle is $5\fdg 2\pm 1\fdg 0$ at the core distance of
0.2 mas and $4\fdg 3\pm 0\fdg 7$ at the core distance of 0.3 mas by adopting a
viewing angle of $\theta=5\fdg 6$.

Limb brightening in jet structures has been detected in a couple of nearby
objects, e.g., M87 (Kim et al. 2018), Cygnus A (Boccardi et al. 2016), 3C 84
(Giovannini et al. 2018). Such a stratified structure can be understood as the
result of a fast internal spine and a slower external sheath in the jet (e.g.,
Ghisellini et al. 2005). The detection of limb brightening close to the
central black hole in these sources indicates that the velocity structure
originates from the jet launching site with the central spine component
possibly being associated with the black-hole-powered jet whereas the sheath
component anchored to the accretion disk. For a black hole mass of
$10^{8.34}\,M_{\odot}$ (Zhou & Cao 2009), a core distance of 0.2–0.3 mas
corresponds to 4.5–6.8$\times 10^{5}$ Schwarzschild Radii (deprojected, for a
viewing angle of $\theta=5.6^{\circ}$) in PKS 1749+096. In this context, it
may seem surprising to detect a limb-brightened underlying structure
originated from the jet base. On the other hand, a velocity structure could
also originate from the interaction of the jet with the ambient medium driven
by Kelvin-Helmholtz instabilities of the jet (e.g., Rossi et al. 2008). In
this scenario, the ambient/jet density contrast is a key parameter determining
the instability evolution and the entrainment properties of the external
medium. Future higher resolution observations would help distinguish between
these two scenarios.

###  3.2 Wobbling of the Inner Jet

In some jetted sources, correlated variations between the inner jet position
angle and flux density in radio, X-rays, and gamma-rays have been observed
(e.g., Liu et al. 2012; Rani et al. 2014, and references therein). Rani et al.
(2014) found a significant correlation between the Fermi-LAT (Large Area
Telescope) gamma-ray flux variations and the position angle variations in the
VLBI jet of S5 0716+714. To explain this correlation, the authors proposed a
scenario where a moving shock propagating down a (bending) relativistic jet
causes significantly increased emission, when the gamma-ray emission and
orientation of the jet flow share the same boosting cone, a correlation
between the two is expected.

Figure 3: Time evolution of the position angle (PA) of the innermost jet
within 0.3 mas of the core (top) and peak flux density (middle), and weekly
averaged gamma-ray flux light curve in the energy range from 0.1 to 300 GeV.
Vertical lines mark the time of the big gamma-ray flare in 2016 July (red
line) and three major gamma-ray brightening events after it.

In Figure 3 (top panel), we show the time evolution of the position angle of
the innermost jet. In calculating the position angle, we first determined the
jet ridgeline, which is the line connecting the maxima of the jet brightness
profiles measured transversally to the local jet direction. Then the mean
position angle of evenly spaced points on the jet ridgeline within 0.3 mas of
the image peak was used to represent the inner jet position angle. The
majority of the position angles range between -30∘ and 20∘. Over the time
scale covered by the observations presented here, it is clear that no regular
or periodic position angle changes exist. Rather, it seems that the inner jet
shows predominantly slower changes in its direction but with occasionally
erratic features (e.g., the rapid swing during 2011–2012 and 2016). Lister et
al. (2013) studied the jet orientation with VLBA monitoring observations at 2
cm and they showed that PKS 1749+096 is one of their sample of sources showing
oscillatory behavior in the inner jet position angle with a fitted period of
12 years. Figure 3 (middle panel) shows the time evolution of the jet peak
flux density. In order to investigate the possible correlation between the
inner jet position angle and the peak flux density, we calculated the
Kendall’s correlation coefficient ($\tau$) and found $\tau$ is 0.26 with a
p-value of $7.2\times 10^{-5}$, indicating a weak correlation between them.

Figure 3 (bottom panel) shows the Fermi-LAT gamma-ray light curve of PKS
1749+096. The Fermi-LAT gamma-ray data were used in the energy range from 0.1
to 300 GeV, and processed using the Fermitools version 1.0.2. We analyzed a
15-degree-radius region of interest around the position of 1749+096, and added
all sources within, which were extracted from the 4FGL catalogue (Abdollahi et
al. 2020). The normalization factors and spectral parameters for sources
within 5 degrees were kept free; while on sources farther than 5 degrees only
their normalization parameter is free to vary. We also applied up-to-date
diffuse and isotropic background models along with the current set of
instrument response functions. We binned the data in 7-days intervals to
increase the signal to noise ratio without compromising temporal resolution.
Only the data points with a Test Statistic (TS) value greater than 25
(significant detection) were used.

Since PKS 1749+096 was only intermittently detectable by the Fermi-LAT and
weakly variable most of the time during the last ten years, a correlation
analysis for the gamma-ray flux and inner jet position angle yields a
Kendall’s $\tau$ of -0.11 with a p-value of 0.02. This indicates that there
are no strong evidence for a correlation between the orientation of the inner
jet and gamma-ray flux, which suggests that the gamma-ray emission in PKS
1749+096 is not driven by orientation-dependent effects, consistent with the
recent findings for radio galaxies and blazars (Angioni et al. 2019, and
references therein). We note that PKS 1749+096 showed unprecedented flares in
2016 July in multiple wavebands, including optical, X-rays, and gamma-rays
(Balonek et al. 2016; Ciprini et al. 2016; Mirzoyan 2016; Schüssler et al.
2017) and the source significantly brightened a few (three) times in gamma-
rays after the major flare in 2016 July. Figure 3 marks the time of these
events with accompanying jet position angle measurements. Interestingly, these
events seemed to occur when the inner jet was along a similar position angle
($\sim$-10∘), although no significant gamma-ray flux increase was seen for
similar jet position angles prior to 2016. Future observations with more
gamma-ray flux measurements would allow further investigation of their
possible connection to the inner jet orientation.

##  4 Summary

In this work, we presented results of a detailed VLBA imaging study of PKS
1749+096 at 7 mm with unprecedented resolution based on a Maximum Entropy
Method technique. We found that the stacked image of the inner jet has a limb-
brightened structure with apparent opening angles of $50\fdg 0\pm 8\fdg 0$ and
$42\fdg 0\pm 6\fdg 0$ at the distance of 0.2 and 0.3 mas (0.9 and 1.4 pc) from
the core, corresponding to an intrinsic jet opening angle of $5\fdg 2\pm 1\fdg
0$ and $4\fdg 3\pm 0\fdg 7$, respectively. The double peaked transverse jet
profile is suggestive of an underlying “spine-sheath” structure possibly
related to the jet launching process or the interaction of the jet with its
ambient medium. We also found that the jet position angle in PKS 1749+096
shows a clear swing phenomenon within the last 10 years, which weakly
correlates with the peak brightness. The lack of correlation between the inner
jet position angle and gamma-ray flux suggests that the gamma-ray emission in
PKS 1749+096 is not sensitive to orientation-dependent effects.

###### Acknowledgements.

We thank the anonymous referee for valuable comments and suggestions. This
study made use of 43 GHz VLBA data from the VLBA-BU Blazar Monitoring Program
(VLBA-BU-BLAZAR; http://www.bu.edu/blazars/VLBAproject.html), funded by NASA
through the Fermi Guest Investigator Program. The VLBA is an instrument of the
National Radio Astronomy Observatory. The National Radio Astronomy Observatory
is a facility of the National Science Foundation operated by Associated
Universities, Inc. L. C. thanks support from the National Key R&D Program of
China (grant No. 2018YFA0404602) and the CAS ’Light of West China’ Program
(grant No. 2018-XBQNXZ-B-021). R.-S. Lu and V. M. Patiño-Álvarez are supported
by the Max Planck Partner Groups at SHAO and INAOE, respectively. R.-S. Lu
acknowledges the support by the Key Program of the National Natural Science
Foundation of China (NSFC grant No. 11933007) and the Research Program of
Fundamental and Frontier Sciences, CAS (grant No. ZDBS-LY-SLH011). This work
is also supported in part by NSFC under grant No. 61931002 & U1731103 and the
Youth Innovation Promotion Association of the CAS under grant No. 2017084.

## References

  * Abdollahi et al. (2020) Abdollahi, S., Acero, F., Ackermann, M., et al. 2020, ApJS, 247, 33
  * Agudo et al. (2007) Agudo, I., Bach, U., Krichbaum, T. P., et al. 2007, A&A, 476, L17
  * Agudo (2009) Agudo, I. 2009, in Astronomical Society of the Pacific Conference Series Vol. 402, Approaching Micro-Arcsecond Resolution with VSOP-2: Astrophysics and Technologies, ed., Y. Hagiwara, E. Fomalont, M. Tsuboi, & M. Yasuhiro, 330 (astro-ph/0803.3388)
  * Akiyama et al. (2017a) Akiyama, K., Kuramochi, K., Ikeda, S., et al. 2017, ApJ, 838, 1
  * Akiyama et al. (2017b) Akiyama, K., Ikeda, S., Pleau, M., et al. 2017, AJ, 153, 159
  * Angioni et al. (2019) Angioni, R., Ros, E., Kadler, M., et al. 2019, A&A, 627, A148
  * Balonek et al. (2016) Balonek, T. J., Stahlin, R. W., Chapman, K. J., et al. 2016, The Astronomer’s Telegram, 9259
  * Becerra Gonzalez et al. (2016) Becerra Gonzalez, J., Thompson, D., & Fermi-LAT Collaboration 2016, The Astronomer’s Telegram, 9231
  * Boccardi et al. (2016) Boccardi, B., Krichbaum, T. P., Bach, U., Bremer, M., & Zensus, J. A. 2016, A&A, 588, L9
  * Buscher (1994) Buscher, D. F. 1994, in IAU Symposium Vol. 158, Very High Angular Resolution Imaging, ed., J. G. Robertson & W. J. Tango, 91
  * Chael et al. (2016) Chael, A. A., Johnson, M. D., Narayan, R., et al. 2016, ApJ, 829, 11
  * Ciprini et al. (2016) Ciprini, S., Gonzalez, J. B., Pivato, G., & Thompson, D. J. 2016, The Astronomer’s Telegram, 9260
  * Coughlan & Gabuzda (2016) Coughlan, C. P., & Gabuzda, D. C. 2016, MNRAS, 463, 1980
  * Finke (2019) Finke, J. R. 2019, ApJ, 870, 28
  * Ghisellini et al. (2005) Ghisellini, G., Tavecchio, F., & Chiaberge, M. 2005, A&A, 432, 401
  * Giovannini et al. (2018) Giovannini, G., Savolainen, T., Orienti, M., et al. 2018, Nature Astronomy, 2, 472
  * Jorstad et al. (2017) Jorstad, S. G., Marscher, A. P., Morozova, D. A., et al. 2017, ApJ, 846, 98
  * Kim et al. (2016) Kim, J.-Y., Lu, R.-S., Krichbaum, T. P., et al. 2016, Galaxies, 4, 39
  * Kim et al. (2018) Kim, J.-Y., Krichbaum, T. P., Lu, R.-S., et al. 2018, A&A, 616, A188
  * Kuramochi et al. (2018) Kuramochi, K., Akiyama, K., Ikeda, S., et al. 2018, ApJ, 858, 56
  * Liu et al. (2012) Liu, X., Mi, L.-G., Liu, B.-R., & Li, Q.-W. 2012, Ap&SS, 342, 465
  * Lister et al. (2013) Lister, M. L., Aller, M. F., Aller, H. D., et al. 2013, AJ, 146, 120
  * Lu et al. (2012) Lu, R.-S., Shen, Z.-Q., Krichbaum, T. P., et al. 2012, A&A, 544, A89
  * Lu et al. (2014) Lu, R.-S., Broderick, A. E., Baron, F., et al. 2014, ApJ, 788, 120
  * Mirzoyan (2016) Mirzoyan, R. 2016, The Astronomer’s Telegram, 9267
  * Pushkarev et al. (2009) Pushkarev, A. B., Kovalev, Y. Y., Lister, M. L., et al. 2009, A&A, 507, L33
  * Pushkarev et al. (2017) Pushkarev, A. B., Kovalev, Y. Y., Lister, M. L., & Savolainen, T. 2017, MNRAS, 468, 4992
  * Rani et al. (2014) Rani, B., Krichbaum, T. P., Marscher, A. P., et al. 2014, A&A, 571, L2
  * Rossi et al. (2008) Rossi, P., Mignone, A., Bodo, G., et al. 2008, A&A, 488, 795
  * Schüssler et al. (2017) Schüssler, F., Seglar-Arroyo, M., Arrieta, M., et al., 2017, in Proceedings of Science, Vol. 301, 35th International Cosmic Ray Conference–ICRC2017 (Busan, Korea), id.652 (astro-ph/1708.01083)
  * Stickel et al. (1988) Stickel, M., Fried, J. W., & Kuehr, H. 1988, A&A, 191, L16
  * Zhou & Cao (2009) Zhou, M., & Cao, X.-W. 2009, , 9, 293