# From Centralized to Self-Supervised: Pursuing Realistic Multi-Agent
Reinforcement Learning

Violet Xiang Logan Cross Jan-Philipp Fränken Nick Haber
Stanford University Corresponding author<EMAIL_ADDRESS>
###### Abstract

In real-world environments, autonomous agents rely on their egocentric
observations. They must learn adaptive strategies to interact with others who
possess mixed motivations, discernible only through visible cues. Several
Multi-Agent Reinforcement Learning (MARL) methods adopt centralized approaches
that involve either centralized training or reward-sharing, often violating
the realistic ways in which living organisms, like animals or humans, process
information and interact. MARL strategies deploying decentralized training
with intrinsic motivation offer a self-supervised approach, enable agents to
develop flexible social strategies through the interaction of autonomous
agents. However, by contrasting the self-supervised and centralized methods,
we reveal that populations trained with reward-sharing methods surpass those
using self-supervised methods in a mixed-motive environment. We link this
superiority to specialized role emergence and an agent’s expertise in its
role. Interestingly, this gap shrinks in pure-motive settings, emphasizing the
need for evaluations in more complex, realistic environments (mixed-motive).
Our preliminary results suggest a gap in population performance that can be
closed by improving self-supervised methods and thereby pushing MARL closer to
real-world readiness.

##  1 Introduction

In real-world environments, agents’ ability to successfully operate depends on
how they navigate and interpret interactions with other agents and entities
they encounter. The dynamics of these interactions are subject to change over
time and vary among individuals, as different agents may have divergent or
mixed motives influencing their behavior. For instance, vendors in a farmer’s
market both compete for customers and cooperate with nearby stalls for shared
resources. Their interactions shift based on immediate needs, like maximizing
sales or maintaining good relations, showcasing the fluid nature of mixed
motives in real-world settings. These interactions are often driven by
underlying motivations, social norms, and evolving relationships.

For agents to be efficient, they must not only witness but also comprehend and
adjust to these intricate dynamics. Existing multi-agent system solutions
predominantly employ centralized strategies, as illustrated in the left panel
of Fig 1. These can involve a centralized training through sharing parameters
(Lowe et al., 2017; Yu et al., 2022) or sharing information that individual
agents would not typically access without explicit communication or insights
into other agents’ non-observable information (Hughes et al., 2018; McKee et
al., 2020, 2021; Baker, 2020). Centralized methods leverage shared information
among a cohort of agents to promote coordination. However, the real world
places agents together with AI entities beyond their training cohort or even
humans, underscoring the need for self-supervised learning, akin to biological
systems. In such settings, agents have limited information, relying on
observable cues to infer states, objectives, and rewards of others. To make
optimal decisions, they must discern the intentions of others and group
dynamics based solely on this observable information.

Figure 1: Illustration of different MARL approaches. Centralized Training:
Agents share parameters, either policy or critic. Reward sharing: Method 1:
Agents distribute their rewards amongst themselves to influence behavior.
Method 2: Agents are aware of others’ rewards and adjust their actions
accordingly. An added term to their reward, such as social value orientation,
serves as an intrinsic motivation, determining if an agent values the success
of others over its own. Self-supervised: These approaches operate with minimal
assumptions, as agents must infer hidden information about other agents solely
from their partially observable egocentric observations.

####

This observational challenge accentuates the importance of adaptive, self-
supervised strategies in realistic multi-agent settings, illustrated in Fig
1b. A cornerstone of self-supervised approaches is intrinsic motivation, a
force that drives actions without external rewards, especially crucial in
sparse-reward environments (Pathak et al., 2017). Methods using curiosity as
intrinsic motivation promote adaptability across diverse tasks and
environments by allowing an agent to seek more information about its
environment by exploring unfamiliar states and improve its model of the
environment (Pathak et al., 2017; Haber et al., 2018; Burda et al., 2018;
Pathak et al., 2019; Kim et al., 2020; Kauvar et al., 2023). While these
epistemic intrinsic motivations effectively drive agents to improve their
model of the environment, they might not always offer definitive guidance on
how agents should behave in relation to others, i.e., helping or undermining
others. (Ndousse et al., 2021) show that integrating an model-based auxiliary
loss, allowing agents to predict upcoming states from observations, promotes
social learning in certain environment even without intrinsic motivation.
Moreover, model-based intrinsic motivations that emphasize interactions with
other agents provide insights into how agents can better model peer behaviors.
Such motivations involve predicting other agents’ actions or states, enhancing
an agent’s ability to align its policy with the behaviors of surrounding
entities. These motivations, whether for exploration, or social influence,
guide decisions and interactions, pushing individuals to cooperate, compete,
or even conform in search of social equilibrium (Jaques et al., 2019; Ma et
al., 2022).

While studies have explored centralized, epistemic, and social intrinsic
motivations, there is a clear need for a head-to-head comparison within a
consistent environment and agent setup. Conducting such an evaluation in a
uniform setting isolates the impact of each motivation type, ensuring
performance or behavioral differences arise from the motivations themselves
and not extraneous factors. Our research bridges this gap. We provide a direct
comparison in two distinct environments: one with mixed-motive dynamics and
another with pure-motive dynamics, all within a low sample complexity setting.
Evaluating in this particular regime ensures readiness for the real world
which is characterized by its unpredictable nature making rapid adaptability a
paramount feature. Agents capable of learning efficiently from minimal samples
can swiftly adapt to emerging challenges without the requirement of extensive
retraining. Similar to biological organisms, agents in real-world scenarios
would benefit immensely from continual and lifelong learning, where the
ability to learn and adapt consistently over extended periods without vast new
data influxes is invaluable. This sheds light on performance differences and
population composition, highlighting the promise of intrinsic motivation in
autonomous agent design. Our contributions are twofold:

  1. 1.

We identify the disparities among centralized and self-supervised MARL models,
highlighting areas for future research.

  2. 2.

We discern the factors behind the population outcome of various models,
identifying a critical need for specialized roles and individual agents’
competence in fulfilling those roles.

##  2 Background

#### Centralized MARL

Centralized Multi-Agent Reinforcement Learning (MARL) often trains agents in a
collective, targeting optimal performance. Method by (Lowe et al., 2017),
individualizes agent policies but centralizes training data with a shared
critic. Method by(Yu et al., 2022) shares a policy network across agents,
optimizing actions for aggregate returns. Some strategies permit decentralized
learning but demand unrealistic data access. For example, (McKee et al., 2021)
uses a reputation-based motivation, specific to the Clean Up environment,
thereby constraining its versatility. (McKee et al., 2020) applies ’social
value orientation’ (SVO) to allocate altruistic or selfish tendencies. Here,
each agent’s SVO is set in a diverse population, where the SVO reflects a
balance between maximizing an agent’s own reward and improving rewards for
fellow agents. While this fosters cooperation, it requires agents to know
peers’ rewards, a limitation somewhat sidestepped by (Li et al., 2023) through
allowing agents to distribute their own reward, but at the cost of explicit
inter-agent communication. Our focus remains on agents understanding peers
strictly via direct observations.

#### Intrinsic motivation in single and multi-agent reinforcement learning

Intrinsic motivation guides agents to effectively navigate open-ended
environments (Oudeyer and Kaplan, 2007; Schmidhuber, 1991). One approach is to
set and achieve self-generated sub-goals coinciding with environment tasks
(Lair et al., 2019; Campero et al., 2020; Forestier et al., 2022; Colas et
al., 2022). Another approach, empowerment (Klyubin et al., 2005), increases an
agent’s control over the environment by maximizing the mutual information
between state and action (Choi et al., 2021). Curiosity is a different form of
intrinsic motivation promoting information-seeking behavior in an agent.
Building on curiosity, (Pathak et al., 2017) rewards exploration based on
forward prediction losses. (Pathak et al., 2019) targets states with high
uncertainty, and (Kim et al., 2020) refines agent models with a curiosity
signal. Extending to multi-agent settings, we analyze with ICM motivation.
Besides these epistemic motivations, there exist motivations tailored to the
social intricacies of multi-agent interactions. (Jaques et al., 2019)
introduces social influence as a motivation, prompting agents to increase
their influence on others’ actions. An agent’s social influence rewarded is
defined by the difference between other agents taking certain actions given
its action and without its action. (Ma et al., 2022) promotes spontaneous
coordination by motivating them to behave in ways that are most expected by
other agents using only local observations and actions. Though these methods
vary, direct comparisons in a unified setting are lacking. Our research offers
this comparison, highlighting the strengths and weaknesses of each motivation.

##  3 Experimental Settings

###  3.1 Environment

To evaluate model performance across varied social dynamics, we selected both
mixed-motive (Cleanup) and pure-motive (Harvest) environments as proposed by
(Hughes et al., 2018). Examples of these environments are shown in Fig 2 111We
adapted environment implementation from this repository. Unlike the standard
matrix games in game theory, these environments utilize spatially and
temporally extended games with embodied avatars.

Figure 2: Clean Up (left) map size is 25x18. Harvest (right) map size is
38x16.

In Clean Up (Fig 2 left), agents are rewarded for consuming apples in an
environment with a polluted river. Initially, apples are not available on the
map until agents start to clean the river. Apple growth is tied to river
cleanliness, with growth rates dropping as pollution increases. Additionally,
the apple growth area is distant from the river, requiring agents to balance
between cleaning and apple consumption. This poses an challenging dynamic as
when they are multiple agents in the environment, agents will have conflicting
motivations (consuming apples vs cleaning the river). In Harvest (Fig 2
right), agents are rewarded for apple consumption in an environment populated
with apples. Apple growth relies on the proximity of other apples, and over-
harvesting leads to non-regeneration. This environment thereby symbolizes the
tragedy of the commons. All models receive inputs in the form of 15x15 square
observations centered around the agents. The policy network for every agent
consists a convolutional neural network (CNN) followed by a gated recurrent
unit (GRU) network. The ICM, ICM-reward, and social influence agents
incorporate distinct world model components for their respective predictions.
For ICM and ICM-reward agents, a separate CNN encoder, coupled with a GRU,
predicts environmental dynamics. For social influence agents we follow the
setup in the original paper (Jaques et al., 2019) by sharing the same CNN
encoder as the policy and adding a separate head with two MLP layers and a GRU
as the MOA model. Differing from only giving one or a few agents with the
ability to influence other agents, we allow all agents in a population to be
rewarded for influencing other agents visible to them. Each model is trained
with episode length of 2000 steps (Clean Up) or 1000 steps (Harvest), and
Bayesian hyper-parameter search is performed for each model to ensure the best
performance. The results in the section 4 are from the best hyperparameters
for each model.

###  3.2 Evaluated Models

A promising model should allow agents to discern optimal strategies depending
on the environment structure, constraints, and emergent social dynamics. To
capture these constraints in a realistic setting, we formulate a Markov game
for $K$ agents as $<S,A,R,P>$. Here:

  * •

$S$ represents the environment state.

  * •

$A$ denotes the joint action set of all agents.

  * •

$R$ is the reward function, defined as $R(S,A)=\\{r_{1},...,r_{K}\\}$.

  * •

$P$ outlines the state transition dynamics, given by $P(S,A)=S^{\prime}$.

At each timestep $t$, an agent $i$ accesses its state $s_{i}^{t}$ via
observation $o_{i}^{t}$, takes an action $a_{i}^{t}$, and receives reward
$r_{i}^{t}$. Additionally, the agent observes actions $a_{j}^{t}$ of other
agents within its neighborhood, defined by $N(i)$. We evaluated these models:

  * •

Independent PPO (IPPO) agents each have their own policy and value network
trained through the proximal policy optimization (PPO) algorithm (Schulman et
al., 2017), targeting the maximization of individual extrinsic rewards from
the environment. They serve as solvers for other models with intrinsic
rewards.

  * •

MAPPO agents use the same solver as the IPPO agents while sharing a single
value and policy network (Yu et al., 2022). The policy network is trained with
trajectories collected from all agents to provide actions based on individual
agents’ local observations, and the value network is trained using the global
state.

  * •

ICM agents each contains a policy parameterized by ${\theta}$ and a world
model parameterized by ${\phi}$ computing forward and inverse dynamics. The
world model $f_{\phi}$ is trained by simultaneously minimizing a forward
prediction loss

$\displaystyle
L_{forward}=\|\mathbf{f_{\phi}(o_{i}^{t},a_{i}^{t})}-\mathbf{o_{i}^{t+1}}\|$
(1)

and an inverse dynamics through separate heads

$\displaystyle
L_{inverse}=\mathbf{cross\\_entropy}(f_{\phi}(o_{i}^{t},o_{i}^{t+1}),a_{i}^{t})$
(2)

Each agent learns its policy based on the following reward formulation using
the forward dynamics loss as an intrinsic reward as it is relevant for policy
exploration.

$\displaystyle r_{i}=r_{i_{ext}}+\alpha r_{i_{int}}\text{ where
}r_{i_{int}}=L_{forward}$ (3)

  * •

ICM-reward agents are trained similarly as the ICM agents mentioned above with
an extra loss for the dynamic model to predict the reward

$\displaystyle
L_{reward}=\|\mathbf{f_{\phi}(o_{i}^{t},a_{i}^{t})}-\mathbf{r_{i}^{t+1}}\|$
(4)

while the policy is optimized with this reward

$\displaystyle r_{i}=r_{i_{ext}}+\alpha r_{i_{int}}\text{ where
}r_{i_{int}}=L_{reward}$ (5)

  * •

Social influence agents: These agents aim to optimize their influence on other
agents’ behaviors. Each agent’s Model of Agents (MOA) predicts actions of
peers based on current observations and prior actions:

$\displaystyle
L_{moa}(i)=\mathbf{cross\\_entropy(f(o_{i}^{t},A^{t-1}),A_{j\neq
i}^{t}})\texttt{ where }A=\\{a_{j}\texttt{ for }j\in N(i)\\}$ (6)

The MOA is used to compute the influence reward by computing the discrepancy
between the marginal policy of other agents and the conditional policy of
other agents given the self agent’s action. Due to the constraint of strict
partial observability, agents use their observation to compute their influence
over all other agents that are within their visibility radius. This
incentivizes agents to increase their action’s causal influence over other
agents’ actions which could lead to coordinated behaviors even without
specific population-level control. This is computing the intrinsic reward:

$\displaystyle c_{i}^{t}=\sum_{j\neq
i}^{K}(D_{KL}[\mathbf{p(a^{t}_{j}|a^{t}_{i},o^{t}_{i})\|p(a^{t}_{j}|o^{t}_{i})}])$
(7)

Individual agents’ policy is then optimized using this reward formulation

$\displaystyle r_{i}=r_{ext}+\alpha r_{int}\text{ where }r_{int}=c_{i}$ (8)

  * •

SVO agents employ a centralized method, emphasizing reward sharing, which
requires agent awareness of others’ rewards at every step. SVO modulates
reward preferences through sampling SVO values from a preset distribution, and
each agent’s SVO determines their inclination to optimize others’ rewards
versus their own. The higher its SVO, the more altruistic an agent is.

$\displaystyle SVO^{t}_{i}=arctan(\frac{\frac{1}{N-1}\sum_{j\neq
i}^{K}r^{t}_{j}}{r^{t}_{i}})$ (9)

Given the pre-specified SVO value ($SVO_{i}^{{}^{\prime}}$) sampled from the
distribution $N(\mu,\sigma)$ for each agent $i$, its policy is optimized using
such reward

$\displaystyle
r^{t}_{i}=r_{ext}-\alpha|SVO^{{}^{\prime}}_{i}-\mathbf{clip}(SVO^{t}_{i})|$
(10)

where the clip function bounds $SVO^{t}_{i}$ within [0, $\pi/2$]. We evaluate
both homogeneous populations (SVO-HO) with consistent SVO values, and
heterogeneous ones (SVO-HE) with diverse SVOs sampled from a distribution.

##  4 Results

Figure 3: Clean Up Results. (a) Population return and equity score computed as
1 - Gini Coefficient, (b) Return and equity by SVO values. The error bars are
standard error.

We trained populations of five agents for each model and evaluated them across
five episodes. The agents are trained up to one million steps but the best
performing epoch, defined as every 5000 environment steps, was selected for
each model (specific environment number are available in supplementary
material 6). Average population (Fig 2a) and equity is depicted in Fig 2a
right panel. To evaluate how equitable the return is across the 5 agents, the
Gini coefficient is computed as
$\frac{\sum_{i=1}^{k}\sum_{k=1}^{k}|r_{i}-r_{j}|}{2m\sum_{i=1}^{k}}$ as in
(Leibo et al., 2021) with a minor difference in handling negative reward
values. Instead of only using positive reward, we shift the minimum negative
reward to zero plus a small $\delta=0.0001$ to ensure that negative returns
also contribute to equity computation.

#### Clean Up

Centralized methods (SVO-HE, SVO-HO, MAPPO) are better than the all other
self-supervised methods as shown in Fig 3a. SVO-HE outperforms other
centralized methods by a large margin. As expected, baseline IPPO agents
exhibited the weakest performance. Agents with intrinsic motivations showed a
marginal improvement over the IPPO baseline. Interestingly, models using
explicit social intrinsic motivation did not surpass those using non-social
intrinsic motivations as ICM performance is very comparable to Social
Influence model as show in Fig 3a. While achieving high rewards is a priority,
it is vital to ensure minimal disparities among agents’ payouts in the
environment. From Fig 3a, it is evident that the SVO-HO model achieved the
highest Gini coefficient scores and thus the most equitable reward
distribution. However, its overall performance, considering the population
returns, was outperformed by the SVO-HE model which has more altruistic
tendency in the optimal SVO hyperparameter and a diverse population.

Two variations of the SVO models are depicted in Fig 3a heterogeneous
population, where the SVO values for five agents are drawn from an Gaussian
distribution with parameters $\mu=75\degree,\sigma=11.9\degree$, which is very
altrusitic, and a homogeneous population featuring five agents with
$30\degree$ SVO values. These values were selected by a hyperparameter search.
The heterogeneous SVO agents outperformed their homogeneous counterparts,
aligning with previous findings (McKee et al., 2020) that suggest diverse SVO
populations excel in mixed-motive settings compared to purely cooperative
groups. To delve deeper into the influence of population SVO configurations,
we analyzed their impact on population returns and equity (Fig 3b).
Interestingly, altruism levels do not directly correlate with returns or
equity. Analyzing the population return data, we observed that when agents
converge to a high-performing population, they excel; otherwise, performance
tends to be suboptimal, suggesting that hyperparameter selection is crucial.

Figure 4: (a) Correlation ($r=0.88$) between the number of waste cleaned
(x-axis) and the population reward (y-axis). (b) z-scored apple eaten (x-axis)
vs waste cleaned (y-axis), this divides agents into four types by quadrant:
top-right Eat More & Clean More, top-left Eat Less & Clean Less, bottom-left
Eat Less & Clean Less and Eat More & Clean Less. (c) Roles in different
population. Each role is defined by which one of the four quadrant an agent is
fallen into on (b).

To discern the factors impacting the performance of different models, we
analyze the relationship of cooperative behavior with population return. Fig
4a reveals a direct correlation ($r=0.88$) between waste cleaned and rewards,
suggesting that it is advantageous for everyone when a group collectively
cleans waste. We categorized agents based on their apple consumption and waste
cleaning relative to the population average (Fig 4b&c). Although one IPPO
agent outperformed its peers, the IPPO population lagged behind SVO-HE, ICM,
and Social Influence due to three underperforming agents. Interestingly, ICM
and SVO-HE both had three agents prioritizing cleaning and two focusing on
eating. However, SVO-HE agents were more proficient and specialized, as
evident in Fig 4a,c, as agents were split between eating and cleaning roles.
This underscores that a population’s effectiveness hinges on agents
specializing in roles and excelling in them.

#### Harvest

We also evaluated all models in Harvest, a pure motive environment distinct
from Clean Up where every agent has an incentive to maintain sustainable
eating. In Harvest, dense apples are readily available without any preliminary
tasks, such as cleaning the river in Clean Up, making it easier to achieve
higher population returns (Fig 2). Fig 5 shows SVO-HO being the best model,
however, there is no substantial difference between SVO models and others
while MAPPO agents perform significantly worse than other model. This
highlights the need for evaluation in environments with mixed-motive social
dynamics. Notably, the optimal SVO parameter in a heterogeneous population
differ between Harvest ($\mu=15$) and Clean Up ($\mu=75$), highlighting the
challenge of pre-specifying a population across environments. An ideal model
should excel in both settings with the same hyperparameter, adapting its
policy through agent interactions. Our study indicates the need for developing
such versatile models.

Figure 5: Harvest Results. (a) Population return and equity score computed as
1 - Gini Coefficient, (b) Return and equity by SVO values. The error bars are
standard error.

##  5 Discussion

We assess various multi-agent models in two environments, each possessing
distinct social dynamics. Centralized approaches surpassed self-supervised
approaches in the mixed-motive setting Clean Up but depends on pre-set
population parameters. When transitioning to a pure motive environment, these
optimal parameters shift. This suggests that population reconfiguration is
needed if environmental dynamics change, highlighting its limitation in
allowing agents to flexibly achieve optimal performance. Additionally, this
method relies on reward sharing mechanisms that may not be possible in every
environment. In Clean Up, a notable gap exists between agents driven by self-
supervised intrinsic motivations and those trained with centralized methods.
Other social intrinsic motivations might bridge this gap. The analysis of
different population composition in Clean Up further suggests the
effectiveness of a multi-agent population is strongly influenced by agents’
role specialization and their proficiency in those roles. The similar
performance of methods in Harvest indicates the importance of evaluation in
mixed-motive environments. Due to resource constraints these models are
trained under a low-sample regime. A major current limitation in MARL is the
large number of samples that is required for populations to converge and reach
equilibrium seen in cooperative dynamics. This provides additional motivation
for investing in methods that can be flexible across environments without the
required population pre-configuration and extensive hyper-parameter searches.
Additionally methods need to be adaptable whenever there is a dynamics change
in the environment. Despite these challenges, our results underscore the gap
between self-supervised and centralized methods and suggests improving self-
supervised MARL methods as a meaningful future direction.

## References

  * Lowe et al. [2017] Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch.  Multi-agent actor-critic for mixed cooperative-competitive environments.  _Advances in neural information processing systems_ , 30, 2017.
  * Yu et al. [2022] Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu.  The surprising effectiveness of ppo in cooperative multi-agent games.  _Advances in Neural Information Processing Systems_ , 35:24611–24624, 2022.
  * Hughes et al. [2018] Edward Hughes, Joel Z Leibo, Matthew Phillips, Karl Tuyls, Edgar Dueñez-Guzman, Antonio García Castañeda, Iain Dunning, Tina Zhu, Kevin McKee, Raphael Koster, et al.  Inequity aversion improves cooperation in intertemporal social dilemmas.  _Advances in neural information processing systems_ , 31, 2018.
  * McKee et al. [2020] Kevin R McKee, Ian Gemp, Brian McWilliams, Edgar A Duéñez-Guzmán, Edward Hughes, and Joel Z Leibo.  Social diversity and social preferences in mixed-motive reinforcement learning.  _arXiv preprint arXiv:2002.02325_ , 2020.
  * McKee et al. [2021] Kevin R McKee, Edward Hughes, Tina O Zhu, Martin J Chadwick, Raphael Koster, Antonio Garcia Castaneda, Charlie Beattie, Thore Graepel, Matt Botvinick, and Joel Z Leibo.  A multi-agent reinforcement learning model of reputation and cooperation in human groups.  _arXiv e-prints_ , pages arXiv–2103, 2021.
  * Baker [2020] Bowen Baker.  Emergent reciprocity and team formation from randomized uncertain social preferences.  _Advances in neural information processing systems_ , 33:15786–15799, 2020.
  * Pathak et al. [2017] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell.  Curiosity-driven exploration by self-supervised prediction.  In _International conference on machine learning_ , pages 2778–2787. PMLR, 2017.
  * Haber et al. [2018] Nick Haber, Damian Mrowca, Stephanie Wang, Li F Fei-Fei, and Daniel L Yamins.  Learning to play with intrinsically-motivated, self-aware agents.  _Advances in neural information processing systems_ , 31, 2018.
  * Burda et al. [2018] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov.  Exploration by random network distillation.  _arXiv preprint arXiv:1810.12894_ , 2018.
  * Pathak et al. [2019] Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta.  Self-supervised exploration via disagreement.  In _International conference on machine learning_ , pages 5062–5071. PMLR, 2019.
  * Kim et al. [2020] Kuno Kim, Megumi Sano, Julian De Freitas, Nick Haber, and Daniel Yamins.  Active world model learning with progress curiosity.  In _International conference on machine learning_ , pages 5306–5315. PMLR, 2020.
  * Kauvar et al. [2023] Isaac Kauvar, Chris Doyle, Linqi Zhou, and Nick Haber.  Curious replay for model-based adaptation.  2023\.
  * Ndousse et al. [2021] Kamal K Ndousse, Douglas Eck, Sergey Levine, and Natasha Jaques.  Emergent social learning via multi-agent reinforcement learning.  In _International Conference on Machine Learning_ , pages 7991–8004. PMLR, 2021.
  * Jaques et al. [2019] Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Z Leibo, and Nando De Freitas.  Social influence as intrinsic motivation for multi-agent deep reinforcement learning.  In _International conference on machine learning_ , pages 3040–3049. PMLR, 2019.
  * Ma et al. [2022] Zixian Ma, Rose Wang, Fei-Fei Li, Michael Bernstein, and Ranjay Krishna.  Elign: Expectation alignment as a multi-agent intrinsic reward.  _Advances in Neural Information Processing Systems_ , 35:8304–8317, 2022.
  * Li et al. [2023] Wenhao Li, Xiangfeng Wang, Bo Jin, Jingyi Lu, and Hongyuan Zha.  Learning roles with emergent social value orientations.  _arXiv preprint arXiv:2301.13812_ , 2023.
  * Oudeyer and Kaplan [2007] Pierre-Yves Oudeyer and Frederic Kaplan.  What is intrinsic motivation? a typology of computational approaches.  _Frontiers in neurorobotics_ , 1:6, 2007.
  * Schmidhuber [1991] Jürgen Schmidhuber.  A possibility for implementing curiosity and boredom in model-building neural controllers.  In _Proc. of the international conference on simulation of adaptive behavior: From animals to animats_ , pages 222–227, 1991.
  * Lair et al. [2019] Nicolas Lair, Cédric Colas, Rémy Portelas, Jean-Michel Dussoux, Peter Ford Dominey, and Pierre-Yves Oudeyer.  Language grounding through social interactions and curiosity-driven multi-goal learning.  _arXiv preprint arXiv:1911.03219_ , 2019.
  * Campero et al. [2020] Andres Campero, Roberta Raileanu, Heinrich Küttler, Joshua B Tenenbaum, Tim Rocktäschel, and Edward Grefenstette.  Learning with amigo: Adversarially motivated intrinsic goals.  _arXiv preprint arXiv:2006.12122_ , 2020.
  * Forestier et al. [2022] Sébastien Forestier, Rémy Portelas, Yoan Mollard, and Pierre-Yves Oudeyer.  Intrinsically motivated goal exploration processes with automatic curriculum learning.  _The Journal of Machine Learning Research_ , 23(1):6818–6858, 2022.
  * Colas et al. [2022] Cédric Colas, Tristan Karch, Olivier Sigaud, and Pierre-Yves Oudeyer.  Autotelic agents with intrinsically motivated goal-conditioned reinforcement learning: a short survey.  _Journal of Artificial Intelligence Research_ , 74:1159–1199, 2022.
  * Klyubin et al. [2005] Alexander S Klyubin, Daniel Polani, and Chrystopher L Nehaniv.  Empowerment: A universal agent-centric measure of control.  In _2005 ieee congress on evolutionary computation_ , volume 1, pages 128–135. IEEE, 2005.
  * Choi et al. [2021] Jongwook Choi, Archit Sharma, Honglak Lee, Sergey Levine, and Shixiang Shane Gu.  Variational empowerment as representation learning for goal-conditioned reinforcement learning.  In _International Conference on Machine Learning_ , pages 1953–1963. PMLR, 2021.
  * Schulman et al. [2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Leibo et al. [2021] Joel Z Leibo, Edgar A Dueñez-Guzman, Alexander Vezhnevets, John P Agapiou, Peter Sunehag, Raphael Koster, Jayd Matyas, Charlie Beattie, Igor Mordatch, and Thore Graepel.  Scalable evaluation of multi-agent reinforcement learning with melting pot.  In _International conference on machine learning_ , pages 6187–6199. PMLR, 2021.

##  6 Supplementary Material

#### Environmental steps during training

All models are trained up to one million environment steps by dividing them
into 200 epochs, each with 5000 steps in the environment. To compare across
models, we select the best-performing epoch on the test episodes, and list the
number of steps occurre in Table 1 for Clean Up and Table 2 for Harvest.

Table 1: Training Steps - Clean UP Model | Environment Steps
---|---
IPPO | 762000
ICM-R | 276000
ICM | 348000
Influence | 726000
SVO-HE | 222000
SVO-HO | 258000
MAPPO | 255000
Table 2: Training Steps - Harvest Model | Environment Steps
---|---
IPPO | 594000
ICM-R | 1000000
ICM | 816000
Influence | 966000
SVO-HE | 900000
SVO-HO | 690000
SVO-HO | 630000

Implementation for this work can be found at
https://github.com/violetxi/EmergentSocialDynamics


#  ByteStack-ID: Integrated Stacked Model Leveraging Payload Byte Frequency
for Grayscale Image-based Network Intrusion Detection

Irfan Khan  Yasir Ali Farrukh  Department of Marine Engineering Technology, in
joint appointment with
Electrical and Computer Engineering Department
and Computer Science & Engineering Department,
Texas A&M University
Galveston, TX, USA
<EMAIL_ADDRESS>Clean and Resilient Energy Systems (CARES) Lab
Texas A&M University
College Station, TX, USA
<EMAIL_ADDRESS>Syed Wali
Clean and Resilient Energy Systems (CARES) Lab
Texas A&M University
College Station, TX, USA
<EMAIL_ADDRESS>
###### Abstract

In the ever-evolving realm of network security, the swift and accurate
identification of diverse attack classes within network traffic is of
paramount importance. This paper introduces ”ByteStack-ID,” a pioneering
approach tailored for packet-level intrusion detection. At its core,
ByteStack-ID leverages grayscale images generated from the frequency
distributions of payload data, a groundbreaking technique that greatly
enhances the model’s ability to discern intricate data patterns. Notably, our
approach is exclusively grounded in packet-level information, a departure from
conventional Network Intrusion Detection Systems (NIDS) that predominantly
rely on flow-based data. While building upon the fundamental concept of
stacking methodology, ByteStack-ID diverges from traditional stacking
approaches. It seamlessly integrates additional meta learner layers into the
concatenated base learners, creating a highly optimized, unified model.
Empirical results unequivocally confirm the outstanding effectiveness of the
ByteStack-ID framework, consistently outperforming baseline models and state-
of-the-art approaches across pivotal performance metrics, including precision,
recall, and F1-score. Impressively, our proposed approach achieves an
exceptional 81% macro F1-score in multiclass classification tasks. In a
landscape marked by the continuous evolution of network threats, ByteStack-ID
emerges as a robust and versatile security solution, relying solely on packet-
level information extracted from network traffic data.

###### Index Terms:

Network Intrusion Detection, Stacking, Machine Learning, Cybersecurity,
Internet of things.

##  I Introduction

The 21st century has ushered in an era of unprecedented era of technological
advancement, reshaping our lives through rapid progress in computation and
communication technologies [1]. However, this rapid advancement has also
expanded the cyber-threat landscape, emphasizing the paramount importance of
security in our increasingly interconnected digital world [2]. Furthermore,
technologies like the Internet of Things (IoT), where a myriad of low-cost,
resource-constrained devices converge on the global stage, further exemplify
this trend with an estimated 24.1 billion connected devices projected by 2030,
compared to 500 million in 2003 [3]. IoT’s transformative power is evident as
it revolutionizes how people and devices interact within connected ecosystems
[4].

In this interconnected society, communication and information systems underpin
critical infrastructure and personal devices. IoT’s extensive deployment,
encompassing nearly 30% of attacks in communication networks [5], poses
significant security challenges due to diverse architectures and communication
channels [2]. Network Intrusion Detection Systems (NIDS) play a crucial role
in identifying and mitigating cyberattacks within a layered defense strategy.
Recent advancements emphasize automated cyberattack identification, with
machine learning (ML) algorithms gaining traction [6, 7]. In ML, ensemble
methods are prominent [8], harnessing the collective power of multiple models
like Bagging, Boosting, and Stacking to enhance accuracy [9].

This paper introduces ”ByteStack-ID,” an innovative approach employing an
integrated stacking model that leverages network traffic payload information
as grayscale images for IoT intrusion detection. Unlike flow-based approaches,
ByteStack-ID operates at the packet level, inspecting packet payloads to
reveal malicious network flows. While flow-based methods focus on lower-level
threats within the TCP/IP protocol stack, potentially overlooking higher-level
threats [10], packet-based approaches, such as ByteStack-ID, offer a more
comprehensive solution. Our contributions in this paper encompass:

  1. 1.

Introduction of an innovative integrated stacking approach, featuring deep
concatenated 2D CNN models.

  2. 2.

Development of a novel method for transforming high-dimensional packet payload
data into grayscale images based on byte frequency.

  3. 3.

Conducting a comprehensive quantitative analysis that compares our proposed
method not only with state-of-the-art approaches but also with base models.

##  II Related Work

While extensive research has been conducted in the domain of NIDS, a
predominant focus has been on flow-based network traffic analysis. While some
approaches have delved into packet-level data, addressing the challenge of
handling the extensive feature space inherent in packet payloads, we have
narrowed our review to such studies that specifically leverage packet-level
information.

In packet-level intrusion detection, Wang et al. introduced the HAST-II
architecture, employing deep Convolutional Neural Networks (CNNs) and Long
Short-Term Memory (LSTM) networks to convert packets into two-dimensional
grayscale images, enhancing traffic classification [11]. Likewise, Zhang et
al. presented a multi-layer IDS model, incorporating CNNs and gcForest. Their
novel P-Zigzag algorithm converts raw packet-based network traffic into
grayscale images, facilitating initial detection and further classification
[12]. Furthermore, the authors in [13], proposed an LSTM-based autoencoder
through exploiting payloads and calculating reconstruction errors.

Regarding deep learning techniques using packets as text input, Liu et al.
introduced PL-CNN and PL-RNN models. PL-RNN employs the LSTM network, trained
on the first $n$ characters of payloads [14]. Packet2Vec, on the other hand,
employs Word2Vec, a shallow neural network, to generate packet vectors.
N-grams are utilized to create a dictionary, and packet vectors, with a length
of 128, serve as features for classification using Random Forest (RF) [15].
Extending the Word2Vec concept, Hassan et al. proposed PayloadEmbeddings,
utilizing vector representations of bytes in network packet payloads for
anomaly detection with a k-Nearest Neighbor (kNN) classifier [16]. Vidal et
al. introduced EsPADA, employing the n-gram technique ($n=3$) to extract
features from payloads and classify packets based on global similarity scores
(GSCs) [17]. However, it is imperative to acknowledge that the performance of
NIDS at the packet level remains an area in need of further enhancement,
particularly in the methods used to transform packet-level information into
effective features. This underscores the pressing need for continued progress
and innovation in this field.

##  III Methodology

In this section, we offer a comprehensive overview of our methodology,
starting with the dataset we used in our experiments. We then explain the
preprocessing steps we took to prepare the data for training and testing,
including the conversion of the data into grayscale images. Afterward, we
describe our integrated stacked model and how we trained it.

###  III-A Dataset and Preprocessing

There are several publicly available dataset within the NIDS field, yet a
significant proportion of these datasets predominantly center around header
information from network packets, emphasizing flow-based network traffic
analysis. Unfortunately, datasets encompassing actual payload data are sparse,
primarily due to privacy-related concerns. Additionally, some datasets that do
incorporate packet data are often found in a raw, unlabeled format, making
them less accessible for ongoing research.

For our study, we utilized the CIC-IDS2017 dataset [18], which was developed
by the University of New Brunswick in 2017. This dataset comprises simulated
packet-based and bidirectional flow-based network traffic, encompassing modern
attacks and benign traffic. The dataset is available in two formats: the
original packet capture (PCAP) files (packet-based data) and CSV files (flow-
based data). Since we focused primarily on packet-based network traffic, we
opted to work with the raw packets from CIC-IDS2017.

Working with raw packet data poses a challenge due to the absence of labels.
To overcome this limitation, we harnessed our developed tool, ”Payload-Byte”
[19], to label the raw packets from the CIC-IDS2017 dataset. Payload-Byte
utilizes the five-tuple features, including Source IP, Destination IP, Source
Port, Destination Port, and Protocol, to match packets with labeled flow-based
data instances.

After labeling the packets, our subsequent task was to extract the payload
information while discarding extraneous data. Features like IP addresses in
the network layer and port fields in the transport layer were omitted due to
potential bias and their dynamic nature, rendering them unsuitable for
training machine learning models. Consequently, we focused on the payload
contents as our primary feature of interest. Given the variable payload sizes
in each packet, we standardized their length to a maximum of 1500 bytes,
aligning with the common 1500-byte packet size limit [20]. This extracted
payload was then unified into a single comprehensive feature, subsequently
divided into 1500 individual features based on bytes. This transformation
involved converting the hexadecimal representation of each byte into an
integer within the range of 0 to 255, effectively creating a unique feature
for every byte. To maintain consistency, we employed zero padding for packets
with payloads containing fewer than 1500 bytes, ensuring uniform feature
vector structures.

Figure 1: Pictorial representation of the processing pipeline for generating
grayscale images from raw network packets. The calculated packet’s payload
byte distribution is normalized with respect to the highest frequency within
each packet.

Additionally, we conducted preprocessing steps as part of our data
preparation. This encompassed the removal of duplicate instances and the
exclusion of instances lacking payload data. To manage dataset size
effectively, we implemented under-sampling, which aided in reducing the number
of benign instances. For a more comprehensive understanding of these
preprocessing procedures and the functionality of Payload-Byte, we refer
interested readers to our prior work [19].

Subsequently, we transformed our processed data into grayscale images,
leveraging payload byte frequency. The data was then split into three subsets:
$D_{1}$, $D_{2}$ and $D_{3}$ with the ratio 50:20:30 respectively. The subset
$D_{1}$ is utilized for training base learner models, whereas $D_{2}$,
intended for the meta-learner model. The subset $D_{3}$ is set aside for
evaluating the proposed approach.

###  III-B Payload Transformation into Images

Our approach to transforming network traffic packet payloads into grayscale
images hinges on the byte frequency distribution within each packet’s payload
data. This transformation addresses a core challenge in dealing with packet-
based network traffic—managing extensive data volumes and complex feature
spaces. By converting the data into images, we effectively handle high-
dimensional data without the need for intricate feature engineering. An
overview of the entire process, starting from raw packet data to image
generation, is depicted in Fig. 1.

The transformation process begins with capturing raw packets and extracting
their payload data. Following this, we calculate the byte-wise frequency
distribution for each packet. This distribution serves as a unique
fingerprint, capturing the frequency of each byte within the payload and
providing a distinct representation of the packet’s content.

To create meaningful grayscale images, we normalize the byte frequency
distribution. Instead of using a generic normalization factor, we adopt a
packet-specific approach. This means that each packet’s byte frequency
distribution is normalized based on the highest frequency observed within that
specific packet. This tailored normalization strategy ensures that the
resulting grayscale images accurately represent the relative importance of
byte occurrences within each packet.

Utilizing the normalized frequency distributions, our next step involves
crafting the grayscale images. Each image is designed with dimensions of 16x16
pixels, providing an organized canvas for visualization. Within this 16x16
grid, we linearly arrange the byte frequency values, accommodating all 256
possible byte values. Fig. 2 presents examples of packet-level network traffic
after transformation.

Figure 2: Visualization of grayscale images generated from raw network
packets, derived from payload byte frequency distribution. The examples shown
are randomly selected for illustration. Figure 3: Model architecture of the
proposed approach—comprising 15 distinct base learner models, each trained for
a specific attack class using a one-vs-all approach. Integrated stacking is
achieved through meta learner with additional dense layers integrated into
non-trainable concatenated base learners, forming an integrated stacking
model.

###  III-C Integrated Stacked Model

Our methodology, ByteStack-ID, builds upon the Stacking ensemble technique
[21], which leverages two categories of learners: Base Learners and Meta
Learners. Stacking utilizes the classification outputs of these base learners,
often referred to as level-0 learners, as inputs for a meta-learner or level-1
learner. This approach excels in handling highly diverse datasets, offering
various methods of data classification and typically outperforming standalone
classifiers [22].

Polikar’s foundational work [23] outlines four primary strategies for
diversifying learning in stacking: using different training data, adjusting
training parameters, incorporating diverse features for training, and
combining various types of classifiers. In line with this, our ByteStack-ID
approach embraces diversity by employing separate training sets for each base
learner, treating them as binary classifiers.

Our methodology introduces a unique strategy for the meta-learner. Unlike the
conventional approach, where additional models are trained as meta-learners
using the outputs of base learners to enhance predictions, we integrate the
last layer of each base learner through the meta-learner. This integration
creates a larger neural network, treating the stacking ensemble as a unified
and cohesive model. Hence, we refer to our approach as an Integrated Stacked
Model.

In our approach, each base learner undergoes individual training and is then
integrated into additional layers treated as the meta-learner. During the
meta-learner’s training, the weights for each layer of the base learners are
frozen, allowing only the weights of the meta-learner to be learned. This
strategic approach significantly enhances the overall performance and
adaptability of our intrusion detection model. For a detailed training
mechanism, please refer to Algorithm 1.

For a visual representation of ByteStack-ID and its architecture, please see
Fig. 3. In this figure, we illustrate 15 distinct base learners, all trained
for binary classification, with each base learner addressing a one-vs-all
classification problem. After training all the base learners, we integrate
them with the meta-learner model layers. Instead of using the last layer of
each base learner model, we employ the second last layer for integration with
the meta-learner. Subsequently, the integrated model or meta-learner is
trained using the $D_{2}$ sub-dataset.

Figure 4: Model architecture of the adopted deep concatenated 2D-CNN for base
learners—consisting of four blocks with input and output concatenation for
enhanced feature extraction.

####  III-C1 Architecture of Base Learner

For our base learners, we have built upon the architecture previously
introduced in our prior work [24], which leverages the concept of input and
output concatenation. The proposed model employs a combination of multiple
Convolutional Neural Networks (CNNs) in a cascaded or concatenated manner,
enabling the model to acquire insights at multiple levels of abstraction and
representations from the input data. Our model architecture aligns with the
established 2D-CNN baselines. As per common practice, the number of filters in
our proposed model progressively increases across layers, empowering the
network to capture increasingly complex and abstract features. You can find a
detailed layer-wise progression within the model in Table I.

TABLE I: Detailed Model Architecture Specifications for Base Learners

| Layer Type
---|---
Input | InputLayer (16, 16, 1)
Block#1 | BatchNormalization()
Activation(’Relu’)
Conv2D(8, (3,3), padding=’same’)
BatchNormalization()
Activation(’Relu’)
Conv2D(8, (3,3), padding=’same’)
Block#2 | BatchNormalization()
Activation(’Relu’)
Conv2D(16, (3,3), padding=’same’)
BatchNormalization()
Activation(’Relu’)
Conv2D(16, (3,3), padding=’same’)
Block#3 | BatchNormalization()
Activation(’Relu’)
Conv2D(32, (3,3), padding=’same’)
BatchNormalization()
Activation(’Relu’)
Conv2D(32, (3,3), padding=’same’,strides=(2,2))
Block#4 | Flatten()
Dense(64, activation=Activation(’Relu’))
Dropout(rate=0.2)
Dense(32, activation=Activation(’Relu’))
Dropout(rate=0.2)
Dense(16, activation=Activation(’Relu’))
Output | Dense(1, activation=’Sigmoid’)

Furthermore, our model incorporates a distinctive structural element, wherein
the output of block 1 is fused with the input data, effectively forming the
input for block 2. This process is iterated for block 3, where the output of
block 2 is concatenated with the input of block 2. However, it’s worth noting
that this same process is not repeated for block 4. This deviation is due to
the nature of block 4, which relies on dense layers rather than convolutional
layers. Since dense layers operate on flattened feature representations,
concatenating the images at this stage would not yield significant benefits in
terms of model performance. A pictorial representation of the model is
illustrated in Fig. 4

Algorithm 1 Training of ByteStack-ID

1:Input: Raw Network Traffic Packet

2:Output: Multiclass Classification $\rightarrow$ Attack Class

3:Step 1: Dataset Division

4:$D_{1}$: Base Learner Dataset $\leftarrow$ 50% of the Dataset

5:$D_{2}$: Meta Learner Dataset $\leftarrow$ 20% of the Dataset

6:$D_{3}$: Evaluation Dataset $\leftarrow$ 30% of the Dataset

7:Step 2: Packets into Grayscale Images

8:Extract payload from each packet and calculate byte-wise frequency
distribution

9:Normalize frequency distribution using the highest frequency observed within
each packet

10:Generate grayscale images with dimensions of 16x16 pixels

11:Step 3: Base Learner Model Training

12:for $i=1$ to $N$ do $\triangleright$ $N$ = Number of Attack Classes

13: Transform labels of $D_{1}$ into one-vs-all labels $\triangleright$ 1 =
Specific Attack Class; 0 = Rest of the classes

14: Train the $i_{th}$ base learner model using the transformed $D_{1}$

15: Freeze the weights of all layers

16:Step 4: Meta Learner Model Training

17:Remove the last layer of each base learner model

18:Integrate the meta learner dense layers with the non-trainable layers of
all base learner models

19:Train the integrated model using the $D_{2}$ dataset

####  III-C2 Architecture of Meta Learner

Our adoption of integrated stacking simplifies the meta-learner’s
architecture. Instead of introducing an additional model as a meta-learner,
which typically trains on the outputs of base learners, our approach directly
integrates the dense layers of the meta-learner with the second last layer of
each base learner. This choice is accompanied by freezing the layers of the
base learner, ensuring that only the weights of the meta-learner layers are
trainable. When we apply the $D_{2}$ sub-dataset to the integrated model, the
training process exclusively targets the meta-learner layers.

For a visual representation of the Integrated Meta Learner, please see Fig. 5.
This diagram provides a clear illustration of the integration process within
our approach. Furthermore, the comprehensive training process of our approach
is outlined in Algorithm 1.

Figure 5: Model architecture of the integrated meta learner and its
integration with base learners. Green blocks represent additional layers of
the meta learner integrated with all base learners, with base learners’
weights frozen as non-trainable.

##  IV Results and Discussion

To evaluate the effectiveness of our proposed ByteStack-ID approach, we
conducted comprehensive experiments comparing its performance with several
baseline models using packet-level data. Our model comparison included well-
established algorithms such as Random Forest (RF), Logistic Regression,
AdaboostClassifier, a Deep Neural Network (DNN) with three layers, and a
combination of 1D-CNN and Long Short-Term Memory (LSTM) with five layers.
Additionally, we included a base model with a similar architecture to
ByteStack-ID, but it underwent a single training iteration, unlike our two-
step base and meta learner training. This inclusion allowed us to investigate
the impact of integrated stacking on model performance.

The outcomes of our experiments are presented in Table II, where we evaluated
each model’s performance based on key metrics, including macro F-1 score,
precision, and recall. By analyzing the results, it is apparent that the base
model, utilizing transformed grayscale images, outperformed all other baseline
models. Nevertheless, our ByteStack-ID approach consistently achieved superior
results across all evaluated metrics, underscoring its effectiveness.

TABLE II: Performance comparison with baseline models

Models | Precision | Recall | F1-Score
---|---|---|---
Random Forest | 62% | 56% | 58%
Logistic Regression | 48% | 42% | 42%
AdaboostClassifier | 37% | 54% | 40%
DNN | 56% | 51% | 52%
CNN_LSTM | 61% | 56% | 56%
Base Model | 81% | 75% | 78%
ByteStack-ID | 84% | 79% | 81%

To further assess the efficacy of our proposed approach, we conducted a
comprehensive comparison with state-of-the-art methods employing packet-level
data. Our selection criteria prioritized approaches with experiments on the
same dataset, reporting precision, recall, accuracy, and F-1 Score. Despite
potential differences in input processing and data extraction, our aim was to
provide a holistic evaluation of packet-level data methods.

To ensure a fair comparison, we identified works operating under similar
experimental conditions. Detailed results are available in Fig. 6. These
results unequivocally demonstrate our approach consistently outperforming
other state-of-the-art methods utilizing network traffic packet data. This
compelling evidence underscores our approach’s effectiveness and superiority
in classifying attack classes using packet level network traffic data.

Figure 6: Comparative analysis of the proposed approach against state-of-the-
art methods utilizing packet-level data. Performance evaluation utilizes
metrics including macro F-1 score, precision, and recall.

##  V Conclusion

In this paper, we present ByteStack-ID a novel method for the NIDS utilizing
packet-level network traffic. One of the key innovations of our proposed
method lies in the utilization of grayscale images derived from payload data
frequency distributions, enhancing the model’s ability to detect subtle
patterns. ByteStack-ID deviates from conventional stacking methods by
integrating additional layers directly into the concatenated base learners,
unifying the base learner and meta learner into a single, unified model.

Our empirical findings demonstrate ByteStack-ID’s exceptional effectiveness,
surpassing baseline models and state-of-the-art approaches. As network threats
continue to evolve, ByteStack-ID emerges as a valuable asset in enhancing
security measures and proactively mitigating emerging threats within critical
network infrastructures. With its pioneering techniques and outstanding
performance, ByteStack-ID takes a prominent position at the forefront of
intrusion detection research at packet-level network traffic.

## Acknowledgment

Research reported in this paper was supported by an Early-Career Research
Fellowship from the Gulf Research Program of the National Academies of
Sciences, Engineering, and Medicine. The content is solely the responsibility
of the authors and does not necessarily represent the official views of the
Gulf Research Program of the National Academies of Sciences, Engineering, and
Medicine.

## References

  * [1] Y. A. Farrukh, S. Wali, I. Khan, and N. D. Bastian, “Detecting unknown attacks in iot environments: An open set classifier for enhanced network intrusion detection,” _arXiv preprint arXiv:2309.07461_ , 2023.
  * [2] F. Meneghello, M. Calore, D. Zucchetto, M. Polese, and A. Zanella, “Iot: Internet of threats? a survey of practical security vulnerabilities in real iot devices,” _IEEE Internet of Things Journal_ , vol. 6, no. 5, pp. 8182–8201, 2019.
  * [3] M. Hatton, “The iot in 2030: Which applications account for the biggest chunk of the 1.5 trillion opportunity?” 2020.
  * [4] S. Wali and I. Khan, “Explainable ai and random forest based reliable intrusion detection system,” 2021.
  * [5] J. C. Jensen, “The first workshop on cyber-physical systems education.” [Online]. Available: https://cps-vo.org/node/7266
  * [6] Y. A. Farrukh, Z. Ahmad, I. Khan, and R. M. Elavarasan, “A sequential supervised machine learning approach for cyber attack detection in a smart grid system,” in _2021 North American Power Symposium (NAPS)_. IEEE, 2021, pp. 1–6.
  * [7] I. Khan, S. Wali, and Y. A. Farrukh, “Radiant: Reactive autoencoder defense for industrial adversarial network threats,” _Available at SSRN 4572172_.
  * [8] D. Gupta and R. Rani, “Improving malware detection using big data and ensemble learning,” _Computers & Electrical Engineering_, vol. 86, p. 106729, 2020.
  * [9] I. Syarif, E. Zaluska, A. Prugel-Bennett, and G. Wills, “Application of bagging, boosting and stacking to intrusion detection,” in _Machine Learning and Data Mining in Pattern Recognition: 8th International Conference, MLDM 2012, Berlin, Germany, July 13-20, 2012. Proceedings 8_. Springer, 2012, pp. 593–602.
  * [10] W. Lee, S. J. Stolfo, P. K. Chan, E. Eskin, W. Fan, M. Miller, S. Hershkop, and J. Zhang, “Real time data mining-based intrusion detection,” _Proceedings - DARPA Information Survivability Conference and Exposition II, DISCEX 2001_ , vol. 1, pp. 89–100, 2001.
  * [11] W. Wang, Y. Sheng, J. Wang, X. Zeng, X. Ye, Y. Huang, and M. Zhu, “Hast-ids: Learning hierarchical spatial-temporal features using deep neural networks to improve intrusion detection,” _IEEE access_ , vol. 6, pp. 1792–1806, 2017.
  * [12] X. Zhang, J. Chen, Y. Zhou, L. Han, and J. Lin, “A multiple-layer representation learning model for network-based attack detection,” _IEEE Access_ , vol. 7, pp. 91 992–92 008, 2019.
  * [13] B. A. Pratomo, P. Burnap, and G. Theodorakopoulos, “Unsupervised approach for detecting low rate attacks on network traffic with autoencoder,” in _2018 international conference on cyber security and protection of digital services (Cyber Security)_. IEEE, 2018, pp. 1–8.
  * [14] H. Liu, B. Lang, M. Liu, and H. Yan, “Cnn and rnn based payload classification methods for attack detection,” _Knowledge-Based Systems_ , vol. 163, pp. 332–341, 2019.
  * [15] E. L. Goodman, C. Zimmerman, and C. Hudson, “Packet2vec: Utilizing word2vec for feature extraction in packet data,” _arXiv preprint arXiv:2004.14477_ , 2020.
  * [16] M. Hassan, M. E. Haque, M. E. Tozal, V. Raghavan, and R. Agrawal, “Intrusion Detection Using Payload Embeddings,” _IEEE Access_ , vol. 10, pp. 4015–4030, 2022.
  * [17] J. M. Vidal, M. A. S. Monge, and S. M. M. Monterrubio, “Espada: Enhanced payload analyzer for malware detection robust against adversarial threats,” _Future Generation Computer Systems_ , vol. 104, pp. 159–173, 2020.
  * [18] I. Sharafaldin, A. H. Lashkari, and A. A. Ghorbani, “Toward Generating a New Intrusion Detection Dataset and Intrusion Traffic Characterization,” 2018.
  * [19] Y. Farrukh, I. Khan, S. Wali, D. A. Bierbrauer, J. Pavlik, and N. D. Bastian, “Payload-byte: A tool for extracting and labeling packet capture files of modern network intrusion detection datasets,” 2022.
  * [20] R. J. Anthony, “The Resource View,” _Systems Programming_ , pp. 203–276, 1 2016.
  * [21] D. A. Bierbrauer, A. Chang, W. Kritzer, and N. D. Bastian, “Cybersecurity anomaly detection in adversarial environments,” _arXiv preprint arXiv:2105.06742_ , 2021.
  * [22] R. Odegua, “An empirical study of ensemble techniques (bagging, boosting and stacking),” in _Proc. Conf.: Deep Learn. IndabaXAt_ , 2019.
  * [23] R. Polikar, “Ensemble based systems in decision making,” _IEEE Circuits and systems magazine_ , vol. 6, no. 3, pp. 21–45, 2006.
  * [24] Y. A. Farrukh, S. Wali, I. Khan, and N. D. Bastian, “Senet-i: An approach for detecting network intrusions through serialized network traffic images,” _Available at SSRN 4370422_.