# RaidEnv: Exploring New Challenges in Automated Content Balancing for Boss
Raid Games

Hyeon-Chang Jeon∗, In-Chang Baek∗, Cheong-mok Bae, Taehwa Park, Wonsang You,
Taegwan Ha, Hoyun Jung, Jinha Noh, Seungwon Oh, Kyung-Joong Kim equal
contribution $*$This paper was produced by the IEEE Publication Technology
Group. They are in Piscataway, NJ.Manuscript received April 19, 2021; revised
August 16, 2021.

###### Abstract

The balance of game content significantly impacts the gaming experience.
Unbalanced game content diminishes engagement or increases frustration because
of repetitive failure. Although game designers intend to adjust the difficulty
of game content, this is a repetitive, labor-intensive, and challenging
process, especially for commercial-level games with extensive content. To
address this issue, the game research community has explored automated game
balancing using artificial intelligence (AI) techniques. However, previous
studies have focused on limited game content and did not consider the
importance of the generalization ability of playtesting agents when
encountering content changes. In this study, we propose RaidEnv, a new game
simulator that includes diverse and customizable content for the boss raid
scenario in MMORPG games. Additionally, we design two benchmarks for the boss
raid scenario that can aid in the practical application of game AI. These
benchmarks address two open problems in automatic content balancing, and we
introduce two evaluation metrics to provide guidance for AI in automatic
content balancing. This novel game research platform expands the frontiers of
automatic game balancing problems and offers a framework within a realistic
game production pipeline.

###### Index Terms:

boss raid game environment, content generation, game playtesting, MMORPG

††publicationid: pubid: 0000–0000/00$00.00 © 2021 IEEE

##  I Introduction

Game content balancing is a crucial process in the game industry when
releasing new game content. Unbalanced game content can cause player
dissatisfaction and frustration. To mitigate this issue, game companies employ
balancing processes to prevent overpowered content that diminishes the game
experience. Prior to content release, game testers evaluate the balancing and
provide reports to the game designer, who verifies whether the content aligns
with their intentions. Even after content release, game designers analyze game
log data and readjust the content to achieve rebalancing.

However, several game companies rely on human testers repeatedly playing new
game content to measure its difficulty. This approach has limitations as it
does not allow testers to acquire sufficient data owing to time constraints.
Moreover, the labor-intensive and expensive nature of this process limits the
collection of extensive data from game testers. Recently, several scholars
have attempted to employ machine learning techniques, such as in popular
puzzle games [1, 2, 3] and card-based real-time strategy games [4], for
testing game content.

Automatic content balancing (ACB) is an automated technique that aims to
readjust or recombine game content to ensure a balance of the game. Within the
game research community, ACB is regarded as a promising solution to address
this challenge by leveraging AI player and generator (i.e., balancer) agents
as game testers and designers, respectively. The ACB process involves two
repetitive phases to automating game design tasks and quality assurance: (1)
generating new game content using machine learning methods known as procedural
content generation with machine learning (PCGML) [5] In ACB, there are two
repetitive phases to automating game design tasks and quality assurance: (1)
generating new game content with machine learning (PCGML) [5] methods and (2)
evaluating the content through playtesting with an AI player. These automated
sequences allow generator models to be trained through extensive trial-and-
error iterations, surpassing the limited trials feasible with human testers.
The quality of the generated content depends on the AI player’s robustness in
playing diverse game contents and fairly evaluating them [6]. Similarly, the
content generator should possess the ability to generate various contents
according to designer-specified requirements.

In this paper, we present RaidEnv, a game environment that aims to encompass
the features found in commercial-level MMORPG games, with a specific focus on
the boss raid scenario, a popular content type in MMORPGs. This highly
customizable environment includes machine learning interfaces for playtesting
and procedural content generation (PCG) agents. Our contribution includes the
proposal of two benchmarks applicable to automatic content balancing (ACB)
problems. These benchmarks consist of (1) training generalized playtesting
agents and (2) training controllable content generation. We conduct the
benchmarks within RaidEnv, providing deep reinforcement learning (DRL) methods
as baselines, in addition to handwritten heuristics. Each benchmark was
individually analyzed, and a combined analysis is presented at the end of this
paper. The major contributions of this study are summarized as follows:

  * •

We developed RaidEnv, an integrated game simulator that supports extensive
customization, facilitating content generation, and playtesting benchmarks.

  * •

We propose a benchmark for evaluating playtesting agents, emphasizing their
generalization ability when encountered with content variations.

  * •

We propose a benchmark for skill content generation, focusing on
controllability, and diversity aspects.

To introduce the new environment and benchmarks, we have organized the
sections as follows: In Section II, wwe review previous studies to compare the
features of our environment, covering existing game environments, and
balancing scenarios. Section III provides a detailed description of the new
game environment, outlining its key features, and components. In Section IV,
we present our first benchmark, which focuses on the development of robust
playtesting agents. To adequately evaluate the performance of the generated
content, we proposed adjusted test performance metrics considering the varying
difficulty with content evolution. In the following Section V, we introduce
the PCG benchamrk involves skill content generation and utilizes the
playtesting agent proposed in the first benchmark. Using agent-based
simulations, the utility of the PCG models was evaluated in terms of
benchmarking controllability and diversity. Finally, the comprehensive results
obtained from both benchmarks are discussed to highlight the potential
implications of this research.

TABLE I: Summary of the environments for contents generation and playtesting Environment | Ref. | PCG Problem | Multiagent Problem | Balancing Problem
---|---|---|---|---
Hearthstone | [7] | Card Deck Building | - | -
Ludii | [8] | Game Rule | - | -
GVGAI | [9] | Rule, Map Layout | - | -
VizDoom | [10, 11] |  | Skill, Map Layout
---
- | -
Mario AI Framework | [12, 13] | Map Layout | - | -
Google Research Football | [14] | - | High-performance | -
Fever Basketball | [15] | - | High-performance | -
Heroes Charge | [16] | - | - | Single-content Balancing (Unit)
Red Alert | [17] | - | - | Single-content Balancing (Unit)
Sports Game | [18] | - | - | Match Making
Dungeon Replicants I & II | [19, 20] | - | - | Experience-driven Playtesting
SMAC I & II | [21, 22] | Scenario | Environmental Generalization | -
Melting Pot | [23] | Map Layout | Ad-hoc Generalization | -
Neural MMO | [24] | Massive Map Layout | High-performance | -
Tower Defense | [25] | Map Layout | - | Multi-content Balancing (Spawn, Unit)
Pokémon | [26] | Card Deck Building | High-performance | Automatic Team Assembly
Our Environment |  | Skill, Stats |  | Environmental Generalization
---
Multi-content Balancing
Figure 1: Overall game components in the RaidEnv environment. The game
features and environmental variables are illustrated in hierarchical structure
and the same-level groups are colorized in same color (e.g., light red and
light green). White box components are the modifiable variables and the
colorized boxes are not modifiable, but grouped similar functional variables.
Character Stats is the characteristic of the playtesting agents and the
Skill(s) is ownable component by an agent. Each agent could have 1-3
executable skills and all the components are fully differentiate by the agent.

##  II Related Works

We summarize the existing game research platforms in Table I and the
development purpose of them. In Table I, the balancing problem column
indicates which problem was handled; the platforms that support content
generation or multi-agent interfaces are additionally summarized for comparing
the detailed features. As shown in Table I, the platforms have various problem
definitions and only one (i.e., Pokémon) handled all three features: content
generation, multiagent, and balancing. That is, our platform has huge
scalability for the game studies and propose new case study for novel
problems.

###  II-A Automated Content Balancing

In previous studies (refer to Balancing Problem in Table I), four
representative balancing problems have been proposed: automatic team assembly,
matchmaking, experience-driven playtesting, and single-/multi-content
balancing; The last problem is particularly relevant to our work. Automatic
team assembly aims to predict the power of a team [27] and identify
overpowering champion combinations [26] in battler games. Matchmaking studies
[18] predict competitive combinations of two or more players, considering
skill rating systems such as TrueSkill [28] and Elo [29]). Experience-driven
playtesting studies aim to generate diverse behavioral playtesting logs from a
game. In [19, 20], the authors explore a deep player behavior modeling
approach, by training imitation models with the playtesting logs of 213
players. Single- and multi-content balancing focuses on regulating game object
parameters to ensure fair gameplay or intended symmetry. Evolutionary methods
are employed in [17, 30] to balance units (characters) in real-time games. In
particular, [30] simulates the game with playtesting agents of varying levels
of proficiency.

Although the developed content generation task is similar to single-content
balancing problems [16, 17], it was applied in a different game genre, MMORPG.
Although our environment supports multi-content balancing for various game
elements such as skills and stats, we demonstrated content generation for a
single content to simplify the problem size. Although this research shares
certain similarities with the VGC AI competition [26], it exhibits notable
differences. First, the VGC AI competition focused on deck-building rather
than altering the content itself, aiming to determine the possible
combinations. Second, they primarily focused on the framework, whereas our
focus lies on evaluation. Furthermore, this research emphasizes the robust
playtesting of playtesting agents. We propose a balancing study in a
cooperative game scenario, specifically a boss raid that encompasses various
game design components such as character class and executable skills. We
provide baseline approaches based on the latest DRL methods. Notably, we
highlight the significance of playtesting agents’ generation ability in game-
balancing studies. Additionally, the proposed platform—utilizing a popular
machine learning framework—have been published as an open-source model for
game researchers to generate accessibility for multi-content generation tasks.
This platform presents numerous opportunities for exploring novel balancing
scenarios among the game research community.

###  II-B Multiplayer Game AI Environments

Multiplayer game AI environments primarily focus on designing algorithms that
determine the behavior of each agent within a game system involving
cooperation, competition, or both among two, or more players. Various
environments have been proposed to facilitate multiplayer game AI studies,
addressing specific problems using multiplayer game AI algorithms, as depicted
in Table I. Several multiplayer game environments have concentrated on
achieving superhuman-level AI performance in diverse games. For example, the
Starcraft Multi-Agent Challenge (SMAC)[21], based on the RTS game StarCraft
II, offers a range of unit configurations, emphasizing long-term decision-
making, and partially observable contexts. In the sports game genre, Google
Research Football (GRF)[14] and the Fever Basketball environment[15] aim to
develop high-performing agents in sports-related games. Recently, the research
focus has shifted toward environments that prioritize performance
generalization, as observed in Melting Pot[23] and SMACv2[22]. Melting Pot
focuses on generalizing agent performance when playing against previously
unseen agents. It encompasses sequential social dilemmas (SSD), competitive
games, and cooperative games. SMACv2 aims to generalize agent performance in
unseen environments by providing procedurally generated unit locations, types,
and numbers. Our study shares similarities with previous work[22] in terms of
the goal of achieving strong performance in unseen environments. However, our
focus is on developing robust playtesting agents capable of adapting to a
wider range of game contents. Additionally, our environment introduces varying
levels of difficulty through changing game content.

###  II-C Procedural Contents Generation Environments

Procedural content generation (PCG) studies have explored various types of
game content generation, including game level layout, game rule generation,
and card deck generation. Game level layout generation has been extensively
researched in popular video games such as Angry Birds [31, 32], Super Mario
Bros [33, 34], and Minecraft [35]. The objective of level generation is to
create playable game content that can be explored and completed by players.
Game rule generation involves automatically generating the mechanics,
dynamics, and constraints of a game. This area of research has been proposed
in platforms such as general video game AI (GVGAI) [9] and Ludii [8],
utilizing game description languages. Card deck generation focuses on
searching for optimal combinations of game cards and has been studied in
representative card simulation games such as Hearthstone [7] and Pokemon [9].
To expand the scope of PCG research to extensive game contents, new
customizable game elements must be considered. However, a limited number of
studies have focused on generating game content such as skills and player
characteristics. Therefore, this study introduces a novel content type for
procedural generation and demonstrates the generation using reinforcement
learning. Game features such as skills, character classes, and items are
common across various game genres, with numerous features shared within the
same genre. To ensure the generality of our framework, we extract several
features from commercial games and provide them as controllable parameters.
This approach offers new perspectives on generating game content and opens up
new possibilities in the game research community.

##  III RaidEnv: The Boss Raid Environment

###  III-A Boss Raid Scenario

Boss raids are a prevalent form of cooperative multiplayer content in most
MMORPG games. The objective of a raid is to defeat a powerful boss within a
limited time frame in a dungeon, while playing alongside other players, with
the aim of obtaining valuable rewards. Boss raid content is centered around
the challenge of overcoming a formidable boss with limited resources and time.
Typically, bosses are designed to be stronger than a single player, featuring
complex attack patterns that pose a significant challenge in solo encounters.
Certain attack patterns necessitate player cooperation, such as spreading out
to minimize damage when the boss launches an area attack. This highlights the
importance of real-time communication between players. As a fundamental
principle in multiplayer game design [36, 37], role differentiation is applied
to boss raid content for assigning specific responsibilities to individual
players. In the boss raid scenario, role differentiation is achieved by
diversifying the skills and abilities of players, with well-defined roles such
as tankers, dealers, and healers being widely standardized. Commercial MMORPG
games often offer numerous character roles to enhance the cooperative gameplay
experience. For instance, World of Warcraft
(WoW)111https://worldofwarcraft.blizzard.com/ features over 30 specialized
roles.

The increasing complexity of role types and game content poses challenges for
game designers in predicting the outcomes of game updates and managing the
combined effects of various elements. Therefore, the use of automatic content
balancing (ACB) techniques becomes crucial for assisting game designers and
developers in this complex landscape. However, only a few studies have
attempted to apply ACB in the context of boss raids, and existing attempts
considered an inadequate number of customizable features to facilitate
comprehensive balancing studies. Consequently, the RaidEnv simulator offers a
valuable contribution by providing a customization interface that enables the
generation of diverse character roles through game simulation. The
implementation details of the boss raid scenario are described in the
following section.

###  III-B Contents Definition

To abstract boss raid content, we have categorized the game elements into
three components: player class, character statistics, and skill parameters.
Figure 1 provides an overview of the design process for player classes, and
the specific details of each component are described below:

Class: The player class is defining characteristic that imparts individuality
to player characters and assigns the distict roles within the game. The class
of a character consists of two key elements: statistics and a set of skills.
For example, the Tanker class specializes in withstanding enemy attacks,
possessing unique statistics such as high health points and defensive bonuses
compared to other classes. In addition, tankers possess skills that enable
them to mitigate incoming damage and protect their allies.

Statistics (Stats): Statistics represent the internal attributes of characters
in RPGs and are typically represented as numerical values. RPGs, ranging from
traditional tabletop games to modern MMORPGs, utilize various statistics to
define character attributes. These statistics can be gained permanently or
temporarily through character growth, equipment, consumables, or special
abilities. Examples of statistics include health/mana points, strength,
intelligence, dexterity, critical chances, and more.

Skill: While modern games employ different terms such as abilities, talents,
or traits, in our environment, we simplify these concepts into the common
notion of skills. Skills are possessed by characters and can be categorized
into two types: active skills and passive skills. Active skills are initiated
by a character’s action, allowing the player to enact decisions regarding
which skill to activate from their available set of skills. Conversely,
passive skills are latent abilities that have an impact even when not directly
activated by the character. Skills can be differentiated by analyzing
conditions for activation, target affected, effects produced, and other
relevant factors.

The environment offers numerous customization features, including 17 character
statistics, and 17 player skill parameters. These variables can be included or
excluded as needed, providing a highly versatile game research platform. For
the PCG track in this study, we have chosen four representative skill
parameters, as generating all skill parameters simultaneously would introduce
a complex problem size. However, the environment supports customizable
variable across multiple game components, allowing for future exploration of
multi-content balancing.

###  III-C RaidEnv: The Game Simulation Environment

Figure 2: The snapshot of the RaidEnv environment. The blue ones are the
player agents and the green one is the boss agent.

The RaidEnv environment is implemented using the Unity platform, and for
machine learning training, we utilized Unity ML-Agents Toolkit [38]. With the
RaidEnv environment as a foundation, we have designed several scenarios
tailored to each benchmark. During each training step, the single unity agent
environment receives continuous observations (e.g., scalars, ray perception
sensors) and samples discrete actions for learning within the environment. The
RaidEnv environment encompasses a basic MMORPG boss raid scenario, consisting
of one enemy agent and three player agents tasked with defeating the boss.
Each player agent possesses one skill targeted at the enemy. To efficiently
gather playtesting results during training, parallelization of the environment
is necessary. Therefore, we have included multiple arenas within the
environment to enable parallel playtesting. Furthermore, We created interfaces
to promote learning among multiplayer agents and PCG, including functions to
compile game logs for tracing action steps and episodes.

The Boss in the boss raid scenario serves as the opponent non-player character
(NPC). As mentioned in Section III-A, the boss is intentionally designed to be
more powerful than a single player, fostering cooperation among the players.
The boss has 10 times the health points of a player agent and is equipped with
two strong skills with ranges of 6 and 12. The boss agent follows simple
behavioral policy: it selects the closest player agent as its target, moves
toward that player, and launches an attack when the skill is available. Figure
2 provides a snapshot of the environment, featuring one boss NPC, and three
player agents In summary, the RaidEnv environment consists of one static boss
NPC and three customizable player agents. Two benchmarks are proposed within
this setting, and the overall process is depicted in Figure 3. We will first
delve into the playtesting aspect in Section IV, followed by a discussion on
content generation in Section V.

Figure 3: The proposed two benchmarks in the RaidEnv. The overall balancing
process includes training playtesting agents and content generation with the
playtesting agents process. In this work, we separated the two process as
benchmarks to clarify the benchmark tasks. The integrated process (Automated
Contents Balancing) is considered as the future work by integrating the
results from two benchmarks.

##  IV Benchmark 1: Playtesting Agent with Changing contents

In this section, we present a playtesting agent trained using multiagent
reinforcement learning (MARL) and evaluate its content robustness to different
game contents. To effectively utilize the playtesting agent in various
scenarios, the agent should manifest adaptability to varing content.
Therefore, we assess whether the agent maintains consistent performance across
different skill parameter value settings within the environment.

Furthermore, to reflect the variation in difficulty resulting from content
variations, we propose the use of a generalization score metric that considers
difficulty adjustments.

###  IV-A Environment Settings for Playtesting Agent

Agent In this section, we introduce the MARL agent, MA-POCA[39]. MA-POCA
addresses the posthumous credit assignment problems, which arises when an
agent is removed from the environment during training. This problem is
particularly critical in games where agent death is a common occurrence.
Specifically, MA-POCA is constructed using the counterfactual multi-agent
policy gradient (COMA)[40], incorporating the self-attention layer[41] in the
network to handle posthumous credit assignment problems. The objective of MA-
POCA involves utilizing a counterfactual baseline[42] and updating for agent
$i$ is performed as follows:

$\displaystyle\mathcal{J}^{i}_{critic}=\tau^{i}-\sum_{a}Q(s,(a^{-i},a))$ (1)
$\displaystyle\mathcal{J}^{i}_{actor}=log\pi_{i}(a^{i}|s)Q(s,(a^{-i},a^{i}))$
(2)

where $\tau^{i}$ represents the temporal difference target, $Q$ denotes the
state-value function, and $\pi_{i}$ represents the policy of agent $i$,
applicable to both homogeneous, and heterogeneous agent settings.

State At each time step, an agent collects observations on the boss and the
status of all team members, including position, velocity, health, and
remaining skill cooldown. Additionally, the agent gathers information on
existing skill projectiles. Furthermore, to facilitate generalized behavior
across skill parameters, the agent obtains parameters such as range, damage,
and cast time associated with the skills. All features are normalized based on
their maximum values.

Action Agents have various actions, including move, rotate, and execute
skills. In our setup, each agent can execute a single action per time step,
such as stay, move forward, move backward, turn right, turn left, move left,
move right, and execute skill.

Reward Agents receive individual rewards from the environment based on the
amount of damage they inflict on the boss. The damage reward is calculated as
$damage\times 0.01$. Additionally, agents receive group rewards from the
environment, which are set to $1.0$ when the boss is defeated. Moreover, we
introduce a cooperative element by incorporating a back attack reward. A back
attack occurs when one agent distracts the boss while other agents turn around
and attack the boss’ vulnerable rearside. The reward for successful back
attack is calculated as $damage\times 0.012$, which is higher than the reward
for regular attacks.

###  IV-B Playtesting Agent Evaluation Setup

####  IV-B1 Baselines

RaidEnv requires a playtesting agent that can handle the complexity of MMORPG
games while demonstrating robustness to content variations for PCG
playtesting. We employ three playtesting agents to validate the robustness of
content variations: the reinforcement learning agent utilizing the MA-POCA
algorithm[39], the playtesting-heuristic (PT-HR) agent, and the playtesting-
random (PT-RD) agent.

  * •

MA-POCA agent makes a decision about actions to defeat the Boss and receives
rewards based on combat performance.

  * •

PT-HR agent follows a simple strategy: it moves around the boss agent and
randomly selects available skills during the combat. The agent maintains a
maximum attack range from the boss and avoids its attacks. This is the
baseline for comparing the multi-agent algorithm. Implementation details are
provided in Appendix A.1.

  * •

PT-RD agent selects actions from the action space by uniformly sampling
distributions.

####  IV-B2 Experiment Setup

In this benchmark, we conducted two experiments: comparing the heuristic
playtesting agent to the MARL agent using fixed content parameter settings and
evaluating the agents‘ robustness. In the first experiment, we compare the
basic performance of the PT-RD, PT-HR, and MA-POCA algorithms across various
content parameter values, specifically the win rate for skill ranges of 5, 9,
13, and 17. In the second experiment, we validate the agents’ robustness by
varying the number of content parameter values, which reflects the diversity
of the environment [43]. At the beginning of each episode, the content
parameter value is sampled from the environment, played with the sampled
parameters, and used for learning about changing content parameters. In this
experiment, The content parameters are set as skill ranges, with each agent
having 1, 2, 3, and 4 skills. MA-POCA1 agent is sampled from skill ranges 5,
MA-POCA2 agent is sampled from skill ranges 5 and 9, MA-POCA3 agent is sampled
from skill ranges 5, 9, and 13, while MA-POCA4 is sampled from skill ranges 5,
9, 13, and 17.

To evaluate the change of difficulty fairly, To evaluate the change of
difficulty fairly, we made the population by extracting representative scores
for a given level. The difficulty of level can be determined using various
game logs such as churn rate[44, 45], success rate[2, 46]. Herein, we evaluate
the level of difficulty based on the win rate of games.

####  IV-B3 Evaluation Metrics

Utilizing the approach [43] used to measure the generalization of performance
on environment diversity, we measured the test gap between the heuristic
playtesting agent and the MA-POCA agents with various skill range parameter
values. We measured the performance of a setting that the trained skill range
parameter settings contain in common, while the contents parameter setting
that any agent has not seen was selected for the test.

Unlike the approach in [43], where the game difficulty remained constant
across parameter changes, in our case varied with parameter changes.
Therefore, these differences must be considered in-game difficulty. For
example, the scenario becomes more challenging if the attack range of the
skill is reduced by adjusting its range parameters, and we need to account for
this difference. We used the term $score^{unseen}$ to denote the agent‘s
performance in the unseen contents parameter setting. Thereafter, We generated
a population (N = 5) by training with the unseen contents parameter setting
and obtained the average performance $score^{unseen}_{population}$ of the
population. We then calculated the adjusted score $AdjustedScore$ using the
following formula:

$\displaystyle
AdjustedScore=\frac{score^{unseen}}{score^{unseen}_{population}}$ (3)

In this metric, we defined the score as the win rate which represents explicit
observable results and We make the MA-POCA populations in skill ranges 3, 4,
19, and 20, using range parameter values.

###  IV-C Experimental Result & Discussion

####  IV-C1 MARL vs Heuristic

TABLE II: Win rate between MARL and Heuristic agent Parameter | $Range$
---|---
Value | 5 | 9 | 13 | 17
MA-POCA |  | 0.485
---
($\pm{0.013}$)
| 0.857
---
($\pm{0.143}$)
| 0.990
---
($\pm{0.006}$)
| 0.990
---
($\pm{0.008}$)
PT-Heuristic |  | 0.089
---
($\pm{0.017}$)
| 0.371
---
($\pm{0.025}$)
| 0.698
---
($\pm{0.021}$)
| 0.830
---
($\pm{0.089}$)
PT-Random |  | 0.000
---
($\pm{0.000}$)
| 0.002
---
($\pm{0.002}$)
| 0.029
---
($\pm{0.006}$)
| 0.058
---
($\pm{0.013}$)

In this section, we present a comparison of the average win rate among the MA-
POCA agent, the playtesting heuristic agent (PT-HR), and the playtesting
random agent (PT-RD) over 500 games. The objective is to demonstrate why RL
agents can be used to obtain a more reliable test result. The MA-POCA agent
trained 50 million steps for every 5 runs, according to the skill range
setting. As in Table II, MA-POCA agents both showed better results than PT-HR
and PT-RD in the corresponding settings regardless of the range. In detail,
the PT-HR agent shows a worse result as the skill range is decreasing. This
indicates that the PT-HR has limitations on solving difficult levels of the
game, while complex MA-POCA agent solves it well. The PT-RD agent cannot win
even one game in range 5 because the game is much more difficult because the
agents should get close to the boss. On the contrary, as range is increasing,
even simple PT-RD agents can win the boss, which indicates the game is really
easy. The MA-POCA agent converged successfully, unlike the relatively fast
convergence of skill ranges 13 and 17 settings, skill ranges 5 and 9 settings
require more steps in convergence on training(See Table V in Appendix F).

####  IV-C2 Robustness of MARL Agent

Figure 4: Figure of adjusted score results in test skill range settings. The
red line indicates the average adjusted score of the population (N=5) in
unseen settings. Adjusted score is the ratio of the test performance of the
trained agent to the average performance of the population. As the adjusted
score achieves a higher test adjusted score, the score is near 1 (sometimes
even higher than 1).

We conducted each MA-POCA1, MA-POCA2, MA-POCA3, and MA-POCA4 5 runs training
50 million steps per set, which is based on the fastest convergence achieved
by the MA-POCA1 agent. After the training, the performance on skill ranges 3,
4, 19, and 20, which were unseen during training were evaluated and compared.
Figure 4, illustrates the comparison between the trained agents and non-
learning PT-HR agents for each MA-POCA agent. These figures depict the
relative game difficulty achieved compared to the average performance of the
population (N = 5), which serves as an indicator of the game difficulty at
each skill range. In this case, the population was created by training
separate MA-POCA agents for each range.

Overall, MA-POCA agents outperformed heuristic agents in the test content
parameter settings. Among the MA-POCA agents, MA-POCA3 exhibited the highest
average performance for the test content parameter settings (3, 4, 19, and 20)
(refer to Table V in Appendix F). This indicates successful learning of
strategies for different situations such as the boss’ melee range, between the
boss’ melee and distant skills, and beyond the boss’ melee range. In contrast,
MA-POCA4 displayed intensive learning of long-distance skills but struggled
with short-distance encounters. MA-POCA1 demonstrated consistent performance
across all test content parameter settings because of its reliance on a single
strategy. Notably, all agents struggled to learn in the range 3 setting,
suggesting room for further improvement and development.

####  IV-C3 Discussion

Figure 5: Heatmap of the average game position occupancy of MA-POCA1, MA-POCA3
in 500 games. The results show that MA-POCA1 agent does not change strategy
well even skill range changes. On the contrary, MA-POCA3 agent recognized the
change of parameter so that change the strategy of movement.

We analyzed the movement patterns of MA-POCA1 and MA-POCA3 agents based on the
number of sampled skill ranges. We played 500 games for each agent and
averaged their occupied locations. As depicted in Figure 5, the MA-POCA1
agent, which only observed a skill range of 5, exhibited consistent behavior
regardless of skill range changes. In contrast, the MA-POCA3 agent recognized
the variation in skill ranges and adapted different strategies accordingly.

Although MARL algorithms have demonstrated successful learning across all
content difficulties, a fully robust agent has yet to be achieved.
Specifically, MA-POCA4 encountered difficulties in learning even with seen
contents (range 5) (refer to Table V in Appendix F). This suggests that the
MA-POCA4 agent struggled to adapt to challenging environments due to
overfitting caused by its ease in learning from simpler environments.
Furthermore, in the case of the content parameter range 3, representing
extremely challenging levels, the agent failed to generalize altogether.
Nevertheless, the MA-POCA agent showcases its strength in slightly more
challenging environments. We anticipate that learning can be accelerated
through approaches such as curriculum learning.

##  V Benchmark 2: Contents Generation

In this section, we present three baselines for the content generation task in
the boss raid scenario. The primary objective of designing procedural content
generation (PCG) agents is to generate or rebalance game content to achieve
specific target game outcomes. For instance, a game designer may request the
generator agent to produce a skill that yields a desired win rate (e.g., 60%),
and the agent should adjust the skill parameters to achieve the designated win
rate. This approach can be applied to both generating new game content from
scratch and rebalancing existing content. We introduce two notations for this
task: (1) current win rate, $W_{c}$, which represents the simulated result
with arbitrary game content, and (2) target win rate, $W_{t}$, which is the
desired outcome of the generation process. The ultimate goal of this task is
to minimize the difference between $W_{c}$ and $W_{t}$ by adjusting the skill
parameter values.

As an example of skill content generation, we propose a deep reinforcement
learning (DRL) method. The DRL framework for PCG was initially introduced in
[47], and the authors of [48] further enhanced this framework to improve its
controllability. As a machine learning approach for content generation, we
adopt a DRL-based baseline utilizing the controllable PCGRL framework
presented in the previous study [47]. The specific details of the DRL
implementation for our work are described below. To evaluate the generated
content, we employ the heuristic playtesting agent discussed in the previous
section (refer to Appendix A.1), and we refer to this agent as the playtesting
heuristic (PT-HR) agent to avoid confusion with the heuristic PCG agent.

###  V-A Environment Settings for DRL

Agent: PCGRL (PCG via RL) [47] proposed a new framework for generating game
content by training a DRL-based generative model. The state is defined as the
scalar or tabular representation of game content; the action involves
modifying the content; and the reward is measured based on the completeness of
the generated content. Subsequent works, such as controllable PCGRL have been
introduced for two-dimensional [48] and three-dimensional games [35] by
designing goal-oriented reward functions that calculate the distance to a
designer-defined value. Controllable PCGRL enables the training of a
generative model to produce a desired output specified by the designer, therby
enhancing the usability of PCGRL models. In this study, we utilize
controllable PCGRL to demonstrate the skill generation task in Section V,
training the generative model to generate diverse game outcomes.

State: The state for the generator includes the skill parameter values
required for generating the target skill. The generator agent receives four
scaled values representing the balancing parameters: cool time, range, damage,
and cast time. In our setup, each value is scaled between a minimum and
maximum value. For example, the cool time, range is defined as 0.5-0.6, the
skill range is set from 1 to 20, the damage range is between 0.5 and 1, and
the cast time falls within the interval of 0.5-1.5. For a comprehensive list
of all environmental variables, please refer to Table VI in Appendix VII-C.

Action: The generator has an action space of $N$, where each action consists
of five discrete values. Here, $N$ represents the number of skills, and the
action determines the parameter updates. We consider four parameters for skill
generation ($N$ = 4): cool time, range, damage, and cast time. The action
determines the increment or decrement of each parameter using a hand-crafted
scaler. The granularity of the scaler was set to 0.16% of each parameter range
allowing for decrease (-) or increases (+) of [-0.16%, -0.08%, 0.0%, +0.08%,
+0.16%] within each parameter range.

Reward: The reward function is designed to provide non-sparse reward signals
to the generator. The generator receives positive rewards when it updates the
skill parameters to achieve the target game win rate ($g$, goal). To design
the reward function for a controllable generator, we adopt a well-designed
reward system proposed in the PCGRL framework [48, 35]. The L1 norm distance
to the goal, $l$, is calculated as $l_{t}=abs(||g_{t}-p_{t}||_{L1})$, where
$p_{t}$ represents the playtested win rate at time $t$. The reward at time $t$
is then calculated as $r_{t}=l_{t-1}-l_{t}$. A positive reward value indicates
that the updated skill is closer to the target win rate. In summary, the
generator was trained to adjust the skill parameters for achieving the
designer-defined playtesting result.

Notably, $p$ is measured by simulating the generated skills multiple times
under the same conditions. In our setup, we repeat the simulation ($N$ = 100)
using the playtesting-heuristic agent (PT-HR) described in Section IV-B, and
we report the average value of $p$ in the experimental results. The use of the
PT-HR agent is preferred in PCG studies due to its reasonable performance and
lower computational cost.

###  V-B Generator Evaluation Setup

####  V-B1 Baselines

We compare the performance of three content generator models in the content
generation experiment: the proposed PCG agent utilizing the proximal policy
optimization (PPO) algorithm [49], PCG-heuristic (PCG-HR) agent, and PCG-
random (PCG-RD) agent. PPO is a popular DRL algorithm known for its stability
and has been widely used in previous content generation studies [47, 35]. To
provide performance comparisons for the PPO agent, we include a simple
heuristic agent and a random agent.

  * •

PPO agent is based on the DRL approach and is implemented using the PCGRL
framework, as detailed in Section V-A. PPO agent enacts decisions to update
skill parameters and receives rewards based on the playtesting results.

  * •

PCG-HR agent randomly adjusts one of the four skill parameters by comparing
the playtesting and target win rates. Specifically, The agent increases or
decreases the parameter value according to a predefined rule based on human
knowledge. The details of the implementation can be found in Algorithm 2 in
Appendix VII-A.

  * •

PCG-RD agent adjusts the parameters without considering the target win rate.
The agent randomly samples one action from five possible action branches in
uniform probability. Each random action is applied to the four skill
parameters.

####  V-B2 Experiment Setup

We evaluated the performance of the generator models, PPO, PCG-HR, and PCG-RD.
We compiled 300 game results at each step to ensure an accurate evaluation of
the generated contents. The heuristic playtesting agent (PT-HR in Section
IV-B) was employed during the playtesting process. Additionally, this section
focuses on demonstrating the generation of skill contents, and the heuristic
method speeds up the experiment time and enhances the reliability of the
evaluation.

Since PPO is trained from a randomly initialized neural network, we conduct
the experiment five times to mitigate performance variations caused by initial
weights. After training the PPO models for 20K steps, we generate 1K skills
for each model, resulting in a total of 5K samples (5 models $\times{}$ 1K
samples = 5K samples). All experimental results are summarized by averaging
the simulated game results across the five seed runs.

####  V-B3 Evaluation Metrics

To evaluate the contents generated by PPO, PCG-HR, and PCG-RD agents is
performed using two commonly used metrics in PCG studies: (1) Controllability
and (2) Diversity. A lack of controllability implies that the agent cannot
generate contents that fulfill the game designer’s requirements, while a lack
of diversity results in repetitive content that can only be used once.

Controllability Controllability is measured by evaluating the error between
the designer’s desired target win rate ($W_{t}$) and the simulated results
obtained from the generated skills ($W_{c}$). The error measurement for
controllability is defined in Equation 4, where $N$ represents the number of
samples.

$\displaystyle WinrateError=\sum_{i}^{N}|W_{i,t}-W_{i,c}|$ (4)

$W_{t}$ denotes the target win rate desired by the designer, and $W_{c}$
represents the win rate of the generated skill. The equation calculates the
mean error of the win rates of skills relative to the target value. The lower
error indicates that the skills produced by the generator are more closely
aligned with the designer’s requirements.

Diversity Diversity measures the variety of styles in which the contents are
generated while still satisfying the intended conditions. A higher diversity
enriches the gameplay experience and provides designers with a wider range of
options. The procedure for sampling skills is the same as in controllability;
however, we apply a root mean squared error (RMSE) $<0.1$ filter to evaluate
the diversity of the contents. This filtering ensures that the evaluated
diversity represents the skill parameter ranges satisfying the designer-
defined win rates, thereby establishing the diversity metric as a more
meaningful benchmark for usability. Thus, diversity assesses the extent to
which the generator offers diverse options to the designer based on the
designer’s desired conditions. A higher value indicates that the generator has
generated contents with diverse skill parameter ranges, resulting in various
game experiences even with the same balanced results ($W_{t}$). To simplify
the interpretation of skill diversity, we employ principal component analysis
(PCA) [50], a dimension reduction technique, to represent the four parameters
as a single value.

###  V-C Experimental Result & Discussion

####  V-C1 Controllability

TABLE III: Descriptive statistics on the controllability of the generative models |  | Generated ($W_{c}$) | RMSE ($\|W_{t}-W_{c}\|$)
---|---|---|---
|  | Mean ($\pm$SD) | Mean ($\pm$SD)
| Target
---
($W_{t}$)
Method |  |  |  |
0.1 | PCG-HR |  0.107 ($\pm{}$0.065) |  0.044 ($\pm{}$0.049)
PPO |  0.149 ($\pm{}$0.242) |  0.156 ($\pm{}$0.191)
PCG-RD |  0.354 ($\pm{}$0.389) |  0.335 ($\pm{}$0.321)
0.2 | PCG-HR |  0.206 ($\pm{}$0.098) |  0.074 ($\pm{}$0.065)
PPO |  0.266 ($\pm{}$0.343) |  0.268 ($\pm{}$0.225)
PCG-RD |  0.366 ($\pm{}$0.381) |  0.333 ($\pm{}$0.249)
0.3 | PCG-HR |  0.297 ($\pm{}$0.108) |  0.079 ($\pm{}$0.073)
PPO |  0.217 ($\pm{}$0.189) |  0.169 ($\pm{}$0.119)
PCG-RD |  0.418 ($\pm{}$0.382) |  0.348 ($\pm{}$0.196)
0.4 | PCG-HR |  0.416 ($\pm{}$0.122) |  0.089 ($\pm{}$0.085)
PPO |  0.377 ($\pm{}$0.315) |  0.276 ($\pm{}$0.153)
PCG-RD |  0.381 ($\pm{}$0.394) |  0.362 ($\pm{}$0.154)
0.5 | PCG-HR |  0.489 ($\pm{}$0.133) |  0.099 ($\pm{}$0.089)
PPO |  0.435 ($\pm{}$0.334) |  0.296 ($\pm{}$0.167)
PCG-RD |  0.390 ($\pm{}$0.391) |  0.379 ($\pm{}$0.144)
0.6 | PCG-HR |  0.592 ($\pm{}$0.132) |  0.097 ($\pm{}$0.090)
PPO |  0.606 ($\pm{}$0.309) |  0.276 ($\pm{}$0.138)
PCG-RD |  0.409 ($\pm{}$0.395) |  0.405 ($\pm{}$0.168)
0.7 | PCG-HR |  0.684 ($\pm{}$0.113) |  0.088 ($\pm{}$0.073)
PPO |  0.652 ($\pm{}$0.354) |  0.292 ($\pm{}$0.205)
PCG-RD |  0.387 ($\pm{}$0.397) |  0.445 ($\pm{}$0.239)
Mean | PCG-HR |  - |  0.081 ($\pm{}$0.078)
PPO |  - |  0.248 ($\pm{}$0.183)
PCG-RD |  - |  0.373 ($\pm{}$0.221)

The descriptive results for measuring the controllability of the generated win
rate targets ($W_{t}$) are listed in Table III. Cells with a blue background
color indicate lower values, while red cells indicate higher values. We
trained the PPO model using seven target parameters ranging from 0.1 to 0.7
and compared its performance with the mentioned approaches. Each of the three
models generated $N=1,000$ skill parameters for each target ($W_{t}$) and was
evaluated using the heuristic playtesting agent (PT-HR).

According to this criterion, the PCG-HR agent demonstrated the best
performance, followed by the PPO agent in second place, and the PCG-RD agent
in last place. The PCG-HR agent consistently achieved the lowest RMSE values
across all target win rate conditions. On average, its error in win rate
estimation was approximately 8.1%. Surprisingly, the PPO agent performed worse
than the heuristic agent. However, we observed that the PPO model was trained
to minimize win rate errors compared to the random agent. The suboptimal
performance of the PPO agent can be attributed to the presence of noise in the
reward signal, indicating that the win rate measurement could be improved by
increasing the number of simulations. The stochastic reward signal had a
negative impact on the learning capabilities of the DRL-based models.

Despite its second-place ranking in this evaluation, the PPO agent has an
advantage as a learning-based algorithm when it comes to skill sampling.
Unlike the PCG-HR agent, which requires a simulation process before executing
an action, the PPO agent can sample actions without relying on playtesting
results. This feature reduces the computational cost associated with content
generation. Consequently, the DRL-based method (PPO) can efficiently sample
the skills if its performance is improved to match that of the heuristic
agent.

####  V-C2 Diversity

TABLE IV: Descriptive statistics on the diversity of the generated contents |  | Range |  | Cool
---
Time
| Cast
---
Time
Damage | PCA
|  | ($\pm$SD) | ($\pm$SD) | ($\pm$SD) | ($\pm$SD) | ($\pm$SD)
| Target
---
($W_{t}$)
Method |  |  |  |  |
0.1 | PCG-HR |  0.258 |  0.300 |  0.269 |  0.265 |  0.302
PPO |  0.315 |  0.482 |  0.258 |  0.472 |  0.625
PCG-RD |  0.266 |  0.325 |  0.294 |  0.283 |  0.336
0.2 | PCG-HR |  0.262 |  0.324 |  0.249 |  0.288 |  0.298
PPO |  0.351 |  0.381 |  0.292 |  0.428 |  0.421
PCG-RD |  0.284 |  0.329 |  0.279 |  0.316 |  0.359
0.3 | PCG-HR |  0.271 |  0.312 |  0.265 |  0.292 |  0.312
PPO |  0.218 |  0.398 |  0.261 |  0.349 |  0.488
PCG-RD |  0.268 |  0.331 |  0.264 |  0.308 |  0.329
0.4 | PCG-HR |  0.252 |  0.312 |  0.233 |  0.278 |  0.304
PPO |  0.180 |  0.377 |  0.364 |  0.476 |  0.559
PCG-RD |  0.277 |  0.333 |  0.287 |  0.292 |  0.358
0.5 | PCG-HR |  0.257 |  0.316 |  0.236 |  0.269 |  0.310
PPO |  0.204 |  0.393 |  0.391 |  0.405 |  0.519
PCG-RD |  0.278 |  0.328 |  0.280 |  0.269 |  0.326
0.6 | PCG-HR |  0.247 |  0.313 |  0.236 |  0.269 |  0.303
PPO |  0.105 |  0.284 |  0.214 |  0.212 |  0.258
PCG-RD |  0.261 |  0.317 |  0.302 |  0.306 |  0.385
0.7 | PCG-HR |  0.257 |  0.293 |  0.227 |  0.263 |  0.293
PPO |  0.194 |  0.271 |  0.447 |  0.406 |  0.625
PCG-RD |  0.262 |  0.322 |  0.258 |  0.305 |  0.358
Mean | PCG-HR |  0.258 |  0.310 |  0.246 |  0.275 |  0.303
PPO |  0.237 |  0.375 |  0.328 |  0.402 |  0.513
PCG-RD |  0.271 |  0.326 |  0.281 |  0.297 |  0.351

Table IV presents the descriptive results for measuring the diversity of the
generated content characteristics. Different background colors represent each
parameter type, with darker colors indicating higher values. We provide the
standard deviation (SD) values for each skill parameter in separate columns of
Table IV. The SD value is calculated using $N=300$ samples for each parameter,
and the samples are filtered based on a threshold (RMSE$<0.1$) to measure
diversity only with valid skills that meet the designer’s requirements. A
higher SD value indicates that the generator generated contents with diverse
skill parameter ranges, resulting in various game experiences even with the
same balanced results ($W_{t}$). To better understand the diversity of the
skills, which can be challenging with separated variance results, we employed
the principal component analysis (PCA) [51] technique, a dimension reduction
technique, to represent the four parameters in a one-dimensional space. By
representing the skills in this space, we measured the variance of the skills
using a single value that encompasses all the parameters of the skill content.
The average SD is calculated across all win rates. The best values are marked
in bold, and PPO outperformed the other models in diversity, showing the best
performance in 6 out of 7 conditions. This result implies that the PPO agent
has a good ability to provide rich options for designers even when given a
specific condition.

####  V-C3 Discussion

In this section, we summarize the two benchmark results, controllability, and
diversity, and conduct further analysis. The controllability results indicate
that the heuristic agent outperformed other methods, including PPO, and random
agents. In the evaluation of diversity, we proposed a method to measure
content diversity using variances of multiple parameters. In diversity
criteria, the PPO agent outperformed other methods, showing the highest
variance in generated skill parameters. For further analysis, we utilized
t-distributed stochastic neighbor embedding (t-SNE) [52], a popular
dimensional reduction technique for data visualization. Figure 6 illustrates
examples of visualizations for the heuristic and PPO agents, respectively.

Figure 6: Visualization of generated skills in a two-dimensional plot using
t-SNE, based on range parameter. Totally, 2,100 entries (300 samples $\times$
7 target win rates) are involved, respectively. Note that gray dots denote all
win rates, and colorized dots denote a specific target win rate (in this case,
$W_{t}=0.3$).

This study sheds light on the contrasting distribution patterns between the
heuristic and PPO approaches in skill generation. The heuristic approach
exhibits a continuous distribution (Heuristic in Fig. 6), indicating a wide
range of skills. In contrast, the PPO approach shows clustering (PPO in Fig.
6), suggesting a narrower focus on skill creation despite its distinct
characteristics. This implies that RL is capable of identifying and leveraging
specific skill parameters to achieve optimal results. However, as the RL aims
to determine an optimal solution, its exploration of alternative states may be
restricted. Instead of seeking a single optimal solution, an alternative
approach is needed to encourage the algorithm to discover different states and
their corresponding skill parameters. The skill parameters generated through
PPO exhibit clustering compared to the PCG-HR approach. However, it can be
observed that they are generally spread out, indicating higher degree of
diversity in PPO which can also be verified numerically.

In this study, the heuristic algorithm was successfully employed by selecting
and implementing four skill parameters, enabling experiments to be conducted
at a usable level. However, commercial game environments possess complex
variables that are challenging to hand-craft, necessitating a more
comprehensive, and automated approach. Future research should focus on
developing methodologies that allow RL to autonomously discover states and
corresponding skill parameters in complex environments, rather than solely
manipulating parameter values. Accordingly, the adaptability of the algorithm
can be enhanced to a broader range of scenarios and its effectiveness can be
improved for generating optimized skill parameters.

In conclusion, this study offers valuable insights into the distribution
patterns and characteristics of heuristic and PPO approaches in skill
parameter generation. The heuristic approach showcases versatility by
generating a wide range of skills, while the PPO approach exhibits distinct
characteristics albeit with a narrower focus. To overcome limitations and
enhance the effectiveness of RL, future research should explore approaches
that foster the exploration of diverse states, rather than solely pursuing a
singular optimal solution.

##  VI Conclusion & Future Work

In this paper, we introduce two benchmarks for training playtesting agents and
generator models. We have developed a multiplayer game environment,
specifically a boss raid benchmark, based on several MMORPG games. The
customizable game components are designed to resemble the settings found in
commercial-level games. The content generation track presents a novel approach
to generating game skills, while the playtesting track emphasizes the
importance of generalization ability when evaluating newly generated content.
We anticipate that the development of the generate-and-test process will
facilitate the integration of research-level algorithms into the industry.
Furthermore, this environment offers game researchers the opportunity to
explore new studies with highly customizable game variables.

Note that the current benchmarks in RaidEnv provide a simplistic setting. They
are based on the assumption of a homogeneous agent, focused on target damage-
based skills, and feature relatively simple content mechanisms, which limit
the demonstration of more complex game transitions. Future endeavors should
consider forming benchmarks that involve heterogeneous agents, such as various
classes (tankers, proximity dealers, distance dealers), robustness to
different content parameters, and content with more intricate mechanisms
(bosses, skills, etc.), which would better align with the collaborative nature
of the game industry.

Our framework offers numerous avenues for future work in multiagent systems
and content generation, which were not explored in this study. Firstly, we
acknowledge the limitation of content types generated in this work, as it only
focuses on skill contents as an example. However, the environment encompasses
a wide range of game contents, including player characteristics, and 15 game
skills, which were not utilized in the experiment. Increasing the controllable
parameters may raise the complexity of action spaces, but it could enrich the
benchmarks and provide valuable insights into controlling generative models to
produce reliable outputs. Diversifying the generated contents, characters, and
skills would create more diverse cooperative raid benchmarks. Secondly,
training robust agents in heterogeneous settings remains a task for future
exploration. In this work, we employed a homogeneous agent setting where the
characteristics and skills are the same for all agents during playtesting.
However, in real boss raid scenarios, the performance of playtesting agents
may decrease when their teammates vary. Therefore, the issue of generalization
to different teammates becomes a significant challenge. Although we have
utilized a limited set of features in our environment to maintain reasonable
benchmark difficulty, we believe that this environment holds the potential to
serve as a research platform that closely resembles commercial games.

##  VII Appendix

###  VII-A PCG & Playtesting Heuristic Agent Algorithm

Algorithm 1 showed the mechanism of playtesting heuristic (PT-HR) agent, and
Algorithm 2 showed the mechanism of PCG-heuristic (PCG-HR) agent.

Algorithm 1 Heuristic Playtesting Agent

Heuristic agent H, Target agent T,

Heuristic agent skill $S_{\textit{H}}$, Heuristic agent position
$P_{\textit{H}}$,

Target agent position $P_{\textit{T}}$

if $P_{\textit{T}}-P_{\textit{H}}<S_{\textit{H}}.range$ then $\triangleright$
Keep maximum skill range

Away from T

else

Close to T

end if

if $RandomNumber/2\equiv 0$ then $\triangleright$ Move around target

Move Clockwise to T

else

Move CounterClockwise to T

end if

if $P_{\textit{T}}-P_{\textit{H}}<S.range\ \land\ !{\textit{H}}.\text{Moving}\
\land\ !{\textit{H}}.\text{Casting}$ then

Execute(skill) $\triangleright$ Attack target with skill

end if

Algorithm 2 Heuristic PCG Agent

Input: Current skill parameters $S$
Current winrate $W_{c}$, Target winrate $W_{t}$

$S\leftarrow$ $S\prime$ $\triangleright$ Copy the skill parameters

$selection\leftarrow$ random value between 0 and 3

if $W_{c}<W_{t}$ then $\triangleright$ Make the game easier

switch $selection$ do

case $0$ $\triangleright$ Increase the traveling range

S.range $\leftarrow$ S.range $+$ 1.5

case $1$ $\triangleright$ Decrease the cool time

S.cast $\leftarrow$ S.cooltime $-$ 10.0

case $2$ $\triangleright$ Decrease the cast time

S.cost $\leftarrow$ S.casttime $-$ 0.33

case $3$ $\triangleright$ Increase the attacking damage

S.damage $\leftarrow$ S.damage $+$ 0.33

else

switch $selection$ do

case $0$ $\triangleright$ Increase the traveling range

S.range $\leftarrow$ S.range $-$ 1.5

case $1$ $\triangleright$ Decrease the cool time

S.cast $\leftarrow$ S.cooltime $+$ 10.0

case $2$ $\triangleright$ Decrease the cast time

S.cost $\leftarrow$ S.casttime $+$ 0.33

case $3$ $\triangleright$ Increase the attacking damage

S.damage $\leftarrow$ S.damage $-$ 0.33

end if

###  VII-B Playtesting Agent Result

TABLE V: Test and Train performance with adjustment of level difficulty | Parameter | $Range$ |
---|---|---|---
| $\mathcal{S}$ | 3(Hard) | 4(Hard) | 5(Train) | 19(Easy) | 20(Easy) | Test Avg.
| 1 |  | 0.004 ($\pm{0.005}$)
---
| 0.553 ($\pm{0.085}$)
---
| 1.049 ($\pm{0.063}$)
---
| 0.597 ($\pm{0.055}$)
---
| 0.597 ($\pm{0.048}$)
---
| 0.506 ($\pm{0.310}$)
---
MA-POCA | 2 |  | 0.028 ($\pm{0.033}$)
---
| 0.553 ($\pm{0.224}$)
---
| 1.004 ($\pm{0.122}$)
---
| 0.523 ($\pm{0.082}$)
---
| 0.525 ($\pm{0.079}$)
---
| 0.473 ($\pm{0.306}$)
---
| 3 |  | 0.013 ($\pm{0.024}$)
---
| 0.369 ($\pm{0.192}$)
---
| 0.630 ($\pm{0.268}$)
---
| 0.826 ($\pm{0.109}$)
---
| 0.778 ($\pm{0.151}$)
---
| 0.511 ($\pm{0.351}$)
---
| 4 |  | 0.000 ($\pm{0.000}$)
---
| 0.016 ($\pm{0.015}$)
---
| 0.049 ($\pm{0.031}$)
---
| 0.980 ($\pm{0.017}$)
---
| 0.971 ($\pm{0.022}$)
---
| 0.443 ($\pm{0.477}$)
---
PT-Heuristic |  | 0.013 ($\pm{0.007}$)
---
| 0.115 ($\pm{0.015}$)
---
| 0.199 ($\pm{0.038}$)
---
| 0.89 ($\pm{0.011}$)
---
| 0.868 ($\pm{0.021}$)
---
| 0.441 ($\pm{0.396}$)
---
TABLE VI: The detailed description of the controllable parameters on RaidEnv

Parameter | Name | Data Type | Range | Description
---|---|---|---|---
Character Statistics | Status | Health Point | integer | [0, 1000] | Maximum number that agent can be dealing with the damage
Manna Point | integer | [0,100] | The resource that used when agent active a skill
Spell Power | integer | [0, 100] | Reference number used when calculating the skill damage
Movement Speed | float | [1, 2] | Move speed of the agent
Attack | Power | integer | [0, 100] | Reference number used when calculating the melee attack damage
Range | integer | [1, 10] | Maximum distance of the melee attack can be reached
Speed | float | [1, 2] | Number of melee attacks can be activated per second
Defensive | Armor | integer | [0, 100] | Ratio that agent dealt reduced damage
Evasion | integer | [0, 100] | Probability that agent can be dodge melee attack or skill
Parry | float | [0, 100] | Probability that agent can block melee attack or skill
Primary | Strength | integer | [1, 100] | Measure of physical power and proficiency. Affects agent melee attack damage, etc.
Agility | integer | [1, 100] | Measure of swiftness. Affects agent evasion, accuracy, etc.
Intelligence | integer | [1, 100] | Measure of mental power and magical prowess. Affect agent magical attack damage, spell effectiveness, etc.
Secondary | Critical | integer | [0, 100] | Chance that agent ability can more effect than normal activation
Haste | integer | [0, 100] | Determine the agent’s action swiftness as in casting, cool time, attack speed, etc
Versatility | integer | [0, 100] | Increase the agent’s attack and defensive abilities effect
Mastery | integer | [0, 100] | Enhance some abilities effect to emphasize character class mechanism
Skill | Information | Name | string | - | Name given to each skill
Trigger Type | enum | [0, 1] | Determine the skill is active skill or passive skill
Magic School | enum | [0, 9] | Determine the visual effects of the skill(e.g. fire, frost etc)
Hit Type | enum | [0, 1] | Determine the skill is melee attack or skill
Target Type | enum | [0, 2] | Type of target which the skill can be used on (e.g. target, non-target, region)
Projectile Speed | float | [0, 50] | Speed of the projectile that reaches to the target
Affect on Ally | bool | True/False | Determine that a agent can activate a skill to ally
Affect on Enemy | bool | True/False | Determine that a agent can activate a skill to enemy
Condition | Cool Time | float | [0, 60] | Amount of time required to Activate the skill again
Cast Time | float | [0, 2] | Amount of time required to execute the skill after activation
Cost | float | [0, 100] | Amount of Mana required to Activate the skill
Range | float | [1, 20] | Maximum distance of the skill can be reached
Charge | integer | [1, 3] | Number of skill can be activated multiple times regardless of the Cool time
Cast on moving | bool | True/False | Determine that an agent can activate a skill during the moving
Cast on Casting | bool | True/False | Determine that an agent can activate a skill during casting another skill
Cast on Channeling | bool | True/False | Determine that an agent can activate a skill during executing another skill
Coefficient | Value | float | [0, 2] | Scaling factor that determines the damage of the skill

Table V showed the performance on train and test content parameters. The
results showed that MA-POCA4 and MA-POCA3 are not trained perfectly in trained
settings(seen). Especially, MA-POCA4 has shown that failure to learn about the
strategy when their skill has short distance. Overall, MA-POCA3 showed the
best average performance in unseen environments. However, since all agents are
failure to generalize in skill range 3, it indicates this benchmark is
challenging.

###  VII-C Full Environment Description

The RaidEnv platform offers extensive customization options for various game
features, which can be controlled using different variables. Figure 1 provides
an overview of all the game components, while Table I summarizes the detailed
descriptions, data types, and ranges associated with these components. The
range values presented in the table were determined by the authors but can be
adjusted by other researchers as needed.

## References

  * [1] E. R. Poromaa, “Crushing candy crush: predicting human success rate in a mobile game using monte-carlo tree search,” 2017.
  * [2] S. F. Gudmundsson, P. Eisen, E. Poromaa, A. Nodet, S. Purmonen, B. Kozakowski, R. Meurling, and L. Cao, “Human-like playtesting with deep learning,” in _2018 IEEE Conference on Computational Intelligence and Games (CIG)_. IEEE, 2018, pp. 1–8.
  * [3] F. Lorenzo, S. Asadi, A. Karnsund, L. Cao, T. Wang, and A. H. Payberah, “Use all your skills, not only the most popular ones,” in _2020 IEEE Conference on Games (CoG)_. IEEE, 2020, pp. 682–685.
  * [4] T. Liu, Z. Zheng, H. Li, K. Bian, and L. Song, “Playing card-based rts games with deep reinforcement learning.” in _IJCAI_ , 2019, pp. 4540–4546.
  * [5] A. Summerville, S. Snodgrass, M. Guzdial, C. Holmgård, A. K. Hoover, A. Isaksen, A. Nealen, and J. Togelius, “Procedural content generation via machine learning (pcgml),” _IEEE Transactions on Games_ , vol. 10, no. 3, pp. 257–270, 2018.
  * [6] L. Gisslén, A. Eakins, C. Gordillo, J. Bergdahl, and K. Tollmar, “Adversarial reinforcement learning for procedural content generation,” in _2021 IEEE Conference on Games (CoG)_. IEEE, 2021, pp. 1–8.
  * [7] A. Dockhorn and S. Mostaghim, “Introducing the hearthstone-ai competition,” _arXiv preprint arXiv:1906.04238_ , 2019.
  * [8] M. Stephenson, E. Piette, D. J. Soemers, and C. Browne, “Ludii as a competition platform,” in _2019 IEEE conference on Games (CoG)_. IEEE, 2019, pp. 1–8.
  * [9] D. Perez-Liebana, J. Liu, A. Khalifa, R. D. Gaina, J. Togelius, and S. M. Lucas, “General video game ai: A multitrack framework for evaluating agents, games, and content generation algorithms,” _IEEE Transactions on Games_ , vol. 11, no. 3, pp. 195–214, 2019.
  * [10] M. Wydmuch, M. Kempka, and W. Jaśkowski, “ViZDoom Competitions: Playing Doom from Pixels,” _IEEE Transactions on Games_ , vol. 11, no. 3, pp. 248–259, 2019, the 2022 IEEE Transactions on Games Outstanding Paper Award.
  * [11] M. Kempka, M. Wydmuch, G. Runc, J. Toczek, and W. Jaśkowski, “ViZDoom: A Doom-based AI research platform for visual reinforcement learning,” in _IEEE Conference on Computational Intelligence and Games_. Santorini, Greece: IEEE, Sep 2016, pp. 341–348, the Best Paper Award.
  * [12] A. Khalifa, “Mario-AI-Framework,” https://github.com/amidos2006/Mario-AI-Framework, Jul 11 2022, [Online; accessed 2023-03-13].
  * [13] S. Karakovskiy and J. Togelius, “The mario ai benchmark and competitions,” _IEEE Transactions on Computational Intelligence and AI in Games_ , vol. 4, no. 1, pp. 55–67, 2012.
  * [14] K. Kurach, A. Raichuk, P. Stańczyk, M. Zając, O. Bachem, L. Espeholt, C. Riquelme, D. Vincent, M. Michalski, O. Bousquet _et al._ , “Google research football: A novel reinforcement learning environment,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 04, 2020, pp. 4501–4510.
  * [15] H. Jia, Y. Hu, Y. Chen, C. Ren, T. Lv, C. Fan, and C. Zhang, “Fever basketball: A complex, flexible, and asynchronized sports game environment for multi-agent reinforcement learning,” _arXiv preprint arXiv:2012.03204_ , 2020.
  * [16] W. Wang and R. Zhang, “Improved game units balancing in game design through combinatorial optimization,” in _2021 IEEE International Conference on e-Business Engineering (ICEBE)_. IEEE, 2021, pp. 64–69.
  * [17] M. Preuss, T. Pfeiffer, V. Volz, and N. Pflanzl, “Integrated balancing of an rts game: Case study and toolbox refinement,” in _2018 IEEE Conference on Computational Intelligence and Games (CIG)_. IEEE, 2018, pp. 1–8.
  * [18] S. M. Nikolakaki, O. Dibie, A. Beirami, N. Peterson, N. Aghdaie, and K. Zaman, “Competitive balance in team sports games,” in _2020 IEEE Conference on Games (CoG)_. IEEE, 2020, pp. 526–533.
  * [19] J. Pfau, A. Liapis, G. Volkmar, G. N. Yannakakis, and R. Malaka, “Dungeons & replicants: automated game balancing via deep player behavior modeling,” in _2020 IEEE Conference on Games (CoG)_. IEEE, 2020, pp. 431–438.
  * [20] J. Pfau, A. Liapis, G. N. Yannakakis, and R. Malaka, “Dungeons & replicants ii: automated game balancing across multiple difficulty dimensions via deep player behavior modeling,” _IEEE Transactions on Games_ , 2022.
  * [21] M. Samvelyan, T. Rashid, C. S. De Witt, G. Farquhar, N. Nardelli, T. G. Rudner, C.-M. Hung, P. H. Torr, J. Foerster, and S. Whiteson, “The starcraft multi-agent challenge,” _arXiv preprint arXiv:1902.04043_ , 2019.
  * [22] B. Ellis, S. Moalla, M. Samvelyan, M. Sun, A. Mahajan, J. N. Foerster, and S. Whiteson, “Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning,” 2022. [Online]. Available: https://arxiv.org/abs/2212.07489
  * [23] J. Z. Leibo, E. A. Dueñez-Guzman, A. Vezhnevets, J. P. Agapiou, P. Sunehag, R. Koster, J. Matyas, C. Beattie, I. Mordatch, and T. Graepel, “Scalable evaluation of multi-agent reinforcement learning with melting pot,” in _International conference on machine learning_. PMLR, 2021, pp. 6187–6199.
  * [24] J. Suarez, Y. Du, P. Isola, and I. Mordatch, “Neural mmo: A massively multiagent game environment for training and evaluating intelligent agents,” _arXiv preprint arXiv:1903.00784_ , 2019.
  * [25] P. Beau and S. Bakkes, “Automated game balancing of asymmetric video games,” in _2016 IEEE conference on computational intelligence and games (CIG)_. IEEE, 2016, pp. 1–8.
  * [26] S. Reis, L. P. Reis, and N. Lau, “Vgc ai competition-a new model of meta-game balance ai competition,” in _2021 IEEE Conference on Games (CoG)_. IEEE, 2021, pp. 01–08.
  * [27] Y.-J. Gong, J.-X. Guo, D.-L. Lin, Y.-L. Zuo, J.-C. Liang, L.-J. Luo, X.-X. Shao, C. Zhou, and M.-T. Li, “Automated team assembly in mobile games: A data-driven evolutionary approach using a deep learning surrogate,” _IEEE Transactions on Games_ , 2022.
  * [28] R. Herbrich, T. Minka, and T. Graepel, “Trueskill™: a bayesian skill rating system,” _Advances in neural information processing systems_ , vol. 19, 2006\.
  * [29] E. Arpad, “The rating of chessplayers, past and present,” _Arco Pub_ , vol. 216, 1978.
  * [30] K. Sorochan and M. Guzdial, “Generating real-time strategy game units using search-based procedural content generation and monte carlo tree search,” _arXiv preprint arXiv:2212.03387_ , 2022.
  * [31] M. Kaidan, T. Harada, C. Y. Chu, and R. Thawonmas, “Procedural generation of angry birds levels with adjustable difficulty,” in _2016 IEEE Congress on Evolutionary Computation (CEC)_. IEEE, 2016, pp. 1311–1316.
  * [32] L. N. Ferreira and C. F. M. Toledo, “Tanager: A generator of feasible and engaging levels for angry birds,” _IEEE Transactions on Games_ , vol. 10, no. 3, pp. 304–316, 2017.
  * [33] T. Shu, J. Liu, and G. N. Yannakakis, “Experience-driven pcg via reinforcement learning: A super mario bros study,” in _2021 IEEE Conference on Games (CoG)_. IEEE, 2021, pp. 1–9.
  * [34] V. Volz, J. Schrum, J. Liu, S. M. Lucas, A. Smith, and S. Risi, “Evolving mario levels in the latent space of a deep convolutional generative adversarial network,” in _Proceedings of the genetic and evolutionary computation conference_ , 2018, pp. 221–228.
  * [35] Z. Jiang, S. Earle, M. Green, and J. Togelius, “Learning controllable 3d level generators,” in _Proceedings of the 17th International Conference on the Foundations of Digital Games_ , 2022, pp. 1–9.
  * [36] M. Seif El-Nasr, B. Aghabeigi, D. Milam, M. Erfani, B. Lameman, H. Maygoli, and S. Mah, “Understanding and evaluating cooperative games,” in _Proceedings of the SIGCHI conference on human factors in computing systems_ , 2010, pp. 253–262.
  * [37] N. P. Zea, J. L. G. Sánchez, F. L. Gutiérrez, M. J. Cabrera, and P. Paderewski, “Design of educational multiplayer videogames: A vision from collaborative learning,” _Advances in Engineering Software_ , vol. 40, no. 12, pp. 1251–1260, 2009.
  * [38] A. Juliani, V.-P. Berges, E. Teng, A. Cohen, J. Harper, C. Elion, C. Goy, Y. Gao, H. Henry, M. Mattar, and D. Lange, “Unity: A general platform for intelligent agents,” _arXiv preprint arXiv:1809.02627_ , 2020.
  * [39] A. Cohen, E. Teng, V.-P. Berges, R.-P. Dong, H. Henry, M. Mattar, A. Zook, and S. Ganguly, “On the use and misuse of absorbing states in multi-agent reinforcement learning,” _arXiv preprint arXiv:2111.05992_ , 2021.
  * [40] J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson, “Counterfactual multi-agent policy gradients,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 32, no. 1, 2018.
  * [41] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [42] D. H. Wolpert and K. Tumer, “Optimal payoff functions for members of collectives,” _Advances in Complex Systems_ , vol. 4, no. 02n03, pp. 265–279, 2001.
  * [43] K. R. McKee, J. Z. Leibo, C. Beattie, and R. Everett, “Quantifying the effects of environment and population diversity in multi-agent reinforcement learning,” _Autonomous Agents and Multi-Agent Systems_ , vol. 36, no. 1, p. 21, 2022.
  * [44] S. Roohi, A. Relas, J. Takatalo, H. Heiskanen, and P. Hämäläinen, “Predicting game difficulty and churn without players,” in _Proceedings of the Annual Symposium on Computer-Human Interaction in Play_ , 2020, pp. 585–593.
  * [45] S. Roohi, C. Guckelsberger, A. Relas, H. Heiskanen, J. Takatalo, and P. Hämäläinen, “Predicting game difficulty and engagement using ai players,” _Proceedings of the ACM on Human-Computer Interaction_ , vol. 5, no. CHI PLAY, pp. 1–17, 2021.
  * [46] J. T. Kristensen, A. Valdivia, and P. Burelli, “Estimating player completion rate in mobile puzzle games using reinforcement learning,” in _2020 IEEE Conference on Games (CoG)_. IEEE, 2020, pp. 636–639.
  * [47] A. Khalifa, P. Bontrager, S. Earle, and J. Togelius, “Pcgrl: Procedural content generation via reinforcement learning,” in _Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment_ , vol. 16, no. 1, 2020, pp. 95–101.
  * [48] S. Earle, M. Edwards, A. Khalifa, P. Bontrager, and J. Togelius, “Learning controllable content generators,” in _2021 IEEE Conference on Games (CoG)_. IEEE, 2021, pp. 1–9.
  * [49] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” 2017.
  * [50] S. Wold, K. Esbensen, and P. Geladi, “Principal component analysis,” _Chemometrics and intelligent laboratory systems_ , vol. 2, no. 1-3, pp. 37–52, 1987.
  * [51] H. Abdi and L. J. Williams, “Principal component analysis,” _Wiley interdisciplinary reviews: computational statistics_ , vol. 2, no. 4, pp. 433–459, 2010.
  * [52] L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” _Journal of machine learning research_ , vol. 9, no. 11, 2008.