# Test and Evaluation Framework for Multi-Agent Systems of Autonomous
Intelligent Agents
††thanks: This paper includes funded research conducted through the System
Engineering Research Center.

Erin Lanus1, Ivan Hernandez1, Adam Dachowicz2, Laura Freeman1, Melanie
Grande2, Andrew Lang2, Jitesh H. Panchal2, Anthony Patrick3, Scott Welch1
1Virginia Tech, Arlington, VA 22309, USA  2Purdue University, West Lafayette,
IN 47907, USA  3George Mason University, Fairfax, VA 22030, USA

###### Abstract

Test and evaluation is a necessary process for ensuring that engineered
systems perform as intended under a variety of conditions, both expected and
unexpected. In this work, we consider the unique challenges of developing a
unifying test and evaluation framework for complex ensembles of cyber-physical
systems with embedded artificial intelligence. We propose a framework that
incorporates test and evaluation throughout not only the development life
cycle, but continues into operation as the system learns and adapts in a
noisy, changing, and contended environment. The framework accounts for the
challenges of testing the integration of diverse systems at various
hierarchical scales of composition while respecting that testing time and
resources are limited. A generic use case is provided for illustrative
purposes and research directions emerging as a result of exploring the use
case via the framework are suggested.

###### Index Terms:

systems engineering, statistical models, software engineering, artificial
intelligence, design of experiments, combinatorial interaction testing

##  I Introduction

The United States engages in numerous strategic initiatives to increase the
use of Artificial Intelligence (AI) to support strategic priorities. Achieving
complex mission needs requires AI to be integrated and deployed in multi-agent
systems of autonomous intelligent agents (AIAs). These systems, if proven to
be reliable, trustworthy, and safe, have the potential to be used in high-
stakes contexts, often with lack of human intervention and under changing
mission and environmental needs. This research is motivated by the challenge
of testing AIAs as compared to static, deterministic systems.

Test and evaluation (T&E) of multi-agent systems of AIAs presents unique
challenges due to the dynamic environments of the agents, adaptive learning
behaviors of individual agents, the complex interactions among the agents, the
complex interactions between agents and the operational environment, the
difficulty in testing black-box machine learning models, and rapidly evolving
AI algorithms. Currently, no unifying framework exists for T&E of multi-agent
systems of AIAs. Existing frameworks for T&E of complex engineered systems [1]
fail to account for these unique challenges.

T&E is a difficult topic to study as different fields of engineering have
evolved their test strategies to meet the specific needs of that field. For
example, the reliability community has a robust literature on reliability
testing [2], the software community has methods for software testing [3], and
manufacturing has methods for testing the consistence of their processes [4].
However, complex systems require the integration of many methods to fully
characterize system capabilities and understand how they will perform in the
actual operational environment. The United States Department of Defense (DoD)
has a mature T&E process due to the nature of the technologies they must
ensure perform adequately and are safe before fielding. These test processes
are documented in the Defense Acquisition Guidebook [1] and provide a
comprehensive overview of these processes, but notably missing is any guidance
on how processes should account for AIA challenges.

The development of multi-agent systems of AIAs involves taking an
interdisciplinary approach, with each discipline providing its own methods,
tools, techniques, priorities, and expertise. Consequently, the collection of
accompanying T&E strategies across a multi-agent system is heterogeneous, and
it is not clear how individual T&E strategies for components or subsystems
should be combined to provide a comprehensive T&E framework for a multi-agent
system of AIAs. Furthermore, new system capabilities, applications,
properties, and behaviors emerge at the intersection of multi-agent,
autonomous, and intelligent systems.

New constructs within T&E are necessary to facilitate addressing these new
challenges in a manageable framework used for systematic analysis. For
example, a multi-agent system-of-systems requires additional testing across a
hierarchical scale as component subsystems are integrated. Testing must be
conducted on “local” factor levels specific to an individual agent as well as
“global” factor levels representing the combined interactions of the different
agents along with environmental conditions. The dynamic nature of multi-agent
interactions can have effects apparent at the lower hierarchical scale of a
given agent and also produce emergent phenomena at higher hierarchical scale.
Additionally, an increase in the number of agents, each with its own
parameters, exponentially increases the number of tests that might be
conducted to support a comprehensive evaluation. Finally, these AIAs will have
the ability to learn over time, so test strategies that continuously evaluate
both the local and global scale over time are needed. The increase in agents
and parameters requires more time and resources for conducting T&E. We
hypothesize that improving efficiency and coverage in a distributed, dynamic
learning environment is essential to a T&E framework for multi-agent systems
of AIAs.

In this work, we propose a unifying framework for T&E of systems of multiple
AIAs to guide the systematic development of test plans. The framework is
informed by three major concepts that address the unique needs of this
context: 1) field of study, 2) hierarchy of test, and 3) test plan efficiency.
Collectively and along with an expanded systems engineering verification and
validation model, these concepts describe how to define a _slice_ of the
process during a phase of the system design, development, and deployment life
cycle in order to identify goals of the test and inform creation of a
comprehensive test plan. The rest of the paper is organized as follows. An
illustrative use case of a satellite system is presented in § II. The VTP
model on which the framework is built is presented in § III. Testing
procedures drawn from fields of study utilized in building these systems of
systems and how they are addressed in the framework are discussed in § IV. How
to conduct integration testing as components are merged into subsystems and
subsystems into systems as a hierarchical approach to testing is considered in
§ V. Maximizing knowledge gained with limited testing resources as the goal of
test plan efficiency is discussed in § VI. The complete framework is given in
§ VII. Finally, conclusions and research directions emerging as a result of
exploring the use case via the framework are suggested § VIII.

##  II Illustrative Use Case

To provide an illustrative practical backdrop, we employ a generic use case of
a satellite network composed of a heterogeneous set of AIAs reporting to a
central controller and acting autonomously to conduct broad area search and
point detection (see Fig. 1). At the local hierarchical scale, each satellite
is composed of component subsystems such as sensors, actuators, and software
including deterministic control software as well as AI software that is
expected to change after deployment as a result of adapting to changing
environments and knowledge acquisition. Each of these subsystems could be
further decomposed into smaller components, such as a piece of hardware or a
function within a program. At the global hierarchical scale, the system can be
described by the types, number, and positions of each satellite, additional
state information such as if the satellite has been damaged or its software
has been compromised, and connectivity of each satellite with each other and
the ground station. Last, operational environmental conditions can be
specified such as lack of visibility, presence of adversarial powers and their
capabilities, and presence and location of observational targets.

Figure 1: Illustrative use case of a multi-agent satellite system

##  III The VTP Model

Figure 2: The VTP framework extends the “Vee” model to include testing
throughout system deployment and a feedback loop

T&E must be integrated throughout not only the system development process, but
also the system life cycle. The Systems Engineering “Vee” model [5] is a
mature model of systems engineering that serves as a sound starting point for
development of a unifying framework (see the left third of Fig. 2). In the
“Vee” model, each system-level in the hierarchy is paired with a corresponding
level of verification and validation. Requirements and test plans are created
from the beginning of the development process, rather than waiting until the
entire system is developed. For example, at the beginning of the project when
requirements for the entire system are identified, tests that verify system
performance are designed though they are not executed until near the end. The
system is thus designed top down, walking down the left side of the “Vee” and
tests in the corresponding slice are defined simultaneously; however, test
execution is conducted “bottom up” as the subsystems are built and the system
is integrated, walking up the right side of the “Vee.”

Since the “Vee” model is based on the principle of hierarchical decomposition,
certain assumptions of the “Vee” model do not hold for testing multi-agent
system of AIAs. Specifically, most systems development life cycles assume
well-defined phases, such as concept studies, technology development,
preliminary design, final design, fabrication, assembly, test, launch,
operations & sustainment, and closeout [6]. The phases in the development
process are akin to the phases in the the Software Engineering Waterfall Model
that include requirement identification, design, implementation, testing, and
maintenance. These models assume that all requirements against which the
system will be tested are able to be listed during a requirements phase, and
that requirements can be decomposed and traced to individual components or
subsystems. Last, maintenance allows for “fixing” a product during deployment,
but does not consider that behavior of the agents could change after
deployment in the field.

These assumptions do not hold for multi-agent systems of AIAs. Specifically,
it may not be possible to define how the system should respond in all
environmental conditions as the environment is constantly changing. For
example, it may not be possible to define in advance the threat capabilities
of advanced adversaries. Requirements may be achieved by multiple combinations
of subsystems. A given task may be achieved by different ensembles of
satellites given their heterogeneous capabilities and positions. After
deployment, the satellite software may be modified by code pushes from the
ground station, but it may also change through learning behaviors as the
embedded AI acquires knowledge from interacting with its environment and
through collaborative decision making with other satellites in the
constellation.

Despite the challenges of sequential design or “big design up front” models,
incremental or iterative models have limited applicability for systems that
cannot be easily recalled once deployed. That is, development of subsystems
such as components on an individual satellite or software may be incrementally
designed, developed, and tested while on the ground, but a clear demarcation
for deployment of the system occurs when the satellite is launched. Thus, our
framework assumes the Systems Engineering “Vee” model up to deployment, though
several iterations of the “Vee” could occur before the deployment cutoff. We
then extend this model with a “T” phase to include testing throughout
operation to detect or respond to events. Such events could include a change
to the mission objective requiring a code push from the ground station to
particular satellites and executing all pertinent tests from the previous
phase as well as new tests to address the code changes. Embedded AI software
may also adapt as a result of learning, and tests must be conducted to ensure
the system is learning the “right” actions. As the system encounters debris or
exhibits hardware degradation over time, tests must be run periodically to
identify faults and enact mitigation strategies. Last, communication systems
are inherently susceptible to adversarial attack, and software may include
intrusion detection algorithms that may need to be updated with new
signatures. Embedded AI should be tested for resilience to newly discovered
vulnerabilities or maturity issues such as data drift.

Last, the model should process feedback. The “P” phase includes a loop back to
the deployment phase due to changes in the system from learning for systems
that are currently deployed. This loop can also extend further to inform the
next phase of system development. The full VTP model on which the framework is
built is presented in Fig. 2. In the figure, the dashed lines delineate the
sub-phases within design, development, and deployment, and these sub-phases
provide timeline context for a slice of the process.

##  IV Field of study

Each AIA is a cyber-physical system with embedded AI. T&E of the composed
system should draw from scientifically-based testing techniques for each
subsystem and thus consider the peculiarities of testing for AI, deterministic
software, electronic hardware, and mechanical systems [7, 8].

For embedded AI, T&E methods should measure inherent weaknesses of the
algorithms in use. For example, the use of neural networks in learning
requires an evaluation strategy that measures the performance sensitivity to
transformations or noise added to the input. This is needed to detect attacks
such as data poisoning and measure the impact on mission success in contended
environments.

The AI functionality of the software is also supported by other deterministic
software, such as functions to receive input from sensors and control
actuators in order to interact with the agent’s environment, as well as to
communicate with other agents within the multi-agent system. For code under
development, white box methods that emphasize structural code coverage (e.g.,
statement, decision, condition, branch, and path coverage) can be employed.
For commercial-out-of-the-box software (COTS) or vendor-supplied software,
black-box techniques are required and include equivalence partitioning,
boundary value analysis, decision tables, state transition testing, use case
testing, and combinatorial techniques.

In testing physical components, statistical analysis of response variables
ascribe variance to different independent variables, or _factors_ , and to
estimate the effect of different factor _levels_ on system performance. Design
of experiments (DOE) is a systematic approach to choosing a set of test cases
to ensure adequate coverage of the operational space, determine how much
testing is enough, and provide an analytical basis for assessing test
adequacy. DOE has also proven useful in testing complex systems with embedded
software [9]. Alternatively, optimal learning [10] is an approach that begins
with an initial set of tests to establish some information about the system. A
Bayesian surrogate of the objective function is trained and the next tests are
chosen based on a heuristic that combines exploration of the test space
exhibiting the most uncertainty with exploitation of areas of the test space
that maximize the objective function.

Additionally, test strategies must evaluate the fully integrated system-of-
systems and its ability to execute tasks autonomously. While each subsystem
may reside within one field of study (e.g., software) and thus testing for the
subsystem may follow known testing strategies for that field, the composed
system spans multiple disciplines and T&E must consider all together. Some
tests may need to be designed that account for the interaction of disparate
systems. For example, a satellite may need to learn that visibility issues
affecting a sensor can be overcome by changes to its position and thus tests
requiring orchestrated interaction of sensors, embedded AI, control software,
and actuators are all required to evaluate this behavior.

Last, the above list is not exhaustive. Depending on the use case in which the
AIAs are employed, additional fields may be considered. Testing AIAs teamed
with humans or with significant human-in-the-loop components should consult
the psychology testing literature for designing tests that address the variety
of unique challenges such as attention issues along the human-computer
interface and how humans and computers can express and understand
collaborative goals. Further, even without human involvement, psychologist SME
consultation can be beneficial to establish benchmarks for learning behaviors
of AIAs and in testing for collaborative behaviors of ensembled AIAs.

##  V Hierarchy of test

Rather than waiting until the complete system is built to test, testing is
conducted throughout the development process in order to detect and correct
flaws as early as possible. Unit testing is conducted on the smallest testable
components, using simulated inputs or digital environments when necessary. As
components are integrated into subsystems, the expectation is that components
work as intended, but there may be interactions among them causing failures or
interaction effects on performance. As an example, suppose some COTS control
software is employed that does not expect the range of inputs produced by a
given sensor on the satellite. When used in conjunction, the code may crash or
unexpected behavior may occur.

Combinatorial interaction testing (CIT) creates test suites to systematically
detect failures caused by combinations of interacting components up to a given
size of interaction[11]. To use available tools such as the Automated
Combinatorial Testing for Software to generate test suites [3], testers must
identify the components (synonymous with factors in DOE), the levels at which
the components should be tested, the maximum interaction size called the
_strength_ , and any _constraints_ , combinations of component levels that
must not be tested together. This process requires that components or factors
of interest are known to the tester, and continuous levels must be discretized
in order to use CIT.

As a complex system of systems, a system that undergoes integration testing in
one phase of testing becomes a component in the next phase. For example, at
the local scale of the hierarchy, payload sensors, actuators, AI algorithm,
and control software are component subsystems integrated into an AIA
satellite, and integration verification and validation testing is performed to
ensure the system performs as expected. Once deployed into a constellation,
testing moves up the hierarchy, and each satellite becomes a subsystem within
the global system. Interaction testing considers failures at the top system or
mission scale, such as whether communication relays between satellites and the
ground station are operational or whether the combined sensor footprint of the
constellation is sufficient for a given tactical operation. Factors at the
global scale of the hierarchy may also include environmental factors that
cannot be controlled but can be observed during a test to evaluate their
impact on mission performance, such as storms affecting visibility. Other
factors may be simulated, such as adversarial attacks.

After deployment, failures occurring at the global scale of the hierarchy may
be caused by interactions of subsystems immediately lower in the hierarchy
along with global scale factors, such as storms affecting sensor visibility of
a particular subset of satellites or interfering with communication with a
ground station. CIT methods include fault localization techniques that can be
used to identify interactions causing the fault [11]. In some cases, a problem
with a component system further down the hierarchy may be causing an error to
propagate up to higher systems. Techniques may need to be utilized to step
down through the involved interactions at each scale of the hierarchy to
locate the component causing the fault.

##  VI Test plan efficiency

Rigorous testing under a variety of conditions provides a degree of assurance
that the system will perform as expected. The test input space is defined by
identification of system and environmental factors of interest and choosing a
range of levels for each factor. The points chosen for a test plan thus result
in some coverage of the multi-dimensional test input space. Each test incurs
some cost and, in most scenarios, both time and resources for testing are
limited. Different techniques prioritize efficiency versus coverage.

A full factorial design from DOE is a test suite including all combinations of
factors and levels, providing exhaustive testing and the ability to conduct an
analysis of variance and characterise the effect of factor levels on system
performance. In most complex systems with many factors with multiple levels,
exhaustive testing is infeasible. Fractional factorial designs select a
fraction of the factorial design with the result that some effects are aliased
with others and variance cannot be fully attributed. The choice of fraction
can lead to aliasing between main effects and higher order interaction effects
that are not expected to be significant and thus provide sufficient system
knowledge with fewer test points.

Optimal learning is a technique that also has the goal of reducing uncertainty
and determining the best factor levels for improving system performance. By
choosing tests adaptively via the exploration-exploitation heuristic policy,
optimal learning can discover the ideal factor settings with fewer tests.
However, it relies on other knowledge such as physical laws and prior
experience to ensure that the results from fewer tests can be used to make
claims about performance in the rest of the operating space.

An often employed technique in CIT is to design a test suite from a covering
array, a combinatorial array where the columns represent factors, the rows
represent tests, and the values in each cell represent the level set for that
factor in that test. Every combination of values for up to a given strength of
interacting features appears in some test in the array; thus, a covering array
guarantees to detect failures due to interactions of up to the strength
specified. One row covers many interactions when the strength is smaller than
the total number of factors and so produces a test suite with a fraction of
the number of tests in the full factorial. The number of tests required grows
logarithmically in the number of factors. While using a covering array as a
test suite provides coverage of the test space in terms of interactions, it
does not guarantee that the cause of the fault detected can be identified.

Humans frequently interact with complex systems as part of the environment or
system. Between-subject and within-subject experimental design strategies
allow for the humans contribution to the test outcome to be characterized and
separated in order to reduce aliasing with factors and levels in the
environment [12]. Between subject designs randomly assign humans to unique
combinations of factors and levels, while within-subject designs provide a
basis for comparison by matching individuals to combinations of factors and
levels.

Each of the above strategies has unique strengths that are leveraged by our
framework to systematically cover the input space at both local and global
scale, integrate information across test slices, and determine the right test
size at various stages of development and deployment. Specifically, finding
flaws early in development can prevent extensive testing later that must step
down to components lower in the hierarchy. Additionally, at the lower
hierarchical scale, such as within a single function of code of a sensor, the
test input space is smaller. Tests may be faster and more automatable, such as
by running a script or using simulated inputs, and test oracles may be
computable. As subsystems are composed, testing emphasis shifts to identifying
faulty interactions among components. At the global scale, the fully composed
system is tested in terms of achieving mission objective, and it may be
necessary to focus on only the most critical factor levels.

These facts support conducting exhaustive testing at the component scale but
sparser testing at the global scale. DOE full factorial or fractional
factorial designs are likely best used for components lower in the hierarchy.
CIT techniques can prevent exponential growth in test points as interactions
among components are considered during integration. At the global scale,
correct system behavior may not even be definable, particularly after
deployment in changing conditions and after acquisition of knowledge by
embedded AI components. In this case, optimal learning may be employed to
estimate system boundaries and adaptively select tests as needed and as
resources allow.

As shown in Fig. 3, the framework makes a trade-off between the number of test
points used and the fidelity of measuring the system. Moving up the hierarchy
towards a completely realized system, testing achieves better fidelity, but
becomes more difficult, and fewer test points are possible.

Figure 3: Number of test points exchanged for increased fidelity as testing
moves up the system composition hierarchy.

##  VII Test Design Framework

The framework does not specify a series of tests to run. Instead, the
framework helps inform comprehensive test plan design by outlining the
considerations to address. These considerations will change at each slice in
the VTP model leading to different test designs. Additionally, the “P” phase
of the VTP model requires that information learned during operational test
slices is incorporated into future tests plans. For example, a system could
employ a combinatorial interaction testing strategy with automated tests of
each agent in its fielded state in an attempt to monitor if any had been
compromised by an adversary. The outcomes of these tests will be incorporated
into future algorithm updates and result in a new round of independent tests
to verify the successful implementation of updates to the agents.

The process by which the framework guides considerations to result in a test
plan is graphically represented in Fig. 4. It begins with identification of
the current phase of the life cycle for the system under test (SUT) and guides
identification of the field of study and hierarchy of test. Hierarchy of test
informs how to identify the components inside the SUT and which component
levels should be tested. Identification of components, levels, and their
interactions defines the test input space. Test plan efficiency guides the
focus of the test given the cost of each test run at the current scale and
assumes systems lower in the hierarchy function as expected. The framework
process also includes identification of goals of the test and reasonable test
methods. The field of study for the SUT and hierarchy of test inform the goal
of the test. Goals are combined with test plan efficiency to determine
appropriate test methods. Finally, the collected knowledge through all prior
steps guides creation of the test plan.

Figure 4: Process by which framework guides considerations towards a
comprehensive test plan

In Fig. 5, three example scenarios within the multi-agent system of AIAs life
cycle are provided. A non-exhaustive list of fields of study involved in the
SUT and from which testing strategies should be drawn is given.

Figure 5: High-level overview of how each framework concept contributes to
guiding test plan development

Recall our use case, a satellite network tasked with conducting broad area
search and point detection. Fig. 1 depicts the satellite network designed to
observe shipping traffic. The network conducts broad area search to understand
normal traffic patterns and detect activities that could indicate potential
illegal shipping activities. Once an anomaly is detected, the network has
multiple objectives: continue broad area search and maintain a track on the
anomalous vessel. Using this context, we can walk through Fig. 5 and propose
reasonable test design structures.

The first row identifies an early phase of testing focused on the AI
algorithm. This algorithm could be black-box or fully specified. In our use
case, consider the central controller, whose objective is to task the
satellites in the network to both provide wide area coverage and maintain
tracks. Our test goal is to assess the reliability, robustness, performance,
and biases of the central controller. Because testing will be conducted via
simulations, tests are relatively affordable, and we can afford a strategy
that allows both comprehensive coverage via CIT and investigation of any areas
of high uncertainty via optimal learning (tests can be sequentially added at
low cost). The CIT design could leverage historical shipping data crossed with
CIT at a high strength (covering many interactions) to embed anomalous traffic
into the historical data. The factors in the CIT test set may include vessel
country of origin, vessel type, vessel size, and geospatial location. Optimal
learning is used to augment the CIT in areas of high uncertainty or that show
large performance changes in the AI algorithm.

The second row of Fig. 5 shows how the considerations change when moving up to
the satellite scale of testing. Testing must now consider not only algorithm
performance, but also how that algorithm integrates with the system’s
additional software, electronic hardware, and mechanical systems. This
necessitates identification of new factors and test design strategy. The
knowledge gained in testing the AI via CIT/Optimal learning is leveraged to
identify a subset of scenarios for input into the system-centric test design.
Here we may use a hardware-in-the-loop test facility where satellite sensors
are given simulated inputs, but all additional aspects are real (e.g.,
simulating the space deployment). Factors include AI performing scenario
(potentially binned into low/medium/high based on the outcomes of the previous
AI testing), active adversary (yes/no), and weather impact on inputs (e.g.,
clouds, rain), etc. Experimental designs are used that focus on understanding
how system performance changes as a function of the simulated inputs combined
with various executions of the AI algorithm.

Finally, once the system is deployed, we may need to understand mission
accomplishment of the fully connected system. In Fig. 5, we focus on the human
integration at a ground control station as important to mission
accomplishment. Here our test size is limited by the number of humans that are
qualified mission controllers for the constellation. We use a within-subject
design to assign different ground teams to various scenarios on the deployed
system. The tasks are controlled by focusing the constellation on certain
parts of the ocean for an operating period.

The three scenarios provide a hypothetical example of how to use the elements
in the test design framework, coupled with the slice in the VTP model to
develop a test design strategy that pairs the goal of the test, derived from
the field of study and the hierarchy of test with desired test efficiencies,
for a complex system-of-systems at multiple scales.

##  VIII Conclusions and Future directions

The framework we propose in this work specifies the three concepts of field of
study, hierarchy of test, and test plan efficiency to be considered in each of
the design, development, and deployment phases of the VTP model in order to
guide the creation of a comprehensive test plan.

This unifying framework synthesizes T&E methodology and is generalizable to
many contexts involving multi-agent systems of AIAs. We have provided examples
throughout of how the framework would be applied to the use case of a
constellation of satellites conducting both broad area search and point
detection in a series of tests. Evaluation of the framework against additional
use cases is needed to assess its usefulness for this category of complex
systems and to identify any features needed to assist in guiding the creation
of test plans.

Context-specific challenges in this work led to the formulation of two new
research questions. Testing the integration of a system-of-systems composed of
subsystems of the same type may lead to tests that appear “symmetrical” such
that multiple tests effectively represent the same configuration. For example,
consider testing the integration of a constellation of satellites having two
satellites, $S_{1}$ and $S_{2}$, with identical settings for all components,
but appearing in different locations in orbit, say $p_{x}$ and $p_{y}$. That
is, they have the same payload, algorithms, control software, and actuators.
If the satellites are listed as factors in a CIT test suite, without using
constraints, tests should be generated including both combinations
$\\{(S_{1},p_{x}),(S_{2},p_{y})\\}$, $\\{(S_{1},p_{y}),(S_{2},p_{x})\\}$ in
larger strength interactions, though at early phases of testing before
learning or damage has occurred, both satellites are interchangeable and thus
both observations are not necessary. When testing is expensive, it may be
desired to observe and analyze only the test cases needed. Thus, a test suite
generation tool that avoids producing these kinds of symmetrical tests is
needed. Using constraints to solve this challenge and generate a test suite
can be computationally expensive in the presence of a large number of
constraints such as for a large constellation. Taking inspiration from
sequence covering arrays, we hypothesize that a partial ordering on specified
factors could be used to build covering arrays without the need to remove
symmetrical tests.

By evaluating the framework in the context of the satellite use case, we
propose to examine how much of the variance in success and failure is captured
by this framework and whether mission success at the top of the hierarchy can
be predicted by the results of testing at the low, task-scale of the
hierarchy. As tests at the global scale may be costlier and, in some cases,
only possible after the constellation has been launched into space, preventing
most changes to the system, early warnings predicting mission-scale success or
failure provide immense value.

## References

  * [1] _Defense Acquisition Guidebook_ , Defense Acquisition University, Fort Belvoir, VA, 2020.
  * [2] E. A. Elsayed, “Overview of reliability testing,” _IEEE Transactions on Reliability_ , vol. 61, no. 2, pp. 282–291, 2012.
  * [3] “Automated combinatorial testing for software,” https://csrc.nist.gov/projects/automated-combinatorial-testing-for-software.
  * [4] C. A. Lowry and D. C. Montgomery, “A review of multivariate control charts,” _IIE transactions_ , vol. 27, no. 6, pp. 800–810, 1995.
  * [5] D. Buede, _The engineering design of systems : models and methods_. Hoboken, N.J: John Wiley & Sons, 2009.
  * [6] NASA, “NASA systems engineering handbook revision 2,” 2017. [Online]. Available: https://www.nasa.gov/connect/ebooks/nasa-systems-engineering-handbook
  * [7] R. Lanotte, M. Merro, A. Munteanu, and L. Viganò, “A formal approach to physics-based attacks in cyber-physical systems (extended version),” _arXiv preprint arXiv:1902.04572_ , 2019.
  * [8] S. Guo and D. Zeng, _Cyber-Physical Systems: Architecture, Security and Application_. Springer, 2019.
  * [9] L. J. Freeman and C. Warner, “Informing the warfighter—why statistical methods matter in defense testing,” _Chance_ , vol. 31, no. 2, pp. 4–11, 2018.
  * [10] W. B. Powell and I. O. Ryzhov, _Optimal Learning_. Wiley, 2012.
  * [11] D. R. Kuhn, R. Bryce, F. Duan, L. S. Ghandehari, Y. Lei, and R. N. Kacker, “Chapter one - combinatorial testing: Theory and practice,” ser. Advances in Computers, A. Memon, Ed. Elsevier, 2015, vol. 99, pp. 1 – 66.
  * [12] G. Charness, U. Gneezy, and M. A. Kuhn, “Experimental methods: Between-subject and within-subject design,” _Journal of Economic Behavior & Organization_, vol. 81, no. 1, pp. 1–8, 2012.


# Machine Learning-based Methods for Reconfigurable Antenna Mode Selection in
MIMO Systems

Yasaman Abdollahian1, Ehsan Tohidi1, Martin Kasparick1,
Li Wang2, Ahmet Hasim Gokceoglu2, and Slawomir Stanczak1  1Fraunhofer Heinrich
Hertz Institute, Berlin, Germany,
2Huawei Technologies, Stockholm, Sweden
Email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

MIMO technology has enabled spatial multiple access and has provided a higher
system spectral efficiency (SE). However, this technology has some drawbacks,
such as the high number of RF chains that increases complexity in the system.
One of the solutions to this problem can be to employ reconfigurable antennas
(RAs) that can support different radiation patterns during transmission to
provide similar performance with fewer RF chains. In this regard, the system
aims to maximize the SE with respect to optimum beamforming design and RA mode
selection. Due to the non-convexity of this problem, we propose machine
learning-based methods for RA antenna mode selection in both dynamic and
static scenarios. In the static scenario, we present how to solve the RA mode
selection problem, an integer optimization problem in nature, via deep
convolutional neural networks (DCNN). A Multi-Armed-bandit (MAB) consisting of
offline and online training is employed for the dynamic RA state selection.
For the proposed MAB, the computational complexity of the optimization problem
is reduced. Finally, the proposed methods in both dynamic and static scenarios
are compared with exhaustive search and random selection methods.

###### Index Terms:

Reconfigurable antennas, MIMO, Neural Network, Multi-Armed-Bandit.

##  I Introduction

In wireless communication, one way to increase the SE is by employing MIMO
technology. Needless to say, increasing the number of antennas in MIMO raises
new challenges, such as the higher complexity of hardware, e.g., a higher
number of RF chains, and software [1, 2, 3]. RA has been known as a feasible
solution to reduce the number of RF chains. RA can be implemented with
different procedures, such as modifications in radiation patterns, antenna
polarizations, frequency, or combinations of them [4], [5]. This
reconfigurability can be done with the asset of RF switches, which leads to
changes in impedance and antenna radiation properties [6]. Here, we
concentrate on RAs that are equipped with different radiation patterns. When
employing RA in wireless systems, algorithms for RA state selection that
maximize the channel capacity are required. However, finding an optimal RA
state is an NP-hard problem. Therefore, several studies concentrate on the RA
state selection problem [7, 8].

###  I-A Related Works

Recently, machine learning-based methods in wireless communications are
accumulating more interests in different areas, such as MIMO channel
denoising, beam selection, cognitive radio, etc. Moreover, they can be
employed to tackle complex and non-convex problems with lower complexity.
Also, they can be implemented on graphics processing units (GPUs) to reach the
solution in a more energy-efficient manner [9].

This work [10] presents a joint antenna selection and beamforming design in
which a semi-definite optimization algorithm for this non-convex problem is
solved through supervised training of a neural network model. The authors in
[11] present a two-step algorithm for antenna selection and beamforming design
that are implemented as classification and prediction problems. The antenna
selection task is carried out in the first step, and the beamforming matrix is
designed in the second step. The proposed ML-based method for both tasks is
convolutional neural networks (CNN). The authors in [12] propose a deep
learning-based method for designing the analog beamforming matrix for mm-wave
multiple input single output (MISO). The proposed network training is
unsupervised in which, during the training, the SE of the system is calculated
via an analog beamformer matrix that is the output of the network. A
comparison between conventional supervised and unsupervised deep learning
methods for beamforming design is given in [9]. In the supervised case, the
loss function is the mean squared error (MSE) between the predicted labels and
ground truth ones. For the unsupervised case, the SE is defined as the loss
function, combined with an estimated covariance matrix for each user. In [13],
deep power control for a wireless communication system is proposed based on
CNNs. In this work, both supervised and unsupervised learning approaches are
used to train the neural network to achieve a higher spectral or energy
efficiency with a reduced training time. The work in [14] uses an unsupervised
learning residual neural network (ResNet) method for jointly designing the
hybrid beamforming and antenna selection in the downlink MIMO. The training of
the system consists of three main parts; in the first part, by considering the
fixed antenna selection matrix, the beamforming matrix is predicted. In the
second part, the extracted beamforming matrix is used to train the model for
the antenna selection task, and in the last part, both systems are trained
jointly. The mutual goal of the whole system is maximizing the mutual
information that is calculated through network training. As for the dynamic RA
mode selection in [15], a reinforcement learning (RL) based method, namely
Multi-Armed-bandit (MAB), is proposed. The proposed MAB policy is upper
confidence bound (UCB), in which the RA states are selected from the five
possible states. As [16] shows, the UCB policy does not achieve satisfactory
performance in the case of having a larger number of states in comparison to
the Thompson Sampling (TS). As a result, by increasing the number of RA
states, the problem becomes more complicated, and for that, in [17], a
hierarchical TS (HTS) policy is proposed to select the best RA states with the
help of clustering methods. In this model, the RA states are divided into a
number of clusters, and in each cluster, the HTS is executed to extract the
optimal RA states. This algorithm convergence faster in comparison to the MAB-
TS and UCB with less computational complexity. The paper [18] employs MAB for
RA state selection plus channel prediction for unexplored states in the case
of using both policies of UCB and TS.

###  I-B Contributions and Novelties

We study the performance of a multi-user MIMO downlink system via a joint
design of RA mode selection and digital beamforming to enhance the system
performance in both static and dynamic MIMO scenarios. Our contributions are
as follows: In the static scenario, $(i)$ we design a deep convolutional
neural network (DCNN) with an unsupervised learning approach for RA mode
selection. For this model, we propose a customized activation function for the
integer optimization problem of selected RA modes, plus a customized loss
function for SE calculation, and $(ii)$ we design a DCNN as the combination of
transfer learning and a semi-supervised learning approach for RA mode
selection. This model reduces the network training time and boosts the
system’s performance. In the dynamic scenario, $(iii)$ we apply MAB with
different policies. For that, we include modifications to TS and UCB policies
to adapt them to our problem, and $(iv)$ we apply offline training for MAB
models to reduce the complexity of RA state selection. This means that both
feature reduction and clustering methods are employed to find the
representative RA states for the online training.

###  I-C Outline

The rest of the paper is organized as follows. Section 2 defines the proposed
system model for the joint design of RA mode selection and beamforming
matrices, followed by the problem formulation. The static and dynamic RA mode
selections are described in Section 3 and 4, respectively. Simulation results
are presented in Section 5. Finally, conclusions are given in Section 6.

##  II System Model and Problem Formulation

We consider the downlink of a single-cell MIMO system. The BS is equipped with
a uniform planar array antenna (UPA) with RA technology that has $N_{T}$
elements with $N_{P}$ different radiation patterns (modes). The BS is equipped
with $N_{RF}$, RF chains to transmit $N_{S}$ data streams. The RX side is
equipped with $N_{R}$ antennas for each user. In this paper, we consider both
single (SU) and multi-user (MU) scenarios. At the BS, the transmitted symbols
are processed by a digital beamformer $\bm{F}_{BB}\in\mathbb{C}^{N_{RF}\times
N_{S}}$, and then pass through $\bm{F}_{RF}\in\mathbb{C}^{N_{T}\times
N_{RF}}$. This RF precoder determines the connection of $N_{T}$ antennas to
the $N_{RF}$ RF chains, which are considered in this paper without any phase-
shifting operations. The transmit signal $\bm{x}\in\mathbb{C}^{N_{T}\times 1}$
can be defined with respect to the power-constrained beamformers as:

$\bm{x}=\bm{F}_{RF}\bm{F}_{BB}\bm{s},$ (1)

where $\bm{s}\in\mathbb{C}^{N_{s}\times 1}$ is the vector of transmitted
symbols. The spectral efficiency of the system for each user $k=1,2,\cdots,K$
at each OFDM subcarrier $f\in\\{1,\cdots,N_{f}\\}$, and each time step $t$ can
be defined as:

$\displaystyle R(\bm{F}_{RF},$
$\displaystyle\bm{F}_{BB},\bm{\nu})=\sum_{k=1}^{K}\log_{2}(|\bm{I}+\frac{\rho}{N_{s}}\bm{C}^{-1}\bm{H}_{k}[f,t,\bm{\nu}]\bm{F}_{RF}[t]$
$\displaystyle\bm{F}_{BB,k}[f,t]\bm{F}_{BB,k}^{H}[f,t]\bm{F}_{RF}^{H}[t]\bm{H}_{k}^{H}[f,t,\bm{\nu}]|),$
(2)

where
$\bm{C}={\mathbf{H}}_{k}[f,t]{\bm{F}}_{\text{RF}}[t]\sum_{\bar{k}=1,\bar{k}\neq
k}^{K}{\bm{F}}_{\text{BB},\bar{k}}[f,t]{\bm{F}}_{\text{BB},\bar{k}}^{H}[f,t]\\\
{\bm{F}}_{\text{RF}}^{H}[t]{\mathbf{H}}_{k}^{H}[f,t]+\sigma_{n}^{2}\bm{I}$ is
the covariance matrix of both interference and noise. We denote each RA state
by $\bm{\nu}=[\nu_{1},\cdots,\nu_{T}]\in\mathcal{M}^{N_{T}\times 1}$ which
contains the modes of $N_{T}$ antennas. The general formulation of SE
including all subcarriers and time steps is presented in (2). However, for the
sake of simplifying the illustration, we focus on one subcarrier with one and
multiple time steps for static and dynamic scenarios, respectively. The main
aim of the system is to maximize the SE through an optimal design of the
beamforming matrix and RA mode selection.

##  III Static RA Mode Selection

The RA state selection in static scenarios can be defined via a binary
decision-making matrix $\bm{W}\in\\{0,1\\}^{N_{T}\times N_{P}}$, meaning that
if $W_{m,\nu}=1$, the $m$th antenna has selected mode $\nu$ and $0$ otherwise.
Therefore, the channel matrix after RA mode selection can be defined as
follows:

$\displaystyle\bm{H}[\bm{W}]$
$\displaystyle=\sum_{\nu=1}^{N_{P}}\bm{H}_{s}[\nu]\mbox{\boldmath\bf
Diag}\,(\bm{W}_{\nu})$ (3) s.t.
$\displaystyle\sum_{\nu=1}^{N_{P}}W_{m,\nu}=1,~{}\forall m$
$\displaystyle\bm{W}\in\\{0,1\\}^{N_{T}\times N_{P}}$

where $\mbox{\boldmath\bf Diag}\,(\cdot)$ defines the diagonal matrix
operator. $\bm{W}_{\nu}$ denotes the $\nu$th column of $\bm{W}$, and
$\bm{H}_{s}[\nu]$ defines the channel when all transmit antennas select the
specific mode $\nu$. We provide a model in which deep learning can be adopted
in MIMO for accomplishing the joint problem of RA mode selection and digital
beamforming design. The proposed DCNN model is depicted in Fig.1. With respect
to the convolutional operators in the proposed DCNN model, the spatial
features of the channel are extracted to capture the beneficial properties of
digital precoding and RA mode selection matrices. These convolutional
operators are mainly applicable to image-processing tasks. As a result, we
transfer the channel coefficient matrix in image shape to apply convolutional
layers on it. Here, the input layer contains the effective channel matrix in
which its real and imaginary parts are separated and stacked as a 2D image. To
extract the RA mode selection matrix in the last layer, we propose a
customized softmax activation function in order to choose the optimum RA modes
for each RF chain. In other words, the proposed customized activation function
(4) tries to find the suitable RA mode selection matrix as defined in (3).
This means that it shifts the output to one-hot encoder variables by
increasing $\alpha$ through each epoch’s training.

$f(x)=\frac{\exp{(\alpha x)}}{\sum_{i=1}^{n}\exp{(\alpha x)}}$ (4)

Figure 1: DCNN architecture in the proposed scheme

###  III-A Learning Approach

We consider two learning approaches for the proposed DCNN, namely unsupervised
and semi-supervised learning. It should be noted that applying unsupervised
learning for DCNN in multi-user scenarios requires more training time and a
deeper network. Therefore, to reduce the training time, we attempt to apply
semi-supervised learning in combination with transfer learning in the second
approach.

####  III-A1 Unsupervised Learning

Regarding this learning approach, we customize the loss function to solve (2).
For the multi-user scenario, we design the digital beamforming matrix via
[19], which eliminates the users’ interference by finding the null space of
the right singular vector of the users’ channels. Following the beamforming
design, the SE of the system is calculated according to (2). During each
epoch’s training, the DCNN attempts to minimize the negative of SE with
respect to the predefined optimizer method.

####  III-A2 Semi-Supervised Learning

We adopt the idea of combining semi-supervised and transfer learning in [13].
We propose to use a transfer learning method so that, in the first step, we
train the DCNN network in a supervised manner and then use the trained DCNN
model for the unsupervised task. Thereby, less time is needed for training the
main task, which is the RA mode selection with unsupervised learning. For the
supervised DCNN, the RA modes are extracted via the exhaustive search method
and transferred to DCNN as ground truth labels. During learning, the binary
cross entropy loss [20] between ground truth and predicted labels are
minimized. After initialization of the weights, biases, and training, the
network is used for unsupervised learning. We use the same architecture for
semi-supervised and unsupervised DCNN, where only the learning procedure is
changed.

##  IV Dynamic RA Mode Selection

In this section, we assume a dynamic scenario with a mobile UE. A moving UE
results in a time-varying channel. As a result, the RA state selection is more
challenging compared to the static scenario. Dynamic RA mode selection can be
accounted as an online mode selection, meaning that in each time step, the RA
mode has to be updated. It should be noted that the problem of RA mode
selection and RA state selection are equivalent.

We aim to develop a low-complex algorithm consisting of offline and online
training. The offline training comprises the two steps of feature reduction
and clustering methods. During offline training, we extract the important
features of the effective channel via the PCA method [21]. In the next step,
we apply the K-means algorithm as a clustering method [21] to group similar RA
states. In other words, by choosing an appropriate number of clusters, the
within cluster’s sum of squares distances (WCSS) will be negligible. Following
that, the representative clusters’ points (the points that have lower WCSS)
are extracted to be used in the next step, which is online training. Thus, the
representative points obtained in the offline training procedure could be
considered as the states of the dynamic scenario, in which we employ MAB. For
MAB, similar to other RL methods, we need to assign a policy and derive the
rewards and punishments of states from that. Here, we modify two policies of
UCB [22] and TS [23]. The main responsibility of MAB is to provide a balance
between exploration and exploitation (EE) for arms selection. For our problem,
the RA states are considered as the MAB arms, and each arm has a reward that
is the SE of the system. The aim is to select the arm that leads the system to
a higher reward.

###  IV-A MAB Policies

####  IV-A1 UCB

This policy requires the average of all instantaneous reward values up to the
present time step named as $\bar{R_{i}}(t)$. Moreover, UCB needs the number of
times up to the current time step that each state has been selected
$n_{i}(t)$. These two values are updated regarding each arm and time step as
follows:

$\bar{R_{i}}(t)=\begin{cases}\frac{\bar{R_{i}}(t-1)n_{i}(t-1)+R_{i}(t)}{n_{i}(t-1)+1}&\text{if
state $i$ is selected }\\\ \bar{R_{i}}(t-1)&\text{otherwise}\end{cases}$ (5)

and

$n_{i}(t)=\begin{cases}n_{i}(t-1)+1&\text{if state $i$ is selected }\\\
n_{i}(t-1)&\text{otherwise}\end{cases}$ (6)

Here, we customize the UCB algorithm to provide a balance between EE for our
problem. This balancing is defined via the $\gamma$ factor, which is an
arbitrary positive real number, applied to the exploitation term of EE.

Algorithm 1 UCB Policy [15]

1:$n_{i}=0$, $\bar{R_{i}}=0$;

2:Select each state at least once and update $n_{i},\bar{R_{i}}$

3:for $t=1$ to $N_{\text{{time step}}}$ do

4: Select a RA state that maximizes
$\gamma\bar{R_{i}}+\sqrt{\frac{2\ln{(t})}{n_{i}}}$

5: Update $\bar{R_{i}}$, $n_{i}$ for state i, w.r.t to (5), (6)

6:end for

####  IV-A2 TS

This policy assumes that the mean reward of each arm has a particular
distribution; typically, the Beta distribution is considered. At each time
step, an arm from the Beta distribution is selected, and if the reward
regarding that arm is larger than the Bernoulli reward, the success
probability of that state is updated; otherwise, the failure probability is
increased.

Regarding the TS policy, we propose four terms for EE tradeoff balancing.
These terms are $\lambda$, $\omega$, the scalar values from real positive
numbers for Beta distribution parameters, and $\Delta S$, $\Delta F$ for
success and failure updates of the Beta distribution. The $S_{i}$ and $F_{i}$
parameters represent the number of successes and failures of each state.

Algorithm 2 Thompson Sampling Policy [18]

1:$S_{i}=0$, $F_{i}=0$;

2:for $t=1$ to $N_{\text{{time step}}}$ do

3: For each arm $i=1,\ldots,N$ sample $\eta_{i}(t)$ from
Beta$(S_{i}+\lambda,F_{i}+\omega)$ distribution.

4: Play arm $i(t):=\arg\max\eta_{i}(t)$ and observe reward $\tilde{r_{t}}$

5: Perform Bernouli trail with probability $\tilde{r_{t}}\in[0,1]$ and observe
output $r_{t}\in\\{1,0\\}$

6: if $r_{t}=1$ then

7: $S_{i}=S_{i}+\Delta S$

8: else

9: $F_{i}=F_{i}+\Delta F$

10: end if

11:end for

##  V Simulation Results

In this section, we evaluate the performance of the proposed methods, that is,
DCNN for RA mode selection in static scenarios, compared with an exhaustive
search method, and MAB with mentioned policies for dynamic scenarios.

###  V-A Static Scenario

Regarding the DCNN implementation, we use TensorFlow Keras in Python. We
consider QuaDRiGa channels [24] for our model.For the static scenario, we
generate $20000$ samples in which UEs are positioned uniformly distributed in
$50-100m$. The SE is evaluated as a performance metric. For the single-user
case, the proposed DCNN model consists of six Conv2D layers, each three of
which have a filter size of $64$ and $128$ with a kernel size of $2\times 2$,
respectively. Then, after the flattening layer, two pairs of fully connected
(FC) and dropout layers with the size of $512$ and $0.1$ are used. For all
layers, we use the ReLU activation function [25], and for the output layer, we
apply the customized activation (4). The number of epochs and batch size are
$200$ and $1024$, respectively.

Regarding the DCNN architecture for the two-user scenario, we consider nine
Conv2D layers, each three of which have a filter size of $32$, $64$, and
$128$, respectively, with a kernel size of $2\times 2$. We use the tanh as a
non-linear activation function for all layers besides the output layer that is
equipped with the customized function. Next, the extracted features matrix is
flattened and followed by a pair of fully FC layers with $128$ units. Finally,
a dropout layer with the rate of $0.1$ for decision-making and training
regularization is considered. The number of epochs and batch sizes are $50$
and $64$, respectively.

Fig. 2 depicts the performance of both single-user and multi-user (two users)
for LOS RA mode selection. The comparison shows DCNN training and validation
performances regarding the network training part. The predicted loss via DCNN
is denoted as DCNN_test_continuous, and the SE of the system is calculated via
the predicted labels, which is plotted as DCNN_test_discrete. We observe that
the predicted results almost reach the exhaustive search method.

Figure 2: Training and testing DCNN unsupervised loss versus epochs and
compared with exhaustive search method (a) LOS_SU, (b) LOS_MU

As shown in Fig.3, the proposed DCNN semi-supervised learning for a two-user
scenario can almost reach the optimum results, similar to the unsupervised
one. However, here, we use half the sample size and fewer training epochs
compared to the unsupervised DCNN. The proposed system model is similar to the
unsupervised one, except that the number of epochs and batch sizes of the
supervised step are $20$ and $128$, and for the unsupervised step are $20$ and
$64$, respectively. Regarding the supervised DCNN, the stochastic gradient
descent (SGD) optimizer with a $0.01$ learning rate and for the unsupervised
DCNNs, the Adam optimizer with a learning rate of $0.0001$ is considered.

Figure 3: Training and testing DCNN semi-supervised loss versus epoch (a)
supervised learning, (b) unsupervised learning for RA_MU_LOS

###  V-B Dynamic Scenario

In the dynamic scenario, during offline training, we extract the PCA
components ($100$ from $1600$ features) from the dataset. For finding a
suitable number of clusters, we consider two metrics of WCSS and Silhouette
[26]. As can be seen from Fig. 4, $100$ clusters could separate RA states in a
proper way.

Figure 4: (a) Within sum of square distances (WCSS), (b) Silhouette metric for
cluster number selection.

Then, regarding the online training, two trajectories are considered for the
UE. Fig. 5 depicts the result for a linear trajectory that comprises two
segments with the UE moving at the speed of $30km/h$. For the MAB-UCB policy,
we set $\gamma=5$. For the MAB-TS, the parameters are set as $\lambda=1$,
$\omega=1$, $\Delta S=10$, and $\Delta F=50$. To assess dynamic RA state
selection via MAB algorithms, we compare the result with random selection as a
baseline in which, at each time step, RA mode is chosen from a uniform random
distribution. As the upper bound, we consider both the exhaustive search
method (max_all_modes) and the best state from the clustered representative
data points (max_selected_modes), which maximizes the SE without considering
the EE term. Fig. 5 illustrates the cumulative reward (cumulative
$\bar{R_{i}}$) comparison of this trajectory where the best of the selected
states reaches the best of all states. In other words, $100$ clusters are
sufficient to represent the states in this trajectory. In addition, MAB-UCB
and MAB-TS are outperforming the random selection method, with the MAB-UCB
performing better than the MAB-TS by increasing the time step.

Figure 5:  (a) User trajectory, (b) Cumulative reward (SE) comparison for MAB
methods, random selection, and exhaustive search.

In Fig. 6, a circular trajectory with a radius of $10m$ and a UE with a
velocity of $30km/h$ is considered. As can be seen from the cumulative reward
plot, the MAB-UCB in circular trajectory is close to the upper bound and
outperforms the MAB-TS.

Figure 6: (a) User trajectory, (b) Cumulative reward (SE) comparison for MAB
methods, random selection, and exhaustive search.

Moreover, Fig. 7 shows the effect of number of clusters on the MAB
performance.

Figure 7: Cumulative reward (SE) versus the number of clusters

##  VI Conclusion

In this paper, we propose ML-based algorithms to solve the RA mode selection
in both static and dynamic scenarios. In the static scenario, DCNNs for RA
mode selection and digital beamforming design with both unsupervised and semi-
supervised learning procedures are proposed in order to reduce the complexity
of addressing the non-convex optimization problem. To facilitate these
implementations, we customize the loss function and activation function of the
network. In the case of RA mode selection in the dynamic scenario, employing
offline training to extract the important features and clusters results in
complexity reduction as well as improved performance. In addition, MAB
policies are modified to fit the proposed approach.

## References

  * [1] T. L. Marzetta and H. Yang, _Fundamentals of massive MIMO_. Cambridge University Press, 2016.
  * [2] E. Tohidi, A. Hariri, H. Behroozi, M. M. Nayebi, G. Leus, and A. P. Petropulu, “Compressed-domain detection and estimation for colocated mimo radar,” _IEEE Transactions on Aerospace and Electronic Systems_ , vol. 56, no. 6, pp. 4504–4518, 2020.
  * [3] E. Tohidi, M. Coutino, S. P. Chepuri, H. Behroozi, M. M. Nayebi, and G. Leus, “Sparse antenna and pulse placement for colocated mimo radar,” _IEEE Transactions on Signal Processing_ , vol. 67, no. 3, pp. 579–593, 2019.
  * [4] J. Costantine, Y. Tawk, S. E. Barbin, and C. G. Christodoulou, “Reconfigurable antennas: Design and applications,” _Proceedings of the IEEE_ , vol. 103, no. 3, pp. 424–437, 2015.
  * [5] I. Bahceci, M. Hasan, T. M. Duman, and B. A. Cetiner, “Efficient channel estimation for reconfigurable mimo antennas: Training techniques and performance analysis,” _IEEE Transactions on Wireless Communications_ , vol. 16, no. 1, pp. 565–580, 2016.
  * [6] R. L. Haupt and M. Lanagan, “Reconfigurable antennas,” _IEEE Antennas and Propagation Magazine_ , vol. 55, no. 1, pp. 49–61, 2013.
  * [7] H. Li, Q. Liu, Z. Wang, and M. Li, “Joint antenna selection and analog precoder design with low-resolution phase shifters,” _IEEE Transactions on Vehicular Technology_ , vol. 68, no. 1, pp. 967–971, 2018.
  * [8] X. Zhai, Y. Cai, Q. Shi, M. Zhao, G. Y. Li, and B. Champagne, “Joint transceiver design with antenna selection for large-scale mu-mimo mmwave systems,” _IEEE Journal on Selected Areas in Communications_ , vol. 35, no. 9, pp. 2085–2096, 2017.
  * [9] H. Huang, Y. Song, J. Yang, G. Gui, and F. Adachi, “Deep-learning-based millimeter-wave massive mimo for hybrid precoding,” _IEEE Transactions on Vehicular Technology_ , vol. 68, no. 3, pp. 3027–3032, 2019.
  * [10] T. X. Vu, S. Chatzinotas, V.-D. Nguyen, D. T. Hoang, D. N. Nguyen, M. Di Renzo, and B. Ottersten, “Machine learning-enabled joint antenna selection and precoding design: From offline complexity to online performance,” _IEEE Transactions on Wireless Communications_ , vol. 20, no. 6, pp. 3710–3722, 2021\.
  * [11] A. M. Elbir and K. V. Mishra, “Joint antenna selection and hybrid beamformer design using unquantized and quantized deep learning networks,” _IEEE Transactions on Wireless Communications_ , vol. 19, no. 3, pp. 1677–1688, 2019\.
  * [12] T. Lin and Y. Zhu, “Beamforming design for large-scale antenna arrays using deep learning,” _IEEE Wireless Communications Letters_ , vol. 9, no. 1, pp. 103–107, 2019.
  * [13] W. Lee, M. Kim, and D.-H. Cho, “Deep power control: Transmit power control scheme based on convolutional neural network,” _IEEE Communications Letters_ , vol. 22, no. 6, pp. 1276–1279, 2018.
  * [14] Z. Liu, Y. Yang, F. Gao, T. Zhou, and H. Ma, “Deep unsupervised learning for joint antenna selection and hybrid beamforming,” _IEEE Transactions on Communications_ , vol. 70, no. 3, pp. 1697–1710, 2022.
  * [15] N. Gulati and K. R. Dandekar, “Learning state selection for reconfigurable antennas: A multi-armed bandit approach,” _IEEE Transactions on Antennas and Propagation_ , vol. 62, no. 3, pp. 1027–1038, 2013.
  * [16] T. Zhao, M. Li, and G. Ditzler, “Online reconfigurable antenna state selection based on thompson sampling,” in _2019 International Conference on Computing, Networking and Communications (ICNC)_. IEEE, 2019, pp. 888–893.
  * [17] T. Zhao, M. Li, and M. Poloczek, “Fast reconfigurable antenna state selection with hierarchical thompson sampling,” in _ICC 2019-2019 IEEE International Conference on Communications (ICC)_. IEEE, 2019, pp. 1–6.
  * [18] T. Zhao, M. Li, and Y. Pan, “Online learning-based reconfigurable antenna mode selection exploiting channel correlation,” _IEEE Transactions on Wireless Communications_ , vol. 20, no. 10, pp. 6820–6834, 2021.
  * [19] Q. H. Spencer, A. L. Swindlehurst, and M. Haardt, “Zero-forcing methods for downlink spatial multiplexing in multiuser mimo channels,” _IEEE transactions on signal processing_ , vol. 52, no. 2, pp. 461–471, 2004.
  * [20] U. Ruby and V. Yendapalli, “Binary cross entropy with deep learning technique for image classification,” _Int. J. Adv. Trends Comput. Sci. Eng_ , vol. 9, no. 10, 2020.
  * [21] C. M. Bishop and N. M. Nasrabadi, _Pattern recognition and machine learning_. Springer, 2006, vol. 4, no. 4.
  * [22] P. Auer, N. Cesa-Bianchi, and P. Fischer, “Finite-time analysis of the multiarmed bandit problem,” _Machine learning_ , vol. 47, no. 2, pp. 235–256, 2002.
  * [23] S. Agrawal and N. Goyal, “Analysis of thompson sampling for the multi-armed bandit problem,” in _Conference on learning theory_. JMLR Workshop and Conference Proceedings, 2012, pp. 39–1.
  * [24] S. Jaeckel, L. Raschkowski, K. Börner, and L. Thiele, “Quadriga: A 3-d multi-cell channel model with time evolution for enabling virtual field trials,” _IEEE transactions on antennas and propagation_ , vol. 62, no. 6, pp. 3242–3256, 2014.
  * [25] A. F. Agarap, “Deep learning using rectified linear units (relu),” _arXiv preprint arXiv:1803.08375_ , 2018.
  * [26] T. Thinsungnoena, N. Kaoungkub, P. Durongdumronchaib, K. Kerdprasopb, and N. Kerdprasopb, “The clustering validity with silhouette and sum of squared errors,” _learning_ , vol. 3, no. 7, 2015.