paper has 30 alternatives divided and completed the ranking after the $2nd$
decision-level. The presented ranking results have a similar trend to the
other two methods, while only $1st$ and $2nd$ decision-levels of S3W-GDM of
computation are used, greatly improving the efficiency of the decision-making
process. However, the difference is due to the fact that the two compared
methods use more comprehensive decision-making information, while the proposed
method only uses partial information. Therefore, all conditional attributes
were added using the same parameters to execute the algorithm. As can be
observed from the bottom subplot in Fig. 12 the decision on the alternative is
more rational after using the information from all the conditional attributes.
In particular, Method 1 shows a trend more similar to the method proposed in
this paper in terms of the ranking of some alternatives. For GDM problems with
a large number of alternatives, it is more rational and intuitive to perform
further decision-making by classifying the results of the alternatives.
Methods 1 and 2 lack a relevant process that brings semantic interpretation to
the decision results. The S3W-GDM method fills such a gap. The specific
classification of this data set will be shown in the sensitive analysis.

Type 3. Comparison of emergency logistics provider selection

This evaluation data set contains 5 emergency logistics providers in the food
and beverage industry
$U=\left\\{{{x_{1}},{x_{2}},{x_{3}},{x_{4}},{x_{5}}}\right\\}$ and 6
evaluation attributes: cost($a_{1}$), product level($a_{2}$), quick response
ability of supply($a_{3}$), quick response ability of transport($a_{4}$),
management($a_{5}$), and reputation($a_{6}$). The attribute weight results of
optimization model-based distance under DHHFLTS environment is used in the
comparison here to eliminate the impact might happen. The vector of weight is
$w=\left\\{{0.1011,0.1017,0.2591,0.1305,0.165,0.2426}\right\\}$.

Table 13 shows a comparison of the ranking results of the several methods
under this data set. Method 3 applied the normalized projection-based distance
and bidirectional projection to DHHFLTS. These improvements in the distance
measurements bring about better superiority and rationality in the ranking
results. This ranking result is consistent with the traditional Methods 4-6.
Method 7, as well as the methods proposed in this paper, are then consistent.
The difference lies mainly in the ranking of alternatives $x_{1}$, $x_{3}$,
$x_{5}$. However, there is a lack of a dynamic decision-making process and
quick initial judgment for the characteristics of emergency decision-making.
It is well known that emergency decision-making has a higher demand for
efficiency. The multi-level S3W-GDM provides a more conclusive semantic
interpretation after the completion of the $3rd$ decision-level ranking.
Although there are differences from most models, $x_{4}$ as the best supplier
is reflected in the positive domain of the partition where it is located. It
is worth noting that the method proposed in this paper uses only half of the
decision information at this decision-level. For emergency decision-making
scenarios, S3W-GDM provides a priori a solution as a decision-making result,
supporting the rapid development of the action.

Table 13: The ranking of logistics provider selection by different methods. Method | Ranking
---|---
Method 3 | ${x_{4}}>{x_{2}}>{x_{5}}>{x_{1}}>{x_{3}}$
Method 4 | ${x_{4}}>{x_{2}}>{x_{5}}>{x_{1}}>{x_{3}}$
Method 5 | ${x_{4}}>{x_{2}}>{x_{5}}>{x_{1}}>{x_{3}}$
Method 6 | ${x_{4}}>{x_{2}}>{x_{5}}>{x_{1}}>{x_{3}}$
Method 7 | ${x_{4}}>{x_{2}}>{x_{1}}>{x_{3}}>{x_{5}}$
S3W-GDM after the $3rd$ decision-level | ${x_{4}}>{x_{1}}>{x_{3}}>{x_{2}}>{x_{5}}$
S3W-GDM after the $4th$ decision-level | ${x_{4}}>{x_{2}}>{x_{1}}>{x_{3}}>{x_{5}}$

Type 4. Comparison of Sichuan liquor brand assessment

This evaluation data set contains 5 Sichuan liquor brands: Wuliangye($x_{1}$),
Luzhou Old Cellar($x_{2}$), Ichiro liquor ($x_{3}$), Tuopai liquor($x_{4}$)
and Jian Nan Chun($x_{5}$). The cognitions of consumers are used as a starting
point to investigate four attributes: product price($a_{1}$), product
classification($a_{2}$), consumer group($a_{3}$), and distribution
channel($a_{4}$). The attributes weights vector is
$w=\left\\{{0.1,0.3,0.2,0.4}\right\\}$.

From the Table 14, these methods can be used to examine the liquor brand data
set to improve the rationality of the decision results. In the analysis of
different methods’ rankings of alternatives, it is observed that Method 4-6
display identical ranking patterns, suggesting similarities in their
evaluation criteria. Method 7 and S3W-GDM provide an alternative ranking
result reveal that these methods may employ different decision-making logics
or prioritize differently. S3W-GDM presents a ranking almost similar to that
of Method 7, both after the computation of all attribute subsets has been
considered and only after the completion of the $3rd$ decision-level. The only
difference is the position of alternative $x_{4}$. The ranking results
obtained by the S3W-GDM method after the $4th$ decision-level differ from
Methods 4-6 in the order of alternatives $x_{1}$ and $x_{5}$. Methods 4-6 tend
to prefer Wuliangye($x_{1}$) as the top alternative, indicating its widespread
acceptance, while the varying rankings of Jian Nan Chun($x_{5}$) reflect
significant differences in evaluations across methods. The reason for the
differences in the methods proposed in this paper goes back to the setup of
this data set itself. This evaluation comes from consumers’ perceptions of
Sichuan liquor brands. Wuliangye($x_{1}$) is well known as a high-end brand.
However, Jian Nan Chun($x_{5}$), a mid-to-high end brand, is currently showing
a rapid growth trend, becoming the “meat and potatoes” of the liquor market.
On the one hand, its price and grade are more in line with the rational
consumption concepts of young people. On the other hand, the occupation
position in the market rises higher than the space of the low-end categories.
Evaluating the condition attributes of Sichuan liquor brand, the largest
weight is the distribution channel($a_{4}$), and Jian Nan Chun($x_{5}$) does
have better distribution channels, gradually becoming the best occupied brand
in the mid-to-high-end market. Based on the perspective of distribution
channels, the findings of the S3W-GDM method should be of better reference
value in providing adjustment strategies for Sichuan liquor enterprises.

Table 14: The ranking of liquor brand by different methods. Method | Ranking
---|---
Method 4 | ${x_{1}}>{x_{5}}>{x_{3}}>{x_{4}}>{x_{2}}$
Method 5 | ${x_{1}}>{x_{5}}>{x_{3}}>{x_{4}}>{x_{2}}$
Method 6 | ${x_{1}}>{x_{5}}>{x_{3}}>{x_{4}}>{x_{2}}$
Method 7 | ${x_{5}}>{x_{1}}>{x_{3}}>{x_{2}}>{x_{4}}$
S3W-GDM after the $3rd$ decision-level | ${x_{5}}>{x_{1}}>{x_{4}}>{x_{3}}>{x_{2}}$
S3W-GDM after the $4th$ decision-level | ${x_{5}}>{x_{1}}>{x_{3}}>{x_{4}}>{x_{2}}$

###  6.2 Sensitivity analysis

The subsection will display how parameter variation affects the decision
results. sensitivity of Breast Cancer Coimbra Data Set. For presentation
purposes, the Breast Cancer Coimbra Data Set with more alternatives is used
here as the sensitivity analysis. The main study is the variation of the
Gaussian kernel parameter $\sigma$ and the neighborhood cut parameter $\kappa$
for different relative gain parameters $\eta$. In Introduction 1 and Section
3, extensive discussion has taken place regarding the parameter of relative
gains. The typical value range for this parameter is [0, 1]. In this data set,
the experiments are conducted through varying $\eta$ from 0 to 1 with an
interval of 0.1. The conclusion is that when $\eta\leq 0.5$, all alternatives
are completely classified into the positive and negative regions at the $1st$
decision-level, which is evidently unreasonable. Given the large number of
alternatives, although the proposed method aims to enhance decision
efficiency, the limited decision information received at the $1st$ decision-
level results in significant errors if classification is based solely on the
first most important conditional attribute. Consequently, this study does not
consider $\eta\leq 0.5$ for this data set. When $\eta$ is 0.6, although the
classification of alternatives begins to show a general pattern, it remains
unstable with variations in $\sigma$ and $\kappa$, and subsequential changes
do not follow a consistent pattern. When $\eta\geq 0.7$, the classification of
alternatives stabilizes, and the subsequent variations in $\sigma$ and
$\kappa$ conform to the discussions in Section 4.3.

The purpose of this work is to observe $\sigma$ and $\kappa$ that are related
to the sequential process, namely the gaussian kernel parameter and the
neighborhood cut parameter with the relative gain parameter of 0.7, 0.8, and
0.9. Then the interval in which these two parameters are observed is [0,1],
with a step size of 0.1. Fig. 13 provides a detailed illustration of the
variations in $\sigma$ and $\kappa$. Subfigures (a), (b), and (c) demonstrate
the variations in the number of boundary region alternatives as a function of
the combination of $\sigma$ and $\kappa$. Even with different parameter
settings, the boundary region alternatives exhibit a stable pattern at this
decision-level. Notably, the closer the combination of $\sigma$ and $\kappa$
approaches (1, 1), the greater the number of alternatives in the boundary
region. This indicates that the decision conditions become stricter, aligning
with the semantic interpretation of variations in these two parameters.

(a) $\eta=0.7$.

(b) $\eta=0.8$.

(c) $\eta=0.9$.

Fig. 13: . Distribution of decision-level with the variation of parameters.
Fig. 14: . Classification results under different combinations of parameters.

Next, to control for variables, the classification results of 30 patients
under $\eta=0.7$ are presented in Fig. 14. This is to illustrate the
classification trends under different parameter settings. Fig. 14 illustrates
the classification of alternatives under four different combinations of
$\sigma$ and $\kappa$. The four sets of parameter combinations are (0.9,0.9),
(0.8,0.8), (0.8,0.7), and (0.9,0.7). Each parameter combination results in
distinct classification trends. This is due to the ability to adjust and vary
$\sigma$ and $\kappa$ at each decision-level. In this study, the value of
$\sigma$ is fixed to maintain sensitivity in the calculations. By adjusting
the changes in $\kappa$ at each decision-level, ensuring that the $\kappa$
values gradually increase, the accuracy of the decision results is ensured,
leading to diverse classification results while maintaining the overall trend.
In each set of sequential processes, $\kappa$ in this way changes at the
decision-level in the interval [0.8,1]. Regarding the properties of these two
parameters, a $\sigma$ value closer to 1 results in a more sensitive Gaussian
kernel function, while a $\kappa$ value closer to 1 indicates stricter
equivalence division among alternatives. Both settings contribute to the
accuracy of the final classification results.

In general, the yellow area tends to shrink as the decision-making process
progresses. Conversely, the blue and red areas may either remain constant or
expand as the decision-making stages advance. These two performance
characteristics accurately reflect the actual decision-making situation.
Furthermore, the more alternatives that are divided, the more varied the
results will be. Different parameters can be adjusted to achieve different
outcomes based on the specific decision-making scenario. The model’s
classification rationality is demonstrated through sensitivity analysis.

###  6.3 Discussion

Since the method proposed in this paper is a dynamic decision-making model to
a greater extent, unlike static decision-making which collects all the
decision-making information at once, mutli-level S3W-GDM makes decisions by
increasing the granularity of the decision-making information in a level-by-
level progression, which on the one hand improves the efficiency of decision-
making, and on the other hand provides a buffer to reduce the risk of
erroneous decision-making when the decision-making information is insufficient
to support the decision.

The differences between the proposed method and others are listed as follows:

(1) Classical GDM methods [33, 47] or multi-attribute decision-making methods
[19, 15, 14, 13, 16] usually use 2WD methods. These methods rank alternatives
based on scores to produce decision results, but lack semantic interpretation
of the decision results. In contrast, the proposed method employs a S3WD
process that provides meaningful explanations for situations such as medical
diagnosis and emergency logistics service provider selection.

(2) In classical GDM methods [33, 47], information fusion is typically handled
by aggregation operators that combine all experts’ information at once. Non-
operator-based information fusion methods consider information fusion from
different perspective but still follow a holistic fusion approach. The
proposed S3W-GDM method, however, combines the concept of multi-granularity
with conventional thinking. It introduces a coarse-to-fine granularity
approach to information fusion, where initial decisions are made using coarse-
grained information, followed by progressively finer-grained analysis to
refine the decisions. This multi-level fusion approach enhances the decision-
making process’s efficiency and accuracy. For breast cancer diagnosis, the
S3W-GDM method utilizes a multi-level granularity approach that first focuses
on key attributes and then continuously improves decision making to improve
decision making efficiency. For emergency logistics provider selection, the
S3W-GDM method offers rapid a priori solutions, crucial in emergencies. The
S3W-GDM method achieves stable classification by the $3rd$ decision-level with
only half the decision information, enhancing efficiency under time
constraints.

(3) The most methods [33, 19, 15, 14, 13] compared in this study do not
adequately address the qualitative expression of decision preferences,
particularly under DHHFLTS environment. Classical methods lack work on the
transition from qualitative evaluations to quantitative changes, leading to
potential biases in decision-making. The proposed S3W-GDM method addresses
this gap by effectively capturing and incorporating qualitative decision
preferences into the decision-making process, ensuring a balanced and
comprehensive evaluation.

The advantages between the proposed method and others are summarized as
follows:

(1) By incorporating a S3WD process and multi-granularity information fusion,
the S3W-GDM method provides comprehensive and interpretable decision results.
This approach not only ranks the alternatives but also classifies them into
positive, boundary, and negative regions, offering a clear semantic
interpretation of the results.

(2) The S3W-GDM method uses a coarse-to-fine granularity approach, pioneering
a new model of information fusion. Initial decisions are made swiftly using
coarse-grained key attributes, providing a rapid preliminary assessment.
Further refinements are then applied to alternatives requiring additional
analysis, utilizing finer-grained information. This multi-level fusion ensures
that assessments are balanced and comprehensive, providing an effective way to
use qualitative evaluation information.

(3) The novel S3WD of DHHFLTS method to address the problem of uncertainty in
decision alternatives by redesigning the computation of conditional
probabilities without relying on decision attributes, this approach improves
the accuracy. It incorporates relative utility into the evaluation of each
alternative, capturing individual psychological behaviour and also improving
decision-making accuracy.

##  7 Conclusion and future work

With the progress of society and information science, GDM problems are
becoming increasingly complex. Classical GDM methods, which rely on
aggregation operators to fuse information from different attributes and
decision-makers at once, significantly increase the decision burden and
constrain efficiency. Moreover, these problems often exhibit vagueness,
hesitation, and variation, adding to their complexity. Existing relative works
rarely take these characteristics into account while improving decision-making
efficiency by changing the paradigm of information fusion. Accordingly, the
work of this paper is summarised as follows. First, constructing a
neighborhood relation matrix based on derived similarity degrees between
alternatives and combining it with the outranking relation to refine
conditional probability calculations. Then, designing a new “loss function”
model for decision risk based on relative perceived utility, incorporating
regret theory (RT). This includes defining expert decision tables and multi-
level granular extraction and aggregation of evaluations. These two steps
establish the foundation of the novel S3WD of DHHFLTS model. Further, the
paper demonstrates the most efficient operator for aggregation in the
decision-level information fusion process, defines a multi-level granular
structure, and proposes decision-making strategies and semantic
interpretations for each level. The efficiency and rationality of the
established method are validated through illustrative example and comparative
analysis with other methods.

In future research, the following three points will be emphasized. With the
development of information science, the volume of tool-focused data for
solving complex problems has become larger and larger. Therefore, the DHHFLTS
as a kind of natural language word computing needs to raise the level of
dealing with decision-making problems to a large-scale group[48, 21], deal
with a larger volume of data through machine learning or deep learning
algorithms[49, 22], and promote the development of the integration of computer
science and technology and management science engineering. Additionally, when
the volume of data becomes larger, how to effectively allocate computing
resources will also become an important issue. Finally, no matter what kind of
decision-making the ultimate goal is to reach a consensus[30], the future will
be centered on multi-granularity to do consensus research.

## Acknowledgement

This work was supported by the National Natural Science Foundation of China
(No.62276
038, No. 62221005), the Joint Fund of Chongqing Natural Science Foundation for
Innovation and Development under Grant (No.CSTB2023NSCQ-LZX0164), the
Chongqing Talent Program (No.CQYC20210202215), the Chongqing Municipal
Education Commission (HZ2021008), and the Doctoral Talent Training Program of
Chongqing University of Posts and Telecommunications (No.BYJS202213),.

## References

  * [1] E. J. Zirkzee, G. M. Steup-Beekman, E. L. Bollen, E. Baptist, T. M. Slee, M. V. Huisman, H. A. Middelkoop, J. Luyendijk, M. A. van BUCHEM, T. W. Huizinga, et al., Prospective study of clinical phenotypes in neuropsychiatric systemic lupus erythematosus; multidisciplinary approach to diagnosis and therapy, The Journal of rheumatology 39 (11) (2012) 2118–2126.
  * [2] E. Herrera Viedma, I. Palomares, C. C. Li, F. J. Cabrerizo, Y. Dong, F. Chiclana, F. Herrera, Revisiting fuzzy and linguistic decision making: Scenarios and challenges for making wiser decisions in a better way, IEEE Transactions on Systems, Man, and Cybernetics: Systems 51 (1) (2020) 191–208.
  * [3] R. M. Rodríguez, L. Martínez, F. Herrera, A group decision making model dealing with comparative linguistic expressions based on hesitant fuzzy linguistic term sets, Information Sciences 241 (2013) 28–42.
  * [4] Q. Pang, H. Wang, Z. Xu, Probabilistic linguistic term sets in multi-attribute group decision making, Information Sciences 369 (2016) 128–143.
  * [5] Y. Xu, A. Xu, J. M. Merigó, H. Wang, Hesitant fuzzy linguistic ordered weighted distance operators for group decision making, Journal of Applied Mathematics and Computing 49 (2015) 285–308.
  * [6] P. Liu, F. Teng, Some muirhead mean operators for probabilistic linguistic term sets and their applications to multiple attribute decision-making, Applied Soft Computing 68 (2018) 396–431.
  * [7] D. Kahneman, A. Tversky, Prospect theory: An analysis of decision under risk, in: Handbook of the fundamentals of financial decision making: Part I, World Scientific, 2013, pp. 99–127.
  * [8] G. Loomes, R. Sugden, Regret theory: An alternative theory of rational choice under uncertainty, The economic journal 92 (368) (1982) 805–824.
  * [9] P. Liu, Y. Li, An extended multimoora method for probabilistic linguistic multi-criteria group decision-making based on prospect theory, Computers & Industrial Engineering 136 (2019) 528–545.
  * [10] S. Zhang, J. Zhu, X. Liu, Y. Chen, Regret theory-based group decision-making with multidimensional preference and incomplete weight information, Information Fusion 31 (2016) 1–13.
  * [11] Z. H. Chen, W. Luo, An integrated interval type-2 fuzzy rough technique for emergency decision making, Applied Soft Computing 137 (2023) 110150.
  * [12] Y. Sun, J. Mi, J. Chen, W. Liu, A new fuzzy multi-attribute group decision-making method with generalized maximal consistent block and its application in emergency management, Knowledge-Based Systems 215 (2021) 106594\.
  * [13] X. Gou, H. Liao, Z. Xu, F. Herrera, Double hierarchy hesitant fuzzy linguistic term set and multimoora method: A case of study to evaluate the implementation status of haze controlling measures, Information Fusion 38 (2017) 22–34.
  * [14] X. Gou, Z. Xu, H. Liao, F. Herrera, Multiple criteria decision making based on distance and similarity measures under double hierarchy hesitant fuzzy linguistic environment, Computers & Industrial Engineering 126 (2018) 516–530.
  * [15] Z. Liu, X. Zhao, L. Li, X. Wang, D. Wang, A novel multi-attribute decision making method based on the double hierarchy hesitant fuzzy linguistic generalized power aggregation operator, Information 10 (11) (2019) 339.
  * [16] X. Gou, Z. Xu, H. Liao, F. Herrera, Probabilistic double hierarchy linguistic term set and its use in designing an improved vikor method: The application in smart healthcare, Journal of the Operational Research Society 72 (12) (2021) 2611–2630.
  * [17] P. Liu, M. Shen, W. Pedrycz, Magdm framework based on double hierarchy bipolar hesitant fuzzy linguistic information and its application to optimal selection of talents, International Journal of Fuzzy Systems (2022) 1–23.
  * [18] J. Montserrat-Adell, Z. Xu, X. Gou, N. Agell, Free double hierarchy hesitant fuzzy linguistic term sets: An application on ranking alternatives in gdm, Information Fusion 47 (2019) 45–59.
  * [19] Z. Liu, D. Wang, Y. Zhao, X. Zhang, P. Liu, An improved electre ii-based outranking method for madm with double hierarchy hesitant fuzzy linguistic sets and its application to emergency logistics provider selection, International Journal of Fuzzy Systems 25 (4) (2023) 1495–1517.
  * [20] X. Gou, X. Xu, F. Deng, W. Zhou, E. Herrera-Viedma, Medical health resources allocation evaluation in public health emergencies by an improved oreste method with linguistic preference orderings, Fuzzy Optimization and Decision Making 23 (1) (2024) 1–27.
  * [21] X. Cheng, Z. Xu, X. Gou, A large-scale group decision-making model considering risk attitudes and dynamically changing roles, Expert Systems with Applications 245 (2024) 123017.
  * [22] X. Cheng, K. Zhang, T. Wu, Z. Xu, X. Gou, An opinions-updating model for large-scale group decision-making driven by autonomous learning, Information Sciences 662 (2024) 120238.
  * [23] Y. Yao, The dao of three-way decision and three-world thinking, International Journal of Approximate Reasoning 162 (2023) 109032.
  * [24] Y. Yao, Granular computing and sequential three-way decisions, in: International conference on rough sets and knowledge technology, Springer, 2013, pp. 16–27.
  * [25] Y. Liu, L. Zhu, R. M. Rodríguez, Y. Yao, L. Martínez, Three-way group decision-making with personalized numerical scale of comparative linguistic expression: An application to traditional chinese medicine, IEEE Transactions on Fuzzy Systems (2024).
  * [26] Y. Yang, M. Q. Jie, Z. S. Chen, Dynamic three-way multi-criteria decision making with basic uncertain linguistic information: A case study in product ranking, Applied Soft Computing 152 (2024) 111228.
  * [27] X. Yang, T. Li, D. Liu, H. Fujita, A multilevel neighborhood sequential decision approach of three-way granular computing, Information Sciences 538 (2020) 119–141.
  * [28] J. Hu, W. Cao, P. Liang, A novel sequential three-way decision model for medical diagnosis, Symmetry 14 (5) (2022) 1004.
  * [29] Q. Zhang, C. Yang, G. Wang, A sequential three-way decision model with intuitionistic fuzzy numbers, IEEE transactions on systems, man, and cybernetics: systems 51 (5) (2019) 2640–2652.
  * [30] M. Wang, D. Liang, Z. Xu, Sequential three-way multiple attribute group decisions with individual attributes and its consensus achievement based on social influence, Information Sciences 518 (2020) 286–308.
  * [31] Y. Wang, B. Sun, X. Zhang, Q. Wang, Bwm and multimoora-based multigranulation sequential three-way decision model for multi-attribute group decision-making problem, International Journal of Approximate Reasoning 125 (2020) 169–186.
  * [32] R. Krishankumar, K. Ravichandran, V. Shyam, S. Sneha, S. Kar, H. Garg, Multi-attribute group decision-making using double hierarchy hesitant fuzzy linguistic preference information, Neural Computing and Applications 32 (2020) 14031–14045.
  * [33] R. Krishankumar, L. Subrajaa, K. Ravichandran, S. Kar, A. B. Saeid, A framework for multi-attribute group decision-making using double hierarchy hesitant fuzzy linguistic term set, International Journal of Fuzzy Systems 21 (2019) 1130–1143.
  * [34] X. Gou, H. Liao, Z. Xu, R. Min, F. Herrera, Group decision making with double hierarchy hesitant fuzzy linguistic preference relations: Consistency based measures, index and repairing algorithms and decision model, Information Sciences 489 (2019) 93–112.
  * [35] D. E. Bell, Regret in decision making under uncertainty, Operations research 30 (5) (1982) 961–981.
  * [36] J. Quiggin, Regret theory with general choice sets, Journal of Risk and Uncertainty 8 (1994) 153–165.
  * [37] N. Luo, Q. Zhang, L. Yin, Q. Xie, C. Wu, G. Wang, Three-way multi-attribute decision-making under the double hierarchy hesitant fuzzy linguistic information system, Applied Soft Computing (2024) 111315.
  * [38] Q. Hu, L. Zhang, D. Chen, W. Pedrycz, D. Yu, Gaussian kernel based fuzzy rough sets: model, uncertainty measures and applications, International Journal of Approximate Reasoning 51 (4) (2010) 453–471.
  * [39] Q. Hu, D. Yu, Z. Xie, Neighborhood classifiers, Expert systems with applications 34 (2) (2008) 866–876.
  * [40] J. Ye, J. Zhan, Z. Xu, A novel decision-making approach based on three-way decisions in fuzzy information systems, Information Sciences 541 (2020) 362–390.
  * [41] Y. Yao, Three-way decisions with probabilistic rough sets, Information sciences 180 (3) (2010) 341–353.
  * [42] F. Jia, P. Liu, A novel three-way decision model under multiple-criteria environment, Information Sciences 471 (2019) 29–51.
  * [43] D. Liang, M. Wang, Z. Xu, Heterogeneous multi-attribute nonadditivity fusion for behavioral three-way decisions in interval type-2 fuzzy environment, Information Sciences 496 (2019) 242–263.
  * [44] W. Lei, W. Ma, B. Sun, Multigranulation behavioral three-way group decisions under hesitant fuzzy linguistic environment, Information Sciences 537 (2020) 91–115.
  * [45] J. Ye, B. Sun, J. Zhan, X. Chu, Variable precision multi-granulation composite rough sets with multi-decision and their applications to medical diagnosis, Information Sciences 615 (2022) 293–322.
  * [46] R. R. Yager, Fusion of ordinal information using weighted median aggregation, International journal of approximate reasoning 18 (1-2) (1998) 35–52.
  * [47] R. Zhang, Z. Xu, X. Gou, An integrated method for multi-criteria decision-making based on the best-worst method and dempster-shafer evidence theory under double hierarchy hesitant fuzzy linguistic environment, Applied intelligence 51 (2021) 713–735.
  * [48] M. Tang, H. Liao, From conventional group decision making to large-scale group decision making: What are the challenges and how to meet them in big data era? a state-of-the-art survey, Omega 100 (2021) 102141.
  * [49] R. X. Ding, I. Palomares, X. Wang, G.-R. Yang, B. Liu, Y. Dong, E. Herrera-Viedma, F. Herrera, Large-scale decision-making: Characterization, taxonomy, challenges and future directions from an artificial intelligence and applications perspective, Information fusion 59 (2020) 84–102.


11institutetext:  University of North Carolina Charlotte, Charlotte, NC, USA
22institutetext: Carnegie Mellon University, Pittsburgh, PA, USA
33institutetext: Pacific Northwest National Laboratory, Richland, WA 99354
33email<EMAIL_ADDRESS>33email<EMAIL_ADDRESS>33email:
<EMAIL_ADDRESS>
# Constraints Satisfiability Driven Reinforcement Learning for Autonomous
Cyber Defense††thanks: Accepted at International Conference on Autonomous
Intelligent Cyber-defence Agents (AICA 2021).

Ashutosh Dutta 11 Ehab Al-Shaer 22 Samrat Chatterjee 33112233

###### Abstract

With the increasing system complexity and attack sophistication, the necessity
of autonomous cyber defense becomes vivid for cyber and cyber-physical systems
(CPSs). Many existing frameworks in the current-state-of-the-art either rely
on static models with unrealistic assumptions, or fail to satisfy the system
safety and security requirements. In this paper, we present a new hybrid
autonomous agent architecture that aims to optimize and verify defense
policies of reinforcement learning (RL) by incorporating constraints
verification (using satisfiability modulo theory (SMT)) into the agent’s
decision loop. The incorporation of SMT does not only ensure the
satisfiability of safety and security requirements, but also provides constant
feedback to steer the RL decision-making toward safe and effective actions.
This approach is critically needed for CPSs that exhibit high risk due to
safety or security violations. Our evaluation of the presented approach in a
simulated CPS environment shows that the agent learns the optimal policy fast
and defeats diversified attack strategies in 99% cases.

##  1 Introduction

With wide applications spanning from national critical infrastructures (e.g.,
smart grid, transport) to personal domains (e.g., home automation system,
healthcare), cyber and CPS systems become more susceptible to cyber attacks
due to misconfigurations, unpatched, or unknown vulnerabilities. Moreover,
attacks like Advanced Persistent Threat (APT) are well-resourced and highly
sophisticated to cause serious and large damage for critical infrastructures
within relatively a short time [13]. Therefore, automating proactive defense
such as penetration testing and risk identification, and reactive defense such
as intrusion response is a key to maintain the integrity and security of these
systems.

Developing autonomous agents for cyber defense is one of the most promising
solutions to achieve real-time monitoring and response against advanced
attackers with minimal human involvement. Autonomous cyber defense agents
(ACDA) have capabilities to not only timely respond to malicious actions but
also adapt their decision-making dynamically to cope with changes of
environment or attack strategies. On the other hand, to guarantee the mission
safety, ACDA actions must be shown provably correct according to the mission,
operation, and business requirements.

Researchers have applied game theory [4], sequential decision process [6, 8],
and reinforcement learning [5, 9] to optimize defense response planning.
However, these works have limited real-world applications due to struggling to
converge while having numerous requirements. Several works apply constraint
satisfaction problems (CSP) [13, 12] to optimize planning considering all
requirements as constraints. However, these works rely on static models for
critical parameters which may be very hard to formulate, even
probabilistically, due to lack of domain specific data. Moreover, static
assumptions on attackers’ exploitation capabilities restrict attack behavior
unrealistically. Therefore, current state-of-the-art of autonomous cyber
defense lacks a framework that can optimize defense planning at real-time
while satisfying all various requirements.

In this paper, we present a new hybrid autonomous agent architecture that
optimizes defense policies through incorporating the feedback on constraint
satisfiability into the decision loop. We formulate the defense optimization
problem as a Sequential Decision Process (SDP) [11], where defense
effectiveness depends on stochastic environment behavior, adaptive attack
strategies, and mission-oriented safety and security requirements. However,
ACDA usually lacks domain-specific experience and data to predict attack
behaviors or characterize defense effectiveness. To accomplish this goal, we
develop a novel approach, named Constrained Satisfiability-driven
Reinforcement Learning (CSRL) approach, to solve the SDP through learning the
environment based on interactive experience with the environment. CSRL employs
model-free Reinforcement Learning [11] to optimize the defense decision-
making, and applies Satisfiability Modulo Theory (SMT) [3] for constraints
satisfiability verification to provide verifiability and refinement of the
defense actions according to safety and security properties. The incorporation
of SMT architecture guides the agent’s RL algorithm towards safe and effective
defense planning.

Our CSRL approach decouples the policy optimization and constraint satisfying
modules to address the challenge of computation complexity. Instead of feeding
constraints directly to the optimizer, the policy is updated based on the
satisfiability of current constraint set by computed defense actions. This
approach does not only make the agent computationally feasible for real-time
defense optimization in a constrained environment, but also offers flexibility
in integrating new or evolved requirements into decision-making. Moreover, the
agent reasons over environment feedback to deduce potential requirements that
may remain undefined or vague due to dynamic factors and incomplete domain
knowledge. Also, the unsatisfiability feedback improves the convergence of
agent’s policy update through steering it to satisfiable regions.

Autonomous defense agents for CPSs will highly need to adopt the CSRL approach
in order to avoid safety and security violations. CPS usually exhibits many
safety and security requirements that defense action must not violate to
maintain the expected behavior of the infrastructure. We develop a use case
scenario that simulates a CPS environment to assess the presented agent
architecture. We show in our experiments that our agent converges to optimal
planning at reasonable time windows despite having no prior knowledge, and the
trained agent defeats attackers with diversified strategies within few time-
sequences in 99% cases. Hence, the outcome of our evaluation demonstrates the
applicability of our agent for real-world cyber applications.

##  2 Overview of Integrated Reasoning Framework for Autonomous Agent

Fig. 1 illustrates our framework that takes the State Space $S$ (set of
possible states), Defense Action Space $A$ (set of possible defense actions),
and optional previous policy (if any) as inputs. The Constraint Formulation
(cons-form) module composes initial Constraint Set by formulating known
business requirements or expert knowledge. At the start of time-sequence $t$,
State Characterization module characterizes the current observation to a state
(distinct environment condition) and sends to both Policy Optimizer and
Constraints Learner (cons-learner). For the given state, Policy Optimizer
recommends the optimal action to Constraints Learner and Pre Execution
Satisfiablity (pre-sat) modules.

Figure 1: Autonomous Agent Architecture and Workflow. The environment contains
an Attacker who observes the environment and strategically executes attack
actions. Note# An arrow (input) ending at dotted box specifies that all
modules inside that box receives the input.

The pre-sat module checks if the recommended action can be deployed without
violating any constraint of a specific subset of constraint set (i.e.,
received from cons-form module). If it fails, the agent sends a Penalty to the
policy optimizer. The optimizer updates the policy immediately based on the
penalty and recommends new action during the same time $t$. Notably, the state
remains unchanged due to not executing the action. In contrary, if the action
satisfies all constraints, the agent executes it on environment as Pre-
Satisfied Defense Action. In response or concurrently, the attacker executes
his next attack action.

Such attack and defense interplays trigger observations, based on which, the
agent infers the action impact. Then, it checks whether the impact conforms
with dynamic or remaining set of requirements (unchecked at pre-sat) at Post
Execution Satisfiability (post-sat) module. If it does not satisfy, the agent
sends a Penalty to the optimizer; otherwise, it quantifies the rewards to send
to the optimizer. The policy optimizer updates the policy based on these
rewards or penalties. Moreover, based on such interactive experience involving
action execution and receiving feedback, the agent’s cons-learner learns or
deduces new requirements that are sent to cons-form module to compose new
constraint set.

##  3 Autonomous Agent Architecture

This section describes components of Autonomous Agent architecture at Fig. 1.

###  3.1 Constraints Formulation

Generally, cyber infrastructures such as CPSs contain diversified requirements
that the computed defense strategy must not violate to maintain its expected
behavior, safety, and security. These requirements can be business and
mission-oriented, expert knowledge (e.g., historical experience), and others.
Notably, expert knowledge may include commands of the network administrator,
for example, keeping at least 20% free resources to avoid hardware failures.
The Constraint Formulation (cons-form) module in Fig. 1 formulates all such
requirements as SMT constraints.

Alongside user-given business requirements and expert knowledge, this module
updates the constraint set when the agent learns new requirements based on its
interactions with environment. Moreover, it modifies the set due to change of
any business or existing requirements. Therefore, by this module, the agent’s
decision optimization can easily cope with the evolvement of requirements.

###  3.2 Constraints Learner

It is generally infeasible to know all constraints initially due to lack of
deep domain knowledge or data, whereas some requirements can only be known
after going into the operations due to environmental variability [7]. For
example, defining constraints for power system state estimation requires
determining confidence on measured data. However, such domain-specific
confidence depending on the likelihood of errors or sensor failures can only
be computed by analyzing operational behaviors. Besides, deterministic
approaches to specify such uncertain behaviors tend to be overly conservative.
Hence, critical infrastructures such as autonomous vehicles, smart grid
nowadays endeavor to learn behavioral information from the environment.

Our agent actively learns new requirements using feedback (i.e., rewards,
observations) of environment. In Fig. 1, the Constraints Learner (cons-
learner) module receives rewards or penalty as consequences of recently
recommended defense actions. By analyzing rewards achieved at specific states,
the agent deduces which group of actions should be avoided or preferred at
particular environment conditions. For instance, if termination of a specific
communication always induces intolerable business loss, the agent easily
understands that the communication should remain untouched. However, before
concluding observations to any such constraint, the agent must observe
consequences of that action for multiple similar events due to non-
deterministic environment behavior. Though we consider a static value for that
required number of events, we plan to determine it dynamically in future
extension. Moreover, there are ongoing research efforts to mine such
constraints at real-time from observations such as runtime event logs or
physical world information [7].

###  3.3 Constraints Checker

The Constraint Checker (cons-checker) module (dotted box at Fig. 1) uses SMT
to check whether the recent defense strategy, recommended by Policy Optimizer,
satisfies all formulated constraints or not. Our approach detaches cons-
checker from the Policy Optimizer, because incorporating all requirements
explicitly into optimization algorithm not only hardens the convergence of
optimal policies but also may induce computational infeasibility. Therefore,
rather than considering constraints directly, the optimizer considers
rewards/penalty, computed based on the satisfiability of current constraint
set by recent defense actions. This module performs constraints satisfiability
verification in the following two phases:

####  3.3.1 (1) Pre Execution Satisfiability Checker:

This module verifies if there is any planning that can implement the
recommended defense action without violating any constraint of Pre-satisfiable
constraint set (pre-cons). For example, if the recommended action wants
traffic monitoring at several critical links, it checks whether any monitoring
plan can achieve that within affordable energy. Importantly, pre-cons either
do not rely on uncertain and dynamic environment factors or consider uncertain
factors probabilistically. For example, Smart Grid considers various critical
packet/message delay constraints [12] by predicting packet delay, because it
cannot be determined certainly due to unanticipated network effects such as
changes in load balancing, or hardware failures.

In Fig. 1, the Pre Satisfiability (pre-sat) module checks the conformity of
the recommended defense action with current pre-cons. Based on the
satisfiability of these constraints, two following cases appear:

(a) Not Satisfied: If the recommended action fails to satisfy any constraint
of pre-sat, the agent immediately sends a Penalty to the policy optimizer
without executing the action. This is unlike traditional reinforcement
learning approaches that update the policy only after executing the action.

(b) Satisfied: If the recommended action satisfies all pre-sat constraints, it
is executed as Pre-Satisfied Defense Action on the environment.

Our approach of not executing unsatisfiable actions makes the agent’s RL
exploration (exploration of action impacts) more effective by (1) avoiding
execution of irrelevant (ineffective for current condition) actions that
induce disastrous impact on real environment, and (2) offering flexibility for
more explorations.

####  3.3.2 (2) Post Execution Satisfiability Checker

This module checks the satisfiability of a subset of constraints, termed as
Post-satisfiable constraint set (post-cons), after executing the pre-satisfied
defense action on the environment. It is beneficial for any cyber system with
following properties:

1\. Constraints with dynamic or uncertain factors: Certain verification of
these constraints demands interactions with the environment, because
scrutinizing impacts of actions on these dynamic factors require executing
them. Importantly, even though such a constraint may be satisfied
probabilistically at pre-sat module, the agent checks its satisfiability as
post-cons.

2\. Numerous Constraints: Verifying all constraints at runtime before
executing an action may not be feasible for ensuring real-time defense
optimization. Hence, the decision framework can only verify subset of
constraints to ensure bounded computational overhead, and the remaining
constraints need to be verified after the action execution.

After executing the action, the Post Satisfiability (post-sat) module at Fig.
1 receives observations from the environment, and checks if the impact of
action conforms all post-cons. Based on satisfiability, following cases
appear:

(a) Not Satisfied: If the executed defense action cannot satisfy any of post-
cons, the agent sends a Penalty to the policy optimizer for that action.

(b) Satisfied: If it satisfies all post-cons, the agent forwards the recent
observations to Reward Calculation for quantifying the action payoffs and
impact.

###  3.4 Policy Optimizer

Policy optimizer optimizes defense policy by maximizing action payoffs
(rewards), that recommends an optimal defense action for a particular state.
Due to no or limited knowledge about the environment initially, the agent
applies Reinforcement Learning (RL) that updates the defense policy based on
rewards or penalty received as feedback [11]. Besides exploiting the previous
experience or knowledge, RL algorithms optimally explore the consequences of
other unexplored actions (i.e., RL-exploration). Thus, our agent applying RL
computes optimal policy through learning the environment based on interactive
experience.

The agent defines the environment and interactions using State space $S$,
Observation space $O$, Action space $A$, and Reward function $R$. As shown in
Fig. 1, the Policy Optimizer recommends the defense action for the current
state and receives feedback. This module uses Proximal Policy Optimization
(PPO) [10] as RL algorithm, that shows better performance for continuous
control tasks with two advantages: (1) constraining policy update within a
small range to avoid drastic deviation from old policy, and (2) performing
multiple epochs on same minibatch data [10]. The first advantage helps the
agent to cope with sensor noises or errors, whereas the second one aids to
cope with the delayed feedback. PPO optimizes a clipped surrogate objective
function, $L^{CLIP}(\theta)$, using Eqn. 1.

$L^{CLIP}(\theta)=\mathbb{E}_{t}[min(r_{t}(\theta)A_{t},clip(r_{t}(\theta),1-\epsilon,1+\epsilon)A_{t})]\vspace{-.6em}$
(1)

where, $\theta$ represents policy parameter, $\epsilon$ is clip-range hyper-
parameter, and $\pi_{\theta}$ and $\pi_{old}$ represents new and old
stochastic policies respectively. Moreover,
$r_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{old}(a_{t}|s_{t})}$
specifies the likelihood ratio, where $\pi_{\theta}(a_{t}|s_{t})$ specifies
the probability of executing $a_{t}$ at state $s_{t}$ by $\pi_{\theta}$.
Notably, PPO clips $r_{t}(\theta)$ if outside of $[1-\epsilon,1+\epsilon]$ to
restrain large update. It formulates the advantage function $A_{t}$ by Eqn. 2,
considering $V(s_{t+l})$ (i.e., expected reward of state $s_{t+l}$ at time
$t+l$) as baseline value to lower variance.

$A_{t}=\sum_{l=0}^{T}\gamma^{l}(r_{t+l}+\gamma
V(s_{t+l+1})-V(s_{t+l}))\vspace{-0.8em}$ (2)

where, $\gamma\in[0,1)$ is discount factor that weighs future value, $r_{t+1}$
is the current reward or penalty, and $T$ is decision-horizon length until
$\gamma^{l}>0$.

PPO applies Advantage Actor Critic (A2C) approach [10] to optimize
$L^{CLIP}(\theta)$, where Critic estimates $V(s_{t})$ of $s_{t}$, and Actor
optimizes the policy based on $A_{t}$.

###  3.5 State Characterization

State represents a distinct condition of the environment based on critical
environmental factors. Based on recent observations, the agent characterizes
the current environment condition to a particular state for deciding the next
optimal action. Symptoms observed from the environment or network may reveal
the underlying state certainly or partially.

Importantly, most of model-free RL algorithms implicitly address uncertainties
associated with observations due to a partial observability, which is unlike
the explicit Belief calculation (probabilistic inference of current state) in
model-based SDP. The applied PPO algorithm for our policy optimzation uses the
characterized observation to decide the next action.

###  3.6 Rewards and Penalty Calculator

Reward quantifies the payoff of a defense action $a_{d}$ and provides as
feedback to the policy optimizer. Understandably, higher reward to a action
for a state bias the optimizer to select that action due to its objective of
maximizing rewards. Our agent assigns two types of rewards to $a_{d}$: (1)
Penalty if $a_{d}$ fails to satisfy any pre or post constraint, and (2) Reward
otherwise.

The Reward Calculation module at Fig. 1 uses current observations to quantify
rewards (can also be negative) based on the current status of the environment,
improvement or degradation of CPS performance, user feedback on offered
services, defense cost (includes deployment cost and negative impact), and
others. For a stochastic environment, reward function depends on multiple
uncertain factors, and the administrator may change the weight of certain
parameters or introduce new parameters based on his/her refined knowledge or
new data. Whereas, the Penalty Calculation quantifies the Penalty based on
severity of constraints violation.

##  4 Evaluation

This section describes the setup of experiments that are conducted to assess
the agent’s performance and discusses these experiments’ outcome.

###  4.1 Experiment Setup

This section describes the use case and simulation parameters of our
experiment.

Use Case Scenario: We consider a CPS (e.g., smart grid) setting that
accommodates anomaly-based detectors, to monitor critical connections among
heterogeneous devices and provide probabilistic risk scores based on anomalous
behavior. These detectors consume varying energy based on required
computation, and all detectors cannot be enabled at a time due to limited
energy. A device’s risk score is the mean of all scores provided by enabled
detectors at its multiple connections considering same accuracy of all
detectors. There are two terminating states: (1) Attack-goal-state when the
attacker compromise at least 50% of all devices, and (2) Attack-end-state when
the agent removes the attacker from all compromised devices.

Attack Model: The attacker aims to reach the attack goal state by propagating
from compromised devices to connected (neighbor) devices. We consider three
types of attackers: (1) Naive attacker who randomly explores $N$ compromised
nodes to propagate, (2) Stealthy attacker who strategically selects
$\frac{N}{2}$ compromised nodes to explore while generating lower risk scores,
and (3) Aggressive attacker who is stealthy and can explore $N$ machines.

Agent’s Objective: The agent may restart and reimage a device if its risk
score is above than a threshold. However, such threshold needs to be
dynamically selected to balance the trade-off between false positive (benign
device identified as compromised) and false negative (compromised devices
identified as benign) rate, considering current attack strategies and the
number of enabled detectors. Therefore, the agent aims to dynamically compute
the optimal threshold to reimage compromised devices and optimally enable
detectors for efficient monitoring after satisfying all constraints at real-
time.

RL Model Primitives: The agent’s defense space $A$ includes 3 qualitative
levels for increasing or decreasing anomaly threshold $\delta_{d}$ (6 actions)
followed by reimaging, 3 levels for increasing or decreasing enabled detector
ratio $f$ of a device (6 actions), reimaging of devices, and do nothing. The
state space $S$ consists of distinct compositions of 6 qualitative ratio
levels of compromised devices (e.g., less than 50%) with 3 levels (e.g., low
number) of enabled-detector (18 states), and 2 terminating states.
Importantly, a device, compromised or not, can be known certainly only after
reimaging it; hence, the state characterization based on currently observed
risk scores is uncertain. The agent’s reward function $R$ is formulated using
the following equation:

$\vspace{-0.3em}R(s,a)=-b_{r}\times C_{r}-d_{r}\times C_{i}+H_{t}\times
I_{w}-H_{g}\times C_{v}$ (3)

where, $b_{r}$ is benign (non-compromised) devices reimaged, $d_{r}$ is the
number of reimaged devices, boolean $H_{t}=1$ if the attack ends, boolean
$H_{g}=1$ if the attack reaches goal state, $C_{r}$, $C_{i}$, and $C_{v}$ are
costs, and $I_{w}$ is the incentive.

Constraints: Pre-cons contains two vital requirements: (1) bounded expected
energy consumption at a time by enabled detectors, and (2) enabling at least
$l$ detectors for each device. To clarify, for a recommended action such as
lowly increase $f$, the agent verifies if any detector-subset can satisfy all
constraints. As Post-cons, it checks whether (1) real energy consumption and
(2) loss due to reimaging benign devices are within tolerable limits.

Implementation: We use Python3.6 to implement the framework and attack model
that generates numerous attack scenarios to train and test the agent. We
consider two topologies: Topo 1 with 100 devices, and Topo 2 with 200 devices.
Our detectors’ risk score distributions for compromised devices follow power
law, whose tails stretch towards lower ends with increased attack
stealthiness. We use OpenAI Gym [1] to simulate the environment, and use PPO2
and MlpPolicy libraries of Stable Baselines [2] to implement PPO.

###  4.2 Results

We investigate (1) how efficiently the agent defends diversified attack
strategies, (2) how fast the agent converges to optimal defense planning, and
(3) how much benefits the constraints satisfying module offers.

Agent’s Learning Curve: Fig. 2 illustrates the progression of agent’s learning
during training, where an Episode consists of 1000 time-sequences.

Figure 2: Reward (Normalized) w.r.t. training progression.

Here, for instance, rewards of plot (1,1) are normalized based on maximum
reward achieved against Naive attacker at topo 1. As we can see, the agent
converges faster for topo 2 despite slow start, due to more satisfiable
plannings and opportunities to explore before termination state. Within 50
episodes, it reaches 87% reward against Stealthy attacker (plot (2,2)), while
plot (2,1) reaches only 68% reward. Though convergences are slower against
Aggressive attacker, the agent reaches more than 80% rewards within 110
episodes in all scenarios.

Figure 3: CDF of Required Time to reach Attack End State.

Time to End Attack: Fig. 3 shows a Cumulative Distribution Function (CDF) that
describes how long the trained agent takes to remove attacker from all devices
during test settings. For instance, a point (25,75) for plot (2,1) specifies
that the agent stops the attacker propagation within 25 time sequences at 75%
cases. Importantly, the rate of attacker’s reaching to Attack Goal State is
much lower than 1%. The agent terminates attack propagation within 100 time
sequences in all cases except against Naive attacker at topo 2 whose
distribution tail stretches until 175 time-sequences. It stops Aggressive
attackers within 25-27 time sequences, while the stealthy attacker
comparatively persists longer.

Figure 4: Mean Reward Comparison between approaches with and without Pre
Execution Satisfiability.

Reward Comparison: Fig. 4 shows the benefit of Pre-sat module for topo 1 and
2, where rewards are normalized by the incentive ($I_{w}$) of attack ending.
The agent with Pre-sat always achieves more rewards, which is maximum (70%)
against Stealthy attacker and minimum (17%) against Naive attacker at topo 2.
Interestingly, though the agent terminates Aggressive attacker faster (at Fig.
3), it executes comparatively expensive actions to defend them.

##  5 Conclusion and Future Directions

Optimizing defense policies dynamically is a challenging tasks due to
uncertainties of environment, strategical and adaptive attacks, and various
safety and security requirements. In this paper, we present an architecture of
Autonomous Defense Agent that optimizes defense planning at real-time using
model-free Reinforcement Learning, while guaranteeing satisfaction of all
requirements using SMT-based constraints satisfiability verification.
Moreover, our agent reasons over environmental observations to deduce new
requirements and learn defense consequences. Our evaluation shows that our
trained agent can defeat diversified attack strategies efficiently without
requiring prior deep knowledge. Our approach is flexible to incorporate new
and modified requirements easily into decision-making, and offers better
scalability for real-time defense optimization in a constrained stochastic
environment with dynamic or uncertain properties.

This architecture creates many interesting future research directions. First,
our agent now learns new requirements based on rewards, but it will be
interesting to find out how automated approaches can be developed to learn new
requirements from network symptoms (e.g., logs, packets traces, and others).
Besides, it is important to understand how much confidence the agent should at
least have before introducing any new requirement. Second, defense payoffs may
not always be observed immediately, and feedback such as user-complains may
arrive several days later. We plan to investigate approaches to integrate the
likelihood of such delayed feedback efficiently into policy optimization.
Third, we would like to assess the scalability of the agent for higher
dimensions of requirements, state space, and defense space of real-world
applications.

## References

  * [1] Gym.  https://gym.openai.com/.
  * [2] Stable Baseline.  https://stable-baselines.readthedocs.io.
  * [3] Clark Barrett and Cesare Tinelli.  Satisfiability modulo theories.  In Handbook of Model Checking, pages 305–343. Springer, 2018.
  * [4] Cuong T Do et al.  Game theory for cyber security and privacy.  ACM Computing Surveys (CSUR), 50(2):1–37, 2017.
  * [5] Panfili et al.  A game-theoretical approach to cyber-security of critical infrastructures based on multi-agent reinforcement learning.  In 2018 26th Mediterranean Conference on Control and Automation (MED), pages 460–465. IEEE, 2018.
  * [6] Zhisheng Hu, Minghui Zhu, and Peng Liu.  Online algorithms for adaptive cyber defense on bayesian attack graphs.  In Proceedings of the 2017 Workshop on moving target defense, pages 99–109, 2017.
  * [7] Thomas Krismayer, Rick Rabiser, and Paul Grünbacher.  A constraint mining approach to support monitoring cyber-physical systems.  In International Conference on Advanced Information Systems Engineering, pages 659–674. Springer, 2019.
  * [8] Erik Miehling, Mohammad Rasouli, and Demosthenis Teneketzis.  A pomdp approach to the dynamic defense of large-scale cyber networks.  IEEE Transactions on Information Forensics and Security, 13(10):2490–2505, 2018.
  * [9] Thanh Thi Nguyen and Vijay Janapa Reddi.  Deep reinforcement learning for cyber security.  arXiv preprint arXiv:1906.05799, 2019.
  * [10] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347, 2017.
  * [11] Richard S Sutton and Andrew G Barto.  Reinforcement learning: An introduction.
  * [12] Wenye Wang and Zhuo Lu.  Cyber security in the smart grid: Survey and challenges.  Computer networks, 57(5):1344–1371, 2013.
  * [13] Kaiming Xiao et al.  Dynamic defense strategy against stealth malware propagation in cyber-physical systems.  In IEEE INFOCOM 2018-IEEE Conference on Computer Communications, pages 1790–1798. IEEE, 2018.