Deep Configuration Performance Learning: A Systematic Survey and Taxonomy

Jingzhi Gong

Loughborough University

Tao Chen

University of Birmingham

Performance is arguably the most crucial attribute that reflects the quality of a configurable software system. However, given the increasing scale and complexity of modern software, modeling and predicting how various configurations can impact performance becomes one of the major challenges in software maintenance. As such, performance is often modeled without having a thorough knowledge of the software system, but relying mainly on data, which fits precisely with the purpose of deep learning. In this paper, we conduct a comprehensive review exclusively on the topic of deep learning for performance learning of configurable software, covering 948 searched papers spanning six indexing services, based on which 85 primary papers were extracted and analyzed. Our results summarize the key topics and statistics on how the configuration data is prepared; how the deep configuration performance learning model is built; how the model is evaluated and how they are exploited in different tasks related to software configuration. We also identify the good practice and the potentially problematic phenomena from the studies surveyed, together with insights on future opportunities for the field. To promote open science, all the raw results of this survey can be accessed at our repository: <https://github.com/ideas-labo/DCPL-SLR>.

§ INTRODUCTION

Configuration is pervasive in software systems, ranging from enterprise web applications to robotic software. The intention thereof is simple: by allowing software systems to be flexibly configured in different ways, it enables them much better applicability over a wide range of domains and a better ability to cope with varying performance requirements <cit.>, such as latency, throughput, and energy consumption. For example, a popular web server Tomcat can adjust different configuration options (e.g., ), the values of which are likely to profoundly influence its throughput. Nevertheless, excessive configurability comes with a cost.  <cit.> have discovered that over 59% of the performance bugs nowadays are due to inappropriate configurations. On the other extreme, software engineers find it generally difficult to adjust the configuration options in order to adapt the performance, therefore most of the options are often ignored, leaving the potential for performance boost untapped <cit.>. The key cause behind the aforementioned issues is the difficulty of knowing how configurations impact the performance beforehand since software systems are complex in nature. But what if there is a model that can establish the correlation between configuration and performance?

Configuration performance modeling—a highly active research field over the last decade—has been becoming an important topic in the software performance research landscape. The goal therein is exactly to build a model that takes a configuration as input and predicts the likely performance before we deploy that configuration. This can create many advancements in the related domains. For example, in software performance testing, one can easily identify the key configuration options that would likely cause a performance bottleneck by investigating a model; in configuration tuning, it is straightforward to simply evaluate and compare different configurations on the model, instead of having to deploy and run the configuration on the actual systems, leading to a dramatic reduction of cost.

However, building an accurate performance model for the configuration of software systems is challenging. Classic performance models have been relying on analytical methods <cit.>, but soon they become ineffective due primarily to the soaring complexity of modern software systems. In particular, there are two key reasons which prevent the success of analytical methods: (i) analytical models often work on a limited set of configurations options <cit.>, but the number of configurations options and the complexity continues to increase. For example, Hadoop has only 17 configurations options in 2006, but it was increased by 9$\times$ more to 173 at 2013 <cit.>; similarly, MySQL has 461 configuration options at 2014, in which around 50% of them are of complex types. (ii) Their effectiveness is highly dependent on assumptions about the internal structure and the environment of the software being modeled. However, many modern scenarios, such as cloud-based systems, virtualized and multi-tenant software, intentionally hide such information to promote ease of use, which further reduces the reliability of the analytical methods <cit.>.

As an alternative, machine learning-based configuration performance models have been explored over the past decade, e.g., liner regression <cit.>, decision tree <cit.>, and random forest <cit.>, which work on arbitrary types of configuration options and do not rely on heavy human intervention <cit.>. Unlike analytical methods, machine learning is data-driven since it seeks to learn the patterns from the configuration data, hence generalizing the correlation between configuration and performance. This problem, namely configuration performance learning, has been gaining momentum in recent years <cit.>.

Among others, a particular type of data-driven configuration performance learning relies on deep learning, i.e., those that make use of deep neural networks <cit.>. Compared with statistical machine learning, deep learning often exhibits several advantages:

* Representation learning: They are capable of extracting the underlying representations from the configuration options even without careful feature engineering, which allows them to learn directly from raw data, enabling end-to-end learning.

* State-of-the-art accuracy: They consist of multiple layers of interconnected neurons, enabling them to capture the nonlinear and complex relationships between the configuration options and performance.

* Generalizability: Deep learning models can leverage pre-trained knowledge from related tasks and quickly adapt to the target task through fine-tuning, hence they often have better generalizability.

Indeed, recent studies have demonstrated the benefits of deep learning for modeling configuration performance. For example,   <cit.> propose , a deep neural network model combined with $L_1$ regularization to address the sparse performance functions, and  <cit.> invent a hierarchical interaction neural network model called  that achieves state-of-the-art accuracy. Yet, despite the importance of such research direction, to the best of our knowledge, there has been little work on a systematic survey that covers the full spectrum of deep configuration performance learning. The current reviews related to this topic mainly focus on either general machine learning models <cit.>, or deep learning in the general context of software engineering <cit.>. Undoubtedly, systematically reviewing state-of-the-art studies on this particular research field can provide vast benefits, including summarizing the common categories, revisiting the important concepts, and, more importantly, discussing novel perspectives on the good practices and “bad smells[In software engineering, the bad smell is a metaphor that denotes the symptoms of code/software that can lead to a deeper problem. In our context, it refers to problematic practices that could lead to serious threats to validity and/or to the sustainability of the research field.]” of the field, and providing insights for future opportunities.

Cumulative number of primary studies on deep configuration performance learning models.

To bridge such a gap, in this paper, we conduct a systematic literature review that covers 948 papers from six online repositories and 52 venues, published between 2013 and 2023, based on which we extract 85 prominent studies for data extraction and analysis. The results confirm that the significance of deep learning for configuration performance and its associated challenges has led to a notable increase in research efforts in this field. As shown in Figure <ref>, clearly, there is a growing trend of papers in this field. In particular, 75% of the work has been published since 2019, indicating the emergence of this new paradigm for configuration performance learning.

In particular, we extensively examine details in all stages of the pipeline for deep configuration performance learning. The major contributions of this survey include:

* A taxonomy that categorizes the techniques used and key concerns in deep configuration performance learning with up to 13 findings of the trends.

* Comprehensive summaries of the key approaches used in the deep learning pipeline for configuration performance, including preparation, modeling, evaluation, and application, together with discussions on their benefits and shortcomings.

* Articulation on the good practices and bad smells observed from the findings.

* Gaps identified from existing studies, offering insights into the future opportunities for this particular thread of research.

The rest of the paper is organized as follows. Section <ref> presents the background information about deep learning for performance modeling and discusses related work. Section <ref> presents our review methodology. Section <ref>, <ref>, <ref>, and <ref> summarizes the taxonomy and results of the research questions. Sections <ref>, <ref>, and <ref> discuss the good practices and bad smells discovered, future opportunities of the field, and threats to validity, respectively. Finally, Section <ref> concludes the paper.

§ BACKGROUND AND RELATED WORK

In this section, we provide a brief overview of the configurable software systems and the deep learning pipeline. We also differentiate this work and other relevant surveys.

§.§ Configurations in Software Systems

To meet the performance requirements, a configurable software system permits possible configuration options to be adjusted at design time or at runtime. For example, around one-third of the configuration options of MySQL, a popular configurable database system, can be changed at runtime, such as ; the remaining ones, e.g., , need to be fixed a priori to the deployment[<https://dev.mysql.com/doc/refman/8.0/en/server-system-variable-reference.html>]. Table <ref> presents a dataset for VP8, which includes $m$ configurations, each associated with a measured performance metric (runtime).

It has been widely acknowledged that the configuration options have great impacts on the software performance <cit.>. Indeed, inappropriate configurations often cause serious performance bugs, which is the key reason why the users became frustrated enough to (threaten to) switch to another product <cit.>. At the same time, simply using default configurations does little help, for instance,  <cit.> show that the default settings on Hadoop can actually result in the worst possible performance. In this regard, a fundamental question to ask is: given a configuration, what is the performance of the software systems?.

Indeed, one solution is to directly profile the software system for all possible configurations when needed. This, however, is impractical, because (i) the number of possible configurations may be too high. For example, MySQL has more than millions of possible configurations. (ii) Even when such a number is small, the profiling of a single set of configurations can be too expensive.  <cit.> report that it could take weeks of running time to benchmark and profile even a simple system. All these issues have called for a computational model, which is cheap to evaluate yet accurate, that captures the correlation between configuration options to a performance attribute.

§.§ Problem Formulation of Deep Configuration Performance Learning

Unlike traditional analytical models, data-driven modeling such as deep learning takes a black-box manner that relies on observing a small amount of the software’s actual behaviors under certain conditions, in order to infer a statistical model to predict the performance without heavy human intervention <cit.>. This is also what enables it to overcome the issues of heavy assumptions on the configuration options and the system's characteristics in analytical methods <cit.>.

Without loss of generality, deep configuration performance learning seeks to build a function $f$ such that:

\begin{equation}
y = f(\mathbf{\overline{x}})
\end{equation}

whereby $y$ is the performance attribute that is of concern; $\mathbf{\overline{x}}$ is a configuration consists of the values for $n$ configuration options, i.e., $\mathbf{\overline{x}}=\{x_1,x_2,...,x_n\}$. Taking the simplest fully connected deep neural network as an example, $f$ is represented as multiple layers of interconnected neurons, where neurons are activated as:

\begin{equation}
a_j^{l+1}=\sigma(\sum_i a_i^l w_{ij}^{l,l+1} + b_j^{l+1})
\end{equation}

where $\sum$ runs over all the lower layer neurons that
are connected to neuron $j$. $i$ is the activation of a
neuron $i$ in the previous layer, and where $a_i^l w_{ij}^{l,l+1}$ is the contribution of neuron $i$ at layer $l$ to the activation of the neuron $j$ at layer $l + 1$. The function $\sigma$ is a nonlinear monotonously increasing activation function, e.g., a sigmoid function; $w_{ij}^{l,l+1}$ is the weight and $b_j^{l+1}$ is the bias term.

To build function $f$, the training in deep learning aims to find the set of weights for different neurons from all the layers such that a loss function can be minimized. For example, the mean squared error below is one possible loss function since configuration performance learning is essentially a regression:

\begin{equation}
\mathbfcal{L}(\theta) =  {1 \over m} \sum^m_{i=1}{(f(\overline{x}_i) - y_i')^2}
\end{equation}

$f(\overline{x}_i)$ and $y_i'$ denote the predicted and actual performance values for the $i$th sample across $m$ data points in the training dataset.

§.§ Related Surveys

There are a few related surveys in the domain of configuration performance modeling. Among others,  <cit.> conduct a review of configuration performance prediction using Queuing Network, and  <cit.> review how performance model can be used to help software development.  <cit.> emphasize the significance of configuration performance modeling in SE tasks and calls for further research to enhance the predictive capabilities of performance models, and  <cit.> seek to understand to what extent are sampling strategies sensitive to configuration performance prediction of configurable systems. A recent work <cit.> classifies software configuration performance learning studies into 6 categories, provides a mapping of them within these categories, and highlights potential weaknesses of the literature. Further, surveys on configuration performance modeling for specific categories of configurable software systems exist, for example, there are reviews on the performance models for distributed systems <cit.> and  <cit.> survey the configuration performance prediction methods for parallel applications published between 2005 and 2020. Similarly,  <cit.> review 34 studies on configuration performance modeling for multi-core systems, and configuration performance learning and tuning techniques for mobile application systems are reviewed by  <cit.>. However, the above work does not focus on how deep learning can be used in the performance model building for configurable software in general.

On the other hand, several surveys have been conducted on the application of deep learning in software engineering. For example,  <cit.> investigates the deep learning-related techniques for software engineering tasks, including the deep learning models, data preprocessing methods, SE tasks, and model optimization methods, while a review on deep learning is also conducted by  <cit.>, which provides a research roadmap and guidelines for future exploration in the context of software engineering.  <cit.> investigates the reproducibility and applicability of deep learning studies in SE, and observes that a significant number of them have overlooked this challenge. A novel work <cit.> explores the use of machine/deep learning techniques in various software engineering tasks and the challenges and differences between machine and deep learning. Yet, these studies do not focus on a specific SE task like configuration performance learning but rather provide a broad review of software engineering with a restricted depth on each topic.

The most similar prior study to our work is probably a review by  <cit.> that explores the application objectives, sampling, learning and measuring methods, and evaluation related to machine learning models for configuration performance modeling. However, we focus on deep learning models, which is a specific type of machine learning, and cover a wider range of techniques such as preprocessing, hyperparameter tuning, and sparsity handling methods. Besides, they did not discuss and justify the good practices and bad smells in the field.

In summary, our work differs from the above in the following aspects:

* We focus explicitly on deep learning-based configuration performance modeling for configurable software in general and capture the latest trends in the past 10 years.

* We review aspects that have not been summarized before, e.g., the use of data preprocessing, encoding scheme, sparsity handling, hyper-parameter tuning, statistical test and effect size test, runtime environments, and public artifact.

* We disclose and justify the positive and potentially problematic practices, which have not been revealed previously, with a summary of knowledge gaps to shed light on future works in the field.

§ RESEARCH METHODOLOGY

To bridge the aforementioned gap, we conduct a systematic literature review that covers the papers published between 2013 and 2023. This period was chosen because we seek to concentrate on the latest trends, mitigating the noises from the old and disappeared practices in the field. In particular, our review methodology follows the best practice of systematic literature review for software engineering <cit.>, as shown in Figure <ref>.

Overview of the literature review protocol.

§.§ Automatic Search

As can be seen, in Stage 1 we conducted an automatic search over six highly influential indexing services, i.e., ACM Library, IEEE Xplore, Google Scholar, ScienceDirect, SpringerLink, and Wiley Online, as they have been used in a number of systematic literature reviews in the field of software engineering, software configuration, and performance modeling <cit.>.

The consideration behind formulating the search string was multi-faceted, as the aim was to identify primary studies that:

* are relevant to software systems research;

* incorporate deep learning or neural network as part of the model;

* utilize configurations as the input for deep configuration performance models;

* apply the deep configuration performance models in some practical domains, e.g., predicting the performance of software or systems.

To that end, different combinations of keywords in these four areas were explored to determine the optimal search string. This ensures that the search results exclude redundant and irrelevant works as much as possible, retrieving papers that align with the research focus on the intersection of software configuration performance prediction and deep learning models.

Specifically, we compile the search string as below:

raster columns=1,
raster rows=1
[myhbox=Search String]   (“software" OR “system") AND “configuration" AND (“performance prediction" OR “performance modeling" OR “performance learning" OR “configuration tuning") AND (“deep learning" OR “neural network")

For all indexing services, the search resulted in 948 studies, accounting for duplicates and excluding non-English documents. Among these, a total of 414 studies were obtained from Google Scholar, 172 studies from SpringerLink, 114 studies from Wiley Online, 98 studies from ACM Library, 94 studies from ScienceDirect, and 56 studies from IEEE Xplore.

§.§ Removing Duplication and Filtering Irrelevant Studies

Next, in Stage 2, we aim to ensure that only unique and highly relevant studies will be considered for further analysis in our review. To achieve this, we first filter out any duplicate studies by carefully examining the titles of the identified papers. This ensures that each study included in our review is distinct and contributes unique insights to the body of knowledge. Subsequently, we conducted a brief evaluation to eliminate any documents that were clearly irrelevant to configuration performance learning. For instance, we excluded studies focused on human and student performance, which is solely relevant to educational outcomes. Similarly, we disregarded papers centered on the performance of, e.g., physical systems, fuel systems, and football systems, as they are not directly aligned with the scope of our investigation.

As a result of these rigorous filtering procedures, we identified 193 highly relevant candidate studies.

§.§ Applying Inclusion and Exclusion Criteria

Through a detailed review of the candidate studies, Stage 3 focuses on applying various criteria to further extract a set of more representative works.

Firstly, we design a set of inclusion criteria to ensure that studies selected for detailed analysis align closely with the core themes of configuration performance learning. Specifically, in our methodology, a study is temporarily selected as a primary study if it meets all of the following inclusion criteria:

* The paper presents a configuration performance modeling approach using deep learning algorithm(s).

* The paper explicitly states how the model built can be used, e.g., for predicting or analyzing the performance of a software system.

* The paper has at least one section that explicitly specifies the learning algorithm(s) used.

* The paper contains quantitative experiments in the evaluation with details about how the results were obtained.

The above is designed to maintain the relevance of the review and reduce the efforts required in the subsequent analysis. For example, the studies should ensure that they provide explicit details about the learning algorithms employed, as this is important for the understanding, analysis, and application of the model. Besides, it is essential for the inclusion of quantitative experiments to ensure a robust evaluation process, enhancing the depth and reliability of the selected studies.

Next, we applied the following exclusion criteria on the previously included study, which would be removed if it meets any:

* The paper is not software or system engineering-related.

* The paper is not published in a peer-reviewed public venue.

* The paper is a survey, review, tutorial, case study, or an empirical type of work.

* The paper is a short and work-in-progress work, i.e., shorter than 8 double-column or 15 single-column pages.

By excluding papers unrelated to software or systems, we ensure that the selected studies directly relate to the research focus of this survey. The requirement for peer-reviewed publication enhances the credibility of the included studies, while the exclusion of certain types of works (survey, review, tutorial, case study, or empirical) helps to filter out literature that may not align with our objective of investigating performance prediction using deep learning. Lastly, the limitation on the length of papers ensures that the selected studies possess sufficient depth and detail for a comprehensive analysis.

Finally, we identified 85 primary studies for detailed data collection[Raw data of all primary studies can be found at our repository: <https://github.com/ideas-labo/DCPL-SLR>.].

§.§ Collecting and Extracting Data

Pipeline for deep configuration performance learning.

§.§.§ Research Questions

In the last stage, we derive the research questions (RQs) of this survey following the deep learning pipeline in Figure <ref>. Specifically, we formalize the workflow of deep learning for learning software configurations and performance into four key phases. The first phase is to process the raw configuration data collected and make preparations for their training with the deep learning models. To understand this, we ask the first research question:

RQ1: How to prepare raw configuration data for learning?

Next, various deep learning algorithms might have been utilized to learn the preprocessed configuration data, handling different challenges in configuration performance prediction, such as sparsity, network structure, and network optimization. To summarize this, we seek to understand:

RQ2: How can the deep configuration performance model be built?

A crucial part of deep configuration performance learning is how to evaluate the trained deep configuration performance models, with a specific evaluation method, a set of testing samples, and a particular accuracy metric, where this process might be repeated for multiple subject systems. Therefore, the next question is:

RQ3: How are the deep configuration performance models evaluated?

Finally, the deep learning models need to be exploited in practical configuration scenarios, adapt to different environmental conditions, and enable researchers to replicate or reproduce the results. To that end, the last research question aims to understand:

RQ4: How to exploit the deep configuration performance model?

§.§.§ Data Items

After formulating the research questions, we have developed a comprehensive list of data items that we aim to collect during the review process. The summary of these data items is presented in Table <ref>, which includes a total of 22 data items, each serving a specific purpose related to the corresponding research question. In this section, we explain the design rationales behind them and clarify the procedure for extracting and classifying the data from each item.

In the initial stage, we employ data items $I_1$ to $I_4$ to gather meta-information of the reviewed studies, such as the title, author, and publication information. This meta-information enables us to accurately reference and organize the reviewed studies, thereby establishing a robust foundation for further analysis and interpretation.

To address RQ1, we seek to examine the key processes in preparing configuration data, where it is common to pre-process the raw configuration data, encode the data into the most appropriate form, and select a subset of informative samples for model training. To capture this information, we design data items $I_5$ to $I_7$ to extract the corresponding data preparation methods. For instance, by using $I_5$ we extract the pre-processing methods used in each primary study according to the fundamental types of methods, e.g., min-max scaling and z-score, which are all used to normalize the scale of the configuration options, can be classified into the same category.

For answering RQ2, we first review the methods used to deal with data sparsity in configuration data via $I_8$. Next, $I_9$ to $I_{13}$ are for understanding how the deep learning models are chosen and trained according to their most general type. For example, on $I_{11}$, hyperparameter tuning methods can be set into three categories:

* using the default hyperparameter settings without any tuning;

* tuning the hyperparameters via manual effort and domain knowledge;

* or relying on automated heuristic methods in the tuning.

To examine the evaluation-related techniques in RQ3, we do not only examine the evaluation procedure ($I_{14}$) and metrics ($I_{15}$) but also the methods of statistical validation that ensure the statistical difference between the comparisons ($I_{16}$ and $I_{17}$). The number and domain of subject systems used in the evaluation, which may influence the generalizability of the conclusions, are also examined using $I_{18}$ and $I_{19}$, respectively.

Lastly, for RQ4, we identify three data items to examine the exploitation information regarding the deep configuration performance models. In particular, $I_{20}$ is employed to understand the application domains of the performance models, $I_{21}$ summarizes the environmental conditions considered, as they significantly decide the generalizability and robustness of the deep configuration performance models, and $I_{22}$ is applied to explore the data for replication and reproduction of the proposed models.

§.§.§ Data Collection

The data collection process aims to collect detailed information from the primary works with the data items presented in Table <ref>, and the collection protocol is the same as commonly used by the literature reviews in Software Engineering <cit.>. Specifically, the protocol involves all the authors with three iterations.

* In the first iteration, each of the authors conducted initial data collection independently, read carefully throughout the 85 primary studies, extracted the data according to each data item, summarized the data into a table of 22 columns and 85 rows, and conducted preliminary classification. Notably, there were situations where certain data items could not be found (e.g., the encoding schemes were often ignored); the data items were not clearly stated (e.g., the sampling methods for training/testing data for evaluation were sometimes vague, with only abstract descriptive words like “case study” or “real dataset”); or the data needed to be conjectured from the contexts (e.g., the domains of subject software systems were often not specified and extra searches were needed to be performed for data collection). In these cases, the corresponding data items were marked for further investigation in the next iteration.

* In the second iteration, the authors reviewed and cross-checked each other's data summary tables, ultimately integrating them into a unified table. The unclear data items that were marked in the previous iteration were rechecked and we arranged meetings to reach a common agreement for each table cell where the collected data from the authors were different. This is achieved through author discussions; further investigations from the existing literature; or consultation with external researchers/authors of the reviewed studies. For instance, the authors discussed and decided to add a category of “unknown” for encoding schemes since a large number of studies omitted the justification of the choice of their encoding method, and investigations on search engines like Google were conducted to collect the domain of subject systems.

* Finally, the goals for the third iteration are to summarize the statistics of the integrated table from the second iteration; count the number of studies for each technique associated with every data item; and aggregate similar techniques into broader categories. For example, in the domain of subject systems, it was found that the number of studies that apply configuration performance models for video encoding software and image processing software are four and three, respectively, and they are combined as the category “multimedia processing” as these software systems share much more similar characteristics comparing with others.

This has finally led to the data for this work, which was discussed among all authors to make comprehensive classifications.

A taxonomy of deep configuration performance learning. The number in the bracket denotes the corresponding number of studies.

§.§ The Taxonomy

With the collected statistics using the RQs, data items, and data collection protocol mentioned in the previous sections, we formalize into a taxonomy of deep configuration performance learning as shown in Figure <ref>. As can be seen, we found a variety of categories for the RQs and their data items. Some of these are specific algorithms (e.g., for the deep learning models used) while some others are of more general types (e.g., for the preprocessing methods). We will present detailed data statistics and introduce each category of techniques for deep configuration performance modeling in what follows.

§ RQ1: HOW TO PREPARE RAW CONFIGURATION DATA FOR LEARNING?

To improve the accuracy and reliability of the deep configuration performance model, data preparation is necessary <cit.>. In this section, we particularly review the preprocessing, encoding, and sampling when preprocessing configuration data that are commonly used by the approaches of deep configuration learning in the community.

§.§ Preprocessing Configuration Data

Data preprocessing methods play a fundamental role in preparing data for deep learning problems <cit.>, and understanding the preprocessing techniques employed in the context of configuration performance learning is crucial for researchers to gain insights and enhance the quality and reliability of the data.

In general, data preprocessing is the process of transforming raw data into a computable and noise-free format. This can be crucial for deep configuration performance learning since the configuration options might be of diverse scale, nature, and types. As an example, for the video encoding software VP8, there are binary configuration options like  (0 or 1), categorical options like  (no/low/medium/high sharpening), and numerical options such as  (from 0 to 1000).

Table <ref> summarizes different preprocessing methods identified from our review. As can be seen, while various methods have been employed, a significant number of studies (44 out of 85) opt to use the raw configuration performance dataset without any preprocessing. Although this way may save effort and costs in the short term, it raises extra requirements on the quality of the datasets to ensure reliable results.

In contrast, normalization techniques have been applied in 32 studies, which are fundamental in data preprocessing to ensure that the scales of configuration options are brought to a standardized range, thereby preventing any particular options from dominating the learning due to its larger values <cit.>. Several widely used normalization techniques for deep configuration performance learning include: Min-max scaling <cit.>, which transforms numerical options to a specific range, typically between 0 and 1, by subtracting the minimum value and dividing by the range of the option. Z-score <cit.> transforms numerical options by subtracting the mean and dividing by the standard deviation of the option, resulting in a distribution with a mean of 0 and a standard deviation of 1, and allowing for easier comparison and interpretation across different options. Although normalization techniques have gained popularity and offer several advantages, it is crucial to acknowledge their limitations. For example, scaling the data might disrupt the original distribution and decrease the performance of certain deep learning algorithms that depend on the inherent characteristics of the original data. Additionally, normalization techniques like min-max scaling can be sensitive to outliers, as extremely big values of the configuration options can significantly impact the range of the scaled values, potentially compressing the majority of the data into a narrow range. Indeed, the  option of the stencil-grid solver HSMGP can range from 64 to 4096, while the largest value of the remaining configuration options is 6.

Methods for dimensionality reduction have been used in nine studies, which help to simplify the data representation, remove redundant or irrelevant options, and overcome the curse of dimensionality, thereby improving computational efficiency and visualization of the configuration data. For example, Principal Component Analysis (PCA) <cit.> transforms the original options into a new set of uncorrelated variables called principal components, which are linear combinations of the original options and are ordered by the amount of variance they explain. Among others, embedding <cit.> involves mapping the original complex data, including the tabular configuration data, to a lower-dimensional space, where each point retains its relative position and proximity to other points as much as possible. Indeed, <cit.> also state that it is crucial to embed the extreme high-dimensional configuration space into low-dimension to better model the interactions between the options and performance.  However, these methods may also require additional resources to find the best hyperparameters for the best reduction outcomes. They also could make the model difficult to interpret since the original option space is destroyed.

Further, five studies have focused on anomaly detection techniques, aiming at identifying data absences, errors, and outliers within a configuration dataset. For example, <cit.> filter out samples where the runtime is zero in the SPEC CPU 2017 dataset, and <cit.> opt to remove configurations with performance values beyond the $99$th percentile. Examples include: Smoothing <cit.>, which aims at capturing the underlying trends, patterns, or regularities in the configuration data while suppressing or filtering out the noise, and Isolation Forest <cit.> algorithm that constructs random decision trees to measure the average number of steps required to isolate an instance, allowing it to identify anomalies as instances of configuration that require fewer steps for isolation. Their limitations may include the potential loss of important details.

Another four studies have explored featurization approaches, where raw data unsuitable for machine/deep learning are transformed into meaningful options that can be used for modeling. For instance, discretization, as highlighted by <cit.>, involves converting continuous options into discrete and representative categories, to save efforts on exploring the huge configuration space. Particularly, the authors discretize the reach rate of a Content Delivery Network by calculating its fifth power, such that the reach rate is projected into several discrete intervals, and each interval is assigned a unique value. Yet, these methods may result in information loss, especially in situations where the software system is highly complex, and they often rely heavily on domain experts.

Besides, imbalance processing approaches have been applied in three studies, which are crucial in dealing with imbalanced performance datasets with regard to the configurations, where the number of samples for different configuration values is significantly uneven. Indeed, in the configuration dataset of the widely used software Apache <cit.>, there are 128 samples with value 1 for the configuration option , while only 64 samples with value 0, resulting in imbalance with this option. To handle this, oversampling methods could be used, which increase the number of instances from the minority class to balance the dataset. For example, <cit.> employ SMOTE (Synthetic Minority Over-sampling Technique), which enhances the minority group by interpolating between neighboring instances, to balance the imbalanced divisions of samples within their “divide-and-learn” framework.

Lastly, two studies remove multicollinearity by addressing highly correlated options, e.g., Kendall’s rank correlation  <cit.>, which quantifies the strength and direction of the association between two ranked options, is used to remove highly correlated options and ensure that the predictors are independent or have a minimal correlation, which eventually reduces the cost of building deep configuration performance model.

Finding : The default datasets, without any preprocessing, are used by most studies (44 out of
85). Among the rest preprocessing methods, normalization is the most popular in handling configuration data, i.e., in 32 studies.

§.§ Encoding Configuration Options

The procedure immediately followed by data preprocessing is encoding, which is concerned with converting the configuration data into a more appropriate format for learning. Indeed, as discovered by <cit.>, the impacts of encoding schemes are non-trivial, thereby the encoding scheme can considerably influence the learning outcome.

Figure <ref> provides a summary of the encoding schemes surveyed. Among others, label encoding <cit.>, emerges as one of the most popular schemes, employed by 52 primary studies. Label encoding assigns a unique numerical label to each value of the option. As such, label encoding preserves the ordinal relationship between categories but does not introduce additional dimensions like one-hot encoding. Although label encoding can simplify the representations of configuration options, it also introduces a random numerical order for those categorical options, e.g., for the highly configurable database system MongoDB, its configuration option  has a set of configurable values $(str\_l1,$ $str\_l2,str\_l3)$, and label encoding will convert them into numeric values of $(0,1,2)$ <cit.>. This might potentially disrupt the inherent relationships between the configuration options and lead to worse predictions.

Similarly, scaled label encoding <cit.>, which is a common variation of label encoding, is chosen by 22 primary studies in our survey. Different from classic label encoding, the encoded configuration options of software systems are normalized into a specific range, e.g., between 0 and 1, to overcome the limitations of label encoding. For instance, the  option of VP8 will be label encoded from 0 to 1000, while the binary option can be only 0 and 1. This difference in scale may result in significant sparsity for performance models, which can be effectively handled by scaled label encoding.

In contrast, one-hot encoding <cit.>, employed in 14 studies, transforms discrete options into a vector of binary values, where each element is represented as a binary form. For instance, for the video encoding system VP8, options like , which has categorical values no/low/medium/high sharpening, will be one-hot encoded into four binary variables. In this way, it enables deep learning models to appropriately interpret and capture relationships between different values. While one-hot encoding has been proven to be beneficial for the accuracy of performance models, it also leads to the highest training overheads <cit.>.

We find that two studies employ graph encoding that leverages graph representations, such as the computational graphs of neural networks <cit.>, to represent configurations. For example, <cit.> use graphs to represent the computational stages and operations of a deep neural network to better capture the internal interactions while modeling the runtime of the deep learning pipeline. Graph encoding can lead to high-dimensional representations when dealing with a large set of options, potentially increasing the complexity and computational requirements.

Notably, a substantial portion of the work (i.e., 65 studies) opt to not explicitly specify encoding schemes used. Therefore, we have to inspect their codes or contact the authors to collect the data.

Finding : Label encoding is applied to convert configuration values for learning in 52 studies, being the most common one, followed by scaled label encoding and one-hot encoding, used in 22 and 14 primary studies, respectively. Noteworthy, the majority of studies, 65 in particular, do not explicitly state their encoding schemes.

Distribution of the encoding schemes (one study might use items from multiple categories).

Distribution of the sampling strategies (one study might use items from multiple categories).

§.§ Sampling Configurations for Learning

Sampling, which is typically performed to guide the raw configuration data collection, is also a critical step in data preparation, as the quality of the collected configuration samples determines the prediction accuracy of the deep learning models <cit.>.

Figure <ref> shows the categories of the sampling strategies used in the reviewed studies. We can first notice that an overwhelming majority of the studies surveyed, specifically 66 out of 85, employ random sampling as a prevalent data sampling technique in their research. Random sampling involves selecting data instances from the original configuration dataset in an unbiased and random manner. The widespread adoption of random sampling highlights its efficiency and effectiveness in creating balanced and diverse samples for training <cit.>. However, random sampling may not efficiently capture the less frequent values of options, and these values may be underrepresented in the selected set, leading to bad generalizability of the deep learning models. For instance, for the web server software Apache, the value of the option  is 1 in 128 configurations, while the value 0 only appears 64 times in this option, therefore the training samples collected by random sampling will have a higher possibility to miss the value 0, especially when the training sample size is small, leading to bad accuracy when the model needs to predict new configurations with 0 for .

In contrast, 14 of the primary studies follow a common practice of utilizing the default sets of training and testing samples that are pre-defined from the benchmark dataset <cit.>,

which allows for a direct comparison of their models' performance against the existing results. For instance, <cit.> only consider the training and testing workloads from the SPEC CPU 2017 benchmark. While they provide a standardized reference for representing specific scenarios or workloads, their applicability to real-world or diverse environments may be limited.

Two studies apply active learning to obtain the samples <cit.>, which seek to select the most informative samples to be measured for training, thereby accelerating the learning process and reducing the costs of data collection. Yet, active learning methods may put high demand on uncertainty estimates or heuristics to select samples.

For the strategies that are part of the minority, each of the following is adopted by one primary study: Stratified sampling <cit.> involves dividing the population into homogeneous subgroups based on specific characteristics and then sampling from each subgroup proportionally to its representation in the overall population. It also requires prior knowledge about the population's stratification variables and their distribution. Pair-wise sampling <cit.> seek to sample data that achieve maximum coverage of interaction effects between pairs of options while minimizing the number of samples required to prevent biases. Indeed, when dealing with systems that have a large number of configurations, pair-wise sampling may become impractical due to the sheer number of possible combinations. Fractional factorial design <cit.>, on the other hand, provides a way to select a subset of factor combinations that represent the most important effects. While it allows for investigating a large number of options with a reduced number of samples, its generalizability is limited by the assumption of linearity and consistency of the configurations. Lastly, Model-based sampling <cit.> builds a surrogate model capable of estimating uncertainty to effectively guide the search for selecting optimal training configurations. As a result, the accuracy of the surrogate model is critical.

Finding : Random sampling is the most widely employed strategy for selecting configuration samples as observed in 66 studies, followed by using the predefined samples from the benchmark as exploited by 14 studies. Other sampling strategies form the minority.

§ RQ2: HOW CAN THE DEEP CONFIGURATION PERFORMANCE MODEL BE BUILT?

A deep learning model can be trained once the data preparation has been completed. To understand the state-of-the-art at this stage for deep configuration performance learning, this section summarizes details on how deep learning can be effectively built to learn configuration performance, offering insights into best practices and potential problems in this field.

§.§ Handling Configuration Sparsity and Preventing Overfitting

As large-scale software systems generate vast amounts of data, sparsity arises when valuable information is scarce in the options or unevenly distributed across the configuration space. As a result, overfitting occurs when deep learning models excessively learn the configuration data used for training, leading to poor generalization on unseen samples <cit.>. For instance, the option  which enables GPU processing for the video codec x264 has a large influence on the runtime, and deep learning models tend to assign a large weight to this option while ignoring the influence of others, leading to overfitted predictions.

Table <ref> presents the taxonomy of mechanisms used to handle sparsity and overfitting. Notably, 32 out of 85 studies have not adopted any specific mechanism to handle sparsity, which could harm the accuracy of deep configuration performance models.

Among the remaining studies, 39 primary research papers have utilized feature selection techniques as a means to deselect the undesired configuration options, hence mitigating sparsity. In particular, feature sparsity is observed when there exists a set of non-influential configuration options, which adds redundant parameters to the configuration performance model. Therefore, by eliminating these options with feature selection, the performance model will be resilient from sparsity. However, it is worth noting that 24 out of the 39 papers simply rely on human effort to select the most important options, which is inefficient and lacks generalizability. Among others, four studies have applied correlation-based feature selection techniques, such as correlation analysis <cit.>, that leverages statistical measures to determine the strength of relationships between options. Four other studies have used tree-based feature selection, which inherently enables analyzing the structure and splits of the trees to determine the most informative options for prediction. For instance, extra trees regression <cit.> involves constructing multiple decision trees with randomized splits to estimate feature importance, and <cit.> utilize the ensemble nature of random forest to rank options based on their contribution to the model's predictive power. Moreover, each of the following feature selection methods is used by 2 primary studies: Filter-based feature selection techniques evaluate each feature independently of the target option and rank them based on predefined criteria. An example is the minimum Redundancy Maximum Relevance Feature Selection (mRMR) <cit.>, which seeks a balance between selecting options that have high relevance to the performance while minimizing redundancy among selected options; Neural Network-based feature selection applies neural network layers to identify relevant features, which have the inherent ability to learn and extract informative options from high-level representations. For example, CNNs <cit.> (Convolutional Neural Network) are commonly used in image-related tasks to extract local patterns and spatial hierarchies, and GNNs <cit.> (Graph Neural Networks) are particularly effective for structured data represented as graphs, which analyze relationships and dependencies among nodes in a graph to capture important patterns and interactions. Wrapper-based feature selection <cit.> uses Recursive Feature Elimination (RFE) to iteratively eliminate options with the least importance, while evaluating the model's performance using cross-validation. In particular, <cit.> select 10 most important options like ,  and  in the SPEC CPU2017 dataset using RFE with four different estimators. Hybrid model also exists as proposed by <cit.>. The core is to combine the outputs of multiple learning algorithms to guide the feature selection process. Finally, metaheuristic feature selection <cit.>, which combines whale optimization algorithm and simulated annealing to construct a search-based option selection algorithm named EMWS, is explored in one study.

Alternatively, 14 studies have employed regularization techniques, which introduce additional constraints or penalties to the model's objective function, aiming to control sparsity, reduce overfitting, and enhance the robustness of the performance models.

Several commonly used examples are: $L_1$ regularization <cit.>, also known as Lasso regularization, adds a penalty term proportional to the absolute value of the model's coefficients, encouraging sparsity in the model by driving some coefficients to zero and reducing the impact of irrelevant features. For instance, after performing $L_1$ regularization on the configuration options of the video encoding software VP8, the coefficients of options like  and  are reduced to zero because they have trivial influence on the performance (runtime), while the coefficient of options like  is changed from $10^{16}$ to $1.9 \times 10^{4}$, significantly reducing the sparsity of configurations.
$L_2$  regularization <cit.>, also called Ridge regularization, adds a penalty term proportional to the squared magnitude of the model's coefficients. It encourages smaller coefficients and can help mitigate multicollinearity issues by reducing the impact of highly correlated options. However, regularization methods come with the trade-off of introducing additional parameters that require tuning to achieve optimal results, which may incur higher costs in terms of time, computational resources, and expertise needed to find the best parameter values <cit.>.

Meanwhile, data processing techniques like dimension extraction have also been leveraged to eliminate unnecessary representations and information from the original dataset in nine studies of deep configuration performance learning. Unlike feature selection techniques, which only focus on mitigating specific options, dimension extraction approaches can achieve simultaneous handling of features, which is crucial when there are interactions or dependencies among features. This is because they work on the overall weight magnitudes, allowing the deep learning models to strike a balance between the negative impacts of the features and their potential interactions with others.  For example, <cit.> embed source codes of configurable software systems into a latent space representation to extract the most informative features and use together for configuration performance modeling, thereby reducing the sparse representations of configuration data.

Dropout techniques are widely used to eliminate insignificant model parameters during the neural network training process in eight prominent studies. Particularly, dropout <cit.> works by randomly deactivating a fraction of neurons during the training phase of a neural network. By doing so, dropout prevents the network from relying too heavily on specific neurons, forcing it to learn more robust and generalized features. In addition, <cit.> applied ensemble pruning of networks to prune the top 30% of the ensemble networks to prevent overfitting. Compared to feature selection and regularization, the advantages of dropout are its simplicity and computational efficiency, which is particularly useful for training large neural networks. Yet, this also causes a lack of interpretability, as dropout does not provide explicit information about which specific options or neurons are important or unimportant.

Furthermore, although a number of studies have focused on addressing feature sparsity, where a small subset of features holds significant influence, there are very few studies that deal with sample sparsity, in which samples exhibit substantial variations in performance, resulting in a non-smooth distribution. For example, <cit.> have shown that by dividing the samples of the video codec VP8 according to the option , there are two clusters of samples with significantly diverse distributions, i.e., the division with  = 0 has runtime between 0 and 10000 ms, while the other ranges from 10000 to 60000. In light of this gap, they proposed a framework based on the idea of “divide-and-learn”, i.e., , which divides the original dataset into distinct divisions based on hidden characteristics and learns a local deep learning model for each division to mitigate sample sparsity <cit.>.

Finding : A considerable number of studies do not explicitly handle the problem of sparsity and overfitting
(32 out of 85). Among the studies that deal with this challenge,
manual feature selection is the most common method (24 studies).

§.§ Choosing Deep Learning Model

It is not hard to anticipate that there will be a variety of models applied to deep configuration performance learning, as shown in Table <ref>. Specifically, the majority of primary studies (61 out of 85) employ a Feedforward Neural Network (FNN). Among these, the most prevalent approach is Multilayer Perceptron (MLP) in 54 studies. MLP is a fundamental neural network consisting of interconnected neurons organized in multiple layers. These neurons apply activation functions to the weighted sum of their inputs, introducing non-linear transformations to the data. <cit.> propose the utilization of regularization techniques in regularized Deep Neural Networks (rDNN), such that additional penalties are introduced during training to remove insignificant features, control feature sparsity of the model and thereby prevent overfitting. <cit.> combines rDNN with the divide-and-learn framework to further address the sample sparsity issues. In addition, Kernel Extreme Learning Machines (KELM) are also used in two studies, aiming at simplifying the training process of neural networks, which randomly initializes the weights between the input and hidden layers, then applies a kernel function to obtain the transformed feature representation, and analytically determines the weights between the hidden and output layers based on the transformed features. A most recent work <cit.> employs a Hierarchical Interaction Neural Network (HINN) that leverages a hierarchical structure for performance learning, where lower-level layers typically capture low-level options, and higher-level layers learn to combine these low-level options into more complex representations. Further, dynamic neural networks and radial basis function neural networks are each utilized in one study.

However, FNNs cannot handle sequential or temporal data effectively in many real-world tasks, such as natural language processing and time series prediction. To mitigate this, Recurrent Neural Networks (RNN) are specifically designed to maintain an internal memory or hidden state that persists across time steps, allowing them to capture information from past inputs. Among our review scope, RNNs have been employed in nine works. For example, Long Short-Term Memory (LSTM) is a dedicated version of the recurrent neural network with internal memory cells, which allows them to selectively retain or forget information based on the input and previous context by using specialized units called gates, and therefore overcome the limitations of traditional RNNs in capturing and remembering long-term dependencies in sequential data. For example, <cit.> employ LSTM to tune the performance of file systems like Ext4, F2FS, and PMFS, where the operations and workloads are dynamically changing over time and it is crucial to apply time-related deep learning models to model the temporal data.

On the other hand, Convolutional Neural Networks (CNN) have been studied in nine research papers to process grid-like structured data, such as images and videos, which can not be handled by FNNs. The hierarchical feature learning capability of the convolutional layers allows CNNs to automatically learn and extract meaningful representations from the input data. Notably, a particular variant of CNNs, known as Residual Networks (ResNets), has been exploited by <cit.>, which incorporate novel architectural designs with residual connections to address the issue of vanishing gradients during training. As an example, <cit.> leverage the ability of CNN that can disclose the hidden and complex correlations among different options to predict the performance of high-performance computing systems like RISC-V.

In addition, Graph Neural Networks (GNN) and its variant DAG-transformer have been used in six studies, which utilize graph convolutions and message-passing techniques to capture complex relationships and dependencies within interconnected entities, e.g., graphs, which are mathematical structures composed of nodes connected by edges. For example, <cit.> utilize GNN to predict the inference latency, energy, and memory footprint of DNN inference, because it can directly process arbitrary DNN TFlite graphs and has good generalizability.

Further, Generative Adversarial Networks (GAN) is an adversarial learning approach that is primarily designed for generative modeling tasks, aiming at learning the underlying distribution of the training data and generating new samples that resemble the training data. It consists of two neural networks: a generator network and a discriminator network, which are trained in an adversarial manner, with the generator trying to produce more realistic samples to deceive the discriminator, and the discriminator trying to become better at distinguishing real from generated samples. For instance, GAN is used by <cit.> to automatically generate configuration samples for different software systems like Kafka, Spark, and MySQL to save performance measurement costs.

Meta-learning and reinforcement learning methods are explored in one study, respectively. Specifically, Model-Agnostic Meta-Learning (MAML) pre-trains meta-model's parameters using the known software environments to facilitate fast adaptation to new environments, allowing for efficient generalization and rapid adaptation. On the other hand, reinforcement learning approaches like Q-learning networks are used to approximate the action-value function (Q-function), which represents the expected cumulative reward for taking a specific action from a given state and following a particular policy. As an example, <cit.> utilizes a Q-network to deal with the dynamic changes of workloads while predicting the configuration performance for multitier web systems like RUBiS.

Distribution of the reasons for choosing deep learning model.

In addition to the statistics of deep learning models, our survey also investigates whether the studies have provided an explanation for their choice of specific learning models, which is captured in Figure <ref>. Among the 85 studies analyzed, it was found that only 38 of them have provided explicit reasons for the utilization of a specific learning model, accounting for approximately 45% of the total studies examined. On the other hand, a significant portion of the studies, 47 in total, do not offer any justification or rationale behind their choice of learning model.

Finding : Various deep learning models have been applied for deep configuration performance learning, within which
the most common one is the multilayer perception (a kind of FNN) from 54 out of 85 studies.
However, it is worth noting that nearly 55% of the primary studies do not provide
explicit justifications for their choice of deep learning model.

§.§ Optimzing Learning Loss

The effective training of deep learning models heavily relies on the choice of optimization methods that reduce the loss of training the activation functions, as the loss determines how the model's parameters are updated during training, while activation functions introduce non-linearity and enable complex representations within the neural network.

Figure <ref> provides data on the optimization methods employed in the examined studies. A notable observation is that: a significant number of studies, 46 in total, omitted the specification of the loss optimization method employed <cit.>.

Despite this omission, Adaptive Moment Estimation (Adam) optimizer emerges as the most prevalent optimization method, being utilized in 24 studies <cit.>,

which is an adaptive optimization algorithm that maintains adaptive learning rates based on the first and second moments of the gradients. Stochastic gradient descent (SGD), which is an iterative optimization algorithm that updates the model parameters by computing gradients on small randomly sampled subsets of the training data, stands as the second most frequently used method, appearing in nine studies <cit.>.

Back-propagation, which involves computing and propagating the gradients of the loss function with respect to the deep learning model's parameters from the output layer to the input layer, is used in three works <cit.>. Resilient backpropagation (Rprop), which individually determines the appropriate step size for each parameter based on the sign of the gradient during training <cit.>, and root mean square propagation (RMSprop), an extension of SGD that adapts the learning rate for each parameter based on the root mean square of recent gradients <cit.>, are both applied in two studies. Besides, all the rest methods are used by one study, i.e., Broyden-Fletcher-Goldfarb-Shanno (BFGS) <cit.>, Limited-memory BFGS <cit.>, and Levenverg-Marquardt <cit.>.  When dealing with the high-dimensional, sparse, and non-linear configuration performance data of highly configurable software, Adam is a promising choice among others as it combines adaptive learning rates and momentum, efficiently handling sparse gradients and noisy data <cit.>.

Optimization methods

Activation functions

Distribution of the loss function optimization techniques (one study might use items from multiple categories).

In Figure <ref>, an overview of the activation functions employed in the surveyed studies is provided. Similar to the data of optimization methods, a significant number of studies, 46 out of 85, did not explicitly report the activation function employed. Despite this omission, Rectified Linear Unit (ReLU) emerges as the most prevalent, utilized in 30 studies <cit.>,

which is known for its simplicity and effectiveness in handling non-linearities. Sigmoid activation function follows as the second most commonly used function appearing in nine studies <cit.>,

and it can map the output of a deep learning model to a probability-like value between 0 and 1. In addition, the hyperbolic tangent (tanh) activation function, which is a smooth, symmetric, and nonlinear function that maps the input to a continuous range between -1 and 1 <cit.>, and linear activation functions are employed in four and three studies, respectively <cit.>, while softmax is used in two studies <cit.>, which ensures the output values lie in the range of 0 to 1 and sum up to 1, representing valid probabilities.  In the context of configuration performance learning, where it is often the case that only a subset of configuration options holds significant influence over performance, ReLU offers a natural solution by effectively zeroing out negative inputs, effectively focusing on the important configuration options <cit.>.

Finding : Despite that 46 of the primary studies omit the justifications of optimization and activation
functions, we observe several different techniques, where the Adam optimizer and
ReLu activation function stands out as the most common one, used
in 24 and 30 papers, respectively.

§.§ Tuning Hyperparameters

Hyperparameter tuning plays a critical role in the accuracy and generalization ability of deep learning models, especially for configuration performance learning, where the interactions between the configurations and performance are complex and can change largely and nonlinearly across software <cit.>. However, determining the optimal values for hyperparameters is a challenging task as the hyperparameter space is often huge and the tuning is time-consuming.

Figure <ref> presents an analysis of the hyperparameter tuning methods utilized in the examined studies. It is most worth noting that, a great portion of 52 studies, rely on human experts and domain knowledge to tune the hyperparameters. Among others, six primary studies simply rely on the default hyperparameter settings <cit.>, where models are trained using pre-defined configurations without any explicit tuning.

In addition, some simple search methods have been applied to tune the hyperparameters. For instance, grid search, which exhaustively tests over a predefined grid of values to find the best configuration, is the most widely used technique and is applied in 20 studies <cit.>;

ablation analysis <cit.>, a way that gradually increases the number of layers and selects the hyperparameter setting with the highest AUC value, and random search <cit.>, where the search path is randomly sampled from the hyperparameter space, are each employed by one study.

Hyperparameter tuning methods.

Simple and complex tuning methods.

Distribution of the hyperparameter tuning methods.

Then, more complex hyperparameter tuning methods, which leverage more complicated heuristics such as probabilistic models, evolutionary algorithms, or learning-based approaches to intelligently explore the hyperparameter space, are applied in five studies. For example, Bayesian optimization, a method that uses probability models to efficiently search for optimal hyperparameters by considering the performance of previously evaluated configurations, is employed in two studies <cit.>. Additionally, each of the following heuristics is explored by one study, respectively: genetic algorithm <cit.>, which leverage the principles of natural selection to iteratively search for optimal hyperparameter configurations; model selection <cit.> focuses on the process of choosing the best model from a set of candidate models based on their performance; and neural architecture search <cit.>, which automates the search for optimal neural network architectures by exploring different hyperparameter configurations.

Finding : Manual hyperparameter tuning is preferred by the majority of studies (52 out of 85). For the automated tuning method,
grid search is the predominately popular one (20 studies).

§ RQ3: HOW ARE THE DEEP CONFIGURATION PERFORMANCE MODELS EVALUATED?

To understand how to evaluate the deep learning models for configuration performance learning, in this section, we summarize the procedure, metrics, statistical validation, and subject software systems that are commonly used in comparing the models.

§.§ Following Evaluation Procedures and Metrics

Evaluation methods

Top 10 accuracy metrics

Distribution of the model evaluation procedures and metrics (one study might use items from multiple categories).

In Figure <ref>, we provide a summary of the evaluation procedure and metrics used. Among others, the most prevalent evaluation procedures is bootstrap, utilized by 30 out of the 85 studies <cit.>,

which is a resampling technique that involves creating multiple datasets by randomly sampling from the original data, allowing for robust estimation of model performance. Hold-out evaluation, another widely used method, is employed in 27 studies <cit.>.

This approach involves splitting the dataset into training and testing sets based on a specific percentage. Cross-validation, which involves iteratively partitioning the dataset into multiple subsets for training and testing, is used by 21 studies <cit.>.

Additionally, we find that seven studies opt to use the default evaluation pipeline provided by benchmark datasets, simplifying the evaluation process <cit.>.
For instance, <cit.> run a subset of Linpack benchmarks on different hardware platforms to collect the power consumption data, representing a variety of industry-standard workloads, to build the models in the offline learning stage, while using the rest benchmarks for evaluating the online identification stage.

Figure <ref> presents the frequency of usage for the top 10 mostly used accuracy metrics. Among the identified metrics, Mean Absolute Percentage Error (MAPE), also referred to as Mean Relative Error (MRE), emerges as the most commonly used in 27 studies <cit.>.

MAPE/MRE measures the percentage difference between the predicted and actual values. Mean Absolute Error (MAE) follows as the second most frequently employed metric, appearing in 17 studies <cit.>,

which calculates the average absolute difference between the predicted and actual values, offering a straightforward measure of model accuracy. Root Mean Square Error (RMSE) <cit.>

and Mean Squared Error (MSE) <cit.>

are utilized in 14 and 12 studies, respectively, providing insights into the model's ability to capture both small and large errors. The coefficient of determination (R$^{2}$)—a metric indicating the proportion of variance in the dependent variable explained by the model—is used in nine studies <cit.>.

Additionally, performance improvement <cit.> and optimal performance <cit.>, which are computed using the performance resulting from a selected configuration, are used by five and four works, respectively. Notably, while these two metrics do not quantify the accuracy of deep configuration models directly, they are practical to evaluate how the deep configuration performance models can impact performance optimization, performance testing, and configuration tuning tasks. For instance, <cit.> measure the increment in the latency and energy consumption to examine the effectiveness of their model. Lastly, relative error (RE), error rate (ER) <cit.>, and normalized mean absolute error (NMAE) <cit.> are employed in three, two and two studies, respectively.

Finding : Bootstrap is the most prevalent evaluation procedure, which is utilized in 30 out of 85
studies. On the other hand, the most popular accuracy metric is MAPE/MRE in 27 studies.

§.§ Leveraging Statistical Validation

Statistical test methods

Effect size test methods

Distribution of the statistical validation methods.

Since the deep learning models are stochastic in nature, evaluation in deep configuration performance learning is often conducted with repeated runs. Therefore, understanding their statistical significance is important. Indeed, when evaluating a deep configuration performance model, the results of two runs with exactly the same model settings and training/testing sample sizes could still be different, due to the stochastic loss optimization and the randomness in selecting the training and testing configurations, e.g., when evaluating with the compression tool Lrzip, two runs under the same conditions might resulted a MRE of 17.49% or 70.44% — a nearly $4\times$ difference <cit.>.

In Figure <ref>, we survey the methods used for statistical validation. Surprisingly, the majority of the studies, 79 out of 85, do not utilize any specific statistical test methods to measure the significance of the results compared. Among the studies that do apply statistical tests, the Scott-Knott Effect Size Difference (ESD) test emerged as the most commonly used, appearing in three studies <cit.>, which is a non-parametric test that groups data into distinct subset with significant difference. Additionally, the Wilcoxon signed-rank test was employed in two studies <cit.>, providing a non-parametric test for comparing two paired sets of data. Furthermore, the t-test <cit.> and the Wilcoxon rank-sum <cit.> test were each used in one study, offering comparisons for two independent data groups.

Similarly, Figure <ref> reveals a phenomenon in the utilization of effect size tests for evaluating deep configuration performance learning, where the majority of the studies, 80 out of 85, do not employ any specific effect size tests. Among the rest studies, the Scott-Knott ESD test is used in three studies <cit.>, which can identify significant differences on the effect size between groups in addition to its ability of assessing the statistical significance. Further, <cit.> utilized Cliff's Delta, a non-parametric effect size measure that quantifies the ordinal association between two variables.

Finding : A significant portion of 93% and 95% studies omit the usage of statistical and effect
size tests, respectively. Scott-Knott ESD test is used in 3 studies, standing out as the most common method for both cases.

§.§ Evaluating Configurable Software Systems

A crucial factor in evaluating deep configuration performance models is to ensure the evaluation covers a good range of subject systems. Figure <ref> provides the distribution of the number of systems considered in the primary studies. Among others, 37 studies evaluate a single software, 21 studies evaluate two software, 12 studies assess three systems, four studies evaluate four subject systems, and the remaining 11 studies evaluate five or more systems.

The taxonomy of the domains of the subject software systems in the primary studies is shown in Figure <ref>. Among these domains, 38 prominent studies build deep configuration performance models for distributed computing systems, which typically involve distributing computational tasks across multiple interconnected devices or nodes, enabling collaboration and parallel processing for improved computing performance, including domains like cloud computing <cit.>, big data processing <cit.>, and high-performance computing <cit.>.

# subject systems considered

Domains of subject systems

Distribution of the subject systems (one study might use items from multiple categories).

Scientific computing systems, used in 17 primary studies, play a crucial role in solving complex scientific and research problems by utilizing advanced algorithms and numerical methods, including domains like matrix multiplication <cit.>, multi-grid solvers <cit.>, stencil-grid solvers <cit.>, and the training of deep learning models <cit.>, enabling researchers and scientists to analyze and model intricate data sets efficiently.

16 studies focus on the modeling of data management systems that facilitate the storage, organization, retrieval, and manipulation of data, including software systems that provide structured data storage and query capabilities <cit.>, I/O libraries that facilitate efficient input/output operations with data <cit.>, file systems that manage file storage and access <cit.>, and compression tools that reduce data size for efficient storage and transmission <cit.>.

Furthermore, Web service software like web server systems, involving the utilization of internet-based technologies to provide a platform for hosting websites, is explored in 11 studies <cit.>.

Additionally, Programming tools and multimedia processing are each used in seven studies. Programming tools include Python programs <cit.> and compilers, which translate high-level programming languages into machine-readable instructions to allow the code to be executed by the computer <cit.>. Multimedia processing systems include image processing <cit.>, which focuses on techniques for enhancing, modifying, and analyzing images, as well as video encoding<cit.>, which involves the compression and encoding of video data to optimize storage and transmission while maintaining high-quality playback.

Hardware computing systems, explored in six studies, comprise diverse domains such as CPU computing <cit.>, GPU applications that leverage graphics processing units for general-purpose computing <cit.>, chip multi-processors <cit.>, and multicore systems <cit.>, all of which contribute to enhancing computational capabilities, efficiency, and parallel processing within computer systems <cit.>.

Finally, five research works examine embedded systems and network systems, respectively. Especially, embedded computing includes mobile systems <cit.>, which are designed for smartphones and tablets, Internet of Things (IoT) communication systems that facilitate connectivity and data exchange among IoT devices <cit.>, large-scale commercial systems used in areas such as e-commerce and finance <cit.>, and operating systems <cit.>; and network systems refer to the diverse domains associated with the establishment, operation, and administration of computer networks, enabling efficient communication and data exchange between devices. This includes traditional network infrastructure <cit.>, as well as emerging architectures like software-defined networking <cit.> and wireless sensor networks <cit.>.

Finding : Most commonly, the primary studies on deep configuration performance learning evaluate one configurable software system only (37
out of 85 studies). Furthermore, among the various domains explored, distributed computing
emerges as the most popular domain for subject systems, utilized in 38 studies.

§ RQ4: HOW TO EXPLOIT THE DEEP CONFIGURATION PERFORMANCE MODEL?

The successful application of deep learning models trained on configuration data relies on considering various factors, including application purposes, adaptability to dynamic software running environments, and the availability of materials for reproducibility. In this section, we study those factors involved in the exploitation and dissemination of the deep configuration performance models.

§.§ Using the Model

A deep configuration performance model, once built, can be used in different stages of the software engineering practices. Table <ref> shows that out of the 85 papers surveyed, 57 studies emphasize the domain of general prediction where the key has been ensuring the accuracy and reliability of the deep configuration performance model, without focusing on any particular domain in which the model can be used.

19 studies apply the deep learning models for performance tuning, which aims at optimizing the performance by finding the optimal configuration. It involves tasks such as configuration auto-tuning <cit.>, where software configurations are automatically adjusted to fit performance requirements, and self-adaptive systems which dynamically adapt to varying conditions to achieve optimal performance <cit.>. In such a context, the deep configuration performance model directly serves as the surrogate for the tuning algorithm.

Performance testing tasks are considered by six studies. They involve assessing the system's behavior for specific configurations or under different environments to identify potential bottlenecks or performance issues. For example, configuration bug prediction seeks to identify potential bugs or issues that are caused by incorrect configuration, leading to performance degradation <cit.>. Vulnerabilities detection focuses on identifying security vulnerabilities that could affect system performance or compromise its functionality <cit.>.

Additionally, tasks on scheduling have been addressed by two primary studies, which seek to allocate the available resources, such as CPU, memory, or network bandwidth, in an optimal manner to maximize system performance <cit.>. The unique property is that there are often low-level metrics, such as the CPU cycle or cache rate, that are considered together with the performance attributes. Meanwhile, one study applies performance models for development <cit.>, involving designing and implementing the best set of configurations for developing the architecture of deep learning models or software.

Finding : The applications of deep configuration performance models are mainly for fulfilling five purposes, in which the most common one is the general prediction for performance analysis.

§.§ Considering Multiple Runtime Environments

Configurable software systems would inevitably run under multiple, and potentially changing environments. For example, a database system may experience both read-heavy and write-heavy workload <cit.>. Similarly, the hardware between the testing and production infrastructure might be drastically different <cit.>, especially during the modern DevOps era. It is therefore natural to understand how such a fact has been taken into account for deep configuration performance learning.

As in Figure <ref>, our survey results indicate that, out of the 85 papers surveyed, 56 studies do not explicitly consider the challenges posed by multiple and dynamic software running environments in their application and design of performance models <cit.>. In other words, they rely on using the data from a single environment to build the configuration performance models and predict therein.

This suggests that a considerable portion of the research in deep learning-based configuration performance learning has focused primarily on static or controlled settings, without accounting for the real-world complexities of dynamic, multiple environments. Nevertheless, among the 35 studies that acknowledge and consider dynamic environments in their application of performance models, 19 studies exclusively focus on exploring the impact of different workloads on configuration performance <cit.>. Additionally, seven studies concentrate solely on investigating the influence of different hardware settings <cit.>, and one study specifically examines the impact of different software versions <cit.>. Lastly, two studies delve into the implications of mixed environment types, i.e., one considers three workloads combined with two hardware settings <cit.> and the other considers 12 workloads combined with two hardware settings <cit.>.

Finding : More than half of the 85 primary studies (66%) do not consider multiple and dynamic environments. Among others, workload stands out as the most commonly considered factor (23%).

Distribution of the environment types.

Distribution of the availability of artifacts.

§.§ Publishing Source Code and Data

The availability of public repositories plays an essential role in promoting replicability, transparency, and development in research. This is particularly crucial for practical research fields such as configuration performance learning. The results of our survey in Figure <ref> reveal that a significant number of studies do not provide an open-access repository containing the necessary resources for replication. In particular, out of the 85 papers surveyed, a total of 67 studies do not make their code, data, or experiment results publicly available <cit.>. This indicates a lack of emphasis on open science practices and hinders the ability of other researchers and practitioners to validate and build upon the findings. On the other hand, 18 studies offer an open-access repository, of which 14 provide both source code and datasets <cit.>, and four of them only provide source code <cit.>.

Finding : Only 21% (28 out of 85) of the primary studies provide public repositories. An even lower proportion of them, i.e., 16%, make both source code and datasets publicly available.

§ DISCUSSION: THE COMMON PRACTICE OF CURRENT WORK

This section provides an in-depth discussion of the common practice identified from the results of the existing work. Through this, we aim to summarize the good behaviors that all of us can follow and alarm the “bad smell” that should be better dealt with in future work of configuration performance learning. Specifically, we discuss the common practice observed with respect to different themes of the research questions, namely Preparation, Modeling, Evaluation, and Application.

§.§ Preparation

Drawing on the results from Section <ref>, we observe various phenomena related to different phases within the preparation stage of deep configuration performance learning. Specifically, there are several good practices:

* Configuration performance data often exhibits some characteristics that are difficult to learn by deep learning models, and addressing these challenges can lead to improved model performance, which puts high demands on data preprocessing to improve the data quality. Therefore, it is a good practice that a variety of characteristics have been tackled in the primary studies. For example, the configuration space of software is demonstrated to be high-dimensional and sparse, which has been mitigated by regularization and dimension extraction techniques; and the primary studies also reveal that raw performance datasets often have outliers, imbalance, and multicollinearity issues. By addressing these issues, these studies have enhanced the effectiveness and efficiency of performance models and provided insights for future studies.

* Encouragingly, it has been demonstrated that the choice of encoding scheme is critical since it profoundly impacts the prediction accuracy and time consumption of deep learning models <cit.>. For instance, one-hot encoding tends to yield the highest accuracy albeit with longer training times compared to other schemes, while label encoding often leads to faster training times. Thereby, they provide practical rules for choosing the best encoding scheme and enable future researchers to make more informative decisions.

* The configuration space can be enormous, while the resources available for collecting training samples are often limited. Therefore, it is crucial to carefully select the most appropriate configurations to train performance models. However, one of the challenges lies in determining the criteria for sample selection. In the current literature, a good practice is that a few studies have addressed this issue by employing heuristic sampling methods tailored to specific requirements. Especially, <cit.> leverage SMBO to guide the sampling process with a surrogate Gaussian Process model to minimize the uncertainty of the samples in each iteration. By combining with various surrogate models, SMBO can potentially yield diverse sampling results, providing potential for future studies to investigate.

On the other hand, the bad smells we observed are as follows:

* It is observed that a majority of the primary studies (44 out of 85) have paid inadequate attention to the importance of preprocessing methods. Neglecting these crucial steps can severely limit the performance and effectiveness of deep learning models <cit.>. Therefore, it is imperative for future studies to recognize the significance of preprocessing techniques and incorporate them systematically to ensure optimal model performance.

* The influence of encoding schemes is underestimated by a majority of the studies, which solely rely on an ad-hoc way without providing justifications for the encoding schemes. This oversight can be detrimental, as it leaves other researchers unclear of which encoding schemes is suitable, potentially resulting in ambiguous data formats for deep learning tasks <cit.>.

* Despite the availability of various sampling strategies, a significant portion of the studies tends to focus on the most straightforward approach, namely random sampling. While random sampling is effective in representing the distribution of the dataset, it can lead to overfitting in highly sparse systems, as many of the features may prove ineffective for software performance analysis <cit.>. This can be even more problematic considering the fact that there exist invalid configurations <cit.>. Therefore, researchers need to explore alternative sampling methods in such scenarios to enhance the accuracy of the performance models.

§.§ Modeling

Recall from Section <ref>, the modeling stage is also a major stage in deep configuration performance learning for which we confirm several common practices. In particular, our analysis reveals the following positive practices:

* Encouragingly, a majority of studies recognize the significance of addressing sparsity and overfitting, which are critical for causing the degradation of predictive models, urging the application of a diverse range of handling techniques <cit.>. However, it is worth noting that most of these techniques are option selection methods as the configuration options often have redundant information and most of them have little influence on the software performance. Yet, <cit.> discover that except for feature/option sparsity, which has been realized by most studies, the distribution of samples in the configuration landscape is also sparse—a significant property in the configuration performance learning problem that is worth further investigating.

* Section <ref> shows that the very basic version of DNN, i.e., MLP is the most commonly used learning model being employed by a total of 54 studies. This is evidence that a simple deep learning model is useful for learning software configurations and performance. This derives the exploitation of more complex DNN, for example, rDNN <cit.> is a DNN armed with regularization to address the problem of feature sparsity, and ResNet <cit.>—a type of neural network that incorporates residual connections to overcome the challenges of vanishing gradient for configurations embed into high dimensional representation. This demonstrates the potential of the deep learning models by addressing specific challenges and sheds light on future studies on improving the existing models.

* With the complicated architectures of deep learning models, one challenge for hyperparameter tuning is the explosion of search space. As such, we are pleased to reveal that 33 primary studies have addressed this challenge by exploring different heuristic methods to guide the tuning of hyperparameters while training deep learning performance models. For instance, <cit.> employ the idea of natural selection and evolution to find the best population of hyperparameter configurations, which is robust to noise and outliers as the overall fitness is based on the collective behavior of the population rather than individual solutions. This diverse range of heuristic methods reflects the researchers' efforts to explore different approaches for optimizing hyperparameters and enhancing model performance, which should be kept by future researchers.

* In configuration performance learning tasks, the best optimization method for reducing the loss in training differs depending on the systems and performance attribute. For example, tasks with sparse features may benefit from adaptive optimization methods like Adam, while SGD is often used in situations where computational resources are limited. Thus, given that the relationship between configurations and performance of highly configurable software is often sparse, it is encouraging to find that 24 studies address this problem by leveraging Adam optimizer.

* Similarly, choosing the appropriate activation function can impact the DL model's performance, e.g., ReLU is effective in capturing complex, non-linear relationships. Hence, it is a promising behavior to see that ReLu is used in 30 primary studies to handle the non-linearity specializing the problem of configuration performance prediction.

Conversely, we have identified the following bad smells:

* Given that sparsity has been addressed by 52 out of 85 papers, 24 of them rely on human efforts to select the correct feature sets to reduce sparsity, which is of low efficiency and low generalizability. In addition, there are still 33 works that did not acknowledge the sparsity issue. Moreover, only one study focused on solving the sparsity problem from the perspective of sample distributions, which could be a key aspect of future research.

* Recall from Section <ref>, there are 55% studies (47 out of 85) that provide no justification for their choices of deep learning models, which is a concern. It is essential for researchers to clarify their reasons for selecting specific models, as this enhances the understanding of the research motivations and contributions <cit.>. In the meantime, the lack of explicit explanations of model selection could be harmful to the community as researchers may miss out on important considerations or potential limitations.

* A great portion (52 out of 85) of papers choose to tune the hyperparameters manually, which is time-consuming and not generalizable. Besides, a subset of six studies relies solely on default hyperparameters that may not be optimal for all scenarios. This limitation may restrict the exploration of the optimal hyperparameter settings and hinder the potential of deep learning models <cit.>. As such, it is important for researchers to consider automatic tuning methods in order to maximize the performance and robustness of deep learning models.

* Surprisingly, more than half of the studies (46 out of 85) omitted information on their choices of optimization and activation approaches. This omission poses a significant challenge to understanding and reproducing the deep learning models' results and findings <cit.>. Therefore, justification for these decisions should be provided in future works, as it allows practitioners to comprehend the rationale behind the decision-making.

§.§ Evaluation

Several noticeable common practices can be observed in the evaluation aspect of deep configuration performance learning from Section <ref>. Notably, the positive behaviors encompass:

* A variety of metrics for accuracy has been applied is a rather positive sign for the community, as  <cit.> point out, no single measure gives an unambiguous indication of the modeling performance. More importantly, we have discovered that all the primary studies have leveraged some form of repeated evaluation procedure. This is crucial to ensure that the results were not produced due to chance or biased samples.

* Almost all studies have conducted comparisons between their models with existing works, and a few of them have utilized statistical tests and effect size tests to ensure the statistical significance and effect size difference of the comparison. This practice is vital in deep learning research as it enhances the reliability and robustness of the findings.

* Another positive observation is that 48 studies have considered multiple subject systems in their experiments. This is significant because evaluating the proposed model on software from different domains, scales, and performance indicators demonstrates its generalizability and robustness <cit.>.

* Moreover, it is encouraging to see that the primary studies have examined a variety of domains. This diversity in domain coverage enhances the credibility and applicability of the findings across various contexts, providing broader insights for developers and practitioners.

On the contrary, the bad smells consist of:

* We found that MAPE is the most widely used metric, due primarily to the fact that it is unit-free and easy to calculate. However, it is known that MAPE suffers the problem of being asymmetric on the error above and below the actual value <cit.>, where the former receives a much greater penalty. As a result, such a strong bias tends to be problematic.

* A notable concern arises from the fact that a significant majority of studies, around 93% (79 out of 85), did not conduct any statistical tests, and 95% of studies did not measure the effect size. This represents a significant problem as it introduces a potential source of randomness and uncertainty in the comparison results, limiting the ability to draw reliable conclusions and increasing the risk of causing severe external threats to validity. This highlights the need for greater awareness and adoption of statistical analysis in deep learning research.

* It is worth noting that a significant proportion of studies (44% of 85) evaluate their models on only one software, which can be considered a potential threat to the validity of the results and findings of the study.
Indeed, software systems of different characteristics can exhibit unique performance behavior due to factors such as architecture, complexity, and usage patterns. Therefore, it is important for future studies to evaluate their models on multiple software systems, spanning various domains, scales, and performance indicators, to ensure the robustness and learning ability of the proposed performance models.

§.§ Application

According to Section <ref>, applying the deep learning model to predict configuration performance is also subject to good practices and bad smells. Particularly, we confirm a few good practices:

* The best practice in the model applications is their widespread adoption across various domains and for different tasks. This diversity is critical as it showcases the potential of performance models in addressing diverse challenges, and provides practical insights for researchers in different fields <cit.>.

* The most encouraging trend is that 29 out of 85 studies have actively considered multiple and dynamic environments, demonstrating the robustness of models. Acknowledging that software systems run under multiple and potentially varying environments is an important property as the behavior of software could be completely different across the environments <cit.>. By accounting for dynamic workloads, versions, and hardware in real-world scenarios, these studies contribute to a more comprehensive understanding of how performance models can be applied in practical settings.

* 19 out of the total studies have embraced the principles of open science by providing a public repository that includes source codes and datasets. This is a good practice as it allows researchers to replicate or reproduce the experiment results, build further research based on the existing knowledge, and collaborate with each other <cit.>.

Meanwhile, the negative smells encompass:

* A potential concern is that 57 studies only consider pure prediction tasks, while these studies contribute valuable insights into performance prediction, it raises the question of whether these models can effectively be applied to other tasks beyond prediction, in other words, the applicability and generalizability of performance models across different domains and tasks remain unclear, especially when the usefulness of the model is merely measured by accuracy metrics.

* Another bad smell is that 66% of 85 primary studies have solely focused on datasets from a single environment without considering dynamic factors that can impact model performance, which means there is a risk of overestimating the model's capabilities or encountering unexpected issues when applied in real-world scenarios <cit.>. Therefore, it is essential for future research to address this limitation by actively incorporating dynamic environments into their evaluation processes.

* It is concerning to note that a significant number of studies (78%) still missed the opportunity to promote open science by not providing public repositories. This is a limitation as it restricts the replicability, reproducibility, and development of the community, which could be addressed by future studies. Further, among the 19 studies that offer an open-access repository, only 14 of them provide both source codes and datasets.

§ FUTURE RESEARCH OPPORTUNITIES

Our survey has revealed several key knowledge gaps and promising directions that are worth further investigation in future studies, which we elaborate on below.

§.§ Model-Based Sampling for Deep Configuration Performance Learning

From RQ1 (Section <ref>), we see that while several sampling strategies have been employed in the studies reviewed, it is evident that a significant majority of them rely on random sampling, which highlights a significant opportunity for future research to explore and develop more effective sampling strategies. Random sampling, while straightforward, may not always yield the most informative or representative samples within the configuration space <cit.>.
By exploring alternative sampling techniques that select samples based on specific criteria or heuristics, researchers can potentially improve the efficiency and effectiveness of the models <cit.>. This is especially important for deep learning configuration performance since it is known that the deep learning model is sensitive to the quality of training data <cit.>.

Among others, addressing this gap by investigating model-based sampling methods that can provide an explicit acquisition function to reliably and accurately guide the sampling process is a promising future direction. For instance, Sequential Model-Based Optimization (SMBO) <cit.>, such as Bayesian Optimization, effectively explores the sample space by quantifying the uncertainty of the samples to guide the search towards potentially optimal regions <cit.>. In particular, using a surrogate model, such as a Gaussian process, to sequentially select new samples based on the current model's predictions and uncertainty estimates, would enable efficient exploration of the sample space. However, to the best of our knowledge, there has been little work that investigates this thread of research.

§.§ Explainable Deep Configuration Performance Learning.

Through RQ2 (Section <ref>), we reveal that various types of deep learning models have been used to learn configuration performance, however, almost all of the proposed deep learning models lack explainability. Unlike analytical performance models, which enable researchers to analyze the importance of options and the interactions between them <cit.>, deep learning models learn the configuration performance in a black-box manner, which could be harmful to the reliability of the results.

In contrast, certain statistical machine learning performance models are naturally interpretable. For instance,  <cit.> propose SPLConqueror leverages a linear model to learn configuration performance, enabling one to quantify the influence of each configuration; <cit.> have examined Random Forest (RF) and Classification and Regression Tree (CART) for performance learning, highlighting the importance of options and the decision boundaries to cluster the configurations. Several studies have explored Bayesian theory to analyze the uncertainty of predictions that aid the explainability <cit.>.

Yet, addressing the above is challenging for deep learning model due to their nonlinear and high-dimensional representation nature. Luckily, explainable deep learning has been studied in various domains, such as pattern recognition tasks including medical diagnosis, face recognition, and self-driving cars <cit.>, including those model-agnostic ones like Local Interpretable Model-agnostic Explanation (LIME) <cit.> and SHapley Additive exPlanations (SHAP) <cit.>. We can directly leverage the advance of the current explainable extension of deep learning, and specialize them in the context of configuration performance. However, achieving explainable deep configuration performance models faces several challenges, e.g.,

* How to quantify the internal elements of a deep learning model, e.g., activation function, with respect to their contributions according to the type and characteristics of the configuration options.

* How to equip a deep learning model with an intrinsic method that produces explanations on the configuration performance alongside its outputs.

* How to visualize the configuration options and their contributions toward the final prediction with a deep learning model.

Pushing the research toward addressing any of the above would be of immediate interest and benefit to the field.

§.§ Configuration Performance Modeling with Few-Shot Deep Learning

Deep neural networks for learning configuration performance often require a substantial amount of labeled data, which can be challenging and time-consuming to obtain, as the availability of labeled data is often limited due to factors such as the complexity of software systems, the cost of performance measurements, and the lack of expert knowledge.

A promising direction of most efficient deep learning approaches is to complement it with few-shot learning, which aims at training to quickly adapt and generalize to new tasks or classes with only a few labeled configuration samples or even a single configuration, e.g., siamese networks, memory-augmented networks, transfer learning, meta-learning, and augmentation strategies <cit.>. For example, augmentation techniques like Generative Adversarial Networks have been used in three studies to generate additional training samples to augment the available configuration dataset <cit.>, and siamese neural networks are able to differentiate between pairs of examples, making them effective for few-shot learning, which have been successfully used for image processing tasks <cit.>.

Despite the above, few-shot deep learning still imposes several challenges for learning configuration data due to the complexity and specialty of configurable software systems. For example, the selected most representative data samples are difficult to quantify, obscuring effective generalizability in the few-shot deep learning model. Therefore, further research is needed to unlock the full potential of few-shot deep learning for configuration performance modeling by enhancing robustness, generalizability, and transferability.

§.§ Interactive Deep Configuration Performance Learning.

While deep learning models have demonstrated effectiveness in configuration performance learning, they come with inherent challenges, notably the substantial training overhead when compared to other machine learning models. The resource-intensive nature of training deep learning models poses limitations on the adaptability and reliability, especially when quick adaptation to newly measured data or real-time updates is required. This issue underscores the urgent need for innovative solutions that can efficiently update and optimize deep learning models, paving the way for more updated and reliable systems.

To address these challenges, interactive deep learning techniques have emerged as a powerful solution, which enables continuous learning and adaptation optionally with the support of humans, overcoming the limitations associated with the static nature of traditional deep learning models and fully benefiting from the domain expertise. By incorporating real-time updates and leveraging user feedback, interactive deep learning ensures that models remain accurate and up-to-date using data filtered by human knowledge. This adaptability is particularly crucial in applications where the underlying data distribution may change over time as the new measurements become available <cit.>. Despite being a promising direction related to the human-in-the-loop techniques for configuration performance modeling, interactive updates of deep configuration performance models are currently under-explored.

§.§ Deep Configuration Performance Learning under Multiple Dynamic Environments

From RQ4, it is evident that a majority, specifically 66%, do not address the dynamic and varying nature of software running environments, which may significantly limit the effectiveness and generalizability of deep configuration performance models. In particular, <cit.> systematically analyzes the influence of workloads on configurable software systems and discloses that varying the workload does not only affect the performance value but also affects the relationships between the configurations and performance.

This observation highlights the need to address this gap by developing deep learning methodologies that can effectively handle configuration data across multiple environments. Little work has tackled the above by using, e.g., multi-task learning <cit.>, which learns the common representations between multiple tasks simultaneously; transfer learning <cit.>,

which seek to leverage the common information gained from related environments, enabling models to adapt and generalize to the target environments; or meta-learning <cit.>,

which pretrain a set of model parameters via meta-training and can adapt quickly or unseen tasks or environments. Particularly, a recent work <cit.> proposes a sequential meta-learning framework, namely , that trains a meta-model with multiple meta-environments in an optimal order such that the pre-trained model can generalize to new environments well. Yet, most of the existing work only assumes homogeneous configuration performance learning, i.e., for different environments, the configuration options need to remain the same. Further, those works do not cater to online learning <cit.>, where the data is continually used to update the model as it becomes available. As a result, deep learning models that can learn from multiple environments and are capable of self-updating at runtime are in high demand for configuration performance learning <cit.>.

§ THREATS TO VALIDITY

When conducting a systematic literature review, it is important to consider potential threats to validity that could affect the reliability and generalizability of the findings. We now discuss the threats to the validity related to this work.

§.§ Sampling Bias

Sampling bias refers to the potential threats of the results due to the selection of studies. In this survey, sampling bias could arise from the selection of primary papers from indexing services or the inclusion and exclusion of certain papers. To mitigate this threat, we have codified specific rules, procedures and criteria in the search protocol, which is based on the guidance provided by  <cit.>. Specifically, the sampling of studies is done by a comprehensive automatic search via six popular indexes in software engineering to cover a wide range of papers, then the application of domain knowledge to remove the redundant and irrelevant studies, and filtering them with the carefully crafted inclusion and exclusion criteria to further remove potential bias, as summarized in Figure <ref>.

§.§ Internal Threats

Internal threats to validity relate to issues within the study design or data analysis that could affect the accuracy and reliability of the results. In this survey, potential internal threats could include inconsistencies in data extraction, subjectivity in data analysis, or biased interpretation of the findings. To address these threats, we have followed a systematic and rigorous approach to data extraction and analysis. By clearly defining research questions, it ensures that the data extraction process is focused and consistent. Additionally, we have limited this by employing multiple reviewers and by conducting three iterations of independent paper reviews among the authors. Error checks and investigations were also conducted to correct any issues found during the search. Any discrepancy in the results was discussed until an agreement can be reached.

§.§ External Threats

External threats to validity are usually related to the generalizability and applicability of the findings beyond the specific context of the survey. In this survey, external threats could arise from the limited time range from 2013 to 2023, or the limited scope of the papers using deep learning for performance modeling. To mitigate these threats, we have searched 948 studies from six indexing services, we clearly defined the inclusion and exclusion criteria for the selection of papers, based on which we extracted 85 prominent primary studies for detailed analysis. By including a diverse range of studies from different indexing services, we provide a broader perspective on deep configuration performance learning.

§ CONCLUSION

In this work, we present a comprehensive survey on the increasingly popular topic of deep configuration performance learning, covering 85 prominent works from 948 studies found on six indexing services. We provide detailed taxonomy, together with discussion on the various techniques involved, according to the key phases in building a deep configuration performance model, i.e., preparation, modeling, evaluation, and applications. Our results reveal several good practices and bad smells of the existing studies in the field.

More importantly, we highlight promising research opportunities for this particular research field, namely:

* model-based sampling methods for deep configuration performance learning.

* explainable deep configuration performance model.

* interactively learned deep configuration performance model.

* deep few-shot learning for modeling configuration performance.

* deep configuration performance learning under multiple and dynamic environments.

By highlighting the identified trends and future directions of deep learning for configuration performance modeling—we hope to inspire sustainable growth on this particular topic.

#1     #1#1#1     #1     #1     #1     #1#1          #1#1

[Acher, Martin, Lesoil, Blouin,
Jézéquel, Khelladi, Barais, and PereiraAcher
et al2022]
authorpersonMathieu Acher, personHugo
Martin, personLuc Lesoil, personArnaud Blouin,
personJean-Marc Jézéquel,
personDjamel Eddine Khelladi, personOlivier Barais,
and personJuliana Alves Pereira.
Feature subset selection for learning huge
configuration spaces: the case of linux kernel size. In
booktitleSPLC '22: 26th ACM International Systems and
Software Product Line Conference, Graz, Austria, September 12 - 16, 2022,
Volume A, editorpersonAlexander Felfernig,
personLidia Fuentes, personJane Cleland-Huang,
personWesley K. G. Assunção,
personAndreas A. Falkner, personMaider Azanza,
personMiguel Á. Rodríguez Luaces,
personMegha Bhushan, personLaura Semini,
personXavier Devroey, personCláudia Maria Lima
Werner, personChristoph Seidl, personViet-Man Le,
and personJosé Miguel Horcas (Eds.).
publisherACM, pages85–96.

[Agarwal, Singhvi, Malakar, and
BynaAgarwal et al2019]
authorpersonM. Agarwal, personD.
Singhvi, personP. Malakar, and personS.
Byna. year2019.
Active Learning-based Automatic Tuning and
Prediction of Parallel I/O Performance. In booktitle2019
IEEE/ACM Fourth International Parallel Data Systems Workshop (PDSW).

[Alabed and YonekiAlabed and Yoneki2021]
authorpersonSami Alabed and personEiko
Yoneki. year2021.
High-Dimensional Bayesian Optimization with
Multi-Task Learning for RocksDB. In
booktitleEuroMLSys@EuroSys 2021, Proceedings of the 1st
Workshop on Machine Learning and Systemsg Virtual Event, Edinburgh, Scotland,
UK, 26 April, 2021, editorpersonEiko Yoneki and
personPaul Patras (Eds.). publisherACM,

[Alaya, Bussy, Gaïffas, and
GuillouxAlaya et al2019]
authorpersonMokhtar Z. Alaya, personSimon
Bussy, personStéphane Gaïffas, and
personAgathe Guilloux. year2019.
Binarsity: a penalization for one-hot encoded
features in linear supervised learning.
journalJ. Mach. Learn. Res.  volume20
(year2019), pages118:1–118:34.

authorpersonEthem Alpaydin.
booktitleIntroduction to machine learning.

[Ateeq, Afzal, Anjum, and KimAteeq
et al2022]
authorpersonMuhammad Ateeq,
personMuhammad Khalil Afzal, personSheraz Anjum,
and personByung-Seo Kim. year2022.
Cognitive quality of service predictions in
multi-node wireless sensor networks.
journalComput. Commun.  volume193
(year2022), pages155–167.

[Bai, Wang, Liu, Liu, Song, Sebe, and KimBai
et al2021]
authorpersonXiao Bai, personXiang Wang,
personXianglong Liu, personQiang Liu,
personJingkuan Song, personNicu Sebe, and
personBeen Kim. year2021.
Explainable deep learning for efficient and robust
pattern recognition: A survey of recent developments.
journalPattern Recognit.  volume120
(year2021), pages108102.

[Balsamo, Marco, Inverardi, and
SimeoniBalsamo et al2004]
authorpersonSimonetta Balsamo,
personAntinisca Di Marco, personPaola Inverardi,
and personMarta Simeoni. year2004.
Model-Based Performance Prediction in Software
Development: A Survey.
journalIEEE Trans. Software Eng.
volume30, number5 (year2004),

[Balsamo, Persone, and InverardiBalsamo
et al2003]
authorpersonSimonetta Balsamo,
personVittoria De Nitto Persone, and personPaola
Inverardi. year2003.
A review on queueing network models with finite
capacity queues for software architectures performance prediction.
journalPerform. Evaluation volume51,
number2/4 (year2003), pages269–288.

[Bao, Liu, Wang, and FangBao
et al2019]
authorpersonLiang Bao, personXin Liu,
personFangzheng Wang, and personBaoyin Fang.
ACTGAN: Automatic Configuration Tuning for
Software Systems with Generative Adversarial Networks. In
booktitle34th IEEE/ACM International Conference on
Automated Software Engineering, ASE 2019, San Diego, CA, USA, November
11-15, 2019. publisherIEEE, pages465–476.

[Blott, Preußer, Fraser, Gambardella,
O’brien, Umuroglu, Leeser, and VissersBlott et al2018]
authorpersonMichaela Blott, personThomas B
Preußer, personNicholas J Fraser, personGiulio
Gambardella, personKenneth O’brien, personYaman
Umuroglu, personMiriam Leeser, and personKees
Vissers. year2018.
FINN-R: An end-to-end deep-learning framework for
fast exploration of quantized neural networks.
journalACM Transactions on Reconfigurable Technology
and Systems (TRETS) volume11, number3
(year2018), pages1–23.

[Brunnert, van Hoorn, Willnecker, Danciu,
Hasselbring, Heger, Herbst, Jamshidi, Jung, von Kistowski, Koziolek,
Kroß, Spinner, Vögele, Walter, and WertBrunnert
et al2015]
authorpersonAndreas Brunnert,
personAndré van Hoorn, personFelix Willnecker,
personAlexandru Danciu, personWilhelm Hasselbring,
personChristoph Heger, personNikolas Herbst,
personPooyan Jamshidi, personReiner Jung,
personJóakim von Kistowski, personAnne
Koziolek, personJohannes Kroß, personSimon
Spinner, personChristian Vögele,
personJürgen Walter, and personAlexander
Wert. year2015.
Performance-oriented DevOps: A Research Agenda.
journalCoRR  volumeabs/1508.04752

[Cao, Pun, and ChenCao et al2022]
authorpersonQi Cao, personMan-On Pun,
and personYi Chen. year2022.
Deep Learning in Network-Level Performance
Prediction Using Cross-Layer Information.
journalIEEE Trans. Netw. Sci. Eng.
volume9, number4 (year2022),

[Cengiz, Forshaw, Atapour-Abarghouei, and
McGoughCengiz et al2023]
authorpersonMehmet Cengiz, personMatthew
Forshaw, personAmir Atapour-Abarghouei, and
personAndrew Stephen McGough. year2023.
Predicting the Performance of a Computing System
with Deep Networks. In booktitleProceedings of the 2023
ACM/SPEC International Conference on Performance Engineering, ICPE 2023,
Coimbra, Portugal, April 15-19, 2023,
editorpersonMarco Vieira, personValeria
Cardellini, personAntinisca Di Marco, and
personPetr Tuma (Eds.). publisherACM,

[Chai, Tripathy, Zhou, Gope, Fedorov, Matas,
Brooks, Wei, and WhatmoughChai et al2023]
authorpersonYuji Chai, personDevashree
Tripathy, personChuteng Zhou, personDibakar Gope,
personIgor Fedorov, personRamon Matas,
personDavid Brooks, personGu-Yeon Wei, and
personPaul Whatmough. year2023.
PerfSAGE: Generalized Inference Performance
Predictor for Arbitrary Deep Learning Models on Edge Devices.
journalarXiv preprint arXiv:2301.10999

[Chai, Zhang, Shen, and GuChai
et al2022]
authorpersonYitian Chai, personHongyu
Zhang, personBeijun Shen, and personXiaodong Gu.
Cross-Domain Deep Code Search with Meta Learning.
In booktitle44th IEEE/ACM 44th International Conference on
Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022.
publisherACM, pages487–498.

[Chen, Chen, and LiChen
et al2024]
authorpersonPengzhou Chen, personTao Chen,
and personMiqing Li. year2024.
MMO: Meta Multi-Objectivization for Software
Configuration Tuning.
journalIEEE Transactions on Software Engineering

[Chen, Haque, Liu, and YangChen
et al2022]
authorpersonSimin Chen, personMirazul
Haque, personCong Liu, and personWei Yang.
DeepPerform: An Efficient Approach for Performance
Testing of Resource-Constrained Neural Networks. In
booktitle37th IEEE/ACM International Conference on
Automated Software Engineering, ASE 2022, Rochester, MI, USA, October
10-14, 2022. publisherACM, pages31:1–31:13.

authorpersonTao Chen. year2019.
All versus one: an empirical comparison on
retrained and incremental machine learning for modeling performance of
adaptable software. In booktitleProceedings of the 14th
International Symposium on Software Engineering for Adaptive and
Self-Managing Systems, SEAMS@ICSE. pages157–168.

authorpersonTao Chen.
Lifelong Dynamic Optimization for Self-Adaptive
Systems: Fact or Fiction?. In booktitleIEEE International
Conference on Software Analysis, Evolution and Reengineering, SANER 2022,
Honolulu, HI, USA, March 15-18, 2022. publisherIEEE,

authorpersonTao Chen.
Planning Landscape Analysis for Self-Adaptive
Systems. In booktitleInternational Symposium on Software
Engineering for Adaptive and Self-Managing Systems, SEAMS 2022, Pittsburgh,
PA, USA, May 22-24, 2022, editorpersonBradley R.
Schmerl, personMartina Maggio, and personJavier
Cámara (Eds.). publisherACM/IEEE,

[Chen and BahsoonChen and Bahsoon2017a]
authorpersonTao Chen and personRami
Bahsoon. year2017a.
Self-Adaptive and Online QoS Modeling for
Cloud-Based Software Services.
journalIEEE Trans. Software Eng.
volume43, number5 (year2017),

[Chen and BahsoonChen and Bahsoon2017b]
authorpersonTao Chen and personRami
Bahsoon. year2017b.
Self-Adaptive Trade-off Decision Making for
Autoscaling Cloud-Based Services.
journalIEEE Trans. Serv. Comput.
volume10, number4 (year2017),

[Chen, Bahsoon, Wang, and YaoChen
et al2018b]
authorpersonTao Chen, personRami Bahsoon,
personShuo Wang, and personXin Yao.
To Adapt or Not to Adapt?: Technical Debt and
Learning Driven Self-Adaptation for Managing Runtime Performance. In
booktitleProceedings of the 2018 ACM/SPEC International
Conference on Performance Engineering, ICPE 2018, Berlin, Germany, April
09-13, 2018, editorpersonKatinka Wolter,
personWilliam J. Knottenbelt, personAndré van
Hoorn, and personManoj Nambiar (Eds.).
publisherACM, pages48–55.

[Chen, Bahsoon, and YaoChen
et al2018a]
authorpersonTao Chen, personRami Bahsoon,
and personXin Yao. year2018a.
A Survey and Taxonomy of Self-Aware and
Self-Adaptive Cloud Autoscaling Systems.
journalACM Comput. Surv. volume51,
number3 (year2018), pages61:1–61:40.

[Chen, Bahsoon, and YaoChen
et al2020]
authorpersonTao Chen, personRami Bahsoon,
and personXin Yao. year2020.
Synergizing Domain Expertise With Self-Awareness in
Software Systems: A Patternized Architecture Guideline.
journalProc. IEEE volume108,
number7 (year2020), pages1094–1126.

[Chen, Li, Bahsoon, and YaoChen
et al2018d]
authorpersonTao Chen, personKe Li,
personRami Bahsoon, and personXin Yao.
FEMOSAA: Feature-Guided and Knee-Driven
Multi-Objective Optimization for Self-Adaptive Software.
journalACM Trans. Softw. Eng. Methodol.
volume27, number2 (year2018),

[Chen and LiChen and Li2021]
authorpersonTao Chen and personMiqing
Li. year2021.
Multi-objectivizing software configuration tuning.
In booktitleESEC/FSE '21: 29th ACM Joint European
Software Engineering Conference and Symposium on the Foundations of Software
Engineering, Athens, Greece, August 23-28, 2021,
editorpersonDiomidis Spinellis,
personGeorgios Gousios, personMarsha Chechik, and
personMassimiliano Di Penta (Eds.). publisherACM,

[Chen and LiChen and Li2023a]
authorpersonTao Chen and personMiqing
Li. year2023a.
Do Performance Aspirations Matter for Guiding
Software Configuration Tuning? An Empirical Investigation under Dual
Performance Objectives.
journalACM Trans. Softw. Eng. Methodol.
volume32, number3 (year2023),

[Chen and LiChen and Li2023b]
authorpersonTao Chen and personMiqing
Li. year2023b.
The Weights Can Be Harmful: Pareto Search versus
Weighted Search in Multi-objective Search-based Software Engineering.
journalACM Trans. Softw. Eng. Methodol.
volume32, number1 (year2023),

[Chen and LiChen and Li2024]
authorpersonTao Chen and personMiqing
Li. year2024.
Adapting Multi-objectivized Software Configuration
journalFSE'24: Proceedings of the ACM on Software
Engineering (PACMSE) volume1, numberFSE.

[Chen, Li, and YaoChen
et al2018c]
authorpersonTao Chen, personMiqing Li,
and personXin Yao. year2018c.
On the effects of seeding strategies: a case for
search-based multi-objective service composition. In
booktitleProceedings of the Genetic and Evolutionary
Computation Conference, GECCO 2018, Kyoto, Japan, July 15-19, 2018,
editorpersonHernán E. Aguirre and
personKeiki Takadama (Eds.). publisherACM,

[Chen, Li, and YaoChen et al2019]