# Specialization of Run-time Configuration Space at Compile-time: An
Exploratory Study

Xhevahire Tërnava  Univ Rennes, CNRS, Inria, IRISAF-35000 RennesFrance
<EMAIL_ADDRESS>,  Mathieu Acher  Univ Rennes, CNRS, Inria, IRISA
Institut Universitaire de France (IUF)F-35000 RennesFrance
<EMAIL_ADDRESS>and  Benoit Combemale  Univ Rennes, CNRS, Inria,
IRISAF-35000 RennesFrance<EMAIL_ADDRESS>
(2018)

###### Abstract.

Numerous software systems are highly configurable through run-time options,
such as command-line parameters. Users can tune some of the options to meet
various functional and non-functional requirements such as footprint,
security, or execution time. However, some options are never set for a given
system instance, and their values remain the same whatever the use cases of
the system. Herein, we design a controlled experiment in which the system’s
run-time configuration space can be specialized at compile-time and
combinations of options can be removed on demand. We perform an in-depth study
of the well-known x264 video encoder and quantify the effects of its
specialization to its non-functional properties, namely on binary size, attack
surface, and performance while ensuring its validity. Our exploratory study
suggests that the configurable specialization of a system has statistically
significant benefits on most of its analysed non-functional properties, which
benefits depend on the number of the debloated options. While our empirical
results and insights show the importance of removing code related to unused
run-time options to improve software systems, an open challenge is to further
automate the specialization process.

program specialization, performance, run-time variability, configuration
options, debloating

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††copyright: none

##  1\. Introduction

Modern software systems are highly configurable and expose to the users their
abundant configuration options. By setting configuration options’ values, a
software system is customized for different users to reach specific
functionality or performance goals (_e.g._ , execution time, energy
consumption, quality of the result) without the need to modify its source
code. Such systems have a large and different number of options. A
considerable number of their options are set at run-time, for example, a
recent version of curl has about $205$ compile-time options and $242$ run-time
options 111Documentation of curl: https://curl.se/., then the x264 video
encoder has $39$ compile-time options and $162$ run-time options
222http://www.chaneru.com/Roku/HLS/X264_Settings.htm.. author=R3,inline,
linecolor=orange!50,backgroundcolor=orange!50,bordercolor=orange!50,]l. 72f It
would be interesting to know, how many (run-time + compile-time + ..) options
these tool have in total. Are most of them run-time options? Run-time options
are typically accessible via command-line parameters, configuration files, or
menu preferences. Such run-time variability is a strength since system’s users
have the flexibility to tune their systems, owing to their specific use case
and deployment constraints.

However, _(i)_ there is evidence that ”a significant percentage (up to
$54.1\%$) of parameters are rarely set by any user” (Xu et al., 2015) and thus
they unnecessarily ”bloat” the software. author=R2,inline,
linecolor=red!50,backgroundcolor=red!50,bordercolor=red!50,]The overall
motivation seems to be ”there are a lot of options and they are rarely used”.
I find this motivation somewhat problematic. On the one hand, yes, great, it
gives context to which options may or may not be worth keeping/maintaining.
Though, from evolving programming languages, and perhaps more specifically
APIs, we know that ”every feature will be used by someone”. Thus, that there’s
not direct openly accessible evidence that something is used, is a very weak
signal. Any deprecation, whether language feature or API, or even the change
of internal, undocumented, unspecified behavior will break some users. Thus,
for widely adopted systems it’s never ”oh this isn’t used”, but rather a cost-
benefit calculation. Do we gain enough reduction in maintenance reduction,
performance improvement, or similar to justify breaking some users? Some
options are never tuned and their values remain the same for a given usage,
for example, depending on the user, the performed activities, and the targeted
environment. For instance, a user or an organization may choose always to
encode a video with x264 in an (ultra-)fast way. In this case, its run-time
configuration option of --cabac is always disabled, as it offers better
compression, but requires more processing power to encode and decode a video.
Apart from that, _(ii)_ it is common to think that a library or a
configuration option that is unused today can be used in the future. In fact,
quite the opposite is true. Pieces of evidence show that the libraries that
are unused today in a software system (86% of them) are unlikely to be ever
used in the future (Soto-Valero et al., 2021). Then, the unused
functionalities in a software system can threaten its security, slow down its
performance, affect its reliability, or increase its maintenance (Qian et al.,
2020). Thus, run-time variability is sometimes unnecessary and does not have
to be embedded.

Based on this observation, we bring up the idea of _specializing the run-time
configuration space_ of a software system. The goal is to retain only a subset
of configurations that meet a functional or a performance requirement and thus
discard the rest. Specifically, we aim to debloat run-time options that are
never used within the source code, at compile-time. author=R3,inline,
linecolor=orange!50,backgroundcolor=orange!50,bordercolor=orange!50,]The
paper, in most parts, describes the advantages of compile-time variability
over run-time variability. However, there could be more room for describing
the drawbacks, e.g., managing and distributing the variants of the compiled
tools and the need for recompiling the tool to choose an option that was not
intended/foreseen to be chosen Which options are unused depend on the system’s
usage context and they are inputs to the specialization process. Our
hypothesis is that the code of unused run-time options is a manifestation of
code bloat that may increase the binary size, the attack surface, or slow down
the system. author=R1,inline,
linecolor=purple!50,backgroundcolor=purple!50,bordercolor=purple!50,]I did not
fully understand how the quality criteria for evaluation have been chosen? Why
is attack surface an important consideration for a video encoder? For
instance, options that are never used induce dead code that can be eliminated
(_e.g._ , by a compiler), thus improving the overall system. This paper aims
to investigate this hypothesis and explore to what extent removing run-time
options can bring benefits to the non-functional properties of software. To
the best of our knowledge, quantifying the benefits (if any) has received
little attention.

Specializing the configuration space of a software system has a long tradition
since the seminal paper of Czarnecki  _et.al._ , (Czarnecki et al., 2005b, a)
and others (_e.g._ , (Hubaux et al., 2013; Amand et al., 2019; Acher et al.,
2018; Harman et al., 2014; Temple et al., 2016; Capilla and Bosch, 2011;
Temple et al., 2017; Martin et al., 2021)). However, most of the works focused
on the specialization of variability models where constraints among individual
options (or across several options) are added to enforce the configuration
space. A missing part we investigate in this paper is to propagate this
specialization to the source code of a configurable system. Specialization can
also be seen as a debloating problem where a subset of run-time options to
specialize are bloat of a configurable system. Several debloating approaches
are proposed in the literature. Most of them provide a way for debloating the
unused functionalities from the external libraries or from high-level
”features” of a system (Jiang et al., 2018; Sharif et al., 2018; Quach and
Prakash, 2019). Yet, to the best of our knowledge, debloating run-time options
of the configurable systems has received little attention. An important
specificity of our problem is that the specialization should be flexible and
is itself configurable. That is, not all options are set once and for all: in
contrast, existing debloating techniques specialize the system under study
with a full configuration. author=R1,inline,
linecolor=purple!50,backgroundcolor=purple!50,bordercolor=purple!50,]On page
2, the authors differentiate their proposal from existing work like so: ”That
is, not all options are set once and for all: in contrast, existing debloating
techniques specialize the system under study with a full configuration.” I
don’t understand how this is a significant and substantive difference? What
changes substantively when only a subset of the existing options is
considered? Our proposal is to annotate run-time options with compilation
directives throughout the source code. As such, the configuration space of
run-time options can be specialized at compile-time and (combinations of)
options can be removed on-demand. In the future, the support for automation of
these processes asks for software language engineering (SLE) techniques.

To realize the idea of specializing a configuration space, several new
challenges need to be addressed, such as (i) to locate the run-time options
within the source code and (ii) to take care of the system’s validity after
its specialization. Our research methodology is to statically annotate run-
time options to mitigate the risk of synthesizing an unsound and incomplete
specialization. In this way, this controlled effort limits the introduction of
errors that could bias the benefits (if any) of specialization on non-
functional properties. In addition, we establish a ground truth for future
automatic program specializers and we report on insights from our experience
on the case of x264.

The contributions of this paper are as follows:

  * •

We propose a means for configuring the specialization of a software system
through debloating its unused run-time configuration options at compile-time;

  * •

We analyze the resulting non-functional properties (_i.e._ , binary size,
attack surface, and performance) under two specialization scenarios of x264,
showing improvements of the x264 configurable system without sacrificing its
interoperability and validity;

  * •

We made available the data for reproducing the experiments and call to
replicate our study for further confirming or refuting our results
333Companion page: https://anonymous.4open.science/r/x264-6ED7.

##  2\. Background and motivating case study

In this section, we introduce the basic concepts of configurable software
systems and the system used as a case study.

author=R2,inline,
linecolor=red!50,backgroundcolor=red!50,bordercolor=red!50,]Throughout reading
the paper, I was also wondering e.g. ”but isn’t this what software product
lines do” and ”how does this all relate to slicing etc”. The paper may benefit
from moving some of the related work techniques to the background section so
that the reader may know early on how these techniques relate to the proposed
work.

###  2.1. Background

Users can configure a software system by setting the values of its numerous
options. author=R3,inline,
linecolor=orange!50,backgroundcolor=orange!50,bordercolor=orange!50,]It is not
entirely clear what the assumptions on the run-time options to apply the
proposed procedure are.
\- What about cross-influences between options? (e.g., –foo can only be chosen
if –bar is chosen, too)
\- What about ”non-binary” options, such as, –low, –medium, –high, where only
one can be chosen?
\- What about non-Boolean options of a program (e.g., -size 64)? Sec. 2.1 does
not exclude them but the example only contains Boolean options and the
algorithm is limited to Boolean options, too. Usually, an option carries a
particular functionality, has a type of value (namely, Boolean, integer, or
string), and a binding time (namely, compile-time or run-time). A
_configuration_ is an assignment of values to options at their specific
binding time. Default values are also assigned to some options. Hence, users
can build their custom software by (de-)activating some options at compile-
time or run-time, but not only at these times (Bosch and Capilla, 2012). The
interest of compile-time options is to embed in the resulting executable
program what is necessary for a given use case. Typically, preprocessor
directives (_e.g._ , #ifdef in C/C++ projects) are used to implement them. As
for the run-time options, they are (de-)activated prior to, or even during,
the execution. Command-line parameters, plugins, or configuration files are
examples of mechanisms that users can rely on to control them. In this work,
we considered the run-time options, which are set as command-line parameters
during the software’s load-time. Within the source code, run-time options are
implemented by ordinary control statements, such as if statements, and
locating them in code can be challenging as they are implicit. Nowadays
software systems, such as x264 video encoder, are configurable and often
provide compile-time and run-time options to customize them.

###  2.2. Motivating case study

As a motivating system for this study is chosen the software system of x264
444http://www.chaneru.com/Roku/HLS/X264_Settings.htm. It is a command-line
video encoder implemented in C, which has been used for the past 5 years to
evaluate the debloating approaches (Soto-Valero, 2021), is studied among the
highly configurable systems (_e.g._ , (Guo et al., 2013; Siegmund et al.,
2015; Temple et al., 2017; Jamshidi et al., 2017; Alves Pereira et al., 2020;
Lesoil et al., 2021a)), and has plenty of run-time options. We use the x264 at
its recent commit db0d417 555x264 software system:
https://github.com/mirror/x264. It has 114,475 lines of code (LoC) 666Measured
using the clocl tool: https://github.com/AlDanial/cloc, 237 files, 39 compile-
time options, and 162 run-time options. For example, it is possible to
deactivate the support of mp4 format at compile-time by using the --disable-
lsmash option during the system build, as in the following.

⬇

$ ./configure \--disable-lsmash && make

After the system’s build, the run-time options can be set. They have an effect
on the properties of the encoded video, namely on its encoding time, bitrate,
and frame rate. Here is a possible usage of three run-time options in x264.

⬇

$ x264 \--cabac \--mbtree \--mixed-refs -o vid.264 vid.y4m

The used --cabac, --mbtree, and --mixed-refs options are Boolean. They also
have their variant to negate their functionality, for example, --mbtree has
--no-mbtree. author=R3,inline,
linecolor=orange!50,backgroundcolor=orange!50,bordercolor=orange!50,]l. 230ff:
Why is it important to mention that the Boolean options have negated variants?
Why are the negated variants needed? E.g., what is the difference between
choosing –no-mbtree and choosing neither –mbtree nor –no-mbtree?

x264 is a free software application for encoding video streams into the H.264
compression format. It is often used as a library in large systems, such as in
space flight hardware 777AVN443 encoder: https://www.tvbeurope.com/production-
post/visionary-provides-hd-encoders-for-international-space-station. where its
_(i)_ binary size and _(ii)_ performance matters. Besides, _(iii)_ a
vulnerability in the x264’s (H.264 decoder) function could be used to attack a
larger system, such as the case with the Cisco Meeting Servers 888 A
vulnerability: https://www.cvedetails.com/cve/CVE-2017-12311/.. Nowadays, the
security of modern software systems is mostly threatened internally, that is,
by reusing their existing code, without the need for code injection (Sharif et
al., 2018). This kind of attack allows an attacker to execute arbitrary code
on a victim’s machine. In this attack, the attacker is able to chain some
small code sequences (called gadgets) and threaten the security of the system.
Basically, the exploited code sequences by the attacker end in a return (RET
or JMP) instruction. Therefore, one of the commonly used metrics for measuring
the attack surface in a system is the number of code reuse gadgets that are
available and which can be exploited by an attacker (Brown and Pande, 2019;
Kurmus et al., 2011; Sharif et al., 2018).

Motivated by similar requirements, we analyse the effects of specializing x264
on these three non-functional properties, namely on its binary size, attack
surface, and performance.

##  3\. Specialization approach

author=R3,inline,
linecolor=orange!50,backgroundcolor=orange!50,bordercolor=orange!50,]The paper
could describe more clearly (e.g., as a table/figure?) the effect of the
preprocessor directives. For example: OPTION_YES 1, OPTION_NO 1: run-time
option not set during compile-time

###  3.1. The vision for debloating unused run-time options at compile-time

At the system level, the usual approach is to keep as much variability as
possible. All run-time options may somehow be needed one day. Furthermore,
packages, binaries, or build instructions force users to take them all.
However, this variability is sometimes unnecessary: in a given context, some
options may never be tuned and thus they always have the same value for all
actual use cases of a given instance of the software. For instance, --cabac
might always have the _true_ value. In this case, its corresponding variable
in the source code becomes a constant. By setting the option of --no-cabac
will speed up the video encoding time, but the video quality may deteriorate.
Hence, the tune of run-time options depends on the objectives and constraints
of the user. Keeping an option while actually always using the same value
leads to missed opportunities. First, by knowing that an option has the same
value, compilers do not have to make some assumptions and the generated code
can be improved. Specifically, compilers can further apply optimizations and
constant folding, including the propagation of constants and the removal of
dead code. Secondly, the presence of unused options at run-time is likely (i)
to increase the executable binary size, since the related code will be
included; (ii) can increase the attack surface (_e.g._ , the number of code-
reuse gadgets (Kurmus et al., 2011)); and (iii) even slow down the system. The
vision is that the proactive removal of the unused run-time options can be
beneficial to several non-functional properties of software.

To technically support this vision, developers should have the means to
debloat run-time options, for example, by lifting them at compile-time, and
thus specialize the original program. With configurable specialization,
developers can build specialized packages (binaries) out of the specialized
program. For example, one can envision delivering new variants of x264:
_x264-fast_ – with fast encoding time, _x264-hq_ – with high quality video
encoding, _x264-secured_ – with fewer code-reuse gadgets, or _x264-tinyfast_ –
a small binary size yet fast x264. Importantly, they do not necessarily
specify a full and complete configuration: not all options are subject to
specialization since a part of the variability can be preserved at run-time
for addressing several usages. But, some of the unneeded options that may
improve the non-functional properties and the performance of the system can be
eliminated from code. Our idea is inspired by the multi-staged configuration
(Czarnecki et al., 2004; Hubaux et al., 2013) for a step-wise specialization
of variability models. In our case, specialization occurs at the code level
and regarding the run-time options.

###  3.2. Process for program specialization

Figure 1. Overall process for debloating run-time options through program
specialization

To quantify the benefits of such a vision, we report in this paper about our
experiments in debloating run-time options of x264 through program
specialization (_cf._ Figure 1). To achieve our vision, one should be able to
specialize x264 by removing run-time configuration options on demand. This
implies removing the parts of source code that implement such options, which
is different from simply disabling those options in a configurator or at the
command-line parameter level. Moreover, to be able to remove the unused run-
time options, they first need to be located in the source code.
author=R1,inline,
linecolor=purple!50,backgroundcolor=purple!50,bordercolor=purple!50,]I did not
understand the difference between T1 and T2. Hence, we propose a removal
approach which is based on 4 tasks: (T1) _we first comprehend the
implementation of run-time options_ , then (T2) _we annotate the corresponding
code which implement a given run-time option_ , and finally (T3) _we apply a
partial evaluator_ taking care of the actual code removal. In addition to
these three tasks, we conducted the last task (T4) which consists in
_validating the program specialization_ by comparing the resulting debloated
program with the original one according to a given configuration of run-time
options.

The experiments conducted for these 4 tasks in the context of run-time options
in C-based systems, taking x264 as an illustrative example, are presented in
the following. As a partial evaluator, we decided to use the preprocessor
switches (Baxter and Mehlich, 2001) and all the optimizations (_e.g._ , dead
code elimination) provided by GCC (in grey in Figure 1). A partial evaluator
is a method for simplifying programs when program parameters are known (Jones
et al., 1993; Baxter and Mehlich, 2001). For instance, when it is known that a
run-time configuration option in a given system will always be unset. In this
study, the program parameters are the given run-time options for removal
whereas their removal and code optimization is completed by the GCC compiler.
Hence, the annotations of the source code consist in adding the required C
preprocessor directives to let GCC remove the code corresponding to the given
run-time options. Moreover, the validation of the program specialization is
not realized at the program level but at the binary level after the complete
compilation of the original program.

####  T1: Comprehending the implementation of run-time options

To debloat unused run-time options later, we first had to comprehend their
implementation within the codebase. During this process, we identified the
following patterns how they are handled in the source code. (A) The x264, and
most of the C-based systems, uses the getopt.h
999https://www.gnu.org/software/libc/manual/html_node/Getopt.html which has
the functionalities to automate some of the chores involved in parsing typical
Unix command-line options. (B) A corresponding variable is usually used to
represent a run-time option. In this case, we manually analyse the data-flow
corresponding to this variable and locate all parts of the code used to
implement that particular run-time option. (C) Some functions were dedicated
to handling a single run-time option. Hence, removing all the functions calls
will eventually lead to the removal of the complete functions considered as
dead code. (D) Then, some of the options have dependencies, for example,
--cabac and --no-cabac have an alternative logic. This suggests that x264 will
be unsound if both these options are removed. In order to provide inputs for
future works about automation (see Section 7), these first experiments have
been conducted manually to identify the patterns how run-time options are
handled within a codebase.

####  T2: Annotating source code

author=R1,inline,
linecolor=purple!50,backgroundcolor=purple!50,bordercolor=purple!50,]Given
that evry option is captured in a variable in the C code, would it not be
simpler to just replace the variable with a constant and let the compiler do
the rest? author=R2,inline,
linecolor=red!50,backgroundcolor=red!50,bordercolor=red!50,]l.97 ”partial
evaluation”: the term is used throughout again and again, but by my
understanding you are simply using the preprocessor, and do not actually have
an actual partial evaluator applied to the code, beside the standard
preprocessor/compiler combination. The problem here is that my expectations
are set at the level: ”oh, they are going to do something complicated, which
might solve some very specific problems”, but in the end it’s just that a big
word is used for relying on the standard compiler. This doesn’t seem an ideal
way to present this work. We now present our approach of removing unused run-
time options. First, we lift run-time options to compile-time through
annotating them with preprocessor directives. Then, the objective is to let
the preprocessor and optimizations of GCC remove the source code related to
the values of a given run-time option. A way to make explicit options’ values
is to annotate them in places where they are implemented (_i.e._ , to locate
them). For this purpose, we propose to use C preprocessor directives, namely,
#define, #if, and #endif. They are special instructions directed to the
preprocessor, which is part of the compiler, in order to be processed. Because
of their prominent and widespread nature, we chose them as they are pretty
light to learn, use, and make it possible to remove the enclosed code.

Algorithm 1 Delimiting a Boolean run-time option

1:compilable $\theta$ C-based system with run-time options

2:compilable $\theta$ with delimited run-time option(s)

3:$removeoption.h\in source\\_files$ in $\theta$

4:for option $\in$ $\theta_{runtime\\_options}$ do

5: for option $=true,false$ do // _e.g._ , _true_ = --cabac, _false_ = --no-
cabac

6: Within $removeoption.h$

7: $\\#$define OPTION_YES 1 // _e.g._ , $\\#$define CABAC_YES 1

8: $\\#$define OPTION_NO 1 // _e.g._ , $\\#$define CABAC_NO 1

9: Set OPTION_YES 0 $\Rightarrow$ $\theta_{specialized}\subset\theta$

10: repeat

11: Find code that implement its $true$ value

12: Enclose it within $\\#$if OPTION_YES...$\\#$endif

13: until $\theta_{specialized}$ is valid

14: Set OPTION_YES 1,

15: Set OPTION_NO 0 $\Rightarrow$ $\theta_{specialized}\subset\theta$

16: repeat

17: Find code that implement its $false$ value

18: Enclose it within $\\#$if OPTION_NO...$\\#$endif

19: until $\theta_{specialized}$ is valid

20: Set OPTION_NO 1

21: end for

22:end for

⬇

1/* File removeoption.h */

2#ifndef CABAC_YES

3#define CABAC_YES 0

4#endif

5#ifndef CABAC_NO

6#define CABAC_NO 1

7#endif

8/* The rest of the directives are omitted */

⬇

1/* File encoder/encoder.c */

2/*The previous code is omitted*/

3#if CABAC_YES //option \--cabac

4 if( h->param.b_cabac )

5 x264_cabac_init( h );

6#endif

7#if CABAC_YES && CABAC_NO

8 else

9#endif

10#if CABAC_NO //option \--no-cabac

11 x264_cavlc_init( h );

12#endif

13/*The rest of the code is omitted*/

⬇

1/* File common/cabac.c */

2/*The previous code is omitted*/

3#if CABAC_YES

4void x264_cabac_init( x264_t *h ) {

5 int ctx_count = CHROMA444 ? 1024 : 460;

6 for( int i = 0; i < 4; i++ ) {

7 // 8 lines of computation

8 }

9}

10#endif

11/*The rest of the code is omitted*/

Figure 2. An excerpt from the removeoption.h file with 2 defined preprocessor
directives (left). The encoder.c file where --cabac and --no-cabac are
implemented (middle). The cabac.c file where the directive is used at the
function level (right)

author=R1,inline,
linecolor=purple!50,backgroundcolor=purple!50,bordercolor=purple!50,]I
struggled with a lot of the notation in Algorithm 1. What does ”Set OPTION_YES
1” mean, especially when a bit later we have ”Set OPTION_YES 1”? What does
Line 6 mean?

_Algorithm 1_ shows the manual approach that we conducted to delimit Boolean
run-time options in x264, one at a time, by annotating their true and false
values. First, we make sure that the system is compilable. A removeoption.h
file is then added. It contains the defined preprocessor directives, which are
used to annotate and remove the desired run-time options. Hence, for each run-
time option that is going to be removed (lines $1-2$, $18-19$), we add a new
preprocessor directive (in lines $4-5$ are added two, _e.g._ , for --cabac and
--no-cabac), which directives can take the value $1$ or $0$. Then, we set the
directive of the first option to $0$ (line $6$) and annotate its respective
code (lines $7-10$). To make sure that all parts of code that are relevant to
an option are annotated, we check the validity of the given software system
after removing each option (lines $10$,$16$). As soon as the system is valid
(_i.e._ , we annotate all parts of code that implement the first option, and
only those parts), we set the value to $1$ (line $11$). Then, in lines $12-17$
we repeat the same process, but now for the second option. The algorithm is
repeated for as many Boolean options as it is necessary and it can be extended
for not Boolean options. The process ends as soon as the system after the
option’s removal is valid.

_Illustrative example._ To illustrate Algorithm 1, we provide a concrete
example from the x264 system. x264 has more than one hundred run-time options,
one of which is the important Boolean option of --cabac. In x264, both values
of a Boolean run-time option use different names, such as --cabac and --no-
cabac, therefore we will refer to them as run-time options instead of values
of an abstract option. In Figure 2 (middle) is shown a code snippet that
implements these options. It has a conditional if statement with two branches
where both options --cabac and --no-cabac are involved. When the condition is
true (lines $4-5$), that is, the user chose to encode a video with the option
of --cabac, then the function on the second branch for --no-cabac (line $11$)
becomes dead code and thus can be removed. Further, a compiler can also
realize that some functions inside it are never called and will further
optimize the code.

author=R2,inline,
linecolor=red!50,backgroundcolor=red!50,bordercolor=red!50,]When reaching Fig.
2, I was wondering why the if statement on line 4 hasn’t been considered.
While I don’t assume that this specific statement has any performance impact,
it’s unclear to me that the way the preprocessor has been used is really going
to extract all performance benefits possible. For this branch for instance,
I’d assume that the true branch is always taken when CABAC_YES=1 and
CABAC_NO=0.

In order to locate and remove the related code of an unused option, as it is
--no-cabac, we proposed Algorithm 1. Specifically, we first make sure that the
system of x264 is compilable. We then add the removeoption.h file and two
directives, namely CABAC_YES and CABAC_NO, as shown in Listing 2 (left).
Further, each of them are used to annotate all lines of code related to
options --cabac and --no-cabac, respectively, as is given in Listing 2
(middle) in lines $3$, $6-7$, $9-10$, and $12$. We used preprocessor
directives at different granularity levels and in different combinations. In
Listing 2 (middle) and (right) is shown their usage to annotate the related
code of --cabac and --no-cabac at statement and function levels, respectively.
In this way, we continue to annotate statically the related parts of code for
one option at a time until the system of x264 becomes again compilable and
valid. We check the validity of the program when an option is removed, that
is, of its specialized version. author=R4,inline,
linecolor=cyan!40,backgroundcolor=cyan!40,bordercolor=cyan!40,]The decision to
add preprocessor annotations for conditional compilation but to not modify the
code any further [line 463] should be justified and discussed better. In
particular, it seems that ironing out some leftover conditional statements
that now refer to a constant value, could be a good source for optimisation,
and it can be done automatically by partially evaluation ‘if‘ conditions. It
should be emphasized that, while we add the run-time options’ annotations, we
kept unmodified the existing code of the system. For this reason, we add two
directives in combination as in lines $7$ and $9$ in Listing 2 (middle). This
way of annotating allows the user to remove one of the options by setting the
values $(0,1)$ or $(1,0)$ in Listing 2 (left) for each directive respectively.
Setting both directives to $0$ is unacceptable as at least one of the options
must be part of the system. Whereas, when both directives are set to $1$ then
both options are kept available at run-time.

The part of the approach to ensure system validity and doing its automated
specialization are given in the following.

####  T3 & T4: Debloating validation

The added preprocessor directives make it possible to remove an unused option
in a given context, and thus to specialize the entire given program. But, when
annotating an option, we have to make sure that all relevant parts in the
source code of a given option are annotated, otherwise removing only some of
its lines of code will make the system uncompilable. Then, being only
compilable is not enough. The program must also be valid, meaning that the
remained options in the program should be intact and the system delivers the
same functionality as before when the option was part of the code but
unset/disabled. Hence, we check program validity during the annotation of each
option. In particular, we check whether the program after removing a given
option (_e.g._ , --cabac) (A) is compilable, (B) preserves the rest of its
behavior, and (C) is interoperable, that is, when a user tries to use the
removed option then the system will provide a pertinent warning message.

The whole validity check in x264 is automated. First, the original and
specialized program are used to encode eight carefully chosen videos (_cf._
Section 4.4) by the ten built-in presets 101010A preset is a set with run-time
configuration options that are set at once. in x264 and record the resulting
video sizes (A). author=R2,inline,
linecolor=red!50,backgroundcolor=red!50,bordercolor=red!50,]l.604 ”10
presets”: here I was wondering whether these are built into x264 or provided
by you. It becomes a little more clear later, but confused me here. In the
next check (B), if the size in bytes of each encoded video by the original and
specialized program is exactly the same and if using the removed option
(_e.g._ , of --cabac) at run-time in the specialized program is not possible
anymore we then conclude that the specialized program is valid. In case that
at least one of these two conditions is false, then the program is invalid,
meaning that the annotated option (_e.g._ , of --cabac) needs to be further
improved. Regarding the interoperability of the specialized program (C), for
example, trying to encode a video by setting the removed option of --cabac
will show a warning message, which notifies the user that the option of
--cabac is no longer available. The validation process is specific to our
program of interest, here x264. However, the methodology is general and
essentially requires to develop a testing procedure (_e.g._ , oracle (Bruce et
al., 2018; Barr et al., 2014)) capable of comparing two programs (_e.g._ ,
their outputs) – the original and the specialized ones.

##  4\. Experimental design

We outline the design of our experiment via the hypothesis, system
specialization scenarios, and experimental settings.

###  4.1. Hypothesis

We present three hypotheses each to be supported or refuted by our analysis.
The first hypothesis concerns the binary size, the second one the attack
surface, and the third one the performance of a software system. Our null
hypotheses are labeled $H_{01}$, $H_{02}$, and $H_{03}$, respectively. Their
respective alternative hypotheses are labeled $H_{A1}$, $H_{A2}$, and
$H_{A3}$.

$\bm{H_{01}:}$:

The baseline software system and its specialization are not significantly
different with respect to their binary size.

$\bm{H_{A1}:}$:

The specialized software system has smaller binary size than its baseline.

$\bm{H_{02}:}$:

The baseline software system and its specialization are not significantly
different with respect to their attack surface (_i.e._ , $\\#$ code reuse
gadgets).

$\bm{H_{A2}:}$:

The specialized software system has smaller attack surface (_i.e._ , $\\#$
code reuse gadgets) than its baseline.

$\bm{H_{03}:}$:

The baseline software system and its specialization do not perform
significantly different with respect to their encoding time, bitrate, and
frame rate.

$\bm{H_{A3}:}$:

The specialized software system performs better than its baseline with respect
to their encoding time, bitrate, and frame rate.

To statistically evaluate our hypotheses, we build up to 20 specializations of
x264. Then, we measure their binary size, attack surface (_i.e._ , $\\#$ code
reuse gadgets), and performance with respect to the encoding time, bitrate,
and frame rate of 8 input videos. Finally, we do a statistical hypothesis
testing by using the Wilcoxon signed-rank test (Wilcoxon et al., 1963), which
is one of the most used statistical test in software engineering (Dybå et al.,
2006). We reject any of the null hypothesis at a common significance level of
$\alpha=0.05$ and do one sided (”greater”/”less”) testing.

###  4.2. Baseline system

The system of x264 comes with a default configuration, that is, each of its
compile-time and run-time options is either enabled or disabled by default. In
the case of external libraries, provided as compile-time options, x264 has six
external libraries which are enabled by default. Therefore, for our study, we
installed all of them in our used environment. Aside from this, the other
compile-time and run-time options are kept to their default values. We refer
to such an x264’s version with default configuration as the baseline system.
Put it differently, the baseline system has the current configuration that it
has in its respective git repository (_cf._ its commit, Section 2.2). In the
following, we will refer to its measured properties as the baseline binary
size, baseline number of gadgets, baseline encoding time, baseline bitrate,
and baseline frame rate.

Table 1. Run-time options and presets to specialize x264 Options Presets |  ultrafast |  superfast |  veryfast |  faster |  fast |  medium |  slow |  slower |  veryslow |  placebo
---|---|---|---|---|---|---|---|---|---|---
|  $S_{11}$ |  $S_{12}$ |  $S_{13}$ |  $S_{14}$ |  $S_{15}$ |  $S_{16}$ |  $S_{17}$ |  $S_{18}$ |  $S_{19}$ |  $S_{20}$
--no-mixed-refs $S_{1}$ |  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$ |
$\circ$ |  $\circ$ |  $\circ$ |  $\circ$ |  $\circ$ |  $\circ$
--no-mbtree $S_{2}$ |  $\bullet$ |  $\bullet$ |  $\circ$ |  $\circ$ |  $\circ$
|  $\circ$ |  $\circ$ |  $\circ$ |  $\circ$ |  $\circ$
--no-cabac $S_{3}$ |  $\bullet$ |  $\circ$ |  $\circ$ |  $\circ$ |  $\circ$ |
$\circ$ |  $\circ$ |  $\circ$ |  $\circ$ |  $\circ$
--no-weightb $S_{4}$ |  $\bullet$ |  $\circ$ |  $\circ$ |  $\circ$ |  $\circ$
|  $\circ$ |  $\circ$ |  $\circ$ |  $\circ$ |  $\circ$
--no-psy $S_{5}$ |  $\circ$ |  $\circ$ |  $\circ$ |  $\circ$ |  $\circ$ |
$\circ$ |  $\circ$ |  $\circ$ |  $\circ$ |  $\circ$
--mixed-refs $S_{6}$ |  $\circ$ |  $\circ$ |  $\circ$ |  $\circ$ |  $\bullet$
|  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$
--mbtree $S_{7}$ |  $\circ$ |  $\circ$ |  $\bullet$ |  $\bullet$ |  $\bullet$
|  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$
--cabac $S_{8}$ |  $\circ$ |  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$
|  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$
--weightb $S_{9}$ |  $\circ$ |  $\bullet$ |  $\bullet$ |  $\bullet$ |
$\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$
--psy $S_{10}$ |  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$ |
$\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$

###  4.3. System specialization

The following are the sample of used run-time options to specialize x264 and
its two specialization scenarios.

#### The sample of run-time options.

As given in Section 2.2, x264 has $162$ run-time configuration options and 10
built-in presets. A preset is a group of run-time options that are commonly
set to quickly encode a video. In x264, only $22$ options are used to build
its $10$ presets. For this exploratory study, we chose a sample of $10$ run-
time options in x264 that are mainly part of these presets and may help thus
to address our hypothesis. In Table 1 are given the $10$ available presets in
x264 and the presence of the $10$ chosen run-time options into them. In
particular, we chose $9$ options that are part of at least one preset, $1$
option that is not a part of any preset (--no-psy), and $1$ option that is not
part of the presets (--psy). For example, the preset ultrafast can be used as
in the following and it includes up to $17$ options.

⬇

$ x264 \--preset=ultrafast -o <output_video> <input_video>

We have chosen to analyze $5$ of its options, which will be set at once by
this preset and hence marked with filled circles ($\bullet$) in Table 1. The
$5$ other chosen options, namely --mixed-refs, --cabac, --mbtree, --weightb,
and --no-psy, we have chosen to specialize the system of x264 regarding this
preset. These options are marked with unfilled circles ($\circ$) in Table 1,
meaning that they will remain unset in the case when users use x264 to always
encode videos with the ultrafast preset. In the companion page we provide a
table with details regarding the set and unset options’ directives in each
specialization.

#### Independent and dependent variables.

To conduct our experiment, we have as control variables: the compile-time
options in the subject system, the number and kind of used videos, and the
used environment. Then, we can identify the independent variable: the remained
run-time options in x264 after its specialization. The dependent variables for
the study were the measures of binary size, number of code reuse gadgets,
encoding time, bitrate, and frame rate.

#### Specialization scenarios.

By using the 10 chosen run-time options, we follow two scenarios for
specializing x264.

author=R1,inline,
linecolor=purple!50,backgroundcolor=purple!50,bordercolor=purple!50,]How were
the 20 specialisations chosen?

####  $Scenario_{1}$:

x264 is specialized by removing one run-time option at a time. With the $10$
considered options, there are $10$ possible specializations of x264. They
consist of the specializations regarding one of the 10 run-time options
(_i.e._ , horizontal aspect) given in the first column of Table 1.

####  $Scenario_{2}$:

x264 is specialized by removing one preset at a time. In this case, we
specialize x264 regarding the $10$ presets by removing the set of unused
options within each specific preset. Based on Table 1, they consist of the
vertical specializations. For instance, towards an x264-fast version, x264
will be specialized regarding the fast preset by removing its $5$ unused
options, which are marked with ”$\circ$”. Hence, in this scenario, there are
$10$ other possible specializations of x264. But, they can be grouped into
four specializations, for example, veryfast and faster have the same used and
unused options and thus both specializations look the same. Despite this, we
still analyze them separately as their performance may differ because of their
other (un)set options that are different and are kept out of the study. For
instance, ultrafast has $17$ options, but only $10$ of them are analyzed.

In total, there is the baseline system of x264 and its 20 possible
specializations with $Scenario_{1}$ and $Scenario_{2}$. In the following, we
will refer to them as $S_{0}$, for the baseline, and $S_{1}$ –$S_{20}$, for
the specialized systems.

Table 2. The properties of the eight used videos Name | Size | Length | Name | Size | Length
---|---|---|---|---|---
$[.mkv]$ | $[MB]$ | $[sec.]$ | $[.mkv]$ | $[MB]$ | $[sec.]$
V_1_720x480 | 108.4 | 13 | V_5_640x360 | 172.8 | 20
V_2_480x360 | 155.6 | 20 | V_6_640x360 | 41.5 | 20
V_3_640x360 | 165.5 | 19 | V_7_640x360 | 207.4 | 20
V_4_640x360 | 172.5 | 19 | V_8_624x464 | 217.2 | 20

###  4.4. Experiment settings

We conducted our experiment on a Linux workstation running Fedora 33 with
Intel Core i7-10610U CPU and 15.3 GiB of memory. In all cases, we use the gcc
$10.3.1$ compiler to build the baseline and specialized systems of x264. Some
of the done measurements (_cf._ $H_{03}$) require an input, namely a video.
Instead of a random selection, we carefully chose $8$ videos based on a recent
work by Lesoil _et.al._ , (Lesoil et al., 2021b), which properties are given
in Table 2. These 8 videos have approximately similar size, resolution, and
length, therefore they can be considered as a constant but they are still
different videos. Actually, these video are evaluated to be as representative
enough of 1,300 input videos in the YT UCG dataset 111111UCG dataset:
https://media.withyoutube.com/. (Lesoil et al., 2021b). author=R2,inline,
linecolor=red!50,backgroundcolor=red!50,bordercolor=red!50,]For the
evaluation, why were 8 videos chosen? You seem to be subsetting a larger
corpus, which one might consider cheery picking. It would be good to motivate
the number. Why not 10? And how did you decide whether they are
representative? This part of the methodology seems under-explained.
author=R2,inline,
linecolor=red!50,backgroundcolor=red!50,bordercolor=red!50,]For the evaluation
I was also wondering whether link-time optimization is used, and whether the
compilation is deterministic, or whether you might be measuring some type of
non-determinism including but not limited to effects from parallel
compilation, link-order differences etc. Some of the observed difference could
be explainable by such effects. Besides, considering the suggestions by DB.
Stewart (Stewart, 2001), the encoding time of videos is measured by using the
time method, and the measurements are repeated 5 times. author=R2,inline,
linecolor=red!50,backgroundcolor=red!50,bordercolor=red!50,]Similarly, l.964
has me worried that the benchmarks are not deterministic. I do not know
whether there’s any use of non-deterministic input sources in x264, but if
frame rates differ by such amounts, it may call the measured numbers into
question. Could you please clarify what is going on here? It would be also
good to know how the benchmarking is done exactly. You mention that you
measure 5 times. Though, this is only explained in sec. 8 instead of where it
should be, in the methodology section. It would also be important to know
whether you recompile the binaries in between to avoid measuring random code
placement changes? How do you avoid measuring turbo boost effects, thermal CPU
budgets, and Address Space Layout Randomization effects? To prevent side
effects, all experiments are run sequentially and as the only process in the
workstation.

Figure 3. The binary size and the found number of gadgets in $S_{0}$
(baseline) and $S_{1}-S_{20}$ (specializations). $(mean\pm std)$ for binary
size - $S_{0}:(3,096,176\pm 0)$, $S_{1-20}:(3,020,417\pm 63,641)$, and for
gadgets - $S_{0}:(106,495\pm 0)$, $S_{1-20}:(103,414\pm 2,696)$

##  5\. Results

In this section, we present the obtained results.

author=R3,inline,
linecolor=orange!50,backgroundcolor=orange!50,bordercolor=orange!50,]The
overall results of the study (i.e., the tool is usually smaller and faster)
are no surprise and the exact numbers are too specific to the example to be
generalizable. Nonetheless, the paper contains three larger tables with
quantitative results. It would be valuable to describe in more detail how the
study itself can be used for further research (e.g., as basis for engineering
metrics about run-time vs. compile-time variability?)

Table 3. The average and standard deviation of encoding time in seconds [s] of $8$ videos by the baseline ($S_{0}$) and specialized systems ($S_{1}-S_{20}$) System | Unit | ultrafast | superfast | veryfast | faster | fast | medium | slow | slower | veryslow | placebo | $H_{03}$
---|---|---|---|---|---|---|---|---|---|---|---|---
$S_{0}$ | [s] | $0.28\pm 0.02$ | $0.48\pm 0.01$ | $0.74\pm 0.03$ | $1.03\pm 0.04$ | $1.41\pm 0.04$ | $1.73\pm 0.04$ | $2.76\pm 0.08$ | $4.97\pm 0.16$ | $9.85\pm 0.37$ | $35.03\pm 0.72$ |
$S_{1}$ | [s] | - | - | - | - | $1.39\pm 0.03$ | $1.77\pm 0.05$ | $2.85\pm 0.10$ | $5.08\pm 0.18$ | $9.86\pm 0.26$ | $35.32\pm 0.46$ |
$S_{2}$ | [s] | - | - | $0.67\pm 0.02$ | $0.96\pm 0.01$ | $1.29\pm 0.02$ | $1.63\pm 0.04$ | $2.61\pm 0.06$ | $4.78\pm 0.18$ | $9.27\pm 0.42$ | $33.23\pm 0.69$ |
$S_{3}$ | [s] | - | $0.46\pm 0.02$ | $0.73\pm 0.02$ | $1.02\pm 0.02$ | $1.43\pm 0.03$ | $1.79\pm 0.05$ | $2.80\pm 0.06$ | $5.00\pm 0.18$ | $9.96\pm 0.26$ | $35.34\pm 0.76$ |
$S_{4}$ | [s] | - | $0.46\pm 0.01$ | $0.73\pm 0.01$ | $1.03\pm 0.01$ | $1.43\pm 0.03$ | $1.78\pm 0.05$ | $2.92\pm 0.10$ | $5.07\pm 0.19$ | $10.11\pm 0.32$ | $35.09\pm 0.48$ |
$S_{5}$ | [s] | $0.29\pm 0.01$ | $0.46\pm 0.02$ | $0.75\pm 0.01$ | $1.06\pm 0.03$ | $1.46\pm 0.04$ | $1.83\pm 0.04$ | $2.87\pm 0.08$ | $5.05\pm 0.19$ | $10.09\pm 0.37$ | $35.20\pm 0.63$ |
$S_{6}$ | [s] | $0.30\pm 0.01$ | $0.47\pm 0.02$ | $0.75\pm 0.02$ | $1.05\pm 0.01$ | - | - | - | - | - | - |
$S_{7}$ | [s] | $0.28\pm 0.01$ | $0.45\pm 0.01$ | - | - | - | - | - | - | - | - |
$S_{8}$ | [s] | $0.28\pm 0.01$ | - | - | - | - | - | - | - | - | - |
$S_{9}$ | [s] | $0.29\pm 0.02$ | - | - | - | - | - | - | - | - | - |
$S_{10}$ | [s] | - | - | - | - | - | - | - | - | - | - |
Avr. $S_{1-10}$ | [s] |  $0.29\pm 0.01$ $\uparrow$ |  $0.46\pm 0.01$ $\downarrow$ |  $0.72\pm 0.03$ $\downarrow$ |  $1.02\pm 0.04$ $\downarrow$ |  $1.40\pm 0.07$ $\downarrow$ |  $1.76\pm 0.08$ $\uparrow$ |  $2.81\pm 0.12$ $\uparrow$ |  $4.99\pm 0.13$ $\uparrow$ |  $9.86\pm 0.34$ $\uparrow$ |  $34.84\pm 0.90$ $\downarrow$ |  $\neg$R
$S_{11}$ | [s] | $0.29\pm 0.01$ | - | - | - | - | - | - | - | - | - |
$S_{12}$ | [s] | - | $0.47\pm 0.02$ | - | - | - | - | - | - | - | - |
$S_{13,14}$ | [s] | - | - | $0.66\pm 0.03$ | $0.95\pm 0.03$ | - | - | - | - | - | - |
$S_{15-20}$ | [s] | - | - | - | - | $1.24\pm 0.04$ | $1.53\pm 0.03$ | $2.56\pm 0.07$ | $4.70\pm 0.16$ | $9.21\pm 0.40$ | $33.55\pm 0.54$ |
Avr. $S_{11-20}$ | [s] |  $0.29\pm 0.0$ $\uparrow$ |  $0.47\pm 0.0$ $\downarrow$ |  $0.66\pm 0.0$ $\downarrow$ |  $0.95\pm 0.0$ $\downarrow$ |  $1.24\pm 0.0$ $\downarrow$ |  $1.53\pm 0.0$ $\downarrow$ |  $2.56\pm 0.0$ $\downarrow$ |  $4.70\pm 0.0$ $\downarrow$ |  $9.21\pm 0.0$ $\downarrow$ |  $33.55\pm 0.0$ $\downarrow$ | R
Avr. $S_{1-20}$ | [s] |  $0.29\pm 0.01$ $\uparrow$ |  $0.46\pm 0.01$ $\downarrow$ |  $0.71\pm 0.04$ $\downarrow$ |  $1.01\pm 0.05$ $\downarrow$ |  $1.37\pm 0.09$ $\downarrow$ |  $1.72\pm 0.12$ $\downarrow$ |  $2.77\pm 0.15$ $\uparrow$ |  $4.95\pm 0.16$ $\downarrow$ |  $9.75\pm 0.41$ $\downarrow$ |  $34.62\pm 0.96$ $\downarrow$ | R
$\%$ of $S_{1-20}$ |  | $1.39\%$ | $-4.13\%$ | $-4.08\%$ | $-2.12\%$ | $-2.76\%$ | $-0.33\%$ | $0.31\%$ | $-0.49\%$ | $-1.07\%$ | $-1.18\%$ |

###  5.1. The binary size of specialized system

To address $H_{01}$, we measured the binary size of x264 in bytes. At first,
under the same experiment settings, we measured the baseline binary size of
x264. Then, we specialized x264 following $Scenario_{1}$ and $Scenario_{2}$,
measured their respective binary size, and compared them with the baseline
binary size. Finally, we statistically test the $H_{01}$ null hypothesis.

The obtained results in Figure 3 (left) show that each specialized system has
a smaller binary size than the baseline system. The smallest binary size have
the specialized systems by $Scenario_{2}$ ($S_{11}$ \- $S_{20}$), as they are
specialized by more than one run-time option, than those by $Scenario_{1}$
($S_{1}$ \- $S_{10}$). In percentage, compared to the baseline binary size,
the specialized systems $S_{1}$ \- $S_{10}$ have a reduced binary size between
$0.001\%$ and $5.416\%$, whereas $S_{11}$ \- $S_{20}$ have a reduced binary
size between $3.526\%$ and $6.365\%$. Concretely, the x264’s binary size is
reduced by $2.45\%$ on average, or up to $6.37\%$ (based on $10$ analysed
options). To avoid repetition here, all of these data in percentage are also
given later in Table 5.

Moreover, the calculated Wilcoxon signed-rank test shows that the p-value is
less than $\alpha=0.05$ ($t=210$, $p=9.54\cdot 10^{-7}$) 121212Where $t$ is
the sum of the ranks of the differences above or below zero., therefore
$H_{01}$ is rejected in favor of $H_{A1}$. This suggests that specializing a
software system regarding its run-time options will statistically
_significantly_ reduce its binary size. This reduction can be useful for
resource-constraint devices. Although there is no direct or strong correlation
between binary size and other non-functional properties, the reduction
suggests that many paths of the code are eliminated and can be beneficial to
the system to be run.

###  5.2. The attack surface of specialized system

To measure the attack surface, we counted the number of code reuse gadgets in
the baseline and $20$ specialized systems of x264. For this reason, we used
the well-known ROPgadget 131313ROPgadget tool:
https://github.com/JonathanSalwan/ROPgadget tool and counted the overall
gadgets in the system’s binary. Then, we statistically test the $H_{02}$ null
hypothesis.

The resulting overall number of code reuse gadgets in Figure 3 (right) show
that the overall number of gadgets are most often reduced (the blue bars)
compared to the baseline number of gadgets (the orange bar). The specialized
systems $S_{1}$ \- $S_{10}$ have fewer gadgets than the baseline system,
between $-0.42\%$ and $6.60\%$, whereas specialized systems $S_{11}$ \-
$S_{20}$ have fewer gadgets, between $4.35\%$ and $7.18\%$. The $-0.42\%$
indicates that some specialized systems by a single option (specifically, the
5 red ones, see also Table 5) can have a small increase in the number of
gadgets despite that their binary size is reduced. This is something that is
also claimed by (Brown and Pande, 2019) that could happen, but not all gadgets
in a system can be chained by an attacker and threaten the system.

However, the calculated Wilcoxon signed-rank test shows that the $H_{02}$ is
not rejected for $S_{1}$ \- $S_{10}$ ($t=35$, $p=0.25$), whereas it is
rejected for $S_{11}$ \- $S_{20}$ ($t=55$, $p=9.77\cdot 10^{-4}$). In general,
it is rejected for $S_{1}$ \- $S_{20}$ in favor of the $H_{A2}$ ($t=190$,
$p=3.54\cdot 10^{-4}$). This suggests that it is better to specialize a
software system by multiple run-time options (_i.e._ , $Scenario_{2}$), as
this will statistically _significantly_ reduce its attack surface (by $2.9\%$
on average, or up to $7.18\%$ based on 10 options in x264), and hence its
security will be improved. Moreover, the attack surface gets reduced much
quicker, for $2.9\%$ on average, than the binary size, for $2.5\%$ on average.

Table 4. The average and standard deviation of bitrate in [kb/s] and frame rate in [fps] of $8$ videos by $S_{0}$ and $S_{1}-S_{20}$ System | Unit | ultrafast | superfast | veryfast | faster | fast | medium | slow | slower | veryslow | placebo | $H_{03}$
---|---|---|---|---|---|---|---|---|---|---|---|---
$S_{0}$ | [kb/s] | $1,800\pm 0$ | $1,167\pm 0$ | $905\pm 0$ | $946\pm 0$ | $947\pm 0$ | $935\pm 0$ | $928\pm 0$ | $851\pm 0$ | $843\pm 0$ | $787\pm 0$ |
Avr. $S_{1-10}$ | [kb/s] |  $1,800\pm 0$ $\leftrightarrow$ |  $1,167\pm 0$ $\leftrightarrow$ |  $912\pm 15$ $\uparrow$ |  $948\pm 6$ $\uparrow$ |  $946\pm 3$ $\downarrow$ |  $935\pm 2$ $\downarrow$ |  $926\pm 4$ $\downarrow$ |  $849\pm 4$ $\downarrow$ |  $849\pm 14$ $\uparrow$ |  $791\pm 9$ $\uparrow$ |  $\neg$R
Avr. $S_{11-20}$ | [kb/s] |  $1,800\pm 0$ $\leftrightarrow$ |  $1,167\pm 0$ $\leftrightarrow$ |  $938\pm 0$ $\uparrow$ |  $959\pm 0$ $\uparrow$ |  $941\pm 0$ $\downarrow$ |  $931\pm 0$ $\downarrow$ |  $919\pm 0$ $\downarrow$ |  $841\pm 0$ $\downarrow$ |  $874\pm 0$ $\uparrow$ |  $806\pm 0$ $\uparrow$ |  $\neg$R
Avr. $S_{1-20}$ | [kb/s] |  $1,800\pm 0$ $\leftrightarrow$ |  $1,167\pm 0$ $\leftrightarrow$ |  $916\pm 17$ $\uparrow$ |  $950\pm 7$ $\uparrow$ |  $945\pm 3$ $\downarrow$ |  $934\pm 2$ $\downarrow$ |  $925\pm 5$ $\downarrow$ |  $848\pm 5$ $\downarrow$ |  $853\pm 16$ $\uparrow$ |  $793\pm 10$ $\uparrow$ |  $\neg$R
$\%$ of $S_{1-20}$ |  | $0.00\%$ | $0.00\%$ | $1.22\%$ | $0.46\%$ | $-0.22\%$ | $-0.15\%$ | $-0.33\%$ | $-0.36\%$ | $1.24\%$ | $0.83\%$ |
$S_{0}$ | [fps] | $2,257\pm 157$ | $1,313\pm 43$ | $881\pm 26$ | $590\pm 13$ | $436\pm 9$ | $336\pm 8$ | $218\pm 6$ | $136\pm 5$ | $69\pm 2$ | $18\pm 1$ |
Avr. $S_{1-10}$ | [fps] |  $2,276\pm 53$ $\uparrow$ |  $1,303\pm 25$ $\downarrow$ |  $875\pm 28$ $\downarrow$ |  $583\pm 23$ $\downarrow$ |  $436\pm 32$ $\uparrow$ |  $337\pm 24$ $\uparrow$ |  $220\pm 18$ $\uparrow$ |  $136\pm 8$ $\uparrow$ |  $69\pm 4$ $\downarrow$ |  $18\pm 1$ $\uparrow$ |  $\neg$R
Avr. $S_{11-20}$ | [fps] |  $2,242\pm 0$ $\downarrow$ |  $1,301\pm 0$ $\downarrow$ |  $957\pm 0$ $\uparrow$ |  $634\pm 0$ $\uparrow$ |  $500\pm 0$ $\uparrow$ |  $396\pm 0$ $\uparrow$ |  $259\pm 0$ $\uparrow$ |  $154\pm 0$ $\uparrow$ |  $77\pm 0$ $\uparrow$ |  $19\pm 0$ $\uparrow$ | R
Avr. $S_{1-20}$ | [fps] |  $2,271\pm 49$ $\uparrow$ |  $1,303\pm 23$ $\downarrow$ |  $888\pm 42$ $\uparrow$ |  $591\pm 29$ $\uparrow$ |  $446\pm 39$ $\uparrow$ |  $347\pm 32$ $\uparrow$ |  $226\pm 23$ $\uparrow$ |  $139\pm 10$ $\uparrow$ |  $70\pm 5$ $\uparrow$ |  $18\pm 1$ $\uparrow$ | R
$\%$ of $S_{1-20}$ |  | $0.59\%$ | $-0.73\%$ | $0.83\%$ | $0.30\%$ | $2.47\%$ | $3.07\%$ | $3.68\%$ | $2.74\%$ | $1.95\%$ | $2.84\%$ |

###  5.3. The performance of specialized system

Next, we examine how a specialized system performs to the end-users. Namely,
as x264 is a video encoder, we measure the performance of a specialized x264
by using three metrics: the video encoding time, bitrate, and frame rate.

For this purpose, we encoded $8$ videos (_cf._ Table 2) using the baseline and
each of the $20$ specialized systems of x264, but only using their available
presets. For example, $S_{6}$ is specialized regarding the --mixed-refs (_cf._
Table 1). We should encode videos only with $4$ presets in it, namely, with
ultrafast, superfast, veryfast, and faster. The rest of the presets should not
be used in $S_{6}$ as they require the option of --mixed-refs, which is
removed from them. In case that they are used, the resulting system’s
performance is expected to be distorted, and thus unrealistic. Therefore, only
the first $4$ presets (_i.e._ , with ”$\circ$”) are available during the
encoding with $S_{6}$. As another example, the $S_{20}$ that is specialized
regarding the placebo preset should encode videos only with fast, medium,
slow, slower, veryslow, and placebo as they share the same used and unused
options. This way of encoding is followed by all $20$ specialized systems. As
for the baseline system, we use all $10$ presets with it.

As for the measured values, we first computed the average of values in $5$
repeated measurements, that is, the average encoding time, bitrate, and frame
rate of each video by each available preset in each system. Next, we computed
the average encoding time, bitrate, and frame rate of all $8$ videos within
each preset and system. Results in Tables 3 and 4 show the average values with
the standard deviation of specialized systems by the $Scenario_{1}$,
$Scenario_{2}$, and their overall average, per preset. The dash (”-”) in Table
3 is used to mark the unmeasured values for the unavailable presets in each
specialized system. It can be noticed that $S_{10}$ has no values, meaning
that it has no available presets as all of them require the option of --psy,
which is removed for the specialization of $S_{10}$. To save space, the
results in Table 4 are more condensed, showing only these three averages for
bitrate and frame rate. In percentage, they are more detailed in Table 5.

#### Encoding time.

The overall Average $S_{1-20}$ in Table 3 shows that in $8$ from $10$ presets
the encoding time is improved or decreased between $0.33\%$ and $4.13\%$,
whereas it is increased or worsened in only $2$ presets and for less, for
$0.31\%$ and $1.39\%$. Actually, the increased time in ultrafast and slow is
very small, for $3.9$ and $8.4$ milliseconds, respectively. Still, as our used
system to measure them has a higher resolution (of $1$ microsecond), we
believe that the increased encoding times are a consequence of our x264
specializations. Looking at the individual specializations, the $S_{15}$ has
the most improved encoding time, for $11.97\%$. This means that because of its
removed unused options the preset fast in $S_{15}$ becomes even faster.
Moreover, the encoding time measurements have always less than $0.96$ seconds
in terms of standard deviations.

#### Bitrate.

As shown in Table 4, the number of presets with an equal, increased, or
decreased bitrate is the same. Among the specializations, $S_{19}$ has the
most increased bitrate, for $3.71\%$, indicating that the video compression
will be most significantly improved by this specialization because of its
removed unused run-time options. In our measurements, the bitrate standard
deviations are minor, less than 17 [kb/s].

#### Frame rate.

A more notable improvement can be observed in the frame rate in the Average
$S_{1-20}$, in Table 4. The frame rate is improved in $9$ from the $10$
presets, between $0.30\%$ and $3.68\%$, whereas only in superfast it is
worsened, for only $0.73\%$. As for the specializations, the $S_{17}$ has the
most increased frames per second in videos, for about $18.65\%$, meaning that
the video compression will be improved by this specialization. The frame rate
deviates more in our measurements, up to $157$ [fps], because by default it is
auto-detected.

The key observations on the x264’s performance are that removing even only $5$
of its unused run-time options will improve the video encoding time in $80\%$
of the presets (for up to $4.13\%$), will improve the bitrate in $40\%$ of the
presets (for up to $1.24\%$), and will improve the frame rate in $90\%$ of the
presets (for up to $3.86\%$). In practical terms, using a specialized system
in those predefined configurations, a video is encoded faster and exhibits a
better frame rate and bitrate than the original x264. Moreover, the calculated
Wilcoxon signed-rank test given in the last column in Tables 3 and 4 shows
that $H_{03}$ is not rejected for $S_{1}$ \- $S_{10}$ for encoding time
($t=28$, $p=0.5$), bitrate ($t=10$, $p=0.16$), and frame rate ($t=26$,
$p=0.46$). But, it is rejected for $S_{11}$ \- $S_{20}$ for encoding time
($t=54$, $p=1.95\cdot 10^{-3}$) and frame rate ($t=7$, $p=0.02$). In general,
it is rejected for all specializations $S_{1}$ \- $S_{20}$ in favor of
$H_{A3}$ for encoding time ($t=51$, $p=0.01$) and frame rate ($t=7$,
$p=0.02$), but not for bitrate ($t=10$, $p=0.16$). Hence, specializing a
software system regarding its unused run-time options could improve its
performance (in x264, it does _not significantly_ improve its bitrate, but it
statistically _significantly_ improves its encoding time and frame rate).

###  5.4. The trade-off among the system properties

Table 5. A comparison of changes, in $\%$, of five properties System | Binary size | Gadgets | Encoding time | Bitrate | Frame rate
---|---|---|---|---|---
$S_{1}$ | $-0.270\%$ | $-0.51\%$ | $0.89\%$ | $0.00\%$ | $-1.78\%$
$S_{2}$ | $-0.001\%$ | $0.18\%$ | $-5.38\%$ | $0.96\%$ | $8.53\%$
$S_{3}$ | $-3.254\%$ | $-4.00\%$ | $0.87\%$ | $0.00\%$ | $-0.65\%$
$S_{4}$ | $-0.001\%$ | $-0.02\%$ | $1.06\%$ | $0.00\%$ | $-1.88\%$
$S_{5}$ | $-0.001\%$ | $0.42\%$ | $1.29\%$ | $0.00\%$ | $-2.26\%$
$S_{6}$ | $-0.403\%$ | $0.29\%$ | $1.07\%$ | $0.00\%$ | $-2.91\%$
$S_{7}$ | $-0.543\%$ | $-0.81\%$ | $-4.73\%$ | $0.00\%$ | $1.86\%$
$S_{8}$ | $-5.416\%$ | $-6.60\%$ | $-0.14\%$ | $0.00\%$ | $2.32\%$
$S_{9}$ | $-0.003\%$ | $0.05\%$ | $1.58\%$ | $0.00\%$ | $2.77\%$
$S_{10}$ | $-0,001\%$ | $0.18\%$ | NaN | NaN | NaN
$S_{11}$ | $-6,365\%$ | $-7.18\%$ | $1.71\%$ | $0.00\%$ | $-0.68\%$
$S_{12}$ | $-4.202\%$ | $-4.88\%$ | $-3.07\%$ | $0.00\%$ | $-0.91\%$
$S_{13}$ | $-3.659\%$ | $-4.43\%$ | $-9.58\%$ | $2.50\%$ | $8.15\%$
$S_{14}$ | $-3.659\%$ | $-4.43\%$ | $-9.58\%$ | $2.50\%$ | $8.15\%$
$S_{15}$ | $-3.526\%$ | $-4.35\%$ | $-11.97\%$ | $-0.67\%$ | $14.77\%$
$S_{16}$ | $-3.526\%$ | $-4.35\%$ | $-11.28\%$ | $-0.45\%$ | $17.64\%$
$S_{17}$ | $-3.526\%$ | $-4.35\%$ | $-7.24\%$ | $-0.98\%$ | $18.65\%$
$S_{18}$ | $-3.526\%$ | $-4.35\%$ | $-5.41\%$ | $-1.09\%$ | $13.91\%$
$S_{19}$ | $-3.526\%$ | $-4.35\%$ | $-6.59\%$ | $3.71\%$ | $12.52\%$
$S_{20}$ | $-3.526\%$ | $-4.35\%$ | $-4.25\%$ | $2.49\%$ | $8.41\%$
Avr. | $-2.447\%$ | $-2.89\%$ | $-3.40\%$ | $0.36\%$ | $5.47\%$

The obtained results in Sections 5.1, 5.2, and 5.3 show that different
specialized systems of x264 have notably different improvements on two non-
functional properties – namely, on the binary size and attack surface – and on
two performance measurements – namely, on the video encoding time and frame
rate. Therefore, to understand which removed options most significantly
improve the system in all or in most of these five aspects (including the
cases with an improvement in the bitrate), we compared the changes of these
five properties (from the baseline) for each specialized system. Specifically,
we compared the changes regarding the binary size and number of gadgets given
in Table 3, the encoding time given in Table 3, and bitrate with frame rate
given in Table 4. The results in percentage are given in Table 5.

It can be observed that the specialized systems $S_{11}-S_{20}$ have far more
improved values in each of the five aspects than the $S_{1}-S_{10}$. In fact,
the maximum improvements in five aspects are achieved with the specializations
$S_{11}-S_{20}$, which percentages are colored in green. The red-colored
percentages show that the number of gadgets and frame rate are worsened the
most in the set of systems $S_{1}-S_{10}$, whereas the encoding time and
bitrate are worsened the most in the set $S_{11}-S_{20}$. The percentages in
orange show the systems with the smallest improvement on the binary size, as
there are only improvements. Whereas the NaN values are the unmeasured values,
as all presets use the run-time option --psy which is used to specialize
$S_{10}$, hence measuring them is unrealistic. It is interesting to note that
specializations that provide an improvement or not are different. Therefore,
one should find a trade-off among them and choose the specialization that best
meets their requirements. For instance, in resource-constraint devices and in
those where security is more important, but the encoding time and frame rate
are flexible, then $S_{11}$ is a good choice. But, in case that encoding time
matters more than the other properties then $S_{15}$ best fulfills this
criterion. In fact, there are 4 specializations that have an improvement in
all five properties - namely, $S_{13}$, $S_{14}$, $S_{19}$, and $S_{20}$.
These specializations can be chosen when an improvement is expected on all
five properties. Besides these changes within every single specialized system,
the overall average in the last row in Table 5 reveals that each of the five
properties gets improved by specializing the x264 system.

The more unused run-time options are removed the greater the benefits gained.
Still, these benefits may vary in the system’s binary size, attack surface,
and performance, therefore finding their trade-offs to a given usage context
is necessary.

##  6\. Related work

We discern the four following areas of related work that are particularly
relevant to our investigation.

#### Configurable systems and their non-functional properties

Software product line engineering and variability management is a well-
established research area that led in the past decade to several techniques
for supporting _e.g._ , orthogonal variability management and derivation of
custom variants (Borba et al., 2012; Apel et al., 2013; Pohl et al., 2005;
Meinicke et al., 2020; Berger et al., 2015). In practice, variability is often
moved from compile-time to load-time (_a.k.a._ , encoding variability) (von
Rhein et al., 2016). The consequence is to deploy the whole product line (or
configurable system) in the delivered software systems. In our work, we follow
the opposite path: we aim to move from run-time to compile-time. We
specifically explore the benefits of applying the derivation process at
compile-time (instead of at run-time) in order to specialize legacy software
systems, with a particular focus on non-functional properties. Besides, there
are numerous works about the non-functional properties and performance of
configurable systems (_e.g._ , see (Pereira et al., 2019; Jamshidi et al.,
2017; Siegmund et al., 2015; Guo et al., 2013; Zhang et al., 2015; Valov et
al., 2015; Temple et al., 2016; Lesoil et al., 2021c)). The goal of this line
of research is to study the performance modeling of configurable systems
without removing run-time options and without altering the original source
code. In contrast, we provide an approach for debloating a configurable system
regarding its run-time options. We also show the impact of their removal in
the system’s non-functional properties.

#### Software debloating

Furthermore, software debloating has been previously explored, for example, to
reduce the size of deployed containers (Rastogi et al., 2017), or the attack
surface of specific programs (Quach et al., 2018; Sharif et al., 2018; Xin et
al., 2020a, b; Koo et al., 2019; Heo et al., 2018; Soto-Valero, 2021). Often,
proposed approaches debloat compiled binaries of a system, or debloat a system
by removing its unused libraries or its stale feature toggles. To our best
knowledge, none of them debloat software regarding the unused run-time
configuration options. Existing approaches are all inheriting of existing
works in program specialization (McNamee et al., 2001). In (Koo et al., 2019),
an approach is proposed to remove feature-specific shared libraries that are
needed only when certain configuration directives are specified by the user in
configuration files. In contrast, we target run-time options within the source
code.

#### Program slicing

It is another longstanding method for automatically decomposing programs by
analyzing their data flow and control flow (Weiser, 1984). Program slicing
techniques require the identification of a variable of interest, and the
benefits have been demonstrated in the context of white-box activities, such
as debugging, testing, maintenance, and understanding of programs. However, as
in our study, configuration options are usually given from an external (black-
box) view of the program, and it is challenging to relate internal variables
to such high-level configuration options. This has been demonstrated in
(Lillack et al., 2014). In practice, we also observed much more complex data
and control flows that would limit the applicability of such an approach.
Hence we believe that such tools can be helpful to partly _support_ the task
of annotating run-time options in the source code (_cf._ Section 7).

#### Performance

Performance has been investigated at run-time, _e.g._ , speculative execution
(Gabbay and Gabbay, 1996) and ahead-of-time compilation (Proebsting et al.,
1997). Such speedups come in complement to the proposed specialization
approach applied at compile-time. Since the specialization is applied at the
source level, existing approaches for optimizing compilers come also in
complement to a previous debloating. Performance has been also investigated in
the context of approximate computing (Mittal, 2016), which provides unsound
transformations, still useful in the context of possible trade-offs with
accuracy. For example, loop perforation (Misailovic et al., 2010) is a useful
interpolation technique while unrolling relaxable loops (_e.g._ , signal
processing). In contrast, we explore a sound, compile-time derivation of a
specialized configuration space.

In general terms and in contrast to the related works, we propose in this
paper a unique experimental study that combines the system’s _variability
management_ and _configuration debloating_ with a specific focus on its _non-
functional properties_ , including its _performance_. The program slicing
techniques with the identified patterns to implement run-time options (_cf._
T1) and program transformations can be helpful in the future to automate the
specialization process.

##  7\. Discussions

#### Replicability and confirmatory studies.

In the companion webpage 141414Companion page:
https://anonymous.4open.science/r/x264-6ED7, we provide the annotated source
code of x264 as well as the commits together with comments and related issues
that document how we annotated it during a period of 12 weeks. All measured
data for each specialization of x264 are available, including a dockerfile and
a usage guide for reproducing the experiment. We consider that it is fully
possible to reproduce/replicate our results and extend our work with, for
example, other metrics.

One of the contributions of this paper is the design of the study and the
experimental protocol. The exploratory nature of this study was mainly here to
generate hypotheses about the possible benefits of specializing the run-time
configuration space of a system. Our results call to further investigate this
idea on other software systems: Can we observe similar benefits in another
application domains and software systems? Is there another non-functional
property for which the specialization pays off? What are run-time options
worth to be removed? Is it always safe for the unused run-time options to be
changed at compile-time (their pros and cons), _e.g._ , the system should be
re-build to enable a compile-time option? We believe our study can be
replicated and we encourage researchers to do so. There are several aspects of
the experience that deserve special attention, particularly in view of the
lessons learned from conducting this case study.

#### Manual vs. automated specialization.

We did rely on a semi-automated specialization process by manually adding
annotations to the source code and then automatically specializing the x264.
The choice of a semi-automated process was on purpose and for the sake of the
experiments since our primary goal was to control in a fine-grained way the
removal of the code. One lesson learned is that manually tracing run-time
options is an error-prone activity. It is easy to forget an annotation or
break some functionalities. Hence, the added T4 task automatically checks that
the remaining configuration space of the specialized system remains equivalent
to the configuration space of the original system.

The major benefit of our semi-automated approach is that we have a ground
truth. It is now possible to envision the use of more automated, perhaps more
aggressive, techniques and possibly compare the outcome. There are two
possible directions. The first approach is to use automated techniques to
support developers during the task of manually tracing the options in the
source code (corresponding to T1 in Figure 1). Works such as feature location
or program slicing can be helpful (_e.g._ , see (Lillack et al., 2014; Ribeiro
et al., 2010; AL-Msie’deen et al., 2013; Weiser, 1984; Rubin and Chechik,
2013; Strüber et al., 2019)), but annotating already implemented run-time
options deserves specific attention. Our experience with x264 is that the
control flow of parameters’ values is hard to follow, run-time options are
scattered in different files and functions, and there are patterns of
annotation that deserve some formalization to either discipline or automate
the effort (corresponding to T2). A second approach is to fully automate the
specialization, possibly at the binary level. As discussed earlier, most of
the debloating techniques have been designed to entirely specialize a system
(_i.e._ , on a fixed configuration or use case). Our debloating occurs at the
options level and for different use case specializations, hence it is unclear
how to adapt existing techniques. The risk of automation is that the removal
of the code can be too aggressive (unsound) or miss some opportunities
(incomplete). In particular, guaranteeing that the remaining configurations
preserve their behavior is challenging (corresponding to T2 and T3). A third
challenge consists in validating the specialization process. In our study, we
compare the outputs (videos) on some representative configurations of the
original system and the debloated system. Other validation and comparison
procedures can be employed depending on the specificity of the kinds of
outputs and software systems (corresponding to T4).

#### Why and when specialization pays off.

Results show that specialization has varying effects on performance-like
properties. There are statistically significant improvements, but we also
notice one insignificant improvement (on bitrate). We hypothesize that there
is a complex interplay between the compiler, the annotated code to remove, and
the run-time usage. Understanding the reasons why and when compilers can
leverage the new specialized source code is an open issue. It can help to
prioritize the options worth debloating.

author=R1,inline,
linecolor=purple!50,backgroundcolor=purple!50,bordercolor=purple!50,]The
differences in quality properties between the different variants appear very
small. Is it really worth the whole effort? This should be discussed in a more
differentiated manner.

##  8\. Threats to validity

#### External threats.

With respect to generalization, we use only a single software subject in a
specific domain (video compression) that is implemented in C language. x264
has been studied in several papers on configurable software (_e.g._ , (Guo et
al., 2013; Siegmund et al., 2015; Temple et al., 2017; Jamshidi et al., 2017;
Pereira et al., 2019)), but not in the context of specializing configurable
systems and debloating run-time options. Our exploratory research aims to
investigate a problem that has not been studied or thoroughly investigated in
the past: the potential benefits of removing run-time options. As with any
exploratory case study, we cannot draw any statistically generalizing
conclusions from such studies (Stol and Fitzgerald, 2018). However, such
generalization of findings is not the goal of such studies – instead, we aim
to develop an understanding and propose hypotheses about other similar
configurable systems.

#### Internal threats.

The performance, such as the video’s encoding time by x264, may differ
depending on the inputs processed. To mitigate this threat, we use 8 videos
considered as representatives of 1,300 videos coming from the YouTube UGC
dataset. Then, the measurements are repeated 5 times. A threat to validity is
also the set of run-time options we consider. Choosing different options may
have a different effect on the performance measurements. Still, we have
selected a diverse set of options with potentially different impacts on non-
functional properties and which are part of the presets that are widely used
and represent a variety of usage in x264.

Another threat to our experiments is the annotation of run-time options, which
can be incomplete and not valid. An incomplete annotated option may impact
different properties (_e.g._ , binary size) and break existing
functionalities, not related to the removed options. To mitigate this threat,
we manually and thoroughly annotated the source code. We systematically
reported on progress through GitHub issues and concrete meetings for
clarifying some subtle cases about _e.g._ , the meaning of debloating. In
terms of validity, we used oracles that control that some remaining
configurations produce the exact same videos. In terms of completeness, we
chose to not consider the assembly code of x264 when annotating options. A few
lines of code and options are involved, suggesting little to negligible impact
on our results.

author=R1,inline,
linecolor=purple!50,backgroundcolor=purple!50,bordercolor=purple!50,]How did
you check completeness of annotations? Was there some independent review of
annotations?

##  9\. Conclusion

We introduced the idea of specializing in a controlled and configurable way
the unused run-time configuration space of a software system. With our
proposed approach, we synthesize a specialized source code at compile time
without the run-time options considered unnecessary while preserving other
functionalities and remaining configurations. Using the well-known video
encoder of x264 as a subject, we showed how to specialize a system by changing
the binding time of its run-time options at compile-time. We experimented with
10 of its options over 8 inputs (videos) and over 10 presets of the system’s
configurations. The results show that specializing a system regarding its
unused run-time options brings statistically significant benefits _w.r.t._
three non-functional properties, namely binary size, attack surface, and
performance (except for bitrate). For instance, removing the single option of
--cabac in x264 reduces the binary size by about 7$\%$, its attack surface by
8$\%$, and the video encoding time by 3$\%$. The combination of options during
the specialization process can lead to further benefits (up to 18% for frame
rate). We believe that developers, operationals, and users can use our
approach to specify and then remove run-time options that are never used or
changed for their specific use cases. For example, run-time options involving
specific algorithms that do not meet the requirements of the application
represent a ”bloat” and have no reason to be embedded.

Our exploratory study naturally calls to replicate the approach and results in
other software engineering contexts. We also brought a new problem to the
community: instead of adding options, the challenge of specialization is to
remove code related to options. As reported in our experience, this is a
change that requires revisiting code analysis and feature location techniques
traditionally designed for adding or maintaining options, not necessarily for
removing them. Another future work direction is also to measure the change of
the system’s non-functional properties when some of its compile-time and run-
time options are both removed.

## References

  * (1)
  * Acher et al. (2018) Mathieu Acher, Paul Temple, Jean-Marc Jezequel, José A Galindo, Jabier Martinez, and Tewfik Ziadi. 2018.  VaryLaTeX: Learning paper variants that meet constraints. In _12th International Workshop on Variability Modelling of Software-Intensive Systems_. ACM, 83–88.  https://doi.org/10.1145/3168365.3168372
  * AL-Msie’deen et al. (2013) Ra’Fat AL-Msie’deen, Abdelhak Seriai, Marianne Huchard, Christelle Urtado, Sylvain Vauttier, and Hamzeh Eyal Salman. 2013.  Feature location in a collection of software product variants using formal concept analysis. In _International Conference on Software Reuse_. Springer, 302–307.  https://doi.org/10.1007/978-3-642-38977-1_22
  * Alves Pereira et al. (2020) Juliana Alves Pereira, Mathieu Acher, Hugo Martin, and Jean-Marc Jézéquel. 2020\.  Sampling effect on performance prediction of configurable systems: A case study. In _International Conference on Performance Engineering_. ACM, 277–288.  https://doi.org/10.1145/3358960.3379137
  * Amand et al. (2019) Benoit Amand, Maxime Cordy, Patrick Heymans, Mathieu Acher, Paul Temple, and Jean-Marc Jézéquel. 2019.  Towards learning-aided configuration in 3D printing: Feasibility study and application to defect prediction. In _13th International Workshop on Variability Modelling of Software-Intensive Systems_. ACM, 1–9.  https://doi.org/10.1145/3302333.3302338
  * Apel et al. (2013) Sven Apel, Alexander Von Rhein, Philipp Wendler, Armin Größlinger, and Dirk Beyer. 2013\.  Strategies for product-line verification: Case studies and experiments. In _35th International Conference on Software Engineering_. IEEE, 482–491.  https://doi.org/10.1109/ICSE.2013.6606594
  * Barr et al. (2014) Earl T Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2014.  The oracle problem in software testing: A survey.  _Transactions on Software Engineering_ 41, 5 (2014), 507–525.  https://doi.org/10.1109/TSE.2014.2372785
  * Baxter and Mehlich (2001) Ira D Baxter and Michael Mehlich. 2001.  Preprocessor conditional removal by simple partial evaluation. In _8th Working Conference on Reverse Engineering_. IEEE, 281–290.  https://doi.org/10.1109/WCRE.2001.957833
  * Berger et al. (2015) Thorsten Berger, Daniela Lettner, Julia Rubin, Paul Grünbacher, Adeline Silva, Martin Becker, Marsha Chechik, and Krzysztof Czarnecki. 2015.  What is a feature? a qualitative study of features in industrial software product lines. In _19th International Conference on Software Product Line_. ACM, 16–25.  https://doi.org/10.1145/2791060.2791108
  * Borba et al. (2012) Paulo Borba, Leopoldo Teixeira, and Rohit Gheyi. 2012\.  A theory of software product line refinement.  _Theoretical Computer Science_ 455 (2012), 2–30.  https://doi.org/10.1016/j.tcs.2012.01.031
  * Bosch and Capilla (2012) Jan Bosch and Rafael Capilla. 2012.  Dynamic variability in software-intensive embedded system families.  _Computer_ 45, 10 (2012), 28–35.
  * Brown and Pande (2019) Michael D Brown and Santosh Pande. 2019.  Is less really more? Towards better metrics for measuring security improvements realized through software debloating. In _12th USENIX Workshop on Cyber Security Experimentation and Test_.
  * Bruce et al. (2018) Bobby R Bruce, Justyna Petke, Mark Harman, and Earl T Barr. 2018\.  Approximate oracles and synergy in software energy search spaces.  _Transactions on Software Engineering_ 45, 11 (2018), 1150–1169.  https://doi.org/10.1109/TSE.2018.2827066
  * Capilla and Bosch (2011) Rafael Capilla and Jan Bosch. 2011.  The promise and challenge of runtime variability.  _Computer_ 44, 12 (2011), 93–95.  https://doi.org/10.1109/MC.2011.382
  * Czarnecki et al. (2004) Krzysztof Czarnecki, Simon Helsen, and Ulrich Eisenecker. 2004\.  Staged configuration using feature models. In _Software Product Lines_. Springer, 266–283.  https://doi.org/10.1007/978-3-540-28630-1_17
  * Czarnecki et al. (2005a) Krzysztof Czarnecki, Simon Helsen, and Ulrich Eisenecker. 2005a.  Formalizing cardinality-based feature models and their specialization.  _Software Process: Improvement and Practice_ 10, 1 (2005), 7–29.  https://doi.org/10.1002/spip.213
  * Czarnecki et al. (2005b) Krzysztof Czarnecki, Simon Helsen, and Ulrich Eisenecker. 2005b.  Staged configuration through specialization and multilevel configuration of feature models.  _Software Process: Improvement and Practice_ 10, 2 (2005), 143–169.  https://doi.org/10.1002/spip.225
  * Dybå et al. (2006) Tore Dybå, Vigdis By Kampenes, and Dag IK Sjøberg. 2006\.  A systematic review of statistical power in software engineering experiments.  _Information and Software Technology_ 48, 8 (2006), 745–755.  https://doi.org/10.1016/j.infsof.2005.08.009
  * Gabbay and Gabbay (1996) Freddy Gabbay and Freddy Gabbay. 1996.  _Speculative execution based on value prediction_.  Technical Report. EE Department TR 1080, Technion - Israel Institute of Technology.
  * Guo et al. (2013) Jianmei Guo, Krzysztof Czarnecki, Sven Apel, Norbert Siegmund, and Andrzej Wasowski. 2013\.  Variability-aware performance prediction: A statistical learning approach. In _28th International Conference on Automated Software Engineering_. IEEE, 301–311.  https://doi.org/10.1109/ASE.2013.6693089
  * Harman et al. (2014) Mark Harman, Yue Jia, Jens Krinke, William B. Langdon, Justyna Petke, and Yuanyuan Zhang. 2014\.  Search based software engineering for software product line engineering: A survey and directions for future work. In _18th International Software Product Line Conference_. ACM, 5–18.  https://doi.org/10.1145/2648511.2648513
  * Heo et al. (2018) Kihong Heo, Woosuk Lee, Pardis Pashakhanloo, and Mayur Naik. 2018\.  Effective program debloating via reinforcement learning. In _2018 Conference on Computer and Communications Security_. ACM, 380–394.  https://doi.org/10.1145/3243734.3243838
  * Hubaux et al. (2013) Arnaud Hubaux, Patrick Heymans, Pierre-Yves Schobbens, Dirk Deridder, and Ebrahim Khalil Abbasi. 2013\.  Supporting multiple perspectives in feature-based configuration.  _Software & Systems Modeling_ 12, 3 (2013), 641–663.  https://doi.org/10.1007/s10270-011-0220-1
  * Jamshidi et al. (2017) Pooyan Jamshidi, Norbert Siegmund, Miguel Velez, Christian Kästner, Akshay Patel, and Yuvraj Agarwal. 2017.  Transfer learning for performance modeling of configurable systems: An exploratory analysis. In _32nd International Conference on Automated Software Engineering_. IEEE, 497–508.  https://doi.org/10.1109/ASE.2017.8115661
  * Jiang et al. (2018) Yufei Jiang, Qinkun Bao, Shuai Wang, Xiao Liu, and Dinghao Wu. 2018.  RedDroid: Android application redundancy customization based on static analysis. In _29th International Symposium on Software Reliability Engineering_. IEEE, 189–199.  https://doi.org/10.1109/ISSRE.2018.00029
  * Jones et al. (1993) Neil D Jones, Carsten K Gomard, and Peter Sestoft. 1993\.  _Partial evaluation and automatic program generation_.  Peter Sestoft.
  * Koo et al. (2019) Hyungjoon Koo, Seyedhamed Ghavamnia, and Michalis Polychronakis. 2019\.  Configuration-driven software debloating. In _12th European Workshop on Systems Security_. ACM, 1–6.  https://doi.org/10.1145/3301417.3312501
  * Kurmus et al. (2011) Anil Kurmus, Alessandro Sorniotti, and Rüdiger Kapitza. 2011\.  Attack surface reduction for commodity OS kernels: Trimmed garden plants may attract less bugs. In _4rth European Workshop on System Security_. ACM, 1–6.  https://doi.org/10.1145/1972551.1972557
  * Lesoil et al. (2021a) Luc Lesoil, Mathieu Acher, Arnaud Blouin, and Jean-Marc Jézéquel. 2021a.  Deep software variability: Towards handling cross-layer configuration. In _15th International Working Conference on Variability Modelling of Software-Intensive Systems_. ACM, 1–8.  https://doi.org/10.1145/3442391.3442402
  * Lesoil et al. (2021b) Luc Lesoil, Mathieu Acher, Arnaud Blouin, and Jean-Marc Jézéquel. 2021b.  The Interaction between inputs and configurations fed to software systems: An empirical study.  _Preprint arXiv:2112.07279_ (2021).
  * Lesoil et al. (2021c) Luc Lesoil, Mathieu Acher, Xhevahire Tërnava, Arnaud Blouin, and Jean-Marc Jézéquel. 2021c.  The interplay of compile-time and run-time options for performance prediction. In _25th International Systems and Software Product Line Conference-Volume A_. ACM, 100–111.  https://doi.org/10.1145/3461001.3471149
  * Lillack et al. (2014) Max Lillack, Christian Kästner, and Eric Bodden. 2014\.  Tracking load-time configuration options. In _29th International Conference on Automated Software Engineering_. ACM, 445–456.  https://doi.org/10.1145/2642937.2643001
  * Martin et al. (2021) Hugo Martin, Mathieu Acher, Juliana Alves Pereira, and Jean-Marc Jézéquel. 2021\.  A comparison of performance specialization learning for configurable systems. In _25th Int. Systems and Software Product Line Conference_. ACM, 46–57.  https://doi.org/10.1145/3461001.3471155
  * McNamee et al. (2001) Dylan McNamee, Jonathan Walpole, Calton Pu, Crispin Cowan, Charles Krasic, Ashvin Goel, Perry Wagle, Charles Consel, Gilles Muller, and Renauld Marlet. 2001\.  Specialization tools and techniques for systematic optimization of system software.  _Transactions on Computer Systems_ (2001), 217–251.  https://doi.org/10.1145/377769.377778
  * Meinicke et al. (2020) Jens Meinicke, Chu-Pan Wong, Bogdan Vasilescu, and Christian Kästner. 2020.  Exploring differences and commonalities between feature flags and configuration options. In _42nd International Conference on Software Engineering: Software Engineering in Practice_. ACM, 233–242.  https://doi.org/10.1145/3377813.3381366
  * Misailovic et al. (2010) Sasa Misailovic, Stelios Sidiroglou, Henry Hoffmann, and Martin Rinard. 2010.  Quality of service profiling. In _32nd International Conference on Software Engineering_. ACM, 25–34.  https://doi.org/10.1145/1806799.1806808
  * Mittal (2016) Sparsh Mittal. 2016\.  A survey of techniques for approximate computing.  _Comput. Surveys_ (2016), 1–33.  https://doi.org/10.1145/2893356
  * Pereira et al. (2019) Juliana Alves Pereira, Hugo Martin, Mathieu Acher, Jean-Marc Jézéquel, Goetz Botterweck, and Anthony Ventresque. 2019.  Learning software configuration spaces: A systematic literature review.  arXiv:arXiv:1906.03018
  * Pohl et al. (2005) Klaus Pohl, Günter Böckle, and Frank J. van der Linden. 2005\.  _Software Product Line Engineering: Foundations, Principles and Techniques_.  Springer-Verlag.
  * Proebsting et al. (1997) Todd A Proebsting, Gregg M Townsend, Patrick G Bridges, John H Hartman, Tim Newsham, and Scott A Watterson. 1997.  Toba: Java for applications-A way ahead of time (WAT) compiler. In _COOTS_. ””, 41–54.
  * Qian et al. (2020) Chenxiong Qian, Hyungjoon Koo, ChangSeok Oh, Taesoo Kim, and Wenke Lee. 2020.  Slimium: Debloating the chromium browser with feature subsetting. In _Conference on Computer and Communications Security_. 461–476.  https://doi.org/10.1145/3372297.3417866
  * Quach and Prakash (2019) Anh Quach and Aravind Prakash. 2019.  Bloat factors and binary specialization. In _3rd Workshop on Forming an Ecosystem Around Software Transformation_. ACM, 31–38.  https://doi.org/10.1145/3338502.3359765
  * Quach et al. (2018) Anh Quach, Aravind Prakash, and Lok Yan. 2018.  Debloating software through piece-wise compilation and loading. In _27th USENIX Security Symposium_. USENIX Association, 869–886.  https://www.usenix.org/conference/usenixsecurity18/presentation/quach
  * Rastogi et al. (2017) Vaibhav Rastogi, Drew Davidson, Lorenzo De Carli, Somesh Jha, and Patrick McDaniel. 2017.  Cimplifier: Automatically debloating containers. In _11th Joint Meeting on Foundations of Software Engineering_. ACM, 476–486.  https://doi.org/10.1145/3106237.3106271
  * Ribeiro et al. (2010) Márcio Ribeiro, Humberto Pacheco, Leopoldo Teixeira, and Paulo Borba. 2010.  Emergent feature modularization. In _Object Oriented Programming Systems Languages and Applications Companion_. ACM, 11–18.  https://doi.org/10.1145/1869542.1869545
  * Rubin and Chechik (2013) Julia Rubin and Marsha Chechik. 2013.  A survey of feature location techniques.  In _Domain Engineering_. Springer, 29–58.  https://doi.org/10.1007/978-3-642-36654-3_2
  * Sharif et al. (2018) Hashim Sharif, Muhammad Abubakar, Ashish Gehani, and Fareed Zaffar. 2018.  TRIMMER: Application specialization for code debloating. In _33rd International Conference on Automated Software Engineering_. ACM, 329–339.  https://doi.org/10.1145/3238147.3238160
  * Siegmund et al. (2015) Norbert Siegmund, Alexander Grebhahn, Sven Apel, and Christian Kästner. 2015.  Performance-influence models for highly configurable systems. In _10th Joint Meeting on Foundations of Software Engineering_. ACM, 284–294.  https://doi.org/10.1145/2786805.2786845
  * Soto-Valero (2021) César Soto-Valero. 2021\.  Software debloating papers.  https://www.cesarsotovalero.net/software-debloating-papers.
  * Soto-Valero et al. (2021) César Soto-Valero, Nicolas Harrand, Martin Monperrus, and Benoit Baudry. 2021.  A comprehensive study of bloated dependencies in the maven ecosystem.  _Empirical Software Engineering_ (2021), 1–44.  https://doi.org/10.1007/s10664-020-09914-8
  * Stewart (2001) David B Stewart. 2001\.  Measuring execution time and real-time performance. In _Embedded Systems Conference_ , Vol. 141. Citeseer.
  * Stol and Fitzgerald (2018) Klaas-Jan Stol and Brian Fitzgerald. 2018.  The ABC of software engineering research.  _Transactions on Software Engineering and Methodology_ 27, 3 (2018), 1–51.  https://doi.org/10.1145/3241743
  * Strüber et al. (2019) Daniel Strüber, Mukelabai Mukelabai, Jacob Krüger, Stefan Fischer, Lukas Linsbauer, Jabier Martinez, and Thorsten Berger. 2019\.  Facing the truth: Benchmarking the techniques for the evolution of variant-rich systems. In _23rd International Systems and Software Product Line Conference_. ACM, 177–188.  https://doi.org/10.1145/3336294.3336302
  * Temple et al. (2017) Paul Temple, Mathieu Acher, Jean-Marc Jézéquel, and Olivier Barais. 2017.  Learning contextual-variability models.  _IEEE Software_ (2017).  https://doi.org/10.1109/MS.2017.4121211
  * Temple et al. (2016) Paul Temple, José A Galindo, Mathieu Acher, and Jean-Marc Jézéquel. 2016.  Using machine learning to infer constraints for product lines. In _20th International Systems and Software Product Line Conference_. ACM, 209–218.  https://doi.org/10.1145/2934466.2934472
  * Valov et al. (2015) Pavel Valov, Jianmei Guo, and Krzysztof Czarnecki. 2015\.  Empirical comparison of regression methods for variability-aware performance prediction. In _19th International Conference on Software Product Line_. ACM, 186–190.  https://doi.org/10.1145/2791060.2791069
  * von Rhein et al. (2016) Alexander von Rhein, Thomas Thüm, Ina Schaefer, Jörg Liebig, and Sven Apel. 2016\.  Variability encoding: From compile-time to load-time variability.  _Journal of Logical and Algebraic Methods in Programming_ (2016), 125–145.  https://doi.org/10.1016/j.jlamp.2015.06.007
  * Weiser (1984) Mark Weiser. 1984\.  Program slicing.  _Transactions on Software Engineering_ (1984), 352–357.  https://doi.org/10.1109/TSE.1984.5010248
  * Wilcoxon et al. (1963) Frank Wilcoxon, SK Katti, and Roberta A Wilcox. 1963.  _Critical values and probability levels for the Wilcoxon rank sum test and the Wilcoxon signed rank test_. Vol. 1.  American Cyanamid Pearl River (NY).
  * Xin et al. (2020a) Qi Xin, Myeongsoo Kim, Qirun Zhang, and Alessandro Orso. 2020a.  Program debloating via stochastic optimization. In _42nd International Conference on Software Engineering: New Ideas and Emerging Results_. ACM, 65–68.  https://doi.org/10.1145/3377816.3381739
  * Xin et al. (2020b) Qi Xin, Myeongsoo Kim, Qirun Zhang, and Alessandro Orso. 2020b.  Subdomain-based generality-aware debloating. In _35th International Conference on Automated Software Engineering_. ACM, 224–236.  https://doi.org/10.1145/3324884.3416644
  * Xu et al. (2015) Tianyin Xu, Long Jin, Xuepeng Fan, Yuanyuan Zhou, Shankar Pasupathy, and Rukma Talwadker. 2015.  Hey, you have given me too many knobs!: Understanding and dealing with over-designed configuration in system software. In _10th Joint Meeting on Foundations of Software Engineering_. ACM, 307–319.  https://doi.org/10.1145/2786805.2786852
  * Zhang et al. (2015) Yi Zhang, Jianmei Guo, Eric Blais, and Krzysztof Czarnecki. 2015\.  Performance prediction of configurable software systems by fourier learning (t). In _30th International Conference on Automated Software Engineering_. IEEE, 365–373.  https://doi.org/10.1109/ASE.2015.15