# VAMP: Visual Analytics for Microservices Performance

Luca Traini  0000-0003-3676-0645 University of L’Aquila, Italy
<EMAIL_ADDRESS>,  Jessica Leone  0000-0002-2870-8161 University of
L’Aquila, Italy<EMAIL_ADDRESS>,  Giovanni Stilo
0000-0002-2092-0213 University of L’Aquila, Italy<EMAIL_ADDRESS>and
Antinisca Di Marco  0000-0001-7214-9945 University of L’Aquila, Italy
<EMAIL_ADDRESS>
###### Abstract.

Analysis of microservices’ performance is a considerably challenging task due
to the multifaceted nature of these systems. Each request to a microservices
system might raise several Remote Procedure Calls (RPCs) to services deployed
on different servers and/or containers. Existing distributed tracing tools
leverage swimlane visualizations as the primary means to support performance
analysis of microservices. These visualizations are particularly effective
when it is needed to investigate individual end-to-end requests’ performance
behaviors. Still, they are substantially limited when more complex analyses
are required, as when understanding the system-wide performance trends is
needed.

To overcome this limitation, we introduce vamp, an innovative visual analytics
tool that enables, at once, the performance analysis of multiple end-to-end
requests of a microservices system. vamp was built around the idea that having
a wide set of interactive visualizations facilitates the analyses of the
recurrent characteristics of requests and their relation w.r.t. the end-to-end
performance behavior. Through an evaluation of 33 datasets from an established
open-source microservices system, we demonstrate how vamp aids in identifying
RPC execution time deviations with significant impact on end-to-end
performance. Additionally, we show that vamp can support in pinpointing
meaningful structural patterns in end-to-end requests and their relationship
with microservice performance behaviors.

Microservices, Distributed Tracing, Performance Analysis

††copyright: none††ccs: Software and its engineering Software
performance††ccs: Software and its engineering Maintaining software††ccs:
Software and its engineering Software evolution††ccs: Human-centered computing
Visual analytics††ccs: Human-centered computing Visualization toolkits

##  1\. Introduction

Microservices have emerged as a pivotal change in the software industry,
paving the way to a novel paradigm for structuring the software development
process. This novel approach entails multiple independent teams responsible
“from development to deploy” (O’Hanlon, 2006) of loosely coupled independently
deployable services (Newman, 2015; O’Hanlon, 2006). Due to their modular
nature, microservices are particularly well-suited for the modern software
industry, where rapidly releasing software updates and enhancements is a
critical competitive advantage (Rubin and Rinard, 2016).

Although beneficial in many aspects, microservices also introduce new
challenges, especially when it comes to maintaining consistent software
performance. This complexity arises from various elements. Firstly, the
inherent complexity of these systems often hinders the adoption of proactive
measures for performance assurance(Sridharan, 2017; Veeraraghavan et al.,
2016), such as pre-production performance testing (Jiang and Hassan, 2015;
Laaber and Leitner, 2018; Traini et al., 2022). Secondly, these proactive
measures are often hampered by time and resource constraints due to the
substantial pressure to deliver fast-to-market (Rubin and Rinard, 2016;
Traini, 2022). Thirdly, microservices systems typically exhibit an emergent
performance behavior in the field that is hard to predict in advance
(Veeraraghavan et al., 2016). Finally, these systems undergo continuous
software changes, with multiple releases occurring on a daily basis, and
handle highly variable workloads (Ardelean et al., 2018), which make them more
vulnerable to unforeseen performance regressions (Veeraraghavan et al., 2016;
Traini et al., 2021).

These challenges have led to an increased interest in the concept of
_observability_ (Majors et al., 2022), _i.e.,_ the ability to have a holistic
understanding of the system’s performance by analyzing its logs, traces, and
metrics. Distributed tracing tools (Parker et al., 2020) are today widely used
in practice to enhance observability of microservices systems (Mace, 2017).
These tools track and record the propagation of requests as they flow through
different RPCs and services of a microservices system (Sambasivan et al.,
2016), and provide visual aids to support performance analysis of end-to-end
requests, _e.g.,_ swimlane visualizations (Davidson et al., 2023; Sigelman et
al., 2010; Technologies, 2023).

Despite their utility, distributed tracing tools have recently been criticized
for their limited support for performance analysis (Davidson et al., 2023). A
common use case for these tools is the analysis of the system-wide performance
behavior (Parker et al., 2020), such as understanding the response time
distributions of end-to-end requests (Davidson et al., 2023). However, current
distributed tracing tools often fall short in this area, necessitating a
switch between various visualization tools, which can make the process
cumbersome and time-consuming (Davidson et al., 2023). Indeed, they primarily
focus on the analysis of individual requests, which has limited value unless
it is compared with the performance behavior of the entire corpus of requests
(Anand et al., 2020; Parker et al., 2020; Davidson et al., 2023).

In this paper, we introduce vamp, an innovative visual analytics tool designed
to enhance the performance analysis of microservices systems. vamp extends the
conceptual proposal of Leone and Traini (Leone and Traini, 2023). The
fundamental idea underpinning vamp is to simplify the understanding of the
relationship between request characteristics and end-to-end response time
behavior through interactive charts and color-encoding techniques. vamp
comprises two main visualization components: an interactive tree that
illustrates the workflow in terms of RPCs for multiple end-to-end requests,
and an interactive histogram representing the end-to-end performance behavior
of the requests under analysis. Interaction with these visual components aids
in identifying the unique characteristics of certain RPC execution paths with
respect to some specific system performance behaviors.

We evaluate vamp using 33 datasets derived from TrainTicket (Zhou et al.,
2021), an open-source microservices system widely utilized in previous
software engineering research (Traini and Cortellessa, 2023; Zhang et al.,
2022; Li et al., 2021). Our findings demonstrate that vamp enables the
identification of notable and recurrent request characteristics associated
with specific end-to-end response time behaviors.

A video of vamp in action can be accessed at https://youtu.be/qMVOMt06EJE.

##  2\. Motivation

Figure 1. Swimlane visualization.

The swimlane visualisation is the canonical way to visualize individual
requests within distributed tracing tools (Kaldor et al., 2017; Sigelman et
al., 2010; Technologies, 2023; Davidson and Mace, 2022). Fig. 1 shows a
representative example of a swimlane visualization. The visualization depicts
a timeline of a single request, with RPCs depicted horizontally and sorted
vertically to highlight their relationships. This type of visualization proves
highly beneficial for performance analysis of individual requests, allowing
for a detailed investigation into how each RPC affects the overall end-to-end
response time.

However, these visualizations exhibit certain limitations when it comes to
conducting more complex performance analyses. For instance, observing the
performance of individual end-to-end requests might result in misleading
insights if not contextualized appropriately (Davidson et al., 2023). Indeed,
a request’s response time can only be deemed anomalous when compared with
other requests of the same type (Anand et al., 2020). Additionally, engineers
are often more inclined to investigate recurrent response time trends rather
than focusing on the performance of individual requests (Parker et al., 2020).
Diverse end-to-end response time behaviors may be associated with specific
request characteristics, such as particular RPC execution paths or RPC
performance behaviors. Consequently, engineers may wish to identify these
characteristics to uncover potential performance issues, and gather a more
comprehensive picture of the system performance (Parker et al., 2020;
Krushevskaja and Sandler, 2013; Cortellessa and Traini, 2020).

Distributed tracing tools currently lack sufficient support for this type of
analysis, which often necessitates the concurrent use of multiple
visualizations and tools, such as Jaeger (Technologies, 2023) and Kibana
(Elastic, 2023b) (Davidson et al., 2023). A naive strategy involves initially
recognizing repetitive performance behaviors for further investigation,
followed by the examination of individual requests to characterize relevant
performance behaviors. This can be accomplished by detecting “modes” within
the end-to-end response time distribution (for instance, using Kibana), which
represent meaningful recurring performance behaviors. Following this, samples
of requests associated with each mode can be extracted and examined
individually (for instance, using Jaeger’s swimlanes) to identify distinct
characteristics that contribute to specific performance behaviors or modes.

However, this method can be particularly laborious as it requires manual
inspection and comparison of multiple requests across diverse visualizations
and tools. Moreover, even when the method is successful, it may not provide a
satisfactory level of confidence. Indeed, determining the specific
characteristics associated with a particular distribution mode necessitates
verifying that these characteristics appear _exclusively_ in requests that
exhibit this particular end-to-end response time behavior. This task can be
challenging by using current tools.

Figure 2. End-to-end response time distribution.

Consider the scenario illustrated in Fig. 2, which represents the distribution
of end-to-end response times for a specific type of request, such as loading a
website homepage. As can be observed in the figure, requests demonstrate four
distinct response time behaviors, _i.e.,_ modes. Suppose that the rightmost
mode is characterized by a unique request characteristic, specifically an RPC
that exhibits slower execution time111Henceforth, the term _execution time_
will be used to denote the response time of a generic RPC. Conversely, _end-
to-end response time_ will refer to the response time of the root RPC that
triggers all subsequent RPC invocations. . That is, this specific RPC shows
increased execution time in all requests belonging to the rightmost mode
(_e.g.,_ due to an expensive task), but not in others. With the current
distributed tracing tools, identifying patterns like this can be particularly
challenging. Current distributed tracing tools lack targeted methods to
simplify the analysis of RPC attributes, such as execution time, and their
relationship with end-to-end response time.

##  3\. vamp

vamp aims to enhance performance analysis of microservices systems by
simplifying the investigation of attributes pertaining to specific RPC and
their relationship with end-to-end response time. In this section, we first
introduce the core insights that underpin vamp, its primary visual components,
and the interaction modality. Then, we describe how these visual components
fit within the vamp dashboard, and detail the vamp architecture and
implementation.

###  3.1. Visual Components

The core insight behind vamp is to make explicit the relationship between RPC
attribute values and end-to-end response time. RPC attributes could refer to
several aspects, such as the _frequency_ of RPC invocation within a request,
or the associated _execution time_. vamp leverages two main interactive
components to highlight this relationship: a _tree_ and a _histogram_. The
tree provides an aggregated view of the requests’ workflows in terms of RPC
invocations, while the histogram displays a traditional distribution plot of
the end-to-end response time. Users can interact with the tree to examine how
specific attribute values, related to a particular RPC execution path,
influence the end-to-end response time; we refer to this as _forward
analysis_. Conversely, starting from the histogram, users can investigate how
specific end-to-end response time behaviors are associated with certain RPC
attribute values; this is referred to as _backward analysis_. In the
following, we will first describe in detail the characteristics of these two
main visual components. Then, in the subsequent subsection, we will detail the
interaction modality of vamp.

(a) Tree

(b) Histogram

Figure 3. Visual Components

####  3.1.1. Tree

This visualization component takes inspiration from the Jaeger comparison tool
(Farro, 2018), which allows users to compare two end-to-end requests and
highlight their structural differences. We have redesigned this approach by
extending its capabilities beyond the comparison of two requests, thereby
allowing aggregated analysis of multiple end-to-end requests. In a nutshell,
the vamp tree provides an aggregated view of the RPC workflows performed by a
set of end-to-end requests, as shown in Fig. 3(a). Each node of the tree
represents a RPC invocation within a specific execution path, where the
leftmost node represents the root RPC, and edges indicate direct RPC
invocation. For instance, in Fig. 3(a) the node labeled as RPCE represents the
execution path RPCA$\rightarrow$RPCB$\rightarrow$RPCE. As can be observed by
the figure, the same RPC can appear in multiple nodes (_e.g.,_ RPCF), as it
can be invoked within multiple different execution paths. A RPC execution path
will appear in the tree if and only if it is present in at least one of the
requests being analyzed. It is worth noting that when a particular RPC invokes
the same RPC multiple times, this leads to a single node in the tree. In other
words, if the RPCA invokes the RPCD multiple times, there will be only one
child node referring to RPCD.

vamp utilizes color encoding to highlight RPC execution paths that are worthy
of investigation based on their attribute values. It currently supports the
analysis of two kinds of attributes: _execution time_ and _frequency_. The
first one denotes the (average) execution time of the RPC within a specific
execution path in each request, while second one indicates the path frequency,
_i.e.,_ how many times it occurs within each request. We use color encoding to
emphasize RPC execution paths with higher variance in their attributes. The
key intuition here is that RPC execution paths showing higher variance in
their attributes are likely to manifest different behaviors that can
potentially affect the end-to-end response time. For instance, a higher
frequency of a particular RPC invocation within a request could result in a
longer end-to-end response time. Or similarly, a slower RPC execution time may
correspond to a prolonged end-to-end response time.

We employ a continuous color scale to depict the variability in the attribute
values associated. This scale is based on the Coefficient of Variation
(CV)(Everitt, 1998), _i.e.,_ a standardized measure of dispersion that is
defined as the ratio of the standard deviation to the mean. As execution times
in distributed systems are well known to be subject to long tails (Dean and
Barroso, 2013), when dealing with this attribute, we apply outlier filtering
by removing execution times values greater than the 99th percentile. A CV of 0
results in a white node, indicating no variability. On the other hand, a CV
greater than or equal to 1 results in a red node, suggesting a high
variability in the attribute values. The shade of color gradually transitions
from white to red as the CV value increases.

####  3.1.2. Histogram

The vamp histogram component (shown in Fig. 3(b)) depicts a traditional
distribution plot of the end-to-end response time. These kinds of
visualizations are frequently used in practice for performance analysis, and
are provided by several tools, _e.g.,_ Kibana (Elastic, 2023b). According to
recent research (Davidson et al., 2023), understanding the distribution of
end-to-end response times stands as core activity in modern performance
analysis practice. The histogram component provided by vamp aims to facilitate
this process by supporting the identification of specific performance
behaviors that are worthy of investigation. The user can visually identify
“modes” in the response time distribution, which indicate meaningful recurring
performance behaviors, to start a targeted investigation on these requests, as
we will detail in the subsequent subsection.

###  3.2. Interaction modalities

(a) Forward analysis

(b) Backward analysis

Figure 4. Interaction modalities

vamp supports bidirectional analysis, allowing users to initiate their
analysis from either the tree (_forward analysis_) or the histogram (_backward
analysis_). In the following, we provide detailed descriptions of both these
interaction modalities.

####  3.2.1. Forward analysis

Fig. 4(a) depicts an illustrative example of forward analysis. By examining
the tree, the user can identify “suspicious” RPC execution paths that exhibit
high variability in the corresponding attribute values. For instance, when
analyzing the execution time attributes, the user can identify RPCs that show
highly varying execution times, and, by clicking on the corresponding node,
they can inspect the recurring execution time behaviors associated with the
path, displayed in the form of a bar chart, as shown in Fig. 4(a). Each bar
refers to a specific execution time range (see y-axis labels), and it shows
the percentage of requests with RPC execution time falling in that range. In
order to identify meaningful ranges, we employ a widely-used clustering
algorithm, namely K-means (Lloyd, 1982). In particular, we run the algorithm
on-the-fly after the user click with $k$ ranging from 2 to 5 and we select the
results showing the highest silhouette score (Rousseeuw, 1987). Each bar
represents a meaningful recurring execution time behavior, and the user can
click on each bar to see how this behavior reflects in the end-to-end response
time. This relation is shown by highlighting in red the area of the
distribution that shows this particular RPC execution time behavior. For
instance, in Fig. 4(a), we can observe that when the selected RPC has an
execution time ranging between 400 and 600 milliseconds, it can lead to end-
to-end response times that range between 700 and 800 milliseconds.
Understanding these kinds of relationships would have been way more
challenging by using currently available tools. It is worth noticing that the
same interaction modality also applies when analyzing different RPC
attributes, such as _occurences_.

####  3.2.2. Backward analysis

In the backward analysis, the user can start its investigation directly from
the histogram component. The user selects a specific range of end-to-end
response time using a slider selector, as shown in Fig. 4(b). This selection
triggers an update in the tree component’s color scheme, shifting its semantic
from variability to divergence. In other words, the updated color scheme will
now denote the degree of divergence in the attribute values of the selected
set of requests (_i.e.,_ those that show end-to-end response time in the
selected range) when compared to those in other requests. A red node indicates
that the corresponding RPC execution path shows considerably different
attribute values in the selected requests when compared to other requests,
suggesting a possible relationship between the selected end-to-end response
time and the RPC execution path. Conversely, a white node indicates similar
attribute values, and therefore a weak relation. We quantify the degree of
divergence using Kullback-Leibler divergence (Boyd and Vandenberghe, 2004),
where values close to 0 indicate nearly identical distributions (white), while
values close to or higher than 1 indicate highly different distributions
(red).

The user can then delve deeper into each RPC execution time behavior by
clicking on the corresponding node. This action lets appear at screen two new
histograms (in the bottom right corner) representing the distributions of the
execution time in the selected RPC execution path, respectively in the
selected requests (in red) and in other requests (in grey). In doing so, the
user can effectively analyze how particular ranges of the end-to-end response
time distribution correlate with specific RPC attribute values.

###  3.3. Dashboard

Figure 5. vamp’s Dashboard

Fig. 5 outlines the vamp dashboard. As can be observed by the figure the two
main visual components, namely the tree and the histogram, are positioned in
the center-left and in the upper-right corners, respectively. The space in the
bottom-right is intentionally left blank and will be used to display
supplementary visualization components during the interaction, _e.g.,_ the bar
chart (for forward analysis) and the two histograms (for backward analysis).

It’s worth noting that vamp is specifically designed to assist in analyzing
requests from the same class, _i.e.,_ those originating from the same root
RPC. As part of this process, the user is required to first select the root
RPC and the RPC attribute (_i.e.,_ execution time or path frequency) to be
investigated, before proceeding with the actual analysis.

The user can select the root RPC using either a dropdown menu A or a search
text-box B . Similarly, the RPC attribute (execution time or frequency) to be
analyzed can be selected using a dropdown menu D . Additionally, the dashboard
includes a date-time range selector C , where the user can specify the start
and end date-times. This feature allows for analyses at different time
granularities (_e.g.,_ monthly, weekly, and daily) or over specific time
ranges known to include system anomalies.

To enhance user experience during the interaction with the tool, vamp supports
pinch gestures to enable zoom in and zoom out of the tree. In addition, it
allows the user to hide the RPCs invoked within a particular execution path by
double-clicking on the related node.

###  3.4. Architecture and Implementation

Figure 6. vamp’s Workflow

Fig. 6 outlines the key architecture components of vamp. The _Trace Collector_
(_i.e.,_ Jaeger (Technologies, 2023)) continuously collects traces from the
microservices system and stores them in a _Trace Storage_ (_i.e.,_
Elasticsearch (Elastic, 2023a)). Given the large volume of data collected each
day, we have devised a _Preprocessing_ step to enhance the efficiency of
interaction with vamp. This preprocessing step operates in batches and is
intended to be executed periodically (_e.g.,_ hourly or daily). For each end-
to-end request (_i.e.,_ trace), vamp recursively reconstructs all the involved
RPC execution paths, along with their attribute values (namely, execution
times and frequencies), and stores them in an _Optimized Trace Storage_ based
on MongoDB. Each path associated with a request is stored as a separate
document in a dedicated MongoDB collection, and includes: the name of the
path, the trace ID, the number of occurrences of the path in the trace, the
observed execution time, the timestamp, and the name of the root RPC.
Similarly, vamp stores the end-to-end response time values, along with related
information, in a separate MongoDB collection. This information includes the
RPC root, the trace ID, the response time value, and the timestamp. This data
reorganization allows for greater flexibility in easily and efficiently
querying the data needed for the vamp dashboard to function properly. As can
be seen from Fig. 6, the _Dashboard_ app directly queries the _Optimized Trace
Storage_ to efficiently generate visualizations.

vamp currently supports distributed traces stored in the Jaeger (Technologies,
2023) format using Elasticsearch (Elastic, 2023a) as _Trace Storage_ , but it
can be easily extended to other technologies. The dashboard and visual
components have been developed using D3.js, which handles the visualization
rendering, and Flask, which serves as the backend service. The preprocessing
scripts are implemented in Python.

##  4\. Evaluation

The conducted evaluation is centered around one main research question: _To
what extent does vamp support performance analysis? _ We want to understand
whether vamp can be successfully utilized to gain insights about the
relationship between request attributes and end-to-end performance response
time.

In the following, we first describe the methodology used to gather the answer.
Then, we report and discuss the results of the experimental evaluation.
Finally, we describe the threats to validity of our study.

###  4.1. Methodology

To achieve our study _goal_ , we generate 33 datasets of distributed traces,
where each dataset reflects a distinct scenario that induces a specific
variation in the relationships between request attributes and end-to-end
response time. Subsequently, we manually analyze each dataset using vamp to
evaluate the effectiveness of our tool in highlighting these relationships.

####  4.1.1. Datasets generation

The 33 datasets are generated from TrainTicket (Zhou et al., 2021), which, as
best as we know at the time of writing, is the largest and most complex open-
source microservice-based system. TrainTicket provides a typical train ticket
booking web service; it involves 41 microservices implemented in four
programming languages, and it utilizes Jaeger (Technologies, 2023) and
Elasticsearch (Elastic, 2023a) for collecting and storing distributed traces.
We have chosen TrainTicket as a representative case system due to its
complexity and because it has been recently used in software engineering
research (Traini and Cortellessa, 2023; Zhang et al., 2022; Li et al., 2021;
Zhou et al., 2021).

Each dataset of our study contains distributed traces related to one specific
_root RPC_ of the system, which are stored on Elasticsearch using the standard
Jaeger format.

To simulate different scenarios that induce different variations in the
relationship between RPC attributes and end-to-end response time, we rely on
two different approaches: (i) we inject synthetic performance issues in
specific RPCs to increase the overall end-to-end response time, and (ii) we
use complex mixtures of varying workloads that may alter the relationships
between RPC execution time/occurrences and end-to-end response time. These two
distinct approaches lead to the generation of two categories of datasets.

The first category of datasets is generated using a methodology similar to the
one presented in (Traini and Cortellessa, 2023; Cortellessa and Traini, 2020).
Initially, the system’s source code is modified to inject random performance
issues. Following this, load-testing sessions are run to simulate user
interactions with the system and generate distributed traces. Each injected
performance issue affects approximately 10% of requests, introducing a delay
into one specific RPC.

To generate a dataset, we first select two random RPCs that will be impacted
by the performance issues. Subsequently, we choose a random delay to increase
the end-to-end response time by $x\%$, where $x\in\\{10,20,30\\}$. In
addition, in half of the datasets, we inject a random delay of $y\%$ (with
$y\in\\{10,20,30\\}$) into an asynchronous RPC, which does not produce any
effect on the end-to-end response time. This is a common practice used to test
the robustness of pattern detection approaches in the context of microservices
systems (Cortellessa and Traini, 2020; Traini and Cortellessa, 2023;
Krushevskaja and Sandler, 2013). After modifying the system accordingly, we
conduct load-testing sessions to generate the distributed trace datasets. Each
load testing session involves 20 synthetic users, simulated by Locust (Jonatan
et al., 2023). Each user makes a request to the system and randomly waits
between 1 and 3 seconds before making the next request. Each session lasts for
20 minutes. Using this methodology, we generate 20 datasets featuring various
combinations of performance issues that affect different RPCs with different
delays. For a more detailed explanation of this process, we refer readers to
the work of Traini and Cortellessa (Traini and Cortellessa, 2023). Due to
space constraints, we do not elaborate further here.

The second kind of dataset does not involve any performance issue injection,
but it is generated using a more elaborate workload generator. Similarly to
recent studies (Liao et al., 2021, 2020) we use load mixtures that involve
multiple types of simulated users (_i.e.,_ load drivers), where each user type
performs different classes of requests on the system. For example, some types
of users may only visit the homepage and subsequently search trains for some
random locations, while others first login into the system and then book
random tickets. Besides this, we also ensure that the number of simulated
users per type keeps changing over time. In this way, workloads will more
closely resemble real-world ones, as they generate mixtures of different
classes of requests that change over time (Ardelean et al., 2018). To this
aim, we slightly modified PPTAM (Avritzer et al., 2019), a workload generator
that involves 5 different user types, to continuously change the number of
users of each type at run-time. Overall, the number of simultaneous users
ranges from a minimum of 20 to a maximum of 31, and the load-testing session
lasts for 1 hour. The workload fluctuations over time are randomly generated
upfront. This process leads to 13 distinct datasets, each one related to a
different API.

The generations of the datasets were done on a bare-metal machine running
Linux Ubuntu 18.04.2 LTS on a dual Intel Xeon CPU E5-2650 v3 at 2.30 GHz, with
a total of 40 cores and 80 GB of RAM. All non-mandatory background processes
except SSH are disabled, and we ensured that no other users interacted with
the dedicated machine during our experiments.

To enhance clarity throughout the rest of the article, we will use specific
notations for different categories of datasets. Datasets characterized by
performance issues (_i.e.,_ first category) will be referred to as
$\widehat{D}_{i}$, where $1\leq i\leq 20$. Conversely, datasets free from
performance issues (_i.e.,_ second category) will be denoted as $D_{i}$, with
$1\leq i\leq 13$.

####  4.1.2. Manual analysis

To assess the effectiveness of our approach, two authors conducted manual
inspections of the 33 distributed trace datasets using vamp. Our evaluation
focused on determining the extent to which vamp facilitated the comprehension
of the relationship between RPC execution time/frequency and end-to-end
response time.

It is worth noting that neither author was aware of the specific performance
issues or the workload variations present in each dataset. This is because the
process for the dataset generation, including performance issue injections and
load testing modifications, was entirely random and automated. Nonetheless,
both authors were familiar with the TrainTicket system before the study.

###  4.2. Results

vamp has proven to be effective in highlighting the relationship between RPC
execution time and end-to-end response time, throughout all the datasets
featuring injected performance issues. The analysis was straightforward for
the majority of the datasets (18 out of 20), demanding minimal interaction
with vamp. In these datasets, both _forward_ and _backward analysis_
demonstrated comparable effectiveness, with no noticeable difference in the
effort needed to understand these relationships. Due to space constraints, we
are unable to present the exhaustive results of our analyses across all
datasets. However, we have included a selection of representative examples
that underscore both the utility and potential challenges associated with
employing vamp. Additionally, for the sake of completeness, we have made
available screenshots capturing interactions with vamp across all the datasets
in a supplementary replication package (Traini et al., 2023).

Figure 7. _Forward analysis_ on _execution time_ for dataset
$\widehat{D}_{2}$.

Fig. 7 showcases an example of _forward analysis_ using the dataset
$\widehat{D}_{2}$. As depicted in the left screenshot, vamp significantly
streamlines the identification of the two RPCs impacted by performance issues,
_i.e.,_ the ones highlighted in bright red. Following this, the user can
select these nodes to investigate correlations between specific RPC execution
times and end-to-end response times. For example, the screenshot on the right
of Fig. 7 reveals that the selected RPC execution path (highlighted in green)
exhibits two distinct execution time behaviors: in 9.87% of the requests, the
RPC getRouteByTripId has an execution time ranging from 27.46 to 33.67
milliseconds, and in the remaining 90.13% of requests, the execution time
ranges from 2.62 to 11.25 milliseconds. This screenshot displays the view of
vamp during the investigation of the first behavior, that is, after clicking
on the corresponding bar (highlighted in red). As evident from the figure,
vamp reveals that all the requests with an execution time ranging from 27.46
to 33.67 milliseconds in the selected RPC execution path fall within a
specific region of the end-to-end response time distribution, as shown by the
red highlight in the histogram. Understanding these kinds of relationships
would have been particularly challenging when using traditional performance
analysis tools.

Figure 8. _Backward analysis_ on _execution time_ for dataset
$\widehat{D}_{9}$.

Fig. 8 offers another example of how vamp allows users to rapidly identify the
RPC responsible for a particular end-to-end response time deviation.
Specifically, this figure demonstrates an instance of a vamp _backward
analysis_ using the dataset $\widehat{D}_{9}$. The left screenshot shows that
by selecting a specific range of end-to-end response times, the user can
immediately pinpoint the RPC execution paths that display significantly
divergent behavior in the execution time (highlighted in bright red). The
screenshot on the right displays the investigation of one of these nodes
(_i.e.,_ the one highlighted in green), illustrating how vamp assists users in
comprehending the correlation between specific RPC execution times and the
selected range of end-to-end response times. For instance, it is noticeable
that when the RPC getRouteByTripId has an execution time exceeding 27
milliseconds, it results in an end-to-end response time that falls within the
range of 137 and 168 milliseconds.

(a)

(b)

Figure 9. _Forward analysis_ on _execution time_ for dataset
$\widehat{D}_{1}$.

Another interesting aspect of vamp is its ability to identify execution time
fluctuations in RPCs that do not have influence on the end-to-end response
time. For instance, Fig. 9 illustrates two distinct execution time behaviors
in the selected RPC calculateSoldTicket: one ranging between 33.42 and 55.95
milliseconds, and another between 1.03 and 14.96 milliseconds. Through the use
of vamp, we were able to easily notice the lack of correlation between the
execution time of this RPC and the end-to-end response time. As illustrated in
Figures 9(a) and 9(b), the selected execution time behaviors (_i.e.,_ the bars
highlighted red) are evenly distributed across the end-to-end response time
distribution, implying a lack of notable correlation with specific regions of
the end-to-end response time. This indicates that even if the RPC execution
time varies drastically from one request to another, it does not have any
significant impact on the end-to-end response time.

In two datasets, specifically $\widehat{D}_{4}$ and $\widehat{D}_{19}$, the
analysis was more complex, requiring a higher number of interactions with
vamp. The peculiarity of these datasets was that the two performance issues
led to an increased end-to-end response time that overlaps within the same
range. This made the connection between the RPC execution time and end-to-end
response time more challenging to understand. We did not include the specific
details of these cases in the paper because of limited space, but we refer the
reader to our supplementary materials (Traini et al., 2023) for the related
screenshots.

(a)

(b)

(c)

Figure 10. _Forward analysis_ on _frequency_ for dataset $D_{2}$.

With regards to the 13 datasets in the second category, we found that a
substantial majority of them - precisely 11 datasets - feature a unique mode
in the end-to-end response time distribution. Given the objectives of our
analysis, these cases were not considered. Consequently, we used two datasets
for our evaluation. The first dataset, $D_{1}$, consists of requests
originating from the root RPC getByCheapest, while the second dataset,
$D_{2}$, comprises requests initiated from queryInfo. vamp enabled us to
characterize the correlation between the frequency of each RPC execution path
and specific modes of the end-to-end response time. Fig. 10 provides an
example of this characterization, illustrating that each mode of the end-to-
end distribution corresponds to a specific number of invocations of a selected
RPC execution path (highlighted in green). For example, as depicted in Fig.
10(c), the right-most mode is characterized by 14 invocations of the path
queryInfo $\rightarrow$ queryForStationId. Similarly, the center mode is
characterized by 6 invocations of this path, while the left-most mode is
marked by 2 invocations. Uncovering such patterns using traditional
observability tools would have been notably challenging.

Summing up, we answer our RQ as follows: vamp proved to be effective in
supporting performance analysis of microservices. In 18 out of the 20 datasets
involving performance issues, we were able to rapidly identify the affected
RPCs, their corresponding execution time behaviors, and their relationship
with end-to-end response time. However, in a few specific cases (2 out of 20
datasets), the analysis proved to be more challenging, necessitating a greater
number of interactions with vamp. Moreover, our evaluation demonstrates how
vamp can facilitate an understanding of how structural differences in requests
(_i.e.,_ varying frequencies of RPC execution paths) influence end-to-end
response time.

###  4.3. Threats to Validity

####  4.3.1. Construct validity

We conducted the evaluation in-house rather than using external participants.
Our familiarity with the tool and the experimental setup could potentially
introduce bias into the evaluation outcomes. To mitigate this threat, we
generated 20 diverse scenarios randomly, each involving various combinations
of performance issues. Additionally, the two authors who conducted the
evaluations were kept uninformed about the RPCs affected by the performance
issues. Using artificial delays as part of the evaluation may not perfectly
mirror real-world performance issues. However, this methodology aligns with
prevailing practices in software engineering research, as evidenced by several
studies (Traini and Cortellessa, 2023; Liao et al., 2021; Laaber and Leitner,
2018; Luo et al., 2021). Furthermore, in contrast to many previous studies
(Liao et al., 2021; Laaber and Leitner, 2018; Luo et al., 2021), which
typically employ a limited set of predetermined regressions with fixed
magnitudes, our approach offers a more comprehensive evaluation on 20 diverse
scenarios involving different combinations of RPCs and delay magnitudes

####  4.3.2. Internal validity

The workloads used in our experimental setup may not be representative of
real-world workloads. To (partially) mitigate this limitation, we perform an
additional analysis using mixtures of continuously changing workloads,
generated through PPTAM (Avritzer et al., 2019). Our evaluation may be subject
to confirmation bias, wherein the authors may unconsciously confirm their pre-
existing beliefs on the effectiveness of vamp. Nonetheless, the results
obtained using vamp unambiguously demonstrate its effectiveness across a
majority of the datasets evaluated. In the interest of transparency and to
enable readers to independently assess this evidence, we have made all
screenshots documenting the use of vamp across the datasets in our study
publicly available (Traini et al., 2023).

####  4.3.3. External validity

We cannot ensure that vamp can achieve the same effectiveness on other
datasets outside our experimental setup (_e.g.,_ real world scenarios).
Nevertheless, through an evaluation on 33 datasets, we have demonstrated that
our approach effectively aids in the performance analysis of microservices
systems. vamp’s efficiency was evaluated on datasets of varying sizes, ranging
from 11181 to 22348 requests. It’s worth noting that real-world microservices
systems may involve a much larger volume of requests. As part of our future
work, we plan to enhance the scalability of vamp by incorporating sampling
techniques and optimizing preprocessing procedures.

##  5\. Related work

Previous research on visualization for distributed systems has primarily
focused on analyzing individual requests or comparing two requests. The
swimlane visualization, a widely used technique to represent individual end-
to-end executions, was originally proposed by Singelman _et al._ (Sigelman et
al., 2010). Today, most distributed tracing tools offer this visualization.
TraVista (Anand et al., 2020) enhances the standard swimlane visualization by
augmenting it with information that assists users in contextualizing the
performance of the analyzed request in relation to others. Beschastnikh _et
al._ (Beschastnikh et al., 2020) introduced a novel visualization tool called
ShiViz, which includes an interactive time-space diagram for visualizing
individual end-to-end executions of a distributed system.

Sambasivan et al. (Sambasivan et al., 2013) studied and compared three
visualization approaches (_i.e.,_ side-by-side view, difference view, and
animation) for comparing two request-flow traces. Jaeger (Technologies, 2023)
provides a feature to visually compare the structural characteristics of two
requests (Farro, 2018).

Several visualizations have also been introduced to analyze the performance
behaviors of multiple end-to-end requests in aggregate. One example is
TransVis by Beck _et al._ (Beck et al., 2021), which provides a visualization
technique for specifying and analyzing transient performance behaviors in
microservice systems. Other examples of visual techniques for aggregate
performance analysis can be found in commercial APM tools (Ahmed et al.,
2016), such as Dynatrace, AppDynamics, or Instana. For instance, Dynatrace’s
Service Flow feature (Inc., 2023) allows to display aggregate workflows of
end-to-end requests along with their associated characteristics.

To the best of our knowledge, despite the many existing visualization
techniques for microservices performance analysis, there is still a lack of
dedicated visualizations to analyze the correlation between requests’
attributes and their end-to-end performance behavior, which is the goal of our
study.

Other related works include the recent Davidson and Mace’s survey (Davidson
and Mace, 2022), which underscores the critical role of visualization within
systems research, and the qualitative interview study conducted by Davidson
_et al._ (Davidson et al., 2023), which highlighted the limitations of current
distributed tracing tools. Davidson _et al._ ’s findings involved several open
research challenges that span multiple research areas, including visualization
research.

##  6\. Conclusion

In this paper, we presented vamp, a novel visual analytics tool for
microservices performance analysis. vamp overcomes the limitations of current
distributed tracing tools by providing a wide set of interactive
visualizations that enables effective performance analysis of multiple end-to-
end requests. Through an evaluation of 33 datasets generated from an
established open-source microservices system, we demonstrate how vamp can be
effectively used to understand the relationship between the RPC attributes and
end-to-end response time. For future work, we plan to enhance the efficiency
of our tool to facilitate its transition to practice. As part of this process,
we intend to validate our future improvements using real-world distributed
traces from large-scale microservices systems, similar to those shared by
Alibaba (Luo et al., 2021). To aid reproducibility we provide the data and
source code needed to replicate our findings (Traini et al., 2023).

###### Acknowledgements.

This work is partially supported by European Union - NextGenerationEU -
National Recovery and Resilience Plan (Piano Nazionale di Ripresa e
Resilienza, PNRR) - Project: “SoBigData.it - Strengthening the Italian RI for
Social Mining and Big Data Analytics” - Prot. IR0000013 - Avviso n. 3264 del
28/12/2021, and by Territori Aperti (a project funded by Fondo Territori,
Lavoro e Conoscenza CGIL CISL UIL).

## References

  * (1)
  * Ahmed et al. (2016) Tarek M. Ahmed, Cor-Paul Bezemer, Tse-Hsun Chen, Ahmed E. Hassan, and Weiyi Shang. 2016.  Studying the Effectiveness of Application Performance Management (APM) Tools for Detecting Performance Regressions for Web Applications: An Experience Report. In _Proceedings of the 13th International Conference on Mining Software Repositories_ (Austin, Texas) _(MSR ’16)_. Association for Computing Machinery, New York, NY, USA, 1–12.  https://doi.org/10.1145/2901739.2901774
  * Anand et al. (2020) Vaastav Anand, Matheus Stolet, Thomas Davidson, Ivan Beschastnikh, Tamara Munzner, and Jonathan Mace. 2020.  Aggregate-Driven Trace Visualizations for Performance Debugging.  arXiv: 2010.13681 [cs] Number: arXiv:2010.13681.
  * Ardelean et al. (2018) Dan Ardelean, Amer Diwan, and Chandra Erdman. 2018.  Performance Analysis of Cloud Applications. In _Proceedings of the 15th USENIX Conference on Networked Systems Design and Implementation_ _(NSDI’18)_. USENIX Association, USA, 405–417.  event-place: Renton, WA, USA.
  * Avritzer et al. (2019) Alberto Avritzer, Daniel Menasché, Vilc Rufino, Barbara Russo, Andrea Janes, Vincenzo Ferme, André van Hoorn, and Henning Schulz. 2019.  PPTAM: Production and Performance Testing Based Application Monitoring. In _Companion of the 2019 ACM/SPEC International Conference on Performance Engineering_ (Mumbai, India) _(ICPE ’19)_. Association for Computing Machinery, New York, NY, USA, 39–40.  https://doi.org/10.1145/3302541.3311961
  * Beck et al. (2021) Samuel Beck, Sebastian Frank, Alireza Hakamian, Leonel Merino, and André van Hoorn. 2021.  TransVis: Using visualizations and chatbots for supporting transient behavior in microservice systems. In _2021 Working Conference on Software Visualization (VISSOFT)_. IEEE, 65–75.
  * Beschastnikh et al. (2020) Ivan Beschastnikh, Perry Liu, Albert Xing, Patty Wang, Yuriy Brun, and Michael D. Ernst. 2020.  Visualizing Distributed System Executions.  _ACM Trans. Softw. Eng. Methodol._ 29, 2 (March 2020).  https://doi.org/10.1145/3375633 Place: New York, NY, USA Publisher: Association for Computing Machinery.
  * Boyd and Vandenberghe (2004) Stephen Boyd and Lieven Vandenberghe. 2004.  _Convex Optimization_.  Cambridge University Press.  https://doi.org/10.1017/CBO9780511804441
  * Cortellessa and Traini (2020) Vittorio Cortellessa and Luca Traini. 2020.  Detecting Latency Degradation Patterns in Service-Based Systems. In _Proceedings of the ACM/SPEC International Conference on Performance Engineering_ _(ICPE ’20)_. Association for Computing Machinery, New York, NY, USA, 161–172.  https://doi.org/10.1145/3358960.3379126
  * Davidson and Mace (2022) Thomas Davidson and Jonathan Mace. 2022.  See it to believe it? The role of visualisation in systems research. In _Proceedings of the 13th Symposium on Cloud Computing_ _(SoCC ’22)_. Association for Computing Machinery, New York, NY, USA, 419–428.  https://doi.org/10.1145/3542929.3563488
  * Davidson et al. (2023) Thomas Davidson, Emily Wall, and Jonathan Mace. 2023.  A Qualitative Interview Study of Distributed Tracing Visualisation: A Characterisation of Challenges and Opportunities.  _IEEE Transactions on Visualization and Computer Graphics_ (2023), 1–12.  https://doi.org/10.1109/TVCG.2023.3241596
  * Dean and Barroso (2013) Jeffrey Dean and Luiz André Barroso. 2013.  The Tail at Scale.  _Commun. ACM_ 56 (2013), 74–80.  http://cacm.acm.org/magazines/2013/2/160173-the-tail-at-scale/fulltext
  * Elastic (2023a) Elastic. 2023a.  Elasticsearch: The Official Distributed Search & Analytics Engine.  https://www.elastic.co/elasticsearch/ ”Accessed 2023-01-15 18:38”.
  * Elastic (2023b) Elastic. 2023b.  Kibana: Explore, Visualize, Discover Data.  https://www.elastic.co/kibana ”Accessed 2023-01-15 17:38”.
  * Everitt (1998) Brian Everitt. 1998.  _The Cambridge Dictionary of Statistics_.  Cambridge University Press, Cambridge, UK.
  * Farro (2018) Joe Farro. 2018.  Trace comparisons arrive in Jaeger 1.7.  https://medium.com/jaegertracing/trace-comparisons-arrive-in-jaeger-1-7-a97ad5e2d05d ”Accessed 2023-01-15 18:16”.
  * Inc. (2023) Dynatrace Inc. 2023.  Dynatrace. Service Flow.  https://www.dynatrace.com/platform/service-flow/ ”Accessed 2023-02-14 14:10:59”.
  * Jiang and Hassan (2015) Zhen Ming Jiang and Ahmed E. Hassan. 2015.  A Survey on Load Testing of Large-Scale Software Systems.  _IEEE Transactions on Software Engineering_ 41, 11 (2015), 1091–1118.  https://doi.org/10.1109/TSE.2015.2445340
  * Jonatan et al. (2023) Heyman Jonatan, Carl Byström, Joakim Hamrén, and Hugo Heyman. 2023.  An open source load testing tool.  https://locust.io ”Accessed 2023-06-10 13:38”.
  * Kaldor et al. (2017) Jonathan Kaldor, Jonathan Mace, Micha\l Bejda, Edison Gao, Wiktor Kuropatwa, Joe O’Neill, Kian Win Ong, Bill Schaller, Pingjia Shan, Brendan Viscomi, Vinod Venkataraman, Kaushik Veeraraghavan, and Yee Jiun Song. 2017.  Canopy: An End-to-End Performance Tracing And Analysis System. In _Proceedings of the 26th Symposium on Operating Systems Principles_ _(SOSP ’17)_. Association for Computing Machinery, New York, NY, USA, 34–50.  https://doi.org/10.1145/3132747.3132749
  * Krushevskaja and Sandler (2013) Darja Krushevskaja and Mark Sandler. 2013.  Understanding Latency Variations of Black Box Services. In _Proceedings of the 22nd International Conference on World Wide Web_ (Rio de Janeiro, Brazil) _(WWW ’13)_. Association for Computing Machinery, New York, NY, USA, 703–714.  https://doi.org/10.1145/2488388.2488450
  * Laaber and Leitner (2018) Christoph Laaber and Philipp Leitner. 2018.  An Evaluation of Open-Source Software Microbenchmark Suites for Continuous Performance Assessment. In _Proceedings of the 15th International Conference on Mining Software Repositories_ (Gothenburg, Sweden) _(MSR ’18)_. Association for Computing Machinery, New York, NY, USA, 119–130.  https://doi.org/10.1145/3196398.3196407
  * Leone and Traini (2023) Jessica Leone and Luca Traini. 2023.  Enhancing Trace Visualizations for Microservices Performance Analysis. In _Companion of the 2023 ACM/SPEC International Conference on Performance Engineering_ (Coimbra, Portugal) _(ICPE ’23 Companion)_. Association for Computing Machinery, New York, NY, USA, 283–287.  https://doi.org/10.1145/3578245.3584729
  * Li et al. (2021) Bowen Li, Xin Peng, Qilin Xiang, Hanzhang Wang, Tao Xie, Jun Sun, and Xuanzhe Liu. 2021.  Enjoy your observability: an industrial survey of microservice tracing and analysis.  _Empirical Software Engineering_ 27, 1 (2021), 25.  https://doi.org/10.1007/s10664-021-10063-9
  * Liao et al. (2020) Lizhi Liao, Jinfu Chen, Heng Li, Yi Zeng, Weiyi Shang, Jianmei Guo, Catalin Sporea, Andrei Toma, and Sarah Sajedi. 2020.  Using Black-Box Performance Models to Detect Performance Regressions under Varying Workloads: An Empirical Study.  _Empirical Softw. Engg._ 25, 5 (sep 2020), 4130–4160.  https://doi.org/10.1007/s10664-020-09866-z
  * Liao et al. (2021) Lizhi Liao, Jinfu Chen, Heng Li, Yi Zeng, Weiyi Shang, Catalin Sporea, Andrei Toma, and Sarah Sajedi. 2021.  Locating Performance Regression Root Causes in the Field Operations of Web-based Systems: An Experience Report.  _IEEE Transactions on Software Engineering_ (2021), 1–1.  https://doi.org/10.1109/TSE.2021.3131529
  * Lloyd (1982) S. Lloyd. 1982.  Least squares quantization in PCM.  _IEEE Transactions on Information Theory_ 28, 2 (1982), 129–137.  https://doi.org/10.1109/TIT.1982.1056489
  * Luo et al. (2021) Shutian Luo, Huanle Xu, Chengzhi Lu, Kejiang Ye, Guoyao Xu, Liping Zhang, Yu Ding, Jian He, and Chengzhong Xu. 2021.  Characterizing Microservice Dependency and Performance: Alibaba Trace Analysis. In _Proceedings of the ACM Symposium on Cloud Computing_. 412–426.
  * Mace (2017) Jonathan Mace. 2017.  _End-to-End Tracing: Adoption and Use Cases_.  Survey. Brown University.  https://cs.brown.edu/people/jcmace/papers/mace2017survey.pdf
  * Majors et al. (2022) C. Majors, L. Fong-Jones, and G. Miranda. 2022.  _Observability Engineering: Achieving Production Excellence_.  O’Reilly Media, Incorporated.  https://books.google.it/books?id=MbmLzgEACAAJ
  * Newman (2015) Sam Newman. 2015.  _Building Microservices_ (1st ed.).  O’Reilly Media, Inc.
  * O’Hanlon (2006) Charlene O’Hanlon. 2006.  A Conversation with Werner Vogels.  _Queue_ 4, 4 (May 2006), 14:14–14:22.  https://doi.org/10.1145/1142055.1142065 Place: New York, NY, USA Publisher: ACM.
  * Parker et al. (2020) Austin Parker, Daniel Spoonhower, Jonathan Mace, Ben Sigelman, and Rebecca Isaacs. 2020.  _Distributed Tracing in Practice: Instrumenting, Analyzing, and Debugging Microservices_.  O’Reilly Media, Incorporated.
  * Rousseeuw (1987) Peter J. Rousseeuw. 1987.  Silhouettes: A graphical aid to the interpretation and validation of cluster analysis.  _J. Comput. Appl. Math._ 20 (1987), 53–65.  https://doi.org/10.1016/0377-0427(87)90125-7
  * Rubin and Rinard (2016) Julia Rubin and Martin Rinard. 2016.  The Challenges of Staying Together While Moving Fast: An Exploratory Study. In _Proceedings of the 38th International Conference on Software Engineering_ _(ICSE ’16)_. Association for Computing Machinery, New York, NY, USA, 982–993.  https://doi.org/10.1145/2884781.2884871
  * Sambasivan et al. (2016) Raja R. Sambasivan, Ilari Shafer, Jonathan Mace, Benjamin H. Sigelman, Rodrigo Fonseca, and Gregory R. Ganger. 2016.  Principled Workflow-centric Tracing of Distributed Systems. In _Proceedings of the Seventh ACM Symposium on Cloud Computing_ _(SoCC ’16)_. ACM, New York, NY, USA, 401–414.  https://doi.org/10.1145/2987550.2987568
  * Sambasivan et al. (2013) R. R. Sambasivan, I. Shafer, M. L. Mazurek, and G. R. Ganger. 2013.  Visualizing Request-Flow Comparison to Aid Performance Diagnosis in Distributed Systems.  _IEEE Transactions on Visualization and Computer Graphics_ 19, 12 (2013), 2466–2475.  https://doi.org/10.1109/TVCG.2013.233
  * Sigelman et al. (2010) Benjamin H. Sigelman, Luiz André Barroso, Mike Burrows, Pat Stephenson, Manoj Plakal, Donald Beaver, Saul Jaspan, and Chandan Shanbhag. 2010.  _Dapper, a Large-Scale Distributed Systems Tracing Infrastructure_.  Technical Report. Google.
  * Sridharan (2017) Cindy Sridharan. 2017.  Testing Microservices, the sane way.  https://copyconstruct.medium.com/testing-microservices-the-sane-way-9bb31d158c16 ”Accessed 2023-01-16 11:14”.
  * Technologies (2023) Uber Technologies. 2023.  Jaeger: Open source, end-to-end distributed tracing.  https://www.jaegertracing.io/ ”Accessed 2023-01-15 14:10:59”.
  * Traini (2022) Luca Traini. 2022.  Exploring Performance Assurance Practices and Challenges in Agile Software Development: An Ethnographic Study.  _Empirical Software Engineering_ 27, 3 (2022), 74.  https://doi.org/10.1007/s10664-021-10069-3 ISBN: 1573-7616.
  * Traini and Cortellessa (2023) Luca Traini and Vittorio Cortellessa. 2023.  DeLag: Using Multi-Objective Optimization to Enhance the Detection of Latency Degradation Patterns in Service-Based Systems.  _IEEE Transactions on Software Engineering_ (2023), 1–28.  https://doi.org/10.1109/TSE.2023.3266041
  * Traini et al. (2022) Luca Traini, Vittorio Cortellessa, Daniele Di Pompeo, and Michele Tucci. 2022.  Towards effective assessment of steady state performance in Java software: are we there yet?  _Empirical Software Engineering_ 28, 1 (2022), 13.  https://doi.org/10.1007/s10664-022-10247-x
  * Traini et al. (2021) Luca Traini, Daniele Di Pompeo, Michele Tucci, Bin Lin, Simone Scalabrino, Gabriele Bavota, Michele Lanza, Rocco Oliveto, and Vittorio Cortellessa. 2021.  How Software Refactoring Impacts Execution Time.  _ACM Trans. Softw. Eng. Methodol._ 31, 2, Article 25 (dec 2021), 23 pages.  https://doi.org/10.1145/3485136
  * Traini et al. (2023) Luca Traini, Jessica Leone, Giovanni Stilo, and Antinisca Di Marco. 2023.  VAMP - Replication Package.  https://github.com/lucatraini/VAMP.
  * Veeraraghavan et al. (2016) Kaushik Veeraraghavan, Justin Meza, David Chou, Wonho Kim, Sonia Margulis, Scott Michelson, Rajesh Nishtala, Daniel Obenshain, Dmitri Perelman, and Yee Jiun Song. 2016.  Kraken: Leveraging Live Traffic Tests to Identify and Resolve Resource Utilization Bottlenecks in Large Scale Web Services. In _12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16)_. USENIX Association, Savannah, GA, 635–651.
  * Zhang et al. (2022) Chenxi Zhang, Xin Peng, Chaofeng Sha, Ke Zhang, Zhenqing Fu, Xiya Wu, Qingwei Lin, and Dongmei Zhang. 2022.  DeepTraLog: Trace-Log Combined Microservice Anomaly Detection through Graph-based Deep Learning. In _2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE)_. 623–634.  https://doi.org/10.1145/3510003.3510180
  * Zhou et al. (2021) Xiang Zhou, Xin Peng, Tao Xie, Jun Sun, Chao Ji, Wenhai Li, and Dan Ding. 2021.  Fault Analysis and Debugging of Microservice Systems: Industrial Survey, Benchmark System, and Empirical Study.  _IEEE Transactions on Software Engineering_ 47, 2 (2021), 243–260.  https://doi.org/10.1109/TSE.2018.2887384