# Characterizing Architecture Related Posts and Their Usefulness in Stack
Overflow

Musengamana Jean de Dieu<EMAIL_ADDRESS>School of Computer Science, Wuhan
University, 430072 Wuhan, China  Peng Liang<EMAIL_ADDRESS>Mojtaba Shahin
<EMAIL_ADDRESS>School of Computing Technologies, RMIT University,
3000 Melbourne, Australia  Arif Ali Khan<EMAIL_ADDRESS>M3S Empirical
Software Engineering Research Unit, University of Oulu, 90014 Oulu, Finland

###### Abstract

Context: Stack Overflow (SO) has won the intention from software engineers
(e.g., architects) to learn, practice, and utilize development knowledge, such
as Architectural Knowledge (AK). But little is known about AK communicated in
SO, which is a type of high-level but important knowledge in development.

Objective: This study aims to investigate the AK in SO posts in terms of their
categories and characteristics as well as their usefulness from the point of
view of SO users.

Method: We conducted an exploratory study by qualitatively analyzing a
statistically representative sample of 968 Architecture Related Posts (ARPs)
from SO.

Results: The main findings are: (1) architecture related questions can be
classified into 9 core categories, in which “architecture configuration” is
the most common category, followed by the “architecture decision” category,
and (2) architecture related questions that provide clear descriptions
together with architectural diagrams increase their likelihood of getting more
than one answer, while poorly structured architecture questions tend to only
get one answer.

Conclusions: Our findings suggest that future research can focus on enabling
automated approaches and tools that could facilitate the search and (re)use of
AK in SO. SO users can refer to our proposed guidelines to compose
architecture related questions with the likelihood of getting more responses
in SO.

###### keywords:

Architectural Knowledge, Architectural Level Element, Architecture Solution,
Stack Overflow, Usefulness

††journal: Journal of Systems and Software

##  1 Introduction

Technical Questions and Answers (Q&A) sites, such as Stack Overflow (SO), have
revolutionized how users seek knowledge on the Internet [1]. SO has shown to
be the most prominent community Q&A site for knowledge sharing and learning in
software development, and SO leverages the knowledge and skills of its users,
such as developers, to share their thoughts and experience by asking various
types of technical questions related to development and providing answers to
these questions. Also, SO users can learn novel techniques and tools from SO
[2]. SO is predominately being used to solve coding problems [3], and these
problems are often not relevant or less interesting to architects because they
focus on lower-level implementation details [3]. However, ever since this site
started growing and being popular, architects have begun to share their
competencies, experience, and design problems by asking architecture related
questions or providing architecture solutions, such as architecture tactics.
In our recent industrial survey on how developers search for architectural
information [4], practitioners acknowledged Q&A sites (e.g., SO) as the most
useful source of architectural information (e.g., benefits and drawbacks of
architecture solutions). Hence, similar to searching and (re)using existing
coding related answers provided in SO to solve programming related problems,
software engineers (e.g., architects and developers) also search and (re)use
existing architecture solutions in SO for addressing their design concerns.
Thus, SO not only accumulates code examples, but also curates a large number
of architecture solutions provided to a wide range of architecture related
questions or design problems [5] [6].

Although SO users discuss high-level knowledge in SO, for instance,
architecture tactics and quality attributes knowledge [5], architecture
knowledge for technology decisions [6], to date the majority of the existing
studies mainly focus on analyzing programming related knowledge in SO posts
from different perspectives. For example, Diamantopoulos and Symeonidis [7]
employed source code information to improve question-answering in SO, and
Zhang et al. [8] investigated the quality of code examples in SO programming
related posts. Little work has focused on analyzing architectural knowledge
provided in Architecture Related Posts (ARPs) in SO. For instance, Bi et al.
[5] mined posts from SO and structured the design relationships between
architectural tactics and quality attributes used in practice. Liu et al. [9]
extracted SO posts and mined the design pattern use scenarios and related
design pattern pairs. Soliman et al. [10] developed a search approach (i.e., a
domain specific search approach) for searching architecture knowledge in SO.
In another work, Soliman et al. [11] conducted an empirical study with 50
software engineers, who used Google to make design decisions using Attribute
Driven Design [12], and they determined how effective web search engines are
to find relevant architectural information from various sources (including SO)
and to capture AK. Malavolta et al. [13] extracted data from five open source
software repositories (including SO), and mined architectural tactics for
energy-efficiency applied by practitioners in real robotics projects. Tian et
al. [14] studied SO users’ conception of architectural smells using SO posts.
The abovementioned studies extracted ARPs from SO and investigated
architecture knowledge (high-level concepts) from different aspects. However,
prior work is only based on architecture related questions and their
associated answers. In contrast, our work covers the entire ARP, including its
question, all comments under the question, all answers associated with the
question, and all comments under the answers. In addition, no prior study has
specifically investigated architectural knowledge provided in ARPs (answers)
with regard to their usefulness. Moreover, there has been no comprehensive
research on exploring architectural knowledge communicated by SO users in
terms of their types, design contexts, characteristics, and usefulness, which
is the focus of this study. Analyzing and understanding how SO users deal with
architecture design concerns in online developer communities, such as SO,
brings three benefits: (1) it provides key insights about the types of design
problems SO users face during their architecture design and the types of
architecture solutions discussed as well as their usefulness, (2) it can help
to know the design contexts in which architecture problems are raised, and (3)
it can help to know the characteristics of architecture problems and solutions
discussed. These benefits provide an opportunity to develop new techniques and
tools that can help SO users search and (re)use architectural knowledge shared
in online developer communities. Therefore, this study aims to complement
prior works by analyzing the characteristics and categories of ARPs in SO as
well as their usefulness from the point of view of SO users. In this study, we
treated usefulness (one quality criterion of posts, e.g., answers, in Q&A
sites [15]) using the definition in [15] (i.e., are the answers useful to
address the questions?).

To achieve the goal of this study (see Section 3.1), we conducted an
exploratory study to investigate various aspects (e.g., categories and
characteristics) of ARPs in SO. More specifically, we extracted 32,182 posts
from SO. We went on to manually filter out irrelevant posts and got 10,423
candidate ARPs. Since 10,423 candidate ARPs were a quite large dataset, and it
was not easy to manually analyze this size of dataset with human effort and
get accurate and comprehensive results, we used the power statistics and
calculated a representative sample size [16] of these 10,423 ARPs. With a 95%
confidence level and 3% margin of error, the final representative sample size
calculated was 968 ARPs. Then, we randomly selected 968 ARPs from the 10,423
ARPs and analyzed them for answering a set of research questions (see Table
1). Specifically, we manually analyzed the 968 ARPs using open coding and
constant comparison from Grounded Theory (GT) [17] to answer those research
questions. The main results and findings of this study are that: (1) SO users
ask a broad spectrum of architecture related questions ranging from
architecture tool to architecture configuration, architecture implementation
to architecture deployment. (2) The useful architecture solutions are
classified into seven categories as a taxonomy (see Figure 4), such as
solution for architecture configuration, solution for architecture
implementation, architecture tactic, and architecture pattern. One observation
is that the identified categories of these posts (questions and answers) cover
almost all the architecting activities that span from the initial stages
(i.e., architectural analysis and synthesis [18]) of architectural creation as
well as the later stages (i.e., architectural implementation and maintenance &
evolution [19]) in a system lifecycle. Thus our identified categories of ARPs
can support the mentioned architecting activities during the architecture
lifecycle, and SO can be considered as one of the sources of architectural
knowledge [6]. We found that architecture related questions that provide clear
descriptions together with architectural diagrams increase their likelihood of
getting more than one answer, while poorly structured architecture questions
tend to only get one answer. (3) SO users frequently use two terms related to
usefulness (i.e., useful and helpful) to explicitly communicate about the
usefulness of certain architecture solutions provided to their associated
architecture related questions.

This study makes the following three contributions: (1) a classification and
characterization of architecture related questions that SO users (e.g.,
developers) asked in SO; (2) a list of identified design contexts in which
architecture related questions were raised; and (3) a classification (i.e., a
proposed taxonomy) and characterization of useful architecture solutions in
SO. Our study findings can be beneficial to various stakeholders. For example,
researchers can refer to our proposed taxonomy of useful architecture
solutions in SO as a guidance to develop new automated approaches and tools
that could mine and locate architecture solutions (e.g., solution for
architecture configuration, see Figure 4) for addressing similar design
concerns (e.g., questions that ask about architecture configuration, see Table
5). This can facilitate SO users to check the questions and solutions that are
relevant to their design concerns. SO can use our results to better adjust its
answers and comments organization mechanisms and enhance the search and
(re)use of useful architecture solutions in SO.

The rest of this paper is structured as follows: Section 2 presents the
background of the study. Section 3 describes the research methodology, and
Section 4 elaborates the study results. Section 5 analyzes the results and
discusses their implications. Section 6 presents the threats to the validity
of the study results, and Section 7 summarizes the related work. Finally,
Section 8 concludes this work with potential areas of future research.

##  2 Background

In this section, we introduce the background concepts used in this study,
including Stack Overflow, architecture knowledge, architecture problem, design
context, and architecture solution.

###  2.1 Stack Overflow

Stack Overflow is one of the websites that make Stack
Exchange111https://stackexchange.com/sites network, which provides a Q&A
platform for its users to share knowledge across various domains (e.g.,
programming, design, statistics, mathematics). SO users exchange knowledge
related to software development by asking questions or providing answers to
existing questions. Among other development knowledge, architecture knowledge,
such as drawbacks and benefits of architecture solutions (e.g., patterns and
tactics) in certain application domains, has been shared at SO to support
architecting activities [20][21]. Mining architecture knowledge in Q&A
websites, specifically in SO, has been the subject of the architecture
research community in recent years, such as architectural knowledge for
technology decisions [6] and architecture tactics and quality attributes [5],
in order to support the architecting process.

###  2.2 Architecture knowledge

Software Architecture (SA) is a set of structures comprising software
elements, the relationships among them, and the properties of the elements and
relationships [22]. Building an architecture of a software system often
requires knowledge, especially architecture knowledge [23], and skills.
Architecture knowledge, such as architecture decisions and their rationale
[24], benefits and drawbacks of architecture solutions [6], is one of the most
important types of knowledge in software development [22]. Architectural
knowledge is often described in various formats, such as textual and graphical
representation [25] and this knowledge is recorded in various sources, such as
books [22], technical blogs and tutorials [11], developer mailing lists (e.g.,
ArgoUML [26]), Q&A sites (e.g., SO [5]). In this study, we investigated
architecture knowledge discussed in SO from various aspects, such as
categories and characteristics of ARPs in SO, SO users’ discussions on the
usefulness of architecture solutions provided in SO.

###  2.3 Architecture problem

Architecture problems (such as “any testable architecture or design pattern
for an MFC application?”222https://tinyurl.com/2z69uzs5) arise during
development when addressing specific architecture design concerns (e.g.,
quality attributes) and their trade-offs [22]. There are various problems
related to architecture design that are asked in SO. In this study, we
investigated the categories of architecture problems/questions, specifically,
the categories of architecture related questions asked in SO (see Section
4.1).

###  2.4 Design context

Design context of a software system comprises the knowledge that an architect
needs to have about the environment (e.g., a hardware platform) in which a
system is expected to operate [27]. Design contexts can be seen as “conditions
that influence design decisions but are not specified explicitly as
requirements” [28]. Harper and Zheng suggested that design contexts are forces
that influence stakeholders’ concerns [29]. There are some works that
categorize design contexts. Bedjeti et al. identified four context categories
of an architecture viewpoint (i.e., platform context, user context,
application context, and organizational context) [27]. Petersen and Wohlin
provided a checklist for documenting design contexts from six perspectives:
product, processes, practices and techniques, people, organization, and market
[30]. Groher and Weinreich studied environmental factors that influence
architecture decision making [31], and they identified eight categories:
company size, business factors, organizational factors, technical factors,
cultural factors, individual factors, project factors, and decision scope. In
our study, we investigated the design contexts that were discussed in
architectural related posts in SO, and we referred to the classification of
design contexts proposed in two existing studies [27][30] (see Section 4.2).

###  2.5 Architecture solution

Architecture solutions are the fundamental building blocks in modern software
design and they are used to address architecture design concerns [22]. There
are various architecture solutions, such as patterns, tactics, and frameworks
for addressing different design concerns. Architecture patterns (e.g.,
Model–View–Controller, Client-Server, Publish-Subscribe patterns) are reusable
solutions to commonly occurring problems in architecture design within given
contexts [22]. Contrarily to changing implementation (e.g., low-level code),
once an architecture solution (e.g., an architecture pattern) is adopted and
implemented, it is quite difficult and costly to change it [22]. Architecture
patterns determine the overall structure and behavior of a software system
[32] and are typically selected early during development for achieving
multiple system requirements (e.g., quality attributes) [22]. In this study,
we studied SO users’ discussions on the usefulness of architecture solutions,
for example, patterns, tactics, frameworks (see Section 4.5), as well as the
categories and characteristics (in Section 4.6) of architecture solutions that
were considered useful in SO.

##  3 Research design

We carried out an exploratory study on various aspects (e.g., categories and
characteristics) of ARPs in SO. In the following subsections, we describe the
details of the research design of this study, including the goal and Research
Questions (RQs) in Section 3.1 and the execution of this study in Section 3.2.

###  3.1 Goal and research questions

The overall goal of this study based on Goal-Question-Metric approach [33] is
“to analyze the ARPs (questions and answers) in SO for the purpose of
investigating their categories, characteristics, and usefulness from the point
of view of SO users in the context of software development in practice”.
Following the goal of this research, we derived six research questions (see
Table 1) that aim to examine four aspects that highlight the question and
answer threads of SO posts, including (1) categorization of architecture
related questions, (2) the design contexts in which architecture related
questions were raised, (3) characterization of architecture related questions
that have more than one answer and characteristics of architecture related
questions that only have one answer, and (4) categorization and
characterization of architecture solutions that are considered useful.

Table 1: Research questions and their rationale Research Question |  Rationale
---|---
RQ1. What architecture related questions are asked in SO? |  Architecture related questions (design problems) are mainly asked to address certain design concerns (e.g., quality attributes of a system) during architecting activities, for example, architectural analysis. SO curates different types of architecture related questions that are raised with various design issues. The answer to this RQ can help researchers to be aware of the areas of interest of SO users in architecture design and help practitioners to get an insight into the architecture related questions asked in SO so that they can provide practical contributions.
RQ2. What are the design contexts in which architecture related questions were raised? |  Design contexts comprise the knowledge about the environments in which systems are expected to operate [27]. Design contexts are indispensable ingredients that can drive the architecture design of a system [27]. A system of similar functionalities can operate differently in different contexts [30]. Although the importance of considering design contexts during architecture design has been recognized, there is limited understanding on what design contexts are considered in architecture design. The answer to this RQ can help researchers and practitioners be aware of typical design contexts in which architecture related questions are raised in SO.
RQ3. What are the characteristics of architecture related questions in SO that have more than one answer? |  One major challenge during architecture design is choosing the right architecture solutions to address the requirements of the systems [34]. Although different architecture solutions act as alternative solutions to similar architecture problems, they differ in terms of their qualities [34]. Therefore, providing more than one answer (e.g., alternative solutions) to architecture problems/questions is important as they provide a wide range of possibilities for making architecture design decisions. With this RQ, we identify and examine the characteristics of architecture related questions that get more than one answer. By characteristics of an architecture related question, we mean certain features, such as architectural diagrams, in the content of the question or question formulation [35], that distinguish the architecture related question to another or make the architecture related question get attraction from SO users and get more than one answer. The answer to this RQ can help researchers and practitioners know what motivates SO users to provide more solutions to these questions, and consequently improve or prevent unanswered architecture related questions in SO.
RQ4. What are the characteristics of architecture related questions in SO that only have one answer? |  Some architecture related questions fail to continuously get attention from SO users by answering them. Similar to RQ3, we want to examine the factors behind this situation. We study the characteristics of architecture related questions that only get one answer. The answer of this RQ can help researchers and practitioners know what demotivates SO users to continue answering these questions and design general guidelines for SO users to compose architecture related questions with the likelihood of getting more responses in SO.
RQ5. What are the types of architecture solutions provided in SO that are considered useful by SO users? |  There are many architecture solutions (e.g., tactics and patterns) to address architecture related questions provided in SO. However, the quality of solutions/answers provided in SO has been a major concern for researchers and practitioners. As elaborated in the related work (see Section 7), this is evident in the growing number of studies, in which the focus is on analyzing the quality of the content in SO posts from different perspectives, for example, code and text. The results of this RQ can help researchers and practitioners be aware of types (a taxonomy) of architecture solutions considered useful in SO.
RQ6. What are the characteristics of architecture solutions in SO that are considered useful by SO users? |  Zhang et al. [8] argued that accepted, highly voted, and frequently viewed SO posts are not always reliable or useful in SO. Identifying the features of architecture solutions that are considered useful helps to better understand what SO users consider when accepting architecture solutions as useful ones, thus providing insights for improving the current answering mechanism of architecture related questions and helping SO users retrieve their desired architecture solutions.

###  3.2 Study Execution

In this subsection, we describe the process of data collection and analysis of
ARPs. Figure 1 shows an overview of the two processes (i.e., data collection
and analysis).

Figure 1: An overview of data collection and analysis

####  3.2.1 Data collection

Our data collection is divided into two phases, namely Phase I: Gathering
architecture related posts and Phase II: Determining the representative sample
size, as detailed below:

Phase I: Gathering architecture related posts

a) Search terms: Before we decided the most suitable terms for capturing posts
relevant to architecture design, we first performed a pilot search with
several terms, namely “architect*” (i.e., “architect”, “architecture”,
“architectural”, and “architecting”) and “design*” (i.e., “design” and
“designing”), within SO. The process was carried out by using a SQL query
through the query interface provided by StackExchange Data
Explorer333https://data.stackexchange.com/stackoverflow/query/new, which is a
web interface that allows the execution of SQL queries on data from Q&A sites,
including SO. After the pilot search with the mentioned terms, we saw that SO
users mostly use the terms “design*” (i.e., “design” and “designing”) in the
programming context in SO, for instance, “singleton design
pattern”444https://tinyurl.com/8yks7nhm. Moreover, we were aware that Soliman
et al. [6] identified distinctive terms between ARPs and pure programming
posts from their studied sample of SO posts. However, in our study, we did not
use those distinctive terms to search ARPs in SO due to the following two
reasons:

(1) The purposes of our work and Soliman et al.’s work in [6] are different.
The purpose of the work in [6] is technology related architecture knowledge
extraction from SO. Specifically, the authors in [6] identified and analyzed
ARPs that mainly discuss architectural knowledge for technology decisions,
such as the pros and cons of a technology solution in a certain application.
In addition, the authors in [6] claimed that they did not find many pure
architectural concepts (such as architectural pattern or tactic) in their
dataset of ARPs. In contrast, our study takes the problems from a wider scope.
Specifically, our study aims to identify and analyze ARPs from SO by looking
at various architectural information, including architecture patterns,
tactics. Therefore, using the specific distinctive terms, such as versus,
alternative, pros, cons, xmpp, that were found in [6] may lead to missing
other relevant ARPs, which may affect the completeness of the retrieved ARPs.

(2) Using the distinctive terms found in Soliman et al.’s work [6] may lead to
bias in the search results. The relevancy and completeness of extracted ARPs
may affect the correctness of the answers to our six RQs (see Table 1). Thus,
including the specific distinctive terms that were found in [6] in the search
queries may lead to the situation that the search results are biased to those
terms.

Therefore, we selected the general terms “architect*” (i.e., “architect”,
“architecture”, “architectural”, and “architecting”) to be used in our search.
It is worth mentioning that we did not use the search terms to search
exclusively through tags only because tags can sometimes be less informative
and ineffective [36]. There are several disadvantages of using tags as the
only approach to determine whether a post is related to a topic. This is due
to the reason that a user who created a post could be unsure about the title
of the most appropriate tag for their discussion, which can lead to the use of
incorrect or irrelevant tags [37]. For example, in this architecture related
post555https://tinyurl.com/mb9y37z4 that asked for an architecture pattern
that can be used in the design of a single webform application, a developer
used tags (“jc#”, “asp.net”, and “web”), and these tags cannot immediately
tell in which contexts (e.g., architecture or programming context) they are
really used. Another problem with user-defined tags is that users may try to
add as many tags as possible (SO allows up to 5 tags) to raise the number of
views and probably increase the probability of getting responses quickly [38].
Thus, while tags can be helpful to capture posts related to architecture
design, using tags exclusively may miss important posts on this topic. Hence,
we decided to add the title and body of the questions into the search. For
example, by following the criteria of the query interface provided by Stack
Exchange666https://data.stackexchange.com/stackoverflow/query/new, for the
term “architect”, we searched in the title, tags, and body of posts by using
this query: SELECT p.Id, p.Tags, p.Title, p.Body as “Questions Body”, p.Score
as “Questions Score”, p.Answercount as “Answer Count” FROM Posts p WHERE
(p.Body like ‘%architect%’ or p.Title like ‘%architect%’ or p.Tags like
‘%architect%’) AND p.Score >0 and p.AnswerCount <>0 ORDER BY p.Score DESC. In
our replication package [39], we provided the complete SQL query used to
search ARPs in SO, such as how the title, tags, and body of a post were
combined during the search. The searching process resulted in 32,182 posts
(see Figure 1). Note that we used the mentioned search terms (i.e.,
“architect”, “architecture”, “architectural”, and “architecting”), not for the
purpose of accumulating all ARPs in SO, but for gathering sufficient data for
a relatively comprehensive analysis to achieve the goal of this study.

b) Filtering ARPs from other posts (i.e., programming and hardware related
posts): We found that SO users use the term “architecture” not only in the
context of software architecture, but also in other contexts, such as hardware
architecture context (e.g., ARM6 CPU
architecture777https://tinyurl.com/f8sjvzz2) and programming context (e.g.,
array architecture888https://tinyurl.com/3ps7b3ek), when describing their
concerns in the SO posts. Therefore, we need to filter the retrieved 32,182
posts and exclude those posts related to programming and hardware
architecture. To do so, we performed context analysis and applied our defined
inclusion and exclusion criteria (see Table 2) to accurately filter and
separate software ARPs from other types of posts mentioned above.

Before the formal post filtering (manual inspection), to reach an agreement
about the inclusion and exclusion criteria (see Table 2), a pilot filtering
was performed whereby the first author took a random sample of 1,000 posts
from the 32,182 posts. He manually checked them with our defined criteria (see
Table 2). The other three authors checked and examined the results so that all
the authors (four authors) of this study could get a consensus on the
understanding of the defined inclusion and exclusion criteria. Thereafter, we
got controversy and misunderstanding on 51 posts from the filtered results.
Such controversy and misunderstanding were discussed between the four authors
of this study till a consensus was reached. The first author carried on with
the formal post filtering based on the inclusion and exclusion criteria. The
process continued till all the 32,182 posts were manually checked. This step
resulted in 10,423 candidate ARPs (see Figure 1). The results from this round
were checked and verified by the other three authors of this study, and it
took us twenty one full days to identify and separate ARPs from programming
and hardware architecture related posts in these 32,182 posts.

Table 2: Inclusion and exclusion criteria for filtering ARPs from programming
and hardware related posts Inclusion criteria
---
I1. An ARP should contain a discussion on software architecture, for example,
architecture design and architecture tactics.
I2. An ARP should contain at least one answer attached to its architecture
related question as we aim to study the factors that make these questions have
more than one answer or only have one answer and the usefulness of their
answers.
I3. An ARP should contain at least one data item that can be extracted
according to the data items defined in Table 4.
Exclusion criteria
E1. An ARP that has a score (i.e., medium number of down/upvote) that is less
than 1 is excluded since we want to make sure that all studied posts have
attracted enough attention from the community [40].

Phase II: Determining the representative sample size

The 10,423 candidate ARPs (filtered from the previous phase (i.e., Phase I))
are a quite large dataset, and it is not realistic to analyze this size of
dataset with human effort and get accurate and comprehensive results. Thus, in
order to get statistically significant results, we used the power statistics
and calculated a representative sample size [16] of these 10,423 ARPs. At a
confidence level of 95%, we set a margin of error (i.e., how much we can
expect our analysis results to reflect the view of the overall dataset) to 3%
for the whole 10,423 ARPs. The final representative sample size calculated is
968 ARPs. Then, we randomly selected 968 ARPs from the 10,423 ARPs and
analyzed them for answering the six RQs (see Table 1). To be more specific,
except for RQ1 and RQ2 on which we used 968 (a representative sample size of
ARPs) to answer them, we used subsets of the 968 ARPs that satisfy our defined
criteria (in the following steps) with respect to the purposes of the
remaining RQs (i.e., RQ3, RQ4, RQ5, and RQ6) (see Table 1). We followed the
following steps to further divide our calculated representative sample size of
ARPs (968) into subsets of the ARPs that are relevant to answering the
remaining RQs:

Step 1: Identification of questions that have more than one answer (RQ3) and
questions that only have one answer (RQ4). As stated in the rationale of these
two RQs (see Table 1), we want to examine the factors that make such
architecture questions get more than one answer or only get one answer. Thus
we considered comments posted on architecture related questions by referring
to these two studies [41][42], in which the authors argued that the quality of
an answer is a combination of both the answer and its associated comments as
comments may provide additional information about the answer, for example,
improvement of answers [42] and obsoleted answers [41]. Therefore, in our
study, we included comments posted on architecture related questions and
studied what makes these posts get more than one answer (RQ3) or only get one
answer (RQ4). More specifically, the first author manually checked 968 ARPs
and their comments, and got 650 architecture related questions (wherein each
question has more than one answer). These questions were used to answer RQ3
(see Figure 1). On the other hand, the first author followed the same
procedure (manual inspection of the 968 ARPs for RQ3) and got 318 architecture
related questions (wherein each question has only one answer). These questions
were used to answer RQ4 (see Figure 1).

Step 2: Identification of ARPs with useful knowledge. For answering RQ5 and
RQ6, we need to identify ARPs with useful knowledge from the representative
random sample of ARPs (968 ARPs). We referred to these two studies (i.e., [41,
42]) to identify ARPs with usefulness knowledge. These studies by Zhang et al.
[41, 42] argued that the quality of an answer is a combination of both the
answer and its associated comments as comments may provide additional
information to support the answer, such as improvement of answers [42] and
obsoleted answers [41]. Therefore, in this study, we included the information
in comments to gain a deep understanding of how SO users discuss the
usefulness of architecture solutions provided to their architecture related
questions in SO. In this study, we did not consider vote score for answering
RQ5 and RQ6 since even highly voted SO posts are not always reliable or useful
as argued by Zhang et al. [8]. We observed that SO users occasionally use
terms related to usefulness, such as “helpful”, in the comments to indicate
that certain architecture solutions provided to their architecture related
questions are useful (see Figure 2). Thus, based on this observation along
with the aid of our defined selection criteria in Table 3, we manually checked
and filtered the 968 ARPs to identify solution threads with useful knowledge
(each solution thread includes all solutions to a question (i.e., accepted &
not-accepted solutions) and all the comments that are associated with the
solutions. Specifically, to filter out ARPs that do not discuss the usefulness
of architecture solutions and reach an agreement about the criteria defined in
Table 3, two authors (i.e., the first and second authors) did a pilot ARPs
filtering. They independently and manually examined a random sample of 20 ARPs
from the 968 ARPs. Similar procedure (i.e., selecting a random sample of data
from a large dataset and subsequent manual filtering) has also been employed
in recent studies, such as [43]. To measure the inter-rater agreement between
the first two authors, we calculated the Cohen’s Kappa coefficient [44] and
got an agreement of 0.898. Note that before a solution was finally included as
a relevant one (i.e., a useful solution), the first and second authors first
read the solution that was commented to be useful/helpful in order to verify
if it is really useful to address the question. Disagreements on the ARPs were
discussed between the two authors till a consensus was reached. Then the first
author carried on to check and filter the remaining ARPs. The number of
resulting ARPs (with useful knowledge) that were used to answer the two RQs
(RQ5 and RQ6) is 324 ARPs (see Figure 1).

Figure 2: An example answer that was commented to be helpful Table 3:
Inclusion and exclusion criteria for identifying ARPs with useful knowledge
Inclusion criterion
---
I1. A comment in an answer thread must contain one of the keywords related to
usefulness, such as “useful”, “helpful”, “beneficial”, “handy”, and
“effective”, and this comment is used to signify the usefulness of the answer.
Exclusion criteria
E1. The keyword related to usefulness, for example, “useful”, “helpful”,
“beneficial”, is used to talk about something else (e.g., a question or answer
itself is related to a “usefulness” topic rather than being a sign that the
answer is likely useful).
E2. An ARP with controversy discussions on the answer (i.e., if there are two
comments in the same answer thread, and one states the usefulness of the
answer while another states its uselessness) is not included.

####  3.2.2 Data extraction and analysis

(1) Data extraction: We performed the data extraction process by identifying
the relevant information to be extracted from 968 ARPs to answer our defined
RQs (see Table 1). In Table 4, we present the data items for which the
relevant information was extracted from the candidate ARPs. It also shows the
RQs that are supposed to be answered using the extracted data. The data
extraction was subsequently followed by data analysis, and these two processes
were conducted and recorded with the aid of MAXQDA (a qualitative data
analysis tool)999https://www.maxqda.com/.

Table 4: Data items to be extracted from the ARPs with their description, analysis approaches, and relevant RQs # |  Data item |  Description |  Data analysis approach |  RQs
---|---|---|---|---
D1 |  Content of the question |  The main content of the question in the ARP |  Open coding & constant comparison |  RQ1
D2 |  Design context |  The design context elaborated in the content of the question in the ARP |  Predefined classifications in [27] and [30] |  RQ2
D3 |  Content of the question and its comments |  The main content of the question and a summary of the question’s comments in the ARP |  Open coding & constant comparison |  RQ3, RQ4
D4 |  Content of the answer |  The main content of the answer in the ARP |  Open coding & constant comparison |  RQ5
D5 |  Content of the answer and its comments |  The main content of the answer and a summary of the answer’s comments in the ARP |  Open coding & constant comparison |  RQ6

(2) Data analysis:

Similarly to several existing studies (e.g., [4]), we used open coding &
constant comparison to answer RQ1 and RQ3-RQ6 in our study. Open coding &
constant comparison are two widely used techniques from Grounded Theory [17]
during qualitative data analysis. Grounded Theory (GT) is a bottom-up approach
and focuses on theory generation, rather than extending or verifying existing
theories [17]. Open coding generates codes for incidents that can be further
classified into concepts and categories [17]. Constant comparison is a
continuous process for verifying the generated concepts and categories. Both
concepts and categories evolve and saturate until they fit the data [17].
Thus, in this study, we employed open coding & constant comparison techniques
from GT to generate the concepts and categories for answering RQ1 and RQ3-RQ6.
Specifically, we used open coding to encode the extracted data items for RQ1
and RQ3-RQ6 (see Table 4) to generate codes. Afterwards, we applied constant
comparison to compare the codes identified in one piece of data with the codes
that emerge from other data to identify the codes which have similar semantic
meanings. We proceeded to group similar codes into high-level concepts and
categories. On the other hand, we employed predefined classifications of
design contexts in [27] and [30] to answer RQ2. We followed the same procedure
(encoding and grouping similar codes into high-level categories) to answer
RQ2.

Before the formal data analysis, the first author conducted a pilot data
analysis for each RQ. Specifically, this analysis process involved the
following steps: (1) The first author selected a random set of 100 ARPs from
the representative sample size calculated (i.e., 968 ARPs). (2) The first
author coded the extracted data (see Table 4) for each RQ. When such posts
were unclear and the first author got confused while coding the extracted
data, physical meetings with the second author were scheduled to solve such
confusion. (3) The first author applied constant comparison and grouped all
the codes into higher-level concepts and turned them into categories and
subcategories. The grouping process was iterative, in which the first author
continuously went back and forth between the concepts, categories,
subcategories, and contents of the questions, answers, and comments to revise
and refine the concepts, categories, and subcategories. (4) Thereafter, other
three authors (the second, third, and fourth authors) checked and validated
the results from the pilot data analysis (i.e., concepts, categories, and
subcategories). The disagreements were resolved in a meeting using a
negotiated agreement approach [45] to improve the reliability of the pilot
data analysis results. The first author carried on with the formal data
analysis and followed similar steps used during the pilot data analysis. In
the following paragraphs, we provide details of the formal data analysis
process:

a) For analyzing RQ1, RQ3, and RQ4

As abovementioned, we used open coding and constant comparison [17] to
manually analyze the extracted data (i.e., content of the question for RQ1 and
content of the question and its comments for RQ3 and RQ4) as shown in Table 4.
With these RQs, we investigated architecture related questions from two
aspects, namely categorization (RQ1) and characterization (RQ3 and RQ4) of
these questions (see Table 1). Specifically, regarding the categorization of
the questions, the first author studied the content of each architecture
related question (from the ARPs that are relevant to answer RQ1 (see Figure
1)) by exploring and identifying their main purposes (e.g., design concerns),
such as asking for help on how to refactor the architecture of a system (e.g.,
refactoring of circular dependencies). Thereafter, the first author summarized
each question’s purpose in a short sentence. Firstly, the first author went on
to encode the summarized sentence. This process was iterative, in which he
continuously applied this technique till all ARPs in the dataset were encoded.
Secondly, the first author applied constant comparison to compare the codes
identified in one summarized sentence with the codes that emerged from other
summarized sentences to check the codes which have similar semantic meanings.
The first author proceeded to group similar codes into high-level concepts,
categories and subcategories. The grouping process was iterative, in which the
first author continuously went back and forth between the concepts,
categories, subcategories, and contents of the questions to revise and refine
both the concepts, categories and their subcategories. To mitigate the
personal bias during the formal data analysis, the other authors (second,
third, and fourth authors) of this study participated in the validation of the
generated codes, concepts, categories, and subcategories. The disagreements
were resolved in a meeting using the negotiated agreement approach [45] to
improve the reliability of the analysis results for RQ1 as during the pilot
data analysis. We finally got 9 high-level categories and 21 subcategories as
the results of RQ1, and these results are fully elaborated in Section 4.1.

As mentioned in Section 3.2.1, for answering RQ3 and RQ4, we manually checked
the 968 ARPs to identify ARPs with more than one answer (RQ3) and ARPs with
only one answer (RQ4). This led to two subsets of the 968 ARPs (i.e., 650 ARPs
and 318 ARPs) relevant to answering RQ3 and RQ4 (see Figure 1). Specifically,
in the formal data analysis, the first author analyzed the questions and their
attached comments (from the ARPs of the two mentioned subsets, 650 ARPs and
318 ARPs, of the 968 ARPs) by encoding the extracted data for the two RQs
(i.e., content of the question and comments as shown in Table 4). He wanted to
study if there might be such factors, for example, question formulation [35]
or certain features in the question (e.g., architecture diagram) that
contribute to such architecture related questions having more than one answer
or only having one answer. For example, when investigating RQ3 (questions with
more than one answer), one community member posted a comment under an
architecture related question saying that “+1 great question, very well-
articulated”101010https://tinyurl.com/dfw27h38, for this comment, he picked a
phrase (i.e., a summary of that comment) “well-articulated”. Subsequently, he
went on to study the content of the question (e.g., how architectural
information is stated in the question) under which this comment was commented,
and then he came up with one code that fits this question. He followed the
same processes (e.g., grouping similar codes into high-level concepts,
categories) that the used when analyzing RQ1 to analyze these two RQs (RQ3 and
RQ4). To mitigate the personal bias, the results from this analysis were
checked and validated by other three authors of this study. As in the analysis
of RQ1, we held a meeting and followed the negotiated agreement approach [45]
to discuss and resolve any disagreement, therefore improving the reliability
of the analysis results for RQ3 and RQ4. In the final analysis, we generated
four characteristics of architecture related questions that have more than one
answer and five characteristics of architecture related questions that are
that only have one answer as the results of RQ3 and RQ4, respectively. The
details about these four characteristics for RQ3 and five characteristics for
RQ4 are provided in Section 4.3 and Section 4.4.

b) For analyzing RQ2

We employed pre-defined classifications in [27] and [30] to answer RQ2.
Specifically, the first author manually analyzed the extracted data for RQ2
(i.e., design contexts, see Table 4) from the questions in the ARPs relevant
to answering RQ2 (see Figure 1). The first author then examined the extracted
data to investigate the design contexts in which architecture related
questions were raised. By referring to the categories of design contexts
presented in the abovementioned studies, three main categories and eight
subcategories were generated from the analyzed ARP questions. The personal
bias was mitigated through the validation of the generated categories and
subcategories with the other three authors of this study. As in the analysis
of the previous RQs, the disagreements were discussed and resolved in a
meeting using the negotiated agreement approach [45] to improve the
reliability of the analysis results for RQ2. The results of RQ2 are presented
in Section 4.2.

c) For analyzing RQ5 and RQ6

These two RQs aim to investigate the usefulness of architecture solutions in
SO. As discussed in Section 3.2.1, in order to answer these two RQs, we
defined a set of criteria (see Table 3) and filtered ARPs with useful
knowledge. For clarity, we examined these two RQs from two aspects:

We investigated how SO users discuss the usefulness of architecture solutions
attributed to their associated architecture related questions. We needed to
gain insights into ways (e.g., terms) SO users may use to communicate the
usefulness of architecture solutions in SO. In addition, understanding SO
users’ discussions on the usefulness of these solutions is important to direct
Q&A platform owners in creating the mechanisms that can help their users to
efficiently and effectively search and (re)use such useful architecture
solutions. To achieve this, we manually checked comments attached to the
solutions in the 968 ARPs with the aid of our defined criteria (see Table 3).
We found that SO users occasionally use terms related to usefulness, such as
“helpful”, in the comments along with other terms (e.g., “very”, “super”) to
explicitly convey how useful they found certain architecture solutions (e.g.,
see Figure 2). As stated in Section 3.2.1, the number of resulting ARPs (with
useful knowledge) that were used to answer the two RQs (RQ5 and RQ6) is 324
ARPs (see Figure 1). It is worth noting that we did not count on the
occurrence of the terms, e.g., “useful” (and similar) stated in comments to
measure the usefulness of such architecture solution given to an architecture
related question in our study. As explained above, we referred to information
in the comments attacked to the solutions to examine SO users’ discussions on
the usefulness of architecture solutions. We mean the reaction of SO users
after seeing and using architecture solutions given to their associated
architecture related questions, for example, see a comment in Figure 2.
Moreover, before such ARPs (solutions) were finally included for analysis, we
first read the solutions commented to be helpful and their associated
questions to check if they are really useful (i.e., are the solutions/answers
useful to address the questions? [15]). In Section 4.5, we provide more
details about the identified terms related to usefulness (e.g., “helpful”)
along with other terms (e.g., “extremely”, “very”) (from 324 ARPs) that SO
users use in comments to explicitly signify how useful they found architecture
solutions provided to their associated questions.

During the data analysis for these two RQs (i.e., the taxonomy of architecture
solutions considered useful (RQ5) and their characteristics (RQ6)), the first
author followed the same procedures (e.g., coding and grouping similar codes
into high-level categories) that were used when analyzing RQ1. One thing to
elucidate when analyzing RQ5 to construct a taxonomy is that the first round
of grouping yielded seven main categories. In the second round, the all the
authors of this study proceeded to further generate subcategories and types
from these seven main categories, ensuring that these main categories, their
subcategories, and types follow an “is a” relationship. The grouping process
was iterative, in which the authors continuously went back and forth between
categories, subcategories, types, and solutions to refine the taxonomy. The
final results of this analysis yielded a taxonomy of 7 main categories, 20
subcategories of which 1 were encoded as “Others” (i.e., refer to codes that
did not fit into the already generated subcategories), and 85 types. Note that
the negotiated agreement approach [45] was used to discuss and resolve any
disagreements. The final taxonomy as the result of RQ5 is elaborated in
Section 4.5. Four characteristics were distilled as the results of RQ6 and are
detailed in Sections 4.6.

Note that while categorizing and characterizing ARPs in SO by reading through
those posts, we observed that a single ARP may contain multiple types of
architecture knowledge. For example, in this
ARP111111https://tinyurl.com/2d8r6w8m from our dataset, an SO user asked about
alternative architecture patterns for Model View Controller (MVC) pattern
(i.e., alternative architecture solutions). In the question body, the user
asked the reasons that could drive someone to decide to use those alternative
architecture patterns over MVC (i.e., architecture decisions and their
rationale), the types of systems that the alternative architecture patterns
are typical used for (i.e., design context), and the pros and cons that come
along with using those alternative architecture patterns (i.e., benefits and
drawbacks of architecture solutions). We encoded such a post with multiple
types of architecture knowledge accordingly. Moreover, while analyzing ARPs in
our dataset, we noted down and then discussed the results (e.g., categories of
architecture related questions and taxonomy of useful architecture solutions)
during the qualitative data analysis. This has led to several interesting
findings and actionable implications for various stakeholders, which are
presented in Section 4 and Section 5, respectively. The dataset collected and
used in this study and the details of data analysis (e.g., coding in MAXQDA)
are available online for replication and validation purposes [39].

##  4 Results

In this section, we present the results to our RQs that we got from data
analysis (see Section 4). The result of each RQ is presented in a dedicated
subsection, ending with the key findings of the corresponding results.

###  4.1 Categories of architecture related questions (RQ1)

Categories of architecture related questions that SO users ask in SO were
determined using the open coding and constant comparison techniques described
in Section 3.2.2. We examined questions in the 968 ARPs to answer this RQ (see
Figure 1). Our data analysis yielded 9 main categories and 21 subcategories of
architecture related questions. Table 5 shows the mentioned categories, their
subcategories, their percentages of occurrence (out of 968 ARP questions), and
count information. As shown in Table 5, architecture configuration (27%, 261
out of 968 ARP questions), architecture decision (19%, 181 out of 968 ARP
questions), and architecture concept (15%, 142 out of 968 ARP questions) are
the top three categories of most frequently asked architecture related
questions. In the following, we report those categories and subcategories.
Where required, we provide an SO question example to support the understanding
of the categories and their subcategories.

Table 5: Categories of architecture related questions, their subcategories,
and their counts & percentages

Category | Subcategory | Count
---|---|---
Architecture configuration (27%, 261) | Architecture configuration with technologies support | 144
Architecture pattern configuration | 117
Architecture decision (19%, 181) | Technology decision | 104
Behavioral decision | 77
Architecture concept (15%, 142) | Architecture overview | 62
Basic architectural concept | 32
Architecture component functionality | 26
Specific architecture pattern | 22
Architecture implementation (12%, 119) | Architecture component implementation | 79
Architecture pattern implementation | 40
Architecture tool (10%, 99) | Architecture modeling tool | 34
Model-based code generation tool | 30
Usage of architecture tool | 21
| Code-based model generation tool | 14
Architecture evolution (6%, 55) | Architecture extension to meet new requirements | 42
Component extension to meet new requirements | 13
Architecture refactoring (5%, 45) | Refactoring of circular dependencies | 21
Refactoring of large components | 13
Refactoring of big ball of mud | 11
Architecture deployment (4%, 34) | Application deployment to meet quality attributes | 24
Application deployment to meet functional requirements | 10
Architecture documentation (3%, 32) | 32

(1) Architecture configuration questions in this category ask about how to
configure components and connectors in software systems. The types of
components and connectors could either belong to certain technologies (e.g.,
Windows Communication Foundation (WCF) and Windows Presentation Foundation
(WPF)) or other architectural concepts (e.g., architecture patterns). This
category is the most common (27%, 261 out of 968 ARP questions) category of
architecture related questions in SO (see Table 5). We further classified this
category into two subcategories, in which architecture configuration with
technologies support surpasses half of the questions (144 out of 261 ARP
questions of the architecture configuration category) that SO users ask in
this category (see Table 5).

  * 1.

Architecture configuration with technologies support is concerned about how to
configure an architecture of an application with specific technologies (e.g.,
WPF, WCF). For instance, in this question121212https://tinyurl.com/yuxjp2su, a
developer asked about how to configure or build a scalable Web-based
application by using WCF: “Does anyone have any experience with how well web
services build with Microsoft’s WCF will scale to a large number of users? The
level I’m thinking of is in the region of 1000+ client users connecting to a
collection of WCF services providing the business logic for our application
(…)”.

  * 2.

Architecture pattern configuration seeks practical guidance on how to
configure a specific architecture pattern (e.g., Model View Controller and
hexagonal architecture patterns) when designing an application to achieve
certain requirements (e.g., functional requirements). For example, in this
question131313https://tinyurl.com/4kn6t27e, a developer asked about how to
configure an application that conforms to a Hexagonal architecture pattern by
stating that: “I’m looking for some guidance or best practices for how to
configure and structure an application which conforms to Hexagonal
architecture that supports multiple (driver) adapters simultaneously (…)”.

(2) Architecture decision: SO users ask this type of questions mostly when
they want to decide between two or more alternative architecture solutions
when deigning their software systems. Among two subcategories identified in
this category, the technology decision subcategory contains the majority of
questions (104 out of 181 ARP questions of the architecture decision category)
that SO users ask in this category (see Table 5).

  * 1.

Technology decision is mainly concerned about choosing between two or more
technology solutions (e.g., frameworks, databases) to meet certain
requirements at the architecture level. Moreover, various aspects can be
considered during this choice, such as technology features, benefits, and
drawbacks [34]. For instance, in this
question141414https://tinyurl.com/k9xzkman, a developer asked about the
reasons that could drive him or her to decide to use Cassandra over HBase for
his/her application by stating that: “HBase is known for being a key-value
store and random reads with .get and .put functions based on the key. Is
Cassandra a better choice for suiting a requirement of key-value store? Can it
support random reads based on key? If so, in which conditions should I choose
Cassandra over HBase in a Spark Streaming application?”.

  * 2.

Behavioral decision is concerned with deciding how certain elements in a
system would interact together to provide some functionality or to satisfy
certain quality attributes [46]. For example, a developer wanted to decide on
either to let clients connect directly to the database or let the connection
go through the web service by asking this
question151515https://tinyurl.com/4pjh3ufk: “Recently I have been developing a
system to run a high secured database (using vb.net and SQL Server 2005). I
want to increase the security of the database so no connection will be made
directly to the database but instead a HttpWebRequest is sent to a web service
which then connects to the database and returns the requested data table in
XML format. My concern is just about the performance, I cannot decide either
to let clients connect directly to the database or let the connection go
through the web service”.

(3) Architectural concept includes theoretical related questions about
software architecture. We divided this category into four subcategories, among
which architecture overview contains the majority of questions (62 out of 142
ARP questions of the architecture concept category) that SO users ask in this
category.

  * 1.

Architecture overview questions are concerned with the information about the
general working mechanism or overview of certain existing architecture. For
instance, in this question161616https://tinyurl.com/2a9hb3ek, a developer
asked about architectural overview of Drupal version 7: “Could someone provide
an architectural overview of the Drupal 7 control flow? Perhaps in the sense
of a flowchart about how a page gets generated (…)”.

  * 2.

Basic architectural concept refers to questions that seek explanations about
basic concepts in software architecture. For instance, in this
question171717https://tinyurl.com/5burxuca, a developer was seeking
explanations about several architecture concepts, such as a architecture
pattern: “Is MVC a pattern or architecture or framework? What is a pattern?
What is an architecture? (…)”.

  * 3.

Architecture component functionality is concerned with the use, purpose, or
functionality of certain components in the architecture. For example, in this
question181818https://tinyurl.com/3x35jzu6, a developer was asking about the
use or purpose of the lifecycle aware component in Android based application:
“We already have a Lifecycle in our Activity/Fragment then why will we use
Lifecycle aware component & kindly guide me the main purpose of it. And if we
use lifecycle aware then why we use lifecycle that we knew already”.

  * 4.

Specific architecture pattern questions ask about particular architecture
patterns that are commonly used in the design of certain applications to
address functional or non-functional requirements. For instance, in this
question191919https://tinyurl.com/2u2df8z5, a developer asked about commonly
used architecture patterns for three-dimensional (3D) video game applications:
“What are some of the more common design patterns used when developing 3D
games? Are there any high-level architectural design patterns that are
commonly used? (…)”.

(4) Architecture implementation questions ask about how to implement a certain
software system according to its architecture design. The architecture design
is refined in detailed design, and then implemented in code. Architecture
implementation category has two subcategories, among which architecture
component implementation occupies the majority of questions (79 out 119 ARP
questions of the architecture implementation category) that SO users ask in
this category.

  * 1.

Architecture component implementation is concerned with how components should
be implemented in the system. For instance, in this
question202020https://tinyurl.com/kp73y3wk: “How to implement a single
component sharing in different modules in Angular 7 while using lazy
loading?”.

  * 2.

Architecture pattern implementation questions are about the ways certain
architecture patterns are implemented with regard to the fundamental design
principles. For example, in this question212121https://tinyurl.com/mpmvwb5c:
“How to implement MVC in Swift? I’ve been building Swift apps where basically
all the functionality is in the ViewController. I know this isn’t the optimal
way to do it because design patterns help you expand the app but I don’t
really understand them (…). How do I go about turning this into a Model-View-
Controller design?”.

(5) Architecture tool: There are various architecture tools (e.g., Enterprise
Architect, Archi, Cloudcraft) that can be used to assist in the architecture
design of a software system. With our dataset, we found architecture related
questions in which SO users ask about these tools and classified them in the
architecture tool category. We further classified this category into four
subcategories, among which architecture modeling tool contains the majority of
questions (34 out of 99 ARP questions of the architecture tool category) that
SO users ask in this category.

  * 1.

Architecture modeling tool questions ask about tools that can enable the
creation or drawing of architectural diagrams to model or represent an
architecture of a software system during the design. For example, in this
question222222https://tinyurl.com/ndf7mrnc: “I am a newbie in TOGAF and I need
to start a first trial. I am trying to model my architecture. Which tool do
you advise me to use in order to model my architecture using TOGAF?”.

  * 2.

Model-based code generation tool refers to questions that ask about
architecture tools that can enable the generation of code from architectural
models. For instance, in this question232323https://tinyurl.com/jwkvtzwc:
“Please suggest me any open source tool to generate C# code from UML designer
(…). My requirement is to have a code generation tool for C#”.

  * 3.

Usage of architecture tool questions look for instructions on how to use
certain architecture tools (e.g., Archi, Microsoft Visio). For instance, in
this question242424https://tinyurl.com/52zffbw3: “How to add UML/layer diagram
to an existing solution in VS 2015 community? There is no architecture menu
there?”.

  * 4.

Code-based model generation tool is concerned with tools that assist in
architectural models generation or recovery from the codebase. For example, in
this question252525https://tinyurl.com/n7rz24mu: “I need to make a UML class
diagram for a project (…) I do not really want to write all the
classes/functions manually, so I was trying to generate the diagram from the
source code but can’t seem to find a way or tool to do it. (…)”.

(6) Architecture evolution: SO users ask this type of architecture related
questions when seeking help on how they can re-architect and expand their
existing architecture for the purposes of achieving certain new requirements
(functional or non-functional requirements). Among two subcategories
identified in this category, the architecture extension to meet new
requirements subcategory contains the majority of questions (42 out of 55 ARP
questions of the architecture evolution category) that SO users ask in this
category (see Table 5).

  * 1.

Architecture extension to meet new requirements is concerned with practical
guidance for expanding an existing architecture of a system to address certain
new functional or non-functional requirements. The changes do not only happen
in one component, but they may happen in almost the whole architecture of the
system. For example, in this question262626https://tinyurl.com/ymbyzcvz: “I am
expanding/converting a legacy Web Forms application into a totally new MVC
application. The expansion is both in terms of technology as well as business
use case (…). The new project has two primary goals: Extensibility (for
currently and future pipeline requirements) and Performance (…). Is there a
way in DDD to achieve both, Extensibility that DDD provides and performance
that DBDD provides?”.

  * 2.

Component extension to meet new requirements includes questions that ask about
the extension of certain architectural components to meet some new functional
or non-functional requirements in existing and running software systems. This
is different from the above subcategory, as here the change or extension
happens in local to a specific component or layer, rather than affecting the
whole architecture of the system. For instance, in this
question272727https://tinyurl.com/3rh9xmhs: “We are currently evaluating CQRS
and Event Sourcing architecture (…). What happens if, after an application has
been up and running for a while, there is a new requirement to add an
additional field to a ViewModel on the ReadModel database? Say, the Customer
Zip Code is required on the CustomerList ViewModel, where it was not
previously”.

(7) Architecture refactoring: SO users ask this type of architecture related
questions when they want to restructure architecture of systems aiming at
improving non-functional attributes of those systems without modifying their
external behaviors. This category includes three subcategories, where most of
the questions (21 out of 45 ARP questions) are related to the subcategory
refactoring of circular dependencies.

  * 1.

Refactoring of circular dependencies is concerned with techniques and tools
that can help remove undesirable circular or cyclic dependency issues among
modules so that layering violations can be addressed and dependency structure
can be improved in the systems. For instance, in this
question282828https://tinyurl.com/475dvbp5: “I am working on the MVC project
where I am following the layered architecture (…). Now, my Business Logic
Layer(BLL) is depending on the Data Access Layer (DAL) which is depending on
BLL because domain objects are inside BLL. So, both are having reference to
each other (…). How can I overcome the circular dependency?”.

  * 2.

Refactoring of large components is concerned with approaches that can help
refactor large components in software systems. For instance, in this
question292929https://tinyurl.com/475dvbp5: “I have a pretty large table
component and I want to separate its body section into new component. Each
time I am trying to do this, the styling of table gets broken (…). I would
like to have exactly this same page after this refactoring. Does anyone know
how to pass styling to this new child component, or how to make thing styling
work again ?”.

  * 3.

Refactoring of big ball of mud: Big ball of mud occurs when a software system
lacks a perceivable, flexible, and appropriate architecture [47]. This
subcategory includes questions that ask about approaches and tools for big
ball of mud refactoring. For instance, in this
question303030https://tinyurl.com/j6rdefeb: “What step would you take to
refactor a ball of mud CF app into something modern and maintainable”.

(8) Architecture deployment collects architecture related questions that ask
about how certain software systems should be deployed in the hosting
environments to meet requirements (e.g., functional and non-functional
requirements). According to our studied dataset, we divided this category into
two subcategories, among which application deployment to meet quality
attributes contains the majority of questions (24 out of 34 ARP questions of
the architecture deployment category) that SO users ask in this category.

  * 1.

Application deployment to meet quality attributes includes architecture
related questions that ask about methods and tools that assist in the
deployment of applications in the hosting environments to meet quality
attributes (e.g., availability and performance). For instance, in this
question313131https://tinyurl.com/4tyd4yt6, a developer asked how to deploy a
microservice based system with zero downtime: “At the moment I’m working on an
application which will be based on the Microservice architecture. As main
technologies, we planned to use Spring Boot and Docker for each Micro Service
development. One of the goals/requirements is to provide a Zero Downtime
Deployment feature for the users (…). Any suggestions on the Zero Downtime
Deployment process? If you have any great ideas for a different architecture
or maybe you’ve used tools which can help us here (…)”.

  * 2.

Application deployment to meet functional requirements covers architecture
related questions that ask about methods and tools that assist in the
deployment of software systems to meet functional requirements. For example,
in this question323232https://tinyurl.com/37dmd6av, a developer asked about
the method s/he can follow in order to deploy his/her microservices based
application in the production environment so that each service of the
application can call each other: “I am trying to deploy my microservices
architecture to production env. Now I have 15 services, 1 Facade Layer, Facade
Layer calls services, gets data, aggregates them, and generates the final
result. Also, services call each other(rarely but yes, they call each other)
(…). So I have decided that I will have 5 Boxes (5 high-end servers). A, B, C,
D, E A will be LVS (for Load Balancing) B & C will host the Facade layer. So
when the request came for Facade, it will come from A and load balanced to B &
C (…). So B & C box will contain each one haproxy instance also since when
Facade Layer calls services, it will be load balanced (…). But my question is
how should I allow my services to call each other? (…)”.

(9) Architecture documentation: This is the only category of architecture
related questions with no subcategories in our studies dataset. The
architecture documentation category includes questions that ask about methods
and tools that assist in the documentation of architecture of software
systems. For instance, in this question333333https://tinyurl.com/msnbc7xb, a
developer asked about the best practices and tools for documenting
architecture of different types of systems: “What are the best practices and
software tools for documenting software design and architecture for PC based
applications based on Java or .NET? Embedded Applications based on VxWorks or
Embedded Linux or Windows CE? (…)”.

Key Findings of RQ1 Finding 1: SO users ask a broad range (9 categories) of
architecture related questions, among which architecture configuration (27%,
261 out of 968 ARP questions), architecture decision (19%, 181 out of 968 ARP
questions), and architecture concept (15%, 142 out of 968 ARP questions) are
the top three categories of most frequently asked architecture related
questions.

###  4.2 Categories of design contexts (RQ2)

This RQ aims to investigate the categories of design contexts in which
architecture related questions were raised. As described in Section 3.2.2, to
answer this RQ, we used a predefined classifications of design contexts from
[27] and [30]) when analyzing the extracted data for RQ2 (i.e., design
contexts) from the 968 ARP questions (see Figure 1). We found that most (71%,
687 out of 968) of our analyzed ARP questions describe their design contexts
(i.e., the knowledge about the environments in which the systems are expected
to operate [27]), and then the responders provided potential solutions with
rationale based on the given design issues and design contexts. In addition,
we identified three main categories and eight subcategories of design
contexts. We report the mentioned categories, their subcategories, their
percentages of occurrence (out of 687 ARP questions), and count information in
Table 6. It is also evident from Table 6 that application context is the most
common (54%, 377 out of 687 ARP questions) category of design contexts, and
organizational context is the least significant category (8%, 56 out of 687
ARP questions).

Table 6: Categories of design contexts, their subcategories, and their counts & percentages Design Context |  Subcategory |  Count
---|---|---
Application context (55%, 377) |  Application domain context |  313
External service context |  64
Platform context (37%, 254) |  Software context |  139
Hardware context |  115
Organizational context (8%, 56) |  Development schedule context |  36
Stakeholders context |  13
Resources context |  7

(1) Application context refers to the software system or product that is to be
designed. It is accessed through a device (platform entity) to deliver
services to end-users [27]. This category includes two subcategories, in which
the application domain context subcategory is the most (313 out of 377 ARP
questions) common one.

  * 1.

Application domain context describes the domain/type of the application that
is being developed (such as E-commerce system, banking system, distributed
system) [30]. Some SO users like to reveal in their architecture related
questions what kind of application domains they are about to design in order
to get potential and relevant architecture solutions that fit their
application domains. For example, in this
question343434https://tinyurl.com/2p93nesu, a developer mentioned that s/he
was designing an E-commerce system: “I am designing an E-commerce using
microservices architecture. Suppose that I have two contexts: a product
catalog, inventory and pricing. It’s seems clear to me that they have a clear
responsibility. But to serve the show case (the product list) I need to make a
request for the product catalog, get a list of ID’s and then use it to query
the Inventory micro services to check inventory status (in stock or stock
out). Besides that I need to make a request to Pricing to get the price of
each product (…). I have been reading about microservices architecture and
when you are dealing with many ‘joins’ it’s possible that the these contexts
should be a single one (…). We can use a domain event to notify ‘search’
microsecond that something has changed. So we can resolve show case with a
single request. This look like a CQRS. Is there a correct approach? Which one
is better ? Trade-offs?”.

  * 2.

External service context refers to specifications of external software
services that the application uses [27]. For example, in this
question353535https://tinyurl.com/2p8phmr6, a developer mentioned that s/he
was designing a system that will require to use Azure or Amazon cloud
services: “Basically my question is on the application architecture. Designing
for hosting is easy but cloud computing adds new challenges (…). I am not
certain what I should do in designing an application for safety engineers, so
a high uptime is important. So, if my application is written in ASP.NET, using
SQL Server, it would seem that my best bet is to design for Azure, but would
Amazon’s solution be a good choice? How would I decide if I should just have
everything on the same system or have the data on Amazon’s cloud and the
ASP.NET on Azure? (…). I decide on the language, does that lock me into a
cloud solution?”.

(2) Platform context comprises the hardware technology a user employs to
access an application, the software it runs, and the network capabilities of
such technology [27]. In our dataset, we identified two platform contexts
(i.e., software and hardware context).

  * 1.

Software context comprises information about the software elements of the
device, such as the Operating System (OS) or other installed applications.
This subcategory collects the ARP questions that describe the software
elements of the device (e.g., OS) on which the planned software system will
need to run in production [48]. We found that some SO users provide this kind
of information when asking architecture related questions. For example, in
this question363636https://tinyurl.com/2p8nd2kw: “I need to build one mobile
application starts with windows phone 7 and then need to convert the
application to other platforms like Android, iOS. The application contains
many screens with data capture and all the data stores it in local storage and
finally, it is passed to a central server. I would like to know how the
architecture needs to be designed (…)”.

  * 2.

Hardware context comprises the platform entity which defines the device
through which the user accesses and uses the application, and can be of
different types, such as desktop, laptop computers, and wearable mobile
devices [27]. The hardware context category gathers ARP questions that mention
hardware technologies (e.g., desktop computers) through which the users access
and utilize the planned applications. For example, in this
question373737https://tinyurl.com/yc2eyvhs, a developer mentioned that s/he
was developing a desktop application: “We want to start develop an
intermediate desktop software. We decided to use the WPF. We don’t want to use
the MVVM pattern. Because we are not familiar with MVVM. Is it true to develop
WPF application without MVVM pattern (using 3 layer architecture but without
MVVM) although does it have better performance than win forms yet?”.

(3) Organizational context refers to the development schedule (e.g., time-to-
market), the people (i.e., stakeholders), or the resources that could
influence the development of software systems. This category includes two
subcategories, in which the development schedule context subcategory is the
most (36 out of 56 ARP questions) common one.

  * 1.

Development schedule describes the time put on software development. For
example, in this question383838https://tinyurl.com/2bu5yu65 a developer
mentioned that the development time for a software project was restricted to
only three months: “For personal and university research reasons I am thinking
of building a simple CRM using a service-oriented architecture (…). The
architecture that I’m designing defines: - WebGUI (a client of the other
services) - AnalyticsService (a service that receives data, analyzes, and
collects it) - CustomerCareService (a service that uses RESTful APIs to apply
CRUD operations (…). What sort of authentication is more suitable for a client
(user token vs OAuth or similar). I’ve about 3 months to do it (…)”.

  * 2.

Stakeholders context describes the people who are involved in the development
of a software system, for example, project managers, owners, architects,
developers, users, among others. For instance, in this
question393939https://tinyurl.com/yra7d3y7 an asker mentioned a number of
developers that was involved in the development of 3D Map application by
stating that: “I’m trying to develop 3D Map, and I found 3 solutions: Use game
engine (like unity) or Use 3D graphic API (OpenGL, etc) or Web app. Is there
another way to do it? And which one of those three solutions (design decision)
is better? (with reason) (..) Developers: 3 programmers”.

  * 3.

Resources context denotes the lack (or availability) of resources (e.g.,
financial or technological competencies) at disposal to develop an application
[49]. For instance, one developer needed to update an application with a tight
budget and stated in this question404040https://tinyurl.com/4pjp3ntj that: “I
have to build a database/image-rich application that’s only going to increase
in size (scalability). I am on a budget, but do have a rather good 3Ghz Xeon
server with 400 GB space. Any ideas? a good way for an individual on a TIGHT
budget”.

Key Findings of RQ2 Finding 2: Most of the SO users (71%, 687 out of 968 ARP
questions) considered design contexts when asking architecture related
questions. Finding 3: Application context is the most common (54%, 377 out of
687) category of design contexts in ARP questions, whereas organizational
context is the least significant design context category (8%, 56 out of 687)
in ARP questions.

###  4.3 Characteristics of architecture related questions that have more than
one answer (RQ3)

Some architecture related questions are continuously getting more attention
from SO users by answering them. This motivated us to investigate why such
architecture related questions get more than one answer in SO by
characterizing those questions. As mentioned in Section 3.2.2, we used two
techniques (i.e., open coding and constant comparison) from Grounded Theory
[17], to examine the characteristics of ARP questions that have more than one
answer from 650 ARPs (a subset of 968 ARPs) (see Figure 1). As discussed in
Section 3.2.2, we referred to the contents of the questions and comments
attached to questions to understand what factors (e.g., question formulation
[35] or certain features in the question (e.g., architectural diagram)) that
contribute to such architecture related questions getting more than one
answer. The outputs of our data analysis generated four common characteristics
of architecture related questions that have more than one answer. We provide
these four common characteristics and their counts & percentages in Table 7,
which shows that well-articulated architectural information is the most (46%,
297 out of 650 ARP questions) frequent characteristic while upvoted
architecture question comes as the least (8%, 51 out of 650 ARP questions)
frequent characteristic of architecture related questions that have more than
one answer. Moreover, we show the numbers of answers of the ARPs that have
more than one answer in Figure 3.

(1) Well-articulated architectural information in the question: The main
reason an architecture related question would continuously be answered is that
its architectural information is well-articulated. This question provides an
overview of the planned software system and its basic principles (e.g., design
contexts and architectural constraints) to help other SO users perceive what
the question is really about (the purpose of the question). For example, a
comment: “+1 great question, I want to say that this is a beautifully and very
well-articulated question!)” was posted under this question
414141https://tinyurl.com/dfw27h38 that illustrates and explains well the
encountered architecture concerns (i.e., architecting a system to meet the
scalability and visualization of data).

(2) Clear description together with architectural diagrams in the question:
Another reason an architecture related question would continuously be answered
is that it is easy to read and understand, for example, the question which
clearly states necessary information about the components and connectors
together with interfaces and relationships to other components. Also,
providing diagrams, such as architectural component diagrams that depict and
clarify the logical architecture view of software systems, contributes to
questions continuously being answered. For example, before one responder
started answering this question 424242https://tinyurl.com/74fr8mwe, this
responder stated that: “Your question is clearly described. Thanks for the
little graph you drew to help clarify the overall architecture (…)”.

(3) Alternative architecture solutions to answer the question: Although
different architecture solutions act as alternative solutions to similar
design problems, they differ in terms of their qualities [34]. For example,
two architecture solutions may both address the interoperability concern but
may differ into addressing performance concern. However, such alternative
architecture solutions are of significant importance as they provide a wide
range of possibilities for choosing and making design decisions on candidate
architecture solutions for certain design issues. In this study, we noticed
that questions that ask about choosing between various architecture solutions,
such as technologies (e.g., databases, frameworks, and programming languages),
are continuously getting new answers (alternative solutions). For example, SO
users were interested in choosing a right combination of message formats with
message transmission techniques in order to achieve quality attributes,
including high performance, availability, scalability, among others, for their
Ruby and Java applications interaction, and they posted a
question434343https://tinyurl.com/44sey2a2: “We have cloud-hosted (RackSpace
cloud) Ruby and Java apps that will interact as follows (….). We are
interested in evaluating both message formatting (such as JSON) as well as
message transmission techniques (RPC, REST, SOAP, etc.). Our criteria are high
performance, availability, scalability (…). What combination of message format
and transmission method would you recommend? Why?”.

(4) Upvoted architecture question: Usually, in SO, if a question is
consistently getting upvote count, its likelihood of being answered or getting
more answers increases [50], and architecture related question is no
exception. In our data analysis, we realized that this factor (upvoted
architecture question) also contributes to architecture related questions
having more than one answer. For example, in this architecture related
post444444https://tinyurl.com/7bc8764a, a responder stated in the title of the
answer thread when s/he was answering the question: “Since this question got
upvoted several times I would like to share what I did in the end (…)”.

Table 7: Characteristics of architecture related questions with more answers and their counts & percentages Characteristic |  Count
---|---
Well-articulated architectural information |  297 (46%)
Clear description together with architectural diagrams in the question |  174 (27%)
Alternative architecture solutions to answer the question |  118 (18%)
Upvoted architecture question |  51 (8%)
Figure 3: Numbers of answers of the ARPs that have more than one answer

Key Findings of RQ3 Finding 4: Well-articulated architectural information is
the most (46%, 297 out of 650 ARP questions) frequent characteristic of
architecture related questions that have more than one answer. Finding 5: The
presence of architectural diagrams (e.g., components diagrams) in the
architecture questions increases the chance of these questions to get more
than one answer.

###  4.4 Characteristics of architecture related questions that only have one
answer (RQ4)

We found out that some architecture related questions gain less attention from
SO users to be continuously answered. Analogous to the two previous RQs (i.e.,
RQ1 and RQ3), we applied two techniques (open coding and constant comparison)
to study the characteristics of questions that only have one answer from 318
ARPs (a subset of 968 ARPs) (see Figure 1). As detailed in Section 3.2.2,
similar to RQ3, we referred to the contents of the questions and the comments
attached to questions to understand what factors demotivate the responders to
continue answering certain architecture related questions. We identified five
common characteristics of those questions with their counts & percentages in
Table 8, which shows that lacking information in the question (39%, 125 out of
318 ARP questions) and poorly structured architecture question (22%, 69 out of
318 ARP questions) are the two major characteristics of architecture related
questions that only have one answer. Below, we elaborate these characteristics
in detail with examples from ARPs.

(1) Lacking information in the question: Architecture related questions that
lack certain significant information (e.g., missing information on components
and connectors together with interfaces and relationships to other components)
fail to attract community members to provide their answers. For example, one
developer pointed out that some information is missing in this architecture
related question454545https://tinyurl.com/btukhpzx: “Your question cannot be
answered without doing (many) assumptions. More information is needed about
the module’s dependencies. Are they stateless? Can you draw a flow of the
requests?”.

(2) Poorly structured architecture question: We found that architecture
related questions that are poorly structured (e.g., not well articulated) fail
to clearly reveal their purposes to the community members so that they can
provide answers. These questions sound unclear, vague, or hard to follow. For
example, in this question464646https://tinyurl.com/3m6fzjbe a developer asked
about how to implement the interactors in Android MVP clean architecture but
failed to clearly structure well his/her question, and another developer came
and commented: “So what exactly is your question?”. The asker came back and
edited the question to make it clear so that other community members could
understand what the question is really about.

(3) Architecture considered as off-topic: Even though architecture related
questions are being asked in SO, SO is mainly designed for programming
information seekers. Therefore, architecture being not directly for
programming related issues but mainly for high-level structure related
concerns, this leads to the situation that architecture related questions get
less attention from the community and consequently only get one answer or
remain unanswered in SO. For example, a developer asked about the overview of
ZeroMQ architecture, and another developer commented under this
question474747https://tinyurl.com/rmpfarjc by saying that: “I’m voting to
close this question as off-topic because it’s not really about programming”.

(4) Proprietary technology in the question: We found a few questions, asking
about proprietary technologies, such as databases and frameworks that are not
widely used, get less attention in SO and consequently get only one answer.
For instance, in this architecture related
post484848https://tinyurl.com/nrz66x3w, a community member claimed to be one
of the technology founders of RethinkDB in the title of the answer when s/he
was answering a question. Other community members kept asking more questions
about that RethinkDB (a not widely used database) in the comment thread, and
those questions have remained unanswered. For example, “Disclaimer: I’m one of
the founders of RethinkDB. Sorry for the longish answer (…). RethinkDB is
designed with a very flexible architecture (…)”.

(5) Duplicate architecture question: Similar to other types of questions
(e.g., programming questions) in SO, some architecture related questions get
few answers (i.e., one answer) or remain unanswered because they are duplicate
architecture questions. Community members do not like to re-answer questions
that were answered before [50]. They would like the askers to review the site
(i.e., to check if their questions have not been posted and answered) before
posting such new questions. For example, a comment: “duplicate of
stackoverflow.com/questions/15142386/…”, was posted under this architecture
related question494949https://tinyurl.com/2p9382hh.

Table 8: Characteristics of architecture related questions with few answers and their counts & percentages Characteristic |  Count
---|---
Lacking information in the question |  125 (39%)
Poorly structured architecture question |  69 (22%)
Architecture considered as off-topic |  51 (16%)
Proprietary technology in the question |  32 (10%)
Duplicate architecture question |  29 (9%)

Key Findings of RQ4 Finding 6: Lacking information in the question (39%, 125
out of 318 ARP questions) and poorly structured architecture question (22%, 69
out of 318 ARP questions) are the top two most frequent characteristics of
architecture related questions that only have one answer.

###  4.5 Taxonomy of architecture solutions that are considered useful (RQ5)

This RQ aims to construct a taxonomy of architecture solutions that are
considered useful in SO. As discussed in Section 3.2.2, when answering this
RQ, we first investigated how SO users discuss the usefulness of architecture
solutions attributed to their associated architecture related questions. We
needed to gain an understanding of the ways (e.g., terms) SO users may use to
communicate the usefulness of architecture solutions in SO. Understanding SO
users’ discussions on the usefulness of these solutions is important to direct
Q&A platform owners in creating the mechanisms that can help SO users to
efficiently and effectively search and (re)use such useful architecture
solutions. We found that SO users frequently use two terms related to
usefulness (i.e., “useful” and “helpful”) along with six other terms (i.e.,
“definitely”, “very”, “really”, “super”, “extremely”, and “incredibly”) in the
comment threads (see Figure 2) to explicitly express their feedback about how
useful they found certain architecture solutions provided to their associated
architecture related questions. Note that SO users may use other ways to
communicate the usefulness of architecture solutions in SO, and we cannot
claim that we have identified all usefulness terms. Secondly, as detailed in
Section 3.2.2, we thoroughly and comprehensively examined the contents of the
solutions from 324 ARPs (a subset of 968 ARPs) with useful knowledge (see
Figure 1) to construct the taxonomy of these solutions. This examination
yielded a taxonomy of 7 main categories, 20 subcategories of which 1 were
encoded as “Others” (i.e., refer to codes that do not fit into the already
generated subcategories), and 85 types (see Figure 4).

Figure 4: Taxonomy of architecture solutions that are considered useful

(1) Solution for architecture configuration: This is the largest category of
architecture solutions in our taxonomy (see Figure 4). The solutions in this
category provide approaches and tools that enable the configuration of
components and connectors of the planned software systems. Among three
subcategories identified in this category, the configuration solution for
domain subcategory collects more than half of the solutions (75 out of 127 ARP
solutions) discussed in this category (see Figure 4).

  * 1.

Configuration solution for domain: This subcategory discusses approaches and
tools for configuring applications of various domains, such as solution for
distributed system configuration, solution for banking system configuration,
solution for E-commerce system configuration, and solution for game system
configuration (see Figure 4). Concerning the solution for distributed system
configuration type, a user asked about how to design and configure a 2/3 tier
distributed application in Java with certain components, including centrally
shared database and multiple fat clients (Swing based Graphical User Interface
clients (GUIs)). S/he needed a simpler approach that could help him or her to
configure those clients so that each client can be informed about data changes
committed to the database by another client. The first solution in this
ARP505050https://tinyurl.com/jfnuke2w is provided based on the application
domain (i.e., distributed application) described in the question. The solution
suggests to configure the application’s components (e.g., database and
clients) by following Java EE distributed container/component-based
architecture by stating that: “(…) Java EE is a distributed
container/component based architecture for the enterprise tier (…) You c/would
design a messaging domain with both topic/subscription based and straight up
Queues. These can be declaratively configured to be durable, or not, etc.
(…)”.

  * 2.

Configuration solution for platform provides approaches and tools that enable
the configuration of components and connectors of applications with regard to
the platforms (e.g., Windows OS) on which these applications will run in
production. We identified seven commonly discussed solution types in this
category, such as solution for Android app configuration, solution for Windows
applications configuration, solution for iOS app configuration, and solution
for cross-platform configuration (see Figure 4). Regarding the solution for
cross-platform configuration type, one SO user needed to design and configure
an application that sends data between two iOS devices (i.e., iPad and iPhone)
with iPad acting as an iBeacon. The first solution in this
ARP515151https://tinyurl.com/ytw52k6p explains how the application’s
components including the iPad and iPhone could be configured by using an
approach that could support Android as well by stating that: “(…) I was forced
into an architecture that would support Android as well, so I switched to
BlueTooth. The iPad acting as an iBeacon also has BlueTooth code that is
looking for ‘peripherals’ with a certain signature. Once the iPhone detects
the iBeacon, the app then starts transmitting a BlueTooth peripheral signal
with the appropriate signature (…)”.

(2) Solution for architecture implementation: The ARP solutions in this
category provide technology solutions, such as frameworks and libraries (see
Figure 4), for implementing diverse architecture designs to address the system
requirements (e.g., quality attributes). According to our dataset, we
classified these technology solutions into three subcategories, among which
framework for architecture implementation contains the majority of solutions
(21 out of 59 ARP solutions) that SO users discuss in this category (see
Figure 4).

  * 1.

Framework for architecture implementation: These solutions gather different
types of frameworks for implementing architecture design, for example,
framework for Web-based application (such as Laravel, Django, Express.js, and
Play frameworks), framework for iOS application (such as SwiftUI, Flutter, and
React Native frameworks), and framework for Windows applications (such as
WinForm, WPF, and UWP frameworks) (see Figure 4). Regarding the framework for
Web-based application type, a user asked about (among other things) a
framework that could facilitate the implementation of REST APIs in a Web-based
application which will serve the content to mobile apps. The third answer in
this ARP525252https://tinyurl.com/2p8pm9rs suggests the Play framework as the
solution to that question by stating that: “Use Play! to do it all. Writing
REST services in Play is very very easy (…)”.

  * 2.

API for architecture implementation accumulates different types of APIs as
solutions to questions that ask about APIs for implementing architecture
design, for instance, API for video game system (such as Pokeapi, Chicken
Coop, Dota2, and Minecraft APIs), API for Web-based application (such as REST,
SOAP, RPC, and Geolocation APIs), and API for facial recognition application
(such as Lambda labs and Microsoft Computer Vision APIs) (see Figure 4).
Concerning the API for Web-based application type, one developer needed an API
to implement a request-response Web application in Service Orientated
Architecture (SOA) in order to meet certain requirements (including high
performance). The second answer in this ARP535353https://tinyurl.com/pakzw2yk
suggests to use REST API over SOAP API by noting that: “(…) consider also
using REST API, it demands less overhead than SOAP, and you can use JSON as
document format which is also more compact than XML, lowering network
throughput requirements (…) SOAP has more fancy features that are not well
supported in all languages, if you use REST you will be more safe here (…)”.

  * 3.

Library for architecture implementation: These solutions recommend various
libraries for implementing architecture, for example, library for linear
algebra application (such as JBLSA, MTJ, OjAlgo, and EJML), library for
computer vision application (such as OpenCV library), library for vector
graphics application (such as DISLIN library), and library for Web-based
application (such as HPPC, Trove, and FastUtil) (see Figure 4). Regarding the
library for vector graphic application type, the first answer in this
ARP545454https://tinyurl.com/259h4f6e suggests Raphaël library as the
solutions to the question that asks about vector graphics application by
saying that: “(…) I chose RaphaëlJS and I have to say it has been an absolute
pleasure to use, and the help is fantastic too (…)”.

(3) Explanation of architecture: The ARP solutions in this category provide
theoretical explanations, purposes, or functionalities of architecture instead
of providing concrete instructions on how to do something (e.g., how to
configure certain architectural components in the system). Explanation of
architecture category consists of four subcategories, among which explanation
of cloud-based architecture is the most (28 out 53 ARP solutions) discussed
subcategory (see Figure 4).

  * 1.

Explanation of cloud-based architecture provides theoretical explanations,
differences, and functionalities of the architecture of cloud computing
services, such as Azure App service functionality explanation, EC2
functionality explanation, and AppHarbor functionality explanation (see Figure
4). For instance, a user asked about the difference between Azure App Service
and the AAzure Service Fabric in terms of functionalities in software
development. The fourth answer in this ARP555555https://tinyurl.com/2p8hnkmm
provides a detailed explanation about the difference between those two Azure
platforms in terms of functionalities in software development as the solution
to that question by stating that: “(…) They’re two separate platforms,
following different development paradigms. The App Service will give you
functionality that Service Fabric doesn’t provide out of the box. Stuff like
auto-scale, authentication, rate limiting, integration with SaaS applications,
etc. (…)”.

  * 2.

Explanation of CMS architecture describes the functionalities of the
architecture of Content Management Systems (CMS), such as Drupal functionality
explanation and TYPO 3 functionality explanation (see Figure 4). For example,
the first answer in this ARP565656https://tinyurl.com/2a9hb3ek provides the
architecture overview of Drupal (together with an architectural diagram) as
the solution to the question that asks about the functionality of Drupal
(e.g., control flow and how a page gets generated) by saying that: “(…)
Although it’s procedural PHP, it’s purely event/listener driven in its
architecture, and there’s no simple ‘flow’ in the main PHP script for you to
look though (…) Drupal’s index.php file functions as a front-side controller
(…)”.

  * 3.

Explanation of database architecture groups the ARP solutions that explain or
describe the architecture of datab”ase systems, for instance, NoSQL database
functionality explanation (such as Apache Cassandra) and Graph database
functionality explanation (such as Nebula Graph) (see Figure 4). For example,
the first answer in this ARP575757https://tinyurl.com/wpvw6j5h provides an
explanation about  Cassandra in terms of data replication to deal with data
failure scenario as the solution to the question that asks about the way
Cassandra handles such a scenario if one node goes down containing the record
(data) a user is querying by stating that: “Cassandra clusters do replicate
data across the nodes. The specific number of replicas is configurable, but
generally production clusters will use a replication factor of 3. This means
that a given row will be stored on three different machines in the cluster (…)
In terms of servicing requests, if a node receives a request for data that it
does not have it will forward that request to the nodes that do own the data”.

  * 4.

Explanation of Web server architecture provides the functionalities or
difference between the architecture of Web servers, for example, Web server
functionality explanation (such as XAMPP, IIS, and WAMP servers) (see Figure
4). For instance, the first answer in this
ARP585858https://tinyurl.com/ysacs8zd provides detailed difference between
XAMPP, WAMP, and IIS servers as the solution to the question that asks about
the difference between those three types of Web severs by expressing that:
“(…) Their (XAMPP and WampServer) differences are in the format/structure of
the package, the configurations, and (…) IIS is a web-server application just
like Apache is, except it’s made by Microsoft and is Windows only (Apache runs
on both Windows and Linux) (…)”.

(4) Architecture tactic: This category of ARP solutions provide and explain
architecture tactics that enable the realization of specific quality attribute
(e.g., performance and security) of software systems. Four subcategories of
architecture tactics were identified in this category, among which tactic for
performance is the most (21 out of 37 ARP solutions) discussed subcategory
(see Figure 4).

  * 1.

Tactic for performance provides and explains architecture tactics that assist
in the realization of the system performance requirements. We identified four
architecture tactics for performance, such as scheduling resources tactic and
in memory caching tactic (see Figure 4). Regarding in memory caching tactic, a
developer wanted to choose a suitable design technique between two data
handling design techniques (i.e., working directly with a database or working
with objects and letting the ORM handle the storage) in order to boost the
performance of the inventory system that should handle thousands of item types
and quantities of each item stored in a database. According to the scenario
elaborated in the question, the first answer in this
ARP595959https://tinyurl.com/289ffurv suggests to apply in-memory caching
architecture tactic with ORM to have the system performance boosted by saying
that “(…) most of the time it is easier to do an SQL query, but an in-memory
cache can really BOOST performance. Yes, it uses memory. Who cares?
Workstations can have 64GB memory these days (…)”.

  * 2.

Tactic for maintainability covers architecture tactics that enable the
maintainability requirements of systems. We identified three maintainability
tactics, such as less coupling tactic and encapsulation through API
introduction tactic (see Figure 4). Concerning the less coupling tactic, a
developer needed to build a scalable, maintainable, and low-latency single
sign-on for all web applications. The first answer in this
ARP606060https://tinyurl.com/22ckrdv4 suggests to apply less coupling tactic
when designing the applications in order to make them maintainable by saying
that: “I would not integrate the authentication on the database level (…) This
might become hard to maintain. I would prefer a loosely coupled approach by
exposing a simple service on your central server that lets the other app
servers run authentication requests (…)”.

  * 3.

Tactic for security provides architecture tactics that help the realization of
the system security requirements. This subcategory includes three architecture
tactics, authentication tactic, limiting access tactic, and communication link
encryption tactic (see Figure 4). Regarding authentication tactic, the first
answer in this ARP616161https://tinyurl.com/bdhk4uud provides and explains
authentication tactic to a question that asked for how to set up two level
authentication approaches of the ‘user JWT’ in microservice based application
by stating that: “(…) You can achieve the two levels of security you require
by using a single user token and claims based authorisation. If a call is made
to the gateway with the user token, the gateway authenticates the call based
on the user token, retrieves the ‘userId’ claim (…)”.

  * 4.

Tactic for availability collects architecture tactics that enable the system
availability requirements. We collected three availability tactics, like data
replication tactic and functional redundancy tactic (see Figure 4). Concerning
data replication tactic, a developer asked how to achieve the availability
requirement for an application that needs to use two Amazon EC2 instances each
with Cassandra database. The first answer in this
ARP626262https://tinyurl.com/2p8db7mu provides and explains the replication
mechanism that could be applied in his/her application (according to the
design scenario described in the question) by saying that: “(…) In your
scenario (since you are in a single DC) you can use SimpleStrategy for your
replication strategy and a Replication Factor (RF) of 2. With this setup, you
will have all data replicated on both nodes. This will make the data available
from either node with a covet”.

(5) Architecture pattern: The ARP solutions in this category provide
architecture patterns for addressing multiple system quality attributes, and
also provide commonly used architecture patterns in certain application
domains (see Figure 4). Among the two subcategories identified in this
category, the architecture pattern suggestion to meet quality attribute
subcategory contains the majority of solutions (17 out of 30 ARP solutions of
the architecture pattern category) that SO users discuss in this category (see
Figure 4).

  * 1.

Architecture pattern suggestion to meet quality attributes collects
architecture patterns for addressing system quality attributes, such as
patterns for modifiability, reusability, and portability (Broker, MVC, and
SOA) (see Figure 4). For example, a SO user asked about the best C#
architecture patterns enabling the communication between separate plugins of a
multi-tenant website wherein modifiability, reusability, and flexibility are
the major concerns. The first answer in this
ARP636363https://tinyurl.com/42m8ts56 recommends SOA pattern as a solution to
that question by noting that: “(…) I might suggest Service Oriented
Architecture. Mostly because it can bend to a business in a very quick and
agile manner. This architecture provides many bonuses: Lightweight, Agile,
Code Re-usability (…)”.

  * 2.

Usage of architecture pattern gathers architecture patterns for questions that
ask about the commonly used architecture patterns in certain application
domains (see Figure 4), such as pattern for time-critical system (Preemptive
Multitasking), pattern for Android application (MVP, MVVM, MVC, Observer), and
pattern for distributed system (SOA, Broker). For example, a user asked about
architecture pattern for time-critical applications. The first answer in this
ARP646464https://tinyurl.com/25y4b9e6 recommends Preemptive Multitasking
pattern to that question by saying that: “(…) This pattern is called
preemptive RTOS, which is capable of handling the events immediately (…)”.

(6) Architecture solution for deployment collects the ARP solutions that
discuss the deployment of architecture of systems in the hosting devices
(either on the Cloud or the local server) in order to address the systems’
requirements. This category consists of two subcategories, among which process
for deployment is the most (9 out of 13 ARP solutions) discussed subcategory
(see Figure 4).

  * 1.

Process for architecture deployment collects the ARP solutions that discuss
the processes for deploying the architecture of applications for the purpose
of achieving the applications’ requirements (e.g., functional or nun-
functional requirements). We identified several processes for architecture
deployment, for example, deployment process with Azure App service, deployment
process with Kubernetes, and deployment process with AppHabor for .NET
applications (see Figure 4). Regarding deployment process with Kubernetes, a
responder provided and explained the deployment process with Kubernetes to an
asker who wanted to deploy a microservices architecture (which was built up
with 15 Spring Boot microservices) on five Kubernetes nodes with one cluster
master. According to the scenario described in the question, the first answer
in this ARP656565https://tinyurl.com/483en2ts suggested to use three cluster
masters at a minimum instead of one cluster master in order to avoid the data
loss and consequently address the system’s availability requirement by saying
that: “(…) one master is not enough. The loss of that VM, the underlying
hardware, or a failure of the services on the master will lead to an outage
for all customers and potentially catastrophic data loss. Run 3 masters at
minimum”.

  * 2.

Tool for architecture deployment collects the tools for deploying architecture
of systems in order to achieve the requirements of the systems. We collected
several tools, such as simulator tool for architecture deployment and tool for
continuous deployment (see Figure 4). Regarding the tool for continuous
deployment, the first answer in this ARP666666https://tinyurl.com/yc7j8z5j
recommends Argo CD tool as the solution to the question that asks about a tool
for microservices architecture continuous deployment on Kubernetes by stating
that: “(…) ArgoCD workflow provides that functionality (…)”.

(7) Solution for architecture documentation: The ARP solutions in this
category provide the approaches and tools that enable the documentation of
architecture (see Figure 4). This category consists of two subcategories,
among which tool for architecture documentation is the most (3 out 5 ARP
solutions) discussed subcategory (see Figure 4).

  * 1.

Tool for architecture documentation suggests the tools that can facilitate the
documentation of architecture, such as tool for Web application architecture
documentation and tool for database architecture documentation (see Figure 4).
Concerning the tool for Web application architecture documentation, the first
answer in this ARP676767https://tinyurl.com/4zm82snw suggests NJsonSchema tool
as the solution to the question that asks about a tool for documenting a
microservices-based application by saying that: “(…) there is NJsonSchema tool
https://github.com/NJsonSchema/NJsonSchema”.

  * 2.

Viewpoint for architecture documentation provides the viewpoints for
architecture documentation, such as development viewpoint for architecture
documentation, and deployment viewpoint for architecture documentation. For
example, the first answer in this ARP686868https://tinyurl.com/2p8ezw7j
provides two viewpoints for architecture documentation (i.e., development
viewpoint for architecture documentation and deployment viewpoint for
architecture documentation) for documenting an architecture that is
implemented with Java.

Key Findings of RQ5 Finding 7: SO users frequently use two terms related to
usefulness (i.e., “useful” and “helpful”), along with six other terms (i.e.,
“definitely”, “very”, “really”, “super”, “extremely”, and “incredibly”) in the
comment threads to explicitly express their feedback about how useful they
found certain architecture solutions provided to their associated architecture
related questions. Finding 8: We derived a taxonomy of useful architecture
solutions consisting of 7 categories, 20 subcategories, and 85 types,
indicating the diversity of useful architecture solutions provided in SO.
Finding 9: Solution for architecture configuration (39%, 127 out 324 ARP
solutions), solution for architecture implementation (18%, 59 out 324 ARP
solutions), explanation of architecture (16%, 53 out 324 ARP solutions), and
architecture tactic (11%, 37 out 324 ARP solutions) are the top four most
frequently discussed categories of useful architecture solutions.

###  4.6 Characteristics of useful architecture solutions (RQ6)

As shown in Figure 2, SO users occasionally leave comments under an
architecture solution to convey that such solution is useful. Hence, this
motivated us to study the characteristics of the architecture solutions that
are considered to be useful. Analogous to RQ5, we used the 324 ARPs (a subset
of 968 ARPs) with useful knowledge (see Figure 1) to analyze the architecture
solutions and their attached comments, and study the characteristics of those
solutions. The qualitative data analysis (see Section 4) identified four
common characteristics of architecture solutions that are considered useful by
SO users. Figure 5 depicts these characteristics along with their counts, in
which complete and comprehensive architecture solution appears to be the most
(34%, 111 out of 324 ARP solutions) frequent characteristic of architecture
solutions that SO users consider to be useful.

(1) Complete and comprehensive architecture solution: A developer may ask more
than one question (sub-questions) in one single architecture related question
in SO. A solution that addresses all sub-questions asked in the question and
provides comprehensive responses (e.g., providing rationale, such as benefits
and drawbacks of the provided architecture solution) to these sub-questions is
considered useful. For example, a developer posted this comment: “+1 for the
most complete, comprehensive useful response I’ve ever seen (…)” under the
first answer in this ARP696969https://tinyurl.com/dfw27h38 that
comprehensively addresses all sub-questions (e.g., difficult to visualize data
in the system architecture implemented with Java and Python) asked in the
question.

(2) Concise explanation with architectural diagrams provides a brief
explanation about the key elements of the architecture solution. Some examples
of these key elements could be the best architecture patterns, tactics, and
technologies (e.g., databases) to be used in order to address the design
concerns described in the question. In addition, providing architectural
diagrams, such as component diagrams to represent and summarize the practical
applicability of the solution also contributes to the architecture solution
being considered useful. For example, one developer asked whether “command
handler” and “command bus” should belong to or be implemented in the
application layer or domain layer in the architecture. At first, in the first
answer of this ARP707070https://tinyurl.com/yh292pn8, a responder provided a
concise and relevant solution. But the asker was not satisfied with this
solution and then s/he commented to request a sequence diagram (which was
provided later) to be associated with the solution for it to be useful:
“Thanks, David. It would be really useful if you could share a sequence
diagram. Appreciate it”.

(3) Detailed architecture solution: These solutions provide and fully describe
all necessary architectural elements (such as patterns, components) and other
various aspects to be considered (e.g., solution trade-offs, constraints, and
alternatives) when addressing the design concerns stated in the question. For
instance, a developer posted this comment: “Thank you for your detailed
answer. This is certainly very helpful” under the second answer in this
ARP717171https://tinyurl.com/pfezm7nn that lists and details all necessary
architectural elements (e.g., quality attributes) and other aspects (e.g.,
pros and cons of the solution, and alternative solutions) that should be
considered when addressing the design concerns (e.g., integrating external
modules (external Web applications) into Drupal or vice versa) stated in the
question.

(4) Summarization of external but relevant content: Answer seekers do not like
to have external links (URLs) only as solutions posted to their questions
since the links may die and the solutions become not accessible and useless.
During our data analysis, we observed that answer seekers prefer to have the
relevant content summary of URLs instead of the URLs only for the architecture
solutions. For example, the first answer in this
ARP727272https://tinyurl.com/2p87vu99 summarizes the content from three URLs
to answer the question which mainly asks about the design approach to follow
in order to address system availability with Cassandra database. A developer
commented under the answer: “Thank you very much for your information. Your
Explanation is sufficient and the links you mentioned are very useful”.

Figure 5: The common identified characteristics of architecture solutions that
are considered useful

Key Findings of RQ6 Finding 10: Complete and comprehensive architecture
solution is the most (34%, 111 out of 324 ARP solutions) frequent
characteristic of architecture solutions that SO users consider to be useful.
Finding 11: The presence of architectural diagrams (e.g., components diagrams)
in the provided architecture solutions increases the chance of these solutions
to be considered useful.

##  5 Discussion

In this section, we revisit the findings of this study by interpreting the
results in Section 5.1 and discussing their implications for various
stakeholders in Section 5.2.

###  5.1 Analysis of the results

####  5.1.1 The delta between our results and the results from prior work

Similar to our study, several studies have analyzed ARPs from SO to mine
architectural knowledge discussed by SO users in order to support architecting
activities. In this section, we discuss the relationship and difference
between our study results and the results in the prior studies (i.e., the
three studies by Soliman et al. [6][51][10]), which are closely related to our
work.

Soliman et al. [6] identified and analyzed ARPs from SO that discuss
architecture knowledge with a focus on technology decisions (one type of
architecture decision [46]). They classified these ARPs based on two
dimensions: the purpose of the question and the solution type of the question.
They further classified the purpose dimension into three subtypes: solution
synthesis, solution evaluation, and multi-purposes, and the solution type
dimension into three subtypes: technology feature, technology bundle, and
architecture configuration. In total, their analysis generated 6 types of
ARPs. Our analysis generated 9 categories and 21 subcategories of ARP
questions (see the results of RQ1 in Table 5), such as architecture
configuration, architecture decision, architecture concept, architectural
implementation, architecture evolution, and architecture refactoring. Some of
the types of ARPs (e.g., solution synthesis, solution evaluation, architecture
configuration) found by Soliman et al. in [6] are aligned with some of our ARP
types, and most of the types of ARPs presented in [6] can be subcategories of
the main categories reported in our work. For example, we have a main category
encoded architecture decision, and this category can cover three types of APRs
(solution synthesis, solution evaluation, and multi-purposes) reported in [6].
Moreover, our analysis generated new categories of ARPs (such as architecture
concept, architecture tool, architecture evolution, architecture refactoring,
architecture deployment, and architecture documentation).

Soliman et al. [51] used the same sample of ARPs that were used in their
previous work (i.e., [6]) and developed an ontology that covers architectural
knowledge concepts in SO. The ontology consists of three main ontology
classes: simple ontology class, composite ontology class, and lexical trigger
ontology class. A simple ontology class is composed of subclasses, for
example, technology solution, architecture pattern, quality attribute,
architecture component, and architecture connector. The composite ontology
class consists of several subclasses, such as architecture configuration,
technology feature, technology benefits and drawbacks, technology user-case,
user request, and design rule. The lexical trigger ontology class has
subclasses, such as difficulty adjectives, advise verbs, value adjectives,
wish verbs, support verbs, versus prepositions. Some subclasses found by the
analysis in [51], such as architecture configuration and architecture pattern,
are aligned with our results of RQ5 (see Figure 4). However, the analysis in
[51] is based on a sample of ARPs that mainly discuss technology information
(e.g., requirements and constraints on technology solutions, technology
benefits and drawbacks, and technology features). Our analysis complements the
work in [51] by adding several new categories, such as architecture tactic,
explanation of architecture, solution for architecture documentation, and
solution for architecture deployment, leading to more comprehensive categories
and subcategories of ARP solutions provided in SO.

Soliman et al. [10] developed a search approach that relies on the
classification approach to provide suitable types of ARPs for each design step
proposed by Kazman and Cervantes [12]. The analysis conducted by Soliman et
al. [10] is also based on the sample of ARPs from their previous work [6], and
some other posts extracted from SO. Specifically, their search approach
classifies SO posts into four types: technology identification, technology
evaluation, features and configuration, and programming posts. The first three
types of posts are ARP types that were reported in their previous study (i.e.,
[6]), and in the first paragraph of this section, we have already described
the difference and similarities between these types of ARPs in [6] and the
types of ARPs in our study (see results of RQ1 in Table 5).

In addition to the abovementioned difference between our study results and the
results reported in [6][51][10], in our study, we investigated a new set of
research questions (RQ2, RQ3, RQ4, RQ5, and RQ6). We explored other types of
architecture knowledge, such as design contexts (RQ2) discussed in
architecture related questions, characteristics of ARPs (questions and
solutions) (RQ3, RQ4, and RQ6), and the usefulness of the ARP solutions (RQ5),
which was not the concern of the abovementioned studies (i.e., [6][51][10]).
Moreover, our analysis covered the entire post, including the question and its
associated comments (RQ3, RQ4), the answers to the question and their
associated comments (RQ5, RQ6). The analysis in the abovementioned studies by
Soliman et al. only focused on questions and answers. Thus, our study results