# Influence of Roles in Decision-Making during OSS Development — A Study of
Python

Pankajeshwara Nand Sharma<EMAIL_ADDRESS>University of
OtagoP.O. Box 1212DunedinOtagoNew Zealand43017-6221 ,  Bastin Tony Roy
Savarimuthu<EMAIL_ADDRESS>University of OtagoP.O. Box
1212DunedinOtagoNew Zealand43017-6221 and  Nigel Stanger
<EMAIL_ADDRESS>University of OtagoP.O. Box 1212DunedinOtagoNew
Zealand43017-6221

(2021)

###### Abstract.

Governance has been highlighted as a key factor in the success of an Open
Source Software (OSS) project. It is generally seen that in a mixed
meritocracy and autocracy governance model, the decision-making (DM)
responsibility regarding what features are included in the OSS is shared among
members from select roles; prominently the project leader. However, less
examination has been made whether members from these roles are also prominent
in DM discussions and how decisions are made, to show they play an integral
role in the success of the project. We believe that to establish their
influence, it is necessary to examine not only discussions of proposals in
which the project leader makes the decisions, but also those where others make
the decisions. Therefore, in this study, we examine the prominence of members
performing different roles in: (i) making decisions, (ii) performing certain
social roles in DM discussions (e.g., discussion starters), (iii) contributing
to the OSS development social network through DM discussions, and (iv) how
decisions are made under both scenarios. We examine these aspects in the
evolution of the well-known Python project. We carried out a data-driven
longitudinal study of their email communication spanning 20 years, comprising
about 1.5 million emails. These emails contain decisions for 466 Python
Enhancement Proposals (PEPs) that document the language’s evolution. Our
findings make the influence of different roles transparent to future (new)
members, other stakeholders, and more broadly, to the OSS research community.

Open Source Software (OSS), influence, roles, decision-making, onion model,
Python, PEP, social network analysis, rationale

††journalyear: 2021††copyright: acmcopyright††conference: Evaluation and
Assessment in Software Engineering; June 21–23, 2021; Trondheim,
Norway††booktitle: Evaluation and Assessment in Software Engineering (EASE
2021), June 21–23, 2021, Trondheim, Norway††price: 15.00††doi:
10.1145/3463274.3463326††isbn: 978-1-4503-9053-8/21/06††ccs: Software and its
engineering Open source model

##  1\. Introduction

The success of OSS has attracted much attention in both research and
managerial practice. These volunteer-based communities are sustained, based on
their governance approach to which decision-making processes are crucial
(lerner2002some, ; fleming2007brokerage, ; giuri2008explaining, ). While there
are good examples of successful OSS projects, e.g., Linux, OpenOffice, and
Python, there are many projects which are not successful ((ehls2017open,
)(lee2019longitudinal, )).

Recently, researchers have investigated the influence of roles, specifically
leaders, in Open Source Software Development (OSSD) to understand what
contributes to the success or failure of OSS projects. Leaders’ “task-oriented
leadership behaviors in forms of technical contributions and relation-oriented
leadership behavior” have been identified as major contributors to success
(mu2019role, ). Similarly, leaders with more contributions usually have more
expertise and provide satisfactory feedback, thus helping to attract more
participants (moon2008role, ). They can initiate dialogues, prompt feedback,
and shape how members discuss a topic (huffaker2010dimensions, ). They also
inspire others to “talk” (panteli2016leaders, ) and engage in conversations
(johnson2015emergence, ). Their findings also suggest that when choosing OSS
projects to contribute to, participants should pay particular attention to the
project leader.

In addition, studies of the OSS group decision-making (GDM) processes (e.g.,
(sharma2020mining, ; mockus2002two, ; mockus2000case, ; mrowka2015decision, ))
have showed initial evidence that the sharing of DM responsibility is likely
to vary across different OSS communities. Recent studies have pointed out that
OSS leadership is more “distributed” rather than individualistic
((thapa2020evaluating, ; gronn2002distributed, )). Also, leaders can be silent
during discussions, while encouraging other members to share their opinions
and facilitate information exchange instead of posting many messages
(panteli2016leaders, ). While the success of OSS projects is normally
attributed to their leaders, the fact that this leadership is distributed
means that the positive performance of a project can also be attributed to the
core participants who conduct day-to-day guidance, direction, and control of
discussions during DM.

Recently, the influence of OSS project leaders has come under the microscope.
Linus Torvalds, the leader of the Linux project, took temporary leave of
absence to focus on understanding how to respond to people’s emotions
(riley2018, ). Conversely, Python’s Benevolent Dictator For Life (BDFL)
permanently stepped down from his post over a proposal (steven2018, ), and
stated that he had never had to fight so hard to get a proposal accepted.

In terms of trying to understand the overall influence of prominent OSS
members in different roles in DM discussions and how decisions are made, most
previous research has only provided snapshots of their involvement
((eseryel2013action, ; li2012leadership, ; johnson2015emergence, ;
barcellini2006users, ; barcellini2005study, ; mockus2002two, )). Thus, there
is a need to examine longitudinal data (using a mix of qualitative and
quantitative approaches) to establish the influence of these members in
various roles who play an integral part in the project’s success, particularly
those projects that employ a mixed governance model during their evolution. At
a high-level, this paper investigates the question: How are members in
different prominent OSSD roles involved in daily DM discussions on OSS
enhancements? Particularly, what is the overall influence of members in these
roles during DM in terms of: (a) who makes decisions, (b) their social roles
during DM discussions (e.g., initiating discussions and answering questions),
(c) their prominence in the social network, and (d) considerations of their
preferences when decisions are made, including comparison of these four
aspects when other members make those decisions.

These research gaps can be addressed by scrutinizing and reflecting on the DM
practices of successful OSS projects. This work bridges the above-mentioned
gaps by discovering the influence of decision-makers during DM discussions
through longitudinal data-driven analysis of the successful Python OSS
project, which has been noted to have a good governance model
(wang2015comparative, ), and has been studied by several researchers
((barcellini2005thematic, ; barcellini2005study, ; sack2006methodological, ;
mahendran2002serpents, ; barcellini2006users, ; keertipati2016exploring, ;
savarimuthu2016process, )). Discussions about 466 Python Enhancement Proposals
from 1.5 million emails are studied to reveal the nature of the roles in DM.

This paper is organized as follows: Section 2 provides the background on
relevant work and presents the research questions, Section 3 presents the
methodology used to extract the influence of Python members, Section 4
presents the results while Section 5 discusses the contributions, and Section
7 presents our conclusions.

##  2\. Background and Research Questions

Three factors that have a detrimental effect on the members of an OSS
community are a change of leadership and governance structures (ehls2017open,
), a lack of DM process transparency, and consent in the DM process
(scacchi2008governance, ). A cross-cutting concern in these three factors is
the nature of roles performed by individuals during DM and the extent to which
these roles positively or negatively influence the governance of a project,
including the key DM steps.

OSS governance is often depicted as an _onion model_ (nakakoji2002evolution, )
that shows the hierarchy of roles, with the significant roles identified in
the middle of the onion (core, with few members such as project leaders and
core-developers) and the outer layers represent non-core, yet useful
activities performed by many members (e.g. bug reporters and software users).

A limitation of the current literature in applying the onion model to OSS
projects with a mixed governance approach is that it has only had a
qualitative focus, where high level DM models are presented based on
interviews and surveys with stakeholders. While qualitative approaches may
offer valuable insights, they also introduce a bias of exclusively relying on
individual opinions (thapa2020evaluating, ). Even when the “actual” DM models
have been studied using a data-driven approach, insights have been fragmented
and limited (thapa2020evaluating, ).

This influence of members can only be established by examining their day-to-
day activities during DM — an approach belonging to the practice theory
perspective, which emphasizes the actual practices employed in everyday
community processes (orlikowski2002knowing, ). We thus examine the following
four aspects using a data-driven study of Python email archives.

_(a) Decision-makers:_ We first focussed on establishing members who have
decision-making roles regarding OSS enhancement proposals. There are two
challenges. First, members progress in their roles within the community
(ye2003toward, ), and thus the actual DM contributions of each member in each
role has to be established. Second, a participant can perform different roles
for different proposals (e.g., author in one but only a participant in
another). This needs to be accommodated in the analysis. Thus, our first
research question is: RQ1: Members in which Python roles make PEP decisions?

_(b) Decision-makers’ social roles in DM discussions:_ Next, we associate
decision-makers with their social roles within OSS enhancement DM discussions.
Social roles are useful because researchers have found the OSS design process
is influenced by the connections members have within their community
(barcellini2005study, ; barcellini2005thematic, ). People in these roles
initiate most discussions (such as question people), reply to a lot of top-
level posts (such as answer people), and (iii) have messages attracting a high
number of replies, and are ever present in discussions. These social roles are
examined in our next research question: RQ2: Do members in certain Python
roles participate more than others in DM discussions in terms of: (i) total
posts, (ii) seeding posts, (iii) replying to posts, (iv) receiving replies,
and (v) involvement in threads, and does this participation change when
someone other than the BDFL makes the decisions?

_(c) Decision-makers’ influence in OSS social network during DM discussions:_
Researchers have used social network anaylsis (SNA) to highlight the influence
and positions of key members, describing the role participants play in
influencing decisions (wonodi2012using, ). Current works have identified core
developers and corresponded them with decision-makers ((sureka2011using, ;
long2006understanding, ; crowston2006core, )), to point out that rather than a
member’s expertise, his/her position in the OSS community structure is more
likely to allow for innovation (dahlander2012core, ). Long and Siau
(long2007social, ) concluded that the social structure of the OSS project
affects DM in communities, while in our previous work ((sharma2017boundary,
)), we found that boundary spanners corresponded with the actual decision-
makers. Meneely at al. (meneely2010improving, ) concluded that the central
developers in a developer network have a high likelihood of being approvers.

None of this prior work, however, has attempted to identify how much influence
the OSS project leader (in a mixed governance model) has in the OSSD network
during DM discussions and how this changes when others make the decisions. We
use the theory of “networked influence” (lee2019longitudinal, ) to investigate
network patterns where the leader influences the behaviour of others to
sustain a successful OSSD network. Thus, our next research question is: RQ3:
What is the influence of members in various Python roles in the decision-
making social network and does it change when someone other than the BDFL
makes the decisions?

_(d) How decisions are made:_ Prior works have employed qualitative approaches
such as interviews to understand OSS project governance
((markus2007governance, ; jensen2010governance, ; mrowka2015decision, ;
Hanganu2012voting, )). It is only recently that researchers have started to
engage data-driven quantitative approaches. Various works ((sharma2020mining,
; savarimuthu2016process, ; keertipati2016exploring, ;
sharma2017investigating, ; sharma2021rationale, ) have unearthed the “actual”
DM processes as opposed to the “advertised” ones in projects with a mixed
governance approach (i.e., consensus-based with dictatorship) with varied
levels of detail. However, the focus of most of these works has been on
decisions themselves (e.g., (li2020automatic, )), and not on “how” they are
made (razavian2019empirical, ; bhat2020evolution, ). Thus, the influence of
OSS participants on how design decisions are made remains underexplored, which
leads to our final research question: RQ 4: How do members from various Python
roles influence PEP decision-making and does it change when someone other than
the BDFL makes the decisions?

##  3\. Methodology

To answer these research questions, we employed a range of methods including
comparative analysis, social network analysis, and content analysis of
discussions for all proposals in the Python OSS project. Python was chosen
because it operates based on a mixed governance model including elements of
both meritocracy and autocracy (sharma2020mining, ). It is also widely used
and popular (popularity2020, ), has a reputation for following good governance
practices (wang2015comparative, ), has garnered significant research interest
(keertipati2016exploring, ; savarimuthu2016process, ), and makes data
available to the public (e.g., email archives and commit repositories
(dam2015mining, ; keertipati2016exploring, )).

The Python language is modified and evolves by means of formal _Python
Enhancement Proposals (PEPs)_. These PEP documents move through various states
at different stages of the DM process. Once the draft is ready, it is
circulated within the Python community (developers and users) for discussion.
The PEP will then eventually be accepted or rejected. This process is outlined
in PEP 1111https://www.python.org/dev/peps/pep-0001/. Our methodological steps
are described next.

Step 1: Data Extraction. The first step was to download all 466 PEPs
(including their metadata) and all email messages related to them. We obtained
the PEPs from Python’s Github repository and stored them in a database. We
considered data from March 1995 to 12th July 2018 — the date when the BDFL
resigned, marking the end of the benevolent dictatorship governance model. We
then extracted all individual email messages from the Python email archives,
and stored them in a MySQL database. We used six mailing list archives
(_python-dev_ , _python-ideas_ , _python-commits_ , _python-checkins_ ,
_python-list_ , and _python-patches_) as they are the leading forums for
Python developer discussions, with _python-dev_ being acknowledged as the
primary forum. The resulting dataset contained a total of 1,553,564 email
messages. During this step, we also assigned PEP numbers to email messages
where PEP numbers or titles were mentioned within the message, or were replies
to these messages.

Step 2: Selecting PEP-related messages. From all the email messages stored in
the database, we selected only those that were assigned a PEP number. This
left a total of 91,311 unique email messages related to these 466 PEPs which
we analysed in our dataset.

Step 3: Identity matching. To ascertain all email messages from the same
author, we used identity matching techniques (wiese2016mailing, ). We first
applied the approach proposed by Bird et al. (bird2006mining, ), which used
several combinations of similarity measures for identity matching. We then
applied a cumulative identity similarity approach proposed in (bird2008latent,
) to the output from the first approach. To find messages written by the PEP
author or BDFL Delegates, we used the PEP metadata available online. For an
example, see the metadata for PEP 1 in the link mentioned in footnote 1.

Step 4: Identifying roles (to answer RQ1). To investigate members and their
Python roles associated with DM for each PEP in each PEP type, we first
established the dates when they were inducted into two specific roles (_PEP
editors_ using (python2018peppurpose, )222This is the last version of PEP 1
before the BDFL stepped down and a steering council was elected. and _Core
Developers_ using the public list of core developers
(pythonofficialcoredevlistpointer, ; pythondevguidelines, ). Then, we
identified another two roles: PEP authors and PEP decision-maker (when an
individual in any of the roles had been made, or volunteered as, the BDFL
Delegate) based on the metadata available about the PEP. The latter
information is available through PEP metadata and in email messages. Finally,
a _PEP contributor_ is one who has contributed to discussions, but does not
belong to any of the other roles. Also, these contributors include members who
are not authors or delegates for a particular PEP.

Step 5: Examining social roles (RQ2). To examine each member’s involvement in
DM discussions (social roles) for PEPs, we focused on five aspects as
mentioned in RQ2.

Step 6: Studying prominence of roles in social networks (RQ3). The influence
of members can be shown using a social network. Using “To” and “From” fields,
we identified the communications of the BDFL, PEP author, and BDFL Delegate
roles. These were then imported into the NodeXL software to construct a social
network of Python members’ interaction and to compute the two following SNA
metrics.

Betweenness centrality measures the centrality of a node in the network by
considering the number of the shortest paths the node is part of. It
demonstrates high levels of social status, power, and managerial roles in both
open source and commercial contexts ((bird2008latent, ) and [18]). Eigenvector
centrality measures how well a node is connected to other well connected
nodes. Thus, it is a fitting measure of a member’s influence in Python.

Step 7: Studying the influence of roles in DM (RQ4). We conducted an in-depth
manual analysis of email messages (spanning multiple mailing lists) relating
to only the 248 PEPs which were either accepted or rejected. The first author
read the corresponding messages and identified the rationale behind how each
PEP was decided using a custom-built GUI tool333A screenshot of the tool used
to analyze messages and other supplementary data can be viewed in the
repository https://github.com/sharmapn/influenceOfRoles. Of these 248 PEPs, we
curated 300 rationale sentences relating to 193 PEPs that had an explicitly
stated rationale behind their decisions. All rationale sentences in the
dataset were verified by the second author (i.e., 100% consensus).

##  4\. Results

###  4.1. RQ 1: Python roles responsible for DM

We identified four distinct roles in Python development: (i) contributors who
help by reporting bugs, proposing ideas, and improving documentation; (ii)
core developers, the key task performers who are heavily involved in
discussing and shaping new proposals and implementing them (e.g., developing
new Python libraries); (iii) PEP editors who are the overseers of PEP quality;
and (iv) BDFL, the final decision maker. We were interested in how each of
these roles were involved in two main DM related roles (the PEP authorship
role, and the Decision-Maker role for a particular PEP).

Figure 1. PEP authorship and decisions by members in different roles for the
three PEP types

Fig. 1 shows the contributions of different roles to PEP authorship and PEP
decisions, broken down by three PEP types (Process, Standards Track and
Informational). PEP authors who volunteer to champion a new idea by writing a
PEP are either contributors or core developers (shown in blue). The number of
contributors who are PEP authors are similar to those of core developers. The
PEP decisions (shown in red), are made by the BDFL or BDFL-delegates (nominees
of the BDFL who are members from the first three roles). We found that this
responsibility lies mostly with the BDFL, supported to a lesser extent by the
senior community members, i.e., the core developers and the PEP editors.

Contributors do not normally make the final decisions on PEPs as they are
relatively new members; indeed, this has only happened twice: PEP 541
(python2017pep541, ) when it was decided that the BDFL Delegate (a
contributor) would only recommend the PEP’s acceptance and PEP 484
(python2014pep484, ), which was decided by a senior contributor who had only
recently accepted the BDFL-Delegate position (python2018e, ). Core developers,
on the other hand, are active as BDFL delegates across all three PEP types.
PEP editors have also served as BDFL delegates who make decisions in Standards
Track and Informational PEPs. More Standards Track PEPs have been decided by
BDFL delegates than the other two PEP types because the BDFL considered the
delegates possessed the most knowledge in certain areas since these PEPs
directly concern modifications to the language compared to the other two PEP
types.

There were 51 PEPs decided by someone other than the BDFL. Of these, 35 were
Standards Track, 10 were Informational, and 6 were Process PEPs. Of these,
almost half (25 PEPs) had the most discussions in the disutils-sig mailing
list. From 2014 (lwn2014, ), Python packaging-related PEPs were discussed here
to fast-track decisions and Nick Coghlan (PEP editor from 2012) decided most
of these.

###  4.2. RQ 2: Decision-makers’ social roles in DM discussions

Table 1 shows the top three Python roles (authors, BDFL and BDFL delegates)
and the eight members who are prominent in Python DM discussions for the 415
PEPs decided by the BDFL. All eight members are Python core developers
(pythondevguidelines, ; sharma2017boundary, ). For comparison, we also have
included (in italics) the contributions of BDFL, PEP authors and the top two
members (Brett Cannon and Nick Coghlan) in the 51 PEPs where the BDFL
Delegates made the decisions. Note the eight members identified by name have
also been found to be prominent boundary spanners (who contribute across
different mailing lists) from another work (sharma2017boundary, ). Full email
contribution data for all 466 PEPs decided either by the BDFL or a BDFL
Delegate and further separation of these contributions by PEP type are
available online3. Table 1 shows the results for questions 2(i-v) in columns
3-7.

_PEP involvement_ — The BDFL contributes to the most number of PEPs (406) when
he makes the decision. Of the nine PEPs in which the BDFL did not participate:
four were eventually withdrawn, one PEP is still active, and one PEP is
reserved for future use. The PEP authors (as a collective) had the second most
involvement, and this was expected as these authors are naturally involved in
discussions of PEPs they author. Sometimes PEP authors disappear after
proposing a PEP idea which explains the 23 PEPs in which they were not
involved in discussions. Two other members contributed to most PEPs — Brett
Cannon and Nick Coghlan.

_Total message posts —_ PEP authors collectively posted the most number of
messages. Their contributions across all PEP types are two and a half times
more than the next highest contributor, the BDFL. The BDFL’s strong
involvement in DM discussion contributions occurs at different stages of
discussions, including messages that summarise what has taken place so far or
to raise his issues with the PEP. His contributions suggest that he is also
driving discussions together with the PEP authors (for each PEP). A third
member is Nick Coghlan, a core developer who has been a BDFL Delegate the most
times (sharma2017boundary, ) and who actively contributed to discussions.

Table 1. Top contributing members and roles during DM discussions across the
415 and 51 PEPs decided by the BDFL and BDFL Delegates, respectively.

| Member
---
or Role
| PEP
---
Invol.
| Total
---
Posts
| Seeding
---
Posts
| Replies
---
Sent
| Replies
---
Received
| Thread
---
Invol.
PEP Authors | 392 | 19173 | 8214 | 10959 | 9921 | 9354
51 | 4190 | 1089 | 3101 | 2798 | 3452
BDFL | 406 | 7757 | 2608 | 5149 | 4608 | 3308
48 | 791 | 248 | 543 | 584 | 328
Brett Cannon | 338 | 3406 | 1841 | 1565 | 1329 | 2231
34 | 286 | 80 | 206 | 185 | 140
Nick Coghlan | 302 | 5440 | 1061 | 4379 | 3571 | 1677
51 | 2092 | 428 | 1664 | 1433 | 628
Barry Warsaw | 274 | 4023 | 2721 | 1302 | 1463 | 2913
Georg Brandl | 270 | 2539 | 1949 | 590 | 581 | 2083
Terry Reedy | 209 | 1201 | 397 | 804 | 655 | 596
Antoine Pitrou | 196 | 1657 | 1083 | 574 | 1300 | 1142
Greg Ewing | 182 | 1606 | 119 | 1487 | 1026 | 377
Paul Moore | 174 | 1225 | 205 | 1020 | 996 | 344
BDFL Delegate | 51 | 1328 | 283 | 1045 | 1045 | 366
Italicised values indicate involvement in PEPs decided by BDFL Delegates.

Figure 2. A social network showing the influence of the BDFL in the
discussions of the 415 PEPs decided by him (left) against the 51 PEPs decided
by the BDFL Delegates (right). The vertex colour and size correspond to the
number of times a member has been involved in a PEP. The edge width
corresponds to the number of connections between two members or roles.

_Seeding posts —_ PEP authors collectively are again most prominent. The PEP
author (as the PEP champion) is responsible for building consensus within the
community, and is generally expected to spearhead PEP discussions. The BDFL
has the second highest numbers, suggesting he is also spearheading
discussions. Barry Warsaw, another core developer, has seeded an almost
similar number to that of the BDFL.

_Replies sent —_ PEP authors again are most prominent, which may be due to
answering questions from the community. Their collective contributions are
twice those of the BDFL. The BDFL also shares his knowledge during DM
discussions by responding to questions about PEP. His posts also raise issues
on the discussion topics present in the seeding posts.

_Replies received —_ These are responses received by members on their message
posts. Again, PEP authors receive the most responses followed by the BDFL. The
fact that the BDFL is second implies that he raises important issues which are
worth responding to, or that members are interested in obtaining his views, by
revealing their positions about issues in their replies.

_Thread involvement —_ This metric highlights how many distinct discussion
threads a member was involved in. Again, we found that PEP authors were the
most prominent, followed by the BDFL. The BDFL’s high thread involvement only
highlights his knowledge sharing trait, which is important for sustaining the
community.

Table 3. Roles and sample sentences that depict the rationale behind their
decisions for accepted and rejected PEPs. The description of each rationale
and their proportions grouped by PEP types are available online.3

Role | State | Rationale | Rationale sentence | PEP
---|---|---|---|---
BDFL | Acc. | Consensus | “The user community unanimously rejected this so I won’t pursue this idea any further” (python2001c, ) | 259
PEP Sum. | Rej. | No Consensus | “Although a number of people favored the proposal there were also some objections ” (python2007b, ) | 3128
Core dev. | Acc. | Lazy Consensus | “If anyone has objections to Michael Hudson’s PEP 264: raise them now.” (python2001b, ) | 264
Core dev. | Rej. | Rough Consensus | “Several people agreed, and no one disagreed, so the PEP is now rejcted. [_sic_]” (python2005b, ) | 265
PEP Sum. | Rej. | Little Support | “It has failed to generate sufficient community support in the six years since its proposal.” (python2001a, ) | 268
Core dev. | Acc. | Majority | “Comments from the Community: The response has been mostly favorable”. (python2002c, ) | 279
BDFL | Rej. | No Majority | “Accordingly, the PEP was rejected due to the lack of an overwhelming majority for change.”(python2003a, ) | 308
PEP Sum. | Rej. | Inept PEPs | “This PEP is withdrawn by the author.”(python2002z, ) | 296
BDFL | Acc. | BDFL Decree | “After a long discussion, I’ve decided to add a shortcut conditional expression to Python 2.5.” (python2005a, ) | 308
BDFL | Acc. | BDFL PNC | “There’s no clear preference either way here so I’ll break the tie by pronouncing false and true.” (python2002a, ) | 285
BDFL | Rej. | BDFL PM | “Python is not a democracy.” (python2004a, ) | 326
PEP Sum. = PEP Summary, BDFL PNC = BDFL Pronouncement after No Consensus, BDFL
PM = BDFL Pronouncement over Majority

###  4.3. RQ 3: Influence in the DM Social Network

Fig. 2 highlights the prominence of members and the two roles in the social
network during PEP decision-making when the BDFL made decisions (left) and
when the BDFL Delegates made decisions (right). Members are filtered by those
having the highest betweenness centrality. Most of the members in both
diagrams are Python core developers who have been BDFL Delegates for PEPs as
previously reported in (sharma2017boundary, ). The diagram on the right shows
the BDFL Delegates both as a collective node and individual nodes.

We can see that collectively, the PEP authors have the most prominent role in
terms of centrality in both social networks. In discussions about PEPs decided
by the BDFL, the BDFL is second most prominent and there is a strong
connection (messages exchanged) between the BDFL and PEP authors. These two
roles are also central and connected to many core developers in the social
network during DM. Nick Coghlan is third most prominent, as he is highly
connected to the PEP authors and the BDFL.

In discussions about PEPs decided by BDFL Delegates, BDFL Delegates are highly
connected to PEP authors. In addition, Nick Coghlan and Paul Moore are more
central and strongly connected to the PEP author and the BDFL Delegates. The
BDFL is not as central in the network, and is not greatly connected to anyone;
however, he is still connected with the PEP Authors and BDFL Delegates.

Table 2 shows the SNA metrics of members when PEP decisions were made by the
BDFL and the Delegates, respectively. The detailed results of SNA metrics in
both cases are contained in the ‘SNA metrics’ sheet within the online
repository3. Focussing on the two SNA metrics, we found that in both cases,
the PEP authors had highest values for betweenness and eigenvector centrality.

Table 2. SNA metrics in PEPs decided by the BDFL and BDFL Delegates Role | BDFL | BDFL Delegate
---|---|---
Betweenness | Eigen. | Betweenness | Eigen.
PEP Author | 737391.753 | 0.012 | 46859.855 | 0.028
BDFL | 261465.886 | 0.010 | 4843.832 | 0.012
Nick Coghlan | 246234.593 | 0.010 | 14696.106 | 0.020
BDFL Delegate | N/A | N/A | 25172.653 | 0.022

The BDFL had the second highest values in the first case, but in the second
case, the Delegates and three members had higher values than the BDFL. Of
these three members, Nick Coghlan has been the BDFL Delegate the most, while
Paul Moore has been on two occasions (sharma2017boundary, ). Thus, their
prominence in these SNA metrics could be attributed to them having high
involvement when making decisions. However, the third member, Steven Aprano,
had high betweenness centrality, meaning he was neither higher than the BDFL
on eigenvector centrality, nor was he high in contributions in the second case
(refer to ’Overall Contributions’ sheet in the repository3).

Thus, based on the SNA metrics, apart from the PEP authors, the members who
are tasked to make PEP decisions play a more central and influential role in
the social network during DM discussions than other members.

###  4.4. RQ 4: Overall Python GDM structure

To find out how Group Decision-Making (GDM) is carried out on PEPs, we
examined PEP messages for explanations on the rationale behind their decisions
(Section 3). In terms of PEP decision-making, a rationale refers to the GDM
basis on which the final decision on the PEP was made and whether the decision
was made using consensus or lazy consensus etc. In this regard, we found 300
sentences containing rationale behind decisions for 248 accepted and rejected
PEPs.

Table 3 shows examples of rationale sentences corresponding to each of the
rationale we manually extracted from the 300 ground truth rationale sentences.
A spreadsheet containing the raw data for these sentences and descriptions of
the rationale is available online3. Manually extracting rationale from the
Python communication datasets was laborious. We spent three months full-time
on the exploratory analysis as it required reading most messages relating to
the 245 PEPs that were analysed, with some messages requiring multiple
readings to fully understand the nuances.

Figure 3. Overall layered PEP DM structure (based on decisions on 248 accepted
and rejected PEPs). Arrows depict the how decisions are made by each entity,
including their percentages. Values in green show the percentage of PEP
decisions made using that criterion. Dark yellow rectangles represent the
BDFL’s independent decisions. IV refers to Informal Voting.

To evaluate the correctness of our results (i.e., that the identified
rationale is correct), we presented our findings to a prominent Python core
developer (name withheld) who had been the lead author or co-author of 22
PEPs, had been BDFL Delegate 14 times (sharma2017boundary, ), and was a former
Python Steering Council member. He replied “I think your list of reasons looks
good”. This demonstrates that the extracted rationale is accurate.

Based on the data on the rationale (and their proportions) behind the PEP
decisions, Fig. 3 depicts the overall layered Python DM hierarchy correlating
the different sets of participants, community preference gathering outcomes,
how decisions are made (emphasising the influence of roles), and the various
decision-makers. In the figure, moving the top of the PEP DM structure is the
same as moving towards the centre of the onion model.

For PEPs, community preference status is gathered from community members. If
the community has completed the preference gathering process, two situations
can arise. First, a collective community view on a PEP has been established
(using consensus, no consensus, lazy consensus, rough consensus, little
support, majority or no majority). In these cases, the BDFL (or his Delegates)
concurred in 99% of the PEPs, which corresponds to 60% of the PEPs in the
dataset, and did not concur in less than 1% of the PEPs. Second, when an
overall community view could not be established after discussions (community
is split), resulting in either no consensus or no majority, the BDFL made a
choice and accepted 15 PEPs and rejected 10 PEPs.

If, however, the community preference gathering process remains incomplete, it
can be for two reasons. First, during some PEP discussions the BDFL intervened
and accepted six PEPs and rejected 12 PEPs using the BDFL Decree. Second, when
it is apparent to the community that some PEPs will not succeed (Inept PEPs),
the discussions decline and decisions on them are just a formality. In 18 such
PEPs, the BDFL intervened during PEP discussions and rejected the PEP (mostly
disliking the syntax). Sometimes the discussions on a PEP are stalled for some
reason and the decision is just a formality (13.3%). The BDFL, his delegate,
or even the trusted senior core developer in charge of maintaining the PEP
repository decides on these PEPs.

While the BDFL makes most of the decisions (73%) about which PEPs can be
accepted into the language, members from other roles can also make these
decisions as BDFL Delegates, as highlighted in Section 4.1. We found three
sets of roles representing BDFL Delegates: contributors (4%), core developers
(55%), and PEP editors (40%). We found that these members always sided with
the collective view of the community. As previously mentioned, some of the
inept PEPs lie inactive for a long period without closure, so some senior core
developers reject these “long standing” PEPs as part of batch cleanup work
(3.3%).

Figure 4. PEP decisions rationale usage by BDFL (on the left) vs. the BDFL
Delegates (right), broken down by PEP type for accepted and rejected PEPs.
Positive values correlate to accepted PEPs and negative values to _rejected_
PEPs.

Most decisions on PEPs are based on the collective view of the community,
i.e., mainly core developers. We believe this is the central aspect of DM in
Python, where the project leader just ratifies (or rubber stamps) most of the
decisions, and these decisions mostly originate from and lie with the views of
the core members of the Python community, i.e., the meritocracy principle.

The substantial number of PEPs where the BDFL has exercised his own preference
(BDFL decree, BDFL pronouncement after no consensus, and BDFL pronouncement
after majority) implies that in the Python community, the BDFL is free to
exercise his choices when necessary. However, most of the decisions taken by
the BDFL are based entirely on a collective community view, such as consensus,
lazy consensus, rough consensus, majority, no majority, and little support.
The Python project leader allows the community to come to a collective view on
proposal outcomes, using any of these rationale, and then he (or sometimes his
delegates) appear to mostly concur with the collective view.

We can see that, in general, the community view is rarely challenged by the
BDFL. Overall, the combination of meritocratic and dictatorship principles in
Python appears to be working well.

It should be noted that the Python leader plays an important role in directing
the project, and is needed to manoeuvre through the community disagreements.
It has been noted that PEPs are accepted “based on whether most core
developers support the PEP” (pythondmcoredevs, ), and wherever there were
disputes between core developers and users, the BDFL normally took the core
developers’ view (e.g., (python2004c, )), but still makes independent
decisions, e.g., PEP 326 (python2014pep326, ) — “Python is not a democracy.”
((python2004a, ; python2015c, )).

Fig. 4 depicts the distribution of rationale used by the BDFL (on the left)
against those by BDFL Delegates (on the right) across the three different PEP
types (Standards Track, Informational, and Process) for accepted and rejected
PEPs. The numbers for Process and Informational PEPs are lower because the
total number of PEPs in these two PEP types is small. The detailed tallies of
DM schemes and rationale grouped by PEP type are available online3. The top 5
rationale used by the BDFL for making decisions were consensus (57), lazy
consensus (43), inept PEPs (42), BDFL decree (17), and BDFL pronouncement
after no consensus amongst developers (15). This implies that generally, the
Python community’s collective view on PEPs are preferred.

When comparing the distribution of rationale used by members with these roles,
it is evident that the BDFL uses a wider variety of rationale (mostly in the
Standards Track PEPs) compared to BDFL Delegates. Thus, we can infer that the
BDFL is responsible for deciding on PEPs where there are discerning opinions
in the community regarding the PEPs, and within these PEPs it requires him to
use various different rationale to reach a decision. In addition, the BDFL
makes most of the decisions where there is variability, i.e., he can reject
PEPs, while the BDFL delegates generally decide on PEPs which are eventually
accepted.

##  5\. Discussion

This study investigated the contributions of Python members in different roles
during the decision-making process.

To achieve this, we investigated four research questions. Our results imply
that Python decision-making responsibility (RQ1) has varied involvement from
members in different roles. Unsurprisingly, the general trend is that
relatively newer community members contribute most to PEP authorship, while
the final decisions on PEPs are mainly made by senior members (core
developers, PEP editors, and the BDFL). New proposals tend to come from new
members who want new functionality to be added to the language. The BDFL
allows these proposals from newer members, while decisions are made by
(mainly) him or his trusted core developers (as his delegates). As we will
discuss shortly, even where BDFL delegates make decisions, he maintains his
influence by participating in discussions.

We next focused on the influence of members in different roles in PEP DM
discussions (RQ2). We found that the Python project leader was more heavily
involved than most members, not only in those PEPs where he made the decision,
but also in those PEPs where he delegated others to decide. The PEP author is
most involved in discussions for all PEPs and is the most prominent in terms
of social roles during DM. The BDFL comes second, but his contribution is
substantial (in the number of emails). In PEPs where the BDFL Delegates made
the decisions, it is the delegates who are second in terms of involvement
(next only to PEP authors), closely followed by the BDFL (in third position).
These results show that the project leader is ever-present as a prominent
contributor in various social roles during DM discussions. This implies that
apart from being responsible for making decisions, the project leader also
needs to be highly involved in community discussions during DM in a maturing
OSS project with a mixed governance model.

We next unearthed prominent members’ influence in the social network to focus
on their proximity and connectedness with other prominent members. In terms of
influence in the social network during DM (RQ3), our findings imply that the
BDFL is closely linked with the proposal authors more than others (even the
BDFL Delegate) when OSS enhancement decisions are made. This implies that he
not only makes the decisions, but is highly involved in the DM discussions. In
the PEPs decided by a BDFL Delegate, one may conclude that the BDFL is still
connected to the actual decision-makers during these decisions (i.e., they do
not make decisions independently).

The SNA graphs suggest that the BDFL is the central and second most
influential member during DM discussions when he makes decisions. Although his
influence is surpassed by other members in discussions when he is not making
the final PEP decision, he is still connected to the PEP author and the BDFL
delegates. This influence might be necessary as he has previously made
decisions for all other PEPs in the project. Based on his expertise, his high
involvement in DM discussions on these select PEPs is crucial in order for the
OSS project to be directed towards its objectives. This responsibility is
recognised and is well adhered to by the BDFL, and thus is a crucial aspect in
the guidance of the project.

Finally, we used content analysis of emails to see how members from these
roles influence DM. In terms of decision-making structure (RQ4), the project
leader keeps control of DM on controversial PEPs, while other members are
responsible for DM on non-controversial PEPs. This emphasises the BDFL’s role
when the Python project was still maturing. While doing so, he shared with the
community members how he made decisions and prepared them for this critical
role by publicly communicating PEP DM rationale.

Our results imply that the BDFL had the most influence and was critical to the
success of the project. However, there was still a community consensus
gathering phase which should not be overlooked, as the collective community
view is mostly preferred in PEP decision-making. At the same time, the BDFL
was needed during disputes and he could overrule the community’s collective
view on occasions.

##  6\. Threats to validity

A threat to internal validity is that we may have missed some email messages
that discussed PEPs, particularly those messages that did not mention a
specific PEP number or PEP title and were not replies to these messages. This
could impact the results for the first three RQs. However, we do not believe
this to be a major issue as PEPs are normally indicated by using their PEP
number or their title. However, we do concede there may have been some
messages that may not have used either of those indications. Also, in this
work we have analysed only email messages. However, discussions could have
been had on other platforms (e.g., IRC channels and Twitter).

A threat to the external validity of the study is the generalisability of our
results to other OSSD communities. Since Python is a major OSS project, we
believe our findings are representative of DM in projects with a mixed
governance approach. However, we suggest broader investigations involving
other OSS projects should be conducted to generalise the results we report
here.

##  7\. Conclusions and Future Work

This paper presents a longitudinal empirical study of how members from
different roles impact decision-making in Python — an OSS project with a mixed
meritocracy and dictatorship governance approach. First, our study identified
the roles of members making decisions. Our work showed that while relatively
junior members can propose OSS enhancements, the final decisions are made by
senior members, i.e., core developers, proposal editors, and, of course, the
project leader. Second, the study revealed the interaction-related activities
pursued in different roles during DM (e.g., seeding posts). This analysis
revealed that proposal authors were the most prominent, followed by the
project leader. Third, using SNA analysis, the work showed that the project
leader was the central node. In cases when someone else makes the decisions,
the influence of these decision-makers were more than the project leader’s.
However, the project leader still contributes to the DM discussions and
communicates with the decision-makers and proposal authors. Fourth, in terms
of group decision-making, we found that the project leader keeps the
responsibility of making OSS enhancement decisions when there is a conflict
within the community. To this end, the decisions made by other members (i.e.,
BDFL Delegates) were on proposals where there is less controversy.

To the best of our knowledge, this study is the first to use a data-driven
approach to examine the influence of OSSD community members who undertake
different roles in DM discussions and their involvement in how OSS design
decisions are made (i.e., GDM structure). Our findings thus enrich the
knowledge surrounding the influence of OSSD members performing various roles
in DM discussions and in how OSSD decisions are made.

A promising area for future research is to undertake a focused study on
analysing the sentiments of members (especially the core and influential ones)
and their roles on a proposal-by-proposal basis to uncover whose views
influence decisions, and how these lead to an outburst of emotions by certain
members (e.g., core members) against others, and maybe even to predict such
scenarios. Another area could be to slice the data, based on time or
milestones reached by the community to investigate how the influence of
members in different roles changed during a certain period.

## References

  * (1) J. Lerner and J. Tirole, “Some simple economics of open source,” _The journal of industrial economics_ , vol. 50, no. 2, pp. 197–234, 2002.
  * (2) L. Fleming and D. M. Waguespack, “Brokerage, boundary spanning, and leadership in open innovation communities,” _Organization science_ , vol. 18, no. 2, pp. 165–180, 2007.
  * (3) P. Giuri, F. Rullani, and S. Torrisi, “Explaining leadership in virtual teams: The case of open source software,” _Information Economics and Policy_ , vol. 20, no. 4, pp. 305–315, 2008.
  * (4) D. Ehls, “Open source project collapse–sources and patterns of failure,” in _Hawaii international conference on system sciences(HICSS)_ , 2017.
  * (5) J. Y.-H. Lee, C.-S. Yang, C. Hsu, and J.-H. Wang, “A longitudinal study of leader influence in sustaining an online community,” _Information & Management_, vol. 56, no. 2, pp. 306–316, 2019.
  * (6) W. Mu, Y. Bian, and J. L. Zhao, “The role of online leadership in open collaborative innovation,” _Industrial Management & Data Systems_, 2019\.
  * (7) J. Y. Moon and L. S. Sproull, “The role of feedback in managing the internet-based volunteer work force,” _Information Systems Research_ , vol. 19, no. 4, pp. 494–515, 2008.
  * (8) D. Huffaker, “Dimensions of leadership and social influence in online communities,” _Human Communication Research_ , vol. 36, no. 4, pp. 593–617, 2010.
  * (9) N. Panteli, “On leaders’ presence: interactions and influences within online communities,” _Behaviour & Information Technology_, vol. 35, no. 6, pp. 490–499, 2016.
  * (10) S. L. Johnson, H. Safadi, and S. Faraj, “The emergence of online community leadership,” _Information Systems Research_ , vol. 26, no. 1, pp. 165–187, 2015.
  * (11) P. Sharma, B. T. R. Savarimuthu, and N. Stanger, “Mining decision-making processes in open source software development: A study of python enhancement proposals (peps) using email repositories,” in _Proceedings of the Evaluation and Assessment in Software Engineering_ , 2020, pp. 200–209.
  * (12) A. Mockus, R. T. Fielding, and J. D. Herbsleb, “Two case studies of open source software development: Apache and Mozilla,” _ACM Transactions on Software Engineering and Methodology (TOSEM)_ , vol. 11, no. 3, pp. 309–346, 2002.
  * (13) A. Mockus, R. T. Fielding, and J. Herbsleb, “A case study of open source software development: The Apache server,” in _Proceedings of the 22nd international conference on Software engineering_. Acm, 2000, pp. 263–272.
  * (14) R. Mrówka, “Decision-making in the process of implementation of open source projects.” _Business Systems & Economics_, vol. 2, no. 2, pp. 39–49, 2015.
  * (15) R. Thapa and S. Vidolov, “Evaluating distributed leadership in open source software communities,” in _In Proceedings of the 28th European Conference on Information Systems (ECIS)_ , 2020.
  * (16) P. Gronn, “Distributed leadership as a unit of analysis,” _The leadership quarterly_ , vol. 13, no. 4, pp. 423–451, 2002.
  * (17) D. Riley. (2018) Linus torvalds takes time out from linux to learn empathy skills - siliconangle. [Online]. Available: https://siliconangle.com/2018/09/16/linus-torvalds-takes-time-linux-learn-empathy-skills
  * (18) S. J. Vaughan-Nichols. (2018) Python language founder steps down - zdnet. [Online]. Available: https://www.zdnet.com/article/python-language-founder-steps-down
  * (19) U. Y. Eseryel and D. Eseryel, “Action-embedded transformational leadership in self-managing global information systems development teams,” _The Journal of Strategic Information Systems_ , vol. 22, no. 2, pp. 103–120, 2013\.
  * (20) Y. Li, C.-H. Tan, and H.-H. Teo, “Leadership characteristics and developers’ motivation in open source software development,” _Information & Management_, vol. 49, no. 5, pp. 257–267, 2012.
  * (21) F. Barcellini, F. Détienne, and J.-M. Burkhardt, “Users’ participation to the design process in an open source software online community,” _arXiv preprint cs/0612009_ , 2006.
  * (22) F. Barcellini, F. Détienne, J.-M. Burkhardt, and W. Sack, “A study of online discussions in an open-source software community,” in _Communities and Technologies 2005_. Springer, 2005, pp. 301–320.
  * (23) J. Wang, P. C. Shih, Y. Wu, and J. M. Carroll, “Comparative case studies of open source software peer review practices,” _Information and Software Technology_ , vol. 67, pp. 1–12, 2015.
  * (24) F. Barcellini, F. Détienne, J.-M. Burkhardt, and W. Sack, “Thematic coherence and quotation practices in OSS design-oriented online discussions,” in _Proceedings of the 2005 international ACM SIGGROUP conference on Supporting group work_. ACM, 2005, pp. 177–186.
  * (25) W. Sack, F. Détienne, N. Ducheneaut, J.-M. Burkhardt, D. Mahendran, and F. Barcellini, “A methodological framework for socio-cognitive analyses of collaborative design of open source software,” _Computer Supported Cooperative Work (CSCW)_ , vol. 15, no. 2-3, pp. 229–250, 2006.
  * (26) D. Mahendran, “Serpents and primitives: An ethnographic excursion into an open source community,” Ph.D. dissertation, Unpublished Master’s Thesis, School of Information Management and Systems, UC Berkeley, 2002.
  * (27) S. Keertipati, S. A. Licorish, and B. T. R. Savarimuthu, “Exploring decision-making processes in Python,” in _Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering_. ACM, 2016, p. 43.
  * (28) B. T. R. Savarimuthu, H. K. Dam, S. Licorish, S. Keertipati, D. Avery, and A. K. Ghose, “Process compliance in open source software development–a study of Python Enhancement Proposals (peps),” 2016.
  * (29) W. Scacchi and C. Jensen, “Governance in open source software development projects: Towards a model for network-centric edge organizations,” California University Irvine Institute for Software Research, Tech. Rep., 2008\.
  * (30) K. Nakakoji, Y. Yamamoto, Y. Nishinaka, K. Kishida, and Y. Ye, “Evolution patterns of open-source software systems and communities,” in _Proceedings of the international workshop on Principles of software evolution_. ACM, 2002, pp. 76–85.
  * (31) W. J. Orlikowski, “Knowing in practice: Enacting a collective capability in distributed organizing,” _Organization science_ , vol. 13, no. 3, pp. 249–273, 2002.
  * (32) Y. Ye and K. Kishida, “Toward an understanding of the motivation of open source software developers,” _IEEE. doi_ , vol. 10, pp. 419–429, 2003.
  * (33) C. B. Wonodi, L. Privor-Dumm, M. Aina, A. Pate, R. Reis, P. Gadhoke, and O. Levine, “Using social network analysis to examine the decision-making process on new vaccine introduction in nigeria,” _Health policy and planning_ , vol. 27, no. suppl_2, pp. ii27–ii38, 2012.
  * (34) A. Sureka, A. Goyal, and A. Rastogi, “Using social network analysis for mining collaboration data in a defect tracking system for risk and vulnerability analysis,” in _Proceedings of the 4th india software engineering conference_. ACM, 2011, pp. 195–204.
  * (35) J. Long, “Understanding the role of core developers in open source software development.” _Journal of Information, Information Technology & Organizations_, vol. 1, 2006.
  * (36) K. Crowston, K. Wei, Q. Li, and J. Howison, “Core and periphery in free/libre and open source software team communications,” in _Proceedings of the 39th Annual Hawaii International Conference on System Sciences (HICSS’06)_ , vol. 6. IEEE, 2006, pp. 118a–118a.
  * (37) L. Dahlander and L. Frederiksen, “The core and cosmopolitans: A relational view of innovation in user communities,” _Organization science_ , vol. 23, no. 4, pp. 988–1007, 2012.
  * (38) Y. Long and K. Siau, “Social network structures in open source software development teams,” _Journal of Database Management (JDM)_ , vol. 18, no. 2, pp. 25–40, 2007.
  * (39) P. Sharma, B. T. R. Savarimuthu, and N. Stanger, “Boundary spanners in open source software development: A study of Python email archives,” in _2017 24th Asia-Pacific Software Engineering Conference (APSEC)_. IEEE, 2017, pp. 308–317.
  * (40) A. Meneely, M. Corcoran, and L. Williams, “Improving developer activity metrics with issue tracking annotations,” in _Proceedings of the 2010 ICSE Workshop on Emerging Trends in Software Metrics_. ACM, 2010, pp. 75–80.
  * (41) M. L. Markus, “The governance of free/open source software projects: Monolithic, multidimensional, or configurational?” _Journal of Management & Governance_, vol. 11, no. 2, pp. 151–163, 2007.
  * (42) C. Jensen and W. Scacchi, “Governance in open source software development projects: A comparative multi-level analysis,” in _IFIP International Conference on Open Source Systems_. Springer, 2010, pp. 130–142.
  * (43) G. Hanganu. (2012) Voting in meritocratic governance projects. [Online]. Available: http://oss-watch.ac.uk/resources/meritocraticgovernancevoting
  * (44) P. Sharma, B. T. R. Savarimuthu, N. Stanger, S. A. Licorish, and A. Rainer, “Investigating developers’ email discussions during decision-making in Python language evolution,” in _Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering_. ACM, 2017, pp. 286–291.
  * (45) P. Sharma, B. T. R. Savarimuthu, and N. Stanger, “Extracting rationale for open source software development decisions — a study of python email archives,” in _2021 43rd International Conference on Software Engineering (ICSE)_. IEEE, 2021.
  * (46) X. Li, P. Liang, and Z. Li, “Automatic identification of decisions from the hibernate developer mailing list,” in _Proceedings of the Evaluation and Assessment in Software Engineering_ , 2020, pp. 51–60.
  * (47) M. Razavian, B. Paech, and A. Tang, “Empirical research for software architecture decision making: An analysis,” _Journal of Systems and Software_ , vol. 149, pp. 360–381, 2019.
  * (48) M. Bhat, K. Shumaiev, U. Hohenstein, A. Biesdorf, and F. Matthes, “The evolution of architectural decision making as a key focus area of software architecture research: a semi-systematic literature study,” in _2020 IEEE International Conference on Software Architecture (ICSA)_. IEEE, 2020, pp. 69–80.
  * (49) Popularity of Programming Language Index. (2020) PYPL Popularity of Programming Language. [Online]. Available: https://pypl.github.io/PYPL.html
  * (50) H. K. Dam, B. T. R. Savarimuthu, D. Avery, and A. Ghose, “Mining software repositories for social norms,” in _Proceedings of the 37th International Conference on Software Engineering-Volume 2_. IEEE Press, 2015, pp. 627–630.
  * (51) I. S. Wiese, J. T. da Silva, I. Steinmacher, C. Treude, and M. A. Gerosa, “Who is who in the mailing list? Comparing six disambiguation heuristics to identify multiple addresses of a participant,” in _2016 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. IEEE, 2016, pp. 345–355.
  * (52) C. Bird, A. Gourley, P. Devanbu, M. Gertz, and A. Swaminathan, “Mining email social networks,” in _Proceedings of the 2006 international workshop on Mining software repositories_. ACM, 2006, pp. 137–143.
  * (53) C. Bird, D. Pattison, R. D’Souza, V. Filkov, and P. Devanbu, “Latent social structure in open source projects,” in _Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering_. ACM, 2008, pp. 24–35.
  * (54) B. Warsaw, J. Hylton, D. Goodger, and N. Coghlan. (2018) PEP 1 – PEP purpose and guidelines. [Online]. Available: https://github.com/python/peps/blob/6208fa0f9b3176c43a69906ad6bbfe68080e19d7/pep-0001.txt
  * (55) Python. Official list of core developers. [Online]. Available: https://discuss.python.org/t/official-list-of-core-developers/924
  * (56) ——. Developer log. [Online]. Available: https://devguide.python.org/developers/
  * (57) Python. (2017) Pep 541 – package index name retention. [Online]. Available: https://www.python.org/dev/peps/pep-0484
  * (58) ——. (2014) Pep 484 – type hints. [Online]. Available: https://www.python.org/dev/peps/pep-0484
  * (59) Python Mailing List Discussions. (2018) [python-committers] Proposing Mark Shannon to be a core developer. [Online]. Available: https://mail.python.org/pipermail/python-committers/2018-May/005368.html
  * (60) Corbet, J. (2014) Python packaging: playing well with others. [Online]. Available: https://lwn.net/Articles/580399/
  * (61) Python Mailing List Discussions. (2001) PEP 259 Rejection — PEP 259 – Omit printing newline after newline. [Online]. Available: https://www.python.org/dev/peps/pep-0259/
  * (62) ——. (2007) PEP 3128 Rejection — PEP 3128 – BList: A Faster List-like Type. [Online]. Available: https://www.python.org/dev/peps/pep-3128
  * (63) ——. (2001) PEP 264 Acceptance — Objections to PEP 264. [Online]. Available: https://mail.python.org/pipermail/python-dev/2001-August/017091.html
  * (64) ——. (2005) PEP 265 & 284 Rejection — python-dev Summary for 2005-06-16 through 2005-06-30 [draft]. [Online]. Available: https://mail.python.org/pipermail/python-dev/2005-July/054624.html
  * (65) ——. (2001) PEP 268 Rejection — PEP 268 - Extended HTTP functionality and WebDAV. [Online]. Available: https://www.python.org/dev/peps/pep-0268/
  * (66) ——. (2002) PEP 279 Acceptance. [Online]. Available: https://mail.python.org/pipermail/python-checkins/2002-April/025640.html
  * (67) ——. (2003) PEP 308 Rejection. [Online]. Available: https://mail.python.org/pipermail/python-checkins/2003-August/037477.html
  * (68) ——. (2002) PEP 296 – Adding a bytes Object Type. [Online]. Available: https://www.python.org/dev/peps/pep-0296/
  * (69) ——. (2005) PEP 308 Acceptance — Conditional Expression Resolution. [Online]. Available: https://mail.python.org/pipermail/python-dev/2005-September/056846.html
  * (70) ——. (2004) PEP 285 Acceptance. [Online]. Available: https://mail.python.org/pipermail/python-checkins/2002-April/025716.html
  * (71) ——. (2004) PEP 326 Rejection — PEP 326 (quick location possibility). [Online]. Available: https://mail.python.org/pipermail/python-dev/2004-January/042306.html
  * (72) Python. (2020) Changing the python language. [Online]. Available: https://devguide.python.org/langchanges/
  * (73) Python Mailing List Discussions. (2004) [Python-Dev] PEP 326 (quick location possibility). [Online]. Available: https://mail.python.org/pipermail/python-dev/2004-January/042300.html
  * (74) Python. (2014) [python-dev] pep 326 (quick location possibility). [Online]. Available: https://mail.python.org/pipermail/python-dev/2004-January/042300.html
  * (75) Python Mailing List Discussions. (2015) no title. [Online]. Available: https://mail.python.org/pipermail/python-list/2015-September.txt