# A Review & Framework for Modeling Complex Engineered System Development
Processes

John Meluso, Jesse Austin-Breneman, James P. Bagrow, & Laurent Hébert-Dufresne
This work has been submitted to the IEEE for possible publication. Copyright
may be transferred without notice, after which this version may no longer be
accessible. Manuscript received [to be completed by editor]. The work of J.
Meluso, J. Bagrow, & L. Hébert-Dufresne was supported in part by Google Open
Source under the Open-Source Complex Ecosystems And Networks (OCEAN)
project.John Meluso is the OCEAN Postdoctoral Fellow at the Vermont Complex
Systems Center at the University of Vermont, Burlington, VT 05405.Jesse
Austin-Breneman is Assistant Professor of Mechanical Engineering at the
University of Michigan, Ann Arbor, MI 48109.James Bagrow is Associate
Professor of Mathematics & Statistics at the University of Vermont,
Burlington, VT 05405.Laurent Hébert-Dufresne is Assistant Professor of
Computer Science at the University of Vermont, Burlington, VT 05405.

###### Abstract

Developing complex engineered systems (CES) poses significant challenges for
engineers, managers, designers, and businesspeople alike due to the inherent
complexity of the systems and contexts involved. Furthermore, experts have
expressed great interest in filling the gap in theory about how CES develop.
This article begins to address that gap in two ways. First, it reviews the
numerous definitions of CES along with existing theory and methods on CES
development processes. Then, it proposes the ComplEx System Integrated
Utilities Model (CESIUM), a novel framework for exploring how numerous system
and development process characteristics may affect the performance of CES.
CESIUM creates simulated representations of a system architecture, the
corresponding engineering organization, and the new product development
process through which the organization designs the system. It does so by
representing the system as a network of interdependent artifacts designed by
agents. Agents iteratively design their artifacts through optimization and
share information with other agents, thereby advancing the CES toward a
solution. This paper describes the model, conducts a sensitivity analysis,
provides validation, and suggests directions for future study.

###### Index Terms:

Systems engineering, large-scale systems, product development, simulation,
complexity theory

##  I Introduction

Society increasingly relies on healthcare systems, stock markets, automotive
systems, national defense programs, and countless other designed systems;
however, these systems continually grow more difficult to develop and manage
due to the complexity within and around them. These complex engineered systems
(CES) are large sets of highly-interacting engineered artifacts with a defined
purpose.111An artifact is any piece of technology designed to serve a specific
purpose [1], often used as “an umbrella term for any technical product of
human minds including physical parts, software, processes, information, etc.”
[2]. Often “difficult to describe, understand, predict, manage, design, or
change” [1] they are composed of many heterogeneous elements, characterized by
nonlinear interactions at multiple levels of organization and abstraction, and
exhibit complex behaviors which emerge from those interactions [3].

To date, government and industry organizations alike manage systems via
systems engineering techniques [4], Deming’s quality control methods [5], and
six sigma principles [6]. However, traditional methods have not kept pace with
the varying scales and increasing interactions between elements of systems.
The cost and schedules of engineering projects have grown exponentially [7, 8,
9], leading to frequent overruns [10, 11], billions of dollars in losses, and
even in lives [12, 13, 14]. Such pervasive failures mean that rather than
“exceptional,” failures have instead become “normal” or expected [15]. Made
more challenging, organizational factors often create the adverse conditions
in which society feels the effects of technical and “human” errors [16].

While traditional systems engineering methods still hold value, leading
government, industry, and academic voices have expressed a “dire need” [17]
for deeper theoretical understandings of CES, their development processes, and
the organizations that create them [18, 10, 11, 8, 19, 17, 20]. Throughout his
work on behalf of the United States National Science Foundation (NSF), the
United States Department of Defense (DoD), and the International Council on
Systems Engineering (INCOSE), Collopy & colleauges identified several areas
“ripe for exploration” to address these needs including abstraction and model-
based systems design [21, 22]. He suggests that “theory could make a real
difference” toward understanding CES development processes (CESDPs) [17].

Researchers have begun to build theory for traditional systems engineering
processes [23] and frameworks for particular CES contexts such as autonomous
vehicles [24] or Internet of Things devices [25, 26]. Nonetheless, few efforts
have suggested frameworks through which to build theoretical understandings of
the complexity involved in CES more broadly and their requisite development
processes [21]. Increasingly, scholars recommend utilizing systems theory to
assess such problems, a set of concepts rooted in complex systems and
management scholarship [27, 28, 29, 30], acknowledging the need for agent-
based methods to explore relative independence, discrete events, and
interactive dynamics [31].

Fortunately, recent advances make such analysis possible. Network theory
facilitates simulated representations of CES architectures [32, 33, 34], and
agent-based modeling grants access to macro-level outcomes of micro-level
decisions and interactions [35, 36, 37]. Systems engineering research has
explored alternative system formulations such as Value-Driven Design [38, 39]
while engineering design research has formulated Multidisciplinary Design
Optimization [40], each of which lends itself to abstract representations of
CES development. Likewise, organizational research is beginning to explore
computational methods of representing organizational processes [41, 42, 43,
44, 45, 46] as engineering follows suit [47, 48, 49, 50, 51, 2].

In light of these advances, this paper builds on the previous work of Meluso
et al. [2] to propose a modeling framework for analyzing how organizations
develop CES. The framework combines techniques from systems engineering,
complex systems, engineering design, and organizational theory to form the
ComplEx System Integrated Utilities Model (CESIUM).

The rest of this article is organized as follows. First, it summarizes the
literature defining CES, the constituent elements of CESDPs, and the methods
for scientifically studying CESDPs (Sec. II). It then describes the model and
simulation methods of CESIUM as an abstract, but grounded, representation of a
CESDP (Sec. III), before characterizing the model through parameter sweeps,
discussing validation, and exemplifying its potential for theory formation
(Sec. IV). The paper closes by generalizing the framework to facilitate the
development of a more robust theory of CES development.

##  II Background

Several disciplines touch, and uniquely define, the complex systems
constructed by people. As such, this section starts by delineating the
definitions of CES (Sec. II-A). From this common understanding, it describes
the constituent elements of CESDPs (Sec. II-B). Then, it describes the methods
through which researchers study CESDPs (Sec. II-C) before detailing the
literature specific to the methods of CESIUM (Sec. II-D).

###  II-A Definitions of Complex Engineered Systems

Scholars continue to debate what to call the systems of interacting artifacts
that people design, and with good reason. Different definitions emphasize
different system characteristics, the relationships between the artifacts
within the system, and the relationship between the system and the outside
world. At the least, it creates something of a “branding problem” when
university departments adopt more than 20 unique names to describe CES [1]. A
few particular terms are worth highlighting, though, as they reveal the core
elements necessary to describe CESDPs.

####  II-A1 Complex Systems

Conceptually, complex systems appear in disciplines from biology [52, 53] to
physics [32] to sociology [54, 55]. The interdisciplinary field of complex
systems grew out of these and several other disciplines in the latter half of
the twentieth century with a few core methods for understanding adaptation,
information, and collective behaviors [56]. Newman [57] notes that “there is
no precise technical definition of a ‘complex system,’ but that most
researchers in the field would probably agree that it is a system composed of
many interacting parts, such that the collective behavior of those parts
together is more than the sum of their individual behaviors.” An important
characteristic of complex systems is what are called emergent behaviors, the
unexpected macroscopic behaviors that often result from simple microscopic
interactions between constituent elements [56, 57]. These
characteristics—significant interactions and emergent behaviors—collectively
suffice to begin defining engineered versions of such systems.222The term
complexity also appears throughout the complex systems literature, along with
the field of complexity science. For the purposes of this article, complexity
refers to the characteristics of a complex system and are thus roughly
synonymous with complex systems, though entire texts exist to define the term
and field more completely [56].

####  II-A2 Complex Engineered Systems

As with complex systems, consensus does not exist about the definition of a
complex engineered system [58]. The simplest definition is “networks of
interconnected components” [34], though the focus therein limits itself to
networks of physical artifacts. Braha et al. [3] define CES as “engineered
systems [that] are composed of many heterogeneous subsystems and are
characterized by observable complex behaviors that emerge as a result of
nonlinear spatio-temporal interactions among the subsystems at several levels
of organization and abstraction.” Put another way, they are complex systems
composed of designed artifacts with interactivity and potential for emergence,
thereby making CES a subset of complex systems.

####  II-A3 Engineering Systems

So far, the definitions of CES do not consider the significant role that
people play is the creation, operation, and evolution of these systems. The
term engineering system largely derives from the Massachusetts Institute of
Technology’s former Engineering Systems Division, now part of the Institute
for Data, Systems, & Society [59], and takes a more social perspective on
engineered systems. Their definition, “a class of systems characterized by a
high degree of technical complexity, social intricacy, and elaborate
processes, aimed at fulfilling important functions in society,” [1]
underscores the social purposes and objectives while acknowledging the
technical, procedural, and social interaction. Much like how information
scholars embrace social subsystems of systems via the term socio-technical
system [60], the term engineering system ensures that human interaction with
technology and other people remains integral, and consequently unique among
the definitions.

####  II-A4 Systems of Systems

Moving closer to practitioners of engineering, the terminology changes yet
again. The DoD provides a formal definition of a system of systems, “a set or
arrangement of systems that results when independent useful systems are
integrated into a larger system that delivers unique capabilities” [61].
International standards bodies provide similar definitions [62]. Papers on
systems of systems often refer to such standards [63, 64] reflecting the
industry-orientation of the term and practice-based attempts to address
complexity [65, 66, 67, 68], though academics utilize similar definitions
[69]. Valid critiques exist on both sides: Bar-Yam [70] notes that traditional
systems engineering practices struggle to manage the many interactions and
interdependencies between systems, though Alderson & Doyle [71] similarly
critique complex systems for not addressing issues of practice.

####  II-A5 Large-Scale Complex Engineered Systems

This practice orientation gives way to another definition seeking to bridge
theory and practice. Large-scale complex engineered systems (abbreviated LSCES
or LaCES) are “engineering projects with significant cost and risk, extensive
design cycles, protracted operational timelines, a significant degree of
complexity, and dispersed supporting organizations” [51]. The term grew out of
a confluence of experts at the NSF, the National Aeronautics and Space
Administration (NASA), academia, and industry to address the challenges
described in Sec. I [72, 73, 10, 39, 11]. Proponents advocated for the
development of systems engineering theory [72, 39], of engineering design
[73], optimization techniques [10, 39, 74], and socio-technical understandings
of systems [10, 11, 74]. Despite efforts to incorporate the various interests
of the academy and industry, the technical and the social, the term remains
limited to the mechanical design community.

####  II-A6 A Working Definition of Complex Engineered Systems

Given these definitions and the various concepts they encompass, this paper
uses the term complex engineered system (still abbreviated CES) defined as
large sets of highly-interacting engineered artifacts with a defined purpose.
This definition includes the interactivity and potential for emergence of
complex systems. While the definition does not explicate human influence, it
presumes that people design, operate, and evolve artifacts. Similarly, while
environments, contexts, and non-designed objects also interact with the
system, they remain peripheral until integrated or modified by people, at
which point they too become artifacts. Through this definition it becomes
possible to advance theory of how CES develop.

###  II-B Constituents of CES Development Processes

This section is not a description of specific development processes, such as
the Systems V [69, 66] or the Toyota Production System [75, 76], but of the
abstracted elements that compose a CESDP. These elements include:

  1. (a)

the organization developing the CES,

  2. (b)

the context in which the organization develops the CES,

  3. (c)

the process through which the organization develops the CES, and

  4. (d)

the complex engineered system itself.

To fully understand how CES develop, researchers and practitioners alike must
understand all four of these elements. The following sections address these
elements in turn.

####  II-B1 Organizational Theory

Management decisions affect organizational performance, and consequently,
system performance. This holds true with subsystem complexity [77], the
balance between project and functional management in a matrix organization
[78], employee incentives [79], managerial responses to failure [80], and even
employee perceptions of procedural justice in top management decisions [81].
The complexity of these systems necessitates organizational involvement, so
leading scholars argue organizational theory is necessary for creating systems
engineering theory as well [19] because engineering is a fundamentally social
activity [8].

One of the core organizational theory texts is Scott & Davis’ Organizations &
Organizing [82]. Historically, three views of organizations emerged: as
rational systems that presume formal relational structures between people who
share a common goal; as natural systems wherein people have multiple (shared
and individual) goals and the relational structure serves as a resource; and
as open systems through which resources pass and interact with shifting
coalitions of people.333In organizational theory contexts, “organizations are
systems of elements, each of which affects and is affected by the others” [82]
much as in technical systems, though here the term includes people,
information, and artifacts. Each theoretical perspective bears similarity with
CES concepts and may serve as foundations upon which to build theory.

####  II-B2 Context

Defining a system’s boundaries is one of the most important tasks in system
definition, and no less important are understanding inputs, outputs, and
external forces or “externalities” [1]. The breadth or narrowness of a
project’s scope can affect performance outcomes [83]. Just as the context of
the system affects its performance, so too does the context of the
organization such as through information technology selection [84], the
influence of marketing on project decisions [85], whether the companies
involved in development are vertically or horizontally integrated [86], and
organizational culture [87]. Theoretical work in this space is primarily
limited to the organization and management sciences to date, though it
certainly merits further inquiry given its importance.

####  II-B3 Development Processes

Practitioners have long suggested that an organization’s choice of development
process matters, from Kelly Johnson and the Lockheed “Skunk Works”[88] to
present-day Silicon Valley start-up culture. Unsurprisingly, system
performance depends on the chosen development process [83]. Why is this the
case? How do different processes achieve better performance than one another?
What development process characteristics make it more or less likely to
achieve an organization’s strategic objectives for “simple” systems let alone
more complex ones?

Currently, insufficient theory exists to answer these and similar questions
[17]. Some knowledge exists in the management sciences about “new product
development” [89, 90, 91, 92] and engineering design researchers are
conducting experiments on micro-level processes that produce better designs
[93, 73, 50, 94, 95]. Still, much remains unknown.

####  II-B4 Theory on Complex Engineered Systems

The majority of existing theory on CES grounds itself in network theory.
Network scholars observed that the internet, the world wide web, power grids,
and other CES could be represented as networks [96, 32, 97, 98, 33].
Engineering scholars have characterized the networks of several CES as well
including spacecraft, vehicles, and software. Theories have begun to form
around structure [99, 34], modularity [100, 101, 102], quality [77, 34],
innovation [103], and the limitations of systems engineering for CES [70]. A
small contingent of work also studies how uncertainty affects system
performance via game theory [104, 105, 51].

###  II-C Methods for Scientifically Studying CESDPs

Building systems engineering theory requires formal methodologies with which
to create those theories. Given the elements of CESDPs described in Sec. II-B,
researchers can employ existing scientific methods used in organizational
scholarship to study organizations, contexts, processes, and CES. As with much
of social science research, organizational research methods tend to fall under
the categories of qualitative and quantitative.

Researchers typically utilize qualitative methods—including interviews [106],
ethnography [107], case studies [108], and grounded theory [109]—to answer
questions of “what happens,” “how it happens,” and “why it happens” in
organizations. Prominent examples with engineering organizations include
assessments of the Three Mile Island [110] and Challenger [111] disasters,
though studies also utilize such methods to broaden and deepen understandings
of non-catastrophic activities in engineering organizations [74, 112, 104,
113, 51, 2].

From understandings of the “whats, hows, and whys,” researchers draw on
quantitative methods—surveys [114], content analysis [115], network analyses
[116, 117], simulations [43, 45], statistical approaches, etc.—to assess the
extent, relationships, and relative impacts of phenomena on macro-level
outcomes of interest. Note, though, that quantitative data are similarly
social in organizational contexts and consequently require care [114, 118].

Unfortunately, gathering data on enough CESDPs to reach statistically
significant conclusions is not feasible today due to limits on the volume of
publicly-accessible data that describe substantial portions of development
processes [2]. In such cases, agent-based models (ABMs) and multi-agent
systems (MAS) are promising tools for assessing the macro-level effects of
micro-level behaviors [55, 119].444The difference between ABMs and MAS is
subtle. ABMs typically emphasize descriptive understandings of the
interactions between agents whereas MAS take normative approaches to
coordinate agents toward the best collective outcome [119]. Both have their
place in CES research depending on the scientific objectives. Through either
of these techniques, researchers can simulate communication [2], game
theoretic interactions [36, 105], and numerous network dynamics [117].
Standard Design of Experiments (DOE) techniques apply for understanding both
the characteristics under study and the parameters of the model [120, 121,
122].555See Sec. II-D2 for more on ABMs.

###  II-D Methods Specific to CESIUM

The availability of diverse methods creates opportunities for combining
methods into representations of CESDPs, as with CESIUM. The following sections
describe the contributions of network theory, agent-based modeling, and design
optimization to CESIUM. 666Sec. II-D was partially adaptated from Sec. II-B of
[2].

####  II-D1 Network Models

Network theory represents systems of people or artifacts as nodes and edges,
as shown in Fig. 1a. Nodes, points connected to one another in pairs, may
represent people, subsystems, artifacts, etc. in a CES. Connections or
interactions between nodes are called edges or ties and can be represented by
an adjacency matrix $A_{ij}$, where $A_{ij}=1$ if an edge exists between nodes
$i$ and $j$ and $A_{ij}=0$ otherwise [117] as in Fig. 1b. When thought of as
interfaces between artifacts, a $A_{ij}$ forms a Design Structure Matrix
representing a CES [34]; thought of as a social network, edges represent
pathways for information diffusion [123]. The edges of an adjacency matrix may
be either directionless (called undirected edges) or directed from one node to
another, in which case $A_{ij}=1$ only if an edge points from node $j$ to node
$i$ [117].

(a)

(b)

(c)

Figure 1: Visual representations of undirected networks generated with a
Holme-Kim preferential attachment algorithm and probability that new edges
will form a triangle $p_{t}=0.9$. (a) Graph of a network with $n=1000$
artifacts. The orange dots represent nodes or artifacts, and the black lines
represent edges or interfaces. (b) An adjacency matrix $A_{ij}$ for a network
with $n=100$ artifacts where $A_{ij}=1$ (black) if an edge exists between
nodes $i$ and $j$ and $0$ (white) if not. Also known as a Design Structure
Matrix. (c) Scale-free degree distribution of a network with $n=1000$
artifacts. Note the approximately-linear, negatively-sloping form of the
distribution on a log-log scale, characteristic of a scale-free degree
distribution [117].

The number of other artifacts that each artifact $i$ interfaces with is called
the degree $k_{i}$ of $i$. A normalized histogram of a network’s degrees is
called a degree distribution [117]. Ample studies have shown that artifacts in
many (but not all) complex systems follow a scale-free degree distribution,
also called power-law or inverse exponential distributions [124, 125, 126,
127, 99, 33, 34]. Scale-free distributions take the form
$p_{k}=c_{1}k^{-c_{2}}$ where $p_{k}$ is the probability of randomly selecting
a node with degree $k$, $c_{1}$ is a constant, and positive constant $c_{2}$
is the exponent of the power law with typical values of $2\leq c_{2}\leq 3$
[117]. The resulting function appears as a negatively-sloping line in a log-
log plot as in Fig. 1c. Braha & Bar-Yam [127, 99] and Sosa et al. [34] suggest
that complex system degree distributions generally follow a power-law with a
cut-off at some large degree reminiscent of a bow-tie structure, making
systems resilient to most small perturbations but vulnerable to certain rare
perturbations [97, 53]. Consequently, scale-free and bow-tie distributions may
each provide unique insights.

A more-recent subject in network theory with significant potential is that of
generative network models that algorithmically construct networks out of basic
rules [117]. One of the most common network generation algorithms is called
preferential attachment which builds a network by connecting new nodes to
existing nodes with an attachment probability proportional to the degree of
the existing node [117]. Several such algorithms exist including those of
Price [128], Barabási & Albert [124], Holme & Kim [129], and Carlson & Doyle
[97], all of which generate networks with scale-free degree distributions
[117, 129]. Recent advances in peer-to-peer network studies allow generative
algorithms to establish hard or soft cut-offs in the distribution [130, 131].
The Holme-Kim preferential attachment algorithm includes a parameter for
tuning node clustering [129], making it useful for simulating the meso-scale
practice of subsystem formation. On the other hand, Carlson & Doyle’s highly
optimized tolerance (HOT) generates bow-tie structures, yielding macro-level
realism. An approach combining the Holme-Kim and HOT approaches would likely
provide the most realistic CES structures, albeit similarly presumptive of
solution structures.

####  II-D2 Agent-Based Modeling

Agent-based modeling is a widely-used, effective, and tested method for
simulating CES [132, 133, 134, 135]. An ABM creates a system of autonomous
decision-making entities called agents which individually assess their
situations and make decisions based on a set of rules [134]. Agents affect
their surroundings through their actions and in doing so, self-organization,
patterns, structures, and behaviors emerge from the “ground up” that were not
explicitly programmed into the models but nevertheless arise through agent-
interactions [135]. This “ground up” agent-centered approach differentiates
ABMs from other system modeling methods such as discrete event simulation and
system dynamic models which take top-down approaches [119]. The ability of
ABMs to demonstrate emergence also makes it ideal for understanding highly-
interacting systems.

Recent applications of ABMs include systems design [136, 137, 138, 51, 2] and
organization studies [139, 140, 141, 142, 2]. INCOSE promotes ABMs as one of
the primary methods through which “to inform trade-off decisions” regarding
“complexity in system design and development” [20]. Because CES are often
composed of many smaller engineered systems that are designed, developed, and
operated by organizations of dispersed, loosely connected people [10], ABMs
facilitate simulation of aggregated artifact development in ways that top-down
models cannot [119].

####  II-D3 Design Optimization

Engineers in various disciplines use design optimization to maximize the
performance of a system, a process of selecting the relative “best”
alternative from among a set of possible designs called the design space
[143]. They do this through objective functions (or utility functions in
agentic contexts), sets of evaluation criteria typically constructed as
functions describing the relationships between independent or decision
variables [143]. Optimization algorithms then explore the design space to find
a global or local minimum (or maximum depending on problem construction) as
efficiently as possible to identify a solution [40].

While the methods of constructing system objectives are beyond the scope of
this paper, one method for searching design spaces remains relevant. Validated
studies have shown that engineers [48, 49] and organizations [139] sample
their design spaces comparably to simulated annealing which can therefore
serve as a modeling representation of human decision-making.

Given this background, the next section combines these concepts to form CESIUM
through the framework established in Sec. II-B.

##  III Methodology

The CESIUM framework simulates a theoretical CESDP by modeling its constituent
elements (Sec. II-B). This section describes how the base instance of CESIUM
represents each element: the system architecture and system boundary (Sec.
III-A), the constituent artifacts (Sec. III-B), the organization of
interacting agents (Sec. III-C), agents’ design processes (Sec. III-D), and
the system development process (Sec. III-E). Thereafter, an example execution
of the base instance follows for a vizualizable system (Sec. III-F) along with
a brief introduction to the framework’s flexibility in simulating variations
on the elements of CESDPs (Sec. III-G).777The implementation of CESIUM
described herein was developed using Python 3 with the NumPy 1.18.1, SciPy
1.4.1, and NetworkX 2.4 packages [144, 145, 146]. The full code for this
article is available at https://github.com/meluso/cesium-framework, and a
clean version of code for the base instance of CESIUM is available at
https://github.com/meluso/cesium-base.888Secs. III-A–III-E were partially
adapted from Sec. III of [2].

###  III-A System Architecture

First, assume that a CES is composed of $n$ interacting artifacts where each
artifact $i\in\\{1,\ldots,n\\}$. The $n$ artifacts interact with one another
in a technical network generated from a scale-free degree distribution via a
Holme-Kim preferential attachment algorithm by adding $h$ edges to each new
node $i$. The probability that the first edge connecting node $i$ to the
network will attach to a specific node $j$ is proportional to that node’s
degree $k_{j}$; the probability that subsequently-added edges will be placed
to form a triangle by connecting $i$ to a node $l$ that is already connected
to $j$ is $p_{t}$ [129]. The result is a network of $n$ interacting artifacts
described by adjacency matrix $A_{ij}$, in this case with no formal hierarchy
and clustering specified by $p_{t}$. Generally, artifacts could also interact
with nodes outside the system boundary. The base instance assumes that the
system boundary contains all factors with sufficient impact on the CES leaving
no exogenous variables, and hence no interaction between the CES and its
context.

###  III-B Artifact Construction

Figure 2: An example of artifact interaction. In this case, the $i^{th}$
artifact is artifact 1 with variable $x_{i}=x_{1}$. Artifact 1 interacts with
$j\in\\{2,4\\}$ and so $x_{j}=[x_{2},x_{4}]$. Therefore, artifact 1’s
objective function is $f_{1}(x_{1},x_{2},x_{4})$.

(a)

(b)

(c)

(d)

Figure 3: Graphical representations of the selected objective functions with
two decision variables. (a) Absolute Sum Function. (b) Sphere Function. (c)
Ackley Function. (d) Levy Function.

In a real-world setting, the design of each artifact $i$ in the system would
depend on numerous contextual and specific factors, say
$\mathbf{v}_{i}=[v_{i1},v_{i2},\ldots]$. Because these factors cannot be known
a priori for innumerable real systems, the model representatively
parameterizes these variables such that the design of each artifact is defined
by a single decision variable $x_{i}(\mathbf{v}_{i})$. Therefore, each $x_{i}$
parameterizes a complex set of inputs, allowing the performance of each
artifact to be represented as an objective function
$y_{i}=f_{i}(x_{i},\mathbf{x}_{j})$, where $j\in\\{1,\ldots,k_{i}\\}$
represents the set of artifacts interfacing with artifact $i$, and
$\mathbf{x}_{j}$ is a vector of the parameterized decision variables of the
$k_{i}$ artifacts as exemplified in Fig. 2. These variables can be expressed
using a combined notation $\mathbf{x}_{i}=[x_{i},\mathbf{x}_{j}]$.

Objective functions can take countless forms in the framework, but because
$\mathbf{x}_{i}$ can be parameterized to any mapping, the simplest objective
topologies can be identified through a Taylor expansion of
$f_{i}(\mathbf{x}_{i})$. At some point $\mathbf{x}_{i0}$, with
$\mathbf{\partial x}_{i}=\mathbf{x}_{i}-\mathbf{x}_{i0}$ and Hermitian matrix
$H_{i}(\mathbf{x}_{i})$, the Taylor expansion of $f_{i}(\mathbf{x}_{i})$ is

$\begin{split}f_{i}(\mathbf{x}_{i})\approx f_{i}(\mathbf{x}_{i0})&+\nabla
f_{i}(\mathbf{x}_{i0})\mathbf{\partial x}_{i}\\\ &+\frac{1}{2}\mathbf{\partial
x}_{i}^{T}H_{i}(\mathbf{x}_{i0})\mathbf{\partial x}_{i}+\cdots\end{split}$ (1)

If $\mathbf{x}_{i}$ is parameterized such that the optima all occur at
$\mathbf{x}_{i}^{*}=\mathbf{0}$, then
$f_{i}(\mathbf{x}_{i}\neq\mathbf{0})>f_{i}(\mathbf{x}_{i}^{*}=\mathbf{0})$ for
every $i$. The simplest such relationship occurs under the condition of
linearity such that $|\nabla f_{i}(\mathbf{x}_{i0})\mathbf{\partial
x}_{i}|=\sum_{m=1}^{k_{i}+1}|x_{m}|$ with all higher order terms equal to 0.
The next simplest then is a quadratic formed by taking only the second term of
the Taylor expansion such that $|\frac{1}{2}\mathbf{\partial
x}_{i}^{T}H_{i}(\mathbf{x}_{i0})\mathbf{\partial
x}_{i}|=\sum_{m=1}^{k_{i}+1}x_{m}^{2}$. More complex relationships, with
multiple minima and asymmetrical objective functions, will of course prove
more realistic though may prove more challenging to formulate and analyze.
However, those complexities can similarly be overcome through the initial
parameterizations of each $x_{i}$.

Hence, to capture a variety of CES, the base instance considers four objective
functions that correspond to the first- and second-order Taylor expansion
terms, a symmetrical multiple-minima function, and an asymmetrical multiple-
minima function as follows:

  1. (a)

The Absolute Sum function, corresponding to the first term of the Taylor
expansion, on domain for all $x_{m}\in[-10.00,10.00]$:

$f_{i}(\mathbf{x}_{i})=\sum_{m=1}^{k_{i}+1}|x_{m}|$ (2)

The optimum $\mathbf{x}_{i}^{*}=(0,\ldots,0)$ minimizes $f_{i}$ for the
Absolute Sum function yielding $f_{i}(\mathbf{x}_{i}^{*})=0$.

  2. (b)

The Sphere function [147], corresponding to the second-order term of the
Taylor expansion, on the recommended evaluation domain for all
$x_{m}\in[-5.12,5.12]$:

$f_{i}(\mathbf{x}_{i})=\sum_{m=1}^{k_{i}+1}x_{m}^{2}$ (3)

with $f_{i}(\mathbf{x}_{i}^{*})=0$ at $\mathbf{x}_{i}^{*}=(0,\ldots,0)$.

  3. (c)

The Ackley function [147], a symmetrical function with multiple minima, on the
recommended evaluation domain for all $x_{m}\in[-32.768,32.768]$, $c_{1}=20$,
$c_{2}=0.2$, and $c_{3}=2\pi$:

$\begin{split}f_{i}(\mathbf{x}_{i})=&-c_{1}\exp\Bigg{(}-c_{2}\sqrt{\frac{1}{k_{i}+1}\sum_{m=1}^{k_{i}+1}x_{m}^{2}}\Bigg{)}\\\
&-\exp\Bigg{(}\frac{1}{k_{i}+1}\sum_{m=1}^{k_{i}+1}\cos(c_{3}x_{m})\Bigg{)}+e^{1}+c_{1}\end{split}$
(4)

with $f_{i}(\mathbf{x}_{i}^{*})=0$ at $\mathbf{x}_{i}^{*}=(0,\ldots,0)$.

  4. (d)

The Levy function [147], an asymmetrical function with multiple minima, on the
recommended evaluation domain for all $x_{m}\in[-10.00,10.00]$, and
$c_{m}=1+\frac{x_{m}-1}{4}$:

$\begin{split}f_{i}\left(\mathbf{x}_{i}\right)=&\sin^{2}\left(\pi
c_{1}\right)\\\
&+\sum_{m=1}^{k_{i}}\left(c_{m}-1\right)^{2}\left(1+10\sin^{2}\left(\pi
c_{m}+1\right)\right)\\\
&+\left(c_{k_{i}+1}-1\right)^{2}\left(1+\sin^{2}\left(2\pi
c_{k_{i}+1}\right)\right)\end{split}$ (5)

with $f_{i}(\mathbf{x}_{i}^{*})=0$ at $\mathbf{x}_{i}^{*}=(1,\ldots,1)$,
though a variable transformation could shift this optima to the origin.

Fig. 3 shows these functions in two dimensions.

One of the advantages of these particular objective functions is that they are
$n$-dimensional, meaning they scale to incorporate the $k_{i}$ decision
variables for each neighbor $j$ of $i$ for any positive integer value of
$k_{i}$. This leaves $n$ coupled objective functions
$\\{f_{1},\ldots,f_{n}\\}$ that comprise the CES being designed.

###  III-C The Organization

CESIUM assumes that the members of an organization (engineers or otherwise)
can be represented by agents in an agent-based model. In the base instance,
one agent represents one engineer. While, the pairings of agents to artifacts
can and do take many forms in real-life, an organization’s structure
approximately reflects the structure of the technical artifacts that those
organizations create. This phenomena, called the mirroring hypothesis or
Conway’s Law [148, 149, 150], means the simplest mapping is one in which the
network of agents in the organization and the network of artifacts in the CES
are synonymous as assumed herein. Each agent is then responsible for one
artifact in the system. As a result, an organization exists wherein engineers
pass information via the technical network.999Sec. V-A discusses more complex
mappings, including variation in the numbers of agents artifacts, as part of
the generalization of CESIUM.

###  III-D Agent Design Process

Next, the model incorporates a design process for the artifacts. Given the
mirroring hypothesis, each agent uses the technical objective function of its
artifact as its utility function, so the objective functions will be spoken of
as belonging to the agents. Each agent seeks to optimize (that is, minimize)
its objective function over a number of turns to reach the best
performance.101010While stepping forward in time through the use of turns is a
common practice in ABMs, the system-level motivation and analog for turns are
described in Sec. III-E.

Again, validated studies have shown that engineers sample their design spaces
similar to optimization using simulated annealing [48, 49, 139]. During each
turn of the model, each agent engineer receives a set of constant input values
$\mathbf{x}_{j}$ from its $k_{i}$ interacting agents to utilize when adjusting
$x_{i}$ to optimize their objective functions. Agents explore the design space
using a simulated annealing algorithm in search of a local optimum
$y_{i}^{*}=f_{i}(x_{i}^{*},\mathbf{x}_{j})$ with a random initial position in
the domain of $x_{i}$, $\omega$ iterations per optimization, initial
temperature of $\tau$, and cooling rate of $\rho$. No cognitive factors affect
agent decision-making. While other design space search algorithms remain
possible, the base instance of CESIUM uses simulated annealing for agent
optimization due to its established validation.

###  III-E System Development Process

With a technique established through which members of an organization develop
each artifact, it becomes feasible to simulate the development of the complete
system. Returning to the beginning of the CESDP: the ABM first initializes a
new system following the method outlined in Sec. III-A. Then, the model steps
through a series of turns where one turn in the ABM represents one design
cycle in a CESDP. From one perspective, turns represent a common development
process technique known as the Shewhart & Deming Cycle [151] wherein members
of an organization iteratively improve and share the design for their subset
of a system; though, turns also serve as a discretized representation of
design refinement more generally.

Each turn, agents exchange information. To facilitate the exchange, CESIUM
stores the latest reported designs of all agents in a system vector
$\mathbf{S}$ as a central repository. At the beginning of each design cycle,
each agent receives $\mathbf{S}$ as a constant input before proceeding to
optimize their variable $x_{i}$ using only the values from their networked
neighbors $\mathbf{x}_{j}$. Then, each agent passes their updated value of
$x_{i}$ back to the system vector for storage in $\mathbf{S}$ and a new design
cycle begins with the updated values as constants.

The model performs these design cycles, iterating through all of the agents in
each cycle, until either the system design converges or the model performs $d$
design cycles. System convergence is calculated from a metric for System
Performance $F$. While many formulations of performance are possible, the base
instance defines $F$ as a sum of the reported objective evaluations of all of
the agents during the current design cycle:

$F(t,\mathbf{f})=\sum_{i=1}^{n}f_{i}(t,\mathbf{x}_{i}(t))$ (6)

Although the $n$ objective functions $f_{i}$ have different magnitudes
depending on the degree $k_{i}$ of each artifact $i$, the System Performance
is assumed to have a greater dependence on components which are more highly
connected so simply adding their contributions exemplifies this behavior. By
this definition, system convergence occurs when the System Performance
consistently changes less than a specified convergence threshold $\varepsilon$
over each of three turns, that is, meeting the following condition:

$\frac{1}{3}\sum_{t^{\prime}=1}^{3}|F(t,\mathbf{f})-F(t-t^{\prime},\mathbf{f})|<\varepsilon$
(7)

Every execution of the model completes a minimum of three design cycles. Upon
convergence, the Number of Design Cycles $N$ measures how long it took the
organization to converge to a design solution, and therefore to complete the
CESDP. This completes the base instance of CESIUM.

###  III-F Example Execution

(a)

(b)

(c)

Figure 4: An example execution of CESIUM for a system with two agents using
the Levy function. White dots and arrows illustrate the progress of
$\mathbf{x}$ over time, thereby affecting the objective evaluations (a)
$f_{1}$ of agent 1, (b) $f_{2}$ of agent 2, and (c) $F$ the overall system
performance. (a) $f_{1}(x_{1},x_{2})$ (b) $f_{2}(x_{2},x_{1})$ (c)
$F(t,f_{1},f_{2})$

To illustrate this complete development process simulation, this section
provides an example execution for a small system which demonstrates how CESIUM
simulates individual and an organizational exploration of their respective
design spaces with varying effects on system outcomes. Because larger numbers
of agents become difficult to visualize, consider a small example system of
the base instance with only two agents as in Fig. 4. Each of the
$i\in\\{1,2\\}$ agents controls one decision variable, $x_{1}$ and $x_{2}$,
respectively. Assuming that the artifacts are neighbors in the system network,
consider the case where agents’ objective functions $f_{1}(x_{1},x_{2})$ and
$f_{2}(x_{2},x_{1})$ are Levy functions, as in Fig. 3d, with System
Performance $F(t,f_{1},f_{2})$.

Unlike the other objectives presented herein, the Levy function is not
symmetric meaning that some $f_{i}(x_{i},x_{j})$ is not necessarily equal to
$f_{i}(x_{j},x_{i})$. Given the ordering of the variables for each agent’s
objective function, Agent 1 has limited control over its outcomes because
changes in $x_{1}$ tend to yield smaller changes in $f_{1}$ than changes in
$x_{2}$ due to the shape of the Levy function, and vice versa for Agent 2.
This configuration then represents a system in which the performance $f_{i}$
of each artifact $i$ is disproportionately affected by design decisions made
for the $k_{j}$ artifacts that artifact $i$ interacts with through the system
network making convergence to the global optimum more challenging.

Figs. 4a & 4b show the progress of each agent toward its individual objective
as a function of the two decision variables, and Fig. 4c shows the system
convergence and progress toward the global optimum. Observe how changes in
$x_{2}$ tended to improve $f_{1}$ even as they produced limited improvement in
$f_{2}$; however, those changes in $x_{2}$ eventually led $x_{1}$ to identify
improved values of $f_{1}$ and consequently $x_{2}$ to do the same for
$f_{2}$. The iterative process of simulated annealing—a combination of
necessarily accepting values of $x_{i}$ that improve $f_{i}$ and
probabilistically accepting values of $x_{i}$ that do not improve
$f_{i}$—gradually improved System Performance as in Fig. 4c. Hence, this
example execution demonstrates how CESIUM simulates both individual and
organization exploration of a design space toward collective system outcomes.

###  III-G Framework Flexibility

Throughout Secs. III-A–III-E, the base instance makes a number of simplifying
assumptions. However, the CESIUM framework requires very few of these. Indeed,
only the elements described in Sec. II-B & forthcoming Sec. V-A need
accounting for, thereby creating the flexibility for researchers to alter any
other qualities described herein so as to enable exploration of other
characteristics, from system architecture topologies to representations of
established development processes.

For example, Meluso et al. [2] recently utilized the framework to simulate
miscommunication in a CESDP. Building on prior findings that practitioners
define the “estimates” they communicate with one another either as
representations of their current design status or future design outcome [152],
the authors introduced a variable $p_{e}$ specifying the probability that an
agent would share future estimates (or conversely $1-p_{e}$ that they would
share current estimates) finding that miscommunication can affect system
performance. Such investigations readily follow from Sec. V, which generalizes
the modeling framework and recommends further avenues of inquiry.

##  IV Model Analysis

TABLE I: Independent and dependent variables for the base instance of CESIUM. Name |  Variable |  Abbreviation |  Type |  Description |  Values
---|---|---|---|---|---
Num. of Nodes |  $n$ |  Num. Nodes |  Independent |  The number of nodes in the generated system during that execution. |  $\\{$50, 100, 500, 1000$\\}$
Objective Function |  $f_{i}$ |  Obj. Fn. |  Independent |  The objective function employed by each agent to evaluate their individual performance. |  $\\{$absolute-sum, sphere, levy, ackley$\\}$
Triangle Probability |  $p_{t}$ |  Tri. Prob. |  Independent |  An input to the Holme-Kim preferential attachment algorithm along with the Num. of Edges Added. The probability that a new edge originating from node $i$ will be placed such that the new edge connects $i$ to a node $l$ which is already connected to node $j$, thereby forming a triangle between $i$, $j$, and $l$. The reverse, $1-$x_prob_triangle, is the probability that new nodes will connect to any existing node according to preferential attachment. |  $\\{$0, 0.1, 0.2, …, 1.0$\\}$
Convergence Threshold |  $\varepsilon$ |  Conv. Thresh. |  Independent |  The system convergence threshold. |  $\\{$0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0$\\}$
Future Estimate Probability |  $p_{e}$ |  Fut. Est. Prob. |  Independent |  Probability that a given agent will utilize a “future” estimate (when $\mu$ is “future”) throughout the system execution, where $1-p_{e}$ is the probability that an agent will utilize a “current” estimate instead. |  $\\{$0, 0.1, 0.2, …, 1.0$\\}$
Num. of Edges Added |  $h$ |  Num. Edg. |  Constant |  The number of edges generated to connect new nodes $i$ to existing nodes $j$ in the system during network generation using the Holme-Kim preferential attachment algorithm. |  2
Maximum Num. of Design Cycles |  $d$ |  Max. Cycl. |  Constant |  The maximum number of design cycles. |  100
Initial Temperature |  $\tau$ |  Init. Temp. |  Constant |  The initial temperature for the dual annealing algorithm in SciPy [145]. |  0.1
Num. of Annealing Iterations |  $\omega$ |  Anneal. Iter. |  Constant |  The maximum number of global search iterations for the dual annealing algorithm. |  1
Estimation Method |  $\mu$ |  Est. Meth. |  Constant |  Specifies the types of estimates generated by agents. A value of “current” allows only current design values. A value of “future” allows probabilistic selection of either “current” design values of $x_{i}$ or “future” predictions of $x_{i}$. |  “future”
Cooling Rate |  $\rho$ |  Cool. Rate |  Constant |  Cooling rate of the dual annealing algorithm. |  2.62
Num. of Design Cycles |  $N$ |  Num. Cycl. |  Dependent |  The number of design cycles that it took the specified execution of CESIUM to converge, bounded by the minimum number of cycles required for convergence and the maximum allowed design cycles. |  [4,100]
System Performance |  $F$ |  Sys. Perf. |  Dependent |  The final system performance for the specified execution. |  Distribution- and objective-specific
TABLE II: Multivariate OLS regressions with robust standard errors of the varied parameters against System Performance. Models II.1 & II.2 include objective function fixed effects; Models II.3 through II.7 instead analyze the objective function subsets of the dataset. More negative values represent more optimal performances of objective functions. | All Functions | Absolute Sum | Sphere | Ackley | Levy
---|---|---|---|---|---
| Model II.1 | Model II.2 | Model II.3 | Model II.4 | Model II.7 | Model II.5 | Model II.6
Dependent Variable | Sys. Perf. | Sys. Perf. | Sys. Perf. | Sys. Perf. | log(Sys. Perf.) | Sys. Perf. | Sys. Perf.
Constant | -1563.787*** | -226.174*** | 0.450*** | 13.298*** | 1.972*** | -3.004*** | 615.742***
| (6.696) | (3.125) | (0.041) | (0.499) | (0.032) | (0.568) | (10.110)
Fn: Sphere | 9.355+ | -30.709*** |  |  |  |  |
| (3.997) | (1.090) |  |  |  |  |
Fn: Ackley | 303.090*** | 20.903*** |  |  |  |  |
| (3.714) | (0.992) |  |  |  |  |
Fn: Levy | 7477.843*** | 1540.989*** |  |  |  |  |
| (8.969) | (6.987) |  |  |  |  |
Num. Nodes$\dagger$ | 4.301*** | 0.529*** | 0.000 | -0.042*** | -0.321*** | 0.731*** | 18.636***
| (0.008) | (0.005) | (0.000) | (0.001) | (0.006) | (0.001) | (0.022)
Tri. Prob. | 6.884 | -17.531*** | -0.250*** | -27.972*** | -1.371*** | -2.032+ | -12.871
| (7.675) | (4.758) | (0.070) | (0.838) | (0.047) | (0.956) | (16.545)
Conv. Thresh.$\dagger$ | 17.287*** | 21.547*** | 3.011*** | 1.697*** | 1.219*** | 15.859*** | 123.995***
| (0.679) | (0.533) | (0.015) | (0.101) | (0.007) | (0.148) | (1.930)
Fut. Est. Prob. | -496.031*** | 398.783*** | -0.278*** | -25.708*** | -3.399*** | -1.357 | -362.163***
| (7.735) | (5.172) | (0.070) | (0.944) | (0.048) | (0.968) | (16.576)
Num. Nodes$\dagger$ $\times$ Tri. Prob. |  | 0.013 | 0.000 | 0.033*** | 0.201*** | -0.023*** | 0.041
|  | (0.007) | (0.000) | (0.001) | (0.008) | (0.002) | (0.027)
Num. Nodes$\dagger$ $\times$ Conv. Thresh.$\dagger$ |  | -0.031*** | -0.001*** | 0.000*** | -0.029*** | -0.019*** | -0.104***
|  | (0.001) | (0.000) | (0.000) | (0.001) | (0.000) | (0.002)
Num. Nodes$\dagger$ $\times$ Fut. Est. Prob. |  | -0.927*** | 0.000** | 0.107*** | 0.688*** | -0.014*** | -3.800***
|  | (0.007) | (0.000) | (0.001) | (0.009) | (0.002) | (0.027)
Tri. Prob. $\times$ Conv. Thresh.$\dagger$ |  | 0.021 | 0.028 | 0.048 | -0.055*** | 0.326 | -0.318
|  | (0.637) | (0.017) | (0.103) | (0.004) | (0.179) | (2.460)
Tri. Prob. $\times$ Fut. Est. Prob. |  | 24.695*** | 0.502*** | 57.744*** | 1.070*** | 4.366* | 36.168
|  | (7.133) | (0.118) | (1.547) | (0.032) | (1.616) | (26.805)
Conv. Thresh.$\dagger$ $\times$ Fut. Est. Prob. |  | -12.102*** | 0.016 | -0.005 | -0.161*** | 0.002 | -48.423***
|  | (0.651) | (0.018) | (0.134) | (0.005) | (0.181) | (2.462)
Fn: Sphere $\times$ Num. Nodes |  | 0.029*** |  |  |  |  |
|  | (0.001) |  |  |  |  |
Fn: Sphere $\times$ Tri. Prob. |  | 14.602*** |  |  |  |  |
|  | (1.037) |  |  |  |  |
Fn: Sphere $\times$ Conv. Thresh. |  | -1.170*** |  |  |  |  |
|  | (0.103) |  |  |  |  |
Fn: Sphere $\times$ Fut. Est. Prob. |  | 47.295*** |  |  |  |  |
|  | (1.354) |  |  |  |  |
Fn: Ackley $\times$ Num. Nodes |  | 0.669*** |  |  |  |  |
|  | (0.001) |  |  |  |  |
Fn: Ackley $\times$ Tri. Prob. |  | -8.667*** |  |  |  |  |
|  | (1.027) |  |  |  |  |
Fn: Ackley $\times$ Conv. Thresh. |  | 5.457*** |  |  |  |  |
|  | (0.104) |  |  |  |  |
Fn: Ackley $\times$ Fut. Est. Prob. |  | -5.120*** |  |  |  |  |
|  | (1.276) |  |  |  |  |
Fn: Levy $\times$ Num. Nodes |  | 16.512*** |  |  |  |  |
|  | (0.009) |  |  |  |  |
Fn: Levy $\times$ Tri. Prob. |  | 21.064+ |  |  |  |  |
|  | (8.670) |  |  |  |  |
Fn: Levy $\times$ Conv. Thresh. |  | 54.089*** |  |  |  |  |
|  | (0.793) |  |  |  |  |
Fn: Levy $\times$ Fut. Est. Prob. |  | -2026.815*** |  |  |  |  |
|  | (8.833) |  |  |  |  |
Num. Obs. | 1355200 | 1355200 | 338800 | 338800 | 338800 | 338800 | 338800
R-squared | 0.618 | 0.969 | 0.722 | 0.125 | 0.663 | 0.886 | 0.945
R-squared Adj. | 0.618 | 0.969 | 0.722 | 0.125 | 0.663 | 0.886 | 0.945
Residual Standard Error | 2824.141 | 798.934 | 5.935 | 64.654 | 1.617 | 92.402 | 1537.809
Significances: + p $<$ 0.05, * p $<$ 0.01, ** p $<$ 0.005, *** p $<$ 0.001
Robust standard errors in parentheses.
$\dagger$ Variable log-transformed for Model II.7 only
TABLE III: Multivariate OLS regressions with robust standard errors of the varied parameters against the Number of Design Cycles. Models III.1 & III.2 include objective function fixed effects; Models III.3 through III.6 instead analyze the objective function subsets of the dataset. | All Functions | Absolute Sum | Sphere | Ackley | Levy
---|---|---|---|---|---
| Model III.1 | Model III.2 | Model III.3 | Model III.4 | Model III.5 | Model III.6
Dependent Variable | Num. Cycles | Num. Cycles | Num. Cycles | Num. Cycles | Num. Cycles | Num. Cycles
Constant | 63.659*** | 66.030*** | 67.403*** | 42.565*** | 98.085*** | 45.007***
| (0.038) | (0.064) | (0.100) | (0.060) | (0.051) | (0.197)
Fn: Sphere | -22.043*** | -21.819*** |  |  |  |
| (0.023) | (0.062) |  |  |  |
Fn: Ackley | 27.487*** | 30.553*** |  |  |  |
| (0.027) | (0.062) |  |  |  |
Fn: Levy | -6.666*** | -21.809*** |  |  |  |
| (0.074) | (0.176) |  |  |  |
Num. Nodes | 0.021*** | 0.025*** | 0.027*** | 0.019*** | 0.010*** | 0.134***
| (0.000) | (0.000) | (0.000) | (0.000) | (0.000) | (0.001)
Tri. Prob. | -0.130** | 0.112 | -0.290+ | -0.057 | -0.034 | -1.588***
| (0.040) | (0.087) | (0.122) | (0.090) | (0.078) | (0.274)
Conv. Thresh. | -2.648*** | -3.977*** | -3.004*** | -2.219*** | -2.905*** | -5.185***
| (0.004) | (0.027) | (0.096) | (0.008) | (0.014) | (0.026)
Fut. Est. Prob. | 1.894*** | -0.656*** | -0.199 | -0.044 | -0.107 | 8.884***
| (0.040) | (0.087) | (0.130) | (0.090) | (0.078) | (0.283)
Num. Nodes $\times$ Tri. Prob. |  | 0.000*** | 0.000*** | 0.000 | -0.003*** | -0.011***
|  | (0.000) | (0.000) | (0.000) | (0.000) |  1 (0.001)
Num. Nodes $\times$ Conv. Thresh. |  | 0.001*** | -0.001*** | 0.000*** | 0.008*** | 0.005***
|  | (0.000) | (0.000) | (0.000) | (0.000) | (0.000)
Num. Nodes $\times$ Fut. Est. Prob. |  | 0.000+ | 0.000** | 0.000*** | 0.000 | -0.020***
|  | (0.000) | (0.000) | (0.000) | (0.000) | (0.001)
Tri. Prob. $\times$ Conv. Thresh. |  | -0.002 | -0.509*** | -0.015 | -0.106*** | -0.010
|  | (0.009) | (0.105) | (0.009) | (0.015) | (0.028)
Tri. Prob. $\times$ Fut. Est. Prob. |  | 0.068 | 0.165 | 0.006 | 0.084 | 1.234***
|  | (0.104) | (0.153) | (0.124) | (0.115) | (0.372)
Conv. Thresh. $\times$ Fut. Est. Prob. |  | 0.307*** | -0.503*** | -0.060*** | -0.046** | 0.736***
|  | (0.009) | (0.150) | (0.009) | (0.015) | (0.032)
Fn: Sphere $\times$ Num. Nodes |  | -0.008*** |  |  |  |
|  | (0.000) |  |  |  |
Fn: Sphere $\times$ Tri. Prob. |  | -0.007 |  |  |  |
|  | (0.069) |  |  |  |
Fn: Sphere $\times$ Conv. Thresh. |  | 0.943*** |  |  |  |
|  | (0.026) |  |  |  |
Fn: Sphere $\times$ Fut. Est. Prob. |  | 0.004 |  |  |  |
|  | (0.069) |  |  |  |
Fn: Ackley $\times$ Num. Nodes |  | 0.012*** |  |  |  |
|  | (0.000) |  |  |  |
Fn: Ackley $\times$ Tri. Prob. |  | -0.415*** |  |  |  |
|  | (0.071) |  |  |  |
Fn: Ackley $\times$ Conv. Thresh. |  | 1.559*** |  |  |  |
|  | (0.026) |  |  |  |
Fn: Ackley $\times$ Fut. Est. Prob. |  | 0.112 |  |  |  |
|  | (0.071) |  |  |  |
Fn: Levy $\times$ Num. Nodes |  | 0.104*** |  |  |  |
|  | (0.000) |  |  |  |
Fn: Levy $\times$ Tri. Prob. |  | -1.022*** |  |  |  |
|  | (0.204) |  |  |  |
Fn: Levy $\times$ Conv. Thresh. |  | -1.104*** |  |  |  |
|  | (0.031) |  |  |  |
Fn: Levy $\times$ Fut. Est. Prob. |  | 11.345*** |  |  |  |
|  | (0.207) |  |  |  |
Num. Obs. | 1355200 | 1355200 | 338800 | 338800 | 338800 | 338800
R-squared | 0.724 | 0.813 | 0.795 | 0.709 | 0.726 | 0.677
R-squared Adj. | 0.724 | 0.813 | 0.795 | 0.709 | 0.726 | 0.677
Residual Standard Error | 14.628 | 12.046 | 8.875 | 7.041 | 6.699 | 18.834
Significances: + p $<$ 0.05, * p $<$ 0.01, ** p $<$ 0.005, *** p $<$ 0.001
Robust standard errors in parentheses.
Figure 5: Random forest feature importances for the objective function subsets
of the executions. Higher bars indicate a more important feature (greater
reduction in error when predicting the outcome). The Number of Nodes and the
Convergence Threshold tended to have the greatest importance on System
Performance and the Number of Design Cycles for most objective functions, the
notable exception being the Sphere Function, the outcomes of which were
significantly affected by the Future Estimate Probability as well.

Model design decisions affect outcomes with any model. But as this section
will demonstrate, the strength of CESIUM as a modeling framework lies in its
ability to expose patterns that emerge across variation in design decisions.
Abstracting away from particular CESDPs to their constituent elements (Sec.
II-B) creates the opportunity to explore the fundamental characteristics of
those elements, such as the number of artifacts in a system, the resources
provided to engineers, or the qualities of the phases of a development
process.

To that end, this section fills three purposes. First, Sec. IV-A characterizes
the model through sensitivity analysis. In so doing, the analysis reveals a
tremendous range of possible outcomes contingent on variables representative
of practical system and design process qualities. Then, Sec. IV-B validates
the framework. Sec. IV-C closes by demonstrating the modeling framework’s
power to identify testable hypotheses and postulate theories describing how
real phenomena in practice affect system outcomes.

###  IV-A Sensitivity Analysis

The authors performed a sensitivity analysis [153] to characterize the
performance of CESIUM with different representations of design processes by
sweeping independent variables, examining their main and interaction effects
via regression, and determining overall feature importances with a random
forest.

Table I contains the independent variables, constants, and dependent
variables; their definitions; and the values varied as part of a fractional
factorial experiment. The subset of independent variables were selected
because  (a) they easily map to real-world concepts (e.g. system size,
development time), and  (b) they were assessed most likely to produce
variation in the dependent variables.  Sec. III-G also mentioned the Future
Estimate Probability $p_{e}$—the probability that agents will exchange
estimates of their future design projections instead of their current design
values [2, 152]. Variable $p_{e}$ was included to demonstrate how deviating
from assumptions can reveal more and less optimal strategies for CESDPs
contingent upon other variables (see Sec. IV-C). Combined, the independent
variable sets consisted of 13,552 unique combinations, each of which ran 100
times for a total of 1,355,200 executions of CESIUM.

The authors first quantified the main and interaction effects by utilizing
multivariate ordinary least squares (OLS) regressions with robust standard
errors [153, 154], as described by Models II.1–7 in Table II for system
performance $F$, and by Models III.1–6 in Table III for the number of design
cycles $N$. To start, Models II.1 & III.1 found main effects for the five
independent variables through OLS regression of the full dataset (Num. Obs.
$=$ 1,355,200) with objective function dummy variables capturing the objective
function fixed effects and the Absolute Sum function represented in the
constant. Model II.1 accounted for 61.8% of the System Performance variance
and Model III.1 for 72.4% of the variance in the Number of Design Cycles.
These two models suggested that CESDP outcomes significantly depend on design
space objective functions, system size (Num. Nodes.), development process
stopping conditions (Conv. Thresh.), and even the type of information
exchanged within development processes (Fut. Est. Prob.). Still, the large
residual standard errors indicated significant variation remained unaccounted
for which could reside in interaction or more complex relationships.

Models II.2 (R-squared: 96.9%) & III.2 (R-squared: 81.3%) included interaction
effects, moderately improving the residuals over Models II.1 & III.1 while
attributing significant variation to most of the interaction terms. The
significant differences between the objective function interaction
coefficients supported that an organization’s ability to develop an optimal
CES significantly depends on the objective functions corresponding to the
design spaces of its constituent artifacts. Models II.3–6 and III.3–6 provided
further support for this hypothesis by isolating the interaction terms and
residual standard errors wthin the objective function data subsets (Num. Obs.
$=$ 338,800 each). The Levy function absorbed the largest residuals, leaving
more-proportionate residuals for the other functions while measuring a variety
of interaction effect amplitudes within the functions. The Sphere function
System Performance (Model II.4) remained an outlier among the regressions,
though, only explaining 12.5% of the variance before log transformations of
the System Performance, Number of Nodes, and Convergence Threshold (Model
II.7, R-squared: 66.3%).

According to the models, nearly all of the system outcomes depend strongly on
the Number of Nodes, the Convergence Threshold, or both, though rarely through
interaction effects. The Number of Nodes tended to increase the Number of
Design Cycles, while the Convergence Threshold consistently revealed an
inverse relationship between System Performance and the Number of Design
Cycles, regardless of the objective function selected. In contrast, the main
and interaction effects of the Triangle and Future Estimate Probabilities
proved scattered with only select cases generating significant effects. For
example, Models II.3 (Absolute Sum), II.4 (Sphere), & II.5 (Ackley) identified
statistically significant effects between the probability of triangle
formation in the system network and the probability of sharing future
estimates (Tri. Prob. $\times$ Fut. Est. Prob.) with magnitudes comparable to
their respective constants, an effect that was limited in the Ackley and
absent from the Levy function.

To further support the previous regressions, and capture any non-linear or
higher-order effects of each variable, a feature importance analysis using a
random forest regressor was applied, shown in Fig. 5. Random forests fit an
ensemble of decisions trees to predict the outcome variable, and feature
importances describe which variables best reduce errors in outcome predictions
when trees split the data on that variable [155]. Random forests were fit
using the default parameters of sci-kit learn v0.24.1 with a maximum depth of
four. The results proved largely consistent with the regressions with a few
noteworthy exceptions. Again, the Number of Nodes and Convergence Thresholds
proved most important. However, the Triangle and Future Estimate Probabilities
only substantially affected the Sphere function performance, suggesting their
effects may be limited to particular topologies. This somewhat contradicts the
regression results which additionally identified an interaction effect for the
Absolute Sum and Ackley functions, suggesting that the regressions for those
functions may have missed nonlinear and/or higher-order effects, or excluded
uncaptured variation in either the network structure or simulated annealing
searches. That said, the contributions of the Triangle and Future Estimate
Probabilities remained significant for the Sphere function.

###  IV-B Validation

Because CESIUM models organizational processes, this work draws from
organization science for its definitions of micro- and macrovalidation.
Microvalidation ensures that the behaviors and mechanisms coded into the model
sufficiently represent, or are “grounded” in, their real-world analogs.
Macrovalidation is divided into face validation—whether results are
superficially distinguishable from real-world conceptual phenomena—and
empirical validation—whether results are statistically calibrated to match
real-world data [156, 157].

To satisfy the conditions of microvalidation, Sec. III grounds each
construction decision of CESIUM in either established research or simplified
abstractions of real-world practices. To satisfy face and empirical
macrovalidation, simulation results would need to correspond to outcomes one
might encounter in practice. While face validation occasionally proves
possible, Sec. II-C cites how insufficient data exists about CESDPs to perform
empirical validation. Indeed, CESIUM is designed to address the absence of
sufficient data by reconstructing the underlying mechanisms governing
development processes and generating data instead.

Consequently, it becomes beneficial to view modeling frameworks like CESIUM
from a different perspective. Carley refers to computational models as
“hypothesis generating machine[s]” [156], in CESIUM’s case generating data
with which to form hypotheses about causal variables in CESDPs. With
sufficient evidence from multiple sources of knowledge, researchers can
validate those hypotheses and they become established systems engineering
theories. Thereafter, systems engineers can devise methods for improving the
outcomes resulting from those relationships.

Contributing to that process, the following section will demonstrate both the
creation of hypotheses and formation of theory by validating the results of
the sensitivity analysis.

###  IV-C Theory Building Demonstration

Consider the results of the sensitivity analysis for each independent
variable:

  1. Result 1:

The Number of Design Cycles tended to increase proportional to the Number of
Nodes, while System Performance varied depending on the objective function.

  2. Result 2:

Both system outcomes varied depending on the Objective Functions, via both
main and interaction effects.

  3. Result 3:

Triangle Probability had small or statistically insignificant effects on
system outcomes for all but the Sphere function, where System Performance
varied as a function of Triangle Probability, Future Estimate Probability, and
interaction between the two.

  4. Result 4:

Increasing the size of Convergence Threshold tended to decrease the Number of
Design Cycles while degrading the System Performance.

  5. Result 5:

Increasing the Future Estimate Probability had small or statistically
insignificant effects on system outcomes for all but the Sphere function,
where System Performance varied through interaction with the Triangle
Probability.

Beginning with Result 1, this result suggests that even if System Performance
varies differently for different objective functions, (larger) systems with
more artifacts will tend to take longer to converge on system designs. This
observation can be posed as a hypothesis for further examination.

######  Hypothesis 1.

The time it takes an organization to converge on a system design increases as
a function of system size.

Scholars can now examine this hypothesis against existing and novel studies to
determine whether it merits conversion into a theory.

Next, note that Results 1-3 & 5 involve variation in system outcomes depending
on the objective functions of the agents. This, too, could easily form a
hypothesis. However, the finding is consistent with extensive research from
Game Theory, Decision Science, and forms the underlying premise of Value-
Driven Design [38], thereby providing at least face validation for the
hypothesis. With abundant evidence from multiple disciplines, the claim is
more accurately stated as an established piece of knowledge in the form of a
theory. Recall that CESIUM assumes that the utilities of the agents designing
system artifacts are synonymous with the objective functions of the artifacts.
Then:

######  Postulate 1.

The outcomes of a complex engineered system development process depend on the
objective functions of the agents who develop system artifacts.

Variation in the other regression coefficients coincident with the objective
functions leaves question as to the relationship between even the most
significant of the remaining variables and system outcomes. Nevertheless, CES
outcomes certainly depended on the selected function giving this first
postulate in combination with existing evidence.

The results lend themselves to two further hypotheses meriting examination.
Result 4 suggests:

######  Hypothesis 2.

Given a system objective function in negative null form, system performance
varies inversely proportionately to system convergence time as a function of
maximum uncertainty required in system performance.

Finally, recall that the Triangle Probability is a proxy for modularity or
clustering in the technical network of the system architecture, and that the
Future Estimate Probability examines the likelihood that an agent will
communicate information about either current design statuses or projected
design outcomes gives the final hypothesis. Combining these abstractions with
Results 3 & 5 gives:

######  Hypothesis 3.

System performance varies as a function of interactions between modularity in
the system architecture and the temporal references of information exchanged
within the organization.

##  V Future Directions

As a framework, CESIUM’s conceptualization of CESDPs presents numerous
opportunities for further exploration including system architectures, system
development processes, artifact design, patterns and contents of interaction,
and objective functions. To facilitate exploration, this section begins by
generalizing the framework (Sec. V-A). Then, it outlines recommended
directions for future inquiry based on the framework to facilitate the
development of theory (Sec. V-B).

###  V-A Framework Generalization

The base instance of CESIUM makes simplifying assumptions about CES and CESDPs
including a scale-free degree distribution of artifacts, no exogenous
variables, one agent per artifact via the mirroring hypothesis, simulated
annealing artifact design, a design cycle development process, and a simple
sum of artifact performances as representative of CES performance. The
following paragraphs relax those assumptions and, in so doing, opens
opportunities for exploration.

Assume that a CES is composed of $n_{a}$ artifacts interacting in a technical
network and system architecture described by adjacency matrix $A_{ij}$. The
$n_{b}$ members of an organization (engineers and otherwise) can be
represented by agents in an agent-based model which exchange information with
one another in a communication network described by adjacency matrix $B_{qr}$.
Each of the $n_{b}$ agents acts upon or “works on” some set of the $n_{a}$
artifacts according to matrix $C_{iq}$. Summing over $q$ then yields the
number of agents that contribute to the design of each artifact
$\lambda_{i}=\sum_{q}C_{iq}$.

Agents apply design decisions to each artifact they work on, influenced by
numerous specific and contextual factors. Describe the decisions made by agent
$w\in\\{1,\ldots,\lambda_{i}\\}$ about the $i$th artifact with a decision
variable vector $\mathbf{v}_{iw}=[v_{iw1},v_{iw2},\ldots]$ of contributions
made by each agent working on the artifact. The contributions of any variables
exogenous to the system can be described by
$\mathbf{u}_{i}=[u_{i1},u_{i2},\ldots]$. As before, these variables can be
representatively parameterized so that each artifact is modeled by a single
decision variable
$x_{i}(\mathbf{v}_{i1},\ldots,\mathbf{v}_{i\lambda_{i}},\mathbf{u}_{i})$ (but
this time of vectors $\mathbf{v}_{iw}$ and $\mathbf{u}_{i}$), thereby mapping
a set of complex factors to a more manageable form.

This parameterized specification for the design process of each artifact makes
it possible to describe the performance for each as $y_{i}$ which is a
function of the artifact’s own design $x_{i}$, the vector of designs
$\mathbf{x}_{j}$ corresponding to the $k_{i}$ interacting artifacts, and a
vector of any exogenous variables $\mathbf{z}_{i}$ that influence performance.
The combined notation $\mathbf{x}_{i}=[x_{i},\mathbf{x}_{j},\mathbf{z}_{i}]$
then allows the artifact’s performance to be described in terms of an
objective function $y_{i}=f_{i}(\mathbf{x}_{i})$. Over time, agents can
individually act toward the improvement of their own objective functions, CES
objective functions, or both. Hence, given one function for each artifact and
another vector of exogenous contributions to the system $\mathbf{\xi}$, multi-
objective functions can be formed quantifying overall outcomes for the CESDP
including system performance at different points in time,
$F(t,f_{1}(\mathbf{x}_{1}),\ldots,f_{i}(\mathbf{x}_{i}),\ldots,f_{n}(\mathbf{x}_{n}),\mathbf{\xi})$.

While abstract in this form, these generalizations enable researchers to vary
assumptions within the CESIUM framework, from which it becomes possible to
explore variable dependencies, pose hypotheses, and form theories about
CESDPs. The following section enumerates some such options.

###  V-B Recommended Investigations

Real technical network distributions vary, so exploring bow-tie structures
generated by HOT (Sec. II-D1) and complete graphs (in which every node is
connected to every other node) could yield novel insights about how artifact
interaction networks shape system outcomes. Again, the clustering parameter of
the Holme-Kim algorithm leaves it well-suited to study subsystem formation and
interaction, closely connecting it to current practices. Of course, artifact
networks are rarely static in practice, so incorporating network evolution
into CESIUM could shed light on how temporal variation affects both agent-
level and system-level outcomes. And because even CES vary dramatically in
size, exploring different scales may reveal how outcomes manifest as a result
of hierarchical decomposition of systems and system of systems interactions.

Development processes themselves also merit exploration including different
permutations of the relationships between artifacts, variable composition,
agent contribution, and design space exploration. Artifacts undergo different
development processes depending on the industry, design process, designer
identity, and designer context [158], each of which could be studied through
CESIUM. Likewise, many CESDPs employ processes that are more involved than the
turn-based Shewhart & Deming Cycle (Sec. III-E), including the Systems V,
agile development, scrum, etc. The ability to explore such varied development
processes may prove one of the greatest opportunities afforded by CESIUM.

The framework also opens possibilities for studying the contents of
interactions between contributors. While researchers have begun to investigate
communication frequency in other contexts, the contents of communication and
interpretation have received limited treatment to date, thereby lending
themselves to questions of feedback, coordination between designers [159], and
different mediums of information exchange.

Different objective constructions could prove extremely informative at the
agent, subsystem, system, and system of systems levels. The functions selected
for this work began to explore concepts such as linear relationships between
variable contributions (Absolute Sum), parabolic relationships (Sphere), local
optima (Ackley), and disproportionate influence by collaborators (Levy),
though innumerable topologies remain possible at each level. System
performance certainly takes forms other than sums of its parts, creating
opportunities to study questions of collective innovation, high-reliability
organizations, and evolving landscapes, from completely abstract to precisely
representative of real-world systems. Hopefully these suggestions will
contribute to the beginning of theory building for CES and CESDPs.

##  VI Conclusion

Complex engineered systems play essential roles in society. However, decades
of increasing system complexity and development process inadequacies have
garnered substantial concern about the capabilities of established systems
engineering processes. Leading experts from government, academic, and industry
organizations now call for the development of theories to guide solutions to
these growing needs. But to date, the notorious difficulty of gathering data
from real-world complex engineered system development processes has stymied
efforts to form and validate generalizable hypotheses about how systems
develop.

This work responds to that challenge by uniting recent advances from systems
engineering, complex systems, engineering design, and organization science to
form the ComplEx System Integrated Utilities Model. Beginning from a review of
terminology, the piece identifies the abstracted elements composing
development processes and integrates techniques for simulating each aspect
into a modeling framework. It then demonstrates the ability of CESIUM to
generate sizeable datasets representing the outcomes of system development
processes from which researchers can form theory. The underlying flexibility
of the framework leaves numerous opportunities for researchers to explore
abstracted phenomena and real-world examples alike toward the construction of
systems engineering theory.

## Acknowledgment

Thanks to all of our colleagues and friends who reviewed this work and
supported us, including Abigail Jacobs, Vanessa Rojano, Mojtaba Arezoomand,
Lynette Shaw, and Maciej Kurant.

## References

  * [1] O. L. de Weck, D. Roos, and C. L. Magee, _Engineering Systems: Meeting Human Needs in a Complex Technological World_. Mit Press, 2011.
  * [2] J. Meluso, J. Austin-Breneman, and L. Shaw, “An Agent-Based Model of Miscommunication in Complex System Engineering Organizations,” _IEEE Systems Journal_ , pp. 1–12, 2019.
  * [3] D. Braha, A. Minai, and Y. Bar-yam, _Complex Engineered Systems: Science Meets Technology_. Berlin/Heidelberg, NETHERLANDS, THE: Springer, 2006.
  * [4] S. J. Kapurch, _NASA Systems Engineering Handbook_ , 2007.
  * [5] W. E. Deming, “Out of the Crisis: Quality, Productivity and Competitive Position,” _Massachusetts Institute of Technology. Center for advanced engineering study_ , 1986.
  * [6] R. G. Schroeder, K. Linderman, C. Liedtke, and A. S. Choo, “Six Sigma: Definition and underlying theory,” _Journal of Operations Management_ , vol. 26, no. 4, pp. 536–554, 2008.
  * [7] N. R. Augustine, _Augustine’s Laws_. American Institute of Aeronautics and Astronautics, 1997.
  * [8] P. D. Collopy, C. L. Bloebaum, and B. L. Mesmer, “The Distinct and Interrelated Roles of Value-Driven Design, Multidisciplinary Design Optimization, and Decision Analysis,” in _12th AIAA Aviation Technology, Integration and Operations (ATIO) Conference and 14th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, 2012, 12th AIAA Aviation Technology, Integration, and Operations (ATIO) Conference and 14th AIAA/ISSMO_ , no. September. Reston, Virigina: American Institute of Aeronautics and Astronautics, Sep. 2012, pp. 1–10.
  * [9] I. Maddox, P. Collopy, and P. A. Farrington, “Value-Based Assessment of DoD Acquisitions Programs,” _Procedia Computer Science_ , vol. 16, pp. 1161–1169, 2013.
  * [10] C. L. Bloebaum, P. D. Collopy, and G. A. Hazelrigg, “NSF/NASA Workshop on the Design of Large-Scale Complex Engineered Systems—From Research to Product Realization,” in _12th AIAA Aciation Technology, Integration, and Operations (ATIO) Conference_ , no. September. Reston, Virigina: American Institute of Aeronautics and Astronautics, Sep. 2012, pp. 1–20.
  * [11] C. L. Bloebaum and A.-M. R. McGowan, “The Design of Large-Scale Complex Engineered Systems: Present Challenges and Future Promise,” in _AIAA Aviation Technology, Integration, and Operations (ATIO) Conference and 14th AIAA/ISSM_ , no. September. Reston, Virigina: American Institute of Aeronautics and Astronautics, Sep. 2012, pp. 1–19.
  * [12] W. P. Rogers, N. A. Armstrong, D. C. Acheson, E. E. Covert, R. P. Feynman, and R. B. Hotz, “Report of the presidential commission on the space shuttle challenger accident,” US Government Accounting Office, Washington, DC, Tech. Rep., Jun. 1986.
  * [13] Y. Bar-Yam, “When systems engineering fails-toward complex systems engineering,” in _SMC’03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483)_ , vol. 2, 2003, pp. 2021–2028 vol.2.
  * [14] H. W. Gehman Jr, J. L. Barry, D. W. Deal, J. N. Hallock, K. W. Hess, G. S. Hubbard, J. M. Logsdon, D. D. Osheroff, S. K. Ride, and R. E. Tetrault, “Columbia Accident Investigation Board Report. Volume 1,” Tech. Rep., Aug. 2003.
  * [15] C. Perrow, _Normal Accidents: Living with High Risk Technologies-Updated Edition_. Princeton university press, 2011.
  * [16] A. M. Madni and S. Jackson, “Towards a Conceptual Framework for Resilience Engineering,” _IEEE Systems Journal_ , vol. 3, no. 2, pp. 181–191, 2009.
  * [17] P. D. Collopy, “Systems engineering theory: What needs to be done,” in _2015 Annual IEEE Systems Conference (SysCon) Proceedings_ , 2015, pp. 536–541.
  * [18] A. A. Minai, D. Braha, and Y. Bar-Yam, “Complex Engineered Systems: A New Paradigm,” in _Complex Engineered Systems: Science Meets Technology_. Berlin, Heidelberg: Springer Berlin Heidelberg, 2006.
  * [19] K. P. Triantis and P. D. Collopy, “A comprehensive basis for systems engineering theory,” in _2014 IEEE International Systems Conference Proceedings_ , 2014, pp. 97–102.
  * [20] D. Salvucci-Favier, “A Complexity Primer for Systems Engineers,” _INCOSE Foundation_ , no. July, pp. 1–5, 2016.
  * [21] P. D. Collopy, “Report on the science of systems engineering workshop,” in _53rd AIAA Aerospace Sciences Meeting_ , 2015, pp. 1865–1865.
  * [22] T. G. Topcu, K. Triantis, R. Malak, and P. Collopy, “An interdisciplinary strategy to advance systems engineering theory: The case of abstraction and elaboration,” _Systems Engineering_ , vol. 23, no. 6, pp. 673–683, 2020.
  * [23] S. Safarkhani, I. Bilionis, and J. H. Panchal, “Toward a Theory of Systems Engineering Processes: A Principal–Agent Model of a One-Shot, Shallow Process,” _IEEE Systems Journal_ , vol. 14, no. 3, pp. 3277–3288, Sep. 2020.
  * [24] S. H. Young, T. A. Mazzuchi, and S. Sarkani, “A Framework for Predicting Future System Performance in Autonomous Unmanned Ground Vehicles,” _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , vol. 47, no. 7, pp. 1192–1206, Jul. 2017.
  * [25] J. Cecil, S. Albuhamood, A. Cecil-Xavier, and P. Ramanathan, “An Advanced Cyber Physical Framework for Micro Devices Assembly,” _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , vol. 49, no. 1, pp. 92–106, Jan. 2019.
  * [26] G. Fortino, C. Savaglio, G. Spezzano, and M. Zhou, “Internet of Things as System of Systems: A Review of Methodologies, Frameworks, Platforms, and Tools,” _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , vol. 51, no. 1, pp. 223–236, Jan. 2021.
  * [27] K. M. Adams, P. T. Hester, J. M. Bradley, T. J. Meyers, and C. B. Keating, “Systems Theory as the Foundation for Understanding Systems,” _Systems Engineering_ , vol. 17, no. 1, pp. 112–123, 2014.
  * [28] C. B. Keating, P. F. Katina, and J. M. Bradley, “Complex system governance: Concept, challenges, and emerging research,” _International Journal of System of Systems Engineering_ , vol. 5, no. 3, pp. 263–288, Jan. 2014.
  * [29] M. S. Poole, “Systems Theory,” in _The SAGE Handbook of Organizational Communication: Advances in Theory, Research, and Methods_. Sage Los Angeles, CA, 2014, pp. 49–74.
  * [30] C. B. Keating, P. F. Katina, R. Hodge, and J. M. Bradley, “Systems Theory: Bridging the Gap Between Science and Practice for Systems Engineering,” _INCOSE International Symposium_ , vol. 30, no. 1, pp. 1017–1031, 2020.
  * [31] P. F. Katina, A. Tolk, C. B. Keating, and K. F. Joiner, “Modelling and simulation in complex system governance,” _International Journal of System of Systems Engineering_ , vol. 10, no. 3, pp. 262–292, Jan. 2020.
  * [32] A.-L. Barabási, R. Albert, and H. Jeong, “Mean-field theory for scale-free random networks,” _Physica A: Statistical Mechanics and its Applications_ , vol. 272, no. 1, pp. 173–187, 1999.
  * [33] A. Clauset, C. Shalizi, and M. Newman, “Power-Law Distributions in Empirical Data,” _SIAM Review_ , vol. 51, no. 4, pp. 661–703, Nov. 2009, doi: 10.1137/070710111.
  * [34] M. E. Sosa, “Where do creative interactions come from? The role of tie content and social networks,” _Organization Science_ , vol. 22, no. 1, pp. 1–21, 2011.
  * [35] T. C. Schelling, “Dynamic models of segregation,” _Journal of mathematical sociology_ , vol. 1, no. 2, pp. 143–186, 1971.
  * [36] R. Axelrod, _The Complexity of Cooperation: Agent-Based Models of Competition and Collaboration_. Princeton University Press, 1997, vol. 3.
  * [37] E. B. Smith and W. Rand, “Simulating macro-level effects from micro-level observations,” _Management Science_ , vol. 64, no. 11, pp. 5405–5421, 2017\.
  * [38] P. D. Collopy and P. M. Hollingsworth, “Value-Driven Design,” _Journal of Aircraft_ , vol. 48, no. 3, pp. 749–759, May 2011.
  * [39] P. D. Collopy, “A Research Agenda for the Coming Renaissance,” in _50th AIAA Aerospace Sciences Meeting Including the New Horizons Forum and Aerospace Exposition_ , no. January. Reston, Virigina: American Institute of Aeronautics and Astronautics, Jan. 2012, pp. 1–10.
  * [40] J. R. R. a Martins and A. B. Lambe, “Multidisciplinary Design Optimization: A Survey of Architectures,” _AIAA Journal_ , vol. 51, no. 9, pp. 2049–2075, 2013.
  * [41] K. M. Carley, “Computational organization science: A new frontier,” _Proceedings of the National Academy of Sciences_ , vol. 99, no. suppl 3, pp. 7257 LP–7262, May 2002.
  * [42] L. Hong and S. E. Page, “Groups of Diverse Problem Solvers Can Outperform Groups of High-Ability Problem Solvers,” _Proceedings of the National Academy of Sciences of the United States of America_ , vol. 101, no. 46, pp. 16 385 LP–16 389, Nov. 2004.
  * [43] J. R. Harrison, Z. Lin, G. R. Carroll, and K. M. Carley, “Simulation modeling in organizational and management research,” _Academy of Management Review_ , vol. 32, no. 4, pp. 1229–1245, Oct. 2007, doi: 10.5465/amr.2007.26586485.
  * [44] G. Fioretti, “Agent-Based Simulation Models in Organization Science,” _Organizational Research Methods_ , vol. 16, no. 2, pp. 227–242, Dec. 2012, doi: 10.1177/1094428112470006.
  * [45] G. P. Morgan and K. M. Carley, “Computational Organizational Theory,” in _International Encyclopedia of the Social & Behavioral Sciences: Second Edition_, 2015, pp. 485–492.
  * [46] J. Clement and P. Puranam, “Searching for structure: Formal organization design as a guide to network evolution,” _Management Science_ , vol. 64, no. 8, 2018.
  * [47] R. M. Crowder, M. A. Robinson, H. P. Hughes, and Y. W. Sim, “The development of an agent-based modeling framework for simulating engineering team work,” _IEEE Transactions on Systems, Man, and Cybernetics Part A:Systems and Humans_ , vol. 42, no. 6, pp. 1425–1439, 2012.
  * [48] C. McComb, J. Cagan, and K. Kotovsky, “Lifting the Veil: Drawing insights about design teams from a cognitively-inspired computational model,” _Design Studies_ , vol. 40, pp. 119–142, 2015.
  * [49] ——, “Drawing Inspiration From Human Design Teams For Better Search And Optimization: The Heterogeneous Simulated Annealing Teams Algorithm,” _Submitted to the Journal of Mechanical Design_ , vol. 138, no. 4, pp. 1–6, Mar. 2016.
  * [50] ——, “Optimizing Design Teams Based on Problem Properties: Computational Team Simulations and an Applied Empirical Test,” _Journal of Mechanical Design_ , vol. 139, no. 4, pp. 041 101–041 101, 2017\.
  * [51] J. Meluso and J. Austin-Breneman, “Gaming the System: An Agent-Based Model of Estimation Strategies and their Effects on System Performance,” _Journal of Mechanical Design_ , vol. 140, no. 12, pp. 121 101–121 109, Sep. 2018, 10.1115/1.4039494.
  * [52] F. Jacob, “Evolution and tinkering,” _Science_ , vol. 196, no. 4295, pp. 1161–1166, 1977.
  * [53] H. Kitano, “Biological robustness,” _Nature Reviews Genetics_ , vol. 5, no. 11, pp. 826–837, 2004.
  * [54] D. J. Watts and S. H. Strogatz, “Collective dynamics of ‘small-world’networks,” _nature_ , vol. 393, no. 6684, p. 440, 1998.
  * [55] E. Bruch and J. Atwell, “Agent-Based Models in Empirical Social Research,” _Sociological Methods & Research_, vol. 44, no. 2, pp. 186–221, Oct. 2013, doi: 10.1177/0049124113506405.
  * [56] M. Mitchell, _Complexity: A Guided Tour_. Oxford University Press, 2009.
  * [57] M. E. J. Newman, “Resource Letter CS–1: Complex Systems,” _American Journal of Physics_ , vol. 79, no. 8, pp. 800–810, Jul. 2011.
  * [58] T. W. Simpson and J. R. R. A. Martins, “Multidisciplinary Design Optimization for Complex Engineered Systems: Report From a National Science Foundation Workshop,” _Journal of Mechanical Design_ , vol. 133, no. 10, pp. 101 002–101 002, 2011.
  * [59] O. de Weck, “MIT Engineering Systems Division R.I.P.: Eulogy for a successful experiment 1998-2015,” _MIT Faculty Newsletter_ , vol. XXVIII, no. 4, pp. 12–16, 2016.
  * [60] P. M. Leonardi, “Materiality, sociomateriality, and socio-technical systems: What do these terms mean? How are they different? Do we need them,” _Materiality and organizing: Social interaction in a technological world_ , vol. 25, 2012.
  * [61] Office of the Deputy Under Secretary of Defense for Acquisition and Technology Systems and Software Engineering, “Systems Engineering Guide for Systems of Systems,” Washington, DC, Tech. Rep., 2008.
  * [62] “System life cycle processes,” ISO/IEC/IEEE International Standard 15288 First edition, May 2015.
  * [63] A. Gorod, B. Sauser, and J. Boardman, “System-of-Systems Engineering Management: A Review of Modern History and a Path Forward,” _IEEE Systems Journal_ , vol. 2, no. 4, pp. 484–499, 2008.
  * [64] S. Ferreira and J. Hess, “Unmanned and Autonomous Systems of Systems Test and Evaluation: Challenges and Opportunities,” Tech. Rep., 2010.
  * [65] D. O. Norman and M. L. Kuras, “Engineering complex systems,” in _Complex Engineered Systems_. Springer, 2006, pp. 206–245.
  * [66] J. O. Clark, “System of Systems Engineering and Family of Systems Engineering from a standards, V-Model, and Dual-V Model perspective,” _2009 3rd Annual IEEE Systems Conference_ , pp. 381–387, Mar. 2009.
  * [67] W. C. Baldwin, B. Sauser, and R. Cloutier, “Simulation approaches for system of systems: Events-based versus agent based modeling,” _Procedia Computer Science_ , vol. 44, no. C, pp. 363–372, 2015.
  * [68] J. Dahmann and G. Roedler, “Systems of Systems Engineering Standards,” _Insight_ , vol. 19, no. 3, pp. 23–26, Oct. 2016, doi: 10.1002/inst.12102.
  * [69] C. Keating, R. Rogers, R. Unal, D. Dryer, A. Sousa-Poza, R. Safford, W. Peterson, and G. Rabadi, “System of Systems Engineering,” _Engineering Management Journal_ , vol. 15, no. 3, pp. 36–45, Sep. 2003, doi: 10.1080/10429247.2003.11415214.
  * [70] Y. Bar-Yam, “Engineering complex systems: Multiscale analysis and evolutionary engineering,” in _Complex Engineered Systems_. Springer, 2006, pp. 22–39.
  * [71] D. L. Alderson and J. C. Doyle, “Contrasting Views of Complexity and Their Implications For Network-Centric Infrastructures,” _IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans_ , vol. 40, no. 4, pp. 839–852, 2010.
  * [72] A. Deshmukh and P. Collopy, “Fundamental Research into the Design of Large-Scale Complex Systems,” in _13th AIAA/ISSMO Multidisciplinary Analysis Optimization Conference_ , ser. Multidisciplinary Analysis Optimization Conferences. American Institute of Aeronautics and Astronautics, Sep. 2010, doi:10.2514/6.2010-9320.
  * [73] K. E. Lewis and P. D. Collopy, “The Role of Engineering Design in Large-Scale Complex Systems,” in _12th AIAA Aviation Technology, Integration, and Operations (ATIO) Conference and 14th AIAA/ISSM_ , 2012.
  * [74] A.-M. R. McGowan, S. Daly, W. Baker, P. Papalambros, and C. Seifert, “A Socio-Technical Perspective on Interdisciplinary Interactions During the Development of Complex Engineered Systems,” _Procedia Computer Science_ , vol. 16, pp. 1142–1151, 2013.
  * [75] P. S. Adler, B. Goldoftas, and D. I. Levine, “Flexibility versus Efficiency? A Case Study of Model Changeovers in the Toyota Production System,” _Organization Science_ , vol. 10, no. 1, pp. 43–68, 1999.
  * [76] S. Spear and H. K. Bowen, “Decoding the DNA of the Toyota Production System,” _Harvard Business Review_ , vol. 77, pp. 97–97, Dec. 1999.
  * [77] B. Gokpinar, W. J. Hopp, and S. M. R. Iravani, “The Impact of Misalignment of Organizational Structure and Product Architecture on Quality in Complex Product Development,” _Management Science_ , vol. 56, no. 3, pp. 468–484, 2010.
  * [78] R. Katz and T. J. Allen, “Project performance and the locus of influence in the R&D matrix,” _Academy of Management journal_ , vol. 28, no. 1, pp. 67–87, 1985.
  * [79] M. Natter, A. Mild, M. Feurstein, G. Dorffner, and A. Taudes, “The effect of incentive schemes and organizational arrangements on the new product development process,” _Management Science_ , vol. 47, no. 8, pp. 1029–1045, 2001.
  * [80] J. P. Eggers and J. H. Suh, “Experience and behavior: How negative feedback in new versus experienced domains affects firm action and subsequent performance,” _Academy of Management Journal_ , vol. 62, no. 2, pp. 309–334, Apr. 2019.
  * [81] H. Li, J. B. Bingham, and E. E. Umphress, “Fairness from the top: Perceived procedural justice and collaborative problem solving in new product development,” _Organization Science_ , vol. 18, no. 2, pp. 200–216, 2007.
  * [82] W. R. Scott and G. F. Davis, _Organizations and Organizing: Rational, Natural, and Open System Perspectives_ , 1st ed. Upper Saddle River, N.J.: Pearson Prentice Hall, 2007, no. Upper Saddle River, N.J. :, rev. of: Organizations / W. Richard Scott. 5th ed. 2003.
  * [83] D. Gerwin and N. J. Barrowman, “An evaluation of research on integrated product development,” _Management Science_ , vol. 48, no. 7, pp. 938–953, 2002.
  * [84] N. Melville, K. Kraemer, and V. Gurbaxani, “Information technology and organizational performance: An integrative model of IT business value,” _MIS quarterly_ , vol. 28, no. 2, pp. 283–322, 2004.
  * [85] K. Atuahene-Gima and F. Evangelista, “Cross-functional influence in new product development: An exploratory study of marketing and R… D perspectives,” _Management science_ , vol. 46, no. 10, pp. 1269–1284, 2000.
  * [86] R. Kapoor and R. Adner, “What Firms Make vs. What They Know: How Firms’ Production and Knowledge Boundaries Affect Competitive Advantage in the Face of Technological Change,” _Organization Science_ , vol. 23, no. 5, pp. 1227–1248, Aug. 2011, doi: 10.1287/orsc.1110.0686.
  * [87] G. S. Saffold, “Culture Traits, Strength, and Organizational Performance: Moving Beyond “Strong” Culture,” _Academy of Management Review_ , vol. 13, no. 4, pp. 546–558, Oct. 1988, doi: 10.5465/amr.1988.4307418.
  * [88] C. L. K. Johnson and M. Smith, _Kelly: More than My Share of It All_. Smithsonian Institution, 1985.
  * [89] B. J. Zirger and M. A. Maidique, “A model of new product development: An empirical test,” _Management science_ , vol. 36, no. 7, pp. 867–883, 1990\.
  * [90] M. A. Schilling, “Technological Lockout: An Integrative Model of the Economic and Strategic Factors Driving Technology Success and Failure,” _Academy of Management Review_ , vol. 23, no. 2, pp. 267–284, Apr. 1998, doi: 10.5465/amr.1998.533226.
  * [91] M. A. Schilling and C. W. L. Hill, “Managing the new product development process: Strategic imperatives,” _Academy of Management Perspectives_ , vol. 12, no. 3, pp. 67–81, Aug. 1998, doi: 10.5465/ame.1998.1109051.
  * [92] L. Mulotte, “Do Experience Effects Vary Across Governance Modes? Evidence from New Product Introduction in the Global Aircraft Industry, 1948–2000,” _Organization Science_ , vol. 25, no. 3, pp. 757–775, Sep. 2013, doi: 10.1287/orsc.2013.0862.
  * [93] J. Austin-Breneman, “Observations of designer behaviors in complex system design,” Ph.D. dissertation, Massachusetts Institute of Technology, 2011.
  * [94] C. Eckert, O. Isaksson, and C. Earl, “Design margins: A hidden issue in industry,” _Design Science_ , vol. 5, pp. e9–e9, 2019.
  * [95] C.-C. Carbon, “Psychology of Design,” _Design Science_ , vol. 5, pp. e26–e26, 2019.
  * [96] S. Irnan and M. Pedram, “Multi-level Network Optimization For Low Power,” in _IEEE/ACM International Conference on Computer-Aided Design_ , 1994, pp. 372–377.
  * [97] J. M. Carlson and J. Doyle, “Highly optimized tolerance: A mechanism for power laws in designed systems,” _Physical Review E_ , vol. 60, no. 2, pp. 1412–1427, Aug. 1999.
  * [98] M. L. Newman, J. W. Pennebaker, D. S. Berry, and J. M. Richards, “Lying Words: Predicting Deception from Linguistic Styles,” _Personality and Social Psychology Bulletin_ , vol. 29, no. 5, pp. 665–675, May 2003, doi: 10.1177/0146167203029005010.
  * [99] D. Braha and Y. Bar-Yam, “The Statistical Mechanics of Complex Product Development: Empirical and Analytical Results,” _Management Science_ , vol. 53, no. 7, pp. 1127–1145, 2007.
  * [100] M. E. Sosa, S. D. Eppinger, and C. M. Rowles, “Identifying Modular and Integrative Systems and Their Impact on Design Team Interactions,” _Journal of Mechanical Design_ , vol. 125, no. 2, pp. 240–240, 2003.
  * [101] ——, “A Network Approach to Define Modularity of Components in Complex Products,” _Journal of Mechanical Design_ , vol. 129, no. 11, pp. 1118–1129, 2007.
  * [102] K. Sinha, E. S. Suh, and O. de Weck, “Integrative Complexity: An Alternative Measure for System Modularity,” _Journal of Mechanical Design_ , vol. 140, no. 5, pp. 51 101–51 111, Mar. 2018, 10.1115/1.4039119.
  * [103] A. Lungeanu and N. S. Contractor, “The effects of diversity and network ties on innovations: The emergence of a new scientific field,” _American Behavioral Scientist_ , vol. 59, no. 5, pp. 548–564, 2015.
  * [104] J. Austin-Breneman, B. Y. Yu, and M. C. Yang, “Biased Information Passing Between Subsystems Over Time in Complex System Design,” _Journal of Mechanical Design_ , vol. 138, no. 1, pp. 011 101–011 101, 2015.
  * [105] G. V. Bhatia, H. Kannan, and C. L. Bloebaum, “A Game Theory approach to Bargaining over Attributes of Complex Systems in the context of Value-Driven Design,” in _54th AIAA Aerospace Sciences Meeting_. Reston, Virginia: American Institute of Aeronautics and Astronautics, Jan. 2016.
  * [106] J. Rowley, “Conducting research interviews,” _Management Research Review_ , vol. 35, no. 3/4, pp. 260–271, Jan. 2012.
  * [107] S. Ybema, D. Yanow, H. Wels, and F. H. Kamsteeg, _Organizational Ethnography: Studying the Complexity of Everyday Life_. SAGE, Aug. 2009.
  * [108] M. Gibbert and W. Ruigrok, “The “What” and “How” of Case Study Rigor: Three Strategies Based on Published Work,” _Organizational Research Methods_ , vol. 13, no. 4, pp. 710–737, Oct. 2010.
  * [109] J. M. Corbin and A. Strauss, _Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory_. SAGE, 2008.
  * [110] C. Perrow, “Normal accident at three Mile Island,” _Society_ , vol. 18, no. 5, pp. 17–26, 1981.
  * [111] D. Vaughan, _The Challenger Launch Decision: Risky Technology, Culture, and Deviance at NASA_. University of Chicago Press, 1997.
  * [112] A.-M. R. McGowan, “Interdisciplinary Interactions During R & D and Early Design of Large Engineered Systems,” Dissertation, University of Michigan, Ann Arbor, MI, 2014.
  * [113] Z. Szajnfarber and E. Gralla, “Qualitative methods for engineering systems: Why we need them and how to use them,” _Systems Engineering_ , vol. 20, no. 6, pp. 497–511, 2017.
  * [114] H. Schuman, “Sense and Nonsense about Surveys,” _Contexts_ , vol. 1, no. 2, pp. 40–47, May 2002, doi: 10.1525/ctx.2002.1.2.40.
  * [115] V. J. Duriau, R. K. Reger, and M. D. Pfarrer, “A Content Analysis of the Content Analysis Literature in Organization Studies: Research Themes, Data Sources, and Methodological Refinements,” _Organizational Research Methods_ , vol. 10, no. 1, pp. 5–34, Jan. 2007.
  * [116] M. K. Ahuja and K. M. Carley, “Network Structure in Virtual Organizations,” _Organization Science_ , vol. 10, no. 6, pp. 741–757, Dec. 1999, doi: 10.1287/orsc.10.6.741.
  * [117] M. Newman, _Networks_. Oxford university press, 2018.
  * [118] C. D’Ignazio and L. F. Klein, _Data Feminism_. MIT Press, 2020.
  * [119] C. M. Macal, “Everything you need to know about agent-based modelling and simulation,” _Journal of Simulation_ , vol. 10, no. 2, pp. 144–156, May 2016, doi: 10.1057/jos.2016.7.
  * [120] J. Sacks, W. J. Welch, T. J. Mitchell, and H. P. Wynn, “Design and Analysis of Computer Experiments,” _Statist. Sci._ , vol. 4, no. 4, pp. 409–423, 1989.
  * [121] S. M. Sanchez and T. W. Lucas, “Exploring the world of agent-based simulations: Simple models, complex analyses,” in _Proceedings of the Winter Simulation Conference_ , vol. 1, 2002, pp. 116–126 vol.1.
  * [122] J.-S. Lee, T. Filatova, A. Ligmann-Zielinska, B. Hassani-Mahmooei, F. Stonedahl, I. Lorscheid, A. Voinov, G. Polhill, Z. Sun, and D. Parker, “The complexities of agent-based modeling output analysis,” _JASSS: the journal of artificial societies and social simulation_ , vol. 18, no. 4, pp. 1–4, 2015.
  * [123] Y. Jiang and J. C. Jiang, “Diffusion in Social Networks: A Multiagent Perspective,” _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , vol. 45, no. 2, pp. 198–213, Feb. 2015.
  * [124] A.-L. Barabási and R. Albert, “Emergence of Scaling in Random Networks,” _Science_ , vol. 286, no. 5439, pp. 509–512, 1999.
  * [125] R. Albert and A.-L. Barabási, “Statistical mechanics of complex networks,” _Reviews of Modern Physics_ , vol. 74, no. 1, pp. 47–97, Jan. 2002.
  * [126] M. E. J. Newman, “The structure and function of complex networks,” _SIAM review_ , vol. 45, no. 2, pp. 167–256, 2003.
  * [127] D. Braha and Y. Bar-Yam, “Topology of large-scale engineering problem-solving networks,” _Physical Review E_ , vol. 69, no. 1, pp. 16 113–16 113, Jan. 2004.
  * [128] D. D. S. Price, “A general theory of bibliometric and other cumulative advantage processes,” _Journal of the American Society for Information Science_ , vol. 27, no. 5, pp. 292–306, Sep. 1976, doi: 10.1002/asi.4630270505.
  * [129] P. Holme and B. J. Kim, “Growing scale-free networks with tunable clustering,” _Physical review E_ , vol. 65, no. 2, pp. 26 107–26 107, 2002\.
  * [130] H. Guclu and M. Yuksel, “Scale-free overlay topologies with hard cutoffs for unstructured peer-to-peer networks,” in _Distributed Computing Systems, 2007. ICDCS’07. 27th International Conference On_. IEEE, 2007, pp. 32–32.
  * [131] D. R. Kumari, H. Guclu, and M. Yuksel, “Ad-hoc limited scale-free models for unstructured peer-to-peer networks,” _Peer-to-Peer Networking and Applications_ , vol. 4, no. 2, pp. 92–105, 2011.
  * [132] V. Grimm, U. Berger, F. Bastiansen, S. Eliassen, V. Ginot, J. Giske, J. Goss-Custard, T. Grand, S. K. Heinz, and G. Huse, “A standard protocol for describing individual-based and agent-based models,” _Ecological modelling_ , vol. 198, no. 1-2, pp. 115–126, 2006.
  * [133] V. Grimm, U. Berger, D. L. DeAngelis, J. G. Polhill, J. Giske, and S. F. Railsback, “The ODD protocol: A review and first update,” _Ecological Modelling_ , vol. 221, no. 23, pp. 2760–2768, 2010.
  * [134] E. Bonabeau, “Agent-based modeling: Methods and techniques for simulating human systems.” _Proceedings of the National Academy of Sciences of the United States of America_ , vol. 99 Suppl 3, no. 90003, pp. 7280–7, May 2002.
  * [135] C. Macal and M. North, “Tutorial on agent-based modelling and simulation,” _J Simul_ , vol. 4, no. 3, pp. 151–162, 2010.
  * [136] J. H. Panchal, “Agent-based modeling of mass-collaborative product development processes,” _Journal of Computing and Information Science in Engineering_ , vol. 9, no. 3, pp. 31 007–31 007, 2009.
  * [137] H. R. Darabi and M. Mansouri, “Governing Competition and Collaboration in Network Industries Using Agent-Based Modeling: A Case-Study of US Air Transportation Network,” _IEEE Systems Journal_ , vol. 11, no. 4, pp. 2118–2127, 2017.
  * [138] J. Soyez, G. Morvan, R. Merzouki, and D. Dupont, “Multilevel Agent-Based Modeling of System of Systems,” _IEEE Systems Journal_ , vol. 11, no. 4, pp. 2084–2095, 2017.
  * [139] K. M. Carley and D. M. Svoboda, “Modeling organizational adaptation as a simulated annealing process,” _Sociological Methods and Research_ , vol. 25, no. 1, pp. 138–168, Aug. 1996, doi: 10.1177/0049124196025001005.
  * [140] K. M. Carley and Z. Lin, “A theoretical study of organizational performance under information distortion,” _Management science_ , vol. 43, no. 7, pp. 976–997, 1997.
  * [141] F. Anjos and R. Reagans, “Commitment, Learning, and Alliance Performance: A Formal Analysis Using an Agent-Based Network Formation Model,” _Journal of Mathematical Sociology_ , vol. 37, no. 1, pp. 1–23, Jan. 2013.
  * [142] B. Jamshidnezhad and K. M. Carley, “Agent-based modelling of quality management effects on organizational productivity,” _Journal of Simulation_ , vol. 9, no. 1, pp. 73–82, Feb. 2015, doi: 10.1057/jos.2014.26.
  * [143] P. Y. Papalambros and D. J. Wilde, _Principles of Optimal Design: Modeling and Computation_. Cambridge university press, 2017.
  * [144] C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. F. del Río, M. Wiebe, P. Peterson, P. Gérard-Marchant, K. Sheppard, T. Reddy, W. Weckesser, H. Abbasi, C. Gohlke, and T. E. Oliphant, “Array programming with NumPy,” _Nature_ , vol. 585, no. 7825, pp. 357–362, Sep. 2020.
  * [145] P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, İ. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, and P. van Mulbregt, “SciPy 1.0: Fundamental algorithms for scientific computing in Python,” _Nature Methods_ , vol. 17, no. 3, pp. 261–272, Mar. 2020.
  * [146] A. Hagberg, P. Swart, and D. S Chult, “Exploring network structure, dynamics, and function using networkx,” Los Alamos National Lab. (LANL), Los Alamos, NM (United States), Tech. Rep. LA-UR-08-05495; LA-UR-08-5495, Jan. 2008.
  * [147] S. Surjanovic and D. Bingham, “Virtual library of simulation experiments: Test functions and datasets,” 2013.
  * [148] A. Cabigiosu and A. Camuffo, “Beyond the ”Mirroring” Hypothesis: Product Modularity and Interorganizational Relations in the Air Conditioning Industry,” _Organization Science_ , vol. 23, no. 3, pp. 686–703, 2012.
  * [149] A. MacCormack, C. Baldwin, and J. Rusnak, “Exploring the duality between product and organizational architectures: A test of the “mirroring” hypothesis,” _Research Policy_ , vol. 41, no. 8, pp. 1309–1324, Oct. 2012.
  * [150] L. J. Colfer and C. Y. Baldwin, “The mirroring hypothesis: Theory, evidence, and exceptions,” _Industrial and Corporate Change_ , vol. 25, no. 5, pp. 709–738, Oct. 2016, 10.1093/icc/dtw027.
  * [151] J. C. Anderson and M. Rungtusanatham, “A theory of quality management underlying the Deming Management Method,” _Academy of Management Review_ , vol. 19, no. 3, pp. 472–509, Jul. 1994.
  * [152] J. Meluso, J. Austin-Breneman, and J. Uribe, “Estimate Uncertainty: Miscommunication About Definitions of Engineering Terminology,” _Journal of Mechanical Design_ , vol. 142, no. 7, Feb. 2020.
  * [153] G. ten Broeke, G. van Voorn, and A. Ligtenberg, “Which Sensitivity Analysis Method Should I Use for My Agent-Based Model?” _Journal of Artificial Societies and Social Simulation_ , vol. 19, no. 1, p. 5, 2016.
  * [154] G. James, D. Witten, T. Hastie, and R. Tibshirani, Eds., _An Introduction to Statistical Learning: With Applications in R_ , ser. Springer Texts in Statistics. New York: Springer, 2013, no. 103, includes index.
  * [155] L. Breiman, “Random Forests,” _Machine Learning_ , vol. 45, no. 1, pp. 5–32, Oct. 2001.
  * [156] K. M. Carley, “Validating computational models,” _Paper available at http://www. casos.cs.cmu.edu/ …_ , vol. 0793, no. September, 1996.
  * [157] U. Wilensky and W. Rand, _An Introduction to Agent-Based Modeling: Modeling Natural, Social, and Engineered Complex Systems with NetLogo_. MIT Press, 2015.
  * [158] S. Chou, “Exploring Designer Context in Engineering Design: The Relationship Between Self, Environment, and Design Methods,” Dissertation, University of Michigan, Ann Arbor, MI, 2020.
  * [159] G. Wen, X. Yu, W. Yu, and J. Lü, “Coordination and Control of Complex Network Systems With Switching Topologies: A Survey,” _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , pp. 1–16, 2020.