# Seamless Digital Engineering: A Grand Challenge Driven by Needs

James S. Wheaton  Department of Systems Engineering
Colorado State University
Fort Collins, Colorado, USA  Daniel R. Herber  Department of Systems
Engineering
Colorado State University
Fort Collins, Colorado, USA

###### Abstract

Digital Engineering currently relies on costly and often bespoke integration
of disparate software products to assemble the authoritative source of truth
of the system-of-interest. Tools not originally designed to work together
become an acknowledged system-of-systems, with their own separate feature
roadmaps, deprecation, and support timelines. The resulting brittleness and
conglomeration of disparate interfaces in the Digital Engineering Ecosystem of
an organization drains resources and impairs efficiency and efficacy. If
Model-Based Systems Engineering were applied to this problem, a complete
system architecture model would be defined, and a purpose-built computing
system-of-systems would be constructed to satisfy stakeholder needs. We have
decades of research in computer science, cybersecurity, software and systems
engineering, and human-computer interaction from which to draw that informs
the design of a Seamless Digital Engineering tooling system, but it would
require starting from a clean slate while carefully adopting existing
standards. In this paper, this problem space and solution space are
characterized, defining and identifying Seamless Digital Engineering as a
grand challenge in Digital Engineering research.

##  1 Introduction

Engineers and other staff and contractors primarily use tools (see Table 1) to
produce engineered systems such as integrated circuits, spacecraft, and
software that meet stakeholder needs. Increasingly, those tools are computer-
based and therefore dependent on the wide range of commercially- and freely-
available computer software and hardware. Digital Engineering (DE), in
particular, is an effort to complete this transition and rely solely on a
digitized Authoritative Source of Truth (ASoT) of the system-of-interest (SoI)
for engineering and associated activities throughout the system development
life cycle. However, due to the heterogeneity and other complexity factors of
those computer-aided engineering (CAE) and ancillary tools, the achievement of
DE is frustrated by integration and usability challenges.

Table 1: Definitions of related engineering terms (emphasis added). Term |  Definition
---|---
Tool |  “The external employment of an unattached or manipulable attached environmental object to alter more efficiently the form, position, or condition of another object, another organism, or the user itself, when the user holds and directly manipulates the tool during or prior to use and is responsible for the proper and effective orientation of the tool.” Shumaker et al. (2011)
Technology |  “Artifacts made through a systematic application of knowledge and used to reach practical goals” Salomon (1984) “in a specifiable and reproducible way” Skolnikoff (1993)
Engineering |  “The creative application of scientific principles to design or develop structures, machines, apparatus, or manufacturing processes, or works utilizing them singly or in combination; or to construct or operate the same with full cognizance of their design; or to forecast their behavior under specific operating conditions; all as respects an intended function, economics of operation and safety to life and property.” Engineers’ Council Professional for Development (1947)
Systems Engineering (SE) |  “Interdisciplinary approach governing the total technical and managerial effort required to transform a set of customer needs, expectations, and constraints into a solution and to support that solution throughout its life.” ISO (2010)
|  “A transdisciplinary and integrative approach to enable the successful
realization, use, and retirement of engineered systems, using systems
principles and concepts, and scientific, technological, and management
methods.” INCOSE Fellows (2019)
Model-Based Systems Engineering (MBSE) |  “The formalized application of modeling to support system requirements, design, analysis, verification and validation activities beginning in the conceptual design phase and continuing throughout development and later life cycle phases.” INCOSE Technical Operations (2007)
Digital Engineering (DE) |  “An enterprise approach for incorporating data-driven infrastructure, methods and tools, innovation, and workforce transformation centered on an authoritative source of truth (ASoT) for engineering data.” Office of the Deputy Assistant Secretary of Defense for Systems Engineering (2018)
|  “DE is a transformation of the implementation of all engineering (and
associated disciplines) across an enterprise to take full advantage of the
digital integration of engineering work, data, knowledge, and wisdom across
that enterprise.” Noguchi et al. (2020)
Authoritative Source of Truth (ASoT) |  “The authoritative source of truth captures the current state and the history of the technical baseline. It serves as the central reference point for models and data across the lifecycle. The authoritative source of truth will provide traceability as the system of interest evolves, capturing historical knowledge, and connecting authoritative versions of the models and data.” Office of the Deputy Assistant Secretary of Defense for Systems Engineering (2018)
Digital Engineering Ecosystem (DEE) |  The “enterprises’ interconnected digital environments, stakeholder-networks, and semantic data that allows the exchange of digital artifacts from an authoritative source of truth to serve the stakeholder communities’ interests.” Office of the Deputy Assistant Secretary of Defense for Systems Engineering (2017)
Digital Engineering System (DES) |  A theoretical engineered system-of-systems that provides the networked computing resources and human-computer interfaces for an enterprise to accomplish all digital engineering activities for successful realization of engineered systems.

Many deficiencies in computing systems are tolerated because workarounds are
available and then implemented, further entrenching their intractability. The
intractability of DE is due to the high number of computing components and
interfaces, the decentralized and inconsistent evolution of those components,
and in particular, a profound lack of a cohesive system architecture model
that systems engineers and other stakeholders would use to inspect, analyze,
validate, verify, and modify the complex digital system. A useful way of
thinking about these digital tooling deficiencies is with the concept of the
_reverse salient_ , which may be defined as “a set of critical problems”
Hughes (1993) or components that “fail to deliver the necessary level of
technological performance thereby inhibiting the performance delivery of the
system as a whole” Dedehayir and Mäkineif (2008), where the SoI is the Digital
Engineering Ecosystem (DEE) that provides a set of affordances that depend on
the underlying computing hardware and software components. The computing
system has not been designed to meet the engineers’ needs, but rather software
components known as applications have been designed for general use. As
systems engineers, we do not tolerate delivering defective systems that
frustrate users with unpredictable behavior and random errors, so if we were
contracted to develop our own Digital Engineering System (DES), we would
naturally apply MBSE with full requirements traceability, and track the
Quality Attributes to guarantee success.

Prior publicly-funded research initiatives have had similar goals. In the
recent past, DARPA has funded such initiatives as CRASH (Clean-Slate Design of
Resilient, Adaptive, Secure Hosts) DARPA (2010a), META-II DARPA (2010b) and
CRAFT (Circuit Realization At Faster Timescales) DARPA (2015), recognizing
that the systemic vulnerabilities and deficiencies of current computer systems
and CAE tools require a clean-slate design to overcome. Norman (1999) also
recognized that “the current paradigm is so thoroughly established that the
only way to change is to start over again”. Given the advancements in Model-
Based Systems Engineering (MBSE), computer science, integrated circuit design,
and manufacturing over the past four decades, we have the opportunity to
envision a fresh DES that meets our needs for safety, security,
comprehensiveness, efficiency, and efficacy. The goal then “is to move from
the current situation of complexity and frustration to one where technology
serves human needs invisibly, unobtrusively: the human-centered, customer-
centered way” Norman (1999), a goal that ought to resonate with systems
engineers and computer users alike.

The primary contribution of this paper is to clarify the definition of
Seamless Digital Engineering in contrast with how DE is currently being
implemented and identify it as a grand challenge in DE research. In Section 2,
historical and related research is summarized in relation to seamless DE.
Section 3 characterizes the DE tooling problem space and current integration
patterns for achieving DE outcomes. Section 4 defines seamless DE so it can be
subject to further SE and DE research. Finally, Section 5 discusses how the
achievement of seamless DE would constitute a grand challenge in DE research
typically beyond the capability of any one organization. Some research
conclusions and a summary of future work follow.

##  2 Related Work

The theoretical and practical foundations for DE have been developing for at
least a half-century, dating back to ARPA-commissioned research by MIT on LISP
machines Lombardi (1964); Bawden et al. (1977); Walker et al. (1987) which
aimed to “help designers get from prototype to product faster” and whose “key
[was] an open architecture and highly-integrated development tools”. Along
with Smalltalk computing systems Ingalls (1981) that represented a similar
trajectory in integrated tool-based computing environments, LISP machines are
virtually nonexistent today, and we risk forgetting the many lessons and
advancements made in those early years of computing. Later research has
focused on ex post facto integration of computer-based tools.

Boehm et al. (1984) described a UNIX-based integrated tooling system that
assembled basic components provided by the OS in combination with custom
software fit-to-task, not unlike what we practice today. Thomas and Nejmeh
(1992) provides an essential formalism of tool integration that decomposes the
concept into four: 1) process integration, 2) data integration, 3)
presentation integration, and 4) control integration. That paper describes
system properties, such as consistency, coherence, nonredundancy, and
synchronization, that are still relevant today, and in particular, “tools are
well integrated when they have a common view of data”. Although these efforts
focus on computer-aided software engineering (CASE), the principles and
lessons are generalizable to DE.

Seamless model-driven systems engineering was first presented in Refs. Broy
(2009); Broy et al. (2010) and later in Broy (2020). Broy et al. (2010)
identifies three ingredients to seamless integration, i.e., “deep, coherent,
and comprehensive integration of models and tools”: 1) a modeling theory that
serves as the basis of the ASoT’s semantic domain, 2) a complete architectural
model that describes product and process, and 3) “a manner to build tools that
conform to the modeling theory and allow the authoring of the product model”,
noting that the scientific basis is available but political barriers thus far
prevent its realization. Broy et al. (2010) contributes critical theories and
observations informed by automotive and aeronautical industry practice, such
as a conceptual model for an integrated model engineering environment based on
horizontal and vertical tooling aspects and the useful notion that currently
available tools impose the current situation rather than the ideal case of
formally-defined modeling theory dictating tool support. The rationale for
this vision may be familiar to DE practitioners, such as: lack of model
integration at both the conceptual and tooling level; transitions between
artifacts are ad-hoc, unclear, and error-prone; redundancy of models that lose
product information, leaving some implicit in engineers’ minds; the high costs
of internally developing tools and the unsatisfactory tool support for domain-
specific modeling in the organization’s core business; and that formal
verification is essential to producing complex systems that are correct-by-
design but that today we only benefit from “islands of success”. Broy et al.
(2010) uses overlapping views of seamless: seamless model-based development,
seamless systems engineering, and seamless system development, and offers the
following definition:

> “Seamless model-based development promises to lift software development to
> higher levels of abstraction by providing integrated chains of models
> covering all phases from requirements to system design and verification. In
> seamless model-based development, modeling is not just an implementation
> method but also a paradigm that provides support throughout the entire
> development and maintenance lifecycle.”

This definition is software development-focused and lacks precision with
respect to ‘higher levels of abstraction’; while it covers the complete system
development lifecycle, it refers to “integrated chains of models” that
somewhat contrasts with DE’s ASoT; lastly, it highlights this approach as a
paradigm as opposed to solely an implementation method. Broy (2020) defines
seamless as an approach to software and system modeling where “all models and
all development steps are in a tight relationship with a clear understanding
how they benefit”, but this definition appears to lack the precision needed to
determine if the modeling approach is in fact seamless. Clarity of what
qualifies as a “tight relationship” among models and steps would improve our
understanding of seamless model-based development. Finally, the first
published use of the term “seamless digital engineering” was by Perzylo et al.
(2020), which describes knowledge integration as a key component to DE, but
that nevertheless leaves the definition implicit and subject to the reader’s
interpretation of seamless.

##  3 Characterizing the Problem Space

As Norman (1999) succinctly notes, “The personal computer is perhaps the most
frustrating technology ever. The computer should be thought of as
infrastructure”. Perhaps we do not think of computers as infrastructure but
rather as suites of products. This mental model may be clouding our vision of
future computing technology that suits our needs as engineers and
communicators. Where much research after LISP machines has assumed the von
Neumann computer architecture and widely-deployed operating systems as
inviolable, a clean-slate approach favors the systems engineering practice of
not making point-design decisions too early in the system development
lifecycle. Let us first identify and characterize the problem space and then
move on to proposing a solution space (see Fig. 1).

Figure 1: Generic Problem-Solution Space (SysML v2 depiction).

###  3.1 Stakeholders and Goals

Table 2: Stakeholders of a DES. Engineer |  Project/Program Manager |  Accountant
---|---|---
Acquirer |  Contractor |  Auditor
Manufacturer |  Technician |  Designer
Supplier |  Regulator |  Technical Writer

Since DE encompasses the entire system development lifecycle, the stakeholders
of a DES are varied and many. We provide here a list (see Table 2) of
potential stakeholders for a DES as a starting point, noting that some
stakeholders listed, such as Engineer, may be further specialized depending on
the role.

The original DE goals Office of the Deputy Assistant Secretary of Defense for
Systems Engineering (2018) are the visionary statements used to guide any DE
implementation effort. The goals provide enough detail for us to link
Stakeholder Expectations to them in a hierarchical, traceable relationship
structure. Enumerated below, the DE goals are among the first elements to add
to our system architecture model.

  1. G1

Formalize the development, integration, and use of models to inform enterprise
and program decision-making.

  2. G2

Provide an enduring, authoritative source of truth (ASoT).

  3. G3

Incorporate technological innovation to improve the engineering practice.

  4. G4

Establish a supporting infrastructure and environments to perform activities,
collaborate, and communicate across stakeholders.

  5. G5

Transform the culture and workforce to adopt and support DE across the
lifecycle.

While some aspects of the above DE goals depend on the organization’s
execution of the DE processes, e.g., “transform the culture and workforce”,
those goals may nevertheless be supported by a comprehensive DES that
incorporates documentation and training resources.

###  3.2 DE is an Adaptation of Existing Tools and Available Computing Systems

Decades of competitive innovation in CAE tools have led to thousands of
different software products in various functional areas throughout the product
development life cycle. Startups entering high-tech fields such as integrated
circuit design must first acquire a set of expensive software tools to begin
their design work, with no guarantee that the various tools will be able to
maintain a coherent system model. DE is an emerging paradigm where an ASoT of
the SoI is maintained for the entire product development lifecycle, supporting
Digital Thread and Digital Twin AIAA Digital Engineering Integration Committee
(2020, 2023), and promising to streamline the engineering of complex systems.
However, it prescribes no particular way of accomplishing this streamlined DE
environment.

The burden of maintaining this organization-wide system of systems drains
organization resources away from engineering systems: requiring data engineers
and information technology administrators, more computing equipment, and the
operational cost of vendor contracts that provide (and may eventually
terminate) licenses and technical support. Moreover, staff is frequently
switching between tools to accomplish their tasks, requiring productivity-
draining context switches and training in each software product while being
subject to vendors’ upgrades that often modify or deprecate functions that
have become integral to the organization’s workflow.

Karban et al. (2016); Delp (2019) outline the OpenCAE tooling system, which
amounts to an instantiation of the DE concept. Over 50 software products and
languages are identifiable from the presentation in 2019. The presenter notes
that the NASA Jet Propulsion Laboratory (JPL) uses “14 server set-ups – over
200 servers” in cloud compute infrastructure and software-as-a-service to
achieve their DEE. An acknowledged system-of-systems (ASoS) of this size and
heterogeneity is likely to exhibit emergent fragility as the various software
products are updated out-of-sync with one another. The academic and industry
communities would benefit from more published research that studies the
incongruities of computing system-of-systems of this kind. While the details
of these deployed DEE are often proprietary, here we can acknowledge their
complexity and the effort required to integrate the various components to
achieve DE’s goals.

###  3.3 Common DE Integration Patterns

Software components can be made to work together typically using what is
referred to as a ‘glue’ or shim component. Even software that exposes the
common UNIX-style plain text interface must nevertheless be integrated using a
shim or glue component that parses the text, interprets any data, assembles a
model, and then perform model transformations while striving to preserve model
invariants though typically without any formal guarantees. The more humans are
involved in these integration processes, the more human error is introduced,
resulting in SoI defects that may compromise cost, schedule, and risk budgets.

####  3.3.1 Import/Export Pattern

The most basic pattern of DE integration is import/export, where the
engineering model is exported to a standardized or proprietary file format and
then imported into another tool (see Fig. 2). This pattern relies on extra
functionality in the tools, clear specification of the file format, and an
accurate implementation thereof. Nevertheless, the pattern suffers when model
information is lost during export and/or import due to incomplete
implementations and incompatible meta-models.

Figure 2: Import/Export method of DE tool integration (SysML v2 depiction).

####  3.3.2 Custom Shim Pattern

When DE tools produce engineering artifacts with known schemas that are stored
in known formats, custom shim software may be written to automate extract,
transformation, load (ETL) operations (see Fig. 3). Examples of such artifacts
include XML-based formats such as XMI and ReqIF, CSV or Excel spreadsheets,
standard SQL databases, and JSON data that may loaded into a NoSQL database.
The common drawback of this method, besides the additional software
engineering effort, is that file formats and database schemas frequently
change and often without due notice, resulting in brittle shims that must be
urgently corrected to maintain the DE ASoS. Custom shims are also written when
integrating software tools that expose RESTful or RPC-style APIs, with similar
drawbacks when API specifications change.

Figure 3: Custom Shim method of DE tool integration (SysML v2 depiction).

####  3.3.3 API Pattern

SysML v2 champions an API-based method of integration by exposing a RESTful
interface (see Fig. 4) to the SysML model of the SoI Bajaj et al. (2022).
Other DE tools are meant to use this RESTful API to query attributes and
properties of the SoI model such that they are no longer duplicated in the
separate tools, thereby reducing opportunities for emergent model definition
divergence.

Figure 4: SysML v2 API method of DE tool integration (SysML v2 depiction).

###  3.4 DE As-Implemented Cannot be Elegant or Seamless

Before the commodification of computer hardware and software, computer-aided
tools were designed to-purpose (SAGE Enticknap and Schuster (1959) is offered
as an illustrative example). Although the cost of such systems was extremely
high due to custom development, they did not have as many computing standards
and conventions to support and thus avoided the complexity of supporting and
interfacing with non-essential or generalized technologies. Nowadays, it is
often assumed that computing systems must support the plethora of computer
technologies provisioned by Microsoft Windows or GNU/Linux to avoid
‘reinventing the wheel’. This decision, while pragmatic, can suffer from a
sunk-cost fallacy that limits the effectiveness of the deployed system and
results in kludge111Kludge is defined as “a badly assembled collection of
parts hastily assembled to serve some particular purpose (often used to refer
to computing systems or software that has been badly put together)” Princeton
University (2011)..

Existing DE and CAE tooling systems lack elegance. System elegance is a
Quality Attribute (QAt) defined as “a system that is robust in application,
fully meeting specified and adumbrated intent, is well-structured, and is
graceful in operation” Watson et al. (2020, 2019, 2014); Madni (2012). The
four characteristics of elegant systems are Griffin (2010):

  * •

System efficacy,

  * •

System efficiency,

  * •

System robustness, and

  * •

Minimizing unintended consequences.

DE as an ASoS lacks efficacy because the tools are nevertheless unaware of the
ASoT and liable to introduce defects. It lacks efficiency because the ASoS
must itself be developed, deployed, and supported as a fixed and operational
cost whose only opportunity for recoupment is the efficiency and quality gains
not otherwise available. Finally, its unintended consequences derive from the
independent governance and development lifecycles of the adopted software
components, whose emergent interactions as the SoS ages are impossible to
predict with precision.

CAE tools, in general, do not consider the complete development lifecycle
activities to be in-scope and only have plug-in and import/export support for
other languages and tools. This scope limitation and typical reliance on
proprietary data formats limit its efficacy and efficiency since the engineer
must context-switch and adapt to the recalcitrant tool rather than the tool
being adaptable to the engineer’s needs. Additionally, import/export
capabilities often have unintended consequences as even international
standards may have ambiguity and often explicitly leave areas of the standard
implementation-defined, so tools will have differing interpretations causing
unintended consequences.

Generally speaking, seamless is defined as “perfectly consistent and coherent”
and “not having or joined by a seam or seams” Princeton University (2011),
i.e., “the fissure or gap formed by the imperfect union of two bodies laid or
fastened together” Whitney and Smith (1906). In systems engineering terms, the
seam is the system architecture element that joins two or more interfaces and
may employ parts and behaviors of its own to satisfy its traced requirements.
So to be seamless, this joining element must not be necessary. This criterion
has only limited satisfaction today, where multiple different model-based
activities are possible from within the same tool (e.g., executable SysML and
instrumented GUIs for system behavior simulations and early prototypes).

###  3.5 Existing Computer Platforms are Nonviable

A driving model for this research is The Error Avalanche as described in
Claxton et al. (2005) and shown in Fig. 5: that not only will errors be
introduced into the delivered system during the system development life cycle
as a consequence of human error—even with the support of CAE tools—but the
computer hardware and software used to design that system may also compromise
the integrity of the system model and introduce subtle defects into the
system. The existence of mercurial cores that silently give the wrong
calculation Hochschild et al. (2021) illustrates this second point, a rather
alarming discovery following the early Pentium FDIV bug Halfhill (1995) and
remotely-exploitable Meltdown Lipp et al. (2020) and Spectre-class Kocher et
al. (2020) CPU microarchitecture flaws that were predicted in Porras and
Lindell (1995).

Figure 5: The Error Avalanche (adapted from Claxton et al. (2005)).

By now it should be well-known that endpoint security is the weakest link in
networked computer architecture and that the current approach to cybersecurity
is nonviable Chiricescu et al. (2013); Reubenstein et al. (2016); von Gernler
(2015); Snowden (2019); Easterly (2023). And so uncritical adoption of
existing computing technologies will eventually compromise enterprise security
and thus compromise the robustness and ‘minimizing unintended consequences’
properties of the DE tooling system. During a time when international
competition is driving the adoption of MBSE and DE, and corporate espionage
and data sabotage are serious risks in the presence of Advanced Persistent
Threats (APTs), a clean-slate DE design must specify the full stack of
computing hardware, firmware, and software to avoid making the same mistakes
that we normally retain for backward-compatibility.

##  4 Defining Seamless Digital Engineering

Prior descriptions of seamless DE or seamless MBE environments have left the
contextual definition of seamless implicit Broy et al. (2010); Perzylo et al.
(2020). In this paper, the following definition of Seamless DE is offered with
the additional criterion of end-to-end formal verification to guarantee
seamless, error-free operation:

> Seamless Digital Engineering (SDE) is a digital engineering tooling paradigm
> that guarantees model coherence and integrity by affording an elegant human-
> computer interface for systems modeling that is end-to-end formally verified
> down thru the computer hardware.

###  4.1 Seamless DE is a Refinement of DE

In order to avoid the unmitigated complexity, hidden costs, and risks of
existing computer systems, we must work to design a clean-slate DES that is
capable of meeting stakeholder needs. There is a great opportunity here to
avoid previous compromises and backward compatibility that have plagued
existing computer systems, which are now the reverse salient that holds back
the envisioned capability of DE and MBSE. Seamless digital engineering refines
the definitions of DE quoted in Table 1 by adding the seamless requirement for
achieving the elegance QAt as discussed above. We propose that this refinement
may guide future research and collaboration toward a standardized and
generational DES.

In light of the research summarized in Section 2, a critical property of a
successful DES must be that the ASoT or model integrity is guaranteed. A
failure of integrity is an unintended change of the model data or “the
condition in which data are identically maintained during any operation, such
as transfer, storage, and retrieval” Hapeman et al. (2001). Whether such a
failure is corrected or not, it will inevitably add cost and risk to the
system development project. The DES should be capable of detecting and
automatically correcting errors wherever possible, leaving humans to their
design activities rather than fixing and cleaning up after broken tools. The
promise of computers in this respect has not yet been fully realized.

All the DE goals Office of the Deputy Assistant Secretary of Defense for
Systems Engineering (2018) are addressable by an adequate system architecture.
A seamless DES would be designed with those goals defined in the system
architecture model to support their bidirectional traceability for
verification and validation. Although DE goal G5 regarding “transform the
culture” is beyond the scope of the system, an adequate architecture model of
the SDE system would include an operational viewpoint showcasing how common
operational processes (workflows) would be accomplished using the system. And
while DE metrics are now collated and processed with great effort, a capable
SDE system would provide toggle-able instrumentation for process and product
quality measurements with automated, configurable dashboards for monitoring DE
effectiveness across the organization or project.

The vision of computers as invisible infrastructure Kay (2011); Norman (1999)
still seems worlds away when compared to our daily experience. And so, with
respect to the presentation and process integration described in Thomas and
Nejmeh (1992), seamless integration must also encompass human-computer
interface (HCI) ergonomics. We may even strive for computing systems that are
delightful to use and help guide and focus the user on their activities and
tasks by incorporating HCI and human factors engineering research into the
design.

###  4.2 Seamless Integration Requirements

Tool integration problems are central to DE transformation efforts at
organizations, and they influence workforce attitudes toward change.
Stakeholders become familiar with their tools—including the quirks—and how
they fit together to satisfy their needs. So naturally, proposed changes to
those workflows are often met with skepticism and resistance. An important
goal of SDE is to smooth out those seams (defined in Section 3.4) so
stakeholders can focus on productive work rather than developing workarounds
that are coping mechanisms for deficient tooling.

Table 3: Transforming tool integration questions Thomas and Nejmeh (1992) into system requirements. Dimension |  Aspect |  Question |  Transformed Requirement(s)
---|---|---|---
Presentation Integration |  Appearance and behavior |  To what extent do two tools use similar screen appearance and interaction behavior? |  The SDE Tools shall conform to the [SDE HCI Standard].
|  Interaction paradigm |  To what extent do two tools use similar metaphors and mental models? |  Following HCI Customization by the Operator, the SDE System shall apply the Operator HCI Configuration uniformly.
Control Integration |  Provision |  To what extent are a tool’s services used by other tools in the environment? |  The SDE System shall provide common functionality as formally-verified System Libraries.
|  Use |  To what extent does a tool use the services provided by other tools in the environment? |  The SDE Tools shall reuse System Libraries for Tool-specific functions.
|  |  |  The SDE Tools shall provide Tool functions conforming to the [SDE Tool Interface Standard].
Process Integration |  Process step |  How well do relevant tools combine to support the performance of a process step? |  The SDE System shall provide the Workflow Definition Tool.
|  Event |  How well do relevant tools agree on the events required to support a process? |  The Workflow Definition Tool shall define the Process Event using [SDE Data Definition Standard].
|  Constraint |  How well do relevant tools cooperate to enforce a constraint? |  During Workflow Definition & Execution, the Workflow Definition Tool shall enforce the Process Constraint on the Workflow-Defined Tool.
Data Integration |  Interoperability |  How much work must be done for a tool to manipulate data produced by another? |  The SDE Tools shall define the applicable Data Schema conforming to the [SDE Data Definition Standard].
|  Nonredundancy |  How much data managed by a tool is duplicated in or can be derived from the data managed by the other? |  The SDE System shall provide the ACID-compliant System Database.
|  Data consistency |  How well do two tools cooperate to maintain the semantic constraints on the data they manipulate? |  The SDE System Database shall maintain semantic constraints of data where available.
|  Data exchange |  How much work must be done to make the nonpersistent data generated by one tool usable by the other? |  The SDE Tools shall write transactions to the SDE System Database for each change performed by the Operator.
|  Synchronization |  How well does a tool communicate changes it makes to the values of nonpersistent, common data? |  The SDE System Database shall provide the Transaction Log applicable to the SDE Tool.

The tool integration evaluation questions from Thomas and Nejmeh (1992) may be
transformed into system requirements as in Table 3. These requirements serve
to define the problem space (Fig. 1) with respect to SDE tool integration. One
aspect commonly discussed in the context of DEE is interoperability, and
integration patterns such as a custom shim (Fig. 3) are often developed to
transform data without having access to data schemas and definitions. The
transformed requirement with respect to interoperability states that SDE Tools
shall define the data schema used according to a standard such that other
tools may parse it directly and avoid rework and mistakes. Likewise, the
Control Integration dimension leads to the requirement that SDE Tools provide
what is essentially a model-generated Interface Control Document for other
tools to consume and use directly. This level of architecture openness and
model-based interface documentation would help resolve many tool integration
challenges.

As SysML v2 requirement definitions, the transformed requirements in Table 3
may then be reused as requirement usages in various levels of the system
architecture providing bidirectional traceability. SysML v2 requirements are
modeled as constraints so that the system either satisfies the requirement or
not. While the questions in Table 3 often ask “how well?”, in a clean-slate
architecture, the components can be designed to satisfy these measures as
requirements. The tool integration questions may also be modeled directly as
SysML v2 concerns that are similar to requirements but include one or more
stakeholders and allow the relevant requirement(s) to frame the concern.
Further SysML v2 meta-modeling could capture the dimensions and aspects in
Table 3, giving them ontological definitions and relations consistent with
systems engineering practice.

###  4.3 Seamless Integration Design Pattern

A seamless integration design pattern (see Fig. 6) uses system-wide standard
interfaces, data schemas, and object representations in a live DES to
accomplish the integration of SoI models to form the ASoT. By drastically
reducing the number of interfaces, tools are designed to speak the same
language. With self-documenting interfaces, tools may query other tools to
discover and perform the available functions. When the ASoT is queried by a
tool, a view is returned including the unique identifiers—or rather—handles of
live objects. In this integration design pattern, it is important to note that
data is not copied and divorced from its source that requires later merging
and reconciliation.

Figure 6: Seamless design pattern of DE tool integration (SysML v2 depiction).

###  4.4 Seamless DE Architecture Tenets

Seamless DE is based on a set of insights or architecture tenets that will
guide its detailed architecture definition. These tenets are derived from
observations of extant computing systems and existing research in computer
science, human-computer interaction, and cybersecurity. Figure 7 shows how
these architecture tenets may be modeled as SysML v2 concerns with their own
subjects, stakeholders, attributes, and constraints — to be framed by system
requirements in a reference architecture model. In summary, the architecture
tenets that enable SDE are: 1) Seamless Models, 2) Composable Data, 3) Live
Objects, 4) Seamless Workflows, and 5) Clean-slate Cybersecurity.

By packaging and presenting these insights together, they may be better
understood than if analyzed individually. In accordance with systems
engineering principles, capturing system architecture information as model
data that is then presented in a variety of stakeholder views is superior to
doing one or the other alone. Again, further SysML v2 meta-modeling may better
capture, connect, and present the SDE architecture tenets — our intent is to
demonstrate how modeling these subjects can aid stakeholder understanding that
improves system development outcomes. Rather than existing as implicit
knowledge harbored by system designers, the SDE architecture tenets are
explicit knowledge captured in the ASoT.

Figure 7: Seamless DE Architecture Tenets as packaged concern definitions
(SysML v2 depiction).

##  5 A Grand Challenge Driven by Needs

Grand challenges have historically been proposed to identify important
research directions in a field and to help focus research and development
efforts. Hoare (2003) has clarified the criteria of what makes a research
challenge a ‘grand challenge’ in particular. Table 4 summarizes how SDE, as
proposed by the authors of this paper, fits those criteria. By proposing SDE
as a DE research direction using this framework, it may be better understood
in the context of past grand challenges.

A grand challenge inspires collaboration across multiple fields and challenges
researchers to adopt a transdisciplinary approach. Transdisciplinary systems
engineering Madni (2019) is especially well-suited for tackling the mess
Ackoff (1997) of DE. Seamless DE draws on research in computer science,
computer engineering, human-computer interaction, knowledge engineering,
formal methods, and MBSE to envision a DES designed to meet stakeholder needs.
Systems engineering researchers are encouraged to learn from these fields and
apply that learning with systems praxis IFSR (2012) to the SDE grand
challenge.

As a grand challenge, the work involved is beyond the capability of any one
enterprise. The economics are challenging and would require investment from
multiple organizations working in concert. The formal verification effort, in
particular, has not been achieved at the proposed scale — although we may rest
assured that a sufficiently specified component that is formally verified is
’finished,’ unlike the software we use from day-to-day, allowing engineers to
progress steadfastly. Another salve is that with clean-slate design,
workarounds of legacy systems are unnecessary, and we may dream beyond such
reverse salients as the central processing unit (CPU) and POSIX standard
interfaces Walli (1995). Emulating open-source software projects may not be
enough to manage the complexity, and new approaches to open-source systems
engineering may need to be tested. Lastly, thinking on a generational time
scale may alleviate the apprehension of accomplishing such a megaproject.

Table 4: Evaluation against criteria for a grand challenge Hoare (2003). Criterion |  Rationale
---|---
Fundamental |  Elegant and correct engineering tools are fundamental to development of complex systems.
Astonishing |  An all-encompassing DE tooling system does not exist, but it is at least conceivable and astonishing if realized.
Testable |  Systems engineering test & evaluation methods apply, and the prototype system shall be continuously verified and validated.
Inspiring |  People want better tools because it makes their work easier. The scope is grand and inspiring.
Understandable |  Varieties of computer-aided tools are widely in use, and they may be understandably synthesized.
Useful |  The application of SE in this way shall inform computer science and related fields, as well as being eminently useful as a tool.
Historical |  This challenge is related to early research in computing contracted by the US DoD, and ties in with former grand challenges in computing.
International |  Everyone is welcome to contribute as everyone may potentially benefit from an open-source tool.
Revolutionary |  Systems, software and computer engineering could experience a paradigm shift as the approaches augment each other within a cohesive tooling system.
Research-directed |  Advanced research is drawn upon and brought into an elegant system architecture in a scope commercially infeasible.
Challenging |  While isolated tools and methods are identifiable, their synthesis will present new challenges and open new inquiries for research.
Feasible |  Recent advancements in MBSE, programming languages and formal verification make a SDE system development feasible.
Incremental |  By using MBSE to specify system components and behaviors, and the program itself, progress may be made incrementally and in parallel efforts.
Co-operative |  Researchers are already working in the relevant fields, and may wish to contribute in this new context.
Competitive |  Point-design decisions are unclear in some cases, such as the best type system or virtual machine, and competitive entries may be evaluated against the architecture metrics.
Effective |  Potentially it will change how we advance DE research.
Risk-managed |  Even failures are valuable research outcomes. By using SE, we can reorient at every spiral.

##  6 Conclusions and Future Work

Rather than coping with fragile, byzantine computing systems for implementing
DE, we could have a DES by virtue of a computing system that is designed to
satisfy user needs. Systems engineers are especially well-suited for tackling
this design problem, but it will require a transdisciplinary systems
engineering Madni (2019) and an international collaborative approach. While
many would contest a clean-slate architecture approach as infeasible, there
are interacting emergent effects of modern networked computing systems that
cannot be resolved with patches or by adding more components: namely,
cybersecurity, reliability, and HCI considerations. By incorporating decades
of research in computing and adjacent fields, we can design in the required
functionality at the system level and eliminate unnecessary complexity that
triggers The Error Avalanche (Fig. 5). A clean-slate design effort of this
nature has been proposed in the past DARPA (2010a); Chiricescu et al. (2013),
but it warrants renewed investment and research through the DE lens.

We contend that under the umbrella of “Seamless Digital Engineering Grand
Challenge”, the necessary resources can be assembled to design a DES that
meets DE practitioners’ needs. In this paper, we defined SDE as a paradigm
(Section 4) to help guide future research, and we distinguished it from prior
uses in the literature with the inclusion of the elegance QAt and end-to-end
formal verification for guaranteeing ASoT coherence and integrity. Seamless
integration requirements (Table 3) were derived from tool integration concerns
in the literature to demonstrate how SE methods may address this problem. A
seamless design pattern of DE tool integration (Fig. 6) was depicted in SysML
v2 to contrast this paradigm with existing tool integration patterns (Figs. 2,
3, and 4) that cope with tooling seams. Finally, SDE architecture tenets (Fig.
7) were derived to provide a concise set of concerns that guide system design.

As a grand challenge, significant work lies ahead. Comprehensive research is
needed in eliciting stakeholder needs, transforming those needs into a finite
set of system requirements, developing a reference architecture with SysML v2,
and constructing prototypes for early evaluation. SysML v2 has been chosen to
position this work in line with the next-generation of MBSE tools. An open-
source reference architecture model is being developed to invite further
feedback and collaboration. In this way, DE researchers and practitioners from
around the world can contribute to a seamless DES that inspires and gratifies.

## References

  * Shumaker et al. [2011] Robert W Shumaker, Kristina R Walkup, and Benjamin B Beck.  _Animal Tool Behavior: The Use and Manufacture of Tools by Animals_.  John Hopkins University Press, 5 2011.  ISBN 978-0801898532.
  * Salomon [1984] Jean-Jacques Salomon.  What is technology? the issue of its origins and definitions.  _History and Technology, an International Journal_ , 1(2):113–156, 1984.  ISSN 0734-1512.  doi:10.1080/07341518408581618.
  * Skolnikoff [1993] Eugene B. Skolnikoff.  _The Elusive Transformation: Science, Technology, and the Evolution of International Politics_.  Princeton University Press, 1993.  ISBN 9780691037707.  doi:10.1515/9781400820924.
  * Engineers’ Council Professional for Development [1947] Engineers’ Council Professional for Development.  _Canons of Ethics for Engineers_.  Engineers’ Council for Professional Development, New York, 1947.
  * ISO [2010] ISO.  Systems and software engineering — vocabulary.  ISO/IEC/IEEE Standard 24765:2010, International Organization for Standardization, Geneva, CH, 12 2010.
  * INCOSE Fellows [2019] INCOSE Fellows.  Briefing to incose board of directors.  Briefing, International Council on Systems Engineering, January 2019\.
  * INCOSE Technical Operations [2007] INCOSE Technical Operations.  Incose systems engineering vision 2020, version 2.03.  Technical Report INCOSE-TP-2004-004-02, International Council on Systems Engineering, Seattle, WA, September 2007.
  * Office of the Deputy Assistant Secretary of Defense for Systems Engineering [2018] Office of the Deputy Assistant Secretary of Defense for Systems Engineering.  Digital engineering strategy.  Technical report, US Department of Defense, Washington, DC, June 2018\.  URL https://apps.dtic.mil/sti/pdfs/AD1068564.pdf.
  * Noguchi et al. [2020] Ryan A Noguchi, Marilee J Wheaton, and James N Martin.  Digital engineering strategy to enable enterprise systems engineering.  In _INCOSE International Symposium_ , volume 30, pages 1727–1741, September 2020.  doi:10.1002/j.2334-5837.2020.00815.x.
  * Office of the Deputy Assistant Secretary of Defense for Systems Engineering [2017] Office of the Deputy Assistant Secretary of Defense for Systems Engineering.  Digital engineering ecosystem.  DAU Glossary Definition, Defense Acquisition University, 2017.
  * Hughes [1993] Thomas Parke Hughes.  _Networks of Power: Electrification in Western Society, 1880-1930_.  JHU Press, 1993.
  * Dedehayir and Mäkineif [2008] Ozgur Dedehayir and Saku J Mäkineif.  Dynamics of reverse salience as technological performance gap: an empirical study of the personal computertechnology system.  _Journal of Technology Management & Innovation_, 3(3):55–66, 2008.  doi:10.4067/S0718-27242008000100006.
  * DARPA [2010a] DARPA.  Clean-slate design of resilient, adapative, secure hosts (crash).  Broad Agency Announcement DARPA-BAA-10-70, Defense Advanced Research Projects Agency, Arlington, VA, June 2010a.
  * DARPA [2010b] DARPA.  Meta-ii.  Broad Agency Announcement DARPA-BAA-10-59, Defense Advanced Research Projects Agency, Arlington, VA, April 2010b.
  * DARPA [2015] DARPA.  Circuit realization at faster timescales (craft).  Broad Agency Announcement DARPA-BAA-15-55, Defense Advanced Research Projects Agency, Arlington, VA, August 2015.
  * Norman [1999] Donald A. Norman.  _The Invisible Computer: Why Good Products Can Fail, the Personal Computer is So Complex, and Information Appliances are the Solution_.  MIT Press, 1999.  ISBN 9780262640411.
  * Lombardi [1964] Lionello A Lombardi.  Lisp as the language for an incremental computer.  Technical Report 51-64, Sloan School of Management, Massachusetts Institute of Technology, Cambridge, MA, USA, 3 1964.  URL http://hdl.handle.net/1721.1/48305.
  * Bawden et al. [1977] Alan Bawden, Richard Greenblatt, Jack Holloway, Thomas Knight, et al.  Lisp machine progress report.  Technical report, Artificial Intelligence Lab, Massachusetts Institute of Technology, Cambridge, MA, USA, 1977.  URL https://apps.dtic.mil/sti/citations/ADA062178.
  * Walker et al. [1987] J.H. Walker, D.A. Moon, D.L. Weinreb, and M. McMahon.  The symbolics genera programming environment.  _IEEE Software_ , 4(6):36–45, 11 1987.  ISSN 1937-4194.  doi:10.1109/MS.1987.232087.
  * Ingalls [1981] Daniel HH Ingalls.  Design principles behind smalltalk.  _BYTE Magazine_ , 6(8):286–298, 8 1981.
  * Boehm et al. [1984] Barry W Boehm, Maria H Penedo, E Don Stuckle, Ronald D Williams, and Arthur B Pyster.  A software development environment for improving productivity.  _Computer_ , 17(06):30–44, 6 1984.  doi:10.1109/MC.1984.1659160.
  * Thomas and Nejmeh [1992] Ian Thomas and Brian A. Nejmeh.  Definitions of tool integration for environments.  _IEEE Software_ , 9(2):29–35, 3 1992.  doi:10.1109/52.120599.
  * Broy [2009] Manfred Broy.  Seamless model driven systems engineering based on formal models.  In Karin Breitman and Ana Cavalcanti, editors, _Formal Methods and Software Engineering_ , pages 1–19, Berlin, Heidelberg, 2009. Springer Berlin Heidelberg.  ISBN 978-3-642-10373-5.  doi:10.1007/978-3-642-10373-5_1.
  * Broy et al. [2010] Manfred Broy, Martin Feilkas, Markus Herrmannsdoerfer, Stefano Merenda, and Daniel Ratiu.  Seamless model-based development: From isolated tools to integrated model engineering environments.  _Proceedings of the IEEE_ , 98(4):526–545, 4 2010\.  doi:10.1109/JPROC.2009.2037771.
  * Broy [2020] Manfred Broy.  Seamless model-based system development: Foundations.  In Jonathan P. Bowen, Zhiming Liu, and Zili Zhang, editors, _Engineering Trustworthy Software Systems: 5th International School, SETSS 2019, Chongqing, China, April 21–27, 2019, Tutorial Lectures_ , pages 1–9, Cham, 2020. Springer International Publishing.  ISBN 978-3-030-55089-9.  doi:10.1007/978-3-030-55089-9_1.
  * Perzylo et al. [2020] Alexander Perzylo, Ingmar Kessler, Stefan Profanter, and Markus Rickert.  Toward a knowledge-based data backbone for seamless digital engineering in smart factories.  In _2020 25th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA)_ , volume 1, pages 164–171, 9 2020\.  doi:10.1109/ETFA46521.2020.9211943.
  * AIAA Digital Engineering Integration Committee [2020] AIAA Digital Engineering Integration Committee.  Digital twin: Definition & value.  Technical report, American Institute of Aeronautics and Astronautics, December 2020.
  * AIAA Digital Engineering Integration Committee [2023] AIAA Digital Engineering Integration Committee.  Digital twin: Reference model, realizations & recommendations.  Technical report, American Institute of Aeronautics and Astronautics, January 2023.
  * Karban et al. [2016] Robert Karban, Frank G Dekens, Sebastian Herzig, Maged Elaasar, and Nerijus Jankevičius.  Creating system engineering products with executable models in a model-based engineering environment.  In _Modeling, Systems Engineering, and Project Management for Astronomy VII_ , volume 9911, pages 96–111. SPIE, 2016.  doi:10.1117/12.2232785.
  * Delp [2019] Christopher Delp.  Open model-based engineering environments.  Presentation, NASA Jet Propulsion Laboratory, Pasadena, CA, USA, April 2019.
  * Bajaj et al. [2022] Manas Bajaj, Sanford Friedenthal, and Ed Seidewitz.  Systems modeling language (sysml v2) support for digital engineering.  _Insight_ , 25(1):19–24, 2022.  doi:10.1002/inst.12367.
  * Enticknap and Schuster [1959] R. G. Enticknap and E. F. Schuster.  Sage data system considerations.  _Transactions of the American Institute of Electrical Engineers, Part I: Communication and Electronics_ , 77(6):824–832, 1 1959\.  ISSN 2379-674X.  doi:10.1109/TCE.1959.6372899.
  * Princeton University [2011] Princeton University.  About wordnet.  _WordNet, Princeton University_ , June 2011.  URL https://wordnet.princeton.edu/.
  * Watson et al. [2020] MD Watson, BL Mesmer, and PA Farrington.  Engineering elegant systems: The practice of systems engineering.  Technical report, National Aeronautics and Space Administration, Huntsville, Alabama, USA, 6 2020.
  * Watson et al. [2019] Michael D. Watson, Bryan Mesmer, and Phillip Farrington.  Engineering elegant systems: Postulates, principles, and hypotheses of systems engineering.  In Stephen Adams, Peter A. Beling, James H. Lambert, William T. Scherer, and Cody H. Fleming, editors, _Systems Engineering in Context_ , pages 495–513, Cham, 6 2019. Springer International Publishing.  ISBN 978-3-030-00114-8.  doi:10.1007/978-3-030-00114-8_40.
  * Watson et al. [2014] Michael D Watson, Michael Griffin, Phillip A Farrington, Laird Burns, Wes Colley, Paul Collopy, John Doty, Stephen B Johnson, Richard Malak, Joey Shelton, et al.  Building a path to elegant design.  In _Proceedings of the International Annual Conference of the American Society for Engineering Management._ , Virginia Beach, VA, USA, 10 2014\. American Society for Engineering Management (ASEM).  ISBN 9781634399890.
  * Madni [2012] Azad M. Madni.  Elegant systems design: Creative fusion of simplicity and power.  _Systems Engineering_ , 15(3):347–354, 7 2012\.  doi:10.1002/sys.21209.
  * Griffin [2010] Michael D Griffin.  How do we fix systems engineering?  In _61st International Astronautical Congress_ , volume 27, Prague, Czech Republic, 2010.  September 27—October 1.
  * Whitney and Smith [1906] W.D. Whitney and B.E. Smith.  _The Century Dictionary and Cyclopedia: Dictionary_ , volume 7 of _The Century Dictionary and Cyclopedia_.  Century Company, 1906.
  * Claxton et al. [2005] John D Claxton, Christina Cavoli, and Collie Johnson.  Test and evaluation management guide.  Technical report, Defense Acquisition University, Fort Belvoir, VA, 2005\.  URL https://apps.dtic.mil/sti/pdfs/ADA436591.pdf.
  * Hochschild et al. [2021] Peter H. Hochschild, Paul Turner, Jeffrey C. Mogul, Rama Govindaraju, Parthasarathy Ranganathan, David E. Culler, and Amin Vahdat.  Cores that don’t count.  In _Proceedings of the Workshop on Hot Topics in Operating Systems_ , HotOS ’21, pages 9–16, New York, NY, USA, 6 2021. Association for Computing Machinery.  ISBN 9781450384384.  doi:10.1145/3458336.3465297.
  * Halfhill [1995] Tom R Halfhill.  An error in a lookup table created the infamous bug in intel’s latest processor.  _BYTE, Mar_ , 3 1995.
  * Lipp et al. [2020] Moritz Lipp, Michael Schwarz, Daniel Gruss, Thomas Prescher, Werner Haas, Jann Horn, Stefan Mangard, Paul Kocher, Daniel Genkin, Yuval Yarom, et al.  Meltdown: Reading kernel memory from user space.  _Communications of the ACM_ , 63(6):46–56, 2020\.  doi:10.1145/3357033.
  * Kocher et al. [2020] Paul Kocher, Jann Horn, Anders Fogh, Daniel Genkin, Daniel Gruss, Werner Haas, Mike Hamburg, Moritz Lipp, Stefan Mangard, Thomas Prescher, et al.  Spectre attacks: Exploiting speculative execution.  _Communications of the ACM_ , 63(7):93–101, 2020\.  doi:10.1145/3399742.
  * Porras and Lindell [1995] O Sibert PA Porras and R Lindell.  The intel 80$\times$86 processor architecture: Pitfalls for secure systems.  In _Proc. IEEE Symp. Security and Privacy_ , pages 211–222, 1995\.  ISBN 9780818670152.  doi:10.1109/SECPRI.1995.398934.
  * Chiricescu et al. [2013] Silviu Chiricescu, André DeHon, Delphine Demange, Suraj Iyer, Aleksey Kliger, Greg Morrisett, Benjamin C. Pierce, Howard Reubenstein, Jonathan M. Smith, Gregory T. Sullivan, Arun Thomas, Jesse Tov, Christopher M. White, and David Wittenberg.  Safe: A clean-slate architecture for secure systems.  In _2013 IEEE International Conference on Technologies for Homeland Security (HST)_ , pages 570–576, 11 2013.  doi:10.1109/THS.2013.6699066.
  * Reubenstein et al. [2016] Howard Reubenstein, Theophilos Giannakopoulos, Silviu Chiricescu, Amanda Strnad, and Joseph Fahey.  Semantically aware foundation environment (safe) for clean-slate design of resilient, adaptive secure hosts (crash).  Technical report, BAE Systems, Burlington, United States, 2 2016.  URL https://apps.dtic.mil/sti/pdfs/AD1007956.pdf.
  * von Gernler [2015] Alexander von Gernler.  Towards a clean slate: Attempting to preserve civil liberty in the post-snowden age.  _it-Information Technology_ , 57(3):203–207, 2015\.  doi:10.1515/itit-2015-0008.
  * Snowden [2019] E. Snowden.  _Permanent Record_.  Henry Holt and Company, 2019.  ISBN 9781250237248.
  * Easterly [2023] Jen Easterly.  Cisa director easterly remarks at carnegie mellon university.  Speech, February 2023.  URL https://www.cisa.gov/cisa-director-easterly-remarks-carnegie-mellon-university.
  * Hapeman et al. [2001] ER Hapeman, WR Zeuch, JA Crandall, SM Carioti, SD Barclay, and C Underkoffler.  Telecom glossary 2000.  American National Standard T1. 523-2001., Alliance for Telecommunications Industry Solutions, 2001.  URL https://glossary.atis.org/glossary/data-integrity/.
  * Kay [2011] Alan C. Kay.  A personal computer for children of all ages.  In _Proceedings of the ACM Annual Conference - Volume 1_ , ACM ’72, New York, NY, USA, 2011. Association for Computing Machinery.  ISBN 9781450374910.  doi:10.1145/800193.1971922.
  * Hoare [2003] Tony Hoare.  The verifying compiler: A grand challenge for computing research.  In _International Conference on Compiler Construction_ , pages 262–272. Springer, 2003.  doi:10.1145/602382.602403.
  * Madni [2019] Azad M. Madni.  Transdisciplinary systems engineering: Exploiting disciplinary convergence to address grand challenges.  _IEEE Systems, Man, and Cybernetics Magazine_ , 5(2):6–11, April 2019.  ISSN 2333-942X.  doi:10.1109/MSMC.2019.2899957.
  * Ackoff [1997] Russell L Ackoff.  Systems, messes and interactive planning.  _The Societal Engagement of Social Science_ , 3(1997):417–438, 1997.  doi:10.9783/9781512819069-021.
  * IFSR [2012] IFSR.  The systems praxis framework, developed as a joint project of incose and isss.  Technical report, International Federation for Systems Research (IFSR), Vienna, Austria, 2012.  URL http://systemspraxis.org/framework.pdf.
  * Walli [1995] Stephen R. Walli.  The posix family of standards.  _StandardView_ , 3(1):11–17, March 1995.  ISSN 1067-9936.  doi:10.1145/210308.210315.