# Command Line Interface Risk Modeling

###### Abstract

Protecting sensitive data is an essential part of security in cloud computing.
However, only specific privileged individuals have access to view or interact
with this data; therefore, it is unscalable to depend on these individuals
also to maintain the software. A solution to this is to allow non-privileged
individuals access to maintain these systems but mask sensitive information
from egressing. To this end, we have created a machine-learning model to
predict and redact fields with sensitive data. This work concentrates on Azure
PowerShell, showing how it applies to other command-line interfaces and APIs.
Using the F5-score as a weighted metric, we demonstrate different
transformation techniques to map this problem from an unknown field to the
well-researched area of natural language processing. Keywords: Command-line
interface, sensitive information, secure, machine learning, bag of words, term
frequency-inverse document frequency, word embedding.

Dr Anthony L Faulds, Microsoft<EMAIL_ADDRESS>https://orcid.org/0000-0002-7209-2472

##  1 Introduction

Sovereign Cloud is a unique offering by Microsoft that allows customers access
to the features of Azure Cloud but with isolated physical resource
infrastructure. Some Sovereign Clouds have varying levels of internet access
depending on the customer’s risk aversion. Each customer defines the
requirements of a user to gain access to the system. These requirements can
range from receiving certification to having government clearance. Users who
fulfill the appropriate requirements and have administrator-level access are
called operators. Maintaining the Azure Cloud requires directly responsible
individuals (DRI) who can support and debug issues related to different Azure
services.

Unfortunately, the number of individuals that are both a DRI, have Azure
expertise, and an operator, have the necessary credentials is a small group of
people. This group of people is difficult to hire, train and grow because
clearances can take months, and learning Azure enough to support a systems
infrastructure can take months. One way to solve this is to separate the
operator and DRI roles. As a result, DRIs can increase as Microsoft hires more
developers in the Azure infrastructure. In addition, DRIs can interact
directly with the Sovereign Cloud, but sensitive information must be redacted
from their interactions. To solve the issue of redacting sensitive
information, the Command And Query (CNQ) team developed a command-line
interface (CLI) to allow DRIs to send commands to the Sovereign Cloud while
using a combination of ML and operator feedback to redact sensitive
information returned in CLI responses. This solution enables the number of
DRIs to scale without requiring the number of operators to increase at the
same rate or requiring that DRIs also be operators.

PowerShell is a CLI that enables scripting and automation. Azure Cloud has
created PowerShell commands that allocate resources, query resources, connect
to resources, deletes resources, and more.

Sensitive information is data in the Sovereign Cloud or how to connect to the
data, connection strings, and passwords. Each Sovereign Cloud customer has its
definition of sensitive information, but some pieces of data are universally
sensitive across all customers.

We show how we can map this problem to more widely researched NLP problems.
The result is that each feature becomes a document with words inside that
document. CLI risk prediction is a unique problem, and this work explores
different ways to map the CLI command and its response to machine learning
problems. Although we show how to map CLI data to documents, similar to the
data sets in the NLP modeling space, CLI is different because it does not hold
to traditional grammar rules and requires some brevity. For example, variable
names and commands are long enough to be self-describing but shorter than it
is context compared to a typical sentence.

Although this technique is specific to Powershell, the same process can apply
to other CLIs and APIs. For example, changing from binary class to multiclass
classifier can map this problem to determine various types of Personally
Identifiable Information (PII) within command responses. In addition, we can
adjust from predicting the probability of sensitive data to predicting the
likelihood that a command can cause a system outage.

##  2 Problem Definition

In security, it is crucial to constrain the access or use of sensitive
information. This information should have limited transmission or visibility
except for those who are allowed to see this data. Masking this data from the
view of non-qualified individuals is one approach to security. It will enable
non-qualified individuals or systems to maintain secure systems but limits the
information seen.

For Sovereign Cloud, most clients have similar definitions of sensitive
information. For example, Key Vault information is considered sensitive
information. Everything from passwords, connection strings, and certificates
are examples of the many things a Key Vault may store. PowerShell has commands
that can create resources that respond with connection string information and
also have commands to fetch these fields specifically.

As Sovereign Cloud customers evolve, things like the server names,
geographical locations, or PII included in the system will fall under
sensitive information in the future.

For this work, we want to map the PowerShell command-line redaction to a well-
known problem. Since there are no numbers, it seems best to map this work to
an NLP problem. Furthermore, ML models predict sensitivity per field to allow
fine-grain resolution for redacting fields. Therefore a single PowerShell
command is mapped to multiple predictions, one per response.

##  3 Related Work

There are many approaches to NLP problems. Even early approaches converted
words into numbers, or vectors for numerical analysis (luhn1957statistical, )
(teller2000speech, ) (jones2004statistical, ). For this work, we compare
several of those to show the best combination.

The bag of words (BOW) method is a simplified text representation
(joachims1998text, ) (zhang2010understanding, ). It ignores word order and
grammar and stores the count of words as a feature. The word counts can use
all vocabulary in the training data set or a specified dictionary. Next, BOW
converts a string of words into an integer vector with counts of the word in
the document.

Term frequency-inverse document frequency (TF-IDF) extends BOW from word count
to include information about how many words are in a document and how
frequently the words are found across documents (robertson2004understanding, )
(vinokourov2002inferring, ) (rajaraman2011mining, )
(lavelli2004distributional, ). Beel, Gipp, Langer, and Breitinger have
documented the popularity and usefulness of this algorithm (beel2016paper, ).
TF-IDF is a combination of two components, TF and IDF. For TF, this is the
ratio of the count of a single word in a document over the number of words in
the document. Closer to 1.0 indicates how prevalent the word is in a document
compared to other terms in that document. IDF is the inverse of how dominant a
word is in all documents. For example, a word like the being in all documents
would give it a unary IDF of 1, whereas contemplate being used in 1 of 1000
documents would give it a unary IDF of 1000. Multiplying these two numbers
together results in TD-IDF, a combination of word frequency relative to
document length and inverse popularity of the word. This combination results
in large TD-IDF values only when a word is rare or used a lot in a single
document. TF-IDF is used as inputs or features to different techniques, from
clustering to visual search of videos (moon2013survey, ) (sivic2008efficient,
) (ramos2003using, ).

Another approach that helps aid in understanding similar words is word
embedding. Mikolov, Chen, Corrado, and Dean developed a method of using neural
networks to project words into a vector space (mikolov2013distributed, )
(mikolov2013efficient, ). This technique transforms words into a real-value
vector in which words close in vector distance have similar meanings. For
example, BOW and TF-IDF treat secret and secrets as two different words. For
this work, these words are the same. Therefore, word embedding provides a
unique approach to encoding the information better. For this problem, we also
want the model to generalize well. Our training set is a small fraction of the
commands we want to predict. The naming convention of commands spans many
teams. So it is vital to capture the information without needing to train on
every variation of the word secret

##  4 Tokenizing the CLI

PowerShell commands are combined with response names and types to map this
problem to what is known as a document in the NLP space. All three values are
in Pascal Case, so we can separate words by the first capital letter, removing
non-alphanumeric values. For example, New-AzKeyVault becomes New Az Key Vault.
Powershell contains several variables with multiple upper-case characters in a
row. For this, we split the upper case into one separate word. For example,
complicated names like New-AzVMConfig become New Az VM Config. The result is
that each feature becomes a document with words inside that document.

The system for PowerShell is an azure command that has a response. This
response is a hierarchal set of variables where each variable has a type and a
value 1. For this work, we want to redact or allow each variable individually.
Therefore, the information we have for each variable includes 1) variable
name, 2) variable type, 3) variable value, 4) variable parent name hierarchy,
and parent variable type hierarchy. The variable name is the name of the
variable name as assigned by the system. Variable type is the object type,
such as int, float, string, or custom object. Finally, the variable value is
the actual value returned and set in that variable.

Finally, to turn it into words and sentences, we implement tokenization on
non-alpha numerics and go from lowercase to uppercase. Table 1 shows the
original PowerShell command. This single command is mapped to 4 items for
machine learning with features and labels, as shown in Table 2. Finally, Table
3 demonstrates concatenating and tokenization to get something that looks like
documents used for NLP.

    Command:
    Get-AzLocation

    Response:
    Location: eastasia
    DisplayName: East Asia
    Providers: {Microsoft.Devices,
        Microsoft.Cache, ...}

Table 1: Example PowerShell command Command | Module | Field Name | Field Type | Parent Name | Parent Type
---|---|---|---|---|---
Get-AzLocation | Resources |  | PSResource- |  |
|  |  | ProviderLocation |  |
Get-AzLocation | Resources | Location | string |  | PSResourceProviderLocation
Get-AzLocation | Resources | DisplayName | string |  | PSResourceProviderLocation
Get-AzLocation | Resources | Providers | System.Collections.- |  | PSResourceProviderLocation
|  |  | Generic.List$<$string$>$ |  |
Table 2: Example list of machine learning features Document
---
Get Az Location;Resources;;PS Resource Provider Location;;
Get Az Location;Resources;Location;string;;PS Resource Provider Location
Get Az Location;Resources;Display Name;string;;PS Resource Provider Location
Get Az Location;Resources;Providers;System Collections Generic List string;;PS
Resource Provider Location
Table 3: Conversion of commands to documents

For this work, we omit variable values. There are three reasons we overlook it
in model training and prediction. The first is that the variable value can be
any type. Therefore it may not be a string or number. Casting all these values
to strings may be cumbersome, or setting up a model to handle the native type
is difficult. Second, we are predicting whether the variable value is
sensitive and do not want this value to egress the Sovereign Cloud. If we
include this value in our training set, we limit where and how we can train
the model. The training data will have sensitive data, so human interaction
with the training data is not allowed. It makes the data more challenging to
handle. Finally, the most sensitive data are passwords, which are strings of
random characters. It is unlikely that the model will pick that up as a signal
to indicate a password as much as a field name of Password or Passwd.

We desire to preprocess the data set to produce a similar data set to NLP
problems. All the features are text and, if combined, resemble a sentence. In
NLP, words are in a spoken or written language. For this work, we use words to
mean the combination of letters determined by our tokenization process.
Although similar to a sentence, it does not contain correct English grammar.
The coding standard for variable names and types in PowerShell is Pascal Case.
Pascal Case is where a variable name or type is a combination of words that
join into one word, and the first letter is in uppercase while the rest are in
lowercase. For example, a Pascal case for the phrase virtual machine name is
VirtualMachineName. The initial preprocessing tokenizes between each lowercase
letter and an uppercase letter.

##  5 NLP Techniques

###  5.1 Bag of Words

For this work, we explore five techniques for converting text to numbers for
use in machine learning models. This technique creates a dictionary based on
all the words in the training set. Each vocabulary item has a positional
value. The transformation turns each item from a sentence to an array where
the array is the vocabulary length, and the word count found in the document
is assigned to the positional value.

Equation 1 shows the term frequency ($tf$) of each term ($t$) per document
($d$).

$tf(t,d)=\frac{f_{t,d},}{\sum_{t^{\prime}\in d}{f_{t^{\prime}d}}}$ (1)

The advantage is that this transformation is easy to implement. It also
reinforces items that have words used frequently. One downside of BOW is that
document lengths can significantly affect counts. A lengthy document can
naturally have more of a single word, even though that may not reinforce its
importance in predicting that document. However, this does not significantly
affect this problem because all documents are close to the same length. BOW
also does not consider the ineffectiveness of frequently used words or a
word’s relative position.

###  5.2 TF-IDF

TF-IDF resolves some of the BOW disadvantages. This transformation works
similarly to BOW but includes a factor that decreases the value of a word-
based if used more frequently across all documents. Equation 4 shows the IDF
based on the number of document occurrences of the term ($n_{t}$) and the
number of documents ($N$).

$N=|D|$ (2)

$n_{t}=|\\{d\in D:t\in d\\}|$ (3)

$idf(t,D)=\log{\frac{N}{n_{t}}}$ (4)

$tfidf(t,d,D)=tf(t,d)*idf(t,D)$ (5)

Less frequently used TF-IDF still has the disadvantage of not considering word
order.

###  5.3 Word Embedding

Word embedding (WE) is a technique that converts each word to a unit vector
with a specified number of dimensions. The mapping of word embeddings results
in vectors that are close in Euclidean distance having similar meanings. Since
each word in order is converted to a vector, concatenating all the vectors
results in a feature set sensitive to word order.

WE creates unique vectors for each word. The size of the vector can be tuned
to get the best results. WE algorithm is shown in figure 1. Mikolov, Chen,
Corrado, and Dean (mikolov2013efficient, ) developed the Word2Vec algorithm.

Using a pre-trained Word2Vec model converts words into similar vectors that
are not similar. For example, the initial for Azure, Az, and the initial for
the identifier Id can be misinterpreted as Arizona and Idaho, respectively. A
pre-trained Word2Vec model sees these are similar items and therefore assigns
them similar vectors. To mitigate this, we trained a Word2Vec using all of the
Azure Powershell commands and responses. Since this is more akin to
unsupervised learning, we do not need labels. This trained model performs
better and learns relationships with this specialized vocabulary in Azure
Powershell.

0: max words $m$, vector length per word $l$

0: float matrix $V$

1:  _Initialize:_ $v_{k}=0\ \forall k\epsilon{1,...,m*l}$, $i=0$

2: W = vector of words from tokenization of document

3: $J\leftarrow\|W\|$

4: for $j=1,2,...,J$ do

5: $w\leftarrow Word2Vec(W_{j})$

6: for $k=1,2,...,l$ do

7: $v_{i+k}=w_{k}$

8: end for

9: $i\leftarrow i+1$

10: if $i>l*m$ then

11: END LOOP

12: end if

13: end for

Algorithm 1 WE algorithm

Table 4 shows a length-3 vector used for word embedding and the euclidean
distance of each word from the WE of password. Using WE, the word passwords is
closer to password than to private. This demonstrates an advantage of WE in
that similar words have similar vector representations.

Word | WE 3 | Distance
---|---|---
password | [0.1, 0.2, 0.5] | 0.00
passwords | [0.2, 0.1, 0.4] | 0.17
private | [0.9, -0.1, 0.1] | 0.94
Table 4: WE example

The disadvantage of word embeddings is not being able to handle homonyms.
Words spelled the same but with different meanings have the same encoding.
Acronyms in variable names and commands cause similar issues. For example, VM
could mean virtual machine for one PowerShell command and virtual memory for
another. In NLP problems, the word order in a sentence can help a model
determine the word’s meaning. Also, using modeling for parts of speech help in
this problem. These are not as helpful in this domain because commands and
variable names while being self-descriptive, are not complete sentences, and
there is no requirement to use correct grammar. Also, a system like
PowerShell, and future CLIs that will be supported, are built by multiple
teams without overarching coding standards.

###  5.4 BOW Per Feature

Another approach explored is using BOW per feature (BOW-PF). For example, the
word Connection in the field name or parent field name is more important than
the word Connection found in the command’s name. This hybrid approach adds a
type of word order sensitivity. For this transformation, we train the BOW
transformation on the original data set to use the entire training vocabulary.
From there, we use BOW to transform each feature separately. Instead of using
the semicolon concatenated document, we use the features from Table 2. This
data creates six times as many features, but now BOW counts are per feature.

The BOW transform trains on all features as tokenized sentences. Apply the
transform to each of the features separately. This results in transformed
features of the size: original feature size * vocabulary size (v). The first
feature will be the count of the first vocabulary word in the first feature.
The second feature will be the count of the second vocabulary word in the
first feature. The 1+v feature will be the count of the first vocabulary word
in the second feature.

1:  _Parameters:_ m = max words, C = vector of features each containing a
document, n = number of features

2:  _Initialize:_ $v_{i,j}=0\ \forall i\epsilon{1,..,n}j\epsilon{1,...,m*l}$,
$i=0$

3: VOCAB = unique list of words across all features in the data set

4: $J\leftarrow\|C\|$

5: for $j=1,2,...,J$ do

6: for $k=1,2,...,n$ do

7: $w\leftarrow BOW(c_{j,k},VOCAB)$

8: for $l=1,2,...,\|w\|$ do

9: $v_{j+k}=w_{l}$

10: end for

11: end for

12: end for

Algorithm 2 BOW-PF algorithm

This transformation has similar advantages to BOW. Where it differs is the
added advantage of separating word count per feature so that the model can
learn the nuances of words in command name vs. variable name. However, a
disadvantage is that it creates more features and can take an ML model longer
to learn. In future work, this can be resolved by adding a feature selection
step that could reduce the number of features before training a final model.

###  5.5 TF-IDF Per Feature

Similar to the BOW-PF, the TF-IDF per feature (TF-IDF-PF) trains the transform
using the TF-IDF on the documents shown in Table 3. Then applies that
transform to each feature separately. This process also has six times as many
features. The advantages include TD-IDF’s ability to scale word importance to
word prevalence across documents.

1:  _Parameters:_ m = max words, C = vector of features each containing a
document, n = number of features

2:  _Initialize:_ $v_{k}=0\ \forall k\epsilon{1,...,m*l}$, $i=0$

3: VOCAB = unique list of words across all features in the data set

4: $J\leftarrow\|C\|$

5: for $j=1,2,...,J$ do

6: for $k=1,2,...,n$ do

7: $w\leftarrow TF-IDF(c_{j,k},VOCAB)$

8: for $l=1,2,...,\|w\|$ do

9: $v_{j+k}=w_{l}$

10: end for

11: end for

12: end for

Algorithm 3 TF-IDF-PF algorithm

In the future, we would like to explore reducing the features used for BOW-PF
and TF-IDF-PF. One idea is to use BOW applied to the features as a sentence.
Train using a tree model and use feature importance to reduce the vocabulary.
Then apply the reduced vocabulary BOW transform to each feature separately.
Doing this will decrease training time. It should also help with the
generalization of the model. This technique excludes rare features that can
make the model overfit.

##  6 Experiments

###  6.1 Machine Learning Models

To compare different preprocessing steps, we want to pick a couple of
different machine learning models to compare results. Some model types work
differently in handling hundreds of features, which causes underfitting or
overfitting.

Logistic Regression (LR) is a prime candidate as a baseline for comparison. It
trains quickly relative to other types of models. It is also easier to analyze
the relationship between features and the label. A more advanced method is
AdaBoost Trees (BT)(freund1997decision, ). This method trains multiple trees
on the training data, with each subsequent group of trees trained on a
weighting based on incorrectly classified examples. The last method used is
Neural Networks (NN). Each of these methods can be refined to build optimal
F5-Scores. This work aims not to find the best modeling technique but to
determine which transformation technique works well in each model type. The
same hyperparameters are used across all transformations for each model type
to compare transformation methods accurately.

###  6.2 Evaluation Data Sets

We use a manually labeled data set containing over 60,000 entries derived from
1,420 commands. Those commands make up 4% of all Azure PowerShell commands. In
addition, we labeled items that allow for authentication, authorization, or
specifically designated sensitive for this data set. This includes connection
strings, certificates, passwords, and values stored in a Key Vault. This does
not include IP, URLs, or ports. These items aid in the connection to a
resource but do not provide authentication or authorization.

Name | Count
---|---
Azure Command Coverage | 1,420
Training size | 62,579
Redacted | 2,155
Table 5: Training data statistics

Some libraries have more redacted fields.

###  6.3 Metrics

The model should attempt to redact all sensitive fields. It is understood by
the customer and us that this is impossible, whether it be because of
statistically based models that could miss some sensitive information or human
mistakes. To that end, we work with the customer to provide the best solution
and work to asymptotically approach 100% recall in partnership with the
customer. False positives are not a major issue. If something is redacted that
should not be, it could hinder the DRI’s work. False negatives are a bigger
issue. They allow DRIs to see sensitive information. Sensitive information is
defined as passwords, connection strings, and other objects that are keys to
information or keys to resources. For this work, we are currently assuming
DRIs as good users, and the system is reducing information to just a need-to-
know basis.

Therefore we need a metric that encapsulates the critical importance of
recall. We must balance that will precision. If too many values are hidden, it
is difficult for a person to do maintenance. Therefore we have to balance
security with usefulness. A house without windows or doors is the most secure
but also difficult to live within.

The best option is to use the $F_{\beta}$ score shown in Equation 6. We want a
continuous function that takes into account a tradeoff between precision and
recall. This is a practical solution but does not provide a single metric if
we want to increase recall. Therefore one of the best single metrics for this
work is the $F_{\beta}$ score. This equation specifies a tradeoff relationship
between recall and precision. For this work, we define that recall is five
times more important than precision. For an f-score to be better, if we
increase recall by one percent, that is worth 5% false positives (decrease in
precision.

$F_{\beta}=(1+\beta^{2})*\frac{precision*recall,}{(\beta^{2}*precision)+recall}$
(6)

Each model has a range of $F\beta$-scores determined by a score threshold.
This demonstrates the differences between the models based on these curves.

Because models are statistically derived, often taking samples of training
data to improve the loss, we run each model 20 times and take the average max
f5-score. With this, we also run statistical significance.

For this work, we propose several metrics for comparison. One often-used
metric is the area under the ROC curve (AUC). This is the area under the
receiver operator curve (ROC). ML models output probabilities of an event. The
split of this probability defines one of two categories based on a threshold
for a two-categorical system. Over that value, we select one. Under we select
the other. As the threshold changes, the metrics change. AUC is a single
metric across all the threshold’s possible values. It tends to indicate
whether a model has better overall thresholds than another. AUC is a valuable
metric in some problem domains. However, sometimes, our data is highly skewed.
As a result, AUC can yield misleading results.

For our work, we are most interested in precision and recall. Sensitive data
should never leave the Sovereign Cloud. For that reason, recall is critical.
For comparing models, we want to hold recall at 0.99 or higher and continue to
increase precision. This number means that the model redacts sensitive
information 99% of the time at the cost that it could block nonsensitive data,
which may be helpful for a DRI to do their work. Therefore, we want to
maintain a high standard of 0.99 or higher and continually improve at not
blocking nonsensitive data.

###  6.4 Vocabulary Size

Each of the transformations discussed uses a vocabulary of words. In NLP,
words are a combination of letters that make a word in a specified spoken or
written language. We refer to a combination of letters as a word for this
work, although acronyms and shortened words are frequently used in
programming. Therefore our vocabulary differs a little from words found in the
English language.

The vocabulary or number of unique words in the training set determines the
number of features for BOW and TF-IDF. The features for BOW are the counts of
each vocabulary word defined by equation 1. The features of TF-IDF are the
same in the count but determined by equation 5. For WE, fixed-length vectors
represent each word. The average number of words in the tokenized command data
set is 78. In order to make a fair comparison to the other transformations,
the size of the WE final vector is equal to the vocabulary size of BOW and TF-
IDF. This experimentation uses a length of 20 float vectors per word.

###  6.5 Feature Importance

Table 7 and 8 display the differences in the importance of different
vocabulary words using BOW and TF-IDF transform. One crucial difference and
something important to this problem is that certificate and certificates are
treated as two separate words. For BOW, the word, certificate, is a rarer
word, so it is not as crucial as certificates. On the other hand, TF-IDF
emphasizes the importance of the IDF multiplier and makes certificate almost
as important. WE algorithm handles this by making the vector encoding of the
two words very close in Euclidean distance. As a result, each algorithm does a
little better at emphasizing the importance or lack of difference between
certificate and certificates.

Transform | Feature Size
---|---
BOW | 1,559
TF-IDF | 1,559
WE | 1,560
BOW-PF | 9,354
TF-IDF-PF | 9,354
Table 6: Feature size Word | Importance
---|---
key | 0.071
certificates | 0.041
encryption | 0.032
secret | 0.032
network | 0.029
name | 0.022
vault | 0.021
compute | 0.021
management | 0.020
string | 0.019
Table 7: Feature importance for BOW Word | Importance
---|---
certificates | 0.049
key | 0.042
certificate | 0.041
name | 0.037
string | 0.032
osprofile | 0.031
az | 0.030
encryption | 0.024
system | 0.023
id | 0.022
Table 8: Feature importance for TF-IDF

BOW-PF introduces a partial word order. It weighs words used in the field name
and parent name more strongly than the module name or the actual command. The
feature importance of BOW-PF is shown in table 9. Interestingly, the model
weights the word key high in the field name.

Feature | Word | Importance
---|---|---
Field Name | key | 0.0562
Parent Type | network | 0.0343
Field Name | data | 0.0253
Field Name | certificate | 0.0239
Parent Name | network | 0.0208
Field Name | name | 0.0206
Parent Type | service | 0.0188
Field Name | string | 0.0186
Field Type | encryption | 0.0184
Parent Type | psmanaged | 0.0184
Table 9: Feature importance for BOW-PF

TF-IDF-PF has a similar finding as TF-IDF, where the plural of a word has
similarly high importance, shown in table 10.

It is also worth noting that string is an essential word in all models. Almost
all secrets come from strings or complex string types. Very little sensitive
information is stored in integers, booleans, or enumerated types.

Feature | Word | Importance
---|---|---
Field Name | key | 0.0630
Field Type | collections | 0.0308
Command | az | 0.0283
Field Name | data | 0.0280
Parent Type | osprofile | 0.0252
Field Type | azure | 0.0212
Parent Type | models | 0.0211
Field Type | system | 0.0203
Parent Type | cluster | 0.0190
Field Name | keys | 0.0189
Table 10: Feature importance for TF-IDF-PF

The training data is built by manually labeling 1500 commands and each of the
response attributes. The data is split into 80% training and 20% validation
data. The split is done by Azure Powershell command so that the validation set
has commands that have not been used in training. This should yield a good
estimate of the generalizability of the algorithm and how well it works on
unseen commands.

Each model is run 20 times, with the results averaged. Displayed in figures
1(a) are the runs of each transform. Some depict using logistic regression,
and others depict using boosted trees. First, each model trains using the
training set. Next, calculating the maximum F5-score is done using the test
set. Finally, using the trained model and the threshold for the max F5-score,
the validation set is used to determine F5-score, as seen in new examples.

###  6.6 Metric Comparison

(a) LR

(b) BT

(c) NN

(d) LR zoomed

(e) BT zoomed

(f) NN zoomed

Figure 1: F5-Score for transforms and models.

$F_{\beta}$-score changes with the threshold selected. This model must make a
binary decision, so we choose the threshold to give us the max F5-Score. We
use the test data set to determine the threshold with the max F5-Score. We
then use the validation set to test the model and threshold against unseen
examples to determine how well the model will generalize and how well it will
work in a production setting.

Each curve shows the F5-Score, as calculated from equation 6, for all
thresholds. The point on the curve is the max F5-Score overall threshold
value. It should be noted that a curve with a plateau top has a robust
threshold. Slight changes in threshold do not change F5-Score much. It also
implies that unseen examples are not sensitive to slight changes in threshold
value. A sharp peak at the maximum F5-Score implies an unstable maximum. Using
this model against this or other data sets could result in very different
F5-Scores if anything is slightly perturbed.

For the logistic regression example, TF-IDF is the weakest. BOW is effective,
while BOW-PF is the most effective. The PF is a form of word importance that
weights the words. TF-IDF does not work as well in highly skewed data. The
word frequencies help predict both the positive and negatively labeled items.
Infrequent words help identify risky items, but most items are not risky, so
words that help predict non-risky items are frequent, and therefore TF-IDF
decreases the weights.

BT has more of a peak to most curves. Unlink LR BT is discontinuous. The
inference of a BT model with perturbation of features causes different parts
of the tree to be traversed. The ordering of worst-to-best algorithms stays
relatively consistent across models. BT’s ability to handle discontinuous
functions allows it to fit the data better, resulting in a higher F5-Score.
BOW-PF still performs the best as it uses word counts, does not discount
frequently used words, and includes a type of word ordering captured in the
Per-Feature part of the algorithm.

This work demonstrates how BOW-PF and TF-IDF-PF can be used to apply a word
order that is not in BOW or TF-IDF. It is recommended that if you use LR or
BT, BOW-PF is the transformation that should be used. BOW-PF captures the use
of words in each feature, for example, emphasizing the field name having the
word key over the command having the word key. The inverse of the document
frequency is not handled well in LR and BT. NN, on the other hand, picks up
the nuances of the inverse of document frequency and handles TF-IDF and TF-
IDF-PF better than the BOW counterparts.

Transform | LR | BT | NN
---|---|---|---
BOW | 0.937 | 0.955 | 0.952
TF-IDF | 0.788 | 0.954 | 0.962
WE | 0.904 | 0.956 | 0.933
BOW-PF | 0.969 | 0.987 | 0.990
TF-IDF-PF | 0.929 | 0.984 | 0.995
Table 11: Max F5-Scores for transforms

##  7 Conclusion and Future Work

This work demonstrates how to convert PowerShell commands to the well-
researched NLP domain. From there, we compare popular techniques and find that
BOW-PF performs best at creating features for different types of models. This
work exploited the sentence-like structure of commands, variable names, and
variable types. Using F5-Score as a metric for comparison allows us to
summarize system goals into one value for optimizing across thresholds and in
subsequent retraining.

Slight adaptions to this work yield a system that works across different CLIs.
Systems other than PowerShell will be more complex because the output is not
in a structured programming object. Also, the variables’ names are Pascal-case
for PowerShell, but other CLIs will require different transformations to
convert commands and variables to words. This will require more work to
convert as even if the responses are structured as objects and the variables’
names are Pascal-case, the names are variables are still created by different
development teams. It is similar to the NLP space of handling different
languages. For example, one CLI might call a password variable Password,
whereas another CLI interface might call it Psswd. The system will need to be
trained to pick up new vocabulary.

This work can extend to APIs. Powershell, behind the scenes, makes API calls
to pass commands and receive back the response. Powershell turns those JSON or
XML responses into objects. This system is built on structured response
objects. APIs have similar structures in which there is an endpoint, similar
to a command, and a response, often a JSON object, analogous to the response
object of PowerShell. Therefore with little work, the risk modeling can be
moved to API space to redact or mask values returned in API calls.

## References

  * (1) H. P. Luhn, “A statistical approach to mechanized encoding and searching of literary information,” IBM Journal of research and development, vol. 1, no. 4, pp. 309–317, 1957.
  * (2) V. Teller, “Speech and language processing: an introduction to natural language processing, computational linguistics, and speech recognition,” 2000\.
  * (3) K. S. Jones, “A statistical interpretation of term specificity and its application in retrieval,” Journal of documentation, 2004.
  * (4) T. Joachims, “Text categorization with support vector machines: Learning with many relevant features,” in European conference on machine learning, pp. 137–142, Springer, 1998.
  * (5) Y. Zhang, R. Jin, and Z.-H. Zhou, “Understanding bag-of-words model: a statistical framework,” International journal of machine learning and cybernetics, vol. 1, no. 1, pp. 43–52, 2010.
  * (6) S. Robertson, “Understanding inverse document frequency: on theoretical arguments for idf,” Journal of documentation, 2004.
  * (7) A. Vinokourov, N. Cristianini, and J. Shawe-Taylor, “Inferring a semantic representation of text via cross-language correlation analysis,” Advances in neural information processing systems, vol. 15, 2002.
  * (8) A. Rajaraman and J. D. Ullman, Mining of massive datasets.  Cambridge University Press, 2011.
  * (9) A. Lavelli, F. Sebastiani, and R. Zanoli, “Distributional term representations: an experimental comparison,” in Proceedings of the thirteenth ACM international conference on Information and knowledge management, pp. 615–624, 2004.
  * (10) J. Beel, B. Gipp, S. Langer, and C. Breitinger, “Paper recommender systems: a literature survey,” International Journal on Digital Libraries, vol. 17, no. 4, pp. 305–338, 2016.
  * (11) A. Moon and T. Raju, “A survey on document clustering with similarity measures,” International Journal of Advanced Research in Computer Science and Software Engineering, vol. 3, no. 11, pp. 599–601, 2013.
  * (12) J. Sivic and A. Zisserman, “Efficient visual search of videos cast as text retrieval,” IEEE transactions on pattern analysis and machine intelligence, vol. 31, no. 4, pp. 591–606, 2008.
  * (13) J. Ramos et al., “Using tf-idf to determine word relevance in document queries,” in Proceedings of the first instructional conference on machine learning, vol. 242, pp. 29–48, New Jersey, USA, 2003.
  * (14) T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” Advances in neural information processing systems, vol. 26, 2013.
  * (15) T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” arXiv preprint arXiv:1301.3781, 2013\.
  * (16) Y. Freund and R. E. Schapire, “A decision-theoretic generalization of on-line learning and an application to boosting,” Journal of computer and system sciences, vol. 55, no. 1, pp. 119–139, 1997.


# Robot Design: Formalisms, Representations, and the Role of the Designer

Alexandra Q. Nilles, Dylan A. Shell, and Jason M. O’Kane  Alexandra Q. Nilles
<EMAIL_ADDRESS>is with the Department of Computer Science at the
University of Illinois, Urbana-Champaign, Illinois, USA. Dylan A. Shell
<EMAIL_ADDRESS>is with the Department of Computer Science and
Engineering, Texas A&M University, College Station, Texas, USA. Jason M.
O’Kane<EMAIL_ADDRESS>is with the Department of Computer Science and
Engineering, University of South Carolina, Columbia, South Carolina, USA.

> “Civilization advances by extending the number of operations we can perform
> without thinking about them.”
>  — Alfred North Whitehead

##  I Introduction

Robotics is a-changin’. In the very recent past, if a robot worked at all, one
had cause to be happy. But, moving beyond thinking merely about robots, some
researchers have begun to examine the robot design process itself. We are
beginning to see a broadening of scope from the products to the process, the
former stemming from expertise, the latter being how that expertise is
exercised.

The authors have participated in this discussion by helping to organize two
related workshops—the RSS 2016 Workshop on Minimality and Design Automation,
and the RSS 2017 Workshop on Minimality and Trade-offs in Automated Robot
Design. These workshops brought together researchers with a broad range of
specializations within robotics, including manipulation, locomotion, multi-
robot systems, bio-inspired robotics, and soft robotics; and who are
developing lines of research relevant to automated design, including formal
methods, rapid prototyping, discrete and continuous optimization, and
development of new software interfaces for robot design. Insights from these
workshops heavily inform the discussion we present here.111A full discussion
of all the participants and research directions raised in the workshops lies
beyond the scope of this abstract. See http://minimality.mit.edu/ for full
speaker lists and more information. This objective of this paper is to distill
the following essential idea from those experiences:

> The information abstractions popular within robotics, designed as they were
> to address insulated sub-problems, are currently inadequate for design
> automation.

To that end, this paper’s first aim is to draw together multiple
threads—specifically those of formalization, minimality, automation, and
integration—and to argue that robot design questions involve some of the most
interesting and fundamental challenges for the discipline. While most efforts
in automating robot design have focused on optimization of hardware, robot
design is also inextricably linked to the design of the internal state of the
robot, how that internal state interacts with sensors and actuators, and how
task specifications are designed within this context. Focusing attention on
those considerations is worthwhile for the study of robot design because they
are currently in a critical intellectual sweet spot, being out of reach
technically, but only just.

The second ingredient of this paper forms a roadmap. It emphasizes two
aspects: (1) the role of models in robot design, a reprise of the old chestnut
about representation in robotics (namely, that “the world is its own best
model” [1]); (2) a consideration of the human-element within the envisioned
scheme.

##  II Four Challenges for Design Automation

From our experiences with robot design and the work toward automation of such,
four themes emerge which recur in many different problem spaces.

###  II-A Formalization: Toward Executable Robotics Theories

Useful formalism builds symbolic models that enable chains of deduction in
order to make predictions and guarantees about robot performance. Robot design
problems exhibit a great deal of structure: we typically use a narrow set of
available hardware components, there are units for quantifying functionality
provided by components, and there are increasingly expressive languages for
providing functional specifications in classes of tasks [2, 3].

Once knowledge is abstracted in this way, it becomes re-usable through the
formation of libraries and tools. Models provide a way to give expression to
assumptions and guarantees; they are also the starting point of a language of
operations (such as composition, refinement, and compression) to achieve
higher levels of competency while managing complexity. State-of-the-art
techniques remain quite piecemeal, limited in the aspects of the problems they
encompass. Further, a great deal of current knowledge is tied up in
mathematical form, without being made machine usable or readable—consider the
criteria used to choose a particle filter rather than an (extended) Kalman
filter. If this expertise were encoded formally, in terms of model assumptions
and resource trade-offs, software could provide a pose estimate on the basis
of domain properties without the roboticist being concerned about the details.

Efforts to formalize robotics are moving toward more than “on paper”
formalisms which capture the structure inherent in robotic systems. We cannot
ignore the work being done in formal synthesis techniques [4] and related
efforts to encode our knowledge about robots and physical systems in an
executable form. In turn, robotics provides a plethora of benchmarks and
motivating examples for researchers in formal methods for embedded/hybrid/open
systems, beyond the classic applications such as thermostats and airplanes. In
turn, these researchers can benefit by engaging with prior work on formalizing
robotics. The results of these collaborations have major implications for the
power and correctness of automated design systems.

###  II-B Minimality: Toward an Understanding of Robot Power

One decades-long line of research poses the question of what tasks a given
robot can complete, or the inverse question, what kinds of robots are capable
of completing a given task. If we imagine ranking robots by some measure of
complexity (their ability to sense, actuate, and compute), at the bottom of
this ranking is a robot with no sensors, no actuators, and no ability to keep
track of state. This robot is quite useless, except perhaps as a paperweight.
Then, as we augment the robot with sensors, actuators, and computational
power, at some point it becomes capable of accomplishing tasks. The
theoretical boundaries of this “design space” are not well understood, despite
considerable work in this direction [5, 6, 7]. As the robot design process
becomes more automated, this line of work becomes more relevant — human
designers may want to provide functional or informational specifications (“I
want a robot that can pick up this type of box” or “I want a robot that can
find my keys in my living room”).

The theme of minimality is more than design-automation-through-optimization,
where we may ask how to design the smallest robot meeting some design
constraints, or one with the fewest number of linkages. This line of work is
useful, and advancing at an exciting pace (see, for example, the work of
Spielberg _et al._ [8] for an interesting example on simultaneous optimization
of robot design and motion strategy). However, it relies on highly-trained
humans to design the underlying models and design constraints. It also does
not help us explore the space of robots which may have quite different body
geometries and hardware designs, but which are all equivalent in their power
to complete a certain task. Thus, the theme of minimality intersects with
formalization, since better formalisms for describing and comparing the
functionality of robots are needed to reason about the theoretical limits of
such systems. Progress in this area would directly impact the power and
correctness of automated design tools.

Many of the problems in this space are computationally hard, in the sense of
NP-hardness [9, 10]. However, this is not a reason to give up! Often,
constraints on the problem space can bring design problems back into a
tractable realm, and advances in generic solvers for problems like integer
programming and satisfiability-modulo-theories (SMT) mean that solving these
problems is becoming more feasible in practice.

###  II-C Automation: Toward Tractable, Realizable Designs

The prior two questions — _how_ robotic systems can be computably represented,
and _what_ are the information requirements of robotic tasks — are tied to the
question of automating the design and fabrication of robotic hardware.

One way to tackle the problem of representation is through modularity and
standardization. In the robot design space, standardization is largely driven
by hardware manufacturers. Robot designers generally, and academics or
hobbyists especially, are design robots by choosing from a range of off-the-
shelf components. This constraint, along with work toward modular and
composable robot designs, helps tame the computational complexities involved.
When the information requirements of a given task have been identified, it is
easier to design a satisfactory robot from a finite collection of components
than to choose a design from the infinite space of all sensors, actuators, and
body geometries.

Of course, the proliferation of new fabrication technologies is challenging
this view of the problem. Task satisfaction and hardware design algorithms are
becoming increasingly integrated, such as a system which automatically places
winch-tendon networks in a soft robot based on a user-specified movement
profile [11]; or a system which evolves shapes of automatically fabricated
wire robots in order to achieve different specified locomotion tasks [12]; or
a system which compiles high-level specifications into laser-cut schematics
and mechanical and wiring diagrams [13]. The flexibility inherent in these
approaches will require new approaches to parameterizing and formalizing the
very large design space available to us.

###  II-D Integration: Toward an End-to-End System

One of the largest challenges in robotics is the integration of different
components and control structures into one robotic system which functions
correctly, and ideally, has some guarantees on its performance. We must
integrate mechanical, electrical, computational, and material systems, and
also must reason across multiple levels of abstraction.

Similar to how computer systems have an abstraction “stack” (transistors to
byte code to programming languages to abstract reasoning over models of
programs), robotics has its own similar abstraction stack, which must give
more attention to the physical reality of the robotic system (from material
properties, to component implementations, to dataflow protocols, to abstract
sensor, actuator, and state representations, to task-level reasoning).222Note
that this “design stack” has complicated dependencies between layers —
advances in low-level hardware enable new choices at all levels of
abstraction. This is true of traditional computer architecture as well, though
perhaps more true in robotics. The recent impact of IMU availability on how
robots are designed and programmed is a choice example.

Identifying distinct layers of this abstraction stack, and the assumptions
therein, is a crucial challenge facing roboticists and is especially crucial
for those working on the frontiers of design automation. If this challenge is
not met, then high-level automation tools (such as formal logic
specifications) which aim to provide safety and security guarantees will be
useless due to mismatches with the physical implementations.

The difficulty of this integration task leads us to believe that robot design
will continue to be an iterative, experiment-driven process for the
foreseeable future, and tools which automate parts of the design process
should enable this workflow pattern. This is especially true wherever
fabrication is a time-consuming or otherwise expensive process. Even 3D
printing, the consummate rapid prototyping tool, can involve spending hours
waiting for a print job to finish - only to realize a flaw in the design when
dynamics are taken into account! The need for rapid feedback and prototyping
is also great for robotic tasks which rely heavily on environmental
interaction. Our current simulation technology is not up to the task of
determining if a given robot design can navigate a sandy, rocky desert, for
example.

##  III A tentative roadmap

> “Design activity… is a processes of ‘satisficing’ rather than optimising;
> producing any one of what might well be a large range of satisfactory
> solutions rather than attempting to generate the one hypothetically-optimum
> solution.”
>
> — Nigel Cross, _Designerly Ways of Knowing_ [14]

###  III-A The Role of Models

One frequent point of the discussion in the aforementioned RSS workshops on
automated robot design was the role of models in the robot design process.
Some roboticists argued that a ”build first, model later (if at all)” approach
is the most effective method for robot design, and that automation efforts
should focus on making the prototyping process as fast as possible. Our lack
of understanding of the physics and “unknown unknowns” in hardware
implementations make most models nearly useless in the design process. Even
very high-fidelity simulations often act completely differently than physical
robots, and designer time is often better spent making actual prototypes and
observing their behavior.

The issue of what role models play in design also extends to optimization-
based approaches. This approach generally uses continuous and discrete
optimization algorithms to adjust robot morphology, sensor configurations, and
motion strategies. However, an algorithm which is optimizing the number and
placements of legs on a mobile robot will never spontaneously decide to try
using wheels instead. Often, the process of prototyping robots reveals
constraints of the task and available hardware that are not apparent at the
beginning of the design process. It is very unlikely that robot designers will
create a perfect optimization problem or other formal specification in a first
attempt. As a result, the design process is inherently iterative, regardless
of its degree of automation.

The following are design decisions (“forks in the road”) that creators of
robot design tools can ask themselves to ensure thoughtful consideration of
the models used and their integration with the rest of the robot design
“stack”:

  * •

What assumptions does the tool make about what types of robots are being
designed?

  * •

Are modeling assumptions communicated clearly to users and (if applicable) at
the API level?

    * –

For example, the Unified Robot Description Format (URDF), often used for ROS
robot models, only allows kinematic tree body types, and thus is unable to
specify robots with closed kinematic chains, but this assumption is clearly
communicated in the documentation [15].

  * •

Are modeling assumptions enforced by the software (perhaps through type
systems, model checkers, a test set, etc), or does that responsibility fall to
the user? What kind of feedback does the tool give when these assumptions are
broken?

  * •

Does the tool attempt to give meaning to designs (such as visualization or
dynamical simulations) before they are fabricated?

  * •

How does the tool interact with the rest of the robot design ecosystem? Can
the tool leverage or bolster existing free and open-source technologies?

###  III-B The Role of Humans

The role of the human in the design process will not, and should not, ever be
completely eliminated. The human role may become extremely high level, perhaps
even to the point where we have systems which autonomously infer new robot
designs. (Imagine, for example, an assistant which notices you performing some
repetitive task and offers a robotic solution.) But humans will still play a
role in the _design of the design tools_ , embedding our biases and
preferences. In the more immediate future, our current design technologies
rely heavily on human input for specification design, and for the insights
that an experienced designer can leverage. This expertise is required at all
levels of the robot design stack.

As in all creative fields, robotics has a plethora of design tools for
different types of users and at different levels of abstraction. As research
into automated robot design continues, we must objectively study the effects
of different interface and architecture decisions on how these tools are used,
instead of relying on our intuition. For example, the literature is mixed on
whether visual programming languages are easier to use for novice programmers,
despite a widespread belief that they are better for children and other
novices [16, 17].

Figure 1: A variety of information flows have been proposed for the way
computer-aided design of robots might proceed. They are distinguished by where
the representation of the robot design is stored (in a computer or in a human
brain), and by the direction and effect of queries. [First] A model of
traditional synthesis. [Second] Automated design, where the automated system
queries a human to resolve incompatibilities or underspecified components of
the design. [Third] Interactive design tools to answer human designer queries,
such as, “what’s the lightest sensor that can detect the color blue?”. This
category may broad enough to encompass many rapid-prototyping systems - “if I
3D print this wheel, will it be strong and light enough for my robot car?”
[Fourth] Interactive design tools, where human questions inform the formalized
solution. For example, a human may query the system, “what happens if I make
the legs twice as long?” which will change the current design solution on the
computer. [Fifth] Socratic model: an automated system which asks questions
and/or provides suggestions to inspire a human designer. (This figure combines
contributions from Andrea Censi, Ankur Mehta, and the present authors).

The following are guiding questions for creators of automated design tools.
Many of these questions are inspired by and explored more deeply in the Human-
Computer Interaction literature, which provides a rich resource for creators
of design tools.

  * •

Who are the intended users of the tool? What other groups of people may find
the tool interesting?

  * •

What part of the user workflow does the tool replace, or what new workflows
does it enable?

  * •

How do users interact with the tool?

    * –

Directly (CAD software, programming language, etc) or indirectly (3D printer,
robot component database, low-level instruction set, etc)? Even indirect
interactions are important to consider, for example, when a user finds their
CAD design won’t fit on a print bed, or a change in a low-level instruction
changes what is possible in a high-level interface.

    * –

Modality of interaction: graphical or text based? What kind of feedback does
the user get from the tool, especially when they specify something impossible
or introduce a bug?

Several interesting examples of these interaction modalities have been
explored already, including:

  * •

Interactive (click and drag) design of morphology and gait with immediate
visual feedback, with fabrication blueprints generated after design is
finalized [11].

  * •

Formal specification in code, followed by a compiler which detects possible
problems with specification and suggests changes to user if the specification
has inconsistencies [18].

  * •

Giving “early meaning” to partial designs via dynamical simulations,
visualization, and haptic interactions with simulated components [19], and
allowing composition of these modules, such as those used in popupCAD [20].

As a baseline, we would like to automate repetitive and time-consuming tasks,
and leave the more creative parts of the workflow intact for the human
designer. The ideal case is that a new automated design tool would enable new
forms of human creativity, such as the way electronic music tools have enabled
new methods of human-driven composition [21].

##  IV Conclusion

In this abstract, we presented a broad vision for a future in which the
process of designing robots transitions from a laborious, error-ridden process
driven almost exclusively the cleverness and determination of expert human
designers, to one in which automated design tools play a significant role in
the process. Our position is that the abstractions commonly used within
robotics will have to be extended in order to address questions that are
essential for automating hardware realization and fabrication, questions
dealing most fundamentally with information and representation.

## References

  * [1] R. A. Brooks, “Intelligence without reason,” in _Proc. International Joint Conference on Artificial Intelligence_ , 1991.
  * [2] Y. Shoukry, P. Nuzzo, I. Saha, A. L. Sangiovanni-Vincentelli, S. A. Seshia, G. J. Pappas, and P. Tabuada, “Scalable lazy SMT-based motion planning,” in _Proc. IEEE Conference on Decision and Control_ , 2016.
  * [3] E. Plaku and S. Karaman, “Motion planning with temporal-logic specifications: Progress and challenges,” _AI Communications_ , vol. 29, no. 1, pp. 151–162, 2016.
  * [4] H. Kress-Gazit, M. Lahijanian, and V. Raman, “Synthesis for robots: Guarantees and feedback for robot behavior,” _Annual Review of Control, Robotics, and Autonomous Systems_ , 2018.
  * [5] M. Blum and D. Kozen, “On the power of the compass (or, why mazes are easier to search than graphs),” in _Proc. Annual Symposium on Foundations of Computer Science_ , 1978.
  * [6] B. R. Donald, “On information invariants in robotics,” _Artificial Intelligence_ , vol. 72, no. 1-2, pp. 217–304, 1995.
  * [7] J. M. O’Kane and S. M. LaValle, “On comparing the power of robots,” _International Journal of Robotics Research_ , vol. 27, no. 1, pp. 5–23, 2008\.
  * [8] A. Spielberg, B. Araki, C. Sung, R. Tedrake, and D. Rus, “Functional co-optimization of articulated robots,” in _Proc. IEEE International Conference on Robotics and Automation_ , 2017.
  * [9] J. M. O’Kane and D. A. Shell, “Concise planning and filtering: hardness and algorithms,” _IEEE Transactions on Automation Science and Engineering_ , vol. 14, no. 4, pp. 1666–1681, 2017.
  * [10] J. Ziglar, R. Williams, and A. Wicks, “Context-aware system synthesis, task assignment, and routing,” _arXiv preprint arXiv:1706.04580_ , 2017.
  * [11] J. M. Bern, K.-H. Chang, and S. Coros, “Interactive design of animated plushies,” _ACM Transactions on Graphics_ , vol. 36, no. 4, p. 80, 2017.
  * [12] D. Cellucci, R. MacCurdy, H. Lipson, and S. Risi, “1d printing of recyclable robots,” _IEEE Robotics and Automation Letters_ , vol. 2, no. 4, pp. 1964–1971, 2017.
  * [13] A. Mehta, J. DelPreto, and D. Rus, “Integrated codesign of printable robots,” _Journal of Mechanisms and Robotics_ , vol. 7, no. 2, 2015.
  * [14] N. Cross, “Designerly ways of knowing: Design discipline versus design science,” _Design issues_ , vol. 17, no. 3, pp. 49–55, 2001.
  * [15] Open Source Robotics Foundation, “XML robot description format (URDF),” 2012, accessed 19-March-2018. [Online]. Available: http://wiki.ros.org/urdf/XML/model
  * [16] T. R. Green and M. Petre, “When visual programs are harder to read than textual programs,” in _Proc. European Conference on Cognitive Ergonomics_ , 1992.
  * [17] T. W. Price and T. Barnes, “Comparing textual and block interfaces in a novice programming environment,” in _Proceedings of the eleventh annual International Conference on International Computing Education Research_. ACM, 2015, pp. 91–99.
  * [18] T. Tosun, G. Jing, H. Kress-Gazit, and M. Yim, “Computer-aided compositional design and verification for modular robots,” in _Robotics Research_. Springer, 2018, pp. 237–252.
  * [19] D. M. Aukes, B. Goldberg, M. R. Cutkosky, and R. J. Wood, “An analytic framework for developing inherently-manufacturable pop-up laminate devices,” _Smart Materials and Structures_ , vol. 23, no. 9, p. 094013, 2014.
  * [20] D. M. Aukes and R. J. Wood, “PopupCAD: a tool for automated design, fabrication, and analysis of laminate devices,” in _Micro-and Nanotechnology Sensors, Systems, and Applications_ , 2015.
  * [21] E. Miranda, _Composing music with computers_. CRC Press, 2001.