# Supporting Information

STREAMLINE: An Automated Machine Learning Pipeline for Biomedicine Applied to
Examine the Utility of Photography-Based Phenotypes for OSA Prediction Across
International Sleep Centers

Supporting Information

Ryan J. Urbanowicz1,4*, Harsh Bandhey1, Brendan T. Keenan2, Greg Maislin2, Sy
Hwang3, Danielle L. Mowery4, Shannon M. Lynch5, Diego R. Mazzotti6,7, Fang
Han8, Qing Yun Li9, Thomas Penzel10, Sergio Tufik11, Lia Bittencourt11,
Thorarinn Gislason12,13, Philip de Chazal14, Bhajan Singh15,16, Nigel
McArdle15,16, Ning-Hung Chen17, Allan Pack2, Richard J. Schwab2, Peter A.
Cistulli18, Ulysses J. Magalang19

1Department of Computational Biomedicine, Cedars Sinai Medical Center, Los
Angeles, CA
2Division of Sleep Medicine, Department of Medicine, University of
Pennsylvania, Philadelphia, PA
3Institute for Biomedical Informatics, University of Pennsylvania,
Philadelphia, PA
4Department of Biostatistics Epidemiology and Informatics, University of
Pennsylvania, Philadelphia, PA
5Cancer Prevention and Control Program, Fox Chase Cancer Center, Philadelphia,
PA
6Division of Medical Informatics, Department of Internal Medicine, University
of Kansas, Lawrence, KS
7Division of Pulmonary, Critical Care, and Sleep Medicine, Department of
Internal Medicine, University of Kansas, Lawrence, KS
8Division of Sleep Medicine, Peking University People’s Hospital, Beijing,
China
9Department of Respiratory and Critical Care Medicine, Ruijin Hospital,
Shanghai Jiao Tong University School of Medicine, Shanghai, China
10Interdisciplinary Center of Sleep Medicine, Charite University Hospital,
Berlin, Germany
11Department of Psychobiology, Federal University of Sao Paulo, Sao Paulo,
Brazil
12Department of Sleep Medicine, Landspitali University Hospital, Reykjavik,
Iceland
13University of Iceland, Faculty of Medicine, Reykjavik, Iceland
14Charles Perkins Center, Faculty of Engineering, University of Sydney,
Sydney, Australia
15Department of Pulmonary Physiology and Sleep Medicine, Sir Charles Gairdner
Hospital, Nedlands, Australia
16University of Western Australia, Crawley, Australia
17Department of Pulmonary and Critical Care Medicine, Sleep Center, Linkou
Chang Gung Memorial Hospital, Taoyuan, Taiwan
18Charles Perkins Centre, Faculty of Medicine and Health, University of Sydney
and Royal North Shore Hospital, Sydney, Australia
19Division of Pulmonary, Critical Care, and Sleep Medicine, Department of
Medicine, Ohio State University Wexner Medical Center, Columbus, OH

*Corresponding author: R.J.U<EMAIL_ADDRESS>
700 N. San Vicente Blvd., Pacific Design Center Suite G541E, Los Angeles, CA
90069

section[0em] subsection[2em] subsubsection[4em]

###### Contents

  1. S.1: AutoML Survey/Comparison
  2. S.2 Materials & Methods
    1. S.2.1 STREAMLINE Detailed Pipeline Walkthrough
      1. S.2.1.3: Phase 3 - Feature Importance Estimation
      2. S.2.1.4: Phase 4 - Feature Selection
      3. S.2.1.6: Phase 6 - Post-Analysis
      4. S.2.1.7: Phase 7 - Compare Datasets
      5. S.2.1.8: Phase 8 - Replication
      6. S.2.1.9: Phase 9 – Summary Report
    2. S.2.2: Benchmark Datasets
      1. S.2.2.3: Multiplexer Benchmark Data
      2. S.2.2.4: XOR Benchmark Data
    3. S.2.3.1: Development and Replication Datasets
    4. S.2.3.2: OSA Data Analyses with STREAMLINE
    5. S.2.4: Study-Wide STREAMLINE Run Parameters
  3. S.3 Results
    1. S.3.1: HCC Benchmark Results
      1. S.3.1.1: Simulated ‘Custom’ HCC Validates Automated Data Processing
      2. S.3.1.2: Original HCC Benchmark Results
    2. S.3.2: GAMETES Benchmark Results
    3. S.3.4: XOR Benchmark Results
    4. S.3.5: Obstructive Sleep Apnea Results
      1. S.3.5.2: Automated Data Processing
  4. S.4 Discussion
    1. S.4.1 Benchmarking Challenges

## S.1: AutoML Survey/Comparison

We surveyed a total of 19 automated machine learning (AutoML) tools and
libraries (including STREAMLINE). “Tools” represent a more self-contained
(i.e. ready-to-run) AutoML approach with little to no pipeline implementation
required, while “libraries” generally are designed to simplify the coding
required to design and run a machine learning analysis pipeline (but can
include automation of some aspects of the pipeline (e.g. algorithm selection
and hyperparameter optimization being most common).

We limited our survey to include popular and relevant open-source (i.e non-
enterprise) AutoML since they are free to access/use and their documentation
and capabilities are largely transparent. We also limited our survey to AutoML
designed (at minimum) to analyze structured data (i.e. tabular data) for
binary classification tasks (but most could also do multi-class and regression
tasks).

This survey focused on documentation provided within the respective GitHub
repositories of each AutoML approach, including README’s, User’s Guides, and
other documentation links within each repository. In each case we focused on
the current release (as of September 2023) in the ‘main’ branch of each
repository. Table S1 summarized basic information on each of the 24 AutoML
tools surveyed.

Table S2 provides a high-level comparison of the scope and capabilities of
each AutoML approach including (1) the data types they can work with, (2) the
AutoML objective(s), e.g. classification, regression, etc., (3) aspects
impacting ease-of-use, convenience, and parallizability for those with and
without programming experience, and (4) what aspects of the ML analysis
pipeline have some degree of automation. Some of these characterizations rely
on our best subjective interpretation of the respective AutoML documentation,
as to whether the AutoML approach automates (to at least some degree) the
specified element (i.e. row in the table). This table does not capture many
nuances of the different AutoML approaches, including; (1) the quality and
efficiency of their implementation (2) the quality and degree of detail
provided in the documentation, (3) other uncommon or more specific
capabilities and (4) the degree to which each capability is satisfied, e.g.
how many options are available?, how much user input is required for setup?,
what assumptions are being made? To date, we have only directly used
STREAMLINE, TPOT, and ALIRO, thus characterizations of other approaches
exclusively rely on their documentation, therefore we apologize to developers
if we have inadvertently mischaracterized any of these capabilities, and would
be happy to receive feedback. Note that the characterization of STREAMLINE’s
capabilities reflects those most recent release used in this current
publication (0.3.4), rather than the original published version (0.2.5).

light-gray AutoML Tool/Library |  Release Version |  Initial Release |  Latest Release |  ML Package Reliance | Code Language | Code Link | Paper Citation
---|---|---|---|---|---|---|---
STREAMLINE |  0.3.4 |  2022 |  2023 |  sklearn | python | https://github.com/UrbsLab/STREAMLINE | (1)
Auto-Keras |  1.1.0 |  2023 |  2023 |  keras | python | https://github.com/keras-team/autokeras | (2)
H20-3 |  NA |  2023 |  2023 |  | java | https://github.com/h2oai/h2o-3 | NA
MLme |  NA |  2023 |  2023 |  sklearn | python | https://github.com/FunctionalUrology/Mlme | (3)
LAMA |  0.3.7.3 |  2021 |  2023 |  sklearn | python | https://github.com/sb-ai-lab/LightAutoML | (4)
FEDOT |  0.7.2 |  2021 |  2023 |  sklearn | python | https://github.com/aimclub/FEDOT | (5)
FLAML |  2.1.0 |  2020 |  2023 |  Apache-Spark | python | https://github.com/microsoft/FLAML | NA
ALIRO |  0.19 |  2020 |  2023 |  sklearn | javascript, python | https://github.com/EpistasisLab/Aliro | (6)
PYCARET |  3.1.0 |  2020 |  2023 |  sklearn | python | https://github.com/pycaret/pycaret | NA
Auto-Gluon |  0.8.2 |  2020 |  2023 |  sklearn | python | https://github.com/autogluon/autogluon | (7)
MLIJAR-supervised |  1.0.2 |  2019 |  2023 |  sklearn | python | https://github.com/mljar/mljar-supervised | NA
Ludwig |  0.8.3 |  2019 |  2023 |  | python | https://github.com/ludwig-ai/ludwig/ | (8)
TPOT |  0.12.1 |  2016 |  2023 |  sklearn | python | https://github.com/EpistasisLab/tpot | (9, 10, 11)
Auto-Sklearn |  0.15.0 |  2015 |  2023 |  sklearn | python | https://github.com/automl/auto-sklearn | (12)
Auto-PyTorch |  0.2.1 |  2021 |  2022 |  Py-Torch | python | https://github.com/automl/Auto-PyTorch | (13)
GAMA |  23.0.0 |  2020 |  2022 |  | python | https://github.com/openml-labs/gama | (14)
Hyperopt-sklearn |  NA |  2014 |  2022 |  sklearn | python | https://github.com/hyperopt/hyperopt-sklearn | (15)
Auto-WEKA |  2.6.4 |  2013 |  2022 |  WEKA | java | https://github.com/automl/autoweka | (16)
RECIPE |  NA |  2017 |  2021 |  sklearn | python, C, JavaScript | https://github.com/laic-ufmg/Recipe | (17)
ML-Plan |  0.2.3 |  2018 |  2020 |  sklearn, WEKA | java | https://github.com/starlibs/AILibs | (18)
TransmogrifAI |  0.7.0 |  2018 |  2020 |  Apache-Spark | Scala | https://github.com/salesforce/TransmogrifAI | NA
MLBox |  0.8.1 |  2017 |  2019 |  sklearn | python | https://github.com/AxeldeRomblay/MLBox | (19)
Xcessiv |  0.5.1 |  2017 |  2017 |  sklearn | javascript, python | https://github.com/reiinakano/xcessiv | NA
Auto_ML |  2.7.0 |  2016 |  2017 |  sklearn | python | https://github.com/ClimbsRocks/auto_ml | NA

Table S1: Summary of AutoML Tools and Libraries Surveyed. Key logistical
information for popular and relevant AutoML tools and/or libraries. Not all
methods included a publication or pre-print by the developers (but were still
included). Methods are sorted in descending year based on latest release and
initial release (with STREAMLINE at the top for convenience). Empty cells
indicate that we were unsure or unable to confirm the respective value.

Below we explain our definitions of each capability category (note that the
documentation for different AutoML’s sometimes use different terms and term
definitions):

  * •

‘Data Types’: Indicates whether respective data types can be utilized.

  * •

‘Target’: Indicates what ML tasks can be automated. Note that anomaly
detection (as an objective) is different than outlier detection (as a data
cleaning element). Also note that Auto-sklearn appeared to be unique in its
capability to handle multi-label tasks and PYCARET appeared to be unique in
its capability to handle anomaly detection.

  * •

‘Ease of Use’:

    * –

‘Codeless implementation: Identifies whether the user is required to utilize
any level of coding to implement the pipeline itself, i.e. to set up the
analysis pipeline, specify algorithms or hyperparameters, utilize (or not)
different elements of an ML pipeline, and in what order.

    * –

‘Codeless running: Identifies whether has the capability to run the AutoML
from a GUI, Web-interface, or a preconfigured notebook, at most having to
adjust pipeline run parameters and file paths, and not requiring command line
usage. I.e. is the AutoML able to be used by someone with no coding
experience.

    * –

‘GUI’: Is there a downloadable or web-based graphical user interface? Note
this does not necessarily mean that the AutoML can be used without coding
(which is the case for Xcessiv).

    * –

‘Generates Figures/Viz’: Does the tool provide automation for the generation
of results figures, model vizualizations, and or other comparisons? This can
either be run automatically as part of the pipeline, or automated secondarily
when asked for by the user.

    * –

‘Outputs/organizes results’: The approach automates or facilitates generating
/downloading/ saving/ accessible-archiving of individual models, performance
results, and visualizations. The focus here is allowing the user to easily
examine and take a deeper dive into their results.

    * –

‘Summary report’: The approach automates or facilitates the generation and
presentation of some type of global analysis summary, e.g. documenting run
parameters, key results, key visualizations, etc.

    * –

‘Recommender system’: The approach includes some higher-level AI or decision
making to suggest or decide what analysis configurations to try.

    * –

‘Parallelizable’: Is there a clear and documented approach in place to
parallelize the analysis, either on a compute cluster, multi-core CPU, or GPU.

  * •

‘Automated Pipeline Elements’:

    * –

‘Pipeline config. discovery’: The approach provides some automation for
determining/discovering what pipeline elements to include and in what order –
beyond some pre-specified pipeline of elements.

    * –

‘Feature Extraction: Automates the extraction of structured features from
unstructured data (e.g. text, images, time-series, etc) so that the structured
features can be analyzed by algorithms designed to run on structured data
only.

    * –

‘Exploratory Data Analysis’: Automates multiple aspects of an exploratory data
analysis (without requiring manual implementation) including but not limited
to the data dimensions, data missingness, feature types, outcome balance or
distribution, univariate analysis, feature correlations, clustering, etc.

    * –

‘Categorical Encoding’: Based on automatic detection or user specification of
feature types (e.g. categorical vs. quanatiative), any categorical features
are automatically appropriately encoded for the user (e.g. one-hot-encoding),
so that they are treated appropriately in ML modeling.

    * –

‘Numerical Encoding’: Can the pipeline automatically work with data features
that include text-values (e.g. Hair color – ‘brown’, ‘black’, ‘blond’, ‘red’),
and ideally encode and treat ordinal vs. categorical features appropriately.
Users may still need to specify a given feature’s type as this often requires
domain knowledge.

    * –

‘Basic Data Cleaning’: Is there automation of some basic data cleaning
elements (excluding missing value imputation), e.g. removing rows with missing
outcome, removing redundant instances or features (i.e. perfect/high
correlation), outlier detection/removal, etc.

    * –

‘Feature Engineering’: This term is often confusingly used synonymously with
feature extraction, feature learning, and even feature transformation. In
modern days its most accurately defined as utilizing domain knowledge to
manually ‘engineer’ new features from the dataset (without using knowledge of
the target outcome), e.g. taking drug start/stop dates and engineering a drug
duration variable. Here we consider feature engineering in a slightly more
flexible manner – that does not necessarily require specific problem domain
knowledge (but is still distinct from feature extraction, feature, learning,
and feature transformation). We define it here as the generation of new
features from existing structured features (which may or may not then be
removed from the dataset). Here we consider this to include elements of data
encoding (including one-hot-encoding, where multiple new features replace an
original feature), adding new features generated by dimensionality reduction
(e.g. PCA), multivariate interaction encoding, or missing value encoding to
explore feature missingness as a predictive variable. The key here is that new
features are added to the dataset, and these features were generated without
knowledge of the target outcome.

    * –

‘Missing Val. Imputation’: Automatically imputes or is capable of ignoring
missing values in the dataset so that modeling can be applied. Could be simple
mean/median/mode imputation or more sophisticated multi-variate imputation.

    * –

‘Feature Transformation’: One or more features in the dataset can be
automatically transformed, e.g. normalization, scaling, or any other
transformation that does not directly impact the dimensionality of the dataset
(i.e. the number of features in the dataset)

    * –

‘Feature Learning’: Prior to ML modeling, the approach automatically learns
new features that are some combination of existing features, where knowledge
of the outcome is available. I.e. generating new informative features that can
help simpler algorithms identify more complex relationships in the data, e.g.
Multi-Factor Dimensionality reduction, or using earlier baseline models as
inputs for downstream modeling (similar but not necessarily the same as
‘stacking’, a type of ensembling)

    * –

‘Feature Selection’: The AutoML automates ‘feature selection’ (the removal of
non or less informative features) prior to modeling (i.e. we do not consider
any feature selection internally conducted by a given ML modeling algorithm to
be relevant here)

    * –

‘Algorithm Selection’: The AutoML automates the utilization and comparison of
multiple ML algorithms for modeling. Different AutoML approaches were found to
have access to a widely varying number of potential algorithms to apply
(anywhere from 2, to all algorithms available in a given ML library, e.g.
scikit-learn). Additionally while some AutoML’s allowed the user to manually
add other algorithms (including STREAMLINE), we do not consider this as a
criteria here.

    * –

‘Deep Learning Included’: The AutoML includes the availability of some form of
deep learning, beyond a standard artificial neural network (ANN) that can be
extended beyond a depth of three layers in a hyperparameter sweep. E.g.
STREAMLINE can be adapted to look beyond 3 layers using scikit-learns ANN,
however we do not consider it as including deep learning here.

    * –

‘Hyperparam. Optimization’: Does the AutoML allow automatic search, seeking to
optimize one or more hyperparameters of the included ML algorithms? We
consider neural architecture search to be included here. This is one of the
most commonly automated elements of AutoML.

    * –

‘Models for each CV part.’: This aspect is focused on comparing the models
generated by different ML algorithms. Specifically are individual models, each
trained on a respective CV training partition, saved/output/evaluated. Most
AutoML tools use internal CV either for the hyperparmeter optimization, or to
internally estimate model generalizability before either being evaluated on a
hold out dataset (i.e. which could be termed evaluation, testing, or
replication data). However this aspect asks if individual models are output by
the autoML as part of a more global CV analysis, rather than a single model.

    * –

‘Model Evaluation’: Does the AutoML automate model evaluation (either on a
testing partition, or globally held out data, e.g. replication data) across a
variety of metrics, and easily present or summarize these results for the
user? This differs from the ‘summary report’ in that it is specifically
focused on summarizing and presenting model performance metrics either upon
command request, within a GUI, or saved as output.

    * –

‘Ensembling’: Does the AutoML automate some form of ensembling (e.g. voting,
bagging, stacking), apart from what could be considered ‘feature learning’.
More specifically is the AutoML capable of generating an ensemble model using
some or all other models trained by the different ML modeling algorithms. This
could be across CV partitions or one model for each algorithm. Ideally this
capability would also include comparing ensemble model performance to models
generated by individual ML algorithms (but this was not always the case).

    * –

‘Significance Comparisons’: Does the AutoML automatically conduct statistical
significance analysis comparing model and or dataset performance? STREAMLINE
appears to be unique in this regard.

    * –

‘Directly Compare Datasets’: Does the AutoML support running multiple datasets
through the pipeline at once and facilitate ML performance comparisons between
those multiple datasets?

    * –

‘Replication Analysis’: Does the AutoML tool make it easy to re-evaluate a
model on some additional holdout or new data (including target outcome
information), beyond the primary hold out data. This often includes
saving/archiving the trained model(s), and ideally should include applying the
same data processing that was applied to the training data (e.g. imputation,
feature transformations, feature engineering, feature selection, etc.) to
ensure the model is applicable to that new data. This was difficult to clearly
determine from the documentation of most other AutoMLs.

    * –

‘Model Deploy (Predictions)’: Does the AutoML make it easy to apply the model
to making predictions on data that does not include target outcome
information, e.g. making predictions for a competition, or in deploying the
model to make real-world predictions (e.g. clinical decision making, or risk
prediction). Ideally this would also include some form of explanation
indicating the factors or reasoning from the model that led to a given
prediction.

Table S2: A High-Level Comparison of AutoML Tools and Libraries. 24 AutoML
methods are compared with respect to various aspects of capabilities and scope
of automation. ‘YES’ entries are highlighted in green, and ‘NO’ entries are
highlighted in red. Ambiguous entries and those that we could not confidently
determine from the documentation are left white and/or blank. Note, that it
should not be concluded from this table that AutoMLs with more capabilities
are necessarily better with respect to overall performance, ease of use, or
adoption of data science best practices (in general or for specific
applications).

## S.2 Materials & Methods

### S.2.1 STREAMLINE Detailed Pipeline Walkthrough

In this section we explain: (1) what STREAMLINE does, (2) what happens in
during each phase, (3) why it’s designed the way it has been, (4) what user
options are available to customize a run, and (5) what to expect when running
a given phase. Phases 1-6 make up the core automated pipeline, with Phase 7
and beyond being run optionally based on user needs. Phases are organized to
both encapsulate related pipeline elements, as well as to address practical
computational needs. STREAMLINE includes reliable default run parameters so
that it can easily be used ’as-is’, but these parameters can be adjusted for
further customization. We refer to a single run of the entire STREAMLINE
pipeline as an ’experiment’, with all outputs saved to a single ’experiment
folder’ for later examination and re-use.

To avoid confusion on ’dataset’ terminology we briefly review our definitions
here:

1\. Target dataset \- A whole dataset (minus any instances the user may wish
to hold out for replication) that has not yet undergone any other data
partitioning and is intended to be used in the training and testing of models
within STREAMLINE. Could also be referred to as the ’development dataset’.

2\. Training dataset \- A generally larger partition of the target dataset
used in training a model

3\. Testing dataset \- A generally smaller partition of the target dataset
used to evaluate the trained model

4\. Validation dataset \- The temporary, secondary hold-out partition of a
given training dataset used for hyperparameter optimization. This is the
product of using nested (aka double) cross-validation in STREAMLINE as a
whole.

5\. Replication dataset \- Further data that is withheld from STREAMLINE
phases 1-7 to (1) compare model evaluations on the same hold-out data and (2)
verify the replicability and generalizability of model performance on data
collected from other sites or sample populations. A replication dataset should
have at least all of the features present in the target dataset which it seeks
to replicate.

In the sections below, specific STREAMLINE run parameters are indicated by
italics. Further details on STREAMLINE installation, run modes, run
instructions, run parameters, output files, and additional code to do more
with the output is given in the STREAMLINE documentation (
https://urbslab.github.io/STREAMLINE/index.html).

#### S.2.1.1: Phase 1 - Data Exploration & Processing

This phase; (1) provides the user with key information about the target
dataset(s) they wish to analyze, via an initial exploratory data analysis
(EDA) (2) numerically encodes any text-based feature values in the data, (3)
applies basic data cleaning and feature engineering to process the data, (4)
informs the user how the data has been changed by the data processing, via a
secondary, more in-depth EDA, and then (5) partitions the data using k-fold
cross validation.

  * •

Parallelizability: Runs once for each target dataset to be analyzed

  * •

Run Time: Typically fast, except when evaluating and visualizing feature
correlation in datasets with a large number of features

##### Initial EDA

Characterizes the orignal dataset as loaded by the user, including: data
dimensions, feature type counts, missing value counts, class balance, other
standard pandas data summaries (i.e. describe(), dtypes(), nunique()) and
feature correlations (pearson).

For precision, we strongly suggest users identify which features in their data
should be treated as categorical vs. quantitative using the
categorical_feature_path and/or quantitative_feature_path run parameters.
However, if not specified by the user, STREAMLINE will attempt to
automatically determine feature types relying on the categorical_cutoff
parameter. Any features with fewer unique values than categorical_cutoff will
be treated as categorical, and all others will be treated as quantitative.

  * •

Output: (1) CSV files for all above data characteristics, (2) bar plot of
class balance, (3) histogram of missing values in data, (4) feature
correlation heatmap

##### Numerical Encoding of Text-based Features

Detects any features in the data with non-numeric values and applies scikit-
learn’s ‘LabelEncoder’ to make them numeric as required by scikit-learn
machine learning packages.

##### Basic Data Cleaning and Feature Engineering

Applies the following steps to the target data, keeping track of changes to
all data counts along the way:

1\. Remove any instances that are missing an outcome label (as these cannot be
used while conducting supervised learning)

2\. Remove any features identified by the user with ignore_features_path (a
convenience for users that may wish to exclude one or more features from the
analysis without changing the original dataset)

3\. Engineer/add ’missingness’ features. Any original feature with a missing
value proportion greater than featureeng_missingness will have a new feature
added to the dataset that encodes missingness with 0 = not missing and 1 =
missing. This allows the user to examine whether missingness is not at random
(MNAR) and is predictive of outcome itself.

4\. Remove any features that have invariant values (i.e. they are always the
same), or that have only one value in addition to missing values. Then remove
any features with a missingness greater than cleaning_missingness. Afterwards,
remove any instances in the data that may have a missingness greater than
cleaning_missingness.

5\. Engineer/add one-hot-encoding for any categorical features in the data.
This ensures that all categorical features are treated as such throughout all
aspects of the pipeline. For example, a single categorical feature with 3
possible states will be encoded as 3 separate binary-valued features
indicating whether an instance has that feature’s state or not. Feature names
are automatically updated by STREAMLINE to reflect this change.

6\. Remove highly correlated features based on correlation_removal_threshold.
Randomly removes one feature of a highly correlated feature pair (Pearson).
While perfectly correlated features can be safely cleaned in this way, there
is a chance of information loss when removing less correlated features.

  * •

Output: CSV file summarizing changes to data counts during these cleaning and
engineering steps.

##### Processed Data EDA

Completes a more comprehensive EDA of the processed dataset including:
everything examined in the initial EDA, as well as a univariate association
analysis of all features using Chi-Square (for categorical features), or Mann-
Whitney U-Test (for quantitative features).

  * •

Output: (1) CSV files for all above data characteristics, (2) bar plot of
class balance, (3) histogram of missing values in data, (4) feature
correlation heatmap, (5) a CSV file summarizing the univariate analyses
including the test applied, test statistic, and p-value for each feature, (6)
for any feature with a univariate analysis p-value less than sig_cutoff (i.e.
significant association with outcome), a bar-plot will be generated if it is
categorical, and a box-plot will be generated if it is quantitative.

##### k-fold Cross Validation (CV) Partitioning

For k-fold CV, STREAMLINE uses ’Stratified’ partitioning by default, which
aims to maintain the same/similar class balance within the ’k’ training and
testing datasets. The value of ’k’ can be adjusted with cv_partitions.
However, using partition_method, users can also select ’Random’ or ’Group’
partitioning.

Of note, ’Group’ partitioning requires the dataset to include a column
identified by match_label. This column includes a group membership identifier
for each instance which indicates that any instance with the same group ID
should be kept within the same partition during cross validation. This was
originally intended for running STREAMLINE on epidemiological data that had
been matched for one or more covariates (e.g. age, sex, race) in order to
adjust for their effects during modeling.

This strategy of performing k-fold cross validation in order to train (and
ultimately evaluate k models for each algorithm is different from most other
AutoML tools, which typically only use k-fold CV internally as a way to try
and optimize model generalizability during optimization prior to final hold
out data evaluation. This approach has the advantage of yielding more than one
model to later allow statistical comparisons between algorithms and datasets,
but it has the disadvantage of having a smaller number of instances being used
to optimize the model(s) during training/optimization.

  * •

Output: CSV files for each training and testing dataset generated following
partitioning. Note, by default STREAMLINE will overwrite these files as the
working datasets undergo imputation, scaling and feature selection in
subsequent phases. However, the user can keep copies of these intermediary CV
datasets for review using the parameter overwrite_cv.

#### S.2.1.2: Phase 2 - Imputation and Scaling

This phase conducts additional data preparation elements of the pipeline that
occur after CV partitioning, i.e. missing value imputation and feature
scaling. Both elements are ’trained’ and applied separately to each individual
training dataset. The respective testing datasets are not looked at when
running imputation or feature scaling learning to avoid potential data
leakage. However, the learned imputation and scaling patterns are applied in
the same way to the testing data as they were in the training data. Both
imputation and scaling can optionally be turned off using the parameters
impute_data and scale_data, respectively for some specific use cases, however
imputation must be on when missing data is present in order to run most
scikit-learn modeling algorithms, and scaling should be on for certain
modeling algorithms learn effectively (e.g. artificial neural networks), and
for if the user wishes to infer feature importances directly from certain
algorithm’s internal estimators (e.g. logistic regression).

  * •

Parallelizability: Runs ’k’ times for each target dataset being analyzed
(where k is number of CV partitions)

  * •

Run Time: Typically fast, with the exception of imputing larger datasets with
many missing values

##### Imputation

This phase first conducts imputation to replace any remaining missing values
in the dataset with a ’value guess’. While missing value imputation could be
reasonably viewed as data manufacturing, it is a common practice and viewed
here as a necessary ’evil’ in order to run scikit-learn modeling algorithms
downstream (which mostly require complete datasets). Imputation is completed
prior to scaling since it can influence the correct center and scale to be
used.

Missing value imputation seeks to make a reasonable, educated guess as to the
value of a given missing data entry. By default, STREAMLINE uses ’mode
imputation’ for all categorical values, and multivariate imputation for all
quantitative features. However, for larger datasets, multivariate imputation
can be slow and require a lot of memory. Therefore, the user can deactivate
multiple imputation with the multi_impute parameter, and STREAMLINE will use
median imputation for quantitative features instead.

##### Scaling

Second, this phase conducts feature scaling with scikit-learn’s
‘StandardScalar’ to transform features to have a mean at zero with unit
variance. This is only necessary for certain modeling algorithms, but it
should not hinder the performance of other algorithms. The primary drawback to
scaling prior to modeling is that any future data applied to the model will
need to be scaled in the same way prior to making predictions. Furthermore,
for algorithms that have directly interpretable models (e.g. decision tree),
the values specified by these models need to be un-scaled in order to
understand the model in the context of the original data values. STREAMLINE
includes a ‘Useful Notebook’ that can generate direct model visualizations for
decision tree and genetic programming models. This code automatically un-
scales the values specified in these models so they retain their
interpretability.

  * •

Output: (1) Learned imputation and scaling strategies for each training
dataset are saved as pickled objects allowing any replication or other future
data to be identically processed prior to running it through the model. (2) If
overwrite_cv is False, new imputed and scaled copies of the training and
testing datasets are saved as CSV output files, otherwise the old dataset
files are overwritten with these new ones to save space.

#### S.2.1.3: Phase 3 - Feature Importance Estimation

This phase applies feature importance estimation algorithms (i.e. Mutual
information (MI) and MultiSURF (20), found in the ReBATE software package)
used as filter-based feature selection algorithms. Both algorithms are run by
default, however the user can deactivate either using do_mutual_info or
do_multisurf, respectively. MI scores features based on their univariate
association with outcome, while MultiSURF scores features in a manner that is
sensitive to both univariate and epistatic (i.e. multivariate feature
interaction) associations.

For datasets with a larger number of features (i.e. > 10,000) we recommend
turning on the TuRF (21) wrapper algorithm with use_turf, which has been shown
to improve the sensitivity of MultiSURF to interactions, particularly in
larger feature spaces. Users can increase the number of TuRF iterations (and
performance) by decreasing turf_pct from 0.5, to approaching 0. However, this
will significantly increase MultiSURF run time.

Overall, this phase is important not only for subsequent feature selection,
but as an opportunity to evaluate feature importance estimates prior to
modeling outside of the initial univariate analyses (conducted on the entire
dataset). Further, comparing feature rankings between MI, and MultiSURF can
highlight features that may have little or no univariate effects, but that are
involved in epistatic interactions that are predictive of outcome.

  * •

Parallelizability: Runs ’k’ times for each algorithm (MI and MultiSURF) and
each target dataset being analyzed (where k is number of CV partitions)

  * •

Run Time: Typically reasonably fast, but takes more time to run MultiSURF, in
particular as the number of training instances approaches the default
instance_subset parameter of 2000 instances, or if this parameter set higher
in larger datasets. This is because MultiSURF scales quadratically with the
number of training instances.

  * •

Output: CSV files of feature importance scores for both algorithms and each CV
partition ranked from largest to smallest scores.

#### S.2.1.4: Phase 4 - Feature Selection

This phase uses the feature importance estimates learned in the prior phase to
conduct feature selection using a ’collective’ feature selection approach
(22). By default, STREAMLINE will remove any features from the training data
that scored 0 or less by both feature importance algorithms (i.e. features
deemed uninformative). Users can optionally ensure retention of all features
prior to modeling by setting filter_poor_features to False. Users can also
specify a maximum number of features to retain in each training dataset using
max_features_to_keep (which can help reduce overall pipeline runtime and make
learning easier for modeling algorithms). If after removing ’uninformative
features’ there are still more features present than the user specified
maximum, STREAMLINE will pick the unique top scoring features from one
algorithm then the next until the maximum is reached and all other features
are removed. Any features identified for removal from the training data are
similarly removed from the testing data.

  * •

Parallelizability: Runs ’k’ times for each target dataset being analyzed
(where k is number of CV partitions)

  * •

Run Time: Fast

  * •

Output: (1) CSV files summarizing feature selection for a target dataset (i.e.
how many features were identified as informative or uninformative within each
CV partition) and (2) a barplot of mean feature importance scores (across CV
partitions). The user can specify the maximum number of top scoring features
to be plotted using top_fi_features.

#### S.2.1.5: Phase 5 - Machine Learning (ML) Modeling

At the heart of STREAMLINE, this phase conducts (1) machine learning modeling
using the training data, (2) model feature importance estimation (also with
the training data), and (3) model evaluation on testing data. STREAMLINE
uniquely includes 3 rule-based machine learning algorithms: ExSTraCS, XCS, and
eLCS. These ’learning classifier systems’ have been demonstrated to be able to
detect complex associations while providing human interpretable models in the
form of IF:THEN rule-sets. The ExSTraCS algorithm was developed by our
research group to specifically handle the challenges of scalability, noise,
and detection of epistasis and genetic heterogeneity in biomedical data
mining.

  * •

Parallelizability: Runs ’k’ times for each algorithm and each target dataset
being analyzed (where k is number of CV partitions)

  * •

Run Time: Slowest phase, but can be sped up by reducing the set of ML methods
selected to run, or deactivating ML methods that run slowly on large datasets

##### Model Selection

The following 16 scikit-learn compatible ML modeling algorithms are currently
included as options:

  1. 1.

Naive Bayes (NB) (23) (scikit-learn)

  2. 2.

Logistic Regression (LR) (24) (scikit-learn)

  3. 3.

Elastic Net (EN) (25) (scikit-learn)

  4. 4.

Decision Tree (DT) (26) (scikit-learn)

  5. 5.

Random Forest (RF) (27) (scikit-learn)

  6. 6.

Gradient Boosting (GB) (28) (scikit-learn)

  7. 7.

XGBoost (XGB) (29) (scikit-learn compatible)

  8. 8.

LGBoost (LGB) (30) (scikit-learn compatible)

  9. 9.

CatBoost (CGB) (31) (scikit-learn compatible)

  10. 10.

Support Vector Machine (SVM) (32) (scikit-learn)

  11. 11.

Artificial Neural Network (ANN) (33) (scikit-learn)

  12. 12.

K-Nearest Neighbors (k-NN) (34) (scikit-learn)

  13. 13.

Genetic Programming (GP) (35) (GPLearn – scikit-learn compatible)

  14. 14.

Educational Learning Classifier System (eLCS) (36) (scikit-learn compatible)

  15. 15.

‘X’ Classifier System (XCS) (37) (scikit-learn compatible)

  16. 16.

Extended Supervised Tracking and Classifying System (ExSTraCS) (38) (scikit-
learn compatible)

NB, LR, EN, DT, RF, GB, SVM, ANN, and k-NN are all part of the standard
scikit-learn package. XGB, LGB, CGB, GP, eLCS, XCS, and ExSTraCS are all
separately imported scikit-learn compatible packages.

The first step is to decide which modeling algorithms to run. By default,
STREAMLINE applies 14 of the 16 algorithms (excluding eLCS and XCS) it
currently has built in. eLCS and XCS are currently experimental
implementations of rule-based ML algorithms. Users can specify a specific
subset of algorithms to run using algorithms, or alternatively indicate a list
of algorithms to exclude from all available algorithms using exclude.
STREAMLINE is also set up so that more advanced users can add other scikit-
learn compatible modeling algorithms to run within the pipeline (as explained
in ‘Adding New Modeling Algorithms’ in the documentation). This allows
STREAMLINE to be used as a rigorous framework to easily benchmark new modeling
algorithms in comparison to other established algorithms.

Modeling algorithms vary in the implicit or explicit assumptions they make,
the manner in which they learn, how they represent a solution (as a model),
how well they handle different patterns of association in the data, how long
they take to run, and how complex and/or interpretable the resulting model can
be. To reduce runtime in datasets with a large number of training instances,
users can specify a limited random number of training instances to use in
training algorithms that run slowly in such datasets using training_subsample.

In the STREAMLINE demo, we run only the fastest/simplest three algorithms
(Naive Bayes, Logistic Regression, and Decision Tree), however these
algorithms all have known limitations in their ability to detect complex
associations in data. We encourage users to utilize a wider variety of the 14
established available algorithms, in their analyses, to give STREAMLINE the
best opportunity to identify a best performing model for the given task (which
is effectively impossible to predict for a given dataset ahead of time).

We recommend users utilize at least the following set of algorithms within
STREAMLINE: Naive Bayes, Logistic Regression, Decision Tree, Random Forest,
XGBoost, SVM, ANN, and ExSTraCS as we have found these to be a reliable set of
algorithms with an array of complementary strengths and weaknesses on
different problems. ExSTraCS is a rule-based machine learning algorithm in the
sub-family of algorithms called learning classifier systems (LCS) that has
been specifically adapted to the challenges of biomedical data analysis (38).

##### Hyperparameter Optimization

Most machine learning algorithms have a variety of hyperparameter options that
influence how the algorithm runs and performs on a given dataset. In designing
STREAMLINE we sought to identify the full set of important hyperparameters for
each algorithm, along with a comprehensive range of possible settings and
hard-coded these into the pipeline. STREAMLINE adopts the ‘Optuna’ package to
conduct automated Bayesian optimization of hyperparameters for most algorithms
by default. The evaluation metric ’balanced accuracy’ is used to optimize
hyperparameters as it takes class imbalance into account and places equal
weight on the accurate prediction of both class outcomes. However, users can
select an alternative metric with primary_metric and whether that metric needs
to be maximized or minimized using the metric_direction parameter. To conduct
the hyperparameter sweep, Optuna splits a given training dataset further,
applying 3-fold cross validation to generate further internal training and
validation partitions with which to evaluate different hyperparameter
combinations.

Users can also configure how Optuna operates in STREAMLINE with n_trials and
timeout which controls the target number of hyperparameter value combination
trials to conduct, as well as how much total time to try and complete these
trials before picking the best hyperparameters found. To ensure
reproducibility of the pipeline, note that timeout should be set to ‘None’
(however this can take much longer to complete depending on other pipeline
settings).

Notable exceptions to most algorithms; Naive Bayes has no parameters to
optimize, and rule-based (i.e. LCS) ML algorithms including ExSTraCS, eLCS,
and XCS can be computationally expensive thus STREAMLINE is set to use their
default hyperparameter settings without a sweep unless the user updates
do_lcs_sweep and lcs_timeout. While these LCS algorithms have many possible
hyperparameters to manipulate they have largely stable performance when
leaving most of their parameters to default values. Exceptions to this include
the key LCS hyperparameters (1) number of learning iterations, (2) maximum
rule-population size, and (3) accuracy pressure (nu), which can be manually
controlled without a hyperparameter sweep by lcs_iterations, lcs_N, and
lcs_nu, respectively.

  * •

Output: CSV files specifying the optimized hyperparameter settings found by
Optuna for each partition and algorithm combination.

##### Train ’Optimized’ Model

Having selected the best hyperparameter combination identified for a given
training dataset and algorithm, STREAMLINE now retrains each model on the
entire training dataset using those respective hyperparameter settings. This
yields a total of ’k’ potentially ’optimized’ models for each algorithm.

  * •

Output: All trained models are pickled as python objects that can be loaded
and applied later.

##### Model Feature Importance Estimation

Next, STREAMINE estimates and summarizes model feature importance scores for
every algorithm run. This is distinct from the initial feature importance
estimation phase, in that these estimates are specific to a given model as a
useful part of model interpretation/explanation. By default, STREAMLINE
employes scikit-learn’s ‘permutation feature importance’ for estimating
feature importances scores in the same uniform manner across all algorithms.
Some ML algorithms that have a build in strategy to gather model feature
importance estimates (i.e. LR,DT,RF,XGB,LGB,GB,eLCS,XCS,ExSTraCS). The user
can instead use these estimates by setting use_uniform_fi to ‘False’. This
will direct STREAMLINE to report any available internal feature importance
estimate for a given algorithm, while still utilizing permutation feature
importance for algorithms with no such internal estimator.

  * •

Output: All feature importance scores are pickled as python objects for use in
the next phase of the pipeline.

##### Evaluate Performance

The last step in this phase is to evaluate all trained models using their
respective testing datasets. A total of 16 standard classification metrics
calculated for each model including: balanced accuracy, standard accuracy, F1
Score, sensitivity (recall), specificity, precision (positive predictive
value), true positives, true negatives, false postitives, false negatives,
negative predictive value, likeliehood ratio positive, likeliehood ratio
negative, area under the ROC, area under the PRC, and average precision of
PRC.

  * •

Output: All evaluation metrics are pickled as python objects for use in the
next phase of the pipeline.

#### S.2.1.6: Phase 6 - Post-Analysis

This phase combines all modeling results to generate summary statistics files,
generate results plots, and conduct non-parametric statistical significance
analysis comparing ML performance across CV runs.

  * •

Output (Evaluation Metrics):

    * –

Testing data evaluation metrics for each CV partition - CSV file for each
modeling algorithm

    * –

Mean testing data evaluation metrics for each modeling algorithm - CSV file
for mean, median, and standard deviation

    * –

ROC and PRC curves for each CV partition in contrast with the average curve -
ROC and PRC plot for each modeling algorithm

    * –

The generation of these plots can be turned off with exclude_plots

    * –

Summary ROC and PRC plots - compares average ROC or PRC curves over all CV
partitions for each modeling algorithm

    * –

Boxplots for each evaluation metric - comparing algorithm performance over all
CV partitions

      * *

The generation of these plots can be turned off with exclude_plots

  * •

Output (Model Feature Importance):

    * –

Model feature importance estimates for each CV partition - CSV file for each
modeling algorithm

    * –

Boxplots comparing feature importance scores for each CV partition - CSV file
for each modeling algorithm

    * –

Composite feature importance barplots illustrating mean feature importance
scores across all algorithms (2 versions)

      * *

Feature importance scores are normalized prior to visualization

      * *

Feature importance scores are normalized and weighted by mean model
performance metric (balanced accuracy by default)

        * ·

The metric used to weight this plot can be changed with metric_weight

    * –

Number of top scoring features illustrated in feature importance plots is
controlled by top_model_fi_features

  * •

Output (Statistical Significance):

    * –

Kruskal Wallis test results assessing whether there is a significant
difference for each performance metric among all algorithms - CSV file

      * *

For any metric that yields a significant difference based on sig_cutoff,
pairwise statistical tests between algorithms will be conducted using both the
Mann Whitney U and the Wilcoxon Rank Test.

    * –

Pairwise statistical tests between algorithms using both the Mann Whitney
U-test and the Wilcoxon Rank Test - CSV file for each statistic and
significance test.

    * –

Parallelizability: Runs once for each target dataset being analyzed.

    * –

Run Time: Moderately fast - turning off some figure generation can make this
phase faster

#### S.2.1.7: Phase 7 - Compare Datasets

This phase should be run when STREAMLINE was applied to more than one target
dataset during the ’experiment’. It applies further non-parametric statistical
significance testing between datasets to identify if performance differences
were observed among datasets comparing (1) the best performing algorithms or
(2) on an algorithm-by-algorithm basis. It also generates plots to compare
performance across datasets and examine algorithm performance consistency.

  * •

Parallelizability: Runs once - not parallelizable

  * •

Run Time: Fast

  * •

Output (Comparing Best Performing Algorithms for each Metric):

    * –

Kruskal Wallis test results assessing whether there is a significant
difference between median CV performance metric among all datasets focused on
the best performing algorithm for each dataset - CSV file

      * *

For any metric that yields a significant difference based on sig_cutoff
pairwise statistical tests between datasets will be conducted using both the
Mann Whitney U-test and the Wilcoxon Rank Test

    * –

Pairwise statistical tests between datasets focused on the best performing
algorithm for each dataset using both the Mann Whitney U-test and the Wilcoxon
Rank Test - CSV file for each significance test.

  * •

Output (Comparing Algorithms Independently):

    * –

Kruskal Wallis test results assessing whether there is a significant
difference for each median CV performance metric among all datasets - CSV file
for each algorithm

      * *

For any metric that yields a significant difference based on sig_cutoff,
pairwise statistical tests between datasets will be conducted using both the
Mann Whitney U-test and the Wilcoxon Rank Test

    * –

Pairwise statistical tests between datasets using both the Mann Whitney U-test
and the Wilcoxon Rank Test - CSV file for each significance test and algorithm

#### S.2.1.8: Phase 8 - Replication

This phase of STREAMLINE is only run when the user has further hold out data ,
i.e. one or more ’replication’ datasets, which will be used to re-evaluate all
models trained on a given target dataset. This means that this phase would
need to be run once to evaluate the models of each original target dataset
Notably, this phase would be the first time that all models are evaluated on
the same set of data which is useful for more confidently picking a ’best’
model and further evaluating model generalizability and it’s ability to
replicate performance on data collected at different times, sites, or
populations.

To run this phase the user needs to specify the filepath to the target dataset
to be replicated with dataset_for_rep as well as the folderpath to the folder
containing one or more replication datasets using rep_data_path.

This phase begins by conducting an initial exploratory data analysis (EDA) on
the new replication dataset(s), followed by uniquely processing versions of
the replication dataset(s) to correspond with each original CV training
dataset, yielding the same number of features (but not necessarily the same
number of instances). This processing includes cleaning of instances in
replication data that have no outcome or with a missingness over the user-
defined threshold. Then to ensure that the same features exist in the
replication data that existed in the respective model’s original training set,
feature engineering creates the same one-hot-encoded categorical features, as
well as the same ‘missingness features’ previously generated. If a new
categorical feature values is observed (that didn’t exist in the original
training data for one-hot-encoding) this new value is treated as ‘missing’ in
replication data by assigning 0’s to all one hot encoded values for that
feature. Similarly, with respect to feature cleaning, the same features
removed in respective training data are removed from the corresponding
replication dataset (i.e. features marked for exclusion or features removed
due to high missingness, high feature correlation, or removed during feature
selection. Then EDA is repeated to confirm processing of the replication
dataset and generate a feature correlation heatmap, however univariate
analyses are not repeated on the replication data.

Next all models previously trained for the target dataset are re-evaluated
across all metrics using each replication dataset with results saved
separately. Similarly, all model evaluation plots (with the exception of
feature importance plots) are automatically generated. As before non-
parametric statistical tests are applied to examine differences in algorithm
performance.

  * •

Parallelizability: Runs once for each replication dataset being analyzed for a
given target dataset.

  * •

Run Time: Moderately fast

  * •

Output: Similar outputs to Phase 1 minus univariate analyses, and similar
outputs to Phase 6 minus feature importance assessments.

#### S.2.1.9: Phase 9 – Summary Report

This final phase generates a pre-formatted PDF report summarizing (1)
STREAMLINE run parameters, (2) key exploratory analysis for the processed
target data, (3) key ML modeling results (including metrics and feature
importances), (4) dataset comparisons (if relevant), (5) key statistical
significance comparisons, and (6) runtime. STREAMLINE collects run-time
information on each phase of the pipeline and for the training of each ML
algorithm model.

Separate reports are generated representing the findings from running Phases
1-7, i.e. ’Testing Data Evaluation Report’, as well as for Phase 8 if run on
replication data, i.e. ’Replication Data Evaluation Report’.

  * •

Parallelizability: Runs once - not parallelizable

  * •

Run Time: About a minute

  * •

Output: One or more PDF reports

### S.2.2: Benchmark Datasets

Much of the text describing the GAMETES and x-bit multiplexer datasets below
directly quotes our previous STREAMLINE paper [1], which we are replicating in
the benchmarking analyses below. Note that since the STREAMLINE codebase has
changed, even though we are using the same random seed for these analyses, the
CV partitions will not be the same as previous analyses, so the results are
not expected to be exactly the same (i.e. we expect to see different
performance for different algorithms and different CV partitions due to
differences in data partitioning as well as stochastic differences in
hyperparameter optimization sweep). However, we expect similar overall
performance.

#### S.2.2.1: HCC Benchmark Data

##### Real-World HCC Dataset

The first benchmarking dataset (hcc_data.csv) is an example of a real-world
biomedical classification task. This is a Hepatocellular Carcinoma (HCC)
dataset taken from the UCI Machine Learning Repository. It includes 165
instances, 49 features, and a binary class label. It also includes a mix of
categorical and quantitative features (however all categorical features are
binary), about 10% missing values, and class imbalance, i.e. 63 deceased
(class = 1), and 102 survived (class 0).

##### Custom Simulated HCC Dataset

The second benchmarking dataset (hcc_data_custom.csv) is similar to the first,
but we have made a number of modifications to it in order to test the data
cleaning and feature engineering functionalities of STREAMLINE.

Modifications include the following:

  * •

Removal of covariate features (i.e. ‘Gender’ and ‘Age at diagnosis’)

  * •

Addition of two simulated instances with a missing class label (to test basic
data cleaning)

  * •

Addition of two simulated instances with a high proportion of missing values
(to test instance missingness data cleaning)

  * •

Addition of three simulated, numerically encoded categorical features with 2,
3, or 4 unique values, respectively (to test one-hot-encoding)

  * •

Addition of three simulated, text-valued categorical features with 2, 3, or 4
unique values, respectively (to test one-hot-encoding of text-based features)

  * •

Addition of three simulated quantiative features with high missingness (to
test feature missingness data cleaning)

  * •

Addition of three pairs of correlated quanatiative features (6 features added
in total), with correlations of -1.0, 0.9, and 1.0, respectively (to test high
correlation data cleaning)

  * •

Addition of three simulated features with (1) invariant feature values, (2)
all missing values, and (3) a mix of invariant values and missing values.

These simulated features and instances have been clearly identified in the
feature names and instances IDs of this dataset.

##### Custom Simulated HCC Replication Dataset

The last demo dataset (hcc_data_custom_rep.csv) was simulated as a mock
replication dataset for hcc_data_custom.csv. To generate this dataset we first
took hcc_data_custom.csv and for 30% of instances randomly generated realistic
looking new values for each feature and class outcome (effectively adding
noise to this data). Furthermore, we simulated further instances that test the
ability of STREAMLINE’s one-hot-encoding to ignore new (as-of-yet unseen)
categorical features values during STREAMLINE’s replication phase. If this
were to happen, the new value would be ignored (i.e. no new feature columns
added).

Modifications included adding a simulated instance that includes a new (as-of-
yet unseen) value for the following previously simulated features:

  * •

The binary text-valued categorical feature

  * •

The 3-value text-valued categorical feature

  * •

The binary numerically encoded categorical feature

  * •

The 3-value numerically encoded categorical feature

  * •

We also added a new, previously unseen feature value to each of the invariant
feature columns.

The code to generate the additional features and instances within the custom
hcc_data_custom.csv and hcc_data_-custom_rep.csv can be found in the notebook
at /data/Generate_Expanded_HCC_dataset.ipynb.

##### HCC Analysis With STREAMLINE

We applied STREAMLINE to analyze and compare both the HCC and custom HCC
datasets. For the replication phase we evaluated the custom HCC dataset using
it’s corresponding simulated custom replication dataset to validate the
efficacy of replication data preparation and model evaluation. See the first
page of ‘hcc_exp_STREAMLINE_Report.pdf’ in the supplementary materials for all
STREAMLINE run parameters used in this analysis.

We repeated this analysis twice in parallel using a fixed number of Optuna
trials and no time limit to confirm the replicability of STREAMLINE.

#### S.2.2.2: GAMETES Benchmark Data

We evaluated 6 genomics datasets simulated with GAMETES (39) using a newer
version of the code extended to simulated additive and heterogeneous
associations (https://github.com/UrbsLab/GAMETES/tree/v2.2). Each dataset was
modeled with a different underlying pattern of association to illustrate how
STREAMLINE can be utilized to evaluate the strengths and weaknesses of
different ML algorithms in different contexts. Each dataset simulated 100
single nucleotide polymorphisms (SNPs) as features and included 1600 instances
using a minor allele frequency of 0.2 for relevant features and the ‘easiest’
model architecture generated for each configuration using the GAMETES
simulation approach. All irrelevant features were randomly simulated with a
minor allele frequency between 0.05 and 0.5. We detail dataset differences
here (A) Univariate association between a single feature and outcome with 1
relevant and 99 irrelevant features and a heritability of 0.4, (B) Additive
combination of 4 univariate associations with 4 relevant and 96 irrelevant
features and heritability of 0.4 for each relevant feature, (C) Heterogeneous
combination of 4 univariate associations, i.e. each univariate association is
only predictive in a respective quarter of instances (with all else the same
as dataset B), (D) Pure 2-way feature interaction with 2 relevant and 98
irrelevant features and heritability of 0.4, (E) Heterogeneous combination of
2 independent pure 2-way feature interactions with 4 relevant and 96
irrelevant features and heritability of 0.4 for each 2-way interaction (used
in above sections for figure examples), and (F) Pure 3-way feature interaction
with 3 relevant and 97 irrelevant features and heritability of 0.2 (i.e. the
most difficult dataset).

##### GAMETES Analysis With STREAMLINE

We applied STREAMLINE to analyze and compare all 6 GAMETES datasets, treating
all features as quantitative unless it had only two possible values in the
training (largely in-line with the original 0.2.5 STREAMLINE analysis which
was not set up to apply one-hot-encoding). No replication data was applied in
this analysis as the ground truth in these dataset simulations is known. See
the first page of ‘gametes_exp_STREAMLINE_Report.pdf’ in the supplementary
materials for all STREAMLINE run parameters used in this analysis.

When STREAMLINE conducts modeling in the datasets discussed so far, ExSTraCS,
which was trained without a hyperparameter sweep, is set to run with
hyperparameters (nu = 1, rule population size =2K, and training iterations =
200K), where the nu setting represents the emphasis on discovering and keeping
rules with maximum accuracy. Previous work suggests that nu = 1 is more
effective in noisy data, while a nu of 5 or 10 is more effective in clean data
(i.e. no noise) (38).

#### S.2.2.3: Multiplexer Benchmark Data

We applied STREAMLINE to 6 different x-bit multiplexer (MUX) binary
classification benchmark datasets (i.e. 6, 11, 20, 37, 70, and 135-bit) often
utilized to evaluate ML algorithms such as RBML (20, 36, 37, 38). Like GAMETES
dataset ‘E’, these MUX datasets involve both feature interactions and
heterogeneous associations with outcome, but differently involve binary
features and a clean association with outcome. The ‘x’-bit value denotes the
number of relevant features underlying the association, and increasing values
from 6 to 135 dramatically scale up the complexity of the underlying pattern
of association. Specifically, solving the 6-bit problem involves modeling 4
independent 3-way interactions, while the 135-bit problem involves modeling

128 independent 8-way interactions. In these datasets, all features are
relevant to solving the problem, where a subset of features serves as ‘address
bits’ which point to one of the remaining ‘register bits’. The value of that
corresponding register bit indicates the correct outcome (i.e. 0 or 1). No
replication data was applied in this analysis as the ground truth in these
dataset simulations is known.

Previous work benchmarking MultiSURF on these datasets indicated that these
address bits would yield the largest feature importance scores of all features
in the dataset, as these features are important for the correct prediction of
every instance in the dataset (20).

In other previous work, the ExSTraCS algorithm was the first ML demonstrated
to directly model the 135-bit problem successfully, in a dataset including 40K
training instances, with hyperparameters (nu = 10, rule population size =10K,
and training iterations = 1.5 million) (38). ExSTraCS has the advantage over
other ML algorithms of using the feature importance estimates from MultiSURF
In this study, 6 to 135-bit datasets were generated with 500, 1000, 2000,
5000, 10000, and 20000 instances, respectively, with 90% of each used by
STREAMLINE for training (due to 10-fold CV). In previous work, it was
demonstrated that larger numbers of instances are required to solve or nearly
solve increasingly complex MUX problems. In contrast with modeling on the
noisy datasets, for the MUX datasets, ExSTraCS was assigned the following
hyperparameter settings: nu = 10, rule population size =5K, and training
iterations = 500K. As such, we do not expect ExSTraCS to be able to solve the
135-bit problem in this analysis.

#### S.2.2.4: XOR Benchmark Data

We evaluated 4 simulated ‘XOR’ datasets with models manually generated and
respective datasets simulated with GAMETES [39] using
(https://github.com/UrbsLab/GAMETES/tree/v2.2). This benchmark was not
previously applied to STREAMLINE v0.2.5. These datasets are designed to
examine the capability of ML algorithms (and the STREAMLINE pipeline as a
whole) to detect increasingly high dimensional pure n-way interactions (i.e.
2-way, 3-way, 4-way, and 5-way). Each dataset simulated 20 single nucleotide
polymorphisms (SNPs) as features and included 1600 instances. Unlike the other
GAMETES datasets (described above), but like the multiplexer datasets, these
data are simulated to have a perfectly clean signal, such that it is possible
to train models with 100% testing accuracy. These are the same XOR datasets
previously used to benchmark the ReBATE suite of Relief-based feature
selection algorithms, including ‘MultiSURF (20). All features in this analysis
were treated as quantitative unless it had only two possible values in the
training (largely in-line with the original 0.2.5 STREAMLINE analysis which
was not set up to apply one-hot-encoding). No replication data was applied in
this analysis as the ground truth in these dataset simulations is known.

In that previous work, we found that MultiSURF failed prioritize the correct
predictive features in the case of 4-way and 5-way pure interactions. More
specifically those feature actually scored the most ‘negative’ of all feature
values. Traditionally, Relief-based algorithms are applied by prioritizing the
most positively scoring features, and or removing features with a score <= 0,
and this is how we have implemented feature selection with MulitSURF in
STREAMLINE. However, based on the results in [20] as well as in the current
study (see S.3.4), we hypothesize that Relief-based algorithms should
prioritize features based on the absolute value of their scores in order to
detect higher order interactions (i.e. 4-way interactions and above).
Preliminary work supports this hypothesis, and future work will seek to
improve STREAMLINE’s current feature selection strategy to better account for
the possible presence of high order interactions (which currently most
research and benchmarking of algorithms does not explicitly consider), by
taking the absolute value of MultiSURF scoring into account.

### S.2.3: Obstructive Sleep Apnea Data

Here we provide a detailed summary of the SAGIC OSA dataset a total of 3111
instances, unique instance IDs, class/outcome (encoded by AHI threshold), and
features including demographics (DEM), comorbidities (DX), symptoms (SYM),
craniofacial measures from photos (CF), and intraoral measures from photos
(IO). Table S3 provides summary statistics for original SAGIC dataset. Table
S4 provides a key explaining all features, their group, type and any
underlying encoding. A total of 85 features are available in datasets with all
feature groups. The two Mallampati score classifications include “Frontal,
Tongue Extended Maximally” (FTEM), and “Tongue in Mouth, No Phonation”
(TIMNPH) (40).

Table 1: * gray!20Feature |  gray!20Count |  gray!20Missing Values |  gray!20Unique Values |  gray!20Mean |  gray!20Standard Deviation |  gray!20Min |  gray!2025% |  gray!20Median |  gray!2075% |  gray!20Max
---|---|---|---|---|---|---|---|---|---|---
subnum |  3111 |  0 |  3111 |  NA |  NA |  NA |  NA |  NA |  NA |  NA
class (AHI$\geq$15) |  3111 |  0 |  2 |  0.45323 |  0.497888 |  0 |  0 |  0 |  1 |  1
class (AHI$\geq$5) |  3111 |  0 |  2 |  0.688525 |  0.463171 |  0 |  0 |  1 |  1 |  1
age |  3111 |  0 |  73 |  49.97653 |  13.93849 |  17 |  39 |  50 |  60 |  89
male |  3111 |  0 |  2 |  0.604629 |  0.489009 |  0 |  0 |  1 |  1 |  1
bmi |  3111 |  0 |  2426 |  29.75954 |  7.049058 |  17.03923 |  25.01352 |  28.3737 |  32.92469 |  75.30469
caucasian |  3104 |  7 |  2 |  0.308956 |  0.462137 |  0 |  0 |  0 |  1 |  1
aframer |  3104 |  7 |  2 |  0.037371 |  0.1897 |  0 |  0 |  0 |  0 |  1
asian |  3104 |  7 |  2 |  0.378222 |  0.485021 |  0 |  0 |  0 |  1 |  1
csamer |  3104 |  7 |  2 |  0.209407 |  0.406951 |  0 |  0 |  0 |  0 |  1
other |  3104 |  7 |  2 |  0.066044 |  0.248399 |  0 |  0 |  0 |  0 |  1
medhx_hbp |  2537 |  574 |  2 |  0.44659 |  0.497237 |  0 |  0 |  0 |  1 |  1
medhx_cad |  2517 |  594 |  2 |  0.097735 |  0.297016 |  0 |  0 |  0 |  0 |  1
medhx_heart_fail |  2520 |  591 |  2 |  0.04127 |  0.198953 |  0 |  0 |  0 |  0 |  1
medhx_stroke |  2526 |  585 |  2 |  0.034046 |  0.181383 |  0 |  0 |  0 |  0 |  1
medhx_ldl |  2489 |  622 |  2 |  0.451185 |  0.497711 |  0 |  0 |  0 |  1 |  1
medhx_dm |  2525 |  586 |  2 |  0.127129 |  0.333183 |  0 |  0 |  0 |  0 |  1
medhx_afib |  2293 |  818 |  2 |  0.060183 |  0.237878 |  0 |  0 |  0 |  0 |  1
loud_snore |  2571 |  540 |  5 |  2.054842 |  1.729045 |  0 |  0 |  2 |  4 |  4
gasping |  2554 |  557 |  5 |  1.287001 |  1.558197 |  0 |  0 |  0 |  3 |  4
choke |  2554 |  557 |  5 |  1.100235 |  1.512753 |  0 |  0 |  0 |  2 |  4
map_index |  2582 |  529 |  15 |  1.487284 |  1.3244 |  0 |  0 |  1.333333 |  2.666667 |  4
upper_face_depth |  3108 |  3 |  86 |  10.0705 |  0.983102 |  6.7 |  9.4 |  10.1 |  10.7 |  14.9
mid_face_depth |  3106 |  5 |  85 |  10.7872 |  0.980476 |  7.8 |  10.1 |  10.8 |  11.4 |  15.1
low_face_depth |  3110 |  1 |  94 |  12.91204 |  1.149739 |  9.5 |  12.1 |  12.9 |  13.6 |  19
max_depth_angle |  3104 |  7 |  355 |  84.44965 |  6.580233 |  8.9 |  80.2 |  84.4 |  88.7 |  107.6
mand_depth_angle |  3108 |  3 |  337 |  77.01638 |  6.260158 |  8.9 |  72.8 |  76.95 |  81.3 |  97.6
max_mand_rel_angle |  3106 |  5 |  191 |  7.445419 |  2.978375 |  -3.3 |  5.5 |  7.3 |  9.4 |  18.3
facial_axis_angle |  3108 |  3 |  417 |  30.97631 |  7.939419 |  6.1 |  25.5 |  30.7 |  36.4 |  60.8
photo_upper_face_ht |  3109 |  2 |  60 |  7.211444 |  0.626629 |  5.3 |  6.8 |  7.2 |  7.6 |  9.9
nose_ht |  3105 |  6 |  52 |  4.995208 |  0.501182 |  3.4 |  4.7 |  5 |  5.3 |  11.4
photo_low_face_ht |  3107 |  4 |  67 |  6.168555 |  0.695872 |  3.9 |  5.7 |  6.1 |  6.6 |  9.3
mand_length |  3108 |  3 |  89 |  7.77684 |  1.028879 |  4.4 |  7.1 |  7.7 |  8.4 |  13
mand_length_dia |  3108 |  3 |  85 |  9.548668 |  1.001957 |  6.6 |  8.9 |  9.5 |  10.1 |  13.8
post_mand_ht |  3110 |  1 |  74 |  6.737971 |  0.947212 |  3.7 |  6.1 |  6.7 |  7.4 |  10.5
mand_plane_angle |  3109 |  2 |  422 |  11.29391 |  7.808057 |  -15.9 |  6 |  11.3 |  16.3 |  46.5
photo_mand_angle |  3073 |  38 |  512 |  114.158 |  10.24584 |  10.1 |  108.6 |  114.3 |  120.1 |  159.5
max_mand_box_area |  3106 |  5 |  479 |  54.66754 |  9.55144 |  10.1 |  47.9 |  54.2 |  60.875 |  105.6
cranial_base_tri_area |  3109 |  2 |  463 |  66.1554 |  9.033765 |  4.2 |  60.2 |  65.5 |  71.6 |  113.9
max_tri_area |  3107 |  4 |  478 |  70.8823 |  9.490196 |  4.3 |  64.4 |  70.4 |  76.7 |  117.1
mand_tri_area |  3111 |  0 |  419 |  39.48905 |  8.109848 |  7.2 |  33.8 |  38.8 |  43.9 |  82
mid_cran_fossa_vol |  3105 |  6 |  889 |  109.475 |  21.77432 |  7.3 |  94.6 |  107.2 |  122.39 |  217.2
head_posit_angle |  3109 |  2 |  386 |  12.65574 |  7.282256 |  -11.2 |  8.2 |  13.2 |  17.5 |  35.5
max_volume |  3104 |  7 |  1344 |  159.8664 |  69.70423 |  5.9 |  130.375 |  152 |  177.6 |  1565.3
mand_vol |  3111 |  0 |  1513 |  142.8338 |  49.46381 |  6.9 |  107.2 |  138.3 |  172.3 |  365.8
max_mand_vol |  3104 |  7 |  1892 |  302.4981 |  94.83212 |  6.9 |  244.3 |  290.2 |  345.15 |  1753.6
ant_mand_ht |  3111 |  0 |  58 |  3.918666 |  0.545831 |  2.1 |  3.6 |  3.9 |  4.3 |  7
man_nas_ang |  3106 |  5 |  290 |  40.04895 |  4.977194 |  8.5 |  36.8 |  39.9 |  43.4 |  60.9
man_sub_ang |  3104 |  7 |  364 |  57.18392 |  6.654066 |  7 |  52.7 |  57.1 |  61.4 |  84.4
hea_inc_ang |  3108 |  3 |  377 |  19.68565 |  7.200343 |  -5.3 |  14.8 |  19.8 |  24.7 |  42
cer_ang |  2836 |  275 |  751 |  154.3691 |  18.35592 |  8.3 |  141.985 |  154.345 |  166.2 |  262.3
face_width |  3111 |  0 |  88 |  13.11694 |  0.954734 |  4.8 |  12.5 |  13.1 |  13.7 |  17.6
mand_width |  3111 |  0 |  97 |  11.00771 |  1.208481 |  5.4 |  10.2 |  10.9 |  11.8 |  17.2
intercant_width |  3099 |  12 |  44 |  3.341488 |  0.402077 |  2.3 |  3.1 |  3.3 |  3.6 |  5.1
biocular_width |  3099 |  12 |  67 |  8.711707 |  0.728143 |  6.7 |  8.2 |  8.6 |  9.1 |  12
nose_width |  3110 |  1 |  44 |  3.952251 |  0.425781 |  2.7 |  3.7 |  3.9 |  4.2 |  6.2
lat_fac_ht |  3107 |  4 |  80 |  10.31225 |  0.984748 |  7.5 |  9.6 |  10.3 |  11 |  14.3
man_wid_len_ang |  3111 |  0 |  405 |  74.33594 |  7.909022 |  10.6 |  68.8 |  73.9 |  79.5 |  103.9
fac_wid_fac_dep_ang |  3107 |  4 |  282 |  62.32664 |  4.798661 |  12.3 |  59.2 |  62.2 |  65.4 |  79.2
fac_wid_low_fac_dep_ang |  3111 |  0 |  258 |  54.6636 |  4.162312 |  12.4 |  51.9 |  54.6 |  57.2 |  71.8
eye_width |  3099 |  12 |  62 |  2.68511 |  0.352662 |  1.65 |  2.45 |  2.65 |  2.85 |  4.15
face_eye_width |  3099 |  12 |  960 |  4.952678 |  0.616253 |  3.2 |  4.529776 |  4.938776 |  5.348332 |  7.210526
face_height_width |  3105 |  6 |  1170 |  0.854631 |  0.101542 |  0.584416 |  0.8 |  0.848 |  0.907143 |  4.270833
mand_width_length |  3108 |  3 |  1333 |  0.711906 |  0.104744 |  0.398438 |  0.642857 |  0.709677 |  0.776596 |  2.407407
face_upper_lower |  3105 |  6 |  611 |  0.816738 |  0.098053 |  0.530864 |  0.75 |  0.8125 |  0.877193 |  1.282051
face_height_depth |  3104 |  7 |  1125 |  1.039679 |  0.100866 |  0.766667 |  0.971963 |  1.031747 |  1.098928 |  2.594937
cranial_mand_area |  3109 |  2 |  2945 |  1.714797 |  0.272733 |  0.583333 |  1.527972 |  1.690073 |  1.87766 |  4.04
face_nose_width |  3110 |  1 |  686 |  3.346333 |  0.33184 |  2.22807 |  3.121951 |  3.333333 |  3.552632 |  5.032258
open_wid_tim_nph |  3085 |  26 |  2909 |  5.66042 |  0.854845 |  3.516 |  5.042679 |  5.599256 |  6.227113 |  10.02625
ver_mou_ope |  3080 |  31 |  2908 |  4.671383 |  0.916921 |  2.141 |  4.012679 |  4.616064 |  5.261602 |  8.531904
open_area_tim_nph |  3080 |  31 |  3040 |  19.71676 |  6.062227 |  4.523 |  15.3255 |  18.88386 |  23.3019 |  47.19128
tongue_width_tim_nph |  2754 |  357 |  2606 |  4.744686 |  0.596651 |  2.179 |  4.36924 |  4.756935 |  5.141194 |  6.713906
open_wid_f_te_m |  3000 |  111 |  2838 |  5.820935 |  0.715308 |  3.688 |  5.322976 |  5.79759 |  6.286839 |  9.03
tongue_length_f |  2955 |  156 |  2820 |  5.382951 |  1.107808 |  1.28297 |  4.6385 |  5.378722 |  6.129566 |  9.477
tongue_width_f_te_m |  3026 |  85 |  2831 |  5.035716 |  0.600539 |  3.145094 |  4.626886 |  5.02608 |  5.412897 |  7.64877
ton_are |  2954 |  157 |  2922 |  22.84309 |  6.453854 |  5.264 |  18.3635 |  22.44592 |  26.68929 |  63.407
tongue_length_p_te_m |  3042 |  69 |  2865 |  4.272597 |  0.805157 |  1.36538 |  3.763103 |  4.281134 |  4.799724 |  8.895
tongue_area_p_te_m |  3041 |  70 |  2956 |  5.698665 |  1.979176 |  0.082545 |  4.31 |  5.530136 |  6.842665 |  29.382
tongue_thickness_p_te_m |  3036 |  75 |  2717 |  1.533286 |  0.311344 |  0.48 |  1.324 |  1.511668 |  1.723153 |  2.923578
tongue_curvature_p_te_m |  3042 |  69 |  2915 |  5.012658 |  1.208547 |  1.258 |  4.116 |  4.983812 |  5.860505 |  8.987571
mal_score_ftem_c1 |  2836 |  275 |  2 |  0.048307 |  0.214453 |  0 |  0 |  0 |  0 |  1
mal_score_ftem_c2 |  2836 |  275 |  2 |  0.068759 |  0.253088 |  0 |  0 |  0 |  0 |  1
mal_score_ftem_c3 |  2836 |  275 |  2 |  0.150917 |  0.358031 |  0 |  0 |  0 |  0 |  1
mal_score_ftem_c4 |  2836 |  275 |  2 |  0.732017 |  0.442987 |  0 |  0 |  1 |  1 |  1
mal_score_timnph_c1 |  2943 |  168 |  2 |  0.06524 |  0.24699 |  0 |  0 |  0 |  0 |  1
mal_score_timnph_c2 |  2943 |  168 |  2 |  0.067618 |  0.251132 |  0 |  0 |  0 |  0 |  1
mal_score_timnph_c3 |  2943 |  168 |  2 |  0.086646 |  0.281364 |  0 |  0 |  0 |  0 |  1
mal_score_timnph_c4 |  2943 |  168 |  2 |  0.780496 |  0.413981 |  0 |  1 |  1 |  1 |  1
Table S3: Summary Statistics for SAGIC Obstructive Sleep Apnea Data. Includes the entire original dataset prior to replication data partitioning, and internal STREAMLINE k-fold CV partitioning. |  |  |  |  |  |  |  |  |  |

Table 2: * gray!20Column Name |  gray!20Feature Group |  gray!20Description |  gray!20Feature Type/Encoding
---|---|---|---
subnum |  NA |  unique SAGIC instance ID including site identifier |  NA
class (AHI$\geq$15) |  NA |  OSA binary outcome encoded by AHI$\geq$15 or AHI$\geq$5 |  Binary/Categorical (0=No OSA, 1=OSA)
class (AHI$\geq$5) |  NA |  OSA binary outcome encoded by AHI$\geq$5 |  Binary/Categorical (0=No OSA, 1=OSA)
age |  DEM |  What is your age? |  Quantitative
male |  DEM |  Are you male? |  Binary/Categorical (0=Female,1=Male)
bmi |  DEM |  Body Mass Index kg/m**2 |  Quantitative
caucasian |  DEM |  Caucasian Race/Ethnicity |  Binary/Categorical(0=Not, 1=Caucasian)
aframer |  DEM |  African American Race/Ethnicity |  Binary/Categorical(0=Not, 1=African Am.)
asian |  DEM |  Asian Race/Ethnicity |  Binary/Categorical(0=Not, 1=Asian)
csamer |  DEM |  Central/South American Race/Ethnicity |  Binary/Categorical(0=Not, 1=Cen/South Am.)
other |  DEM |  Other Race/Ethnicity |  Binary/Categorical(0=Not, 1=Other)
medhx_hbp |  DX |  Have you been diagnosed with high blood pressure? |  Binary/Categorical(0=No, 1=Yes)
medhx_cad |  DX |  Ever Diagnosed with CAD (Angina, MI, Heart Attack)? |  Binary/Categorical(0=No, 1=Yes)
medhx_heart_fail |  DX |  Have you ever been diagnosed with heart failure? |  Binary/Categorical(0=No, 1=Yes)
medhx_stroke |  DX |  Have you ever had a stroke? |  Binary/Categorical(0=No, 1=Yes)
medhx_ldl |  DX |  Have you ever been told that you have high cholesterol and/or triglycerides? |  Binary/Categorical(0=No, 1=Yes)
medhx_dm |  DX |  Have you been diagnosed with diabetes? |  Binary/Categorical(0=No, 1=Yes)
medhx_afib |  DX |  Have you ever been diagnosed with atrial fibrillation? |  Binary/Categorical(0=No, 1=Yes)
loud_snore |  SYM |  Frequency of loud snoring symptom from the Multivariable Apnea Prediction (MAP) Index |  Ordinal/Quantitative: 0 = None/Don’t Know, 1 = Rarely (less than once a week), 2 = Sometimes (1-2 times a week), 3 = Frequently (3-4 times a week), 4 = Always (5-7 times a week)
gasping |  SYM |  Frequency of snorting or gasping symptom from the Multivariable Apnea Prediction (MAP) Index |  Ordinal/Quantitative: 0 = None/Don’t Know, 1 = Rarely (less than once a week), 2 = Sometimes (1-2 times a week), 3 = Frequently (3-4 times a week), 4 = Always (5-7 times a week)
choke |  SYM |  Frequency of breathing stops or witnessed apneas from the Multivariable Apnea Prediction (MAP) Index |  Ordinal/Quantitative: 0 = None/Don’t Know, 1 = Rarely (less than once a week), 2 = Sometimes (1-2 times a week), 3 = Frequently (3-4 times a week), 4 = Always (5-7 times a week)
map_index |  SYM |  Multivariable Apnea Prediction (MAP) Index Score |  Quantitative (0-4)
upper_face_depth |  CF |  Upper Face Depth |  Quantitative
mid_face_depth |  CF |  Middle Face Depth |  Quantitative
low_face_depth |  CF |  Lower Face Depth |  Quantitative
max_depth_angle |  CF |  Maxillary Depth Angle |  Quantitative
mand_depth_angle |  CF |  Mandibular Depth Angle |  Quantitative
max_mand_rel_angle |  CF |  Maxillary-Mandibular Relationship Angle |  Quantitative
facial_axis_angle |  CF |  Facial Axis Angle |  Quantitative
photo_upper_face_ht |  CF |  Upper Face Height |  Quantitative
nose_ht |  CF |  Nose Height |  Quantitative
photo_low_face_ht |  CF |  Lower Face Height |  Quantitative
mand_length |  CF |  Mandibular Length |  Quantitative
mand_length_dia |  CF |  Mandibular Length Diagonal |  Quantitative
post_mand_ht |  CF |  Posterior Mandibular Height |  Quantitative
mand_plane_angle |  CF |  Mandibular Plane Angle |  Quantitative
photo_mand_angle |  CF |  Mandibular Angle |  Quantitative
max_mand_box_area |  CF |  Maxillary-Mandibular Box Area |  Quantitative
cranial_base_tri_area |  CF |  Cranial Base Triangular Area |  Quantitative
max_tri_area |  CF |  Maxillary Triangular Area |  Quantitative
mand_tri_area |  CF |  Mandibular Triangular Area |  Quantitative
mid_cran_fossa_vol |  CF |  Middle Cranial Fossa Volume |  Quantitative
head_posit_angle |  CF |  Natural Head Position Angle |  Quantitative
max_volume |  CF |  Maxillary Volume |  Quantitative
mand_vol |  CF |  Mandibular Volume |  Quantitative
max_mand_vol |  CF |  Maxillary-Mandibular Volume |  Quantitative
ant_mand_ht |  CF |  Anterior Mandibular Height |  Quantitative
man_nas_ang |  CF |  Mandibular-Nasion Angle |  Quantitative
man_sub_ang |  CF |  Mandibular-Subnasion Angle |  Quantitative
hea_inc_ang |  CF |  Head Base Inclination Angle |  Quantitative
cer_ang |  CF |  Cervicomental Angle |  Quantitative
face_width |  CF |  Face Width |  Quantitative
mand_width |  CF |  Mandibular Width |  Quantitative
intercant_width |  CF |  Intercanthal Width |  Quantitative
biocular_width |  CF |  Biocular Width |  Quantitative
nose_width |  CF |  Nose Width |  Quantitative
lat_fac_ht |  CF |  Lateral Face Height |  Quantitative
man_wid_len_ang |  CF |  Mandibular Width to Length Angle |  Quantitative
fac_wid_fac_dep_ang |  CF |  Face Width to Middle Face Depth Angle |  Quantitative
fac_wid_low_fac_dep_ang |  CF |  Face Width to Lower Face Depth Angle |  Quantitative
eye_width |  CF |  Eye Width ([biocular_width - intercant_width]/2) |  Quantitative
face_eye_width |  CF |  Face Width :: Eye Width |  Quantitative
face_height_width |  CF |  Face Height::Width |  Quantitative
mand_width_length |  CF |  Mandibular Width::Length |  Quantitative
face_upper_lower |  CF |  Upper Face Height::Lower Face Height |  Quantitative
face_height_depth |  CF |  Face Height::Depth |  Quantitative
cranial_mand_area |  CF |  Cranial Mandibular Area |  Quantitative
face_nose_width |  CF |  Face Width :: Nose Width |  Quantitative
open_wid_tim_nph |  IO |  Mouth Opening Width (Tongue in Mouth, No Phonation) |  Quantitative
ver_mou_ope |  IO |  Vertical Mouth Opening (Tongue in Mouth, No Phonation) |  Quantitative
open_area_tim_nph |  IO |  Mouth Opening Area (Tongue in Mouth, No Phonation) |  Quantitative
tongue_width_tim_nph |  IO |  Tongue Width (Tongue in Mouth, No Phonation) |  Quantitative
open_wid_f_te_m |  IO |  Mouse Opening Width (Frontal, Tongue Extended) |  Quantitative
tongue_length_f |  IO |  Tongue Length (Frontal, Tongue Extended) |  Quantitative
tongue_width_f_te_m |  IO |  Tongue Width (Frontal, Tonuge Extended) |  Quantitative
ton_are |  IO |  Tongue Area (Frontal, Tongue Extended) |  Quantitative
tongue_length_p_te_m |  IO |  Tongue Length (Profile, Tongue Extended) |  Quantitative
tongue_area_p_te_m |  IO |  Tongue Area (Profile, Tongue Extended) |  Quantitative
tongue_thickness_p_te_m |  IO |  Tongue Thickness (Profile, Tongue Extended) |  Quantitative
tongue_curvature_p_te_m |  IO |  Tongue Curvature (Profile, Tongue Extended) |  Quantitative
mal_score_ftem_c1 |  IO |  Mallampati Score Class I (FTEM) |  Binary/Categorical (0=No, 1=Yes)
mal_score_ftem_c2 |  IO |  Mallampati Score Class II (FTEM) |  Binary/Categorical (0=No, 1=Yes)
mal_score_ftem_c3 |  IO |  Mallampati Score Class III (FTEM) |  Binary/Categorical (0=No, 1=Yes)
mal_score_ftem_c4 |  IO |  Mallampati Score Class IV (FTEM) |  Binary/Categorical (0=No, 1=Yes)
mal_score_timnph_c1 |  IO |  Mallampati Score Class I (TIMNPH) |  Binary/Categorical (0=No, 1=Yes)
mal_score_timnph_c2 |  IO |  Mallampati Score Class II (TIMNPH) |  Binary/Categorical (0=No, 1=Yes)
mal_score_timnph_c3 |  IO |  Mallampati Score Class III (TIMNPH) |  Binary/Categorical (0=No, 1=Yes)
mal_score_timnph_c4 |  IO |  Mallampati Score Class IV (TIMNPH) |  Binary/Categorical (0=No, 1=Yes)
Table S4: Variable Key - SAGIC Obstructive Sleep Apnea Data |  |  |

lightgray Dataset ID | Feature Set(s) Included | AHI cutoff | Full Dataset Name
---|---|---|---
A | DEM |  AHI$\geq$15 events/hour | A_GE15_DEM
B | DX |  AHI$\geq$15 events/hour | B_GE15_DX
C | SYM |  AHI$\geq$15 events/hour | C_GE15_SYM
D | CF |  AHI$\geq$15 events/hour | D_GE15_CF
E | IO |  AHI$\geq$15 events/hour | E_GE15_IO
F | DEM+DX |  AHI$\geq$15 events/hour | F_GE15_DEMDX
G | DEM+DX+SYM |  AHI$\geq$15 events/hour | G_GE15_DEMDXSYM
H | DEM+DX+SYM+CF |  AHI$\geq$15 events/hour | H_GE15_DEMDXSYMCF
I | DEM+DX+SYM+IO |  AHI$\geq$15 events/hour | I_GE15_DEMDXSYMIO
J | DEM+DX+SYM+CF+IO |  AHI$\geq$15 events/hour | J_GE15_DEMDXSYMCFIO
gray!20 K | DEM |  AHI$\geq$5 events/hour | K_GE5_DEM
gray!20 L | DX |  AHI$\geq$5 events/hour | L_GE5_DX
gray!20 M | SYM |  AHI$\geq$5 events/hour | M_GE5_SYM
gray!20 N | CF |  AHI$\geq$5 events/hour | N_GE5_CF
gray!20 O | IO |  AHI$\geq$5 events/hour | O_GE5_IO
gray!20 P | DEM+DX |  AHI$\geq$5 events/hour | P_GE5_DEMDX
gray!20 Q | DEM+DX+SYM |  AHI$\geq$5 events/hour | Q_GE5_DEMDXSYM
gray!20 R | DEM+DX+SYM+CF |  AHI$\geq$5 events/hour | R_GE5_DEMDXSYMCF
gray!20 S | DEM+DX+SYM+IO |  AHI$\geq$5 events/hour | S_GE5_DEMDXSYMIO
gray!20 T | DEM+DX+SYM+CF+IO |  AHI$\geq$5 events/hour | T_GE5_DEMDXSYMCFIO
Table S5: Constructed SAGIC Datasets for Analysis

#### S.2.3.1: Development and Replication Datasets

Included in the SAGIC dataset instance identifiers is the site/location where
the data was collected.

Below we provide the unique study/data collection site IDs:

  * •

BER = Germany (Berlin)

  * •

BRA = Brazil

  * •

ICE = Iceland

  * •

OSU = USA (Ohio State)

  * •

PEN = USA (UPenn)

  * •

WES = Australia (Perth)

  * •

SYD = Australia (Sydney)

  * •

TAI = Taiwan

  * •

BEI = China (Beijing)

Once the 20 datasets listed in Table S4 were constructed we applied a Jupyter
Notebook ‘OSA_Replication_Data_-Splitter.ipynb’ (available upon request),
which split each of these datasets into respective development and replication
sets. All data partitioning in this study (both here and during internal
STREAMLINE CV partitioning) randomly assigned instances to partitions, i.e.
instances from a given site are not kept together as a hold out evaluation
set, in order to avoid possible underlying site-specific collection
differences/biases. For datasets with the same binary OSA class definition
(i.e. AHI$\geq$15 or AHI$\geq$5), the exact same set of instances comprise the
development and replication sets to ensure within-group comparability.
However, between these two dataset groups, different instances are included in
development and replication sets because different instances are labeled as
having either a 0 (no OSA) or 1 (OSA) outcome with a subsequently different
degree of ‘class balance’.

Specifically, AHI$\geq$15 datasets all have 4̃5.3% OSA (i.e. class 1), and
AHI$\geq$5 datasets all have 6̃8.9% OSA (i.e. class 1). Thus while AHI$\geq$15
datasets are relatively ‘balanced’, AHI$\geq$5 datasets have a more
significant degree of imbalance which is taken into account during model
evaluation. After this initial partitioning, AHI$\geq$15 development datasets
have 2178 instances and replication datasets have 933 instances, while
AHI$\geq$5 development datasets have 2179 instances and replication datasets
have 932 instances.

#### S.2.3.2: OSA Data Analyses with STREAMLINE

Refer to ‘sagic_exp_STREAMLINE_Report.pdf’ (Page 1) for all STREAMLINE run
parameter settings used in this analysis. We ran this complete analysis across
all 20 development datasets using STREAMLINE phases 1-7, and then ran separate
phase 8 replication analyses for each of the 20 respective replication
datasets.

Subsequently, we re-ran phase 7 ‘dataset comparison’ (on it’s own, without
retraining models) on different subsets of the 20 datasets to generate simpler
and more targeted comparisons between (1) AHI$\geq$15 vs. AHI$\geq$5 datasets,
(2) all datasets with just one feature group included, and (3) datasets with
just one feature group included for AHI$\geq$15 vs. AHI$\geq$5 datasets,
separately.

### S.2.4: Study-Wide STREAMLINE Run Parameters

While STREAMLINE run parameters for each ‘experiment’ are documented on the
first page of their respective PDF reports (in supplemental materials), here
we document the relevant STREAMLINE run parameters that were consistent across
all experiments and analyses.

  * •

CV Partitions: 10

  * •

Partition Method: Stratified

  * •

Statistical Significance Cutoff: 0.05

  * •

Random Seed: 42

  * •

Use Mutual Information: True

  * •

Use MultiSURF: True

  * •

Use TURF: False

  * •

Max Features to Keep: 2000

  * •

Top Features to Display: 40

  * •

Export Feature Importance Plot: True

  * •

Engineering Missingness Cutoff: 0.5

  * •

Cleaning Missingness Cutoff: 0.5

  * •

Correlation Removal Threshold: 1.0

  * •

List of Exploratory Analysis Ran: [’Describe’, ’Univariate Analysis’, ’Feature
Correlation’]

  * •

List of Exploratory Plots Saved: [’Describe’, ’Univariate Analysis’, ’Feature
Correlation’]

  * •

Use Data Scaling: True

  * •

Use Data Imputation: True

  * •

Use Multivariate Imputation: True

  * •

Overwrite CV Datasets: True

  * •

Artificial Neural Network: True

  * •

Category Gradient Boosting: True

  * •

Decision Tree: True

  * •

Elastic Net: True

  * •

ExSTraCS: True

  * •

Extreme Gradient Boosting: True

  * •

Genetic Programming: True

  * •

Gradient Boosting: True

  * •

K-Nearest Neighbors: True

  * •

Light Gradient Boosting: True

  * •

Logistic Regression: True

  * •

Naive Bayes: True

  * •

Random Forest: True

  * •

Support Vector Machine: True

  * •

XCS: False

  * •

eLCS: False

  * •

Primary Metric: balanced_accuracy

  * •

Uniform Feature Importance Estimation (Models): True

  * •

Hyperparameter Sweep Number of Trials: 200

  * •

Hyperparameter Timeout: 900

  * •

Stats and Figure Settings:

  * •

Export ROC Plot: True

  * •

Export PRC Plot: True

  * •

Export Metric Boxplots: True

  * •

Export Feature Importance Boxplots: True

  * •

Metric Weighting Composite FI Plots: balanced_accuracy

  * •

Top Model Features To Display: 40

## S.3 Results

Note, that all results figures below and in main text were automatically
generated by STREAMLINE (some were minorly edited in Adobe Illustrator to move
legends or create multi-panel figures. STREAMLINE is also set up with
supporting code in the form of ‘Useful Notebooks’ allowing users to re-
generate plots to their own formatting specifications. Included in these
supplementary materials are PDF reports for each STREAMLINE ‘experiment’.
However, detailed results files, models, and other automatically saved
STREAMLINE output files can also be made available upon request
(ryan.urbanowiz@cshs.org).

### S.3.1: HCC Benchmark Results

#### S.3.1.1: Simulated ‘Custom’ HCC Validates Automated Data Processing

First, examination of the hcc_data_custom data processing results validated
the efficacy of the STREAMLINE phase 1 data processing (i.e. cleaning and
feature engineering. Specifically, all simulated features and instances
designed to be cleaned or engineered were done so correctly. Referring to
‘hcc_exp_STREAMLINE_Report.pdf’ (Page 3) note that hcc_data does not have any
changes to feature or instance count as the result of cleaning or engineering.
However, on (Page 4) we observe the following correct data processing of the
simulated hcc_data_custom dataset:

  * •

C1: 2 instances missing an outcome label are removed (note that class counts
don’t change because those class labels were missing to begin with)

  * •

E1: 3 categorical features are added to the dataset representing newly
engineered missingness features for the 3 features in this dataset with a
missingness > 0.5.

  * •

C2: 6 features are removed (4 categorical and 2 quantitative) including the
original 3 features with high missingness and 3 features with invariant
values.

  * •

C3: 2 instances are removed with missingness > 0.5

  * •

E2: One-hot-encoding replaces 4 non-binary categorical features with binary
features for each feature’s possible categories. Specifically 14 features are
added and 4 are removed yielding a net gain of 10 binary categorical features.

  * •

C4: 2 pairs of features in this data are perfectly correlated. One of each
pair is removed for a net reduction of two quantitative features

Now referring to ‘hcc_exp_STREAMLINE_Replication_Report.pdf’ (Page 2) which
summarizes the replication evaluation of hcc_data_custom using
hcc_data_custom_rep, we correctly observed that C1 removes 3 instances missing
the class label, and R1 processes the rest of this data in the same way as for
hcc_data_custom, yielding the same number of features as well as confirming
Phase 8’s ability to handle as-of-yet unobserved categorical values occurring
in the replication data (i.e. replacing them with NA for binary categorical
features, and adding 0 values for all one-hot-encoded features for that
original feature.

#### S.3.1.2: Original HCC Benchmark Results

Refer to ‘hcc_exp_STREAMLINE_Report.pdf’ (Pages 2,3,5,6,&7) for EDA and
modeling results for the hcc_data dataset. We review major results below.

Figure S1 includes ROC and PRC plots comparing mean algorithm performance. XGB
yielded the best ROC-AUC of 0.800 (0.127). In this dataset there is a
reasonable degree of class imbalance (63 deceased, and 102 survived in the
original dataset) so it is best here to focus instead balanced accuracy and on
PRC-AUC results. EN yielded the best balanced accuracy of 0.728 (0.067) and
SVM yielded the best PRC-AUC of 0.718 (0.107). Figure S2 gives the PRC plot
for SVM comparing each of 10-fold CV models trained, and illustrating
reasonably high variance in performance across partitions. This is reasonably
expected given the small sample size of this dataset and highlights the
importance of evaluating models across CV partitions in order to evaluate the
impact of partitioning-based sample bias. Figure S3 gives permutation-based
model feature importance estimates for each algorithm (averaged across all
10-CV models for each algorithm).

Figure S1: Original HCC Data - Comparing Mean Algorithm Performance. (A) ROC
plot comparing algorithm performance. (B) PRC plot comparing algorithm
performance. For both, each line represents mean algorithm performance across
10 CV models. Area under the curve (AUC) is provided for both, and average
precision (APS) is included for PRC. Figure S2: Original HCC Data - PRC Plot
of SVM performance across individual CV Models.  Figure S3: Original HCC Data
– Mean Model Feature Importance Estimates Across all Algorithms. This
composite feature importance plot illustrates normalized and (balanced
accuracy)-weighted mean FI scores summed across all algorithms. Only the top
40 features are displayed.

Table S6 compares the performance of our previous STREAMLINE HCC benchmarking
(v0.2.5) [1] with the current (v0.3.4). Mean model performances were similar,
if not slightly higher, however the performance metric standard deviation was
also significantly larger, suggesting a larger degree of random partitioning
sample bias than in in the earlier benchmarking run. Replication data was not
available for the HCC benchmarking, and the dataset was too small to
reasonably hold out further data for replication, however these results
suggest that this new release of STREAMLINE is performing as expected,
yielding similar top performance metrics, and similar top model feature
importance estimates.

light-gray | v0.2.5 | v0.3.4
---|---|---
light-gray Metric | Best Algorithm | Benchmark Testing Evaluation | Best Algorithm | Benchmark Testing Evaluation
ROC AUC | SVM | 0.777 (0.049) | XGB | 0.800 (0.127)
PRC AUC | CGB | 0.635 (0.076) | SVM | 0.718 (0.107)
Balanced Accuracy | RF | 0.724 (0.013) | EN | 0.728 (0.067)
F1 Score | RF | 0.662 (0.008) | EN | 0.667 (0.094)

Table S6: Best Mean Algorithm Performance Between STEAMLINE versions on HCC.
Each value represents the mean metric value (with standard deviation) across
10-fold CV trained models for that algorithm. Here the ‘best’ model is
selected for each individual metric for comparison.

Notably the results highlight the fact that there is no clear ‘winning’
algorithm that performs best across all evaluation metrics or CV partitions,
and on this particular dataset most algorithms perform similarly well with
respect to PRC-AUC, with the exception of DT and GP.

### S.3.2: GAMETES Benchmark Results

Refer to ‘gametes_exp_STREAMLINE_Report.pdf’ for EDA and modeling results for
all 6 of the simulated GAMETES datasets. We review major results below. These
datasets are balanced, so we can confidently focus on ROC-AUC as the key
performance metric. Table S7 compares the performance of our previous
STREAMLINE GAMETES benchmarking (v0.2.5) [1] with the current (v0.3.4). As
expected, mean model ROC-AUCs were very similar between the two analyses.

light-gray | v0.2.5 | v0.3.4
---|---|---
light-gray GAMETES Dataset | Best Algorithm | Benchmark Testing Evaluation | Best Algorithm | Benchmark Testing Evaluation
A - univariate | RF | 0.842 (0.022) | XGB | 0.847 (0.035)
B – uni-4add | XGB | 0.983 (0.009) | XGB | 0.983 (0.007)
C – uni-4het | CGB | 0.672 (0.040) | CGB | 0.671 (0.042)
D - 2way-epistasis | ExSTraCS | 0.854 (0.028) | ExSTraCS | 0.852 (0.025)
E - 2way epi+2het | ExSTraCS | 0.740 (0.040) | ExSTraCS | 0.746 (0.026)
F – 3way epistasis | ExSTraCS | 0.564 (0.114) | SVM | 0.511 (0.042)

Table S7: Best Mean Algorithm ROC-AUC Between STEAMLINE versions on GAMETES
Datasets. Each value represents the mean metric value (with standard
deviation) across 10-fold CV trained models for that algorithm. Here the
‘best’ model is selected for each individual metric for comparison.

Figure S4 similarly compares mean model performance across all algorithms and
datasets using STREAMLINE v0.2.5 vs v0.3.4. Similar to what was observed for
v0.2.5 [1], dataset (A), i.e. univariate, all algorithms performed similarly
well, with NB (instead of KNN) performing the least well. For dataset (B),
i.e. additive univariate, all algorithms performed similarly well, with SVM
(instead of NB) performing the least well. For dataset (C), i.e. heterogeneous
univariate, CGB, XGB, EN, LGB, RF, GB, and ExSTraCS performed best, while NB,
KNN, and GP performed the least well. For dataset (D), i.e. pure 2-way
epistasis, we observed the first dramatic performance differences with
ExSTraCS, CGB, LGB, XGB, and GB performing similarly at the top, while NB, LR,
and DT entirely failed to detect the 2-feature interaction. For dataset (E),
i.e. heterogeneous pure 2-way epistasis, ExSTraCS and CGB stood out as top
performers, while again NB, LR, and DT entirely failed. Lastly, for dataset
(F), i.e. pure 3-way epistasis, all algorithms struggled with this noisy
complex association. Notably in this run even ExSTraCS failed to detect the
correct noisy complex association. As would be expected, no algorithm stands
out as being ideal across all data scenarios.

Figure S4: GAMETES Dataset Performance Comparison. (A). STREAMLINE v0.2.5
analysis (B) STREAMLINE v0.3.4 analysis. Each algorithm line represents mean
ROC-AUC across 10-fold CV models. Each plot’s formatting is different given
that it was automatically generated with different versions of the AutoML, and
the newest version has added the EN algorithm.

Also, as before, model feature importance estimates correctly prioritized the
ground-truth important features in these simulations. For example, Figure S5
illustrates the correct identification of features M0P0 and M0P1 (involved in
a pure, epistatic interaction) out of a total 100 features in the original
dataset. Additionally, Figure S6 illustrates the correct identification of
features M0P0, M0P1, M1P0, and M1P1 (which represent two independently
simulated 2-way epistatic interactions, i.e. M0P0xM0P1, M1P0xP1P1, that are
each predictive in half of the simulated instances of the original dataset).
We can observe here how different algorithms succeed or fail in their ability
to handle both interactions and heterogeneous associations.

Figure S5: GAMETES Dataset D (2-Way Epistasis) Model Feature Importance. This
composite feature importance plot illustrates normalized and (balanced
accuracy)-weighted mean FI scores summed across all algorithms. Only the top
40 features are displayed. Figure S6: GAMETES Dataset E (2-Way Epi+2 Het)
Model Feature Importance. This composite feature importance plot illustrates
normalized and (balanced accuracy)-weighted mean FI scores summed across all
algorithms. Only the top 40 features are displayed. In this dataset only M0P0,
M0P1, M1P0, and M1P1 are predictive.

In summary, these results suggest that this new release of STREAMLINE is
performing as expected, yielding similar top performance metrics, and similar
top model feature importance estimates.

### S.3.3: Multiplexer Benchmark Results

Refer to ‘multiplexer_exp_STREAMLINE_Report.pdf’ for EDA and modeling results
for all 6 of the benchmark multiplexer datasets. We review major results
below. These datasets are also balanced, so we can confidently focus on ROC-
AUC as the key performance metric. Table S8 compares the performance of our
previous STREAMLINE multiplexer benchmarking (v0.2.5) [1] with the current
(v0.3.4). As expected, mean model ROC-AUCs were very similar between the two
analyses. We attribute minor performance differences to any pipeline
variations based on the use of ‘random’, i.e. stochasticity, including the
partitioning of training/testing CV sets, which will not be exactly the same
in running v0.2.5 and v0.3.4.

light-gray | v0.2.5 | v0.3.4
---|---|---
light-gray Multiplexer Dataset | Best Algorithm | Benchmark Testing Evaluation | Best Algorithm | Benchmark Testing Evaluation
A – 6-bit | 10-way tie | 1.000 (0.000) | 11-way tie | 1.000 (0.000)
B – 11-bit | 8-way tie | 1.000 (0.000) | RF | 0.982 (0.013)
C – 20-bit | 5-way tie | 1.000 (0.000) | ExSTraCS | 0.968 (0.026)
D – 37-bit | ExSTraCS | 1.000 (0.000) | ExSTraCS | 0.938 (0.021)
E – 70-bit | ExSTraCS | 0.992 (0.016) | ExSTraCS | 0.908 (0.051)
F – 135-bit | LGB | 0.559 (0.016) | SVM | 0.556 (0.009)

Table S8: Best Mean Algorithm ROC-AUC Between STEAMLINE versions on
Multiplexer Datasets. Each value represents the mean metric value (with
standard deviation) across 10-fold CV trained models for that algorithm. Here
the ‘best’ model is selected for each individual metric for comparison.

Figure S7 similarly compares mean model performance across all algorithms and
datasets using STREAMLINE v0.2.5 vs v0.3.4. Similar to what was observed in
v0.2.5 (1), for the 6-bit dataset, most algorithms solved the problem
similarly well with the exception of EN, LR, and NB which are not expected to
handle complex interactions in data. These algorithms performed similarly
poorly on all other (more complicated) multiplexer datasets. For the 11-bit
dataset, GP joined the list of poor performing algorithms, and unlike in
v0.2.5, no algorithm solved this dataset with 100% testing accuracy (likely
due to variations in CV partitioning), with RF performing slightly best. For
the 20-bit dataset, CGB, ExSTraCS, XGB, GB, LGB, and RF performed best, with
DT and K-NN significantly dropping in performance. For the 37-bit dataset,
ExSTraCS and LGB performed best with XGB and GB seeing some more significant
drop in performance. For the 70-bit dataset, ExSTraCS performed far better
than any other algorithm (see Figure S8).

Figure S7: Multiplexer Dataset Performance Comparison. (A). STREAMLINE 0.2.5
analysis (B) STREAMLINE 0.3.4 analysis. Each algorithm line represents mean
ROC-AUC across 10-fold CV models. Each plot’s formatting is different given
that it was automatically generated with different versions of the AutoML, and
the newest version has added the EN algorithm. Figure S8: 70-bit Multiplexer
ROC Plot Comparing Mean Algorithm Performance. A composite feature importance
plot illustrating the mean, normalized and balanced-accuracy weighted
permutation feature importance estimates for each algorithm. Each line
represents mean algorithm performance across 10 CV models. Area under the
curve (AUC) is provided.

Figure S9 illustrates how ExSTraCS correctly prioritizes ‘address’ bit
features in model testing. And lastly, for the 135-bit dataset, no algorithm
performed well, but SVM performed slightly better than others. While ExSTraCS
had previously been demonstrated to be able to perform well even on the
135-bit multiplexer problem, in this analysis only 18K instances were
available for model training, when previous work required 40K instances for
ExSTraCS to closely solve the 135-bit MUX where a staggering 4.36e40 unique
binary instances make up the problem space (38). In this particular benchmark
analysis, ExSTraCS stands out clearly as the best-performing algorithm.
Overall this analysis highlights the capability of RBML algorithms such as
ExSTraCS to perform competitively and sometimes better than other well-
established ML modeling approaches.

Figure S9: 70-bit Multiplexer – Mean Model Feature Importance Estimates Across
all Algorithms. This composite feature importance plot illustrates normalized
and (balanced accuracy)-weighted mean FI scores summed across all algorithms.
Only the top 40 features are displayed. In this dataset all features are
predictive, but A_0, A_1, A_2, A_3, A_4, and A_5 are expected to be most
predictive.

### S.3.4: XOR Benchmark Results

Refer to ‘xor_exp_STREAMLINE_Report.pdf’ for EDA and modeling results for all
4 of the benchmark xor datasets. We review major results below. These datasets
are also balanced, so we can confidently focus on ROC-AUC as the key
performance metric. Table S9 and Figure S10 highlight the performance of
STREAMLINE v0.3.4 on these datasets that had not been previously evaluated
using STREAMLINE v0.2.5. For the 2-way dataset, 8 algorithms solve it
perfectly, and DT, KNN, and SVM do reasonably well, with EN, LR, and NB
failing. For the 3-way dataset, 5 algorithms (including CGB, ExSTraCS, XGB,
GB, and LGB) solve it perfectly, and RF performing reasonably well, with all
other algorithms failing. For the 4 and 5-way datasets, no algorithm performed
well. This result was not initially expected, as previous work applying
ExSTraCS suggested it should be well suited to detecting even higher-order
interactions.

light-gray | v0.3.4
---|---
light-gray Multiplexer Dataset | Best Algorithm | Benchmark Testing Evaluation
A – 2-way | 8-way tie | 1.000 (0.000)
B – 3-way | 5-way tie | 1.000 (0.000)
C – 4-way | DT | 0.519
D – 5-way | LR | 0.542
Table S9: Best Mean Algorithm ROC-AUC in STEAMLINE v0.3.4 on XOR Datasets.
Each value represents the mean metric value (with standard deviation) across
10-fold CV trained models for that algorithm. Here the ‘best’ model is
selected for each individual metric for comparison. Figure S10: XOR Dataset
Performance Comparison. STREAMLINE v0.3.4 analysis. Each algorithm line
represents mean ROC-AUC across 10-fold CV models.

However, this failure can be explained (not as an issue with all modeling
algorithms) but due to the default strategy implemented by STREAMLINE to
remove features during feature selection. Specifically, in the current
implementation of STREAMLINE, phase 4 automatically removes any features from
the respective training and testing datasets that yielded a score of 0 or
lower for both the mutual information and MultiSURF algorithms (when filter-
feat is set to True, i.e. the current default setting). This is based on the
current working assumption that only a positive score for these algorithms is
indicative of a potentially important feature. As mentioned above, we
previously observed that MultiSURF would yield highly negative scores for
predictive features involved in a high order interaction (i.e. 4-way and
above) (20). We thus noticed that predictive features were being removed prior
to ML modeling in the 3-way and 4-way dataset analyses (see Figure S11).

Figure S11: 4-way XOR Pre-Modeling Median Feature Importance Scores. (A)
Mutual Information, (B) MultiSURF. In this dataset, simulated predictive
features include M0P1, M0P2, M0P3, and M0P4.

Thus, we re-ran this experiment setting STREAMLINE’s filter-feat parameter to
False, to keep all features in the dataset regardless of initial feature
importance scoring by mutual information or MultiSURF. Table S10 and Figure
S12 presents the results of this follow-up analysis. In this secondary
analysis, performance results for algorithms were very similar for 2-way and
3-way, but as expected for the 4-way XOR, ExSTraCS now stood out as performing
close to 100%, with CGB and GB doing fairly well, at around 82%. Further
behind was XGB and LGB achieving closer to 60% and all algorithms failing.
Figure S13 illustrates model feature importance estimates for this 4-way XOR
dataset, showing how the models that performed best focused on the correct
interacting predictive features. On the 5-way XOR dataset, no algorithms
performed well, but out of these, ExSTraCS performed best. This is most likely
a result of the ‘curse of dimensionality’ and having a relatively small sample
size available to detect such a high order interaction (41). It is likely that
with a larger number of instances, ExSTraCS (and possibly other ML algorithms
evaluated) would be able to perform better on this benchmark problem.

light-gray | v0.3.4 – no feature selection
---|---
light-gray Multiplexer Dataset | Best Algorithm | Benchmark Testing Evaluation
A – 2-way | 7-way tie | 1.000 (0.000)
B – 3-way | 5-way tie | 1.000 (0.000)
C – 4-way | ExSTraCS | 0.98 ()
D – 5-way | ExSTraCS | 0.522 ()
Table S10: Best Mean Algorithm ROC-AUC in STEAMLINE v0.3.4 on XOR Datasets (no
feature selection). Each value represents the mean metric value (with standard
deviation) across 10-fold CV trained models for that algorithm. Here the
‘best’ model is selected for each individual metric for comparison.  Figure
S12: XOR Dataset Performance Comparison (no feature selection). STREAMLINE
v0.3.4 analysis. Each algorithm line represents mean ROC-AUC across 10-fold CV
models.  Figure S13: XOR 4-way Dataset Mean Model Feature Importance Estimates
Across all Algorithms. This composite feature importance plot illustrates
normalized and (balanced accuracy)-weighted mean FI scores summed across all
algorithms. In this dataset, simulated predictive features include M0P1, M0P2,
M0P3, and M0P4.

These results highlight the need to further improve the feature selection
strategy utilized in Phase 4 of STREAMLINE in future releases, in particular
to address the (likely) rare, but possible, presence of features involved in
high order interactions with no univariate effects. Future work will
specifically examine the reliability of employing absolute value feature
ranking with MultiSURF and it’s impact on the detection of simple univariate
effects, as well as increasingly complex interactions, and then adopt the most
effective strategy within STREAMLINE.

### S.3.5: Obstructive Sleep Apnea Results

Refer to ‘sagic_exp_STREAMLINE_Report.pdf’ for EDA and modeling results for
all 20 variations of the SAGIC datasets. EDA and model performance results for
respective replication datasets (A-T) are available upon request to the
corresponding author (as their inclusion exceeded arxiv file size limits).
Select plots and results are presented below to (1) demonstrate the diverse
set of automated figures generated by STREAMLINE over subsequent phases, and
(2) highlight major findings.

#### S.3.5.1: Initial Exploratory Data Analysis

Figure S14 first illustrates the difference in class balance in SAGIC
development datasets (before data processing) where OSA is defined as either
AHI$\geq$15 or AHI$\geq$5 events/hour. While AHI$\geq$15 datasets are close to
balanced, i.e. Class 0 = 1098 and Class 1 = 1080 (and can be primarily
evaluated with ROC-AUC as the primary metric), AHI$\geq$5 have reasonably
imbalanced classes, i.e. Class 0 = 606 and Class 1 = 1573 (and should be
primarily evaluated with PRC-AUC as the primary metric). Note, these
respective class balances have been maintained in the corresponding
replication datasets for each (stratified partitioning used).

Figure S14: Bar Plot of Class Balance in OSA Development Datasets (Before
Processing). (A) Class counts for all 10 development datasets where OSA is
defined as having AHI$\geq$15 events/hour. These are closely balanced
datasets. (B) Class counts for all 10 development datasets where OSA is
defined as having AHI$\geq$5 events/hour. These datasets have a reasonably
large degree of class imbalance with more OSA than non-OSA.

Figure S15 give an example of missing value counts in one of the 20 OSA
datasets analyzed (i.e. J) prior to processing. This plot is the same for all
20 datasets (prior to processing). The majority of the 85 total features had
no missing values, however some other features had a fairly high degree of
missingness, with ‘medhx_afib’ having the largest missing value count of 548.

Figure S15: Histogram of Missing Value Counts Across Features in
J_GE15_DEMDXSYMCFIO Development Dataset. This dataset includes all features
(across all feature groups) and a total of 2178 instances prior to data
processing.

Figure S16 gives the Pearson correlation heatmap for one of the 20 OSA
datasets analyzed (i.e. J) across all 85 features in the OSA data prior to
processing. These correlations are the same for all AHI$\geq$15 datasets, and
very similar for all AHI$\geq$5 datasets. There were no perfect correlations
in any of the datasets, so no features were automatically removed by
STREAMLINE using the default correlation cutoff of 1.0. However, we note that
there are a number of reasonably high positive and negative correlations among
features in this dataset with the strongest correlation observed between
mand_tri_area and mand_length_dia (0.959), in all datasets.

Figure S16: Pearson Feature Correlations in J_GE15_DEMDXSYMCFIO Development
Dataset. This dataset includes all features (across all feature groups) and a
total of 2178 instances prior to processing.

#### S.3.5.2: Automated Data Processing

Table S11 and Table S12 summarize the development datasets characteristics
before and after automated STREAMILNE data processing (i.e. cleaning and
feature engineering). Since in these datasets we had already one-hot-encoded
categorical features, and no features had a missingness > 50%, the only
processing to take place on any of these datasets was removing instances with
high missingness (i.e. > 50%). STREAMLINE automatically removes instances with
more than 50% missing values (by default), intended to avoid potential data
quality issues when imputing missing values with multiple imputation when many
other feature values for an instance are also missing. Note that even after
processing, AHI$\geq$15 datasets remained relatively balanced, and AHI$\geq$5
datasets remained largely imbalanced (i.e. class ratio).

lightgray Dataset |  Pre/Post Proc. |  Instances |  Total Features |  Categorical Features |  Quant. Features |  Missing Values |  Missing Percent |  Class 0 |  Class 1 |  Class Ratio
---|---|---|---|---|---|---|---|---|---|---
A |  Pre |  2178 |  8 |  6 |  2 |  20 |  0..0011 |  1098 |  1080 |  1.02
light-gray A |  Post |  2174 |  8 |  6 |  2 |  0 |  0 |  1097 |  1077 |  1.02
B |  Pre |  2178 |  7 |  7 |  0 |  2877 |  0.1887 |  1098 |  1080 |  1.02
light-gray B |  Post |  1794 |  7 |  7 |  0 |  229 |  0.0182 |  840 |  954 |  0.88
C |  Pre |  2178 |  4 |  0 |  4 |  1442 |  0.1655 |  1098 |  1080 |  1.02
light-gray C |  Post |  1810 |  4 |  0 |  4 |  10 |  0.0014 |  855 |  955 |  0.90
D |  Pre |  2178 |  46 |  0 |  46 |  360 |  0.0036 |  1098 |  1080 |  1.02
light-gray D |  Post |  2178 |  46 |  0 |  46 |  360 |  0.0036 |  1098 |  1080 |  1.02
E |  Pre |  2178 |  20 |  8 |  12 |  2183 |  0.0501 |  1098 |  1080 |  1.02
light-gray E |  Post |  2139 |  20 |  8 |  12 |  1719 |  0.0402 |  1090 |  1049 |  1.04
F |  Pre |  2178 |  15 |  13 |  2 |  2897 |  0.0887 |  1098 |  1080 |  1.02
light-gray F |  Post |  1810 |  15 |  13 |  2 |  306 |  0.0113 |  852 |  958 |  0.89
G |  Pre |  2178 |  19 |  13 |  6 |  4339 |  0.1048 |  1098 |  1080 |  1.02
light-gray G |  Post |  1834 |  19 |  13 |  6 |  546 |  0.0157 |  867 |  967 |  0.90
H |  Pre |  2178 |  65 |  13 |  52 |  4699 |  0.0332 |  1098 |  1080 |  1.02
light-gray H |  Post |  2178 |  65 |  13 |  52 |  4699 |  0.0332 |  1098 |  1080 |  1.02
I |  Pre |  2178 |  39 |  21 |  18 |  6522 |  0.0768 |  1098 |  1080 |  1.02
light-gray I |  Post |  2166 |  39 |  21 |  18 |  6281 |  0.0743 |  1092 |  1074 |  1.02
J |  Pre |  2178 |  85 |  21 |  64 |  6882 |  0.0372 |  1098 |  1080 |  1.02
light-gray J |  Post |  2178 |  85 |  21 |  64 |  6882 |  0.0372 |  1098 |  1080 |  1.02
Table S11: AHI$\geq$15 Development Dataset Characteristics Pre vs. Post STREAMLINE Processing.  lightgray Dataset |  Pre/Post Proc. |  Instances |  Total Features |  Categorical Features |  Quant. Features |  Missing Values |  Missing Percent |  Class 0 |  Class 1 |  Class Ratio
---|---|---|---|---|---|---|---|---|---|---
K |  Pre |  2179 |  8 |  6 |  2 |  25 |  0.0014 |  606 |  1573 |  0.39
light-gray K |  Post |  2174 |  8 |  6 |  2 |  0 |  0 |  605 |  1569 |  0.39
L |  Pre |  2179 |  7 |  7 |  0 |  2866 |  0.1879 |  606 |  1573 |  0.39
light-gray L |  Post |  1798 |  7 |  7 |  0 |  233 |  0.0185 |  447 |  1351 |  0.33
M |  Pre |  2179 |  4 |  0 |  4 |  1408 |  0.1615 |  606 |  1573 |  0.39
light-gray M |  Post |  1820 |  4 |  0 |  4 |  10 |  0.0014 |  449 |  1371 |  0.33
N |  Pre |  2179 |  46 |  0 |  46 |  336 |  0.0034 |  606 |  1573 |  0.39
light-gray N |  Post |  2179 |  46 |  0 |  46 |  336 |  0.0034 |  606 |  1573 |  0.39
O |  Pre |  2179 |  20 |  8 |  12 |  2170 |  0.0498 |  606 |  1573 |  0.39
light-gray O |  Post |  1961 |  20 |  8 |  12 |  1673 |  0.0391 |  603 |  1534 |  0.41
P |  Pre |  2179 |  15 |  13 |  2 |  2891 |  0.0885 |  606 |  1573 |  0.39
light-gray P |  Post |  1812 |  15 |  13 |  2 |  302 |  0.0111 |  450 |  1362 |  0.39
Q |  Pre |  2179 |  19 |  13 |  6 |  4299 |  0.1028 |  606 |  1573 |  0.39
light-gray Q |  Post |  1840 |  19 |  13 |  6 |  566 |  0.0162 |  457 |  1383 |  0.33
R |  Pre |  2179 |  65 |  13 |  52 |  4635 |  0.0327 |  606 |  1573 |  0.39
light-gray R |  Post |  2179 |  65 |  13 |  52 |  4635 |  0.0327 |  606 |  1573 |  0.39
S |  Pre |  2179 |  39 |  21 |  18 |  6469 |  0.0761 |  606 |  1573 |  0.39
light-gray S |  Post |  2168 |  39 |  21 |  18 |  6244 |  0.0738 |  600 |  1668 |  0.36
J |  Pre |  2179 |  85 |  21 |  64 |  6805 |  0.0367 |  606 |  1573 |  0.39
light-gray J |  Post |  2179 |  85 |  21 |  64 |  6805 |  0.0367 |  606 |  1573 |  0.39
Table S12: AHI$\geq$5 Development Dataset Characteristics Pre vs. Post
STREAMLINE Processing.

#### S.3.5.3: EDA Univariate Analyses on Processed Development Data

Here, we focus on the univariate analyses conducted in the two development
datasets (after processing) with all feature subgroups (i.e. J and T). Table
S13 and Table S14 give the top features univariately associated with outcome
for AHI$\geq$15 and AHI$\geq$5 datasets, respectively. In these datasets, top
features were all quantitative (i.e. evaluated with the Mann-Whitney U Test),
however all categorical features were automatically evaluated with Chi-Square
Test.

light-gray Feature | p-value | Test-Statistic | Test-Name
---|---|---|---
face_width | 1.046e-51 | 371046.5 | Mann-Whitney U Test
mand_width | 1.414e-50 | 373520.5 | Mann-Whitney U Test
bmi | 1.082e-38 | 402018.5 | Mann-Whitney U Test
max_mand_vol | 9.671e-36 | 407717.0 | Mann-Whitney U Test
max_tri_area | 2.541e-34 | 412405.0 | Mann-Whitney U Test
cer_ang | 8.807e-31 | 342422.0 | Mann-Whitney U Test
cranial_base_tri_area | 1.530e-28 | 429862.5 | Mann-Whitney U Test
mand_vol | 3.194e-28 | 431267.5 | Mann-Whitney U Test
mand_tri_area | 6.906e-27 | 435382.0 | Mann-Whitney U Test
max_mand_box_area | 1.250e-26 | 434456.5 | Mann-Whitney U Test
Table S13: Top Univariate Analyses for J_GE15_DEMDXSYMCFIO. Top 10 features, ranked by p-value. light-gray Feature | p-value | Test-Statistic | Test-Name
---|---|---|---
bmi | 1.060e-42 | 296375.5 | Mann-Whitney U Test
mand_width | 1.339e-40 | 301119 | Mann-Whitney U Test
face_width | 5.267e-38 | 307115 | Mann-Whitney U Test
cer_ang | 6.350e-33 | 258706.5 | Mann-Whitney U Test
max_mand_vol | 1.464e-30 | 323896.5 | Mann-Whitney U Test
mand_vol | 1.623e-27 | 333591 | Mann-Whitney U Test
max_tri_area | 1.865e-26 | 336101 | Mann-Whitney U Test
loud_snore | 1.721e-25 | 214783.5 | Mann-Whitney U Test
mand_tri_area | 4.203e-24 | 343357 | Mann-Whitney U Test
lat_fac_ht | 4.635e-24 | 342626 | Mann-Whitney U Test
Table S14: Top Univariate Analyses for T_GE5_DEMDXSYMCFIO. Top 10 features,
ranked by p-value.

Focusing on the most significant univariate association for each table, Figure
S17 gives a boxplot summary between classes for face_width, and Figure S18
gives a boxplot summary between classes for bmi.

Figure S17: Most Significant Univariate Association for J_GE15_DEMDXSYMCFIO
Development Dataset (face_width). Boxplot comparing face_width values between
class 0 and class 1 as defined by AHI$\geq$15. Figure S18: Most Significant
Univariate Association for T_GE5_DEMDXSYMCFIO Development Dataset (bmi).
Boxplot comparing bmi values between class 0 and class 1 as defined by
AHI$\geq$5.

#### S.3.5.4: Pre-Modeling Feature Importance Estimation

Figure S19 and Figures S20 give pre-modeling FI estimates for the two
development datasets with all feature subgroups (i.e. J and T) after
processing, CV partitioning, imputation, and scaling. These scores represent
the median FI calculated by either Mutual Information (sensitive to univariate
effects) or MultiSURF (sensitive to both univariate and interaction effects)
over 10-fold CV training datasets. For dataset J in Figure S19, the three top
scoring features are the same for both FI estimation algorithms, i.e.
mand_width, face_width, and bmi. For dataset T in Figure S20, the top scoring
feature for mutual information was bmi, but for MultiSURF it was cer_ang.

Figure S19: J_GE15_DEMDXSYMCFIO Pre-Modeling Median FI Scores.Median FI scores
across 10-fold CV training datasets (A) Mutual Information, (B) MultiSURF.
Figure S20: T_GE5_DEMDXSYMCFIO Pre-Modeling Median FI Scores. Median FI scores
across 10-fold CV training datasets (A) Mutual Information, (B) MultiSURF.

#### S.3.5.5: Development Data Testing Evaluations of Modeling

Figure 2 (main text) gives a global summary of model testing performance (with
14 ML algorithms) across all 20 OSA datasets differing in their encoding of
OSA outcome (as either AHI$\geq$15 or AHI$\geq$5), and/or their inclusion of
different available features sets. Table S15 gives pairwise performance metric
comparisons (using Wilcoxon Signed-Rank Test) between datasets having the same
underlying features, e.g. A vs. K, indicate that balanced accuracy and ROC-AUC
were consistently (but not always significantly) higher for models trained on
AHI$\geq$5 data. Notably, when focusing on PRC-AUC, GP models perform far
better than any other algorithm for datasets B, C, E, L, M, and O (see Figure
S21). However, closer inspection of other evaluation metrics for these GP
models reveal that they are predicting class 1 for all testing instances (i.e.
they have the maximum number of possible false positives, and thus are not
useful models). Therefore, we replaced these GP models in our assessment with
the next best scoring algorithm (as indicated by GP->’AlgorithmName’ in Table
S15. Even after this substitution, AHI$\geq$5 datasets still yielded
consistently and significantly higher PRC-AUC.

light-gray Data 1 Data 1 Data 2 Data 2 light-gray Metric Data 1 Data 2 Feature
Set(s) Statistic p-value Best Median Best Median light-gray Algorithm
(AHI$\geq$15) Algorithm (AHI$\geq$5) Balanced Accuracy A K DEM 15 0.232422 CGB
0.674879 XGB green!250.702773 B L DX 22 0.625 GP 0.55878 XGB green!250.57963 C
M SYM 8 yellow!500.048828 LGP 0.623776 XGB green!250.654582 D N CF 22 0.625 LR
0.678416 XGB green!250.693432 E O IO 15 0.232422 LR 0.60168 EN
green!250.612221 F P DEM+DX 24 0.769531 XGB 0.661979 SVM green!250.688487 G Q
DEM+DX+SYM 19 0.431641 ExSTraCS 0.690846 ExSTraCS green!250.695652 H R
DEM+DX+SYM+CF 5 yellow!500.019531 ExSTraCS 0.694347 ExSTraCS green!250.718696
I S DEM+DX+SYM+IO 16 0.275391 ExSTraCS 0.685458 ExSTraCS green!250.709368 J T
blue!25DEM+DX+SYM+CF+IO 8 yellow!500.048828 CGB 0.692929 blue!25EN
green!250.718936 ROC-AUC A K DEM 14 0.193359 ExSTraCS 0.738696 RF
green!250.768363 B L DX 19 0.431641 LGB 0.561623 LGB green!250.58465 C M SYM
16 0.275391 GB 0.660045 EN green!250.679587 D N CF 18 0.375 RF 0.734304 XGB
green!250.767255 E O IO 25 0.845703 SVM 0.640454 RF green!250.653842 F P
DEM+DX 8 yellow!50 0.048828 LGB 0.720548 ANN green!250.761928 G Q DEM+DX+SYM
16 0.275391 LGB 0.742128 EN green!250.761972 H R blue!25DEM+DX+SYM+CF 3
yellow!500.009766 ANN 0.766822 blue!25XGB green!250.79297 I S DEM+DX+SYM+IO 16
0.275391 LGB 0.7554 CGB green!250.778981 J T DEM+DX+SYM+CF+IO 7 yellow!50
0.037109 ANN 0.760819 GB green!250.792941 PRC-AUC A K DEM 0 yellow!500.001953
CGB 0.726915 CGB green!250.873214 B L DX 3 yellow!500.001152 GP 0.678069 GP ->
DT green!250.819413 C M SYM 3 yellow!500.000157 GP 0.708505 GP -> LGB
green!250.862589 D N CF 0 yellow!500.001953 XGB 0.719624 RF green!250.87526 E
O IO 4 yellow!500.000157 EN 0.625999 GP -> RF green!250.815426 F P DEM+DX 0
yellow!500.001953 RF 0.735926 CGB green!250.889826 G Q DEM+DX+SYM 0
yellow!500.001953 LGB 0.749936 EN green!250.89404 H R DEM+DX+SYM+CF 0
yellow!500.001953 CGB 0.752545 XGB green!250.894904 I S DEM+DX+SYM+IO 0
yellow!500.001953 XGB 0.732568 SVM green!250.892398 J T
blue!25DEM+DX+SYM+CF+IO 0 yellow!500.001953 RF 0.756443 blue!25EN
green!250.896618

Table S15: Pairwise Performance Comparisons of AHI$\geq$15 vs. AHI$\geq$5
Datasets with Same Feature Set(s). Pairwise tests using the Wilcoxon Signed
Rank Test. The testing metrics of the 10 CV models trained by the algorithm