11institutetext:  Data Science & Sharing Team, National Institute of Mental
Health, Bethesda, MD, USA 22institutetext: Machine Learning Team, National
Institute of Mental Health, Bethesda, MD, USA 33institutetext: Naval
Postgraduate School, Monterey, CA, USA

# Reliability and predictability of phenotype information from functional
connectivity in large imaging datasets

Jessica Dafflon 11 2 2 Dustin Moraczewski 11 Eric Earl 11 Dylan M. Nielson 22
Gabriel Loewinger 22 Patrick McClure 33 Adam G. Thomas 11 Francisco Pereira 22

###### Abstract

One of the central objectives of contemporary neuroimaging research is to
create predictive models that can disentangle the connection between patterns
of functional connectivity across the entire brain and various behavioral
traits. Previous studies have shown that models trained to predict behavioral
features from the individual’s functional connectivity have modest to poor
performance. In this study, we trained models that predict observable
individual traits (phenotypes) and their corresponding singular value
decomposition (SVD) representations – herein referred to as _latent
phenotypes_ from resting state functional connectivity. For this task, we
predicted phenotypes in two large neuroimaging datasets: the Human Connectome
Project (HCP) and the Philadelphia Neurodevelopmental Cohort (PNC). We
illustrate the importance of regressing out confounds, which could
significantly influence phenotype prediction. Our findings reveal that both
phenotypes and their corresponding latent phenotypes yield similar predictive
performance. Interestingly, only the first five latent phenotypes were
reliably identified, and using just these reliable phenotypes for predicting
phenotypes yielded a similar performance to using all latent phenotypes. This
suggests that the predictable information is present in the first latent
phenotypes, allowing the remainder to be filtered out without any harm in
performance. This study sheds light on the intricate relationship between
functional connectivity and the predictability and reliability of phenotypic
information, with potential implications for enhancing predictive modeling in
the realm of neuroimaging research.

##  1 Introduction

One of the central objectives of contemporary neuroscience is to elucidate a
connection between brain structure/function and behavior. This connection
could provide clinically relevant metrics, and be used for prognosis and
treatment decisions. In order to identify such connections, many neuroimaging
studies have focused on predicting observable characteristics or behavior –
broadly, phenotypes – from brain measurements. Despite the popularity of this
question, most studies report a small correlation of $r=0.1-0.4$ between the
phenotypes and their predictions from brain measurements; moreover, the
prediction performance for similar or related phenotype measures may vary
substantially across studies, (Greene and Constable, 2023; Sui et al., 2020;
Poldrack et al., 2020; Pervaiz et al., 2020; Genon et al., 2022).

The situation described above can be attributed to several factors, for
instance intrinsic variations of scanning equipment, noise at various stages
of acquisition, or natural and pathological differences between study
participants, among others. Moreover, the effect of these factors can be
exacerbated by variations in sample size across studies. In general, the use
of machine learning methods in typical clinical neuroimaging datasets is
severely constrained by their size (tens to low hundreds of subjects), in
comparison with their dimensionality (tens of thousands of features).
Determining how much data is needed to train models to predict brain-behavior
associations is still an open question. Recent research by Marek et al. (2022)
suggested that at least a thousand subjects are needed to accurately measure
univariate brain-behavior associations, and that small studies fail to
replicate and produce inflated effect sizes. This controversial study prompted
an avalanche of responses showing that the brain-behavior relationships could
be predicted from smaller datasets by using multivariate models and hold-out
samples (Spisak et al., 2023) and thoughtfully choosing the brain data and
behavioral targets of prediction (Cecchetti and Handjaras, 2022; Gratton et
al., 2022; Rosenberg and Finn, 2022).

In addition, recent studies (Nikolaidis et al., 2022; Gell et al., 2023) argue
that increasing sample size might improve the prediction performance, but not
be sufficient for consistent results. They argue that the reliability of the
prediction targets will limit predictive performance, and that the reliability
of clinical and cognitive phenotypes should be better assessed. In particular,
Nikolaidis et al. (2022) showcased that, while inconsistency in results is
aggravated by small sample sizes, the use of more reliable measures can allow
for more robust estimates. The authors’ suggested solution to improve
reliability is to aggregate across multiple raters and/or measurements.

Traditionally, brain-behavior models have primarily focused on individual
behaviors or symptoms examined in isolation. The increasing recognition of the
integrated nature of the brain has led to a methodological shift, however, and
researchers have begun to consider the complex interplay of variables across
various demographic, clinical, and behavioral phenotypes (Holmes and Patrick,
2018). Following this line of thought, Chen et al. (2023, 2022) averaged the
scores of selected behavioral phenotypes into three categories (Cognition,
Personality, and Mental Health). Their main idea was that the combination
would reflect not only a single phenotypic measure and its noise, but a
hopefully more robust combination of many measures. In addition, because
scores across different behavioral tasks may be correlated, many studies
process them using dimensionality reduction techniques like factor analysis
(Ooi et al., 2022; Schöttner et al., 2023) and principal component analysis
(Chopra et al., 2022). The goal of dimensionality reduction is to find a
reduced set of (potentially orthogonal) latent variables that contain most of
the information contained in the original phenotypes. Ooi et al. (2022) showed
that latent phenotypes derived from factor analysis were more predictable than
the original phenotypes in the HCP and ABCD datasets. Therefore, in this
study, we attempted to replicate this finding in a different dataset using a
different dimensionality reduction algorithm: singular value decomposition
(SVD). We chose to use SVD because it produces the optimal low-rank
approximation to the data, in addition to having other desirable properties
that will be discussed later in this paper. In addition, Ooi et al. (2022)
only used the top three components. Therefore, an additional question that was
left unanswered, and we address in this paper, is what is the impact of using
all latent phenotypes and if the latter might be more reliable than the
former, and hence lead to better predictive results. This is particularly
relevant as prior research has identified a limited set of latent phenotypes
that connect brain imaging data with a wide array of phenotypes, including
cognition, mental health, demographic characteristics, and clinical phenotypes
(Chen et al., 2022; Smith et al., 2015; Goyal et al., 2022; Miller et al.,
2016).

An additional motivation for this work is analyzing the robustness of findings
in the literature when using a different dimensionality reduction method and
applying this dimensionality reduction to more than one dataset. There is an
inherent flexibility in the process of designing and executing scientific
experiments and, in particular, analytical pipelines, and this can be yet
another source of variability in the performance of brain-behavior prediction
models. It is now widely acknowledged that a single research question can be
approached through a diverse array of analytical pipelines, often resulting in
different outcomes (Dafflon et al., 2022; Botvinik-Nezer et al., 2020; Carp,
2012). In young research areas, such as functional neuroimaging, in which many
of the ground truths are yet to be discovered, there are limited foundations
to favor one option over its alternatives. As a result, researchers must
consider various factors when designing imaging analyses, including data
preprocessing, feature selection, functional connectivity computation, and
prediction model of choice. A study by Pervaiz et al. (2020) explored the
impact of various choices on model performance, provided insights into the
influence of these choices, and offered recommendations about which
parcellations to use, metrics for estimating functional connectivity and how
those choices depend on the target of interest. It’s worth noting, however,
that while this study employed a substantial sample size, it primarily focused
on a limited set of cognitive measures (fluid intelligence and neuroticism
score), age, and sex. Consequently, whether the findings and suggestions
generalize to other phenotypes remains uncertain. Many studies individually
focused on one dataset or one phenotype prediction, and the absence of a gold
standard complicates comparisons between studies that adopt different
analytical approaches.

In this study, we examined the questions above in the context of predicting
rich phenotypic information from functional connectivity data, in the Human
Connectome Project (HCP) and Philadelphia Neurodevelopmental Cohort (PNC)
datasets. First, we analyzed the importance of regressing out
confounding111While not confounds in the strict definition of confounding from
the causal inference literature (Hernan and Robins, 2023), we refer to
inflation of phenotype prediction by age and sex as confounding in this paper
to reflect the common usage in the machine learning field. covariates such as
age or sex at birth, since predictive performance for variables correlated
with these may be inflated by the known predictability of age and sex from
functional connectivity.

We then attempted to replicate previous findings that latent phenotypes are
more predictable than original phenotypes (Ooi et al., 2022). Specifically, we
compared the performance of models predicting latent phenotypes from
functional connectivity to the performance of models predicting the original
phenotypes from functional connectivity.

Since the reliability of the predicted variable limits the maximal observable
predictive performance, we also evaluated the reliability of the latent
phenotypes across different participant samples, and the relationship between
reliability and predictability from functional connectivity. Finally, we
tested whether the phenotype information predictable from functional
connectivity is primarily contained in reliable latent phenotypes. To this
effect, we reconstructed phenotypes from the subset of reliable latent
variables, and evaluated the performance of models trained to predict these
reconstructed phenotypes from functional connectivity. Overall, our study
provides insights into the relationship between prediction accuracy and
reliability and extends our understanding of the extent to which latent, and
reconstructed phenotypes can be predicted from resting state fMRI.

##  2 Methods

###  2.1 Datasets

In the analyses described in this paper, we used resting state fMRI data and
phenotype information from two datasets: the Human Connectome project (Van
Essen et al., 2012) and Philadelphia Neurodevelopmental Cohort (Satterthwaite
et al., 2016). We will focus on predicting phenotypes only from functional
connectivity (FC) measures derived from the fMRI data, as prediction models
derived from FC outperform other modalities in general (Ooi et al., 2022). We
provide dataset-specific details in the following subsections, and describe
the common extraction of FC matrix after that. The code used for the analysis,
together with a list of all subjects used for training, validation, and hold-
out, can be found at the following repo: https://github.com/JessyD/bblocks-
phenotypes. Our work relies on the recently released “dataset-phenotypes” tool
111https://github.com/nimh-dsst/dataset-phenotypes, which outputs BIDS tabular
phenotypic data dictionaries and transforms tabular phenotypic data to BIDS
TSVs for common neuroimaging datasets.

####  2.1.1 Human Connectome Project (HCP)

We used the behavioral and imaging data from the HCP Young Adult 1200 Subject
release (Van Essen et al., 2012)(N=1071 ; 485 males/586 females aged $28.8\pm
3.7$ years). As described by Barch et al. (2013), the functional connectivity
data was acquired with a 32-channel head coil on a 3T Simens Skyra with TR=720
s, TE=33.1 ms, flip angle=52 deg, BW=2290 Hz/Px, in-plane FOV=208 × 180 mm, 72
slices, 2.0 mm isotropic voxels, with a multi-band acceleration factor of 8.

Subjects were chosen if their imaging preprocessing pipeline was completed
without error. While for PNC, we removed frames that were considered as high-
motion using their framewise displacement and DVARS (as described below), we
did not use quality control metrics to select HCP subjects. Subjects were
split into a training set (N=856; 385 males/471 females aged $28.88\pm 3.70$
years), a validation set (N=108; 45 males/63 females aged $28.16\pm 3.50$
years), and a separate hold-out set (N=107; 55 males/52 females aged $28.69\pm
3.96$). The hold-out set was not used for the results reported in this paper,
and is being kept for a future pre-registered analysis. Subjects were assigned
to the training, validation, and hold-out sets in a two-stage procedure.
First, all families with two or more siblings were included in the training
set. This was done to prevent leakage of information between twins or other
siblings, which could happen if they were split between training and the other
two datasets. Second, all subjects were assigned at random to the three
datasets, until the specified number of participants was obtained in each.

##### Imaging Data

We used the cleaned version of the HCP S1200 release, and used the
grayordinate resting-state functional MRI data processed with ICA-FIX and MSM-
All provided by Glasser et al. (2013a), a pipeline which was developed and
optimized for this dataset.

##### Behavioral Phenotypes

We used 83 phenotypes scores (Appendix ALABEL:tab:phenotypes-hcp includes a
full list of all scores used), which span across behavioral domains of
cognition and personality and some additional variables that we included to be
consistent with prior work. To facilitate comparisons, we included all
behavioral measures previously used by Ooi et al. (2022), Kong et al. (2019)
and He et al. (2020). All behavioral scores were individually z-scored across
participants, and outliers that were above and below 3 standard deviations
were treated as missing data as described below. We incorporated behavioral
phenotypes categorized as “age-adjusted behaviors”, which had undergone prior
age adjustments by the HCP team, alongside the raw, unadjusted behaviors. The
difference between “age-adjusted behaviors” and “unadjusted” behavior is that,
while the Unadjusted Scale Score evaluates a participant’s performance
compared to the entire NIH Toolbox Normative Sample, the Age-Adjusted Scale
Score assesses a participant’s performance in the context of a specific age
group within the Toolbox Norming Sample (e.g., 18-29 or 30-35) (Slotkin et
al., 2012). Both the outliers and the missing data were imputed using the
IterativeImputer function from `scikit-learn` (Pedregosa et al., 2011), which
models each feature as a function of other features, and uses the model to
predict its missing values, iterating over features in a round-robin fashion.
To make sure that we keep the development and validation sets completely
separate, we trained the imputation function on the development set and
applied it to the validation set.

####  2.1.2 Philadelphia Neurodevelopmental Cohort (PNC)

As described by Satterthwaite et al. (2016), the PNC dataset was acquired
using a 3T Siemens TIM Trio, with a 32-channel head coil with TR=3000 ms,
TE=32 ms, flip angle=90 deg, BW=2056, FOV=192 x 192 mm, voxel resolution of
3x3x3mm with 46 slices. For our analysis, we used 1082 subjects. Similar to
the HCP dataset, we split the subjects into a development set (N=864; 404
males/460 females aged $15.670\pm 3.376$ years), validation set (N=109; 48
males / 61 females aged $16.231\pm 3.364$), and an unused hold-out (N=109, 47
males/ 62 females aged $15.97\pm 3.38$) dataset reserved for future pre-
registered analyses. Subjects were assigned to the training, validation, and
hold-out sets in the same two-stage process used in the HCP data.

##### Imaging Data

We preprocessed the PNC dataset using fmriprep version 21.0.2 (Esteban et al.,
2019) (additional information about all software used can be found in Appendix
A.5.1). For motion censoring, time points were considered individually. If a
time point exceeded 0.2 mm frame-wise displacement or a derivative root mean
square (DVARS) above 75, it was marked as a point to be censored. Intervals of
less than five points between censor points were also censored. The six
estimated head-motion parameters, their derivatives, the average signal within
the anatomically-derived white matter, and cerebrospinal fluid masks obtained
from fmriprep were used as nuisance variables and regressed from the fMRI
signal before we computed the FC.

##### Behavioral Phenotypes

Although the PNC dataset includes surveys, cognitive and clinical phenotypes,
for this analysis we did not use surveys or clinical phenotypes. We selected
39 summary scores from cognitive tasks from the PNC dataset (Satterthwaite et
al., 2016; Gur et al., 2010; Roalf et al., 2014). See Appendix A. 2 for a full
list of all scores used. Similar to the procedure used for the HCP dataset,
missing values and outliers were imputed using the IterativeImputer function
from `scikit-learn`, and each measure was z-scored across participants.

###  2.2 Generation of datasets for prediction experiments

We constructed functional connectivity matrices by computing the Pearson
correlations between the average time series for each pair of brain regions.
Regions were defined using the 17-network Schaefer 400 parcellation (Schaefer
et al., 2018). If more than one resting state session was available for a
subject, we averaged the resulting functional connectivity matrices across
sessions and used the resulting one in our analyses. If any of the runs had
more than 50% of the runs flagged as time points to be censored, it was
excluded from the computation of functional connectivity. Across this paper,
we will refer to the data matrix $X^{n\times d}$), where $n$ is the number of
subjects available and $d$ the number of pairwise connections after
vectorizing the lower diagonal (in our case 79800). These pairwise connections
will be the input features for the prediction model described in Section 2.5.
The process is illustrated in Figure 1.

Figure 1: Schematic overview of how the data matrix is constructed. First, the
brain activity is parcellated using an atlas of choice (here we used the
Schaefer 400 parcellation). The functional connectivity matrix is created by
computing the Pearson correlation between the average resting-state fMRI time
series in each pair of regions. Because the FC matrix is symmetric between its
upper and lower triangular entries, we flatten only the lower triangular
entries of the functional connectivity matrix.

###  2.3 Dimensionality reduction of phenotype data

####  2.3.1 Linear dimensionality reduction

The following paragraphs compare different types of linear dimensionality
reduction. They introduce those methods and compare them to SVD, the method we
used in further analysis.

In our experiments, we structure phenotype data as a matrix with $n$ rows
(participants) and $d$ columns (phenotype measurements). Each row is therefore
a $d$-dimensional observation for a participant. A linear dimensionality
reduction method expresses that matrix as a product of a $n\times k$ latent
variable matrix, and a $k\times d$ matrix that expresses observation vectors
as a linear combination of latent variables (often called the mixing or
loading matrix). The latent variable matrix can be viewed as a reduced
dimensionality representation of the original matrix. The underlying
assumption behind this framing is that each observation does not vary
independently in $d$ different ways. Instead, those $d$ measures are driven by
one or more latent variables, and that relationship holds across participants.
If measurements were the summary scores of different instruments, the latent
variables could correspond to factors that drive variance across them; in that
case, latent variables would be helpful to elicit relationships between
instruments. Aside from their role in psychology research, latent variables
are potentially useful as prediction targets from imaging, as discussed
earlier. This is because they are estimated from multiple phenotype measures
they are associated with, and hence, are potentially less noisy.
Factorizations can also be used for denoising, in that the product of the
latent variable matrix and the mixing matrix can be viewed as an approximation
that tries to keep the essential characteristics of the data. This is usually
done by considering fewer than the maximum number of latent variables that can
be estimated from the data matrix.

####  2.3.2 Linear dimensionality reduction methods

There are many linear dimensionality reduction methods. They differ primarily
in their assumptions about the relationship between latent variables, and
their connection to the measured variables. While it is beyond the scope of
this paper to review all the methods available, we briefly do so for those
that have been used for experiments similar to those we carry out. Factor
Analysis (FA) finds latent variables (factors) and a mixing matrix (loadings)
such that between factor correlation is minimized. FA is used in the
development of psychometric questionnaires, where the goal is to relate
answers to questions to hypothesized latent constructs driving behavior. FA
usually includes a step called varimax rotation (Kaiser, 1958), where the goal
is to further transform the factor/loading relationship so that each factor
loads on as few phenotype measurements as possible, to facilitate
interpretability.

Independent Component Analysis (ICA) finds latent variables (components) and a
mixing matrix such that the factors are statistically independent, not
orthogonal, but uncorrelated. Neither FA nor ICA have an intrinsic criterion
for choosing the number of dimensions $k$ analogous to the percentage of
variance explained in PCA and SVD.

Principal Component Analysis (PCA) finds successive orthogonal projection
directions of the data that maximize variance after projection. Each latent
variable is defined as the projection of all the data points into one
dimension; the mixing matrix is derived as part of the process of finding the
projection directions. The intuition for this approach is that important
latent variables should drive a lot of variance in the observations. Similar
to FA, a varimax rotation can also be applied to a PCA to simplify the
interpretation of the components.

Singular Value Decomposition (SVD), the technique we will use in this paper,
finds the same projection directions as PCA if the inputs to the SVD are mean-
centered. Both PCA and SVD share a useful characteristic that other methods do
not have: the percentage of variance explained provides an order of importance
of the latent variables. In addition, SVD has a number of practical advantages
over PCA (e.g., it does not require computation of a covariance matrix), as
well as useful theoretical properties. The one that is most important for this
paper is that the data reconstruction through SVD, using $k$ latent variables,
is the best rank $k$ approximation of the data, in the least squares sense. We
provide an illustrated introduction to SVD, its mathematical properties, and
its relationship with PCA in Section 5.2 of the Appendix. Finally, there are
methods such as reduced rank regression that implicitly perform an SVD/PCA of
a dataset of target variables, as part of a multivariate, multiple regression
model. Given that the results are similar to those obtained by computing an
SVD of the target variables, predicting each latent variable independently,
and reconstructing predictions of the targets, we do not consider these
further.

###  2.4 Controlling for age/sex

Before training the prediction models we regressed out age and sex at birth
terms from each phenotype measure in both datasets: age, sex,
$\textit{sex}^{2}$, and the interactions $\textit{age}\times\textit{sex}$ and
$\textit{age}^{2}\times\textit{sex}$. Sex at birth was coded as a 0/1 binary
variable and age was z-scored. The phenotype measure values used in the
prediction experiments were the residual values after fitting this regression
model. The purpose of regressing age and sex information out of the phenotype
targets was to determine the degree to which their predictability was a
combination of a) their being predictable from age/sex (e.g. a developmental
phenotype measure) and b) age/sex being predictable from the imaging data. We
deliberately did not regress age/sex information out of the imaging data. Our
plan is to train neural networks that predict age/sex, in tandem with other
phenotypes where age/sex have been regressed out. For that application,
age/sex should not be regressed out of the imaging data. As these experiments
are meant to produce baseline results using linear models, against which
neural network results can be compared, we opted not to regress age/sex out of
the imaging data in this case as well.

We also conducted a paired two-sided t-test to determine if there was a
significant difference in performance before and after adjusting for age and
sex. Before running the t-test, we calculated the correlations between the
actual and predicted values for models both with and without the regression
adjustments. We then applied the Fisher z-transformation to these correlations
and calculated the average across multiple repetitions.

###  2.5 Prediction experiments

##### Prediction model

We used a Ridge regression model to find the linear relationship between the
imaging data and each of the selected behavioral phenotypes, or the latent
variables derived from them. We chose a model with $L_{2}$-regularisation, as
implemented by the `scikit-learn` library (Pedregosa et al., 2011), version
1.2.2, as it can handle ill-posed problems that result from extremely
correlated features. The reason for only using a linear prediction model in
this work is that our primary goal is to obtain baseline results for future
nonlinear models.

##### Experimental setup

The predictive results reported reflect the mean and standard deviation –
which is an estimate of the standard error of the mean – across 100 bootstrap
samples, taking age, gender and family information into account. In this
procedure, we first split the training data into a part that would be
resampled (80% of the total dataset) and a part that would be used to evaluate
the performance of models with different regularization parameters, which was
kept fixed (10% of the total dataset). The remaining 10% was kept as a hold-
out dataset and was not used in this study. We did not use nested cross-
validation. For each resampled training set, we found the optimal setting of
the $l_{2}$-regularisation term ($\alpha$) using a grid search over the search
space [$10^{2}$, $10^{7}$], applying the model to the fixed part of the
dataset used for evaluation. The model with the best $l_{2}$-regularisation
term was then applied to the validation set, yielding one result of the 100 in
the sample for predicting that particular phenotype measure. We performed the
SVD of the phenotypes within each resampled training set. These were z-scored
in a column-wise fashion, i.e., we computed the mean and standard deviation
across the training set to normalize that specific phenotype, so that the
features with large means or variance would not dominate the results, and
repeated this procedure for all phenotypes. We then used the mean and standard
deviation from the training dataset to z-score the phenotype measures in the
validation set.

##### Model evaluation

The performance of each model was evaluated using the coefficient of
determination ($r^{2}$) and Pearson’s correlation between the predicted
measure and the actual measure in the validation set. While Pearson’s
correlation assumes values between -1 and 1, the $r^{2}$ can assume values
between $(-\infty,1]$ where 1 corresponds to the best possible score, as it is
being computed on a separate dataset.

We used the Autorank library (Herbold, 2020; Demšar, 2006) to evaluate if
there was any statistical difference in the performance of the prediction
algorithms using all of the components or a subset thereof. The Autorank
library first assesses the normality and heteroscedasticity of the data before
selecting the most suitable group level and post-hoc test to determine
differences between the groups. The family-wise significance level for these
tests was set at $\alpha$ = 0.05. We provided inputs to the Autorank library
by calculating the correlation between the predictions and the actual values,
applying the Fisher-z transformation to these correlations, and then computing
the average correlation across multiple bootstraps.

###  2.6 Comparing predictive performance on original phenotypes and latent
phenotypes

The latent phenotypes are composed of linear combinations of the original
phenotypes, so directly comparing predictive performance between the two is
not straightforward. Instead, we computed absolute difference between z-scored
values of the prediction and true phenotypes (i.e., the error) for the
component with the highest prediction, taking the average over all bootstraps.
We also computed the same difference for the best performing latent phenotype.
We then used a paired t-test to test the difference in error between the best
performing latent phenotype and the best performing original phenotype.

###  2.7 Latent phenotypes reliability analysis

We conducted analyses to assess the reliability of the latent phenotypes
produced by the SVD separately in the HCP and PNC datasets. Specifically, we
wanted to determine how similar each latent phenotype would be if the SVD was
conducted in distinct sets of subjects with the same phenotypic measures. We
accomplished this by combining the training and validation subsets and then
randomly splitting these combined datasets into two halves (without
replacement) 1000 times. Splits were applied at the family level so that all
members of a family were in the same subsample in every splitting. Since the
latent phenotypes produced by the SVD are ordered by percent variance
explained, the order may differ between samples. To ensure that the order was
consistent across all subsample, we first ran an SVD on the combined dataset,
saving the latent phenotypes. Then the latent variables from each subsample
were reordered to match the order of latent phenotypes in the combined
dataset. Components were matched via the Gale–Shapley stable marriage
algorithm (Gale and Shapley, 1962) using the correlation between latent
phenotype weights as the preferences. Once the latent phenotypes from each
subsample within a split were aligned, we took the correlation between the
latent phenotype’s weights on the features (following the SVD notation, this
would be $V_{T}$) as our measure of latent phenotype reliability. In all cases
age and sex were regressed out of the phenotypes, values more than three
standard deviations from the mean were censored, and missing values were
imputed as described above prior to running the SVDs. These preprocessing
steps were carried out separately in each subsample. Empirical 95% confidence
intervals were determined from the distribution of inter-subsample
correlations across splits. Correlation coefficients were Fisher z-transformed
prior to averaging across splits and then transformed back for
plotting/analysis.

##  3 Results

###  3.1 Predictability of phenotype measures from functional connectivity

The goal of this experiment was to ascertain the degree to which each
phenotype measure is predictable from functional connectivity, in both HCP and
PNC.

As described in Section 2.5, we trained ridge regression models to predict
each phenotype measure across 100 bootstrap resamplings; we then used them to
generate predictions for participants in the validation set, which was fixed
rather than resampled. This resampling scheme allows us to estimate the
variability in performance across potential training sets for the fixed
validation set. We averaged prediction results across resamplings, and used
those results to estimate the standard deviation of the mean estimate. Figure
2.a and Figure 3.a show the average correlation between the phenotype
predictions and their true values, with the measures sorted by predictability.
We can see that for the HCP data the most predictable phenotype is _Strength
Unadjusted_ , which measures the Grip Strength (Figure 2a). Because of the
relation between strength and sex at birth, we repeated the same analysis
after regressing out age and sex and their interaction as confounds. Appendix
Figure 8 shows the corresponding results using $r^{2}$ as the metric of
performance.

Figure 2: Predictability of phenotype variables in the HCP validation set
before regressing out confounds (a), after confound removal (b), reconstructed
using only a subset of the latent phenotypes (c), and of the latent phenotypes
(d). Predictability is quantified as the correlation between true phenotypes
and prediction from functional connectivity data. Panel c shows the
correlation of the predicted phenotypes with true values when the model was
trained with a subset of latent phenotypes (1, 2, 3, 4, 5 and all, represented
by the different shades of green). The shaded regions represent the standard
deviation across resamplings. The phenotypes (y-axis) are ordered by the
prediction performance, with the exception of (d), which is ordered by the
variance explained by the latent phenotypes. (a) Before regressing out, the
phenotypes with the best performance is “Strength unadjusted”; after
regressing age and sex, cognitive phenotypes have the best performance. In
addition, using latent phenotypes (d) or untransformed phenotypes (b) does not
impact the performance substantially. Figure 3: Predictability of phenotype
variables in the PNC validation set, before regressing out confounds (a),
after confound removal (b), reconstructed using only a subset of the latent
phenotypes (c), and of the latent phenotypes (d). Predictability is determined
by the correlation between actual phenotypes and predictions from functional
connectivity. The shaded regions represent the standard deviation across
resamplings. Similarly to Figure 2, the correlation is higher before
regressing out the confounds (a) than in the remaining plots (b-d). As HCP and
PNC have different prediction ranges, the x-axes here are different than in
Figure 2

###  3.2 Predictability of phenotype measures from functional connectivity
after regressing the confounds

Figure 4: Beta-weights of the linear regression used to regress out age, sex,
and their quadratic interactions from HCP (a) and PNC (b) over the repetions.
(a) Dotted lines include pairs of phenotypes included in the analysis that
were both Unadjusted and Age-adjusted (_AgeAdj_).

One of the issues in determining how predictable a phenotype measure is from
imaging is the presence of potential confounds, such as participant sex at
birth, age, and time of scan, which are to some extent predictable from
imaging data. Given this, a phenotype measure might be predictable from
imaging data because a) the confound can be predicted from imaging data, and
b) the phenotype measure can be predicted from the confound. To try to
minimize the impact of confounding variables on our predictions, we regressed
out age, sex, and age$\times$sex interaction, as well as their squares, from
each phenotypic measure.

Figure 4 illustrates the regression beta coefficients for each phenotype
measure in HCP (Figure 4a) and PNC (Figure 4b). The magnitude of the beta
coefficient indicates the strength of the relationship between the independent
variable and the dependent variable. A list of all the phenotypes used, the
acronyms and a short description can be found in Table LABEL:tab:phenotypes-
hcp (HCP) and Table 2 (PNC).

In the HCP analysis, we did not see a large dependence between the variables
and age, and the highest beta coefficient was observed for “Strength
Unadjusted”, where there was a high interaction with sex. This aligns with our
expectations as HCP is a young adult cohort while PNC is a neurodevelopmental
cohort. Note that we included behavioral phenotypes that had previously been
adjusted for age by the HCP team (referred to as “age-adjusted behaviors”) and
the raw unadjusted behaviors. Our initial aim was to compare the beta
coefficients between the adjusted and unadjusted variables provided by the HCP
consortia. We observed that if we had trained our model without correcting for
the confounds, some of the predictive performance would be due to their
presence. For the PNC dataset (Figure 4b), the highest beta coefficients are
observed between age and Total Correct Response for All Test Trials (PEDT_A),
Median Response Time for All Test Trials (PADT_T), and Total Raw Score
(WRAT_CR_RAW).

We also compared predictive performance for phenotypes in which we regressed
out age and sex confounds or not. Regressing out age and sex results in a
significant decrease in mean prediction across all phenotypes (before
regression: 0.172 $\pm$ 0.143 (mean $\pm$ standard deviation); after
regression 0.096 $\pm$ 0.095; t-statistic:-6.37, value:1.75e-07, df:38)

###  3.3 Predictability of SVD latent variables from functional connectivity

As described earlier, SVDs were fit to each resampled training set and used to
produce latent phenotype scores for that training set and the fixed validation
set. Contrary to our expectations, we found that SVD latent phenotypes were
not more predictable than individual phenotype measures (correlation in Figure
2d and Figure 3d; $r^{2}$ in Appendix Figure 8d and Figure 9d). We evaluated
this by performing a paired t-test to compare the absolute error between
predicted and true z-scored values for the most predictable latent factor and
the most predictable phenotypes. In both datasets the error was not
significantly different between the best phenotype (HCP: phenotype mean error
(std):0.890 (0.600)/ PNC: phenotype mean error (std): 0.891 (0.654)) and the
best latent phenotype (HCP: latent phenotype mean error (std): 1.001 (0.645) /
PNC: latent phenotype mean error (std): 0.950 (0.641); HCP: t-statistic=-1.78,
p-value=0.078, df=107 / PNC: t-statistic=-1.138, p-value=0.258, df=108).

Figure 5: Explained variance for each of the latent phenotype variables
obtained after applying an SVD to the behavioral variables from HCP (green,
left) and PNC (blue, right). In this experiment, the SVD was computed using
the training and the validation data. The bar graphs show the variance
explained by each latent variable, while the line plots show the cumulative
variance explained. For both datasets, a large proportion of the variance is
explained by the first two latent phenotypes; however, most latent phenotypes
are needed to reach 95% of the variance, as represented by the dashed line in
the figure. Note that the x-axis for these two plots are different as HCP has
83 phenotypes and PNC 39.

###  3.4 Low-dimensional representations of phenotype variables and their
reliability

One of our goals was to understand the dependency structure between different
phenotype measures, as captured through the SVD of the dataset. This analysis
is important to determine if the phenotypes can be represented using a smaller
set of uncorrelated latent phenotypes.

For both datasets, we can see that while the first two latent phenotypes
explain about 30% of the variance, this decays rapidly for successive latent
phenotypes (Figure 5). To attain a comprehensive 95% explanation of the
variance, a substantial number of latent phenotypes are required. Considering
that each dataset uses a diverse array of tests to cover various cognitive
domains, we were not surprised to see that many of the components were
required to explain 95% of the variance.

Figure 6: Latent phenotypes reliability for both datasets HCP (in green; left)
and PNC (in blue; right). The loadings of the first 5 SVD components were
reliably identified within different splits using the Gale–Shapley stable
marriage algorithm. The shaded area represents the 95% confidence intervals.

We also conducted an experiment to examine the reliability of the latent
phenotypes, i.e., how many of them would replicate if the SVD was applied to
two independent samples drawn from the same population. To that effect, we
simulated this situation with an experiment where we repeatedly split the
phenotype dataset into two halves (i.e., sub-samples), and independently
computed the SVD of each split. To ensure that the order was consistent across
all subsample, we first ran an SVD on the combined dataset, saving the latent
phenotypes. We then reordered the latent phenotypes in each split to match the
order of latent phenotypes in the original dataset based on the Gale–Shapley
stable marriage algorithm (Gale and Shapley, 1962) using the correlation
between latent phenotype weights as the preferences. The matching procedure
was repeated for 1000 random splits of the data. The first five latent
phenotypes in each dataset had an inter-split $r^{2}$ greater than 0.2 in 95%
or more or random splits (Figure 6, Figure 11 in the appendix for
correlation). Both datasets also had some reliable low-variance latent
phenotypes (69-73 and 83 in HCP, 35-39 in PNC). Outside of these latent
phenotypes, the inter-split correlation of the weights was low. We postulate
that this could be due to the loadings for remaining factors being driven by
sample idiosyncrasies or the remaining relationships between phenotype
measures being weak enough that noise can perturb their estimation. The
predictability analysis reveals that the first three HCP latent phenotypes and
the fifth HCP latent phenotype, along with the first PNC component, exhibit a
correlation mean between true and predicted values that is above 0.15 (Figures
2 and 3). That predictability decays quickly beyond them with an increase in
the prediction performance for a few of the last components that do not
explain much variance.

###  3.5 Predictability of original phenotypes after training on phenotypes
reconstructed from latent space

The previous sections investigated the predictability of individual phenotypes
measures, and of latent phenotypes derived from them. We saw that only the
first five latent phenotypes in each dataset were reliable (the first 5
components explain 40% of the variance in HCP and 45% on PNC), but explaining
95% of the variance would require most of the latent phenotypes (53 on the HCP
and 28 in PNC). If we assume that variance explained by unreliable latent
phenotypes will not be predictable from functional connectivity data, then
these results suggest the intriguing possibility that most of the variance
present across phenotype measures might not be predictable from functional
connectivity data (at least using linear prediction models in a moderately
sized sample of about 1000 subjects for both HCP and PNC). We can test this by
comparing the predictive performance of models trained using progressively
larger proportions of the variance in the original phenotypes to the
performance of models trained on the original phenotypes themselves. If a
model trained to predict a reduced variance representation of the phenotypes
does as well as models trained with the original phenotypes, then this will
indicate that that reduced variance is the maximally predictable portion of
the variance.

To test this possibility, we took advantage of the fact that the SVD of a
dataset can be used to generate the best possible rank $k$ approximation of
that dataset, in the least squares sense ( Appendix, section 5.2). This
corresponds to reconstructing each phenotype measure by considering only those
relationships to all other phenotype measures that explain the most variance.
Specifically, we used the first five latent phenotypes, as those were
reliability identified (Figure 6), to produce rank-1 through rank-5
approximate reconstructions of the phenotype measures in each resampling
training set. We then evaluated models trained to predict the reconstructed
phenotypes against the true, original phenotypes in the validation set. For
both datasets, the reconstructed prediction of rank-1 to rank-5 was very
similar to that obtained using the original phenotypes. (PNC: Figure 3c, HCP:
Figure 2c).

To perform a quantitative comparison between the 5 reconstruction models and
the baseline one, we performed a critical difference analysis (Demšar, 2006).
This method is used to statistically compare the prediction performance of
several models across many different prediction problems (phenotype measures,
in this case), to determine which subsets have significant differences in
performance relative to other subsets. In the case of HCP data, the post-hoc
Tukey HSD test revealed a significant decline in performance when employing a
single latent phenotype compared to all other groups (Figure 12a).
Nevertheless, there was no statistically significant difference in performance
across all measures when employing 2, 3, 4, 5, or all latent phenotypes. For
the PNC dataset, there were no significant differences in performance between
utilizing all latent phenotypes and any of the latent phenotypes.

##  4 Discussion

##### Phenotype measures are predictable in both HCP and PNC.

In this study, we examined prediction of behavioral phenotypes from functional
connectivity data. Several studies have trained different kinds of models to
predict a variety of phenotype targets, but have mainly focused on one dataset
(Chen et al., 2023; Bertolero and Bassett, 2020) or have used PCA with varimax
transformation for dimensionality reduction (Ooi et al., 2022). As a first
experiment, we tested phenotype prediction from functional connectivity in
both the Human Connectome Project (HCP) and Philadelphia Neurodevelopmental
Cohort (PNC) datasets. We found that phenotypes can be predicted from resting
state functional connectivity in both datasets (Figure 2b and Figure 3b).
Similar to other studies, we observed correlations between predicted and true
values in the range of 0.1-0.4 for the majority of phenotypes. For example,
the top three phenotypes that could be predicted with the highest performance
were Working memory score across all memory tasks (WM_Task_ACC), 2-min walk
endurance test (Endurance_Unadj), Attention problems (ASR_ATTN_T) in the HCP
(Figure 2b) and Professional Verbal Reasoning Test Total Correct Responses for
All Test Trials (PVRT_CR) Wide Range Achievement Test’s Wide Range Assessment
Test 4 Total Raw Score (WRAT_CR_RAW) and Primary Mental Abilities total
correct response (PMAT_CR) in the PNC dataset (Figure 3b).

##### Regression of the confounds is important.

Studies predicting phenotypes from imaging data do not always control for
demographic confounds in their analyses (Chyzhyk et al., 2022). While this may
make sense in some cases (as we discuss below), failure to control for these
confounding variables can inflate estimates of how predictable some phenotypes
are if those phenotypes are correlated with the demographic confounds. To
evaluate the degree to which this could be an issue, we considered the two
most common potential confounds: age and sex. We regressed age, sex, age
squared and the interactions of age and age squared with sex from our
prediction targets – phenotype measures – and evaluated differences in
predictive performance. This showed that confounds can inflate the prediction
results if not accounted for; this is particularly visible, for instance, in
the relation between the phenotypic variable “strength unadjusted” and sex in
the HCP dataset ($r=0.6$ before; $r=0.1$ after adjustment; Figure 2a and
Figure 2b). It is also important to note that it is essential to handle
confounding separately for the training and test data. Otherwise, attempting
to control for these variables across the entire dataset can undermine the
independence of the training and testing data (Chyzhyk et al., 2022).

Finally, it is worth noting that it may be useful to keep the confounding
variables in either the prediction targets or the imaging data, depending on
the purpose of the prediction model. One example would be age, which is
sometimes regressed out as a confounding variable, and sometimes added as an
additional predictor to explain part of the variance. For example, in the
context of predicting Alzheimer’s disease, not regressing age out of imaging
might help prediction models that can consider combinations of age and other
features. It is still an open question, and left as an additional researcher’s
degree of freedom, if the confounds should be removed from the image, targets,
or both. In this manuscript, we explored the effects of removing the confounds
from the phenotypes but future work should consider regressing the confounds
from the images.

##### SVD latent variables are not more predictable than individual phenotype
measures.

Many of the phenotype measures collected in large batteries are correlated to
some extent. As discussed earlier, dimensionality reduction techniques can be
applied to transform phenotype measures into a latent variable representation.
These variables are usually uncorrelated or independent and, informally,
explain different aspects of the phenotype measures. Beyond the scientific
reasons for generating them, latent variables may also be cleaner than
individual phenotype measures, if the individual measures are taken to be
noisy measurements of an underlying construct. And, finally, reducing
dimensionality also reduces the number of prediction targets to consider.

Several studies have constructed uncorrelated latent variable representations,
with each variable explaining different aspects of the phenotypes (Schöttner
et al., 2023; Ooi et al., 2022; Chen et al., 2022). For example, Ooi et al.
(2022) computes a factor analysis of the behavioral scores and predicts the
latent phenotypes instead of the raw phenotypes. They observed that latent
phenotypes, in particular the first three, are more predictable than using
individual measures and that functional connectivity outperforms other
modalities to predict behavioral phenotypes. In this study, we also
transformed the phenotype measures into a latent variable representation, and
used those latent variables as targets for prediction experiments. Figure 2d
and Figure 3d illustrates that even though the first components have the
highest prediction performance, the obtained performance is not higher than
that obtained for the untransformed phenotype measures (Figure 2b and Figure
3b). One important point is to make a distinction between predictability and
reliability, as a perfectly reliable phenotype might still not be predictable
or, even worse, might not be related to the phenomena being studied. Finn and
Rosenberg (2021) illustrates this point by making an analogy between
functional connectivity (FC) and barcodes. Barcodes are unique patterns, but
they have no intrinsic connection to any noteworthy traits of the items they
label. Therefore barcodes can have excellent accuracy for identification
purposes while offering limited value in predicting behavior. In simpler
terms, FC might be one-of-a-kind yet fail to provide meaningful insights (high
reliability but low validity).

##### The majority of SVD latent variables are needed to explain 95% of
phenotype variance, but only the first few can be reliably estimated.

Phenotype prediction is a challenging task, with performance often being
negligible, in terms of $r^{2}$, if not undistinguishable from chance level.
Ooi et al. (2022) showed that latent variables derived from phenotypes using
factor analysis could be more predictable than using the untransformed
phenotypes in the HCP and ABCD datasets. We wanted to assess if this finding
would replicate with a different dimensionality reduction algorithm, and
generalize to other datasets.

The present study employed SVD because it is the most efficient approach to
producing low-rank data approximations. The questionnaires used to produce the
phenotype measures in each study are chosen to span a range of domains of
cognition. However, it is quite likely that each phenotype measure draws from
more than one latent construct, leading to a complex correlation structure
between them. This correlation structure – identified with SVD, PCA, or FA –
allows robust identification of latent variables, and has been used to
guarantee the performance of transfer learning algorithms (He et al., 2022).
The resulting latent variables may also be more interpretable because they are
uncorrelated.

For both datasets, the majority of the latent variables were required to
account for 95% of the variance; at the same time, the first variables explain
more variance than most others (Figure 5). Because of this behavior, we
hypothesised that the predictability of each latent variable would be
correlated with how much variance it explains. As discussed above and visible
from Figure 2d, the observed phenomenon is more complicated than that. Over
the first few latent variables, there was no clear relationship between the
amount of variance explained and predictability. This is particularly visible
for the HCP data, where the first latent variable, which explains 16% of the
variance, was not the most predictable one. At the same time, predictability
decays rapidly after the first few latent variables (Figure 2d).

This latter finding prompted us to study the reproducibility of the loadings
associated with each latent variable. To do this, we used the Gale–Shapley
stable marriage algorithm (Gale and Shapley, 1962), where we used the
correlation to match latent phenotypes across different splits. After further
investigation, we observed that, for both datasets, only the loadings for the
first five components were reliably identified on 1000 repetitions (Figure 6),
and reliability quickly decays for the remaining factors. We think that due to
the mathematical properties of the SVD (that requires all components to be
orthonormal), we see an increase of reliability in the low-variance phenotypes
again. The increase in reliability for the first components might be related
to their increased predictability compared to the other latent phenotypes.
Further highlighting this complex relation between predictability and
reliability, we also noticed that, in the HCP dataset, some of the variables
explaining the least variance are most predictable. This experiment is
important as it showcases that, beyond five latent phenotypes, there would be
no guarantee of finding the same latent phenotypes in different samples. If we
cannot even ensure reliable splits within the same dataset, our ability to
uncover relationships in out-of-sample associations becomes increasingly
uncertain. Our results raise an interesting question: why do latent features
beyond the fifth one become unreliable, and why do they explain so little
variance? One simple guess would be that starting from the fifth component,
what is predominantly being captured is either noise or variance that occurs
on very few subjects, as opposed to meaningful, structured information
inherent in the data.

##### A low-dimensional representation of phenotype variables has similar
predictability to the original phenotypes.

The unreliability from the fifth latent phenotype onward prompted us to
question how much information could be predicted using only the reliable
latent variables. Figure 2c and Figure 3c demonstrate that only a few latent
variables are necessary to adequately reconstruct the training set phenotype
measures, and achieve similar performance on the validation set as a model
trained using the original phenotype measures. This finding suggests that
relations between phenotype measures that are present across many participants
may be captured primarily on the first few latent variables. Therefore, SVD
could serve as a denoising algorithm, providing means to work with fewer
latent variables in training but still yielding as good or better prediction
performance as using the original phenotype measures.

The relationship between noise and reliability and predictive models is
crucial but, while the importance of sample size for statistical power is
widely acknowledged, the consideration of measurement reliability when
estimating the required sample sizes remains an under-addressed aspect (Zuo et
al., 2019). Following this logic, Gell et al. (2023); Nikolaidis et al. (2022)
propose that focusing on the reliability of phenotypic measurements during
target selection and choosing only those with high reliability might improve
the performance of brain-behavior models. However, due to the notable
variability in reliability, the proposed sample size requirements may not
universally apply (Rosenberg and Finn, 2022; Chen et al., 2023). As a
significant consequence, when the reliability of a measurement diminishes, a
larger number of participants will be needed to accurately identify a
correlation between two measures (Zuo et al., 2019). Some studies have found
that a larger sample size helps obtain a higher prediction performance, but
carefully choosing reliable targets has a bigger impact on the model’s
performance (Gell et al., 2023; Nikolaidis et al., 2022). One remaining
question is whether using low-rank SVD reconstructions of the phenotype
measures lessens the sample sizes needed to obtain a given prediction
performance. This could be tested by running resampling experiments with
smaller sample sizes, but this is beyond the scope of the present paper.

In this paper, we did not assess the reliability of the features that are
important for prediction. Many works that trained brain-behavior models use a
variety of interpretation techniques – e.g., feature weights or saliency maps
– to analyze which brain regions were more relevant for prediction (Finn et
al., 2015; Jiang et al., 2020; Chen et al., 2022). While reliability of the
prediction targets has not been the focus of some of these studies (Finn et
al., 2015; Jiang et al., 2020; Chen et al., 2022)., the feature weights of
models frequently show moderate to low test-retest reliability as well.
Surprisingly, a linear transformation to these feature weights (i.e., Haufe-
transformed weights (Haufe et al., 2014)) demonstrated greater reliability
compared to untransformed weights (Tian and Zalesky, 2021; Chen et al., 2023).

##### Limitations of the current work

To overcome the scarcity of data, a few studies have used transfer learning to
improve the performance of brain-behavior models (Holderrieth et al., 2022; He
et al., 2022; Chopra et al., 2022; Wulan et al., 2024) and shown that
information from large datasets with rich phenotypic data can be used to
improve performance on smaller datasets. However, there are still many
implementation details that need to be further investigated before being able
to transfer phenotype models from large to small datasets successfully.
Generalizability, the broad applicability of a model, is closely linked to the
concept of transfer learning. The main difference is that while
generalizability refers to the model’s ability to perform well on new, unseen
data, in transfer learning, a model predicting a phenotype is first trained on
a large dataset and then the model is refined to improve its performance on
specific smaller datasets, combining the general information learned from the
larger dataset with information specific to the smaller dataset. A few studies
have evaluated the generalizability of models trained on one dataset to other
datasets, for a variety of brain-behavioral phenotypes, and have shown that
fluid intelligence is one of the few targets for which models have shown
moderate performance when tested on new datasets (Wu et al., 2022; Tong et
al., 2022). In this study, we did not explore transfer learning, as we
explored the idea shown by previous research (Nikolaidis et al., 2022; Gell et
al., 2023) that by using more reliable prediction targets (in our case
obtained from latent components), we could improve the model prediction but as
we have seen the using latent phenotypes did not improve the performance.

When pre-processing the phenotypes, we z-scored all phenotypes, set values
that were above and below 3 standard deviations to zeros and treated those
values as missing by the imputation algorithm. This choice of how to deal with
outliers could have an impact on our analysis and was one of the possible
researcher degrees of freedom in our analysis. In particular, by making this
choice, we are eliminating the most severe scores. A possible alternative
would have been to use winsorizing instead of treating the severe scores as
missing. Another limitation of our preprocessing stems from the fact that we
used different processing pipelines from HCP and PNC. While for the HCP we
were using the already pre-processed data for HCP, we pre-processed the data
using fMRIPrep. Because of this choice, we had additional information about
motion parameters for PNC and used this information to censor time points that
had excessive motion, but we did not censor time points with excessive motion
in HCP.

Another limitation of the current work stems from our reliance on a ridge
regression model, as this can only capture a linear relationship between the
functional connectivity data used as input and the phenotype measures used as
prediction targets. It is still debated if the brain-behavior relationship can
be better predicted using linear, and therefore more explainable models, or
benefit from the non-linear models. While Bertolero and Bassett (2020) showed
that deep neural networks can model simple and complex relationships between
brain and behavior, He et al. (2020) defended that deep neural networks and
kernel regression yield similar levels of accuracy when it comes to predicting
behavior and demographics through functional connectivity analysis. On the
other hand, Pervaiz et al. (2020) and Schulz et al. (2020) showed that simple
linear models perform very similarly to more complex ones. The second
limitation is the use of a linear method (SVD) to derive latent variables from
the phenotype measures. SVD captures covariance relationships, a linear
measure of association between those measures. It is possible that there are
also non-linear relationships between the phenotypes, which could be
identified with non-linear dimensionality reduction approaches such as auto-
encoders. Here, we chose to use both a linear model and a linear
dimensionality reduction approach to set a baseline for future analyses and
facilitate comparison with previous work.

In conclusion, we showed the importance of controlling for age and sex
confounds when creating a brain-behavior model. Failure to remove them could
lead to artificially inflated prediction results. Our main contribution is
showing that, by reconstructing phenotype variables using the first few latent
SVD variables, we could obtain a very similar performance training a model on
the original variables. This suggests that most of the information about
phenotype that can be identified using a linear model trained on functional
connectivity data is present only in the first latent phenotypes. We suggest
that future research should further explore the usage of latent variables
derived from phenotypes.

##### Acknowledgements

This research was supported by the National Institute of Mental Health
Intramural Research Program: ZIC-MH002968 (Dylan Nielson, Gabriel Loewinger,
Patrick McClure, Francisco Pereira) and ZIC-MH002960 (Jessica Dafflon, Eric
Earl, Dustin Moraczewski, Adam Thomas). This research utilized the
computational resources of the NIH HPC Biowulf cluster (http://hpc.nih.gov).

##### Authorship contribution statement

Each author contributed as follows:
JD: data curation, design the study, analyzed the data, contributed to the
manuscript
DMN: design the study, analyzed the data, contributed to the manuscript
DM: design the study, data curation, contributed to the manuscript
EE: data curation, contributed to the manuscript
GL: methodology feedback, contributed to the manuscript
AGT: design the study, contributed to the manuscript
PM: design the study, contributed to the manuscript
FP: data curation, design the study, contributed to the manuscript

##### Data and Code availability

All code is available at https://github.com/JessyD/brain-phenotypes. There the
user can also find the lists of participants’ IDs and behavior scores utilized
for both datasets. Both PNC and HCP are publicly available and can be obtained
after acceptance of their respective data agreement.

## References

  * Abraham et al. (2014) A. Abraham, F. Pedregosa, M. Eickenberg, P. Gervais, A. Mueller, J. Kossaifi, A. Gramfort, B. Thirion, and G. Varoquaux.  Machine learning for neuroimaging with scikit-learn.  _Frontiers in Neuroinformatics_ , 8, 2014.  ISSN 1662-5196.  doi: 10.3389/fninf.2014.00014.  URL https://www.frontiersin.org/articles/10.3389/fninf.2014.00014/full.
  * Alexander et al. (2017) L. M. Alexander, J. Escalera, L. Ai, C. Andreotti, K. Febre, A. Mangone, N. Vega-Potler, N. Langer, A. Alexander, M. Kovacs, S. Litke, B. O’Hagan, J. Andersen, B. Bronstein, A. Bui, M. Bushey, H. Butler, V. Castagna, N. Camacho, E. Chan, D. Citera, J. Clucas, S. Cohen, S. Dufek, M. Eaves, B. Fradera, J. Gardner, N. Grant-Villegas, G. Green, C. Gregory, E. Hart, S. Harris, M. Horton, D. Kahn, K. Kabotyanski, B. Karmel, S. P. Kelly, K. Kleinman, B. Koo, E. Kramer, E. Lennon, C. Lord, G. Mantello, A. Margolis, K. R. Merikangas, J. Milham, G. Minniti, R. Neuhaus, A. Levine, Y. Osman, L. C. Parra, K. R. Pugh, A. Racanello, A. Restrepo, T. Saltzman, B. Septimus, R. Tobe, R. Waltz, A. Williams, A. Yeo, F. X. Castellanos, A. Klein, T. Paus, B. L. Leventhal, R. C. Craddock, H. S. Koplewicz, and M. P. Milham.  An open resource for transdiagnostic research in pediatric mental health and learning disorders.  _Sci Data_ , 4:170181, Dec 2017.  doi: 10.1038/sdata.2017.181.
  * Avants et al. (2008) B. Avants, C. Epstein, M. Grossman, and J. Gee.  Symmetric diffeomorphic image registration with cross-correlation: Evaluating automated labeling of elderly and neurodegenerative brain.  _Medical Image Analysis_ , 12(1):26–41, 2008\.  ISSN 1361-8415.  doi: 10.1016/j.media.2007.06.004.  URL http://www.sciencedirect.com/science/article/pii/S1361841507000606.
  * Barch et al. (2013) D. M. Barch, G. C. Burgess, M. P. Harms, S. E. Petersen, B. L. Schlaggar, M. Corbetta, M. F. Glasser, S. Curtiss, S. Dixit, C. Feldt, D. Nolan, E. Bryant, T. Hartley, O. Footer, J. M. Bjork, R. Poldrack, S. Smith, H. Johansen-Berg, A. Z. Snyder, D. C. Van Essen, and WU-Minn HCP Consortium.  Function in the human connectome: task-fmri and individual differences in behavior.  _Neuroimage_ , 80:169–89, Oct 2013.  doi: 10.1016/j.neuroimage.2013.05.033.
  * Behzadi et al. (2007) Y. Behzadi, K. Restom, J. Liau, and T. T. Liu.  A component based noise correction method (CompCor) for BOLD and perfusion based fmri.  _NeuroImage_ , 37(1):90–101, 2007.  ISSN 1053-8119.  doi: 10.1016/j.neuroimage.2007.04.042.  URL http://www.sciencedirect.com/science/article/pii/S1053811907003837.
  * Bertolero and Bassett (2020) M. A. Bertolero and D. S. Bassett.  Deep neural networks carve the brain at its joints.  _arXiv preprint arXiv:2002.08891_ , 2020.
  * Botvinik-Nezer et al. (2020) R. Botvinik-Nezer, F. Holzmeister, C. F. Camerer, A. Dreber, J. Huber, M. Johannesson, M. Kirchler, R. Iwanir, J. A. Mumford, R. A. Adcock, et al.  Variability in the analysis of a single neuroimaging dataset by many teams.  _Nature_ , 582(7810):84–88, 2020.
  * Carp (2012) J. Carp.  On the plurality of (methodological) worlds: estimating the analytic flexibility of fmri experiments.  _Frontiers in neuroscience_ , 6:149, 2012.
  * Cecchetti and Handjaras (2022) L. Cecchetti and G. Handjaras.  Reproducible brain-wide association studies do not necessarily require thousands of individuals.  _PsyArXiv_ , 2022.
  * Chen et al. (2022) J. Chen, A. Tam, V. Kebets, C. Orban, L. Q. R. Ooi, C. L. Asplund, S. Marek, N. U. Dosenbach, S. B. Eickhoff, D. Bzdok, et al.  Shared and unique brain network features predict cognitive, personality, and mental health scores in the abcd study.  _Nature communications_ , 13(1):2217, 2022.
  * Chen et al. (2023) J. Chen, L. Q. R. Ooi, T. W. K. Tan, S. Zhang, J. Li, C. L. Asplund, S. B. Eickhoff, D. Bzdok, A. J. Holmes, and B. T. Yeo.  Relationship between prediction accuracy and feature importance reliability: An empirical and theoretical study.  _NeuroImage_ , 274:120115, 2023.
  * Chopra et al. (2022) S. Chopra, E. Dhamala, C. Lawhead, J. A. Ricard, E. R. Orchard, L. An, P. Chen, N. Wulan, P. Kumar, A. Rubenstein, J. Moses, L. Chen, P. Levi, A. Holmes, K. Aquino, A. Fornito, I. Harpaz-Rotem, L. T. Germine, J. T. Baker, B. T. Yeo, and A. J. Holmes.  Reliable and generalizable brain-based predictions of cognitive functioning across common psychiatric illness.  _medRxiv_ , 2022.  doi: 10.1101/2022.12.08.22283232.  URL https://www.medrxiv.org/content/early/2022/12/15/2022.12.08.22283232.
  * Chyzhyk et al. (2022) D. Chyzhyk, G. Varoquaux, M. Milham, and B. Thirion.  How to remove or control confounds in predictive models, with applications to brain biomarkers.  _GigaScience_ , 11:giac014, 2022.
  * Cox and Hyde (1997) R. W. Cox and J. S. Hyde.  Software tools for analysis and visualization of fmri data.  _NMR in Biomedicine_ , 10(4-5):171–178, 1997\.  doi: 10.1002/(SICI)1099-1492(199706/08)10:4/5¡171::AID-NBM453¿3.0.CO;2-L.
  * Dafflon et al. (2022) J. Dafflon, P. F. Da Costa, F. Váša, R. P. Monti, D. Bzdok, P. J. Hellyer, F. Turkheimer, J. Smallwood, E. Jones, and R. Leech.  A guided multiverse study of neuroimaging analyses.  _Nature Communications_ , 13(1):3758, 2022.
  * Dale et al. (1999) A. M. Dale, B. Fischl, and M. I. Sereno.  Cortical surface-based analysis: I. segmentation and surface reconstruction.  _NeuroImage_ , 9(2):179–194, 1999.  ISSN 1053-8119.  doi: 10.1006/nimg.1998.0395.  URL http://www.sciencedirect.com/science/article/pii/S1053811998903950.
  * Demšar (2006) J. Demšar.  Statistical comparisons of classifiers over multiple data sets.  _The Journal of Machine learning research_ , 7:1–30, 2006\.
  * Earl et al. (2023) E. Earl, J. Dafflon, R. Salamanca-Giron, J. Faskowitz, A. Basavaraj, D. Moraczewski, F. Pereira, and A. Thomas.  Big neuroimaging dataset BIDS tabular phenotype tools, June 2023.  URL https://doi.org/10.5281/zenodo.8084023.
  * Esteban et al. (2018a) O. Esteban, R. Blair, C. J. Markiewicz, S. L. Berleant, C. Moodie, F. Ma, A. I. Isik, A. Erramuzpe, M. Kent, James D. andGoncalves, E. DuPre, K. R. Sitek, D. E. P. Gomez, D. J. Lurie, Z. Ye, R. A. Poldrack, and K. J. Gorgolewski.  fmriprep.  _Software_ , 2018a.  doi: 10.5281/zenodo.852659.
  * Esteban et al. (2018b) O. Esteban, C. Markiewicz, R. W. Blair, C. Moodie, A. I. Isik, A. Erramuzpe Aliaga, J. Kent, M. Goncalves, E. DuPre, M. Snyder, H. Oya, S. Ghosh, J. Wright, J. Durnez, R. Poldrack, and K. J. Gorgolewski.  fMRIPrep: a robust preprocessing pipeline for functional MRI.  _Nature Methods_ , 2018b.  doi: 10.1038/s41592-018-0235-4.
  * Esteban et al. (2019) O. Esteban, C. J. Markiewicz, R. W. Blair, C. A. Moodie, A. I. Isik, A. Erramuzpe, J. D. Kent, M. Goncalves, E. DuPre, M. Snyder, et al.  fmriprep: a robust preprocessing pipeline for functional mri.  _Nature methods_ , 16(1):111–116, 2019.
  * Evans et al. (2012) A. Evans, A. Janke, D. Collins, and S. Baillet.  Brain templates and atlases.  _NeuroImage_ , 62(2):911–922, 2012.  doi: 10.1016/j.neuroimage.2012.01.024.
  * Finn and Rosenberg (2021) E. S. Finn and M. D. Rosenberg.  Beyond fingerprinting: Choosing predictive connectomes over reliable connectomes.  _NeuroImage_ , 239:118254, 2021.
  * Finn et al. (2015) E. S. Finn, X. Shen, D. Scheinost, M. D. Rosenberg, J. Huang, M. M. Chun, X. Papademetris, and R. T. Constable.  Functional connectome fingerprinting: identifying individuals using patterns of brain connectivity.  _Nature neuroscience_ , 18(11):1664–1671, 2015\.
  * Fonov et al. (2009) V. Fonov, A. Evans, R. McKinstry, C. Almli, and D. Collins.  Unbiased nonlinear average age-appropriate brain templates from birth to adulthood.  _NeuroImage_ , 47, Supplement 1:S102, 2009.  doi: 10.1016/S1053-8119(09)70884-5.
  * Gale and Shapley (1962) D. Gale and L. S. Shapley.  College admissions and the stability of marriage.  _The American Mathematical Monthly_ , 69(1):9–15, 1962.
  * Gell et al. (2023) M. Gell, S. B. Eickhoff, A. Omidvarnia, V. Kueppers, K. R. Patil, T. D. Satterthwaite, V. I. Mueller, and R. Langner.  The burden of reliability: How measurement noise limits brain-behaviour predictions.  _BioRxiv_ , pages 2023–02, 2023.
  * Genon et al. (2022) S. Genon, S. B. Eickhoff, and S. Kharabian.  Linking interindividual variability in brain structure to behaviour.  _Nature Reviews Neuroscience_ , 23(5):307–318, 2022.
  * Glasser et al. (2013a) M. F. Glasser, S. N. Sotiropoulos, J. A. Wilson, T. S. Coalson, B. Fischl, J. L. Andersson, J. Xu, S. Jbabdi, M. Webster, J. R. Polimeni, D. C. Van Essen, and M. Jenkinson.  The minimal preprocessing pipelines for the human connectome project.  _NeuroImage_ , 80:105–124, 2013a.  ISSN 1053-8119.  doi: https://doi.org/10.1016/j.neuroimage.2013.04.127.  URL https://www.sciencedirect.com/science/article/pii/S1053811913005053.  Mapping the Connectome.
  * Glasser et al. (2013b) M. F. Glasser, S. N. Sotiropoulos, J. A. Wilson, T. S. Coalson, B. Fischl, J. L. Andersson, J. Xu, S. Jbabdi, M. Webster, J. R. Polimeni, D. C. Van Essen, and M. Jenkinson.  The minimal preprocessing pipelines for the human connectome project.  _NeuroImage_ , 80:105–124, 2013b.  ISSN 1053-8119.  doi: 10.1016/j.neuroimage.2013.04.127.  URL http://www.sciencedirect.com/science/article/pii/S1053811913005053.
  * Gorgolewski et al. (2011) K. Gorgolewski, C. D. Burns, C. Madison, D. Clark, Y. O. Halchenko, M. L. Waskom, and S. Ghosh.  Nipype: a flexible, lightweight and extensible neuroimaging data processing framework in python.  _Frontiers in Neuroinformatics_ , 5:13, 2011.  doi: 10.3389/fninf.2011.00013.
  * Gorgolewski et al. (2018) K. J. Gorgolewski, O. Esteban, C. J. Markiewicz, E. Ziegler, D. G. Ellis, M. P. Notter, D. Jarecka, H. Johnson, C. Burns, A. Manhães-Savio, C. Hamalainen, B. Yvernault, T. Salo, K. Jordan, M. Goncalves, M. Waskom, D. Clark, J. Wong, F. Loney, M. Modat, B. E. Dewey, C. Madison, M. Visconti di Oleggio Castello, M. G. Clark, M. Dayan, D. Clark, A. Keshavan, B. Pinsard, A. Gramfort, S. Berleant, D. M. Nielson, S. Bougacha, G. Varoquaux, B. Cipollini, R. Markello, A. Rokem, B. Moloney, Y. O. Halchenko, D. Wassermann, M. Hanke, C. Horea, J. Kaczmarzyk, G. de Hollander, E. DuPre, A. Gillman, D. Mordom, C. Buchanan, R. Tungaraza, W. M. Pauli, S. Iqbal, S. Sikka, M. Mancini, Y. Schwartz, I. B. Malone, M. Dubois, C. Frohlich, D. Welch, J. Forbes, J. Kent, A. Watanabe, C. Cumba, J. M. Huntenburg, E. Kastman, B. N. Nichols, A. Eshaghi, D. Ginsburg, A. Schaefer, B. Acland, S. Giavasis, J. Kleesiek, D. Erickson, R. Küttner, C. Haselgrove, C. Correa, A. Ghayoor, F. Liem, J. Millman, D. Haehn, J. Lai, D. Zhou, R. Blair, T. Glatard, M. Renfro, S. Liu, A. E. Kahn, F. Pérez-García, W. Triplett, L. Lampe, J. Stadler, X.-Z. Kong, M. Hallquist, A. Chetverikov, J. Salvatore, A. Park, R. Poldrack, R. C. Craddock, S. Inati, O. Hinds, G. Cooper, L. N. Perkins, A. Marina, A. Mattfeld, M. Noel, L. Snoek, K. Matsubara, B. Cheung, S. Rothmei, S. Urchs, J. Durnez, F. Mertz, D. Geisler, A. Floren, S. Gerhard, P. Sharp, M. Molina-Romero, A. Weinstein, W. Broderick, V. Saase, S. K. Andberg, R. Harms, K. Schlamp, J. Arias, D. Papadopoulos Orfanos, C. Tarbert, A. Tambini, A. De La Vega, T. Nickson, M. Brett, M. Falkiewicz, K. Podranski, J. Linkersdörfer, G. Flandin, E. Ort, D. Shachnev, D. McNamee, A. Davison, J. Varada, I. Schwabacher, J. Pellman, M. Perez-Guevara, R. Khanuja, N. Pannetier, C. McDermottroe, and S. Ghosh.  Nipype.  _Software_ , 2018.  doi: 10.5281/zenodo.596855.
  * Goyal et al. (2022) N. Goyal, D. Moraczewski, P. A. Bandettini, E. S. Finn, and A. G. Thomas.  The positive–negative mode link between brain connectivity, demographics and behaviour: a pre-registered replication of smith et al . (2015).  _Royal Society Open Science_ , 9(2), Feb. 2022.  ISSN 2054-5703.  doi: 10.1098/rsos.201090.  URL http://dx.doi.org/10.1098/rsos.201090.
  * Gratton et al. (2022) C. Gratton, S. M. Nelson, and E. M. Gordon.  Brain-behavior correlations: Two paths toward reliability.  _Neuron_ , 110(9):1446–1449, 2022.
  * Greene and Constable (2023) A. S. Greene and R. T. Constable.  Clinical promise of brain-phenotype modeling: A review.  _JAMA psychiatry_ , 2023.
  * Greve and Fischl (2009) D. N. Greve and B. Fischl.  Accurate and robust brain image alignment using boundary-based registration.  _NeuroImage_ , 48(1):63–72, 2009.  ISSN 1095-9572.  doi: 10.1016/j.neuroimage.2009.06.060.
  * Gur et al. (2010) R. C. Gur, J. Richard, P. Hughett, M. E. Calkins, L. Macy, W. B. Bilker, C. Brensinger, and R. E. Gur.  A cognitive neuroscience-based computerized battery for efficient measurement of individual differences: standardization and initial construct validation.  _Journal of neuroscience methods_ , 187(2):254–262, 2010.
  * Haufe et al. (2014) S. Haufe, F. Meinecke, K. Görgen, S. Dähne, J.-D. Haynes, B. Blankertz, and F. Bießmann.  On the interpretation of weight vectors of linear models in multivariate neuroimaging.  _Neuroimage_ , 87:96–110, 2014.
  * He et al. (2020) T. He, R. Kong, A. J. Holmes, M. Nguyen, M. R. Sabuncu, S. B. Eickhoff, D. Bzdok, J. Feng, and B. T. Yeo.  Deep neural networks and kernel regression achieve comparable accuracies for functional connectivity prediction of behavior and demographics.  _NeuroImage_ , 206:116276, 2020.  ISSN 1053-8119.  doi: https://doi.org/10.1016/j.neuroimage.2019.116276.  URL https://www.sciencedirect.com/science/article/pii/S1053811919308675.
  * He et al. (2022) T. He, L. An, P. Chen, J. Chen, J. Feng, D. Bzdok, A. J. Holmes, S. B. Eickhoff, and B. T. Yeo.  Meta-matching as a simple framework to translate phenotypic predictive models from big to small data.  _Nature neuroscience_ , 25(6):795–804, 2022.
  * Herbold (2020) S. Herbold.  Autorank: A python package for automated ranking of classifiers.  _Journal of Open Source Software_ , 5(48):2173, 2020.
  * Hernan and Robins (2023) M. Hernan and J. Robins.  _Causal Inference: What If_.  Chapman & Hall/CRC Monographs on Statistics & Applied Probab. CRC Press, 2023.  ISBN 978-1-4200-7616-5.  URL https://books.google.com/books?id=_KnHIAAACAAJ.
  * Holderrieth et al. (2022) P. Holderrieth, S. Smith, and H. Peng.  Transfer learning for neuroimaging via re-use of deep neural network features.  _medRxiv_ , pages 2022–12, 2022.
  * Holmes and Patrick (2018) A. J. Holmes and L. M. Patrick.  The myth of optimality in clinical neuroscience.  _Trends in cognitive sciences_ , 22(3):241–257, 2018.
  * Jenkinson et al. (2002) M. Jenkinson, P. Bannister, M. Brady, and S. Smith.  Improved optimization for the robust and accurate linear registration and motion correction of brain images.  _NeuroImage_ , 17(2):825–841, 2002.  ISSN 1053-8119.  doi: 10.1006/nimg.2002.1132.  URL http://www.sciencedirect.com/science/article/pii/S1053811902911328.
  * Jiang et al. (2020) R. Jiang, V. D. Calhoun, L. Fan, N. Zuo, R. Jung, S. Qi, D. Lin, J. Li, C. Zhuo, M. Song, et al.  Gender differences in connectome-based predictions of individualized intelligence quotient and sub-domain scores.  _Cerebral cortex_ , 30(3):888–900, 2020.
  * Kaiser (1958) H. F. Kaiser.  The varimax criterion for analytic rotation in factor analysis.  _Psychometrika_ , 23(3):187–200, 1958.
  * Kennedy et al. (2016) D. N. Kennedy, C. Haselgrove, J. Riehl, N. Preuss, and R. Buccigrossi.  The nitrc image repository.  _Neuroimage_ , 124(Pt B):1069–1073, Jan 2016\.  doi: 10.1016/j.neuroimage.2015.05.074.
  * Klein et al. (2017) A. Klein, S. S. Ghosh, F. S. Bao, J. Giard, Y. Häme, E. Stavsky, N. Lee, B. Rossa, M. Reuter, E. C. Neto, and A. Keshavan.  Mindboggling morphometry of human brains.  _PLOS Computational Biology_ , 13(2):e1005350, 2017.  ISSN 1553-7358.  doi: 10.1371/journal.pcbi.1005350.  URL http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1005350.
  * Kong et al. (2019) R. Kong, J. Li, C. Orban, M. R. Sabuncu, H. Liu, A. Schaefer, N. Sun, X.-N. Zuo, A. J. Holmes, S. B. Eickhoff, and B. T. T. Yeo.  Spatial topography of individual-specific cortical networks predicts human cognition, personality, and emotion.  _Cereb Cortex_ , 29(6):2533–2551, Jun 2019.  doi: 10.1093/cercor/bhy123.
  * Lam et al. (2022) K. C. Lam, B. W. Mahony, A. Raznahan, and F. Pereira.  Interpretable (meta) factorization of clinical questionnaires to identify general dimensions of psychopathology.  2022\.
  * Lanczos (1964) C. Lanczos.  Evaluation of noisy data.  _Journal of the Society for Industrial and Applied Mathematics Series B Numerical Analysis_ , 1(1):76–85, 1964.  ISSN 0887-459X.  doi: 10.1137/0701007.  URL http://epubs.siam.org/doi/10.1137/0701007.
  * Marek et al. (2022) S. Marek, B. Tervo-Clemmens, F. J. Calabro, D. F. Montez, B. P. Kay, A. S. Hatoum, M. R. Donohue, W. Foran, R. L. Miller, T. J. Hendrickson, et al.  Reproducible brain-wide association studies require thousands of individuals.  _Nature_ , 603(7902):654–660, 2022.
  * Miller et al. (2016) K. L. Miller, F. Alfaro-Almagro, N. K. Bangerter, D. L. Thomas, E. Yacoub, J. Xu, A. J. Bartsch, S. Jbabdi, S. N. Sotiropoulos, J. L. Andersson, et al.  Multimodal population brain imaging in the uk biobank prospective epidemiological study.  _Nature neuroscience_ , 19(11):1523–1536, 2016\.
  * Nikolaidis et al. (2022) A. Nikolaidis, A. A. Chen, X. He, R. Shinohara, J. Vogelstein, M. Milham, and H. Shou.  Suboptimal phenotypic reliability impedes reproducible human neuroscience.  _BioRxiv_ , pages 2022–07, 2022.
  * Ooi et al. (2022) L. Q. R. Ooi, J. Chen, S. Zhang, R. Kong, A. Tam, J. Li, E. Dhamala, J. H. Zhou, A. J. Holmes, and B. T. T. Yeo.  Comparison of individualized behavioral predictions across anatomical, diffusion and functional connectivity mri.  _Neuroimage_ , 263:119636, Nov 2022.  doi: 10.1016/j.neuroimage.2022.119636.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, et al.  Scikit-learn: Machine learning in python.  _the Journal of machine Learning research_ , 12:2825–2830, 2011.
  * Pervaiz et al. (2020) U. Pervaiz, D. Vidaurre, M. W. Woolrich, and S. M. Smith.  Optimising network modelling methods for fmri.  _NeuroImage_ , 211:116604, 2020.  ISSN 1053-8119.  doi: https://doi.org/10.1016/j.neuroimage.2020.116604.  URL https://www.sciencedirect.com/science/article/pii/S1053811920300914.
  * Poldrack et al. (2020) R. A. Poldrack, G. Huckins, and G. Varoquaux.  Establishment of best practices for evidence for prediction: a review.  _JAMA psychiatry_ , 77(5):534–540, 2020.
  * Power et al. (2014) J. D. Power, A. Mitra, T. O. Laumann, A. Z. Snyder, B. L. Schlaggar, and S. E. Petersen.  Methods to detect, characterize, and remove motion artifact in resting state fmri.  _NeuroImage_ , 84(Supplement C):320–341, 2014\.  ISSN 1053-8119.  doi: 10.1016/j.neuroimage.2013.08.048.  URL http://www.sciencedirect.com/science/article/pii/S1053811913009117.
  * Pruim et al. (2015) R. H. R. Pruim, M. Mennes, D. van Rooij, A. Llera, J. K. Buitelaar, and C. F. Beckmann.  Ica-AROMA: A robust ICA-based strategy for removing motion artifacts from fmri data.  _NeuroImage_ , 112(Supplement C):267–277, 2015\.  ISSN 1053-8119.  doi: 10.1016/j.neuroimage.2015.02.064.  URL http://www.sciencedirect.com/science/article/pii/S1053811915001822.
  * Roalf et al. (2014) D. R. Roalf, R. E. Gur, K. Ruparel, M. E. Calkins, T. D. Satterthwaite, W. B. Bilker, H. Hakonarson, L. J. Harris, and R. C. Gur.  Within-individual variability in neurocognitive performance: age-and sex-related differences in children and youths from ages 8 to 21.  _Neuropsychology_ , 28(4):506, 2014.
  * Rosenberg and Finn (2022) M. D. Rosenberg and E. S. Finn.  How to establish robust brain–behavior relationships without thousands of individuals.  _Nature Neuroscience_ , 25(7):835–837, 2022.
  * Satterthwaite et al. (2013) T. D. Satterthwaite, M. A. Elliott, R. T. Gerraty, K. Ruparel, J. Loughead, M. E. Calkins, S. B. Eickhoff, H. Hakonarson, R. C. Gur, R. E. Gur, and D. H. Wolf.  An improved framework for confound regression and filtering for control of motion artifact in the preprocessing of resting-state functional connectivity data.  _NeuroImage_ , 64(1):240–256, 2013.  ISSN 10538119.  doi: 10.1016/j.neuroimage.2012.08.052.  URL http://linkinghub.elsevier.com/retrieve/pii/S1053811912008609.
  * Satterthwaite et al. (2016) T. D. Satterthwaite, J. J. Connolly, K. Ruparel, M. E. Calkins, C. Jackson, M. A. Elliott, D. R. Roalf, R. Hopson, K. Prabhakaran, M. Behr, H. Qiu, F. D. Mentch, R. Chiavacci, P. M. A. Sleiman, R. C. Gur, H. Hakonarson, and R. E. Gur.  The philadelphia neurodevelopmental cohort: A publicly available resource for the study of normal and abnormal brain development in youth.  _Neuroimage_ , 124(Pt B):1115–1119, Jan 2016\.  doi: 10.1016/j.neuroimage.2015.03.056.
  * Schaefer et al. (2018) A. Schaefer, R. Kong, E. M. Gordon, T. O. Laumann, X.-N. Zuo, A. J. Holmes, S. B. Eickhoff, and B. T. Yeo.  Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity mri.  _Cerebral cortex_ , 28(9):3095–3114, 2018.
  * Schöttner et al. (2023) M. Schöttner, T. A. W. Bolton, J. Patel, A. T. Nahálka, S. Vieira, and P. Hagmann.  Exploring the latent structure of behavior using the human connectome project’s data.  _Sci Rep_ , 13(1):713, Jan 2023.  doi: 10.1038/s41598-022-27101-1.
  * Schulz et al. (2020) M.-A. Schulz, B. T. T. Yeo, J. T. Vogelstein, J. Mourao-Miranada, J. N. Kather, K. Kording, B. Richards, and D. Bzdok.  Different scaling of linear models and deep learning in UKBiobank brain images versus machine-learning datasets.  _Nature Communications_ , 11(1), aug 2020.  doi: 10.1038/s41467-020-18037-z.  URL https://doi.org/10.1038%2Fs41467-020-18037-z.
  * Slotkin et al. (2012) J. Slotkin, M. Kallen, J. Griffith, S. Magasi, J. Salsman, C. Nowinski, et al.  Nih toolbox.  _Technical Manual.[Google Scholar]_ , 2012.
  * Smith et al. (2015) S. M. Smith, T. E. Nichols, D. Vidaurre, A. M. Winkler, T. E. Behrens, M. F. Glasser, K. Ugurbil, D. M. Barch, D. C. Van Essen, and K. L. Miller.  A positive-negative mode of population covariation links brain connectivity, demographics and behavior.  _Nature neuroscience_ , 18(11):1565–1567, 2015\.
  * Spisak et al. (2023) T. Spisak, U. Bingel, and T. D. Wager.  Multivariate bwas can be replicable with moderate sample sizes.  _Nature_ , 615(7951):E4–E7, 2023.
  * Sui et al. (2020) J. Sui, R. Jiang, J. Bustillo, and V. Calhoun.  Neuroimaging-based individualized prediction of cognition and behavior for mental disorders and health: methods and promises.  _Biological psychiatry_ , 88(11):818–828, 2020\.
  * Tian and Zalesky (2021) Y. Tian and A. Zalesky.  Machine learning prediction of cognition from functional connectivity: Are feature weights reliable?  _NeuroImage_ , 245:118648, 2021.  ISSN 1053-8119.  doi: https://doi.org/10.1016/j.neuroimage.2021.118648.  URL https://www.sciencedirect.com/science/article/pii/S1053811921009216.
  * Tong et al. (2022) X. Tong, H. Xie, N. Carlisle, G. A. Fonzo, D. J. Oathes, J. Jiang, and Y. Zhang.  Transdiagnostic connectome signatures from resting-state fmri predict individual-level intellectual capacity.  _Translational psychiatry_ , 12(1):367, 2022.
  * Tustison et al. (2010) N. J. Tustison, B. B. Avants, P. A. Cook, Y. Zheng, A. Egan, P. A. Yushkevich, and J. C. Gee.  N4itk: Improved n3 bias correction.  _IEEE Transactions on Medical Imaging_ , 29(6):1310–1320, 2010.  ISSN 0278-0062.  doi: 10.1109/TMI.2010.2046908.
  * Van den Akker et al. (2021) O. Van den Akker, S. Weston, L. Campbell, B. Chopik, R. Damian, P. Davis-Kean, A. Hall, J. Kosie, E. Kruse, J. Olsen, et al.  Preregistration of secondary data analysis: A template and tutorial.  _Meta-psychology_ , 5:2625, 2021.
  * Van Essen et al. (2012) D. C. Van Essen, K. Ugurbil, E. Auerbach, D. Barch, T. E. J. Behrens, R. Bucholz, A. Chang, L. Chen, M. Corbetta, S. W. Curtiss, S. Della Penna, D. Feinberg, M. F. Glasser, N. Harel, A. C. Heath, L. Larson-Prior, D. Marcus, G. Michalareas, S. Moeller, R. Oostenveld, S. E. Petersen, F. Prior, B. L. Schlaggar, S. M. Smith, A. Z. Snyder, J. Xu, E. Yacoub, and WU-Minn HCP Consortium.  The human connectome project: a data acquisition perspective.  _Neuroimage_ , 62(4):2222–31, Oct 2012.  doi: 10.1016/j.neuroimage.2012.02.018.
  * Wu et al. (2022) J. Wu, J. Li, S. B. Eickhoff, F. Hoffstaedter, M. Hanke, B. T. Yeo, and S. Genon.  Cross-cohort replicability and generalizability of connectivity-based psychometric prediction patterns.  _Neuroimage_ , 262:119569, 2022.
  * Wulan et al. (2024) N. Wulan, L. An, C. Zhang, R. Kong, P. Chen, D. Bzdok, S. B. Eickhoff, A. J. Holmes, and B. T. Yeo.  Translating phenotypic prediction models from big to small anatomical mri data using meta-matching.  _bioRxiv_ , pages 2023–12, 2024.
  * Zelenina et al. (2023) M. Zelenina, D. Pine, A. Stringaris, and D. Nielson.  Validation of cbcl depression scores of adolescents in three independent datasets.  2023\.
  * Zhang et al. (2001) Y. Zhang, M. Brady, and S. Smith.  Segmentation of brain MR images through a hidden markov random field model and the expectation-maximization algorithm.  _IEEE Transactions on Medical Imaging_ , 20(1):45–57, 2001.  ISSN 0278-0062.  doi: 10.1109/42.906424.
  * Zuo et al. (2019) X.-N. Zuo, T. Xu, and M. P. Milham.  Harnessing reliability for neuroscience research.  _Nature human behaviour_ , 3(8):768–771, 2019\.

##  5 Appendix

###  5.1 Additional details on fmriprep pre-processing pipeline

Results included in this manuscript come from preprocessing performed using
_fMRIPrep_ 21.0.2 (Esteban et al. (2018b); Esteban et al. (2018a);
RRID:SCR_016216), which is based on _Nipype_ 1.6.1 (Gorgolewski et al. (2011);
Gorgolewski et al. (2018); RRID:SCR_002502).

Preprocessing of B0 inhomogeneity mappings

A total of 1 fieldmaps were found available within the input BIDS structure. A
_B0_ nonuniformity map (or _fieldmap_) was estimated from the phase-drift
map(s) measure with two consecutive GRE (gradient-recalled echo) acquisitions.
The corresponding phase-map(s) were phase-unwrapped with prelude (FSL
6.0.5.1:57b01774). A total of 1 T1-weighted (T1w) images were found within the
input BIDS dataset. The T1-weighted (T1w) image was corrected for intensity
non-uniformity (INU) with N4BiasFieldCorrection (Tustison et al., 2010),
distributed with ANTs 2.3.3 (Avants et al., 2008, RRID:SCR_004757), and used
as T1w-reference throughout the workflow. The T1w-reference was then skull-
stripped with a _Nipype_ implementation of the antsBrainExtraction.sh workflow
(from ANTs), using OASIS30ANTs as target template. Brain tissue segmentation
of cerebrospinal fluid (CSF), white-matter (WM) and gray-matter (GM) was
performed on the brain-extracted T1w using fast Zhang et al. (2001). Brain
surfaces were reconstructed using recon-all (FreeSurfer 6.0.1,
RRID:SCR_001847, Dale et al., 1999), and the brain mask estimated previously
was refined with a custom variation of the method to reconcile ANTs-derived
and FreeSurfer-derived segmentations of the cortical gray-matter of Mindboggle
(RRID:SCR_002438, Klein et al., 2017). Volume-based spatial normalization to
two standard spaces (MNI152NLin2009cAsym, MNI152NLin6Asym) was performed
through nonlinear registration with antsRegistration (ANTs 2.3.3), using
brain-extracted versions of both T1w reference and the T1w template. The
following templates were selected for spatial normalization: _ICBM 152
Nonlinear Asymmetrical template version 2009c_ [Fonov et al. (2009),
RRID:SCR_008796; TemplateFlow ID: MNI152NLin2009cAsym], _FSL’s MNI ICBM 152
non-linear 6th Generation Asymmetric Average Brain Stereotaxic Registration
Model_ [Evans et al. (2012), RRID:SCR_002823; TemplateFlow ID:
MNI152NLin6Asym].

Functional data preprocessing

For each of the 3 BOLD runs found per subject (across all tasks and sessions),
the following preprocessing was performed. First, a reference volume and its
skull-stripped version were generated using a custom methodology of
_fMRIPrep_. Head-motion parameters with respect to the BOLD reference
(transformation matrices, and six corresponding rotation and translation
parameters) are estimated before any spatiotemporal filtering using mcflirt
(FSL 6.0.5.1:57b01774, Jenkinson et al., 2002). BOLD runs were slice-time
corrected to 1.47s (0.5 of slice acquisition range 0s-2.94s) using 3dTshift
from AFNI (Cox and Hyde, 1997, RRID:SCR_005927). The BOLD time-series
(including slice-timing correction when applied) were resampled onto their
original, native space by applying the transforms to correct for head-motion.
These resampled BOLD time-series will be referred to as _preprocessed BOLD in
original space_ , or just _preprocessed BOLD_. The BOLD reference was then co-
registered to the T1w reference using bbregister (FreeSurfer) which implements
boundary-based registration (Greve and Fischl, 2009). Co-registration was
configured with six degrees of freedom. Several confounding time-series were
calculated based on the _preprocessed BOLD_ : framewise displacement (FD),
DVARS and three region-wise global signals. FD was computed using two
formulations following Power (absolute sum of relative motions, Power et al.
(2014)) and Jenkinson (relative root mean square displacement between affines,
Jenkinson et al. (2002)). FD and DVARS are calculated for each functional run,
both using their implementations in _Nipype_ (following the definitions by
Power et al., 2014). The three global signals are extracted within the CSF,
the WM, and the whole-brain masks. Additionally, a set of physiological
regressors were extracted to allow for component-based noise correction
(_CompCor_ , Behzadi et al., 2007). Principal components are estimated after
high-pass filtering the _preprocessed BOLD_ time-series (using a discrete
cosine filter with 128s cut-off) for the two _CompCor_ variants: temporal
(tCompCor) and anatomical (aCompCor). tCompCor components are then calculated
from the top 2% variable voxels within the brain mask. For aCompCor, three
probabilistic masks (CSF, WM and combined CSF+WM) are generated in anatomical
space. The implementation differs from that of Behzadi et al. in that instead
of eroding the masks by 2 pixels on BOLD space, the aCompCor masks are
subtracted a mask of pixels that likely contain a volume fraction of GM. This
mask is obtained by dilating a GM mask extracted from the FreeSurfer’s _aseg_
segmentation, and it ensures components are not extracted from voxels
containing a minimal fraction of GM. Finally, these masks are resampled into
BOLD space and binarized by thresholding at 0.99 (as in the original
implementation). Components are also calculated separately within the WM and
CSF masks. For each CompCor decomposition, the _k_ components with the largest
singular values are retained, such that the retained components’ time series
are sufficient to explain 50 percent of variance across the nuisance mask
(CSF, WM, combined, or temporal). The remaining components are dropped from
consideration. The head-motion estimates calculated in the correction step
were also placed within the corresponding confounds file. The confound time
series derived from head motion estimates and global signals were expanded
with the inclusion of temporal derivatives and quadratic terms for each
(Satterthwaite et al., 2013). Frames that exceeded a threshold of 0.5 mm FD or
1.5 standardised DVARS were annotated as motion outliers. The BOLD time-series
were resampled into standard space, generating a _preprocessed BOLD run in
MNI152NLin2009cAsym space_. First, a reference volume and its skull-stripped
version were generated using a custom methodology of _fMRIPrep_. The BOLD
time-series were resampled onto the following surfaces (FreeSurfer
reconstruction nomenclature): _fsaverage6_ , _fsaverage_. Automatic removal of
motion artifacts using independent component analysis (ICA-AROMA, Pruim et
al., 2015) was performed on the _preprocessed BOLD on MNI space_ time-series
after removal of non-steady state volumes and spatial smoothing with an
isotropic, Gaussian kernel of 6mm FWHM (full-width half-maximum).
Corresponding “non-aggresively” denoised runs were produced after such
smoothing. Additionally, the “aggressive” noise-regressors were collected and
placed in the corresponding confounds file. _Grayordinates_ files (Glasser et
al., 2013b) containing 91k samples were also generated using the highest-
resolution fsaverage as intermediate standardized surface space. All
resamplings can be performed with _a single interpolation step_ by composing
all the pertinent transformations (i.e. head-motion transform matrices,
susceptibility distortion correction when available, and co-registrations to
anatomical and output spaces). Gridded (volumetric) resamplings were performed
using antsApplyTransforms (ANTs), configured with Lanczos interpolation to
minimize the smoothing effects of other kernels (Lanczos, 1964). Non-gridded
(surface) resamplings were performed using mri_vol2surf (FreeSurfer).

Many internal operations of _fMRIPrep_ use _Nilearn_ 0.8.1 (Abraham et al.,
2014, RRID:SCR_001362), mostly within the functional processing workflow. For
more details of the pipeline, see the section corresponding to workflows in
_fMRIPrep_ ’s documentation.

###  5.2 Singular Value Decomposition (SVD)

####  5.2.1 Definition

Given a $n\times d$ data matrix X, the SVD of X is

$X=USV^{\prime}$

where $U$ is $n\times k$, $S$ is $k\times k$, and $V$ is $d\times k$, for a
given number of latent variables $k$. The matrices $U$ and $V$ are called the
left and right singular vector matrices (the columns are the singular
vectors), and $S$ is the singular value matrix (the diagonal contains the
singular values).

The matrices $U$, $S$, and $V$ have various useful properties:

  * •

$U$ and $V$ are orthonormal: their columns have norm 1, and are orthogonal to
each other.

  * •

As orthonormal matrices, $U$ and $V$ can be viewed as bases for the spaces
spanned by columns or rows of X, respectively, and where coordinates for X
would be $SV^{\prime}$ or $US$.

  * •

S is a diagonal matrix, with different values in each entry of the diagonal.

  * •

$k$ can only be as large as $\mathtt{min}(n,d)$; if $X$ is of rank lower than
that – which can happen if columns or rows are linearly dependent – it can be
smaller.

####  5.2.2 Dimensionality reduction with SVD

In the dimensionality terminology used in the paper, $US$ is the latent
variable matrix, and $V^{\prime}$ is the mixing matrix. The variables in the
low-dimensional space are uncorrelated, which makes them suitable for various
applications (e.g., visualization, and preprocessing for other methods that
assume uncorrelated variables).

In addition, the matrix $V$ can be used both to convert $X$ (or other
examples) in the $d$-dimensional space into the $k$-dimensional space, as

well as convert any examples in the low-dimensional space back into the
$d$-dimensional space by multiplying them by $V^{\prime}$.

Figure 7 (top) shows how each individual entry $x_{i,j}$ in matrix $X$ is
reconstructable as a multiplication of the $i^{th}$ row of $U$ ($u_{i,:})$,
scaled by the entries in the diagonal of $S$, and by the $j^{th}$ column of
$V^{\prime}$.

An essential step to take into consideration before applying SVD is
normalization as (1) it ensures scale-invariance in the SVD results,
preventing larger-scale variables from dominating the analysis and distorting
the interpretation of relationships between variables; (2) without
normalization, it becomes challenging to understand the relative importance
and contributions of variables due to varying scales, making the
interpretation less intuitive; (3) it enhances the numerical stability of SVD
computations, reducing the risk of numerical errors that can arise when data
scales vary widely.

Figure 7: Top: The support vector decomposition of matrix $X$ as a product of
orthonormal matrix $U$, diagonal matrix $S$, and the transpose of orthonormal
matrix $V$. Each entry $x_{ij}$ in matrix $X$ is reconstructable as a
multiplication of the $i^{th}$ row of $U$ ($u_{i,:}$, scaled by the entries in
the diagonal of $S$, by the $j^{th}$ column of $V^{\prime}$. Bottom: The
matrix $X$ can be expressed as a sum of rank 1 matrices $R_{1},\ldots,R_{k}$,
where $R_{l}$ is the outer product $u_{:,l}\otimes v^{\prime}_{:,l}$
multiplied by the scalar $s_{l,l}$.

####  5.2.3 Relationship between SVD and the data matrix

SVD provides the best approximation of the matrix X at any specified rank
(i.e., one can take the full SVD and expand it to show that it reconstructs X
as a sum of $k$ rank 1 matrices, in other words, $R_{1},\ldots,R_{k}$:

$\displaystyle X$ $\displaystyle=$ $\displaystyle R_{1}+\ldots+R_{k}$
$\displaystyle=$ $\displaystyle
U_{:,1}S_{1,1}V_{:,1}^{\prime}+\ldots+U_{:,k}S_{k,k}V_{:,k}^{\prime}$

as shown in Figure 7 (bottom)). Each of these matrices explains some of the
variance in $X$. If we sort the dimensions of the SVD from those that explain
the most to those that explain the least, adding $k$ matrices will yield the
best rank $k$ approximation of $X$, in terms of minimizing squared error.

If we take the diagonal of the matrix S, ${\bf s}=\mathtt{diag}(S)$, then the
percentage of variance accounted for by dimension $i$ is
$\frac{s_{i}^{2}}{\sum_{j=1}^{k}s_{j}^{2}}$. Adding up the variance for
dimensions $1-i$ will give you the total variance explained by the rank $i$
approximation. This is useful in data compression (best approximation),
denoising (many dimensions with little variance are capturing noise), or
matrix completion (if there are missing entries, the SVD reconstruction can be
a good multivariate imputation method, subject to certain conditions). All of
this results from the fact that, if minimizing squared error to the original
matrix $X$, the SVD with $k$ latent phenotypes is the best rank $k$
approximation of $X$.

####  5.2.4 Relationship between PCA and SVD

To prove the relationship between PCA and SVD, we assume that the dataset $X$
has been centered, i.e. where each column has mean 0. The covariance matrix
for a centered dataset X is $C=\frac{1}{n}X^{\prime}X$. Principal Component
Analysis is a spectral decomposition of $C$, i.e.

$C=E\Lambda E^{\prime}$

where $E$ is a $d\times d$ matrix, and $\Lambda$ is a diagonal matrix. The $E$
is an orthonormal matrix (column vectors are norm 1, and orthogonal to each
other), where columns are called eigenvectors. The diagonal of $\Lambda$,
${\bf\lambda}=\mathtt{diag}(\Lambda)$ contains the corresponding eigenvalues.
The matrix $E$ acts as a basis for reconstructing the covariance matrix E. The
fraction of variance explained by eigenvector $i$ is
$\frac{\lambda_{i}}{\sum_{j=1}^{d}\lambda_{j}}$.

The relationship between SVD and PCA becomes clear if we consider that C can
be expressed in terms of the SVD matrices, as follows

$\displaystyle C=\frac{1}{n}X^{\prime}X$ $\displaystyle=$
$\displaystyle\frac{1}{n}VS^{\prime}U^{\prime}USV^{\prime}$ $\displaystyle=$
$\displaystyle V\frac{1}{n}S^{\prime}SV^{\prime}\ (\mathrm{as\
}U^{\prime}U=I)$ $\displaystyle=$ $\displaystyle V\frac{1}{n}S^{2}V^{\prime}\
(\mathrm{as\ }S=S^{\prime})$

so the eigenvector matrix $E$ is the same as the right singular vector matrix
$V$ (with some caveats if X is not full rank).

###  5.3 List of all the phenotypes scores used from HCP and PNC

The tables below contain the variable names used in our study, the description
of what it refers to, and which category the questionnaire comes from.

##### Used phenotypes from the HCP Dataset

Variable | Description | Category
---|---|---
AngAffect_Unadj*222*: Those were used by Ooi et al. (2022); Kong et al. (2019); He et al. (2020). All papers used the same variables. | Anger-Affect Survey: Unadjusted | Emotion (NIH Toolbox)
AngAggr_Unadj* | Anger-Physical Aggression Survey: Unadjusted | Emotion (NIH Toolbox)
AngHostil_Unadj* | Anger-Hostility Survey: Unadjusted | Emotion (NIH Toolbox)
ASR_Aggr_T | Aggressive Behavior Gender and Age Adjusted T-score | Psychiatric/Life Function
ASR_Anxd_Pct | Anxious/Depressed Gender and Age Adjusted T-score | Psychiatric/Life Function
ASR_Attn_T | Attention Problems Gender and Age Adjusted T-score | Psychiatric/Life Function
ASR_Extn_T | Externalizing Gender and Age Adjusted T-score | Psychiatric/Life Function
ASR_Intn_T | Internalizing Gender and Age Adjusted T-score | Psychiatric/Life Function
ASR_Intr_T | Intrusive Gender and Age Adjusted T-score | Psychiatric/Life Function
ASR_Rule_T | Rule Breaking Behavior Gender and Age Adjusted T-score | Psychiatric/Life Function
ASR_Soma_T | Somatic Complaints Gender and Age Adjusted T-score | Psychiatric/Life Function
ASR_TAO_Sum | Sum of Thought, Attention, and Other Problems Raw Score | Psychiatric/Life Function
ASR_Thot_T | Thought Problems Gender and Age Adjusted T-score | Psychiatric/Life Function
ASR_Totp_T | Total Problems Gender and Age Adjusted T-score | Psychiatric/Life Function
ASR_Witd_T | Withdrawn Gender and Age Adjusted T-score | Psychiatric/Life Function
CardSort_Unadj* | Dimensional Change Card Sort Test: Unadjusted | Cognition (NIH Toolbox)
CogCrystalComp_AgeAdj | Cognition Crystallized Composite: Age Adjusted | Cognition (NIH Toolbox)
CogEarlyComp_AgeAdj | Cognition Early Childhood Composite: Age Adjusted | Cognition (NIH Toolbox)
CogFluidComp_AgeAdj | Cognition Fluid Composite: Age Adjusted | Cognition (NIH Toolbox)
CogTotalComp_AgeAdj | Cognition Total Composite Score: Age Adjusted | Cognition (NIH Toolbox)
DDisc_AUC_200 | Area Under the Curve for Delay Discounting of $200 | Cognition
DDisc_AUC_40K* | Area Under the Curve for Delay Discounting of $40,000 | Cognition
Dexterity_Unadj* | 9-hole Pegboard Dexterity Test: Unadjusted | Motor (NIH Toolbox)
Emotion_Task_Face_Acc* | Accuracy % during FACE blocks in EMOTION task | In-Scanner Task
EmotSupp_Unadj* | Emotional Support Survey: Unadjusted | Emotion (NIH Toolbox)
Endurance_Unadj* | 2-minute Walk Endurance Test: Unadjusted | Motor (NIH Toolbox)
ER40ANG* | Number of Correct Anger Identifications | Emotion (Penn ER Test)
ER40FEAR* | Number of Correct Fear Identifications | Emotion (Penn ER Test)
ER40NOE* | Number of Correct Neutral Identifications | Emotion (Penn ER Test)
ER40SAD* | Number of Correct Sad Identifications | Emotion (Penn ER Test)
ER40_CR* | Number of Correct Responses | Emotion (Penn ER Test)
ER40_CRT | Correct Responses Median Response Time (ms) | Emotion (Penn ER Test)
FearAffect_Unadj* | Fear-Affect Survey: Unadjusted | Emotion (NIH Toolbox)
FearSomat_Unadj* | Fear-Somatic Arousal Survey: Unadjusted | Emotion (NIH Toolbox)
Flanker_Unadj* | Flanker Test: Unadjusted Scale Score | Cognition (NIH Toolbox)
Friendship_Unadj* | Friendship Survey: Unadjusted | Emotion (NIH Toolbox)
GaitSpeed_Comp* | 4-Meter Walk Gait Speed Test: Computed Score | Motor (NIH Toolbox)
InstruSupp_Unadj* | Instrumental Support Survey: Unadjusted | Emotion (NIH Toolbox)
IWRD_TOT* | Word Memory: Total Number of Correct Responses | Cognition (Penn Tests)
Language_Task_Math_Avg | Language Task MATH difficulty level* | Language
Language_Task_Story_Avg | Language Task STORY difficulty level* | Language
LifeSatisf_Unadj* | General Life Satisfaction Survey: Unadjusted | Emotion (NIH Toolbox)
ListSort_Unadj* | List Sorting Working Memory Test: Unadjusted | Cognition (NIH Toolbox)
Loneliness_Unadj* | Loneliness Survey: Unadjusted | Emotion (NIH Toolbox)
Mars_Final* | Mars Final Contrast Sensitivity Score | Sensory
MeanPurp_Unadj* | Meaning and Purpose Survey Unadjusted | Emotion (NIH Toolbox)
MMSE_Score* | Mini Mental Status Exam Total Score | Alertness
NEOFAC_A* | Agreeableness | Personality (NEO-FFI)
NEOFAC_C* | Conscientiousness | Personality (NEO-FFI)
NEOFAC_E* | Extraversion | Personality (NEO-FFI)
NEOFAC_N* | Neuroticism | Personality (NEO-FFI)
NEOFAC_O* | Openness to Experience | Personality (NEO-FFI)
Odor_Unadj* | Odor Identification Age 3+ Unadjusted | Sensory (NIH Toolbox)
PainInterf_Tscore* | Pain Interference Survey Age 18+: T-score | Sensory (NIH Toolbox)
PercHostil_Unadj* | Perceived Hostility Survey: Unadjusted | Emotion (NIH Toolbox)
PercReject_Unadj* | Perceived Rejection Survey: Unadjusted | Emotion (NIH Toolbox)
PercStress_Unadj* | Perceived Stress Survey: Unadjusted | Emotion (NIH Toolbox)
PicSeq_AgeAdj | Picture Sequence Memory Test: Age-Adjusted | Cognition (NIH Toolbox)
PicVocab_AgeAdj | Picture Vocabulary Test: Age-Adjusted | Cognition (NIH Toolbox)
PicVocab_Unadj* | Picture Vocabulary Test: Unadjusted | Cognition (NIH Toolbox)
PMAT24_A_CR* | Progressive Matrices Number of Correct Responses | Cognition (Penn Tests)
PosAffect_Unadj* | Positive Affect Survey: Unadjusted | Emotion (NIH Toolbox)
ProcSpeed_AgeAdj | Pattern Comparison Processing Speed: Age-Adjusted | Cognition (NIH Toolbox)
PSQI_Score* | Pittsburgh Sleep Questionnaire Total Score | Alertness
ReadEng_AgeAdj | Oral Reading Recognition Test: Age Adjusted | Cognition (NIH Toolbox)
ReadEng_Unadj* | Oral Reading Recognition Test: Unadjusted | Cognition (NIH Toolbox)
Relational_Task_Acc* | Average accuracy % during RELATIONAL task | In-Scanner Task
Sadness_Unadj* | Sadness Survey: Unadjusted | Emotion (NIH Toolbox)
SCPT_SEN* | Short Continuous Performance Test: Sensitivity | Cognition (Penn Tests)
SCPT_SPEC* | Short Continuous Performance Test: Specificity | Cognition (Penn Tests)
SelfEff_Unadj* | Self-Efficacy Survey: Unadjusted | Emotion (NIH Toolbox)
Social_Task_Perc_Random* | Overall % of stimuli that the subject rated as random | In-Scanner Task
Social_Task_Perc_TOM* | Overall % of stimuli that received a social rating | In-Scanner Task
Strength_Unadj* | Grip Strength Test: Unadjusted | Motor (NIH Toolbox)
Taste_Unadj* | Regional Taste Intensity Age 12+ Unadjusted | Sensory (NIH Toolbox)
VSPLOT_TC* | Variable Short Penn Line Orientation Test: Total Correct | Cognition (Penn Tests)
WM_Task_Acc* | Accuracy across all conditions in working memory task | In-Scanner Task

##### Used phenotypes from the PNC Dataset

Variable | Description | Category
---|---|---
LNB_MCR | Number of Correct Responses to for 1-Back and | Letter N-Back (LNB)
| 2-Back Trials |
LNB_MRTC | Mean of the Median Response Time for Correct | Letter N-Back (LNB)
| Responses for 1-Back (TP) and for 2-Back (TP) Trials |
MP_MP2RTCR | Median Response Time for Correct Mouse Click Responses | Motor Praxis (MP)
PADT_A | Total Correct Responses for All Test Trials by genus | Penn Age Differentiation (PADT)
PADT_PC | Percent Correct Responses for All Test Trials by genus | Penn Age Differentiation (PADT)
PADT_SAME_PC | Percent Correct Responses to Test Trials with | Penn Age Differentiation (PADT)
| No Age Difference, by genus |
PADT_T | Median Response Time for All Test Trials, by genus | Penn Age Differentiation (PADT)
PCET_ACC2 | Calculated Accuracy Measure | Penn Conditional Exclusion (PCET)
PCET_CAT | Number of Categories Achieved | Penn Conditional Exclusion (PCET)
PCPT_T_FP | Total of Incorrect Responses to Number Trials (TP) | Penn Continuous Performance (PCPT)
| and Letter Trials (FP) |
PCPT_T_FPRT | Median Response Time for Incorrect Responses to | Penn Continuous Performance (PCPT)
| Number Trials (TP) and Letter Trials (TP) |
PCPT_T_TP | Total of Correct Responses to Number Trials (TP) | Penn Continuous Performance (PCPT)
| and Letter Trials (TP) |
PCPT_T_TPRT | Median Response Time for Correct Responses to Number | Penn Continuous Performance (PCPT)
| Trials (TP) and Letter Trials (TP) |
PEDT_A | Total Correct Responses for All Test Trials, by genus | Penn Emotion Differentiation (PEDT)
PEDT_ANG_PC | Percent of Correct Responses for Anger Trials, by genus | Penn Emotion Differentiation (PEDT)
PEDT_FEAR_PC | Percent of Correct Responses for Fear Trials, by genus | Penn Emotion Differentiation (PEDT)
PEDT_HAP_PC | Percent of Correct Responses for Happy Trials, by genus | Penn Emotion Differentiation (PEDT)
PEDT_PC | Percent of Correct Responses for All Test Trials, by genus | Penn Emotion Differentiation (PEDT)
PEDT_SAD_PC | Percent of Correct Responses for Sad Trials, by genus | Penn Emotion Differentiation (PEDT)
PEDT_SAME_PC | Percent Correct Responses to Test Trials with Neutral | Penn Emotion Differentiation (PEDT)
| Difference, by genus |
PEDT_T | Median Response Time for All Test Trials, by genus | Penn Emotion Differentiation (PEDT)
PEIT_CR | Total Correct Responses for All Test Trials, by genus | Penn Emotion Identification (PEIT)
PEIT_CRT | Median Response Time for Total Correct Test Trial | Penn Emotion Identification (PEIT)
| Responses, by genus |
PFMT_IFAC_RTC | Median Response Time for Total Correct Test | Penn Face Memory (PFMT)
| Trial Responses |
PFMT_IFAC_TOT | Total Correct Responses for All Test Trials | Penn Face Memory (PFMT)
PLOT_PC | Percent Correct Responses for All Test Trials, by genus | Penn Line Orientation (PLOT)
PLOT_TC | Total Correct Responses for All Test Trials, by genus | Penn Line Orientation (PLOT)
PLOT_TCRT | Median Response Time for Correct Trials, by genus | Penn Line Orientation (PLOT)
PMAT_CR | Total Correct Responses for All Test Trials, by genus | Primary Mental Abilities (PMAT)
PVRT_CR | Total Correct Responses for All Test Trials, by genus | Penn Verbal Reasoning (PVRT)
PVRT_RTCR | Median Response Time for Correct Verbal Reasoning | Penn Verbal Reasoning (PVRT)
| Responses, by genus |
PVRT_RTER | Median Response Time for Incorrect Verbal Reasoning | Penn Verbal Reasoning (PVRT)
| Responses, by genus |
PWMT_KIWRD_RTC | Median Response Time for Total Correct Test Trial | Penn Word Memory (PWMT)
| Responses |
PWMT_KIWRD_TOT | Total Correct Responses for All Test Trials | Penn Word Memory (PWMT)
VOLT_SVT | Total Correct Responses for All Test Trial | Visual Object Learning (VOLT)
VOLT_SVTCRT | Median Response Time for Total Correct Responses | Visual Object Learning (VOLT)
VOLT_SVTIRT | Median Response Time for Incorrect Responses | Visual Object Learning (VOLT)
WRAT_CR_RAW | Total Raw Score | Wide Range Achievement (WRAT)
WRAT_CR_STD | Total Standard Score | Wide Range Achievement (WRAT)

###  5.4 $r^{2}$ Results

Figure 8: Predictability of phenotype variables in the validation set,
quantified by the coefficient of determination ($r^{2}$, rather than by
correlation between predicted and true values as in Figure 2) between
predictions and true values, for HCP before regressing the covariates (a),
after regression (b), reconstruction (c), and of the latent phenotypes (d).
The shaded regions represent the standard deviation across resamplings. Figure
9: Predictability of phenotype variables in the validation set, quantified by
the coefficient of determination ($r^{2}$ by correlation between predicted and
true values as in Figure 3) between predictions and true values, for PNC
before regressing the covariates (a), after regression (b), reconstruction
(c), and of the latent phenotypes (d). The shaded regions represent the
standard deviation across resamplings.

###  5.5 Beta-coefficients for the linear regression

Figure 10: Beta-weights of the linear regression used to regress out age, sex,
and their quadratic interactions from HCP (a) PNC (b). (a) Dotted lines
include pairs of phenotypes that were both Unadjusted and Age-adjusted
(_AgeAdj_). These results are the same as those reported in Figure 4 but
values, where the mean is above $\pm$ 0.1, are annotated in the plot. Each
annotated entry corresponds to the mean (std) over the 100 repetitions.

###  5.6 How many latent phenotypes do we need to have an accurate prediction?

Figure 11: Marriage results using the inter-split correlation. In contrast to
the plot in the main text, where we report the $r^{2}$, here we report the
inter-split correlation. Figure 12: Critical difference diagrams for HCP
(left) and PNC (right). For the HCP data, the post-hoc Tukey HSD showed that
using just one latent phenotype had a significantly worse performance than all
other groups. There is, however, no significant difference in using all or a
subset of latent phenotypes for the PNC dataset

##  6 Pre-registration of replication on HBN dataset

###  6.1 Reliability and predictability of phenotype information from
functional connectivity on the Healthy Brain Network (HBN) dataset

This section follows the guidelines from OSF for a secondary data analysis
template, which was inspired by Van den Akker et al. (2021). Where necessary
to improve comprehension, the question from the template form is copied into
this for clarity in _italic_.

###  6.2 Description

Neuroimaging research aims to unravel the intricate relationship between
patterns of functional connectivity across the entire brain and various
behavioral phenotypes. The pre-registration described in this appendix seeks
to build, extend, and evaluate the generalizability of our analyses described
in the associated main document (see sections 2 through 4), extend our
analysis, and evaluate the generalizability of our results to a new dataset,
specifically the Healthy Brain Network (HBN). In other words, the work
described in the main part of this document was exploratory, and with the
analyses that we will carry out with the HBN dataset, we aim to confirm our
findings on an independent dataset. Similar to the exploratory work described
above, our main objective is to train predictive models for phenotypes and
compare them to the prediction of latent phenotypes (obtained from a singular
value decomposition (SVD) of the phenotypes). In the associated preprint we
evaluated: (i) the predictability of phenotypes and latent phenotypes, (ii)
the reliability of the latent phenotypes, and (iii) the impact of controlling
for age/sex confounds in the prediction. With this new study, we want to
evaluate if the previous findings generalize to an entirely new dataset
acquired from different scanners and with different phenotypes.

###  6.3 Contributors

Jessica Dafflon, Dustin Moraczewski, Eric Earl, Dylan M. Nielson, Gabriel
Loewinger, Patrick McClure, Adam G. Thomas, and Francisco Pereira (See
Reliability and predictability of phenotype information from functional
connectivity in large imaging datasets for details)

###  6.4 License

CC0 1.0 Universal

###  6.5 Subjects

Medicine and Health Sciences, Psychiatry and Psychology Computer Science, Life
Sciences, Machine Learning

###  6.6 Study Information

####  6.6.1 Research Questions

_List each research question included in this study. When specifying your
research questions, it is good practice to use only two new concepts per
research question. For example, split up your questions into a simple format:
‘Does X lead to Y?’ and ‘Is the relationship between X and Y moderated by Z?’.
By splitting up the research questions here, you can more easily describe the
statistical test for each research question later._

  1. 1.

How does controlling for age/sex confounds impact the prediction of phenotypes
from functional connectivity in HBN?

  2. 2.

How reliable are the latent phenotypes estimated with SVD across different
sub-samples of the HBN participants?

  3. 3.

Are latent phenotypes obtained by applying linear transformation (SVD) to
phenotypes more predictable from functional connectivity than the original
phenotypes on the HBN dataset?

  4. 4.

Do models predicting phenotype variables reconstructed from the latent SVD
variables with functional connectivity perform as well as models predicting
the original phenotype variables with functional connectivity in the HBN data?

####  6.6.2 Hypotheses

_For each of the research questions listed in the previous section, provide
one or more specific and testable hypothesis. Please make clear whether the
hypotheses are directional (e.g., A $>$ B) or non-directional (e.g., A $\neq$
B). If directional, state the direction. You may also provide a rationale for
each hypothesis._

  1. 1.

How does controlling for age/sex confounds impact the phenotypes prediction?
\- Hypothesis: In accordance with the literature (Chyzhyk et al., 2022) and
the results we obtained on our exploratory analysis (Figure 4), failure to
remove age and sex confounds will lead to a higher correlation coefficient for
variables that are strongly correlated with age/sex. Accordingly, we expect
that predictive performance will be lower after regressing age/sex out of the
phenotypes. Though our expectation is directional, we will test this with a
two-sided paired t-test on the predictive performance with and without
regressing out age and sex.
\- Rationale: Information from sex and age can drive the predictability of
specific phenotypes. If not regressed out, those confounds can inflate the
predictability of some phenotypes. This is particularly visible in the
variable ‘Strength Unadjusted’ from our exploratory analysis of the HCP
dataset. If sex is not regressed out, it is the most predictable variable
(Figure 2a), but most of its predictability is given by the sex information.

  2. 2.

How reliable are the latent phenotypes estimated with SVD across different
samples?
\- Hypothesis: Only the first five latent phenotypes (in order of explained
variance) will be reliable across different samples, with subsequent latent
phenotypes showing decreased reliability. For a complete description of how
our reliability metric was computed, see section 2.7. We specifically define a
reliable component as having an empirical lower 95% confidence interval on the
$r^{2}$ greater than 0.2. The empirical confidence interval will be calculated
from 1,000 random splits of the data.
\- Rationale: Our previous analyses of the HCP and PNC datasets have
demonstrated that while the first few latent phenotypes are reliably
identified across different samples, the reliability diminishes for subsequent
latent phenotypes. We anticipate a similar pattern in the HBN dataset, where
the initial latent phenotypes should be reliably identified across different
dataset splits.

  3. 3.

Can we obtain more predictable phenotypes if we use a linear transformation to
latent phenotypes (SVD) on HBN?
\- Hypothesis: Applying a linear dimensionality reduction (SVD) to obtain
latent phenotypes on the HBN dataset will not decrease the predictability
compared to directly using phenotypes. We will evaluate their predictability
by comparing the correlation of the prediction for the untransformed
phenotypes and those transformed using SVD with a two-sided paired t-test.
Again our expectation is directional but we are using a two-sided case because
it will still be of interest if either the latent or the original phenotypes
have significantly greater performance.
\- Rationale: Given our exploratory analysis with HCP and PNC where we did not
observe an increase in predictability by using latent phenotypes, in
comparison with the original phenotype variables, we do not expect to see that
the former will be more predictable.

  4. 4.

Are phenotype variables reconstructed from the SVD latent variables as
informative for training a model as the original phenotype variables?
\- Hypothesis: The reconstruction of phenotypes using a subset of latent
phenotypes, particularly the first few components, will exhibit similar or
enhanced predictability compared to using all latent phenotypes. To evaluate
if there is any statistical difference in the performance of the prediction
algorithms on using all of the components or a subset thereof, we will use the
autorank library (Herbold, 2020; Demšar, 2006). See sections 2.5 and 3.5 for a
more detailed description.
\- Rationale: Linear transformations such as SVD aim to capture the underlying
structure within complex datasets by creating uncorrelated variables (latent
phenotypes) that explain as much variance as possible. Conversely, the
original data can be reconstructed from the most informative latent
phenotypes, retaining essential information while discarding redundant or
noisy signals. If used to denoise the phenotype variables in the training set
of prediction models, the denoising can potentially lead to improved
predictive performance. Therefore, we expect that prediction performance of
models trained on reconstructed phenotypes will not decrease and may even
surpass that of models trained using the original phenotype variables.

###  6.7 Data Description

####  6.7.1 Datasets used

_Name and briefly describe the dataset(s), and if applicable, the subsets of
the data you plan to use. Useful information to include here is the type of
data (e.g., cross-sectional or longitudinal), the general content of the
questions, and some details about the respondents. In the case of longitudinal
data, information about the survey’s waves is useful as well. Mention the most
relevant information so that readers do not have to search for the information
themselves._
In the first part of this manuscript, we conducted an exploratory analysis
using the HCP (Van Essen et al., 2012) and PNC dataset (Satterthwaite et al.,
2016), for our confirmatory analysis, described in this pre-registration, we
will use the Healthy Brain Network (HBN) dataset (Alexander et al., 2017).
While the HCP dataset includes young adults aged between 22 and 35, the PNC
dataset includes children aged 8 to 21. Both PNC and HBN are
neurodevelopmental cohorts acquired in the Philadelphia and New York
metropolitan areas, respectively. The HBN dataset contains neuroimaging and
phenotypic data gathered across a diverse population of children and
adolescents (ages 5-21). The dataset includes, among other modalities,
functional MRI, diffusion MRI and electroencephalography collected
longitudinally. It also includes extensive behavioral assessments, cognitive
tests and psychiatric diagnostics. For this analysis, we will use the resting