# Improving machine learning-derived photometric redshifts and physical
property estimates using unlabelled observations

Humphrey, A.,1,2 Cunha, P. A. C.1,3 Paulino-Afonso, A.,1 Amarantidis, S.,4
Carvajal, R.,5,6 Gomes, J. M.,1 Matute, I.,5,6 Papaderos, P.5,6
1 Instituto de Astrofísica e Ciências do Espaço, Universidade do Porto, CAUP,
Rua das Estrelas, Porto, 4150-762, Portugal
2 DTx – Digital Transformation CoLAB, Building 1, Azurém Campus, University of
Minho, 4800-058 Guimarães, Portugal
3 Faculdade de Ciências da Universidade do Porto, Rua do Campo de Alegre,
4150-007 Porto, Portugal
4 Institut de Radioastronomie Millimétrique (IRAM), Avenida Divina Pastora 7,
Local 20, E-18012, Granada, Spain
5 Departamento de Física, Faculdade de Ciências, Universidade de Lisboa,
Edifício C8, Campo Grande, PT1749-016 Lisboa, Portugal
6 Instituto de Astrofísica e Ciências do Espaço, Faculdade de Ciências,
Universidade de Lisboa, Tapada da Ajuda, PT-1349-018 Lisboa,
Portugal
E-mail<EMAIL_ADDRESS>(AH)

(Accepted 5 December 2022. Received 15 November 2022; in original form 9 June
2022)

###### Abstract

In the era of huge astronomical surveys, machine learning offers promising
solutions for the efficient estimation of galaxy properties. The traditional,
‘supervised’ paradigm for the application of machine learning involves
training a model on labelled data, and using this model to predict the labels
of previously unlabelled data. The semi-supervised ‘pseudo-labelling’
technique offers an alternative paradigm, allowing the model training
algorithm to learn from both labelled data and as-yet unlabelled data. We test
the pseudo-labelling method on the problems of estimating redshift, stellar
mass, and star formation rate, using COSMOS2015 broad band photometry and one
of several publicly available machine learning algorithms, and we obtain
significant improvements compared to purely supervised learning. We find that
the gradient-boosting tree methods CatBoost, XGBoost, and LightGBM benefit the
most, with reductions of up to $\sim$15 % in metrics of absolute error. We
also find similar improvements in the photometric redshift catastrophic
outlier fraction. We argue that the pseudo-labellng technique will be useful
for the estimation of redshift and physical properties of galaxies in upcoming
large imaging surveys such as Euclid and LSST, which will provide photometric
data for billions of sources.

###### keywords:

methods: statistical – galaxies: photometry – galaxies: fundamental parameters
– galaxies: distances and redshifts

††pubyear: 2022††pagerange: Improving machine learning-derived photometric
redshifts and physical property estimates using unlabelled
observations–Improving machine learning-derived photometric redshifts and
physical property estimates using unlabelled observations

##  1 Introduction

The Universe and its evolution cannot be understood without first
understanding galaxies, and their evolution. A key endeavour in extragalactic
astrophysics is the estimation of the redshift and physical properties of
galaxies (e.g., stellar mass, star-formation rate, etc.), thereby allowing
galaxies to be placed in evolutionary sequences and frameworks, and
facilitating the testing of galaxy evolution models (e.g., Förster Schreiber &
Wuyts, 2020).

Galaxy property estimation methods have traditionally used software that fits
spectral templates to observed spectral energy distributions or spectra (e.g.,
Arnouts et al., 1999; Bolzonella, Miralles, & Pelló, 2000; Cid Fernandes et
al., 2005; Ilbert et al., 2006; da Cunha et al., 2008; Noll et al., 2009;
Laigle et al., 2016; Gomes & Papaderos, 2017; Carnall et al., 2018; Johnson et
al., 2021); while generally effective and physically motivated, template
fitting tends to be expensive and often does not scale well to very large
datasets (i.e., $\gg 1{\rm M}$ objects), since the computation time typically
scales linearly with the number of sources to be fitted.

Supervised machine learning methods represent an alternative that is
increasingly applied to the problems of galaxy classification and property
estimation. In this paradigm, a learning algorithm infers a function to map
observations to labels, using examples of observation and label pairs. The
function is subsequently used to predict the labels of new (unlabelled)
examples (see, e.g., Baron, 2019). A major benefit of such methods is their
usually greater scalablity compared to template fitting methods: Once trained,
machine learning models can usually be applied to large volumes of unlabelled
data at a negligible computational cost (e.g., Hemmati et al., 2019).

There are now numerous studies using supervised machine learning in the field
of extragalactic astrophysics. Among its most frequent uses is in the
separation of sources into different classes, for which there is now a rich
body of literature. For instance, supervised learning has been successfully
used by various authors to perform binary classification with the goal of
selecting sources with a specific set of characteristics (e.g., Cavuoti et
al., 2014), or for multi-class problems such as star/galaxy/quasar
classification (e.g., Bai et al., 2019; Clarke et al., 2020; Cunha & Humphrey,
2022)

Supervised machine learning has also been extensively used to estimate galaxy
redshifts from imaging or photometric data, via regression (e.g., Collister &
Lahav, 2004; Brescia et al., 2013; Cavuoti et al., 2017; Pasquet et al., 2019;
Razim et al., 2021; Guarneri et al., 2021; Carvajal et al., 2021; Cunha &
Humphrey, 2022). While reducing the required computational time is often among
the main motivations for using machine learning to estimate redshift, there
are now indications that these methods can significantly outperform
traditional template-fitting methods, in some regions of colour-redshift
space, in terms of the accuracy estimated redshifts (Euclid Collaboration:
Desprez et al., 2020).

Other key uses of supervised learning in extragalactic astronomy are the
estimation galaxy physical properties (e.g., Bonjean et al., 2019; Delli
Veneri et al., 2019; Mucesh et al., 2021; Simet et al., 2021), and
morphological classification of galaxies from images (e.g., Dieleman, Willett,
& Dambre, 2015; Huertas-Company et al., 2015; Domínguez Sánchez et al., 2018;
Tuccillo et al., 2018; Nolte et al., 2019; Bowles et al., 2021; Bretonnière et
al., 2021). A few studies have also explored schemes for combining results
from traditional methods with machine learning methods, resulting in improved
predictions (e.g., Cavuoti et al., 2017; Fotopoulou & Paltani, 2018).

One weakness of the standard supervised learning paradigm is that only
labelled data can be used for model training, and potentially useful
information that may exist within the unlabelled data remains invisible to,
and thus unused by, the training algorithm. As such, it is interesting to
consider methodologies whereby models can be trained with an awareness of the
content of unlabelled test data.

The ‘pseudo-labelling’ technique is a semi-supervised machine learning method
that makes use of unlabelled data to refine supervised machine learning models
(Lee, 2013; Slijepcevic & Scaife, 2021). By allowing model training to make
use of information about the structure and compostion of the unlabelled data,
the resulting model can more accurately predict labels for this data. In this
paper, we demonstrate the application of pseudo-labelling to the estimation of
galaxy properties using supervised machine learning and broad-band photometry.

We begin by giving a brief overview of the pseudo-labelling methodology in
$\S$2. The data and its preprocessing are described in $\S$3. The application
of pseudo-labelling to machine-learning-based estimation of photometric
redshifts is presented in $\S$5.1. Examples of its application to the
estimation of stellar mass and star formation rate are presented in $\S$5.2.
Our conclusions and final remarks are given in $\S$6.

##  2 The pseudo-labelling methodology

Pseudo-labelling is a semi-supervised technique which may be incorporated into
supervised machine-learning pipelines, leveraging information contained within
unlabelled data such that stronger models may be produced (Lee, 2013). The
main improvements from this technique arise by allowing supervised learning
algorithms to obtain a clearer picture of the structure of the overall dataset
from which the training and hold-out data are drawn. This includes allowing
the learning algorithm to recognise when ‘outlier’ examples in the training
set are in fact members of a significant population that is worthwhile to
model rather than to ignore.

In its simplest form, the pseudo-labelling workflow is as follows. First, a
supervised machine learning model is trained on a set of labelled training
data. The resulting model is then used to predict the labels for the
unlabelled (test) data. Subsequently, a new, larger training set is created by
appending the test data to the original training data. When creating this new
training set, the examples that come from the test set are labelled with their
predicted labels (the ‘pseudo-labels’). The labels associated with the
original training data remain unaltered, however. This yields a significantly
larger training set, where the labels used for training are now a mixture of
ground-truth values, and values predicted by the aforementioned model.

A new model is trained on this new training set, and is used to predict new
labels for the test set examples. This process may be repeated for an
arbitrary number of iterations, each time updating the pseudo-labels to their
most recent predicted values, prior to training the next model. After the
desired number of iterations has been completed, the resulting model is used
to make the final label predictions for the test set (and other hold-out sets,
if applicable). Our pseudo-labelling workflow is illustrated in Fig. 1.

A real world application of this workflow might be the prediction of
photometric redshifts using broad-band photometry from the Euclid Wide Survey
(Euclid Collaboration: Scaramella et al., 2021).

In this study, we perform 50 iterations of pseudo-labelling before stopping.
This was decided in order to have a reasonable balance between computation
time and number of iterations. The results presented in this work required
$\sim 20{\rm h}$ of computation time on a workstation with an Intel Core i7
11700K 8-core CPU (3.6-5.0 GHz), 32 GB of RAM, and an NVIDIA RTX A6000 48GB
GPU. However, the code can also be executed without a GPU, or with less
powerful hardware, with some minor modifications.

We test the pseudo-labelling methodology with five open-source learning
algorithms. Three of these are gradient-boosting tree (GBT) algorithms:
CatBoostRegressor111https://catboost.ai (Prokhorenkova et al., 2018),
LGBMRegressor222https://lightgbm.readthedocs.io (Ke et al., 2017), and
XGBoost333https://xgboost.readthedocs.io (Chen & Guestrin, 2016). While there
are subtle differences between them, the GBT methods all have a fundamentally
similar approach to model training: they combine multiple regression trees in
series, with each new tree fitting the residuals from the previous trees
(‘boosting’; see Friedman, 2001).

From the Python Scikit-Learn444https://scikit-learn.org package (Pedregosa et
al., 2011) we also use the popular tree-ensemble method RandomForestRegressor
(Breiman, 2001). This learning algorithm combines multiple weak regression
trees into a single, stronger regression model by averaging the predictions
from all the individual trees. To reduce the similarity between the
constituent trees, each tree is built using a subset of the features and
training examples that is selected at random (with replacement).

Also from Scikit-Learn, we test the distance-based method KNeighborsRegressor.
Using the $k$-nearest neighbours (KNN) algorithm, this is a non-parametric
supervised method, which uses the similarity (typically defined by the
Euclidean distance) to training examples to predict the labels for test
examples. For regression problems, label predictions are usually made by
averaging the $k$-nearest labels from the training set, where $k$ is an
integer specified by the user.

The learning algortithms were optimized to give competitive results for the
prediction problems posed herein, but were not optimized exhaustively, since a
test of which algorithm performs best for galaxy physical property estimation
is not the goal of this study. The Python scripts used in this work are
publicly available at Github555https://github.com/humphrey-and-the-
machine/pseudo-labelling. The readme file for this repository gives
instructions for how to modify the code for use without a GPU, and/or with
less powerful computing hardware..

We emphasize that this methodology differs significantly from model stacking
workflows (e.g., Wolpert, 1992; Zitlau et al., 2016; Euclid Collaboration:
Humphrey et al., 2022; Cunha & Humphrey, 2022); whereas pseudo-labelling
involves appending unlabelled data (with model-predicted labels) to the
training data, stacking usually only makes use of labelled data for model
training.

Figure 1: Flow diagram illustrating the pseudo-labelling proceedure that we
have used. Labelled data is used to train an initial model; the model predicts
(pseudo) labels for unlabelled data; this pseudo-labelled data is appended to
the training data; the resulting dataset used to train a new model, predict
new labels, update the pseudo-labels, and so on, until stopping criteria are
reached (50 iterations in our case).

Figure 2: Histograms for photometric redshift, stellar mass, and star
formation rate for the selected sample. Figure 3: Estimating photometric
redshifts using pseudo labelling with CatBoost, LightGBM, XGBoost,
KNeighborsRegressor or RandomForestRegressor. The zeroth iteration corresponds
to the initial model trained only on the training set; the subsequent 50
iterations correspond to the pseudo-labelling process. The solid blue line
shows the evolution of the photometric redshift metrics as a function of the
pseudo labelling iteration, for the test set used for pseudo-labelling. The
dashed orange shows the evolution of the metrics for the hold-out set, which
was not used for model training or pseudo labelling. The results shown in this
figure correspond to the averaged metrics from 10 runs with different random
seeds for the random train / test / hold-out split, and where applicable, for
the learning algorithms also. Left column: Normalised median absolute
deviation. Middle column: Catastrophic outlier fraction. Right column: The
overall bias of the prediction errors. Figure 4: Similar to Fig. 3, but for
the estimation of $Log_{10}M_{\star}$. Left column: Mean absolute error.
Middle column: Median absolute error. Right column: $R^{2}$ score.  Figure 5:
Similar to Figs. 3 and 4, but for the estimation of $Log_{10}{\rm SFR}$.
Figure 6: Quantifying the improvement, if any, obtained by applying pseudo-
labelling. Each panel shows the maximum improvement obtained, relative to the
zeroth iteration, for each learning algorithm, during the 50 iterations of
pseudo-labelling. Negative values (i.e., reduced errors) indicate an
improvement, and positive values indicate that model quality worsened.

##  3 Data and preprocessing

The dataset used in this work is derived from the COSMOS2015 multiwavelength
catalogue (Laigle et al., 2016). We selected a subset of $135,374$ galaxies
from the catalogue using the following criteria:

  * •

(i) Galaxies not detected in X-rays (TYPE=0);

  * •

(ii) photometric redshifts in the range $0<z<9.8$;

  * •

(iii) $Y$, $J$, and $H$ $\leq 24$ AB mag.

These criteria were designed to (i) remove active galaxies where possible,
(ii) remove sources with a problematic photometric redshift, and (iii) remove
faint sources with low signal to noise photometry.

To create the features for model training, we extracted the $u$, $B$, $V$,
$r$, $i_{+}$, $z_{+}$, $Y$, $J$, $H$ and $K_{s}$ broad band magnitudes, as
measured in the 3″aperture. In the interest of simplicity and repeatability,
we have used the COSMOS2015 MAG_APER3 measurements without applying any
transformations or corrections. All unique colour combinations were then
computed, resulting in 55 features (10 magnitudes and 55 colours).

Missing values represent 0.03% of these data. At the time of writing, there is
no established best-practise for imputing missing photometry for use with
machine learning; many of the traditional imputation methods used by machine
learning practioners (e.g., adopting the mean or median value) have the
potential to result in unphysical spectral energy distributions, particularly
when a photometry value is missing due to the presence of a spectral break in
the spectrum of the source. We opted to impute missing photometry values with
the maximum measured magnitude of the other galaxies in the same band (e.g.,
Carvajal et al., 2021; Curran, 2022); while this does not inform the learning
algorithms that the magnitude value is a special case, it does at least place
sources that have missing values at approximately correct locations within the
colour-magnitude hypercube (cf. Euclid Collaboration: Humphrey et al., 2022,
who imputed ‘magic values’ to allow learning algorithms to treat missing
values differently when beneficial).

We have adopted the COSMOS2015 values of PHOTOZ, MASS_BEST, and SFR_BEST as
the ground truth values of the redshift, $log_{10}(M_{*}/M_{\odot})$, and
$log_{10}{\rm(SFR)}$, respectively666In the interest of brevity, we
hereinafter use $M_{*}$ and SFR to mean the log of the quantities.. These
quantities are not ‘ground-truth’ in the conventional sense, in that they were
originally produced by applying the Le Phare template-fitting code to 30
COSMOS2015 photometry bands (Laigle et al., 2016). As such, these quantities
are subject to both random and systematic errors with respect to their ‘true’
values. Figure 2 shows histograms for photometric redshift, stellar mass, and
star formation rate for the selected sample.

The dataset was then randomly split into ‘train’, ‘test’, and ‘hold-out’ sets
with a 12:28:10 ratio (24%, 56%, and 20%), using the Scikit-Learn function
train_test_split. The train set (or training set) is used for model training,
with the ground truth labels visible to the learning algorithm. Finally, all
55 features in the three data subsets (train, test, hold-out) were
standardized by removing their mean and scaling to unit variance, using the
mean and standard deviation calculated from the train set.

The test set simulates unlabelled data that is to be used for pseudo-
labelling; our hold-out set simulated unlabelled data that is not used for
pseudo-labelling, but which we use to test how well models generalise to data
that was not used for training or pseudo-labelling (note that our definition
of the test and hold-out sets differs slightly from some of the literature).
For the test and hold-out sets, the ground truth labels are not visible to the
learning algorithms, and are used only to evaluate metrics of model
performance.

##  4 Metrics of model quality

To evaluate our regression models, we use several different metrics. In the
case of photometric redshift estimation, we use the normalized median absolute
deviation (NMAD) as measure of accuracy, which we calculate as

$\mbox{NMAD}=1.48\,\mbox{median}\,\left(\frac{\,|z_{\rm phot}-z_{\rm
ref}|\,}{1+z_{\rm ref}}\right)\,,$ (1)

where $z_{\rm phot}$ is the redshift predicted by the model, and $z_{\rm ref}$
is the reference photometric redshift used as the ‘ground truth’. The NMAD is
roughly analogous to the standard deviation: $\sigma\approx\kappa\,{\rm MAD}$,
where MAD is the median absolute deviation, and $\kappa$ is a constant scale
factor; $\kappa\approx 1.48$ for normally distributed data. An advantage of
the NMAD metric is that it is less affected by outliers than is the classical
standard deviation.

It is also of interest to use a metric that quantifies the number of outliers
among the redshift predictions. Thus, we adopt the following criterion to
classify redshift estimates as catastrophic outliers:

$\frac{|z_{\rm phot}-z_{\rm ref}|}{1+z_{\rm ref}}>0.15.$ (2)

In addition, we define the bias of the photometric redshift estimates as

$\mbox{bias}=\mbox{median}\left(\frac{z_{\rm phot}-z_{\rm ref}}{1+z_{\rm
ref}}\right).$ (3)

To quantify the errors in predicting $M_{*}$ and SFR, we use the mean absolute
error, the median absolute error, and the coefficient of determination
$R^{2}$. The mean absolute error is calculated as

$\mbox{mean absolute
error}=\frac{\sum\limits_{i=1}^{n}(\hat{y_{i}}-y_{i})}{n}$ (4)

where $y_{i}$ is the true value of the label of the $i$-th sample,
$\hat{y_{i}}$ is the predicted value, and $n$ is the total number of samples.
Similarly, the median absolute error is calculated as

$\mbox{median absolute error}=\mbox{median}\left(\hat{y_{i}}-y_{i}\right).$
(5)

The $R^{2}$ score is calculated as

$R^{2}=\frac{\sum\limits_{i=1}^{n}(\hat{y_{i}}-y_{i})}{\sum\limits_{i=1}^{n}(y_{i}-\bar{y})}$
(6)

where $\bar{y}$ is $y_{i}$ averaged over $n$ samples. Unlike the previous
metrics, higher values of $R^{2}$ indicate a better model, up to a maximum
value of 1, with no minimum value.

##  5 Results and discussion

In Figs. 3-5, we show the evolution of selected metrics of model quality
versus iteration number, when applying pseudo-labelling to the estimation of
photometric redshift (Fig. 3), stellar mass (Fig. 4), and SFR (Fig. 5). Each
quantity was predicted separately, using a dedicated single-output model.

The curves correspond to the metrics of model quality, averaged over 10 runs,
using different random seeds. The zeroth iteration corresponds to the initial
model trained using only the train set; subsequent iterations pertain to
models training using pseudo-labelling. In Fig. 6, we also show the maximum
improvement obtained during the 50 iterations of pseudo-labelling.

The behaviour of the different learning algorithms varies substantially when
applying pseudo-labelling. CatBoostRegressor, LightGBMRegressor, and
XGBoostRegressor all experienced significant improvements in model quality.
Conversely, pseudo-labelling only offered a marginal improvement to the
KNeighborsRegressor models, and appears to be significantly detrimental to the
RandomForestRegressor models.

The reasons why the different learning algorithms respond differently to
pseudo-labelling are complex, but can be thought of as to a trade-off between
the enlargement of the training sample, which usually improves model quality,
and the inclusion of some incorrect (pseudo) labels, which can degrade model
quality (e.g., Euclid Collaboration: Humphrey et al., 2022). A more detailed
description of the results is given in the following subsections.

###  5.1 Photometric redshift

####  5.1.1 Gradient boosted tree models

It is clear from Fig. 3 that pseudo-labelling can significantly improve the
NMAD and outlier fraction metrics for photometric redshift estimates, but this
depends on the learning algorithm used, and on the number of iterations of
pseudo-labelling that are performed.

The greatest improvements ($\ga 5\%$) are obtained when using a gradient
boosted tree method (GBT; CatBoost, LightGBM, or XGBoost; see the first 3 rows
of Figs. 3-5). In these cases, the NMAD decreases rapidly in the first $\sim
3$ iterations of pseudo-labelling, after which the NMAD curve either flattens-
off (CatBoost), experiences an up-turn (LightGBM), or continues in a gradual
decline (XGBoost). All three GBT algorithms show a broadly similar evolution
in the catastrophic outlier fraction, with an intially steep decline that
becomes shallower as the pseudo-labelling iterations proceed.

A noteworthy result is that the improvements in the NMAD and outlier fraction
of the test set predictions are accompanied by improvements in the hold-out
set predictions (see, e.g., Fig. 3 panels a,b), indicating that improved
generalisation was achieved not only for the test set (used for the pseudo-
labelling), but also to the hold-out set (not involved in model training).
However, the improvements in NMAD and outlier fraction are smaller for the
hold-out set than for the test set.

The initial models (zeroth iteration) result in predictions with a generally
low redshift bias, which the pseudo-labelling process further improves in the
cases of CatBoost and XGBoost.

####  5.1.2 Other models

The results from using pseudo-labelling with the other learning algorithms
(KNeighborsRegressor or RandomForestRegressor) are less clear-cut. In the case
of KNeighborsRegressor (Fig. 3 panels d,j,p), the first $\sim$2 iterations
provide only a marginal improvement in NMAD, while the outlier fraction shows
a sharp initial decrease followed by a reversal back up towards, but not
reaching, the starting value. A marginal improvement is also present in the
bias.

The RandomForestRegressor (Fig. 3 panels f,l,r) stands out as the learning
algorithm that did not receive any benefit from pseudo-labelling. Even from
the first iteration of pseudo-labelling, the values of the NMAD, outlier
fraction, and bias are always significantly higher than when pseudo-labelling
is not used (the zeroth iteration). In effect, we see a gradual accumulation
of errors due to the introduction of label noise.

###  5.2 Stellar mass and SFR

The results from applying pseudo-labelling to the estimation of $M_{*}$ (Fig.
4) and SFR (Fig. 5) are broadly analogous to those described in $\S$5.1. For
the GBT methods, there is a rapid decrease in mean absolute error and median
absolute error values during the initial $\sim 5$ iterations of pseudo-
labelling. The subsequent evolution in these two metrics differs between the
three GBT methods. In the case of CatBoost, a constant value is quickly
reached. For (LightGBM), there is a reversal and increase in the metric
values. In the case of XGBoost, the metric values continue their gradual
decrease, albeit with a shallower gradient. Likewise, the $R^{2}$ scores when
using the GBT methods show an initially significant increase in model quality
(higher values), followed by an evolution that depends on which learning
algorithm was used.

For KNeighborsRegressor the test set predictions improve marginally after a
single iteration of the pseudo-labelling procedure, as shown by the decrease
in the mean absolute error, median absolute error, and $R^{2}$ score. However,
from the second iteration onward model quality dimishes, such that by the
tenth iteration the aforementioned metrics have values that are significantly
worse than the initial (zeroth iteration) model. For the hold-out set, we find
little, if any, improvement in model quality with the application of pseudo-
labelling.

Similar to the results for photometric redshift estimation, in the case of
RandomForest the mean absolute error, median absolute error, and $R^{2}$ score
show a substantial reduction in model quality as the result of application of
pseudo-labelling.

##  6 Concluding remarks

We have explored the application of the semi-supervised pseudo-labelling
methodology to the problem of estimating galaxy redshift, stellar mass, and
star-formation rate from broad band photometry and colours. The dataset used
is derived from the COSMOS2015 multiwavelength catalogue of Laigle et al.
(2016). Our tests simulate the case of a photometric dataset produced by a
large-scale imaging survey, for which labels are available only for a small
subset of the sources; the objective is to examine whether pseudo-labelling
can be used to improve machine learning models for the prediction of the
missing labels, compared to the simple case where only the labelled data is
used for model training.

We have demonstrated that our methodology can result in regression models that
are able to predict redshift, stellar mass, and star-formation rate
significantly more accurately, with typical improvements in metrics of
absolute error of $\sim$5–15%, by making use of information present in
unlabelled data.

In cases where improved predictions are obtained for the test data used for
the pseudo-labelling, the predictions for hold-out data (not used for pseudo-
labelling) also show significant, but typically smaller, improvement. In other
words, pseudo-labelling also results in improved generalisation to previously
unseen hold-out data.

An important caveat that must be stated is that the performance of our
implementation of pseudo-labelling, in terms of how well it improves
supervised machine learning models for galaxy properties, depends on the
learning algorithm with which it is used. For instance, we find that models
produced with the gradient boosting tree methods CatBoostRegressor,
LightGBMRegressor, and XGBoostRegressor experience the greatest improvements
from the use of pseudo-labelling. Conversely, models trained with the popular
tree-ensemble method RandomForestRegressor were negatively affected by the
application pseudo-labelling.

This study is intended to be a proof-of-concept for the use of pseudo-
labelling for the estimation of the properties of astronomical sources. As
such, we have not exhaustively tested all available learning algorithms, nor
have we exhaustively optimised the hyperparameters of the learning algorithms
we use herein. In a future study, we will explore in greater detail the
application of pseudo-labelling (and other variants thereof) to the
classification and property estimation of astronomical sources.

Finally, we emphasize that the pseudo labelling technique is likely to be
useful for enhancing the quality of estimates of galaxy redshift and physical
properties in large upcoming (and ongoing) imaging surveys such as UNIONS
(Chambers et al., 2020), LSST (Ivezić et al., 2019) and Euclid (Laureijs et
al., 2011), which will produce rich, but challengingly vast datasets in need
of computationally efficient labelling.

## Acknowledgements

We thank the referee for their careful reviews and useful comments. This work
was supported by Fundação para a Ciência e a Tecnologia (FCT) through grants
UID/FIS/04434/2019, UIDB/04434/2020, UIDP/04434/2020 and PTDC/FIS-
AST/29245/2017, EXPL/FIS-AST/1085/2021, and an FCT-CAPES Transnational
Cooperation Project. R.C. acknowledges support from the FCT through the
Fellowship PD/BD/150455/2019 (PhD:SPACE Doctoral Network PD/00040/2012). We
also acknowledge support from NVIDIA in the form of a GPU under the NVIDIA
Academic Hardware Grant Program. In the development of this work, we have made
use of the Pandas (McKinney, 2010), Numpy (Harris et al., 2020), and Dask
(Rocklin, 2015) packages for Python.

## Data Availability

All data and methods used in this study are publicly available at
https://github.com/humphrey-and-the-machine.

## References

  * Arnouts et al. (1999) Arnouts, S., Cristiani, S., Moscardini, L., et al. 1999, MNRAS, 310, 540
  * Bai et al. (2019) Bai, Y., Liu, J., Wang, S., et al. 2019, AJ, 157, 9
  * Baron (2019) Baron, D. 2019, arXiv:1904.07248
  * Bolzonella, Miralles, & Pelló (2000) Bolzonella, M., Miralles, J.-M., & Pelló, R. 2000, A&A, 363, 476
  * Bonjean et al. (2019) Bonjean, V., Aghanim, N., Salomé, P., et al. 2019, A&A, 622, A137
  * Bowles et al. (2021) Bowles, M., Scaife, A. M. M., Porter, F., Tang, H., & Bastien, D. J. 2021, MNRAS, 501, 4579
  * Breiman (2001) Breiman, L. 2001, Mach. Learn., 45, 1
  * Brescia et al. (2013) Brescia, M., Cavuoti, S., D’Abrusco, R., Longo, G., & Mercurio, A. 2013, ApJ, 772, 140
  * Bretonnière et al. (2021) Bretonnière, H., Boucaud, A., & Huertas-Company, M. 2021, arXiv:2111.15455
  * Carnall et al. (2018) Carnall, A. C., McLure, R. J., Dunlop, J. S., et al. 2018, MNRAS, 480, 4379
  * Carvajal et al. (2021) Carvajal, R., Matute, I., Afonso, J., et al. 2021, Galaxies, 9, 86
  * Cavuoti et al. (2017) Cavuoti, S., Tortora, C., Brescia, M., et al. 2017, MNRAS, 466, 2039
  * Cavuoti et al. (2014) Cavuoti, S., Brescia, M., D’Abrusco, R., Longo, G., & Paolillo, M. 2014, MNRAS, 437, 968
  * Chambers et al. (2020) Chambers, K., Unions Team including Pan-Starrs Team, & CFIS Team 2020, American Astronomical Society meeting 235, id. 154.04. Bulletin of the American Astronomical Society, Vol. 52, No. 1
  * Chen & Guestrin (2016) Chen, T., Guestrin, C., 2016 arXiv:1603.02754v3
  * Cid Fernandes et al. (2005) Cid Fernandes, R., Mateus, A., Sodré, L., et al. 2005, MNRAS, 358, 363
  * Clarke et al. (2020) Clarke, A. O., Scaife, A. M. M., Greenhalgh, R., & Griguta, V. 2020, A&A, 639, A84
  * Collister & Lahav (2004) Collister, A. A., & Lahav, O. 2004, PASP, 116, 345
  * Cunha & Humphrey (2022) Cunha, P. A. C. & Humphrey, A. 2022,
  * da Cunha et al. (2008) da Cunha, E., Charlot, S., & Elbaz, D. 2008, MNRAS, 388, 1595
  * Curran (2022) Curran, S. J. 2022, MNRAS, 512, 2099
  * Delli Veneri et al. (2019) Delli Veneri, M., Cavuoti, S., Brescia, M., Longo, G., & Riccio, G. 2019, MNRAS, 486, 1377
  * Dieleman, Willett, & Dambre (2015) Dieleman, S., Willett, K. W., & Dambre, J. 2015, MNRAS, 450, 1441
  * Domínguez Sánchez et al. (2018) Domínguez Sánchez, H., Huertas-Company, M., Bernardi, M., et al. 2018, MNRAS, 476, 3661
  * Euclid Collaboration: Desprez et al. (2020) Euclid Collaboration: Desprez, G., Paltani, S., et al. 2020, A&A, 644, A31
  * Euclid Collaboration: Humphrey et al. (2022) Euclid Collaboration: Humphrey A., Bisigello L., Cunha P. A. C., Bolzonella M., Fotopoulou S., Caputi K., Tortora C., et al., 2022, A&A, accepted. arXiv:2209.13074
  * Euclid Collaboration: Scaramella et al. (2021) Euclid Collaboration, Scaramella, R., Amiaux, J., et al. 2022, A&A, 662, A112
  * Friedman (2001) Friedman J., 2001, The Annals of Statistics, Vol. 29, No. 5
  * Förster Schreiber & Wuyts (2020) Förster Schreiber, N. M. & Wuyts, S. 2020, ARA&A, 58, 661
  * Fotopoulou & Paltani (2018) Fotopoulou, S., & Paltani, S. 2018, A&A, 619, A14
  * Gomes & Papaderos (2017) Gomes, J. M., & Papaderos, P. 2017, A&A, 603, A63
  * Guarneri et al. (2021) Guarneri, F., Calderone, G., Cristiani, S., et al. 2021, MNRAS, 506, 2471
  * Harris et al. (2020) Harris, C. R., Millman, K. J., van der Walt, S. J., et al. 2020, Nature, 585, 357
  * Hemmati et al. (2019) Hemmati, S., Capak, P., Pourrahmani, M., et al. 2019, ApJ, 881, L14
  * Huertas-Company et al. (2015) Huertas-Company, M., Gravet, R., Cabrera-Vives, G., et al. 2015, ApJS, 221, 8
  * Ilbert et al. (2006) Ilbert, O., Arnouts, S., McCracken, H. J., et al. 2006, A&A, 457, 841
  * Ivezić et al. (2019) Ivezić, Ž., Kahn, S. M., Tyson, J. A., et al. 2019, ApJ, 873, 111
  * Johnson et al. (2021) Johnson, B. D., Leja, J., Conroy, C., et al. 2021, ApJS, 254, 22
  * Ke et al. (2017) Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.-Y., 2017, LightGBM: A Highly Efficient Gradient Boosting Decision Tree, in Advances in Neural Information Processing Systems, vol. 30, 3146
  * Laigle et al. (2016) Laigle, C., McCracken, H. J., Ilbert, O., et al. 2016, ApJS, 224, 24
  * Laureijs et al. (2011) Laureijs, R., Amiaux, J., Arduini, S., et al. 2011, [[arXiv]1110.3193]
  * Lee (2013) Lee, D., “Pseudo-Label: The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks.” ICML 2013 Workshop: Challenges in Representation Learning (WREPL), Atlanta, Georgia, USA, 2013 [eprint]
  * McKinney (2010) McKinney, w., 2010, Data Structures for Statistical Computing in Python, in Proceedings of the 9th Python in Science Conference, pp 51
  * Mucesh et al. (2021) Mucesh, S., Hartley, W. G., Palmese, A., et al. 2021, MNRAS, 502, 2770
  * Noll et al. (2009) Noll, S., Burgarella, D., Giovannoli, E., et al. 2009, A&A, 507, 1793
  * Nolte et al. (2019) Nolte, A., Wang, L., Bilicki, M., Holwerda, B., & Biehl, M. 2019, arXiv e-prints, arXiv:1903.07749
  * Pasquet et al. (2019) Pasquet, J., Bertin, E., Treyer, M., et al. 2019, A&A, 621, A26
  * Pedregosa et al. (2011) Pedregosa, F., et al., 2011, Journal of Machine Learning Research, 12, 2825
  * Prokhorenkova et al. (2018) Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A.V., Gulin, A., 2018, Advances in Neural Information Processing Systems, 31, 6638
  * Razim et al. (2021) Razim, O., Cavuoti, S., Brescia, M., Riccio, G., Salvato, M., Longo, G., 2021, MNRAS, 507, 5034
  * Rocklin (2015) Rocklin, M., 2015, Dask: Parallel Computation with Blocked Algorithms and Task Scheduling, in Proceedings of the 14th Python in Science Conference, pp 130
  * Simet et al. (2021) Simet, M., Chartab, N., Lu, Y., & Mobasher, B. 2021, ApJ, 908, 47
  * Slijepcevic & Scaife (2021) Slijepcevic, I. V. & Scaife, A. M. M. 2021, arXiv:2111.04357
  * Tuccillo et al. (2018) Tuccillo, D., Huertas-Company, M., Decencière, E., et al. 2018, MNRAS, 475, 894
  * Zitlau et al. (2016) Zitlau, R., Hoyle, B., Paech, K., et al. 2016, MNRAS, 460, 3152
  * Wolpert (1992) Wolpert, D.H, 1992, Neural Networks, 5(2), 241-259


# Reconstructing Retinal Visual Images from 3T fMRI Data Enhanced by
Unsupervised Learning

###### Abstract

The reconstruction of human visual inputs from brain activity, particularly
through functional Magnetic Resonance Imaging (fMRI), holds promising avenues
for unraveling the mechanisms of the human visual system. Despite the
significant strides made by deep learning methods in improving the quality and
interpretability of visual reconstruction, there remains a substantial demand
for high-quality, long-duration, subject-specific 7-Tesla fMRI experiments.
The challenge arises in integrating diverse smaller 3-Tesla datasets or
accommodating new subjects with brief and low-quality fMRI scans. In response
to these constraints, we propose a novel framework that generates enhanced 3T
fMRI data through an unsupervised Generative Adversarial Network (GAN),
leveraging unpaired training across two distinct fMRI datasets in 7T and 3T,
respectively. This approach aims to overcome the limitations of the scarcity
of high-quality 7-Tesla data and the challenges associated with brief and low-
quality scans in 3-Tesla experiments. In this paper, we demonstrate the
reconstruction capabilities of the enhanced 3T fMRI data, highlighting its
proficiency in generating superior input visual images compared to data-
intensive methods trained and tested on a single subject.

Index Terms—  Functional Magnetic Resonance Imaging, Generative Adversarial
Network, Visual Cortex

##  1 Introduction

To elucidate the intricate mechanisms of visual encoding and decoding
processes in the human brain, researchers have employed various methodologies,
ranging from measurements of brain activity to the application of deep
learning techniques. Noteworthy contributions to the field include studies
focused on the reconstruction of visual inputs from brain activity [1, 2, 3,
4, 5, 6]. These investigations have significantly advanced our comprehension
of the intricate relationship between neural activity and the reconstruction
of visual stimuli.

Building upon recent advancements in visual representations, contemporary
investigations have delved into the domain of functional Magnetic Resonance
Imaging (fMRI). These studies aim to capture neural responses during carefully
designed experiments where subjects engage with specific visual stimuli, such
as natural scene images. The exploration spans a spectrum of datasets, from
smaller-scale repositories like BOLD5000 [7], Deep Image Reconstruction [8],
and the Generic Object Decoding Dataset (GOD) [9], to more expansive datasets
like the Natural Scenes Dataset (NSD) [10] and the recently introduced Natural
Object Dataset (NOD) [11]. This diversified approach enables investigations of
unprecedented scale, resolution, and signal-to-noise ratio (SNR), allowing
researchers to navigate the intricate landscape of reconstructing visual input
images from fMRI signals.

Commencing with relatively limited sample sizes and lightweight algorithms
[12, 13], this line of investigation has transitioned toward the intricate
domain of deep learning models. Researchers are now dedicated to decoding and
reconstructing visual experiences through the utilization of advanced
techniques, including generative adversarial networks (GAN) and self-
supervised learning directly from fMRI data [2, 3, 6, 14]. Recent studies have
incorporated additional information by integrating semantic annotations of
input images to augment the decoding process [1, 15, 16]. This progressive
shift reflects an increased commitment to capturing the intricate nuances of
neural representations, emphasizing the integration of cutting-edge
methodologies to enhance the reconstructed visual experiences based on fMRI.

However, most high-performance studies necessitate subject-specific training
or fine-tuning from scratch and high-resolution (7T) fMRI scans to mitigate
the challenges posed by individual brain variability and the considerable
number of parameters in generative models [15, 14, 16]. From the standpoint of
a practical fMRI decoding framework, these approaches encounter limitations
when applied to new subjects who have not undergone extensive high-Tesla
experiments. This constraint restricts their practical applicability in
scenarios where the model is trained on subjects with ample 7T fMRI data but
is subsequently employed to decode subjects with shorter and lower resolution
(3T) fMRI scans.

Addressing this challenge, we propose a novel framework designed for the
reconstruction of visual stimulus images utilizing enhanced 3T fMRI data
across subjects generated by the Optimal Transportation Guided GAN (OT-GAN)
[17, 18]. Unlike conventional approaches that rely on training and testing
exclusively on single subjects with high-Tesla fMRI, our architecture is
trained on a large group of subjects spanning two distinct datasets: NOD with
3T fMRI and NSD with 7T fMRI. The integrated framework demonstrates the
capacity to generate enhanced 3T fMRI data of comparable quality to 7T data
from subjects who have only undergone short and low-resolution 3T fMRI
experiments. Notably, our approach achieves superior performance in terms of
Fréchet Inception Distance (FID) [19] and human judgment when compared to
models that were trained and tested on single subject.

##  2 CONTRIBUTIONS STATEMENT

Our study makes the following key contributions:

  * •

OT-GAN for retinotopic fMRI enhancement: We introduce our OT-GAN tailored for
generating enhanced 3T fMRI data from the NOD dataset in the shared space of
7T fMRI, aligning with each shared visual input.

  * •

Linear Regression Training: We train two linear regressions on the original 7T
fMRI NSD data and generated enhanced 3T fMRI data. These regressions are
designed to map the collection of visual inputs and semantic annotations to
their respective latent presentations, facilitating subsequent reconstruction.

  * •

Reconstruction on untrained subjects: We leverage Stable Diffusion [20] to
generate reconstructed images from 3T fMRI of an untrained subject. Our
results demonstrate superior image quality and FID score compared to subject-
specific methods.

These contributions collectively advance the field by proposing an effective
framework for generating enhanced 3T fMRI data, training linear regressions
for latent representation mapping, and employing Stable Diffusion for high-
quality image reconstruction.

##  3 Methods

Fig.1 illustrates the framework of our proposed method.

(a)

(b)

Fig. 1: Illustration of proposed framework: (a) overview of the entire
pipeline, red arrows indicate models requiring training, (b) overview of the
proposed OT-GAN structure.

###  3.1 Dataset and Splitting

We employed two distinct datasets in our study: the Natural Scenes Dataset
(NSD) [10] and Natural Object Dataset (NOD) [11].

Natural Scenes Dataset (NSD): NSD offers high resolution 7-Tesla fMRI data
from 30 to 40 sessions. During these sessions, subjects viewed natural scene
images sourced from MS-COCO dataset [21], which provides images with rich
contextual information and detailed annotations, enhancing the depth of the
visual stimuli compared to previous fMRI studies.

Natural Object Dataset (NOD): NOD consists of 3-Tesla fMRI data from 1 to 6
sessions, involving 30 subjects viewing images from both the MS-COCO and
ImageNet datasets.

Our fMRI preprocessing is grounded in the assumption that subjects performing
the same visual tasks yield similar fMRI recordings. We identified 70 images
that were shared between the NSD and NOD experiments. This shared set enables
us to extract corresponding fMRI data for training our GAN model, facilitating
the generation of enhanced 3-Tesla data from the low-quality NOD records to
match the 7-Tesla NSD records. For the NSD dataset, all 8 subjects contribute
to our 7T data. For the NOD dataset, we selected 9 out of 30 subjects who
participated in the short MS-COCO imaging session to provide our 3T data.

During the GAN training, trials related to the 70 shared images from the 8 NSD
subjects are used as 7T fMRI, and trials related to the 70 shared images of
the first 8 NOD subjects are used as 3T fMRI. For training the linear
regression model for image reconstruction, a combination of all 7T fMRI and
enhanced 3T fMRI generated by each trial from 3T fMRI is utilized. Instead of
testing on untrained images, as done in some previous studies [15, 16, 14],
our framework is evaluated on a new, untrained subject with low-quality fMRI
(i.e., the 9th subject in NOD dataset). His 3T fMRI data is fed to the GAN for
enhancement, and the enhanced fMRI data is then used to reconstruct visual
input images. For a comprehensive overview, refer to the detailed dataset
splitting provided in Table 1.

Dataset | subjects | fMRI Trials*
---|---|---
NOD (3T) | $9(8+1)$ | $9\times 10$
NSD (7T) | $8$ | $8\times 3$
Train (7T + enhanced 3T) | $8+8$ | $8\times 10+8\times 3$
Test (enhanced 3T) | 1 | $10$
Table 1: Collection of NSD, NOD dataset and train-test split, * are counting
number of trials for each shared image.

###  3.2 fMRI Preparation

To ensure the proper functioning of our GAN model, it is imperative that the
space of 3T and 7T fMRI trials is correctly aligned. Consequently, we opted
for the 32k fsLR surface space. For subjects in the NOD dataset, individual
fMRI data underwent preprocessing and registration onto the 32k fsLR surface
using the ciftify toolbox [22]. This process involved applying a high-pass
temporal filter (cutoff = $128$ s) and a spatial smoother (FWHM = $4$ mm).
Further details on the preprocessing steps can be found in the dataset
description [11]. As for subjects in the NSD dataset, the original researchers
provided preprocessed individual fMRI data on the fsaverage surface, which was
subsequently transformed to 32k fsLR using the neuromaps toolbox [23]. All
mappings of fMRI trials are stored separately for the left and right
hemispheres $V$, treated as two distinct channels $C$.

###  3.3 fMRI Generation with Unpaired OT-GAN

Denote $a\sim\mathbb{P}_{X}$ and $b\sim\mathbb{P}_{Y}$ as two probability
measurements on source and target distributions. Given a loss function
$\mathcal{L}:(x,y)\rightarrow\mathbb{R}_{+}$, we have the Monge’s optimal
transport mapping $G_{\theta}:Y\rightarrow X$ with optimal transportation cost
between $b\rightarrow a$ will be:

$\displaystyle G^{*}_{\theta}=$
$\displaystyle\inf_{f:a(x)=b(G^{-1}_{\theta}(x))}\int_{Y}\mathcal{L}(y,G_{\theta}(y))db(y)$
(1) $\displaystyle\theta^{*}=$
$\displaystyle\mathop{\arg\min}\limits_{\theta:\mathbb{P}_{X}=\mathbb{P}_{G_{\theta}(Y)}}\mathbb{E}_{y\sim\mathbb{P}_{Y}}\mathcal{L}(y,G_{\theta}(y))$
(2)

The parametric optimal transport map could be considered as a deep neural
network with a well-defined unsupervised training schedule on unpaired data.
Equation 2 can be relaxed to an unconstrained optimization by adding a
Lagrange multiplier term $\lambda
d(\mathbb{P}_{X},\mathbb{P}_{G_{\theta}(Y)})$, where $d$ corresponds to the
divergence between two distributions with all element smaller than $0$. We can
calculate the adversarial divergence by Wasserstein-1 distance [17] using
$D_{\mu}$ as a parametric discriminator:

$\displaystyle\mathbb{W}_{1}(\mathbb{P}_{X},\mathbb{P}_{G_{\theta}(Y)})=\sup_{||D_{\mu}||\leq
1}$ $\displaystyle\mathbb{E}_{x\sim\mathbb{P}_{X}}(D_{\mu}(x))-$
$\displaystyle\mathbb{E}_{y\sim\mathbb{P}_{Y}}(D_{\mu}(G_{\theta}(y)))$ (3)

In summary, high alignment between low-quality and high-quality fMRI could be
achieved by trading off consistency between enhanced and high-quality fMRI.
The two parametric mappings $G_{\theta}$ and $D_{\mu}$ could be trained
through a GAN structure by optimizing the combined objective function:

$\displaystyle\max_{\theta}\min_{\mu}E_{y\sim\mathbb{P}_{Y}}\mathcal{L}(y,G_{\theta}(y))+\lambda\mathbb{W}_{1}(\mathbb{P}_{X},\mathbb{P}_{G_{\theta}(Y)})$
(4)

We choose the conventional MSE as our loss function
$\mathcal{L}(y,G_{\theta}(y))=\mathbb{E}||y-G_{\theta}(y)||^{2}$. To maintain
the hidden structural information in the original fMRI, we use a U-shape
Network to pass underlying information across down-sample layers. Residual
Channel Attention Blocks (RCAB) [17, 18] are modified to 1D convolution and
then used to replace naive connections. Down-sample and up-sample blocks are
scaling the input dimension by $\times 1/2$ or $\times 2$, respectively. And
the discriminator is adapted from [24].

###  3.4 Image Reconstruction based on Diffusion Model

The Diffusion Model (DM) [25] is a probabilistic generative model that
recovers sampled variables from Gaussian noise to a specific learned
distribution through iterative denoising. In each diffusion step, the input
$\boldsymbol{x}$ is blurred by the addition of small Gaussian noise $\epsilon$
iteratively over time points $t$. This process can be summarized as
$\boldsymbol{x}_{t}=\sqrt{\alpha}x_{0}+\sqrt{1-\alpha_{t}}\epsilon_{t}$. The
denoising step employs a neural network $f\theta(\boldsymbol{x}_{t},t)$,
typically a U-Net, to restore the previous input. The learning objective
converges to $f\theta(\boldsymbol{x}_{t},t)\simeq\epsilon_{t}$ [25]. In the
Latent Diffusion Model (LDM), an encoder-decoder pair is applied before and
after the diffusion-denoising network, accelerating the process by
transferring the original pixel space calculation into a smaller encoded
latent space.

Recent studies [20] have demonstrated that this method can be improved by
incorporating conditional input into the neural network, such as semantic
annotations of the original input, to generate high-quality natural images.
Given the performance of Stable Diffusion [20], it becomes an ideal choice for
testing our enhanced 3T fMRI data in visual image reconstruction tasks.

Drawing inspiration from a recent study’s straightforward training framework
[16], we utilized the pre-trained Stable Diffusion v1.4 and kept all its
weights frozen. Two linear neural networks were applied to our combined fMRI
training set to generate the latent representation of corresponding visual
images and their annotations, respectively. After training, the enhanced 3T
fMRI from the 9th subject of the NOD dataset was used for generating test
images. The enhanced fMRI from his ten 3T scan trials when viewing each image
was averaged and then applied to the two networks. The generated latent
representation of visual and semantic inputs was subsequently used for
reconstruction in the pre-trained Stable Diffusion.

##  4 Results

Our reconstruction are displayed in Figure 2. For each input visual images, we
generated five different reconstruction, and we present the one with the least
FID. The FID [19] is a widely used metric for assessing image generation
quality by measuring the distance between ground-truth images and
reconstructions. Compared to recent reconstruction methods [15, 14, 16], our
framework produces considerably high-quality natural scenes with a test
subject that hasn’t been included in training.

(a) Ground Truth

(b) Reconstructed Image

Fig. 2: Ground truth and corresponding reconstruction from the 9th subject of
the NOD dataset.

We assess our performance using FID across the entire set of 70 shared images.
The metric serves to compare our results with two studies that focus on NSD
dataset: Lin et al. [15] and Takagi et al. [16], as outlined in Table 2. Our
FID scores demonstrate the superiority of our approach over previous methods,
which were trained and tested on a single subject with a lengthy 7T fMRI scan.
Notably, our framework excels when tested on an untrained new subject who has
only undergone a light-weight 3T fMRI experiment.

Method | FID
---|---
Lin et al. | $45.02$
Takagi et al. | $52.14$
Proposed | 40.48
Table 2: Comparison of FID score between NSD reconstruction on 70 shared
images

##  5 Conclusion

In this study, we introduced a robust OT-guided unpaired GAN model designed to
generate enhanced 3T fMRI scans. These scans exhibit comparable quality and
reconstruction performance to native 7T fMRI scans, as demonstrated across two
distinct datasets. The enhanced 3T fMRI data effectively preserves both visual
and semantic information, subsequently employed in a straightforward linear
regression model for visual imaging reconstruction utilizing Stable Diffusion.

Our approach underscores its capability to undertake reconstruction tasks
requiring extensive and high-quality fMRI scans, particularly for new subjects
with only brief and lower-quality fMRI scans. Through the adept utilization of
a well-trained GAN model, our method proves promising in addressing such
challenges. In subsequent work, we envisage modifying our framework to address
complex tasks, such as solving population receptive field (pRF) mapping and
facilitating brain disease diagnoses, all within the application of brief
3-Tesla fMRI experimental settings.

##  6 COMPLIANCE WITH ETHICAL STANDARDS

This research study was conducted retrospectively using human subject data
available in open access by [21, 10, 11]. Ethical approval was not required as
confirmed by the license attached with the open-access data.

##  7 Acknowledgments

The work was partially supported by NSF (DMS-1413417 & DMS-1412722) and NIH
(R01EY032125 & R01DE030286).

## References

  * [1] Fang et al.,  “Reconstructing perceptive images from brain activity by shape-semantic gan,”  Advances in Neural Information Processing Systems, vol. 33, pp. 13038–13048, 2020.
  * [2] Ren et al.,  “Reconstructing seen image from brain activity by visually-guided cognitive representation and adversarial learning,”  NeuroImage, vol. 228, pp. 117602, 2021.
  * [3] Le et al.,  “Brain2pix: Fully convolutional naturalistic video reconstruction from brain activity,”  BioRxiv, pp. 2021–02, 2021.
  * [4] Seeliger et al.,  “Generative adversarial networks for reconstructing natural images from brain activity,”  NeuroImage, vol. 181, pp. 775–785, 2018.
  * [5] Parthasarathy et al.,  “Neural networks for efficient bayesian decoding of natural images from retinal neurons,”  Advances in Neural Information Processing Systems, vol. 30, 2017.
  * [6] Gaziv et al.,  “Self-supervised natural image reconstruction and large-scale semantic classification from brain activity,”  NeuroImage, vol. 254, pp. 119121, 2022.
  * [7] Chang et al.,  “Bold5000, a public fmri dataset while viewing 5000 visual images,”  Scientific data, vol. 6, no. 1, pp. 49, 2019.
  * [8] Shen et al.,  “Deep image reconstruction from human brain activity,”  PLoS computational biology, vol. 15, no. 1, pp. e1006633, 2019.
  * [9] Horikawa and Kamitani,  “Generic decoding of seen and imagined objects using hierarchical visual features,”  Nature communications, vol. 8, no. 1, pp. 15037, 2017.
  * [10] Allen et al.,  “A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence,”  Nature neuroscience, vol. 25, no. 1, pp. 116–126, 2022.
  * [11] Gong et al.,  “A large-scale fmri dataset for the visual processing of naturalistic scenes,”  Scientific Data, vol. 10, no. 1, pp. 559, 2023.
  * [12] Kay et al.,  “Identifying natural images from human brain activity,”  Nature, vol. 452, no. 7185, pp. 352–355, 2008.
  * [13] Naselaris et al.,  “Bayesian reconstruction of natural images from human brain activity,”  Neuron, vol. 63, no. 6, pp. 902–915, 2009.
  * [14] Chen et al.,  “Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 22710–22720.
  * [15] Lin et al.,  “Mind reader: Reconstructing complex images from brain activities,”  Advances in Neural Information Processing Systems, vol. 35, pp. 29624–29636, 2022.
  * [16] Takagi and Nishimoto,  “High-resolution image reconstruction with latent diffusion models from human brain activity,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 14453–14463.
  * [17] Wang et al.,  “Optimal transport for unsupervised denoising learning,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 2, pp. 2104–2118, 2022.
  * [18] Zhu et al.,  “Optimal transport guided unsupervised learning for enhancing low-quality retinal images,”  Proc IEEE Int Symp Biomed Imaging, vol. 2023, Apr 2023.
  * [19] Heusel et al.,  “Gans trained by a two time-scale update rule converge to a local nash equilibrium,”  Advances in neural information processing systems, vol. 30, 2017.
  * [20] Rombach et al.,  “High-resolution image synthesis with latent diffusion models,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10684–10695.
  * [21] Lin et al.,  “Microsoft coco: Common objects in context,”  in Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer, 2014, pp. 740–755.
  * [22] Dickie et al.,  “Ciftify: A framework for surface-based analysis of legacy mr acquisitions,”  Neuroimage, vol. 197, pp. 818–826, 2019.
  * [23] Markello et al.,  “Neuromaps: structural and functional interpretation of brain maps,”  Nature Methods, vol. 19, no. 11, pp. 1472–1479, 2022.
  * [24] Ledig et al.,  “Photo-realistic single image super-resolution using a generative adversarial network,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4681–4690.
  * [25] Ho et al.,  “Denoising diffusion probabilistic models,”  Advances in neural information processing systems, vol. 33, pp. 6840–6851, 2020.