# Task Selection for AutoML System Evaluation

Jonathan Lorraine Google  Nihesh Anderson Google  Chansoo Lee Google  Quentin
De Laroussilhe Google  Mehadi Hassen Google

###### Abstract

Our goal is to assess if AutoML system changes - i.e., to the search space or
hyperparameter optimization - will improve the final model’s performance on
production tasks. However, we cannot test the changes on production tasks.
Instead, we only have access to limited descriptors about tasks that our
AutoML system previously executed, like the number of data points or features.
We also have a set of development tasks to test changes, ex., sampled from
OpenML with no usage constraints. However, the development and production task
distributions are different leading us to pursue changes that only improve
development and not production. This paper proposes a method to leverage
descriptor information about AutoML production tasks to select a filtered
subset of the most relevant development tasks. Empirical studies show that our
filtering strategy improves the ability to assess AutoML system changes on
holdout tasks with different distributions than development.

##  1 Introduction

Prob. of selectionSelected model type Figure 1:  We show the distribution of
selected model types for the prod. tasks and dev. tasks. The choices are
random forest (RF), deep neural networks (DNN), gradient-boosted decision tree
(GBDT), AdaNet, Linear-Feature-Cross (Cross), and a linear model.

Successful deployment of machine learning models requires many design choices,
often requiring expertise. Automatic Machine Learning (AutoML) systems
mechanize this process. A typical AutoML system consists of many components,
including feature engineering, model selection & optimization [1, 2, 3], and
hyperparameter optimization [4, 5, 6, 7].

Providers usually deploy an AutoML system as a service, which they want to
improve by rolling out changes and assessing system performance. Due to
providers limited visibility into _production tasks_ (abbreviated prod.)
supplied by actual clients, it is impossible to run system experiments on
these tasks. Instead, providers have a set of open-sourced _development tasks_
(abbreviated dev.) [8, 9, 10, 11, 12, 13, 14, 15, 16] which can be used to
assess system changes.

However, the distribution of the dev. tasks may be wildly different from the
received prod. tasks – see (App.)endix Fig. 1 and 2. Fig. 1 displays the
distribution of selected model types for a set of prod. and dev. tasks,
showing a significant difference, which could – for example – lead us to
pursue DNN improvements when a random forest is used most often in prod.
Therefore, testing the impact of a change on real users is a non-trivial task
unaddressed by existing papers on benchmarking [1, 2, 3, 17, 18, 19]. Our
contributions in this paper include:

  1. 1.

Proposing a framework to select a filtered subset of dev. tasks so the
performance delta – on a change to our AutoML system – is similar between
filtered and prod. tasks.

  2. 2.

Proposing task selection methods and comparing their trade-offs in large-scale
experiments.

  3. 3.

Showing specific examples of methods which improve our ability to accurately
assess AutoML system changes when we have task distribution shift, as in
production setups.

Density$\log_{10}$ of number of input features$\log_{10}$ of number of data
points Figure 2:  We show histograms of features for both the prod. and dev.
task distributions, illustrating significant differences.

##  2 Background and Key Concepts

Notations used in the paper are summarized in Appendix Table 1.

The Production AutoML System: The key components of our AutoML system include
the search space, hyperparameter optimizer, and how we assess performance. The
AutoML system is a service used on varied machine learning tasks for
commercial applications, supporting multi-modal datasets with image, natural
language text, and tabular data. Specific details are in App. Sec. C.1,
summarized here: We use a conditional search space, with a root parameter for
the algorithm choice (Fig. 1) and sub-parameters conditioned on the root
choice. A wide range of hyperparameters are tuned, including optimizer, model
choice, regularization, and feature engineering. Vizier [20] is our
hyperparameter optimizer, supporting user-specified parameters controlling the
HPO.

Changes to the AutoML System: AutoML service providers constantly make
_changes_ to the AutoML system in order to improve performance. By _change_ ,
we mean moving from an AutoML system with a _baseline setup_ to a _modified
setup_. Generally, we can view each setup as an arbitrary AutoML system; thus,
the change can encode almost any reasonable system modification. System
changes can be partitioned into those affecting the AutoML system/search
space, and those for the hyperparameter optimizer. We look at both in our
experiments in Sec. 4.1.

Tasks: AutoML service providers receive a stream of tasks from their end-
users. Here, a _task_ is a dataset and a _problem statement_ specifying what
is to be solved and the relevant metric. For example, stating that we have a
binary classification problem for some specified label, and the goal is
maximizing AUC. Our systems run on two sets of tasks: _Development tasks_
(dev.) are gathered from various open-source repositories (such as OpenML
[8]), have no restrictions and thus can be easily used to evaluate our
service. The second set is _Production tasks_ (prod.), which are those
received by our service from a client and _can only be run for the exact
client-specified purpose_. This setup has parallels with meta-learning (ex.,
task splits), but with additional issues and information/constraints available
for providers.

Unique Production Issues for AutoML Service Providers: AutoML service
providers have limited visibility into the uses of the ML system in product
applications. This could, for example, be to comply with privacy regulations
and the terms of service. Similarly, providers may not want to risk evaluating
experiments on model changes in live production environments. These
constraints force providers to assess system performance on dev. tasks.

System descriptors: AutoML service providers often have access to common
_descriptors_ about the executed tasks. These include _task descriptors_ ,
containing info about only the task, such as the number of data points or
number/modality of features. Alternatively, _system descriptors_ contain info
about both the task and our AutoML system. For example, the performance (ex.,
accuracy) of a model on a task with some hyperparameter values.

Using Filters for Production Deployment DecisionsDesigning Filters for
Production Deployment Decisions Figure 3:  _Left:_ How a filter is used to
select relevant filtered, development tasks (given production task
descriptors), which are evaluated on a system change for yes/no deployment
decisions. _Right:_ The filter is tuned for effective deployment decisions,
but we can not evaluate changes on production – see Sec. 2 – so we select
holdout tasks with no constraints to use as a proxy.

##  3 Problem Setup

We propose a method allowing us to better assess AutoML system changes in
prod. tasks by using their task descriptor to select relevant dev. tasks. More
formally, we assume that when making a roll-out decision on a change, the
service provider has descriptors for: (a) dev. tasks on the baseline setup.
(b) prod. tasks on the baseline setup. (c) dev. tasks on the modified setup.

Assessing AutoML system changes then boils down to the following steps – see
Fig. 3, left. (1) Filtering a subset of dev. tasks that more closely match
prod. tasks. (2) Assessing the change on the filtered subset of dev. tasks.
(3) Roll out the change to production if the modified setup improves over the
baseline setup on the filtered subset of dev. tasks. In this paper, we propose
a framework to design filters to perform step 1 of the above.

Since we are limited in using prod. tasks for the purpose of this paper, we
divide the dev. tasks into two sets as a proxy. From here on, _train tasks_
represent the role of dev. tasks and _holdout tasks_ are a proxy for prod.
tasks. Fig. 3 illustrates this setup. In App. Sec. D we describe a method to
assess if a change to our AutoML system improves performance across a
distribution of tasks by first measuring improvement on each task, then
aggregating improvement measures across tasks.

###  3.1 Filtering Tasks

The first step to assessing AutoML system changes is filtering dev. tasks that
closely match the holdout tasks (a proxy for prod. tasks). Our filtering
problem uses descriptor info from holdout tasks to select a filtered subset of
the dev. tasks. This section presents different filtration strategies and then
shows a method to evaluate the strategies.

Algorithm 1 simFilter(length, simMetric)

1:def filter(trainTasks, holdoutTasks):

2: sims = simMetric(trainTasks,holdoutTasks)

3: mostSimTasks = argsort(sims)[:length]

4: return trainTasks[mostSimTasks]

5:return filter $\triangleright$ Note - this returns a function

####  3.1.1 Similarity Filters

First, we design filters for a single holdout task, which we later aggregate
into filters for multiple tasks. Intuitively, _(sim)ilarity filter_ methods
take a way to measure the similarity between training tasks and a holdout task
– denoted a _similarity metric_ – then returns the top $n$ tasks. Alg. 1 shows
a skeleton for the proposed similarity filter parameterized by the number of
returned tasks (_filter length_) and the similarity metric. The returned item
is a filter function taking train tasks and a holdout task, then returning a
filtered subset of the train tasks.

Algorithm 2 distanceSimMetric(trainTasks,holdoutTask)

1:featureDistances = empty list

2:for trainTask in trainTasks do

3: $d$ = distance(trainTask.descriptor, holdoutTask.descriptor)

4: add $d$ to featureDistances

5:trainTaskSimilarities = $\nicefrac{{1}}{{\textnormal{featureDistances}}}$

6:return trainTaskSimilarities

####  3.1.2 Similarity Metrics

Our similarities metrics use the different types of info available to AutoML
service providers in out setup, including: (a) task descriptors, (b) system
descriptors. We also describe a similarity forming a heuristic performance
bound. Sec. 4.4 lists other metrics that could be considered, but we did not
use.

Task descriptor similarity: For descriptors like the number of data points or
features, we can reasonably rank train tasks in their euclidean distance from
the holdout task, which we simply use as the similarity metric (Alg. 2).

Performance descriptor similarity: For descriptors like the performance on
different hyperparameter values, we can not simply compute a euclidean
distance to the holdout task as with task descriptors. Instead we propose to
use the following intuition: similar tasks have nearby qualities for
hyperparameter values. Instead of evaluating train tasks on hyperparameter
configurations used for the holdout tasks on the baseline setup, we estimate
train tasks performance with a surrogate model. Then the correlation between
the predicted quality and the actual quality from the holdout tasks is used as
the similarity. Specific details are in App. Sec. D.3.

Oracle Similarity: We would like a (relatively tight) upper bound on the
possible performance of a similarity filter, to see how performant our filters
are. Intuitively, we approximate this by constructing a filter using extra
info about holdout tasks, which cannot be accessed via stored descriptors.
Specifically, we re-run the holdout tasks with changes and simply compute
correlations in task quality. See App. Sec. D.3 for more details. This
strategy is only a heuristic a upper bound and _cannot be used on actual prod.
tasks_ , but proves useful in our experiments nonetheless.

####  3.1.3 Constructing Filters for Multiple Holdout Tasks:

Our prior sections looked at filters with one holdout task. However, we want
filters for multiple holdout tasks because we have multiple tasks in prod. A
simple approach for transforming a single holdout task filter is taking the
union of the filtered tasks over every holdout task. However, this does not
control the number of filtered tasks and can include rarely selected train
tasks. We apply a filter for each holdout task and have them vote on the
selected tasks. Once the votes are collected, we return the top $n$ tasks for
a user-specified $n$. App. Alg. 9 shows our voting method, and App. Sec. D.3.1
discusses design choices.

####  3.1.4 How to evaluate a filter with a loss

Now that we have a set of filters defined, we want to evaluate them. An ideal
filter has the same result for changes on the filtered and holdout tasks.
Hence, comparing the evaluation results with changes should inform us of
filter strength. We show an example system change evaluating method in App.
Alg. 4 (_EvalSystemChange_).

We propose a filter evaluation technique (_EvalFilter_) in Alg. 3 via a loss
comparing system change results. _EvalFilter_ inputs a change to evaluate, a
set of training & holdout tasks, and a filter, then returns the scalar
difference between the change’s loss on filtered and holdout tasks.

We measure the difference between the change’s loss on the filtered and
holdout tasks with a $\log$-loss because this is a common choice for comparing
($\log$)probs – ex., as in logistic regression – and our change’s loss is a
($\log$)probability. The $\log$-loss is $t\log(y)+(1-t)\log(1-y)$, where $y,t$
are the probs. of improvement on the filtered and holdout tasks, respectively.
Fig. 4 shows this loss with our oracle similarity and simple baselines,
illustrating the range of values we should expect.

Algorithm 3 EvalFilter(filter, trainTasks, holdoutTasks,
change=(baselineSetup, modifiedSetup))

1:filteredTasks = filter(trainTasks, holdoutTasks.descriptors)

2:$y$ = EvalSystemChange(filteredTasks, change) $\triangleright$ The filtered
task improvement probability

3:$t$ = EvalSystemChange(holdoutTasks, change) $\triangleright$ The holdout
task improvement probability

4:return logLoss = $t\log(y)+(1-t)\log(1-y)$ $\triangleright$ Can clip $y,t$
to prevent $\infty$

DensityFilter loss for various filters Figure 4:  We show two examples of
contrasting filters with Alg. 6 using the loss from Alg. 3. For each filter,
we show the distribution of loss values from different samplings of train
tasks and holdout tasks, along with a solid vertical line at the mean.

##  4 Experiments

First, we investigate the setup for our problem in Sec. 4.1. Next, we explore
our filtering problem in Sec. 4.2, showing that our filters improve over
standard approaches. App. Sec. E contains additional details for our
experiments, including computational requirements.

###  4.1 Experimental Setup

To preface the filtering results in Sec. 4.2, we first provide our setup,
including: (a) changes made to our AutoML system. (b) the train and holdout
task partitions. In our experiments we focus on binary classification with AUC
as the quality, and say a system has performed better – on a single run, for a
single task – if the quality measured is higher.

Changes made to the AutoML system for experiments: We take a default setup and
contrast with modified setups as in App. Alg. 4. Our default uses a full,
conditional search space for a fixed wall-clock time budget. We use a diverse
range of modifications – see App. Sec. E.2.1 – including setups to assess
changes to: (1) search space, restricting to only a DNN. (2) hyperparameter
optimizer by turning on transfer learning from [20]. (3) Vizier optimizer
budget allowing $5\times$ more wall-clock time for hyperparameter queries. (4)
the entire AutoML system by changing the underlying library used to implement
the different learning algorithms. App. Fig.. 8 displays the quality
distribution for setups on various tasks, showing changes affect model quality
in varied ways. Some tasks have positively correlated qualities, while some
are anti-correlated, showing a simple similarity metric used in App. Alg. 8.

Training and holdout task selection for experiments: Since we cannot use the
prod. tasks for this paper, we divide the dev. tasks into two sets, _train
tasks_ and _holdout tasks_. We do this in two major ways for the results
presented here: (a) The dev. tasks are assigned into either train or holdout
at random, so the distribution of train and holdout tasks is similar. (b) The
dev. tasks are partitioned into all OpenML tasks for train and all other tasks
for holdout to simulate distribution shift. See App. Table 2 for more details
on specific tasks.

###  4.2 Filtering Problem Results

First, we compare different filters in Sec. 4.2.3. Then, we use filters when
holdout tasks are from a different distribution in Sec. 4.2.4, demonstrating
our proposed filters improve our ability to assess AutoML system change
performance effects on holdout tasks with realistic distribution shifts. We
also include investigations of design choices including evaluating AutoML
system changes (Sec. 4.2.1), filter strength (Sec. 4.2.2), and which changes
filtration is useful for (Fig. 5).

####  4.2.1 Investigating design choices for evaluating changes to AutoML
Systems

Alg. 4 evaluates our system changes and has arguments of the evaluation tasks
and system change (from a baseline setup to a modified setup). App. Fig. 10
contrasts the improvement probability for different AutoML system changes and
task sizes, showing how the improvement probability depends on the change we
make to our system. Some changes – like increasing the compute budget – almost
always improve performance, showing an example where the selected task does
not affect the result.

We also display different numbers of tasks, showing how Alg. 4 behaves as we
vary the filter and holdout sizes, verifying we can compare improvements on
the differently sized filter and holdout as in later experiments (ex., Fig. 6
or 7). App. Fig. 11 contrasts runs with a probability estimate using one or
multiple task evaluations, showing that more evaluations allow us to assess
the improvement probability better.

DensityImprovement probability for holdout and randomly filtered tasksHoldout
$\approx$ Train TasksChange $=$ Default $\to$ DNN OnlyHoldout $\neq$ Train
TasksChange $=$ Default $\to$ DNN OnlyHoldout $\approx$ Train TasksChange $=$
Default $\to$ $5\times$ compute Figure 5:  We look at the distribution of
evaluate change results on a baseline random filter – i.e., the improvement
probability given the change. _Left:_ The resulting distribution when
restricting the system search space to DNN only, and we sample our train and
holdout tasks the same distribution. We contrast this with other setups
highlighted in red. _Middle:_ The holdout tasks are sampled from a different
distribution than train as specified in Section 4.1. _Right:_ We vary the
system change to increase the compute budget by $5\times$.

####  4.2.2 Investigating how we evaluate filters

We now wield our ability to evaluate a change’s loss in our AutoML system for
evaluating filters. Alg. 3 computes a loss for a filter given a set of train
tasks, holdout tasks, and a system change with the difference between losses
on the filtered and holdout tasks (for the change). This loss depends on the
number of filtered and holdout tasks, how the train and holdout tasks differ,
and the system change. As such, we vary these in our experiments.

Fig. 5, left, evaluates the random task filter with the same train and holdout
task distributions, showing the same results distributions with moderate
variance. Here, filtering will only provide benefit if we have a limited
number of tasks output by the filter.

Fig. 5, middle, evaluates the random task filter while varying the train and
holdout task distributions, showing that random selection performs poorly
under mismatch. Here, we can benefit from a better filtering approach than the
baseline strategy of random.

Fig. 5, right, evaluates the random filter while varying the AutoML system
change showing all filters perform equally well – with near $0$ loss as in
App. Fig. 13 – if a change always improves performance. Here, filtering will
provide little to no benefit for any number of filtered tasks, because all
tasks perform similarly.

App. Fig. 12 repeats these plots while varying the number of filtered and
holdout tasks, showing we can compare the improvement probability distribution
for differing task sizes.

####  4.2.3 Comparing filtering strategies on 1 holdout task

Fig. 6 shows that various filtration strategies offer improvements in filter
loss over a random filter baseline, even as we vary the filter lengths,
similarities, and system changes. When we select all tasks, then all filters
perform equally. Here, performance descriptor similarity is the best feasible
filter for non-maximal tasks sizes, which we hypothesis is due to using extra
info about the system, and not just the task.

Also, App. Fig. 14, 15 and 16 show the best filtration will depend on the
chosen system change by repeating this for more filtering strategies and
system changes. Further, App. Fig. 16 verifies that most filter loss
differences pass a significance test.

####  4.2.4 Comparing filtering strategies on multiple holdout tasks:

We construct multi-holdout-task filters using single-holdout-task filters
(App. Alg. 9). We are particularly interested in holdout tasks with a
distribution shift as a proxy for production. Fig. 7 shows the absolute filter
performance for various filters (each line), numbers of holdout tasks (left
$\to$ right), and task distributions (top $\to$ bottom). Performance for all
filters was similar at about $8$ selected tasks, so we display at most $12$.
We use $30$ total tasks allocating the remaining $18$ to holdout.

In Fig. 7, top – where we sample the train and holdout tasks from the same
distribution – we do not see a benefit from filtering. The maximal filter size
– where all strategies are equal, and we use all tasks – performs best because
using more tasks is better when distributions match.

Most importantly, in the bottom of Fig. 7 – where we sample the train and
holdout tasks from different distributions – filtering improves our ability to
assess AutoML system change’s affects on performance with larger holdout
sizes. Notably, the best filters only uses $3$ tasks, with the # data points
then performance descriptor similarity.

Loss diff. from random baselineNumber of tasks selected by filterChange $=$
Default $\to$ DNN OnlyApproximate Oracle Similarity Filter Figure 6:  We
display the $5$-sample mean loss differences between filter strategies and a
random baseline. _Left:_ We show filter strategies in each color – see App.
Fig. 14 for other system changes. _Right:_ We show the largest possible loss
difference for system changes in each color with our heuristic upper bound –
see App. Fig. 15 for other filters.

###  4.3 Takeaways

We see clear performance boosts from the filtering strategies when we have a
limited number of holdout tasks, or the holdout task distribution is different
from training. As such, we have the following takeaways for practitioners:

  1. 1.

If there are many prod. tasks, and the distribution differs from dev.,
filtering helps – Fig. 7, bottom. Task (with $\\#$ datapoints) and performance
descriptor similarity performed best.

  2. 2.

Using filtering, we saw similar performance using only $\approx 20\%$ of the
tasks, providing a use-case for cheaper benchmarking with only $\approx 20\%$
the cost _even with no prod. distribution shift_ – Fig. 7.

  3. 3.

The best filter will depend on the system changes we use – Sec. 4.2.3.

###  4.4 Future Directions

Our primary goal is showing our filtering paradigm allows us to create simple
filters that can select valuable tasks for assessing AutoML system changes –
not to construct complex filters. The next step is designing better filters
because our simple baselines performed best. More complicated filters – like
KNNs – could rank tasks using all available descriptor information.
Alternatively, looking at more sophisticated metrics like problem complexity
measures could be fruitful.

We also found that simple task similarity metrics performed best, but these
encode minimal information about the task. Investigating why these simple
metrics performed robustly is interesting for future work. Also, we explored a
range of changes to the system, hoping our results apply to other AutoML
systems, but this should be experimentally verified. Finally, our filtering
methods readily scale to magnitudes larger development task sets, so looking
at how we should vary filtering in this regime may be necessary.

##  5 Limitations & Broader Impact

Limitations: We used the improvement probability to assess system performance
when evaluating system changes, but this might not accurately reflect a
provider’s risk tolerances. Users should select an appropriate scalar measure
of system performance. Another limitation is that using improvement
probability may not give helpful answers for changes that consistently improve
performance, like increasing compute budget. There are other performance
metric choices, like changes in the mean quality. Also, we explored a limited
number of ways to measure task distribution mismatch – there are various other
strategies.

Holdout $\approx$ Train TasksMean filter loss$1$ holdout task$8$ holdout
tasksAll $18$ holdout tasksHoldout $\neq$ Train TasksMean filter lossNumber of
tasks selected by filter Figure 7:  We show different filters’ absolute
performance as we vary the number of holdout tasks, where we sample the
holdout tasks from varying distributions described in Sec. 4.1. _Top:_ The
random baseline always performs better with more tasks. All methods perform
equally well with many holdout tasks, as expected due to matching
distributions. _Bottom:_ Our filters improve performance over the random
baseline for any number of holdout tasks.

Broader Impact Statement: While there are important impacts from using AutoML
in general, we focus on a specific impact of our filtering formulation: Our
method allows us to assess changes to our system using fewer development
tasks, which could help reduce the environmental impact of benchmarking our
system. For example, our best-performing filter used only $20\%$ of the
available training tasks. However, this could create unintended biases by
using fewer datasets. Also, our method could help service users who are
specifically concerned with privacy. We should further investigate which
groups most benefit from this.

##  6 Related work

Our related work covers: AutoML broadly, AutoML systems we hope to apply our
method on, then methods to evaluate those AutoML systems. Additional related
work is in App. Sec.B.

AutoML: [21] contains an overview of key AutoML components – including
hyperparameter optimization [22], meta-learning [23], and neural architecture
search [24].

AutoML Systems: We can apply our filters to arbitrary AutoML systems used for
production. This includes various publicly available AutoML systems like Auto-
WEKA [25], Auto-MEKA [26], Hyperopt-Sklearn [27], Auto-sklearn [28], Auto-Net
[29], TPOT [30], adaptive TPOT [31], the automatic statistician [32], AlphaD3m
[33], H20 [34], SmartML [35], ML-Plan [36], Mosaic [37], RECIPE [38], Alpine
Meadow [39], ATM [40], Rafiki [41] We focus on systems for multiple models,
but some systems focus on DNNs including MetaQNN [42], Auto-Keras [43], and
the various papers on Neural Architecture Search [44, 45, 46, 47, 48].

Evaluating AutoML Systems: There are various proposed benchmarks including
HPO-B [2], HPOBench [3], AutoML Benchmark [49], and other varied studies [1,
17, 18, 19, 50]. [51] outline fundamental AutoML evaluation practices.
Competitions are another method to provide common goals for AutoML systems,
including the AutoML Challenges [52], the AutoDL challenge [53] and NeurIPS
BBO Challenge [54].

Filtering for AutoML: There are fewer works on filtering strategies in AutoML.
The closest work we are aware of in this vein is Oboe [55] using collaborative
filtering on the matrix of qualities for many tasks for time-constrained HO.

##  7 Conclusion

We were motivated by assessing if AutoML system changes – i.e., the search
space or HPO – will improve the final output model’s performance on a separate
set of prod. tasks. But, we cannot run the system changes prod. tasks, so we
assess them on dev. tasks. However, the set of dev. and prod. tasks differ,
leading us to pursue changes improving dev. and not prod. We proposed the
filtering problem, leveraging available descriptor info about holdout tasks to
select useful dev. tasks. Then we proposed various filtration strategies,
which we used in large-scale empirical studies. We showed that the filters
improved our ability to assess if system changes improve performance on
holdout tasks from different distributions than training, such as in prod. We
hope this helps build the set of benchmarking strategies for more
sophisticated and realistic setups, allowing people to better deploy AutoML
systems. We also believe this provides a fruitful avenue of research in
stronger filtration strategies, leveraging the broad body of work on task
relationship learning.

## Acknowledgements

We would also like to thank Sagi Perel, and Luke Metz for feedback on this
work and acknowledge the Python community [56, 57] for developing the tools
that enabled this work, including numpy [58, 59, 60], Matplotlib [61] and
SciPy [62].

## References

  * Zöller and Huber [2021] Marc-André Zöller and Marco F Huber.  Benchmark and survey of automated machine learning frameworks.  _Journal of artificial intelligence research_ , 70:409–472, 2021.
  * Arango et al. [2021] Sebastian Pineda Arango, Hadi S Jomaa, Martin Wistuba, and Josif Grabocka.  Hpo-b: A large-scale reproducible benchmark for black-box hpo based on openml.  _arXiv preprint arXiv:2106.06257_ , 2021.
  * Eggensperger et al. [2021] Katharina Eggensperger, Philipp Müller, Neeratyoy Mallik, Matthias Feurer, René Sass, Aaron Klein, Noor Awad, Marius Lindauer, and Frank Hutter.  Hpobench: A collection of reproducible multi-fidelity benchmark problems for hpo.  _arXiv preprint arXiv:2109.06716_ , 2021.
  * Bergstra et al. [2011] James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl.  Algorithms for hyper-parameter optimization.  _Advances in neural information processing systems_ , 24, 2011.
  * Hutter et al. [2011] Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown.  Sequential model-based optimization for general algorithm configuration.  In _International conference on learning and intelligent optimization_ , pages 507–523. Springer, 2011.
  * Snoek et al. [2015] Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams.  Scalable bayesian optimization using deep neural networks.  In _International conference on machine learning_ , pages 2171–2180. PMLR, 2015.
  * Springenberg et al. [2016] Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter.  Bayesian optimization with robust bayesian neural networks.  _Advances in neural information processing systems_ , 29, 2016.
  * Vanschoren et al. [2013] Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo.  Openml: Networked science in machine learning.  _SIGKDD Explorations_ , 15(2):49–60, 2013.  doi: 10.1145/2641190.2641198.  URL http://doi.acm.org/10.1145/2641190.2641198.
  * Kohavi [1996] Ron Kohavi.  Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid.  In _Proceedings of the Second International Conference on Knowledge Discovery and Data Mining_ , KDD’96, page 202–207. AAAI Press, 1996\.
  * Moro et al. [2014] Sérgio Moro, Paulo Cortez, and Paulo Rita.  A data-driven approach to predict the success of bank telemarketing.  _Decision Support Systems_ , 62:22–31, 2014.  ISSN 0167-9236.  doi: https://doi.org/10.1016/j.dss.2014.03.001.  URL https://www.sciencedirect.com/science/article/pii/S016792361400061X.
  * Candillier and Lemaire [2012] Laurent Candillier and Vincent Lemaire.  Design and analysis of the nomao challenge - active learning in the real-world.  In _Proceedings of the ALRA : Active Learning in Real-world Applications, Workshop ECML-PKDD 2012, Friday, September 28, 2012, Bristol, UK_ , page to appear, 2012.
  * Guyon et al. [2004] Isabelle Guyon, Steve Gunn, Asa Ben-Hur, and Gideon Dror.  Result analysis of the nips 2003 feature selection challenge.  volume 17, 01 2004.
  * Johnson et al. [2013] Brian Johnson, Ryutaro Tateishi, and Nguyen Hoan.  A hybrid pansharpening approach and multiscale object-based image analysis for mapping diseased pine and oak trees.  _International Journal of Remote Sensing_ , 34:6969–6982, 10 2013.  doi: 10.1080/01431161.2013.810825.
  * Borkan et al. [2019] Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman.  Nuanced metrics for measuring unintended bias with real data for text classification.  _CoRR_ , abs/1903.04561, 2019.  URL http://arxiv.org/abs/1903.04561.
  * Zhang et al. [2015] Xiang Zhang, Junbo Zhao, and Yann LeCun.  Character-level Convolutional Networks for Text Classification .  _arXiv:1509.01626 [cs]_ , September 2015.
  * Diemert Eustache, Betlei Artem et al. [2018] Diemert Eustache, Betlei Artem, Christophe Renaudin, and Amini Massih-Reza.  A large scale benchmark for uplift modeling.  In _Proceedings of the AdKDD and TargetAd Workshop, KDD, London,United Kingdom, August, 20, 2018_. ACM, 2018.
  * Bischl et al. [2017] Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Frank Hutter, Michel Lang, Rafael G Mantovani, Jan N van Rijn, and Joaquin Vanschoren.  Openml benchmarking suites.  _arXiv preprint arXiv:1708.03731_ , 2017.
  * Balaji and Allen [2018] Adithya Balaji and Alexander Allen.  Benchmarking automatic machine learning frameworks.  _arXiv preprint arXiv:1808.06492_ , 2018.
  * Elshawi et al. [2019] Radwa Elshawi, Mohamed Maher, and Sherif Sakr.  Automated machine learning: State-of-the-art and open challenges.  _arXiv preprint arXiv:1906.02287_ , 2019.
  * Golovin et al. [2017] Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and David Sculley.  Google vizier: A service for black-box optimization.  In _Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining_ , pages 1487–1495, 2017.
  * Hutter et al. [2019a] Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren, editors.  _Automated Machine Learning - Methods, Systems, Challenges_.  Springer, 2019a.
  * Feurer and Hutter [2019] Matthias Feurer and Frank Hutter.  Hyperparameter optimization.  In Hutter et al. [63], pages 3–38.
  * Vanschoren [2019] Joaquin Vanschoren.  Meta-learning.  In Hutter et al. [63], pages 39–68.
  * Elsken et al. [2019] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter.  Neural architecture search.  In Hutter et al. [63], pages 69–86.
  * Kotthoff et al. [2019] Lars Kotthoff, Chris Thornton, Holger H. Hoos, Frank Hutter, and Kevin Leyton-Brown.  Auto-weka: Automatic model selection and hyperparameter optimization in weka.  In Hutter et al. [63], pages 89–103.
  * de Sá et al. [2018] Alex GC de Sá, Alex A Freitas, and Gisele L Pappa.  Automated selection and configuration of multi-label classification algorithms with grammar-based genetic programming.  In _International Conference on Parallel Problem Solving from Nature_ , pages 308–320. Springer, 2018.
  * Komer et al. [2019] Brent Komer, James Bergstra, and Chris Eliasmith.  Hyperopt-sklearn.  In Hutter et al. [63], pages 105–121.
  * Feurer et al. [2019] Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Tobias Springenberg, Manuel Blum, and Frank Hutter.  Auto-sklearn: Efficient and robust automated machine learning.  In Hutter et al. [63], pages 123–143.
  * Mendoza et al. [2019] Hector Mendoza, Aaron Klein, Matthias Feurer, Jost Tobias Springenberg, Matthias Urban, Michael Burkart, Max Dippel, Marius Lindauer, and Frank Hutter.  Towards automatically-tuned deep neural networks.  In Hutter et al. [63], pages 145–161.
  * Olson and Moore [2019] Randal S. Olson and Jason H. Moore.  Tpot: A tree-based pipeline optimization tool for automating machine learning.  In Hutter et al. [63], pages 163–173.
  * Evans et al. [2020] Benjamin Evans, Bing Xue, and Mengjie Zhang.  An adaptive and near parameter-free evolutionary computation approach towards true automation in automl.  In _2020 IEEE Congress on Evolutionary Computation (CEC)_ , pages 1–8. IEEE, 2020.
  * Steinrucken et al. [2019] Christian Steinrucken, Emma Smith, David Janz, James Lloyd, and Zoubin Ghahramani.  The automatic statistician.  In Hutter et al. [63], pages 175–188.
  * Drori et al. [2021] Iddo Drori, Yamuna Krishnamurthy, Remi Rampin, Raoni de Paula Lourenco, Jorge Piazentin Ono, Kyunghyun Cho, Claudio Silva, and Juliana Freire.  Alphad3m: Machine learning pipeline synthesis.  _arXiv preprint arXiv:2111.02508_ , 2021.
  * LeDell and Poirier [2020] Erin LeDell and Sebastien Poirier.  H2o automl: Scalable automatic machine learning.  In _Proceedings of the AutoML Workshop at ICML_ , volume 2020, 2020\.
  * Maher and Sakr [2019] Mohamed Maher and Sherif Sakr.  Smartml: A meta learning-based framework for automated selection and hyperparameter tuning for machine learning algorithms.  In _EDBT: 22nd International Conference on Extending Database Technology_ , 2019.
  * Mohr et al. [2018] Felix Mohr, Marcel Wever, and Eyke Hüllermeier.  Ml-plan: Automated machine learning via hierarchical planning.  _Machine Learning_ , 107(8):1495–1515, 2018.
  * Rakotoarison et al. [2019] Herilalaina Rakotoarison, Marc Schoenauer, and Michèle Sebag.  Automated machine learning with monte-carlo tree search.  _arXiv preprint arXiv:1906.00170_ , 2019.
  * de Sá et al. [2017] Alex GC de Sá, Walter José GS Pinto, Luiz Otavio VB Oliveira, and Gisele L Pappa.  Recipe: a grammar-based framework for automatically evolving classification pipelines.  In _European Conference on Genetic Programming_ , pages 246–261. Springer, 2017.
  * [39] Zeyuan Shang, Emanuel Zgraggen, and Tim Kraska.  Alpine meadow: A system for interactive automl.
  * Swearingen et al. [2017] Thomas Swearingen, Will Drevo, Bennett Cyphers, Alfredo Cuesta-Infante, Arun Ross, and Kalyan Veeramachaneni.  Atm: A distributed, collaborative, scalable system for automated machine learning.  In _2017 IEEE international conference on big data (big data)_ , pages 151–162. IEEE, 2017.
  * Wang et al. [2018] Wei Wang, Sheng Wang, Jinyang Gao, Meihui Zhang, Gang Chen, Teck Khim Ng, and Beng Chin Ooi.  Rafiki: Machine learning as an analytics service system.  _arXiv preprint arXiv:1804.06087_ , 2018.
  * Baker et al. [2016] Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar.  Designing neural network architectures using reinforcement learning.  _arXiv preprint arXiv:1611.02167_ , 2016.
  * Jin et al. [2019] Haifeng Jin, Qingquan Song, and Xia Hu.  Auto-keras: An efficient neural architecture search system.  In _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_, pages 1946–1956, 2019.
  * Zoph and Le [2016] Barret Zoph and Quoc V Le.  Neural architecture search with reinforcement learning.  _arXiv preprint arXiv:1611.01578_ , 2016.
  * Pham et al. [2018] Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean.  Efficient neural architecture search via parameters sharing.  In _International conference on machine learning_ , pages 4095–4104. PMLR, 2018.
  * Maziarz et al. [2019] Krzysztof Maziarz, Mingxing Tan, Andrey Khorlin, Kuang-Yu Samuel Chang, Stanisław Jastrzębski, Quentin de Laroussilhe, and Andrea Gesmundo.  Evolutionary-neural hybrid agents for architecture search.  2019\.
  * Adam and Lorraine [2019] George Adam and Jonathan Lorraine.  Understanding neural architecture search techniques.  _arXiv preprint arXiv:1904.00438_ , 2019.
  * Jastrzębski et al. [2018] Stanisław Jastrzębski, Quentin de Laroussilhe, Mingxing Tan, Xiao Ma, Neil Houlsby, and Andrea Gesmundo.  Neural architecture search over a graph search space.  _arXiv preprint arXiv:1812.10666_ , 2018.
  * Gijsbers et al. [2019] Pieter Gijsbers, Erin LeDell, Janek Thomas, Sébastien Poirier, Bernd Bischl, and Joaquin Vanschoren.  An open source automl benchmark.  _arXiv preprint arXiv:1907.00909_ , 2019.
  * He et al. [2021] Xin He, Kaiyong Zhao, and Xiaowen Chu.  Automl: A survey of the state-of-the-art.  _Knowledge-Based Systems_ , 212:106622, 2021.
  * Milutinovic et al. [2020] Mitar Milutinovic, Brandon Schoenfeld, Diego Martinez-Garcia, Saswati Ray, Sujen Shah, and David Yan.  On evaluation of automl systems.  In _Proceedings of the ICML Workshop on Automatic Machine Learning_ , 2020.
  * Guyon et al. [2019] Isabelle Guyon, Lisheng Sun-Hosoya, Marc Boullé, Hugo Jair Escalante, Sergio Escalera, Zhengying Liu, Damir Jajetic, Bisakha Ray, Mehreen Saeed, Michèle Sebag, Alexander Statnikov, Wei-Wei Tu, and Evelyne Viegas.  Analysis of the automl challenge series 2015-2018.  In Hutter et al. [63], pages 191–236.
  * Liu et al. [2021] Zhengying Liu, Adrien Pavao, Zhen Xu, Sergio Escalera, Fabio Ferreira, Isabelle Guyon, Sirui Hong, Frank Hutter, Rongrong Ji, Julio CS Jacques Junior, et al.  Winning solutions and post-challenge analyses of the chalearn autodl challenge 2019.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 43(9):3108–3125, 2021.
  * Turner et al. [2021] Ryan Turner, David Eriksson, Michael McCourt, Juha Kiili, Eero Laaksonen, Zhen Xu, and Isabelle Guyon.  Bayesian optimization is superior to random search for machine learning hyperparameter tuning: Analysis of the black-box optimization challenge 2020.  In _NeurIPS 2020 Competition and Demonstration Track_ , pages 3–26. PMLR, 2021.
  * Yang et al. [2019] Chengrun Yang, Yuji Akimoto, Dae Won Kim, and Madeleine Udell.  Oboe: Collaborative filtering for automl model selection.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 1173–1183, 2019.
  * Van Rossum and Drake Jr [1995] Guido Van Rossum and Fred L Drake Jr.  _Python reference manual_.  Centrum voor Wiskunde en Informatica Amsterdam, 1995.
  * Oliphant [2007] Travis E Oliphant.  Python for scientific computing.  _Computing in Science & Engineering_, 9(3):10–20, 2007.
  * Oliphant [2006] Travis E Oliphant.  _A guide to NumPy_ , volume 1.  Trelgol Publishing USA, 2006.
  * Van Der Walt et al. [2011] Stefan Van Der Walt, S Chris Colbert, and Gael Varoquaux.  The NumPy array: A structure for efficient numerical computation.  _Computing in Science & Engineering_, 13(2):22–30, 2011.
  * Harris et al. [2020] Charles R Harris, K Jarrod Millman, Stéfan J van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J Smith, et al.  Array programming with NumPy.  _Nature_ , 585(7825):357–362, 2020.
  * Hunter [2007] John D Hunter.  Matplotlib: A 2d graphics environment.  _Computing in Science & Engineering_, 9(3):90–95, 2007.
  * Jones et al. [2001] Eric Jones, Travis Oliphant, Pearu Peterson, et al.  SciPy: Open source scientific tools for Python.  2001\.
  * Hutter et al. [2019b] Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren, editors.  _Automatic Machine Learning: Methods, Systems, Challenges_.  Springer, 2019b.
  * Domke [2012] Justin Domke.  Generic methods for optimization-based modeling.  In _Artificial Intelligence and Statistics_ , pages 318–326. PMLR, 2012.
  * Maclaurin et al. [2015] Dougal Maclaurin, David Duvenaud, and Ryan Adams.  Gradient-based hyperparameter optimization through reversible learning.  In _International conference on machine learning_ , pages 2113–2122. PMLR, 2015.
  * Lorraine and Duvenaud [2018] Jonathan Lorraine and David Duvenaud.  Stochastic hyperparameter optimization through hypernetworks.  _arXiv preprint arXiv:1802.09419_ , 2018.
  * Mackay et al. [2018] Matthew Mackay, Paul Vicol, Jonathan Lorraine, David Duvenaud, and Roger Grosse.  Self-tuning networks: Bilevel optimization of hyperparameters using structured best-response functions.  In _International Conference on Learning Representations_ , 2018.
  * Lorraine et al. [2020] Jonathan Lorraine, Paul Vicol, and David Duvenaud.  Optimizing millions of hyperparameters by implicit differentiation.  In _International Conference on Artificial Intelligence and Statistics_ , pages 1540–1552. PMLR, 2020.
  * Lorraine et al. [2022a] Jonathan P Lorraine, David Acuna, Paul Vicol, and David Duvenaud.  Complex momentum for optimization in games.  In _International Conference on Artificial Intelligence and Statistics_ , pages 7742–7765. PMLR, 2022a.
  * Raghu et al. [2021] Aniruddh Raghu, Jonathan Lorraine, Simon Kornblith, Matthew McDermott, and David K Duvenaud.  Meta-learning to improve pre-training.  _Advances in Neural Information Processing Systems_ , 34:23231–23244, 2021.
  * Vicol et al. [2022] Paul Vicol, Jonathan P Lorraine, Fabian Pedregosa, David Duvenaud, and Roger B Grosse.  On implicit bias in overparameterized bilevel optimization.  In _International Conference on Machine Learning_ , pages 22234–22259. PMLR, 2022.
  * Lorraine et al. [2022b] Jonathan Lorraine, Paul Vicol, Jack Parker-Holder, Tal Kachman, Luke Metz, and Jakob Foerster.  Lyapunov exponents for diversity in differentiable games.  In _Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems_ , pages 842–852, 2022b.
  * Močkus [1975] Jonas Močkus.  On bayesian methods for seeking the extremum.  In _Optimization techniques IFIP technical conference_ , pages 400–404. Springer, 1975.
  * Snoek et al. [2012] Jasper Snoek, Hugo Larochelle, and Ryan P Adams.  Practical bayesian optimization of machine learning algorithms.  _Advances in neural information processing systems_ , 25, 2012.
  * Kandasamy et al. [2020] Kirthevasan Kandasamy, Karun Raju Vysyaraju, Willie Neiswanger, Biswajit Paria, Christopher R Collins, Jeff Schneider, Barnabas Poczos, and Eric P Xing.  Tuning hyperparameters without grad students: Scalable and robust bayesian optimisation with dragonfly.  _J. Mach. Learn. Res._ , 21(81):1–27, 2020.
  * Wang et al. [2021] Zi Wang, George E Dahl, Kevin Swersky, Chansoo Lee, Zelda Mariet, Zack Nado, Justin Gilmer, Jasper Snoek, and Zoubin Ghahramani.  Automatic prior selection for meta bayesian optimization with a case study on tuning deep neural network optimizers.  _arXiv preprint arXiv:2109.08215_ , 2021.
  * Feurer et al. [2015] Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter.  Methods for improving bayesian optimization for automl.  In _Proceedings of the International Conference on Machine Learning_ , 2015.
  * Thornton et al. [2013] Chris Thornton, Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown.  Auto-weka: Combined selection and hyperparameter optimization of classification algorithms.  In _Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 847–855, 2013.
  * Friedman and Popescu [2008] Jerome H Friedman and Bogdan E Popescu.  Predictive learning via rule ensembles.  _The annals of applied statistics_ , 2(3):916–954, 2008.
  * Cortes et al. [2017] Corinna Cortes, Xavier Gonzalvo, Vitaly Kuznetsov, Mehryar Mohri, and Scott Yang.  Adanet: Adaptive structural learning of artificial neural networks.  In _International conference on machine learning_ , pages 874–883. PMLR, 2017.
  * Song et al. [2022] Xingyou Song, Sagi Perel, Chansoo Lee, Greg Kochanski, and Daniel Golovin.  Open source vizier: Distributed infrastructure and api for reliable and flexible blackbox optimization.  In _First Conference on Automated Machine Learning (Main Track)_ , 2022.

## Appendix

##  Appendix A Glossary

Table 1: Glossary Search Space | The set of all possible hyper-parameters.
---|---
Task | The specification of the dataset and the learning problem.
(Prod.)uction tasks | Tasks observed in the production environment, presented by customers.
(Dev.)elopment tasks | Tasks used for development, constituting open-source datasets.
Filter | An algorithm for choosing a subset.
Train tasks | A subset of the dev. tasks used for designing a filter.
Holdout tasks | A subset of the dev. tasks used as a
| representative of production tasks for evaluating a filter.
Filtered tasks | Subset of train tasks chosen by a filter.
Quality | A scalar quantity depicting the performance of a model on a task.
Baseline Setup | Some default AutoML setup.
Modified Setup | A non-default AutoML setup.
Change | A combination of a baseline and modified setup.
UB | Upper bound
DNN | Deep Neural Network

##  Appendix B Additional Related work

Hyperparameter optimization in AutoML: There are various scalable methods for
HO using more information than quality evaluations [64, 65, 66, 67, 68, 69,
70], which can have their own pitfalls[71, 72]. However, we focus on black-box
HO like Bayesian optimization [6, 73, 74, 75, 76]. [77] focus on Bayesian
optimization for AutoML. We use Vizier [20], which supports various black-box
optimizers including Bayesian Optimization.

##  Appendix C Additional Background

###  C.1 The Production AutoML System

####  C.1.1 The Search Space

Our AutoML system explores jointly algorithm selection and hyper-parameters
[78], modeled as a conditional search space, with a root parameter for the
algorithm choice and conditional sub-parameters conditioned on the choice of
the root node. Our included learning algorithms are Feed-forward Neural
Networks (DNNs), Random Forests, Gradient Boosted Decision Trees, Linear
models, Sparse Linear Model with feature crosses (Cross) similar to RuleFit
[79], and ensembles of Linear and DNNs using AdaNet [80].

####  C.1.2 The Hyperparameter Optimizer

We use Vizier [20, 81] to optimize our hyperparameters, which supports various
user-specified parameters controlling the HPO. Vizier supports various
underlying HPO algorithms – ex., random search, and Bayesian optimization. We
can specify parameters controlling the underlying HPO algorithms – including
exploration rate, whether to use transfer learning and how to apply early
stopping.

####  C.1.3 Assessing System Performance

We focus on binary classification problems for simplicity here, but our work
generalizes for an arbitrary scalar quality measure. Specifically, we measure
the AUC on the validation subset for the best model selected from the search
space by our hyperparameter optimizer. The system has performed better if we
measure a higher quality on a single run for a single task.

##  Appendix D The Filtering Problem

Evaluating Changes to the AutoML System: To assess if some change to our
AutoML system improves performance across a distribution of tasks, we first
measure improvement on each task and then aggregate the improvement measures
across tasks. Note that quality measures between tasks are not directly
comparable. For example, even if two tasks use the same quality metric, such
as AUC, the scale might be different. In other words, an AUC of $0.92$ might
be suitable for one task, while non-suitable for another task where $0.99$ AUC
is possible. To be able to compare quality across tasks, we measure
improvement on a task with the probs. of the quality improving (Alg. 4). To
aggregate the improvement measure (i.e., probs. of improvement) across tasks,
we could simply take the mean of the probabilities. Instead, we use the mean
of logits – a simple transformation mapping the probs. to $(-\infty,\infty)$ –
then going back to probs. with the inverse-logit (Alg. 4).

Algorithm 4 EvalSystemChange(tasks, change=(baselineSetup, modifiedSetup))

1:for task in tasks do $\triangleright$ compute every taskLoss for each task

2: get baselineQualities from baselineSetup $\triangleright$ An array of
qualities – ex., from a database

3: get modifiedQualities for each run with modifiedSetup

4: probImproved = fraction of pairings with modifiedQuality > baselineQuality

5: taskLoss = logit(probImproved) $\triangleright$ Loss is log-odds of
improvement probability with change

6:return expit(meanTaskLoss) $\triangleright$ Aggregate with mean, and take
inverse logit for a probability

###  D.1 How Filters work

a

Algorithm 5 abstractFilter(trainTasks, holdoutTaskDescriptors)

1:Construct a list L containing elements of trainTasks, given
holdoutTaskDescriptors

2:return L

###  D.2 How to Compare Filters

a

Algorithm 6 contrastFilters(newFilter, baselineFilter, baselineSetup,
modifiedSetup, taskPartitions)

1:Initialize loss storage arrays newFilterLosses, baselineFilterLosses

2:for each trainTasks, holdoutTasks in taskPartitions do

3: newFilterLoss = EvalFilter(newFilter, trainTasks, holdoutTasks,
baselineSetup, modifiedSetup)

4: baselineFilterLoss = EvalFilter(newFilter, trainTasks, holdoutTasks,
baselineSetup, modifiedSetup)

5: Store newFilterLoss and baselineFilterLoss

6:Compute summary of loss distributions – ex., the difference in the mean
losses, or a significance test to see if newFilter’s loss is lower,
significantly.

7:return filter loss distribution summary

###  D.3 Similarity Metrics

a

Performance descriptor similarities: Alg. 7 uses a surrogate model to estimate
the performance of the hyperparameter configurations on a train task by taking
the hyperparameter values as input features. The intuition is that similar
tasks perform similarity on the same hyperparameter values. Instead of
actually evaluating the train tasks on all hyperparameter configuration used
for the holdout tasks on the baseline setup we estimate training performance
using a surrogate model.

The surrogate model is trained for each train task on past runs on all
baseline setups. Then the trained surrogate model are used to predict what
performance metric will be for the hyperparameter values that were applied to
holdout task tasks on the baseline setup. These predictions give an idea of
how the train tasks would have performed on the set of hyperparameter values
used for the holdout tasks. The correlation between the predicted
quality/objective metric and the actual metrics from the holdout tasks are
computed and used as a similarity measure between the train tasks and the
holdout task. We outline this method in Alg. 7.

Algorithm 7 performanceDescriptorSimilarity(trainTasks, holdoutTask)

1:trainTaskSimilarities = empty list

2:Train surrogate models for each task in trainTasks.

3:holdoutTaskHparamConfigs, holdoutTaskQuality = fetch past runs for
holdoutTask for baseline setup

4:for trainTask in trainTasks do

5: surrogateModel = Fetch surrogate model for trainTask

6: estimatedQuality = surrogateModel(holdoutTaskHparamConfigs)

7: similarity = Compute a correlation coefficient for holdoutTaskQuality and
estimatedQuality

8: store similarity for task in trainTaskSimilarities

9:return trainTaskSimilarities

Approximate oracle similarity: When developing our filters, it would be
desirable to have a (relatively tight) upper-bound on the possible performance
given our limited access to info about the holdout tasks. For example, once we
approach this performance then we know it is not worth attempting to design
better filters. This also allows us to better gauge the magnitude of
improvement over a random task selection baseline.

We propose a heuristic to approximate an upper-bound on the filter performance
using limited descriptors about holdout data, with a filter which uses the
unrestricted descriptors about holdout data. This setup allows re-running the
holdout data with AutoML system changes. If we can access this info about
holdout tasks, then we wouldn’t need to filter. We would simply evaluate our
changes on holdout tasks as is standard practice. Nonetheless, this is useful
to bound possible filter performance.

Specifically, in App. Alg. 8 we compute the similarity by simply computing the
correlation in qualities between a train task and test task – sampled over
different setups to our AutoML system. The choice of setups to sample over,
and the correlation coefficient to compute are user-specified. This is
heuristic with no strong guarantees, but proves useful in our experiments.
Experiments for App. Alg. 8 are in Sec. E.2.2.

Algorithm 8 oracleSimilarity(trainTasks, holdoutTask)

1:trainTaskSimilarities = empty list

2:for trainTask in trainTasks do

3: trainQualities, holdoutQualities = empty lists

4: for setup in systemSetups do

5: store trainQuality = Run the train task with the setup

6: store holdoutQuality = Run the holdout task with the setup

7: similarity = compute a correlation coefficient for trainQualities and
holdoutQualities

8: store similarity for task in trainTaskSimilarities

9:return trainTaskSimilarities

####  D.3.1 Constructing Filters for Multiple Holdout Tasks

For Alg. 9, the number of filtered tasks for the inner, single-task filter and
the outer multi-task filter can be different, but we keep them the same for
simplicity in our experiments. Also, each filtered task receives a single vote
of equal weight. We can use more complicated strategies like weighted votes –
ex., based on similarity scores – but these were unnecessary for our
experiments.

Algorithm 9 VotingFilter(trainTasks, testTasks, filterLength, innerFilter)

1:initialize voteCounts = dictionary of how many votes each train task
received

2:for testTask in testTasks do

3: filteredTasks = innerFilter(trainTasks, testTask)

4: for filteredTask in filteredTasks do

5: voteCounts[filteredTask] += 1

6:return the top filterLength tasks with the most votes in voteCounts

####  D.3.2 How to compare filters:

Once we can evaluate a filter’s loss, we use this to compare different
filters. App. Alg. 6 shows a skeleton for our method of comparing filters,
with relevant experiments in Sec. 4.2.3. We look at two simple strategies:
First, we compute the cross-entropy loss for each filter – i.e., the filter
log-loss averaged over different train and test task partitions – and report
the difference in cross-entropies as the magnitude of improvement between
filtering strategies (Fig. 6). Second, we use a significance test on the
difference in cross-entropy for the filters (Fig. 16).

##  Appendix E Experiments

###  E.1 Compute Requirements

The total compute requirement is equivalent to running $60k$ CPUs for $1$ day,
consuming $10$ PiB of memory on average.

Table 2: Training details Datasets | OpenML [8] (Phishing Websites, Adult [9],
---|---
| Bank Marketing [10], Churn, Electricity, Kr-vs-kp, Numerai28.6,
| Sick, Spambase, Nomao [11], Jm1, Internet Advertisements, Phoneme,
| Madelon [12], Bioresponse, Wilt [13]), Civil Comments [14], Imdb,
| Census, Yelp sentiment [15], Criteo [16]
Search Space | Random Forest, Linear, Deep Neural Networks, Gradient Boosted
| Decision Trees, AdaNet, Linear Feature Cross
Data Splits | 80-10-10 split, uniformly at random.

###  E.2 Experimental Setup

####  E.2.1 Changes made to the AutoML system for experiments

a

Quality for OpenML jm1 taskQuality for OpenML sick taskQuality for OpenML
ozone taskQuality for OpenML sick task Figure 8:  We show of the distribution
of qualities for our setups on various pairs of OpenML tasks. Each setups
qualities are shown in a different color. For each task and setup, we have
multiple observations over re-runs, so we plot every pairing of the tasks
qualities for each setup.

####  E.2.2 Approximate Oracle Similarity Results

Fig. 9 display the matrix of similarities from Alg. 8, computed using the
system setups from Sec. E.2.1, showing non-trivial task relationships.

Task IndexTask Index Figure 9:  We show the matrix of approximate oracle
similarities where each row and column correspond to a single task.

###  E.3 Filtering Problem Results

####  E.3.1 Investigating design choices for evaluating changes to AutoML
Systems

a

DensityImprovement probability for $n$ tasksChange $=$ Default $\to$ DNN
OnlyChange $=$ Default $\to$ $5\times$ compute Figure 10:  We contrast runs of
_EvalSystemChange_ (Alg. 4) showing the improvement probability for different
AutoML system changes as we vary the number of tasks.  DensityImprovement
Prob. for $n$ tasks$1$ quality evaluation$5$ quality evaluations Figure 11:
We contrast runs of Alg. 4 with a probability estimate using a single task
evaluation, with a probability estimate using multiple task evaluations.
Density$1$ filtered task, $1$ holdout taskImprovement Prob. for Holdout and
Filtered Tasks$10$ filtered tasks, $1$ holdout task$10$ filtered tasks, $10$
holdout tasks Figure 12:  We look at the distribution of evaluate change
results – i.e., the improvement probability given the change – on both the
randomly filtered tasks and the holdout tasks, as we vary the number of
filtered and holdout tasks. For all results we use $5$ quality samples.
Takeaway: We should account for how we select the number of filtered tasks and
holdout tasks when assessing filters.

####  E.3.2 Investigating how we evaluate filters

a

DensityFilter LossChange $=$ Default $\to$ DNN OnlyChange $=$ Default $\to$
$5\times$ compute Figure 13:  We show the distribution of filter loss values
from different samplings of train tasks and holdout tasks, for the changes in
Fig. 5.

####  E.3.3 Comparing different filtering strategies

a

Loss Diff. from Random BaselineNumber of Tasks Selected by FilterChange $=$
Default $\to$ DNN OnlyChange $=$ Default $\to$ Diff. AutoML System Figure 14:
The mean of the differences between a filter strategy with a random baseline –
multiple filter strategies are displayed in each color.  Loss Diff. from
Random BaselineNumber of Tasks Selected by FilterApproximate Oracle Similarity
FilterPerformance Descriptor Similarity Filter Figure 15:  The mean of the
differences between a filter strategy with a random baseline – multiple system
changes are displayed in each color. Takeaway: The approximate oracle
similarity filter always improves performance over the random baseline. The
performance descriptor similarity improves performance over the random
baseline when assessing changing to the search space, but not for changes to
the implementation.  Loss Diff. from Random BaselineNumber of Tasks Selected
by FilterPerformance Descriptor FilterChange $=$ Default $\to$ DNN Only Figure
16:  The mean of the differences in quality, relative to a random baseline,
and filtered to only show values with a significance level greater than $.05$
between a filter strategy with a random baseline – for different changes in
setups. Takeaway: The significance filtered results are similar to their
respective unfiltered results in Fig. 14 and 15.


# Author guidelines for ISBI proceedings manuscripts

# Rethinking Dual-Domain Undersampled MRI reconstruction: domain-specific
design from the perspective of the receptive field

###### Abstract

Undersampled MRI reconstruction is crucial for accelerating clinical scanning.
Dual-domain reconstruction network is performant among SoTA deep learning
methods. In this paper, we rethink dual-domain model design from the
perspective of the receptive field, which is needed for image recovery and
K-space interpolation problems. Further, we introduce domain-specific modules
for dual-domain reconstruction, namely k-space global initialization and
image-domain parallel local detail enhancement. We evaluate our modules by
translating a SoTA method DuDoRNet under different conventions of MRI
reconstruction including image-domain, dual-domain, and reference-guided
reconstruction on the public IXI dataset. Our model DuDoRNet+ achieves
significant improvements over competing deep learning methods.

Index Terms—  MRI reconstruction, dual-domain, receptive field, K-space

##  1 Introduction

Magnetic resonance imaging (MRI) is a non-invasive and flexible imaging
modality widely used in clinical practice. Complete K-space measurements lead
to unbearable acquisition time while fewer measurements lead to aliasing and
blurring in the image. Under-sampled MRI reconstruction aims to reconstruct
the high-quality, clean MRI image from its low-quality, aliased counterpart.
Previously, Compressed Sensing (CS) and Parallel Imaging (PI) accelerated MRI
reconstruction 2-3 times. Since the revolutionary work [1], convolutional
neural networks (CNN) have become the primary workhorse for under-sampled MRI
reconstruction. As the success of Transformer [2] is now indisputable in
computer vision, Transformer has shown great potential for undersampled MRI
reconstruction as well [3, 4, 5, 6, 7].

Many methods [8, 9, 10] focus on adapting novel architecture designs of image-
domain neural networks. Customizing conventional CNNs or ViT to MRI further
benefits MRI reconstruction, including K-space data consistency (DC) [11, 12],
dual-domain recurrent learning [13, 14], over-complete representation [15].
SwinMR [3] and DSFormer [4] pioneer Swin transformer [16] as a strong backbone
in single-contrast and multi-contrast MRI reconstruction, respectively.
ReconFormer [5] develops a recurrent pyramid Transformer. Feng et al. [6]
proposed a task transformer network for joint MRI reconstruction and super-
resolution.

Among all SoTA methods, dual-domain networks show strong performance by
considering MRI reconstruction in both image and K-space domains. DuDoRNet
[14] presents a dual-domain [13] recurrent learning strategy and used a
dilated residual dense CNN to eliminate non-local aliasing artifact in the
image domain. DuDoCAF [7] leverages the query-key mechanism of MSA. Yet their
models use the same networks for both domains, neglecting the distinct
properties of image recovery and k-space recovery problem.

In our study, (1) we analyze the feasibility of K-space interpolation from the
perceptive of the receptive field; (2) we propose two novel domain-specific
modules, including image-domain parallel local detail enhancement and k-space
global initiation; and (3) extensive experiments on the public IXI-dataset
validate the superiority of our model, DuDoRNet+, under multiple settings of
MRI reconstruction including image-domain, dual-domain, and dual-domain
undersampled MRI reconstruction guided by a reference protocol.

##  2 Analysis and Method

###  2.1 Problem Formulation

Let $k_{u}\in\mathbb{C}^{mn}$ and $k_{f}\in\mathbb{C}^{mn}$ be the under-
sampled and fully-sampled k-space signal respectively;
$i_{u}\in\mathbb{C}^{mn}$, $i_{f}\in\mathbb{C}^{mn}$, and
$i_{r}\in\mathbb{C}^{mn}$ be the under-sampled, fully-sampled and
reconstructed image signal respectively; $M\in\mathbb{R}^{mn}$ be the binary
k-space mask for acceleration.

The under-sampled MRI reconstruction can be formulated as an image recovery
problem [1, 15, 17, 18, 3] with K-space DC as a regularisation term [11]:

$\mathop{\arg\min}\limits_{\theta_{i}}\left(\left\|i_{f}-\mathcal{P}_{i}\left(i_{u};\theta_{i}\right)\right\|_{2}^{2}\right.\\\
\left.+\lambda\left\|k_{u}-M\odot\mathcal{F}\left(\mathcal{P}_{i}\left(i_{u};\theta_{i}\right)\right)\right\|_{2}^{2}\right),$
(1)

where $\mathcal{F}$ is 2D discrete Fourier Transform and the approximation
function $\mathcal{P}_{x}(\cdot;\theta_{x})$ is used to predict a
reconstructed signal $x_{r}$ given its parameter $\theta_{x}$ and any under-
sampled input. A few works [14, 13, 7] leverage the relationship between image
and k-space domain using Fourier Transform pairs
($\mathcal{F},\mathcal{F}^{-1}$) and transform MRI reconstruction into a
multi-variable optimization problem described as

$\begin{split}\mathop{\arg\min}\limits_{\theta_{i},\theta_{k}}&\left(\left\|k_{f}-\mathcal{P}_{k}\left(\mathcal{F}\left(\mathcal{P}_{i}\left(i_{u};\theta_{i}\right)\right);\theta_{k}\right)\right\|_{2}^{2}\right.\\\
&+\left\|i_{f}-\mathcal{P}_{i}\left(\mathcal{F}^{-1}\left(\mathcal{P}_{k}\left(k_{u};\theta_{k}\right)\right);\theta_{i}\right)\right\|_{2}^{2}\\\
&+\left.\lambda\left\|k_{u}-M\odot\mathcal{F}\left(\mathcal{P}_{i}\left(\mathcal{F}^{-1}\left(\mathcal{P}_{k}\left(k_{u};\theta_{k}\right)\right);\theta_{i}\right)\right)\right\|_{2}^{2}\right),\end{split}$
(2)

and solve it using a dual-domain recurrent learning strategy [14].

Yet most of existing works design the approximation function
$\mathcal{P}_{x}(\cdot;\theta_{x})$ using neural networks whose architectures
mainly concentrate on the image recovery problem, neglecting the distinct
properties of k-space recovery problem. We argue that a hand-crafted design on
dual-domain approximation functions is crucial for solving this multi-variable
optimization problem since image and k-space bear different properties and the
tasks in two spaces shall be distinct further.

###  2.2 Solving K-space recovery problem from the perspective of the
receptive field

Organs scanned by MRI (e.g brains, knees) are usually rich in contextual
details, where a larger receptive field can improve the image reconstruction
performance in general. On the other hand, undersampled MRI reconstruction is
a K-space recovery problem and has been considered as a local interpolation in
many CS algorithms [19]. However, when MRI is under-sampled with a higher
rate, the average number of measurements in a fixed local region decreases.

Consider a set of K-space measurements after accelerating MRI with rate $a$
using 1D Cartesian sampling pattern. Conventionally, a central fraction of
K-space measurements called auto-calibration region is fully sampled,
accounting for a fraction of $R_{acs}$; while the other measurements are
sampled randomly.

For the non-calibration measurements, the feasibility of interpolation depends
on the number of k-space scanning lines in a reachable local region $K$ of
size $k*k$, in which the minimal number of data points is two. The probability
$P$ of a feasible interpolation is

Fig. 1: Probability of a feasible interpolation $P$ w.r.t. the size of
receptive field and acceleration rate $a$ when $R_{acs}$=0.125.

$P=1-(p)^{k}-C_{1}^{k}(p)^{k-1}(1-p),$ (3)

where $p$ is the possibility that none of the K-space lines are measured,
determined by $R_{acs}$ and $a$:

$p=1-\frac{\frac{1}{a}-R_{acs}}{1-R_{acs}}.$ (4)

An illustration of the distribution of $P$ w.r.t $(K,a)$ given a fixed
$R_{acs}$ is shown in figure 1. When accelerating with high rate $(>4)$, the
probability of feasible interpolation is lower than $50\%$ with , not to say a
successful interpolation.

Using ConvNets for MRI reconstruction [15] [14] can achieve large receptive
field by upsampling, dilation convolution or large kernels. However, dilation
convolution does not lead to an increase of lower bound for doing
interpolation in k-space since dilation increases receptive field with aligned
atrous part. Upsampling heavily burdens the GPU. Also, piling up large
convolution kernels harms model convergence.

###  2.3 Method

Fig. 2: Framework of a recurrent block in DuDoRNet+. It is constructed by a
domain-specific Shallow Feature Extraction (X-SFE), Global Feature
Refinement(GFR) and 4-stage domain-specific building blocks(X-$S_{i}$).
Building blocks’ color follow the notation in the color table. Convolution is
followed by ReLU unless noted.

We choose a SoTA dual-domain MRI reconstruction model DuDoRNet [14] for
evaluating our domain-specific designs. Our designs improve its dual domain
recurrent blocks by customizing global and local structures based on domain-
specific properties. Our model, DuDoRNet+, is illustrated in Figure 2,
including domain-specific Shallow Feature Extraction (X-SFE), Global Feature
Refinement(GFR), and 4-stage domain-specific building blocks (X-$S_{i}$) as
the backbone. Global residual learning and global feature fusion in DuDoRNet
are preserved in ours.

The overall pipeline goes as follows:

$F_{-1}=Conv^{3}(x_{u}),F_{0}^{C}=Conv^{3}(F_{-1}).$ (5)

where $Conv^{k}$ denotes a convolution operation with kernel size $k*k$ and
$F_{-1}$ denotes the first extracted feature used for global residual
learning. $F_{0}^{C}$ denotes the intermediate extracted feature by
convolution. The second extracted feature $F_{0}$ is the input of X-$S_{i}$.
For image domain, $F_{0}=F_{0}^{C}$.

For K-space, a Global Initiation Module (K-GLIM) is introduced based on the
idea of a sketchy K-space initiation:

$F_{0}=P_{K-GLIM}(F_{0}^{C})=P_{K-GLIM}(Conv^{3}(F_{-1})).$ (6)

Global Feature Fusion (GFF) fuses features from X-BB’s and is input for global
residual learning:

$F_{i}=P_{X-S_{i}}(F_{i-1}),$ (7)
$x_{r}=P_{GFR}(F_{-1}+P_{GFF}(concat(F_{1},F_{2},F_{3},F_{4}))),$ (8)

where GFF consists of 1x1 and 3x3 convolutions and GFR consists of two 3x3
convolutions, creating the refined reconstructed image/K-space $x_{r}$.

####  2.3.1 K-GLIM: K-space global initiation

We propose K-space global initialization module (K-GLIM) placed at the
beginning of the K-space network. K-GLIM is applied on the transformed
features of two convolution layers in SFE, providing channel-wise interaction
and a global view.

The main part of K-GLIM is a Channel-wise Multi-head Self-attention (C-MSA).
Instead of performing pixel-level or patch-level attention in a conventional
spatial attention way, C-MSA is performed on the transpose of pixel-level
tokens. Similar to MSA, C-MSA is an extension of Self-Attention (C-SA) in
which $h$ times SA operations are done. C-SA can be expressed as

$\boldsymbol{z}_{j}^{c}=\sum_{i}\operatorname{Softmax}\left(\frac{\boldsymbol{Q^{T}K}}{\sqrt{d}}\right)_{i}\boldsymbol{V}_{i,j}^{T},$
(9)

and its computational complexity is $O(6(hw)C^{2})$, linear to $(hw)$. C-MSA
naturally captures global information and interactions for visual recognition
tasks and complements (windowed-)spatial attention. Compared with channel-wise
convolution, it is data-specific and fine-grained.

####  2.3.2 I-PLDE: Image parallel local detail enhancement

Recovering high-quality images from undersampled MRI measurements can be
divided into two sub-tasks in the image domain: artifact removal and detail
refinement. The former requires large receptive field due to the global
property of signal aliasing, yet the latter task is much more a local task.
Since the receptive field of a network enlarges with stacked convolutions, the
detail refinement ability is possibly harmed with features propagating.

To this end, we propose the I-PLDE module, a parallel branch emphasizing local
detail, inspired by the “divide-and-conquer” idea in [20, 21]: modeling
locality and long-range dependencies in parallel and fusing the features to
account for both. Depthwise convolution .Different from ViTAE, we adopt stage-
level parallel connection and use sequential dilated convolution (DRDB) for
pyramid feature extraction instead of multiple parallel dilated convolution of
different dilation rates. This design is customized to tiny models since a
wide network design leads to a less stable training process and lower feature
representation ability. I-PLDE consists of a 1x1 convolution to match the
hidden dimension with its parallel branch, three stacked depth-wise
convolution layers, and a window embedding operation. SiLU is used for non-
linear activation following the convention in [20, 21]. The output of I-PLDE
$F_{i}^{CE}$ is added with DRDN’s output. ViTAE’s design is not successful in
our scenarios since the tiniest model it constructs has about 5 times the
parameters of ours and different availability of large datasets.

##  3 Experiment

###  3.1 Dataset and training

Our evaluation is carried out on the Multi-Contrast IXI
dataset111https://brain-development.org/ixi-dataset/, CC BY-SA 3.0 license. We
use all 575 subjects with paired T2-PD and uniformly sample 14 slices from
each subject volume. We split the dataset patient-wise into training,
validation and testing set with a ratio of 7 : 1 : 2, corresponding to 5628
training images, 812 validation images, and 1610 test images each protocol.
Images are center-cropped from 256x256 to 224x224. Code is written in Pytorch
and experiments are performed using an NVIDIA GeForce RTX 3090. As for the
rest, we follow the same experiment settings in DuDoRNet [14].

###  3.2 Performance evaluation

We compare our methods with other baseline deep learning methods in three
conventions of MRI reconstruction: image-domain [22, 14, 23, 15, 3], dual-
domain [22, 14, 4] and reference-protocol-guided dual-domain reconstruction
[24, 23, 14, 4, 7]. Considering backbone design, [23, 24] adopt Dense-Unet;
[4, 7, 3] share similar Swin-Transformer backbones derived from SwinIR [16].
All models besides UNet [22] have $\approx$ 1M parameters and UNet has 2M. All
models are recurred twice and a DC is added at the end of each recurrent
block. The recurrent time of local residual blocks in OUCR [15] is set to 5,
complying with their default setting; yet DCs at the end of their local
recurrent structure are discarded while those at the end of their global
recurrent structure are preserved for fairness. All models are trained for 100
epochs and the hyper-parameters of each method are tuned on the validation set
with test data held out for final evaluation. We consider 1D Cartesian
sampling pattern with an acceleration rate ranging from 4 to 8; the center
sampling fraction $R_{acs}$ is set to 0.125. Peak signal-to-noise ratio (PSNR)
and structural similarity index (SSIM) are used as the quantitative evaluation
metrics.

###  3.3 Results on undersampled MRI reconstruction

Acceleration | 4x |  |  | 8x |  |
---|---|---|---|---|---|---
| PSNR | SSIM | MSE | PSNR | SSIM | MSE
Zero Padding | 25.17±1.80 | 80.20±6.34 | 223.51 | 24.16±1.79 | 79.90±6.37 | 282.28
Unet | 31.73±1.98 | 95.48±2.12 | 53.03 | 26.98±2.01 | 91.43±3.75 | 157.41
Dense-Unet | 32.49±2.23 | 96.47±1.79 | 46.10 | 27.70±1.97 | 92.16±3.50 | 132.27
OUCR | 32.52±2.26 | 96.57±1.76 | 45.90 | 27.80±2.00 | 92.33±3.44 | 129.40
SwinIR | 32.75±2.27 | 96.69±1.71 | 43.60 | 27.78±2.08 | 92.49±3.45 | 130.76
DuDoRNet_I | 32.95±2.28 | 96.75±1.69 | 41.82 | 28.00±2.04 | 92.76±3.32 | 123.94
Ours_I | 33.01±1.98 | 96.80±1.59 | 40.36 | 28.02±2.04 | 92.80±3.36 | 122.78
Dual-DenseUnet | 32.76±2.22 | 96.67±1.65 | 43.21 | 27.68±1.93 | 91.99±3.45 | 131.75
Dual-SwinIR | 33.37±2.44 | 97.09±1.57 | 38.86 | 27.73±2.02 | 92.16±3.50 | 131.70
DuDoRNet(w/ ref) | 33.00±2.31 | 96.81±1.63 | 41.46 | 27.84±2.05 | 92.24±3.46 | 128.73
Ours (w/ ref) | 33.43±2.40 | 97.02±1.59 | 35.84 | 28.13±2.10 | 92.68±3.41 | 125.27
DuDoRNet | 40.45±2.59 | 99.26±0.51 | 7.86 | 38.38±2.52 | 98.93±0.72 | 12.52
Ours | 40.59±2.55 | 99.29±0.50 | 7.61 | 38.46±2.52 | 98.96±0.69 | 12.00

Table 1: Quantitative comparison (PSNR(dB), SSIM(%), MSE(*1e-5)) with baseline
methods on the IXI-dataset. The first, second and third part correspond to
image-domain reconstruction of PD, dual-domain reconstruction of PD and dual-
domain reconstruction of PD with a reference protocol T2. The best results are
marked as red.

In Table 1, we demonstrated PD reconstruction evaluations using ×4, ×8
acceleration in three common approaches of MRI reconstruction: image-domain,
dual-domain, and T2-guided dual-domain MRI reconstruction. The best results
under the same setting and acceleration rate are colored with red. Our method
achieves the best performances under all settings and acceleration rates.

###  3.4 Ablation Study

We examine I-PLDE and K-GLIM on undersampled PD reconstruction without
reference guidance. For each component, we apply it to image-domain and dual-
domain subsequently. If there is a performance drop in either way, we apply it
on K-space only to observe the performance. The reason for this design is that
we weigh the utility of the image recovery network and the synergy between
image and K-space reconstruction networks over K-space reconstruction since
the image recovery network alone gives better results than K-space recovery
network thus a default choice when reconstructing in a single domain. As shown
in Table 2, applying them to the other domain leads to performance drops.

4x PD |  |  | Domain(s) to put the module |  |
---|---|---|---|---|---
| Modules |  | Image | K-space | Both
| K-GLIM | I-LDE | PSNR, SSIM | PSNR,SSIM | PSNR, SSIM
DuDoRNet |  |  | - | - | 33.00, 96.81
DuDoRNet w/ | ✓ |  | 32.89, 96.57 | 33.37, 96.87 | 33.34, 96.84
DuDoRNet w/ |  | ✓ | 33.16,96.83 | 32.76, 96.79 | 33.11, 96.83
Ours | ✓ | ✓ | - | - | 33.43,97.02

Table 2: Domain-wise quantitative evaluation (PSNR, SSIM(%)) of hybrid
structure and domain-specific modules. The best results are marked as red.

##  4 Conclusion

We rethink dual-domain MRI reconstruction from the perspective of the
receptive field and propose two domain-specific modules. By introducing
domain-specific designs to a SoTA dual-domain MRI reconstruction model
DuDoRNet, our model DuDoRNet+ surpasses popular deep learning methods in three
common settings of MRI reconstruction. In the future, we will further validate
our domain-specific design by transforming other SoTA models. Future work also
includes extending our model from single-coil to multi-coil reconstruction and
validating our model’s utility under different sampling patterns.

## References

  * [1] Shanshan Wang, Zhenghang Su, Leslie Ying, Xi Peng, Shun Zhu, Feng Liang, Dagan Feng, and Dong Liang,  “Accelerating magnetic resonance imaging via deep learning,”  in ISBI, 2016.
  * [2] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.,  “An image is worth 16x16 words: Transformers for image recognition at scale,”  in ICLR, 2021.
  * [3] Jiahao Huang, Yingying Fang, Yinzhe Wu, Huanjun Wu, Zhifan Gao, Yang Li, Javier Del Ser, Jun Xia, and Guang Yang,  “Swin transformer for fast mri,”  Neurocomputing, vol. 493, pp. 281–304, 2022.
  * [4] Bo Zhou, Neel Dey, Jo Schlemper, Seyed Sadegh Mohseni Salehi, Chi Liu, James S Duncan, and Michal Sofka,  “Dsformer: a dual-domain self-supervised transformer for accelerated multi-contrast mri reconstruction,”  in WACV, 2023.
  * [5] Pengfei Guo, Yiqun Mei, Jinyuan Zhou, Shanshan Jiang, and Vishal M Patel,  “Reconformer: Accelerated mri reconstruction using recurrent transformer,”  arXiv preprint arXiv:2201.09376, 2022.
  * [6] Chun-Mei Feng, Yunlu Yan, Huazhu Fu, Li Chen, and Yong Xu,  “Task transformer network for joint mri reconstruction and super-resolution,”  in MICCAI, 2021.
  * [7] Jun Lyu, Bin Sui, Chengyan Wang, Yapeng Tian, Qi Dou, and Jing Qin,  “Dudocaf: Dual-domain cross-attention fusion with recurrent transformer for fast multi-contrast mr imaging,”  in MICCAI, 2022.
  * [8] Guang Yang, Simiao Yu, Hao Dong, Greg Slabaugh, Pier Luigi Dragotti, Xujiong Ye, Fangde Liu, Simon Arridge, Jennifer Keegan, Yike Guo, et al.,  “Dagan: deep de-aliasing generative adversarial networks for fast compressed sensing mri reconstruction,”  IEEE transactions on medical imaging, vol. 37, no. 6, pp. 1310–1321, 2017.
  * [9] Kyong Hwan Jin, Michael T McCann, Emmanuel Froustey, and Michael Unser,  “Deep convolutional neural network for inverse problems in imaging,”  IEEE Transactions on Image Processing, vol. 26, no. 9, pp. 4509–4522, 2017.
  * [10] Tran Minh Quan, Thanh Nguyen-Duc, and Won-Ki Jeong,  “Compressed sensing mri reconstruction using a generative adversarial network with a cyclic loss,”  IEEE transactions on medical imaging, vol. 37, no. 6, pp. 1488–1497, 2018.
  * [11] Jo Schlemper, Jose Caballero, Joseph V Hajnal, Anthony Price, and Daniel Rueckert,  “A deep cascade of convolutional neural networks for mr image reconstruction,”  in IPMI, 2017.
  * [12] Chen Qin, Jo Schlemper, Jose Caballero, Anthony N Price, Joseph V Hajnal, and Daniel Rueckert,  “Convolutional recurrent neural networks for dynamic mr image reconstruction,”  IEEE transactions on medical imaging, vol. 38, no. 1, pp. 280–290, 2018.
  * [13] Taejoon Eo, Yohan Jun, Taeseong Kim, Jinseong Jang, Ho-Joon Lee, and Dosik Hwang,  “Kiki-net: cross-domain convolutional neural networks for reconstructing undersampled magnetic resonance images,”  Magnetic resonance in medicine, vol. 80, no. 5, pp. 2188–2201, 2018.
  * [14] Bo Zhou and S Kevin Zhou,  “Dudornet: learning a dual-domain recurrent network for fast mri reconstruction with deep t1 prior,”  in CVPR, 2020.
  * [15] Pengfei Guo, Jeya Maria Jose Valanarasu, Puyang Wang, Jinyuan Zhou, Shanshan Jiang, and Vishal M Patel,  “Over-and-under complete convolutional rnn for mri reconstruction,”  in MICCAI, 2021.
  * [16] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte,  “Swinir: Image restoration using swin transformer,”  in WACV, 2021.
  * [17] Eric Z Chen, Terrence Chen, and Shanhui Sun,  “Mri image reconstruction via learning optimization using neural odes,”  in MICCAI 2020, 2020.
  * [18] Dongwook Lee, Jaejun Yoo, Sungho Tak, and Jong Chul Ye,  “Deep residual learning for accelerated mri using magnitude and phase networks,”  IEEE Transactions on Biomedical Engineering, vol. 65, no. 9, pp. 1985–1995, 2018.
  * [19] Mark A Griswold, Peter M Jakob, Robin M Heidemann, Mathias Nittka, Vladimir Jellus, Jianmin Wang, Berthold Kiefer, and Axel Haase,  “Generalized autocalibrating partially parallel acquisitions (grappa),”  Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine, vol. 47, no. 6, pp. 1202–1210, 2002.
  * [20] Yufei Xu, Qiming Zhang, Jing Zhang, and Dacheng Tao,  “Vitae: Vision transformer advanced by exploring intrinsic inductive bias,”  NeurIPS, 2021.
  * [21] Qiming Zhang, Yufei Xu, Jing Zhang, and Dacheng Tao,  “Vitaev2: Vision transformer advanced by exploring inductive bias for image recognition and beyond,”  IJCV, 2023.
  * [22] Olaf Ronneberger, Philipp Fischer, and Thomas Brox,  “U-net: Convolutional networks for biomedical image segmentation,”  in MICCAI, 2015.
  * [23] Chun-Mei Feng, Yunlu Yan, Geng Chen, Yong Xu, Ying Hu, Ling Shao, and Huazhu Fu,  “Multi-modal transformer for accelerated mr imaging,”  IEEE Transactions on Medical Imaging, 2022.
  * [24] Lei Xiang, Yong Chen, Weitang Chang, Yiqiang Zhan, Weili Lin, Qian Wang, and Dinggang Shen,  “Ultra-fast t2-weighted mr reconstruction using complementary t1-weighted information,”  in MICCAI 2018, 2018.