# Enhancing Question Answering on Charts Through Effective Pre-training Tasks

Ashim Gupta1, $\dagger$, , Vivek Gupta2, Shuo Zhang3,
Yujie He3, Ning Zhang3, Shalin Shah3, $\ddagger$
1University of Utah, Salt lake, UT
2University of Pennsylvania, Philadelphia, PA
3Bloomberg, New York, NY
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Work done during summer internship at Bloomberg

###### Abstract

To completely understand a document, the use of textual information is not
enough. Understanding visual cues, such as layouts and charts, is also
required. While the current state-of-the-art approaches for document
understanding (both OCR-based and OCR-free) work well, a thorough analysis of
their capabilities and limitations has not yet been performed. Therefore, in
this work, we addresses the limitation of current VisualQA models when applied
to charts and plots. To investigate shortcomings of the state-of-the-art
models, we conduct a comprehensive behavioral analysis, using ChartQA as a
case study. Our findings indicate that existing models particularly
underperform in answering questions related to the chart’s structural and
visual context, as well as numerical information. To address these issues, we
propose three simple pre-training tasks that enforce the existing model in
terms of both structural-visual knowledge, as well as its understanding of
numerical questions. We evaluate our pre-trained model (called MatCha-v2) on
three chart datasets - both extractive and abstractive question datasets - and
observe that it achieves an average improvement of 1.7% over the baseline
model.

Enhancing Question Answering on Charts Through Effective Pre-training Tasks

Ashim Gupta1, $\dagger$,††thanks:  Work done during summer internship at
Bloomberg , Vivek Gupta2, Shuo Zhang3, Yujie He3, Ning Zhang3, Shalin Shah3,
$\ddagger$ 1University of Utah, Salt lake, UT 2University of Pennsylvania,
Philadelphia, PA 3Bloomberg, New York, NY<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Understanding and extracting insights from charts and plots is a fundamental
aspect of data analysis that is critical for various domains, including
finance, healthcare, and scientific research. To bridge the gap between raw
data and actionable knowledge, question answering (QA) systems tailored for
charts and plots have gained increasing attention in recent years. These
systems aim to enable users to pose natural language questions about the
content of visual data representations, such as bar graphs, line charts, and
scatterplots, and receive informative answers. However, despite the remarkable
progress made in developing such QA systems, there remains a significant
challenge: their performance often falls short when subjected to human-
generated questions. This discrepancy between machine performance and human
expectations underscores the need for a comprehensive investigation into the
limitations of existing models and the development of strategies to address
their shortcomings.

To shed light on the shortcomings of current chart-based QA systems (Masry et
al., 2022), this paper first undertakes a detailed checklist-based behavioral
analysis (Ribeiro et al., 2020; Bhatt et al., 2021; Rogers et al., 2021).
Choosing the models trained on the ChartQA dataset (Masry et al., 2022) for
the representative case study, we systematically evaluate model responses
against examples constructed from the checklist of expected behaviors,
allowing us to pinpoint the areas in which these models falter. The checklist
is designed to assess various dimensions of chart-based question answering,
including the ability to interpret the structural and visual context of the
chart, handle questions requiring numerical reasoning, and offer meaningful
insights that align with human expectations. Concretely, our analysis reveals
that current state-of-the-art models perform poorly on two types of questions.
The first type of questions are those that pertain to the visual aspects of a
chart (e.g., color), while the second type are questions that require
application of numerical operators to numerical items present in the chart
(e.g., average, etc.).

To address these two shortcomings, we propose a set of three pre-training
tasks: Visual-Structure prediction, Summary Statistics prediction, and
Numerical operator prediction. Through evaluation on three chart question
answering datasets, we find that models fine-tuned after using this pre-
training outperform the baseline model by more than 1.7 percentage points in
an absolute sense.

In summary, our contributions are: 1) We perform a checklist-based behavioral
analysis of the current state-of-the-art chart question answering systems to
identify issues and challenges faced by such systems. 2) We propose three
simple, yet effective, pre-training tasks to address these issues. The new
pre-trained model outperforms the baseline systems significantly.

##  2 Behavioral Analysis via Checklist

We first describe the procedure for constructing the checklist for chart-based
question answering systems and then discuss the results and observations.

###  2.1 Checklist For ChartQA

To perform the perturbation analysis for the chart- based models, we choose
the popular ChartQA dataset, where the objective is to answer questions based
on the information provided in the accompanying charts. This real-world
dataset consists of four (4) different subsets: (a) Statista-H, (b) Pew, (c)
Our World In Data (OWID), and (d) OECD. However, in all our checklist analysis
and evaluation, we only use one subset of data, namely OWID, as the library
(owid-grapher) to generate the charts and apply perturbations is
available111https://github.com/owid/owid-grapher-py. All the other data
subsets were either manually curated or hard-coded and hence cannot be
perturbed. To construct the checklist for behavioral analysis, we leverage 400
different charts and the corresponding data points sourced from Our World In
Data. To ensure the accuracy and reliability of our analysis, we manually
design three distinct types of templates: Structural & Visual, Data
Extraction, and Numerical QA.

#### Structural and Visual

The Structural & Visual templates are crafted to assess the model’s
understanding of chart structures and visual elements. For instance, we
evaluate whether the model can discern between different types of charts, such
as bar charts or line charts, and if it can recognize various colors used in
the charts.

#### Data Extraction

The Data Extraction templates gauge the model’s ability to accurately retrieve
data values from the charts.

#### Numerical QA

Last, the Numerical QA templates are tailored to assess the model’s
proficiency in answering both simple and complex numerical questions related
to the data points present in the charts.

In total, we have 33 distinct QA templates belonging to these three classes. A
few examples are shown in Table 1 and detailed examples are presented in the
Appendix in Table LABEL:tab:app:failure.

###  2.2 Evaluation and Results

As mentioned earlier, our focus is to perform behavioral analysis using the
checklist for the chart-based models on the ChartQA dataset to first under-
stand their extent of chart understanding and then pinpoint areas where these
models require improvement.

#### Models Evaluated

Our focus in this work are the two large, recently introduced chart pre-
trained models in MatCha, and DePlot + LLM. While MatCha is an end-to-end
chart-to-text pre-trained model, DePlot + LLM is a pipelined approach where
DePlot first converts an input chart into its textual representation in the
form of a table and then performs few-shot question answering via a Large
Language Model (LLM). In our experiments, we use the flan-ul2 (Tay et al.,
2022) with 20 billion parameters as the LLM for DePlot + LLM evaluation. We do
not evaluate models such as VisionTapas (Masry et al., 2022), Vl-T5 (Cho et
al., 2021) since they are harder to work with than the two models we use.

#### Evaluation Metric

For each of the templates we use, we measure the model’s _Failure Rate_ , i.e.
the number of examples where model prediction does not match the expected/gold
output. We present examples of some failure cases along with failure rates in
Table 1.

#### Results

The results are shown in Table 1. Notably, the Structural & Visual templates
exhibit alarmingly high failure rates, prompting an in-depth investigation
into the underlying causes. One plausible explanation for these high failure
rates is the absence of explicit enforcement of structural and visual
information in the pre-training tasks for models like MatCha and DePlot. This
absence underscores a critical gap in the models’ understanding of fundamental
chart structures and visual elements, posing a significant challenge in their
interpretation of complex data visualizations.

In stark contrast, the models demonstrate a significantly higher proficiency
in handling templates focused on data extraction. The lower failure rates
observed in this category highlight the models’ capability to accurately
extract data points from diverse charts, indicating a relatively robust
performance in tasks requiring precise information retrieval.

We also notice intriguing disparities in the models’ performance concerning
numerical QA templates. While the failure rates soar for complex mathematical
operations, such as intricate calculations involving multiple operators, a
marked improvement is observed in tasks involving simpler operations like
finding maximum or minimum values. This nuanced discrepancy suggests that
while the models struggle with advanced numerical reasoning, they exhibit a
more stable grasp on elementary mathematical concepts. This insight not only
sheds light on the specific challenges faced by these models in handling
complex mathematical operations but also underscores their potential strengths
in addressing simpler, more straightforward numerical queries.

##  3 Experiments

Figure 1: An unperturbed chart without any modifications from the ChartQA
dataset. The chart comes from the OWID website. Our aim is to perturb these
charts and evaluate models using our proposed checklist.

Figure 2: A perturbed chart where the bars are sorted in descending order. We
find that the models (especially MatCha) are sensitive to the order in which
the bars appear.

Figure 3: A perturbed chart where a bar unrelated to the question is added.
Adding an unrelated entity/bar to the chart reduces performance.

| Chart Capability
---
& Template Description
Failure Rate (%)
|  | MatCha | DePlot | MatCha-v2 | DePlot-v2
Structural & Visual |  |  Colors in Chart: Is a certain color present or absent?
---
| 98.9
---
99.5 | 15.6 | 23.5
Chart Type: Is it bar plot or a line plot? | 99.4 | 74.9 | 2.2 | 8.4
Data Extraction | Extract Entity Name from Original Chart (a) | 33.9 | 1.6 | 31.2 | 1.8
Extract Entity Value from Original Chart (a) | 63.5 | 0.8 | 25.6 | 1.1
Extract Value from Perturbed Chart - Sort Descending Order (b) | 13.3 | 1.6 | 12.4 | 1.5
Extract Value from Perturbed Chart - Add Irrelevant Bar (c) | 35.0 | 1.1 | 31.2 | 1.1
Numeric Q/A |  Operator: Sum | 99.5 | 45.8 | 62.3 | 44.5
Operator: ArgMax | 16.3 | 15.2 | 11.5 | 16.1
Operator: Average + Comparison | 83.4 | 65.2 | 47.5 | 61.0

Table 1: A partial selection of Template based tests for the ChartQA using our
checklist. We report Failure Rate (in %) for each of the templates. The
proposed v2 versions significantly decrease the failure rate. Please refer to
Table LABEL:tab:app:failure in the Appendix for the examples of each type of
template.

###  3.1 Proposed Pre-training Tasks

As discussed earlier, we proposed a comprehensive approach to address the
challenges identified through checklist-based analysis. In this section, we
introduce three distinct pre-training tasks designed to enhance the MatCha
model’s chart understanding capabilities.

#### Visual Structure Prediction

The first task, termed Visual Structure Prediction, demands the model predict
intricate details of input charts, encompassing chart types (such as bar,
line, etc.), colors associated with chart entries, and even chart titles.

#### Summary Statistics Prediction

The second task, Statistics Prediction, focuses on refining numerical
question-answering by training the model to predict summary statistics like
mean, maximum, and minimum values from the chart data.

#### Numerical Comparison

Finally, the numerical comparison task requires the model to compare values
from different chart entries and predict relationships such as greater than,
smaller than or equal to. By engaging in these tasks, the MatCha model
undergoes targeted pre-training to bolster its chart comprehension abilities,
paving the way for more accurate and insightful responses in question-
answering scenarios.

#### Extension to DePlot

Since DePlot is a chart-to-table generation model, we only pre-train the
DePlot model on the first two pre-training tasks of Visual Structure
Prediction and Summary Statistics Prediction.

#### Pre-training Details

We pre-train both models on the charts extracted from the training data of the
ChartQA dataset. We pre-train each model for only one epoch as we saw
significant reduction in accuracy on the validation set. In addition, we use
batch size of six (6) and use all the other hyperparameters as suggested by
(Liu et al., 2023). We denote our proposed versions of MatCha and DePlot with
the v2 suffix.

#### Computing Infrastructure Used

All of our experiments required access to GPU accelerators. We ran our
experiments on two machines: NVIDIA Tesla V100 (16 GB VRAM) and Tesla V100 (32
GB VRAM).

#### Improved QA on Checklist

We first measure the effectiveness of these two pre-training tasks on the
proposed checklist. As shown in Table 1, the proposed variations of the two
models (called v2 versions) significantly decrease failure rates across all
template and question types.

###  3.2 Chart Question Answering Evaluation

To assess the effectiveness of our proposed pre-training tasks on actual
question answering tasks, we conduct experiments on the ChartQA dataset.
Furthermore, we extend our evaluation to a different question-answering
dataset named PlotQA, wherein charts and associated questions are derived from
disparate sources. This cross-dataset evaluation enables us to gauge the
model’s adaptability and generalizability beyond its original training data.

Both ChartQA and PlotQA are extractive question-answering datasets, where an
answer is retrieved by combining entries from the chart. In addition, we
explore the model’s performance in abstractive question-answering, a more
complex task necessitating detailed descriptive responses, using the OpenCQA
dataset. Notably, both ChartQA and PlotQA focus on extractive question-
answering, demanding precise extraction of information from the source data,
while OpenCQA necessitates the generation of more extensive, contextually rich
answers. Through these evaluations, we gain a holistic understanding of the
MatCha model’s capabilities, from basic chart comprehension to nuanced and
elaborate question answering across diverse datasets and question types.

###  3.3 Evaluation Metrics and Datasets

#### Datasets

We evaluate the effectiveness of our proposed pre-training methods on the
three chart related datasets. Here is a brief summary of each dataset along
with how we use it to evaluate a model’s chart understanding capabilities:

Chart QA The Chart Question Answering (QA) (Masry et al., 2022) dataset, as
the name suggests, is a natural language query and answer generation dataset
with one key difference from most NLP datasets – it also provides the visual
representation of data or the chart image which contain richer information
such as layout, colors, etc. It contains approximately 28,000 training
examples and comes with two evaluations sets: Augmented and Human. The
Augmented set was constructed using a question generation system, while the
Human set was made entirely from human annotations.

Plot QA The Plot Question Answering (QA) dataset (Methani et al., 2020) is
also a visual question answering dataset based on real-world scientific charts
and question-answer pairs collection from crowd-sourced templates. This
dataset has 80% QA pairs whose answer is either not present in the chart or
not in vocabulary, which means it contains a nice breath of data variability.
PlotQA contains approximately 5 million training examples. To reduce the
computational burden, we sample approximately 25,000 of those randomly sampled
examples for our evaluation.

OpenCQA The Open Chart Question Answering (CQA) dataset (Kantharaj et al.,
2022) is designed specifically for open-ended question answering on charts.
These questions span from summarizing the trend observed in the chart to
describing and comparing a certain attribute over the period considered in the
chart. OpenCQA contains approximately 7,200 training examples.

#### Evaluation Metrics

For ChartQA and PlotQA, we follow the original authors and use relaxed
accuracy (correct answer within a tolerance of 5%), while we use ROUGE metrics
for OpenCQA, as it is a generative text-heavy dataset.

Model | ChartQA | PlotQA | OpenCQA
---|---|---|---
| Aug | Human |  |
MatCha | 77.0 | 28.7 | 52.0 | 29.19
DePlot + Flan UL2 | 69.4 | 22.4 | 50.2 | 36.52
MatCha-v2 (Ours) | 78.3 | 30.1 | 55.8 | 29.6
DePlot-v2 + Flan UL2 (Ours) | 71.5 | 24.2 | 51.1 | 35.47

Table 2: Evaluation Results for the proposed pre-training methods. We measure
accuracies for the ChartQA and PlotQA datasets, while ROUGE is used for the
OpenCQA dataset. As can be observed, our proposed pre-training methods
significantly improves MatCha model (called MatCha-v2) across all three
datasets. We highlight the entries where proposed variation provides the
improvement over the baseline.

###  3.4 Results with Pre-training

As can be observed from Table 2, we see consistent improvements for the MatCha
model across all three evaluation datasets. Perhaps surprisingly, we observe a
much larger improvement for PlotQA than for the ChartQA dataset from where
pre-training data was used. We also gain improvements for the OpenCQA dataset,
which requires long-form question answering. The improvement for DePlot-v2 is
smaller than that for MatCha-v2, as the component responsible for question
answering is the LLM that remains unchanged.

Since our pre-training procedure uses charts sourced from the ChartQA dataset,
the evaluation on two other datasets forms an out-of-domain evaluation.
Particularly on PlotQA, where charts are from different sources than those
used in ChartQA, we see more significant improvements than ChartQA. Finally,
although performance improvements are less significant on the OpenCQA dataset,
this is not entirely unexpected, as the checklist we devised was for
extractive QA. As such, the pre-training tasks motivated from those results
are also more suited for extractive QA.

##  4 Conclusion

In this work, we present a detailed study to evaluate the chart understanding
capabilities of two current state-of-the-art QA models. We propose a detailed
checklist that can be used to high- light the current shortcomings of these
models. Broadly, we evaluate end-to-end QA models like MatCha and pipeline
based models like DePlot + LLM. These help us identify avenues of improvement.
Using them, we show that adding relevant pre-training tasks improves the
performance of the model to achieve performance improvements across three
datasets. Across the three tasks and datasets we consider, our pre-training
methods help MatCha achieve an average improvement of 1.7% points.

## References

  * Bhatt et al. (2021) Shaily Bhatt, Rahul Jain, Sandipan Dandapat, and Sunayana Sitaram. 2021.  A case study of efficacy and challenges in practical human-in-loop evaluation of NLP systems using checklist.  In _Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval)_ , pages 120–130, Online. Association for Computational Linguistics.
  * Cho et al. (2021) Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. 2021.  Unifying vision-and-language tasks via text generation.  In _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 1931–1942. PMLR.
  * Kantharaj et al. (2022) Shankar Kantharaj, Xuan Long Do, Rixie Tiffany Leong, Jia Qing Tan, Enamul Hoque, and Shafiq Joty. 2022.  OpenCQA: Open-ended question answering with charts.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11817–11837, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Liu et al. (2023) Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, and Julian Eisenschlos. 2023.  MatCha: Enhancing visual language pretraining with math reasoning and chart derendering.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 12756–12770, Toronto, Canada. Association for Computational Linguistics.
  * Masry et al. (2022) Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022.  ChartQA: A benchmark for question answering about charts with visual and logical reasoning.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 2263–2279, Dublin, Ireland. Association for Computational Linguistics.
  * Methani et al. (2020) Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. 2020.  Plotqa: Reasoning over scientific plots.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 1527–1536.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32.
  * Ribeiro et al. (2020) Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020.  Beyond accuracy: Behavioral testing of NLP models with CheckList.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4902–4912, Online. Association for Computational Linguistics.
  * Rogers et al. (2021) Anna Rogers, Timothy Baldwin, and Kobi Leins. 2021.  ‘just what do you think you’re doing, dave?’ a checklist for responsible data use in NLP.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 4821–4833, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Tay et al. (2022) Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. 2022.  Unifying language learning paradigms.  _arXiv preprint arXiv:2205.05131_.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_.

##  Appendix A Appendix

###  A.1 Implementation Details

We use batch size of six (6) to train the MatCha models due to computational
constraints. Additionally, all the models that require training (e.g., MatCha)
were trained up to five epochs due to computational constraints.

We did not experiment with VisionTapas (Masry et al. (2022)) as we could not
run the publicly released implementation due to a missing dependency. We
created a github issue that was not addressed 222https://github.com/vis-
nlp/ChartQA/issues/9.

We train all our models using the Transformers library from Hugging Face (Wolf
et al., 2019) with the PyTorch back-end (Paszke et al., 2019).

[floatPos=p,capWidth=1,capPos=t,rotAngle=90,objectPos=l]table  Chart
Capability & Template Description Failure Rate (%) Example with Prediction (P)
and Gold (G) MatCha DePlot MatCha-v2 DePlot-v2 Structural & Visual Colors in
Chart: Is a certain color present or absent? 98.9 99.5 15.6 23.5 Q: Is there a
bar of orange color in the given chart? P: yes, G yes Q: Is there a bar of
blue color in the given chart? P: yes, G: no Chart Type: Is it bar plot or a
line plot? 99.4 74.9 2.2 8.4 Q: Is it a bar plot or line plot? P: line plot,
G: bar Data Extraction Extract Entity Name from Original Chart (a) 33.9 1.6
31.2 1.8 Q: Which country has wasting percentage as 9.8? P: Fiji, G: Fiji
Extract Entity Value from Original Chart (a) 63.5 0.8 25.6 1.1 Q: What is the
wasting percentage of Fiji in the given chart? P: 3.8, G: 9.8 Extract Value
from Perturbed Chart 13.3 1.6 12.4 1.5 Q: What is the wasting percentage of
Fiji in the given chart? \- Sort Descending Order (b) P: 0.8, G: 9.8 Extract
Value from Perturbed Chart 35.0 1.1 31.2 1.1 Q: What is the wasting percentage
of Fiji in the given chart? \- Add Irrelevant Bar (c) P: 0.8, G: 9.8 Numerical
QA Operator: Sum 99.5 45.8 62.3 44.5 Q: What is the sum of wasting percentage
for Nigeria and Sri Lanka bars? P: 23.5, G: 35.6 Operator: ArgMax 16.3 15.2
11.5 16.1 Q: Which country has the highest wasting percentage? P:
Malnutrition, G: Lesotho Operator: Average + Comparison 83.4 65.2 47.5 61.0 Q:
How many countries have wasting percentage more than average of all the
countries? P: 2, G: 3 A partial selection of Template based tests for the
ChartQA using our checklist. We report Failure Rate (in %) for each of the
templates. The proposed v2 versions significantly decrease the failure
rate.tab:app:failure


# HyperPredict: Estimating Hyperparameter Effects for Instance-Specific
Regularization in Deformable Image Registration.

Aisha L. Shuaibu
<EMAIL_ADDRESS>
University of Sussex, Predictive Analytics Lab (PAL) Ivor J. A. Simpson
https://orcid.org/0000-1111-2222-3333<EMAIL_ADDRESS>
University of Sussex, Predictive Analytics Lab (PAL)

###### Abstract

Methods for medical image registration infer geometric transformations that
align pairs, or groups, of images by maximising an image similarity metric.
This problem is ill-posed as several solutions may have equivalent
likelihoods, also optimising purely for image similarity can yield implausible
deformable transformations. For these reasons regularization terms are
essential to obtain meaningful registration results. However, this requires
the introduction of at least one hyperparameter, often termed $\lambda$, that
serves as a trade-off between loss terms. In some approaches and situations,
the quality of the estimated transformation greatly depends on hyperparameter
choice, and different choices may be required depending on the characteristics
of the data. Analyzing the effect of these hyperparameters requires labelled
data, which is not commonly available at test-time. In this paper, we propose
a novel method for evaluating the influence of hyperparameters and
subsequently selecting an optimal value for given pair of images. Our
approach, which we call HyperPredict, implements a Multi-Layer Perceptron that
learns the effect of selecting particular hyperparameters for registering an
image pair by predicting the resulting segmentation overlap and measures of
deformation smoothness. This approach enables us to select optimal
hyperparameters at test time without requiring labelled data, removing the
need for a one-size-fits-all cross-validation approach. Furthermore, the
criteria used to define optimal hyperparameter is flexible post-training,
allowing us to efficiently choose specific properties (e.g. overlap of
specific anatomical regions of interest, smoothness/plausibility of the final
displacement field). We evaluate our proposed method on the OASIS brain MR
standard benchmark dataset using a recent deep learning approach (cLapIRN) and
an algorithmic method (Niftyreg). Our results demonstrate good performance in
predicting the effects of regularization hyperparameters and highlight the
benefits of our image-pair specific approach to hyperparameter selection.

Keywords: Deformable Image Registration, Hyperparameter Selection,
Regularization.

##  1 Introduction

Deformable image registration has been an active field of research for
decades, as it is a fundamental process utilized in various medical imaging
studies. The process involves aligning images of different modalities and time
points for comparison and measurement of changes over time. It is very common
for medical images to vary in their spatial resolution and orientation; as a
result, non-linear image registration plays a vital role in various clinical
applications such as image-guided treatment delivery, pre-operative/post-
operative assessment comparison, disease monitoring, disease diagnosis, and
population analysis (Meng et al., 2022).

In medical images, the quality of registration largely depends on two
conditions: (a) accurate alignment of anatomical structures, (b) a smooth and
plausible displacement field. Achieving a balance between these factors is
accomplished by utilizing hyperparameters that act as a trade-off between both
objectives. Hence, selecting optimal hyperparameter values is a critical
aspect when evaluating image registration methods.

The most quantitative method of selecting these hyperparameters is by
performing grid or random search on the validation data using a discrete set
of hyperparameter values. In this approach, additional data containing
anatomical annotations is employed. By computing segmentation overlap of
anatomical regions between the ground truth labels and the inferred
registration, an optimal value is selected based on the hyperparameter that
maximizes a criteria _across the entire population_. We argue that this
approach may lead to sub-optimal choice as data-specific optimal
hyperparameter differ significantly depending on various factors such as, the
morphological similarity between the images and the anatomical structure of
interest, among other factors. Furthermore, researchers may be inclined to
adopt values from existing literature that may not be suitable for their
specific dataset or registration task, equally leading to sub-optimal results.

In this paper, we present a novel method, HyperPredict, an efficient tool for
instance-specific optimal hyperparameter selection in registration tasks. Our
proposed approach enables greater flexibility at test time and can be
leveraged in scenarios where labelled scans are scarce. In summary our
contributions are as follows:

  * •

We propose a novel and efficient method of _learning_ the effect of
registration hyperparameters. This is achieved by learning the parameters of a
network that maps a set of hyperparameters and image pair to desired
evaluation metrics (derived from segmentation overlap and smoothness of
deformation field).

  * •

Our method is flexible and allows the efficient choice of optimal parameters
based on a defined criteria.

  * •

We test our proposed method using two different registration algorithms
(cLapIRN and Niftyreg), our code is publicly available at
https://github.com/aisha-lawal/hyperpredict.

The structure of the paper is as follows. Section 2 introduces deformable
image registration, it’s mathematical formulation and hyperparameters in
medical image registration tasks, Section 3 describes related work. In Section
4 we present our method. Sections 5 and 6 describe experimental results and
discuss insights. Finally, we present limitations of our method and conclude
in Sections 7 and 8 respectively.

##  2 Background

Deformable Image Registration:  Conventional registration algorithms
(Ashburner, 2007; Beg et al., 2005; Avants et al., 2008b; Modat et al., 2014)
solve the optimization problem for each volume pair by iteratively improving
estimates for the desired transformation such that the loss function is
minimized. However, solving a pairwise optimization problem is computationally
expensive and can be very slow in practical medical applications. Over the
years, deep learning image registration methods (DLIR) (Mok and Chung, 2020b,
2021a; Balakrishnan et al., 2019) have been introduced to circumvent this by
optimizing the parameters of a network, the general process aims to establish
a dense, non-linear correspondence between pairs of images, such as magnetic
resonance (MR) images or computerised tomography (CT) scans, by learning the
optimal spatial transformation between the image pair that enhances
similarity. The process of registering two images can be formulated as an
optimization problem generally expressed as:

${\phi^{*}}=\underset{\phi}{\arg\min}\mathcal{L}_{\text{sim}}(f,m\circ\phi)$
(1)

Equation 1 seeks to minimize a loss that consists of two parameters, _f_ and
_$m\circ\phi$_. Where _m_ and _f_ denote the moving and fixed image
respectively, _$\phi$_ represents the deformation field, _$\phi^{*}$_ is the
optimal registration field that maps the pixels/voxels from _m_ to _f_.

Intuitively, deformable image registration poses a significant challenge due
to its inherently ill-posed nature. Reliance solely on surrogate measures such
as image similarity can be insufficient as they do not have the ability to
differentiate between accurate and inaccurate registrations (Rohlfing, 2011).
For example, given an image pair, _m_ and _f_ , registration algorithms seek
to find the transformation that deforms the moving image, mapping the
coordinates of _m_ to _f_ ; however, ground-truth deformation fields do not
exist to serve as a reference point, and without any restrictions placed on
the transformation properties, the cost function becomes poorly conditioned.
Thus, to ensure tractability, registration algorithms employ some
regularization that imposes a constraint on the estimated deformation field -
reducing the set of possible solutions. The quality of the registration
largely depends on the regularization weight, _$\lambda$_ , that serves as a
trade-off between the quality of the registration and how smooth the
deformation field is. Thus, the cost function can be reformulated as follows:

${\phi^{*}}=\underset{\phi}{\arg\min}\mathcal{L}_{\text{sim}}(f,m\circ\phi)+{\lambda}\mathcal{L}_{\text{reg}}(\phi)$
(2)

Description of the notation is similar to Equation 1, with $\lambda$ denoting
the regularization weight, $\mathcal{L}_{\text{sim}}$ and
$\mathcal{L}_{\text{reg}}$ represent the dissimilarity and imposed
regularization function respectively. The aim is to optimize both the
registration quality, $\mathcal{L}_{\text{sim}}$, while having a smooth
deformation field, $\mathcal{L}_{\text{reg}}$. The choice of loss function for
$\mathcal{L}_{\text{sim}}$ depends on various factors including the intensity
distribution and contrast of the image. Commonly used functions are mean
squared error, mutual information (Viola and Wells III, 1997), and normalized
cross correlation (Avants et al., 2008a), which capture different aspects of
similarity. For $\mathcal{L}_{\text{reg}}$, a diverse range of regularisers
can be employed to enforce spatially smooth deformations, examples include,
linear elasticity, bending energy, and learned priors.

Evaluation Metrics: Quantifying the accuracy of non-rigid image registration
is inherently difficult. As such, no gold standard for evaluating deformable
registration exists. However, with _additional labelled data_ , we can make an
effort to quantify the quality of the registration with some degree of
confidence. A common approach as a means of evaluation is to compute the
segmentation overlap of different anatomical regions between _m_ and _f_. The
Dice score (Dice, 1945) is one way to do this. While the Dice score between
anatomical structures is a reliable proxy for evaluation, a high Dice score
does not imply a biologically plausible registration. This is because a
deformation field with overlapping voxels can still yield a high Dice score
despite potentially unrealistic deformations (Rohlfing, 2011). Therefore, in
addition to the Dice score, assessing the deformation’s diffeomorphic property
is equally important in order to preserve the topology of the features in the
transformed image. Hence, the determinant of the Jacobian $|J_{\phi}|$ serves
as a measure of smoothness of the deformation field.

The experiments presented in this paper employ the Dice score and number of
folded voxels (derived from $|J_{\phi}|$) as the desired evaluation metrics.
However, it is worth emphasizing that our methodology is not limited to these
specific choices. Our approach is flexible and can be easily extended to
accommodate a wide range of hyperparameters and evaluation metrics. Depending
on specific application or research objective, our method can be adopted to
fit the desired metrics, such as Hausdorff distance and Target Registration
Error (TRE) amongst others.

Hyperparameter Selection: Hyperparameter optimization algorithms tackle the
challenge of jointly optimizing both model hyperparameters and model weights
through a validation and training objective respectively. The simplest
approach involves treating model training as a block-box function and
employing methods such as grid search, random search, or sequential search
(Bergstra and Bengio, 2012). Other methods include manual fine-tunning and
Bayesian optimization (Bergstra et al., 2011; Turner et al., 2021; Snoek et
al., 2012; Mockus, 1998). Although effective, these methods can be inefficient
as they require repetitive optimization procedures for each hyperparameter
value. We describe existing methods in Section 3.

Hyperparameters in Registration: In non-rigid image registration, the number
of hyperparameters to be optimized depends on the specific registration
algorithm and objective. For example, cLapIRN, (Mok and Chung, 2021a) regulate
the smoothness of the deformation field using a _single_ registration
hyperparameter, denoted as $\lambda$. On the other hand, algorithms such as
ConvexAdam, (Siebert et al., 2021) and Niftyreg, (Modat et al., 2014) employ a
_set of_ hyperparameters to optimize the registration process. In Niftyreg,
spacing for spline interpolation, bending energy, and linear elasticity are
some of the regularization options to govern the diffeomorphic properties of
the deformation field. In such scenarios, especially when dealing with
algorithms with high computational complexity, employing the above methods to
tune multiple parameters becomes impractical.

Motivated by the challenge above, HyperPredict presents a more efficient
method for selecting optimal hyperparameter values. During training,
HyperPredict learns the effect of the hyperparameter on the evaluation
metrics. By utilizing a registration algorithm, the target values of both
metrics (described above) are obtained for backpropagation. At test time,
given an input {_m, f, $\lambda$_}, and without having true segmentation, the
model predicts the metrics associated with the input. Based on a specific
criteria (defined in method section), we select optimal parameter,
$\lambda^{*}$, and use that for registration.

##  3 Related Work

As described in Section 2, registration methods typically optimize a data term
and a weighted regularization term. Hence, to be able to minimize the error of
the registered results, it becomes crucial to optimize the independent
hyperparameters of the registration model. Despite the advantages that come
with DLIR methods, they still face challenges in navigating the trade-off
parameter within the objective function. Traditionally, the selection of
regularization parameter values relied on a trial and error approach
(Ashburner, 2007; Andersson et al., 2007; Rueckert et al., 1999). In this,
users manually search for parameters that yield satisfactory results for a
given dataset. This process often involves fine-tuning multiple parameters,
which can be time consuming, particularly in hierarchical registration
methods. An example is studies conducted in (Rühaak et al., 2017), their
registration method utilized four distinct parameters. To capture the effect
of each parameter, they conducted separate experiments, in each, one parameter
was varied while the other three remained fixed. This iterative process
allowed them to empirically determine an optimal value for all four
parameters. Although ad hoc parameter-tuning may produce satisfactory
outcomes, it requires expert domain knowledge to effectively guide the tuning
process (Joshi et al., 2004; Vialard et al., 2011; Ma et al., 2008; Wang et
al., 2019).

Cross-validation is another standard practice employed for selecting
registration hyperparameters (Balakrishnan et al., 2019; Mok and Chung,
2020a). Using grid search, the parameters that yield the best performance on a
given dataset is selected for subsequent registrations. As a result, it
utilizes a _fixed_ level of regularization across the entire set of image
pairs, with the assumption that all image pairs require the same degree of
regularization. Additionally, this process necessitates availability of
manually labelled dataset that accurately represents the testing data of
interest. This method of parameter selection requires substantial
computational resources and human effort, which may result in sub-optimal
parameter choices.

Employing a hierarchical Bayesian model to infer the regularization parameter
is another method demonstrated in studies by (Risholm et al., 2012) and
(Simpson et al., 2012). The general strategy of Bayesian approach leverages a
probabilistic model to explore and evaluate the performance of hyperparameters
that result in improved registration. This is achieved by characterising the
posterior distribution using techniques like Markov Chain Monte Carlo (MCMC)
or Variational Inference. (Risholm et al., 2013) considered scenarios where
the confidence of both observed data and model priors are unknown, aiming to
tackle the challenge of finding an objective trade-off between the two terms.
They characterised the posterior distribution using Metropolis-Hastings and
treated the hyperparameters as latent variables approximately marginalized
over. A similar approach is proposed in (Wang and Zhang, 2021; Zhang et al.,
2013). Given sufficient samples, MCMC yields a good characterization of the
posterior, however, the computational demands and complexity of Markov chain
is a restricting factor that limits the feasibility of this approach. While
Variational Bayes(VB) has less computational burden, it may compromise on the
quality of estimates. (Simpson et al., 2015) utilized a spatially adaptive
prior to limit unwanted regularization on the estimated transformation. One
limitation is that the uncertainty estimates quantify the variability in the
displacement field based on the inferred hyperparameters, but they do not
account for the uncertainty in the registration caused by variations in the
hyperparameters themselves (Le Folgoc et al., 2016).

Recent advancements in deep learning registration methods such as HyperMorph,
(Hoopes et al., 2021) and cLapIRN, (Mok and Chung, 2021b) propose an approach
that eliminates the need for repeatedly training models in search of
hyperparameters that boost model performance. They do this by learning the
effect of hyperparameters on the deformation field. Specifically, cLapIRN
proposes a conditional image registration method that learns the conditional
features that are strongly correlated with specific regularization values. On
the other hand, HyperMorph leverages a secondary network to generate
conditioned weights for the primary network, in order to learn the impact of
registration hyperparameters on the deformation field. To quantify the choice
of hyperparameter (selected arbitrarily), registration methods require labels.
Rather than learning the entire registration process, (Niethammer et al.,
2019), focus on learning a spatially adaptive regularizer within a
registration model to preserve the desired level of regularity, however, this
is not done on a pair-wise basis. A similar method is adapted in (Vialard and
Risser, 2014) using a learning based approach.

To summarize the challenges above; depending on the selected method, tuning
and selection of optimal hyperparameters involves dealing with one or more of
the following (a) computational expense, (b) time consuming nature of the
process, (c) requires labelled data at test time which involves human effort
and may not readily available, (d) ineffective in selecting optimal
hyperparameter (assumes a fixed value throughout the entire dataset). While
both amortized inference and algorithmic approaches to registration allow for
hyperparameter selection at test time, efficiently selecting an optimal value
at test time remains a challenge, which we aim to solve with our proposed
HyperPredict.

We emphasize that the goal of HyperPredict is not to function as a
registration tool or to directly compare it with existing ones, but rather to
serve as a means of aiding registration algorithms in selecting appropriate
hyperparameters.

##  4 Method

Current unsupervised learning-based image registration methods that learn the
effect of hyperparameters define a network $h_{\psi}$(_m_ , _f_ , _$\lambda$_)
= $\phi$, where $h_{\psi}$ is parameterized by a convolutional neural network
(CNN). Our presented approach takes advantage of these registration algorithms
but uniquely learns a function $g_{\theta}$ that captures the correlation
between a hyperparameter value and the evaluation metric for an image pair.
Given $e_{o}$(an encoded representation of _m_ and _f_) and sampled
hyperparameter value, $\lambda$ (drawn from a log-normal distribution, i.e.,
$\lambda\sim\mathrm{LogNormal}(\mu,\sigma)$), we parameterize our proposed
method as a function, $g_{\theta}(e_{o},\lambda)$ with a Multi-Layer
Perceptron (MLP). The proposed method works with any registration algorithm
that incorporates a regulariser in its smoothness during optimization.

Figure 1 presents an overview of our method. The network architecture during
training is comprised of an encoder, a registration algorithm, and the MLP. At
train time, we freeze the pretrained weights of the encoder and registration
algorithm. Based on the encoded representation and hyperparameter passed as
input, the MLP learns to predict the Dice coefficient and number of folded
voxels as the evaluation metrics. We use the deformation field, _$\phi$_ ,
from the registration algorithm and the Dice score between the warped moving
label, _s(m)_ $\circ$ $\phi$, and fixed label, _s(f)_ , to enable the
computation of the loss. At test time, given unseen images and regularization
weight, {_m, f, $\lambda$_}, we obtain the predicted metrics by evaluating
$g_{\theta}$($e_{o}$, $\lambda$). All notations are illustrated in Figure 1.

Figure 1: Overview of the method: (a) We learn the parameters $g_{\theta}$
(yellow box) of an MLP that maps the encoded representation ($e_{o}$) of an
image pair (_m, f_) and a set of hyperparameters ($\lambda$) to evaluation
metrics (Dice and folded voxels). At train time, we optimize $g_{\theta}$ by
computing the target metrics of the input (blue boxes and dashed blue lines)
using a registration algorithm. (b) At test time, given a set of input, we
predict the effect for each hyperparameter value. We select the optimal
hyperparameter, $\lambda^{*}$, and use that in the registration algorithm to
obtain a desired warped image (red box). The green shapes (encoder and
registration) are pretrained.

When inferring the optimal weight, $\lambda^{*}$ (as in Figure 1(b)), we can
impose selection criteria. For instance, we can consider only results that
restrict significant discontinuities in the deformation field. This is
achieved by constraining the percentage of number of folded voxels (nfv)
allowed. Based on this constraint, the optimal weight, $\lambda^{*}$, that
corresponds to the highest Dice score is obtained (across image pairs or
selected labels). Mathematically, this can be expressed as;

$\lambda^{*}=\underset{\lambda}{\arg\max}(\text{Dice}(\lambda)\mathds{1}[\text{nfv}<\text{N0.5\%}])$
(3)

Equation 3 determines the optimal value, ($\lambda^{*}$) by maximizing the
Dice score while ensuring %nfv is less than 0.5% of total folded voxels (N).
The derived $\lambda^{*}$ is used for registration. In the following sections,
we describe our method in detail.

###  4.1 HyperPredict Architecture

This section describes the architecture used in our experiments. Let moving
image, _m_ and fixed image, _f_ represent an image pair, each defined over a
3D spatial domain, $\Omega$ $\subset$ $\mathbb{R}^{3}$. Since CNNs capture
abstract representation of images (Szeliski, 2022), we leverage this to define
our encoder. The encoder takes _m_ and _f_ as input and outputs an encoded
representation, _$e_{o}$_ , of the image pair. For simplicity, our experiments
utilize two pre-trained registration models as encoders; Symnet, (Mok and
Chung, 2020a) and cLapIRN, (Mok and Chung, 2021a). It is important to note
that we need an encoder that could take in a pair of 3D-volumes, the
conventional approach of dealing with this is by utilizing a registration
model. We performed preliminary experiments using 2D implementation of masked
autoencoder (He et al., 2022) to learn a self-supervised encoding, we found
this very slow since they are not designed to work with 3D data.

The encoder is represented as $\mathbb{R}^{N,C_{m},D_{m},H_{m},W_{m}}\text{x}\
\mathbb{R}^{N,C_{f},D_{f},H_{f},W_{f}}\rightarrow\mathbb{R}^{N,C_{e_{o}},D_{e_{o}},H_{e_{o}},W_{e_{o}}}$.
With _N, C, D, H, W_ , indicating the batch size, channels, depth, height and
width respectively. The subscripts _m, f, $e_{o}$_, denote moving, fixed and
encoded representation respectively. Using Symnet as an encoder, the resulting
representation, _$e_{o}$_ has dimensions of 1x56x10x12x14. We flatten the last
three dimensions (D,H,W) and compute the mean across the channels. The result
is concatenated with $\lambda$ and passed as input to the MLP. We explored
other summary statistics, such as concatenating the mean, max, and min across
the channels, comparable results are presented in Appendix C.

Throughout our experiments, we parameterize the function $h_{\psi}(.,.)$ with
two different registration algorithms; cLapIRN, and Niftyreg. Both of which
accept hyperparameters as input that condition the smoothness of the
registered image. At each iteration, the same set of image pair and
hyperparameter is passed to the encoder and registration algorithm
simultaneously, the loss is computed between the predicted values from the MLP
and target values from the registration.

###  4.2 Loss Functions

In this section, we describe the loss function and evaluation metrics of the
model. We optimize the MLP parameters, $g_{\theta}$ using stochastic gradient
descent to minimize the loss. Mathematically, the objective function at each
iteration is defined as;

$\emph{$L$}=\mathcal{L}_{\text{overlap}}+\alpha\mathcal{L}_{\text{nfv}}$ (4)

$\mathcal{L}_{\text{overlap}}$ and $\mathcal{L}_{\text{nfv}}$ are the two
components of the loss _$L$_ , representing the Dice loss and nfv loss
respectively. This is a multitask learning problem, hence we define $\alpha$
as a weighting term, balancing the relative magnitude of both losses, Appendix
F shows the relative effect of training with different values of $\alpha$.
From the figure, 1.0 proves to be optimal for learning the metrics of both
cLapIRN and Niftyreg registration methods. Both components of the loss are
evaluated using the mean squared error defined in Equations 6 and 7 below.

Assessing the quality of registration is non-trivial due to competing
objectives. Low regularization parameter can allow for close matching of
appearance at the cost of anatomically-implausible and highly irregular
deformations. Similarly, high value of smoothness regularization enables
smooth deformations with sub-optimal alignment. In practice, amongst other
evaluation metrics, the Dice score and $|J_{\phi}|$ are used to access the
result of the registration. In our method, we focus on these two metrics.

Dice score: Acquiring accurate human annotated anatomical segmentations is a
tedious task, hence the annotations available are leveraged during training.
For a well aligned image, regions in _f_ and _m $\circ$ $\phi$_ corresponding
to the same anatomical region should overlap well. The Dice score quantifies
the overlap between the structures. Let _s(f) i_ and _s(m) i_ represent labels
of volumes _f_ and _m_ of structure _i_ respectively, we define the target
Dice score as;

$\text{Dice}(\emph{s(f)${}_{i}$},\emph{s(m)${}_{i}$}\circ\phi)=2*\frac{|\emph{s(f)${}_{i}$}\cap(\emph{s(m)${}_{i}$}\circ\phi)|}{|\emph{s(f)${}_{i}$}|+|\emph{s(m)${}_{i}$}\circ\phi|}=\hat{y}_{i}$
(5)

Where a score of 1 indicates a perfect match between anatomical labels, and 0
means there is no overlap. Thus, we can mathematically define
$\mathcal{L}_{\text{overlap}}$ as MSE between the predicted Dice, $y_{i}$ and
the target Dice, $\hat{y_{i}}$ given as;

$\mathcal{L}_{\text{overlap}}=\frac{1}{|N|}\sum_{i\in
N}(y_{i}-\hat{y}_{i})^{2}$ (6)

Since we also aim to evaluate the effect of hyperparameter on each label, we
predict 35 different Dice scores, each representing the degree of overlap for
an anatomical region in the brain.

Number of folded voxels: We obtain the target number of folded voxels from the
registration by computing the Jacobian determinant of the deformation field
$|J_{\phi}|$ for each input, this gives us an understanding of its
diffeomorphic properties. Where negative values of $|J_{\phi}|$ indicates non-
plausible mapping, values less than 1 means contraction around that local
region and values greater than 1 denotes expansion within that region. In
practice, the displacement field is represented in an n + 1D image, meaning
for each voxel p within the spatial domain, $\Omega$, the displacement u(p)
represents a shift that aligns _f_(p) and [_m $\circ$ $\phi$_](p) to
corresponding anatomical regions. The target number of folded voxels, $x_{i}$,
is derived by computing the sum where $|J_{\phi}|<$ 0 acorss the entire
displacement field. Hence, $\mathcal{L}_{\text{nfv}}$ in Equation 4 can be
formally defined as;

$\mathcal{L}_{\text{nfv}}=\frac{1}{|N|}\sum_{i\in N}(x_{i}-\hat{x}_{i})^{2}$
(7)

Where $x_{i}$ and $\hat{x_{i}}$ denote the predicted and target nfv
respectively. Throughout our experiments, we express nfv as a percentage,
i.e., %nfv.

##  5 Results

We evaluate our method on 3D brain MR scans. To demonstrate our contributions,
we train and evaluate HyperPredict models using two pre-trained convolutional
encoders and two registration models as previously mentioned. Subsequent
sections provide a comprehensive description of the experimental setup and the
experiments conducted.

Dataset and Pre-processing: We demonstrate our method on brain MRI
registration task. We use 414 T1-weighted brain scans from the OASIS dataset
(Brains, 2020; Marcus et al., 2007). The dataset contains cross-sectional
collection of subjects (men and women) aged from 18 to 96. The pre-processed
version of OASIS dataset is obtained from (Hoopes et al., 2021). They
performed standard pre-processing steps using FreeSurfer, (Fischl, 2012)
including; affine spatial normalization, skull stripping, sub-cortical
structures segmentation, and finally, cropping the resulting images to 160 x
192 x 224. The dataset includes sub-cortical segmentation maps of 35 different
anatomical structures for each volume, generated by FreeSurfer. We consider
Freesurfer a silver-standard method for generating automatic brain
segmentations (Puonti et al., 2016). We used the resulting segmentation maps
during training to compute the target segmentation overlap between the image
pair. We divide this dataset into train, validation and test sets of sizes
254, 80, and 80 respectively.

Implementation: To improve the computational efficiency of our experimental
process, we precompute registrations using both cLapIRN and Niftyreg between
all images pairs using randomly selected regularization weights, sampled from
a log-Normal distribution. We save the resulting metrics of these
registrations in a csv file, which is used to train the MLP network.

The parameterization of $g_{\theta}$ is based on three linear layers, each
followed by a LeakyReLU (Maas et al., 2013) activation function except the
final layer. Our proposed method is implemented in PyTorch, we adopt Adam
optimizer, (Kingma and Ba, 2014) with a learning rate of $10^{-4}$. The
training process follows the description in Section 4.1.

We ran our model using both the SymNet and cLapIRN encoders. The results
presented in Appendix C demonstrates that SymNet achieved a lower Mean
Absolute Error for both the Dice score and %nfv. Hence, for all experiments
detailed in this section, we show results for Symnet as an encoder. To
experiment HyperPredict’s ability to generalize across different registration
methods, we show results for HyperPredict on cLapIRN and Niftyreg registration
methods separately. We label HyperPredict trained with both methods as
HyperPredict${}_{\text{clap}}$ and HyperPredict${}_{\text{nr}}$ respectively.
We denote cLapIRN _single_ registration hyperparameter as $\lambda$. Niftyreg
offers various regularization options, we use spacing, _sx_ , bending energy,
_be_ , and linear elasticity, _le_ in our experiments.

###  5.1 Experiment 1: Accuracy of HyperPredict

This experiment aims to validate our proposed method by assessing if, and how
accurately a HyperPredict model is able to predict the desired evaluation
metrics. We check to see if through the registration algorithms, HyperPredict
learns to capture the effect of a wide range of hyperparameter values
simultaneously. We present a validation plot that shows our ability to predict
over both metrics for selected values of hyperparameters on the test dataset.
Additionally, we present Bland-Altman plots to evaluate the agreement between
predicted values from HyperPredict and the resulting registration metrics.

Figure 2: Error Distribution Plots on Both HyperPredict Models. Left:
Difference in predicted and target dice coefficients for selected
hyperparameter values. Right: Difference in predicted and target %nfv for
selected hyperparameter values. For visualization purpose, we display results
for selected values. Figure 3: Validating HyperPredict. Left: Bland-Altman
plots showing the agreement between the predicted vs target Dice scores across
the entire population for both models (each point represents the average Dice
for a particular image pair). Right: Bland-Altman plots showing the agreement
between the predicted vs target %nfv for the entire population.

Experiment Setup and Results: First, we separately train two HyperPredict
models, one for each registration algorithm (cLapIRN and Niftyreg) using the
steps outlined in method section. To measure the models predictive
performance, we analyze the residual plots between the predicted and target
metrics. To do this, we sample arbitrary discrete values of hyperparameters;
$\lambda\in$ [0.05, 0.075, 0.1, 0.125, 0.150, 0.2, 0.5, 1], and _be_ $\in$
[0.0001, 0.001, 0.0075, 0.05, 0.075, 0.1, 0.125], we observed more variance
with smaller values of bending energy. We use the sampled hyperparameter
values to run image pairs in the test set on HyperPredict${}_{\text{{clap}}}$
and HyperPredict${}_{\text{{nr}}}$ respectively. Similarly, for each test pair
and hyperparameter value, we derive the corresponding target metrics from both
registration methods. Finally, for the range of hyperparameter values
selected, we compute the difference between the predicted and target for both
metrics on each HyperPredict model. We present the error distribution plots in
Figure 2.

To assess the agreement between the predicted and target metrics, we use
Bland-Altman plots. For HyperPredict${}_{\text{{clap}}}$, we generate 200 log-
linearly spaced values between -10 and 0, $\lambda\in[-10,0]$. Similarly, for
HyperPredict${}_{\text{{nr}}}$, we sample 200 log-linearly spaced values for
the bending energy, $be\in[-10,0]$, we use a fixed value of 5 for the spacing,
$sx\in[5]$ (Segmentation overlap metrics indicate that the smallest spacing is
better in Niftyreg, see Figure 9 (left) in Appendix A, but we compromise for
computational efficiency), and a fixed value of 0.01 for linear elasticity,
_le_. For image pairs in the test dataset, with
HyperPredict${}_{\text{{clap}}}$, we run each of the 200 sampled
hyperparameter value, $\lambda$ with the pair to obtain the predicted metrics.
Next, we select the optimal $\lambda$ value per image pair and pass it to the
cLapIRN registration to obtain the target metrics. The same procedure is done
for HyperPredict${}_{\text{{nr}}}$, using sampled _be_ , and fixed values of
_le_ and _sx_ as parameters.

In Figure 3 we present the pairwise differences between the predicted and
target metrics plotted against their average, for both
HyperPredict${}_{\text{{clap}}}$ (top) and HyperPredict${}_{\text{{nr}}}$
(bottom). The plots reveal a close agreement between results from the
predicted and target metrics for HyperPredict${}_{\text{{clap}}}$, with narrow
upper and lower boundaries, indicating that majority of the differences fall
within an acceptable range. Furthermore, the figure demonstrates that our
method is able to yield metrics similar to those obtained through cLapIRN
registration for each image pair. We found an average difference in the
predicted and target Dice scores to be 0.015$\pm$0.014 and 0.028$\pm$0.026 for
the %nfv.

Using HyperPredict${}_{\text{{nr}}}$, we observe reasonable agreement between
the predicted and target dice scores (bottom left), however, our model
exhibits some limitations in accurately predicting %nfv (bottom right). The
determined difference is 0.018$\pm$0.013 and 0.099$\pm$0.109 for Dice and %nfv
respectively.

###  5.2 Experiment 2: HyperPredict vs Cross-Validation

The standard approach to obtaining an optimal hyperparameter is by performing
cross validation. Hence, this section aims to evaluate the effect of inferring
instance specific hyperparameters using HyperPredict. We achieve this by
comparing the results obtained from our method with standard cross-validation.

Experiment Setup and Results: In this section, we conduct three experiments.

Step 1: In the first experiment, we compare the Dice scores obtained using our
method on selected anatomical structures and %nfv with that of cross-
validation. For HyperPredict${}_{\text{{clap}}}$ and cross-validation on
cLapIRN, we leverage the low computational burden offered by HyperPredict to
test multiple hyperparameter values on a single pair. The sampling method is
similar to the previous experiment. At test time, we run the sampled values
for each subject in the test set through the trained
HyperPredict${}_{\text{{clap}}}$ network. Using the criteria defined in
Equation 3, we select the optimal hyperparameters across specific labels.
Finally, we register the image pair using the derived optimal values to obtain
the corresponding target Dice coefficient and %nfv.

Figure 4: HyperPredict vs Cross-Validation. Left: Comparison of the Dice
Coefficient on both methods for selected labels across the population. Right:
Comparison of the %nfv on both methods across the population.

Since in practice it is not feasible to run 200 hyperparameter values per
image pair on a registration algorithm in search of the optimal
hyperparameter, we perform cross-validation using 8 discrete values of
$\lambda$ selected arbitrarily, $\lambda$ $\in$ [0.05, 0.075, 0.1, 0.125,
0.150, 0.2, 0.5, 1], cross-validation on cLapIRN found optimal value,
$\lambda^{*}$, to be 0.075. We use this for registration at test time across
the _entire_ test dataset.

We repeat steps the above steps using HyperPredict${}_{\text{{nr}}}$ and
Niftyreg registration algorithm. For cross-validation, we maintain a fixed
spacing value of 5, a fixed linear elasticity of 0.01, and discretely sample
the bending energy, _be_ $\in$ [0.0001, 0.001, 0.0075, 0.05, 0.1, 0.2, 0.5,
1]. Cross-validation on Niftyreg found optimal value, _be_ , to be 0.0075.

We compare the results of HyperPredict and cross-validation for both
experiments using the Dice coefficient scores and %nfv. In Figure 4 (top row),
we present results of HyperPredict${}_{\text{{clap}}}$ vs cross-validation on
cLapIRN for both metrics. The bottom row of the figure depicts results
obtained from Niftyreg.

From the Figure, HyperPredict${}_{\text{{clap}}}$ demonstrates comparable Dice
scores with those obtained through cross-validation, with HyperPredict
slightly outperforming cross-validation on a few anatomical structures.
Interestingly, we find that our method of selecting optimal hyperparameter
values leads to a reduction in folded voxels, indicating more desirable
(plausible) registrations (top right). Similarly, the experiment with Niftyreg
shows comparable dice scores between both methods, and the image pair with the
highest %nfv has a value less than that of cross-validation.

Figure 5: HyperPredict vs Cross-Validation. Top: Worse case analysis on
HyperPredict${}_{\text{{clap}}}$ Bottom: Worse case analysis on
HyperPredict${}_{\text{{nr}}}$.

Step 2: For the second experiment, we perform a best and worst case analysis
in Figure 5, where we plot the value of the metric derived from HyperPredict
against cross-validation for each image pair. We present results for
HyperPredict${}_{\text{{clap}}}$ and HyperPredict${}_{\text{{nr}}}$. For all
four plots depicted in the figure, the best cases are represented with a blue
marker, indicating samples where HyperPredict out-performs cross-validation,
red markers represent regions where cross-validation surpasses our proposed
method, and equal cases are represented as black points that lie on the dotted
slope.

Step 3: In the final experiment comparing cross-validation with HyperPredict,
we run the registration of a single image pair using the optimal $\lambda^{*}$
= 0.075 derived from cross-validation on cLapIRN. Similarly, we perform the
registration on the same pair using the optimal $\lambda^{*}$ = 0.102 obtained
from HyperPredict${}_{\text{{clap}}}$ for that specific pair. The outcomes of
these two approaches are illustrated in Figure 6. We record the dice score
prior to registration as 0.425. The dice scores of both methods after
registration are 0.755 and 0.751 respectively, and their corresponding folded
voxels as 0.73% and 0.48%.

Figure 6: Cross Validation vs HyperPredict on cLapIRN registration: Top: Using
optimal value of $\lambda^{*}$ = 0.075 derived from cross validation to
register an image pair. Bottom: Using optimal value of $\lambda^{*}$ = 0.102
derived from HyperPredict${}_{\text{{clap}}}$ to register the same image pair.

###  5.3 Experiment 3: Analyzing the Distribution of Hyperparameters

The objective of this experiment is to assess the distribution of
regularization values for the two variants of HyperPredict:
HyperPredict${}_{\text{{clap}}}$ and HyperPredict${}_{\text{{nr}}}$, and also
identifying the best values to utilize for registration. We present the
results obtained when optimizing for two instance-specific cases; across
structures and across subjects. For visualization purposes, we present results
of selected sub-cortical anatomical regions (Thalamus, Caudate, Putamen,
Pallidum, Hippocampus, Amygdala, Accumbens).

Experiment Setup and Results: Maintaining the defined criteria and
experimental setup in previous experiments, this section is subdivided into
two distinct experiment.

Figure 7: Distribution of Optimal Regularization Weights. Left: across labels
Right: across subjects. For HyperPredict${}_{\text{{clap}}}$(top row) and
HyperPredict${}_{\text{{nr}}}$(bottom row). The black-dashed lines in the plot
represents the optimal value obtained through cross-validation method on both
registration.

Step 1: The first experiment seeks to validate our hypothesis that optimal
values differ on a subject and structure basis. We infer the results of
HyperPredict${}_{\text{{clap}}}$ and HyperPredict${}_{\text{{nr}}}$ using the
test dataset and sampled values of $\lambda$ and _be_ respectively. From the
result, we obtain the distribution of optimal hyperparameters across subjects
(right) and selected labels (left) presented in Figure 7.

Figure 8: Varying regularization weights per anatomical structure. Right:
Average Dice scores plotted against sampled regularization weights for
selected anatomical regions. Left: Registration of an image pair using optimal
parameters, with Thalamus and Amygdala as intended regions of interest.

Step 2: In the second experiment we identify label specific optimal
regularization weights. For an image pair, HyperPredict derives 35 different
Dice scores, each for an anatomical region. We leverage on this to observe the
effect of utilizing instance-specific optimal values on corresponding regions.
With the test results obtained from HyperPredict${}_{\text{{clap}}}$, we
compute the optimal Dice scores across each label. Figure 8 (right)
illustrates the varying optimal weights for selected structures. We visualize
registration results (left) for two regions; Thalamus and Amygdala, using
their optimal values, 0.09 and 0.16 respectively. In both registrations, we
obtain an overall dice score of 0.75 and 0.73 across the entire image.
However, when optimizing for Thalamus (using 0.09), the Dice score of Thalamus
yields 0.89 and that of Amygdala was 0.80. Similarly, when optimizing for
Amygdala (using 0.16), we obtained 0.78 and 0.80 for Thalamus and Amygdala
respectively. Notice we obtain higher Dice scores for the specific region we
optimize for, showing the significance of instance-specific regularization.
The red-dashed line in the figure represents the optimal value obtained
through cross-validation method, 0.075. We present result for
HyperPredict${}_{\text{{nr}}}$ in Appendix A, Figure 9 (right).

###  5.4 Computational Efficiency at Test Time

HyperPredict demonstrates notable computational efficiency at test time. A
single model is capable of evaluating the effects of hundreds of
hyperparameter values without imposing significant computational burden. In
Table 1, we present a summary that compares the test time of HyperPredict with
other baseline registration methods. It is important to note that while
HyperPredict by itself is not a registration method, it acts as a tool that
facilitates the selection of optimal hyperparameters. Therefore, the purpose
of the comparison is to highlight that unlike HyperPredict, baseline methods
are unable to infer the effect of hundreds (or thousands) of hyperparameters
within a short period of time - which is a hindrance when trying to select the
optimal hyperparameter.

The runtime for methods under our proposed framework (utilizing Symnet
encoder) is presented, with testing conducted using 8000 distinct
hyperparameter values (HP) and includes the time it takes with and without
registration (w/reg). HyperPredict with registration (w/reg) is the time it
takes to run all 8000 values and using the optimal for registrating the image
pair (either with cLapIRN or Niftyreg method), while HyperPredict without
registration (w/o reg) is the time taken to test HyperPredict with 8000
values. We also present results of HyperMorph (Hoopes et al., 2021),
highlighted in Section 3 as an additional baseline method, we asterisks it to
show that the result is adopted from the paper.

Table 1: Comparison between HyperPredict and other registration methods at test time. Results are presented in seconds.  Method |  Test Time(1HP) | Test Time(8000 HP)
---|---|---
w/o reg |  w/reg
HyperMorph* |  2.1 |  - |  16800
Niftyreg |  15.0 |  - |  120000
HyperPredict${}_{\text{nr}}$ |  - |  0.84 |  15.84
cLapIRN |  0.10 |  - |  800
HyperPredict${}_{\text{clap}}$ |  - |  0.84 |  0.94

##  6 Discussion

Our experiments demonstrated for the first time that the effects of a
regularization hyperparameters can be predicted for pairwise image
registration. We employ two metrics as a means of evaluating the registration;
the dice score (for each segmented region), and the number of folded voxels
(nfv), however, our framework could be extended to incorporate additional
registration quality metrics.

Accuracy of HyperPredict: In our experiments, we utilize two registration
algorithms: cLapIRN and Niftyreg to derive target values for training
HyperPredict. Analysis from the residual plots in Experiment 1 reveals that
the difference between the predicted and true values are centered around zero.
This suggests that on average, our predictive model is making unbiased
predictions and does not consistently overestimate or underestimate the
predicted metrics. Similarly, the Bland-Altman plots show a good correlation
between HyperPredict and cLapIRN registration. With Niftyreg, we find
comparable results between the predicted and target dice coefficients,
however, our model demonstrates sight limitations in accurately predicting the
%nfv. We hypothesize that this effect may be as a result of the use of a more
complex similarity function (normalised mutual information) and the
algorithmic heuristics, that makes it less predictable than cLapIRN. Another
contribution to the sub-optimal predictive performance on %nfv may be the
choice of encoder. Currently, we employ two off-the-shelf convolutional
encoders. The experiment presented in Appendix C shows that the choice of
encoder influences how effective HyperPredict is in estimating the desired
metrics, it may be that: (a) the encoded representation itself does not
contain sufficient information for HyperPredict${}_{\text{nr}}$ to infer the
%nfv and/or (b) the use of a fixed convolutional encoder limits its ability to
capture certain representations of the input specific to the task. In future,
it is worth investigating the restrictions associated with encoder choice.

HyperPredict vs Cross-Validation: Cross-validation is the principle existing
method for hyperparameter selection. The experiment results in Figure 4
comparing HyperPredict and cross-validation show comparable dice scores using
HyperPredict${}_{\text{clap}}$ and HyperPredict${}_{\text{nr}}$, with both
HyperPredict approaches deriving slightly better results on a number of
anatomical regions (Pallidum and Amygdala). We also find that
HyperPredict${}_{\text{clap}}$ produces registration with substantially less
voxel folding than cross-validation (Figure 4 top-right). This implies that
our approach is able to capture the variability between subjects and identify
an improved instance-specific level of regularization.

We perform an additional comparison between both methods by investigating the
proportion of examples where HyperPredict over-performs, under-performs, or
yields comparable results to cross validation, presented in Figure 5, we term
this as better, worse and equal cases respectively in the Figure. For both
HyperPredict${}_{\text{clap}}$ and HyperPredict${}_{\text{nr}}$, the Dice
coefficient are comparable with HyperPredict${}_{\text{nr}}$ having slightly
more cases where HyperPredict performs better. HyperPredict${}_{\text{clap}}$
demonstrates a significant improvement over cross-validation in terms of %nfv,
exhibiting lower %nfv values for nearly all subjects (top-right). On the other
hand, when comparing HyperPredict${}_{\text{nr}}$ with cross-validation, there
are slightly more instances where cross-validation outperforms
HyperPredict${}_{\text{nr}}$ (bottom-right), the reason for this may be
attributed to the limitation in the predictive ability of
HyperPredict${}_{\text{nr}}$.

Finally, the registration of the same image pair using the regularization
weight of 0.102 and 0.075 derived from HyperPredict${}_{\text{clap}}$ and
cross-validation shows (in Figure 6) that while we arrive at a comparable dice
score of 0.751 and 0.755 respectively, our method performs better in terms of
%nfv, with corresponding values of 0.48% and 0.73% respectively. This shows
that utilizing HyperPredict enables us to arrive at more plausible deformation
fields.

Distribution of Regularization Weights: Our study finds that different
regularization weights yield optimal results depending on the subject and
specific anatomical regions of interest. Our proposed framework has the
flexibility of selecting such task-specific regularization values at test time
as shown in Figures 7 and 8 (right), that can then be used for registration.
The effect of such instance specific values is observed in Experiment 3
(Figure 8, left). When we apply the optimal regularization weight of
independent anatomical regions (Thalamus and Amygdala) during registration, we
derive higher dice scores for that particular region of interest.

We conduct an additional supplementary experiment presented in Appendix A that
learns the effect of multiple hyperparameters. We utilize the Niftyreg
algorithm due to its ability to incorporate a diverse range of regularization
parameters, specifically we learn the effect of two of them simultaneously;
bending energy (_be_), and linear elasticity (_le_). We present a heat map in
Appendix A, Figure 10 depicting the combined impact of both parameters on the
quality metrics across subjects. We observe from the figure that lower values
of bending energy and linear elasticity results in our method predicting
higher dice scores with more irregularities in the deformation field and vice-
versa.

Computational Efficiency: Finally, we analyse the computational efficiency of
HyperPredict at test time. Exploiting a single trained HyperPredict model at
test time means that for any arbitrary image pair, we are able to infer the
effect of a range of hyperparameter values from a continuous interval without
the need of labelled data while limiting the computational burden. It is
important to note that our method is an _amortized inference approach_ that
incurs an initial computational cost at train time, but results in a
computationally efficient prediction at test time. Additionally, without our
method a typical approach of selecting an optimal value at test time would
involve running multiple independent registrations with different values. This
approach becomes impractical as it restricts the search space of values and
incurs additional computational costs for each independent registration.

Comparing the run time of a registration model using a single hyperparameter
on an image pair to HyperPredict (testing thousands of values on the same
image pair) reveals a notable difference in computational efficiency at test
time. Table 1 shows that HyperPredict takes less than 0.84 seconds to predict
the registration metrics for a single image pair with _8000 different_
hyperparameter values. This is approximately 8 times less the time it takes a
cLapIRN registration model to derive the results from a _single_
hyperparameter value. The first column on the table presents the run time
using a single hyperparameter value on various registration methods, the aim
of HyperPredict is to test multiple values, hence we don’t show results for
our method in this column. In the second column, we show the run time with and
without registration for 8000 distinct hyperparameter values. With
HyperPredict, we can efficiently test and select the best value from all 8000
options. On the other hand, baseline registration methods necessitate more
computational resources since the registration using all 8000 values need to
be executed and the optimal one chosen based on the registration outcomes.
Evidence of this is shown in the last column (w/reg), where it takes
approximately 16800 seconds for HyperMorph to run 8000 distinct values on a
single image pair, Niftyreg takes around 120000s (compared to
HyperPredict${}_{\text{nr}}$ that takes 0.84 seconds to run all 8000 values
and an additional 15s to run registration on the optimal value). A similar
pattern can be observed between cLapIRN and HyperPredict${}_{\text{clap}}$,
with runtime of 800 seconds and 0.94 seconds, respectively. HyperPredict
enables a fast and more efficient method of hyperparameter selection at test
time based on a flexible criteria, presenting a substantial advantage over
baseline parameter selection methods.

##  7 Limitations and Future Work

The findings from HyperPredict demonstrate the efficacy of our approach in
aiding the selection of hyperparameters that align with the unique attributes
of the input. One limitation of our approach is the computational cost during
training, as it involves running multiple registrations to optimize the MLP.
Although, we perform further experiment presented in Appendix E that
illustrates that HyperPredict can learn from 25% of the available data while
achieving satisfactory result. This significantly decreases the overall
registrations that need to be run when utilizing our approach. To circumvent
this computational cost at train time and for reproducibility, we ran multiple
registrations (for both cLapIRN and Niftyreg) across all image pairs with
sampled hyperparameter values. We have made the csv files containing
registration results publicly accessible in our GitHub repository.

Our findings indicate that regularization weights vary across different
anatomical structures, it is also likely that narrower regions in the brain
are more difficult to register, future works will look into learning the
importance of different regions and the effect it has on registration.
Additionally, we plan to explore other hyperparameters, such as cost
functions, as well as incorporating diverse datasets like Alzheimer’s and lung
data to generalize HyperPredict. Furthermore, experiments in Appendix B and C
depicts the influence of model architecture and encoded representations on
metric prediction, we intend to explore alternative encoded representations of
image pairs. This can include deriving an image representation from a network
that predicts the overlap between pairs of images, fine-tuning existing
models, or utilizing self-supervised representation learning, we can then
analyze the effect any of the methods have on the predicted result. Finally,
in computing the number of folded voxels, we considered the entire image -
including regions outside the labels, another problem formulation could be to
learn the diffeomorphic properties of the image for only regions in the image
where anatomical structures exist.

We have made our code, registration results and guidelines available on GitHub
and we invite the community to contribute by replicating our experiments on
different datasets, conducting further evaluations on new experiments, or
expanding the applicability of HyperPredict to different use-cases.

##  8 Conclusion

The accuracy of non-linear registration algorithms heavily depends on the
choice of hyperparameter values, which may vary based on various factors.
Thus, selecting a hyperparameter depending on specific use-case is an
essential component of image registration. In this study, we propose
HyperPredict, an efficient method of learning the effect of hyperparameters on
registration, one that eliminates the need of running multiple registrations
at test time in search for an optimal hyperparameter. HyperPredict utilizes an
MLP that takes an image pair and desired set of hyperparameter as input and in
return, predicts the evaluation metric that corresponds to the input. Our
experiments show that by training a single HyperPredict model, we capture the
effect of a range of hyperparameter values on an image pair. With this
ability, at test time HyperPredict is able to predict the registration results
of an image pair with thousands of registration hyperparameter values, giving
us the ability to select the optimal value for a specific use-case. In
summary, we have presented a novel approach we believe is more efficient in
aiding the selection of optimal hyperparameter in image registration. While
the experiments presented in this paper is specific to the domain of medical
imaging, the general idea of HyperPredict can be adopted in other
applications.

Ethical Standards

The work follows appropriate ethical standards in conducting research and
writing the manuscript. We train our models with publicly available data.

Conflicts of Interest

We declare we don’t have conflicts of interest.

## References

  * Andersson et al. (2007) Jesper LR Andersson, Mark Jenkinson, Stephen Smith, et al.  Non-linear registration, aka spatial normalisation fmrib technical report tr07ja2.  _FMRIB Analysis Group of the University of Oxford_ , 2(1):e21, 2007.
  * Ashburner (2007) John Ashburner.  A fast diffeomorphic image registration algorithm.  _Neuroimage_ , 38(1):95–113, 2007.
  * Avants et al. (2008a) Brian B Avants, Charles L Epstein, Murray Grossman, and James C Gee.  Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain.  _Medical image analysis_ , 12(1):26–41, 2008a.
  * Avants et al. (2008b) Brian B Avants, Charles L Epstein, Murray Grossman, and James C Gee.  Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain.  _Medical image analysis_ , 12(1):26–41, 2008b.
  * Balakrishnan et al. (2019) Guha Balakrishnan, Amy Zhao, Mert R Sabuncu, John Guttag, and Adrian V Dalca.  Voxelmorph: a learning framework for deformable medical image registration.  _IEEE transactions on medical imaging_ , 38(8):1788–1800, 2019.
  * Beg et al. (2005) M Faisal Beg, Michael I Miller, Alain Trouvé, and Laurent Younes.  Computing large deformation metric mappings via geodesic flows of diffeomorphisms.  _International journal of computer vision_ , 61(2):139–157, 2005.
  * Bergstra and Bengio (2012) James Bergstra and Yoshua Bengio.  Random search for hyper-parameter optimization.  _Journal of machine learning research_ , 13(2), 2012.
  * Bergstra et al. (2011) James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl.  Algorithms for hyper-parameter optimization.  _Advances in neural information processing systems_ , 24, 2011.
  * Brains (2020) OASIS Brains.  Open access series of imaging studies, 2020.
  * Dice (1945) Lee R Dice.  Measures of the amount of ecologic association between species.  _Ecology_ , 26(3):297–302, 1945.
  * Fischl (2012) Bruce Fischl.  Freesurfer.  _Neuroimage_ , 62(2):774–781, 2012.
  * He et al. (2022) Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16000–16009, 2022.
  * Hoopes et al. (2021) Andrew Hoopes, Malte Hoffmann, Bruce Fischl, John Guttag, and Adrian V Dalca.  Hypermorph: Amortized hyperparameter learning for image registration.  In _Information Processing in Medical Imaging: 27th International Conference, IPMI 2021, Virtual Event, June 28–June 30, 2021, Proceedings 27_ , pages 3–17. Springer, 2021.
  * Joshi et al. (2004) Sarang Joshi, Brad Davis, Matthieu Jomier, and Guido Gerig.  Unbiased diffeomorphic atlas construction for computational anatomy.  _NeuroImage_ , 23:S151–S160, 2004.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Le Folgoc et al. (2016) Loic Le Folgoc, Herve Delingette, Antonio Criminisi, and Nicholas Ayache.  Quantifying registration uncertainty with sparse bayesian modelling.  _IEEE transactions on medical imaging_ , 36(2):607–617, 2016.
  * Ma et al. (2008) Jun Ma, Michael I Miller, Alain Trouvé, and Laurent Younes.  Bayesian template estimation in computational anatomy.  _NeuroImage_ , 42(1):252–261, 2008.
  * Maas et al. (2013) Andrew L Maas, Awni Y Hannun, Andrew Y Ng, et al.  Rectifier nonlinearities improve neural network acoustic models.  In _Proc. icml_ , volume 30, page 3. Atlanta, GA, 2013.
  * Marcus et al. (2007) Daniel S Marcus, Tracy H Wang, Jamie Parker, John G Csernansky, John C Morris, and Randy L Buckner.  Open access series of imaging studies (oasis): cross-sectional mri data in young, middle aged, nondemented, and demented older adults.  _Journal of cognitive neuroscience_ , 19(9):1498–1507, 2007.
  * Meng et al. (2022) Mingyuan Meng, Lei Bi, Michael Fulham, David Dagan Feng, and Jinman Kim.  Enhancing medical image registration via appearance adjustment networks.  _NeuroImage_ , 259:119444, 2022.
  * Mockus (1998) Jonas Mockus.  The application of bayesian methods for seeking the extremum.  _Towards global optimization_ , 2:117, 1998.
  * Modat et al. (2014) Marc Modat, David M Cash, Pankaj Daga, Gavin P Winston, John S Duncan, and Sébastien Ourselin.  Global image registration using a symmetric block-matching approach.  _Journal of medical imaging_ , 1(2):024003–024003, 2014.
  * Mok and Chung (2020a) Tony CW Mok and Albert Chung.  Fast symmetric diffeomorphic image registration with convolutional neural networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4644–4653, 2020a.
  * Mok and Chung (2020b) Tony CW Mok and Albert CS Chung.  Large deformation diffeomorphic image registration with laplacian pyramid networks.  In _Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part III 23_ , pages 211–221. Springer, 2020b.
  * Mok and Chung (2021a) Tony CW Mok and Albert CS Chung.  Conditional deformable image registration with convolutional neural network.  In _Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part IV 24_ , pages 35–45. Springer, 2021a.
  * Mok and Chung (2021b) Tony CW Mok and Albert CS Chung.  Conditional deformable image registration with convolutional neural network.  In _Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part IV 24_ , pages 35–45. Springer, 2021b.
  * Niethammer et al. (2019) Marc Niethammer, Roland Kwitt, and Francois-Xavier Vialard.  Metric learning for image registration.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8463–8472, 2019.
  * Puonti et al. (2016) Oula Puonti, Juan Eugenio Iglesias, and Koen Van Leemput.  Fast and sequence-adaptive whole-brain segmentation using parametric bayesian modeling.  _NeuroImage_ , 143:235–249, 2016.
  * Risholm et al. (2012) Petter Risholm, Firdaus Janoos, Jennifer Pursley, Andriy Fedorov, Clare Tempany, Robert A Cormack, and William M Wells.  Selection of optimal hyper-parameters for estimation of uncertainty in mri-trus registration of the prostate.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2012: 15th International Conference, Nice, France, October 1-5, 2012, Proceedings, Part III 15_ , pages 107–114. Springer, 2012.
  * Risholm et al. (2013) Petter Risholm, Firdaus Janoos, Isaiah Norton, Alex J Golby, and William M Wells III.  Bayesian characterization of uncertainty in intra-subject non-rigid registration.  _Medical image analysis_ , 17(5):538–555, 2013.
  * Rohlfing (2011) Torsten Rohlfing.  Image similarity and tissue overlaps as surrogates for image registration accuracy: widely used but unreliable.  _IEEE transactions on medical imaging_ , 31(2):153–163, 2011.
  * Rueckert et al. (1999) Daniel Rueckert, Luke I Sonoda, Carmel Hayes, Derek LG Hill, Martin O Leach, and David J Hawkes.  Nonrigid registration using free-form deformations: application to breast mr images.  _IEEE transactions on medical imaging_ , 18(8):712–721, 1999.
  * Rühaak et al. (2017) Jan Rühaak, Thomas Polzin, Stefan Heldmann, Ivor JA Simpson, Heinz Handels, Jan Modersitzki, and Mattias P Heinrich.  Estimation of large motion in lung ct by integrating regularized keypoint correspondences into dense deformable registration.  _IEEE transactions on medical imaging_ , 36(8):1746–1757, 2017.
  * Siebert et al. (2021) Hanna Siebert, Lasse Hansen, and Mattias P Heinrich.  Fast 3d registration with accurate optimisation and little learning for learn2reg 2021.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , pages 174–179. Springer, 2021.
  * Simpson et al. (2012) Ivor JA Simpson, Julia A Schnabel, Adrian R Groves, Jesper LR Andersson, and Mark W Woolrich.  Probabilistic inference of regularisation in non-rigid registration.  _NeuroImage_ , 59(3):2438–2451, 2012.
  * Simpson et al. (2015) Ivor JA Simpson, Manuel Jorge Cardoso, Marc Modat, David M Cash, Mark W Woolrich, Jesper LR Andersson, Julia A Schnabel, Sébastien Ourselin, Alzheimer’s Disease Neuroimaging Initiative, et al.  Probabilistic non-linear registration with spatially adaptive regularisation.  _Medical image analysis_ , 26(1):203–216, 2015.
  * Snoek et al. (2012) Jasper Snoek, Hugo Larochelle, and Ryan P Adams.  Practical bayesian optimization of machine learning algorithms.  _Advances in neural information processing systems_ , 25, 2012.
  * Szeliski (2022) Richard Szeliski.  _Computer vision: algorithms and applications_.  Springer Nature, 2022.
  * Turner et al. (2021) Ryan Turner, David Eriksson, Michael McCourt, Juha Kiili, Eero Laaksonen, Zhen Xu, and Isabelle Guyon.  Bayesian optimization is superior to random search for machine learning hyperparameter tuning: Analysis of the black-box optimization challenge 2020.  In _NeurIPS 2020 Competition and Demonstration Track_ , pages 3–26. PMLR, 2021.
  * Vialard and Risser (2014) François-Xavier Vialard and Laurent Risser.  Spatially-varying metric learning for diffeomorphic image registration: A variational framework.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2014: 17th International Conference, Boston, MA, USA, September 14-18, 2014, Proceedings, Part I 17_ , pages 227–234. Springer, 2014.
  * Vialard et al. (2011) François-Xavier Vialard, Laurent Risser, Darryl D Holm, and Daniel Rueckert.  Diffeomorphic atlas estimation using karcher mean and geodesic shooting on volumetric images.  In _MIUA_ , pages 55–60, 2011.
  * Viola and Wells III (1997) Paul Viola and William M Wells III.  Alignment by maximization of mutual information.  _International journal of computer vision_ , 24(2):137–154, 1997.
  * Wang and Zhang (2021) Jian Wang and Miaomiao Zhang.  Bayesian atlas building with hierarchical priors for subject-specific regularization.  In _Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part IV 24_ , pages 76–86. Springer, 2021.
  * Wang et al. (2019) Jian Wang, Wei Xing, Robert M Kirby, and Miaomiao Zhang.  Data-driven model order reduction for diffeomorphic image registration.  In _Information Processing in Medical Imaging: 26th International Conference, IPMI 2019, Hong Kong, China, June 2–7, 2019, Proceedings 26_ , pages 694–705. Springer, 2019.
  * Zhang et al. (2013) Miaomiao Zhang, Nikhil Singh, and P Thomas Fletcher.  Bayesian estimation of regularization and atlas building in diffeomorphic image registration.  In _International conference on information processing in medical imaging_ , pages 37–48. Springer, 2013.

##  A Experiments on Niftyreg

Figure 9:  Left: Niftyreg yields higher Dice coefficients with smaller spacing
values. Right: Varying optimal bending energy across selected labels. Figure
10: Heatmap depicting the combined effect of both hyperparameters; bending
energy and linear elasticity on the dice score and %nfv across subjects.

Figure 10 is a supplementary experiment to investigate the combined effect of
multiple regularization on each subject. To conduct this experiment, we train
an additional HyperPredict${}_{\text{{nr}}}$ model following the methodology
outlined in Section 5. However, in this iteration the model specifically
focuses on learning the influence of both the bending energy and linear
elasticity. Both of which are sampled from a log-normal distribution and serve
as input to the MLP at train time. At test time, for both _be_ and _le_ , we
sample 200 log-linearly spaced values between -10 and 0 for each
hyperparameter; this means each subject is tested with a combination of 40,000
values. Without a selection criteria, we present a heatmap above that shows
the simultaneous effect of both bending energy and linear elasticity on the
Dice coefficient and the folded voxels across subjects.

Figure 11: Applying No Restriction on %nfv: Since the criteria used to define
an optimal value is flexible after training, we change our criteria to
consider no restrictions on the %nfv in this experiment, and compare the
results with the constrained version. Bottom Left: Dice scores plotted against
values of $\lambda$, the red line is the optimal value inferred with
restriction on %nfv for this specific image pair (Pair 1). Bottom Right:
Comparing warped images with and without restricted %nfv for image Pair 1. The
non-constrained warped image (brain image on the right) has a closer
similarity to the fixed image compared to the constrained warped image (brain
image on the left), this is made evident by the presence of a higher dice
score (the red marker depicts visual similarities). However, we expect a less
smooth warped image, as highlighted by the increased %nfv from 0.4% to 2.1%,
we depict this visually using blue markers.

##  B HyperPredict Architecture

Figure 12: Effect of HyperPredict model on two different network architecture
(using one and two hidden layers). Top: Results using
HyperPredict${}_{\text{clap}}$. Bottom: Results using
HyperPredict${}_{\text{nr}}$

.

##  C Comparing Summary Statistics and Encoder type

Figure 13: Comparing summary statistics and encoder types. Mean absolute error
(MAE) of Dice scores (left) and %nfv (right) using two different summary
statistics (taking the _mean_ across the channels compared to concatenating
the _min_ , _max_ , and _mean_ across the channels) and two different
convolutional encoders (cLapIRN and Symnet).

##  D Combined Evaluation of Dice Coefficient and %nfv

Figure 14: Distribution plot of Dice coefficient and %nfv on selected values
of $\lambda$ (left) and bending energy (right), for
HyperPredict${}_{\text{clap}}$ and HyperPredict${}_{\text{nr}}$ respectively.
We show results of a sub-sample for visualization purpose. Both HyperPredict
methods predict higher irregularities in the deformation field as the
hyperparameter value gets smaller.

##  E Data Size

Figure 15: Training HyperPredict with different number of training samples
(100%, 50% and 25% of image pairs) Top: Mean absolute error of dice
coefficient and %nfv on HyperPredict${}_{\text{clap}}$ Bottom: Mean absolute
error of dice coefficient and %nfv on HyperPredict${}_{\text{nr}}$

##  F Sensitivity Analysis

Figure 16: Sensitivity analysis result on $\alpha$ using varying values for
training. We experimented with smaller values of $\alpha$, for visualization
purposes, we present results for selected four. Top: MAE of dice coefficient
and %nfv on HyperPredict${}_{\text{clap}}$ Bottom: MAE of dice coefficient and
%nfv on HyperPredict${}_{\text{nr}}$