# Specialist or Generalist? Instruction Tuning for Specific NLP Tasks

Chufan Shi♠ , Yixuan Su♢,♣ , Cheng Yang♠ , Yujiu Yang♠ , Deng Cai♡
♠Tsinghua Shenzhen International Graduate School, Tsinghua University
♢Language Technology Lab, University of Cambridge ♣Cohere
♡Tencent AI Lab
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
This work was completed during an internship at Tencent AI Lab. Corresponding
author.

###### Abstract

The potential of large language models (LLMs) to simultaneously perform a wide
range of natural language processing (NLP) tasks has been the subject of
extensive research. Although instruction tuning has proven to be a data-
efficient method for transforming LLMs into such generalist models, their
performance still lags behind specialist models trained exclusively for
specific tasks. In this paper, we investigate whether incorporating broad-
coverage generalist instruction tuning can contribute to building a specialist
model. We hypothesize that its efficacy depends on task specificity and skill
requirements. Our experiments assess four target tasks with distinct coverage
levels, revealing that integrating generalist instruction tuning consistently
enhances model performance when the task coverage is broad. The effect is
particularly pronounced when the amount of task-specific training data is
limited. Further investigation into three target tasks focusing on different
capabilities demonstrates that generalist instruction tuning improves
understanding and reasoning abilities. However, for tasks requiring factual
knowledge, generalist data containing hallucinatory information may negatively
affect the model’s performance. Overall, our work provides a systematic guide
for developing specialist models with general instruction tuning. Our code and
other related resources can be found at
https://github.com/DavidFanzz/Generalist_or_Specialist.

##  1 Introduction

The latest generation of large language models (LLMs), such as ChatGPT OpenAI
(2022) and GPT4 OpenAI (2023), are often referred to as generalist models for
their exceptional generalizability to perform various natural language
processing (NLP) tasks. Recent studies Taori et al. (2023); Zhou et al.
(2023); Gudibande et al. (2023) suggest that (1) the foundation of their
superior performance (i.e., knowledge and capabilities) is predominantly
acquired during large-scale unsupervised pre-training; and (2) instruction
tuning Sanh et al. (2022); Wei et al. (2022a); Mishra et al. (2021); Ouyang et
al. (2022) is an incredibly data-efficient method for unleashing the power of
LLMs to complete realistic NLP tasks. However, under rigorous evaluation, the
performance of those instruction-following generalist models often falls short
compared to traditional task-specific specialist models Jiao et al. (2023b);
Qin et al. (2023); Fang et al. (2023); Liu et al. (2023). Recently, there has
also been a growing trend towards developing specialist models using
instruction tuning Jiao et al. (2023a); Wang et al. (2023b); Zhang et al.
(2023); Cheng et al. (2023); Wang et al. (2023a).

In this paper, we study how to better harness the power of LLM for specific
NLP tasks using instruction tuning. Our research is motivated by the existence
of various broad-coverage general-purpose instruction-following datasets Taori
et al. (2023); Peng et al. (2023); Labs (2023); Xu et al. (2023); Zhou et al.
(2023); Su et al. (2023b) and their surprising efficiency for turning LLMs
into capable instruction-following generalists. For instance, Zhou et al.
(2023) show that merely one thousand supervised input-output pairs are
necessary to build a competent generalist. In contrast to general-purpose
instruction tuning, our preliminary experiments show that a sufficiently large
set of task-specific data is still required for transforming an LLM into a
superior specialist. This leads us to a pivotal research question: How to
better unleash the power of LLMs for specific NLP tasks by marrying the best
of two worlds? More specifically, can general-purpose instruction-following
datasets aid in the transformation of an LLM into a specialist? If so, when
and how?

We hypothesize the answers to the previous questions depend on (1) how
specific the target task is; and (2) what skills the target task requires. To
test this hypothesis, we first assess four target tasks with distinct levels
of coverage. Our findings reveal that integrating general instruction
tuning—that is, training with generalist data enhances the model’s performance
on specific NLP tasks with broad task coverage, particularly when the amount
of task-specific training data is limited. To gain a deeper understanding of
the improvements elicited by training with generalist data, we subsequently
examine three target tasks that focus on distinct skill sets. Our results
suggest that general instruction tuning improves the model’s understanding and
reasoning capabilities. However, when it comes to tasks that demand factual
knowledge from the LLM, instructional data generated through self-instruct
Wang et al. (2022a) harms the model’s performance due to the intrinsic
hallucinations brought by such data creation approach.

In sum, to the best of our knowledge, our work is the first effort to present
a systematic guide for building and improving specialist models with general
instruction tuning.

##  2 Background: Instruction Tuning

In recent years, large language models (LLMs) have undergone rapid development
and have dominated the field of natural language processing (NLP) Radford et
al. (2018); Devlin et al. (2019); Radford et al. (2019); Brown et al. (2020).
Today’s LLMs, such as ChatGPT OpenAI (2022) and GPT-4 OpenAI (2023), can
perform complex and diverse tasks in the unified form of following natural
language instructions. Generally, these models are trained in three separate
stages: (1) large-scale unsupervised pre-training on raw text; and (2)
instruction tuning via supervised learning Sanh et al. (2022); Wei et al.
(2022a); Mishra et al. (2021); Su and Collier (2022); Su et al. (2023b); and
(3) reinforcement learning from human feedback Stiennon et al. (2020); Bai et
al. (2022); Ouyang et al. (2022). Recent studies Zhou et al. (2023); Gudibande
et al. (2023) argued that almost all capabilities of LLMs are learned during
unsupervised pre-training, and instruction tuning with a limited amount of
supervised data is sufficient. However, this observation refers to the process
of constructing general-purpose instruction-following models—generalists. In
the following, we separately introduce broad-coverage “generalist” and task-
specific “specialist” instruction tuning.

#### Generalist Instruction Tuning.

Early attempts on instruction tuning (Wang et al., 2022b; Sanh et al., 2022;
Wei et al., 2022a; Chung et al., 2022, inter alia) transform a range of public
NLP datasets into an instructional format, with a few manually crafted
templates for each task. They then fine-tune an LLM on a portion of the
transformed data and evaluate on another set of held-out tasks. Each work
affirms that the model’s generalization ability to unseen tasks improves when
increasing the task and template diversity. However, template-based
instructions are not sufficiently diverse for building a truly competent
generalist Ouyang et al. (2022). In contrast, state-of-the-art generalist
models such as ChatGPT OpenAI (2022) are trained with proprietary instructions
collected from real human users. In the pursuit to replicate the success of
ChatGPT, various open-source broad-coverage instruction-tuning datasets are
proposed. Some are gathered via crowd-sourcing Labs (2023); Zhou et al. (2023)
while others use the outputs from strong proprietary models Taori et al.
(2023); Peng et al. (2023); Xu et al. (2023); Su et al. (2023a); Li et al.
(2023) with techniques such as self-instruct Wang et al. (2022a). Existing
results suggest that these models can achieve near parity with proprietary
models in various aspects Chiang et al. (2023); Zhou et al. (2023); Taori et
al. (2023).

#### Specialist Instruction Tuning.

There is also an emerging trend to continue instruction tuning on specific NLP
tasks, such as machine translation Jiao et al. (2023a), information extraction
Wang et al. (2023b), medical QA Wang et al. (2023a); Fleming et al. (2023),
and writing-assistant Zhang et al. (2023). These works typically transform
existing task-specific datasets into the same instructional format as
generalist instruction tuning and yield better model performance in specific
tasks. Different from previous work, this study aims to provide a
comprehensive and in-depth investigation of the role of generalist instruction
data in specialist instruction tuning.

Our work is most related to the initial studies on the cross-task
generalization of instruction tuning such as FLAN Wei et al. (2022a). The
differences between our work and previous work are: (1) we use broad-coverage
generalist data, while they use template-based data; and (2) they focus on
zero/few-shot performance on unseen tasks, while we assume an adequate amount
of task-specific training data is available.

##  3 Incorporating Specialist Training with Generalist Training

###  3.1 Data Collection

We sort the instruction-following data into two groups: (1) specialist data
and (2) generalist data.

#### Specialist data.

primarily originates from existing NLP datasets with a focus on particular
tasks. To facilitate our research, we mainly utilize the SuperNI dataset Wang
et al. (2022b), a comprehensive benchmark containing 1,616 NLP datasets
coupled with their respective natural language instructions, as the source of
specialist data. The details are described in Section 4.1. We also leverage
existing question answering datasets Kwiatkowski et al. (2019); Berant et al.
(2013); Joshi et al. (2017) , reading comprehension datasets Lai et al. (2017)
reasoning datasets Bowman et al. (2015); Talmor et al. (2019); Ling et al.
(2017) to evaluate different aspects of model skills, detailed in Section 5.1.

#### Generalist data

is characterized by its extensive scope and diversity. For our research, we
select two representative broad-coverage general-purpose datasets:
GPT4-Instruct Peng et al. (2023) and LIMA Zhou et al. (2023). GPT4-Instruct
Peng et al. (2023) contains 52k unique instruction-response pairs, where the
instructions are collected through self-instruct Wang et al. (2022a) and the
responses are generated by GPT-4 OpenAI (2023). LIMA Zhou et al. (2023)
consists of 1k carefully curated instruction-response pairs derived from
human-authored community questions and answers. Notably, we emphasize that
GPT4-Instruct serves as an example of generalist data synthesized by LLMs and
LIMA represents another distinct example of generalist data written by humans.

#### Unified Format.

We follow the template used in Stanford’s Alpaca project Taori et al. (2023)
(See Appendix A). Each instance in the generalist and specialist data is
transformed in a pair of {instruction, response}.

###  3.2 Training

#### Specialist/Generalist Data Combination.

For each target task, we construct the training and test set with 50k and 5k
instances, respectively. For target tasks that span over multiple datasets, we
uniformly sample training/test instances from the corresponding datasets such
that each dataset has an equal proportion. For generalist data, we consider
the GPT4-Instruct and LIMA datasets as discussed above. We first train models
on generalist data and then specialist data. We vary the amounts of specialist
data across {2k, 4k, 6k, 8k, 10k} to study the effect of generalist data under
different circumstances of data scarcity.

#### Model and Training Details.

We conduct our experiments with the popular LLaMA 7B and 13B models Touvron et
al. (2023). For training on generalist data, we follow the original setups in
the respective papers Zhou et al. (2023); Taori et al. (2023). Specifically,
for GPT4-Instruct, we train for 3 epochs with a batch size of 128, while for
LIMA, we train for 15 epochs with a batch size of 64. In the subsequent
specialist training phase, we train for 3 epochs with a batch size of 128. In
both stages, we use the Adam optimizer Kingma and Ba (2015) with a learning
rate of 2e-5 and utilize the standard language modeling objective:

$\mathcal{L}=-\frac{1}{|\bm{y}|}\sum_{i=1}^{|\bm{y}|}\log
p_{\theta}(y_{i}|\bm{x},\bm{y}_{<i}),$

where $\theta$ denotes the model parameters and $\\{\bm{x},\bm{y}\\}$ is an
instruction-response pair.

##  4 Experiments I: The Coverage of the Target Tasks

Figure 1: Comparison of models trained with different combinations of
specialist and generalist data across different tasks. We report Rouge-L for
SuperNI and accuracy for other levels.

###  4.1 Coverage Taxonomy

To assess our model’s performance on a variety of target tasks with distinct
levels of generality, we construct a hierarchy of four specialist tasks using
the SuperNI dataset Wang et al. (2022b). This taxonomy encompasses tasks with
varying scopes of coverage, as detailed below.

#### SuperNI

(multiple tasks, multiple formats). At the most comprehensive level, we
incorporate all the English tasks from the SuperNI dataset, which encompasses
a total of 756 datasets. Unlike LIMA and GPT4-Instruct, which accommodate a
broad spectrum of user-oriented inquiries, the datasets in SuperNI focus on
specific NLP tasks distilled from real-world demands. Therefore, we treat them
as specialist data at the highest coverage level.

#### Classification

(multiple tasks, single format). The tasks in SuperNI can be grouped based on
their task types, such as classification, summarization, and question
answering. For the second level, we focus on the classification subset.
Specifically, we select 252 classification datasets. To measure the model’s
cross-task generalization capability, we allocate 223 datasets for training
and reserve the remaining 29 datasets as held-out datasets for evaluation.

#### Sentiment

(single tasks, multiple domains). The classification tasks selected above can
be further categorized based on their specific topics, such as sentiment
analysis, toxic language detection, commonsense categorization, and others.
Among these, we designate 32 sentiment analysis datasets as the third level.

#### Yelp

(single tasks, single domain). The sentiment analysis datasets mentioned above
span various domains, such as movie and restaurant reviews. At the most fine-
grained level, we choose the Yelp dataset (Zhang et al., 2015) as the
representative task to evaluate the model’s performance in a highly
specialized domain.

Task Coverage | GPT4-Instruct | LIMA | specialist
---|---|---|---
SuperNI | 25.54 | 12.65 | 54.92
Classification | 53.20 | 46.84 | 80.02
Sentiment | 68.66 | 51.46 | 90.71
Yelp | 91.68 | 65.52 | 98.11
Table 1: The performance of generalists and specialists on tasks of different
coverage levels on LLaMA-7B. The specialists are trained with 10k task-
specific instances. For SuperNI, the performance is measured by Rouge-L, while
the others are measured by accuracy.

###  4.2 Evaluation Setup

For the SuperNI level, we follow the same evaluation protocol as in Wang et
al. (2022b) and report Rouge-L Lin (2004). For the decoding strategy, we adopt
greedy search with a maximum generation length of 512.111We leave the study on
more advanced decoding methods (Holtzman et al., 2019; Su et al., 2022; Yang
et al., 2023) as future work. For the Classification, Sentiment, and Yelp
levels, we follow previous studies Brown et al. (2020); Sanh et al. (2022) and
utilize a classification with options approach, where we prompt the model with
a set of options and compute the likelihood of each option being the response.
The one with the highest probability is taken as the model’s prediction, and
we report the model’s accuracy.

###  4.3 Main Results

#### Generalist models lag behind specialist models across all coverage
levels.

We compare generalist models that are solely trained on generalist data (i.e.,
LIMA or GPT4-Instruct) to those specialist models that are solely trained on
specialist data (the 10k training instances we collect for each coverage
level), using LLaMA-7B. From the results presented in Table 1, we can see that
generalist models fall short in performance when compared to specialist models
on all coverage levels. Notably, even as the coverage level becomes more
encompassing, the performance gap between generalist models and specialist
models does not shrink. For instance, on the most specific Yelp task, the
specialist model outperforms the generalist model (GPT4-Instruct) by 6.43%
absolute points. On the SuperNI task, the performance gap between the
specialist and the generalist (GPT4-Instruct) is 29.38. These results validate
the necessity of specialist tuning for specific NLP tasks.

#### Transforming an LLM into a superior specialist demands a substantial
amount of task-specific data.

Figure 1 depicts the performance of specialist models on different tasks with
varying numbers of training data (from 2k to 10k). From the results, we see
that, for tasks with broader coverage (e.g. SuperNI and Classification), the
model’s performance does not seem to converge with the 10k training instances.
Even for narrow tasks such as Sentiment, at least 10k task-specific data is
required to fully unlock the LLM’s potential. These results reveal the data-
hungry nature of building specialist models.

Figure 2: Results on held-out tasks (Classification) with LLaMA-7B.

#### Generalist data can improve specialist performance when the task coverage
is broad.

Figure 1 also demonstrates that the inclusion of generalist data consistently
results in performance improvements for both SuperNI and Classification across
LLaMA 7B and 13B models. On average across different settings of specialist
data, the introduction of generalist data leads to an improvement of 0.96 for
LLaMA-7B and 0.74 for LLaMA-13B on SuperNI tasks, while for Classification
tasks, it results in an enhancement of 1.53% for LLaMA-7B and 0.82% for
LLaMA-13B. It is also worth noting that LIMA only has 1k instances, but it can
even help improve performance when the number of specialist data is 10$\times$
larger. However, the results are the opposite for Sentiment and Yelp. For
instance, the introduction of LIMA leads to a minor performance degeneration
on Sentiment with 2k specialist data (a reduction of 0.25% for LLaMA-7B and
0.56% for LLaMA-13B). In the case of the Yelp task, the impact of including
generalist data (both GPT4-Instruct and LIMA) appears to be minimal on the
overall model performance.

#### The performance gain is most evident when the amount of specialist data
is limited.

We can see that the performance gap between specialists trained with and
without generalist data shrinks as the amount of specialist data increases.
For example, at the Classification level, when the specialist data comprises
only 2k instances, the inclusion of GPT4-Instruct enhances LLaMA-7B’s accuracy
from 65.36% to 71.31% (+5.95%) and LLaMA-13B’s accuracy from 70.59% to 73.13%
(+2.54%). However, when the number of specialist data reaches 10k instances,
the addition of GPT4-Instruct only leads to smaller improvements, from 80.02%
to 80.17% (+0.15%) for LLaMA-7B, and from 81.01% to 81.93% (+0.92%) for
LLaMA-13B, respectively.

#### The performance gain is less pronounced when the model scale is larger.

As shown in Figure 1, when comparing the results of the 7B and 13B models, the
trend of change in the effect of integrating generalist data is consistent for
both models. However, it is worth noting that as the model scale is larger,
the performance gain is less pronounced. Specifically, when the model scales
up from 7B to 13B, the average improvement achieved by adding GPT4-Instruct on
SuperNI decreases from 1.49 to 0.58, and the improvement in Classification
reduces from 2.00% to 1.18%.

Figure 3: Results using different amounts of generalist data (Classification,
10k specialist data) with LLaMA-7B.

###  4.4 Further Analysis

For a deeper understanding of the impact of generalist data, here we present
additional analyses. Unless otherwise specified, all experiments use LLaMA-7B
as the foundation model.

#### Cross-task Generalization.

For the Classification level, recall that we exclude some classification tasks
when constructing the training data. These tasks can be used as hold-out tasks
to examine the specialist’s cross-task generalization ability. The results are
shown in Figure 2. It can be observed that the accuracy on held-out tasks
fluctuates in small ranges from 50.98% to 57.55% across different amounts of
specialist data. However, upon incorporating LIMA, the average absolute
accuracy improvement on the hold-out task increases by 2.70%, while adding
GPT4-Instruct results in a 6.12% rise in absolute accuracy. This indicates
that generalist data can greatly improve the cross-task generalization of
specialist models.

#### Number of Generalist Data.

To study the effect of the amount of generalist data, we additionally
partition the GPT4-Instruct dataset into five random parts and test the
model’s performance when using different proportions of the dataset. The
experiments are conducted at the Classification level with a fixed quantity of
10k specialist data. As shown in Figure 3, even with only 10k generalist data,
the model’s accuracy is raised from 78.12% to 82.48%. Another interesting
finding is that further increasing the generalist data to 50k merely brings
small improvements (from 82.48% to 84.0%). The results together with our
experiments with LIMA suggest that adding a small number of generalist data is
sufficient to improve the specialist performance.

Figure 4: The results using different test instructions (Classification) with
LLaMA-7B.

#### Cross-instruction Robustness.

In all previous experiments, the models are trained and tested using the same
instructions for each dataset. Now, we assess the model’s robustness when
confronted with alternative instructions that have not appeared during
training. To do so, we employ ChatGPT OpenAI (2022) to generate 20
semantically equivalent instructions based on the original instruction. Figure
4 reports the results of these unseen instructions. As seen, the models
trained with the addition of generalist data exhibit substantial improvement
in average accuracy compared to the models trained with specialist data only.
For instance, when the specialist data is limited to 2k instances,
incorporating generalist data leads to a 6.64% absolute improvement on average
compared to the specialist model. In the meantime, the incorporation of
generalist data also alleviates the performance variation between the best-
performing and worse-performing runs from 4.04% to 2.42%.

##  5 Experiments II: The Required Skills of the Target Tasks

We hypothesize that the model’s ability to perform specific NLP tasks can be
attributed to the mix of several core capabilities. As such, we set up three
target tasks that focus on three key skills which are detailed below.

###  5.1 Skill Taxonomy

Figure 5: Comparison of models trained with different combinations of specialist and generalist data across different tasks. We report F1 score for Factual Knowledge and accuracy for other levels. Task Coverage | GPT4-Instruct | LIMA | specialist
---|---|---|---
Factual Knowledge | 14.78 | 17.28 | 46.37
Understanding | 35.10 | 30.82 | 76.77
Reasoning | 28.02 | 26.58 | 63.40
Table 2: The performance of generalists and specialists on tasks focusing on
different skills. The specialists are trained with 10k task-specific instances
on LLaMA-7B. For Factual Knowledge, the performance is measured by F1 score,
while the others are measured by accuracy.

#### Factual Knowledge

is essential for models to serve information needs. We use three knowledge-
intensive datasets: Natural Questions Kwiatkowski et al. (2019), WebQuestions
Berant et al. (2013), and TriviaQA Joshi et al. (2017). All these three
datasets consist of entity-centric questions, making them suitable for probing
models’ ability to activate and utilize factual knowledge. Following previous
work Brown et al. (2020), we evaluate under the closed-book setting where
models are required to answer questions without the help of any external
knowledge grounding.

#### Understanding

acts as an important perspective as the capability to interpret input text. We
choose the RACE dataset Lai et al. (2017). RACE comprises data collected from
English examinations in China and is specifically designed to assess the
model’s ability to read and deeply comprehend texts in real-world scenarios.

#### Reasoning

is another fundamental ability for models to solve complex tasks. We use the
SNLI Bowman et al. (2015) dataset for implicit reasoning, the CQA Talmor et
al. (2019) for commonsense reasoning, and the AQUA Ling et al. (2017) dataset
for arithmetic reasoning.

###  5.2 Evaluation Setup

For the Factual Knowledge tasks, we use greedy search with a maximum
generation length of 512. We adopt the F1 score as the evaluation metric
following Brown et al. (2020). For the Understating and Reasoning tasks, we
utilize the same classification with options method detailed in Section 3.2
and report the model accuracy.

###  5.3 Results and Analysis

#### Generalist models lag behind specialist models across all task skills.

Similar to Experiment I, we commence by comparing specialist and generalist
models across three target tasks, each concentrating on distinct skills. The
outcomes presented in Table 2 indicate that the generalist models consistently
underperform the specialist models. For the Factual Knowledge task, the
specialist model outperforms the generalist model with a 29.09 points higher
F1 score. For the Understanding task, the specialist model surpasses the
generalist model with a 41.67% increase in accuracy. For the Reasoning task,
the specialist model excels beyond the generalist model, attaining a 35.38%
absolute accuracy difference. Collectively, these findings substantiate the
necessity of specialist tuning for accomplishing specific tasks.

#### Incorporating GPT4-Instruct impairs the model’s factual knowledge, while
integrating LIMA offers benefits.

As illustrated in Figure 5, we observe the varying impact of different
generalist data on the model’s performance in the Factual Knowledge task. In
particular, when GPT4-Instruct is incorporated, the F1 score experiences a
decline. Conversely, when LIMA data is integrated, the F1 witnesses an
increase. We argue that this difference stems from the fact that GPT4-Instruct
is machine-generated, while LIMA is human-authored. The rationale is that
machine-generated data may contain hallucinations, thus impairing the model’s
ability to recall factual knowledge.

To validate our hypothesis, we conduct experiments using additional generalist
datasets, namely Dolly Labs (2023), and Evol-Instruct Xu et al. (2023). Dolly
consists of manually curated data generated by Databricks employees. Evol-
Instruct uses more complex instructions than GPT4-Instruct and collects
responses from ChatGPT Fang et al. (2023). As observed in Figure 6, adding
Dolly does not impair the performance, but incorporating Evol-Instruct leads
to similar performance degradation as GPT4-Instruct. The above results are
consistent with our hypothesis that machine-generated generalist data might
adversely affect the model’s factual knowledgeability due to hallucinations.

For a more rigorous comparison, we use ChatGPT to generate responses for the
1k instructions in LIMA. The new 1k instruction-response pairs form a new
generalist dataset, which we call LIMA-Chat. The only difference between LIMA-
Chat and LIMA is that the responses in LIMA-Chat are machine-generated, while
those in LIMA are human-written. From Figure 6, we can see that LIMA-Chat
indeed harms the performance while LIMA improves the performance. The above
results suggest that the choice of generalist data is crucial for target tasks
that heavily rely on factual knowledge.

Figure 6: Results on the Factual Knowledge with LLaMA-7B.

#### Adding generalist data enhances the understanding ability.

The results of the Understanding task are presented in Figure 5. It is evident
that the addition of GPT4-Instruct greatly improves the model’s performance
when the specialist data is only 2k or 4k instances. However, as the number of
specialist data further increases, the improvement diminishes. This suggests
that the inclusion of generalist data can enhance the model’s comprehension
ability when the specialist data is limited.

#### Adding generalist data enhances the reasoning ability.

We further evaluate the consequences of incorporating generalist data on the
model’s reasoning ability, as demonstrated in Figure 5. Notably, unlike
Understanding, where the improvements from adding generalist data gradually
diminish, the benefits of incorporating generalist data on the Reasoning tasks
are persistent across different amounts of specialist data (an average
improvement of 0.65% on LLaMA-7B and 1.12% on LLaMA-13B). This phenomenon
could be attributed to the fact that the activation of reasoning capabilities
relies on diverse instruction data, and specialist data can be too narrow to
fully unlock the true potential of LLMs.

#### Effect of Model Scale.

For Factual Knowledge, increasing the model size from 7B to 13B results in
more substantial performance improvements compared to increasing the amount of
specialist data. This observation aligns with previous work Brown et al.
(2020), which indicates that an LLM’s knowledge is mostly obtained through its
pre-training. For Understanding, increasing the model size is as beneficial as
adding more specialist data. For Reasoning, increasing the model size does not
yield improvements as noticeable as Factual Knowledge and Understanding. We
speculate that the emergence of strong reasoning abilities requires a larger
model scale Wei et al. (2022b).

#### Generalist data plays a vital role in enhancing a model’s understanding
and reasoning capabilities, thereby increasing its effectiveness in addressing
task-specific objectives.

We dissect the model’s capabilities into three core components: (i) factual
knowledge, (ii) understanding, and (iii) reasoning abilities. We demonstrate
that incorporating generalist data does not improve the model’s factual
knowledge and, in some cases, may even be detrimental if it includes
hallucinated information. Nevertheless, comparative experiments focusing on
understanding and reasoning abilities reveal that generalist data effectively
fosters the model’s comprehension and significantly augments its reasoning
capabilities.

This observed efficacy can be ascribed to the capacity of generalist data to
facilitate the model’s understanding and execution of diverse tasks. The wide
range of instructions embedded within the generalist data stimulates the
model’s comprehension and reasoning faculties, empowering it to grasp specific
requirements associated with various tasks more effectively. Moreover, by
activating the model’s reasoning abilities, it showcases enhanced performance
across an assortment of tasks involving different levels of complexity.

The activation of comprehension and reasoning abilities further broadens the
model’s cognitive capacity, allowing it to derive a more comprehensive
understanding based on existing information pertinent to the given task.
Consequently, the inclusion of generalist data amplifies the model’s task-
specific capabilities, as it becomes adept at utilizing its expanded cognitive
capacity to achieve superior performance.

##  6 Conclusions

In this study, we thoroughly investigated the interaction between specialist
data and generalist data in the context of targeting specific NLP tasks. Our
findings consistently demonstrate that the addition of generalist data leads
to performance improvement when the task coverage is broad. This highlights
the potential benefits of incorporating generalist data, particularly when the
availability of specialist data is limited. Furthermore, we extensively
examined the impact of integrating generalist data on the model’s core
capabilities. Surprisingly, we observed that the inclusion of generalist data
did not enhance the model’s factuality. In fact, generalist data containing
hallucinatory information can have a negative impact. On the other hand, our
experiments also revealed that the introduction of generalist data has
positive effects on the model’s understanding and reasoning abilities.
Overall, our findings highlight the importance of leveraging generalist data
to enhance the understanding and reasoning capabilities of NLP models, thereby
enabling them to tackle various tasks more effectively. However, careful
consideration should be given to the quality and reliability of the generalist
data to avoid adverse effects on the model’s factual knowledge.

## Limitations

While this work aims to provide a comprehensive investigation, we note that we
do not exhaustively cover all possible evaluations. For example, we do not
discuss NLP tasks such as summarization, translation, etc. Instead, we focus
on constructing a hierarchy of four target tasks of different coverage levels
and three target tasks focusing on different core skills. In addition, due to
resource constraints, we only use LLaMA 7B/13B as our foundation models. We
leave the investigation on different types and scales of models to our future
work.

## Acknowledgments

This work was partly supported by the National Key Research and Development
Program of China (No.2020YFB1708200), and the Shenzhen Science and Technology
Program (JSGG20220831110203007).

## References

  * Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _ArXiv preprint_ , abs/2204.05862.
  * Berant et al. (2013) Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013.  Semantic parsing on Freebase from question-answer pairs.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Cheng et al. (2023) Daixuan Cheng, Shaohan Huang, and Furu Wei. 2023.  Adapting large language models via reading comprehension.  _ArXiv preprint_ , abs/2309.09530.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022.  Scaling instruction-finetuned language models.  _ArXiv preprint_ , abs/2210.11416.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_.
  * Fang et al. (2023) Tao Fang, Shu Yang, Kaixin Lan, Derek F Wong, Jinpeng Hu, Lidia S Chao, and Yue Zhang. 2023.  Is chatgpt a highly fluent grammatical error correction system? a comprehensive evaluation.  _ArXiv preprint_ , abs/2304.01746.
  * Fleming et al. (2023) Scott L Fleming, Alejandro Lozano, William J Haberkorn, Jenelle A Jindal, Eduardo P Reis, Rahul Thapa, Louis Blankemeier, Julian Z Genkins, Ethan Steinberg, Ashwin Nayak, et al. 2023.  Medalign: A clinician-generated dataset for instruction following with electronic medical records.  _ArXiv preprint_ , abs/2308.14089.
  * Gudibande et al. (2023) Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023.  The false promise of imitating proprietary llms.  _ArXiv preprint_ , abs/2305.15717.
  * Holtzman et al. (2019) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019.  The curious case of neural text degeneration.  _arXiv preprint arXiv:1904.09751_.
  * Jiao et al. (2023a) Wenxiang Jiao, Jen-tse Huang, Wenxuan Wang, Xing Wang, Shuming Shi, and Zhaopeng Tu. 2023a.  Parrot: Translating during chat using large language models.  _ArXiv preprint_ , abs/2304.02426.
  * Jiao et al. (2023b) Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023b.  Is chatgpt A good translator? A preliminary study.  _ArXiv preprint_ , abs/2301.08745.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7\.
  * Labs (2023) Databricks Labs. 2023.  Dolly: A tool for data versioning and dataops in databricks.  https://github.com/databrickslabs/dolly.  Accessed on June 8, 2023.
  * Lai et al. (2017) Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017.  RACE: Large-scale ReAding comprehension dataset from examinations.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_.
  * Li et al. (2023) Siheng Li, Cheng Yang, Yichun Yin, Xinyu Zhu, Zesen Cheng, Lifeng Shang, Xin Jiang, Qun Liu, and Yujiu Yang. 2023.  AutoConv: Automatically generating information-seeking conversations with large language models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_.
  * Ling et al. (2017) Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017.  Program induction by rationale generation: Learning to solve and explain algebraic word problems.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
  * Liu et al. (2023) Aiwei Liu, Xuming Hu, Lijie Wen, and Philip S Yu. 2023.  A comprehensive evaluation of chatgpt’s zero-shot text-to-sql capability.  _ArXiv preprint_ , abs/2303.13547.
  * Mishra et al. (2021) Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2021.  Natural instructions: Benchmarking generalization to new tasks from natural language instructions.  _ArXiv preprint_ , abs/2104.08773.
  * OpenAI (2022) OpenAI. 2022.  Introducing chatgpt.  https://openai.com/blog/chatgpt.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 technical report.  _ArXiv preprint_ , abs/2303.08774.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35.
  * Peng et al. (2023) Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023.  Instruction tuning with GPT-4.  _ArXiv preprint_ , abs/2304.03277.
  * Qin et al. (2023) Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023.  Is chatgpt a general-purpose natural language processing task solver?  _ArXiv preprint_ , abs/2302.06476.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018.  Improving language understanding by generative pre-training.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8).
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022.  Multitask prompted training enables zero-shot task generalization.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_.
  * Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. 2020.  Learning to summarize with human feedback.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Su and Collier (2022) Yixuan Su and Nigel Collier. 2022.  Contrastive search is what you need for neural text generation.  _arXiv preprint arXiv:2210.14140_.
  * Su et al. (2023a) Yixuan Su, Tian Lan, and Deng Cai. 2023a.  Openalpaca: A fully open-source instruction-following model based on openllama.  https://github.com/yxuansu/OpenAlpaca.
  * Su et al. (2023b) Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. 2023b.  Pandagpt: One model to instruction-follow them all.  _ArXiv preprint_ , abs/2305.16355.
  * Su et al. (2022) Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. 2022\.  A contrastive framework for neural text generation.  _Advances in Neural Information Processing Systems_ , 35:21548–21561.
  * Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019.  CommonsenseQA: A question answering challenge targeting commonsense knowledge.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  Llama: Open and efficient foundation language models.  _ArXiv preprint_ , abs/2302.13971.
  * Wang et al. (2023a) Haochun Wang, Chi Liu, Nuwa Xi, Zewen Qiang, Sendong Zhao, Bing Qin, and Ting Liu. 2023a.  Huatuo: Tuning llama model with chinese medical knowledge.  _ArXiv preprint_ , abs/2304.06975.
  * Wang et al. (2023b) Xiao Wang, Weikang Zhou, Can Zu, Han Xia, Tianze Chen, Yuansen Zhang, Rui Zheng, Junjie Ye, Qi Zhang, Tao Gui, Jihua Kang, Jingsheng Yang, Siyuan Li, and Chunsai Du. 2023b.  Instructuie: Multi-task instruction tuning for unified information extraction.  _ArXiv preprint_ , abs/2304.08085.
  * Wang et al. (2022a) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022a.  Self-instruct: Aligning language model with self generated instructions.  _ArXiv preprint_ , abs/2212.10560.
  * Wang et al. (2022b) Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022b.  Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_.
  * Wei et al. (2022a) Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022a.  Finetuned language models are zero-shot learners.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_.
  * Wei et al. (2022b) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022b.  Emergent abilities of large language models.  _Trans. Mach. Learn. Res._ , 2022.
  * Xu et al. (2023) Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023.  Wizardlm: Empowering large language models to follow complex instructions.  _ArXiv preprint_ , abs/2304.12244.
  * Yang et al. (2023) Haoran Yang, Deng Cai, Huayang Li, Wei Bi, Wai Lam, and Shuming Shi. 2023.  A frustratingly simple decoding method for neural text generation.  _arXiv preprint arXiv:2305.12675_.
  * Zhang et al. (2015) Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  In _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_.
  * Zhang et al. (2023) Yue Zhang, Leyang Cui, Deng Cai, Xinting Huang, Tao Fang, and Wei Bi. 2023.  Multi-task instruction tuning of llama for specific scenarios: A preliminary study on writing assistance.  _ArXiv preprint_ , abs/2305.13225.
  * Zhou et al. (2023) Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023.  Lima: Less is more for alignment.  _ArXiv preprint_ , abs/2305.11206.

##  Appendix A Instruction Template

============ Instruction Format =========== Below is an instruction that
describes a task, paired with an input that provides further context. Write a
response that appropriately completes the request. $\\#\\#\\#$Instruction:
[Task Prompt] $\\#\\#\\#$Input: [Input Text] $\\#\\#\\#$Response: [Output
Text]


# Univalency of certain transform of univalent functions

Milutin Obradović  Department of Mathematics, Faculty of Civil Engineering,
University of Belgrade, Bulevar Kralja Aleksandra 73, 11000, Belgrade, Serbia
<EMAIL_ADDRESS>and  Nikola Tuneski  Department of Mathematics and
Informatics, Faculty of Mechanical Engineering, Ss. Cyril and Methodius
University in Skopje, Karpoš II b.b., 1000 Skopje, Republic of North
Macedonia<EMAIL_ADDRESS>
###### Abstract.

We consider univalency problem in the unit disc ${\mathbb{D}}$ of the function

$g(z)=\frac{(z/f(z))-1}{-a_{2}},$

where $f$ belongs to some classes of univalent functions in ${\mathbb{D}}$ and
$a_{2}=\frac{f^{\prime\prime}(0)}{2}\neq 0$.

###### Key words and phrases:

analytic, univalent, transform

###### 2000 Mathematics Subject Classification:

30C45

##  1\. Introduction

Let ${\mathcal{A}}$ denote the family of all analytic functions $f$ in the
unit disk ${\mathbb{D}}:=\\{z\in{\mathbb{C}}:\,|z|<1\\}$ satisfying the
normalization $f(0)=0=f^{\prime}(0)-1$, i.e., $f$ has the form

(1) $f(z)=z+a_{2}z^{2}+a_{3}z^{3}+\ldots.$

Let $\mathcal{S}$, $\mathcal{S}\subset\mathcal{A}$, denote the class of
univalent functions in ${\mathbb{D}}$, let $\mathcal{S}^{\star}$ be the
subclass of ${\mathcal{A}}$ (and $\mathcal{S}$ which are starlike in
${\mathbb{D}}$ and let $\mathcal{U}$ denote the set of all $f\in{\mathcal{A}}$
satisfying the condition

(2) $\left|\operatorname{U}_{f}(z)\right|<1\qquad(z\in{\mathbb{D}}),$

where

(3) $\operatorname{U}_{f}(z):=\left(\frac{z}{f(z)}\right)^{2}f^{\prime}(z)-1.$

In [5, Theorem 4] the authors consider the problem of univalency for the
function

(4) $g(z)=\frac{(z/f(z))-1}{-a_{2}},$

where $f\in\mathcal{U}$ has the form (1) with $a_{2}\neq 0$. They proved the
following

Theorem A. Let $f\in\mathcal{U}$. Then, for the function $g$ defined by the
expression (4) we have

  * (a)

$|g^{\prime}(z)-1|<1$ for $|z|<|a_{2}|/2$;

  * (b)

$g\in\mathcal{S}^{\star}$ in the disk $|z|<|a_{2}|/2$, and even more

$\left|\frac{zg^{\prime}(z)}{g(z)}-1\right|<1$

in the same disk ;

  * (c)

$g\in\mathcal{U}$ in the disk $|z|<|a_{2}|/2$ if $0<|a_{2}|\leq 1.$

These results are the best possible.

For the proof of the previous theorem the authors used the next representation
for the class $\mathcal{U}$ (see [2] and [4]). Namely, if $f\in\mathcal{U}$
then

(5) $\frac{z}{f(z)}=1-a_{2}z-z\omega(z),$

where function $\omega$ is analytic in ${\mathbb{D}}$ with
$|\omega(z)|\leq|z|<1$ for all $z\in{\mathbb{D}}$. The appropriate function
$g$ from (4) has the form

(6) $g(z)=z+\frac{1}{a_{2}}z\omega(z).$

##  2\. Results

In this paper we consider other cases of Theorem A(c) and certain related
results.

######  Theorem 1.

Let $f\in\mathcal{U}$. Then the function $g$ defined by the equation (4)
belongs to $\mathcal{U}$ in the disc

$|z|<\sqrt{\frac{1-|a_{2}|+\sqrt{|a_{2}|^{2}+2|a_{2}|-3}}{2}},$

i.e., satisfies (2) on this disc, if $\frac{5}{4}\leq|a_{2}|\leq 2$.

###### Proof.

For the first part of the proof we use the same method as in [5]. By the
definition of the class $\mathcal{U},$ i.e., inequality (2), and using the
next estimation for the function $\omega$,

$|z\omega^{\prime}(z)-\omega(z)|\leq\frac{r^{2}-|\omega(z)|^{2}}{1-r^{2}},$

(where $|z|=r$ and $|\omega(z)|\leq r$), after some calculations, we obtain

$\begin{split}|\operatorname{U}_{g}(z)|&=\left|\frac{\frac{1}{a_{2}}\left[z\omega^{\prime}(z)-\omega(z)\right]-\frac{1}{a^{2}_{2}}\omega^{2}(z)}{\left[1+\frac{1}{a_{2}}\omega_{1}(z)\right]^{2}}\right|\\\
&\leq\frac{|a_{2}|\cdot|z\omega^{\prime}(z)-\omega(z)|+|\omega(z)|^{2}}{\left(|a_{2}|-|\omega(z)|\right)^{2}}\\\
&\leq\frac{|a_{2}|\cdot\frac{r^{2}-|\omega(z)|^{2}}{1-r^{2}}+|\omega(z)|^{2}}{\left(|a_{2}|-|\omega(z)|\right)^{2}}\\\
&=:\frac{1}{1-r^{2}}\cdot\varphi(t).\end{split}$

Here,

(7) $\varphi(t)=\frac{|a_{2}|r^{2}-(|a_{2}|-1+r^{2})t^{2}}{(|a_{2}|-t)^{2}}$

and $|\omega(z)|=t$, $0\leq t\leq r$. From here we have that

$\varphi^{\prime}(t)=\frac{2|a_{2}|}{(|a_{2}|-t)^{3}}\cdot\left[r^{2}-(|a_{2}|-1+r^{2})t\right]$

(where $|a_{2}|-t>0$ since $|a_{2}|\geq\frac{5}{4}>1>t$). Next,
$\varphi^{\prime}(t)=0$ for

$t_{0}=\frac{r^{2}}{|a_{2}|-1+r^{2}}$

and $0\leq t_{0}\leq r$ if

$\frac{r^{2}}{|a_{2}|-1+r^{2}}\leq r,$

which is equivalent to

$r^{2}-r+|a_{2}|-1\geq 0.$

The last relation is valid for $\frac{5}{4}\leq|a_{2}|\leq 2$ and every $0\leq
t<1$. It means that the maximal value of the function $\varphi$ on $[0,r]$ is

$\varphi(t_{0})=\frac{(|a_{2}|-1+r^{2})r^{2}}{(|a_{2}|-1)(|a_{2}|+r^{2})}.$

Finally,

$|\operatorname{U}_{g}(z)|\leq\frac{1}{1-r^{2}}\cdot\varphi(t_{0})=\frac{(|a_{2}|-1+r^{2})r^{2}}{(1-r^{2})(|a_{2}|-1)(|a_{2}|+r^{2})}<1$

if

$r^{4}-(1-|a_{2}|)r^{2}+(1-|a_{2}|)<0,$

or if

$r<\sqrt{\frac{1-|a_{2}|+\sqrt{|a_{2}|^{2}+2|a_{2}|-3}}{2}}.$

This completes the proof. ∎

For our next consideration we need the next lemma.

######  Lemma 1.

Let $f\in\mathcal{A}$ be of the form (1). If

(8) $\sum_{2}^{\infty}n|a_{n}|\leq 1,$

then

$\begin{split}|f^{\prime}(z)-1|&<1\qquad(z\in{\mathbb{D}}),\\\\[5.69054pt]
\left|\frac{zf^{\prime}(z)}{f(z)}-1\right|&<1\qquad(z\in{\mathbb{D}})\end{split}$

(i.e. $f\in\mathcal{S}^{\star}$ ), and $f\in\mathcal{U}$.

For the proof of $f\in\mathcal{U}$ in the lemma see [4], while the rest easily
follows.

Further, let $\mathcal{S}^{+}$ denote the class of univalent functions in the
unit disc with the representation

(9) $\frac{z}{f(z)}=1+b_{1}z+b_{2}z^{2}+\ldots,\quad b_{n}\geq
0,\,\,n=1,2,3,\ldots.$

For example, the Silverman class (the class with negative coefficients) is
included in the class $\mathcal{S}^{+}$, as well as the Koebe function
$k(z)=\frac{z}{(1+z)^{2}}\in\mathcal{S}^{+}$. The next characterization is
valid for the class $\mathcal{S}^{+}$ (for details see [3])

(10)
$f\in\mathcal{S}^{+}\quad\Leftrightarrow\quad\sum_{n=2}^{\infty}(n-1)b_{n}\leq
1.$

######  Theorem 2.

Let $f\in\mathcal{S}^{+}$. Then the function $g$ defined by (4) belongs to the
class $\mathcal{U}$ in the disc $|z|<|a_{2}|/2$ and the result is the best
possible.

###### Proof.

Using the representation (9), the corresponding function $g$ has the form

$g(z)=\frac{\frac{z}{f(z)}-1}{-a_{2}}=\frac{\frac{z}{f(z)}-1}{b_{1}}=z+\sum_{2}^{\infty}\frac{b_{n}}{b_{1}}z^{n}\quad(b_{1}\neq
0),$

and from here

$\frac{1}{r}g(rz)=z+\sum_{2}^{\infty}\frac{b_{n}}{b_{1}}r^{n-1}z^{n}\quad(0<r\leq
1).$

Then, after applying Lemma 1, we have

$\begin{split}\sum_{2}^{\infty}n|a_{n}|&=\sum_{2}^{\infty}n\frac{b_{n}}{b_{1}}r^{n-1}\\\
&=\frac{1}{b_{1}}\sum_{2}^{\infty}(n-1)b_{n}\frac{n}{n-1}r^{n-1}\\\
&\leq\frac{2r}{b_{1}}\sum_{2}^{\infty}(n-1)b_{n}\leq\frac{2r}{b_{1}}\leq
1\end{split}$

if $r\leq\frac{b_{1}}{2}=\frac{|a_{2}|}{2}$. It means, by the same lemma, that
$g\in\mathcal{U}$ in the disc $|z|<|a_{2}|/2.$

In order to show that the result is the best possible, let consider the
function $f_{1}$ defined by

(11) $\frac{z}{f_{1}(z)}=1+bz+z^{2},\quad 0<b\leq 2.$

Then, $f_{1}\in\mathcal{S}^{+}$ is of type $f_{1}(z)=z-bz^{2}+\cdots$, so the
function

$g_{1}(z)=\frac{\frac{z}{f_{1}(z)}-1}{b}=z+\frac{1}{b}z^{2}$

is such that

$\left|\left(\frac{z}{g_{1}(z)}\right)^{2}g^{\prime}_{1}(z)-1\right|\leq\frac{\frac{1}{b^{2}}|z|^{2}}{\left(1-\frac{1}{b}|z|\right)^{2}}<1$

when $|z|<b/2$. This implies that $g_{1}$ belongs to the class $\mathcal{U}$
in the disc $|z|<b/2$. On the other hand, since $g^{\prime}_{1}(-b/2)=0$, the
function $g_{1}$ is not univalent in a bigger disc, implying that the result
is the best possible. ∎

######  Theorem 3.

Let $f\in\mathcal{S}$. Then the function $g$ defined by (4) belongs to the
class $\mathcal{U}$ in the disc $|z|<r_{0}$, where $r_{0}$ is the unique real
root of the equation

(12) $\frac{3r^{2}-2r^{4}}{(1-r^{2})^{2}}-\ln(1-r^{2})=|a_{2}|^{2}$

on the interval $(0,1)$.

###### Proof.

We apply the same method as in the proof of the previous theorem. Namely, if
$f\in\mathcal{S}$ has the representation (9), then

(13) $\sum_{n=2}^{\infty}(n-1)|b_{n}|^{2}\leq 1$

(see [1], Theorem 11, p. 193, Vol. 2). Also, using (4), (9) and (13), we have
$a_{2}=-b_{1}$, and

$\frac{1}{r}g(rz)=z+\sum_{2}^{\infty}\frac{b_{n}}{b_{1}}r^{n-1}z^{n},\quad
0<r\leq 1.$

So,

$\begin{split}\sum_{n=2}^{\infty}n|a_{n}|&=\sum_{n=2}^{\infty}n\frac{|b_{n}|}{|b_{1}|}r^{n-1}\\\
&=\frac{1}{|b_{1}|}\sum_{n=2}^{\infty}\sqrt{n-1}\cdot|b_{n}|\cdot\frac{n}{\sqrt{n-1}}\cdot
r^{n-1}\\\
&\leq\frac{1}{|b_{1}|}\cdot\left(\sum_{n=2}^{\infty}(n-1)|b_{n}|^{2}\right)^{1/2}\cdot\left(\sum_{n=2}^{\infty}\frac{n^{2}}{n-1}r^{2(n-1)}\right)^{1/2}\\\
&\leq\frac{1}{|b_{1}|}\left(r^{2}\sum_{n=2}^{\infty}(n-1)(r^{2})^{n-2}+2r^{2}\sum_{n=2}^{\infty}(r^{2})^{n-2}+\sum_{n=2}^{\infty}\frac{1}{n-1}(r^{2})^{n-1}\right)^{1/2}\\\
&=\frac{1}{|b_{1}|}\left[\frac{3r^{2}-2r^{4}}{(1-r^{2})^{2}}-\ln(1-r^{2})\right]^{1/2}\leq
1\end{split}$

if $|z|<r_{0}$, where $r_{0}$ is the root of the equation

$\frac{3r^{2}-2r^{4}}{(1-r^{2})^{2}}-\ln(1-r^{2})=|b_{1}|^{2}(=|a_{2}|^{2}).$

We note that the function on the left side of this equation is an increasing
one on the interval $(0,1)$, so the equation has a unique root when
$0<|a_{2}|\leq 2.$ ∎

## References

  * [1] A. W. Goodman, Univalent functions, Vols. 1-2, Mariner, Tampa, Florida, 1983.
  * [2] Obradović M., Pascu N. N. and Radomir I., A class of univalent functions, Math. Japonica, 44(3) (1996), 565–568.
  * [3] M. Obradović and S. Ponnusamy, Coefficient characterization for certain classes of univalent functions, Bull. Belg. Math. Soc. (Simon Stevin) 16 (2009), 251–263.
  * [4] Obradović M., Ponnusamy S., On the class $\mathcal{U}$, Proc. 21st Annual Conference of the Jammu Math. Soc. and a National Seminar on Analysis and its Application, 11–26, 2011.
  * [5] Obradović M.,Tuneski N., Some properties of the class $\mathcal{U}$, Ann. Univ. Mariae Curie-Skłodowska Sect. A, 73(1) (2019), 45–56.


11institutetext: Technical University of Munich, Germany 22institutetext:
Helmholtz AI and Institute of Machine Learning in Biomedical Imaging, Munich,
Germany 33institutetext: Klinikum Rechts der Isar, Munich, Germany
44institutetext: Imperial College London, United Kingdom 55institutetext:
School of Biomedical Engineering and Imaging Sciences, King’s College London,
United Kingdom

# Unsupervised Analysis of Alzheimer’s Disease Signatures using 3D Deformable
Autoencoders

Mehmet Yigit Avci∗ 11 Emily Chan∗ 112255 Veronika Zimmer 11 Daniel Rueckert
113344 Benedikt Wiestler 33 Julia A. Schnabel 112255 Cosmin I. Bercea 1122

###### Abstract

With the increasing incidence of neurodegenerative diseases such as
Alzheimer’s Disease (AD), there is a need for further research that enhances
detection and monitoring of the diseases. We present _MORPHADE_ (Morphological
Autoencoders for Alzheimer’s Disease Detection), a novel unsupervised learning
approach which uses deformations to allow the analysis of 3D T1-weighted brain
images. To the best of our knowledge, this is the first use of deformations
with deep unsupervised learning to not only detect, but also localize and
assess the severity of structural changes in the brain due to AD. We obtain
markedly higher anomaly scores in clinically important areas of the brain in
subjects with AD compared to healthy controls, showcasing that our method is
able to effectively locate AD-related atrophy. We additionally observe a
visual correlation between the severity of atrophy highlighted in our anomaly
maps and medial temporal lobe atrophy scores evaluated by a clinical expert.
Finally, our method achieves an AUROC of 0.80 in detecting AD, out-performing
several supervised and unsupervised baselines. We believe our framework shows
promise as a tool towards improved understanding, monitoring and detection of
AD. To support further research and application, we have made our code
publicly available at github.com/ci-ber/MORPHADE.

###### Keywords:

Unsupervised learning Registration Classification

**footnotetext: These authors contributed equally to this work

##  1 Introduction

Due to the increased prevalence of neurodegenerative diseases and their
effects on cognitive function, the study of such diseases is a highly active
research field. As the leading cause of dementia [1], Alzheimer’s disease (AD)
is a particular focus of research advancements. However, the complex
pathogenesis and progression mechanisms of AD remain only partially
understood.

Magnetic resonance imaging (MRI) has shown use in the non-invasive tracking of
AD-associated brain changes, such as hippocampal and amygdala atrophy and
ventricular dilation [16, 11]. Notably, several supervised machine learning
methods utilizing MRI have been proposed which yield improvements in AD
identification [24, 15, 23]. However, such methods are restricted by the need
for large, annotated data sets. In contrast, unsupervised anomaly detection
techniques [7, 2, 21, 25] offer a promising solution by modeling the
distribution of healthy brain images to identify and localize anomalies
without relying on labeled data.

Nevertheless, unsupervised approaches face challenges in accurately analyzing
structural abnormalities, particularly regions of atrophy, which are critical
in AD research [5]. Classical techniques using multi atlas-based deformable
registration [13] and morphometry methods [8, 3] have been proposed to analyze
these structural changes. However, such methods allow analysis to be conducted
only on a population-level, for instance as deviations from an atlas.

In this work, we propose Morphological Autoencoders for Alzheimer’s Disease
Detection (MORPHADE), a novel unsupervised anomaly detection framework based
on deformable autoencoders (AEs) [4] which leverages deformation networks to
generate patient-specific anomaly maps from 3D T1-weighted MRI brain scans.
These anomaly maps allow not only AD detection, but also crucially reveal the
location and degree of atrophy. Our main contributions are as follows:

  * •

We use deformation fields in an unsupervised framework to analyze AD-related
changes in the brain. To the best of our knowledge, this is the first use of
such an approach using deep learning in the context of AD.

  * •

We extend deformable autoencoders to 3D, utilize adversarial training and
propose a dual-deformation strategy to improve reconstruction fidelity and the
localization of atrophy.

  * •

We accurately identify AD-affected brain regions, aligning our findings with
clinical expectations.

  * •

We assess AD severity by correlating our findings with clinical medial
temporal lobe atrophy scores, evaluated by a board-certified clinical expert.

  * •

Through comprehensive validation, we demonstrate superior performance in AD
detection compared to unsupervised and even supervised baselines.

##  2 Background

In unsupervised anomaly detection, reconstruction-based frameworks such as
autoencoders (AEs) can be used to learn the distribution of healthy samples
and subsequently identify samples that deviate from this norm as anomalous.
The encoder $E_{\theta}$ maps an input $x$ to a lower-dimensional latent space
and then the decoder $D_{\phi}$ learns to reconstruct from this encoded
representation. The parameters $\theta$, $\phi$ of the AE are optimized given
healthy input data $\chi=\\{x_{i},...,x_{n}\\}$ by minimizing the mean squared
error (MSE) between the inputs and their reconstructions:

$MSE=min_{\theta,\phi}\sum_{i=1}^{N}||x_{i}-D_{\phi}(E_{\theta}(x_{i}))||^{2}\enspace.$
(1)

It is then assumed that during inference, the AE will generate a so-called
pseudo-healthy reconstruction, in which only in-distribution healthy tissue
can be successfully reconstructed and thus any reconstruction errors can be
thought of as anomalies. A subject-specific map of anomalies can then be
obtained by taking the residual between an input $x$ and its reconstruction
$x_{recon}=D_{\phi}(E_{\theta}(x))$ as follows:

$m_{residual}=|x-x_{recon}|\enspace.$ (2)

Deformable Autoencoders (AEs) [4] were proposed as a method to alleviate false
positives in the anomaly maps due to the limited reconstruction capabilities
of traditional AEs. Since the top layers of the AE contain spatial
information, deformable AEs use these layers to estimate a dense deformation
field $\boldsymbol{\Phi}$ that allows local adaptions of the pseudo-healthy
reconstruction to the individual anatomy of the subject. The estimation of the
deformation field is optimized using local normalized cross correlation
(LNCC):

$\mathcal{L}_{morph}=LNCC(x,x_{morph})+{\beta}||\boldsymbol{\Phi}||^{2}\enspace,$
(3)

where $\beta$ is a weight that is kept relatively high to constrain the
deformations to be smooth and local, allowing only small changes to the
reconstructions. We therefore refer to this part of the network as the
constrained deformer. The improved reconstruction, which we refer to as the
morphed reconstruction, $x_{morph}$, can then be obtained by
$x_{morph}=x_{recon}\circ\boldsymbol{\Phi}$.

The authors also propose to use perceptual loss (PL) [12] weighted by the
hyperparameter $\alpha$, in addition to the MSE when optimizing the AE
parameters, to promote reconstructions that closely resemble the training
distribution:

$\mathcal{L}_{recon}=\text{MSE}(x,x_{recon})+\alpha\text{PL}(x,x_{recon})\enspace.$
(4)

##  3 Methods and Materials

Figure 1: Our approach, MORPHADE, integrates a dual-deformation strategy with
a 3D autoencoder and adversarial training. The constrained deformer refines
the reconstruction to generate a residual map with reduced false positives,
while the unconstrained deformer is used to produce a folding map that
highlights anomalies. The residual and folding maps together produce an
anomaly map that allows the localization and assessment of the severity of
atrophy.

We propose MORPHADE, shown in Fig. 1, which builds upon deformable AEs.
Firstly, we employ a 3D convolutional AE to enable the use of 3D images with
the framework. Secondly, since PL uses 2D networks pre-trained on ImageNet, we
employ an adversarial loss [9] to increase the realness of the
reconstructions. We train a discriminator by minimizing this adversarial loss;
therefore, the reconstruction loss becomes:

$\mathcal{L}_{recon}=\text{MSE}(x,x_{recon})+\gamma\text{Adversarial}(x,x_{recon})\enspace,$
(5)

where ${\gamma}$ balances the production of realistic reconstructions while
maintaining pixel-wise accuracy.

Our major extension to the deformable AEs is the use of a dual-deformation
strategy, in which we employ an unconstrained deformer in addition to the
constrained deformer, with the aim of improving the localization of atrophic
regions. As previously stated, the constrained deformer is trained with a high
value of $\beta$ to improve the generation of the pseudo-healthy
reconstructions and thus reduce false positives in the anomaly maps. In
contrast, the unconstrained deformer has the goal of reverting the pseudo-
healthy reconstruction back to its original anomalous state. The deformer is
trained with the same loss as in Eq. 3, but with a low value of $\beta$, which
allows the creation of unconstrained deformation fields. In such deformation
fields, low values of deformation should occur in areas of healthy tissue.
Conversely, in regions of atrophy, the deformation field exhibits foldings, or
areas in which the mapping of the deformation from the pseudo-healthy
reconstruction to the original image is not one-to-one due to the loss of
tissue volume. The determinant of the Jacobian of the deformation map,
$J_{\boldsymbol{\Phi}}$, can be used to determine local volume changes, with
negative values indicating such foldings. Therefore, we highlight the
anomalies by using the negative Jacobian values to generate a map of the
foldings, $m_{foldings}=\max(0,-\det(J_{\boldsymbol{\Phi}}))$.

We finally multiply these foldings pixel-wise with the residual map from the
constrained deformer to generate an anomaly map with reduced false positives
and improved atrophy localization:

$\text{Anomaly Map}=m_{residual}\times m_{foldings}\enspace.$ (6)

Implementation. All networks were trained with Adam optimizer. The
discriminator was trained with a learning rate of $1.0e^{-4}$, otherwise
$5.0e^{-4}$ was used. The framework was first trained with a high value of
$\beta=10$. We motivate this choice in Fig. 2a, where we show that using
decreasing values of $\beta$ during training results in blurrier
reconstructions. Conversely, a high $\beta$ value ensures that the AE does not
overly rely on the deformations to achieve faithful reconstructions, but is
instead forced to learn an accurate representation of the in-distribution
data. After 200 epochs, the weights of these models were kept frozen while the
deformation parameters were optimized for 100 epochs.

At inference, we use a high value of $\beta=10$ to obtain the residual maps
and a low value of $\beta=0.01$ to generate the folding maps. We demonstrate
the need for lower $\beta$ values to produce improved folding maps in Fig. 2b,
where it can be seen that using low values accentuates the anomalous regions
in the brain.

Figure 2: a) During training, a high value of $\beta=10$ constrains the
deformer, promoting the AE to learn to produce less blurry reconstructions. b)
At inference, a lower value of $\beta=0.01$ is used to generate folding maps
(here shown overlayed on the input brain) that enhance the identification of
anomalies.

Dataset and Preprocessing. Data used in the preparation of this article were
obtained from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database
(adni.loni.usc.edu) [19].We used skull-stripped T1-weighted MPRAGE images of
both male and female patients that are registered to the MNI brain template
[17]. Our training set comprised 760 healthy control (HC) samples, with an
additional 95 HC samples utilized for validation purposes. For the supervised
baseline training, an additional 430 AD samples were used. The test set
included 215 HC samples and 200 samples with AD.

##  4 Experiments and Results

Atrophy Localization. We first validate the effectiveness of our method in
identifying atrophy in sub-cortical brain regions affected by AD. To achieve
this, we used the FSL FIRST tool [18] to segment these regions and compute
mean anomaly scores for each, shown in Fig. 3. Our results indicate that AD
patients exhibit notably higher anomaly scores in the hippocampus (left: 0.282
$\pm$ 0.495, right: 0.185 $\pm$ 0.382) and amygdala (left: 0.132 $\pm$ 0.207,
right: 0.108 $\pm$ 0.208) compared to the hippocampus (left: 0.108 $\pm$
0.193, right: 0.069 $\pm$ 0.125) and amygdala (left: 0.066 $\pm$ 0.115, right:
0.072 $\pm$ 0.111) for the healthy controls. These results are in line with
the clinical expectation of these regions being significantly affected by AD
pathology [6], indicating that MORPHADE is able to identify atrophy in
clinically relevant brain regions.

Figure 3: Anomaly scores for subcortical brain regions for Alzheimer’s Disease
(AD) and Healthy Control (HC) samples, showcasing markedly higher scores for
AD samples in the hippocampus and amygdala, consistent with clinical
literature. [6]

Atrophy Severity. We next evaluate the ability of our method to determine the
severity of the localized anomalies by comparing our anomaly maps to medial
temporal lobe atrophy (MTA) scores [20] that were assessed by a senior board-
certified neuroradiologist. These scores range from 0 to 4 and are assigned
based on the degree of structural changes observed in the choroid fissure, the
temporal horn of the lateral ventricle, and the hippocampus. Fig. 4 shows a
visual correlation between the degree of atrophy highlighted in the anomaly
map in these key regions and the MTA scores, demonstrating the utility of our
method in determining the severity of detected anomalies.

Figure 4: Anomaly maps for AD patients alongside their corresponding medial
temporal lobe atrophy (MTA) scores, demonstrating consistent alignment with
AD-related structural changes and clinical MTA assessments.

Pathology Detection. In this section, we assess the capability of MORPHADE in
detecting AD at the patient level. Table 1 shows the Area Under the Receiver
Operating Characteristic curve (AUROC) scores obtained when comparing our
method to various baselines for identifying subjects with AD compared to
healthy control (HC) subjects. Our model achieves an AUROC of 0.80, surpassing
even the 3D supervised baselines ResNet [14] and DenseNet [10], with AUROCs of
0.77 and 0.74, respectively.

Furthermore, we obtain improved performance compared to methods proposed for
unsupervised anomaly detection. These methods are only available in 2D, so
were assessed slice-wise with the final anomaly scores obtained by averaging
over the slices for each patient. f-AnoGAN [21], Ganomaly [2] obtained AUROCs
of 0.70 and 0.72, respectively. We also outperform Brainomaly [22] (AUROC
0.78), a method that is not strictly unsupervised since it requires
pathological samples during training for improved performance.

We also compare our results to a 3D adversarial AE to illustrate the benefit
of utilizing the deformation fields with our method. Fig. 5 shows the
reconstructions and residual maps obtained for both methods in representative
AD and healthy controls (HC) subjects. Our method produces more refined
reconstructions compared to the adversarial AE, shown by the improved MAE and
SSIM scores. Moreover, the residual maps show fewer false positives for the
healthy subject, while accentuating pathological areas for the AD subject.
Using these improved residual maps alone for AD detection achieves a superior
performance of AUROC 0.77 compared to 0.74 obtained by the adversarial AE.

Finally, we demonstrate the utility of our dual-deformation approach, where AD
identification was superior using our method compared to using only the
residual maps from the constrained deformer (AUROC 0.77) or the folding maps
from the unconstrained deformer (AUROC 0.79). Notably, the high performance of
the folding maps underscores their effectiveness in detecting anomalies
without relying on image differences between the input and reconstructions.

Table 1: AUROC scores for the classification of AD and Healthy Controls (HC)
patients. Best results are shown in bold.

Method AD vs. HC $\uparrow$ ResNet (Supervised)[14] 0.77 DenseNet
(Supervised)[10] 0.74 Brainomaly [22] (Mixed Supervision) 0.78 f-AnoGAN [21]
(Unsupervised) 0.70 Ganomaly [2] (Unsupervised) 0.72 Adversarial AE
(Unsupervised) 0.74 MORPHADE (ours) (Unsupervised) $0.80$ \- Only with
residual maps ($\beta$=10) 0.77 \- Only with folding maps ($\beta$=0.01)
$0.79$

Figure 5: A comparison of the performance of MORPHADE ($\beta=10$) with
adversarial AEs for a subject with AD (left) and a healthy control subject
(right). The morphological adjustments facilitated by MORPHADE enhance
reconstruction fidelity, yielding higher Structure Similarity Index (SSIM)
values for our method’s morphed reconstructions compared to those of the
adversarial AE. The residual maps also demonstrate fewer reconstruction errors
for the healthy subject, while highlighting atrophy for the subject with AD.

##  5 Discussion and Conclusion

In this work, we introduced MORPHADE, a novel framework leveraging 3D
deformable AEs for unsupervised analysis of Alzheimer’s Disease using
T1-weighted brain MRI. Our approach is unique in employing deformation fields
within an unsupervised learning context to analyze, localize, and assess the
severity of AD-related atrophy.

Our results demonstrate that MORPHADE can effectively identify and localize
atrophy in clinically relevant brain regions, such as the hippocampus and
amygdala, which aligns with clinical expectations of AD pathology.
Furthermore, the anomaly maps generated by our method show strong visual
correspondence with MTA scores, underscoring the potential of our method in
clinical assessments. Lastly, MORPHADE achieved an AUROC of 0.80 in detecting
AD, outperforming several supervised and unsupervised baselines. This
highlights the robustness of our method without requiring extensive labeled
datasets, addressing a significant limitation in current diagnostic
approaches.

Future work could explore integrating MORPHADE’s deformation metrics with
established AD biomarkers, such as tau protein accumulation and amyloid-beta
levels, to enhance understanding of disease progression. Additionally,
expanding our framework to other neurodegenerative diseases could further
validate its versatility and clinical utility.

In conclusion, MORPHADE offers a promising tool for localization, and severity
assessment of AD-related atrophy, contributing valuable insights into the
progression and diagnosis of neurodegenerative diseases. Our findings suggest
that this approach could significantly enhance the non-invasive monitoring and
understanding of AD, paving the way for improved patient outcomes.

## References

  * [1] 2024 Alzheimer’s Disease facts and figures. Alzheimer’s & Dementia 20(5), 3708–3821 (2024). https://doi.org/10.1002/alz.13809
  * [2] Akcay, S., Atapour-Abarghouei, A., Breckon, T.: Ganomaly: Semi-supervised anomaly detection via adversarial training. In: Computer Vision – ACCV 2018. Lecture Notes in Computer Science, vol. 11363, pp. 622–637. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20893-6_39
  * [3] Ashburner, J., Hutton, C., Frackowiak, R., Johnsrude, I., Price, C., Friston, K.: Identifying global anatomical differences: Deformation-based morphometry. Human Brain Mapping 6(5-6), 348–357 (1998). https://doi.org/10.1002/(SICI)1097-0193(1998)6:5/6<348::AID-HBM4>3.0.CO;2-P
  * [4] Bercea, C.I., Rueckert, D., Schnabel, J.A.: What do AEs learn? Challenging Common Assumptions in Unsupervised Anomaly Detection. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 304–314. Springer (2023). https://doi.org/10.1007/978-3-031-43904-9_30
  * [5] Bercea, C.I., Wiestler, B., Rueckert, D., Julia, S.: Generalizing unsupervised anomaly detection: Towards unbiased pathology screening. International Conference on Medical Imaging with Deep Learning (2023)
  * [6] Breijyeh, Z., Karaman, R.: Comprehensive review on Alzheimer’s Disease: Causes and treatment. Molecules 25(24), 5789 (2020). https://doi.org/10.3390/molecules25245789
  * [7] Chen, X., Konukoglu, E.: Unsupervised detection of lesions in brain MRI using constrained adversarial auto-encoders. arXiv preprint arXiv:1806.04972 (2018)
  * [8] Chung, M., Worsley, K., Paus, T., Cherif, C., Collins, D., Giedd, J., Rapoport, J., Evans, A.: A unified statistical approach to deformation-based morphometry. NeuroImage 14(3), 595–606 (2001). https://doi.org/10.1006/nimg.2001.0862
  * [9] Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks (2014), https://arxiv.org/abs/1406.2661
  * [10] Huang, G., Liu, Z., van der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition (2017). https://doi.org/10.1109/CVPR.2017.243
  * [11] Jack Jr., C.R., Bernstein, M.A., Fox, N.C., Thompson, P., Alexander, G., Harvey, D., Borowski, B., Britson, P.J., L. Whitwell, J., Ward, C., Dale, A.M., Felmlee, J.P., Gunter, J.L., Hill, D.L., Killiany, R., Schuff, N., Fox-Bosetti, S., Lin, C., Studholme, C., DeCarli, C.S., Krueger, G., Ward, H.A., Metzger, G.J., Scott, K.T., Mallozzi, R., Blezek, D., Levy, J., Debbins, J.P., Fleisher, A.S., Albert, M., Green, R., Bartzokis, G., Glover, G., Mugler, J., Weiner, M.W.: The Alzheimer’s Disease Neuroimaging Initiative (ADNI): MRI methods. Journal of Magnetic Resonance Imaging 27(4), 685–691 (2008). https://doi.org/10.1002/jmri.21049
  * [12] Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. vol. 9906, pp. 694–711 (10 2016). https://doi.org/10.1007/978-3-319-46475-6_43
  * [13] Koikkalainen, J., Lötjönen, J., Thurfjell, L., Rueckert, D., Waldemar, G., Soininen, H., Initiative, A.D.N., et al.: Multi-template tensor-based morphometry: application to analysis of Alzheimer’s Disease. NeuroImage 56(3), 1134–1144 (2011). https://doi.org/10.1016/j.neuroimage.2011.03.029
  * [14] Korolev, S., Safiullin, A., Belyaev, M., Dodonova, Y.: Residual and plain convolutional neural networks for 3D brain MRI classification (2017). https://doi.org/10.1109/ISBI.2017.7950647
  * [15] Li, H., Shi, X., Zhu, X., Wang, S., Zhang, Z.: FSNet: Dual interpretable graph convolutional network for Alzheimer’s Disease analysis. IEEE Transactions on Emerging Topics in Computational Intelligence 7(1), 15–25 (2023). https://doi.org/10.1109/TETCI.2022.3183679
  * [16] Liu, M., Zhang, D., Shen, D., the Alzheimer’s Disease Neuroimaging Initiative: Hierarchical fusion of features and classifier decisions for Alzheimer’s Disease diagnosis. Human Brain Mapping 35(4), 1305–1319 (2014). https://doi.org/10.1002/hbm.22254
  * [17] Mazziotta, J.C., Toga, A.W., Evans, A., Fox, P., Lancaster, J.: A probabilistic atlas of the human brain: Theory and rationale for its development: The International Consortium for Brain Mapping (ICBM). NeuroImage 2(2, Part A), 89–101 (1995). https://doi.org/10.1006/nimg.1995.1012
  * [18] Patenaude, B., Smith, S., Kennedy, D., Jenkinson, M.: A Bayesian model of shape and appearance for subcortical brain segmentation. NeuroImage 56(3), 907–922 (2011). https://doi.org/10.1016/j.neuroimage.2011.02.046
  * [19] Petersen, R., Aisen, P., Beckett, L., Donohue, M., Gamst, A., Harvey, D., Jack, C., Jagust, W., Shaw, L., Toga, A., Trojanowski, J., Weiner, M.: Alzheimer’s Disease Neuroimaging Initiative (ADNI). Neurology 74(3), 201–209 (2010). https://doi.org/10.1212/WNL.0b013e3181cb3e25
  * [20] Scheltens, P., Launer, L., Barkhof, F., Weinstein, H., Van Gool, W.: Visual assessment of medial temporal lobe atrophy on Magnetic Resonance Imaging: Interobserver reliability. Journal of Neurology 242, 557–60 (1995). https://doi.org/10.1007/BF00868807
  * [21] Schlegl, T., Seeböck, P., Waldstein, S.M., Langs, G., Schmidt-Erfurth, U.: f-AnoGAN: Fast unsupervised anomaly detection with generative adversarial networks. Medical Image Analysis 54, 30–44 (2019). https://doi.org/10.1016/j.media.2019.01.010
  * [22] Siddiquee, M., Shah, J., Wu, T., Chong, C., Schwedt, T., Dumkrieger, G., Nikolova, S., Li, B.: Brainomaly: Unsupervised neurologic disease detection utilizing unannotated T1-weighted brain MR images. IEEE Winter Conf Appl Comput Vis pp. 7558–7567 (2024). https://doi.org/10.1109/wacv57701.2024.00740
  * [23] Wen, J., Thibeau-Sutre, E., Diaz-Melo, M., Samper-González, J., Routier, A., Bottani, S., Dormont, D., Durrleman, S., Burgos, N., Colliot, O.: Convolutional neural networks for classification of Alzheimer’s Disease: Overview and reproducible evaluation. Medical Image Analysis 63, 101694 (2020). https://doi.org/10.1016/j.media.2020.101694
  * [24] Zhang, Y., Teng, Q., Liu, Y., Liu, Y., He, X.: Diagnosis of Alzheimer’s Disease based on regional attention with sMRI gray matter slices. Journal of Neuroscience Methods 365, 109376 (2022). https://doi.org/10.1016/j.jneumeth.2021.109376
  * [25] Zimmerer, D., Isensee, F., Petersen, J., Kohl, S., Maier-Hein, K.: Unsupervised anomaly localization using variational auto-encoders. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2019. pp. 289–297. Springer International Publishing, Cham (2019). https://doi.org/10.1007/978-3-030-32251-9_32