# Summarization-based Data Augmentation for Document Classification

Yueguan Wang
The University of Tokyo
<EMAIL_ADDRESS>
Naoki Yoshinaga
Institute of Industrial Science,
The University of Tokyo
<EMAIL_ADDRESS>

###### Abstract

Despite the prevalence of pretrained language models in natural language
understanding tasks, understanding lengthy text such as document is still
challenging due to the data sparseness problem. Inspired by that humans
develop their ability of understanding lengthy text from reading shorter text,
we propose a simple yet effective summarization-based data augmentation,
SUMMaug, for document classification. We first obtain easy-to-learn examples
for the target document classification task by summarizing the input of the
original training examples, while optionally merging the original labels to
conform to the summarized input. We then use the generated pseudo examples to
perform curriculum learning. Experimental results on two datasets confirmed
the advantage of our method compared to existing baseline methods in terms of
robustness and accuracy. We release our code and data at
https://github.com/etsurin/summaug.

##  1 Introduction

Although the pretrained language models Devlin et al. (2019); Liu et al.
(2019); He et al. (2020) have boosted the accuracy of various natural language
understanding tasks, the accuracy is still limited for complex tasks with
lengthy input Lin et al. (2023) and fine-grained output Liu et al. (2021),
such as document classification. These tasks require models to find a mapping
between diverse input and output, which models are more likely to suffer from
the data sparseness problem.

To address the data sparseness problem, researchers have studied data
augmentation for text classification tasks. A basic approach is to generate
pseudo training examples from gold examples by perturbing the inputs; those
perturbation include back-and-forth translation Shleifer (2019) and minor
editing of input text Wei and Zou (2019); Karimi et al. (2021) or its hidden
representations Chen et al. (2020, 2022); Wu et al. (2022). These methods
basically echo the information in the original training data, which will not
help much the model learn to read lengthy inputs.

In this study, to effectively develop the model’s ability to comprehend the
content in document classification, we propose a simple yet effective
summarization-based data augmentation, SUMMaug, to generate pseudo,
abstractive training examples for document classification. Specifically, we
apply text summarization to the input of gold examples in document
classification task to obtain abstractive, easy-to-read examples, and merge
fine-grained target labels as needed so that the labels conforms to the
summarized input. Motivated by that we humans gradually develop the ability of
understanding lengthy text from reading shorter text, we use the generated
examples in the context of curriculum learning (surveyed in Soviany et al.
(2022)), namely, curriculum fine-tuning.

We compare our method to a baseline data augmentation Karimi et al. (2021) on
two versions of IMDb dataset with a different number of target labels.
Experimental results confirm that curriculum fine-tuning with SUMMaug
outperforms baseline methods on both accuracy and robustness.

Figure 1: Curriculum fine-tuning for document classification using SUMMaug
data augmentation: prior to the normal finetuning, it fine-tunes a model with
easy-to-learn examples obtained by summarizing the original training examples.

##  2 Related Work

In this section, we first review existing neural models for document
classification, and next introduce existing data augmentation methods for text
classification. We then mention other attempts to leverage summarization for
text classification.

#### Document Classification

In the literature, researchers explore a better neural architecture to
comprehend the lengthy content in document classification; examples include a
graph neural network Zhang and Zhang (2020); Zhang et al. (2022) and a
convolutional attention network Liu et al. (2021). Recently, Transformer
Vaswani et al. (2017)-based models have been revisited Dai et al. (2022) and
reported to outperform the task-specific networks. Since our work is model-
agnostic and orthogonal to the model architecture, we adopt RoBERTa Liu et al.
(2019), a Transformer-based pre-trained model, as the target of evaluation.

#### Data Augmentation for Text Classification

To address the data sparseness problem in text classification, researchers
employ data augmentation, which generates pseudo training examples from the
training examples. Shleifer (2019) leverages back-and-forth translation to
paraphrase the inputs of training examples. Through translating the inputs
into another language and then translating the resulting translation back to
the source language, they obtain the input that are written in different ways
but will have the same meanings conforming to the corresponding target labels.
Xie et al. (2017) perturb the input by deleting and inserting words and
replacing words with their synonyms. Karimi et al. (2021) propose a simple but
more effective perturbation that randomly inserts punctuation marks. Rather
than directly perturbing the input of training examples, some studies add
noises in their continuous representations Chen et al. (2020, 2022); Wu et al.
(2022). However, these method predominantly echo existing training data,
providing minimal assistance in understanding lengthy texts.

I am Anthony Park, Glenn Park is my father. First off I want to say that the
story behind this movie and the creation of the Amber Alert system is a good
one. However the movie itself was poorly made and the acting was terrible. The
major problem I had with the movie involved the second half with Nichole
Timmons and father Glenn Park. The events surrounding that part of the story
were not entirely correct. My father was suffering from psychological
disorders at the time and picked up Nichole without any intent to harm her at
all. He loved her like a daughter and was under the mindset that he was
rescuing her from some sort of harm or neglect that he likely believed was
coming from her mother who paid little attention to her over the 3 plus years
that my father took care of her and summarily raised her so her mother could
frolic about. The movie depicted my father in a manner that he was going to
harm her in some way shape or form. The funny thing is that Nichole had spent
many nights sometimes consecutively at my fathers place while Sharon would be
working or doing whatever she was doing. The reason that my father was
originally thought to be violent was because he had items that could be
conceived to be weapons on his truck. My father was a landscaper. The items
they deemed to be weapons were landscaping tools that he kept in his truck all
the time for work. My recommendation is take this movie with a grain of salt,
it is a good story and based on true events however the details of the movie
(at least the Nichole Timmons - Glenn Park portion) are largely inaccurate and
depict the failure of the director to discover the truth in telling the story.
The funny thing is, that if the director would have interviewed any of
Sharon’s friends who knew the situation they would have stated exactly what I
have posted here.
---
The movie itself was poorly made and the acting was terrible. The events
surrounding that part of the story were not entirely correct. My
recommendation is take this movie with a grain of salt, it is a good story and
based on true events.
Table 1:  An example of original text an generated summary on IMDb dataset.
The first row is the original text while the second row is the generated
summary. Red text are counterparts of summary in the original text.

#### Use of Summarization in Text Classification

Li and Zhou (2020) and Hartl and Kruschwitz (2022) utilize automatically
generated summaries to retrieve fact for fake news detection. Whereas this
approach uses summaries to retrieve knowledge for classification, our approach
leverages summaries in training as easy-to-learn examples, which does not
assume costly summarization in inference.

##  3 SUMMaug

Document classification requires a model to comprehend lengthy text with
dozens of sentences, which is even difficult for humans, especially, children
and second-language learners. Then, how do we humans develop an ability to
comprehend lengthy text? In school, starting from reading short, concise text,
we gradually read longer text.

In this study, we develop a summarization-based data augmentation method for
document classification, SUMMaug, and use it to generate pseudo, abstractive
training examples from gold examples to perform curriculum learning in
document classification.

###  3.1 Summarization-based data augmentation

In SUMMaug, a summarization model $M$ is used to generate pseduo, easy-to-
learn examples for document classification. In this study, we apply an off-
the-shelf summarization model, $M$, to each training pair $\\{x,y\\}$, where
$x$ denotes the document and $y$ denotes the label, and then obtain a concise
summary of $x$, namely, $\hat{x}=M(x)$.

An issue here is how to determine the label for the generated concise summary,
$\hat{x}$. Since the summarization abstracts away detailed information for
classification, the original target label $y$ can be inappropriate especially
when the target labels are fine-grained. We thus define a map function $f$ to
merge the fine-grained categories into a coarse-grained label group, and
obtain the augmented training pair is $\\{\hat{x},f(y)\\}$, as shown in Figure
1.

#### On summarization model

To summarize diverse text handled in document classification, we assume an
off-the-shelf summarization model that can handle documents with diverse
topics. In this study, we choose an off-the-shelf BART Lewis et al.
(2020)-based summarization model fine-tuned on CNN-Dailymail Hermann et al.
(2015) dataset as an implementation of
$M$,111https://huggingface.co/facebook/bart-large-cnn since the writing style
of news reports is suitable for most of the text in daily life. We should
mention that the CNN-Dailymail dataset contains mostly extractive summaries,
and the resulting summarization model will be less likely to suffer from
hallucinations Maynez et al. (2020) that have been reported for a
summarization model trained on abstractive summarization datasets such as XSum
Narayan et al. (2018).

Table 1 exemplifies a summary generated for IMDb datasets. While the original
input (review) exhibits a mild negative sentiment, its compression into a
summary intensifies this sentiment. This observation underscores the
imperative to categorize labels of augmented data into coarser groups.

###  3.2 Learning a Classifier with augmented data

In the literature of data augmentation, the models are basically trained with
the original and augmented training data, since both data are related to the
target task. In our settings, however, the labels will be merged into fewer
labels so that the labels conform to the generated summaries. We thus consider
the following two strategies to utilize the pseudo abstractive training data.

Mixed fine-tuning

We combine the original and pseudo training data to fine-tune a pre-trained
model for classification. In this setting, we do not collapse labels, namely,
$f(y)=y$.

Curriculum fine-tuning

We first finetune a pre-trained model on the pseudo training data, and then
finetune a pre-trained model on the original training data. This strategy is
inspired by curriculum learning Bengio et al. (2009). In this setting, we
collapse labels as needed. When we collapse labels, we discard parameters for
the collapsed labels in the fine-tuning with the original examples.

In the following experiments, we compare two strategies for datasets with
different numbers of labels.

##  4 Experiments

We conduct experiments on two datasets to evaluate our method, thus
demonstrating that: (1) our method shows better accuracy and robustness
compared with baseline methods in both general setting and low-resource
settings; and (2) curriculum fine-tuning plays an important role in achieving
improvements.

###  4.1 Dataset

We use two versions of large-scale movie reviews dataset IMDb for evaluation.
One contains 50,000 movie reviews with a positive or negative label Maas et
al. (2011), while the other involves 10 different labels from rating 1 to 10.
For the IMDB-2 dataset, we split 10% of the training data for validation. For
the IMDb-10 dataset, the same splitting as Adhikari et al. (2019) is used. The
detailed information of the two datasets is shown in Table 2.

Dataset | train | val | test | $C$ | $L$ | $L_{M}$
---|---|---|---|---|---|---
IMDb-2 | 22500 | 2500 | 25000 | 2 | 279.5 | 51.3
IMDb-10 | 108670 | 13432 | 13567 | 10 | 394.2 | 50.2
Table 2:  Details of the IMDb datasets: $C$ denotes the number of classes. $L$
and $L_{M}$ denote the average length of the inputs and the generated
summaries, respectively.

###  4.2 Methods

We use the following three models for evaluation. All models are based on
RoBERTa Liu et al. (2019) with a classification layer.

#### RoBERTa

We finetune a pre-trained RoBERTa222https://huggingface.co/roberta-large on
the original training data as a baseline.

#### RoBERTa + AEDA

We use AEDA Karimi et al. (2021), a strong data augmentation method for text
classification as another baseline. We apply
AEDA333https://github.com/akkarimi/aeda_nlp to the original documents, and
then fine-tune a RoBERTa model on the augmented data and original data.

#### RoBERTa + SUMMaug

We use BART-based summarizer trained on CNN-Dailymail to generate concise
summaries, and fine-tune a RoBERTa model on the augmented data and original
data.

To evaluate the performance of our method in low resource settings, we
randomly select 200 and 1500 samples from the two datasets and train a model
on these sub datasets. However, on the IMDb-10 dataset, we observe that all
models diverge and perform randomly when training data is reduced to 200,
likely due to the challenges of fine-grained classification with rather
limited training data; we thereby do not report the results.

In order to reveal the effectiveness of curriculum fine-tuning, we apply
curriculum fine-tuning not only to SUMMaug but also to AEDA. On the IMDb-10
dataset, we map the labels of the augmented data into coarse-grained ones, as
mentioned in § 3.1. Specifically, labels between 0-4 are mapped into 0
(negative) while labels between 5-9 are mapped into 1 (positive).

Model | The size of training data
---|---
200 | 1500 | all
RoBERTa | 92.191.21 | 94.210.62 | 94.630.56
\+ AEDA (mixed) | 90.911.44 | 94.430.49 | 94.750.66
\+ AEDA (curriculum) | 93.591.16 | 94.260.74 | 95.560.12
\+ SUMMaug (mixed) | 92.940.99 | 94.610.64 | 94.850.62
\+ SUMMaug (curriculum) | 93.360.97 | 94.770.28 | 95.450.17
Table 3:  Classification accuracy${}_{\textrm{stdev.}}$ (%) on IMDb-2: mixed
and curriculum denotes mixed and curriculum fine-tuning. All the results are
averages over five runs. The best results are marked as bold.

###  4.3 Implementation Details

We set the model’s hyperparameters as follows. For experiments on the IMDb-2
dataset, batch size is set to 64 and learning rate is set to 1e-5. For
experiments on the IMDb-10 dataset, following Adhikari et al. (2019), batch
size is set to 16, with learning rate set to 2e-5. Detailed information of
training epochs can be found at Appendix A. All the experiments were conducted
on four NVIDIA Quadro P6000 GPUs with 24GB memory.

The final model for evaluation is selected on the basis of the performance on
validation set. To eliminate the effect of random factors, we report the
average accuracy over five runs.

##  5 Results

Model | The size of training data
---|---
1500 | all
RoBERTa | 39.998.46 | 56.580.34
\+ AEDA (mixed) | 36.5810.64 | 51.2314.39
\+ AEDA (curriculum) | 41.773.01 | 56.631.65
\+ SUMMaug (mixed) | 40.652.71 | 55.812.00
\+ SUMMaug (curriculum) | 42.141.48 | 57.550.29
Table 4:  Classification accuracy${}_{\textrm{stdev.}}$ (%) on IMDb-10. All
the results are averages over five runs. The notations follow Table 3.

Tables 3 and 4 list the results of baseline methods and our proposed method.
Our method outperforms baseline methods in all experimental settings. We
additionally confirm on both datasets that our data augmentation is effective
even when the training data size is small.

#### How robustly does SUMMaug work?

SUMMaug achieves higher classification accuracy across datasets while
improving or maintaining robustness (low standard deviations), whereas the
original AEDA, namely AEDA (mixed), reduces the accuracy on IMDb-2 when 200
training examples are used, and it leads to unstable results on IMDb-10
dataset.

#### Is curriculum fine-tuning effective?

We use mixed fine-tuning with SUMMaug and curriculum fine-tuning with AEDA. We
observe that under mixed fine-tuning method, the data augmented by SUMMaug
exhibited less improvements and even turns to be harmful on the IMDb-10
dataset. Conversely, it turns out that curriculum learning helps the AEDA
method achieve further improvements in some cases while addressing the low
robustness issue. However, curriculum learning with AEDA does not consistently
enhance results because the AEDA augmented data retains the same information
as the original data, which offers limited benefits in improving text
comprehension.

$N$ | $f$ | Accuracy${}_{\textrm{stdev.}}$
---|---|---
2 | [0,0,0,0,0,1,1,1,1,1] | 57.550.29
3 | [0,0,0,1,1,1,1,2,2,2] | 57.470.20
4 | [0,0,0,1,1,2,2,3,3,3] | 57.660.31
5 | [0,0,1,1,2,2,3,3,4,4] | 57.320.60
10 | [0,1,2,3,4,5,6,7,8,9] | 57.200.56
Table 5:  Classification accuracy${}_{\textrm{stdev.}}$ (%) on IMDb-10 with
different label coarsening function under SUMMaug (curriculum) method. $N$
denotes the number of merged label groups while $f$ shows how the original
label 0-9 is mapped into coarse-grained label. All the results are averages
over five runs.

#### How label coarsening affects accuracy?

Table 5 shows the results of SUMMaug (curriculum) under different map function
$f$. The accuracy is comparable when $N\leq 4$, while there’s a noticeable
decline in accuracy, accompanied by decreased stability when label coarsening
is insufficient or not adopted. This is probably because the summaries can
filter out detailed content, which is essential for fine-grained
classification. On the other hand, unlike mixed fine-tuning, in which
potentially noisy augmented data is used throughout the training process, in
the curriculum fine-tuning, the effect of noise diminishes after model turns
to train on the original data. Consequently, it can still achieve improvement
even without label coarsening.

##  6 Conclusion and Future Work

This study explores a novel application of a summarization model and proposes
a simple yet effective data-augmentation method, SUMMaug, for document
classification. It performs curriculum learning-style fine-tuning to first
train a model on concise summaries prior to the fine-tuning on the original
training data. This mirrors the human process of mastering lengthy text
comprehension, through gradual exposure to longer text. Experimental results
on two document classification datasets confirm that SUMMaug enhances both
accuracy and training stability compared to the baseline data augmentation
method. Meanwhile, our method shows effective in low-resource settings.

The future work will focus on searching for the optimal mapping function $f$
and exploring the effect of different summarization models. We will also apply
SUMMaug to other document classification tasks of various domains.

## Limitations

One of the drawbacks of this study is that we do not consider the label
coarsening function $f$ as a hyper-parameter and just choose the simplest one
for experiments. The effect of label coarsening function on accuracy is still
insufficiently explored. For the datasets, despite the different numbers of
labels, the documents used are originally from the same kind of domain, which
is not convincing enough to show that SUMMaug is robust across diverse
classification tasks in different domains.

## Acknowledgements

This work was partially supported by the special fund of Institute of
Industrial Science, The University of Tokyo, and by JSPS KAKENHI Grant Number
JP21H03494.

## References

  * Adhikari et al. (2019) Ashutosh Adhikari, Achyudh Ram, Raphael Tang, and Jimmy Lin. 2019.  DocBERT: BERT for document classification.  _arXiv preprint arXiv:1904.08398_.
  * Bengio et al. (2009) Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009\.  Curriculum learning.  In _Proceedings of the 26th Annual International Conference on Machine Learning_ , ICML ’09, page 41–48, New York, NY, USA. Association for Computing Machinery.
  * Chen et al. (2022) Hui Chen, Wei Han, Diyi Yang, and Soujanya Poria. 2022.  DoubleMix: Simple interpolation-based data augmentation for text classification.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 4622–4632, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Chen et al. (2020) Jiaao Chen, Zichao Yang, and Diyi Yang. 2020.  MixText: Linguistically-informed interpolation of hidden space for semi-supervised text classification.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2147–2157, Online. Association for Computational Linguistics.
  * Dai et al. (2022) Xiang Dai, Ilias Chalkidis, Sune Darkner, and Desmond Elliott. 2022.  Revisiting transformer-based models for long document classification.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 7212–7230, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Hartl and Kruschwitz (2022) Philipp Hartl and Udo Kruschwitz. 2022.  Applying automatic text summarization for fake news detection.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 2702–2713, Marseille, France. European Language Resources Association.
  * He et al. (2020) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020.  Deberta: Decoding-enhanced BERT with disentangled attention.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  _Advances in neural information processing systems_ , 28.
  * Karimi et al. (2021) Akbar Karimi, Leonardo Rossi, and Andrea Prati. 2021.  AEDA: An easier data augmentation technique for text classification.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 2748–2754, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Li and Zhou (2020) Qifei Li and Wangchunshu Zhou. 2020.  Connecting the dots between fact verification and fake news detection.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 1820–1825, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Lin et al. (2023) Yu-Chen Lin, Si-An Chen, Jie-Jyun Liu, and Chih-Jen Lin. 2023.  Linear classifier: An often-forgotten baseline for text classification.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 1876–1888, Toronto, Canada. Association for Computational Linguistics.
  * Liu et al. (2021) Yang Liu, Hua Cheng, Russell Klopfer, Matthew R. Gormley, and Thomas Schaaf. 2021\.  Effective convolutional attention network for multi-label clinical document classification.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 5941–5953, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011.  Learning word vectors for sentiment analysis.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.
  * Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020.  On faithfulness and factuality in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1906–1919, Online. Association for Computational Linguistics.
  * Narayan et al. (2018) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1797–1807, Brussels, Belgium. Association for Computational Linguistics.
  * Shleifer (2019) Sam Shleifer. 2019.  Low resource text classification with ULMFit and backtranslation.  _arXiv preprint arXiv:1903.09244_.
  * Soviany et al. (2022) Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. 2022.  Curriculum learning: A survey.  _International Journal of Computer Vision_ , 130(6):1526–1565.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Wei and Zou (2019) Jason Wei and Kai Zou. 2019.  EDA: Easy data augmentation techniques for boosting performance on text classification tasks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 6382–6388, Hong Kong, China. Association for Computational Linguistics.
  * Wu et al. (2022) Xing Wu, Chaochen Gao, Meng Lin, Liangjun Zang, and Songlin Hu. 2022.  Text smoothing: Enhance various data augmentation methods on text classification tasks.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 871–875, Dublin, Ireland. Association for Computational Linguistics.
  * Xie et al. (2017) Ziang Xie, Sida I. Wang, Jiwei Li, Daniel Lévy, Aiming Nie, Dan Jurafsky, and Andrew Y. Ng. 2017.  Data noising as smoothing in neural network language models.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
  * Zhang et al. (2022) Chong Zhang, He Zhu, Xingyu Peng, Junran Wu, and Ke Xu. 2022.  Hierarchical information matters: Text classification via tree based graph neural network.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 950–959, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Zhang and Zhang (2020) Haopeng Zhang and Jiawei Zhang. 2020.  Text graph transformer for document classification.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8322–8327, Online. Association for Computational Linguistics.

##  Appendix A Detailed training epochs

Table 6 shows detailed training epochs in our experiments. We select training
epochs based on the accuracy on the validation set.

Dataset (fine-tuning method) | The size of training data
---|---
200 | 1500 | all
IMDb-2 (w/o data augmentation) | 70 | 18 | 2
IMDb-2 (mixed) | 70 | 18 | 2
IMDb-2 (curriculum) | 70/70 | 18/18 | 2/2
IMDb-10 (w/o data augmentation) | - | 20 | 4
IMDb-10 (mixed) | - | 20 | 4
IMDb-10 (curriculum) | - | 5/20 | 2/6
Table 6:  Detailed training epochs in our experiments. For curriculum fine-
tuning method, $x/y$ denotes that model is trained $x$ epochs on augmented
data and then $y$ epochs on original data.


# Adaptive Variance Thresholding: A Novel Approach to Improve Existing Deep
Transfer Vision Models and Advance Automatic Knee-Joint Osteoarthritis
Classification

Fabi Prezja  Leevi Annala  Sampsa Kiiskinen  Suvi Lahtinen  Timo Ojala

###### Abstract

Knee-Joint Osteoarthritis (KOA) is a prevalent cause of global disability and
is inherently complex to diagnose due to its subtle radiographic markers and
individualized progression. One promising classification avenue involves
applying deep learning methods; however, these techniques demand extensive,
diversified datasets, which pose substantial challenges due to medical data
collection restrictions. Existing practices typically resort to smaller
datasets and transfer learning. However, this approach often inherits
unnecessary pre-learned features that can clutter the classifier’s vector
space, potentially hampering performance. This study proposes a novel paradigm
for improving post-training specialized classifiers by introducing adaptive
variance thresholding (AVT) followed by Neural Architecture Search (NAS). This
approach led to two key outcomes: an increase in the initial accuracy of the
pre-trained KOA models and a 60-fold reduction in the NAS input vector space,
thus facilitating faster inference speed and a more efficient hyperparameter
search. We also applied this approach to an external model trained for KOA
classification. Despite its initial performance, the application of our
methodology improved its average accuracy, making it one of the top three KOA
classification models.

###### keywords:

Knee Osteoarthritis , Adaptive Variance Threshold , Neural Architecture Search
, Convolutional Neural Network , Disease stages

[1]organization=University of Jyväskylä, Faculty of Information Technology,
city=Jyväskylä, postcode=40014, country=Finland

[3]organization=University of Helsinki, Faculty of Science, Department of
Computer Science, city=Helsinki, country=Finland

[4]organization=University of Helsinki, Faculty of Agriculture and Forestry,
Department of Food and Nutrition, city=Helsinki, country=Finland

[5]organization=University of Jyväskylä, Faculty of Mathematics and Science,
Department of Biological and Environmental Science, city=Jyväskylä,
postcode=40014, country=Finland

##  1 Introduction

Over the last ten years, a significant increase has been observed in the
incorporation of artificial intelligence into the medical field[1, 2], spurred
by the robust expansion in deep machine learning methodologies[3]. Medicine
has been recognized as a key domain for deploying these sophisticated
technologies, with deep learning primarily focused on data analysis and
clinical decision support. These proficient systems, capable in exploring
medical data to identify patterns and relationships, cover many applications.
They’ve displayed considerable advancement in predicting patient outcomes [4,
5, 6], along with enriching diagnostics and disease classification [7, 8, 9,
10, 11]. Apart from data analysis and classification, deep learning has been
successful in data segmentation [12, 13], and has also shown progress in the
generation [14, 15, 16, 17, 18] and anonymization of medical data[19, 20, 21,
22, 23, 24]. Nonetheless, applying these technological developments to
Osteoarthritis (OA) brings its unique set of challenges.

OA, predominantly knee joint osteoarthritis[25, 26, 27] (KOA), is a leading
cause of disability worldwide[28], with estimated expenses amounting to as
much as 2.5% of the gross national product in western countries.[28]. Early
detection is hampered by subtle radiographic indicators and variability in
disease progression[29, 25]. Employing deep learning in KOA classification
[30, 31, 32] heavily relies on diverse and extensive data sets. Yet, obtaining
such data sets is challenging, hindered by patient privacy concerns[33, 34],
limitations in data collection, and the intrinsic progression of OA.
Traditionally, researchers have circumvented these limitations through deep
transfer learning, where pre-trained neural networks are repurposed for a
specific task using smaller data sets. The peak accuracy achieved for multi-KL
classification using radiographic Kellgren and Lawrence grading of OA stands
at 74.81% [35, 25]. This process typically involves end-to-end training of
transfer architectures, which include a feature learning section and a
classification section. In this setup, learned features are usually fed into a
classifier for predictions. However, not all repurposed features may be highly
relevant to the task, potentially cluttering the classifier with unnecessary
or barely varying (near-static) features, while manual classifier design could
increase complexity and raise the risk of under-fitting. However, feature
selection[36] and Neural Architecture Search (NAS)[37, 38] can partly mitigate
these issues. Feature selection refining classifier input and NAS automating
the architecture design process, identifying optimal structures often missed
in manual design, thus increasing potential model efficiency and accuracy.
However, these techniques have yet to be utilized with transfer learning-based
classification for KOA classification.

In this study, we propose a method to amplify the performance of existing KOA
classification models using an adaptive variance threshold feature filtering
technique and neural architecture search. This approach refined the
classifier’s feature space, boosted computational efficiency, and advanced the
initial accuracy of the evaluated models. Significantly, when we applied our
technique to an external model, specifically from Chen et al[39], its average
accuracy improved to 71.14%. This boost propeled the external model to the top
three KOA classification models[25, 35].

##  2 Methods

Our study’s methodology is structured into three clear phases. First, we focus
on data collection and essential pre-processing. In the subsequent phase, we
explore the intricacies of the deep transfer learning approach. The concluding
phase offers a comprehensive overview of the adaptive variance thresholding
method and neural architecture search, detailing the metrics for evaluation.
Figure 1 illustrates the core approach we used with the pre-trained
EfficientNet model for KOA classification.

Figure 1: The study’s methodological pipeline, as applied to the
EfficientNetV2 Base Model, showcases the intervention in the classification
head.

###  2.1 Data Collection

Our study utilized knee joint X-ray images derived from the 2019 Chen, et al.
study[39], obtained initially from the Osteoarthritis Initiative (OAI). The
OAI is a multi-center study focused on biomarkers for knee osteoarthritis that
involved 4796 participants aged between 45 and 79. We used the pre-processed
primary cohort data[40] from the Chen 2019 study, which had undergone
automatic knee joint detection, bounding, and standardization of zoom to
0.14mm/pixel. This process yielded 8260 images (224 x 224 pixels) extracted
from 4130 X-rays, each encompassing both knee joints. The images were
classified using the Kellgren and Lawrence (KL) system[41], as illustrated in
Figure 2. The distribution of KL grades comprised 3253 images for Grade 0,
1495 for Grade 1, 2175 for Grade 2, 1086 for Grade 3, and 251 for Grade 4.

Figure 2: Sample images display different KL grades, from 0 (indicating no OA
signs) to 4 (representing severe OA). As we move from left to right, the
severity of OA intensifies. Within each image, two red markers are present: a
circle pinpointing osteophyte areas and an arrow highlighting joint space
narrowing (JSN).

###  2.2 Image Pre-Processing

We adjusted each right knee joint image to replicate the orientation of a left
knee. We subsequently located and inverted any negative channel images,
leading to 189 instances. The contrast of the image histograms was then
equalized using equation 1. In this equation, we took a given grayscale image
I of $m\times n$ dimensions and used the cumulative distribution function
$cdf$ and pixel value $v$ to derive an equalized value $h(v)$ within the range
$[0,255]$. Here, $cdf_{min}$ represents a non-zero minimum value of the
image’s cumulative distribution, while $m\times n$ refers to the total number
of pixels.

$h(v)=255\frac{cdf(v)-cdf_{min}}{(mn)-cdf_{min}}$ (1)

###  2.3 Chen 2019 External Validation

In order to externally validate our approach, we mirrored the exact models and
data utilized in the Chen et al. 2019 study[39]. We chose the study’s best-
performing VGG19 - Ordinal model (publicly available[40]). We extracted the
feature vectors from this model from the pre-output dense layer, effectively
replicating the conditions under which the original model achieved its top
performance. This step was crucial to ensure a fair comparison and assessment
of the effectiveness of our approach when applied to an external model.

###  2.4 Convolutional Neural Networks

Convolutional Neural Networks (CNNs) [42], a cornerstone in the renaissance of
deep learning [3], have often found applications in the realm of computer
vision. The strength of CNNs lies in their ability to execute convolution
operations between an input and a filter-kernel, thereby highlighting
distinctive features captured in a response known as a feature map. As these
filters slide across inputs, they create layers of complex feature maps, each
representing more abstract concepts from the preceding maps. Mathematically,
given an image I with dimensions $u\times v$ and a filter-kernel H of $s\times
t$ dimensions, the generation of a feature map G by convolving I and H across
axes $u,v$ is expressed as:

$\textbf{{G}}(u,v)=\sum_{s}\sum_{t}\textbf{{I}}(u,v)\textbf{{H}}(u-s,v-t)$ (2)

Usually, the resulting values of the feature map are filtered through an
activation function, which acts to re-map these values using a pre-defined
function. One common example is the Rectified Linear Unit activation
function[43] (ReLu), which sets negative values to zero, thereby offering
computational efficiency by eliminating less essential information. For any
feature map value $z$, the ReLu activation function is defined as:

$f(z)=\max(0,z)$ (3)

Further, a max pooling operation is often employed to down-sample the
convolution result. Consequently, cascades of max pooling and convolution
yield a gradually decreasing length of features. For an image I with
dimensions $u\times v$, the max pooled value ${g(u_{I})}$ for dimension $u$
can be defined as:

$g(u_{I})=\lfloor\frac{{u_{I}}-r}{h}\rfloor+1$ (4)

Here, $u_{I}$ is dimension $u$ from image I, $r$ denotes the pooling window
size, and $h$ signifies the stride value. This streamlining process
contributes to an efficient and effective model for identifying the necessary
classification task features.

###  2.5 Convolutional Neural Network Architecture

A notable deep learning model, EfficientNet[44] , adopts the strategy of
compound scaling, which systematically adjusts depth (number of layers), width
(size of the layers), and resolution (size of the input image). This scaling
process is mathematically encapsulated as follows:

$d=\alpha^{\phi}d_{0},\quad w=\beta^{\phi}w_{0},\quad r=\gamma^{\phi}r_{0}$
(5)

Here, $\alpha,\beta,\gamma$ denote constants, $\phi$ stands for a user-
selected coefficient, and $d_{0},w_{0},r_{0}$ represent the depth, width, and
resolution of the original model, respectively.

A distinctive feature of EfficientNet is the MBConv block, which chains
together transformations in the order of a $1\times 1$ convolution, a depth-
wise convolution, a Squeeze-and-Excitation (SE) operation [45], and a
subsequent $1\times 1$ convolution:

$T_{MB}(\textbf{{I}})=\textbf{{K}}_{2}\ast
SE(\textbf{{D}}\ast(\textbf{{K}}_{1}\ast\textbf{{I}}))$ (6)

In this equation, $\textbf{{K}}_{1}$ and $\textbf{{K}}_{2}$ are $1\times 1$
convolutional filters, D stands for the depth-wise convolutional filter, and
$SE(\cdot)$ signifies the Squeeze-and-Excitation operation. The
EfficientNetV2[46] model extends its predecessor by incorporating a Fused-
MBConv block that fuses the initial $1\times 1$ and depth-wise convolutions
into a single $3\times 3$ convolution, followed by an SE operation and a
concluding $1\times 1$ convolution:

$T_{FMB}(\textbf{{I}})=\textbf{{K}}{2}\ast(SE(\textbf{{K}}{f}\ast\textbf{{I}}))$
(7)

Here, $\textbf{{K}}_{f}$ is the $3\times 3$ convolutional filter that merges
the initial $1\times 1$ and depth-wise convolutions, and $\textbf{{K}}_{2}$ is
the final $1\times 1$ convolutional filter. An activation function follows
each convolution and may include a skip connection. The EfficientNetV2 base
model (B0) and the modifications implemented in our study are illustrated in
Figure 3.

Figure 3: The base architecture of EfficientNetV2, complemented by the changes
we implemented after the pooling stage.

We utilized the EfficientNetV2-M architecture for our model and further
updated it by introducing flattening and a dense layer consisting of 240
neurons. The model was trained over 25 epochs using the Adam optimizer, with
the dataset being partitioned into Training (75%), Validation (15%), and
Testing (15%) sets (patient-wise splits). The minimum validation loss governed
the early stopping criterion. The CNN training and online affine augmentations
(’advanced augmentation preset’) were executed using the open-source Deep Fast
Vision library[47].

###  2.6 Adaptive Variance Thresholding

Variance Thresholding is a popular feature selection method[48, 49] used to
enhance machine learning model performance by removing excess features. It
calculates the variance of each feature and drops those whose variance is
below a specific user-defined threshold. However, the conventional method
requires manually setting the threshold, which is not adaptable to different
datasets and does not generalize across datasets or feature vectors of
different architectures trained even on the same dataset. To overcome this, we
propose Adaptive Variance Thresholding (AVT), which automatically sets the
threshold from a user-defined percentile of the calculated feature variances.
This approach makes the method flexible, as it adapts to the unique
characteristics of each dataset without requiring manual changes. The
following mathematical steps describe the process of Adaptive Variance
Thresholding (AVT):

  1. 1.

Compute the variance of each feature in the feature set, F. We denote this
variance as v, where each element $v_{i}$ represents the variance of the
$i$-th feature:

$\mathbf{v}=\text{Var}(\textbf{{F}}_{i}),\quad\forall i\in{1,2,...,w}$ (8)

Here, $\textbf{{F}}_{i}$ denotes the $i$-th feature in the feature matrix F
with dimensions $w\times z$ (where $w$ is the number of features and $z$ is
the number of samples). $\text{Var}(\textbf{{F}}_{i})$ calculates the variance
of the $i$-th feature, with $\mathbf{v}$ storing these variances.

  2. 2.

Set the threshold $j$ to be the $p$-th percentile of these computed variances:

$j=\text{percentile}(\mathbf{v},p)$ (9)

The function $\text{percentile}(\mathbf{v},p)$ computes the $p$-th percentile
of the variances in $\mathbf{v}$, where $p$ is a user-defined parameter.

  3. 3.

Form a new feature matrix F’, consisting of only those features $i$ whose
variance exceeds the threshold $j$:

$\textbf{{F}}_{i}\in\textbf{{F'}}\text{ if }v_{i}\geq j$ (10)

In the final step, the feature $\textbf{{F}}_{i}$ is included in the new
feature matrix F’ if its corresponding variance $v_{i}$ is greater than or
equal to the threshold $j$. Otherwise, the feature is removed. The power of
the adaptive approach is in its capacity to derive the threshold directly from
the data, allowing it to adjust to the distinct characteristics of each
dataset. In our study, we examined three threshold levels, referred to as Low,
Mid, and High, each representing different percentiles - 1.5%, 50%, and 98.5%,
respectively, given the computational demands associated with this approach;
conducting an exhaustive search was prohibitively computationally demanding.
In essence, we preserved the top 98.5% varying features in the Low setting,
the top 50% in the Mid setting, and only the top 1.5% in the High setting.
These settings explored various scenarios from low to significant
dimensionality reduction. The Python class implementing this method can be
found in the provided materials. It constitutes a modification of the Scikit-
Learn class[50], using NumPy[51] for computations.

###  2.7 Neural Architecture Search

Neural Architecture Search (NAS) [37], a method in machine learning, offers an
automated way to design neural network architectures, diminishing the need for
exhaustive manual tuning. By systematically exploring a range of potential
architectures, NAS pinpoints the ones that yield the best performance for a
specific task. The process involves defining a search space of potential
architectures and using a controller model to generate, assess, and train
these candidate architectures. In our experiment, we applied the AutoKeras[38]
Structured Classifier search approach to identify suitable architectures for
the Adaptive Variance Thresholding (AVT) sets. Notably, the evaluation of NAS
was performed exclusively using validation data. We permitted a maximum of 55
consecutive trials with validation accuracy guiding the early stopping
criterion. Each trial was given a training space of up to 25 epochs. All
identified architectures are detailed in the appendix, and trained versions of
these architectures are available under data availability.

###  2.8 Classification Metrics

Accuracy, a common measure of model performance, is the proportion of correct
predictions (true positives and true negatives) to the total number of
observations. It’s mathematically defined as:

$\text{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}$ (11)

Precision, also referred to as the positive predictive value, denotes the
fraction of correct positive predictions in relation to all positive
predictions made. The calculation for precision is as follows:

$\text{Precision}=\frac{TP}{TP+FP}$ (12)

Recall, alternatively known as sensitivity or true positive rate, represents
the fraction of actual positive instances that the model correctly identified.
The formula for recall is:

$\text{Recall}=\frac{TP}{TP+FN}$ (13)

The F1 score aims to balance precision and recall by taking their harmonic
mean. It’s calculated using the following formula:

$\text{F1
Score}=2\cdot\frac{\text{Precision}\cdot\text{Recall}}{\text{Precision}+\text{Recall}}$
(14)

In these expressions, TP stands for True Positives, TN represents True
Negatives, FP is for False Positives, and FN refers to False Negatives.

##  3 Results

###  3.1 External Chen 2019 Model

This study utilized adaptive variance thresholding (AVT) and neural
architecture search on pre-trained vision neural networks. Table 1 compares
the performance across AVT thresholds on the best pre-trained Chen 2019 model
(baseline), highlighting the impact of these techniques. We observed that the
model’s NAS input dimensionality significantly decreased from 4096 in the
’Baseline’ condition to 62 in the ’High AVT’ condition. Accuracy generally
improved, peaking at 71.14% in the ’High AVT’ scenario. Precision increased to
70.98% in the ’Mid AVT’ condition and slightly declined to 69.76% in the ’High
AVT’ setting. Recall remained relatively steady across all conditions with a
minor increase to 68.95% in the ’Mid AVT’ scenario. The F1-score, crucial in
handling uneven KL distributions, increased until ’Mid AVT’, then slightly
decreased in the ’High AVT’ condition. Figure 4 further details the confusion
matrixes for each condition.

Table 1: Classification report for the Chen model using Chen test data, with and without AVT intervention (baseline). ’Dimensionality’ denotes the vector space size pre-NAS. |  Baseline |  Low AVT |  Mid AVT |  High AVT
---|---|---|---|---
Dimensionality |  4096 |  4034 |  2048 |  62
Accuracy |  69.63% |  68.30% |  70.41% |  71.14%
Precision |  64.79% |  68.71% |  70.98% |  69.76%
Recall |  67.74% |  67.32% |  68.95% |  67.53%
F1-score |  64.96% |  67.96% |  69.86% |  66.88%
Figure 4: The confusion matrices for the external Chen model using Chen test
data. The baseline indicates no intervention, whereas Low, Mid, and High
represent AVT sets. The appendix tables 8,9,10 detail the NAS architecture
discovered for each set. X-axis shows predicted labels; Y-axis shows true
labels.

In the provided confusion matrices and accuracy results, the impact of varying
Adaptive Variance Thresholding (AVT) levels, namely Baseline, Low AVT, Mid
AVT, and High AVT, on the performance of a neural network model is presented.
A general improvement in accuracy was observed as the AVT level increased,
rising from approximately 69.63% at Baseline to 71.14% at High AVT. A detailed
look into the normalized confusion matrices revealed a consistent rise in
correct predictions, notably for the first and third classes, across
increasing AVT levels. An exception was observed for KL2, where correct
predictions declined as AVT increased. On the other hand, the fourth and fifth
classes exhibited stability in model performance with varying AVT levels after
an initial improvement from Baseline to Low AVT.

Different trends surfaced when we focused on the recall values for individual
classes (as presented in Table 2). The model’s proficiency in identifying
instances of KL 0 consistently strengthened with increasing AVT, leading to
the highest recall across all classes and conditions at High AVT (91.39%).
Conversely, KL 1 experienced a peak in recall at Low AVT (36.82%), followed by
a decrease at High AVT (14.86%), lower than its Baseline value. KL 2 also
benefited from higher AVT levels, as indicated by a higher recall at High AVT
(75.39%), despite slight decreases at Low and Mid AVT. The KL 3 class
maintained a relatively stable recall across all AVT levels, peaking slightly
at Low AVT (78.92%). Finally, the recall for KL 4 initially dropped from
Baseline to Low AVT but then held relatively steady for Mid and High AVT.
These recall trends underline the complex interplay between AVT levels and KL
grade-specific performance.

Table 2: Recall report for the external Chen model across classes, derived from the confusion matrix diagonals. KL Label |  Baseline |  Low AVT |  Mid AVT |  High AVT
---|---|---|---|---
0 |  87.01% |  80.13% |  85.13% |  91.39%
1 |  17.57% |  36.82% |  35.14% |  14.86%
2 |  74.94% |  66.22% |  68.01% |  75.39%
3 |  74.89% |  78.92% |  78.03% |  77.58%
4 |  84.31% |  74.51% |  78.43% |  78.43%

###  3.2 EfficientNetV2M Internal Model

The following section will focus on a different neural network model - the
EfficientNetV2M. When looking at Table 3, it was clear that the dimensionality
of the NAS input reduced dramaticaly as we moved from the Baseline to the High
AVT level. Regarding model performance, Accuracy, and Precision peaked at the
Mid AVT level, with Accuracy reaching 65.48% and Precision achieving 67.18%.
However, Precision took an upward leap at High AVT, reaching 79.72%, the
highest among all levels. In contrast, Recall and F1-score exhibited a slight
downward trend after the Baseline, with the lowest values observed at the High
AVT level - 58.99% and 58.35%, respectively. Based on Accuracy and Precision,
the model performance seemed optimal at Mid AVT rather than High AVT,
suggesting that extreme dimensionality reduction might not necessarily lead to
better overall performance for this set-up.

Table 3: Classification report for the EfficientNet model using the test data, with and without AVT intervention (baseline). ’Dimensionality’ denotes the vector space size pre-NAS. |  Baseline |  Low AVT |  Mid AVT |  High AVT
---|---|---|---|---
Dimensionality |  240 |  236 |  120 |  4
Accuracy |  63.23% |  64.27% |  65.48% |  65.08%
Precision |  66.25% |  65.15% |  67.18% |  79.72%
Recall |  66.24% |  63.57% |  64.12% |  58.99%
F1-score |  65.36% |  63.25% |  64.15% |  58.35%

When analyzing the confusion matrices (Figure 5), we observe an overall upward
trend in the diagonal values, indicating an increase in correct predictions on
a higher variance threshold (AVT). Notably, KL 0 and KL 2 experienced
consistent improvement in accuracy. The performance, however, varied for other
classes. While KL 1’s score declined post-Baseline and droped to zero at High
AVT, KL 3 showed a slight inconsistency, and KL 4 displayed a fluctuating
trend, peaking at Baseline and Mid AVT but falling at High AVT. The off-
diagonal elements reveal changes in KL confusion across different AVT levels.
Mainly, confusion between KL 0 and KL 1 decreased with an increasing AVT,
while the confusion between KL 1 and KL 2 increased. The recall table 4
further supplements these findings.

Figure 5: The confusion matrices for the EfficientNet model using the test data. The baseline indicates no intervention, whereas Low, Mid, and High represent AVT sets. The appendix tables 5,6,7 detail the NAS architecture discovered for each set. X-axis shows predicted labels; Y-axis shows true labels. Table 4: Recall report for the EfficientNet model across classes, derived from the confusion matrix diagonals. KL Label |  Baseline |  Low AVT |  Mid AVT |  High AVT
---|---|---|---|---
0 |  77.66% |  87.68% |  89.98% |  92.90%
1 |  39.51% |  17.56% |  14.15% |  0.00%
2 |  51.73% |  56.65% |  61.56% |  62.72%
3 |  68.18% |  64.77% |  60.80% |  68.75%
4 |  94.12% |  91.18% |  94.12% |  70.59%

##  4 Discussion

This study evaluated a novel method for enhancing the performance of pre-
existing models used for diagnosing Knee-Joint Osteoarthritis (KOA) using a
combination of adaptive variance thresholding (AVT) and neural architecture
search (NAS). This effort led to a notable increase in diagnostic accuracy, a
substantial reduction in the NAS input vector space, and an overall
improvement in model efficiency. More specifically, our approach improved
model accuracy on the Chen KOA model, from 69.63% at the baseline to 71.14% at
High AVT. This approach rendered the new solution within the top three
radiographic KOA classification solutions %[25, 35]. We observed that this
enhancement was exceptionally beneficial for KL 0 and KL 2, showing a
consistent increase in recall as the AVT level increased. Our internal model,
EfficientNetV2M, also benefited from the new method. Although the overall
model performance peaked at Mid AVT, we observed a remarkable jump in
Precision at High AVT, suggesting that extreme dimensionality reduction could
still lead to specific performance improvements. However, it is worth noting
that while the adaptive variance thresholding approach successfully enhanced
overall model accuracy, the level of improvement varied among different AVT
levels and ground-truth classes.

Furthermore, as evident from our experiments, the overall performance
improvement on the external model was slightly superior to the internal model.
The disparity may be attributed to the higher initial dimensionality of the
external model, which allowed for less extensive pruning by adaptive variance
thresholding. On the other hand, the internal model with lower initial
dimensionality might have had fewer redundant features to begin with, hence a
lesser margin for improvement. To gain further intuitive understanding, future
work could explore the inverse process of identifying convolutional filter
index based on the selected features post-AVT. This process could be an
intriguing method to visualize the feature maps primarily associated with the
Adaptive Variance Thresholding (AVT) sets.

Neural Architecture Search (NAS) played an instrumental role in this study,
helping optimize the architecture of the classifier models for our specific
task. The application of AVT resulted in a 60-fold reduction in the NAS input
vector space at high levels. This improved the model’s efficiency and
expedited the inference speed and hyperparameter search. Despite these
improvements, it is worth noting that we selected three threshold values
placed at equal intervals. Although the prospect of an increased AVT sampling
resolution could yield potentially superior results, it is essential to
balance this with the considerably higher computational cost it would incur.
In the Neural Architecture Search (NAS) process, we relied on validation
accuracy as our primary evaluative metric. However, applying various
evaluation metrics could potentially yield a diverse range of classifiers.
Therefore, exploring potential meta-metrics which involve weighted
combinations of various metrics may provide an edge in identifying
architectures that are better tailored for specific tasks.

In our study, a significant constraint arose from the lack of label noise
estimates for our dataset. This challenge is especially relevant for studies
such as ours, focused on radiographic Knee Osteoarthritis (KOA). The early
stages of KOA are usually shrouded in diagnostic uncertainty, thus making
substantial label noise not just a mere possibility but a considerable
expectation. Past research[23] corroborated this, revealing significant
discrepancies in labeling among medical experts. Without an estimate for label
noise, our ability to discern the maximum achievable performance on the task
is impeded. Moreover, another level of complexity stems from the potential for
extensive feature noise, which may result from poor-quality radiographs. It
remains unclear how our approach would respond under such circumstances,
adding an extra dimension to the limitations faced in the study.

An additional limitation in our study pertained to the resolution parameters
employed. Our evaluation was centered around transfer learning architectures
and a resolution size of 224 x 224. This raises a question of scale
adaptability: it is uncertain how our methodology would perform when applied
to features from larger receptive fields in the feature learning segment. The
influence of resolution size on model performance, especially when dealing
with higher-resolution inputs, remains a topic unexplored within the confines
of this study. This constitutes another key limitation that future studies
would seek to address.

While our study was rooted in the context of KOA classification, our
methodology’s potential may extend beyond this specific domain. Given the
inherent adaptability of AVT and NAS, our proposed paradigm has the potential
to generalize to other areas of medicine or even entirely different fields
that utilize deep transfer learning. However, such generalizations would
necessitate extensive further research.

In conclusion, this study introduced a novel paradigm that enhanced the
performance of existing deep-learning models for KOA classification. The
proposed combination of AVT and NAS improved model accuracy and efficiency,
and could potentially be applied to other models and diseases, presenting a
promising avenue for future automatic diagnostic research.

## Data Availability

Trained models and the AVT code are available in the Google drive repository.

## References

  * [1] F. Wang, L. P. Casalino, D. Khullar, Deep learning in medicine—promise, progress, and challenges, JAMA internal medicine 179 (3) (2019) 293–294.
  * [2] A. L. Beam, I. S. Kohane, Big data and machine learning in health care, Jama 319 (13) (2018) 1317–1318.
  * [3] Y. LeCun, Y. Bengio, G. Hinton, Deep learning, nature 521 (7553) (2015) 436–444.
  * [4] J. N. Kather, J. Krisam, P. Charoentong, T. Luedde, E. Herpel, C.-A. Weis, T. Gaiser, A. Marx, N. A. Valous, D. Ferber, others, Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study, PLoS medicine 16 (1) (2019) e1002730.
  * [5] P. Courtiol, C. Maussion, M. Moarii, E. Pronier, S. Pilcer, M. Sefta, P. Manceron, S. Toldo, M. Zaslavskiy, N. Le Stang, others, Deep learning-based classification of mesothelioma improves prediction of patient outcome, Nature medicine 25 (10) (2019) 1519–1525.
  * [6] A. Diamant, A. Chatterjee, M. Vallières, G. Shenouda, J. Seuntjens, Deep learning in head \& neck cancer outcome prediction, Scientific reports 9 (1) (2019) 1–10.
  * [7] A. Esteva, B. Kuprel, R. A. Novoa, J. Ko, S. M. Swetter, H. M. Blau, S. Thrun, Dermatologist-level classification of skin cancer with deep neural networks, nature 542 (7639) (2017) 115–118.
  * [8] Z. Han, B. Wei, Y. Zheng, Y. Yin, K. Li, S. Li, Breast cancer multi-classification from histopathological images with structured deep learning model, Scientific reports 7 (1) (2017) 1–10.
  * [9] M. Bakator, D. Radosav, Deep learning and medical diagnosis: A review of literature, Multimodal Technologies and Interaction 2 (3) (2018) 47.
  * [10] F. Prezja, S. Äyrämö, I. Pölönen, T. Ojala, S. Lahtinen, P. Ruusuvuori, T. Kuopio, Improved accuracy in colorectal cancer tissue decomposition through refinement of established deep learning solutions, Scientific Reports 13 (1) (2023) 15879.
  * [11] F. Prezja, L. Annala, S. Kiiskinen, S. Lahtinen, T. Ojala, P. Ruusuvuori, T. Kuopio, Improving performance in colorectal cancer histology decomposition using deep and ensemble machine learning, arXiv preprint arXiv:2310.16954 (2023).
  * [12] F. Isensee, P. F. Jaeger, S. A. A. Kohl, J. Petersen, K. H. Maier-Hein, nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation, Nature methods 18 (2) (2021) 203–211.
  * [13] X. Liu, L. Song, S. Liu, Y. Zhang, A review of deep-learning-based medical image segmentation methods, Sustainability 13 (3) (2021) 1224.
  * [14] M. J. M. Chuquicusma, S. Hussein, J. Burt, U. Bagci, How to fool radiologists with generative adversarial networks? A visual turing test for lung cancer diagnosis, in: 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), IEEE, 2018, pp. 240–244.
  * [15] F. Calimeri, A. Marzullo, C. Stamile, G. Terracina, Biomedical data augmentation using generative adversarial neural networks, in: International conference on artificial neural networks, Springer, 2017, pp. 626–634.
  * [16] M. Frid-Adar, I. Diamant, E. Klang, M. Amitai, J. Goldberger, H. Greenspan, GAN-based synthetic medical image augmentation for increased CNN performance in liver lesion classification, Neurocomputing 321 (2018) 321–331.
  * [17] V. Thambawita, J. L. Isaksen, S. A. Hicks, J. Ghouse, G. Ahlberg, A. Linneberg, N. Grarup, C. Ellervik, M. S. Olesen, T. Hansen, others, DeepFake electrocardiograms using generative adversarial networks are the beginning of the end for privacy issues in medicine, Scientific reports 11 (1) (2021) 1–8.
  * [18] L. Annala, N. Neittaanmäki, J. Paoli, O. Zaar, I. Pölönen, Generating hyperspectral skin cancer imagery using generative adversarial neural network, in: 2020 42nd Annual International Conference of the IEEE Engineering in Medicine \& Biology Society (EMBC), IEEE, 2020, pp. 1600–1603.
  * [19] H.-C. Shin, N. A. Tenenholtz, J. K. Rogers, C. G. Schwarz, M. L. Senjem, J. L. Gunter, K. P. Andriole, M. Michalski, Medical image synthesis for data augmentation and anonymization using generative adversarial networks, in: International workshop on simulation and synthesis in medical imaging, Springer, 2018, pp. 1–11.
  * [20] J. Yoon, L. N. Drumright, M. Van Der Schaar, Anonymization through data synthesis using generative adversarial networks (ads-gan), IEEE journal of biomedical and health informatics 24 (8) (2020) 2378–2388.
  * [21] A. Torfi, E. A. Fox, C. K. Reddy, Differentially private synthetic medical data generation using convolutional gans, Information Sciences 586 (2022) 485–500.
  * [22] S. N. Kasthurirathne, G. Dexter, S. J. Grannis, Generative Adversarial Networks for Creating Synthetic Free-Text Medical Data: A Proposal for Collaborative Research and Re-use of Machine Learning Models, in: AMIA Annual Symposium Proceedings, Vol. 2021, American Medical Informatics Association, 2021, p. 335.
  * [23] F. Prezja, J. Paloneva, I. Pölönen, E. Niinimäki, S. Äyrämö, DeepFake knee osteoarthritis X-rays from generative adversarial neural networks deceive medical experts and offer augmentation potential to automatic classification, Scientific Reports 12 (1) (2022) 1–16.  doi:10.1038/s41598-022-23081-4.
  * [24] F. Prezja, J. Paloneva, I. Pölönen, E. Niinimäki, S. Äyrämö, Synthetic (DeepFake) Knee Osteoarthritis X-ray Images from Generative Adversarial Neural Networks, Mendeley Data, V3 (2022).  doi:10.17632/fyybnjkw7v.3.
  * [25] P. S. Q. Yeoh, K. W. Lai, S. L. Goh, K. Hasikin, Y. C. Hum, Y. K. Tee, S. Dhanalakshmi, Emergence of deep learning in knee osteoarthritis diagnosis, Computational intelligence and neuroscience 2021 (2021).
  * [26] S. Saarakkala, P. Julkunen, P. Kiviranta, J. Mäkitalo, J. S. Jurvelin, R. K. Korhonen, Depth-wise progression of osteoarthritis in human articular cartilage: investigation of composition, structure and biomechanics, Osteoarthritis and Cartilage 18 (1) (2010) 73–81.
  * [27] M. S. Laasanen, J. Töyräs, R. K. Korhonen, J. Rieppo, S. Saarakkala, M. T. Nieminen, J. Hirvonen, J. S. Jurvelin, Biomechanical properties of knee articular cartilage, Biorheology 40 (1, 2, 3) (2003) 133–140.
  * [28] J. Hermans, M. A. Koopmanschap, S. M. A. Bierma-Zeinstra, J. H. van Linge, J. A. N. Verhaar, M. Reijman, A. Burdorf, Productivity costs and medical costs among working patients with knee osteoarthritis, Arthritis care \& research 64 (6) (2012) 853–861.
  * [29] D. J. Hunter, S. Bierma-Zeinstra, Osteoarthritis, The Lancet 393 (10182) (2019) 1745–1759.  doi:https://doi.org/10.1016/S0140-6736(19)30417-9.
URL https://www.sciencedirect.com/science/article/pii/S0140673619304179

  * [30] A. Tiulpin, S. Klein, S. M. A. Bierma-Zeinstra, J. Thevenot, E. Rahtu, J. v. Meurs, E. H. G. Oei, S. Saarakkala, Multimodal machine learning-based knee osteoarthritis progression prediction from plain radiographs and clinical data, Scientific reports 9 (1) (2019) 20038.
  * [31] A. Tiulpin, J. Thevenot, E. Rahtu, P. Lehenkari, S. Saarakkala, Automatic knee osteoarthritis diagnosis from plain radiographs: a deep learning-based approach, Scientific reports 8 (1) (2018) 1727.
  * [32] A. Tiulpin, S. Saarakkala, Automatic grading of individual knee osteoarthritis features in plain radiographs using deep convolutional neural networks, Diagnostics 10 (11) (2020) 932.
  * [33] C. for Disease Control, Prevention, others, HIPAA privacy rule and public health. Guidance from CDC and the US Department of Health and Human Services, MMWR: Morbidity and mortality weekly report 52 (Suppl 1) (2003) 1–17.
  * [34] P. Voigt, A. dem Bussche, The eu general data protection regulation (gdpr), A Practical Guide, 1st Ed., Cham: Springer International Publishing 10 (3152676) (2017) 10–5555.
  * [35] B. Zhang, J. Tan, K. Cho, G. Chang, C. M. Deniz, Attention-based cnn for kl grade classification: data from the osteoarthritis initiative, in: 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), IEEE, 2020, pp. 731–735.
  * [36] S. Khalid, T. Khalil, S. Nasreen, A survey of feature selection and feature extraction techniques in machine learning, in: 2014 science and information conference, IEEE, 2014, pp. 372–378.
  * [37] T. Elsken, J. H. Metzen, F. Hutter, Neural architecture search: A survey, The Journal of Machine Learning Research 20 (1) (2019) 1997–2017.
  * [38] H. Jin, F. Chollet, Q. Song, X. Hu, AutoKeras: An AutoML Library for Deep Learning, Journal of Machine Learning Research 24 (6) (2023) 1–6.
URL http://jmlr.org/papers/v24/20-1355.html

  * [39] P. Chen, L. Gao, X. Shi, K. Allen, L. Yang, Fully automatic knee osteoarthritis severity grading using deep neural networks with a novel ordinal loss, Computerized Medical Imaging and Graphics 75 (2019) 84–92.
  * [40] P. Chen, Knee osteoarthritis severity grading dataset, Mendeley Data 1 (2018) 21–23.
  * [41] J. H. Kellgren, J. Lawrence, Radiological assessment of osteo-arthrosis, Annals of the rheumatic diseases 16 (4) (1957) 494.
  * [42] Y. LeCun, Y. Bengio, others, Convolutional networks for images, speech, and time series, The handbook of brain theory and neural networks 3361 (10) (1995) 1995.
  * [43] V. Nair, G. E. Hinton, Rectified linear units improve Restricted Boltzmann machines, in: ICML 2010 - Proceedings, 27th International Conference on Machine Learning, 2010, pp. 807–814.
  * [44] M. Tan, Q. Le, Efficientnet: Rethinking model scaling for convolutional neural networks, in: International conference on machine learning, PMLR, 2019, pp. 6105–6114.
  * [45] J. Hu, L. Shen, G. Sun, Squeeze-and-excitation networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7132–7141.
  * [46] M. Tan, Q. Le, Efficientnetv2: Smaller models and faster training, in: International conference on machine learning, PMLR, 2021, pp. 10096–10106.
  * [47] F. Prezja, Deep Fast Vision: Accelerated Deep Transfer Learning Vision Prototyping and Beyond, \url{https://github.com/fabprezja/deep-fast-vision} (4 2023).  doi:10.5281/zenodo.7865289.
URL https://doi.org/10.5281/zenodo.7865289

  * [48] Scikit-Learn Developers, Code implementation of VarianceThreshold in Scikit-learn, \url{https://github.com/scikit-learn/scikit-learn/blob/7f9bad99d/sklearn/feature_selection/_variance_threshold.py#L14} (2023).
  * [49] M. Kuhn, K. Johnson, others, Applied predictive modeling, Vol. 26, Springer, 2013.
  * [50] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay, Scikit-learn: Machine Learning in {P}ython, Journal of Machine Learning Research 12 (2011) 2825–2830.
  * [51] C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. F. del Río, M. Wiebe, P. Peterson, P. Gérard-Marchant, K. Sheppard, T. Reddy, W. Weckesser, H. Abbasi, C. Gohlke, T. E. Oliphant, Array programming with {NumPy}, Nature 585 (7825) (2020) 357–362.  doi:10.1038/s41586-020-2649-2.
URL https://doi.org/10.1038/s41586-020-2649-2

## Acknowledgements

The authors extend their sincere gratitude to Kimmo Riihiaho, Rodion Enkel and
Leevi Lind for their exceptional support and invaluable discussions.

## Author contributions statement

Conceptualization: F. P.; Methodology: F. P.; Investigation: F. P.; Data
curation: All authors; Formal analysis: All authors; Writing – original draft:
F. P.; Writing – review & editing: All authors.

## Additional information

Competing interests All authors declare that they have no conflicts of
interest.

## Appendix

### EfficientNetV2M NAS

Table 5: NAS-selected architecture for the Low AVT Set of EfficientNet. Layer Type |  Output Shape |  Parameter Count
---|---|---
Input Layer |  236 |  -
Multi Category Encoding |  236 |  -
Normalization |  236 |  473
Dense Layer |  32 |  7584
ReLU Activation |  32 |  -
Dense Layer |  32 |  1056
ReLU Activation |  32 |  -
Dropout |  32 |  -
Dense Layer |  5 |  165
Softmax Activation |  5 |  -
Table 6: NAS-selected architecture for the Mid AVT Set of EfficientNet. Layer Type |  Output Shape |  Parameter Count
---|---|---
Input Layer |  120 |  -
Multi Category Encoding |  120 |  -
Dense Layer |  16 |  1936
ReLU Activation |  16 |  -
Dense Layer |  32 |  544
ReLU Activation |  32 |  -
Dense Layer |  5 |  165
Softmax Activation |  5 |  -
Table 7: NAS-selected architecture for the High AVT Set of EfficientNet. Layer Type |  Output Shape |  Parameter Count
---|---|---
Input Layer |  4 |  -
Multi Category Encoding |  4 |  -
Normalization |  4 |  9
Dense Layer |  32 |  160
ReLU Activation |  32 |  -
Dense Layer |  32 |  1056
ReLU Activation |  32 |  -
Dense Layer |  5 |  165
Softmax Activation |  5 |  -

###  4.1 Chen VGG19 Ordinal NAS

Table 8: NAS-selected architecture for the Low AVT Set of the external Chen VGG19 Ordinal. Layer Type |  Output Shape |  Parameter Count
---|---|---
Input Layer |  4034 |  -
Multi Category Encoding |  4034 |  -
Normalization |  4034 |  8069
Dense Layer |  32 |  129120
Batch Normalization |  32 |  128
ReLU Activation |  32 |  -
Dense Layer |  32 |  1056
Batch Normalization |  32 |  128
ReLU Activation |  32 |  -
Dense Layer |  5 |  165
Softmax Activation |  5 |  -
Table 9: NAS-selected architecture for the Mid AVT Set of the external Chen VGG19 Ordinal. Layer Type |  Output Shape |  Parameter Count
---|---|---
Input Layer |  2048 |  -
Multi Category Encoding |  2048 |  -
Normalization |  2048 |  4097
Dense Layer |  32 |  65568
Batch Normalization |  32 |  128
ReLU Activation |  32 |  -
Dense Layer |  1024 |  33792
Batch Normalization |  1024 |  4096
ReLU Activation |  1024 |  -
Dense Layer |  128 |  131200
Batch Normalization |  128 |  512
ReLU Activation |  128 |  -
Dropout |  128 |  -
Dense Layer |  5 |  645
Softmax Activation |  5 |  -
Table 10: NAS-selected architecture for the High AVT Set of the external Chen VGG19 Ordinal. Layer Type |  Output Shape |  Parameter Count
---|---|---
Input Layer |  62 |  -
Multi Category Encoding |  62 |  -
Dense Layer |  128 |  8064
ReLU Activation |  128 |  -
Dropout |  128 |  -
Dense Layer |  16 |  2064
ReLU Activation |  16 |  -
Dropout |  16 |  -
Dense Layer |  32 |  544
ReLU Activation |  32 |  -
Dropout |  32 |  -
Dense Layer |  5 |  165
Softmax Activation |  5 |  -