# Language Models Do Hard Arithmetic Tasks Easily and Hardly Do Easy
Arithmetic Tasks

Andrew Gambardella Yusuke Iwasawa Yutaka Matsuo
University of Tokyo
Correspondence<EMAIL_ADDRESS>
###### Abstract

The ability (and inability) of large language models (LLMs) to perform
arithmetic tasks has been the subject of much theoretical and practical
debate. We show that LLMs are frequently able to correctly and confidently
predict the first digit of $n$-digit by $m$-digit multiplication tasks without
using chain of thought reasoning, despite these tasks require compounding
operations to solve. Simultaneously, LLMs in practice often fail to correctly
or confidently predict the last digit of an $n$-digit by $m$-digit
multiplication, a task equivalent to $1$-digit by $1$-digit multiplication
which can be easily learned or memorized. We show that the latter task can be
solved more robustly when the LLM is conditioned on all of the correct higher-
order digits, which on average increases the confidence of the correct last
digit on $5$-digit by $5$-digit multiplication tasks using Llama 2-13B by over
230% (0.13→0.43) and Mistral-7B by 150% (0.22→0.55).

Language Models Do Hard Arithmetic Tasks Easily and Hardly Do Easy Arithmetic
Tasks

Andrew Gambardella††thanks: Correspondence:
<EMAIL_ADDRESS>Yusuke Iwasawa Yutaka Matsuo University
of Tokyo

##  1 Introduction

The development of large language models (LLMs) Brown et al. (2020) has given
new life to the deep learning revolution, and seen mass adoption within not
just the scientific community, but also society at large. These LLMs, being
the first known “general” machine learning model developed by humanity Morris
et al. (2024), have been applied to various tasks dealing with natural
language such as those commonly encountered in school curricula Hendrycks et
al. (2021), and even branching off into tasks such as text-to-image generation
Saharia et al. (2022) and hierarchical planning Wang et al. (2023).

Despite the generality and far-reaching consequences of LLMs, there are still
many significant limitations making difficult the direct application of LLMs
to certain tasks. One such limitation is the poor performance of LLMs on
arithmetic tasks, such as elementary addition, subtraction, multiplication,
and division Nogueira et al. (2021). Not only do modern LLMs perform poorly on
these tasks, but some tasks such as $n$-digit by $m$-digit multiplication and
division, which require compounding operations to solve, appear to be
unlearnable by pure autoregressive transformer architectures unless they
decompose the problem into multiple steps, such as with chain of thought
reasoning Wies et al. (2022); Liu et al. (2023). As such, several solutions
have been proposed, such as fine-tuning so that chain of thought reasoning is
automatically used for problems which require compounding operations Liu et
al. (2023); Kojima et al. (2022) or fine-tuning to call outside tools, such as
a calculator Schick et al. (2024).

While we most likely cannot expect simply training models with more parameters
to allow for the solving of tasks which require compounding operations without
chain of thought, we believe that analyzing the limitations and abilities of
autoregressive LLMs when attempting to solve these tasks directly may shed
light on unknown properties of LLMs. We therefore use Monte Carlo Dropout (MC
Dropout) Gal and Ghahramani (2016) to analyze the performance of LLMs which
were trained with dropout and which have open weights available, such as Llama
2 Touvron et al. (2023) and Mistral Jiang et al. (2023), in carrying out
arithmetic tasks.

MC Dropout allows one to interpret neural networks which were trained with
dropout as Bayesian neural networks, as neural networks trained with dropout
have been shown to be equivalent to a Bayesian approximation to a Gaussian
process. This allows one to obtain empirical Bayesian confidence distributions
over neural network weights or outputs by doing multiple forward passes
through the neural network with dropout on, during test time Gal and
Ghahramani (2016). MC Dropout is one of many ensemble-based methods for
uncertainty quantification Ovadia et al. (2019); Ashukha et al. (2020), and
has been applied to analyze the confidence of transformer architectures
Shelmanov et al. (2021) and to implement tree-based LLM prompting Mo and Xin
(2023).

Our results when applying MC Dropout to Llama 2 and Mistral in arithmetic
tasks were surprising. We found that all models could confidently and
correctly predict the first digit result of $n$-digit by $m$-digit
multiplication problems, despite it most likely being impossible for any
autoregressive LLM to have learned a general algorithm for doing so without
decomposing the problem into multiple steps, as finding this digit in general
requires solving the entire multiplication problem111Consider that the
highest-order digit of $31622776601683793319^{2}$ is $9$, but the highest-
order digit of $31622776601683793320^{2}$ is $1$.. We also found that all
models struggled to correctly output the last digit of $n$-digit by $m$-digit
multiplication problems, despite it being very easy to learn an algorithm for
doing so, as calculating the last digit is equivalent to $1$-digit by
$1$-digit multiplication. Finally, we show that the confidence of LLMs in
predicting the last digit can be increased by conditioning the generation of
the last digit on the correct intervening digits, despite the computation of
the last digit not depending on the correct computations of the higher-order
digits at all.

##  2 Experiments

We evaluate the HuggingFace Wolf et al. (2019) implementations of Llama 2-7B,
Llama 2-13B, and Mistral-7B Touvron et al. (2023); Jiang et al. (2023) in
2-shot settings, where the 2-shot examples are of correct $n$-digit by
$m$-digit multiplications. Sections 2.1 and 2.2 show results on the $3$-digit
by $3$-digit multiplication task $592*392$, and averages over multiple
problems with varying digit length are provided in Section 2.3. Details about
the prompt and hyperparameters are given in Appendix A, details about the
tokenizers for the models are given in Appendix B, and details about the use
of dropout in the training of the models is given in Appendix C.

###  2.1 Unconditional Answer Generation

We first study a version of the problem in which the answer is generated with
the language model conditioned on the few shot examples and the problem to be
solved, but is provided with none of the digits to be generated (i.e., the
normal few-shot arithmetic scenario), which we refer to as “unconditional”
generation in an abuse of terminology. Our main results for these experiments
are in Figures 1 and 2.

In Figure 1 we can see that both Llama 2-7B and Llama 2-13B can confidently
and correctly predict the first digit of the $3$-digit by $3$-digit
multiplication task $592*392$, which equals $232064$. This should be
surprising as it is not immediately apparent from the problem that the first
digit of the solution should be $2$, and the only way to discover this is to
compute the multiplication. As LLMs most likely cannot perform $n$-digit by
$m$-digit multiplication in the general case without decomposing the problem
into steps, the output of the first digit in this case is unlikely to be the
output of a multiplication algorithm learned by the LLM.

Figure 1: Confidence and accuracy of Llama 2-7B and Llama 2-13B predicting the
first digit of the result of $592*392$. Both language models are able to
confidently and correctly predict that the first digit should be $2$, despite
this not being immediately apparent from the problem.

Figure 2: Confidence and accuracy of Llama 2-7B and Llama 2-13B predicting the
sixth digit of the result of $592*392$. Neither are able to predict this digit
confidently, with the mode of the distribution on the “end string” character
in both cases. Both only output $4$ in about 20% of samples, despite it being
immediately apparent that the final digit should be $4$.

Conversely, in Figure 2, we can see that both Llama 2-7B and Llama 2-13B can
neither confidently nor correctly predict the last digit of the same problem,
despite doing so being equivalent to $1$-digit by $1$-digit multiplication.
This is a case in which any reasonable model should be able to confidently and
correctly solve the task, as not only could the algorithm to solve the task be
learned by an autoregressive language model, but the information needed to
solve this task could also very easily be memorized by language models with
billions of weights.

###  2.2 Conditional Answer Generation

Figure 3: Confidence and accuracy of Llama 2-7B and Llama 2-13B predicting the
last digit of the result of $592*392$, when conditioned on the first five
correct digits. The confidence in the correct answer being $4$ doubles for
Llama 2-7B and more than triples for Llama 2-13B, despite the computation of
the last digit not depending on the prior digits being correct at all.

Finally, we contrast the experiments given in Figures 1 and 2 with a third
experiment, in which the LLM is given all digits from the answer except for
the final digit, and is tasked with outputting solely the final digit, which
we refer to as “conditional” generation in an abuse of terminology. Results
for this experiment are given in Figure 3. In this case the confidence in the
correct output doubles for Llama 2-7B and triples for Llama 2-13B, with Llama
2-13B now having most of its probability mass on the correct last digit,
whereas it did not do so when generating the entire string at once (and
therefore often conditioning on incorrect prior digits). The fact that in both
cases, more probability mass is being put on the correct answer should be
surprising, as the computation of this digit does not depend on the
correctness of the higher-order digits in any way.

###  2.3 Ablation Over Digit Length

We provide further ablations over digit length with Llama 2-7B and 13B in
Table 1. Each subtable gives the confidence of the correct digit, averaged
over 10 different $n$-digit by $m$-digit multiplication problems each. We find
that the conclusions shown for a single example in Sections 2.1 and 2.2 hold
over varying multiplication problems and digit lengths in general. We further
provide similar Mistral-7B experiments in Table 2. While Mistral-7B is
stronger at arithmetic tasks than both Llama 2-7B and 13B, the same patterns
and conclusions found for Llama 2-7B and 13B also hold for Mistral-7B.

Llama 2-7B | Llama 2-13B
---|---
|
|
|

n m | 2 | 3 | 4 | 5
---|---|---|---|---
2 |  0.81 |  0.90 |  0.82 |  0.82
3 |  0.91 |  0.78 |  0.88 |  0.92
4 |  0.88 |  0.83 |  0.92 |  0.77
5 |  0.89 |  0.74 |  0.89 |  0.87
(a)

n m | 2 | 3 | 4 | 5
---|---|---|---|---
2 |  0.84 |  0.85 |  0.79 |  0.73
3 |  0.87 |  0.72 |  0.85 |  0.86
4 |  0.84 |  0.83 |  0.78 |  0.78
5 |  0.86 |  0.71 |  0.84 |  0.86
(b)

n m | 2 | 3 | 4 | 5
---|---|---|---|---
2 |  0.52 |  0.34 |  0.16 |  0.20
3 |  0.39 |  0.22 |  0.16 |  0.19
4 |  0.40 |  0.21 |  0.20 |  0.15
5 |  0.33 |  0.20 |  0.15 |  0.11
(c)

n m | 2 | 3 | 4 | 5
---|---|---|---|---
2 |  0.78 |  0.50 |  0.32 |  0.30
3 |  0.56 |  0.40 |  0.24 |  0.17
4 |  0.63 |  0.37 |  0.29 |  0.22
5 |  0.52 |  0.30 |  0.24 |  0.13
(d)

n m | 2 | 3 | 4 | 5
---|---|---|---|---
2 |  0.64 |  0.41 |  0.24 |  0.51
3 |  0.55 |  0.45 |  0.38 |  0.40
4 |  0.43 |  0.33 |  0.38 |  0.36
5 |  0.44 |  0.41 |  0.26 |  0.25
(e)

n m | 2 | 3 | 4 | 5
---|---|---|---|---
2 |  0.82 |  0.66 |  0.48 |  0.57
3 |  0.66 |  0.68 |  0.49 |  0.51
4 |  0.73 |  0.54 |  0.56 |  0.47
5 |  0.70 |  0.54 |  0.50 |  0.43
(f)

Table 1: Llama 2-7B and 13B generation average confidence of the correct first
digit (a, b), unconditional average confidence of the correct last digit (c,
d), and conditional average confidence of the correct last digit (e, f).

n m | 2 | 3 | 4 | 5
---|---|---|---|---
2 |  0.97 ± 0.03 |  0.98 ± 0.03 |  0.98 ± 0.02 |  1.00 ± 0.00
3 |  0.98 ± 0.03 |  1.00 ± 0.00 |  0.94 ± 0.09 |  0.93 ± 0.04
4 |  0.99 ± 0.01 |  0.87 ± 0.15 |  0.98 ± 0.04 |  0.82 ± 0.09
5 |  0.89 ± 0.1 |  0.94 ± 0.11 |  0.95 ± 0.06 |  0.99 ± 0.01
(a)

n m | 2 | 3 | 4 | 5
---|---|---|---|---
2 |  0.74 ± 0.06 |  0.57 ± 0.26 |  0.52 ± 0.29 |  0.41 ± 0.21
3 |  0.87 ± 0.10 |  0.70 ± 0.13 |  0.20 ± 0.12 |  0.11 ± 0.07
4 |  0.44 ± 0.14 |  0.70 ± 0.14 |  0.28 ± 0.23 |  0.30 ± 0.15
5 |  0.70 ± 0.10 |  0.33 ± 0.09 |  0.20 ± 0.13 |  0.22 ± 0.07
(b)

n m | 2 | 3 | 4 | 5
---|---|---|---|---
2 |  0.85 ± 0.23 |  0.83 ± 0.13 |  0.73 ± 0.21 |  0.76 ± 0.23
3 |  0.86 ± 0.13 |  0.85 ± 0.11 |  0.75 ± 0.22 |  0.57 ± 0.32
4 |  0.76 ± 0.17 |  0.62 ± 0.27 |  0.77 ± 0.26 |  0.59 ± 0.26
5 |  0.80 ± 0.18 |  0.68 ± 0.21 |  0.65 ± 0.26 |  0.55 ± 0.35
(c)

Table 2: Mistral-7B generation average and standard deviation confidence of
the correct first digit (a), unconditional average and standard deviation
confidence of the correct last digit (b), and conditional average and standard
deviation confidence of the correct last digit (c).

##  3 Discussion of Results

###  3.1 First Digit

It is most likely impossible for autoregressive LLMs to compute the first
digit of an $n$-digit by $m$-digit multiplication problem without decomposing
the problem into steps, especially given that the answer is being written
starting with the highest-order digit, and calculating the first digit depends
on the correct calculations of the lower-order digits.

LLMs _can_ , however, perform $1$-digit by $1$-digit multiplication. If these
LLMs were to internally round $592$ to $600$ and $392$ to $400$, it could
approximately solve for the highest-order digit in this way, as $600*400$ is a
computation that can be performed by autoregressive language models. We find
it likely that such a computation is occurring inside these LLMs, especially
as stochastic gradient descent is likely to find such “shortcuts.”

###  3.2 Last Digit

Both LLMs failing to predict the last digit when generating the entire string
autoregressively, and their confidence and accuracy in predicting the last
digit increasing when conditioned on correct prior digits, seem to be related,
and could stem from the view that autoregressive language models are
“exponentially diverging diffusion processes,” a view that several researchers
have argued informally LeCun et al. (2023), and has also recently been more
formally proven Dziri et al. (2023). The argument is essentially that if an
autoregressive LLM has some non-zero chance of making a mistake, then repeated
application of that LLM to generate a long string will cause errors to
compound exponentially.

This argument is not fully satisfying, however, for explaining the behavior of
LLMs in predicting the last digit. Not only should
$p(last\\_digit|wrong\\_intervening\\_digits)$ be the same as
$p(last\\_digit|correct\\_intervening\\_digits)$ due to the computation
involved (the last digit not depending on any other digits of the answer at
all), but the fact that LLMs are more correct and more confident when
conditioned on correct digits rather than wrong digits means that LLMs are
able to internally distinguish between the two states, despite not being able
to generate the entire correct string in the first place.

This finding may be related to recent results in the hallucination detection
literature, where it has been noted that the internal states of LLMs can be
used to detect when the conditioning text, including its own outputs, are
wrong Azaria and Mitchell (2023); Chen et al. (2024). It stands to reason that
if the internal states of an LLM differ depending on whether its conditioning
is correct or not, then further outputs which are autoregressively generated
based on these internal states may also differ. In other words, while previous
results show that LLMs may experience exponentially compounding errors, our
finding suggests this may occur not only due to faulty reasoning when using
incorrect intermediate steps, but also when the LLM “realizes” that it had
generated incorrect output, and then “believes” that its task is to continue
to do so. While out of the scope of this paper, we are interested in further
study of this property in particular, and its potential implications.

##  4 Conclusion

Here we present findings on the application of LLMs to arithmetic tasks, seen
through the lens of Monte Carlo Dropout. We found that the abilities of what
LLMs can do in practice, versus what the theory dictates should be possible
for LLMs to do, can be reversed in several cases. In particular, we found that
Llama 2 and Mistral could confidently and correctly output the first digit of
the result of $n$-digit by $m$-digit multiplication tasks despite most likely
being unable to in the general case, whereas they struggled with outputting
the last digit either correctly or confidently, a task which should be easily
learnable. We also found that accuracy and confidence in outputting the last
digit increases when the prior digits are correct, and we believe that this
finding is related to, and could have implications for, recent results in
hallucination detection.

##  5 Limitations

MC Dropout is a technique that is only applicable when neural network weights
are available and the neural network was trained with dropout. These
restrictions limit the number of language models that can be analyzed with the
techniques in this paper significantly, and crucially, state of the art
language models such as GPT-4 OpenAI (2023), Gemini Gemini Team et al. (2023),
and Claude Anthropic (2023) cannot be analyzed in this way by researchers
outside of OpenAI, Google, and Anthropic respectively. Such limitations make
clear the need for researchers to have access to language models with open
weights.

As we have restricted our analysis to Llama 2 and Mistral (which share similar
architectures), it is possible that our findings do not generalize to other
large language models, but given the very small number of existing language
models that can be analyzed in this way, it will be difficult to gauge the
generality of our findings until more language models which were trained with
dropout and have open weights are released.

## References

  * Anthropic (2023) Anthropic. 2023.  Model Card and Evaluations for Claude Models.
  * Ashukha et al. (2020) Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. 2020.  Pitfalls of in-domain uncertainty estimation and ensembling in deep learning.  In _8th International Conference on Learning Representations, ICLR 2020_.
  * Azaria and Mitchell (2023) Amos Azaria and Tom Mitchell. 2023.  The Internal State of an LLM Knows When It’s Lying.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in neural information processing systems 33_ , pages 1877–1901.
  * Chen et al. (2024) Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. 2024.  INSIDE: LLMs’ Internal States Retain the Power of Hallucination Detection.  In _The Twelfth International Conference on Learning Representations_.
  * Dziri et al. (2023) Nouha Dziri, Ximing Lu, Melanie Sclar, Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. 2023.  Faith and Fate: Limits of Transformers on Compositionality.  In _Thirty-seventh Conference on Neural Information Processing Systems_.
  * Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani. 2016.  Dropout as a Bayesian approximation: Representing model uncertainty in deep learning.  In _33rd International Conference on Machine Learning, ICML 2016_ , volume 3, pages 1651–1660.
  * Gemini Team et al. (2023) Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, and others. 2023.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_.
  * Gemma Team (2024) Gemma Team. 2024.  Gemma: Open models based on gemini research and technology.  _arXiv preprint arXiv:2403.08295_.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021.  Measuring Massive Multitask Language Understanding.  _Proceedings of the International Conference on Learning Representations (ICLR)_.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.  Mistral 7B.  _arXiv preprint arXiv:2310.06825_.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  _Advances in neural information processing systems_ , 35:22199–22213.
  * LeCun et al. (2023) Yann LeCun, Brenden Lake, Jacob Browning, David Chalmers, Ellie Pavlick, and Gary Lupyan. 2023.  Debate: Do language models need sensory grounding for meaning and understanding?
  * Liu et al. (2023) Tiedong Liu, Bryan Kian, and Hsiang Low. 2023.  Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks.  _arXiv preprint arXiv:2305.14201_.
  * Mo and Xin (2023) Shentong Mo and Miao Xin. 2023.  Tree of Uncertain Thoughts Reasoning for Large Language Models.  _arXiv preprint arXiv:2309.07694_.
  * Morris et al. (2024) Meredith Ringel Morris, Jascha Sohl-Dickstein, Noah Fiedel, Tris Warkentin, Allan Dafoe, Aleksandra Faust, Clement Farabet, and Shane Legg. 2024.  Levels of AGI: Operationalizing Progress on the Path to AGI.  _arXiv preprint arXiv:2311.02462_.
  * Nogueira et al. (2021) Rodrigo Nogueira, Zhiying Jiang, Jimmy Lin, and David R Cheriton. 2021.  Investigating the Limitations of Transformers with Simple Arithmetic Tasks.  Technical report.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  Technical report.
  * Ovadia et al. (2019) Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, and Jasper Snoek. 2019.  Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift.  In _Advances in Neural Information Processing Systems_ , volume 32.
  * Saharia et al. (2022) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Raphael Gontijo-Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. 2022.  Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding.  In _Advances in Neural Information Processing Systems_ , volume 35.
  * Schick et al. (2024) Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2024.  Toolformer: Language models can teach themselves to use tools.  _Advances in Neural Information Processing Systems_ , 36.
  * Shelmanov et al. (2021) Artem Shelmanov, Evgenii Tsymbalov, Dmitri Puzyrev, Kirill Fedyanin, Alexander Panchenko, and Maxim Panov. 2021.  How certain is your transformer? In _EACL 2021 - 16th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and others. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wang et al. (2023) Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023.  Voyager: An Open-Ended Embodied Agent with Large Language Models.  _arXiv preprint arXiv:2305.16291_.
  * Wies et al. (2022) Noam Wies, Yoav Levine, and Amnon Shashua. 2022.  Sub-Task Decomposition Enables Learning in Sequence to Sequence Tasks.  In _The Eleventh International Conference on Learning Representations_.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and others. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_.

##  Appendix A Prompt Format and Hyperparameters

The exact prompt used in Sections 2.1 and 2.2 is “$111*472=52392$.
$362*194=70228$. $\\{math\\_question\\}=\\{given\\_str\\}$” where
$math\\_question$ is the multiplication task, and $given\\_str$ is the empty
string in Section 2.1 and all but the last digit of the correct answer in
Section 2.2. In Section 2.3 the prompts are randomly generated 2-shot
$n$-digit by $m$-digit multiplication examples in the same format.

We set the dropout rate to be $0.1$, which is the dropout rate commonly used
in GPT applications, and appears to be the dropout rate used to train Llama 2
and Mistral. All sampling from LLMs is done deterministically other than the
stochasticity induced by dropout (i.e., we take argmax over logits). We
collect $100$ samples for each output.

##  Appendix B Tokenization

Both the Llama 2 and Mistral tokenizers have one single token for each digit,
$0$ to $9$, and no digits appear in any tokens other than these. This property
has been shown to be necessary to consistently perform even simple addition
tasks Nogueira et al. (2021).

##  Appendix C Dropout

The use of MC Dropout to model uncertainty in neural networks requires, as a
prerequisite, that the neural networks were trained with dropout. As we do not
know the exact training details of Llama 2 or Mistral, we cannot be fully
assured that they used dropout in training. We do, however, have very strong
reason to believe that they did use dropout during training, due to the fact
that both of these models still output reasonable text when dropout is turned
on. Conversely, the Gemma Gemma Team (2024) HuggingFace code also has dropout,
but when dropout is turned on even to only 10%, the model outputs are entirely
nonsensical (when attempting these experiments with Gemma, we do not even get
numbers as output when dropout is turned on, but do get reasonable output with
dropout turned off). The sort of robustness to neurons being dropped out that
can be seen in Llama 2 and Mistral only occurs in models that were actually
trained with dropout, and thus we can be fairly confident that the use of MC
Dropout here is appropriate.


# Music Auto-tagging with Robust Music Representation Learned via Domain
Adversarial Training

###### Abstract

Music auto-tagging is crucial for enhancing music discovery and
recommendation. Existing models in Music Information Retrieval (MIR) struggle
with real-world noise such as environmental and speech sounds in multimedia
content. This study proposes a method inspired by speech-related tasks to
enhance music auto-tagging performance in noisy settings. The approach
integrates Domain Adversarial Training (DAT) into the music domain, enabling
robust music representations that withstand noise. Unlike previous research,
this approach involves an additional pretraining phase for the domain
classifier, to avoid performance degradation in the subsequent phase. Adding
various synthesized noisy music data improves the model’s generalization
across different noise levels. The proposed architecture demonstrates enhanced
performance in music auto-tagging by effectively utilizing unlabeled noisy
music data. Additional experiments with supplementary unlabeled data further
improves the model’s performance, underscoring its robust generalization
capabilities and broad applicability.

Index Terms—  Robust Music Representation, Music Auto-tagging, Domain
Adversarial Training

##  1 Introduction

Music auto-tagging is the automated process of attaching relevant semantic
labels such as genre, mood, or instrument to musical tracks, usually enabled
by machine learning algorithms. This function is crucial for effective music
information retrieval, personalization, and recommendation systems,
predominantly in music-streaming platforms such as Spotify. These services
rely heavily on clean, pure music tracks and utilize comprehensive metadata
for each track to create a tailored and enriched user experience. This
metadata, originating from clean musical sources, allows for precise alignment
with individual user preferences.

Fig. 1: Feature extraction from clean and noisy music tracks in robust music
representation learning. The extractor aims to produce closely positioned
embeddings for the same track, regardless of audio quality.

Furthermore, music auto-tagging is not only crucial for enhancing search
capabilities and user accessibility in music streaming services, but also
vital for catering to the specific needs of users who demand more personalized
recommendations and detailed search options for music content in video-
streaming platforms like YouTube [1]. Through the use of meaningful semantic
tags, users can more effectively search for specific genres, artists, and
moods, thereby enhancing the overall user accessibility and discoverability of
music contents. However, the challenge is compounded on video-streaming
services where music tracks are frequently mixed with real-world noises like
crowd sounds and applause. This complicates the task for existing auto-tagging
algorithms, which are predominantly trained on clean music tracks. Given the
limited diversity of current tags and the sheer volume of diverse music-
related content uploaded daily, there is a compelling need to advance auto-
tagging techniques. Such improvements will not only make searching more
efficient but also significantly contribute to delivering a more personalized
and enriched user experience across various platforms.

As existing models have difficulty maintaining consistent feature extraction
from both clean and noisy versions of the same track, we propose creating a
dataset that includes both clean and noisy versions of identical music tracks.
To enhance the robustness of music representation, we employ the technique of
Domain Adversarial Training (DAT) [2] which was effective in improving noise
robustness of speech representation with performing downstream tasks such as
Automatic Speech Recognition (ASR) or Speaker Identification (SID) [3]. This
method is designed to condition the feature extractor to be indifferent to
whether a music track is clean or noisy, effectively diminishing the
distinctions between the clean and noise domains. It accomplishes this by
closely aligning the embeddings of identical music tracks across both clean
and noisy versions. As a result, this strategy allows the model to perform
downstream tasks on noisy input representations as effectively as it would on
clean representations. This refined framework is designed to facilitate model
training for the recognition of musical elements within noisy environments,
thereby improving overall model performance across a diverse range of auditory
conditions.

Fig. 2: The proposed architecture and training process. Overall structure is
composed of Feature Extractor (FE, pink), Domain Classifier (DC, yellow), and
Label Predictor (LP, green). The training process is set to 3 steps for 1)
pretraining FE, 2) pretraining DC, and 3) finetuning FE and training LP. In
contrast, for both the baseline and oracle configurations, only the FE and LP
are utilized, leading to a simplified two-step training process.

##  2 RELATED WORKS

In the realm of music auto-tagging, Convolutional Neural Network (CNN)-based
models [4] have demonstrated noteworthy performance, as evidenced by multiple
studies [5, 6, 7, 8, 9]. [5] employed a hybrid model, integrating Recurrent
Neural Networks (RNNs) with CNNs to more effectively capture temporal patterns
in the data. Another study by [6] deployed fully convolutional neural networks
comprising multiple layers but without fully connected layers, thereby
reducing the model’s parameter count. While the majority of research in this
area traditionally utilizes mel-spectrogram inputs, [7] diverged by using
sample-level input without the mel-spectrogram conversion. Adding another
layer of complexity, [9] incorporated a Squeeze-and-Excitation (SE) block into
their sample-level input model, enhancing the extraction of representational
features.

Nevertheless, given that music representations gain interpretability and
significance when modeled sequentially, [10] employed the Transformer
architecture [11] as the backbone model for their study. Further, in
subsequent work, [12] adopted a semi-supervised approach in conjunction with
the Transformer, highlighting the insufficiency of available data specifically
for music auto-tagging. Furthermore, [13, 14, 15], collectively advance the
field of music AI by exploring efficient pre-training strategies for audio
understanding, [14] introducing a novel self-supervised model (MERT) for
nuanced music audio analysis, and [15] establishing MARBLE, a comprehensive
benchmark for evaluating music information retrieval systems.

The Domain Adversarial Training (DAT) [16] approach has been effective in
making speech representation more robust, as shown in previous work [3]. In
this setup, clean audio is used as the source domain, while different types of
distorted audio make up the target domain. By applying the reversed gradient
of the domain classifier’s loss, the feature extractor can be tuned to lessen
the difference caused by these distortions. This adjustment allows for better
performance in downstream tasks using the label predictor. In this paper, we
use the DAT approach from earlier research [3], but with changes in domain
settings and training steps, which we will discuss in subsequent sections.

##  3 METHOD

###  3.1 Architecture

In the proposed architecture, the model is composed of three primary
components: Feature Extractor, Domain Classifier, and Label Predictor.

The Feature Extractor (FE) is first trained to extract general music embedding
from the input audio, then finetuned to blur the distinction between clean and
noisy input. In this paper, we employ CLMR [17], with SampleCNN [18] serving
as the encoder, whose backbone is SimCLR [19]. For FE, we exclusively utilize
the encoder component of CLMR. The output embedding of the Feature Extractor
then subsequently serves as the input for both the Domain Classifier and the
Label Predictor.

The Domain Classifier (DC) is tasked with determining the origin of the
embedding—whether it is derived from a clean or noisy audio source. The
structure of the DC is based on the original DAT research [2]. This module
outputs a scalar that classifies whether the embedding originated from the
clean source input or the noisy target input, which comprises simple fully-
connected layers, accompanied by activation functions and batch normalization.

The Label Predictor (LP) focuses on the downstream task of music auto-tagging
based on the provided embedding. Among the models in the proposed
architecture, the LP stands out with its compact structure and minimal number
of layers and parameters. This module takes the output from the FE as input
and sequentially processes it through two fully-connected layers, with a ReLU
activation function in between.

Fig. 3: Proposed dataset configuration: The music dataset (red) utilizes MTAT
[20] and the music split from Musan [21]. The real-world noise dataset
(yellow) incorporates Audioset [22] and the noise split from Musan Synthesized
samples combining music and noise are designated as target domain data. During
training, source (src) and target (trg) domain samples do not overlap. Note
that the Musan noise dataset is exclusively employed for creating the test
set, while it is not used in the formation of the validation set.

###  3.2 Training Process

Our proposed training methodology incorporates elements from previous work
[3], but introduces an additional pretraining step for the DC, resulting in a
three-step process in total.

The initial stage involves pretraining the FE. This phase allows the FE to
gain a general understanding of both music representations and real-world
noises, employing a contrastive loss function [17, 19] for learning. In this
step, both the encoder and the projector are trained, although only the
encoder is used in subsequent steps. For an input audio $x_{i}$, the FE
extracts the embedding of the audio $e_{i}$ as the result of
$\text{FE}(x_{i})$, which is a 512-dimensional vector.

$\displaystyle(e_{i},e_{j})$ $\displaystyle=\text{FE}(x_{i},x_{j})$
$\displaystyle(h_{i},h_{j})$ $\displaystyle=\mathrm{proj}(e_{i},e_{j})$
$\displaystyle\mathcal{L}_{FE}$
$\displaystyle=\text{ContrastiveLoss}(h_{i},h_{j})$

The second stage focuses on the pretraining of the DC, with the parameters of
the FE being frozen. This separate training step for DC deviates from the
methods outlined in [2] and [3]. We opted for this separation of training
steps after observing a decline in performance when the Domain Classifier (DC)
was trained alongside other components. Given an input embedding vector
$e_{i}$, the DC performs binary classification to determine whether the input
originates from clean or noisy musical audio.

$\displaystyle\hat{d_{i}}$ $\displaystyle=\text{DC}(e_{i})$
$\displaystyle\mathcal{L}_{DC}$
$\displaystyle=\text{BCELoss}(\hat{d_{i}},d_{i})$

The final stage is dedicated to the finetuning of the FE and the training of
the LP. At this stage, the parameters of the DC are frozen, based on the
assessment that it has achieved adequate binary classification performance. In
contrast, LP is trained from scratch. For an input embedding vector $e_{i}$,
the LP outputs a 50-dimensional vector, which corresponds to the
classification of the 50 multi-tags in the MTAT dataset [20]. The combined
loss of the LP and DC informs the fine-tuning of the FE, yielding the total
loss function as described in the equation. Note that the gradient of the DC
is negated which forces the FE to blur the distinction between clean and noisy
domain. Also, $\mathcal{L}_{\text{LP}}^{\text{trg}}$ is not applicable, as tag
labels for the target domain are assumed to be unavailable.

$\displaystyle(e^{\text{src}},e^{\text{trg}})=\text{FE}(x^{\text{src}},x^{\text{trg}}),\quad(\hat{d}^{\text{src}},\hat{d}^{\text{trg}})=\text{DC}(e^{\text{src}},e^{\text{trg}})$
$\displaystyle\mathcal{L}_{\text{LP}}=\text{BCEWithLogitsLoss}(\hat{l}^{\text{src}},l^{\text{src}}),\quad\hat{l}^{\text{src}}=\text{LP}(e^{\text{src}})$
$\displaystyle\mathcal{L}_{\text{Total}}=\mathcal{L}_{\text{LP}}^{\text{src}}+\lambda\\{\mathcal{L}_{\text{DC}}^{\text{src}}+\mathcal{L}_{\text{DC}}^{\text{trg}}\\}$

##  4 DATASET

In our experiments, we used the MTG-Jamendo dataset [23] for the pretraining
of the FE and used the MagnaTagATune (MTAT) dataset [20] for music auto-
tagging tasks. Regarding the size of the full Jamendo dataset, we selected a
subset comprising audio files with both genre and mood/theme tags. For the
real-world noise dataset, Audioset [22] is used after filtering to exclude any
data containing music-related tags, such as those denoting musical notes, for
example, ‘bell’ or ‘ding’. Additionally, the Musan dataset [21] is used as an
extra dataset which is provided in music, speech, and noise splits. We
employed the music and noise splits for training and testing phases,
respectively.

###  4.1 Data Configuration

For the pretraining of the FE, we utilized the Jamendo and Audioset datasets.
The audio samples from Jamendo are subjected to random augmentations, such as
pitch shifting and the application of filters, following the methodology
outlined by [17]. Additionally, to improve the model’s generalization
capabilities with respect to noisy musical audio, we synthesized random
samples from Audioset with the Jamendo audio samples.

Regarding the subsequent steps in our training process, we employed the MTAT
and Audioset datasets under diverse experimental configurations. In the
baseline setting, we assumed that only clean audio samples with corresponding
tags are accessible, which aligns with the existing frameworks for auto-
tagging. For the oracle setup, we assumed the availability of tags for both
clean and noisy audio samples, a condition that is not feasible in real-world
scenarios. In the proposed experimental setting, we made use of clean audio
samples with tags for the source domain, and synthesized noisy samples without
tags for the target domain (proposed (a)). Lastly, to demonstrate the capacity
for further model training and generalization with noisy musical samples, we
also utilized additional noisy samples synthesized from the Musan music split
and Audioset, excluding any associated tags (proposed (b)). This final setting
serves to underscore the efficacy of the proposed architecture in
accommodating real-world audio conditions.

For the validation and test phase, we did not split the MTAT dataset but fully
and repeatedly used the validation and test dataset in five different
conditions. First we used music audio samples without any noise added for
clean source domain. From second to the last condition, we added noise but in
different sound-to-noise ratio (SNR) conditions from -5 to 10.

Table 1: The test AUC and AP metrics for the baseline, oracle, and proposed
configurations (a) and (b), evaluated with the inclusion of either 1 or 2
noises in the synthesized noisy music data.

Table 2: The test AUC and AP metrics with the inclusion of 4 noises.

##  5 EXPERIMENT

In the experiment, the Adam optimizer [24] is used across all training phases.
Specifically, the learning rate is set to 3e-4 for the pretraining of the FE,
and 1e-4 for pretraining DC, finetuning FE, and training LP. The length of
each audio input is fixed to 59,049 samples, in alignment with previous work
[17]. Furthermore, a uniform sample rate of 22,050 is applied across all
training steps, with resampling if necessary.

To simulate various real-world noisy conditions, we synthesized music samples
by combining them with one, two, or four different types of noise samples.
During each data retrieval from the dataset, each music sample is normalized
and randomly mixed with noise samples. The SNR for these mixtures is also
randomly selected from a predefined range of [-10, 10]. For the validation and
test phases, target samples are synthesized according to predefined SNRs as
described in Figure 3.

For the evaluation metrics, we employed the area under the receiver operating
characteristic curve (AUC) and average precision (AP). The experimental test
results for source domain indicate that the baseline performance was either
comparable to or slightly better than our proposed approach (Table 1, 2). This
suggests that incorporating additional noisy music samples could potentially
impact the existing performance adversely. However, as the number of noise
samples increases, our proposed method demonstrates enhanced generalization
and robustness (Table 3). Notably, the proposed (b) configuration consistently
delivered the best performance, underscoring the idea that even a modest
amount of extra, noisy, and unlabeled data can improve the model’s
performance. Although the performance differences between proposed (a) and
proposed (b) may appear subtle, it is important to note that the quantity of
additional data samples per epoch in proposed (b) is approximately 17 times
less than the unlabeled data used in proposed (a).

Additionally, tests conducted on the Musan noise dataset corroborate the
model’s consistent performance across different noise conditions. These
results further support the notion that the FE’s embeddings are robust when
exposed to a variety of noise types, thereby enhancing the model’s overall
generalization capabilities. Lastly, through all the evaluations, the model
demonstrated enhanced performance even under challenging conditions involving
various signal-to-noise ratios, affirming its robustness and utility for music
auto-tagging in noisy scenarios.

##  6 CONCLUSION

In this work, we introduced a novel framework for improving music auto-tagging
performance by leveraging unlabeled noisy music data. We employed Domain
Adversarial Training (DAT) to enhance the robustness of feature extraction,
making it capable of handling both clean and noisy audio inputs effectively.
Our experimental setup included diverse datasets such as MTG-Jamendo,
MagnaTagATune (MTAT), Audioset, and Musan, and incorporated various real-world
noise conditions to simulate realistic scenarios.

Our evaluations, using metrics such as AUC and AP, indicates promising
results. While the incorporation of noisy data had a nuanced effect on
performance, we found that as the variety of noise increased, the model’s
robustness improved, suggesting that our approach has strong generalization
capabilities. In particular, the configuration using extra, unlabeled noisy
data showed performance gains, even when the additional data volume was
relatively small.

Additional tests on the Musan noise dataset corroborated the model’s
consistency and robustness across various noise conditions. These results
affirm that the feature extractor’s embeddings are resilient to noise, thereby
extending the applicability and generalizability of our model for music auto-
tagging in noisy environments.

##  7 Acknowledgement

This work was partly supported by Culture, Sports and Tourism R&D Program
through the Korea Creative Content Agency grant funded by the Ministry of
Culture, Sports and Tourism in 2022 (No.R2022020066, 1/2) and the National
Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT)
(No. RS-2023-00219429, 1/2).

## References

  * [1] Ahyeon Choi, Eunsik Shin, Haesun Joung, Joongseek Lee, and Kyogu Lee,  “Towards a new interface for music listening: A user experience study on youtube,”  arXiv preprint arXiv:2307.14718, 2023.
  * [2] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky,  “Domain-adversarial training of neural networks,”  The journal of machine learning research, vol. 17, no. 1, pp. 2096–2030, 2016.
  * [3] Kuan Po Huang, Yu-Kuan Fu, Yu Zhang, and Hung-yi Lee,  “Improving distortion robustness of self-supervised speech processing tasks with domain adaptation,”  arXiv preprint arXiv:2203.16104, 2022.
  * [4] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton,  “Imagenet classification with deep convolutional neural networks,”  Advances in neural information processing systems, vol. 25, 2012.
  * [5] Keunwoo Choi, György Fazekas, Mark Sandler, and Kyunghyun Cho,  “Convolutional recurrent neural networks for music classification,”  in 2017 IEEE International conference on acoustics, speech and signal processing (ICASSP). IEEE, 2017, pp. 2392–2396.
  * [6] Keunwoo Choi, George Fazekas, and Mark Sandler,  “Automatic tagging using deep convolutional neural networks,”  arXiv preprint arXiv:1606.00298, 2016.
  * [7] Jongpil Lee, Jiyoung Park, Keunhyoung Luke Kim, and Juhan Nam,  “Sample-level deep convolutional neural networks for music auto-tagging using raw waveforms,”  arXiv preprint arXiv:1703.01789, 2017.
  * [8] Jordi Pons, Oriol Nieto, Matthew Prockup, Erik Schmidt, Andreas Ehmann, and Xavier Serra,  “End-to-end learning for music audio tagging at scale,”  arXiv preprint arXiv:1711.02520, 2017.
  * [9] Taejun Kim, Jongpil Lee, and Juhan Nam,  “Sample-level cnn architectures for music auto-tagging using raw waveforms,”  in 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2018, pp. 366–370.
  * [10] Minz Won, Sanghyuk Chun, and Xavier Serra,  “Toward interpretable music tagging with self-attention,”  arXiv preprint arXiv:1906.04972, 2019.
  * [11] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  Advances in neural information processing systems, vol. 30, 2017.
  * [12] Minz Won, Keunwoo Choi, and Xavier Serra,  “Semi-supervised music tagging transformer,”  arXiv preprint arXiv:2111.13457, 2021.
  * [13] Matthew C McCallum, Filip Korzeniowski, Sergio Oramas, Fabien Gouyon, and Andreas F Ehmann,  “Supervised and unsupervised learning of audio representations for music understanding,”  arXiv preprint arXiv:2210.03799, 2022.
  * [14] Yizhi Li, Ruibin Yuan, Ge Zhang, Yinghao Ma, Xingran Chen, Hanzhi Yin, Chenghua Lin, Anton Ragni, Emmanouil Benetos, Norbert Gyenge, et al.,  “Mert: Acoustic music understanding model with large-scale self-supervised training,”  arXiv preprint arXiv:2306.00107, 2023.
  * [15] Ruibin Yuan, Yinghao Ma, Yizhi Li, Ge Zhang, Xingran Chen, Hanzhi Yin, Le Zhuo, Yiqi Liu, Jiawen Huang, Zeyue Tian, et al.,  “Marble: Music audio representation benchmark for universal evaluation,”  arXiv preprint arXiv:2306.10548, 2023.
  * [16] Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, and Mario Marchand,  “Domain-adversarial neural networks,”  arXiv preprint arXiv:1412.4446, 2014.
  * [17] Janne Spijkervet and John Ashley Burgoyne,  “Contrastive learning of musical representations,”  arXiv preprint arXiv:2103.09410, 2021.
  * [18] Jongpil Lee, Jiyoung Park, Keunhyoung Luke Kim, and Juhan Nam,  “Samplecnn: End-to-end deep convolutional neural networks using very small filters for music classification,”  Applied Sciences, vol. 8, no. 1, pp. 150, 2018.
  * [19] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton,  “A simple framework for contrastive learning of visual representations,”  in International conference on machine learning. PMLR, 2020, pp. 1597–1607.
  * [20] Edith Law, Kris West, Michael I Mandel, Mert Bay, and J Stephen Downie,  “Evaluation of algorithms using games: The case of music tagging.,”  in ISMIR. Citeseer, 2009, pp. 387–392.
  * [21] David Snyder, Guoguo Chen, and Daniel Povey,  “Musan: A music, speech, and noise corpus,”  arXiv preprint arXiv:1510.08484, 2015.
  * [22] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter,  “Audio set: An ontology and human-labeled dataset for audio events,”  in 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2017, pp. 776–780.
  * [23] Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, and Xavier Serra,  “The mtg-jamendo dataset for automatic music tagging,”  ICML, 2019.
  * [24] Diederik P Kingma and Jimmy Ba,  “Adam: A method for stochastic optimization,”  arXiv preprint arXiv:1412.6980, 2014.


# SpectralNET: Exploring Spatial-Spectral WaveletCNN for Hyperspectral Image
Classification

Tanmay Chakraborty and Utkarsh Trehan
MALIS - Machine Learning and Intelligent Systems
Lecturer: Dr. Maria A. Zuluaga
Department of Data Science and Engineering, EURECOM, Biot, France.
Emails<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Hyperspectral Image (HSI) classification using Convolutional Neural Networks
(CNN) is widely found in the current literature. Approaches vary from using
SVMs to 2D CNNs, 3D CNNs, 3D-2D CNNs. Besides 3D-2D CNNs and FuSENet, the
other approaches do not consider both the spectral and spatial features
together for HSI classification task, thereby resulting in poor performances.
3D CNNs are computationally heavy and are not widely used, while 2D CNNs do
not consider multi-resolution processing of images, and only limits itself to
the spatial features. Even though 3D-2D CNNs try to model the spectral and
spatial features their performance seems limited when applied over multiple
dataset. In this article, we propose SpectralNET, a wavelet CNN, which is a
variation of 2D CNN for multi-resolution HSI classification. A wavelet CNN
uses layers of wavelet transform to bring out spectral features. Computing a
wavelet transform is lighter than computing 3D CNN. The spectral features
extracted are then connected to the 2D CNN which bring out the spatial
features, thereby creating a spatial-spectral feature vector for
classification. Overall a better model is achieved that can classify multi-
resolution HSI data with high accuracy. Experiments performed with SpectralNET
on benchmark dataset, i.e. Indian Pines, University of Pavia, and Salinas
Scenes confirm the superiority of proposed SpectralNET with respect to the
state-of-the-art methods. The code is publicly available in
https://github.com/tanmay-ty/SpectralNET.

###### Index Terms:

Wavelet CNN; 2-D Convolutional Neural Net (CNN); 3-D Convolutional Neural Net;
SpectralNET; hyperspectral image (HSI); spectral-spatial features; HSI
classification.

##  I Introduction

A Hyperspectral Image (HSI) is a high dimension image cube, where each band
stores the intensity values of the pixels in a particular spectrum [1]. HSI
classification is the task of correctly predicting the different pixel values
associated with the different classes present in a remotely sensed HSI.
Applications include urban development, detection of land changes, military
applications, land cover analysis, crop detection etc. A key feature of HSI is
they contain both spectral and spatial information.

Deep-learning based methods specially CNNs perform extremely well on image
data. In recent works, HSI classification using different CNN models is also
seen besides traditional hand-extracted feature based models [2]. Most models
are based on 2D CNN, and 3D CNN [3]. Due to satisfactory performances of the
two independent models [4], hybrid 3D-2D CNNs have also been proposed in the
literature [5]. FuSENet is another model proposed in literature for HSI
classification [6].

In [7], a band weighing strategy has been proposed that utilizes multiple
binary support vector machines (SVM) in order to maximize the spectral
distances between each class of a remotely sensed HSI. Their method was able
to weight the spectral bands and improve classification results. A similar
approach using SVMs has been proposed in [8], where the authors explored
discrete space model (DSM) to transform continuous spectral features into
discrete feature space, they utilized a composite kernel to take into account
the spectral and spatial features. This pre-processing step improved the
performance of SVMs for HSI classification. Kernel based approaches has also
been found in the literature. In [9], spectral similarity based kernels has
been developed and utilized along with the RBF kernel in a SVM. For the
problem in hand they concluded spectral similarity based kernels outperform
traditional SVM kernels.

The work in [10], adapts and improves the traditional low-rank representation
(LRR) to the HSI classification problem. Locality-and structure-regularized
LRR combines both the spectral and spatial features to explore the local
similarity of pixels. The authors of [11], applied the concept of spectral
gradient for HSI classification. They extracted the spatial features through a
random forest algorithm and spectral features through spectral gradients. Then
they perform a multi-scale fusion to integrate spatial-spectral features for
the SVM to perform classification. The work in [12], introduced deep support
vector machines (DSVM) for HSI classification. The model was able to
outperform most of the state-of-the-art algorithms including all the variants
of traditional SVMs.

In [13], a 3D octave CNN has been proposed which factorizes the mixed
frequency feature map to reduce the spatial redundancy obtained when using a
traditional 3D CNN with HSI. The authors of [14], utilized pseudo 3D blocks
with a densely connected network. Their pseudo 3D blocks can capture both
spectral and spatial features simultaneously compared to a traditional 3D CNN.
The article [15], utilized small 3D patches extracted from the original HSI
cube to train a 3D CNN with 3D kernel. In the following works [16], residual
connections were added to a 3D CNN in order to assimilate both high and low
level features present in a HSI and improve classification results. The work
of [17], studied the effect of dimensionality reduction of HSI on 3D CNNs.
They concluded reducing the dimension of the training image reduced training
time by 60%.

In [18], a 3D-2D CNN has been proposed for HSI classification. As a pre-
processing the authors utilized channel wise shift and channel wise weighting
to highlight the different spectral bands. In [4], 2D-3D CNN has been utilized
with multi band feature fusion mechanism. This mechanism allows them to fuse
both shallow and deep features in spectral band, which improves the feature
vector sent into the dense layer.The work proposed in [19], introduces
adaptive spectral unmixing into a 3D-2D CNN along with a early exit strategy.
The early exit strategy reduces computational cost for easy samples. In [20],
a residual hybrid 3D-2D CNN has been proposed, which has further been improved
in [21] and is currently the state-of-the-art.

Efforts have also been made with Recurrent Neural Networks (RNN), Generative
Adversarial Networks (GAN), Graph CNNs [22], and Squeeze and Excitation
Residual Network [23]. RNNs consider the spectral signature of the HSI as a
sequence in order to learn discriminative features [24].

Even though the 3D-2D CNNs model both the spatial and spectral features from a
HSI cube, their model performance when applied over multiple dataset seems
limited. 3D CNNs are also computationally expensive over 2D CNNs. So a method
involving only 2D CNN as well as the power of extracting both spatial and
spectral features is desirable.

In this article, a 2D wavelet CNN has been proposed for HSI classification.
The work in [25], established wavelet transform as a good feature extractor
for HSI classification task. Thus fusing the wavelet transform into a 2D CNN
model brings out both the spectral and spatial features from a HSI. These
features are then concatenated channel wise and sent as an input to the dense
classification layers of the 2D CNN. The developed model uses Factor Analysis
(FA) as a pre-processing step to reduce the huge dimensionality of HSI. Then
patches are extracted and sent into the CNN. This reduces the training time as
well. The spectral features coming from wavelet transform are computationally
lighter as well compared to a 3D CNN. The model outperforms all previous
models and paves the way for wavelet CNN in multi-resolution image
classification. This model has been named SpectralNET in this paper.

The rest of the paper is arranged in the following way Section II, describes
the SpectralNET model in details, Section III contains our experiments and
discussions, and the paper is concluded in Section IV.

##  II SpectralNET

The conventional 2D CNN can be considered a limited version of a multi-
resolution CNN that can consider both spectral and spatial information [26].
Previous works have been successful in establishing the convolution and
pooling function in a 2D CNN as filtering and downsampling [27]. A basic CNN
can be mathematically represented as the weighed sum of nearest neighbours
with an added constant bias.

###  II-A Background for SpectralNET

Figure 1: Input HSI cube is pre-processed using Factor Analysis (FA) to reduce
the dimention to 3. Patches are extracted from the pre-processed image and
sent as an input to the SpectralNET model. SpectralNET model architecture with
4-level wavelet decomposition of the input HSI patch. The input kernel size is
3x3 with 1x1 padding. The output batch channel size is denoted by the numbers
written after conv. To reduce feature map 3x3 kernels with stride 2 and 1x1
padding are used. The wavelet transformed features are added channelwise. To
prevent the gradient from vanishing projection shortcuts are utilized with 1x1
convolutions. An average pooling layer is used globally after which the output
is sent to the fully connected layers with dropout neurons.

Given an input vector $X_{n}$ with corresponding labels $Y_{n}$ from the
$\mathbb{R}^{n}$ space. In equation 1, $Y_{i}$ is a label from
$\mathit{Y_{n}}$ labels and $X_{i}$ is the corresponding sample from
$\mathit{X_{n}}$. $W_{j}$ is the weight defined by a filtering kernel. $N_{i}$
are neighbouring i data.

$y_{i}=\sum_{j\in N_{i}}W_{j}X_{j}$ (1)

The equation 1 can be simply considered as the convolution of $X_{i}$ and
kernel $W_{j}$ and can be rewritten as $\mathbf{Y=X*W}$. This is called the
convolution layer of a CNN, where $\mathbf{W}$ is in $\mathbb{R}^{o}$. The
output of the convolution layers are typically big and needs to be pooled down
before feeding it to the next layer. The pooling layers are placed in between
convolution layers to perform a filtering operation and reducing the number of
outputs.

This paves the way towards the multi-resolution CNN where the convolution is
performed by a pair of kernels $\mathbf{k_{low}}$ and $\mathbf{k_{high}}$
which generate $\mathbf{X_{low}}$ and $\mathbf{X_{high}}$. The multi-
resolution CNN performs the hierarchical decomposition of the
$\mathbf{X_{low,t}}$ into $\mathbf{X_{low,t+1}}$ and $\mathbf{X_{high,t+1}}$
with different kernels at each step t.

For SpectralNET, the wavelet kernel $\mathbf{K_{high,t}}$ is Haar wavelets and
$\mathbf{K_{low,t}}$ is a scaling function [28]. The 2D haar wavelets utilize
the following four kernels ($f_{L,L}f_{L,H}f_{H,L}f_{H,H}$) for wavelet
transform [29].

$\begin{split}f_{L,L}=\begin{bmatrix}1&1\\\
1&1\end{bmatrix}f_{L,H}=\begin{bmatrix}-1&-1\\\ 1&1\end{bmatrix}\\\
f_{H,L}=\begin{bmatrix}-1&1\\\ -1&1\end{bmatrix}f_{H,H}=\begin{bmatrix}1&-1\\\
-1&1\end{bmatrix}\end{split}$ (2)

A HSI patch x with SxS dimensions when passed through a Haar transform the
(i,j)-th spectrum position value can be written as
$Haar(\mathit{i,j})=x(2i-1,2j-1)+x(2i-1,2j)+x(2i,2j-1)+x(2i,2j)$.

The HSI patch taken as an input is decomposed by the wavelet transform into
sub-bands, these sub-bands are then sent through a convolution layer to learn
the spectral and location features. Note that the sub-bands indicated as high
and low pass filers do not necessarily filter the spectral band in with high
pass and low pass filter. The part of the sub-band is again decomposed in the
next layer by the wavelet transform and sent into the convolution layer. This
process is continued in each layer and the CNN continues to learn the spectral
and spatial features from the HSI patch.

###  II-B SpectralNET Model Description

The input HSI cube having dimension MxNxR is first sent into a layer of Factor
Analysis (FA) to reduce the dimension into MxNxB. Reducing the dimension
reduces training time by 60% [17]. The output vector Y having a dimension 1xMN
take up a class from the available land cover categories denoted by C. The
spectral dimensions are preserved in FA, i.e. MxN, just the bands are reduced
from R to B. Using FA in HSI as a pre-processing step is extremely beneficial,
as FA is able to describe the variability among the different correlated and
overlapping spectrum bands, which helps making the model classify similar
examples better. On the other hand, commonly used Principal Component Analysis
(PCA) based reduction does not directly address this objective in HSI. PCA
provides an approximation to the required factors which do not help to
differentiate similar examples that well. After the FA step is complete,
overlapping 3D patches of size SxSxB are extracted from the pre-processed HSI
and sent into the SpectralNET. SxS is the window size for patch extraction,
for the Indian Pines dataset the patch size has been set at 64x64 and for the
University of Pavia and Salinas Scene dataset the window size has been set at
24x24. The truth values for these patches are determined by the center pixel’s
class category. The values were chosen based on experimentation to maximize
the overall accuracy.

###  II-C Implementation

The proposed SpectralNET model architecture is given in figure 1. The model is
initialized with 3x3 convolution kernels and 1x1 padding. To replace pooling
layers in between convolution a stride of 2 has been utilized. A global mean
pooling has been employed at the end of all the convolution layers before
sending into the dense layer, this prevents overfitting in the model. Dense
connections has been utilized along with projection shortcuts for utilizing
the wavelet transformed data more efficiently [30] [31]. Dense connections
with channel wise concatenation of the decomposed data makes sure that all the
features flow till the end of the model. The model explored two dropout layers
as well along with batch normalization to prevent overfitting. Since the
number of samples are very less in HSI the chances of overfitting are high.
All steps to prevent the model from overfitting needs to be taken. Rectified
Linear Unit (ReLU) has been utilized as the activation function. We explored
the Stochastic gradient descent (SGD) over 150 epochs with a learning rate of
0.01 and momentum of 0.9 to optimize the objective function.

##  III Experiments, results, and discussion

(a) IP Image cube

(b) IP Ground Truth

(c) IP Predictions

(d) IP legend

Figure 2: Image cube, spectral ground truth, and spectral prediction for
Indian Pines Dataset along with legend.

###  III-A Dataset and Training

The experiments were conducted on multiple publicly available benchmark
datasets, Indian Pines (IP), University of Pavia (UP), and Salinas Scene (SA)
111http://lesun.weebly.com/hyperspectral-data-set.html. The detailed
descriptions of the three datasets are given in table I.The classification
spectral layout for IP dataset is given in figure 2.

To perform the experiments, Google colab cloud platform with GPU has been
utilized222https://colab.research.google.com/. Based on our experimental
analysis an optimum learning rate of 0.01 with a momentum of 0.9 was chosen
for the SGD optimizer. For preserving the validity of the results for all
datasets, the bands of the extracted patches have all been set to 3. So, the
patch dimension for IP dataset is 64x64x3 and for UP and SA it is 24x24x3
respectively. The model has been trained for 150 epochs and convergence was
achieved at around 60 epochs.

TABLE I: Detailed description of each dataset used during experiment. Name | Spatial Dimension | Spectral Bands | Wavelength Range | Classes
---|---|---|---|---
IP | 145x145 | 224 | 400nm - 2500nm | 16
UP | 610x340 | 103 | 430nm - 860nm | 9
SA | 512x217 | 224 | 360nm - 2500nm | 16

###  III-B Classification Results

The classification results are given in table II. Three benchmark metrics are
utilized to judge the performance of the proposed model. Overall Accuracy (OA)
gives the total number of correctly classified labels out of the total number
of labels. Average Accuracy (AA) gives the mean of class wise classification
accuracies, and Kappa Accuracy is a measure that correlates the ground truth
and classified values. The results are compared with the state-of-the-art
methods like HybridSN [21] and FuSENET [6], besides SVM, 2D CNN, 3D CNN, M3D
CNN [32] 333https://github.com/eecn/Hyperspectral-Classification. The results
are compared for two sets 10% - 90% random train test split and 30% - 70%
random train test split respectively.

It can be observed from the results that the proposed model outperforms all
state-of-the-art models in both the sets. Even though in the 10% train set the
HybridSN model appears to perform better in SA dataset, that might be because
of the fact it takes a lot more spectral bands as input compared to the
proposed model. It can also be seen from the results that 2D CNN standalone
performs better than 3D CNN in SA dataset. It might be due to the increased
spectral redundancy in the SA dataset compared to the rest. The performance of
FuSENET, HybridSN and SpectralNET is consistently high throughout the three
dataset over M3D CNN. SpectralNET is able to outperform all even with a lot
less spectral bands, i.e. 3, utilized than the state-of-the-art models which
utilize 15, 30 bands. This highlights the merit of using wavelets based
spectral features with a CNN. The time for training the SpectralNET is around
30 minutes which is also comparable to the currently established models.

For more detailed class wise classification results are in the appendix. From
the results it can be established that the performance of SpectralNET is
superior to all the methods currently available for HSI classification.

TABLE II: Classification accuracies (%) of proposed SpectralNET in terms of
OA, Kappa, and AA with varying training data 10% and 30%, respectively

Training Samples | Methods |  | IP dataset |  |  | UP dataset |  |  | SA dataset |
---|---|---|---|---|---|---|---|---|---|---
|  | OA | Kappa | AA | OA | Kappa | AA | OA | Kappa | AA
| SVM | 81.67$\pm$0.65 | 78.76$\pm$0.77 | 79.84$\pm$3.37 | 90.58$\pm$0.47 | 87.21$\pm$0.70 | 92.99$\pm$0.36 | 94.46$\pm$0.12 | 93.13$\pm$0.34 | 93.01$\pm$0.60
| 2D-CNN | 80.27$\pm$1.2 | 78.26$\pm$2.1 | 68.32$\pm$4.1 | 96.63$\pm$0.2 | 95.53$\pm$0.2 | 94.84$\pm$1.4 | 96.34$\pm$0.3 | 95.93$\pm$0.9 | 94.36$\pm$0.5
| 3D-CNN | 82.62$\pm$0.1 | 79.25$\pm$0.3 | 76.51$\pm$0.1 | 96.34$\pm$0.2 | 94.90$\pm$1.2 | 97.03$\pm$0.6 | 85.00$\pm$0.1 | 83.20$\pm$0.7 | 89.63$\pm$0.2
10% | M3D-CNN | 81.39$\pm$2.6 | 81.20$\pm$2.0 | 75.22$\pm$0.7 | 95.95$\pm$0.6 | 93.40$\pm$0.4 | 97.52$\pm$1.0 | 94.20$\pm$0.8 | 93.61$\pm$0.3 | 96.66$\pm$0.5
| FuSENet | 97.11$\pm$0.2 | 97.25$\pm$0.2 | 97.32$\pm$0.2 | 97.65$\pm$0.3 | 97.69$\pm$0.3 | 97.68$\pm$0.4 | 99.23$\pm$0.1 | 99.97$\pm$0.2 | 99.16$\pm$0.1
| HybridSN | 98.39$\pm$0.1 | 98.16$\pm$0.1 | 98.01$\pm$0.2 | 99.72$\pm$0.1 | 99.64$\pm$0.1 | 99.20$\pm$0.1 | 99.98$\pm$0.2 | 99.98$\pm$0.2 | 99.98$\pm$0.1
| SpectralNET | 98.76$\pm$0.2 | 98.59$\pm$0.1 | 98.61$\pm$0.1 | 99.71$\pm$0.1 | 99.62$\pm$0.1 | 99.43$\pm$0.2 | 99.96$\pm$0.2 | 99.96$\pm$0.1 | 99.97$\pm$0.1
| SVM | 87.24$\pm$0.38 | 85.27$\pm$0.45 | 85.15$\pm$1.10 | 95.65$\pm$0.13 | 94.63$\pm$0.17 | 94.60$\pm$0.14 | 94.95$\pm$0.10 | 94.48$\pm$0.11 | 97.93$\pm$0.11
| 2D-CNN | 88.90$\pm$1.3 | 87.01$\pm$1.6 | 85.70$\pm$1.0 | 96.50$\pm$0.4 | 96.55$\pm$0.3 | 96.00$\pm$0.1 | 96.75$\pm$0.6 | 96.71$\pm$0.7 | 98.57$\pm$0.2
| 3D-CNN | 90.23$\pm$0.2 | 89.70$\pm$0.3 | 89.87$\pm$0.1 | 97.90$\pm$0.3 | 97.22$\pm$0.1 | 97.30$\pm$0.1 | 95.54$\pm$0.5 | 94.81$\pm$0.3 | 97.09$\pm$0.6
30% | M3D-CNN | 95.67$\pm$0.1 | 94.70$\pm$0.3 | 94.60$\pm$0.6 | 97.60$\pm$0.2 | 96.50$\pm$0.6 | 98.00$\pm$0.1 | 94.99$\pm$0.3 | 95.40$\pm$0.1 | 96.28$\pm$0.2
| FuSENet | 99.01$\pm$0.2 | 98.60$\pm$0.1 | 98.64$\pm$0.1 | 99.42$\pm$0.2 | 99.21$\pm$0.3 | 99.33$\pm$0.2 | 99.68$\pm$0.2 | 99.74$\pm$0.1 | 99.69$\pm$0.1
| HybridSN | 99.75$\pm$0.1 | 99.71$\pm$0.1 | 99.63$\pm$0.2 | 99.98$\pm$0.1 | 99.98$\pm$0.2 | 99.97$\pm$0.2 | 100 | 100 | 100
| SpectralNET | 99.86$\pm$0.2 | 99.84$\pm$0.2 | 99.98$\pm$0.1 | 99.99$\pm$0.1 | 99.98$\pm$0.1 | 99.98$\pm$0.1 | 100 | 100 | 100

##  IV Conclusion

In a nutshell, a wavelet CNN has been proposed in this work for HSI
classification task. The developed SpectralNET takes into consideration both
spectral and spatial features present in a high dimensional HSI cube using
layers of wavelet decomposition of the input and adding that to the CNN.
Experiments conducted with the three benchmark datasets IP, UP and SA along
with a comparison with the state-of-the-art methods establish the superiority
of the proposed model.

This work has been done in the context of the Machine Learning and Intelligent
System (MALIS) course and it represents the final project report.

## References

  * [1] J. M. Amigo, “Chapter 1.1 - hyperspectral and multispectral imaging: setting the scene,” in Hyperspectral Imaging (J. M. Amigo, ed.), vol. 32 of Data Handling in Science and Technology, pp. 3 – 16, Elsevier, 2020.
  * [2] S. Li, W. Song, L. Fang, Y. Chen, P. Ghamisi, and J. A. Benediktsson, “Deep learning for hyperspectral image classification: An overview,” IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 9, pp. 6690–6709, 2019.
  * [3] X. Yang, Y. Ye, X. Li, R. Y. K. Lau, X. Zhang, and X. Huang, “Hyperspectral image classification with deep learning models,” IEEE Transactions on Geoscience and Remote Sensing, vol. 56, no. 9, pp. 5408–5423, 2018.
  * [4] Z. Ge, G. Cao, X. Li, and P. Fu, “Hyperspectral image classification method based on 2d–3d cnn and multibranch feature fusion,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 13, pp. 5776–5788, 2020.
  * [5] M. Han, R. Cong, X. Li, H. Fu, and J. Lei, “Joint spatial-spectral hyperspectral image classification based on convolutional neural network,” Pattern Recognition Letters, vol. 130, pp. 38 – 45, 2020.  Image/Video Understanding and Analysis (IUVA).
  * [6] S. K. Roy, “Fusenet: fused squeeze-and-excitation network for spectral-spatial hyperspectral image classification,” IET Image Processing, vol. 14, pp. 1653–1661(8), June 2020.
  * [7] C. Yan, X. Bai, P. Ren, L. Bai, W. Tang, and J. Zhou, “Band weighting via maximizing interclass distance for hyperspectral image classification,” IEEE Geoscience and Remote Sensing Letters, vol. 13, no. 7, pp. 922–925, 2016.
  * [8] L. Xie, G. Li, M. Xiao, L. Peng, and Q. Chen, “Hyperspectral image classification using discrete space model and support vector machines,” IEEE Geoscience and Remote Sensing Letters, vol. 14, no. 3, pp. 374–378, 2017\.
  * [9] K. Wang, L. Cheng, and B. Yong, “Spectral-similarity-based kernel of svm for hyperspectral image classification,” Remote Sensing, vol. 12, no. 13, p. 2154, 2020.
  * [10] Q. Wang, X. He, and X. Li, “Locality and structure regularized low rank representation for hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 2, pp. 911–923, 2019\.
  * [11] Z. Chunhui, G. Bing, Z. Lejun, and W. Xiaoqing, “Classification of hyperspectral imagery based on spectral gradient, svm and spatial random forest,” Infrared Physics and Technology, vol. 95, pp. 61 – 69, 2018.
  * [12] O. Okwuashi and C. E. Ndehedehe, “Deep support vector machine for hyperspectral image classification,” Pattern Recognition, vol. 103, p. 107298, 2020.
  * [13] Q. Xu, Y. Xiao, D. Wang, and B. Luo, “Csa-mso3dcnn: Multiscale octave 3d cnn with channel and spatial attention for hyperspectral image classification,” Remote Sensing, vol. 12, no. 1, p. 188, 2020.
  * [14] A. Li and Z. Shang, “A new spectral-spatial pseudo-3d dense network for hyperspectral image classification,” in 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1–7, 2019.
  * [15] M. Ahmad, “A fast 3d cnn for hyperspectral image classification,” 2020.
  * [16] Z. Zhong, J. Li, Z. Luo, and M. Chapman, “Spectral–spatial residual network for hyperspectral image classification: A 3-d deep learning framework,” IEEE Transactions on Geoscience and Remote Sensing, vol. 56, no. 2, pp. 847–858, 2018.
  * [17] N. Laban, B. Abdellatif, H. M. Ebeid, H. A. Shedeed, and M. F. Tolba, “Reduced 3-d deep learning framework for hyperspectral image classification,” in The International Conference on Advanced Machine Learning Technologies and Applications (AMLTA2019) (A. E. Hassanien, A. T. Azar, T. Gaber, R. Bhatnagar, and M. F. Tolba, eds.), (Cham), pp. 13–22, Springer International Publishing, 2020.
  * [18] J. Zheng, Y. Feng, C. Bai, and J. Zhang, “Hyperspectral image classification using mixed convolutions and covariance pooling,” IEEE Transactions on Geoscience and Remote Sensing, pp. 1–13, 2020.
  * [19] B. Fang, Y. Bai, and Y. Li, “Combining spectral unmixing and 3d/2d dense networks with early-exiting strategy for hyperspectral image classification,” Remote Sensing, vol. 12, no. 5, p. 779, 2020.
  * [20] F. Feng, S. Wang, C. Wang, and J. Zhang, “Learning deep hierarchical spatial–spectral features for hyperspectral image classification based on residual 3d-2d cnn,” Sensors, vol. 19, no. 23, p. 5276, 2019.
  * [21] S. K. Roy, G. Krishna, S. R. Dubey, and B. B. Chaudhuri, “Hybridsn: Exploring 3-d–2-d cnn feature hierarchy for hyperspectral image classification,” IEEE Geoscience and Remote Sensing Letters, vol. 17, no. 2, pp. 277–281, 2020.
  * [22] L. Mou, X. Lu, X. Li, and X. X. Zhu, “Nonlocal graph convolutional networks for hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, pp. 1–12, 2020.
  * [23] L. Wang, J. Peng, and W. Sun, “Spatial–spectral squeeze-and-excitation residual network for hyperspectral image classification,” Remote Sensing, vol. 11, no. 7, p. 884, 2019.
  * [24] R. Hang, Q. Liu, D. Hong, and P. Ghamisi, “Cascaded recurrent neural networks for hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 8, pp. 5384–5394, 2019.
  * [25] T. N. Prabhakar and P. Geetha, “Two-dimensional empirical wavelet transform based supervised hyperspectral image classification,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 133, pp. 37 – 45, 2017.
  * [26] S. Fujieda, K. Takayama, and T. Hachisuka, “Wavelet convolutional neural networks,” arXiv preprint arXiv:1805.08620, 2018.
  * [27] S. Fujieda, K. Takayama, and T. Hachisuka, “Wavelet convolutional neural networks for texture classification,” arXiv preprint arXiv:1707.07394, 2017\.
  * [28] X. Wang, “Moving window-based double haar wavelet transform for image processing,” IEEE Transactions on image processing, vol. 15, no. 9, pp. 2771–2779, 2006.
  * [29] P. Liu, H. Zhang, K. Zhang, L. Lin, and W. Zuo, “Multi-level wavelet-cnn for image restoration,” in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 773–782, 2018.
  * [30] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
  * [31] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700–4708, 2017.
  * [32] M. He, B. Li, and H. Chen, “Multi-scale 3d deep convolutional neural network for hyperspectral image classification,” in 2017 IEEE International Conference on Image Processing (ICIP), pp. 3904–3908, 2017.

[Classwise Classification Results] Class wise classification results for IP,
SA and UP datasets are summarised in table III, IV, and V respectively.
Confusion matrix are available in figure 3.

TABLE III: Detailed classification results for Indian Pines Dataset in terms
of Precision, Recall, f1-score, Test Loss, Overall Accuracy, Average Accuracy
and Kappa Accuracy.

Class Labels | Precision | Recall | f1-score | Support
---|---|---|---|---
Alfalfa | 1.00 | 1.00 | 1.00 | 32
Corn-notill | 1.00 | 1.00 | 1.00 | 1000
Corn-mintill | 1.00 | 0.99 | 1.00 | 581
Corn | 1.00 | 1.00 | 1.00 | 166
Grass-pasture | 0.99 | 1.00 | 1.00 | 338
Grass-trees | 1.00 | 1.00 | 1.00 | 511
Grass-pasture-mowed | 1.00 | 0.85 | 0.92 | 20
Hay-windrowed | 1.00 | 1.00 | 1.00 | 335
Oats | 0.78 | 1.00 | 0.88 | 14
Soyabean-notill | 1.00 | 1.00 | 1.00 | 680
Soyabean-mintill | 1.00 | 1.00 | 1.00 | 1719
Soyabean-clean | 1.00 | 1.00 | 1.00 | 415
Wheat | 1.00 | 1.00 | 1.00 | 143
Woods | 1.00 | 1.00 | 1.00 | 886
Buildings-Grass-Trees-Drives | 1.00 | 1.00 | 1.00 | 270
Stone-Steel-Towers | 0.98 | 1.00 | 0.99 | 65
accuracy |  |  | 1.00 | 7175
macro avg | 0.98 | 0.99 | 0.99 | 7175
weighted avg | 1.00 | 1.00 | 1.00 | 7175
Test loss |  |  |  | 0.7%
Average accuracy (%) |  |  |  | 99.98%
Kappa accuracy (%) |  |  |  | 99.84%
Overall accuracy (%) |  |  |  | 99.86%

TABLE IV: Detailed classification results for Salinas Scene Dataset in terms
of Precision, Recall, f1-score, Test Loss, Overall Accuracy, Average Accuracy
and Kappa Accuracy.

Class Labels | Precision | Recall | f1-score | Support
---|---|---|---|---
Brocoli-green-weeds-1 | 1.00 | 1.00 | 1.00 | 1406
Brocoli-green-weeds-2 | 1.00 | 1.00 | 1.00 | 2608
Fallow | 1.00 | 1.00 | 1.00 | 1383
Fallow-rough-plow | 1.00 | 1.00 | 1.00 | 976
Fallow-smooth | 1.00 | 1.00 | 1.00 | 1875
Stubble | 1.00 | 1.00 | 1.00 | 2771
Celery | 1.00 | 1.00 | 1.00 | 2505
Grapes-untrained | 1.00 | 1.00 | 1.00 | 7890
Soil-vinyard-develop | 1.00 | 1.00 | 1.00 | 4342
Corn-senesced-green-weeds | 1.00 | 1.00 | 1.00 | 2295
Lettuce-romaine-4wk | 1.00 | 1.00 | 1.00 | 748
Lettuce-romaine-5wk | 1.00 | 1.00 | 1.00 | 1349
Lettuce-romaine-6wk | 1.00 | 1.00 | 1.00 | 641
Lettuce-romaine-7wk | 1.00 | 1.00 | 1.00 | 749
Vinyard-untrained | 1.00 | 1.00 | 1.00 | 5088
Vinyard-vertical-trellis | 1.00 | 1.00 | 1.00 | 1265
accuracy |  |  | 1.00 | 37891
macro avg | 1.00 | 1.00 | 1.00 | 37891
weighted avg | 1.00 | 1.00 | 1.00 | 37891
Test loss |  |  |  | 0.001%
Average accuracy (%) |  |  |  | 100%
Kappa accuracy (%) |  |  |  | 100%
Overall accuracy (%) |  |  |  | 100%

TABLE V: Detailed classification results for University of Pavia Dataset in
terms of Precision, Recall, f1-score, Test Loss, Overall Accuracy, Average
Accuracy and Kappa Accuracy.

Class Labels | Precision | Recall | f1-score | Support
---|---|---|---|---
Asphalt | 1.00 | 1.00 | 1.00 | 4642
Meadows | 1.00 | 1.00 | 1.00 | 13055
Gravel | 1.00 | 1.00 | 1.00 | 1496
Trees | 1.00 | 1.00 | 1.00 | 2145
Painted metal sheet | 1.00 | 1.00 | 1.00 | 942
Bare soil | 1.00 | 1.00 | 1.00 | 3520
Bitumen | 1.00 | 1.00 | 1.00 | 931
Self-Blocking Bricks | 1.00 | 1.00 | 1.00 | 2577
Shadows | 1.00 | 1.00 | 1.00 | 663
accuracy |  |  | 1.00 | 29944
macro avg | 1.00 | 1.00 | 1.00 | 29944
weighted avg | 1.00 | 1.00 | 1.00 | 29944
Test loss |  |  |  | 0.07%
Average accuracy (%) |  |  |  | 99.98%
Kappa accuracy (%) |  |  |  | 99.98%
Overall accuracy (%) |  |  |  | 99.99%

(a) IP Confusion Matrix

(b) UP Confusion Matrix

(c) SA Confusion Matrix

Figure 3: Confusion matrix for IP, UP, and SA using SpectralNET.