[1]Waqas<EMAIL_ADDRESS>
# Evaluating the method reproducibility of deep learning models in the
biodiversity domain

Waqas Ahmed  Heinz Nixdorf Chair for Distributed Information Systems,
Friedrich Schiller University Jena, Ernst-Abbe-Platz 2, Jena,07743, Thuringia,
Germany  Vamsi Krishna Kommineni  Heinz Nixdorf Chair for Distributed
Information Systems, Friedrich Schiller University Jena, Ernst-Abbe-Platz 2,
Jena,07743, Thuringia, Germany  Department of Functional Biogeography, Max
Planck Institute for Biogeochemistry, Hans-Knoell-Str. 10, Jena, 07745,
Thuringia, Germany  German Centre for Integrative Biodiversity Research (iDiv)
Halle-Jena-Leipzig, Puschstraße 4, Leipzig, 04103, Saxony, Germany  Birgitta
König-Ries  Heinz Nixdorf Chair for Distributed Information Systems, Friedrich
Schiller University Jena, Ernst-Abbe-Platz 2, Jena,07743, Thuringia, Germany
Michael Stifel Center Jena, Friedrich Schiller University Jena, Leutragraben
1, Jena, 07743, Thuringia, Germany  German Centre for Integrative Biodiversity
Research (iDiv) Halle-Jena-Leipzig, Puschstraße 4, Leipzig, 04103, Saxony,
Germany  Jitendra Gaikwad  Heinz Nixdorf Chair for Distributed Information
Systems, Friedrich Schiller University Jena, Ernst-Abbe-Platz 2, Jena,07743,
Thuringia, Germany  Luiz Gadelha  Heinz Nixdorf Chair for Distributed
Information Systems, Friedrich Schiller University Jena, Ernst-Abbe-Platz 2,
Jena,07743, Thuringia, Germany  Sheeba Samuel  Heinz Nixdorf Chair for
Distributed Information Systems, Friedrich Schiller University Jena, Ernst-
Abbe-Platz 2, Jena,07743, Thuringia, Germany  Michael Stifel Center Jena,
Friedrich Schiller University Jena, Leutragraben 1, Jena, 07743, Thuringia,
Germany

###### Abstract

Artificial Intelligence (AI) is revolutionizing biodiversity research by
enabling advanced data analysis, species identification, and habitats
monitoring, thereby enhancing conservation efforts. Ensuring reproducibility
in AI-driven biodiversity research is crucial for fostering transparency,
verifying results, and promoting the credibility of ecological findings. This
study investigates the reproducibility of deep learning (DL) methods within
the biodiversity domain. We design a methodology for evaluating the
reproducibility of biodiversity-related publications that employ DL techniques
across three stages. We define ten variables essential for method
reproducibility, divided into four categories: resource requirements,
methodological information, uncontrolled randomness, and statistical
considerations. These categories subsequently serve as the basis for defining
different levels of reproducibility. We manually extract the availability of
these variables from a curated dataset comprising 61 publications identified
using the keywords provided by biodiversity experts. Our study shows that the
dataset is shared in 47% of the publications; however, a significant number of
the publications lack comprehensive information on deep learning methods,
including details regarding randomness.

## Introduction

In recent years, deep learning methods have been increasingly applied to
understand complex ecological systems, particularly in the field of
biodiversity. These methods have the potential to process and analyze large
amounts of biological data rapidly, leading to significant insights. For
instance, August et al., (2020) demonstrated how AI image classifiers can
create new biodiversity datasets from social media imagery, highlighting the
spatial and taxonomic biases that can influence ecological inference.
Similarly, DL models have been utilized to analyze camera trap images for
wildlife monitoring, enabling researchers to identify species and infer
ecological patterns and processes with high accuracy in Tabak et al., (2019).
However, there is a growing concern about the reproducibility, transparency,
and trustworthiness of research findings produced using deep learning methods
in this domain (Feng et al.,, 2019; GPAI,, 2022).

Reproducibility is essential in scientific research as it allows researchers
to validate and advance methods and results, ensuring the reliability of
scientific claims (Samuel and König-Ries,, 2021). Goodman et al., (2016)
define research reproducibility in three aspects: methods reproducibility,
results reproducibility, and inferential reproducibility. In this paper, we
focus on methods reproducibility, which involves the ability to exactly
replicate the experimental and computational processes using the same data and
tools to achieve the same outcomes. This is especially important in
biodiversity conservation, where decisions can directly impact ecological
health. For example, Norouzzadeh et al., (2018) demonstrated the application
of deep learning for automatically identifying and counting wildlife in camera
trap images, which is vital for monitoring species populations and informing
conservation strategies. However, failure to reproduce these methods
accurately could lead to erroneous population estimates, thereby compromising
conservation efforts and resource allocation. Similarly, in the study by
Rovero et al., (2013), the deployment of camera traps for wildlife monitoring
highlighted the need for reproducible methods to ensure consistent data
collection and analysis across different geographical locations. Inconsistent
methods could result in data that are not comparable, leading to flawed
conclusions and ineffective conservation measures. Moreover, the inability to
reproduce results due to methodological inconsistencies can prevent the
detection of errors that might hide underlying biases or issues in the data.
For instance, Christin et al., (2019) discuss how the reproducibility of deep
learning models in ecological research is challenged by the complexity and
heterogeneity of biodiversity data, which includes interactions between
variables, missing values, and non-linearity. Ensuring reproducibility helps
to uncover these complexities and improve the reliability of ecological models
used for decision-making.

A deep learning pipeline is a structured sequence of processes in training and
deploying a DL model (El-Amir and Hamdy,, 2020). The pipeline typically begins
with data collection and preprocessing, involving tasks such as data cleaning,
normalization, and transformation. Following data preprocessing, the next
stage consists of designing and selecting an appropriate deep learning
architecture, considering factors like model complexity and the nature of the
problem. Subsequently, model training takes place, where the chosen
architecture is trained on the preprocessed data using optimization algorithms
and specific hyperparameter configurations. After training, the model is
evaluated and fine-tuned using various performance metrics to ensure its
effectiveness in solving the targeted problem. The best performing model is
run on the test data. To ensure an unbiased evaluation of the model’s
predictive performance. Finally, the trained model is deployed for real-world
applications or further refinement.

To ensure the reproducibility of the deep learning pipeline, comprehensive
documentation is crucial at each stage. This includes detailed records of the
data collection steps (including providing persistent identifier for each data
point), data preprocessing steps, such as the specific data transformation
techniques applied and any data augmentation strategies employed.
Additionally, it is vital to document the specifics of the chosen deep
learning architecture, including the exact configurations and versions of the
neural network layers utilized. Detailed notes on the hyperparameter values
selected during the model training phase are essential, as well as the
training convergence criteria and the optimization algorithms employed. Proper
documentation of the evaluation metrics used and the testing dataset ensures
the reproducibility of the model’s performance assessment. Finally,
maintaining a record of the software libraries, hardware, frameworks, and
versions utilized throughout the pipeline aids in replicating the experimental
setup accurately.

In this paper, we aim to shed light on the current state of reproducibility of
deep learning methods applied to biodiversity research. We conducted a
systematic literature review to identify publications that use deep learning
techniques in biodiversity research using keywords provided by biodiversity
experts (Abdelmageed et al.,, 2022). We define various reproducibility-related
variables inspired by the current literature. We then curated a dataset of 61
publications and manually extracted reproducibility-related variables, such as
the availability of datasets and code. We also analyzed advanced
reproducibility variables, such as the specific deep learning methods
employed, models, and hyperparameters. Our findings suggest that the
reproducibility of deep learning methods in biodiversity research is generally
low. However, there is a growing trend towards improving reproducibility, with
more and more publications making their datasets and code available. Our study
will contribute to the ongoing discourse on the reproducibility of deep
learning methods in biodiversity research and help to improve the credibility
and impact of these methods in this vital field.

In the following sections, we provide a detailed description of our study. We
start with an overview of the state-of-the-art (“Related Work”). We provide
the methodology of our study (“Methododology”) We describe our results and
discuss the implications of our work (“Discussion”). Finally, we summarize the
key aspects of our study and provide future directions of our research
(“Conclusion”).

## Related Work

Method reproducibility, as mentioned in Goodman et al., (2016), is an
important aspect of progress in any scientific field. Raff, (2019) provides
some insight into reproducibility in ML. The author studied the
reproducibility of 255 ML papers published from 1984 to 2017 and tried to
correlate the reproducibility of the papers with their characteristics.
According to his observation, the main obstacle to reproducibility of results
is insufficient explanation of the method and implementation details when the
source code is not provided. However, some of the defined attributes were very
subjective, such as the algorithmic difficulty of the work or the readability
of the work. While Raff, (2019) provides insight into ML reproducibility in
general, there is also research specifically related to the field of
biodiversity (Feng et al.,, 2019; Schnitzer and Carson,, 2016). They pointed
out that reproducibility is problematic because erroneous data is widespread
and there is a lack of empirical studies to verify previous research findings.
Several authors have pointed out the need for better data management and
reporting practices to ensure the reproducibility of research results (Stark,,
2018; Waide et al.,, 2017; Samuel et al.,, 2021). In Waide et al., (2017), the
authors discuss the challenges associated with managing different data sets in
ecological research, while Leonelli, (2018) further argues that
reproducibility should be a criterion for assessing research quality. However,
setting standards for assessing reproducibility itself is an important issue
for research. Gundersen et al., (2022) in his latest study defined 22
variables that could be categorized according to different degrees of
reproducibility. Three categories were mentioned: Reproducible Data,
Reproducible Experiment and Reproducible Method. Gundersen et al., (2022) used
these variables to evaluate reproducibility support on 13 different open
source machine learning platforms. Similarly, Heil et al., (2021) proposed 3
different standards for reproducibility in machine learning that have
applications in the life sciences. Bronze, silver or gold standards are
defined according to the availability of data, models and code. Tatman et al.,
(2018) went a step further and analyzed ML papers from ICML and NeurIPS
conferences and distinguished levels of reproducibility based on the resources
provided with the paper. To achieve the desired result, they recommended some
practical steps such as providing code and data in an executable environment
where all libraries and dependencies are linked. This allows code or
experiments to run smoothly on a new machine. This recommendation is in line
with the gold standard provided in (Heil et al.,, 2021) that authors should
produce reproducible results with the execution of a single command. In
addition to the general recommendations, Pineau et al., (2021) proposed
standard practices and activities to improve reproducibility in the AI
community. Some of these are the reproducibility programme at the NeurIPS
conference and the ML reproducibility checklist, which provides guidelines to
authors before submitting to conferences. Inspired by Pineau et al., (2021),
some of the major AI conferences (AAAI and IJCAI) have introduced similar
reproducibility checklists. We have considered these guidelines and also
previous work in Gundersen et al., (2022) to develop 10 reproducibility
variables. However, the work in Gundersen et al., (2022) is designed to assess
the reproducibility of different ML platforms, whereas we aim to assess the
reproducibility of biodiversity research. Our variables also include the
aspect of uncontrolled randomness and the statistical information necessary
for reproducibility.

## Methodology

In this section, we will discuss the steps of our work analyzing the
reproducibility of research papers in the biodiversity domain.

### Identification:

To assess the reproducibility of methods in biodiversity research, we first
tried to obtain unbiased and relevant publications. To do this, we adopted
keywords from Abdelmageed et al., (2022), which were provided by biodiversity
experts and were also used to develop a corpus in the field of biodiversity
for large language models (Abdelmageed et al.,, 2022). These keywords are:
”biodivers*” OR ”genetic diversity” OR ”*omic diversity” OR ”phylogenetic
diversity” OR ”population diversity” OR ”species diversity” OR ”ecosystem
diversity” OR ”functional diversity” OR ”microbial diversity” OR ”soil
diversity” AND ”Deep Learning”.

With this search query, Google Scholar suggested more than 8000 articles in a
period from 2015 to 2021. However, we only selected the first 100 results for
our analysis, as manual processing of such a large number of articles was not
possible in a limited time frame and we believe that the characteristics of
these papers are statistically not different from those of the complete set.
We acknowledge that the first 100 articles selected for our analysis might not
represent an entirely unbiased sample of the search results. However, our
selection covers a broad range of publishers and publication years, thus
capturing a diverse cross-section of the available literature. Additionally,
we manually reviewed each selected publication to extract relevant information
on ten defined variables, dedicating approximately 40 minutes per paper. This
thorough manual curation aimed to enhance the reliability of our dataset. This
manually curated dataset will serve as a benchmark for developing an automated
system to extract variable information, enabling future analyses to encompass
a larger number of articles while maintaining methodological rigor and
reproducibility.

### Screening:

Before analyzing the individual articles for reproducibility, we found that
some articles could not be considered further for the following reasons: 1)
duplication, 2) only an abstract was provided or the full article was not
accessible, and 3) some articles were not empirical studies and therefore did
not include experiments. After considering all these limitations, the number
of articles was reduced to 61. Figure 1 shows the distribution of articles
over the period from 2015 to 2021, with most articles being from 2020 and
2021. Figure 2 shows the publisher information for the considered articles,
which are distributed quite randomly across more than ten publishers.

Figure 1: Number of publications considered for collecting the binary
responses as per the reproducibility criteria based on year Figure 2:
Publisher information for the 61 publications selected for collecting the
binary responses as per the reproducibility criteria

### Selection of reproducibility variables:

There is no standard for assessing the reproducibility of DL biodiversity
methods. A common practice for analyzing the reproducibility of research
articles is to rerun the experiments using the same methodology as the
original author. However, this requires a lot of time and computing resources.
Also, sometimes it is not possible to work with the same resources such as
hardware and software. Inspired by (Gundersen et al.,, 2022), we have
formulated a set of 10 indicators that can be used as proxies for the
probability that a result is reproducible. Instead of repeating each
experiment, we look for these variables or factors that are considered
important determinants of the reproducibility. Using the literature and the
reproducibility checklists at conferences such as NeurIPS and AAAI (Pineau et
al.,, 2021), we can divide these variables into four categories. Resource
Information (ReI) details the availability of datasets, source code, open-
source tools, and proprietary model details crucial for reproduction efforts.
Methodological Information (MI) captures the specifics of software and
hardware used, as well as a high-level overview of the deep learning methods
employed. Randomness Information (RaI) addresses aspects of unpredictability
in computational processes, ensuring they are documented for consistency.
Statistical Information (SI) focuses on the rigor of result analysis,
advocating for multiple metrics and averaging techniques for reliable
evaluation. The comprehensive details that define these aspects of
reproducibility are systematically itemized in Table LABEL:Tab:Def

### Reproducibility check:

For all selected papers, we first manually checked the variables for each
paper. To reduce the degree of subjectivity, we (two of the authors) began by
independently recording the binary responses (availability or non-
availability). In the initial phase of our assessment, we encountered notable
discrepancies due to ambiguous interpretations of the definitions of each
variable, leading to inconsistent binary responses. This was quantitatively
evidenced by an average Cohen’s Kappa value (Cohen,, 1960) of 0.54, indicating
a moderate level of agreement and highlighting the initial inconsistencies
between annotators. To address these issues and improve the reliability of our
analysis, we undertook a review and clarification of the variable definitions.
In the end, we obtained the same binary responses for each paper, as shown in
Table LABEL:Tab:Bin-res. These responses formed the basis for analysing
reproducibility. We verified whether the paper covered the functionality of
each previously defined variable. After completing the binary responses for
each paper, we quantified the reproducibility in 5 levels. The idea behind the
different levels is to give independent researchers an insight into the
chances of obtaining accurate results independently. The higher the level of
reproducibility, the more variables are covered and the greater the chances of
reproducing the result. These levels are set considering the time and
computational effort required for reproducibility when certain variables are
not present. However, the criteria for the basic level (Level 1) must be met
to be reproducible according to the definition we refer to Goodman et al.,
(2016). Figure 3 shows the different levels of reproducibility together with
the categories covered by each level. Level 1 should at least cover all
variables defined in ReI. Level 2 should cover all variables defined in ReI
and MI. Level 3 comprises the variables ReI, MI and SI, while Level 4
comprises the variables ReI , MI and RaI and these two levels do not build on
each other. Finally, the highest level of reproducibility combines all
categories at Level 5.

Level 1ReILevel 2ReI+MILevel 3ReI+MI+SILevel 4ReI+MI+RaILevel 5ReI+MI+SI+RaI Figure 3: Number of levels along with the categories covered for respective levels Table 1: Definition of the various reproducibility variables used in this research paper Category |  Variable |  Variable short name |  Variable Description
---|---|---|---
Resource Information (ReI) |  Dataset |  V1 |  Availability of dataset with persistent identifiers.
|  Source Code |  V2 |  Availability of source code to recreate the experiment (e.g. GitHub, GitLab, Zenodo).
|  Open source frameworks or environment |  V3 |  Availability of open source software and hardware tools required to reproduce the work (e.g., docker container, virtual environments).
|  Model architecture |  V4 |  Availability of a deep learning model architecture or the accessibility of its internal working.
Methodological Information (MI) |  Software and Hardware Specification |  V5 |  Availability of information related to the type of hardware used (e.g., GPU), its specifications, and the version of software and libraries used.
|  Methods |  V6 |  Availability of high-level information about the deep learning pipeline including the pre-processing and post-processing steps. The other parameters of the pipeline are assessed as separate variables.
|  Hyper-parameters |  V7 |  Availability of hyperparameters used to train the model (e.g., number of epochs, learning rate, optimizer, etc.).
Randomness Information (RaI) |  Randomness |  R |  Availability of information about weight initialization, data shuffling, data augmentation, data train-test split and cuDNN GPU library.
Statistical Information (SI) |  Averaging result |  S1 |  Availability of multiple model training and averaging them out rather than selecting only the highest value.
|  Evaluation metrics |  S2 |  Availability of more than one evaluated metric (e.g., $R^{2}$ score along with Root Mean Square Error (RMSE) or Mean Absolute Error (MAE) or Loss of the model).

## Results

As discussed in Section Methodology, we divided the ten reproducibility
variables we had defined into four categories 1) Resource Information 2)
Methodological Information 3) Randomness Information, and 4) Statistical
Information. These four categories were further categorized into five
different levels of reproducibility, allowing us to assess the reproducibility
level of the publications more comprehensively.

(a) Binary responses of variables for the category ’Resources’

l

(b) Binary responses of variables for the category ’Methodological
information’

(c) Binary responses of variables for the category ’Randomness’

(d) Binary responses of variables for the category ’Statistical consideration’

Figure 4: Binary responses of the considered reproducible variables in four
categories 1) Resources 2) Methodological information 3) Randomness 4)
Statistical consideration for selected research publication

### Variable level information

Figure 4 contains four plots, one for each category of information. Within
each category, specific reproducibility variables are depicted with binary
counts, illustrated by two overlapping bars: one denoted by ’Yes’ in a brick
orange color and the other by ’No’ in blue, along with the corresponding
number of publications. In total, we made 610 individual judgements for the
ten variables, 353 of these were positive i.e the respective information was
provided, while this was not the case in 257 cases resulting in negative
judgements. Detailed descriptions of all responses can be found at both the
category and individual variable levels.

In the Resources Information category, 29 publications provided a dataset, 16
indicated the code repositories, 58 used open-source frameworks or
environments, and 57 mentioned the model architectures (Figure 4(a)).

For the Methodological Information category, around a quarter of the
publications included details about the hardware and the software (libraries)
they used, all the publications explained methods that were used to build a
machine learning pipeline and 40 publications provided the basic
hyperparameters (Figure 4(b)).

The number of publications that used a random seed in all possible ways
(weight initialization, data shuffling, data augmentation, data train-test
split and cuDNN GPU library) in their code is 3 (Figure 4(c)). Regarding
statistical considerations, 21 publications provided the average result of
multiple model trainings, 54 publications evaluated their models with more
than one evaluation metric. (Figure 4(d)).

Figure 5: Distribution of selected research publications along four categories

### Categorical level information

We have harmonized the binary responses of reproducibility variables for each
categorical level in such a way that all the individual reproducibility
variables in a category must be available (Yes) to mark that the specific
categorical information is available (Yes). If one of the individual
reproducibility variables is unavailable (No), the specific categorical
information is unavailable (No). Mathematically, $V1\&V2\&V3\&V4==Yes$ is used
to denote the availability of resource information (Yes).

Figure 5 depicts the distribution of publications that satisfy the defined
categories. The number of publications that satisfy the Resources,
Methodological, Randomness, and Statistical information are 10, 13, 3, and 20,
respectively.

Figure 6: Bar plot indicating the number of publications satisfying the
different levels of reproducibility

### Reproducibility levels of publications

Per the definitions of the reproducibility levels described in Section
Methodology, Levels 1 and 5 are the lowest and highest, respectively.
According to Figure 6, only one publication fulfils the highest
reproducibility level, while ten publications meet the lowest.

(a) Number of papers meeting criteria for the category ’Resources’ according
to year

l

(b) Number of papers meeting criteria for the category ’Methodological
information’ according to year

(c) Number of papers meeting criteria for the category ’Randomness’ according
to year

(d) Number of papers meeting criteria for the category ’Statistical
consideration’ according to year

Figure 7: Number of papers meeting criteria for the four categories 1)
Resources 2) Methodological information 3) Randomness 4) Statistical
consideration for selected research publications by year

### Reproducibility status of publications by year

The number of publications that meet the defined reproducibility criteria
follows a linear trend with respect to year (Figure 7).

## Discussion

In the Biodiversity field, deep-learning methods are becoming part of many
studies that run large-scale experiments. These gave us the opportunity to
orchestrate and extract the binary responses of 10 reproducibility variables
from 61 publications (Table LABEL:Tab:Def and Table LABEL:Tab:Bin-res). As a
result, we recorded 610 total responses: 353 were positive, and 257 were
negative. All the positive responses were dominated by four variables (230
responses): 1) Open source frameworks or environment, 2) Model architecture,
3) Methods, and 4) Evaluation metrics and the negative responses were
dominated by only one variable: Randomness.

Most of the publications that employed deep learning models use open-source
frameworks or environments like Tensorflow and PyTorch with the programming
language Python/R, and they also provide model architectures either as a
figure/table in the publication or described in the text with respective
citations. Some of the publications used licensed programming languages like
Matlab, comparatively, it is negligible. We looked for high-level information
on the whole deep-learning pipeline in methods, all the publications provided
compact information.

Most of the publications use more than one metric to evaluate their models,
for example, when it comes to regression tasks, they use $R^{2}$ score along
with Root Mean Square Error (RMSE) or Mean Absolute Error (MAE) or Loss of the
model, etc. The information from the publications is in compliance with
reproducible workflow guidelines for the positively dominated variables.

We found datasets only in 29 publications; in the other 32 publications, there
was no tangential information about the dataset. There were some cases where
the authors linked to some data-providing websites to find the data that was
used in the publication, but those websites will change over time and finding
the exact data that was used in the publication is not possible without
providing persistent identifiers of the respective data points.

Source code is one of the fundamental variables of reproducibility. However, a
little more than 25 % of the publications provided their source code.

In 14 publications, authors have provided specifications about both software
and hardware. Without specific information about the hardware and software,
the reproducibility results will change because the random generators work
differently with different hardware and software changes with each version.

Hyperparameters are the values that are chosen to control the model learning
process. This means with each set of hyperparameters, the model will provide a
varied range of results. However, information about basic hyperparameters
(epochs, learning rate, optimizer and loss function) was missing from 21
publications. Reproducing the results of these 21 publications is unfeasible
due to the missing essential hyperparameter information.

Averaging results is also an important aspect of reproducibility, after each
training process, results will change slightly because of the certain random
initializations through the deep learning pipeline. However, in our study, 40
publications didn’t report the multiple training results.

We opted for manually updating the variables by going through each publication
and extracting the required information as a binary response because we did
not find a system or technique that could automatically extract the required
variable information from a publication. Since we are extracting the
information about variables from publications manually, it is only possible to
work with a small dataset, which is also the limitation of this study.

Due to recent developments with Large Language Models (LLMs), we are
considering extracting the reproducible variables information from
publications using LLMs from the year 2022 onwards (Ahmed et al.,, 2023;
Kommineni et al.,, 2024). This will allow us to implement our analysis on
large-scale publications.

## Conclusion

In this paper, we presented our pipeline for assessing the reproducibility of
deep learning methods in biodiversity research. Inspired by the current state
of the art, we established a comprehensive set of ten variables, categorized
into four distinct groups, to effectively quantify the reproducibility of DL
empirical research. Based on the defined categories, we documented the
availability of each variable across 61 selected publications over the period
from 2015 to 2021. From the total 610 responses for the 10 variables, 57.9%
show the availability of the variables in the publication, while the remaining
42.1% are primarily characterized by the absence of randomness-related
information. The highest and lowest reproducibility levels are satisfied by
only one and ten publications, respectively. Given the use of deep learning to
advance biodiversity research, improving reproducibility of the DL methods is
crucial. Considering the limitations of the manual approach and the relatively
small dataset analyzed until 2021, our future endeavors will focus on
implementing a semi-automatic approach that leverages Large Language models
for extracting information on reproducible variables from publications.

## Data availability statement

The data and the code used to extract and analyse the reproducibility
information of Deep Learning methods from publications in the Biodiversity
domain is publicly available: https://github.com/fusion-jena/Reproduce-
DLmethods-Biodiv

Table 2: Binary responses of different reproducibility variables. ’y’ denotes the presence of variable information, while ’n’ signifies the absence of variable information. Publication | V1 | V2 | V3 | V4 | V5 | V6 | V7 | R | S1 | S2
---|---|---|---|---|---|---|---|---|---|---
Klein et al., (2015) | n | n | y | n | n | y | n | n | n | n
Khalighifar et al., (2021) | n | n | y | y | n | y | n | n | n | y
Choe et al., (2021) | n | n | y | y | y | y | y | n | y | y
Mahmood et al., (2016) | y | n | y | y | n | y | n | n | n | y
Younis et al., (2020) | y | y | y | y | y | y | y | n | y | y
Schwartz and Alfaro, (2021) | y | y | y | y | y | y | y | n | y | y
Potamitis, (2016) | n | n | y | y | n | y | n | n | n | n
Chen et al., (2020) | n | n | y | y | n | y | n | n | n | y
Fujisawa et al., (2021) | n | n | y | y | n | y | n | n | y | y
Weinstein, (2018) | n | y | y | y | y | y | y | n | n | y
Chalmers et al., (2021) | n | n | y | y | y | y | n | n | n | y
Guirado et al., (2019) | y | n | y | y | y | y | y | n | y | y
Zualkernan et al., (2020) | n | n | y | y | n | y | y | n | n | y
Villon et al., (2018) | n | n | y | y | n | y | y | n | y | y
Fairbrass et al., (2019) | y | y | y | y | y | y | y | n | n | y
Weinstein et al., (2019) | y | y | y | y | y | y | y | n | n | y
Hu et al., (2020) | n | n | y | y | n | y | n | n | n | y
Alshahrani et al., (2021) | y | n | y | y | n | y | y | n | y | y
Villon et al., (2020) | n | n | y | y | n | y | n | n | y | y
Botella et al., (2018) | y | n | y | y | n | y | y | n | y | y
Salamon et al., (2017) | y | n | y | y | n | y | y | n | y | y
Mac Aodha et al., (2018) | y | y | y | y | y | y | y | n | y | y
Schindler and Steinhage, (2021) | n | n | y | y | n | y | y | n | n | y
Rakshit et al., (2018) | n | n | y | y | n | y | y | n | n | y
Bjerge et al., (2021) | n | y | y | y | n | y | y | y | y | y
Guirado et al., (2017) | n | n | y | y | n | y | n | n | n | y
Hussein et al., 2021a  | n | n | y | y | n | y | y | n | n | y
Rammer and Seidl, (2019) | y | y | y | y | y | y | y | n | n | y
Anand et al., (2021) | n | n | n | y | n | y | n | n | n | y
Zizka et al., (2021) | y | y | y | y | n | y | y | y | n | n
Miele et al., (2020) | n | n | y | y | n | y | n | n | y | n
Demertzis et al., (2018) | n | n | y | y | n | y | n | n | y | y
Malerba et al., (2021) | y | n | y | y | n | y | y | n | n | n
Huang and Basanta, (2021) | n | n | y | y | n | y | y | n | y | y
Campos-Taberner et al., (2020) | y | n | n | y | n | y | n | n | n | y
Rousset et al., (2021) | y | n | y | y | n | y | y | n | n | y
López-Jiménez et al., (2019) | y | n | y | y | n | y | y | n | n | y
Schuettpelz et al., (2017) | n | y | y | y | n | y | y | n | n | y
Schiller et al., (2021) | y | y | y | y | y | y | y | y | y | y
Heredia, (2017) | n | y | y | y | n | y | y | n | y | y
Martins et al., (2021) | y | n | y | y | y | y | y | n | n | y
Browning et al., (2018) | y | n | y | y | n | y | n | n | y | y
Guirado et al., (2020) | y | n | y | y | n | y | y | n | n | y
Ayhan et al., (2020) | n | n | y | y | n | y | y | n | n | y
Loddo et al., (2021) | n | n | y | y | n | y | n | n | y | y
Jamil et al., (2021) | y | n | y | y | n | y | y | n | n | y
Neves et al., (2021) | n | n | y | y | n | y | n | n | n | y
Jin et al., (2021) | y | n | y | n | n | y | n | n | n | y
Mohanty et al., (2016) | y | y | y | y | n | y | y | n | n | y
Xie et al., (2019) | y | n | y | y | n | y | y | n | y | y
Tian et al., (2020) | n | n | n | n | n | y | n | n | y | y
Gimenez et al., (2021) | n | y | y | y | n | y | y | n | n | y
Ortega Adarme et al., (2020) | y | n | y | y | n | y | y | n | n | y
Boer and Vos, (2018) | y | y | y | y | y | y | y | n | n | n
Dunker et al., (2021) | n | n | y | y | n | y | n | n | n | y
Villon et al., (2016) | n | n | y | y | n | y | n | n | n | y
Dyrmann et al., (2021) | n | n | y | y | n | y | y | n | n | y
Arruda et al., (2021) | y | n | y | n | n | y | y | n | n | n
Kislov and Korznikov, (2020) | y | n | y | y | n | y | y | n | n | y
Hussein et al., 2021b  | y | n | y | y | n | y | y | n | n | y
Becker et al., (2021) | n | y | y | y | y | y | y | n | n | y

## References

  * Abdelmageed et al., (2022) Abdelmageed, N., Löffler, F., Feddoul, L., Algergawy, A., Samuel, S., Gaikwad, J., Kazem, A., and König-Ries, B. (2022).  Biodivnere: Gold standard corpora for named entity recognition and relation extraction in the biodiversity domain.  Biodiversity Data Journal, 10.
  * Ahmed et al., (2023) Ahmed, W., Kommineni, V. K., Koenig-ries, B., and Samuel, S. (2023).  How reproducible are the results gained with the help of deep learning methods in biodiversity research?  Biodiversity Information Science and Standards, 7.
  * Alshahrani et al., (2021) Alshahrani, H. M., Al-Wesabi, F. N., Al Duhayyim, M., Nemri, N., Kadry, S., and Alqaralleh, B. A. Y. (2021).  An automated deep learning based satellite imagery analysis for ecology management.  Ecol. Inform., 66(101452):101452.
  * Anand et al., (2021) Anand, A., Pandey, M. K., Srivastava, P. K., Gupta, A., and Khan, M. L. (2021).  Integrating multi-sensors data for species distribution mapping using deep learning and envelope models.  Remote Sens. (Basel), 13(16):3284.
  * Arruda et al., (2021) Arruda, V. L. S., Piontekowski, V. J., Alencar, A., Pereira, R. S., and Matricardi, E. A. T. (2021).  An alternative approach for mapping burn scars using landsat imagery, google earth engine, and deep learning in the brazilian savanna.  Remote Sens. Appl. Soc. Environ., 22(100472):100472.
  * August et al., (2020) August, T. A., Pescott, O. L., Joly, A., and Bonnet, P. (2020).  AI naturalists might hold the key to unlocking biodiversity data in social media imagery.  Patterns, 1(7):100116.
  * Ayhan et al., (2020) Ayhan, B., Kwan, C., Budavari, B., Kwan, L., Lu, Y., Perez, D., Li, J., Skarlatos, D., and Vlachos, M. (2020).  Vegetation detection using deep learning and conventional methods.  Remote Sens. (Basel), 12(15):2502.
  * Becker et al., (2021) Becker, A., Russo, S., Puliti, S., Lang, N., Schindler, K., and Wegner, J. D. (2021).  Country-wide retrieval of forest structure from optical and SAR satellite imagery with deep ensembles.
  * Bjerge et al., (2021) Bjerge, K., Nielsen, J. B., Sepstrup, M. V., Helsing-Nielsen, F., and Høye, T. T. (2021).  An automated light trap to monitor moths (lepidoptera) using computer vision-based tracking and deep learning.  Sensors (Basel), 21(2):343.
  * Boer and Vos, (2018) Boer, M. J. and Vos, R. A. (2018).  Taxonomic classification of ants (formicidae) from images using deep learning.  bioRxiv, page 407452.
  * Botella et al., (2018) Botella, C., Joly, A., Bonnet, P., Monestiez, P., and Munoz, F. (2018).  A deep learning approach to species distribution modelling.  In Multimedia Tools and Applications for Environmental & Biodiversity Informatics, pages 169–199. Springer International Publishing, Cham.
  * Browning et al., (2018) Browning, E., Bolton, M., Owen, E., Shoji, A., Guilford, T., and Freeman, R. (2018).  Predicting animal behaviour using deep learning: GPS data alone accurately predict diving in seabirds.  Methods Ecol. Evol., 9(3):681–692.
  * Campos-Taberner et al., (2020) Campos-Taberner, M., García-Haro, F. J., Martínez, B., Izquierdo-Verdiguier, E., Atzberger, C., Camps-Valls, G., and Gilabert, M. A. (2020).  Understanding deep learning in land use classification based on sentinel-2 time series.  Sci. Rep., 10(1):17188.
  * Chalmers et al., (2021) Chalmers, C., Fergus, P., Wich, S., and Longmore, S. N. (2021).  Modelling animal biodiversity using acoustic monitoring and deep learning.  In 2021 International Joint Conference on Neural Networks (IJCNN). IEEE.
  * Chen et al., (2020) Chen, X., Zhao, J., Chen, Y.-H., Zhou, W., and Hughes, A. C. (2020).  Automatic standardized processing and identification of tropical bat calls using deep learning approaches.  Biol. Conserv., 241(108269):108269.
  * Choe et al., (2021) Choe, H., Chi, J., and Thorne, J. H. (2021).  Mapping potential plant species richness over large areas with deep learning, MODIS, and species distribution models.  Remote Sens. (Basel), 13(13):2490.
  * Christin et al., (2019) Christin, S., Hervet, É., and Lecomte, N. (2019).  Applications for deep learning in ecology.  Methods in Ecology and Evolution, 10(10):1632–1644.
  * Cohen, (1960) Cohen, J. (1960).  A coefficient of agreement for nominal scales.  Educational and psychological measurement, 20(1):37–46.
  * Demertzis et al., (2018) Demertzis, K., Iliadis, L. S., and Anezakis, V.-D. (2018).  Extreme deep learning in biosecurity: the case of machine hearing for marine species identification.  J. Inf. Telecommun., 2(4):492–510.
  * Dunker et al., (2021) Dunker, S., Motivans, E., Rakosy, D., Boho, D., Mäder, P., Hornick, T., and Knight, T. M. (2021).  Pollen analysis using multispectral imaging flow cytometry and deep learning.  New Phytol., 229(1):593–606.
  * Dyrmann et al., (2021) Dyrmann, M., Mortensen, A. K., Linneberg, L., Høye, T. T., and Bjerge, K. (2021).  Camera assisted roadside monitoring for invasive alien plant species using deep learning.  Sensors (Basel), 21(18):6126.
  * El-Amir and Hamdy, (2020) El-Amir, H. and Hamdy, M. (2020).  Deep learning pipeline.  Apress: Berkeley, CA, USA.
  * Fairbrass et al., (2019) Fairbrass, A. J., Firman, M., Williams, C., Brostow, G. J., Titheridge, H., and Jones, K. E. (2019).  CityNet—Deep learning tools for urban ecoacoustic assessment.  Methods Ecol. Evol., 10(2):186–197.
  * Feng et al., (2019) Feng, X., Park, D. S., Walker, C., Peterson, A. T., Merow, C., and Papeş, M. (2019).  A checklist for maximizing reproducibility of ecological niche models.  Nature Ecology & Evolution, 3(10):1382–1395.
  * Fujisawa et al., (2021) Fujisawa, T., Noguerales, V., Meramveliotakis, E., Papadopoulou, A., and Vogler, A. P. (2021).  Image-based taxonomic classification of bulk biodiversity samples using deep learning and domain adaptation.
  * Gimenez et al., (2021) Gimenez, O., Kervellec, M., Fanjul, J.-B., Chaine, A., Marescot, L., Bollet, Y., and Duchamp, C. (2021).  Trade-off between deep learning for species identification and inference about predator-prey co-occurrence: Reproducible R workflow integrating models in computer vision and ecological statistics.
  * Goodman et al., (2016) Goodman, S. N., Fanelli, D., and Ioannidis, J. P. A. (2016).  What does research reproducibility mean?  Science Translational Medicine, 8(341):341ps12–341ps12.
  * GPAI, (2022) GPAI (2022).  Biodiversity and artificial intelligence, opportunities and recommendations report.
  * Guirado et al., (2020) Guirado, E., Alcaraz-Segura, D., Cabello, J., Puertas-Ruíz, S., Herrera, F., and Tabik, S. (2020).  Tree cover estimation in global drylands from space using deep learning.  Remote Sens. (Basel), 12(3):343.
  * Guirado et al., (2017) Guirado, E., Tabik, S., Alcaraz-Segura, D., Cabello, J., and Herrera, F. (2017).  Deep-Learning convolutional neural networks for scattered shrub detection with google earth imagery.
  * Guirado et al., (2019) Guirado, E., Tabik, S., Rivas, M. L., Alcaraz-Segura, D., and Herrera, F. (2019).  Whale counting in satellite and aerial images with deep learning.  Sci. Rep., 9(1):14259.
  * Gundersen et al., (2022) Gundersen, O. E., Shamsaliei, S., and Isdahl, R. J. (2022).  Do machine learning platforms provide out-of-the-box reproducibility?  Future Generation Computer Systems, 126:34–47.
  * Heil et al., (2021) Heil, B. J., Hoffman, M. M., Markowetz, F., Lee, S.-I., Greene, C. S., and Hicks, S. C. (2021).  Reproducibility standards for machine learning in the life sciences.  Nature Methods, 18(10):1132–1135.
  * Heredia, (2017) Heredia, I. (2017).  Large-scale plant classification with deep neural networks.  In Proceedings of the Computing Frontiers Conference, New York, NY, USA. ACM.
  * Hu et al., (2020) Hu, J., Huang, W., Su, Y., Liu, Y., and Xiao, P. (2020).  BatNet++: A robust deep learning-based predicting models for calls recognition.  In 2020 5th International Conference on Smart Grid and Electrical Automation (ICSGEA). IEEE.
  * Huang and Basanta, (2021) Huang, Y.-P. and Basanta, H. (2021).  Recognition of endemic bird species using deep learning models.  IEEE Access, 9:102975–102984.
  * (37) Hussein, B. R., Malik, O. A., Ong, W.-H., and Slik, J. W. F. (2021a).  Automated extraction of phenotypic leaf traits of individual intact herbarium leaves from herbarium specimen images using deep learning based semantic segmentation.  Sensors (Basel), 21(13):4549.
  * (38) Hussein, B. R., Malik, O. A., Ong, W.-H., and Slik, J. W. F. (2021b).  Reconstruction of damaged herbarium leaves using deep learning techniques for improving classification accuracy.  Ecol. Inform., 61(101243):101243.
  * Jamil et al., (2021) Jamil, S., Rahman, M., and Haider, A. (2021).  Bag of features (BoF) based deep learning framework for bleached corals detection.  Big Data Cogn. Comput., 5(4):53.
  * Jin et al., (2021) Jin, L., Yu, J., Yuan, X., and Du, X. (2021).  Fish classification using DNA barcode sequences through deep learning method.  Symmetry (Basel), 13(9):1599.
  * Khalighifar et al., (2021) Khalighifar, A., Brown, R. M., Goyes Vallejos, J., and Peterson, A. T. (2021).  Deep learning improves acoustic biodiversity monitoring and new candidate forest frog species identification (genus platymantis) in the Philippines.  Biodivers. Conserv., 30(3):643–657.
  * Kislov and Korznikov, (2020) Kislov, D. E. and Korznikov, K. A. (2020).  Automatic windthrow detection using very-high-resolution satellite imagery and deep learning.  Remote Sens. (Basel), 12(7):1145.
  * Klein et al., (2015) Klein, D. J., McKown, M. W., and Tershy, B. R. (2015).  Deep learning for large scale biodiversity monitoring.  In Bloomberg Data for Good Exchange Conference.
  * Kommineni et al., (2024) Kommineni, V. K., König-Ries, B., and Samuel, S. (2024).  From human experts to machines: An llm supported approach to ontology and knowledge graph construction.  arXiv preprint arXiv:2403.08345.
  * Leonelli, (2018) Leonelli, S. (2018).  Rethinking reproducibility as a criterion for research quality.  In Including a symposium on Mary Morgan: curiosity, imagination, and surprise, volume 36, pages 129–146. Emerald Publishing Limited.
  * Loddo et al., (2021) Loddo, A., Loddo, M., and Di Ruberto, C. (2021).  A novel deep learning based approach for seed image classification and retrieval.  Comput. Electron. Agric., 187(106269):106269.
  * López-Jiménez et al., (2019) López-Jiménez, E., Vasquez-Gomez, J. I., Sanchez-Acevedo, M. A., Herrera-Lozada, J. C., and Uriarte-Arcia, A. V. (2019).  Columnar cactus recognition in aerial images using a deep learning approach.  Ecol. Inform., 52:131–138.
  * Mac Aodha et al., (2018) Mac Aodha, O., Gibb, R., Barlow, K. E., Browning, E., Firman, M., Freeman, R., Harder, B., Kinsey, L., Mead, G. R., Newson, S. E., Pandourski, I., Parsons, S., Russ, J., Szodoray-Paradi, A., Szodoray-Paradi, F., Tilova, E., Girolami, M., Brostow, G., and Jones, K. E. (2018).  Bat detective—deep learning tools for bat acoustic signal detection.  PLoS Comput. Biol., 14(3):e1005995.
  * Mahmood et al., (2016) Mahmood, A., Bennamoun, M., An, S., Sohel, F., Boussaid, F., Hovey, R., Kendrick, G., and Fisher, R. B. (2016).  Automatic annotation of coral reefs using deep learning.  In OCEANS 2016 MTS/IEEE Monterey. IEEE.
  * Malerba et al., (2021) Malerba, M. E., Wright, N., and Macreadie, P. I. (2021).  A continental-scale assessment of density, size, distribution and historical trends of farm dams using deep learning convolutional neural networks.  Remote Sens. (Basel), 13(2):319.
  * Martins et al., (2021) Martins, J. A. C., Nogueira, K., Osco, L. P., Gomes, F. D. G., Furuya, D. E. G., Gonçalves, W. N., Sant’Ana, D. A., Ramos, A. P. M., Liesenberg, V., dos Santos, J. A., de Oliveira, P. T. S., and Junior, J. M. (2021).  Semantic segmentation of tree-canopy in urban environment with pixel-wise deep learning.  Remote Sens. (Basel), 13(16):3054.
  * Miele et al., (2020) Miele, V., Dussert, G., Cucchi, T., and Renaud, S. (2020).  Deep learning for species identification of modern and fossil rodent molars.
  * Mohanty et al., (2016) Mohanty, S. P., Hughes, D. P., and Salathé, M. (2016).  Using deep learning for image-based plant disease detection.  Front. Plant Sci., 7:1419.
  * Neves et al., (2021) Neves, A. K., Körting, T. S., Fonseca, L. M. G., Soares, A. R., Girolamo-Neto, C. D., and Heipke, C. (2021).  Hierarchical mapping of brazilian savanna (cerrado) physiognomies based on deep learning.  J. Appl. Remote Sens., 15(04).
  * Norouzzadeh et al., (2018) Norouzzadeh, M. S., Nguyen, A., Kosmala, M., Swanson, A., Palmer, M. S., Packer, C., and Clune, J. (2018).  Automatically identifying, counting, and describing wild animals in camera-trap images with deep learning.  Proceedings of the National Academy of Sciences, 115(25):E5716–E5725.
  * Ortega Adarme et al., (2020) Ortega Adarme, M., Queiroz Feitosa, R., Nigri Happ, P., Aparecido De Almeida, C., and Rodrigues Gomes, A. (2020).  Evaluation of deep learning techniques for deforestation detection in the brazilian amazon and cerrado biomes from remote sensing imagery.  Remote Sens. (Basel), 12(6):910.
  * Pineau et al., (2021) Pineau, J., Vincent-Lamarre, P., Sinha, K., Larivière, V., Beygelzimer, A., d’Alché Buc, F., Fox, E., and Larochelle, H. (2021).  Improving reproducibility in machine learning research (a report from the neurips 2019 reproducibility program).  The Journal of Machine Learning Research, 22(1):7459–7478.
  * Potamitis, (2016) Potamitis, I. (2016).  Deep learning for detection of bird vocalisations.  arXiv preprint arXiv:1609.08408.
  * Raff, (2019) Raff, E. (2019).  A step toward quantifying independently reproducible machine learning research.  Advances in Neural Information Processing Systems, 32.
  * Rakshit et al., (2018) Rakshit, S., Debnath, S., and Mondal, D. (2018).  Identifying land patterns from satellite imagery in amazon rainforest using deep learning.
  * Rammer and Seidl, (2019) Rammer, W. and Seidl, R. (2019).  Harnessing deep learning in ecology: An example predicting bark beetle outbreaks.  Front. Plant Sci., 10:1327.
  * Rousset et al., (2021) Rousset, G., Despinoy, M., Schindler, K., and Mangeas, M. (2021).  Assessment of deep learning techniques for land use land cover classification in southern new caledonia.  Remote Sens. (Basel), 13(12):2257.
  * Rovero et al., (2013) Rovero, F., Zimmermann, F., Berzi, D., and Meek, P. (2013).  ” which camera trap type and how many do i need?” a review of camera features and study designs for a range of wildlife research applications.  Hystrix.
  * Salamon et al., (2017) Salamon, J., Bello, J. P., Farnsworth, A., and Kelling, S. (2017).  Fusing shallow and deep learning for bioacoustic bird species classification.  In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE.
  * Samuel and König-Ries, (2021) Samuel, S. and König-Ries, B. (2021).  Understanding experiments and research practices for reproducibility: an exploratory study.  PeerJ, 9:e11140.
  * Samuel et al., (2021) Samuel, S., Löffler, F., and König-Ries, B. (2021).  Machine learning pipelines: Provenance, reproducibility and FAIR data principles.  In Glavic, B., Braganholo, V., and Koop, D., editors, Provenance and Annotation of Data and Processes - 8th and 9th International Provenance and Annotation Workshop, IPAW 2020 + IPAW 2021, Virtual Event, July 19-22, 2021, Proceedings, volume 12839 of Lecture Notes in Computer Science, pages 226–230. Springer.
  * Schiller et al., (2021) Schiller, C., Schmidtlein, S., Boonman, C., Moreno-Martínez, A., and Kattenborn, T. (2021).  Deep learning and citizen science enable automated plant trait predictions from photographs.  Sci. Rep., 11(1):16395.
  * Schindler and Steinhage, (2021) Schindler, F. and Steinhage, V. (2021).  Identification of animals and recognition of their actions in wildlife videos using deep learning techniques.  Ecol. Inform., 61(101215):101215.
  * Schnitzer and Carson, (2016) Schnitzer, S. A. and Carson, W. P. (2016).  Would ecology fail the repeatability test?  BioScience, 66(2):98–99.
  * Schuettpelz et al., (2017) Schuettpelz, E., Frandsen, P., Dikow, R., Brown, A., Orli, S., Peters, M., Metallo, A., Funk, V., and Dorr, L. (2017).  Applications of deep convolutional neural networks to digitized natural history collections.  Biodivers. Data J., 5:e21139.
  * Schwartz and Alfaro, (2021) Schwartz, S. T. and Alfaro, M. E. (2021).  Sashimi: A toolkit for facilitating high‐throughput organismal image segmentation using deep learning.  Methods Ecol. Evol., 12(12):2341–2354.
  * Stark, (2018) Stark, P. B. (2018).  Before reproducibility must come preproducibility.  Nature, 557(7706):613–614.
  * Tabak et al., (2019) Tabak, M. A., Norouzzadeh, M. S., Wolfson, D. W., Sweeney, S. J., VerCauteren, K. C., Snow, N. P., Halseth, J. M., Di Salvo, P. A., Lewis, J. S., White, M. D., et al. (2019).  Machine learning to classify animal species in camera trap images: Applications in ecology.  Methods in Ecology and Evolution, 10(4):585–590.
  * Tatman et al., (2018) Tatman, R., VanderPlas, J., and Dane, S. (2018).  A practical taxonomy of reproducibility for machine learning research.
  * Tian et al., (2020) Tian, J., Wang, L., Yin, D., Li, X., Diao, C., Gong, H., Shi, C., Menenti, M., Ge, Y., Nie, S., Ou, Y., Song, X., and Liu, X. (2020).  Development of spectral-phenological features for deep learning to understand spartina alterniflora invasion.  Remote Sens. Environ., 242(111745):111745.
  * Villon et al., (2016) Villon, S., Chaumont, M., Subsol, G., Villéger, S., Claverie, T., and Mouillot, D. (2016).  Coral reef fish detection and recognition in underwater videos by supervised machine learning: Comparison between deep learning and HOG+SVM methods.  In Advanced Concepts for Intelligent Vision Systems, Lecture notes in computer science, pages 160–171. Springer International Publishing, Cham.
  * Villon et al., (2018) Villon, S., Mouillot, D., Chaumont, M., Darling, E. S., Subsol, G., Claverie, T., and Villéger, S. (2018).  A deep learning method for accurate and fast identification of coral reef fishes in underwater images.  Ecol. Inform., 48:238–244.
  * Villon et al., (2020) Villon, S., Mouillot, D., Chaumont, M., Subsol, G., Claverie, T., and Villéger, S. (2020).  A new method to control error rates in automated species identification with deep learning algorithms.  Sci. Rep., 10(1):10972.
  * Waide et al., (2017) Waide, R. B., Brunt, J. W., and Servilla, M. S. (2017).  Demystifying the landscape of ecological data repositories in the United States.  BioScience, 67(12):1044–1051.
  * Weinstein, (2018) Weinstein, B. G. (2018).  Scene‐specific convolutional neural networks for video‐based biodiversity detection.  Methods Ecol. Evol., 9(6):1435–1441.
  * Weinstein et al., (2019) Weinstein, B. G., Marconi, S., Bohlman, S., Zare, A., and White, E. (2019).  Individual tree-crown detection in RGB imagery using semi-supervised deep learning neural networks.  Remote Sens. (Basel), 11(11):1309.
  * Xie et al., (2019) Xie, J., Hu, K., Zhu, M., Yu, J., and Zhu, Q. (2019).  Investigation of different CNN-based models for improved bird sound classification.  IEEE Access, 7:175353–175361.
  * Younis et al., (2020) Younis, S., Schmidt, M., Weiland, C., Dressler, S., Seeger, B., and Hickler, T. (2020).  Detection and annotation of plant organs from digitised herbarium scans using deep learning.  Biodivers. Data J., 8:e57090.
  * Zizka et al., (2021) Zizka, A., Silvestro, D., Vitt, P., and Knight, T. M. (2021).  Automated conservation assessment of the orchid family with deep learning.  Conserv. Biol., 35(3):897–908.
  * Zualkernan et al., (2020) Zualkernan, I. A., Dhou, S., Judas, J., Sajun, A. R., Gomez, B. R., Hussain, L. A., and Sakhnini, D. (2020).  Towards an IoT-based deep learning architecture for camera trap image classification.  In 2020 IEEE Global Conference on Artificial Intelligence and Internet of Things (GCAIoT). IEEE.


$\delta_{0}>0$, such that ${\mathbf{M}}_{\mathbb{R}}(d_{1}\times
d_{2};l_{1},\ldots,l_{r})\cap\mathfrak{B}_{\delta_{0}}(x_{0})$ is an
$(d_{1}d_{2}-\frac{1}{2}\sum_{j=1}^{r}(l_{j}-1)(l_{j}+2))$-dimensional
manifold. Thus, by an argument that is similar to (3.22), we have

$\displaystyle\mathbb{P}\left(\exists t\in I,\ \mathrm{s.t.\
}\left|s_{J_{j}}^{\beta}(t)\right|=1\ \mathrm{for\ some\ disjoint\
}J_{j}\in[d_{1}]\mathrm{\ with\ }|J_{j}|=l_{j},\forall 1\leq j\leq r\right)$
$\displaystyle=\mathbb{P}\left(Z^{\beta}(t)\in{\mathbf{M}}_{\mathbb{R}}(d_{1}\times
d_{2};l_{1},\ldots,l_{r})\mathrm{\ for\ some\ }t\in I\right)$
$\displaystyle\geq\mathbb{P}\left(W^{\beta}(I)\cap
T_{\beta}^{-1}\left({\mathbf{M}}_{\mathbb{R}}(d_{1}\times
d_{2};l_{1},\ldots,l_{r})\cap\mathfrak{B}_{\delta_{0}}(x_{0})-B^{\beta}\right)\tilde{T}_{\beta}^{-1}\not=\emptyset\right)$
$\displaystyle\geq
c_{5}\mathcal{C}_{d_{1}d_{2}-\sum_{j=1}^{N}\frac{1}{H_{j}}}\left(T_{\beta}^{-1}\left({\mathbf{M}}_{\mathbb{R}}(d_{1}\times
d_{2};l_{1},\ldots,l_{r})\cap\mathfrak{B}_{\delta_{0}}(x_{0})-B^{\beta}\right)\tilde{T}_{\beta}^{-1}\right)>0.$

The proof of (iii) is to apply Lemma 2.3 as the proof of Theorem 1.1, and we
omit the details. This finishes the proof. ∎

###  4.2 The complex case: proof of Theorem 1.4

In this section, we consider the matrix (1.14) for the case $\beta=2$.

The first lemma is the complex version of Lemma 4.1, which shows that
$2d_{1}d_{2}-\sum_{j=1}^{r}(l_{j}-1)(l_{j}+1)$ is an upper bound for the
dimension of ${\mathbf{M}}_{\mathbb{C}}(d_{1}\times
d_{2};l_{1},\ldots,l_{r})$.

######  Lemma 4.4.

Let $\Delta:\mathbb{R}_{\geq 0}^{l}\rightarrow{\mathbf{D}}(d)$ be the function
that maps the vector $u=(u_{1},\ldots,u_{l})\in\mathbb{R}_{\geq 0}^{l}$ to the
diagonal matrix $\Delta(u)=\\{\Delta_{i,j}(u):1\leq i,j\leq d\\}$ given by
(3.1). Then there exist compactly supported smooth functions
$\widehat{\Gamma}_{L}:\mathbb{R}^{d_{1}^{2}+d_{2}^{2}-d_{2}-\sum_{j=0}^{r}l_{j}(l_{j}-1)}\rightarrow\mathbf{U}(d_{1})$
and
$\widehat{\Gamma}_{R}:\mathbb{R}^{d_{1}^{2}+d_{2}^{2}-d_{2}-\sum_{j=0}^{r}l_{j}(l_{j}-1)}\rightarrow\mathbf{U}(d_{2})$,
such that the mapping

$\displaystyle\widehat{G}:\mathbb{R}_{\geq
0}^{l}\times\mathbb{R}^{d_{1}^{2}+d_{2}^{2}-d_{2}-\sum_{j=0}^{r}l_{j}(l_{j}-1)}\rightarrow\mathbb{C}^{d_{1}\times
d_{2}}$

given by

$\displaystyle\widehat{G}(u,v)=\widehat{\Gamma}_{L}(v)\Delta(u)\widehat{\Gamma}_{R}(v)^{*},~{}~{}(u,v)\in\mathbb{R}_{\geq
0}^{l}\times\mathbb{R}^{d_{1}^{2}+d_{2}^{2}-d_{2}-\sum_{j=0}^{r}l_{j}(l_{j}-1)}$

satisfies

$\displaystyle{\mathbf{M}}_{\mathbb{C}}(d_{1}\times
d_{2};l_{1},\ldots,l_{r})\subseteq\left\\{x\in\mathbb{R}^{d^{2}}:\
x\in\mathrm{Im}(\widehat{G})\right\\},$

where $\mathrm{Im}(\widehat{G})$ is the image of the mapping $\widehat{G}$.

###### Proof.

The proof is similar to the proof of Lemma 4.1, and is sketched below.

Since $\mathbf{U}(d_{1})$ and $\mathbf{U}(d_{2})$ are Lie groups of dimension
$d_{1}^{2}$ and $d_{2}^{2}$ respectively, their direct product
$\mathbf{U}(d_{1})\times\mathbf{U}(d_{2})$ is again a Lie group with dimension
$d_{1}^{2}+d_{2}^{2}$. Recall that
$\mathbf{U}(d_{2};l_{1},\ldots,l_{r},l_{0})$ is a Lie group of dimension
$d_{2}+\sum_{j=0}^{r}l_{j}(l_{j}-1)$. For any
$g\in\mathbf{U}(d_{2};l_{1},\ldots,l_{r},l_{0})$, one can write
$g=\mathrm{diag}(g_{d_{1}},g_{l_{0}})$, where
$g_{d_{1}}\in\mathbf{U}(d_{1};l_{1},\ldots,l_{r})$ and
$g_{l_{0}}\in\mathbf{U}(l_{0})$. In case that $l_{0}=0$, we use the convention
$\mathbf{U}(d_{2};l_{1},\ldots,l_{r},l_{0})=\mathbf{U}(d_{1};l_{1},\ldots,l_{r})$
and $g=g_{d_{1}}$. Then one can verify that the mapping
$\left(\mathbf{U}(d_{1})\times\mathbf{U}(d_{2})\right)\times\mathbf{U}(d_{2};l_{1},\ldots,l_{r},l_{0})\to\mathbf{U}(d_{1})\times\mathbf{U}(d_{2})$
given by

$\displaystyle((p_{1},p_{2}),g)\mapsto(p_{1},p_{2})\cdot
g=(p_{1}g_{d_{1}},p_{2}g)$

is the right action of the Lie group
$\mathbf{U}(d_{2};l_{1},\ldots,l_{r},l_{0})$ on the manifold
$\mathbf{U}(d_{1})\times\mathbf{U}(d_{2})$. Moreover, the right action is
smooth, free and proper. Therefore, by Lemma 2.9, the orbit space
$\left(\mathbf{U}(d_{1})\times\mathbf{U}(d_{2})\right)/\mathbf{U}(d_{2};l_{1},\ldots,l_{r},l_{0})$
is a smooth manifold of dimension
$d_{1}^{2}+d_{2}^{2}-d_{2}-\sum_{j=0}^{r}l_{j}(l_{j}-1)$, and the quotient map
$\pi:\mathbf{U}(d_{1})\times\mathbf{U}(d_{2})\longrightarrow\left(\mathbf{U}(d_{1})\times\mathbf{U}(d_{2})\right)/\mathbf{U}(d_{2};l_{1},\ldots,l_{r},l_{0})$
is a smooth submersion. Furthermore, by Lemma 2.6, $\pi$ is an open map.

For any $p\in\mathbf{U}(d_{1})\times\mathbf{U}(d_{2})$, since $\pi$ is a
submersion, by Lemma 2.7, there exists a smooth local section $\sigma_{\pi}$
from an open nieghbourhood of $\pi(p)$ to
$\mathbf{U}(d_{1})\times\mathbf{U}(d_{2})$ that maps $\pi(p)$ to $p$ and that
$\pi\circ\sigma_{\pi}$ is identity. Moreover, by the definition of chart (see
e.g. [Tu2011]*Definition 5.1), there exist $\delta>0$ and a smoothly
diffeomorphism

$\displaystyle\phi:I_{\epsilon}\to\phi(I_{\epsilon})\subset\left(\mathbf{U}(d_{1})\times\mathbf{U}(d_{2})\right)/\mathbf{U}(d_{2};l_{1},\ldots,l_{r},l_{0})\cap\pi(\mathfrak{B}_{\delta}(p))$

satisfying $\phi(0)=\pi(p)$, where

$\displaystyle
I_{\epsilon}=(-\epsilon,\epsilon)^{d_{1}^{2}+d_{2}^{2}-d_{2}-\sum_{j=0}^{r}l_{j}(l_{j}-1)}.$

Then we can construct smooth function

$\displaystyle\Gamma=\left(\Gamma_{L},\Gamma_{R}\right):\mathbb{R}^{d_{2}^{2}+d_{1}^{2}-d_{2}-\sum_{j=0}^{r}l_{j}(l_{j}-1)}\rightarrow{\mathbf{U}}(d_{1})\times{\mathbf{U}}(d_{2})$

that is supported on $I_{2\epsilon}$ and satisfies

$\displaystyle\left(\Gamma_{L}(v),\Gamma_{R}(v)\right)=\Gamma(v)=\sigma_{\pi}\circ\phi(v),\forall
v\in I_{\epsilon}.$

The rest of the proof is similar to that of Lemma 4.1 and hence is omitted. ∎

The following result is the complex version of Lemma 4.2. The proof is similar
to that of Lemma 4.2 and hence is omitted.

######  Lemma 4.5.

Let $A\in{\mathbf{M}}_{\mathbb{C}}(d_{1}\times d_{2};l_{1},\ldots,l_{r})$ with
singular value decomposition $A=P_{L}DP_{R}^{*}$ for some
$P_{L}\in{\mathbf{U}}(d_{1})$, $P_{R}\in{\mathbf{U}}(d_{2})$ and
$D\in{\mathbf{D}}(d_{1}\times d_{2})$, such that
$0\notin\mathbf{Spec}(AA^{*})$ and $|\mathbf{Spec}(AA^{*})|=l$. Then for any
$\epsilon>0$, there exists $\delta>0$, such that for all
$B\in{\mathbf{M}}_{\mathbb{C}}(d_{1}\times d_{2};l_{1},\ldots,l_{r})$
satisfying

$\displaystyle\max_{1\leq i,j\leq d}|A_{i,j}-B_{i,j}|<\delta,$

we have $0\notin\mathbf{Spec}(BB^{*})$ and $|\mathbf{Spec}(BB^{*})|=l$.
Moreover, there exists a singular value decomposition $B=Q_{L}FQ_{R}^{*}$,
where $Q_{L}\in{\mathbf{U}}(d_{1})$, $Q_{R}\in{\mathbf{U}}(d_{2})$ and
$F\in{\mathbf{D}}(d_{1}\times d_{2})$ satisfy

$\displaystyle\max_{1\leq i\leq
d_{1}}|D_{i,i}-F_{i,i}|<\epsilon,\quad\max_{1\leq i,j\leq
d_{1}}\left|\left(Q_{L}\right)_{i,j}-\left(P_{L}\right)_{i,j}\right|<\epsilon,\quad\max_{1\leq
i,j\leq
d_{2}}\left|\left(Q_{R}\right)_{i,j}-\left(P_{R}\right)_{i,j}\right|<\epsilon.$

The following lemma is the complex version of Lemma 4.3, which indicates that
the upper bound $2d_{1}d_{2}-\sum_{j=1}^{r}(l_{j}-1)(l_{j}+1)$ for the
dimension of ${\mathbf{M}}_{\mathbb{C}}(d_{1}\times d_{2};l_{1},\ldots,l_{r})$
is optimal. The proof is similar to the proof of Lemma 4.3, and hence is
omitted.

######  Lemma 4.6.

For $x_{0}\in{\mathbf{M}}_{\mathbb{C}}(d_{1}\times d_{2};l_{1},\ldots,l_{r})$
with $0\notin\mathbf{Spec}(x_{0}x_{0}^{*})$ and
$|\mathbf{Spec}(x_{0}x_{0}^{*})|=l$, there exists $\delta_{0}>0$ such that
${\mathbf{M}}_{\mathbb{C}}(d_{1}\times
d_{2};l_{1},\ldots,l_{r})\cap\mathfrak{B}_{\delta_{0}}(x_{0})$ is a
$(2d_{1}d_{2}-\sum_{j=1}^{r}(l_{j}-1)(l_{j}+1))$-dimensional manifold. In
particular, ${\mathbf{M}}_{\mathbb{C}}(d_{1}\times
d_{2};l_{1},\ldots,l_{r})\cap\mathfrak{B}_{\delta_{0}}(x_{0})$ has positive
$(2d_{1}d_{2}-\sum_{j=1}^{r}(l_{j}-1)(l_{j}+1))$-dimensional Lebesgue measure.

Now we are ready to prove Theorem 1.4.

###### Proof of Theorem 1.4.

The proof is similar to that of Theorem 1.1, which is sketched below.

We first prove Part (i). By Lemma 4.4, there exists a smooth map

$\displaystyle\widehat{G}:\mathbb{R}_{\geq
0}^{l}\times\mathbb{R}^{d_{1}^{2}+d_{2}^{2}-d_{2}-\sum_{j=0}^{r}l_{j}(l_{j}-1)}\rightarrow\mathbb{C}^{d_{1}\times
d_{2}},$

such that ${\mathbf{M}}_{\mathbb{C}}(d_{1}\times
d_{2};l_{1},\ldots,l_{r})\subseteq\mathrm{Im}(\widehat{G})$. Together with
Lemma 4.6, we derive that

$\displaystyle\dim_{{}_{\rm
H}}\left(\mathrm{Im}(\widehat{G})\right)=2d_{1}d_{2}-\sum_{j=1}^{r}(l_{j}-1)(l_{j}+1).$
(4.11)

If

$\displaystyle\sum_{j=1}^{N}\frac{1}{H_{j}}<\sum_{j=1}^{r}(l_{j}-1)(l_{j}+1),$

then by Lemma 2.3, we obtain

$\displaystyle\mathbb{P}\left(\exists t\in I,\ \mathrm{s.t.\
}\left|s_{J_{j}}^{\beta}(t)\right|=1\ \mathrm{for\ some\ disjoint\
}J_{j}\in[d_{1}]\mathrm{\ with\ }|J_{j}|=l_{j},\forall 1\leq j\leq r\right)$
$\displaystyle=\mathbb{P}\left(Z^{\beta}(t)\in{\mathbf{M}}_{\mathbb{C}}(d_{1}\times
d_{2};l_{1},\ldots,l_{r})\mathrm{\ for\ some\ }t\in I\right)$
$\displaystyle\leq\mathbb{P}\left(W^{\beta}(I)\cap
T_{\beta}^{-1}\left(\mathrm{Im}(\widehat{G})-B^{\beta}\right)\tilde{T}_{\beta}^{-1}\not=\emptyset\right)$
$\displaystyle\leq
c_{6}{\mathbf{H}}_{2d_{1}d_{2}-\sum_{j=1}^{N}\frac{1}{H_{j}}}\left(T_{\beta}^{-1}\left(\mathrm{Im}(\widehat{G})-B^{\beta}\right)\tilde{T}_{\beta}^{-1}\right)=0,$

where we use the translation invariance of Hausdorff measure, (4.11),
Corollary 2.1 and Lemma 2.2. If

$\displaystyle\sum_{j=1}^{N}\frac{1}{H_{j}}=\sum_{j=1}^{r}(l_{j}-1)(l_{j}+1),$

then by Lemma 2.5, we can compute the Lebesgue measure

$\displaystyle\lambda_{2d_{1}d_{2}}\left(\left(\mathrm{Im}\left(\widehat{G}\big{|}_{F_{K}}\right)-B^{\beta}\right)^{(r)}\right)\leq
Cr^{\sum_{j=1}^{N}\frac{1}{H_{j}}}$

for all $r>0$ small, and $C$ is a constant the depends only on
$G,d,H_{1},\ldots,H_{N}$. Here, $F_{K}=[0,K]^{l}\times F^{\prime}$ is a
compact interval with $F^{\prime}$ being a compact interval of
$\mathbb{R}^{d_{1}^{2}+d_{2}^{2}-d_{2}-\sum_{j=0}^{r}l_{j}(l_{j}-1)}$
containing the support of both $\widehat{\Gamma}_{L}$ and
$\widehat{\Gamma}_{R}$. By a similar argument as (3.22), for $K$ large, we
obtain

$\displaystyle\mathbb{P}\bigg{(}\left\\{\exists t\in I,\ \mathrm{s.t.\
}\left|s_{J_{j}}^{\beta}(t)\right|=1\ \mathrm{for\ some\ disjoint\
}J_{j}\in[d_{1}]\mathrm{\ with\ }|J_{j}|=l_{j},\forall 1\leq j\leq r\right\\}$
$\displaystyle\quad\quad\quad\quad\quad\quad\quad\quad\cap\left\\{\mathbf{Spec}\left(Z^{\beta}\left(Z^{\beta}\right)^{*}\right)\subset\left[0,K^{2}\right]\right\\}\bigg{)}$
$\displaystyle\leq\mathbb{P}\left(\left\\{Z^{\beta}(t)\in
T_{\beta}^{-1}\left(\mathrm{Im}\left(\widehat{G}\right)-B^{\beta}\right)\tilde{T}_{\beta}^{-1}\mathrm{\
for\ some\ }t\in
I\right\\}\cap\left\\{\mathbf{Spec}\left(Z^{\beta}\left(Z^{\beta}\right)^{*}\right)\subset\left[0,K^{2}\right]\right\\}\right)$
$\displaystyle\leq\mathbb{P}\left(W^{\beta}(I)\cap
T_{\beta}^{-1}\left(\mathrm{Im}\left(\widehat{G}\big{|}_{F_{K}}\right)-B^{\beta}\right)\tilde{T}_{\beta}^{-1}\not=\emptyset\right).$

We deduce from Lemma 2.4 and Corollary 2.2 that

$\displaystyle\mathbb{P}\bigg{(}\left\\{\exists t\in I,\ \mathrm{s.t.\
}\left|s_{J_{j}}^{\beta}(t)\right|=1\ \mathrm{for\ some\ disjoint\
}J_{j}\in[d_{1}]\mathrm{\ with\ }|J_{j}|=l_{j},\forall 1\leq j\leq r\right\\}$
$\displaystyle\quad\quad\quad\quad\quad\quad\quad\quad\cap\left\\{\mathbf{Spec}\left(Z^{\beta}\left(Z^{\beta}\right)^{*}\right)\subset\left[0,K^{2}\right]\right\\}\bigg{)}=0.$

Letting $K\to\infty$, we obtain

$\displaystyle\mathbb{P}\left(\exists t\in I,\ \mathrm{s.t.\
}\left|s_{J_{j}}^{\beta}(t)\right|=1\ \mathrm{for\ some\ disjoint\
}J_{j}\in[d_{1}]\mathrm{\ with\ }|J_{j}|=l_{j},\forall 1\leq j\leq
r\right)=0.$

Next, we prove (ii). We choose $x_{0}\in{\mathbf{M}}_{\mathbb{C}}(d_{1}\times
d_{2};l_{1},\ldots,l_{r})$ satisfying $0\notin\mathbf{Spec}(x_{0}x_{0}^{*})$
and $|\mathbf{Spec}(x_{0}x_{0}^{*})|=l$. By Lemma 4.6, there exists
$\delta_{0}>0$, such that ${\mathbf{M}}_{\mathbb{R}}(d_{1}\times
d_{2};l_{1},\ldots,l_{r})\cap\mathfrak{B}_{\delta_{0}}(x_{0})$ is an
$(2d_{1}d_{2}-\sum_{j=1}^{r}(l_{j}-1)(l_{j}+1))$-dimensional manifold. Thus,
by Lemma 2.3 together with Corollary 2.1 and Lemma 2.2, when

$\displaystyle\sum_{j=1}^{N}\frac{1}{H_{j}}>\sum_{j=1}^{r}(l_{j}-1)(l_{j}+1),$

we have

$\displaystyle\mathbb{P}\left(\exists t\in I,\ \mathrm{s.t.\
}\left|s_{J_{j}}^{\beta}(t)\right|=1\ \mathrm{for\ some\ disjoint\
}J_{j}\in[d_{1}]\mathrm{\ with\ }|J_{j}|=l_{j},1\leq j\leq r\right)$
$\displaystyle=\mathbb{P}\left(Z^{\beta}(t)\in{\mathbf{M}}_{\mathbb{C}}(d_{1}\times
d_{2};l_{1},\ldots,l_{r})\mathrm{\ for\ some\ }t\in I\right)$
$\displaystyle\geq\mathbb{P}\left(W^{\beta}(I)\in
T_{\beta}^{-1}\left({\mathbf{M}}_{\mathbb{C}}(d_{1}\times
d_{2};l_{1},\ldots,l_{r})\cap\mathfrak{B}_{\delta_{0}}(x_{0})-B^{\beta}\right)\tilde{T}_{\beta}^{-1}\not=\emptyset\right)$
$\displaystyle\geq
c_{5}\mathcal{C}_{2d_{1}d_{2}-\sum_{j=1}^{N}\frac{1}{H_{j}}}\left(T_{\beta}^{-1}\left({\mathbf{M}}_{\mathbb{C}}(d_{1}\times
d_{2};l_{1},\ldots,l_{r})\cap\mathfrak{B}_{\delta_{0}}(x_{0})-B^{\beta}\right)\tilde{T}_{\beta}^{-1}\right)>0.$

The proof of (iii) is similar to that in the proof of Theorem 1.1 and is
omitted. This finishes the proof. ∎

## Acknowledgments

The author gratefully acknowledges the financial support of ERC Consolidator
Grant 815703 ”STAMFORD: Statistical Methods for High Dimensional Diffusions”.
Besides, the author would like to acknowledge Jian Song and Jianfeng Yao for
the discussion and helpful suggestions.

## References


$\kappa_{\operatorname{LLT}^{\operatorname{cospin}}}(\bm{\nu},f)=\sum_{\lambda}\sum_{i=1}^{d_{\lambda^{1}/\mu^{1},\lambda^{2}/\mu^{2}}^{\lambda}}c_{\lambda^{1}/\mu^{1},\lambda^{2}/\mu^{2}}^{\lambda;i}[i]_{q}s_{\lambda}.$

Since the LLT cumulant of $1$-colored tuple is simply an LLT polynomial (which
is Schur positive by the result of Grojnowski and Haiman [GH07]) and there are
no other $r$-colorings of a pair of skew partitions, the proof is finished. ∎

######  Remark 4.2.

Note that in this simple case, the coefficient
$[s_{\lambda}]\kappa_{\operatorname{LLT}^{\operatorname{cospin}}}(\bm{\nu},f)$
is explicit assuming that the coefficient
$[s_{\lambda}]\operatorname{LLT}^{\operatorname{cospin}}(\bm{\nu})$ is known.
In our setting, this coefficient was described combinatorially in terms of
inversions of Yamanouchi tableaux by Roberts [Rob14], which, in effect,
provides also the combinatorial interpretation of the coefficient
$[s_{\lambda}]\kappa_{\operatorname{LLT}^{\operatorname{cospin}}}(\bm{\nu},f)$.

An explicit expression for $\operatorname{LLT}(\bm{\nu})$ in the Schur basis
exists also for $\ell(\bm{\nu})=3$ due to Blasiak [Bla16] but it is much more
complicated and, as noticed by Blasiak, there are serious difficulties in
going beyond the case $\ell(\bm{\nu})=3$. Let us recover Blasiak’s result here
[Bla16, Corollary 4.3], so that we can state our conjecture connected to its
cumulant counterpart.

Let $\bm{\nu}=(\lambda^{1}/\mu^{1},\lambda^{2}/\mu^{2},\lambda^{3}/\mu^{3})$.
Blasiak proved that

(35)
$\operatorname{LLT}(\bm{\nu})(q)=\sum_{\lambda}c_{\bm{\nu}}^{\lambda}(q)s_{\lambda},\hskip
42.67912pt\text{where}\hskip
42.67912ptc_{\bm{\nu}}^{\lambda}(q)=\sum_{\begin{subarray}{c}T\in\operatorname{RSST}(\lambda)\\\
\operatorname{Des}_{3}^{\prime}(T)=D^{\prime}(\bm{\nu})\\\
\tilde{c}(\bm{\nu})\text{ - entries of
}T\end{subarray}}q^{\operatorname{inv}_{3}^{\prime}(T)},$

and

  * •

$\operatorname{RSST}(\lambda)$ is the set of _restricted square strict
tableaux_ of shape $\lambda$, i.e., fillings of $\lambda$ whose columns
strictly increase upwards, rows strictly increase rightwards, and the filling
of the cell $(x,y)$ is smaller by at least $3$ than that of
$(x^{\prime},y^{\prime})$ with $x^{\prime}>x$ and $y^{\prime}>y$;

  * •

$\operatorname{Des}^{\prime}_{3}(T)$ is the multiset of pairs
$(T(x,y),T(x^{\prime},y^{\prime}))$ with
$(x,y),(x^{\prime},y^{\prime})\in\operatorname{sh}(T)=\lambda$, such that
$T(x,y)-T(x^{\prime},y^{\prime})=3$, and either $y>y^{\prime}$ and $x\leq
x^{\prime}$, or $x=x^{\prime}+1$, $y=y^{\prime}+1$, and
$T(x^{\prime},y)=T(x,y)-1$;

  * •

$D^{\prime}(\bm{\nu})$ is the multiset of pairs
$(\tilde{c}(x,y),\tilde{c}(x^{\prime},y^{\prime}))$ with
$(x,y),(x^{\prime},y^{\prime})\in\bm{\nu}$, such that
$\tilde{c}(x,y)=\tilde{c}(x^{\prime},y^{\prime})+3$ and $y<y^{\prime}$ and
$x\leq x^{\prime}$;

  * •

$\tilde{c}(\bm{\nu})$ is the sequence of shifted contents of $\bm{\nu}$; and

  * •

$\operatorname{inv}^{\prime}_{3}(T)$ is the number of pairs
$((x,y),(x^{\prime},y^{\prime}))$ with
$(x,y),(x^{\prime},y^{\prime})\in\operatorname{sh}(T)$ with
$0<T(x,y)-T(x^{\prime},y^{\prime})<3$, such that $y>y^{\prime}$ and $x\leq
x^{\prime}$.

Note that the sets $\operatorname{Des}_{3}^{\prime}(T)$ and
$D^{\prime}(\bm{\nu})$ are indeed multisets. For instance, for
$\bm{\nu}=((3,3,3),(1),(1))$, we have

$D^{\prime}(\bm{\nu})=\\{(6,3),(3,0),(3,0),(3,0),(0,-3),(0,-3),(0,-3),(-3,-6)\\}.$

The point $(3,0)$ in $D^{\prime}(\bm{\nu})$ counted with multiplicity $3$
comes from the following pairs $(x,y),(x^{\prime},y^{\prime})\in\bm{\nu}$:
$(2,1),(2,2)\in\bm{\nu}_{1}$, $(2,1),(3,3)\in\bm{\nu}_{1}$, and
$(3,2),(3,3)\in\bm{\nu}_{1}$.

######  Example.

Let $\lambda^{1}/\mu^{1}=\lambda^{3}/\mu^{3}=(1,1)$ and
$\lambda^{2}/\mu^{2}=(2,2)/(2)$ and consider
$[s_{(3,2,1)}]\operatorname{LLT}(\bm{\nu})$ for
$\bm{\nu}=(\lambda^{1}/\mu^{1},\lambda^{2}/\mu^{2},\lambda^{3}/\mu^{3})$.
According to (35), it is counted by restricted square strict tableaux of shape
$(3,2,1)$ with some additional constraints. On the left hand side of Fig. 4,
we show $\bm{\nu}$ with its shifted contents and we give an example of a
restricted square strict tableau $T$ of shape $(3,2,1)$, which satisfies the
constraint $\operatorname{Des}_{3}^{\prime}(T)=D^{\prime}(\bm{\nu})$. We
colored the boxes of $\lambda^{i}/\mu^{i}$, therefore the pairs counting
$\operatorname{inv}^{\prime}_{3}(T)$ can be represented as the edges of a
graph on three vertices (which is shown on two drawings on the right hand
side).

Figure 4. Restricted square strict tableau corresponding to Schur-expansion of
an LLT polynomial of three skew shapes.

Using the notation from Fig. 4, let $e_{i,j}(T)$ denote the number of pairs
$(\square,\square^{\prime})$ contributing to
$\operatorname{inv}_{3}^{\prime}(T)$ with $T(\square)\equiv i$ and
$T(\square^{\prime})\equiv j$ modulo $3$, so that

$\operatorname{inv}_{3}^{\prime}(T)=e_{1,2}(T)+e_{1,3}(T)+e_{2,3}(T).$

We believe that the following is true:

######  Conjecture 4.3.

For any triple of skew diagrams
$\bm{\nu}=(\lambda^{1}/\mu^{1},\lambda^{2}/\mu^{2},\lambda^{3}/\mu^{3})$ and
every triple $\\{i,j,k\\}=\\{1,2,3\\}$ with $i<j$, we have

(36)
$\operatorname{LLT}(\lambda^{i}/\mu^{i},\lambda^{j}/\mu^{j})(q)\cdot\operatorname{LLT}(\lambda^{k}/\mu^{k})(q)=\sum_{\lambda}\left(\sum_{\begin{subarray}{c}T\in\operatorname{RSST}(\lambda)\\\
\operatorname{Des}_{3}^{\prime}(T)=D^{\prime}(\beta)\\\ c(\bm{\nu})\text{ -
entries of }T\end{subarray}}q^{e_{i,j}(T)}\right)s_{\lambda}.$

######  Corollary 4.4.

Assume that 4.3 holds true. Then 2.7 holds true for all $r$-colored triples of
skew shapes.

###### Proof.

The proof follows the same argument as the one used in Theorem 3.10 to show
that

$[s_{\lambda}]\kappa_{\operatorname{LLT}}(\bm{\nu},f)=\sum_{\begin{subarray}{c}T\in\operatorname{RSST}(\lambda)\\\
\operatorname{Des}_{3}^{\prime}(T)=D^{\prime}(\beta)\\\ c(\bm{\nu})\text{ -
entries of }T\end{subarray}}\mathcal{I}_{(G^{T})_{f}}(q),$

where $G^{T}$ is an $f$-colored graph whose vertices are entries of $T$ and we
connect pairs contributing to $\operatorname{inv}_{3}^{\prime}(T)$. ∎

Note that the above argument works for any $r$-colored tuple of shapes
$(\bm{\nu},f)$ and thus, 2.7 suggests the following interesting structure of
the coefficients of LLT-polynomials in the Schur expansion.

######  Problem 4.5.

Let $\bm{\nu}=(\lambda^{1}/\mu^{1},\dots,\lambda^{r}/\mu^{r})$ be an $r$-tuple
of skew Young diagrams. Is it true that for any partition $\lambda$ there
exists a class of graphs $\mathcal{G}^{\bm{\nu}}_{\lambda}$ with the set of
vertices $[1..r]$ such that for any set-partition $\pi\in\mathcal{P}([1..r])$
one has

$[s_{\lambda}]\prod_{B\in\pi}\operatorname{LLT}^{\operatorname{cospin}}((\bm{\nu})^{B})=\sum_{G\in\mathcal{G}^{\bm{\nu}}_{\lambda}}q^{\sum_{B\in\pi}e_{B}},$

where $(\bm{\nu})^{B}:=(\bm{\nu},\operatorname{id}_{[1..r]})^{B}$ and
$\operatorname{id}_{[1..r]}\colon[1..r]\to[1..r]$ is the identity function?

Note that the affirmative answer for this problem implies 2.7 providing its
combinatorial interpretation:

$[s_{\lambda}]\kappa_{\operatorname{LLT}^{\operatorname{cospin}}}(\bm{\nu},f)=\sum_{G\in\mathcal{G}^{\bm{\nu}}_{\lambda}}\mathcal{I}_{(G)_{f}}(q).$

In the next section, we show that 4.5 has an affirmative answer in some
special cases and thus, 2.7 holds true for them.

###  4.1. Unicellular LLT and melting lollipops

Figure 5. The correspondence between unicellular LLT polynomials, Dyck paths
and unit interval graphs. The graph $G(\bm{\nu})$ on the right is the melting
lollipop graph $L^{(2)}_{(5,2)}$ and we display the arrangment of unit
intervals which realizes it as the unit interval graph.

A _Schröder path_ of length $n$ is a path from $(0,0)$ to $(n,n)$ composed of
steps $\uparrow\ =(0,1)$, $\rightarrow\ =(1,0)$, and $\nearrow\ =(1,1)$
(referred to as north, east, and diagonal steps, respectively), which stays
above the main diagonal (i.e., it can touch it, but the diagonal steps lie
strictly above it). Denote by $(i,j)$ the coordinates of the $1\times 1$ box
with upper right vertex in $(i,j)$. It is well known [Hag08] that the
vertical-shape LLT polynomials of homogenous degree $n$ are in bijection with
Schröder paths of length $n$: start from an $\ell$-tuple of vertical shapes
$\bm{\nu}$ of size $n$, label its boxes by their shifted contents and
standardize them, i.e. replace them (in the unique way) by labels in $[1..n]$
such that the order of new labels is the same as the order of shifted
contents. Now construct a Schröder path $F(\bm{\nu})$ such that the box
$(i,j)$ lies below the path if and only if the entry $i$ attacks the entry $j$
in $\bm{\nu}$ and the box $(i,j)$ lies on the diagonal step if the entry $j$
lies directly below the entry $i$. This procedure is clearly invertible and we
denote by $\bm{\nu}(F)$ the tuple of vertical strips associated with the
Schröder path $F$ (see the left side of Fig. 5 and consult [Hag08] for more
details).

A special case of a Schröder path is a _Dyck path_ , that is a path with no
diagonal steps. The corresponding $\ell$-tuple of vertical shapes $\bm{\nu}$
of size $n$ is a sequence of $n$ single boxes (i.e. $\ell=n$) and its LLT
polynomial is called _unicellular_. It is remarkable that the LLT graphs
associated with unicellular LLT polynomials are precisely _unit interval
graphs_ , i.e. they can be realized as the intersection graphs of $n$ unit
intervals on the line (see the right side of Fig. 5).

Note that for every unit interval graph $G$ on $n$ vertices, one has

$\operatorname{LLT}(G)(1)=e_{1}^{n}=\sum_{\lambda\vdash
n}\sum_{T\in\operatorname{SYT}(\lambda)}s_{\lambda}.$

Therefore, it is natural to look for a statistic
$a_{G}:\operatorname{SYT}\to\mathbb{N}$ such that

$[s_{\lambda}]\operatorname{LLT}(G)(q)=\sum_{T\in\operatorname{SYT}(\lambda)}q^{a_{G}(T)}.$

Recall that the descent set $\operatorname{Des}(T)$ of a standard Young
tableau $T\in\operatorname{SYT}(\lambda)$ is given by the values
$i\in[1..n-1]$ for which the entry $i+1$ lies in $T$ in a row above the entry
$i$333it is easy to check that this definition coincides with the previous
definition of $\operatorname{Des}(T)$ given in Section 3.2.3 in the special
case of $\bm{\nu}=(\lambda)$ and $T\in\operatorname{SYT}(\bm{\nu})$ and define

$\overleftarrow{\operatorname{Des}(T)}:=\\{n+1-i\colon
i\in\operatorname{Des}(T)\\}.$

Let $m\geq 1,n$ be nonnegative integers and $0\leq k\leq m-1$. A _melting
lollipop_ $L_{(m,n)}^{(k)}$ is a graph with the vertex set $[1..m+n]$, built
by joining the complete graph on vertices $[1..m]$ with the path on vertices
$[m..m+n]$ (with edges of the form $(i,i+1)$) and erasing edges
$(1,m),(2,m),\dots,(k,m)$. The unit interval graph depicted in Fig. 5 is the
melting lollipop $L^{(2)}_{(5,2)}$.

Recently Huh, Nam and Yoo proved the following theorem [HNY20]:

######  Theorem 4.6.

[HNY20] Let $\mathcal{F}_{n}$ be the family of unit interval graphs with $n$
vertices such that

$\operatorname{LLT}(G)(q)=\sum_{\lambda\vdash
n}\sum_{T\in\operatorname{SYT}(\lambda)}q^{\sum_{i\in\overleftarrow{\operatorname{Des}(T)}}\deg^{G}_{\operatorname{in}}(i)}s_{\lambda}$

for each $G\in\mathcal{F}_{n}$ (here $\deg^{G}_{\operatorname{in}}(i)$ denotes
the number of edges in $G$ incoming to the vertex $i$). Then $\mathcal{F}_{n}$
contains melting lollipops and their disjoint unions.

Melting lollipops contain two extremal cases for which Theorem 4.6 is a
classical result: the complete graph $K_{n}=L_{(n,0)}^{(0)}$ and the path
graph $P_{n}=L_{(1,n-1)}^{(0)}=L_{(2,n-2)}^{(0)}$.

######  Theorem 4.7.

Let $G$ be a melting lollipop graph with $r$ vertices. Then for every set-
partition $\pi\in\mathcal{P}([1..r])$, one has

$[s_{\lambda}]\prod_{B\in\pi}\operatorname{LLT}^{\operatorname{cospin}}(G|_{B})=\sum_{T\in\operatorname{SYT}(\lambda)}q^{\sum_{B\in\pi}e_{B}(G_{\operatorname{in}}^{\overleftarrow{\operatorname{Des}(T)}})},$

where $G_{\operatorname{in}}^{A}$ is a graph obtained from $G$ by removing all
the edges which are not incoming to vertices in $A\subset V$. In particular,
4.5 and 2.7 have an affirmative answer in this case and

$[s_{\lambda}]\kappa_{\operatorname{LLT}^{\operatorname{cospin}}}(G,f)=\sum_{T\in\operatorname{SYT}(\lambda)}\mathcal{I}_{(G_{\operatorname{in}}^{\overleftarrow{\operatorname{Des}(T)}})_{f}}(q).$

###### Proof.

It is enough to notice that

  * •

for every set-partition $\pi\in\mathcal{P}([1..r])$ the graph
$G_{\pi}:=\bigoplus_{B\in\pi}G|_{V_{B}}$ is a disjoint union of melting
lollipops so that

$\prod_{B\in\pi}\operatorname{LLT}(G|_{V_{B}})(q)=\operatorname{LLT}(G_{\pi})(q)=\sum_{\lambda\vdash
n}\sum_{T\in\operatorname{SYT}(\lambda)}q^{\sum_{i\in\overleftarrow{\operatorname{Des}(T)}}\deg^{G_{\pi}}_{\operatorname{in}}(i)}s_{\lambda};$

  * •

the identity

$\sum_{i\in A}\deg^{G}_{\operatorname{in}}(i)=|E(G_{\operatorname{in}}^{A})|$

follows directly from the construction of $G_{\operatorname{in}}^{A}$.

∎

######  Remark 4.8.

Note that the class $\mathcal{F}_{n}$ is strictly smaller than the class of
unit interval graphs on $n$ vertices which can be seen already for $n=4$: the
unit interval graph $G=(V=[1..4],E)$ with $E=\\{(1,2),(2,3),(2,4),(3,4)\\}$
does not belong to $\mathcal{F}_{n}$. On the other hand, we were not able to
find any graph which belongs to $\mathcal{F}_{n}$ and is not a disjoint union
of melting lollipops, and it is tempting to conjecture that these two classes
of graphs coincide.

We finish by discussing a different approach to attacking 2.7. One can try to
find an explicit formula for
$\kappa_{\operatorname{LLT}^{\operatorname{cospin}}}(\bm{\nu},f)$ as a linear
combination of LLT polynomials with coefficients in $\mathbb{Z}_{\geq
0}[q,q^{-1}]$. Note that Schur polynomials are a special case of LLT
polynomials so 2.7 claims that such an expression exists. Nevertheless, we
want to stress out that LLT polynomials are not linearly independent so one
can hope that some expressions are more natural and easier than others. One
particular example where we observed such a natural combinatorial expression
is the unicellular case corresponding to the complete graph, i.e., when
$\bm{\nu}$ is an $r$-colored tuple of $r$ single boxes:
$\lambda^{i}=(1),\mu^{i}=\emptyset$ for all $1\leq i\leq r$. This case might
seem to be trivial at first sight, but one can quickly convince oneself that
this is a false impression. It turned out that the corresponding cumulant
involves beautiful combinatorial objects such as parking functions and it has
a form similar to the formula in the Shuffle Theorem, conjectured in [HHL+05b]
and proved by Carlson and Mellit [CM18]. Before we show the formula we quickly
explain what parking functions are.

Figure 6. The correspondence between tuples of vertical shapes, Schröder paths
and parking functions.

A _parking function_ $P\in\operatorname{PF}_{n-1}$ of size $n-1$ is a function
$P\colon[1..n-1]\to[1..n-1]$ such that for each $i\in[1..n-1]$, one has
$|P^{-1}(i)|\geq i$. One can represent a parking function by drawing a Dyck
path from $(1,1)$ to $(n,n)$ and labeling the boxes to the right of north
steps by distinct integers $[1..n-1]$ in such a way that the labels of boxes
stacked in the same column are upward increasing. Starting from a parking
function $P\in\operatorname{PF}_{n-1}$, convert the corresponding Dyck path of
length $n-1$ into a Schröder path of length $n$ by adding steps
$\uparrow,\rightarrow$ starting from $(0,0)$ and then replacing all the pairs
of consecutive steps $(\rightarrow,\uparrow)$ by $\nearrow$, see the right
side of Fig. 6. The following formula was recently proved by the second
author:

######  Theorem 4.9.

[Kow20] Let $(\bm{\nu},f)=(((1),\dots,(1)),f)$ be an $r$-colored tuple of $r$
single boxes. Then

(37)
$\kappa_{\operatorname{LLT}}(\bm{\nu},f)=\kappa_{\operatorname{LLT}^{\operatorname{cospin}}}(\bm{\nu},f)=\sum_{P\in\operatorname{PF}_{r-1}}\operatorname{LLT}(\bm{\nu}(F(P))),$

where we sum over all parking functions of size $r-1$.

This formula gives a positive expression in terms of vertical-shaped LLT
polynomials, which are Schur-positive (by [GH07]) and $e$-positive after
applying the shift (by [D’A20, AS22]). In particular, Theorem 4.9 gives yet
another proof of 2.7 and also 2.6 in this special case. Although Theorem 4.9
might suggest that there is a combinatorial formula expressing an LLT cumulant
as a positive combination of LLT polynomials, we were not able to find a
pattern allowing us to construct such a formula in general and we leave this
problem for further investigations in the future.

## Acknowledgments

MD would like to thank Erik Carlsson and Fernando Rodriguez Villegas for
interesting discussion on possible connections between Macdonald cumulants and
the work of Hausel, Letellier and Rodriguez Villegas [HLRV11].

## References

  * [Ale21] Per Alexandersson, _LLT polynomials, elementary symmetric functions and melting lollipops_ , J. Algebraic Combin. 53 (2021), no. 2, 299–325. MR 4238181
  * [AMRV19] T. Abdelgadir, A. Mellit, and F. Rodriguez Villegas, _The Tutte polynomial and toric Nakajima quiver varieties_ , Preprint arXiv:1910.01633, 2019\.
  * [AN21] Alex Abreu and Antonio Nigro, _A symmetric function of increasing forests_ , Forum Math. Sigma 9 (2021), Paper No. e35, 21. MR 4252214
  * [AP18] Per Alexandersson and Greta Panova, _LLT polynomials, chromatic quasisymmetric functions and graphs with cycles_ , Discrete Math. 341 (2018), no. 12, 3453–3482. MR 3862644
  * [AS22] Per Alexandersson and Robin Sulzgruber, _A combinatorial expansion of vertical-strip LLT polynomials in the basis of elementary symmetric functions_ , Adv. Math. 400 (2022), Paper No. 108256, 58. MR 4385139
  * [Ber17] François Bergeron, _Open questions for operators related to rectangular Catalan combinatorics_ , J. Comb. 8 (2017), no. 4, 673–703. MR 3682396
  * [Bla16] Jonah Blasiak, _Haglund’s conjecture on 3-column Macdonald polynomials_ , Math. Z. 283 (2016), no. 1-2, 601–628. MR 3489082
  * [CBVdB04] William Crawley-Boevey and Michel Van den Bergh, _Absolutely indecomposable representations and Kac-Moody Lie algebras_ , Invent. Math. 155 (2004), no. 3, 537–559, With an appendix by Hiraku Nakajima. MR 2038196
  * [CD22] Guillaume Chapuy and Maciej Dołęga, _Non-orientable branched coverings, $b$-Hurwitz numbers, and positivity for multiparametric Jack expansions_, Adv. Math. 409 (2022), Paper No. 108645, 1–72.
  * [CM18] Erik Carlsson and Anton Mellit, _A proof of the shuffle conjecture_ , J. Amer. Math. Soc. 31 (2018), no. 3, 661–697. MR 3787405
  * [D’A20] Michele D’Adderio, _$e$ -positivity of vertical strip LLT polynomials_, J. Combin. Theory Ser. A 172 (2020), 105212, 15. MR 4054520
  * [DF17] Maciej Dołęga and Valentin Féray, _Cumulants of Jack symmetric functions and the $b$-conjecture_, Trans. Amer. Math. Soc. 369 (2017), no. 12, 9015–9039. MR 3710651
  * [Doł17] Maciej Dołęga, _Strong factorization property of Macdonald polynomials and higher-order Macdonald’s positivity conjecture_ , J. Algebraic Combin. 46 (2017), no. 1, 135–163. MR 3666415
  * [Doł19] Maciej Dołęga, _Macdonald cumulants, $G$-inversion polynomials and $G$-parking functions_, European J. Combin. 75 (2019), 172–194. MR 3862962
  * [ELW10] Eric Egge, Nicholas A. Loehr, and Gregory S. Warrington, _From quasisymmetric expansions to Schur expansions via a modified inverse Kostka matrix_ , European J. Combin. 31 (2010), no. 8, 2014–2027. MR 2718279
  * [F1́5] Valentin Féray, _Cyclic inclusion-exclusion_ , SIAM J. Discrete Math. 29 (2015), no. 4, 2284–2311. MR 3427040
  * [Ges84] Ira M. Gessel, _Multipartite $P$-partitions and inner products of skew Schur functions_, Combinatorics and algebra (Boulder, Colo., 1983), Contemp. Math., vol. 34, Amer. Math. Soc., Providence, RI, 1984, pp. 289–317. MR 777705
  * [Ges95] by same author, _Enumerative applications of a decomposition for graphs and digraphs_ , Discrete Math. 139 (1995), no. 1-3, 257–271, Formal power series and algebraic combinatorics (Montreal, PQ, 1992). MR 1336842
  * [GH07] Ian Grojnowski and Mark Haiman, _Affine Hecke algebras and positivity of LLT and Macdonald polynomials_ , Preprint, 2007.
  * [GHQR19] Adriano M. Garsia, James Haglund, Dun Qiu, and Marino Romero, _$e$ -positivity results and conjectures_, Preprint arXiv:1904.07912, 2019\.
  * [GJ96] I. P. Goulden and D. M. Jackson, _Connection coefficients, matchings, maps and combinatorial conjectures for Jack symmetric functions_ , Trans. Amer. Math. Soc. 348 (1996), no. 3, 873–892. MR 1325917 (96m:05196)
  * [GT96] A. M. Garsia and G. Tesler, _Plethystic formulas for Macdonald q, t-Kostka coefficients_ , Adv. Math. 123 (1996), no. 2, 144–222.
  * [Hag08] James Haglund, _The $q$,$t$-Catalan numbers and the space of diagonal harmonics_, University Lecture Series, vol. 41, American Mathematical Society, Providence, RI, 2008, With an appendix on the combinatorics of Macdonald polynomials. MR 2371044
  * [Hai01] Mark Haiman, _Hilbert schemes, polygraphs and the Macdonald positivity conjecture_ , J. Amer. Math. Soc. 14 (2001), no. 4, 941–1006. MR 1839919
  * [HHL05a] J. Haglund, M. Haiman, and N. Loehr, _A combinatorial formula for Macdonald polynomials_ , J. Amer. Math. Soc. 18 (2005), no. 3, 735–761. MR 2138143
  * [HHL+05b] J. Haglund, M. Haiman, N. Loehr, J. B. Remmel, and A. Ulyanov, _A combinatorial formula for the character of the diagonal coinvariants_ , Duke Math. J. 126 (2005), no. 2, 195–232. MR 2115257
  * [HLRV11] Tamás Hausel, Emmanuel Letellier, and Fernando Rodriguez-Villegas, _Arithmetic harmonic analysis on character and quiver varieties_ , Duke Math. J. 160 (2011), no. 2, 323–400. MR 2852119
  * [HNY20] JiSun Huh, Sun-Young Nam, and Meesue Yoo, _Melting lollipop chromatic quasisymmetric functions and Schur expansion of unicellular LLT polynomials_ , Discrete Math. 343 (2020), no. 3, 111728, 21. MR 4033624
  * [HS02] Tamás Hausel and Bernd Sturmfels, _Toric hyperKähler varieties_ , Doc. Math. 7 (2002), 495–534. MR 2015052
  * [JLuR00] Svante Janson, Tomasz Ł uczak, and Andrzej Rucinski, _Random graphs_ , Wiley Series in Discrete Mathematics and Optimization, vol. 45, Wiley-Interscience, 2000.
  * [JV13] Matthieu Josuat-Vergès, _Cumulants of the $q$-semicircular law, Tutte polynomials, and heaps_, Canad. J. Math. 65 (2013), no. 4, 863–878. MR 3071084
  * [KN98] Anatol N. Kirillov and Masatoshi Noumi, _Affine Hecke algebras and raising operators for Macdonald polynomials_ , Duke Math. J. 93 (1998), no. 1, 1–39.
  * [Kno97] Friedrich Knop, _Integrality of two variable Kostka functions_ , Journal fur die Reine und Angewandte Mathematik (1997), 177–189.
  * [Kow20] Maciej Kowalski, _LLT cumulants of unicellular Young diagrams, parking functions and Schur positivity_ , Preprint arXiv:2011.15080, 2020.
  * [KS97] Friedrich Knop and Siddhartha Sahi, _A recursion and a combinatorial formula for Jack polynomials_ , Invent. Math. 128 (1997), no. 1, 9–22. MR 1437493
  * [Lee21] Seung Jin Lee, _Linear relations on LLT polynomials and their k-Schur positivity for $k=2$_, J. Algebraic Combin. 53 (2021), no. 4, 973–990. MR 4263640
  * [LLT97] Alain Lascoux, Bernard Leclerc, and Jean-Yves Thibon, _Ribbon tableaux, Hall-Littlewood functions, quantum affine algebras, and unipotent varieties_ , J. Math. Phys. 38 (1997), no. 2, 1041–1068. MR 1434225
  * [LS59] V. P. Leonov and A. N. Shiryaev, _On a method of semi-invariants_ , Theor. Probability Appl. 4 (1959), 319–329. MR 0123345
  * [LT00] Bernard Leclerc and Jean-Yves Thibon, _Littlewood-Richardson coefficients and Kazhdan-Lusztig polynomials_ , Combinatorial methods in representation theory (Kyoto, 1998), Adv. Stud. Pure Math., vol. 28, Kinokuniya, Tokyo, 2000, pp. 155–220. MR 1864481
  * [LV95] Luc Lapointe and Luc Vinet, _A Rodrigues formula for the Jack polynomials and the Macdonald-Stanley conjecture_ , Internat. Math. Res. Notices 1995 (1995), no. 9, 419–424. MR 1360620 (96i:33018)
  * [LV97] by same author, _Rodrigues formulas for the Macdonald polynomials_ , Adv. Math. 130 (1997), no. 2, 261–279. MR 1472319 (99e:05128)
  * [Mac88] I. G. Macdonald, _A new class of symmetric functions_ , Publ. IRMA Strasbourg 372 (1988), 131–171.
  * [NŚ11] Jonathan Novak and Piotr Śniady, _What is a Free Cumulant?_ , Notices of the AMS 58 (2011), no. 2, 300–301.
  * [Rob14] Austin Roberts, _Dual equivalence graphs revisited and the explicit Schur expansion of a family of LLT polynomials_ , J. Algebraic Combin. 39 (2014), no. 2, 389–428. MR 3159257
  * [Sah96] Siddhartha Sahi, _Interpolation, integrality, and a generalization of Macdonald’s polynomials_ , Internat. Math. Res. Notices 1996 (1996), no. 10, 457–471.
  * [SSW03] A. Schilling, M. Shimozono, and D. E. White, _Branching formula for $q$-Littlewood-Richardson coefficients_, vol. 30, 2003, Formal power series and algebraic combinatorics (Scottsdale, AZ, 2001), pp. 258–272. MR 1979794
  * [SW85] Dennis W. Stanton and Dennis E. White, _A Schensted algorithm for rim hook tableaux_ , J. Combin. Theory Ser. A 40 (1985), no. 2, 211–247. MR 814412
  * [The20] The Sage Developers, _Sagemath, the Sage Mathematics Software System (Version 9.2)_ , 2020, https://www.sagemath.org.
  * [Tom21] Foster Tom, _A combinatorial Schur expansion of triangle-free horizontal-strip LLT polynomials_ , Comb. Theory 1 (2021), Paper No. 14, 28. MR 4396219
  * [Tut54] W. T. Tutte, _A contribution to the theory of chromatic polynomials_ , Canadian J. Math. 6 (1954), 80–91. MR 0061366
  * [Wei35] Louis Weisner, _Abstract theory of inversion of finite series_ , Trans. Amer. Math. Soc. 38 (1935), no. 3, 474–484.