Data vs classifiers, who wins?
Cardoso, L. et al.

mode = title]Data vs classifiers, who wins?

1]Lucas F. F. Cardoso[type=editor,

[1]

Conceptualization of this study, Methodology, Software development, Writing - main text

[1]Faculdade de Computação, Universidade Federal do Pará, Belém, Brazil

1]Vitor C. A. Santos[type=editor,

Translation, Result analysis

1]Regiane S. Kawasaki Francês[type=editor,

Text curation, Writing - Original draft preparation

2]Ricardo B. C. Prudêncio[type=editor,

Writing - reference support

[2]Centro de Informática, Universidade Federal de Pernambuco, Recife, Brazil

3]Ronnie C. O. Alves[type=editor,

Orientation, General work review, Writing - Final review

[3]Instituto Tecnológico Vale, Belém, Brazil

[cor1]Corresponding author

* decodIRT automates the evaluation of datasets and algorithms by Item Response Theory.

* Item Response Theory can serve to find the best subset of a benchmark.

* Rating systems and Item Response Theory can be used together to evaluate algorithms.

* Innate ability can be seen as an algorithm's ability to generalize.

* The Glicko-2 system with the IRT allows exploring the concept of innate ability.

The classification experiments covered by machine learning (ML) are composed by two important parts: the data and the algorithm. As they are a fundamental part of the problem, both must be considered when evaluating a model's performance against a benchmark. The best classifiers need robust benchmarks to be properly evaluated. For this, gold standard benchmarks such as OpenML-CC18 are used. However, data complexity is commonly not considered along with the model during a performance evaluation. Recent studies employ Item Response Theory (IRT) as a new approach to evaluating datasets and algorithms, capable of evaluating both simultaneously. This work presents a new evaluation methodology based on IRT and Glicko-2, jointly with the decodIRT tool developed to guide the estimation of IRT in ML. It explores the IRT as a tool to evaluate the OpenML-CC18 benchmark for its algorithmic evaluation capability and checks if there is a subset of datasets more efficient than the original benchmark. Several classifiers, from classics to ensemble, are also evaluated using the IRT models. The Glicko-2 rating system was applied together with IRT to summarize the innate ability and classifiers performance. It was noted that not all OpenML-CC18 datasets are really useful for evaluating algorithms, where only 10% were rated as being really difficult. Furthermore, it was verified the existence of a more efficient subset containing only 50% of the original size. While Randon Forest was singled out as the algorithm with the best innate ability.

IRT Machine Learning Benchmarking OpenML Classification Rating

§ INTRODUCTION

Machine Learning (ML) commonly consists of allowing the computer to be able to recognize patterns through algorithms and then classifying the analyzed data according to these patterns [Kotsiantis et al., 2007]. Several applications in different areas adopt ML models to perform tasks. The types of learning algorithms range from unsupervised to supervised [Monard and Baranauskas, 2003]. Because of this, studies in ML areas have been growing very fast recently. In this work, the focus is on supervised learning algorithms, more precisely on classification algorithms, which are commonly adopted for pattern recognition tasks in several applications [Domingos, 2012].

It is well known that there is not only one supervised learning algorithm, but several types of algorithms that use different types of techniques to perform learning [Monard and Baranauskas, 2003]. Along with this, it is also easy to observe that there are several types of datasets that have particular characteristics. However, there is no specific method capable of asserting the best algorithm to be used for a given dataset. Because of this, it is common to evaluate algorithms empirically.

Empirically evaluating ML algorithms is crucial to understanding the advantages and limitations of available techniques. In this approach, algorithms are usually evaluated using datasets or beenchmarks from online repositories. Following the growth of the area, in recent years different online dataset repositories have emerged, such as OpenML [Vanschoren et al., 2014].

Developed to work as a platform, where ML researchers can share their results, methodologies and datasets used in their experiments, such as evaluating the performance of a classifier against a dataset. Such platform improves the organization and use of information, creating a collaborative environment [Vanschoren et al., 2014]. In addition, OpenML has several reference benchmarks, such as OpenML Curated Classification 2018 (OpenML-CC18), a benchmark for classification tasks that has 72 datasets curated and standardized by the platform [Bischl et al., 2017].

After selection of a benchmark or dataset, ML models can be trained and tested by applying a specific experimental methodology (e.g., cross-validation) and then evaluated using evaluation metrics of interest (e.g., accuracy). This strategy, however, does not allow for an in-depth analysis of the real capacity of the model. Considering that the type of dataset used may be favoring the algorithm, giving the false impression that the classifier is, in fact, the best in relation to the others [Martínez-Plumed et al., 2016]. This situation causes the complexity of the data to be taken into account during the process of evaluating the performance of an algorithm. According Kubat, 2017, only the use of the classical evaluation metrics can not guarantee that the evaluation result is completely reliable. Therefore, it is important that other metrics are applied to result in a more robust assessment.

But in the data vs classifiers fight, who is more important? According to [Domingos, 2012], even a simpler classifier can beat the best classifier if the first classifier has much more training data than the second. At the same time, “Data alone is not enough”, models are a fundamental part of the process and make all the difference in the final learning result. In this way, data and models are two sides of the same coin of machine learning experimentation. So, how to evaluate the classifiers considering the data as well?

In previous works [Prudêncio et al., 2015, Martínez-Plumed et al., 2016, Martínez-Plumed et al., 2019, Song and Flach, 2021] Item Response Theory (IRT) was adopted as a new tool to solve this problem and provide a new and robust approach that allows evaluating both datasets and classifiers. Used in psychometric tests, IRT aims to measure an individual's ability to correctly answer a set of items (e.g., questions) by calculating the individual's probability of answering the item correctly. By considering classifiers as individuals and test instances as items, it is then possible to apply the concepts of IRT in the ML field. In addition, the IRT assesses the individual's ability according to the complexity of the item being answered, through the parameters that describe the item. Thus, when applying IRT in ML, it is possible to simultaneously evaluate datasets and algorithms already considering the complexity of the data when measuring the classifier performance.

Despite the advantages of applying IRT in ML, evaluating benchmarks and algorithms with IRT is not a simple task. Given its characteristics, the evaluation requires analysis of the performance of algorithms and the complexity of datasets at the instance level. Given the above, this work proposes the strategy of using the IRT together with rating systems [Samothrakis et al., 2014] to assist in the evaluation of both classifiers and benchmarks. Such systems are commonly used to assess the “strength” of an individual in a competition (e.g., chess), that is, these systems serve to measure how good the individual is and measure their proficiency in a given activity. Thus, the Glicko-2 [Glickman, 2012] rating system was adopted in order to create a ranking to summarize the results obtained by calculating the IRT.

Preliminary results were obtained in Cardoso et al., 2020, where it was possible to apply the IRT concepts to evaluate a set of 60 datasets from the OpenML-CC18 benchmark, observing the item parameters of difficulty and discrimination, using the developed tool decodIRT. This allowed us to classify the OpenML-CC18 benchmark according to its ability to separate good from bad classifiers and to test the classification power of algorithms.

Given the results already obtained, the following questions were raised: Would it be possible to use the IRT estimators to choose the best benchmark subset within OpenML-CC18? Are there datasets within a benchmark that might not be really good for evaluating learning algorithms? Through this, this work aims to expand the concepts and technologies already explored initially in [Cardoso et al., 2020]. For this, IRT is used to create subsets of OpenML-CC18 datasets, then the combination of IRT and Glicko-2 is applied to generate the classifier rankings. In sequence, each ranking is analyzed, considering the expected performance of the classifiers to evaluate and then choosing a possible subset that is more efficient than the original. In addition, this work also seeks to use the rankings generated for each subset to explore with the Glicko-2 system the existence of an “innate ability” of the evaluated classifiers and then point out which is the best algorithm.

The main contributions of this work are summarized below:

* The proposal of a new methodology to simultaneously evaluate the performance of algorithms and the difficulty of datasets, based on the combination of IRT and Glicko-2.

* The decodIRT tool, developed to automate the process of evaluating datasets and algorithms via IRT.

* Application of the proposed methodology to analyze existing problems in a known benchmark: OpenML-CC18.

* The use of the methodology proposed in OpenML-CC18 to point out the best classifier, exploring the concept of innate ability.

The rest of this work is organized as follows: Section 2 contextualizes the main issues covered in this work, more precisely about classical performance metrics, concept of innate ability, OpenML, Item Response Theory and the Glicko-2 system. Section 3 presents the related work and compares it with the present work. Section 4 presents the methodology used, explains how decodIRT and the Glicko-2 system were used. Section 5 discusses the results obtained. Section 6 presents the final considerations of the work and also brings a self-criticism made by the authors.

§ BACKGROUND

§.§ Classifier abiliity evaluation

In ML it is not enough just to train an algorithm, generate a model and start using it. It is very important to know if the model that was generated was really able to learn to classify correctly. For this, one can apply performance evaluation metrics most common. According to Kubat, 2017 there are different performance metrics and each one can be more interesting than the other depending on the aspect you want to evaluate.

Accuracy and error rate are one of the most used classic metrics. However, the result of a single performance metric can be misleading and not correctly reflect the true capability of a classifier [Kubat, 2017]. In Ferri et al., 2009 the authors experimentally analyzed the behavior of a total of 18 performance metrics. In the work, it is reinforced that the different performance metrics can generate different evaluations about the model's capacity depending on the situation, that is, it depends on the data set used. For example, in situations where there is an imbalance of classes or the dataset has few instances, a given metric may be preferable over the others. Thus, it is important to choose one or more specific metrics that are best able to evaluate the model, always considering the situation of the experiment.

§.§.§ Innate ability

According to Domingos, 2012, the main objective of ML is generalization, that is, the algorithm that best manages to generalize during training is the best one to be chosen. Making an analogy with human life, the preferable algorithm would be one that has the best “innate ability” for learning. Given this situation, the following questions arise: Is it possible to use classical metrics to measure the innate ability of models? Which metric or metrics would be best suited to assess a model's ability? For this, first, it is important to define what would be the innate ability.

The innate can be conceptualized as: “born with”. This means that the innate ability would then be an ability that is already present from the individual's birth. For example, people from a young age have immense talent in a given activity. When translating this concept into the field of supervised learning, the skill of a model would be its ability to learn to classify well, as this is the only activity to be performed. The innate ability would then be the ability of the algorithm to be able to classify well independently of the hyperparameters and datasets used, as this would be a natural ability that the algorithm was “born with”.

Classical metrics aim to evaluate the classifier facing a specific task, that is, whether the classifier can classify a dataset well. This approach only assesses the ability of the algorithm against a single dataset, which does not allow defining whether the model would perform well in a general context, a situation that we seek to explore with the concept of innate ability.

§.§ OpenML

OpenML is a repository that works as a collaborative environment, where ML researchers can automatically share detailed data and organize it to work more efficiently and collaborate on a global scale [Vanschoren et al., 2014]. It also allows ML tasks to be executed with the repository datasets using the preference algorithm and then share the results obtained within the platform, minimizing the double effort. In addition, OpenML also makes it possible for new datasets to be made available by users, challenging the community to run algorithms on the dataset using specific parameters to solve a given ML task (e.g., classification) [Vanschoren et al., 2014].

The platform can be divided into four main classes, namely: Datasets, Tasks, Flows and Runs. In the Datasets class, the existing datasets in OpenML are made available. Tasks describe what to do with the dataset, define what types of inputs are provided, what types of outputs should be returned, and the scientific protocols that can be used. Flows are precisely the learning algorithms that are applied to solve Tasks. While Runs is the application of a given Flow to a given Task [Vanschoren et al., 2014].

In addition to the four main classes, OpenML also has the Study class which allows you to combine the four main classes into studies to share with the online community or simply to keep a record of a work [23]. The Study class also allows the creation of benchmark suites that can be translated as a set of tasks that are selected to evaluate algorithms under specific conditions. Creating benchmarks that enable the experiments performed on them to be clearly reproducible, interpretable and comparable [Bischl et al., 2017].

§.§ OpenMLCC-18 benchmark

The creation of appropriate benchmarks are key part of the research in ML. They are important pieces for the standardization of studies in the area, enabling the community to follow the progress over time, identify which problems are still a challenge and which algorithms are best for certain applications. The lack of standardized benchmarks available results in many studies using their own sets of pre-processed datasets in their own way. This condition makes it difficult to compare and reproduce the results obtained by these studies [Bischl et al., 2017].

In this context, OpenML also has the advantage of providing several reference benchmarks, such as the OpenMLCC-18 [Link to access OpenML-CC18: <https://www.openml.org/s/99>]. Proposed by Bischl et al., 2017, OpenML-CC18 is a classification benchmark composed of 72 existing OpenML datasets from mid-2018 and which aims to address a series of requirements to create a complete reference set. In addition, it includes several datasets frequently used in benchmarks published in recent years.

According to Bischl et al., 2017, the properties used to filter the datasets are: (a) Number of instances between 500 and 100,000; (b) Number of features up to 5000; (c) At least two classes targeted, where no class has less than 20 instances in total; (d) The proportion between minority and majority classes must be above 0.05; (e) Datasets cannot have been artificially generated; (f) Datasets must allow for randomization through a 10-field cross-validation; (g) No dataset can be a subset of another larger dataset; (h) All datasets must have some source or reference available; (i) No dataset should be perfectly classifiable by a single feature; (j) No dataset should allow a decision tree to achieve 100% accuracy in a 10-field cross-validation task; (k) Datasets cannot have more than 5000 features after a one-hot-encoding process on categorical features; (l) The datsets cannot have been created by binarizing regression or multiclass tasks; (m) No dataset can be sparse.

Therefore, it is understood that OpenML has a lot to contribute to research in the field of machine learning. In the previous work [Cardoso et al., 2020] an initial analysis of OpenML-CC18 was performed using IRT, which allowed the generation of new relevant metadata about the complexity and quality of the benchmark, such as the difficulty and discriminative power of the data. In this present work, we seek to deepen this analysis by looking for a subset of datasets within OpenML-CC18 that is as good or perhaps better than the original. Using IRT to find a more efficient benchmark subset that maintains the characteristics of the original.

§.§ Item Response Theory

According to de Andrade et al., 2000, to assess the performance of individuals in a test, traditionally, the total number of correct answers is used to rank the individuals evaluated. Despite being common, this approach has limitations to assess the actual ability of an individual. On the other hand, IRT allows the assessment of latent characteristics of an individual that cannot be directly observed and aims to present the relationship between the probability of an individual correctly responding to an item and their latent traits, that is, their ability in the assessed knowledge area. One of the main characteristics of the IRT is to have the items as central elements and not the test as a whole, the performance of an individual is evaluated based on their ability to hit certain items of a test and not how many items they hit.

Also according to de Andrade et al., 2000, the IRT is a set of mathematical models that seek to represent the probability of an individual to correctly answer an item based on the item parameters and the respondent's ability, where the greater the individual's ability , the chance of success will also be greater. The various proposed models depend fundamentally on three factors:

* The item nature: whether it is dichotomous, in which it is only considered whether the answer is right or wrong. Or if it is non-dichotomous, where more possible answers are considered.

* Number of populations involved, if it is just one or more than one.

* Amount of latent traces being measured.

Logistic models for dichotomous items are the most used, for these items there are basically three types of models, which differ by the number of item parameters used. These are known as 1, 2 and 3 parameter logistic models. The 3-parameter logistic model, called 3PL, is the most complete among the three, where the probability of an individual $j$ correctly answering an item $i$ given their ability is defined by the following equation:

\begin{equation}
P(U_{ij} = 1|\theta_{j}) = c_{i} + (1 - c_{i})\frac{1}{1+ e^{-a_{i}(\theta_{j}-b_{i})}}
\end{equation}

* $U_ {ij}$ is the dichotomous response that can take the values 1 or 0, being 1 when the individual j hits the item i and 0 when he misses;

* $\theta_{j}$ is the ability of the individual j;

* $b_{i}$ is the item's difficulty parameter and indicates the location of the logistic curve;

* $a_{i}$ is the item's discrimination parameter, i.e., how much the item i differentiates between good and bad respondents. This parameter indicates the slope of the logistic curve. The higher its value, the more discriminating the item is;

* $c_{i}$ is the guessing parameter, representing the probability of a casual hit. It is the probability that a respondent with low ability hits the item.

Although theoretically the discrimination parameter can vary from $-\infty$ to $+\infty$, negative discrimination values are not expected. Because this means that the probability of success is greater for individuals with lower ability values, which goes against what is expected by the IRT [de Andrade et al., 2000]. The other two logistic models can be obtained by simplifying the 3PL. For 2PL the guessing parameter is removed, i.e., $c_{i}=0$. And for 1PL the discrimination parameter is also removed, assuming that $a_{i} =1$. To estimate the item parameters, the response set of all individuals for all items that will be evaluated is used.

Unlike the classic assessment approach, the IRT is not designed to generate a final respondent score. Its purpose is to provide a “magnifying glass” that allows you to observe the individual's performance more specifically on each item and estimate a likely ability level in the assessed area. However, when taking a test, it is common to wait for a final score. Therefore, the IRT also has the concept of True-Score [Lord and Wingersky, 1984], which is the sum of the correct probabilities calculated for each item in the test. The True-Score is then used to set a final score that summarizes the respondent's performance. Based on the above, it is understood that IRT can be an adequate approach to assess the real capacity of classifiers and the complexity of datasets.

§.§ Glicko-2 System

Although IRT already has the True-Score calculation as its own metric to generate a final score. It is understood that in order to properly explore the concept of classifiers' ability, it is necessary to apply a more robust evaluation method together with the IRT. Given this, this work proposes the use of rating systems to summarize the data generated by the IRT and define a final score that is capable of measuring the classifiers' capacity. Given the fact that rating systems are widely used to measure an individual's ability in an activity, where rating is the numerical value that measures the ability [Veček et al., 2014].

They are usually used in competitions to measure the “strength” of competitors, where each individual will have their own rating value and after a match this value is updated depending on the result (win, draw or defeat). Among the existing rating systems, Glicko-2 is the update of the Glicko system developed by Mark E. Glickman [Glickman, 2012] to measure the strength of chess players. The Glicko system was developed in order to improve the Elo system [Elo, 1978] taking into account the players' activity period to ensure greater reliability to the rating value [Samothrakis et al., 2014].

In the Glicko-2 system, each individual has three variables used to measure the statistical strength, they are: the rating value R, the rating deviation (RD) and the volatility ($\sigma$). Despite being very approximate, it cannot be said that the rating value perfectly measures an individual's ability, as it is understood that this value may suffer some variation. For this, the Glicko system has the RD, which allows calculating a 95% reliable range of rating variation, using the formula: $[R-2RD, R+2RD]$. This means that there is a 95% chance that the individual's actual strength is within the calculated range. Therefore, the smaller the RD value, the higher the rating precision [Glickman, 2012, Samothrakis et al., 2014].

To measure how much fluctuation the rating is within its RD range, Glicko uses volatility. Thus, the higher the volatility value, the greater the chances of the rating having large fluctuations within its range, and the lower the volatility, the more reliable the rating is. For example, in a dispute between individuals with low volatility values, based on their ratings it is possible to state more precisely who is the strongest [Samothrakis et al., 2014, Veček et al., 2014].

The Glicko-2 system uses the concept of rating period to estimate rating values, which consist of a sequence of matches played by the individual. At the end of this sequence, the Glicko system updates the player's parameters using the opponents' rating and RD along with the results of each game (e.g., 1 point for victory and 0 for defeat). If the individual is being evaluated for the first time, the Glicko system uses standardized initial values, being: 1500 for rating, 350 for RD and 0.06 for volatility [Glickman, 2012].

§ RELATED WORKS

§.§ IRT and ML

As it is a recent approach, there are few works that aim to apply IRT in studies involving the fields of AI. Prudêncio et al., 2015 seek to take the first steps to employ IRT in ML, the aim of this work is to understand the relationship between a dataset considered difficult and the performance of the models. Where they consider that once they get the knowledge that a given classifier performs better in datasets with instances considered difficult, this makes this method preferable over the others. This analysis is compared to the methodology used for psychometric analysis of the proficiency level of students on a test, using the IRT.

In this study, several Random Forests models with different numbers of trees were used to generate the set of responses to estimate the item parameters. For a case study, the Heart-Statlog dataset and the two-parameter logistic model (2PL) were used, focusing on the difficulty parameter. In addition, the work also uses IRT to identify instances considered noise through the intentional insertion of false instances. To compare the performance of classifiers, from the calculation of the hit probability, three different classifiers were used: Naive Bayes, Logistic Regression and Random Forests.

Another work that also employs IRT in ML is Martínez-Plumed et al., 2016. In this work, the objective is also to apply IRT as a method to understand how different classification algorithms behave when faced with difficult instances of a dataset. In addition to trying to verify if the so-called difficult instances are actually more difficult than the others or if they are just noise. Furthermore, it also seeks to provide an overview of IRT and how it can be used to resolve the many issues that exist about machine learning.

This work is a continuation of the work mentioned above, its main differences are the use of several classifiers from 15 families of algorithms to generate the set of answers. As a case study, they use the Cassini and Heart-Statlog datasets. In addition to proposing the use of artificial classifiers to serve as a baseline between optimal and bad classification in a linear way. This time, the three-parameter logistic model (3PL) was chosen. In addition to presenting the concept of Classifier Characteristic Curve (CCC) as a way to visualize and analyze the variation in the classifiers' performance on instances with different values of difficulty and discrimination.

Martínez-Plumed et al., 2019 is the most complete work, as it aims to describe a pipeline of how to apply IRT in machine learning experiments and explores the advantages of its use, with a focus on supervised learning. In the work, the authors discuss how each item parameter can be used to carry out a deeper analysis about the result of the classifiers. In addition, it is also observed the difference in the use of different logistic models of the IRT, where the 3PL presents the most consistent results.

To perform the IRT analyses, this study used a set of 12 real datasets plus an artificial dataset. In addition, 128 classifiers from 11 different algorithm families were used. The objective is to explore why instances have different item parameter values and how this affects the performance of various learning algorithms. At the end of the work, the authors also suggest five main areas of ML in which IRT can be applied, they are: using IRT to improve classifiers; creation of portfolios of algorithms; classifier selection; improve understanding of the complexity of datasets; and evaluation of classifiers using IRT.

Like the present work, Martinez-Plumed and Hernandez-Orallo, 2018 use the IRT to assess benchmarks according to the difficulty and discrimination estimators, but unlike the other works mentioned above, the focus is on reinforcement learning instead of supervised learning. The authors use the benchmarks Arcade Learning Environment (ALE) [Bellemare et al., 2013] and General Video Game AI (GVGAI) [Perez-Liebana et al., 2015], which are remarkable benchmarks that allow observing the performance of AI agents in different problems. In the work, dual indicators are proposed to evaluate both the benchmarks and the AI agents, coming from different reinforcement learning algorithms, using the IRT concepts and proposing the generality indicator. Which can be understood as the ability of an agent to solve all tasks up to a certain level of difficulty, which can be limited by available resources.

The authors apply the IRT 2PL logistic model for dichotomous items. For this, they use the human performance threshold in each analyzed game, where: if the AI agent's performance can equal or surpass the human, then it is considered the correct answer, otherwise it is an incorrect answer. In addition to benchmarking, the authors also use IRT's estimated ability and generality to assess agents. In order to use IRT to create the most suitable benchmarks, by selecting games with high discrimination values and accurately measuring whether the AI agent is really generalizing or is specializing in specific tasks.

§.§ Benchmarking

In addition to OpenML-CC18 made available by OpenML, other works also highlight the importance of creating and maintaining good benchmarks, such as Nie et al., 2019. In this work the authors propose a new benchmark for NLI (Natural Language Inference), in which the benchmark is developed using an iterative human-and-model-in-the-loop adversary procedure. In this format, humans first write problems that models cannot classify correctly. The resulting new hard instances serve to reveal model weaknesses and can be added to the training set to create stronger models. Therefore, the new model undergoes the same procedure to collect weaknesses in several rounds, where after each cycle a new stronger model is trained and a new set of tests is created. This cycle can be repeated endlessly to create stronger models and harder benchmarks after each iteration.

Based on this premise, Facebook launched Dynabench [9], a platform for dynamic data collection and benchmarking. The goal is to use the adversary method to iteratively create SOTA (state of the art) models and benchmarks, so you can create a benchmark that doesn't get outdated over time.

§.§ Comparison with Related Work

Like the studies presented above [Prudêncio et al., 2015, Martínez-Plumed et al., 2016, Martínez-Plumed et al., 2019], this work also seeks to use IRT as a tool for analyzing datasets and classifiers. Among the objectives of this work, we seek to evaluate the well-known benchmark OpenML-CC18 according to the IRT lens, in order to explore its evaluation capacity. Alongside this, it is proposed to use the Glicko-2 rating system in conjunction with IRT as a new strategy to perform a more robust assessment of a classifier's strength and to assess the quality and efficiency of subsets of a benchmark. The generality metric proposed by Martinez-Plumed and Hernandez-Orallo, 2018 can be compared to the concept of innate ability explored in this work. As well as the use of the discrimination parameter to filter and choose which games would be more suitable to compose a specific benchmark is similar to the strategy adopted in this work to create more efficient benchmarks. And like Dynabench [Nie et al., 2019], this work aims to create and maintain quality benchmarks, evaluating their ability to test classifiers through IRT parameters.

Additionally, the decodIRT is presented, which allows automating the process of analyzing classifiers and datasets through IRT. Where datasets are automatically downloaded from the OpenML platform, so the user only needs to choose which dataset they would like to evaluate. The goal is to refine the benchmark choice through IRT, looking for the most robust and difficult set of datasets available in OpenML.

§ MATERIALS AND METHODS

§.§ IRT and machine learning

Although generally applied for educational purposes, IRT has recently been extended to AI and more specifically to ML [Prudêncio et al., 2015, Martínez-Plumed et al., 2016, Martínez-Plumed et al., 2019]. For this, the following analogy is used: the datasets are the tests, the instances of a dataset are the items and the classifiers are the respondents. For this work, the 3PL logistic model for dichotomous items was used because it is the most complete and consistent among the existing ones [Martínez-Plumed et al., 2019]. The item parameters are then used to evaluate the datasets directly, reporting the percentage of difficult instances, with great discriminative power and with a great chance of random hits. In this way, it is possible to have a view of the complexity of the evaluated datasets and how different classifiers behave in the challenge of classifying different datasets.

To calculate the probability of correct answer, you must first estimate the item parameters and the ability of respondents. According to Martínez-Plumed et al., 2016, there are three possible situations. In the first, only the item parameters are known. In the second situation, only the ability of the respondents is known. And in the third, and also the most common case, both the items parameters and the respondents ability are unknown. This work is in the third case and for this situation, the following two-step interactive method proposed by Birnbaum [Birnbaum et al., 1968] is applied:

* At first, the parameters of each item are calculated only with the answers of each individual. Initial respondent capability values can be the number of correct answers obtained. For classifiers, this study used the accuracy obtained as the initial ability.

* Once obtained the items parameters, the ability of individuals can be estimated. For both item parameters and respondent capacity, simple estimation techniques can be used, such as maximum likelihood estimation [Martínez-Plumed et al., 2016].

§.§ decodIRT tool

To build the IRT logistic models and analyze the benchmarks, the decodIRT [Link to the source code: <https://github.com/LucasFerraroCardoso/IRT_OpenML>] tool initially presented in [Cardoso et al., 2020] was used. DecodIRT has as main objective to automate the analysis of existing datasets in the OpenML platform as well as the proficiency of different classifiers. For this, it depends on the probability of correct answer derived from the logistic model of IRT and the item parameters and the ability of respondents.

As can be seen in Figure <ref>, the decodIRT tool consists of a total of four scripts, with three main scripts (within the square) designed to be used in sequence. The first script is responsible for downloading the OpenML datasets, generating the ML models and placing them to classify the datasets. Then, a response matrix is generated, which contains the classification result of all classifiers for each test instance. The response matrix is the input to the second script, which in turn is responsible for calculating the item's parameters. The last script will use the data generated by the previous ones to rank the datasets using the item parameters and estimate the ability, calculate the response probability and the True-Score of each model.

DecodIRT was modified to work as a package too, where the fourth script (outside the square) was developed to facilitate the use of the tool by the user. Running the tool automatically and allowing the user to use IRT estimators to evaluate benchmarks, create benchmark subsets and store them directly in OpenML.

Flowchart of the decodIRT execution.

§.§ decodIRT_OtML

The first script has the function of downloading selected datasets from OpenML and running the models on the datasets to get the answers that are used to estimate the item parameters. As usual, the datasets are divided into a training set and a test set. So the answers from the classifiers are obtained only for the test set. By definition a stratified split of 70/30 is performed, but for very large datasets, the split is handled so that the test set is composed of 500 instances at most. This number of instances is considered large enough for analysis and will be better justified later.

All ML models are generated using Scikit-learn [Pedregosa et al., 2011] as standard library. Three different sets of classifiers are generated. The first set is composed only of Neural Network (MLP) models. Totaling 120 MLP models, where the depth of the networks gradually increases from 1 to 120. The second set is composed of 12 classifiers from different families that are evaluated in this work, they are: Naive Bayes Gaussian standard, Naive Bayes Bernoulli standard, KNN of 2 neighbors, KNN of 3 neighbors, KNN of 5 neighbors, KNN of 8 neighbors, Standard Decision Trees, Random Forests (RF) with 3 trees, Random Forests with 5 trees, Standard Random Forests, Standard SVM and Standard MLP. The models classified as standard means that the standard Scikit-learn hyperparameters were used. All models are trained using 10-field cross-validation.

The third set of models is composed of 7 artificial classifiers. The concept of artificial classifiers is initially presented in Martínez-Plumed et al., 2016, as follows: a optimal classifier (gets all the classifications right), a pessimal one (all misses), a majority (classifies all instances with the majority class), a minority (classify with the minority class) and three random classifiers (sort randomly). This set is used to provide performance threshold indicators for real classifiers.

Despite using OpenML as the base repository, decodIRT also allows the user to use local datasets and define training and testing sets specifically.

§.§ decodIRT_MLtIRT

This script's function is to use the responses generated by the classifiers and estimate the item parameters for the test instances. As stated previously, the logistic model for dichotomous items is used, which means that regardless of the number of classes existing in each dataset, it is only considered if the classifier was right or wrong in the classification of each instance.

To calculate the item parameters, the Ltm package [Rizopoulos, 2006] for the R language is used, which implements a framework containing several mechanisms for the calculation and analysis of the IRT. The Rpy2 package [Gautier, 2008] was used to perform Python communication with the R packages. As mentioned previously, the maximum limit of 500 instances for estimating item parameters was defined. According to Martínez-Plumed et al., 2019, packages that estimate the IRT item parameters may get stuck in a local minimum or not converge if the number of items is too large. This is not strange, as the IRT is used for psychometric tests, it is very unusual for these tests to have such a large number of questions. Thus, it is recommended that less than 1000 instances be used to estimate the parameters.

§.§ decodIRT_analysis

The third script of the main set is in charge of performing the analysis and organizing the data generated by the previous scripts, in order to make the data easier to read. Among the various functions of this script is the creation of dataset rankings by item parameter. Each ranking will organize the datasets according to the percentage of instances with high values for each of the parameters, i.e. values above a certain threshold. For example, the difficulty ranking will sort the datasets by the number of instances with high difficulty values. Limits can be defined by the user.

In case the user does not define any specific limit, default limit values are used, based on Adedoyin et al., 2013. In the cited work, the authors point out that for an item to be considered difficult, the value of its difficulty parameter must be above 1. Very discriminative items have a discrimination value above 0.75. And for guessing the limit value is 0.2. Analyzing the percentages of item parameters is one of the interests of this work.

Before calculating the probability of correct answer of the classifiers for the instances, one must first estimate the ability of the classifiers, as explained in Birnbaum's method. Both to estimate the ability $\theta$ and to calculate the probability of correct answer, use the Catsim package [Meneghetti and Junior, 2017] from Python. For this, the instances are sorted according to their difficulty and divided into 10 groups, then they are used in ascending order to estimate the ability of the classifiers. After this step, the probability of correct answer can then be calculated.

In addition, this script also implements the True-Score [Lord and Wingersky, 1984] concept explained earlier. Aiming to score the performance of classifiers. The True-Score, later, is also used as input for the generation of the rating values of the Glicko-2 system that is used to evaluate the performance and the innate ability of the classifiers.

§.§ decodIRT_create_benchmark

This last script works as a benchmark builder through IRT, it allows the user to create new benchmark sets within the OpenML platform, using the item parameters to choose the best set of datasets. For this, the script uses the decodIRT tool as a library, where the user can choose from OpenML the set of datasets he wants to evaluate with the IRT, which item parameter he wants to use and the cut percentage. For example, the user can choose to choose the difficulty parameter with a percentage of cut of 20%, this means that the new benchmark will be composed of the 20% most difficult datasets from the original set.

To add new benchmarks in OpenML, the platform's Study class is used. This class allows users to create complete studies involving ML, ranging from the set of datasets used to which algorithms and forms of assessment and training were used [24].

§.§ Ranking of classifiers by the Glicko-2 system

Due to the fact that rating systems are commonly used in competitions, to apply the Glicko-2 [Glickman, 2012] system to evaluate the classifiers, it was necessary to simulate a competition between them. The simulated competition is an round-robin tournament, where each classifier will face each other and at the end of the competition will create a ranking with the rating of the models.

The competition works like this: each dataset is seen as a classification period in the Glicko-2 system, so that all classifiers face off in each dataset. To define the winner of each contest, the True-Score values obtained by the models facing each other are used. This happens as follows: if the True-Score value is greater than the opponent's, it is counted as a victory; if the value is lower than that of the opponent, it is counted as a defeat; and if the values are equal, then it sets up a tie between the models. In addition, the Glicko system asks the result of a match to assign a score to the opponents. For this, the scoring system applied in official Chess competitions was used, where victory counts as 1 point, defeat as 0 point and draw counts as 0.5 point.

Thus, after each dataset the rating, RD and volatility values of the classifiers are updated and used as the initial value for the next dataset. Once all datasets are finalized, the final rating values are used to create the final ranking that will be used to evaluate the models.

§.§ OpenML-CC18 datasets

OpenML-CC18 was chosen to be the case study of this work, which has as one of the main objectives to evaluate benchmarks through the IRT lens, in order to give greater reliability in the use of this benchmark. This section will present the datasets that were selected from OpenML-CC18 to be evaluated using the decodIRT tool.

Despite having 72 datasets, only 60 were used in this work. This was for two main reasons:

* The size of the datasets, where 11 have more than 30,000 instances, were soon considered too large and would take a long time to run all decodIRT models;

* Could not generate item parameters for dataset “Pc4”. R's Ltm package could not converge even using just under 500 test instances.

Despite this, the final amount of datasets used still corresponds to 83.34% of the original benchmark. All datasets evaluated are from tabular data and the characterization of the benchmark will be further explored in the next sections.

§.§ Assessment of innate ability

Given the definition of innate ability explained in the previous section, it is understood that its assessment can be done as follows:

* A benchmark composed of several datasets with different characteristics is defined. The chosen benchmark is OpenML-CC18 itself;

* A pool of classifiers composed of algorithms from the same family or from different families is assembled, always keeping the same configuration for each model. For this, the same set of real classifiers as decodIRT will be used;

* Test the different algorithms on the benchmark datasets. Step already performed by decodIRT;

* Different subsets of benchmarks are assembled from the original set. The subsets are assembled from the IRT estimators with decodIRT;

* For each subset of the benchmark, the rating ranking will be generated by the Glicko-2 system;

* The model with the best and most consistent performance is chosen as the one with the best innate ability.

§ RESULTS AND DISCUSSION

The evaluation of the OpenML-CC18 benchmark through the IRT lens was done around the discrimination and difficulty parameters. It is understood that these parameters are directly linked to the data, in comparison with the guessing parameter that is more linked to the performance of the respondents. The objective then is to evaluate the discriminatory power along with the difficulty of the datasets and later use them to evaluate the models performance [All classification results can be obtained at <https://osf.io/wvptb/files/>].

§.§ Decoding OpenML-CC18 Benchmark

It shows the percentages of difficult and very discriminative instances arranged in ascending order. There is a certain percentage of discrimination and a percentage of difficulty that are in the same position on the X axis do not necessarily correspond to the same dataset. “tic-tac-toe”, “credit-approval” and “optdigits” are respectively the datsets with the most difficult instances. While “banknote-authentication”, “analcatdata_authorship” and “texture” are the most discriminative.

When looking at Figure <ref>, it is possible to notice an inversion relationship between the parameters of difficulty and discrimination. So, the rankings generated by the two parameters reveal that the most discriminating datasets are also the least difficult and vice versa [Parameter rankings can be accessed at: <https://osf.io/jpygd/>]. This relationship is consistent with what is expected by the IRT, where it is normal that the easiest instances are good to differentiate the good from the bad classifiers, as it is thought that the more skilled classifiers will hit the easiest instances while the less skillful ones can make mistakes. Through this, it is possible to affirm that the more difficult datasets are not adequate to separate the good and bad classifiers, despite being more challenging. Meanwhile, the easiest datasets are not suitable for testing the classification power of algorithms, but it allows to differentiate the best from the worst well.

Histogram of the number of datasets by the percentage of discriminative and difficult instances.

Among the evaluated datasets, only 7 are really challenging and have more than 50% of difficult instances. While 49 of the total have less than 27% of difficult instances. This means that only 11.67% of the total evaluated datasets are hard while 81.67% have more than 70% easy instances. Therefore, the OpenML-CC18 benchmark should be used with caution and taking into account the purpose of its use. Figure <ref> reinforces the high discriminatory capacity of the benchmark, where only 1/4 of the datasets have less than 50% of low discriminatory instances and more than half of the total have at least 80% of highly discriminatory instances.

From this, it is possible to infer that OpenML-CC18 is not considered as challenging as expected, but it has many appropriate datasets to differentiate the good and bad classifiers. In addition, item parameters allow the benchmark to be chosen more specifically. For example, if the objective is solely to test the algorithms classification power, only the most difficult datasets can be used, disregarding testing with the entire benchmark.

§.§ Classifiers performance on OpenML-CC18

Sum of all True-Scores obtained by the classifiers.

When taking into account only True-Score values
obtained by the classifiers it is already possible to notice a pattern in their respective performances and create a ranking (see Figure <ref>). It is observed that the artificial classifiers assumed extreme positions in the ranking, as expected. For real classifiers, MLP takes the lead, but with a True-Score value very close to Random Forest. Although classifications similar to this one are the most common, in some specific cases the position of the classifiers is inverted and the worst models have the highest True-Score values, as can be seen in Figure <ref>.

The True-Score values obtained for the “jm1” dataset.

This situation can happen due to the occurrence of many instances with negative values of discrimination. As they are not expected by the IRT, negative values usually mean that there is something wrong with the item itself. For psychometric tests, this could mean a poorly formulated and ambiguous question, for example. When placing this concept in the ML field, negative discrimination may indicate some inconsistency in the instance, such as noise or outlier. Therefore, it can be inferred that datasets with many instances with negative discrimination may not be suitable for the formulation of a good benchmark. A possible future work would be to carefully analyze whether dataset characteristics are linked to these situations and how this can affect the performance of models. OpenML already has an extensive set of metadata about its datasets that can be used for this purpose.

§.§ Classifiers performance by Glicko-2

Although the True-Score values obtained already make it possible to observe and evaluate the performance of the classifiers, there is still a large amount of data to be evaluated individually before being able to point out which algorithm has the best innate ability. Therefore, the Glicko-2 system was applied in order to summarize the generated data and identify the “strongest” algorithm. Table <ref> presents the final rating ranking that was obtained.

Classifier rating ranking.

Classifier     Rating     RD     Volatility

optimal                                     1732.56     33.25     0.0603

MLP                             1718.65     31.20     0.0617

RandomForest                    1626.60     30.33     0.0606

RandomForest(5_trees)     1606.69     30.16     0.0621

RandomForest(3_trees)     1575.26     30.41     0.0646

DecisionTree                  1571.46     31.16     0.0674

SVM                                       1569.48     32.76     0.0772

KNeighbors(3)                   1554.15     30.74     0.0646

GaussianNB                                1530.86     31.25     0.0683

KNeighbors(2)                   1528.41     30.40     0.0638

KNeighbors(5)                   1526.10     30.27     0.0630

BernoulliNB                               1494.87     32.64     0.0770

KNeighbors(8)                   1457.78     30.25     0.0638

minority                               1423.01     30.66     0.0631

rand2                                     1374.78     30.27     0.0605

rand3                                     1337.27     30.95     0.0600

rand1                                     1326.38     31.42     0.0610

majority                               1301.08     31.74     0.0666

pessimal                                   1270.46     31.74     0.0603

As in the True-Score ranking (see Figure <ref>) the position of the artificial classifiers is as expected. Optimal leads while the other artificial classifiers have ratings lower than all real classifiers, where it is also the MLP that has the highest rating among the real ones. However, MLP is closer to Optimal's rating than RF is in third place. This situation differs from the True-Score ranking and from what was expected, as it was thought that Optimal would have a rating much higher than the others.

Despite the proximity of the rating of the MLP and Optimal be surprising, the strength of the MLP can be confirmed by observing the low volatility value. Overall, volatility is low for all classifiers, with caveats for SVM and Naive Bayes Bernoulli which have the highest volatility values respectively. This means that SVM and NB Bernoulli have the least reliable rating values of all, so they are more likely to vary widely within their respective RD ranges.

Furthermore, if you consider a high fluctuation in ratings within their RD ranges, the final ranking position may change sharply. For example, considering the largest negative change in MLP's RD, its new rating will be 1656.25. This would allow raters up to 4th place to be able to outperform the MLP should their ratings fluctuate as much upward as possible. However, for raters from 5th position onwards, no model could reach the MLP, even with the maximum fluctuation of their ratings as well.

Therefore, it is understood that there are groups of classifiers that have equivalent strength, where among the first three real classifiers, it is not possible to say precisely which one is the strongest in the challenge proposed by OpenML-CC18. However, this situation also allows us to assume that the innate ability of MLP is better than that of algorithms below 4th position. Because, given the fact that tests were performed with several different datasets that have different IRT estimators, always keeping the model configuration, it can be assumed that the results obtained reflect the innate ability of the learning algorithms.

It is also important to point out that the optimization of models can have a fine-tuning effect on the decision limits of the classifiers, resulting in better performance in the most difficult datasets. However, this approach would not allow for a clean analysis of the models' innate ability.

In order to provide greater credibility for the rating values that were generated, the Friedman test [Pereira et al., 2015] was performed. With this, the aim is to identify whether through the rating values, in fact, it is possible to differentiate the algorithms' innate ability. The Friedman test was calculated using only the rating values of the real classifiers, as they are the focus of the study. Its execution resulted in a p-value of approximately 9.36e-80. The low p-value obtained means that, in fact, different distributions of ratings were found, which allowed the execution of the Nemenyi test [Nemenyi, 1962]. The Nemenyi test is applied to identify which distributions differ from each other. Figure <ref> presents a Heatmap of the Nemenyi test.

Heatmap generated from the Nemenyi Test, using only the rating distributions of the real classifiers.

When analyzing the Heatmap, it is noted that the assumption of the existence of groups is proven. For, the first three classifiers that have the highest ratings also have high p-value. This means that the Nemenyi test cannot differentiate them. And although they have the best performers, they all have a p-value for at least one lower performer. Other classifiers also have a high p-value, even though they are from different algorithm families. Therefore, it is not evident that there is a clear separation of classifiers from different groups. And this leads to believe that, although the Friedman test indicates the existence of distinct groups of classifiers, these differences are not statistically significant to indicate with certainty which algorithm is more skillful.

Based on the above, a deeper analysis of the performance of the algorithms is necessary in order to define which in fact has the greatest innate classification ability. For this, different subsets of OpenML-CC18 were generated, chosen from the estimators of the IRT, to then choose the algorithm that presented the best innate ability. Furthermore, it will also be explored whether, within the generated benchmark subsets, there is one more efficient than the original.

§.§ Efficient benchmark by decodIRT

According to Smith and Martinez, 2014, not all instances of a dataset are equally beneficial for learning. This means that some instances can help more than others in the model induction process, and it is even possible to have instances that hinder learning. By expanding this idea to a higher level, it is possible to imagine that in a benchmark, not all datasets are equally beneficial for evaluating models.

From this, one can imagine that within a benchmark there may be a subset of datasets that is as good, or even better, than the original benchmark. Through this, it is also believed that decodIRT can help to find more efficient benchmarks. Because using the item parameters of the IRT it is possible to select the datasets that present the highest percentage of instances with high values for each one of the item parameters.

According to the results presented above, it is known that the datasets occupy opposite positions in the Discrimination and Difficulty rankings. Therefore, it is understood that to generate a more efficient subset of datasets it is necessary to take into account the values of both parameters.

To assess whether there is a subset of OpenML-CC18 that is equally good or better than the original, three cutoff percentages were defined - 30%, 50% and 70% - to select the most difficult and discriminating set of datasets using the decodeIRT. For example, for cutting 30%, it will be the most discriminating 15% datasets and the most difficult 15% datasets. The assessment of the quality of the subsets is performed by analyzing the Glicko rating rankings generated after each percentage cut.

Classifier rating ranking by benchmark subset 70%.

Classifier     Rating     RD     Volatility

optimal                                     1689.83     31.59     0.0601

RandomForest                             1672.18     30.67     0.0599

MLP                    1643.37     29.99     0.0604

RandomForest(5_trees)     1628.19     30.17     0.0605

RandomForest(3_trees)                                1621.54     30.17     0.0632

KNeighbors(8)                   1604.03     30.17     0.0629

KNeighbors(5)                                       1602.32     30.19     0.0638

KNeighbors(2)                  1580.18     29.90     0.0629

SVM                   1556.00     30.74     0.0677

GaussianNB                   1547.81     31.13     0.0684

DecisionTree                               1540.01     30.55     0.0661

BernoulliNB                   1526.66     31.45     0.0708

KNeighbors(3)     1519.88     29.81     0.0627

majority                               1364.58     30.59     0.0637

rand3                                     1323.52     30.45     0.0600

rand2                                     1315.71     30.42     0.0604

minority                                     1295.84     30.98     0.0611

pessimal                                   1291.72     31.00     0.0606

rand1                               1287.40     31.11     0.0607

By looking at the ratings generated by the 70% set (see Table <ref>). There are some variations in the positions of the real and artificial classifiers. In this new ranking, Random Forest took the lead over MLP, but it still has a rating value very close to the Optimal classifier, a situation that should not occur given the characteristics of Optimal. Another point to consider is the proximity of the rating values in the 70% set. Because, considering the maximum variation of the rantings in their respective RD intervals, the Optimal classifier can be reached by the KNeighbors(2) classifier, which is in the 8th position of the ranking. Such a condition makes it more difficult to try to separate the classifiers by their strength.

However, despite the proximity of ranking between the real classifiers and Optimal, the new benchmark set had a good positive point. The ranting difference between the last real classifier and the first classifier among the artificial ones that occupy the lower part of the table has increased significantly. The original benchmark ranking value (see Table <ref>) has increased from 34.77 to 155.3, which makes real classifiers unreachable for artificial classifiers. And this situation is in line with what was expected given the large differences in performance and probably occurs due to the greater discriminative capacity of the datasets.

For the set of 50%, the final performance of the classifiers and their ranking order is the closest to what was expected (see Table <ref>).

Classifier rating ranking by benchmark subset 50%.

Classifier     Rating     RD     Volatility

optimal                                     1724.29     32.78     0.0601

RandomForest                             1678.31     31.21     0.0600

MLP                    1655.45     30.52     0.0602

RandomForest(5_trees)     1644.98     30.84     0.0600

RandomForest(3_trees)                                1634.40     30.32     0.0618

KNeighbors(2)                  1620.72     30.33     0.0609

KNeighbors(8)                   1593.82     30.30     0.0621

KNeighbors(5)                                       1591.09     30.31     0.0628

KNeighbors(3)     1569.81     30.09     0.0603

BernoulliNB                   1558.71     31.32     0.0672

SVM                   1549.54     31.14     0.0676

DecisionTree                               1522.45     30.66     0.0646

GaussianNB                   1510.21     30.64     0.0635

majority                               1377.13     30.63     0.0623

rand3                                     1287.90     31.64     0.0600

rand1                               1276.45     31.75     0.0602

rand2                                     1274.34     31.41     0.0601

minority                                     1264.92     31.75     0.0601

pessimal                                   1244.41     32.16     0.0604

Despite the difference in some of the ranking positions. The positive points obtained in the cutoff of 70% were maintained in the new classification, as the big difference in ranting between artificial and real classifiers. Furthermore, the position of the artificial classifiers was as expected. Where, the Majority classifier is the one with the highest position, followed by the random three, the Minority and ending with the Pessimal.

Another positive point obtained was the increase in the maximum rating value and consequently the increase in the difference between Optimal and Random Forest, which remained as the best real classifier. Furthermore, it is possible to observe a grouping of classifiers that belong to the same model family. This situation is also consistent with the expected final result, as it is imagined that classifiers from the same family have a similar performance.

The more datasets are filtered for the most discriminating and difficult, the more evident the rating difference between real and artificial classifiers becomes, as can be seen in the ranking with 30% of the benchmark (see Table <ref>). Where the rating difference between Optimal and Random Forest exceeds 100 points.

Classifier rating ranking by benchmark subset 30%.

Classifier     Rating     RD     Volatility

optimal                                     1848.59     34.67     0.0616

RandomForest                             1734.63     32.64     0.0600

MLP                    1697.76     31.68     0.0601

RandomForest(5_trees)     1680.40     31.59     0.0600

KNeighbors(3)     1644.92     31.95     0.0601

KNeighbors(2)                  1624.71     31.73     0.0602

KNeighbors(5)                                       1606.95     31.25     0.0603

KNeighbors(8)                   1564.72     31.33     0.0604

RandomForest(3_trees)                                1560.64     30.96     0.0608

BernoulliNB                   1536.59     31.50     0.0631

GaussianNB                   1531.65     31.62     0.0637

DecisionTree                               1529.08     31.22     0.0620

SVM                   1524.85     31.29     0.0622

majority                               1345.71     31.77     0.0616

rand2                                     1336.60     32.52     0.0601

minority                                     1295.49     33.44     0.0600

rand3                                     1292.05     32.88     0.0601

rand1                               1254.62     34.00     0.0602

pessimal                                   1213.69     35.12     0.0603

However, it is noted that the position of the artificial classifiers has changed. The expected order that was reached by the ranking with a cut of 50% has changed and the artificial Minority classifier has come out of second to last place and has a rating value very close to the Majority classifier. This situation possibly occurs due to the smaller amount of datasets for evaluation and the lower discrimination power in half of the benchmark, as only 18 datasets are used in which 9 have low discrimination values because they are the most difficult. Such condition can be better observed in Table <ref>.

Note that the set of datasets resulting from the 30% cut has the lowest mean and highest standard deviation of Discrimination. And simultaneously presents the highest average percentage of Difficulty, but with the highest standard deviation as well. This results in an unbalanced benchmark set that may not be adequate to evaluate models well. As an example, we have the final position of Random Forest with three trees that appeared in the 5th position in the benchmarks with 100%, 70% and 50% of the total datasets, but which performed below the KNN algorithms for the set of 30%.

When analyzing Table <ref>, it can be seen that the subset of 50% is the one with the most balanced discrimination and difficulty values. Because despite not having the greatest discriminatory power in the average, its final value differs only about 5% from the highest average, which was reached with the original benchmark. In addition, it is the second highest average difficulty subset at 25.19%, which also makes it more challenging.

Comparison between the Discrimination and Difficulty percentages for each subset.

1l     2cDiscrimination        2cDifficulty

1l     Average      S. Deviation     Average     S. Deviation

30%      58.5%            41.91%                33%             33.71%

50%      62.06%           38.72%                25.19%          28.23%

70%      65.16%           35.34%                20.44%          25.26%

100%     67.13%           30.78%                15.93%          22.56%

The empirical analysis of the rating values and the percentages of discrimination and difficulty of each benchmark subset shows that the 50% cut generated the most efficient and balanced subset. This can also be confirmed by evaluating the variance and standard deviation of the RD and Volatility values generated by each rating ranking (see Table <ref>).

For the Glicko system, the lower the RD and Volatility values, the more reliable the ranking and rating values will be. By Table <ref>, it can be seen that the subset of 30% and the original benchmark present, respectively, the highest values of variance and standard deviation of RD. This means, that some classifiers have very high variation ranges, such as the artificial Optimal and Pessimal classifiers. Despite this, the 30% subset has the smallest volatility variations, revealing an imbalance in the subset.

Comparison between RD and Volatility values.

1l     2cRD        2cVolatility

1l     Variance     S. Deviation     Variance     S. Deviation

30%                     1.19          1.49              0.0000013     0.0011

50%                     0.53          0.73              0.0000057     0.0023

70%                     0.26          0.51              0.000010      0.0032

100%                    0.86          0.93              0.000025      0.0050

On the other hand, the subset generated by the 50% cut, again shows up as the most consistent. Because, it presents the second smallest variation of DR, behind only the subset of 70%. And it also has the second smallest volatility variation, just behind the 30% subset. This allows us to infer that the 50% subset generated by decodIRT would be a more efficient and adequate choice than the original OpenML-CC18 benchmark, according to the analysis of the item parameters and the Glicko system.

§.§ Innate ability evaluation

In addition to choosing a more efficient benchmark, creating the subsets allows for a deeper assessment of the models' innate ability. It is observed in the new rankings generated (see Tables <ref>, <ref> and <ref>) that Random Forest maintained the lead ahead of MLP. As the benchmarks subsets, the average difficulty of the benchmarks gradually increases (see Table <ref>). This suggests that RF is probably the most skillful algorithm, as it manages to keep the rating high even in the face of the most difficult datasets. Another point that corroborates the previous statement is volatility. In all benchmark scenarios analyzed (100%, 70%, 50% and 30%), Random Forest is the real classifier with the lowest volatility value, which means that its rating is the more accurate and reliable, so it is less susceptible to possible fluctuation.

The different scenarios also allow evaluating inverse cases, where the classifiers had a drop in performance. The main example is the SVM that dropped from 6th to last position among the real classifiers. Although surprising, the Glicko-2 system already pointed out this possibility, since the SVM had the highest volatility value recorded in all the rankings generated, 0.0772. This then means that SVM was the model with the least reliable rating value.

Although the results are not yet completely conclusive, it is noted that the search for the correct assessment of the algorithms innate ability is valid and that the combination of the use of IRT with the Glicko-2 system can serve as a correct format for this assessment.

§.§ OpenML-CC18 datasets characterization

Creating subsets of a benchmark is a very important task, since a smaller benchmark is computationally less costly. However, one should not only consider the computational cost for this task, it is important that the new benchmark generated is also able to maintain characteristics similar to the original.

To test if the set of 50% generated from the OpenML-CC18 benchmark has similar characterization. The following general dataset characteristics were compiled:

* Data types: if the dataset is composed only of features of numeric or categorical type or if the dataset is mixed (it has features of both types).

* Classification type: whether the dataset is binary or multiclass.

OpenML-CC18 characterization.

Binary     Multiclass     Total

Categorical                  6.66%          6.66%          13.33%

Numeric                    30%          38.33%          68.33%

Mixed                       11.66%          6.66%          18.33%

Total                       48.33%          51.66%          100%

Characterization of the 50% subset.

Binary     Multiclass     Total

Categorical                  6.66%          6.66%          13.33%

Numeric                    16.66%          46.66%          63.33%

Mixed                       13.33%          10%          23.33%

Total                       36.66%          63.33%          100%

When analyzing Tables <ref> and <ref>, it can be observed that the data types of the datasets are kept proportional in total between the two sets of benchmarks. Where, for the new benchmark, the total amount of numeric and mixed type datasets varies only 5% when compared to the values of the original benchmark.

The most significant change is in the type of dataset classification, where the original benchmark is more balanced, especially in terms of total values. While the new benchmark generated presents a higher total percentage of multiclass datasets, with the difference between the amount of binaries being 26.66%. However, this wide classification difference only appears when evaluating datasets of numeric data type. Because, for categorical datasets, the classification proportion is maintained and for mixed datasets the percentage variation is below 4% if compared to the original benchmark.

In addition to the general characterization, more specific characteristics of the datasets of each benchmark were also analyzed. For this purpose, 9 metadata were removed from OpenML for each dataset: number of classes, number of features, percentage of instances with missing values, percentage of categorical features, percentage of numerical features, percentage of binary features, percentage of majority and minority classes and the dimensionality. This data can be accessed in the supplementary material (link to supplementary material).

Dataset metadata analysis of each set was performed by calculating the mean, median and standard deviation. It was observed if these values suffered a lot of change when analyzed against the original benchmark and later on the subset generated by the 50% cut. Table <ref> brings the variation of values between sets in the mean.

Variation between the original and the new benchmark in percentage.

Avg. 100%     Avg. 50%

Nº of Features                          151.06      136.95

Nº of Classes                           6.83     5.25

Perc. Instances W. Mis. Val.     5.64     4.97

Perc. Symbolic Features                 27.24      25.77

Perc. Numeric Features                  72.75       74.22

Perc. Binary Features                   13.10      12.23

Perc. Majority Class                    38.78      46.76

Perc. Minority Class                    16.10      19.27

Dimensionality                          0.86      1.37

Among the analyzed metadata, it is observed that only the Number of Features and the Percentage of the Majority Class suffer considerable variation. Despite the high average number of features, a thorough analysis reveals a standard and median deviation of 335.66 and 25 for the original benchmark and 329.72 and 29 for the new benchmark. This reveals that only a few datasets from both sets have very high numbers of features and that overall there is not much variation, see values very close to the median.

The main change is in the percentage of appearance of the majority class, which undergoes a change of about 20% from the original average value to more in the new benchmark. This shows that the 50% subset has more datasets with unbalanced classes. Furthermore, a direct analysis of the metadata of each dataset revealed that the highest percentages of the majority class belong to the datasets classified as the most discriminating, i.e. they are the easiest.

It was initially thought that highly unbalanced datasets would be considered the most difficult, but the situation is the opposite. However, this situation is not far from general knowledge in ML. One possible explanation is that the IRT defined that models unable to generalize all classes in an unbalanced situation as bad, while models that do are considered good. This would explain the high discrimination values and remain consistent with the natural knowledge of the behavior of models in ML.

§ FINAL CONSIDERATIONS

This work explored the IRT for benchmark evaluation and ranking algorithms. ML benchmarks are commonly used to explore how far ML algorithms can go when dealing with datasets in order to keep the model as strong as possible. Although OpenML-CC18 is designed to be a gold standard, it should be used with caution. Of the 60 datasets evaluated, only 12% have instances that are considered difficult, while half of the benchmark presents 80% of the instances as being very discriminatory. This condition can be a great source for analyzing comparisons, but it is not useful for testing the ability of classifiers. The benchmark assessment methodology is provided and can be replicated by the decodIRT tool in an automated way. Although classifier skills are highlighted by IRT, there was also a problem with innate skill, whether it is possible to define the boundaries between the ML algorithm (by design) and training (optimization). The results of the IRT were exploited by rating systems as used to assess the strength of chess players to establish the ML winner and thereby provide an initial glimpse of a score for the innate ability of the algorithms.

In addition, decodIRT was used to explore whether there is a more efficient benchmark subset than the original and whether it could be obtained from the IRT estimators. After exploring different subsets, the subset consisting of 50% of the total datasets selected on the percentages of discrimination and difficulty was chosen. Because, not only does it allow an evaluation to be carried out with the characterization of the datasets very close to the original, but it also proved to be more suitable for evaluating and separating the strength of the models. Furthermore, the creation of the benchmark subsets allowed us to explore a little more the innate ability of the models. The final result pointed out that Random Forest is the classifier that has the greatest innate ability, making its choice preferable in relation to the evaluated models. Therefore, it was shown that the IRT can also be used for filtering and creating more efficient sets of benchmarks and, based on that, choosing more accurately which is the best classifier. So, in the dispute between data vs classifiers, the final result was a technical draw to decide who is the most important.

Machine learning is usually a constant process of improving data and models to obtain better results. This work can be seen in the same way, as it presents key points that need to be better explored and analyzed. Among them is the concept of difficult. It is known that the difficulty is subjective and strongly depends on the individual. The assertion that a given dataset is more difficult than another must be made with caution as well, as it is directly linked to the models that were used to estimate the difficulty.

Which leads to another key point, the number of classifiers used in the evaluation process is still small and does not encompass all algorithm families. Future work would be to expand the number of different models evaluated. Algorithm evaluation is also a key point that can be better explored by comparing the results obtained by the Glicko-2 system with the most used evaluation metrics.

§ ACKNOWLEDGEMENTS

We thank the Federal University of Pará (UFPA), the Graduate Program in Computer Science (PPGCC) and the Vale Technological Institute (ITV) for supporting development and research. This work had financial support from the Brazilian agency CNPq (Conselho Nacional de Desenvolvimento Científico e Tecnológico). This work was supported by Vale (Genomics Biodiversity project, Grant No. RBRS000603.85) to Ronnie Alves. The funders had no role in the study design, data collection and interpretation, or the decision to submit the work for publication.

§ FLOATS

Figures may be included using the command, in
combination with or without its several options to further control
graphic.  is provided by graphic[s,x].sty
which is part of any standard  distribution.
graphicx.sty is loaded by default.  accepts figures in
the postscript format while pdf accepts *.pdf,
*.mps (metapost), *.jpg and *.png formats.
pdf does not accept graphic files in the postscript format.

The evanescent light - $1S$ quadrupole coupling
($g_{1,l}$) scaled to the bulk exciton-photon coupling
($g_{1,2}$). The size parameter $kr_{0}$ is denoted as $x$ and
the is placed directly on the cuprous oxide sample ($\delta
r=0$, See also Table <ref>).

The  environment is handy for marking up tabular
material. If users want to use multirow.sty,
array.sty, etc., to fine control/enhance the tables, they
are welcome to load any package of their choice and
cas-dc.cls will work in combination with all loaded

This is a test caption. This is a test caption. This is a test
caption. This is a test caption.

Col 1     Col 2     Col 3     Col4

12345     12345     123     12345

12345     12345     123     12345

12345     12345     123     12345

12345     12345     123     12345

12345     12345     123     12345

§ THEOREM AND THEOREM LIKE ENVIRONMENTS

cas-dc.cls provides a few shortcuts to format theorems and
theorem-like environments with ease. In all commands the options that
are used with the  command will work exactly in the same
manner. cas-dc.cls provides three commands to format theorem or
theorem-like environments:

The  command formats a
theorem in 's default style with italicized font, bold font
for theorem heading and theorem number at the right hand side of the
theorem heading.  It also optionally accepts an argument which
will be printed as an extra heading in parentheses.

For system (8), consensus can be achieved with
$\|T_{\omega z}$ ...

\begin{eqnarray}\label{10}
\end{eqnarray}

The  command is the same in
all respects as its  counterpart except that
the font shape is roman instead of italic.  Both
and  commands
automatically define counters for the environments defined.

The  command defines proof environments with
upright font shape.  No counters are defined.

[Adedoyin et al., 2013]
authorAdedoyin, O., authorMokobi, T., et al.,
titleUsing irt psychometric analysis in examining the
quality of junior certificate mathematics multiple choice examination test
journalInternational Journal of Asian Social Science
volume3, pages992–1011.

[de Andrade et al., 2000]
authorde Andrade, D.F., authorTavares, H.R.,
authorda Cunha Valle, R., year2000.
titleTeoria da resposta ao item: conceitos e
journalABE, Sao Paulo .

[Bellemare et al., 2013]
authorBellemare, M.G., authorNaddaf, Y.,
authorVeness, J., authorBowling, M.,
titleThe arcade learning environment: An evaluation
platform for general agents.
journalJournal of Artificial Intelligence Research
volume47, pages253–279.

[Birnbaum et al., 1968]
authorBirnbaum, A., authorLord, F.,
authorNovick, M., year1968.
titleStatistical theories of mental test scores.
journalSome latent trait models and their use in inferring
an examinee’s ability. Addison-Wesley, Reading, MA .

[Bischl et al., 2017]
authorBischl, B., authorCasalicchio, G.,
authorFeurer, M., authorHutter, F.,
authorLang, M., authorMantovani, R.G.,
authorvan Rijn, J.N., authorVanschoren, J.,
titleOpenml benchmarking suites and the openml100.
journalstat volume1050, pages11.

[Cardoso et al., 2020]
authorCardoso, L.F., authorSantos, V.C.,
authorFrancês, R.S.K., authorPrudêncio, R.B.,
authorAlves, R.C., year2020.
titleDecoding machine learning benchmarks, in:
booktitleBrazilian Conference on Intelligent Systems,
organizationSpringer. pp. pages412–425.

[Domingos, 2012]
authorDomingos, P., year2012.
titleA few useful things to know about machine learning.
journalCommunications of the ACM volume55,

[Elo, 1978]
authorElo, A.E., year1978.
titleThe rating of chessplayers, past and present.
publisherArco Pub.

[9]
authorFacebook, .
titleRethinking ai benchmarking.

[Ferri et al., 2009]
authorFerri, C., authorHernández-Orallo, J.,
authorModroiu, R., year2009.
titleAn experimental comparison of performance measures
for classification.
journalPattern Recognition Letters volume30,

[Gautier, 2008]
authorGautier, L., year2008.
titlerpy2: A simple and efficient access to r from
journalURL http://rpy. sourceforge. net/rpy2. html
volume3, pages1.

[Glickman, 2012]
authorGlickman, M.E., year2012.
titleExample of the glicko-2 system.
journalBoston University , pages1–6.

[Kotsiantis et al., 2007]
authorKotsiantis, S.B., authorZaharakis, I.,
authorPintelas, P., year2007.
titleSupervised machine learning: A review of
classification techniques.
journalEmerging artificial intelligence applications in
computer engineering volume160, pages3–24.

[Kubat, 2017]
authorKubat, M., year2017.
titleAn introduction to machine learning.

[Lord and Wingersky, 1984]
authorLord, F.M., authorWingersky, M.S.,
titleComparison of irt true-score and equipercentile
observed-score" equatings".
journalApplied Psychological Measurement
volume8, pages453–461.

[Martinez-Plumed and Hernandez-Orallo, 2018]
authorMartinez-Plumed, F., authorHernandez-Orallo, J.,
titleDual indicators to analyze ai benchmarks: Difficulty,
discrimination, ability, and generality.
journalIEEE Transactions on Games volume12,

[Martínez-Plumed et al., 2016]
authorMartínez-Plumed, F., authorPrudêncio,
R.B., authorMartínez-Usó, A.,
authorHernández-Orallo, J., year2016.
titleMaking sense of item response theory in machine
learning, in: booktitleProceedings of the Twenty-second European
Conference on Artificial Intelligence, pp. pages1140–1148.

[Martínez-Plumed et al., 2019]
authorMartínez-Plumed, F., authorPrudêncio,
R.B., authorMartínez-Usó, A.,
authorHernández-Orallo, J., year2019.
titleItem response theory in ai: Analysing machine
learning classifiers at the instance level.
journalArtificial Intelligence volume271,

[Meneghetti and Junior, 2017]
authorMeneghetti, D.D.R., authorJunior, P.T.A.,
titleApplication and simulation of computerized adaptive
tests through the package catsim.
journalarXiv preprint arXiv:1707.03012 .

[Monard and Baranauskas, 2003]
authorMonard, M.C., authorBaranauskas, J.A.,
titleConceitos sobre aprendizado de máquina.
journalSistemas inteligentes-Fundamentos e
aplicações volume1, pages32.

[Nemenyi, 1962]
authorNemenyi, P., year1962.
titleDistribution-free multiple comparisons, in:
booktitleBiometrics, organizationInternational
Biometric Soc 1441 I ST, NW, SUITE 700, WASHINGTON, DC 20005-2210. p.

[Nie et al., 2019]
authorNie, Y., authorWilliams, A.,
authorDinan, E., authorBansal, M.,
authorWeston, J., authorKiela, D.,
titleAdversarial nli: A new benchmark for natural language
journalarXiv preprint arXiv:1910.14599 .

[23]
authorOpenML, a.
titleAn open, collaborative, frictionless, automated
machine learning environment.

[24]
authorOpenML, b.
titleAn open, collaborative, frictionless, automated
machine learning environment.

[Pedregosa et al., 2011]
authorPedregosa, F., authorVaroquaux, G.,
authorGramfort, A., authorMichel, V.,
authorThirion, B., authorGrisel, O.,
authorBlondel, M., authorPrettenhofer, P.,
authorWeiss, R., authorDubourg, V., et al.,
titleScikit-learn: Machine learning in python.
journalthe Journal of machine Learning research
volume12, pages2825–2830.

[Pereira et al., 2015]
authorPereira, D.G., authorAfonso, A.,
authorMedeiros, F.M., year2015.
titleOverview of friedman’s test and post-hoc analysis.
journalCommunications in Statistics-Simulation and
Computation volume44, pages2636–2653.

[Perez-Liebana et al., 2015]
authorPerez-Liebana, D., authorSamothrakis, S.,
authorTogelius, J., authorSchaul, T.,
authorLucas, S.M., authorCouëtoux, A.,
authorLee, J., authorLim, C.U.,
authorThompson, T., year2015.
titleThe 2014 general video game playing competition.
journalIEEE Transactions on Computational Intelligence and
AI in Games volume8, pages229–243.

[Prudêncio et al., 2015]
authorPrudêncio, R.B., authorHernández-Orallo,
J., authorMartınez-Usó, A., year2015.
titleAnalysis of instance hardness in machine learning
using item response theory, in: booktitleSecond International
Workshop on Learning over Multiple Contexts in ECML 2015. Porto, Portugal, 11
September 2015.

[Rizopoulos, 2006]
authorRizopoulos, D., year2006.
titleltm: An r package for latent variable modeling and
item response theory analyses.
journalJournal of statistical software
volume17, pages1–25.

[Samothrakis et al., 2014]
authorSamothrakis, S., authorPerez, D.,
authorLucas, S.M., authorRohlfshagen, P.,
titlePredicting dominance rankings for score-based games.
journalIEEE Transactions on Computational Intelligence and
AI in Games volume8, pages1–12.

[Smith and Martinez, 2014]
authorSmith, M.R., authorMartinez, T.,
titleReducing the effects of detrimental instances, in:
booktitle2014 13th International Conference on Machine Learning
and Applications, organizationIEEE. pp.

[Song and Flach, 2021]
authorSong, H., authorFlach, P., year2021.
titleEfficient and robust model benchmarks with item
response theory and adaptive testing.
journalInternational Journal of Interactive Multimedia &
Artificial Intelligence volume6.

[Vanschoren et al., 2014]
authorVanschoren, J., authorVan Rijn, J.N.,
authorBischl, B., authorTorgo, L.,
titleOpenml: networked science in machine learning.
journalACM SIGKDD Explorations Newsletter
volume15, pages49–60.

[Veček et al., 2014]
authorVeček, N., authorMernik, M.,
authorČrepinšek, M., year2014.
titleA chess rating system for evolutionary algorithms: A
new method for the comparison and ranking of evolutionary algorithms.
journalInformation Sciences volume277,