# Training Green AI Models Using Elite Samples

Mohammed Alswaitti , Roberto Verdecchia, Grégoire Danoy , Pascal Bouvry and
Johnatan Pecero M. Alswaitti is with the Interdisciplinary Centre for
Security, Reliability and Trust (SnT), University of Luxembourg, L-4364 Esch-
sur-Alzette, Luxembourg<EMAIL_ADDRESS>Verdecchia is
with the Software Technology Laboratory (STLab) of the School of Engineering,
University of Florence, 50121 Firenze FI, Italy. E-mail
<EMAIL_ADDRESS>and P.Bouvry are with the Department of
Computer Science, Faculty of Science, Technology and Medicine, and SnT,
University of Luxembourg, L-4364 Esch-sur-Alzette, Luxembourg.
E-mail:{gregoire.danoy<EMAIL_ADDRESS>is with the Department
of Computer Science, Faculty of Science, Technology and Medicine, University
of Luxembourg, L-4364 Esch-sur-Alzette, Luxembourg. E-mail:
<EMAIL_ADDRESS>
###### Abstract

The substantial increase in AI model training has considerable environmental
implications, mandating more energy-efficient and sustainable AI practices. On
one hand, data-centric approaches show great potential towards training
energy-efficient AI models. On the other hand, instance selection methods
demonstrate the capability of training AI models with minimised training sets
and negligible performance degradation. Despite the growing interest in both
topics, the impact of data-centric training set selection on energy efficiency
remains to date unexplored.

This paper presents an evolutionary-based sampling framework aimed at (i)
identifying elite training samples tailored for datasets and model pairs, (ii)
comparing model performance and energy efficiency gains against typical model
training practice, and (iii) investigating the feasibility of this framework
for fostering sustainable model training practices.

To evaluate the proposed framework, we conducted an empirical experiment
including 8 commonly used AI classification models and 25 publicly available
datasets. The results showcase that by considering 10% elite training samples,
the models’ performance can show a 50% improvement and remarkable energy
savings of 98% compared to the common training practice.

In essence, this study establishes a new benchmark for AI researchers and
practitioners interested in improving the environmental sustainability of AI
model training via data-centric approaches.

###### Index Terms:

AI Model Training, Data-centric, Green AI, Energy Efficiency, Instance
Selection, Evolutionary Algorithms.

##  I Introduction

Artificial intelligence (AI) technologies are continually evolving and
influencing various aspects of people’s lives. Some of the key ways in which
AI is becoming more and more essential include automation[1],
personalization[2], predictive analytics[3], safety and security[4],
accessibility[5], and others. These substantial tasks, which raise humankind’s
standard of living, are nowadays driven by thoroughly trained data-hungry AI
models. Therefore, to ensure that no opportunity is missed, gathering data
inexorably became the gold of a new era in which the competition in the
business market is embodied in collecting as much data as possible[6]. The
quantity of data, collected either privately or publicly accessible, enables
AI community researchers to train a myriad of different AI models, tailored at
an increasing pace for various purposes. Resources such as data availability,
hardware capability, and open-source software enabled an exponential growth of
AI communities, which in turn led to a duplication of computational resources
needed for AI training every 3.4 months since 2012[7]. The explosion in
computational resources required by AI is due both to the training required by
a few remarkably large models (owned by large companies), characterized by
billions of training examples and parameters, and millions of less complex
models (used by researchers and smaller companies) of varying training data
and parameter sizes [6].

To date, several studies focused on presenting efficient algorithms aimed at
optimizing the computational resources required for model training while
maintaining their accuracy at comparable levels [8]. Similarly, other research
focused on using specialized hardware designed to be more power-efficient,
such as low-power CPUs, custom-designed chips, and accelerators [9]. Such
contributions were recently challenged by the overwhelming growth of AI
systems, that led to concerning amounts of greenhouse gas emissions,
accompanied by high operational costs for organizations [10]. By considering
the state of practice, we note that the pursuit of squeezing every bit of
performance to build high-quality AI models comes with significant
environmental implications in terms of energy consumption and carbon
footprint. As documented in the literature, training an AI model can emit as
much carbon as five cars in their lifetimes [11], an equivalent to 125 round-
trip flights between New York and Beijing [12], more than 626,000 pounds of
carbon dioxide [13, 14], and energy consumption equivalent to a trans-American
ﬂight [15].

The growing environmental concerns triggered the AI community to review their
research agenda to include the Green AI term, first defined as “AI research
that yields novel results while taking into account the computational cost”
[16]. The seminal study on Green AI [16] raised concerns regarding the current
trend of favouring accuracy over efficiency in the current practice and raised
a call for Green AI action which requires a joint and collaborative effort to
holistically target the environmental sustainability of AI models (e.g., by
considering data used, model types, experimentation environments, training,
and hyperparameter tuning). As a further outlook, the research concluded that
more data does not necessarily yield more efficient AI model learning
processes, and encouraged the inclusion of the model energy consumption/carbon
footprint as an evaluation metric [16].

According to the latest systematic review on Green AI [17], increasing
research endeavours have emerged in the past few years to tackle AI
environmental sustainability. The majority of Green AI studies focused on
model-related topics, such as monitoring, hyperparameter-tuning, benchmarking,
and deployment. On the contrary, far less attention was given to data-centric
approaches and performance/efficiency trade-offs, while data quality showcased
promising potential in Green AI. One goal of utilizing high-quality data for
training AI models is reaching the minimal training data samples required to
reach a sufficient model performance. Some studies have proved the existence
of training subsets that lead to more energy-efficient yet equally accurate
models [18]. However, the selection and combination of training samples were
only picked at random from a dataset, without any consideration regarding
which specific data samples were responsible for the improvements.

Finding representative training samples for an AI model is known in the
literature with different terminologies, e.g., Instance Selection (IS),
prototype selection, and sampling (which are used interchangeably in this
paper for reducing the number of data observations). Determining these
samples, referred to as elite training samples in this paper, to use them
regularly for training AI models brings many benefits. Among other benefits,
utilizing elite training samples ensures results reproducibility, cuts off
underreported experimentations during model construction, and reduces the cost
of model training in terms of available resources and energy
consumption/carbon footprint. However, the process of finding elite training
samples in a given dataset is both energy and computationally-intensive,
especially in large-scale problems, as it is considered to be an NP-hard
problem that requires optimisation algorithms to be solved.

Evolutionary Algorithms (EAs) are generic population-based metaheuristic
optimisation search algorithms that have been extensively used to solve a wide
range of complex problems [19]. The Differential Evolution (DE) algorithm [20]
has been selected as a prototype in this study due to its simplicity and
flexibility to match the proposed problem formulation. There are several EA
variations and various additional optimisation methods in the literature that
can be customized to address such a problem, each with varying performance
gains and applications. However, the focus of this study is not to delve into
the customization of EA variations. Instead, our study centres on customizing
the DE algorithm as an exemplar EA variant to solve the IS/sampling problem
from a data-centric perspective. More specifically, this study centres on the
task of identifying elite training samples for a given dataset and machine
learning model (as a branch of AI) pairs. the goal of this study is to explore
how elite training samples impact model performance and contribute to reducing
the energy consumed by AI model training. Furthermore, this research aims to
validate the worthiness of the gains in terms of performance and energy
consumption of the whole framework. Hence, the objectives of this study are as
follows:

  * •

To design an elite training sampling framework based on the DE algorithm for
generic data and AI models.

  * •

To investigate the effect of elite training samples on the performance and
energy consumption of AI models.

  * •

To explore the viability of the elite training sampling framework in reducing
AI energy consumption.

The remainder of this paper is structured as follows. Section II presents the
related work to the IS problem from different perspectives. Section III
details the overall approach and the framework design. Section IV describes
the experimentation setup, results analysis, and the feasibility of the
proposed framework. Discussions with future research directions and
conclusions are listed in Section V and Section VI, respectively. Finally, it
is worth pointing out that the detailed experimental results, including the
elite training sample indices for the dataset and model pairs to ensure both
results’ reproducibility and further improvements, are available at
https://github.com/Sustainable-AI-Training/Elite-Training-Samples/.

##  II Related Work

This section presents a background on the IS problem, a taxonomy of the
existing IS techniques in the literature, and the research gaps that motivated
this study.

###  II-A Instance Selection

Reducing data complexity can take two approaches: reducing the number of
features describing each data point or reducing the number of data
observations themselves. Researchers in this context have been challenged to
propose data-reduction techniques that have minimal impact on AI model
inference. One commonly used method for data reduction is referred to as IS,
sampling, or prototype selection. This common technique is considered a data
pre-processing method to reduce the size of the training data for an AI model
while preserving the models’ performance.

IS methods aim to find a minimized training set that is expected to offer a
compromise between reducing the size of the original training set and
maintaining model performance, especially in the case of supervised machine
learning classifiers. This is achieved by selecting the most informative or
representative samples from the dataset [21]. A general description of the
problem is as follows: a dataset $X$ is usually composed of a number $n$ of
$x$ instances or samples that can be described as
${x}_{s}=\left(x_{s1},x_{s2},\dots,x_{sd},x_{sl}\right):s=1,\dots,n$ with a
$d$-dimensional space and class $c$ given by $x_{sl}$ attribute in which
$x_{sj}$ is the value of the $j_{th}$ feature of the ${x}_{th}$ sample. During
the AI model learning and evaluation processes, $X$ is divided into a training
set $TR$ and a testing set $TS$. The goal of IS methods is to find a subset of
selected samples (elite training samples in this study) $E\subseteq TR$ that
is used for training a classifier without a classification accuracy decline
over the testing samples ${TS}$.

Traditional IS methods adopted different data reduction mechanisms such as
eliminating or including instances that degrade classifier performance [22,
23, 24], choosing the best or most influential instances [25, 26], employing
density-based concepts to select instances [27, 28, 29], or proposing centroid
and cluster-based instance selection methods [30, 31]. Several taxonomies of
the IS techniques and categorisations were reported in the literature [32, 33,
34, 35] and their importance to AI models in improving the classification
performance and reducing both the training time and storage requirements were
highlighted. Despite that IS is not new to the AI field, it is a crucial
technique and point of interest in recent research to improve the model
performance, reduce training time, and curb the scaling-up problem when
training AI models. However, IS techniques can be computationally expensive
and often lead to model overfitting [32].

Other evolutionary-based methods that replicate natural evolution were
presented as alternatives to traditional IS methods in the literature [35,
36]. Based on the reported research outcomes, evolutionary-based IS methods
proved their comparable performance against traditional IS methods and gained
popularity due to their implementation flexibility and heuristic search
merits. In the next section, we therefore provide a list of the literature
focusing on evolutionary-based IS methods, which will be the focal point of
interest of this study.

###  II-B Evolutionary-based IS techniques

Evolutionary algorithms proved their efficiency in solving a wide range of
real-world optimisation problems [37]. Researchers continuously propose new or
modified variants of the optimisers and evaluate their performance in terms of
the fitness function, convergence speed and computational complexity on
benchmarking problems. Recently, as the research branch of low carbon and
energy-efficient computation has emerged [38], researchers applied
optimisation algorithms to solve several domain problems in an energy-
efficient manner such as resource allocation, scheduling, and communications
[39, 40, 41, 42].

Instance selection can be formulated as a search or optimisation problem where
EAs can be applied. When using EAs in IS, the goal is to find a subset of
instances that maximizes the performance of an AI model. The EA starts with a
population of randomly generated subsets of the dataset. The fitness of each
subset is evaluated by training a machine learning model on the subset and
then evaluating the model’s performance on a held-out test set. The fittest
subsets are then selected to form the next population. This process is
repeated until a stopping criterion is met. Researchers in the literature
customised EAs and evaluated their effectiveness as an IS technique. The
empirical studies of Cano et al. [43, 44] showed the great potential of EAs as
an IS over traditional methods in terms of reduction rates, model accuracy,
and ease of interpretation. Other proposals employed EAs to address the
scaling-up problem of model training over large-sized datasets [45, 46, 47].
Additionally, different methods showed the potential of IS-based EAs in
reducing the model sensitivity to noisy data [48], class imbalance problems
[49, 50, 51], and model training computational cost [52, 53]. Table I lists
the summary of the proposed IS-based EAs available in the literature, by
providing more details on each proposal method and features, the utilized EA,
and the targeted AI model in the study.

Despite the numerous IS-based EAs proposed in the literature, only a limited
subset (as detailed in the following section) has incorporated energy
consumption and carbon footprint as performance metrics in their experimental
evaluations. Furthermore, no prior studies have accounted for the energy cost
associated with the IS technique itself, which is an integral component of the
model training process and is typically computationally intensive. Therefore,
this research represents a notable contribution by presenting IS methodologies
through a Green AI scope and assessing their potential to reduce energy
consumption during AI model training, encompassing the computational cost of
the IS process.

TABLE I: Summary of evolutionary-based training set selection methods Reference |  Method and Features |  Core Representative EA |  AI Model
---|---|---|---
[43, 44] |  Empirical studies showed the great potential of EAs as an IS over the traditional methods in terms of reduction rates, model accuracy, and ease of interpretation. |  Genetic Algorithm (GA), CHC Adaptive Search Algorithm, Population-Based Incremental Learning (PBIL) |  Nearest neighbour (1-NN) and induction decision tree (C4.5)
[45, 46] |  Alleviating the scaling-up problem using stratification strategy with EAs as IS algorithms over large-size datasets. |  CHC |  1-NN and C4.5
[54] |  Evolutionary IS method to reduce learning time and improve the prediction performance of a stock market analysis model. |  GA |  Artiﬁcial Neural Network (ANN)
[49] |  GA was applied as a sampling technique to handle the imbalanced data effect on the classifier performance. |  GA |  C4.5
[47] |  A memetic algorithm for evolutionary prototype selection with a local search mechanism was proposed to alleviate the scaling-up problem. |  GA |  1-NN
[52] |  An instance selection cooperative coevolution approach coupled with a divide and conquer strategy was proposed to improve both model performance and computational cost. |  Cooperative coevolution |  1-NN
[48] |  DE-based optimisation method was proposed to optimise the positions of the selected prototypes before classifier evaluation to reduce its sensitivity to noisy data. |  DE |  1-NN
[50] |  An evolutionary training set selection called evolutionary sampling for training set selection is developed for class-imbalance undersampling. |  CHC |  C4.5

###  II-C IS from a Data-centric and Green AI perspective

AI models are known to be data-hungry, as the training phase requires a huge
amount of data and intensive computations to get more accurate. It was
observed through real-world AI development that more training data does not
necessarily yield better results. Even more, reducing the training set size
[51] and performing data selection throughout the learning process [55] can
speed up the training process and improve model generalisability. Yet, a
recent development in AI has increased the value of gathering and analysing
quality data and their effects on delivering better AI models. This recent
trend, following the Green AI vision [16], showcases the potential of reducing
the carbon footprint at the data level of the AI model lifecycle.

For instance, accuracy and efﬁciency trade-off insights related to Natural
Language Processing (NLP) tasks were proposed in [56]. The study analysed the
effect of the model size and the input sequence lengths on the trade-off
between accuracy, power, and speed in long-context transformer models during
fine-tuning and inference. It concludes that in summarization tasks, larger
models and shorter input sequence lengths lead to higher accuracy, slower
inference, and energy-efficient results. Conversely, in question-answering
tasks, longer input sequence lengths with simpler models lead to both higher
accuracy and efficiency.

In [57], a data detection and redundancy removal method for the training
dataset was proposed with a negligible effect on reasoning accuracy. The data
sample is considered to be redundant if its distance from the class mean value
is less than a threshold. As a result, a reduced version (by 20%) of the
Bonsai dataset (skin segmentation dataset) is recommended for future model
training to save time and energy consumption. However, the study lacks the
generality to be applied to several real-world data since the conducted
experiments targeted only two datasets with no feasible recommendations for
setting the threshold value. Moreover, solid computational costs in terms of
energy consumption or method complexity are missing.

Moving a step forward towards data-centric techniques, the effect of data
reduction (in both features and data points) on different AI models in terms
of model accuracy and energy consumption was proposed in [18]. The study
reported promising results where energy consumption was significantly reduced
with a slight or absence of accuracy decline through exclusively different
data manipulation setups. Furthermore, the study contrasted the models’ energy
consumption behaviours regardless of any data manipulation. Despite promoting
a data-centric approach for Green AI, the empirical study was conducted with
only one dataset and all the reduction setups either for the number of data
points or features were randomly set without a link between which data caused
the gained optimizations.

###  II-D The Motivations and Contributions of the Study

The results of the previous studies were an eye-opener to the existence of a
small subset of data (elite training samples), which can be effectively used
for training AI models. While existing data-centric studies in the literature
primarily focused on demonstrating the impact of varying data sizes on model
energy consumption and accuracy, none of these studies has established a
relationship that links the data subset to its model accuracy effect. In other
words, while they have reported training data reductions that reached
approximately 80%, they have not answered a crucial question: “Which 20% of
the data should be retained for model training to achieve the same carbon
footprint and model accuracy?”.

Identifying these elite training samples in a given dataset is an AI model-
dependent process that poses an optimisation problem in terms of the tradeoffs
between model accuracy and training data size. For instance, the size and the
selected elite training data that provide a predefined accuracy will differ
from one model to another. Additionally, the energy consumption/carbon
footprint of each model is unique due to the adopted learning mechanism. The
idea for this proposal stemmed from finding the following research gaps in IS
training set selection methods in the literature:

  * •

All previous IS-based EA proposals did not include the cost of the IS process
itself and were oriented to achieve exclusively better model efficiency.

  * •

Most of the proposed approaches evaluated the selected instances on a limited
number of classifiers in their studies or vice versa.

  * •

All studies used a fixed-size testing set (at a maximum of 30% testing split)
which might not reflect the real representativity of the selected instances to
the whole dataset. In other words, no guarantee of model generalisability.

  * •

Results reproducibility or an intermediate state of the selected instances for
further improvement is almost missing.

As a result of the literature search concerning IS methods for training set
selection to improve the performance of AI model, the contributions of this
work are to:

  1. 1.

The design of the first elite EA-based sampling framework to improve energy
efficiency with a new solution representation.

  2. 2.

An empirical investigation of the framework considering a wider collection of
AI models and datasets in terms of model performance metrics, testing
generalisability, and energy consumption

  3. 3.

The evaluation of the proposed framework viability and worthiness in reducing
energy consumption of AI models.

  4. 4.

Providing findings, future insights, and research directions towards
sustainable training of AI models.

It is worthwhile to highlight that in this study, the DE algorithm is selected
as an example of EAs, while endless other EAs exist as optimisers in the
literature and can be customised to the IS problem, with its conventional
implementation and settings. The goal of this research is to demonstrate a new
formulation of the IS problem and how the DE algorithm is customised to solve
it, rather than comparing the performance of EAs as IS methods. Moreover, the
emphasis is on the practicality of deploying IS-based EAs in terms of energy
consumption of the entire framework, rather than focusing solely on the
training phase or enhancing the EAs’ performance through any methods like
parameter tuning, enhanced local/global search techniques, and so on.

##  III Methodology

A dataset $X$, composed of a number $n$ of instances and a number $d$ of
features, is fed to the sampling framework with predefined parameters, namely,
bounds and sample size $q$. The bounds refer to the minimum (starting from
$1$) and the maximum (up to $n$) values associated with the instance indices
forming an input dataset. The sample size $q$ represents the desired training
sample size to be used for AI model training. Moreover, other parameters
concerning the optimiser (the DE in our case) should be set accordingly. Then,
the sampling process by the DE algorithm takes place, following the iterative
evolutionary concept of multiple phases, starting from the initialisation of
random solutions, mutation, and crossover to eventually find the best possible
$q$ sample indices (elite) based on an objective function. In this case, elite
indices are used to reference the associated data samples from $X$ to form
$TR$, which is expected to maximise the model performance. Finally, the
testing pool $TS=X-TR$ contains the remaining data instances to be partially
or fully used to evaluate the model performance. For example, if $n=100$, the
normal training practice is to set $q=70$ and consequently, $TR$ and $TS$
(equal to the whole testing pool in this scenario) are 70 and 30,
respectively. However, in case $q=30$, the training will be done using 30
samples and the testing pool will contain 70 samples. Hence, $TS$ can take a
value between 30 (the normal practice) and 70 (model generalisability). For
practicality, $q$ is set as a percentage ratio (70% or 30% in the previous
examples) concerning the variable input dataset size. It is noteworthy to
point out that, in the proposed approach, the CEILING or FLOOR functions are
used to obtain an integer number of training and testing samples once the
percentage notation is used. Figure 1 reports an overview of the proposed
methodology.

Figure 1: Overview of the proposed methodology.

The sampling framework computational complexity depends mainly on $n$ and $q$.
One of the objectives of this study, rather than observing the performance of
the model once trained with the elite samples, is to answer an energy
efficiency question “How much energy could be saved once an AI model trained
with the elite samples compared to the required energy to find the elite
samples itself?”. To answer this question, the consumed energy by the sampling
framework to find the elite samples is measured for each dataset and model
pair. Further, the consumed energy by training a model with the elite samples
is also recorded in order to be compared with the normal practice, where the
model is usually trained with 70% of the data.

###  III-A The Proposed DE Sampling Framework

In this study, the DE algorithm is selected and customised as a representative
of the EAs to solve the IS problem. Figure 2 illustrates the visual
implementation and the anatomy of each phase of the DE algorithm as a sampling
approach. The process starts with the random initialisation of a number ${p}$
of $P$ solutions (population size), where each solution $P=I_{p1},I_{p2},\dots
I_{pq}$ is represented by $q$ randomly generated indices $I=1,\dots,n$. Then,
the respective training samples from $X$ associated with each solution indices
are used as the training set for the classifier. Based on the initial
objective function values the ${Current}_{best~{}solution}$ is determined to
act as a baseline for the iterative following steps for each solution in the
population. Next, for each iteration over each solution $P$ in the population
$p$, three unique solutions $P_{{a}},P_{{b}},P_{{c}}$ are selected for the
mutation process. This stage creates a mutant solution $M=a+F(b-c)$ that
inherits randomly mixed merits from the three selected solutions with a
controllable mutation factor $F$. Since indices are expected to be integers, a
bound-checking mechanism is done to ensure all $M$ attribute values
$=1,\dots,n$. Some of the simple approaches are to replace the out-of-bounds
attribute value, in case of negative or values exceeding $n$, with a randomly
generated integer $\in\left[1,n\right]$, or by the minimum/maximum bound
values. Afterwards, the mutant solution $M$ and the current solution $P$
exchange some of their attribute pairs with controlled crossover probability
in the crossover process to form a candidate solution. The objective function
of the candidate solution is compared with the previously computed objective
function of the current solution and the fittest retains its place in the
population. The iterative process, through each iteration over the population,
outputs the ${Iteration}_{~{}best~{}solution}$, which is used to update the
$Best_{~{}solution}$ at the iteration level till the maximum iteration number
is reached. After execution termination, the $Best_{~{}solution}$ represents
the indices of the elite samples ${I}_{elite}$.

Figure 2: Detailed workflow of the proposed DE sampling framework.

##  IV Experimental Evaluation

This section describes the experimental design, the classification datasets
and machine learning models used, the phases of the experiments, the findings
and analyses, and the insights gained on the approach proposed in this
research.

###  IV-A Experimental setup

In this study, 8 widely used [58] machine learning models (classifiers) were
selected for analyses, namely, Adaptive Boost, Bagging, Decision Tree,
K-Nearest Neighbour (KNN), Logistic Regression, Naive Bayes, Random Forest,
and Support Vector Machine (SVM). The performance of the models was analyzed
over 25 datasets collected from the UCI Machine Learning Database
Repository111 https://archive.ics.uci.edu/datasets. and KEEL dataset
repository222http://www.keel.es. (Standard classification datasets). To give
an indicator of the generalisability and feasibility of the study findings,
dataset selection took into account varied characteristics such as the number
of observations (#Obs), attributes (#Atts.), classes (#Cl.), and user hits
(#Hits) till the date of the manuscript submission. The description of these
datasets is summarised in Table II.

TABLE II: Description of the Experimental Classification Datasets Dataset |  #Obs. |  #Atts. |  #Cl. |  # Hits |  Dataset |  #Obs. |  #Atts. |  #Cl. |  # Hits
---|---|---|---|---|---|---|---|---|---
Balance |  625 |  4 |  3 |  339692 |  Optdigits |  5620 |  64 |  10 |  393977
Bupa |  345 |  6 |  2 |  235474 |  Page-blocks |  5472 |  10 |  5 |  123945
Cleveland |  303 |  13 |  5 |  2221101 |  Penbased |  10992 |  16 |  10 |  284812
Coil2000 |  9822 |  85 |  2 |  197453 |  Pima |  768 |  8 |  2 |  387040
Contraceptive |  1473 |  9 |  3 |  257663 |  Satimage |  6435 |  36 |  7 |  167574
Glass |  214 |  9 |  7 |  476663 |  Spambase |  4597 |  57 |  2 |  753397
Haberman |  306 |  3 |  2 |  286064 |  Thyroid |  7200 |  21 |  3 |  356055
Heart |  270 |  13 |  2 |  2221151 |  Vehicle |  846 |  18 |  4 |  162750
Iris |  150 |  4 |  3 |  5388432 |  Wdbc |  569 |  30 |  2 |  2000358
Led7digit |  500 |  7 |  10 |  86507 |  Wine |  178 |  13 |  3 |  2194120
Letter |  20000 |  16 |  26 |  493600 |  Wisconsin |  699 |  9 |  2 |  289785
Mammographic |  961 |  5 |  2 |  224384 |  Yeast |  1484 |  8 |  10 |  376116
Monk-2 |  432 |  6 |  2 |  243363 |  |  |  |  |

All study experiments were conducted on a machine with Intel(R) Xeon(R) W-2225
CPU @ 4.10GHz equipped with 1 x Quadro RTX 6000 GPU, 128 GB RAM,
Linux-5.15.0-53-generic-x86_64-with-glibc2.35 OS, and Python
3.10333https://www.python.org/downloads/release/python-3100/.. To measure
energy consumption codecarbon444https://github.com/mlco2/codecarbon., a Python
library that seamlessly tracks the CO2 and energy consumption of an executable
code, was utilized. The reported energy consumption amount represents the
total CPU, GPU and RAM energy consumption during the experiments. All
classifiers used in this study follow the standard implementation with the
default parameter settings in the Python package scikit-
learn555https://scikit-learn.org/stable/whats_new/v1.2.html#version-1-2-0/.
All resources were accessed on 15th October 2023 1.1.2. The DE algorithm
parameters were set according to its standard implementation [20].

To mitigate potential threats to the experimentation validity and ensure that
all experiments have run under the same hardware conditions, this study
follows the majority of the recommendations reported in [18]. Specifically, to
achieve a high precision of the statistical spread of the experimental
results, each experiment was conducted 30 times and the average of all runs
was reported for each evaluation metric. Further, the experiment pairs
(classifier, dataset) were randomly shuffled to reduce the effect of unknown
processes depending on the execution order that might affect the energy
measurements. Additionally, a 5-second CPU-intensive warm-up operation (before
the experiment execution) and 5-second sleep intervals (between runs) have
been introduced to avoid hardware cold boot and to enable the hardware to cool
down, respectively. In each run,— evaluation metrics—the model classification
accuracy, F1-score and training energy consumption in Joules (J) were
recorded.

To have a fair comparison and comprehensive insights, the experimental runs
were divided into two phases. The first phase of the experimentation followed
the common practice (without the sampling framework) of having 30-folds of 70%
training and 30% testing sets for each dataset and model pair. The
permutations of each selected dataset and model pairs were evaluated and
recorded in terms of energy consumption and recognition (accuracy and
F1-score) for 30 runs. This part provided us with the baseline reference for
the following experimentation phases.

The second phase of the experimentation was divided into two parts that make
use of the proposed sampling framework. The first part took the output of the
sampling framework (elite samples) and followed the same experimental
conditions used in the first phase except for the training set percentage
($TR$ was set based on the selected $q$ value) to compare the energy
consumption and recognition of the same dataset and model pairs. Following, in
the second part, the model generalisability was tested using the whole
remaining data ($X-TR$) which was more than 30% to ensure the general
effectiveness of the elite samples on the model’s behaviour. This provides a
new way of testing the performance of AI models with existing squandered data
resources. In total, 18K (8 Classifiers × 25 Datasets × 30 cross-validations ×
3 types of experiments) experimental runs were executed to achieve the study
objectives.

###  IV-B Experimental Results

This section presents the results of the experimental evaluation of the
proposed framework. The obtained results were interpreted in two stages. The
first stage highlights the overall effect of the elite training samples on
models’ recognition performance. The second stage gives detailed insights into
the effect of the elite training samples on individual datasets and model
pairs.

####  IV-B1 Overall Performance of Classifiers Over All Datasets

In this stage, the average performance of each classifier over the 25
experimental datasets is reported as an initial understanding of the obtained
results. It is worth mentioning that the reported results, in terms of
classification accuracy, F1-score and training energy consumption of each
classifier represent the average of the average of the 30 runs. This is a
general performance indicator of the classifier’s performance once exposed to
different training and testing data splits. In what follows, the common
practice of having stratified 70% training and 30% testing data splits through
cross-validation is referred to as normal, whereas the 10% training (the
chosen $q$ value in this work) is referred to as elite. According to the
literature survey in Section II, some studies reported reduction rates of up
to 85% of the training data with reasonable classifier performance. This
motivated the choice of the $q$ value to be only 10% of the whole dataset
which indeed is considered a parameter that needs further tuning in future
studies.

(a)

(b)

Figure 3: Overall Average Classification Accuracy and F1- Score of Classifiers
over all datasets

Figure 3 shows the obtained average performance of the classifiers in terms of
classification accuracy (Figure 3.a) and F1-score (Figure 3.b), where three
types of comparisons are illustrated. The first type of comparison highlights
the effect of elite training samples (10%) on the overall average performance
(in terms of both metrics) of all classifiers compared to the baseline (normal
training 70%) using 30% testing sets. This is represented by comparing the
leftmost and the middle columns of each classifier. We observe that the
overall classifier performance can be categorised as either improved including
the AdaBoost, Logistic Regression, Naive Bayes, and SVM, slightly improved
such as Decision Tree, KNN, and Random Forest, or slightly declined in one
classifier (Bagging Classifier).

Finding 1 (elite samples effect on classifiers performance with 30% test set):
7 out of 8 classifiers can learn from fewer data and give similar or better
performance than the baseline training practice.

The second type of comparison validates the performance generalisability of
the classifiers when trained with the elite samples. Taking a step forward,
the remaining 90% was used as a testing set to ensure that each classifier has
learned the dataset representative information from the elite training samples
and to refute any chance of over/under-fitting incidents. The obtained results
show similar outcomes to the first type comparison results (by comparing the
first and the third columns of each classifier in Figure 3) where all
classifiers succeeded in holding a comparable performance. Another inferred
insight is that despite the increase in testing samples (from 30% to 90%), a
negligible decline in all classifiers’ performance is noticed (by comparing
the second and the third columns of each classifier in Figure 3). Normally the
performance of a classifier decreases as the amount of testing data increases
with less data to train on, so they are less likely to make accurate
predictions. However, the above results confirm both the representative
quality of the elite training samples and the generalisability of the model
performance once exposed to larger inference samples.

Finding 2 (elite samples effect on classifiers performance with 90% test set):
Once trained with elite samples, classifiers were able to retain comparable
performance, and this performance was maintained over larger testing sets
compared to standard practice.

Figure 4: Overall Average Training Energy Consumption of Classifiers over all
datasets.

The third type of comparison reports the overall gains, in terms of the
training energy consumption of the classifiers over all datasets. Figure 4
contrasts the average training energy consumption (in joules) of each
classifier on all datasets before and after elite training sample usage. In
other words, it reports the average amount of jouls needed for each classifier
per training run (calculated based on the datasets involved in this study).
The classifiers with the lowest training energy consumptions are KNN, Naive
Bayes, and Decision Tree. However, based on the obtained results, these
energy-efficient classifiers still benefitted from approximately 60%, 55%, and
59% training energy reduction using elite training samples, respectively.
Further, the results confirm that the reduction in training energy consumption
for all classifiers is bounded between 45% (lowest as in the AdaBoost
classifier) and 90% (highest as in the SVM classifier).

Finding 3 (elite training samples impact on classifiers energy consumption):
Different classifiers exhibit drastic energy consumption savings, even in the
case of energy-efficient classifiers, that can reach up to 90%.

####  IV-B2 Individual Performance of Each Classifier and Dataset Pairs

This stage provides a deeper look at the performance of classifiers on
individual datasets and shows the performance variations of classifiers in
terms of classification accuracy and energy efficiency. Due to the large
number of experiments (datasets, classifiers), validation metrics
(classification accuracy, F1-score, standard deviation, training energy), and
phases (baseline with 70% training and 30% testing, elite training with 30%
testing, and elite training with 90% testing), reporting several raw tabular
results (available at the link provided earlier) will be an eyesore to the
reader. Hence, more compact and visually tabular comparisons are provided in
what follows (similar to the heat map) where the baseline performance is
considered as the reference. In other words, all the results reported in
Tables III-V are percentage values of improvement (positive) or degradation
(negative) compared to the baseline values. Moreover, each table shows a
colour contrast with a legend for easier comparisons, insights, and
conclusions.

Table III shows the detailed conclusions taken from Figure 3.a when columns 1
and 2 are compared, respectively. The percentage of improvement/degradation of
the average accuracy of each classifier and dataset pairs trained with the
elite samples and 30% testing set is reported. In the first category of
classifier performance (inferred as improved earlier), the blue hue that
represents an improvement over the baseline is dominant in the Naive Bayes
classifier making it the most promising candidate for using elite training
samples. Naive Bayes is followed by the Logistic Regression, SVM, and AdaBoost
classifiers, where elite training succeeded in retaining similar classifier
performance (few neutral cases represented by 0%) or enhanced the classifier
performance (in the majority of the cases). However, a negligible decline in
the three classifiers’ performance is noticed in one (yeast), two (Letter and
Mammographic) and three cases (Balance, Letter, and Wdbc), respectively.

The second category (inferred as slightly improved earlier) includes the
Decision Tree, KNN, and Random Forest classifiers. Although the Decision Tree
classifier has only 3 out of 25 declined performance cases, it has a few
neutral cases and a majority of slightly improved cases. For the KNN and
Random Forest classifiers, the number of neutral and slightly declined cases
is higher but with overall slightly improved classifier performance.

The last slightly declined category includes one classifier (the bagging
classifier) that has the majority of slightly declined cases and few neutral
and slightly improved cases leading to an overall slightly declined
performance. The slight performance degradation of the Bagging classifier
performance is due to the fact that using 10% of the dataset, as a training
set, was insufficient for the model to capture all the data representative or
hidden structures, especially in large-sized datasets. However, this
performance is a compromise that someone might take compared to rewarding
gains recorded in the classifier training energy consumption (see Figure 4).

Similarly, Table IV shows the detailed conclusions taken from Figure 3.a when
columns 2 and 3 are compared, respectively. The testing set was increased to
90% (since only 10% has been used as elite training samples) to validate the
performance of the classifiers in terms of over/under-fitting and
generalisability. The obtained results and conclusions were Convergent to what
has been represented in Table III, where all classifiers held the same
performance with a very reasonable overall improvement/decline despite being
exposed to higher inference cases. All of the above observations apply to the
F1-score measure when analyzing classifier performance on individual datasets
(as classification accuracy is not a stand-alone metric).

Finding 4 (elite samples for tailored AI training): Using elite training
samples, the overall performance of a dataset and model pair can witness an
improvement of up to 50% compared to the common training practice.

As previously said, obtaining elite training samples is regarded as an
optimisation problem in this study, and it becomes complex in higher-
dimensional search spaces. Since this study advocates for greener AI model
training, finding the elite training samples itself is an energy-consuming
process and raises questions about its worthiness and how much energy savings
could be gained after including the energy consumed by the sampling framework
(the optimisation process). To answer this question, the number of hits of
each dataset (see Table II) that indicates the number of times the dataset has
been used in a classification task was retrieved. Surely, this number does not
reflect the real situation in the AI community, logically it is far beyond
that, due to the unreported massive offline experiments, multiple runs for the
model’s parameter tuning, etc. However, considering this parameter as a lower
bound gives an estimation of how much energy could have been saved if all
these hits used the elite samples to train the selected classifiers in this
study. In other words, this highlights the importance of not only saving
energy consumption of the current model training but also showing the long-
term effect and the sustainability of model training in terms of required
computations in the future.

TABLE III: Percentage of improvement/degradation of the average accuracy of
each classifier and dataset pairs with elite training and 30% testing set.
TABLE IV: Percentage of improvement/degradation of the average accuracy of
each classifier and dataset pairs with elite training and 90% testing set.

In all study experiments, the average training energy consumption of each
classifier was recorded for the normal training $E_{N_{TR}}$, reduced training
$E_{R_{TR}}$, and the sampling framework $E_{F}$. So the normal amount of
consumed energy $E_{N_{M}}$ by a classifier $M$ on a dataset $X$ is calculated
as follows:

$E_{N_{M}}=E_{N_{TR}}\times X_{\\#hits}.$ (1)

Similarly, the reduced amount of consumed energy $E_{R_{M}}$ including the
sampling framework overhead is calculated as follows:

$E_{R_{M}}=(E_{R_{TR}}\times X_{\\#hits})+E_{F}.$ (2)

Then, the excess/reduction percentage of energy consumption is calculated by
comparing $E_{N_{M}}$ and $E_{R_{M}}$ values calculated from equations (1) and
(2). Table V reports the percentage of training energy consumption reduction
compared to the baseline of each classifier and dataset pairs trained with
elite samples including the framework overhead.

First, the obtained results demonstrated that even in the worst circumstances,
there is still a minor energy reduction (a few examples indicated by white
shades, such as the AdaBoost, Bagging, and Random Forest classifiers paired
with the Glass and Wisconsin datasets). Secondly, energy consumption reduction
was significant in the majority of other cases and reached up to 98% (as in
the SVM and Coil2000 pair). Thirdly, the predicted relationship between the
problem space (the dataset characteristics: #Obs. and #Atts.) and the
percentage of energy consumption reduction was clear. In other words, the more
complex the problem space is, the more energy reduction is gained using the
elite training sample. This conclusion is supported by the reported energy
reduction results that have the dark green hue in all classifiers of the most
complex datasets such as Coil200, Letter, Optdigits, Spambase, Page-blocks,
Thyroid, etc. (see Table II). Conversely, the reported energy reduction
results in a light green or white hue tended to be for simpler problems such
as Wisconsin and Glass for some classifiers only.

In addition, regardless of the problem complexity, some classifiers achieved
more energy gains than others (such as SVM, Decision Tree, and Naive Bayes)
which is a learning mechanism causality. Lastly, it is noteworthy that the
reported results in Table V reflect the feasibility of finding the elite
training samples to train models compared to the conventional training
practice in terms of energy efficiency. We can infer that some classifiers are
more or less sensitive to the number of observations or attributes than
others. A detailed analysis of these relationships has been investigated in
[18] and they are conformant with the results obtained in this study.

Finding 5 (elite samples viability towards Green AI): Using elite training
samples, certain dataset and model pair combinations can achieve remarkable
energy savings of 98% compared to the common training practice.

Taking advantage of this proposal could be a step forward towards greener AI
model training at the data level rather than increasing the complexity of AI
model architectures.

TABLE V: Percentage of training energy consumption reduction of each
classifier and dataset pairs with elite training including the optimiser
overhead.

##  V Discussion and Future Directions

This empirical study aimed at providing a proof of concept that AI models can
learn from fewer exemplar instances in a more energy-efficient manner. Since
the topic lies within an intersection of several domains, e.g., optimisation
theory, instance selection techniques, machine learning models, and energy
efficiency, some of the further steps that lie ahead towards greener AI
practices are:

  * •

A systematic energy-efficiency comparison of IS techniques, including both
traditional and EA-based methods, should be conducted to gain a comprehensive
understanding of the potential trade-offs between model accuracy and energy
required to identify elite samples on a larger scale. This line of research
can include endless modifications, among others, instance selection criteria,
parameter tuning, evaluation metrics, and their applications.

  * •

Arguably, identifying elite samples in a given dataset is an AI model-
dependent process that poses a multi-objective optimisation problem (MOP) in
terms of the tradeoffs between model accuracy and training data size. For
instance, the size and the selected elite training samples that provide a
certain accuracy will differ from one model to another. In addition, the
energy consumption of each model is unique due to the learning mechanism they
implement. Such problems with conflicting objectives are widely solved using
multi-objective evolutionary algorithms (MOEAs) [59]. Therefore, further
investigation is needed to evaluate the MOEA’s capability and feasibility in
finding the Pareto elite training samples matching each dataset and model
pair.

  * •

The size of the decision and objective space of the MOPs define their
complexity. Presently, almost all supercomputers have become heterogeneous and
energy-efficient hardware architectures. Hence, one of the potential research
endeavours is to explore the parallel implementation of the MOEAs as an IS
technique on modern parallel architectures. The combination of parallel MOEAs
with energy-efficient hardware is expected to facilitate solving large-scale
MOPs by providing faster, more accurate and energy-efficient solutions.

  * •

Determining elite training samples associated with a dataset and model pairs
and archiving these pairs for public use, e.g., through a shared green AI
repository, would nullify the need for the AI community to train models
individually on a full training set regularly (repeated and wasted efforts).
In addition, having access to the archived elite samples and model pairs with
the associated evaluation metrics provides an intermediary state where other
researchers can improve further the quality of the elite samples and discover
the correlation/causation between the participating samples and model
performance. Such discovery might lead to faster approaches to finding the
elite samples and substantially contribute towards Greener AI.

##  VI Conclusion

The plethora of data, AI tools, and computing infrastructures facilitate
training myriad sorts of AI models with non-negligible effects on the
environment through energy demands. This paper presented an evolutionary-based
sampling framework to curb the required energy while training AI models with
few but effective training instances, namely, elite training samples. The
problem formulation and the customisation of the differential evolution
algorithm to extract the elite training samples, as an IS problem, were
presented. The effect of the elite training samples was analysed on commonly
used datasets and classifiers in terms of classification performance, energy
consumption, and generalisability. The obtained results demonstrated the great
potential of the proposed framework to retain model performance with
drastically reduced energy demands, especially on large-sized datasets.
Further, the feasibility of employing the proposed framework as a part of the
AI model training routine proved its applicability towards a long-term
sustainable AI outlook. Finally, prospects and research directions that
necessarily need collaborative AI community efforts were listed to encourage
researchers to produce a greener AI future.

## Acknowledgments

This work is partially funded by the joint research programme UL/SnT–ILNAS on
Technical Standardisation for Trustworthy ICT, Aerospace, and Construction.

This work is partially supported by the European Union under the Italian
National Recovery and Resilience Plan (NRRP) of NextGenerationEU, partnership
on “Telecommunications of the Future” (PE0000001 - program “RESTART”).

## References

  * [1] W. Wang and K. Siau, “Artificial intelligence, machine learning, automation, robotics, future of work and future of humanity: A review and research agenda,” _Journal of Database Management (JDM)_ , vol. 30, no. 1, pp. 61–79, 2019.
  * [2] K. B. Johnson, W.-Q. Wei, D. Weeraratne, M. E. Frisse, K. Misulis, K. Rhee, J. Zhao, and J. L. Snowdon, “Precision medicine, ai, and the future of personalized health care,” _Clinical and Translational Science_ , vol. 14, no. 1, pp. 86–93, 2021. [Online]. Available: https://ascpt.onlinelibrary.wiley.com/doi/abs/10.1111/cts.12884
  * [3] E. B. Hansen and S. Bøgh, “Artificial intelligence and internet of things in small and medium-sized enterprises: A survey,” _Journal of Manufacturing Systems_ , vol. 58, pp. 362–372, 2021.
  * [4] Z. Zhang, H. Ning, F. Shi, F. Farha, Y. Xu, J. Xu, F. Zhang, and K.-K. R. Choo, “Artificial intelligence in cyber security: research advances, challenges, and opportunities,” _Artificial Intelligence Review_ , pp. 1–25, 2022.
  * [5] S. Trewin, S. Basson, M. Muller, S. Branham, J. Treviranus, D. Gruen, D. Hebert, N. Lyckowski, and E. Manser, “Considerations for ai fairness for people with disabilities,” _AI Matters_ , vol. 5, no. 3, pp. 40–63, 2019.
  * [6] C.-J. Wu, R. Raghavendra, U. Gupta, B. Acun, N. Ardalani, K. Maeng, G. Chang, F. Aga, J. Huang, and C. Bai, “Sustainable ai: Environmental implications, challenges and opportunities,” _Proceedings of Machine Learning and Systems_ , vol. 4, pp. 795–813, 2022.
  * [7] D. Amodei and D. Hernandez, “AI and compute,” Open AI, May 2018. [Online]. Available: https://openai.com/research/ai-and-compute/
  * [8] J. Cheng, P.-s. Wang, G. Li, Q.-h. Hu, and H.-q. Lu, “Recent advances in efficient computation of deep convolutional neural networks,” _Frontiers of Information Technology & Electronic Engineering_, vol. 19, pp. 64–77, 2018.
  * [9] A. J. Abd El-Maksoud, M. Ebbed, A. H. Khalil, and H. Mostafa, “Power efficient design of high-performance convolutional neural networks hardware accelerator on fpga: A case study with googlenet,” _IEEE Access_ , vol. 9, pp. 151 897–151 911, 2021.
  * [10] A. S. Luccioni and A. Hernandez-Garcia, “Counting carbon: A survey of factors influencing the emissions of machine learning,” _arXiv preprint arXiv:2302.08476_ , 2023.
  * [11] E. Strubell, A. Ganesh, and A. McCallum, “Energy and policy considerations for deep learning in nlp,” _arXiv preprint arXiv:1906.02243_ , 2019.
  * [12] P. Dhar, “The carbon impact of artificial intelligence.” _Nat. Mach. Intell._ , vol. 2, no. 8, pp. 423–425, 2020.
  * [13] E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell, “On the dangers of stochastic parrots: Can language models be too big?” in _Proceedings of the 2021 ACM conference on fairness, accountability, and transparency_ , 2021, pp. 610–623.
  * [14] M. Heikkilä, “We’re getting a better idea of ai’s true carbon footprint,” MIT Technology Review, Nov. 2022. [Online]. Available: https://www.technologyreview.com/2022/11/14/1063192/were-getting-a-better-idea-of-ais-true-carbon-footprint/
  * [15] B. Marr, “Green intelligence: Why data and ai must become more sustainable,” Forbes, Mar. 2023. [Online]. Available: https://www.forbes.com/sites/bernardmarr/2023/03/22/green-intelligence-why-data-and-ai-must-become-more-sustainable/?sh=5747835e7658/
  * [16] R. Schwartz, J. Dodge, N. A. Smith, and O. Etzioni, “Green ai,” _Communications of the ACM_ , vol. 63, no. 12, pp. 54–63, 2020.
  * [17] R. Verdecchia, J. Sallou, and L. Cruz, “A systematic review of green ai,” _Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery_ , p. e1507, 2023.
  * [18] R. Verdecchia, L. Cruz, J. Sallou, M. Lin, J. Wickenden, and E. Hotellier, “Data-centric green ai an exploratory empirical study,” in _2022 international conference on ICT for sustainability (ICT4S)_. IEEE, 2022, pp. 35–45.
  * [19] A. Slowik and H. Kwasnicka, “Evolutionary algorithms and their applications to engineering problems,” _Neural Computing and Applications_ , vol. 32, pp. 12 363–12 379, 2020.
  * [20] R. Storn and K. Price, “Differential evolution–a simple and efficient heuristic for global optimization over continuous spaces,” _Journal of global optimization_ , vol. 11, pp. 341–359, 1997.
  * [21] S. García, J. Luengo, and F. Herrera, _Data preprocessing in data mining_. Springer, 2015, vol. 72.
  * [22] P. Hart, “The condensed nearest neighbor rule (corresp.),” _IEEE transactions on information theory_ , vol. 14, no. 3, pp. 515–516, 1968.
  * [23] D. R. Wilson and T. R. Martinez, “Reduction techniques for instance-based learning algorithms,” _Machine learning_ , vol. 38, pp. 257–286, 2000.
  * [24] D. L. Wilson, “Asymptotic properties of nearest neighbor rules using edited data,” _IEEE Transactions on Systems, Man, and Cybernetics_ , no. 3, pp. 408–421, 1972.
  * [25] D. W. Aha, D. Kibler, and M. K. Albert, “Instance-based learning algorithms,” _Machine learning_ , vol. 6, pp. 37–66, 1991.
  * [26] E. Leyva, A. González, and R. Pérez, “Three new instance selection methods based on local sets: A comparative study with several approaches from a bi-objective perspective,” _Pattern Recognition_ , vol. 48, no. 4, pp. 1523–1537, 2015.
  * [27] J. L. Carbonera, “An efficient approach for instance selection,” in _Big Data Analytics and Knowledge Discovery: 19th International Conference, DaWaK 2017, Lyon, France, August 28–31, 2017, Proceedings 19_. Springer, 2017, pp. 228–243.
  * [28] C. J. Luís and A. Mara, “A density-based approach for instance selection,” in _2015 IEEE 27th International Conference on Tools with Artificial Intelligence (ICTAI)_. IEEE, 2015, pp. 768–774.
  * [29] M. Malhat, M. El Menshawy, H. Mousa, and A. El Sisi, “A new approach for instance selection: Algorithms, evaluation, and comparisons,” _Expert Systems with Applications_ , vol. 149, p. 113297, 2020.
  * [30] J. L. Carbonera and M. Abel, “Efficient instance selection based on spatial abstraction,” in _2018 IEEE 30th international conference on tools with artificial intelligence (ICTAI)_. IEEE, 2018, pp. 286–292.
  * [31] M. Moran, T. Cohen, Y. Ben-Zion, and G. Gordon, “Curious instance selection,” _Information Sciences_ , vol. 608, pp. 794–808, 2022.
  * [32] W. Cunha, F. Viegas, C. França, T. Rosa, L. Rocha, and M. A. Gonçalves, “A comparative survey of instance selection methods applied to nonneural and transformer-based text classification,” _ACM Computing Surveys_ , 2023.
  * [33] N. García-Pedrajas, “Evolutionary computation for training set selection,” _Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery_ , vol. 1, no. 6, pp. 512–523, 2011.
  * [34] S. Garcia, J. Derrac, J. Cano, and F. Herrera, “Prototype selection for nearest neighbor classification: Taxonomy and empirical study,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 34, no. 3, pp. 417–435, 2012.
  * [35] J. A. Olvera-López, J. A. Carrasco-Ochoa, J. F. Martínez-Trinidad, and J. Kittler, “A review of instance selection methods,” _Artificial Intelligence Review_ , vol. 34, pp. 133–143, 2010.
  * [36] J. Derrac, S. García, and F. Herrera, “A survey on evolutionary instance selection and generation,” in _Modeling, Analysis, and Applications in Metaheuristic Computing: Advancements and Trends_. IGI Global, 2012, pp. 233–266.
  * [37] H. A. Bouarara, _Advanced Bioinspiration Methods for Optimisation Problems: Survey and Applications_. IGI Global, 2023, pp. 17–69.
  * [38] R. Verdecchia, P. Lago, C. Ebert, and C. De Vries, “Green it and green software,” _IEEE Software_ , vol. 38, no. 6, pp. 7–15, 2021.
  * [39] F. N. Al-Wesabi, M. Obayya, M. A. Hamza, J. S. Alzahrani, D. Gupta, and S. Kumar, “Energy aware resource optimization using unified metaheuristic optimization algorithm allocation for cloud computing environment,” _Sustainable Computing: Informatics and Systems_ , vol. 35, p. 100686, 2022.
  * [40] C. Lu, Y. Huang, L. Meng, L. Gao, B. Zhang, and J. Zhou, “A pareto-based collaborative multi-objective optimization algorithm for energy-efficient scheduling of distributed permutation flow-shop with limited buffers,” _Robotics and Computer-Integrated Manufacturing_ , vol. 74, p. 102277, 2022.
  * [41] F. Xing, S. He, V. C. Leung, and H. Yin, “Energy efficiency optimization for rate-splitting multiple access-based indoor visible light communication networks,” _IEEE Journal on Selected Areas in Communications_ , vol. 40, no. 5, pp. 1706–1720, 2022.
  * [42] L. Yue, H. Wang, J. Mumtaz, M. Rauf, and Z. Li, “Energy-efficient scheduling of a two-stage flexible printed circuit board flow shop using a hybrid pareto spider monkey optimisation algorithm,” _Journal of Industrial Information Integration_ , vol. 31, p. 100412, 2023.
  * [43] H. F. Cano, José Ramón and L. Manuel, “Using evolutionary algorithms as instance selection for data reduction in kdd: An experimental study,” _IEEE transactions on evolutionary computation_ , vol. 7, no. 6, pp. 561–575, 2003.
  * [44] J. R. Cano, F. Herrera, and M. Lozano, _Instance Selection Using Evolutionary Algorithms: An Experimental Study_. London: Springer London, 2005, pp. 127–152. [Online]. Available: https://doi.org/10.1007/1-84628-183-0_5
  * [45] C. J. Ramón, H. Francisco, and L. Manuel, “Stratification for scaling up evolutionary prototype selection,” _Pattern Recognition Letters_ , vol. 26, no. 7, pp. 953–963, 2005.
  * [46] J. R. Cano, F. Herrera, and M. Lozano, “Evolutionary stratified training set selection for extracting classification rules with trade off precision-interpretability,” _Data & Knowledge Engineering_, vol. 60, no. 1, pp. 90–108, 2007.
  * [47] S. García, J. R. Cano, and F. Herrera, “A memetic algorithm for evolutionary prototype selection: A scaling up approach,” _Pattern Recognition_ , vol. 41, no. 8, pp. 2693–2709, 2008.
  * [48] I. Triguero, S. García, and F. Herrera, “Differential evolution for optimizing the positioning of prototypes in nearest neighbor classification,” _Pattern Recognition_ , vol. 44, no. 4, pp. 901–916, 2011.
  * [49] D. J. Drown, T. M. Khoshgoftaar, and R. Narayanan, “Using evolutionary sampling to mine imbalanced data,” in _Sixth International Conference on Machine Learning and Applications (ICMLA 2007)_. IEEE, 2007, pp. 363–368.
  * [50] S. García, A. Fernández, and F. Herrera, “Enhancing the effectiveness and interpretability of decision tree and rule induction classifiers with evolutionary training set selection over imbalanced problems,” _Applied soft computing_ , vol. 9, no. 4, pp. 1304–1314, 2009.
  * [51] M. Kabkab, A. Alavi, and R. Chellappa, “Dcnns on a diet: Sampling strategies for reducing the training set size,” _arXiv preprint arXiv:1606.04232_ , 2016.
  * [52] N. García-Pedrajas, J. A. Romero del Castillo, and D. Ortiz-Boyer, “A cooperative coevolutionary algorithm for instance selection for instance-based learning,” _Machine Learning_ , vol. 78, pp. 381–420, 2010.
  * [53] A. Rosales-Pérez, S. García, J. A. Gonzalez, C. A. C. Coello, and F. Herrera, “An evolutionary multiobjective model and instance selection for support vector machines with pareto-based ensembles,” _IEEE Transactions on Evolutionary Computation_ , vol. 21, no. 6, pp. 863–877, 2017.
  * [54] K.-j. Kim, “Artificial neural networks with evolutionary instance selection for financial forecasting,” _Expert Systems with Applications_ , vol. 30, no. 3, pp. 519–526, 2006.
  * [55] C. Coleman, C. Yeh, S. Mussmann, B. Mirzasoleiman, P. Bailis, P. Liang, J. Leskovec, and M. Zaharia, “Selection via proxy: Efficient data selection for deep learning,” _arXiv preprint arXiv:1906.11829_ , 2019.
  * [56] P. Ang, B. Dhingra, and L. W. Wills, “Characterizing the efficiency vs. accuracy trade-off for long-context nlp models,” _arXiv preprint arXiv:2204.07288_ , 2022.
  * [57] P. Dhabe, P. Mirani, R. Chugwani, and S. Gandewar, _Data set reduction to improve computing efficiency and energy consumption in healthcare domain_. Springer, 2021, pp. 53–64.
  * [58] G. Bonaccorso, _Machine Learning Algorithms: Popular algorithms for data science and machine learning_. Packt Publishing Ltd, 2018.
  * [59] Y. Tian, L. Si, X. Zhang, R. Cheng, C. He, K. C. Tan, and Y. Jin, “Evolutionary large-scale multi-objective optimization: A survey,” _ACM Computing Surveys (CSUR)_ , vol. 54, no. 8, pp. 1–34, 2021.

|  Mohammed Alswaitti received the MS and PhD degrees in computer engineering
from the University of Science Malaysia (USM), Malaysia in 2011 and 2018. He
is currently a research associate with the parallel Computing and Optimisation
Group of The Interdisciplinary Centre for Security, Reliability and Trust
(SnT) at the University of Luxembourg, Luxembourg. His current research
interests include evolutionary computation, optimisation, data clustering, and
data-centric energy-efficient computing. He has authored more than 20 papers
in refereed journals and conferences in these areas.
---|---
|  Roberto Verdecchia received a double Ph.D. in Computer Science appointed by
the Gran Sasso Science Institute, L’Aquila, Italy, and the Vrije Universiteit,
The Netherlands. He is currently an Assistant Professor at the Software
Technology Laboratory (STLab) of the School of Engineering, University of
Florence, Italy. His research interest focuses on the adoption of empirical
methods to improve software development and system evolution, with particular
interest in the fields of software energy efficiency, technical debt, software
architecture, and software testing. More information is available at
robertoverdecchia.github.io.
---|---
|  Grégoire Danoy received the Industrial Engineering degree in computer
science from Luxembourg University of Applied Sciences, Luxembourg City,
Luxembourg, in 2003, and the master’s and Ph.D. degrees in computer science
from the Ecole des Mines of Saint-Etienne, Saint-Ètienne, France, in 2004 and
2008, respectively. Since 2008, he has been a Research Scientist with the
Parallel Computing and Optimization Group, University of Luxembourg,
Luxembourg. He is currently a Research Scientist with the University of
Luxembourg (UL) and the Deputy Head of the Parallel Computing and Optimization
Group (PCOG). He has published over 100 research articles in international
journals and conferences. He has authored a book entitled Evolutionary
Algorithms for Mobile Ad Hoc Networks (Wiley). His research interests include
exact and approximate methods applied to bio-informatics, cloud computing,
high-performance computing, the Internet of Things, and smart cities.
---|---
|  Pascal Bouvry (Member, IEEE) received the bachelor’s degree in economic and
social sciences and the master’s degree (Hons.) in computer science from the
University of Namur, Belgium, in 1991, and the Ph.D. degree in computer
science from the University of Grenoble (INPG), France, in 1994. He is
currently a Full Professor with the University of Luxembourg, “Chargé de
Mission auprès du Recteur” in charge of the University High-Performance
Computing, heading the Parallel Computing and Optimization Group (PCOG), and
directing the master’s programs in HPC and Technopreneurship. He is also a
Faculty Member with the Interdisciplinary Center of Security, Reliability and
Trust, and active in various scientific committees and technical workgroups.
His research interests include cloud and parallel computing, optimization,
security, and reliability.
---|---
|  Johnatan Pecero Sanchez received a doctorate in computer science from the
University of Grenoble (INPG), France and carried out post-doctoral research
at the University of Luxembourg on energy optimization in data centres, cloud
and HPC. He then became head of the standardization department at the
Luxembourg Institute for Standardization where he contributed to the
development of the normative culture, the establishment of education and
research programs with industrial partners and the University of Luxembourg,
including the Master in Technopreneurship. Since 2021, Johnatan Pecero Sanchez
has been Partnership Development Officer - HPC at the University of Luxembourg
where he is contributing to the development of the EUmaster4HPC project.
---|---


# Transformer-based Model for Word Level Language Identification in Code-mixed
Kannada-English Texts

Atnafu Lambebo Tonja1, Mesay Gemeda Yigezu2, Olga Kolesnikova3,
Moein Shahiki Tash4, Grigori Sidorov5, Alexander Gelbukh6

Instituto Politécnico Nacional (IPN), Centro de Investigación en Computación
(CIC), Mexico City, Mexico
{ 1alambedot2022, 2mgemedak2022, 5sidorov<EMAIL_ADDRESS>
{ 3kolesolga, 4<EMAIL_ADDRESS>
###### Abstract

Using code-mixed data in natural language processing (NLP) research currently
gets a lot of attention. Language identification of social media code-mixed
text has been an interesting problem of study in recent years due to the
advancement and influences of social media in communication. This paper
presents the Instituto Politécnico Nacional, Centro de Investigación en
Computación (CIC) team’s system description paper for the CoLI-Kanglish shared
task at ICON2022. In this paper, we propose the use of a Transformer based
model for word-level language identification in code-mixed Kannada English
texts. The proposed model on the CoLI-Kenglish dataset achieves a weighted
F1-score of 0.84 and a macro F1-score of 0.61.

##  1 Introduction

In recent years, language identification of social media text has been a
fascinating research topic Ansari et al. (2021). Social media platforms have
become more integrated in this digital era and have impacted various people’s
perceptions of networking and socializing Tonja et al. (2022c). This influence
allowed different users to communicate via various social media platforms
using a mix of texts. NLP technology has advanced rapidly in many
applications, including machine translation Tonja et al. (2022b); Yigezu et
al. (2021); Tonja et al. (2021), abusive comment detection Balouchzahi et al.
(2022b), fake news detectionArif et al. (2022); Truică et al. (2022),
aggressive incident detection Tonja et al. (2022a), hope speech detection
Gowda et al. (2022), and others. However, numerous tools have not yet been
created for languages with limited resources or languages with code-mixed
data.

Code-mixing is the use of linguistic units—words, phrases, and clauses—at the
sentence or word level from various languages. In casual communication, such
as social media, it is typically seen. We have access to an enormous amount of
code-mixed data because of the various social media platforms that allow
individuals to communicate Sutrisno and Ariesta (2019). As a result, automatic
language recognition at the word level has become an essential part of
analyzing noisy content in social media. It would help with the automated
analysis of content generated on social media. Currently, in the area of NLP,
different researchers are developing different NLP applications in code-mixed
datasets. Some of the applications are code-mixed sentiments analysis
Balouchzahi et al. (2021b), code-mixed offensive language identification
Balouchzahi et al. (2021a), etc. We took part in the Kanglish shared task
Balouchzahi et al. (2022a), which aims to identify language at the word level
from code-mixed data for Kannada-English texts. For word-level code-mixed
language identification tasks, we used Transformer -based Vaswani et al.
(2017) pre-trained language models (PLMs). Our transformer-based model
consists of BERT Devlin et al. (2018) and its three variants. We used PLMs and
LSTM models for this word-level language identification task.

This paper discusses a Transformer-based model for word-level language
identification in code-mixed Kannada-English texts for the Kanglish shared
task. The paper is organized as follows: Section 2 describes past work related
to this study, section 3 gives an overview of the dataset and its statistics,
section 4 explains the methodology adopted in this study including the
algorithms, section 5 emphasizes on the experimental results and descriptions.
Finally, Section 6 concludes the paper.

##  2 Related Work

Currently, solving NLP problems in code-mixed data is getting attention from
many researchers. For word-level language identification in code-mixed text,
different researchers have suggested various models. Chittaranjan et al.
(2014) proposed a Conditional Random Fields (CRF)- based system for word level
language identification of code-mixed text for four language pairs, namely,
English-Spanish (En-Es), English-Nepali (En-Ne), English-Mandarin (En-Cn), and
Standard Arabic-Arabic (Ar-Ar) dialects. The authors explored various token
levels and contextual features to build an optimal CRF using the provided
training data. The proposed system performed more or less consistently, with
accuracy ranging from 80% to 95% across four language pairs.

Gundapu and Mamidi (2020) also proposed a CRF based model for word-level
language identification in English-Telugu code-mixed data. The authors used
feature extraction as the main task for the proposed model. They used POS-
tags, length of the word, prefix and suffix of focus word, numeric digit,
special symbol, capital letter, and character N-grams (Uni-, Bi-, Trigrams of
words) as features. The proposed CRF-based model had an F1-score of 0.91.

A Support Vector Machines (SVM)-based model for word level language
identification of Tamil-English code-mixed text in social media is proposed by
Shanmugalingam et al. (2018). The authors used dictionaries, double
consonants, and term frequency to identify features. The proposed SVM model
with a linear kernel gave 89.46% accuracy for the language identification
system for Tamil-English code-mixed text at the word level.

Ansari et al. (2021) proposes transfer learning and fine-tuning BERT models
for language identification of Hindi-English code-mixed tweets. The authors
used data from Hindi-English-Urdu code-mixed text for language pre-training
and Hindi-English code-mixed for subsequent word-level language
classification. The authors first pre-trained Hindi-English-Urdu code-mixed
text using BERT and fine-tuned the trained model in downstream Hindi-English
code-mixed word-level language classification. Their proposed model for Hindi-
English code-mixed language identification, both pre-training and fine-tuning
with code-mixed text, gives the best F1-score of 0.84 as compared to their
monolingual counterparts.

##  3 Data

During the experimental phase, we used the CoLI-Kenglish dataset Hosahalli
Lakshmaiah et al. (2022) which consists of English and Kannada words in Roman
script and are grouped into six major categories, namely, Kannada (kn),
English (en), Mixed-language (en-kn), Name, Location and Other. Figure 1 shows
some samples from the dataset used for training.

Figure 1: Training samples

###  3.1 Dataset Statistics

Figures 2 and 3 depict the training and test data distribution statistics with
their assigned tags. The training dataset is slightly imbalanced: 43.9% of the
words were labeled as kn, 30% were labeled as en, 9.28% were labeled as en-kn,
4.76% were labeled as name, 0.68% were labeled as location and 11.2% were
labeled as other. This shows that approximately 73% of the training dataset
was labeled as kn and en. Similarly, in the test dataset, words tagged as en
and kn take a higher number than the rest of the dataset.

Figure 2: Training data distribution with tags Figure 3: Test data
distribution with tags

##  4 Methodology

This section presents a description of the data pre-processing, methodology,
and models used in this work. We used Transformer based pre-trained language
models (PLMs) with the combination of the LSTM model for word level language
identification in Kannada-English code-mixed text. We used PLMs in the
embedding layer of the LSTM model layer.

###  4.1 Pre-processing

Pre-processing is one of the preliminary steps in training NLP tasks, with the
aim of making the training data suitable during the training phase. The
dataset provided by the organizers for this task has passed the basic pre-
processing steps, and we carried out one pre-processing step to prepare the
training data during the experimental phase. We applied label encoding to
tags, to convert the tags into a numeric form. As discussed in section 3, the
dataset contains six tags (kn, en, en-kn, name, location and other). We
converted these tags into numeric values using one-hot encoding.

###  4.2 Proposed Experimental Architecture

Figure 4 shows the experimental architecture of our Transformer-based model
for word level language identification in code-mixed Kannada-English texts.

Figure 4: Experimental architecture for Transformer-based model for word level
language identification in code-mixed Kannada-English texts

As shown in Figure 4, our experimental architecture consists of five steps:

  * •

Step 1 \- preparing labelled data for training, the data set contains words
and their tags as discussed in section 3.

  * •

Step 2 \- we converted the tags into a numeric machine-readable form.

  * •

Step 3 \- after label encoding the representation for each token is fed to
transformer layers to obtain contextualized tokens using PLMs.

  * •

Step 4 \- the embeddings obtained in step-3 are fed into LSTM model to obtain
their corresponding language tag.

We used the following pre-trained language models (PLMs) in the embedding
layer of the LSTM model for our experiment.

  * •

BERT Devlin et al. (2018) \- stands for Bidirectional Encoder Representations
from Transformers. As the name suggests, it is a way of learning
representations of a language that uses a transformer, specifically, the
encoder part of the transformer.

  * •

mBERT Devlin et al. (2018) \- is a Multilingual BERT, it provides sentence
representations for 104 languages, which are useful for many multi-lingual
tasks. Previous work probed the cross-linguality of mBERT using zero-shot
transfer learning on morphological and syntactic tasks.

  * •

XLM-R Conneau et al. (2019) \- uses self-supervised training techniques to
achieve state-of-the-art performance in cross-lingual understanding, a task in
which a model is trained in one language and then used with other languages
without additional training data.

  * •

RoBERTa Liu et al. (2019) is a self-supervised transformers model that was
trained on a large corpus of English data. This means it was pre-trained on
raw texts only, with no human labeling in any way (which is why it can use
lots of publicly available data) and an automatic process to generate inputs
and labels from those texts.

Table 1 shows models used in our experiments and their parameters.

Model |  | Transformer
---
blocks
| Hidden
---
layer size
| Self-attention
---
heads
#Parameters
bert-base-uncased | 12 | 768 | 12 | 110M
bert-base-multilingual-uncased | 12 | 768 | 12 | 110M
xlm-roberta-large | 24 | 1024 | 16 | 355M
roberta-base | 12 | 768 | 12 | 110M
Table 1: Transformers used in this paper and their parameters

##  5 Experiments and Results

This section presents the description of the experimental setups, training
parameters, results, and analysis. We conducted four experiments by replacing
embedding layers with different pre-trained language models, the results are
presented in section 5.2.

###  5.1 Experiments

We used Google colab 111https://colab.research.google.com/ for GPU support
with the Python programming language. Sci-kit-learn 222https://scikit-
learn.org/stable/ and Keras 333https://keras.io/ (with TensorFlow backend)
were utilized for the LSTM model, for PLMs we used Hugging Face
444https://huggingface.co/ transformer libraries. We used PLMs for embedding
and the LSTM model as the classifier, To optimize the model, we used an Adam
optimizer with a batch size of 64 and a learning rate of 0.0001. We used the
maximum number of epochs of 30, with early stopping based on the performance
of the validation set. We also used a dropout of 0.2 to regularize the model.

We added a batch normalization layer to speed up training, and make learning
easier, and a fully-connected output layer with a SoftMax function so that a
probabilistic output of all tags for language identification would be
produced. For further information, all the parameters and their summaries are
depicted in Figure 5. Figure 5 shows our proposed model summary for word-level
language identification in code-mixed Kannada-English texts.

Figure 5: Proposed model summary

###  5.2 Results

Table 2 depicts the overall results (official) of four experiments conducted
in this work. From four experiments, using bert-base-uncased in the embedding
layer with the LSTM model out-performs other pre-trained languages models used
in the embedding layer with the LSTM model with a weighted score of 0.85
precision, 0.84 recall, 0.84 F1-scores and a micro score of 0.62 precision,
0.62 recall, 0.61 F1-scores.

| Weighted Score | Macro score
---|---|---
Model | P | R | F1-score | P | R | F1-score
bert-base-multilingual-uncased | 0.83 | 0.82 | 0.82 | 0.62 | 0.57 | 0.57
xlm-roberta-large | 0.84 | 0.85 | 0.84 | 0.64 | 0.59 | 0.61
roberta-base | 0.83 | 0.8 | 0.81 | 0.63 | 0.55 | 0.52
bert-base-uncased | 0.85 | 0.84 | 0.84 | 0.62 | 0.62 | 0.61

Table 2: Performance of our models on the test set (official results)

The official rank of the top three teams participating in the shared task of
word-level language identification in code-mixed Kannada-English texts is
shown in Table 3. As shown in Table 3 our model ranked second in overall
results among all participant teams.

|  | Weighted Score | Macro score
---|---|---|---
Rank | Team name | P | R | F1-score | P | R | F1-score
1 | tiya1012 | 0.87 | 0.85 | 0.86 | 0.67 | 0.61 | 0.62
2 | Our team | 0.85 | 0.84 | 0.84 | 0.62 | 0.62 | 0.61
2 | Habesha | 0.85 | 0.83 | 0.84 | 0.66 | 0.6 | 0.61
3 | lidoma | 0.83 | 0.83 | 0.83 | 0.64 | 0.56 | 0.58

Table 3: Official rank of top 3 teams Figure 6: Training and validation loss
of BERT based approach Figure 7: Training and validation accuracy of BERT
based approach

Figures 6 and 7 display the training and, validation losses, training, and
validation accuracy of the BERT-based approach for code-mixed language
identification tasks. It is seen that the BERT-based model’s training loss
decreases and stabilizes at a specific point, but the validation loss is not
as stable as the training loss. This shows that the more specialized the model
becomes with training data, the worse it is able to generalize to new data,
resulting in an increase in generalization error.

The above result demonstrates that transformer-based models can give promising
results when applied to NLP tasks like word-level language identification in
code-mixed texts without considering any linguistic features.

##  6 Conclusion

In this paper, we explored the application of BERT-based pre-trained language
models to identify languages at the word level in code-mixed data for Kannada-
English texts. Pre-trained models with a combination of the LSTM model and a
BERT-based model outperformed the others and have shown promising results in
identifying languages in code-mixed Kannada-English texts. Our team achieved
the second place in CoLI-Kanglish: word-level language identification in the
code-mixed Kannada-English texts competition.

## Acknowledgements

The work was done with partial support from the Mexican Government through the
grant A1S-47854 of CONACYT, Mexico, grants 20220852, 20220859, and 20221627 of
the Secretaría de Investigación y Posgrado of the Instituto Politécnico
Nacional, Mexico. The authors thank the CONACYT for the computing resources
brought to them through the Plataforma de Aprendizaje Profundo para
Tecnologías del Lenguaje of the Laboratorio de Supercómputo of the INAOE,
Mexico and acknowledge the support of Microsoft through the Microsoft Latin
America PhD Award.

## References

  * Ansari et al. (2021) Mohd Zeeshan Ansari, MM Sufyan Beg, Tanvir Ahmad, Mohd Jazib Khan, and Ghazali Wasim. 2021.  Language identification of hindi-english tweets using code-mixed bert.  In _2021 IEEE 20th International Conference on Cognitive Informatics & Cognitive Computing (ICCI* CC)_, pages 248–252. IEEE.
  * Arif et al. (2022) Muhammad Arif, Atnafu Lambebo Tonja, Iqra Ameer, Olga Kolesnikova, Alexander Gelbukh, Grigori Sidorov, and AG Meque. 2022.  Cic at checkthat! 2022: multi-class and cross-lingual fake news detection.  _Working Notes of CLEF_.
  * Balouchzahi et al. (2021a) F Balouchzahi, S Bashang, G Sidorov, and HL Shashirekha. 2021a.  Comata oli-code-mixed malayalam and tamil offensive language identification.  In _Working Notes of FIRE 2021-Forum for Information Retrieval Evaluation (Online). CEUR_.
  * Balouchzahi et al. (2022a) Fazlourrahman Balouchzahi, Sabur Butt, Asha Hagde, Noman Ashraf, Shashirekha Hosahalli Lakshmaiah, Grigori Sidorov, and Alexander Gelbukh. 2022a.  Overview of CoLI-Kanglish: Word Level Language Identification in Code-mixed Kannada-English Texts at ICON 2022.  In _19th International Conference on Natural Language Processing Proceedings_.
  * Balouchzahi et al. (2022b) Fazlourrahman Balouchzahi, Anusha Gowda, Hosahalli Shashirekha, and Grigori Sidorov. 2022b.  Mucic@ tamilnlp-acl2022: Abusive comment detection in tamil language using 1d conv-lstm.  In _Proceedings of the Second Workshop on Speech and Language Technologies for Dravidian Languages_ , pages 64–69.
  * Balouchzahi et al. (2021b) Fazlourrahman Balouchzahi, Hosahalli Lakshmaiah Shashirekha, and Grigori Sidorov. 2021b.  Cosad-code-mixed sentiments analysis for dravidian languages.  In _CEUR Workshop Proceedings_ , volume 3159, pages 887–898. CEUR-WS.
  * Chittaranjan et al. (2014) Gokul Chittaranjan, Yogarshi Vyas, Kalika Bali, and Monojit Choudhury. 2014.  Word-level language identification using crf: Code-switching shared task report of msr india system.  In _Proceedings of The First Workshop on Computational Approaches to Code Switching_ , pages 73–79.
  * Conneau et al. (2019) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Unsupervised cross-lingual representation learning at scale.  _arXiv preprint arXiv:1911.02116_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Gowda et al. (2022) Anusha Gowda, Fazlourrahman Balouchzahi, Hosahalli Shashirekha, and Grigori Sidorov. 2022.  Mucic@ lt-edi-acl2022: Hope speech detection using data re-sampling and 1d conv-lstm.  In _Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion_ , pages 161–166.
  * Gundapu and Mamidi (2020) Sunil Gundapu and Radhika Mamidi. 2020.  Word level language identification in english telugu code mixed data.  _arXiv preprint arXiv:2010.04482_.
  * Hosahalli Lakshmaiah et al. (2022) Shashirekha Hosahalli Lakshmaiah, Fazlourrahman Balouchzahi, Anusha Mudoor Devadas, and Grigori Sidorov. 2022.  CoLI-Machine Learning Approaches for Code-mixed Language Identification at the Word Level in Kannada-English Texts.  _acta polytechnica hungarica_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Shanmugalingam et al. (2018) Kasthuri Shanmugalingam, Sagara Sumathipala, and Chinthaka Premachandra. 2018.  Word level language identification of code mixing text in social media using nlp.  In _2018 3rd International Conference on Information Technology Research (ICITR)_ , pages 1–5. IEEE.
  * Sutrisno and Ariesta (2019) Bejo Sutrisno and Yessika Ariesta. 2019.  Beyond the use of code mixing by social media influencers in instagram.  _Advances in Language and Literary Studies_ , 10(6):143–151.
  * Tonja et al. (2022a) Atnafu Lambebo Tonja, Muhammad Arif, Olga Kolesnikova, Alexander Gelbukh, and Grigori Sidorov. 2022a.  Detection of aggressive and violent incidents from social media in spanish using pre-trained language model.  In _Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2022), CEUR Workshop Proceedings. CEUR-WS. org_.
  * Tonja et al. (2022b) Atnafu Lambebo Tonja, Olga Kolesnikova, Muhammad Arif, Alexander Gelbukh, and Grigori Sidorov. 2022b.  Improving neural machine translation for low resource languages using mixed training: The case of ethiopian languages.  In _Mexican International Conference on Artificial Intelligence_ , pages 30–40. Springer.
  * Tonja et al. (2022c) Atnafu Lambebo Tonja, Olumide Ebenezer Ojo, Mohammed Arif Khan, Abdul Gafar Manuel Meque, Olga Kolesnikova, Grigori Sidorov, and Alexander Gelbukh. 2022c.  Cic nlp at smm4h 2022: a bert-based approach for classification of social media forum posts.  In _Proceedings of The Seventh Workshop on Social Media Mining for Health Applications, Workshop & Shared Task_, pages 58–61.
  * Tonja et al. (2021) Atnafu Lambebo Tonja, Michael Melese Woldeyohannis, and Mesay Gemeda Yigezu. 2021\.  A parallel corpora for bi-directional neural machine translation for low resourced ethiopian languages.  In _2021 International Conference on Information and Communication Technology for Development for Africa (ICT4DA)_ , pages 71–76. IEEE.
  * Truică et al. (2022) Ciprian-Octavian Truică, Elena-Simona Apostol, and Adrian Paschke. 2022.  Awakened at checkthat! 2022: fake news detection using bilstm and sentence transformer.  _Working Notes of CLEF_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Yigezu et al. (2021) Mesay Gemeda Yigezu, Michael Melese Woldeyohannis, and Atnafu Lambebo Tonja. 2021\.  Multilingual neural machine translation for low resourced languages: Ometo-english.  In _2021 International Conference on Information and Communication Technology for Development for Africa (ICT4DA)_ , pages 89–94. IEEE.