# Let’s Predict Who Will Move to a New Job

Rania Mkhinini Gahar  OASIS Laboratory
ENIT
University of Tunis El Manar
Tunis, Tunisia
<EMAIL_ADDRESS>Adel Hidri  Computer Department, Deanship
of Preparatory Year and Supporting Studies,
Imam Abdulrahman Bin Faisal University
Dammam, Saudi Arabia
<EMAIL_ADDRESS>Minyar Sassi Hidri  Computer Department, Deanship
of Preparatory Year and Supporting Studies,
Imam Abdulrahman Bin Faisal University
Dammam, Saudi Arabia
<EMAIL_ADDRESS>
###### Abstract

Any company’s human resources department faces the challenge of predicting
whether an applicant will search for a new job or stay with the company. In
this paper, we discuss how machine learning (ML) is used to predict who will
move to a new job. First, the data is pre-processed into a suitable format for
ML models. To deal with categorical features, data encoding is applied and
several MLA (ML Algorithms) are performed including Random Forest (RF),
Logistic Regression (LR), Decision Tree (DT), and eXtreme Gradient Boosting
(XGBoost). To improve the performance of ML models, the synthetic minority
oversampling technique (SMOTE) is used to retain them. Models are assessed
using decision support metrics such as precision, recall, F1-Score, and
accuracy.

###### Index Terms:

Machine Learning, Oversampling, Dummy Encoding, SMOTE.

##  I Introduction

A number of factors, including job market competition and personal
preferences, lead to people changing jobs over the course of their careers.

However, changing jobs is a difficult decision that may be influenced by a
variety of elements, including pay, job description, and location. A
successful professional career requires making smooth job changes.

The objective of this work is to accurately predict if an applicant will move
to a new job or not using supervised Machine learning (ML) models
[AlsaifHFEH22, AlsaifHEFA22, AmmarHH20].

To evaluate the model, several different implementations of classification
were compared to determine which model suits this type of data best. These
were trained on a subset of data originating from available person profiles
collected through web scraping.

The different steps of our approach are as follows:

  * •

Data preprocessing (Data cleaning): in this step, null values are removed from
the training dataset. Input values have been changed to specific required data
types. Categorical variables are encoded to dummy variables.

  * •

Model Building: in this step, several MLAs have been performed including
Random Forest (RF), Logistic Regression (LR), Decision Tree (DT), and eXtreme
Gradient Boosting (XGBoost). Finally, SMOTE is used to improve the performance
of MLAs.

  * •

Model evaluating: in this step, models are assessed using decision support
metrics such as precision, recall, F1-Score, and accuracy.

The remainder of this paper is organized as follows. The methodology adopted
is detailed in section II. Section III evaluates the proposed predictive
model. Conclusions and directions for future work are given in Section IV.

##  II Methodology

MLAs expect inputs or features to be in a single numeric vector. Similarly,
the value to be predicted (label), especially when dealing with categorical
data, must be encoded [AmmarHH20]. Thus, one of the objectives of data
preparation is to obtain the data in the format expected by the MLAs

###  II-A Human Resources data

Publicly available Human Resources (HR) data have been used in this work. The
data has 10 features. Data features include the City Development Index, the
Gender, the Relevant Experience, the Enrolled University, the Education Level,
the Major Discipline, the Total Years of Experience, the Company Size, and the
Company Size. The last feature is the Target which indicates if the employee
seeks a new job or not (0 or 1). The employees are divided into two classes.
The first class contains all those employees who want to move to a new job and
the second class consists of all those who did not seek a new job.

Table I presents the number of instances used to train and test the model.

TABLE I: Training and Test data #Training Data (80%) | #Test Data (20%)
---|---
7164 | 1791

Information about the classes (Job seekers) and Non-job seekers) from test
Data presented in Table I is summarized in Table II.

TABLE II: Data description #Non job seekers | #Job seekers
---|---
1511 | 280

###  II-B Data encoding

A categorical variable takes on values called categories, modalities, or
levels that have no quantitative meaning. For example, the gender of an
individual is a categorical variable with two (or more) modalities: male and
female. Most categorical variables are nominal. These variables are used to
categorize and label attributes. Variables contain different values, and each
value represents a separate category.

Many MLAs are unable to deal with categorical variables. It is therefore
important to encode the data in an appropriate form in order to be able to
preprocess these variables. As you need to fit and evaluate your model, you
need to code the categorical data and convert all input and output variables
to numeric values. Thus, the model will be able to understand and extract the
information that generates the desired result. A different set of data varies
depending on the number of possible values.

Moreover, the way in which this transformation is carried out is very
important. Indeed, the coding of categorical variables generally harms the
performance of learning algorithms. One code may be better than another. For
example, RF model struggles to capture information from categorical variables
with a large number of categories if they are processed with the one-hot
encoding technique.

This is how more specific learning algorithms such as XGBoost came into
existence.

We have used different methods and tricks to manage the categorical variables
present in the dataset used, namely:

  * •

One-hot encoding: It consists of coding each categorical variable with
different Boolean variables (also called dummy variables) which take the
values 0 or 1, indicating whether a category is present in an observation.

Consider a categorical variable $X$ which admits $K$ modalities $m_{1}$,
$m_{2}$,…, $m_{K}$. One hot encoding consists of creating $K$ indicator
variables, i.e. a vector of size $K$ which has 0s everywhere and a 1 at
position $i$ corresponding to modality $m_{i}$. The categorical variable,
therefore, is replaced with $K$ numerical variables.

Some algorithms, in particular some implementations of decision tree (DT)
forests, fail to make the best use of the information contained in these
variables when this number of modalities is too large.

  * •

Reduction in the number of modalities: Business knowledge can help reduce the
number of modalities. Indeed, an understanding of the categories can allow
them to be grouped effectively. A natural grouping is done when the modalities
are hierarchical, that is to say, it is possible to define a new category that
includes other categories. Suppose a variable whose categories are the
districts of a city: these categories can for example be grouped by district,
i.e. the districts of the same district will have the same modality. This is a
fairly common case. However, these groupings can introduce a bias into the
model [3565266].

A second way to get away with a high number of categories is to try to merge
the categories with low counts. Modalities that appear very infrequently in
the data can be combined. A frequency table of the modalities is drawn up, and
those whose frequency is below a certain threshold are put together in the
same other category, for example. Then, a one-hot encoding can be applied to
the new variable.

  * •

Impact encoding: When the number of categories becomes very large, encoding by
dummy variables can become inconvenient. An alternative method to clustering
or truncation of categories consists in characterizing the categories by the
link they maintain with the target variable $y$: this is the encoding impact
[GNAT20213542].

This method is also known under the names: likelihood encoding, target coding,
conditional probability encoding, and weight of evidence.

For a regression problem with target variable $y$, let $X$ be a categorical
variable with $K$ categories $m_{1}$, $m_{2}$,…, $m_{K}$. Each $m_{K}$
category is encoded by its impact value:

$impact(m_{k})=E[y|X=m_{k}]-E[y]$ (1)

$E[y|X=m_{k}]-E[y]$ corresponds to the expectation of the target $y$ knowing
that the variable $X$ is fixed to the modality $m_{k}$. For a training set of
size $n$ containing samples $\\{(x_{i},y_{i})\\}$ independent and identically
distributed, the estimator of this expectation is the mean of the values of
$y_{i}$ for which the modality $x_{i}$ is equal to $m_{k}$:

$E[y|X=m_{k}]=\frac{1}{n_{k}}\sum_{i\in S_{k}}y_{i}$ (2)

where $S_{k}$ is the set of indices $i$ of the observations such that $x_{i}$
is equal to $m_{k}$ and $n_{k}$ the cardinality of this set.

The estimator of the expectation of $y$ is simply its empirical mean:

$E[y]=\frac{1}{n}\sum_{i}^{n}y_{i}$ (3)

  * •

Embedding methods: This method uses deep learning techniques; it draws its
inspiration from models like word2vec on textual data and which gives very
impressive results [goldberg2014word2vec].

This involves creating a representation of each modality of a categorical
variable in a numeric vector of fixed size. The use of embeddings allows among
other things a reduction of the dimensionality since the size of the vector e
can be chosen very small compared to the number of modalities.

Concretely, obtaining these embeddings is done by training a neural network
(often a multilayer perceptron) with only the categorical variables as input.
First, a one-hot encoding is applied to the variable in order to be input to
the network. Generally, one or two concealed coats are sufficient. The first
hidden layer has e neurons. The network is then trained on the same task as
that initially defined. Then, the output of the first hidden layer then
constitutes the vector of embeddings. This vector is then concatenated to the
initial data. This data is then used in the fitting of the final model. There
are various variations in the literature on how to obtain these embeddings. In
addition, nothing prevents putting more than two hidden layers and retaining
the output of the second rather than that of the first. Furthermore, the
network can be trained on a task other than the initial task.

###  II-C Handling Imbalanced Data with SMOTE

Imbalanced datasets hinder the predictive capability of ML models due to the
biased towards the majority class [j.ins.2018].

The preprocessed dataset is imbalanced with a bias towards the majority class.
To optimize the predictive capability of the ML models and enhance the
generalizability of this study, the preprocessed (imbalanced sample) data was
undersampled and oversampled to create a balanced distribution.

Undersampling involves removing cases of the majority class and oversampling
involves adding instances to the minority class [Kotsiantis]. Undersampling
and oversampling have associated benefits and costs. Undersampling can
increase accuracy by decreasing the complexity of the dataset [Faraoun2007].
Conversely, undersampling can hinder predictive performance by using a
diminutive dataset, compared to the full sample [j.ins.2018]. Oversampling
enhances predictability through increased data but can result in over-fitting
if observations are duplicated [Weiss2007].

It is common in machine learning when one of the classes has a much greater or
lesser number of observations than the other classes. In this case, the
distribution of data is unbalanced. Taking class distribution into account is
not possible with MLAs since they increase accuracy by reducing error.
Detecting fraud, anomalies, and facial recognition are common examples of this
problem.

The majority class bias in standard ML techniques like the DT and LR tends to
exclude minorities. Therefore, the minority class is often misclassified as a
majority class, due to their tendency to predict only the majority. We are
more likely to see negligible or very low recall for the minority class if we
have an unbalanced distribution of data in our dataset.

There are mainly 2 widely used algorithms to handle unbalanced class
distribution: SMOTE (Synthetic Minority Oversampling Technique) and Near Miss
Algorithm.

Oversampling SMOTE aims to balance the distribution of classes by randomly
increasing examples of minority classes by reproducing them [1622407].

The algorithm 1 illustrates the diffrent steps of SMOTE.

Algorithm 1 SMOTE Algorithm

1:Definition of the set of minority classes $A$

2:for $x\in A$ do

3: $k$ nearest neighbors of $x$ -¿ Euclidean distance between $x$ and all the
other samples of the set $A$.

4:end for

5:Adjust the sampling rate $N$ according to the unbalanced proportion.

6:for $x\in A$ do

7: Randomly choose $N$ examples (i.e. $x_{1}$, $x_{2}$,…,$x_{n}$) among its
$k$ nearest neighbors

8: Build the set $A_{1}$.

9:end for

10:for $x_{k}\in A_{1}$ $(k=1$, $2$, $3$,…,$N)$ do

11: Generate a new example: $x^{{}^{\prime}}=x+rand(0,1)*\mid x-x_{k}\mid$
where $rand(0,1)$ represents the random number between 0 and 1.

12:end for

The main idea of the NearMiss methods is to keep a set of points of the
majority class which are close to the points of the minority class, in order
to better represent the border which separates $C0$ and $C1$
[TANIMOTO2022117130].

The NearMiss-1 method selects the elements of the majority class which have
the smallest average distance with respect to the $k$ closest examples of the
minority class ($k$ being a tuning parameter to be chosen by the user). The
NearMiss-2 method retains the cases of the majority category which have the
smallest average distance with respect to the $k$ points farthest from the
minority category. Finally, the NearMiss-3 method keeps the $k$ examples of
class $C0$ which are the nearest neighbors of each element of class $C1$
[TANIMOTO2022117130].

The basic intuition about how quasi-neighborhood methods work is presented in
algorithm 2.

Algorithm 2 NearMiss Algorithm

1:Calculate the distances between the majority class and the instances of the
minority class.

2:Selects the next $n$ instances of the majority class that have the smallest
distances with those of the minority class.

3:if There are $k$ instances in the minority class then

4: The closest method will result in $k*n$ instances of the majority class.

5:end if

###  II-D Model Building

Fig. 1 shows the predictive modeling process. Let’s look at these MLAs used to
develop the predictive model.

Figure 1: Predictive modelling process.

  * •

LR [lr]: LR is a statistical model used to study the relationships between a
set of qualitative variables $X_{i}$ and a qualitative variable $Y$. It is a
generalized linear model using a logistic function as a link function.

  * •

DT [dtdt]: DT are among the most widely used non-parametric supervised
learning methods in classification and regression. On the one hand because of
their algorithmic simplicity and on the other hand, because of the ease of
interpreting them and explaining the results generated. Decision trees are
built through an algorithmic approach and can be viewed as a tree with rules
that identify ways to split a data set. The goal is to create a model that
predicts the value of a target variable by learning the decision rules.

  * •

RF [breiman2001random]: This classification algorithm reduces the variance of
predictions from a single decision tree, thereby improving their performance.
For this, it combines many decision trees in a bagging-type approach.

  * •

XGBoost [2939785]: XGBoost is an optimized distributed gradient boosting
method. In spite of the fact that Gradient Boost methods are sequential
algorithms, XGBoost uses multithread processing to search in parallel for the
best split between the features. When compared to other Gradient Boost method
implementations, XGBoost performs well due to the utilization of
multithreading.

##  III Model Evaluation

As for any ML algorithm, we need to be able to evaluate the performance of the
used MLAs in order to decide which algorithm fits our situation best. The
following metrics are used in our approach [AlsaifHH22, FerjaniHF22]:

  * •

Confusion matrix.

  * •

Precision.

  * •

Recall.

  * •

F1 - score.

  * •

Accuracy.

The goal is to predict employees who want to change jobs, so we have two
classes (0 for non-job seekers, and 1 for job seekers).

Fig. 2 shows the confusion matrix for RF (Fig. 2(a)), LR (Fig. 2(b)), DT (Fig.
2(c)), and XGBoost (Fig. 2(d)) without data balancing (SMOTE).

Figure 2: Confusion Matrix without data balancing.

Fig. 3 shows the confusion matrix for LR (Fig. 3(a)) and RF (Fig. 3(b)) with
data balancing (SMOTE).

Figure 3: Confusion Matrix with data balancing.

According to Fig. 3, SMOTE significantly improves the prediction performance.
SMOTE-LR outperforms the competition on all metrics, particularly recall
(56.34%) and accuracy (86.26%).

Table III presents the model performance given recall, precision, F1-score,
and accuracy values.

TABLE III: Model performance. MLA | Precision | Recall | F1-score | Accuracy
---|---|---|---|---
RF | 35.4% | 50.24% | 41.53% | 84%
LR | 40% | 54.9% | 46.28% | 85.48%
DT | 35.71% | 32.05% | 33.78% | 78.11%
XGBoost | 46.8% | 53.9% | 50.09% | 85.42%
SMOTE-RF | 35.71% | 51.62% | 42.22% | 84.75%
SMOTE-LR | 53.9% | 56.34% | 55.09% | 86.26%

According to Table III, SMOTE-LR gives the best recall and Accuracy.

##  IV Conclusion

In this paper, we tried to predict from HR data who will move to a new job. We
performed 6 ML models. The accuracy of each algorithm is evaluated.

The classification rate is assessed using decision ML metrics including
recall, precision, F-1 score, and accuracy. As a general rule, most models
perform well if the proportions of the classes in a dataset are relatively
similar.

Since MLAs struggle to correctly identify the minority class, slight class
imbalances were well managed using SMOTE. SMOTE-LR can be extremely useful for
HR department as it has the highest Recall.

In the future direction, we will apply the Convolutional Neural Network-based
deep learning (CNN-based DL) model to predict who has the attrition to move to
a new job or not.

## References

## References

  * [1]

myBibFile