# Machine-learning classification of astronomical sources: estimating F1-score
in the absence of ground truth

A. Humphrey1,2, W. Kuberski3, J. Bialek3, N. Perrakis3, W. Cools3, N.
Nuyttens3, H. Elakhrass3, P. A. C. Cunha1,4
1Instituto de Astrofísica e Ciências do Espaço, Universidade do Porto, CAUP,
Rua das Estrelas, Porto, 4150-762, Portugal
2 DTx – Digital Transformation CoLAB, Building 1, Azurém Campus, University of
Minho, 4800-058 Guimarães, Portugal
3NannyML NV, Interleuvenlaan 62, 3001 Heverlee, Belgium
4Faculdade de Ciências da Universidade do Porto, Rua do Campo de Alegre,
4150-007 Porto, Portugal

(Accepted 2022 September 29. Received 2022 September 28; in original form 2022
July 3)

###### Abstract

Machine-learning based classifiers have become indispensable in the field of
astrophysics, allowing separation of astronomical sources into various
classes, with computational efficiency suitable for application to the
enormous data volumes that wide-area surveys now typically produce. In the
standard supervised classification paradigm, a model is typically trained and
validated using data from relatively small areas of sky, before being used to
classify sources in other areas of the sky. However, population shifts between
the training examples and the sources to be classified can lead to ‘silent’
degradation in model performance, which can be challenging to identify when
the ground-truth is not available. In this Letter, we present a novel
methodology using the NannyML Confidence-Based Performance Estimation (CBPE)
method to predict classifier F1-score in the presence of population shifts,
but without ground-truth labels. We apply CBPE to the selection of quasars
with decision-tree ensemble models, using broad-band photometry, and show that
the F1-scores are predicted remarkably well (${\rm MAPE}\sim 10\%$,
$R^{2}=0.74-0.92$). We discuss potential use-cases in the domain of astronomy,
including machine-learning model and/or hyperparameter selection, and
evaluation of the suitability of training datasets for a particular
classification problem.

###### keywords:

methods: statistical – quasars: general – galaxies: photometry

††pubyear: 2022††pagerange: Machine-learning classification of astronomical
sources: estimating F1-score in the absence of ground truth–References

##  1 Introduction

Classification of sources is a fundamental activity in modern astronomy,
allowing objects with specific properties, or in a particular evolutionary
phase, to be selected for further study. While colour-colour methods have
traditionally been employed for object classification (e.g., Haro, 1956; Daddi
et al., 2004; Leja, Tacchella, & Conroy, 2019; Bisigello et al., 2020),
template-fitting methods have also become widely used for more detailed
classification and characterisation (e.g., Arnouts et al., 1999; Bolzonella,
Miralles, & Pelló, 2000; da Cunha et al., 2008; Ilbert et al., 2006; Laigle et
al., 2016; Gomes & Papaderos, 2017).

In recent years, supervised machine learning has seen an explosion in
popularity as a tool to classify sources into phenomenological types (e.g.,
Cavuoti et al., 2014; Bai et al., 2019; Clarke et al., 2020; Cunha & Humphrey,
2022), or into morphological classes (e.g., Dieleman, Willett, & Dambre, 2015;
Huertas-Company et al., 2015; Domínguez Sánchez et al., 2018; Tuccillo et al.,
2018; Nolte et al., 2019; Bowles et al., 2021; Bretonnière et al., 2021). The
supervised learning paradigm allows the efficient creation of prediction
functions that can be dramatically more scalable than template-fitting
methods, while often outperforming the traditional colour-colour or template-
fitting methods (Euclid Collaboration: Humphrey et al., 2022).

One potential problem with the standard supervised learning paradigm is the
often tacit assumption that there is no population shift (also known as ’data
drift’) between the training samples and the samples to be classified, and
that model performance will thus not differ between the two samples. In
layman’s terms, when this assumption is true, it means there is no significant
statistical difference between the samples. However, in real-world
applications of supervised learning, population shift is frequently present
and can cause ‘silent failure’, where model predictions are significantly
reduced in quality compared to the original validation results (e.g., El-Hay &
Yanover, 2022; Bennett et al., 2022).

The impact of population shift on machine-learning models in astronomy has
received relatively little attention to date (see, e.g., Vilalta et al., 2019;
Euclid Collaboration: Humphrey et al., 2022). However, population shift is
likely to become increasingly relevant when supervised machine learning
methods are used to classify, or to estimate redshifts and properties, of the
billions of sources that will be detected in surveys such as the Euclid Wide
Survey (Laureijs et al., 2011; Euclid Collaboration: Scaramella et al., 2021),
or the Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST: Ivezić
et al., 2019). For example, extragalactic deep-fields offer rich
multiwavelength data suitable for accurate labelling of training data (e.g.,
COSMOS: Scoville et al., 2007; Laigle et al., 2016), but their relatively
small area may result in population shift with respect to other fields, due to
cosmic or sample variance, and/or due to in-built biases (e.g., bright
foregound objects may be under-represented by design). Therefore, it is
crucial to identify methods to detect and overcome the impact of population
shift on supervised machine learning in astronomy.

Various previous methods exist for the detection of population shifts, or to
assess prediction quality (e.g., Singh Sethi & Kantardzic, 2017; Jiang et al.,
2018; White et al., 2019; Malinin et al., 2020; Angelopoulos & Bates, 2022).
However, the shifts detected by these methods do not necessarily correlate
with changes in model performance, since population shift does not always lead
to a change in model performance; furthermore, reliable estimates of model
performance, in terms of classification metric values, are not provided. Thus,
there is a critical need for a methodology that can reliably estimate metrics
of machine learning model performance, when the test-set ‘ground-truth’ labels
are not available, allowing model failure to be identified and quantified, and
allowing also decisions to be taken concerning the suitability of a model or
training dataset for a particular classification problem.

In this Letter, we explore the application of the NannyML Confidence-Based
Performance Estimation (CBPE) method for the estimation of classification
model performance, in the context of the classification of astronomical
sources. This Letter is structured as follows. In $\S$2, we define the
performance metrics used. Next, in $\S$3 we describe the CBPE methodology for
estimating model performance in the absence of ground-truth. We describe in
$\S$4 the construction of the dataset used in this study, In $\S$5, we explore
the application of CBPE to the identification of quasars from broad band
photometry. We briefly discuss the results and draw conclusions in $\S$6.

##  2 Metric definitions

###  2.1 Classification

To evaluate our classification results, we use the F1-score metric (Dice,
1945; Sørensen, 1948), which is the harmonic mean of the precision and recall.
The F1-score is calculated as:

${\rm F1}=\frac{{\rm TP}}{{\rm TP}+0.5({\rm FP}+{\rm FN})}\,,$ (1)

where TP is the number of true positives, FP is the number of false positives,
and FN is the number of false negatives. In addition, the number of true
negatives will be referred to by the abbreviation TN. The F1-score can have
values between 0 and 1, with 1 being the best score.

###  2.2 Regression

To evaluate the quality of F1-score estimates, we use two regression metrics.
The coefficient of determination $R^{2}$ is calculated as

$R^{2}=1-\frac{\sum\limits_{i=1}^{n}(\hat{y_{i}}-y_{i})^{2}}{\sum\limits_{i=1}^{n}(y_{i}-\bar{y})^{2}}\,,$
(2)

where $y_{i}$ is the true value of the F1-score of the $i$-th sample,
$\bar{y}$ is $y_{i}$ averaged over $n$ samples, and $\hat{y_{i}}$ is the
predicted value. Higher values of $R^{2}$ indicate a better model, up to a
maximum value of 1, with no minimum value. The $R^{2}$ score indicates how
much of the variance in $y$ is explained by the independent variables present
in the model. It is both scale and dataset dependent, which means it is often
not possible to compare $R^{2}$ scores between different datasets, or even
different subsets of the same dataset.

In addition, we use the mean absolute fractional error (MAFE), which is
calculated as:

${\rm
MAFE}=\frac{1}{n}\sum\limits_{i=1}^{n}\left|\frac{\hat{y_{i}}-y_{i}}{y_{i}}\right|\,.$
(3)

MAFE is scale independent, since it provides a measure of relative error. Note
that MAFE is identical to the more commonly used mean absolute percentage
error (MAPE) divided by 100.

##  3 Confidence-based performance estimation

###  3.1 Calculation method

The CBPE method for binary classification is part of the open-source
NannyML111https://github.com/NannyML/nannyml library for Python. In brief,
CBPE uses the confidence (or lack thereof) of a classifier in its ability to
correctly assign the correct class to test-set examples, taking advantage of
the fact that the expected quality of the classification is encoded within the
class probabilities that are predicted by the model. The method takes
calibrated binary class probabilities as input, and predicts the confusion
matrix (i.e., ${\rm\widehat{TP}}$, ${\rm\widehat{TN}}$, ${\rm\widehat{FP}}$,
and ${\rm\widehat{FN}}$), as:

${\rm\widehat{TP}}=\sum{\\{\,\hat{y}\,|\,\hat{y}\geq t\,\\}\ }\,,$ (4)

${\rm\widehat{TN}}=\sum{\\{\,1-\hat{y}\,|\,\hat{y}<t\,\\}\ }\,,$ (5)

${\rm\widehat{FP}}=\sum{\\{\,1-\hat{y}\,|\,\hat{y}\geq t\,\\}\ }\,,$ (6)

${\rm\widehat{FN}}=\sum{\\{\,\hat{y}\,|\,\hat{y}<t\,\\}\ }\,,$ (7)

where $\hat{y}$ is the calibrated probability given by the model of an example
being in class 1, and $t$ is the probability threshold used to define the
boundary between class 0 and class 1. Equations 4-7 can be summarized as
summations of the predicted class probabilities at and above, or below, the
probability theshold $t$.

Subsequently, the F1-score is predicted using equation 1. While it is possible
to predict other metrics, such as precision, recall, specificity, etc., in the
interest of simplicity we restrict the present study to F1-score.

###  3.2 Assumptions

The CBPE method relies on frequentist inference to estimate model performance,
with the following main assumptions. Firstly, it is assumed that the model
scores used as input are calibrated probabilities. While classification
algorithms generally provide class scores in the range 0-1, these scores often
cannot be treated as class probabilities because the training process usually
prioritises the minimization of a loss function, with other considerations
(i.e., probability calibration) being typically less important. This is true
for the learning algorithms used herein, and many other commonly used machine-
learning classification methods. Thus, if a classifier outputs scores that are
not calibrated class probabilities, then calibration must be performed prior
to the application of the CBPE method.

Second, it is assumed that concept drift222We define ‘concept drift’ as a
change in the mapping between the features and the labels, such that two
datasets with statistically identical features have statistically different
labels. This is in contrast to population shift (or ‘data drift’), where the
statistical properties of the features change without changing the mapping
between the features and labels. These terms are sometimes, erroneously, used
interchangeably in the literature. (see, e.g., Bayram et al., 2022) does not
take place between the training data and the data subset for which the model
performance is to be estimated (i.e., the test set). When concept drift is
present, the CBPE method may underestimate changes in model performance, since
it only estimates the performance change due to population shift. Indeed, our
preliminary experiments with other real-world datasets have shown that CBPE
demonstrates reduced predictive accuracy when concept drift is present.

##  4 The dataset

The sample used for this study was selected from the Sloan Digital Sky Survey
(SDSS; Gunn et al., 1998), using an SQL query via the SDSS SciServer CasJobs
system. We selected the top 10,000,000 sources in DR15 meeting the criteria
mode = 1, sciencePrimary = 1, and clean = 1. Sources with any missing
photometry measurements were removed. The data were subsequently undersampled
such that 9 in 10 rows were dropped, resulting in a dataset containing 351,089
sources (209,266 galaxes; 81,536 stars; 60,287 QSOs), allowing efficient model
training on a ‘typical’ laptop computer. The original dataset, along with all
scripts used in this work, are publicly available on
GitHub333https://github.com/humphrey-and-the-machine/f1_prediction..

Rudimentary feature engineering was performed using the modelMag_u,
modelMag_g, modelMag_r, modelMag_i, modelMag_z, and fiberMag_r photometry
measurements. First, we calculated all unique modelMag colour combinations
(e.g., $u-g$ … $i-z$). To obtain a feature with sensitivity to source size, we
also computed modelMag_r-fiberMag_r. These quantities, together with the
$u$,$g$,$r$,$i$,$z$ values of fiberMag, petroMag, modelMag, and psfMag, were
used as the input features for model training (31 in total). Finally, the
dataset was sorted in order of ascending right ascension (RA).

The classification problem considered herein is the selection of quasars using
broad band photometry. Thus, the target variable is a binarized version of
class, where QSOs are assigned binary class 1, and stars or galaxies are
assigned 0.

Fig. 1 shows some of the general characteristics of the sample. As expected,
there are substantial population shifts, with the balance between stars,
galaxies, and quasars showing a strong evolution with RA (Fig. 1, top panel).
In particular, the number of stars relative to other sources is dramatically
higher in the ranges $50\la{\rm RA}\la 110$ deg and $250\la{\rm RA}\la 340$
deg, due to the presence of the Milky Way. Significant shifts are also
apparent in the colours of the three different classes of source (Fig. 1,
lower panel).

Figure 1: General characteristics of the dataset as a function of RA. Top: The fraction of sources in each of the three classes, versus RA. Bottom: The median $g-i$ colour of each class versus RA. The shaded areas show the $\pm 1\,\sigma$ range in $g-i$ for each of the three classes. In both cases, each data point represents a rolling window with a length of 1000 objects, and a step size of 1 object. Table 1: Hyperparameters used to train the classification models. Parameters not listed here kept their default values. Learning algorithm | Hyperparameter | Value
---|---|---
RandomForest | n_estimators | 100
| max_depth | 4
XGBoost | n_estimators | 300
| max_depth | 4
| tree_method | hist
| eval_metric | map
LightGBM | n_estimators | 1000
| max_depth | 10
| learning_rate | 0.01
| colsample_bytree | 0.9
| subsample | 0.8

##  5 Model performance prediction

###  5.1 Model training

Models were generated using three different machine learning algorithms: the
popular RandomForestClassifier method (Breiman, 2001) from the Scikit-Learn
package (Pedregosa et al., 2011), and the gradient-boosting decision tree
methods XGBoostClassifier444https://xgboost.readthedocs.io (Chen & Guestrin,
2016), and LightGBMClassfier555https://lightgbm.readthedocs.io (Ke et al.,
2017).

Since objective of this study is to explore the application of CBPE, rather
than to identify optimal learning algorithms or hyperparameters for the
selection of quasars, the most important hyperparameters were tuned manually
and non-exhaustively, with the objective of producing classification models
that are of an acceptably high-quality. A summary of the hyperparameters used
for model training is given in Table 1.

The models were trained on 1/40 of the sources in the range $150<{\rm RA}<200$
deg. Sources used for model training were not used at any subsequent stage in
the workflow.

###  5.2 Probability calibration

The aim of calibration is to derive and apply a transformation to class
probabilities such that the transformed values correspond to the true
probability of belonging to the class in question. For instance, objects
assigned a class probability of 0.80 should have an 80 per cent probability of
belonging to class 1. However, class probabilities produced by many
classifiers are biased and poorly calibrated. Thus, as discussed in $\S$3.2,
class probabilities must be calibrated before the CBPE is applied.

While there are various different methods to calibrate class probabilities
(see, e.g., Niculescu-Mizil & Caruana, 2005; Kull et al., 2017), we have used
the isotonic regression method, which is appropriate for relatively large
datasets (≳1000 examples). This method fits a non-parametric, isotonic
regressor, which yields a step wise non-decreasing function.

In the interest of simplicity and reproducibility, we have used the Scikit-
Learn function CalibratedClassifierCV, which produces a calibration
transformation simultaneously with the training of the classifier, using
5-fold cross-validation. Note that the same samples cannot be used to train
the classifier and compute the calibration transform.

###  5.3 Results

For each source in the dataset that was not used for model training, the
binary class and the class probability were predicted, using each of the three
models. The F1-score versus RA is shown in Fig. 2 (solid blue line). The RA
range from which the training sample was drawn is highlighted with cyan
shading.

All three models show roughtly similar behaviour of the F1-score with respect
to RA: the F1-score remains near to $\sim$0.8 over most of the RA range, but
drops dramatically to $\sim 0.2$ near to ${\rm RA}\sim 90$ deg and ${\rm
RA}\sim 310$ deg. Clearly, the population shifts seen in Fig. 1 have resulted
in decreased classification performance in RA ranges where the number of stars
is high relative to galaxies and QSOs.

Next, we apply the CBPE methodology as outlined in $\S$3 to the predictions
from the classifier models. The resulting F1-score estimates are shown by the
orange dashed lines in Fig. 2. The estimated score follows the measured score
relatively well, with $R^{2}$-scores of 0.92, 0.76, 0.74, and MAFE scores of
0.12, 0.11, 0.12, for the RandomForestClassifier, XGBoostClassifier, and
LightGBMClassifier cases, respectively. Interestingly, the CBPE method is able
to predict the dramatically low F1-scores at ${\rm RA}\sim 100$ and ${\rm
RA}\sim 300$ deg, but also predicts the shallow gradient in the F1-score in
the range $125<{\rm RA}<250$ deg, and many of the small fluctuations.

Figure 2: General characteristics of the dataset as a function of RA. Top:
Estimated and actual F1-scores versus RA, for the RandomForest model. Middle:
The same, but for the XGBoost model. Bottom: The same, but for the LghtGBM
model. The metrics are calculated in a rolling window of length 1001 objects,
with results from each window plotted at the RA value that corresponds to the
central element of the bin.

##  6 Discussion and conclusions

We have introduced the NannyML CBPE method for the estimation of machine
learning model performance in the absence of ground-truth labels. We have
demonstrated that CBPE is able to predict population shift-driven changes in
the performance of machine-learning models, using the selection of QSOs from
SDSS photometry as an example use-case.

A number of interesting implications arise from this study. Among them is the
potentially game-changing possibility to predict the classification
performance of a model in the absence of ground-truth labels, even in the
presence of substantial population shifts between the training and test sets.
Depending on the experimental set-up employed, the application of CBPE should
allow model-selection to be performed such that models (or hyperparameters)
can be chosen to maximise the expected performance in unlabelled (test) data,
in contrast to the traditional paradigm where performance is optimized using
labelled validation data.

Another potentially fruitful use-case we envisage is the detection of
population shifts in large astronomical datasets, and quantification of their
impact on classification metrics (e.g., Fig. 2). We also argue that such
analyses may be useful when planning wide-field surveys in which machine
learning classifiers are expected to be used. For example, models trained on
different deep, pencil-beam surveys might be evaluated on their performance
over a wider survey area using CBPE, in order to select the model (or training
sample) that is estimated to provide the highest quality result for a desired
classification objective.

While the CBPE method has proven useful to predict changes in model
performance under population shift, it is important to note that under concept
drift, where changes occurs in the mapping between the features and the
target, the performance estimates may be less accurate.

Finally, we acknowledge that the work presented in this Letter represents a
restricted subset of the possible use-cases of the CBPE method in the field of
astronomical source classification. A more detailed analysis and application
to a wider range of classification problems is beyond the scope of this
Letter, and will be the subject of a future paper.

## Acknowledgements

This work was supported by Fundação para a Ciência e a Tecnologia (FCT)
through grants UID/FIS/04434/2019, UIDB/04434/2020, UIDP/04434/2020 and
PTDC/FIS-AST/29245/2017, EXPL/FIS-AST/1085/2021, and an FCT-CAPES
Transnational Cooperation Project. AH also acknowledges support from NVIDIA in
the form of a GPU under the NVIDIA Academic Hardware Grant Program. In the
development of this work, we have made use of the Pandas (McKinney, 2010),
Numpy (Harris et al., 2020), Scipy (Virtanen et al., 2020), Dask (Rocklin,
2015) and NannyML packages for Python.

## Data Availability

The Python and SQL scripts used in this study are publicly available at
https://github.com/humphrey-and-the-machine/f1_prediction.

The data were collected from the SDSS SkyServer / SciServer / CASJobs portal
at https://skyserver.sdss.org/casjobs/. A CSV file containing the dataset used
has been made available online, hosted by MNRAS.

## References

  * Angelopoulos & Bates (2022) Angelopoulos, A. N., Bates, S. 2022, ArXiv e-prints, ArXiv:2107.07511
  * Arnouts et al. (1999) Arnouts, S., Cristiani, S., Moscardini, L., et al. 1999, MNRAS, 310, 540
  * Bai et al. (2019) Bai, Y., Liu, J., Wang, S., et al. 2019, AJ, 157, 9
  * Bayram et al. (2022) Bayram, F., Ahmed, B. S., Kassler, A. 2022, ArViv e-prints, ArXiv:2203:11070
  * Bennett et al. (2022) Bennett, M., Balusu, J., Hayes, K., Kleczyk, E. J. 2022, ArXiv e-prints, ArXiv:2204.10227
  * Bisigello et al. (2020) Bisigello, L., Kuchner, U., Conselice, C. J., et al. 2020, MNRAS, 494, 2337 (B20)
  * Bolzonella, Miralles, & Pelló (2000) Bolzonella, M., Miralles, J.-M., & Pelló, R. 2000, A&A, 363, 476
  * Bowles et al. (2021) Bowles, M., Scaife, A. M. M., Porter, F., Tang, H., & Bastien, D. J. 2021, MNRAS, 501, 4579
  * Breiman (2001) Breiman, L. 2001, Mach. Learn., 45, 1
  * Bretonnière et al. (2021) Bretonnière, H., Boucaud, A., & Huertas-Company, M. 2021, ArXiv e-prints, ArXiv:2111.15455
  * Cavuoti et al. (2014) Cavuoti, S., Brescia, M., D’Abrusco, R., Longo, G., & Paolillo, M. 2014, MNRAS, 437, 968
  * Chen & Guestrin (2016) Chen, T., Guestrin, C., 2016, ArXiv e-prints, ArXiv:1603.02754v3
  * Clarke et al. (2020) Clarke, A. O., Scaife, A. M. M., Greenhalgh, R., & Griguta, V. 2020, A&A, 639, A84
  * Cunha & Humphrey (2022) Cunha, P. A. C. & Humphrey, A. 2022, ArXiv e-prints, ArXiv:2204.02080. doi:10.1051/0004-6361/202243135
  * da Cunha et al. (2008) da Cunha, E., Charlot, S., & Elbaz, D. 2008, MNRAS, 388, 1595
  * Daddi et al. (2004) Daddi, E., Cimatti, A., Renzini, A., et al. 2004, ApJ, 617, 746
  * Dice (1945) Dice, L. R., 1945, Ecology, 26(3), 297
  * Dieleman, Willett, & Dambre (2015) Dieleman, S., Willett, K. W., & Dambre, J. 2015, MNRAS, 450, 1441
  * Domínguez Sánchez et al. (2018) Domínguez Sánchez, H., Huertas-Company, M., Bernardi, M., et al. 2018, MNRAS, 476, 3661
  * El-Hay & Yanover (2022) El-Hay, T., Yanover, C., Proceedings of the Conference on Health, Inference, and Learning. PMLR, 2022, p. 48-62. ArXiv e-prints, ArXiv:2202.13683
  * Euclid Collaboration: Humphrey et al. (2022) Euclid Collaboration, Humphrey, A., et al. 2022, A&A, accepted, ArXiv e-prints, ArXiv:2209.13074
  * Euclid Collaboration: Scaramella et al. (2021) Euclid Collaboration, Scaramella, R., Amiaux, J., et al. 2022, A&A, 662, A112
  * Fawcett (2006) Fawcett, T., Pattern Recognition Letters 27, 2006, 861–874
  * Gomes & Papaderos (2017) Gomes, J. M., & Papaderos, P. 2017, A&A, 603, A63
  * Gunn et al. (1998) Gunn, J. E., Carr, M., Rockosi, C., et al. 1998, AJ, 116, 3040
  * Haro (1956) Haro, G. 1956, Boletin de los Observatorios Tonantzintla y Tacubaya, 2, 8
  * Harris et al. (2020) Harris, C. R., Millman, K. J., van der Walt, S. J., et al. 2020, Nature, 585, 357
  * Huertas-Company et al. (2015) Huertas-Company, M., Gravet, R., Cabrera-Vives, G., et al. 2015, ApJS, 221, 8
  * Ilbert et al. (2006) Ilbert, O., Arnouts, S., McCracken, H. J., et al. 2006, A&A, 457, 841
  * Ivezić et al. (2019) Ivezić, Ž., Kahn, S. M., Tyson, J. A., et al. 2019, ApJ, 873, 111
  * Jiang et al. (2018) Jiang, H., Kim, B., Guan, M. Y., Gupta, M., 2018, 32nd Conference on Neural Information Processing Systems (NIPS 2018), Montréal, Canada. ArXiv e-prints, ArXiv:1805.11783
  * Ke et al. (2017) Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.-Y., 2017, LightGBM: A Highly Efficient Gradient Boosting Decision Tree, in Advances in Neural Information Processing Systems, vol. 30, 3146
  * Kull et al. (2017) Kull, M., Silva Filho, T. M., Flach, P., 2017, Electron. J. Statist., 11 (2), 5052-5080
  * Laigle et al. (2016) Laigle, C., McCracken, H. J., Ilbert, O., et al. 2016, ApJS, 224, 24
  * Laureijs et al. (2011) Laureijs, R., Amiaux, J., Arduini, S., et al. 2011, ArXiv e-prints [[arXiv]1110.3193]
  * Leja, Tacchella, & Conroy (2019) Leja, J., Tacchella, S., & Conroy, C. 2019, ApJ, 880, L9
  * Malinin et al. (2020) Malinin, A., Prokhorenkova, L., Ustimenko, A. 2020, ArXiv e-prints, ArViv:2006.10562
  * McKinney (2010) McKinney, w., 2010, Data Structures for Statistical Computing in Python, in Proceedings of the 9th Python in Science Conference, pp 51
  * Niculescu-Mizil & Caruana (2005) Niculescu-Mizil, A., Caruana, R., 2005, in Proceedings of the 22nd International Conference on Machine Learning, Bonn, Germany
  * Nolte et al. (2019) Nolte, A., Wang, L., Bilicki, M., Holwerda, B., & Biehl, M. 2019, arXiv e-prints, ArXiv:1903.07749
  * Pedregosa et al. (2011) Pedregosa, F., et al., 2011, Journal of Machine Learning Research, 12, 2825
  * Rocklin (2015) Rocklin, M., 2015, Dask: Parallel Computation with Blocked Algorithms and Task Scheduling, in Proceedings of the 14th Python in Science Conference, pp 130
  * Scoville et al. (2007) Scoville, N., Aussel, H., Brusa, M., et al. 2007, ApJS, 172, 1
  * Singh Sethi & Kantardzic (2017) Singh Sethi, T., Kantardzica, M. 2017, ArXiv eprints, ArXiv:1704.00023
  * Sørensen (1948) Sørensen T. 1948, Kongelige Danske Videnskabernes Selskab, 5(4): 1
  * Tuccillo et al. (2018) Tuccillo, D., Huertas-Company, M., Decencière, E., et al. 2018, MNRAS, 475, 894
  * Vilalta et al. (2019) Vilalta, R., Dhar Gupta, K., Boumber, D., et al. 2019, PASP, 131, 108008
  * Virtanen et al. (2020) Virtanen, P., Gommers, R., Oliphant, T. E., et al. 2020, Nature Methods, 17, 261
  * White et al. (2019) White, C., Neiswanger, W., Savani, Y. 2019, 4th workshop on Bayesian Deep Learning (NeurIPS 2019), Vancouver, Canada. E-print: http://bayesiandeeplearning.org/2019/papers/26.pdf


# Automatic Procurement Fraud Detection with Machine Learning

Jin Bai, Tong Qiu

###### Abstract

Although procurement fraud is always a critical problem in almost every free
market, audit departments still have a strong reliance on reporting from
informed sources when detecting them. With our generous cooperator, SF
Express, sharing the access to the database related with procurements took
place from 2015 to 2017 in their company, our team studies how machine
learning techniques could help with the audition of one of the most profound
crime among current chinese market, namely procurement frauds. By representing
each procurement event as 9 specific features, we construct neural network
models to identify suspicious procurements and classify their fraud types.
Through testing our models over 50000 samples collected from the procurement
database, we have proven that such models – despite having space for
improvements – are useful in detecting procurement frauds.

## Keywords

auditing, procurement fraud, machine learning, artificial neural network

##  1 Introduction

Procurement fraud, sometimes called contract fraud, is believed by
professionals to be one of the most common and costly white-collar crime. [1]
It is defined to be an intentional act by one or more individuals among
management, those charged with governance, employees, or third parties,
involving the use of deception to obtain an unjust or illegal advantage. [2]

Fraudulent process in procurement has been bothering various corporations all
over the world, including governmental departments [3] for a long time.
Problems such as collusion between bidders and bid inviters, buyers’
acceptance of bribes and creations of fictitious transactions all contain
great possibilities of causing both financial and assets damages to the
corporations who desire purchases of products from suppliers in the most
economical way.

As stated by Ying, vice president of Higher Education Forensic Accounting
Professional Core Course Textbook Editorial Board, despite the ubiquity and
profoundness of procurement fraud, the main auditing methods used by most
audit departments are still book audit and reports reviewing. [4] During our
contact with SF Express, such facts were admitted by the manager of auditing
department of SF Express as well. In other words, even being the greatest
express company in China, SF Express still lack the ability to execute
initiative fraud auditing, not to mention other corporations who do not have
equal scales as SF Express. After detailed communication with SF auditing
department, we arrived at the conclusion that such currently unavoidable
dependence on informers is mainly caused by both the huge amount of
procurements operated in the company annually and the complicated steps needed
in the procurement process, including bidding, enquiry, contract management
and order management. Apparently, such enormous amount and complication are
making the department unable to carefully audit procurements one by one with
mere human power.

Considering the need for handling data of large scale and relationship between
different steps and elements in a complete procurement process, we figured out
that a machine learning model established with artificial neural network
algorithms could be a handful tool. We decided to rely on computers’ great
ability of process large-scale data and take advantages of specialties of a
artificial neural network including its ability to perform nonlinear computing
to deal with the complex relationships between different steps and elements.
We expected our model to have the ability to inform its users the probability
that one procurement involves fraudulence after a set of data related to that
procurement was inputted.

##  2 Background

###  2.1 History of Machine Learning

Machine learning refers to the study of giving computers “the ability to learn
without being explicitly programmed”. [5] Growing from the field of artificial
intellegence, machine learning researches gradually shifted their focus from
serving as a tool for AI to solving practical problems using models from
probability theory and statistics. [6] Main types of problems for machine
learning include classification, regression, clustering and estimation, which
relate very well with our objectives of procurement fraud detections.

Neural networks arise from a branch of machine learning called deep learning,
which contains multiple hidden layers in the learning models – in contrast to
only one layer in traditional (shallow) models. Compare to the shallow ones,
deep learning models are more capable to handle handle areas superior to human
insights, but also require more computing power in general. [7] The invention
of Nvidia’s CUDA framework on its Graphical Processing Units (GPUs) in late
2000s resulted in the “big bang” of deep learning, as it continues to become a
very hot and trending subject in an enormous number of disciplines. [8]

###  2.2 Prior Work on Machine Learning

Various efforts have already been made to take advantage of machine learning,
especially neural networks, in the area of fraud detection. The most typical
case is the detection of frauds in credit card transactions. A fair number of
scholarly articles have supported the idea that a neural network is a feasible
approach in credit card fraud detection. [9] In addition, Kaggle, a world-
famous data science website, recently hosted a machine learning challenge in
exactly this area. [10] Based on a pre-collected, pre-labelled dataset of
transactions, participants in this challenge are required to come up with the
best machine learning model to identify frauds as accurately as possible.

These articles and events can benefit us a lot when moving this idea into the
area of procurement auditing. They not only build up our confidence that our
approach has a high chance to succeed in auditing, but also provide us with
good neural network models or other learning algorithms as a starting point.
On the other hand, since the Kaggle challenge relies on a detailed and
organized dataset, and since the purpose of the challenge is to find the best
machine learning model, we should always be aware of the two key components in
this study – reliable and comprehensive data and a carefully structured model.

###  2.3 Prior Work on Procurement Fraud Detection

As pointed out by Hugo, Badenhorst-Weiss and Van Rooyen, compared with
researches on other business functions, there has been a shortage of study on
procurement fraud from the risk management perspective.[11] Moreover, the
power of machine learning has not yet been applied to procurement fraud
detection and prediction at the time this study is conducted.

How to actively and effectively detect procurement has been a problem
bothering enterprises and governments for a long time. Previous studies of
procurement fraud investigation stated that purely manual investigation has an
overwhelmingly high demand of well-trained investigators that could be
unaffordable for even government departments. [12]

Prior researchers have also built risk management models for procurement
frauds and successfully proved these models to be effective. [13] However,
they still fail to put auditors on the aggressive side of investigation.

Last but not least, the study group designs the research to be Chinese-market-
oriented due to the fact that all data we use is from a Chinese enterprise. We
base our understanding of procurement fraud on Chinese laws [14] as well as
studies of procurement auditing by other Chinese scholars. [15][16]

##  3 Data Specification

###  3.1 Data Collection

As we are expecting a machine learning model to be used in procurement
auditing, a database containing historical records of real procurements
conducted in one company is indeed necessary for machine training. Therefore,
in order to make sure our machine could be “well trained”, our first step is
to collect data and establish a database with both huge enough scale and a
balanced ratio between the amounts of positive cases (procurements that
involve fraudulence) and negative cases (procurements that do not involve
fraudulence).

Being our cooperator, SF Express provides us with access to records of all
their previously audited procurements dated after January 1st, 2015, as well
as all procurements take place after January 1st, 2015, whose relevant
information is stored in the systems applications and products database (SAP
database).

Considering the extreme asymmetry between the number of positive and negative
cases, the research team decides to alleviate such asymmetry by conducting
sampling on positive cases. As we have around 25000 negative cases in total,
we also pulled 25000 positive cases out of the SAP database, to ensure a $1$
to $1$ ratio. When sampling positive cases, we first choose 5 random dates
between 2015.1.1 and 2017.5.31 and make sure none of the 25000 fraudulence
procurements take place on these dates. We then manually collect 5000
procurements from all procurements happened on each 5 dates and summarize them
into one sheet using Arbutus Analyzer.

For all variable listed below(see _3.3 Input Variables_), we directly used the
raw data stored in SF’s SAP system in order to test the ability of the machine
to function efficiently in real business and to test its multiusability in
different businesses. However, data inputted will be further normalized during
the process of machine learning.(see _5.1 Implementation Details_)

###  3.2 Choice of Variables

As SF Express only provides us with the access to the user interface of SAP
system, other than the admin zone of it, we have no choice but to limit the
number of variables used in machine learning. The research team decide to
choose input variables which satisfy the criterions listed by Robert May,
Graeme Dandy and Holger Maier [17]. That is, all variables must be chosen with
the following five factors considered:

#### _Relevance_

Apparently, all input variables of the model must contain their own relevance
with the output. In our research, input variables need to have at least one
feature that could be the direct cause of fraud or an obvious clue of fraud
detection. Nevertheless, we also predict some variables might not be proved to
have such required features as they were previously studied by linear models
even though non-linear models as Neural Network may actually discover such
features of these variables.

#### _Computational Effort_

With the increase of number of input variable, the server will have to bear
more computational burden. The most direct effect of such burden would be the
substantial expansion of computing time. In pratice, considering the concrete
use of our model, efficiency should always be a necessary trait as the model
is designed mainly for business use. After all, there is no point of using a
machine learing model if it is even less efficient than human censoring.

#### _Training Difficulty_

Another problem that could be caused by the increasing number of variables is
the difficulty for Nueral Network builders to sufficiently train their models.
Redundant and irrelevant variables can slower the training speed since they
increase the number of possible combinations of parameters, which will create
locally optimal error values. Also, the fact that such variables are redundant
and irrelevant can result in longer time for the machine to recognize their
relationship with the error and to successfully map such ambiguous
relationship. Unfortunately, in our study, redundant and irrelevant facts
about procurements occupy a great amount of the whole SF database and thus
result in the very limited number of variables eventually put into study. This
problem will be further discussed in our expectation of future studies.

#### _Dimensionality_

A critical fact that one should learn about artificial neural network is the
_curse of dimensionality_ [18], the fact that as the dimensianlity of the
model increases linearly, the total volumn of the domain of the modelling
problem would increase exponentially. As the completion of one procurement
requires step by step operation that involves the supplier, the purchaser and
the regulatory authorities, the establishment of a multi-dimensional model is
indeed unavoidable. In the study, our team decides to establish a total of
four dimensions for the training machine, including: 1. numerical data of the
procurement 2. information about the supplier 3. information about the
purchaser 4. the property of fraud.

#### _Comprehensibility_

While early researchers and modellers like to refer to artificial neural
network as a ”black box” [19], recent studies ask models to be more self-
explanatory. In particular, the fulfillments of the following 3 purposes are
required:

  1. i.

The inputs should have a certain domain that produce certain outputs, which
can be useful knowledge in the neural network itself.

  2. ii.

The model should be able to verify that the response trends between the input
and output data make sense.

  3. iii.

The model should be able to discover new relationships between inputs and
outputs, which reveal previously unknown insights into the underlying physical
process. [20]

###  3.3 Input Variables

  1. i.

Procurement Serial Number (PSN): This is the serial number used in SAP system
to identify each procurement. With each serial number, details of individual
procurement could be traced. We choose serial number as a variable because
when series of fraud take place, they could have similar or consecutive serial
numbers. Another reason to choose this variable is that similar or consecutive
serial numbers are capable of leading auditors to procurements which belongs
to the same procurement contract or procurement program.

  2. ii.

Procurement Group Number (PGN): The very basic logic to choose procurement
group number as an input variable is that crime and deviance have the
likelihood to repeat in specific circumstances or neighborhoods. [21] With the
memory function of machine learning, we expect the machine to conclude which
type of procurement group would have a higher chance of committing fraud. On
the other hand, superior’s negligence is a cause of fraud. Therefore, we
assume a specific procurement group can have more possibility to conduct fraud
than another group.

  3. iii.

Procurement Organization Number (PON): Procurement organization number as a
variable is chosen for almost the same reason as procurement group number. In
previous procurement fraud studies, researches proved the importance of
management in fraud prevention. [22] Unlike procurement group number, which
refers more to a specific group who organize the procurement in detail,
procurement organization number is rather a representation of the manager who
leads the groups.

  4. iv.

Material Group Number (MGN): We decide to include this input variable based on
the assumption that certain types of fraud like receiving kickbacks are more
likely to happen during the purchase of specific types of products. What’s
more, the material group that a product belongs to also contains information
of both the product itself and its possible compliments and substitutes. Such
information could be helpful in further building of the model when
computations of price elasticities are used to judge the legitimacy of one
procurement.

  5. v.

Net Price (NP): People’s intention to conduct procurement fraud is related
with the profit they could get from such action, which is then related to the
net price of the product being purchased. In early studies of fraud auditing,
net price is proved to have direct relationship with the occurence of fraud as
well as the shrinkage of a company’s economic benefit. [23]

  6. vi.

Purchase Amount (PA): One forgettable fact that is related to procurement
fraud is that fraudsters can gain a large amount of profit even when his (or
her) profit per unit is negligible. Especially in large scale companies who
execute a substantial amount of purchases monthly, a fraudster gaining a
little profit per unit can result in huge losses for the company. Also,
according to experiences of workers of SF auditing department, smallness of
unit profit usually increases the difficulty of manual auditing. However, that
would surely not bother a model who has hundreds of millions of times
computing power.

  7. vii.

Procurement Total Price (PTP): This is an inclusive representation of the
above two factors, which more generally reflect the size of one business. It
will not only provide the machine a detailed insight of how scale of business
is related with possibility of fraud but also give the machine a better idea
on the classification of each case. (See below, _Fraud Type_)

  8. viii.

Fraud Type (FT; If no fraud is contained, this value would be set to 0): We
train the machine with fraud cases that are already classified so that it
would have the ability to predict not only the possibility of fraud but also
what kind of fraud one specific case might belong to. In practice, this
feature would make auditors more efficient when they perform more detailed
examination of a procurement at a later time. Also, this variable would help
the machine discover special relationships between one type of fraud and other
parameters, which exactly meets the need of the 3rd standard of
comprehensiveness listed in _Choice of Variables_.

  9. ix.

Supplier Serial Number (SSN): Previous studies on procurement fraud have found
fraud undetectable because it usually involves collusion between members of
staff and suppliers. [24] With supplier being one important part (as both
conductor and benefiter) of conducting procurement, we expect suppliers who
have a crime history to have more chances to break the law again. In the logic
of machine learning, we feed the machine with suppliers who have crime
histories so that it would automatically build a blacklist which makes the
machine more ”cautious” when same suppliers appear again.

Below is a sample input chart to help readers better understand what our input
actually look like.

Figure 1: Input sample

##  4 Neural Network Model

###  4.1 From Basic Concepts to MLP

A neural network consists of several layers that manipulate on multi-
dimensional arrays of data, often called tensors. [25] Each layer can be
viewed as a function, whose both input and output are tensors of particular
shapes. Internally, it has a set of parameters, which, given an input, compute
the output based on a generalized version of matrix multiplication. Multiple
such layers are stacked together, so that an input tensor of data is “fed
forward“ through these layers to produce the output, which is distinct enough
for a machine to generate different predictions from different inputs.

The training of a neural network is, therefore, a process to improve the
parameters of the layers, so that the output could lead to the right
prediction of a given task. Given an input to the model, the prediction
generated by the output of the network is compared to the “label” – the ground
truth corresponding to that input. If the results do not coincide, a “loss
function” is calculated and applied to every layer to compute its “gradient” –
the optimal direction to update this layer so it can generate outputs for more
accurate predictions. The layer is then slightly adjusted along the gradient
direction, a process called “gradient descent”. [26] The above process is
repeated thousands or millions of times, until the network reaches its
optimum, where, given any input, the model has a highest chance to predict the
correct label for it.

Various types of neural network layers are commonly used for deep learning
researches. Based on the type of the input, a layer can be designed to have
parameters of different sizes and shapes, so that the layer can take maximal
advantage of the input’s spatial patterns. For example, if the input is an
image, a “convolutional layer” can have a set of filters that acts on nearby
pixels to generate output features, so that the spatial similarity between
neighboring pixels is extracted. If the input is an English sentence, a
“recurrent layer” can have a recursive structure within itself, so the
relation between two adjacent words is represented. The choice of layer type
is highly related to how the input data is formatted.

Multi-layer perceptron network (abbr. MLP network) is a common class of neural
networks that consists of linear layers – the most basic type of network
layer. The parameters in a linear layer is simply a matrix: given an input of
a 1-dimensional vector, the layer performs matrix multiplication on the vector
and produces the result as another vector, a process equivalent to linear
transformation. Often times, a fixed activation function acts on the result
vector, so that non-linearity is added to the layer to make it more
adjustable. The MLP network stacks several linear layers together, so that an
input data vector is transformed into a feature vector, consisting of useful
information for the machine to generate predictions. The intermediate linear
layers are often called “hidden layers” of the network. Below is a figure
illustrating an MLP network.

Figure 2: The concept of a MLP network

MLP network is most suitable for input data that can be represented as 1-D
vectors, in which different fields are not correlated with each other. As
discussed in Section 3.2, the data we’ve designed highly resembles this
structure, so MLP network is in fact the best choice of our task.

###  4.2 Our Model Design

Let’s now go back to our task at hand: given a huge database of procurements,
auditors need to first identify any suspicious procurements from the rest,
then give each case a suitable label, so that punitive measures could be
taken. So in fact, two models are needed for the above purposes: a binary
model to distinguish between suspicious procurements and not suspicious ones,
and a multiclass model to classify what kind of fraud is contained in a
procurement already labeled as suspicious.

In fact, the two models can both be formulated as one single classification
problem: the binary model needs to choose a prediction between class (1) this
procurement is suspicious, or class (2) not suspicious, and the multiclass
model needs to choose from a list of classes of possible fraud types. So in
the process of designing our network, we can inherit our model from a given
layer structure that has been proven to work with this type of inputs, and add
a specific layer at the end for each model, in order to accommodate the
different number of output classes. A “softmax layer” serves exactly this
purpose: given a 1-dimensional tensor of many features, it can produce an
output vector $v=(v_{1},\ldots,v_{n})$ of fixed length $n$, where each $v_{i}$
denotes the probability that this input should be classified as belonging to
class $i$.

We decide to use the following structure for our network, provided in the
example of training an MLP network on the MNIST dataset, which achieves state-
of-the-art accuracy of $98.4\%$: [27] [28]

Figure 3: Structure of our MLP network

In the above figure, the network contains two large, dense layers of 512
neurons, each followed by a dropout layer with ratio $d=0.2$. The purpose of
the dropout layers is to prevent overfitting, which is a quite common problem
in MLP networks. Next, the output layer is a softmax layer, as mentioned
above, which fixes the number of output prediction classes, and should have
different sizes for our two models. Note that although the two models share
mainly the same structure, there are no relations between them in the training
and predicting process, since each model is given a different ground truth
label for an input data, and so the gradient descent process would happen in
different directions, diverging the two models eventually to have very
different parameters.

##  5 Implementation, Testing and Results

###  5.1 Implementation Details

In the actual coding, we choose to use two popular neural network libraries to
implement our models: tensorflow and keras. First, tensorflow serves as the
backend of keras, and provides strong GPU computing support when we train our
networks on our large number of data samples. Second, keras is well-known for
its easy, clean abstractions of MLP network layers, which save us a lot of
time when we code up, test, and optimize our neural network models.

In order to better initialize our networks and so achieve better performance,
we decide to normalize all our input data so they fall into the range $[0,1]$.
However, since the procurements are actually conducted at different scales, it
is impossible to find a uniform standard of normalization for all data. So we
make one pass through all samples, record the maximum and minimum values in
each column, and perform linear normalization on each point by the formula:

$normalized=\frac{original-min}{max-min}$ (1)

Our models are tested after training for 20 epochs, with a sample size of
50000, as specified in section 3.1. In addition, we implement 10-fold cross
validation to get the average performance of our models on the whole dataset.

###  5.2 Accuracy Results

Below are the results of our binary model(left) and multiclass model (right),
respectively:

Fold # Loss Accuracy 1 $0.3417$ $0.8297$ 2 $0.2936$ $0.8476$ 3 $0.2755$
$0.9273$ 4 $0.3378$ $0.8235$ 5 $0.3343$ $0.8251$ 6 $0.3376$ $0.7841$ 7
$0.2985$ $0.8460$ 8 $0.3392$ $0.8343$ 9 $0.4273$ $0.8333$ 10 $0.3022$ $0.8974$
Average $0.3288$ 0.8448 Fold # Loss Accuracy 1 $0.0674$ $0.9799$ 2 $0.0896$
$0.9778$ 3 $0.0778$ $0.9770$ 4 $0.0551$ $0.9770$ 5 $0.0730$ $0.9762$ 6
$0.0734$ $0.9778$ 7 $0.0727$ $0.9782$ 8 $0.0800$ $0.9782$ 9 $0.0778$ $0.9762$
10 $0.0859$ $0.9807$ Average $0.0753$ 0.9779 (2)

Besides cross-validation, we’ve also extracted some actual data (different
from the training samples above) to form a testing dataset. We choose 20
random rows from the dataset and record the prediction results of both models
below. If we count the hits and misses, we can see that the accuracies shown
here match quite well with the results obtained from above procedures.

Figure 4: Binary model predictions Figure 5: Multiclass model predictions

Quite obviously, both models perform quite well in the training process. For
the binary model, the result of $84\%$ accuracy is acceptable, based on the
limited number of features. In an actual scenario, $16\%$ wrong prediction
rate means that most of the procurement frauds would be caught by this model,
which is far more efficient than traditional auditing techniques. Still, there
could be ways to improve this result, such as adding more training features,
or using more complicated network structures; it is possible to expect such a
model to hit an accuracy of over $95\%$ if everything went smooth. And for the
multiclass model, $98\%$ accuracy is actually the state-of-the-art performance
of a multiclass MLP network – the classification of fraud types is now working
at its full power.

##  6 Future Research & Conclusions

###  6.1 Future Research

The following subjects are to be studied.

  * •

Different information relevant to a single procurement might be stored in
distinct databases. Such databases usually have different design methods and
lengths of use, which makes data collection unnecessarily tedious. We suggest
future researchers make ”creating a more integrated database” one of their
main targets.

  * •

Historical information related with fraud procurements is collected with high
reliance on reports written manually. This fact significantly affects the
number of procurements that could be used in machine learning. Considering the
large-scale database needed in effective machine learning, more work should be
done to increase the amount of cases that can be used to train the program.

  * •

Since raw data from databases have various formats, normalization measures
need to be taken before we feed data to our models. However, due to the lack
of consistency between columns of data, finding a good normalization method
for all data is exceptionally hard. In our current experiments, we rely on
some pre-determined assumptions on those data to construct different
normalization formulae for different columns. It is urgent that a uniformed,
concise and effective normalization method be worked out in future studies.

###  6.2 Conclusions

This research reveals both the probability of utilizing machine learning in
traditionally manpower-consuming areas like fraud auditing and the
difficulties that such utilization might face. The model testing part
evidently shows that our model has a good performance in predicting the
existence and further classification of fraudulence.

With the SAP database provided by SF Express, our artificial neural network
exhibits decent calculating speed and accuracy, which lead us to the
confidence of further applying machine learning in different types of audits.

A few incompatibilities still exist and further modifications of the model are
required. Also, considering the fact that the database a company use directly
decides the data to be inputted in training, the company itself should bring
more uniformity to the database it use as well.

## Acknowledgments

We would like to offer special thanks to Zhijun Lin, Raymond Li, Peng Su,
Chunlei Zhang, Jiansheng Fan and the auditing department of SF Express for all
information and data concerning procurement fraud, as well as every single
support they provided.

Also, we want to thank our reviewers and all of those who provided comments on
prior drafts of this paper.

## References

  * [1] Caulfield Tom.  The anatomy of procurement fraud.  [Contract Management][54(4)], 2014.
  * [2] The International Federation of Accountants.  International standard on auditing 240, 2009.
  * [3] M Navin and Matthew L Sherman.  Combating procurement fraud.  [Reporter][2016 Vol.43 Issue 2, p57-61], 2016.
  * [4] Ping Ying.  Fraud Audit.  China Financial & Economic Publishing House, 2016.
  * [5] Arthur Samuel.  Some studies in machine learning using the game of checkers.  [IBM Journal of Research and Development], 1959.
  * [6] Pat Langley.  The changing science of machine learning.  [Machine Learning: 82 (3)], 2011.
  * [7] Alex Krizhevsky.  Imagenet classification with deep convolutional neural networks.  [NIPS 2012: Neural Information Processing Systems], 2012.
  * [8] Dean Takahashi.  Nvidia ceo bets big on deep learning and vr.  [Venture Beat], 2016.
  * [9] Sam Maes.  Credit card fraud detection using bayesian and neural networks.  [ResearchGate], 2015.
  * [10] Andrea Dal Pozzolo.  Kaggle data science challenge: Credit card fraud detection, 2016.
  * [11] Hugo W.M.J., Badenhorst-Weiss J.A., and Van Rooyen D.C.  Procurement and supply management. 5th edition.  Pretoria: Van Schaik, 2006.
  * [12] Raymond C. McCann.  Procurement fraud investigative techniques: The need for congressional expansion of the inspector general subpoena power to include the power to compel testimony.  [Public Contract Law Journal][Volume 16, No. 2], 1986.
  * [13] AC Venter.  A procurement fraud risk management model.  [Meditari Accountancy Research][Volume 15, No. 2], 2007.
  * [14] Ministry of Finance, China Securities Regulatory Commission, and National Audit Office.  Basic internal control norms for enterprises, 2008.
  * [15] Shifeng Wang.  Procurement fraud methods and auditing strategies.  [Chinese Market][Volume 44], 2015.
  * [16] Dong Yajun.  Audit of material procurement fraud.  [Chinese Internal Audit][Volume 11], 2006.
  * [17] Robert May, Graeme Dandy, and Holger Maier.  Review of Input Variable Selection Methods for Artificial Neural Networks.  InTech, 2011.
  * [18] Richard E. Bellman.  Dynamic Programming Dover Books on Computer Science.  Courier Corporation, 2013.
  * [19] Farouq S. Mjalli, S.Al-Asheh, and H.E.Alfadala.  Use of artificial neural network black-box modeling for the prediction of wastewater treatment plants performance.  [Journal of Environmental Management][Volume 83, Issue 3], 2007\.
  * [20] Mark W. Cravena and Jude W. Shavlik.  Using neural networks for data mining.  [Future Generation Computer Systems][Volume 13, Issues 2–3], 1997\.
  * [21] Rodney Stark.  Deviant places: A theory of the ecology of crime.  [Criminology][Volume 25, Issues 4], 2006.
  * [22] Robert J. Kenney Jr. and Wendy T. Kirby.  A management approach to the procurement fraud problem.  [Public Contract Law Journal][Volume 15, No. 2], 1984.
  * [23] Xinjun Wei and Jian Sun.  Procurement parity audit.  [Guangdong Audit][Volume 11], 2002.
  * [24] Andrew Durant.  The enemy within.  [Engineering Management][Volume 15, Issue 5], 2005.
  * [25] Ge Rong.  Tensor methods in machine learning.  http://www.offconvex.org/2015/12/17/tensor-decompositions/, 2015\.
  * [26] Faizan Shaikh.  Introduction to gradient descent algorithm (along with variants) in machine learning.  https://www.analyticsvidhya.com/blog/2017/03/introduction-to-gradient-descent-algorithm-along-its-variants/, 2017\.
  * [27] Yann Lecun.  The mnist database.  http://yann.lecun.com/exdb/mnist/.
  * [28] Github user fchollet.  https://github.com/fchollet/keras/blob/master/examples/mnist_mlp.py.