# Is Surprisal in Issue Trackers Actionable?

James Caddy  0000-0002-6254-8982  University of AdelaideAdelaideAustralia
<EMAIL_ADDRESS>,  Markus Wagner  0000-0002-3124-0061  University
of AdelaideAdelaideAustralia<EMAIL_ADDRESS>,  Christoph Treude
0000-0002-6919-2149  University of MelbourneMelbourneAustralia
<EMAIL_ADDRESS>,  Earl T. Barr  0000-0003-0771-7891
University College LondonLondonUnited Kingdom<EMAIL_ADDRESS>and  Miltiadis
Allamanis  0000-0002-5819-9900  Microsoft ResearchCambridgeUnited Kingdom
<EMAIL_ADDRESS>
(2022)

###### Abstract.

##### Background

From information theory, surprisal is a measurement of how unexpected an event
is. Statistical language models provide a probabilistic approximation of
natural languages, and because surprisal is constructed with the probability
of an event occuring, it is therefore possible to determine the surprisal
associated with English sentences. The issues and pull requests of software
repository issue trackers give insight into the development process and likely
contain the surprising events of this process.

##### Objective

Prior works have identified that unusual events in software repositories are
of interest to developers, and use simple code metrics-based methods for
detecting them. In this study we will propose a new method for unusual event
detection in software repositories using surprisal. With the ability to find
surprising issues and pull requests, we intend to further analyse them to
determine if they actually hold importance in a repository, or if they pose a
significant challenge to address. If it is possible to find bad surprises
early, or before they cause additional troubles, it is plausible that effort,
cost and time will be saved as a result.

##### Method

After extracting the issues and pull requests from 5000 of the most popular
software repositories on GitHub, we will train a language model to represent
these issues. We will measure their perceived importance in the repository,
measure their resolution difficulty using several analogues, measure the
surprisal of each, and finally generate inferential statistics to describe any
correlations.

self-information, n-gram, GitHub issues

††copyright: acmcopyright††journalyear: 2022††doi:
XXXXXXX.XXXXXXX††conference: 19th International Conference on Mining Software
Repositories (Registered Reports); May 23–24, 2022; Pittsburgh, Pennsylvania,
United States††price: XX.XX††ccs: Information systems Language models††ccs:
Computing methodologies Anomaly detection††ccs: Information systems Data
mining††ccs: Computing methodologies Maximum likelihood modeling

##  1\. Introduction

Surprisal is a measure in information theory that can quantify how unexpected
and thus how informative a word $w_{t}$ is given the words that precede it
($w_{1},...,w_{t-1}$). A higher word surprisal value indicates that the
current word is less expected given the context. In mathematical terms,
surprisal is defined as the negative logarithm of the word’s conditional
probability of occurrence (Armeni et al., 2017).

In this work, we propose to apply the surprisal measure to software
engineering artefacts, motivated by many researchers arguing that software
developers need to be aware of unusual or surprising events in their
repositories, e.g., when summarizing project activity (Treude et al., 2015),
notifying developers about unusual commits (Leite et al., 2015; Goyal et al.,
2018), and for the identification of malicious content (Gonzalez et al.,
2021). The basic intuition is that catching bad surprises early will save
effort, cost, and time, since bugs cost significantly more to fix during
implementation or testing than in earlier phases (Dawson et al., 2010), and by
extension, bugs cost more the longer they exist in a product after being
reported and before being addressed.

Following recent work on applying natural language techniques to software
engineering data (Hindle et al., 2016), in this work, we investigate whether
the information-theoretic measure of surprisal is actionable when applied to
software repositories. In this study we analyse issues and pull requests
separately, but as a matter of convention, we will refer to these simply as
issues going forward. In our method, the differences are negligible, but
separately they may produce unique analyses.

We investigate two scenarios involving surprisal: its effect on resolution
difficulty and the perceived importance of surprising issues. Prior work
(Kavaler et al., 2017) found that conformance to project-specific language
norms reduces issue resolution time. Assuming that surprising issues do not
conform to such norms, we investigate whether they are more difficult to
resolve. Other work (Mohamed et al., 2018) has analysed what kind of issues
are reopened, and specific issue metrics such as number of comments are shown
to correlate. We conceptualise difficulty in terms of (1) reopened rate, (2)
amount of discussion, and (3) resolution time. Further, prior work established
that developers want to be aware of unexpected issues in their repositories
(Treude et al., 2015), and that high importance issues are more likely to be
included in release notes (Abebe et al., 2016). In this work, we investigate
whether surprising issues are treated with higher importance. We conceptualise
importance in terms of (1) mention in release notes, (2) first issues to be
worked on after a break, (3) issues attracting GitHub reactions (Borges et
al., 2019), and (4) priority labels being assigned to issues.

###  1.1. Motivating Examples

Take for example, a software product sustainment team that receives a steady
stream of issue reports which they must triage. Each of these issues have a
cost increasing at a given rate, which itself might be accelerating. This cost
may be in safety, budget, and/or reputation for example. It is in the interest
of stakeholders that cost is minimised across the lifetime of the product.

In order to predict the cost of the new issues coming in, someone with
experience needs to spend time comparing against previous experience, or
applying metrics based methods. Then time needs to be spent addressing the
issue for the cost to be eliminated.

If surprisal can be used to tell which issues are challenging or notable
without human input, it is possible that efficiency can be gained in the
resolution process. The triage process is better informed with notable issues
that might need to be prioritised sooner. The cost to address an issue,
weighed against its cost to the project, is better informed by the challenge
that the issue presents. Challenging issues can have more people assigned to
resolve it. Information about notable issues can be distributed to other
developers so they know what mistakes to avoid in the future.

Another example is of a new developer just joining a project. In order to
familiarise themselves with the history and most important changes or
developments that have occurred, they would usually have to rely on release
notes or a version history. In the case where the project either lacks release
notes, or in the case where the release notes contain even the most minor
changes, this can be overwhelming and of limited value to the new developer. A
tool based on surprisal could extract the most important or notable changes
from a mature project for this developer, or even tell someone who has been
away from a project for a period what notable things have happened since they
left.

##  2\. Background

In Claude Shannon’s seminal work on information theory he describes radio
signals and their ability to communicate information. In doing so, he makes
the first formal description of information entropy or information
uncertainty.

###  2.1. Uncertainty of an Event

Take two events, represented by the symbols ‘A’ and ‘B’. If we know that these
events are equally likely to occur, we could say that we are equally uncertain
which the next event will be. The information that we gain from observing an
event can be represented by $I(x)=-\log_{2}{P(x)}$, where $P(x)$ is the
probability of $x$ occurring. In this example, we can see that for either
outcome, $P(A)=P(B)=0.5$ and so $I(A)=I(B)=1$. We gain exactly 1 bit of
information, as there are two possible outcomes and either is as likely to
occur.

If we take another example, and modify the probabilities of the outcomes such
that $P(A)=0.0$ and $P(B)=1.0$, we can see that if we observe B, we gain
exactly 0 bits of information from $I(B)=-\log_{2}{1}=0$. This is intuitive,
since we already knew that the event would be B, there was no uncertainty.

Now we take an example where an event is extremely unlikely to happen. When
$P(A)=0.999$ and $P(B)=0.001$, if we observe the event A, we gain
$I(A)=-\log_{2}{0.999}\approx{}0.001$ bits of information. Since it was likely
to happen, we do not receive much information, but still a little. It is then
surprising when we observe the event B, since it is unlikely to happen. The
information we gain from observing B, $I(B)=-log_{2}{0.001}\approx{}9.966$, is
extreme in comparison.

###  2.2. Statistical Language Models

Shannon describes an approximation of the English language by utilising n-gram
statistical language models. Initially he describes a crude unigram model that
selects the next word in a sentence based on their relative frequencies in the
English language. This has an obvious flaw; unigram models assume that each
choice of word is independent from the last. Syntax and grammar demand more
careful choice of words than pure randomness alone, so to account for this,
greater order n-gram models are used.

Bigram models select the next word based on the previous word, and trigrams
select based on the previous two. These better capture the structure of
English, and while higher order models are possible, these tend to overfit the
training data. The language model also becomes exponentially sparser as the
order increases, which will in turn require exponentially more training data
or risk severely underfitting the training data.

With a model that statistically represents the English language in n-grams, we
can now determine how likely a word $w_{t}$ is, given the words that precede
it ($w_{1},...,w_{t-1}$).

###  2.3. Probability Distributions

To determine how closely an issue’s description represents the whole corpus of
descriptions, in an effort to see how surprising or not it is, we can use
cross entropy. To understand cross entropy, it is important to discuss the
underlying concept of entropy. Entropy can be quantified as the average number
of symbols needed to represent an event from a distribution of events. Take
for example, a distribution of events where:

$\displaystyle P(A)$ $\displaystyle=0.5,$ $\displaystyle P(B)$
$\displaystyle=0.25,$ $\displaystyle P(C)$ $\displaystyle=0.25$

This distribution is biased, or skewed in favour of observing the event ‘A’.
We can describe these events with their relative frequencies in a Shannon–Fano
coding (Fano, 1949), where:

$\displaystyle A$ $\displaystyle=\\{0\\},$ $\displaystyle B$
$\displaystyle=\\{01\\},$ $\displaystyle C$ $\displaystyle=\\{10\\}$

In this example, we can see that half of the time, we only require $1$ bit of
information to represent any event. The other half of the time, we require $2$
bits. We can therefore say that the average number of bits required to
represent an event from this distribution (its entropy) is $1.5$. Just as
above, where an event that occurs often can be represented with less bits,
distributions that are more biased, that yield one particular event more
often, present less entropy. Shannon formalises this into the following
equation, using $X$ as the support for random events $x$:

(1) $H(P)=-\sum_{x\ \in\ X}P(x)\times\log(P(x))$

Cross entropy describes how many symbols on average are required to represent
an event from one distribution in a coding optimised for another, if both have
the same support. In our setting, it measures how many symbols are required to
represent an issue’s description, based on the distribution of words observed
($P_{o}$) in the issue, in the true distribution ($P_{\mathit{tt}}$) of words
in all issues.

(2) $H(P_{o},P_{\mathit{tt}})=-\sum_{x\ \in\
X}P_{o}(x)\times\log(P_{\mathit{tt}}(x))$

When $P_{o}=P_{\mathit{tt}}$, Equation 2 is equivalent to Equation 1.
Otherwise, when the observed distribution (the issue’s use of words) differs
from the distribution of the training set (the set of all issues), cross
entropy increases. Higher values of cross entropy therefore imply an issue is
more surprising.

There are potentially other surprisal metrics that provide different
evaluations. More simple measures for instance, might take the minimum,
maximum, or average surprisal of all words in an issue. Cross entropy is a
very well-realised metric of model accuracy in the literature (Niven, 2007),
and it is for this reason that we are using it.

##  3\. Research Questions

A number of research questions and hypotheses have been made to guide the
investigation.

RQ1: How well does information theory’s surprisal, as measured by a
statistical language model, align with perceived surprisal?

With RQ1, we hope to find how statistical language models compare to the human
perception of surprisal, and also which factors of the language model
influence its ability to measure the surprisal of an issue.

RQ2: Are surprising issues correlated with resolution difficulty?

In RQ2, we define “resolution difficulty” as a combination of the following
factors. Difficult to resolve issues are: reopened more often; attract more
discussion prior to resolution; and take longer to be resolved, when compared
to issues with little or no resolution difficulty. This is perhaps an
incomplete list, but will serve as the basis for the report.

RQ2 can be formalised into the following hypotheses:

  1. H2.1:

Surprising issues are more likely to be reopened.

  2. H02.1:

There is no significant difference in how often surprising issues are
reopened, compared to unsurprising issues.

  3. H2.2:

Surprising issues attract more discussion. (Number of people involved)

  4. H02.2:

There is no significant difference in how much discussion surprising issues
draw, compared to unsurprising issues.

  5. H2.3:

Surprising issues attract more discussion. (Number of interactions)

  6. H02.3:

There is no significant difference in how much discussion surprising issues
draw, compared to an unsurprising issue.

  7. H2.4:

Surprising issues take longer to resolve.

  8. H02.4:

There is no significant difference in time to resolve surprising issues,
compared to unsurprising issues.

  9. H2.5:

Surprising issues are difficult, and difficult issues are best represented as
some combination of reopen rate, amount of discussion, and time to resolve.

  10. H02.5:

Surprising issues are difficult, but difficulty is best represented as only
one factor of reopen rate, amount of discussion, or time to resolve.

RQ3: Are surprising pull requests correlated with resolution difficulty?

As in RQ2, RQ3 defines difficulty in the same way but for pull requests rather
than issues. The purpose of RQ3 is to determine if the more structured and
formal contents of pull requests are more suitable than issues for
establishing a relationship between surprisal and difficulty. Pull requests
offer a suitable alternative because they are intended to directly address a
need that would typically be expressed in an issue. Additionally, merged pull
requests are reviewed and thus are unlikely to be duplicated, describe what
they are resolving, and what they address is far less likely to be misreported
as a defect if it is not.

The formal hypotheses for RQ3 are formulated in a manner identical to the
hypotheses of RQ2, since pull requests and issues are functionally identical
in this context. In the interest of brevity, the full text of these hypotheses
H2.1 through H2.5 has been omitted.

RQ4: Are surprising issues correlated with perceived importance?

In RQ4, we define “perceived importance” as a combination of the following
factors. Important issues are: mentioned in release notes; addressed soon
after periods of breaks; attract more GitHub reactions; and have high-priority
labels added. Once again, this is perhaps an incomplete list, but will serve
as the basis for the report.

RQ4 can be formalised into the following hypotheses:

  1. H4.1:

Surprising issues are more likely to be mentioned in release notes.

  2. H04.1:

There is no significant difference in how often surprising issues are
mentioned in release notes, compared to unsurprising issues.

  3. H4.2:

Surprising issues are addressed with priority over unsurprising issues after a
hiatus.

  4. H04.2:

There is no significant difference in the time addressing post-hiatus
surprising issues, compared to unsurprising issues.

  5. H4.3:

Surprising issues attract more GitHub reactions.

  6. H04.3:

There is no significant difference in how many reactions a surprising issue
receives, compared to unsurprising issues.

  7. H4.4:

Surprising issues are more often labelled as high-priority issues.

  8. H04.4:

There is no significant difference in what priority surprising issues are
labelled, compared to unsurprising issues.

In H3.2, we define a hiatus as the top 25% longest times between issue
resolutions per contributor for a particular repository. Priority in this case
is the order in which issues are worked on after a hiatus.

RQ5: Are surprising pull requests correlated with perceived importance?

As before with RQ2 and RQ3, RQ4 focuses on issues, and RQ5 will focus on their
pull request counterparts. The full text of hypotheses H4.1 through H4.4 has
been omitted for brevity.

##  4\. Variables

A summary of the variables involved for all the research questions can be
found in Table 1.

Predictor Variable:

  * •

Surprisal. How surprising the content of the issue is to the language model.
This is calculated as the cross entropy of the issue text and the corpus of
issues.

Response Variables:

  * •

Reopenings. How many times the issue has been reopened. An issue can be
labelled as closed and then reopened for numerous reasons, just as a pull
request can be merged and then reopened. This usually indicates a regression
of functionality, reoccuring bug, or unsuccessful fix (Jiang et al., 2019),
all indications that the issue has additional complexity associated with it.

  * •

Participants. How many individual participants have interacted with the issue.
Every event that takes place on an issue has an actor associated with it. This
actor represents somebody interacting with the issue. If a particular issue
involves multiple assignees for example, it may be a sign that additional
expertise is needed to resolve it. It could also mean that it affects a lot of
people but is not necessarily more difficult. Kavaler et. al. (Kavaler et al.,
2017) show that there is a significant increase in issue resolution time with
an increased number of unique participants.

  * •

Interactions. How many interactions have been made with the issue, including
comments, mentions, taggings, assignments and state changes. A full list of
events is available in the GitHub Issue API documentation (Docs, 2022b). Some
of these interactions are considered a normal part of the resolution process,
although we expect to see more interactions if the issue reveals hidden
complexity over time. Kavaler et. al. (Kavaler et al., 2017) also show there
is an increase in issue resolution time corresponding to the number of
comments made.

  * •

Open State Duration. How long the issue has been unresolved; from first
submission to last closure, or if it has not been closed, the time of
analysis. While some issues may not be difficult but especially time
consuming, we expect to see longer resolution times for issues that are
difficult to diagnose or replicate.

  * •

Mentions in Release Notes. How many times the issue has been mentioned within
release notes on GitHub. Some repositories make no use of GitHub’s Releases
feature, so only the repositories that do, and that mention issues at all in
them will be considered. Highly important issues are more likely to be
included in release notes (Abebe et al., 2016).

  * •

Order of Address. After lengthy breaks of development, whether independently
taken or due to holidays, it is likely that the most pressing of issues in the
backlog are chosen for immediate resolution. Commits after extended breaks
have been described as interesting, in a previous paper (Treude et al., 2015).
Each contributor has a time between addressing issues, so taking the top 25%
of these breaks, and then ordering the issues that they worked on afterwards
gives us an indication of what importance that author places on each issue.

  * •

Reactions. How many reactions have been made on the issue. Reactions give a
quick way for users to interact with an issue. For example, users can express
joy that a particular issue is closed, or frustration if it disrupts them
personally, through reactions. This can be seen as a community rating of
importance, rather than that of the maintainers (Borges et al., 2019).

  * •

Labelling. Many repositories use the GitHub issue labelling system to triage
incoming bug reports and feature requests. Issues are sorted by maintainers
into priorities and labelled as such, from low priority to high priority (Xie
et al., 2021), §3.B. These labels can be seen as the maintainer’s rating of
importance.

Table 1. Variables Variable | Hypotheses |  Description | Measure |  Operationalisation
---|---|---|---|---
Surprisal |  [t]3cmPredictor for |  |  |
all hypotheses | How surprising an issue is to the statistical language model. |  Ratio | Cross entropy of issue, obtained with probability from SLM trained on corpus of all issues. |
Reopenings |  [t]3cmResponse for |  |  |
H1.1, H2.1 | After an issue has been labelled as closed or resolved, it can be reopened either due to a fix being unsuccessful, a regression, or reoccurring bug. |  Ratio | GitHub Issue API’s “reopened” event. |
Participants |  [t]3cmResponse for |  |  |
H1.2, H2.2 | Number of unique individuals involved with the issue. Each event (as described by the GitHub API (Docs, 2022b)) associated with an issue has an actor that initiates it, whether that event is a comment, or a state update. |  Ratio | GitHub Issue API’s event “actor” for each event associated with an issue. |
Interactions |  [t]3cmResponse for |  |  |
H1.3, H2.3 | Number of events associated with the issue. |  Ratio | Count of events, as described by the GitHub Issue API (Docs, 2022b). |
[t]2cmOpen State |  |  |  |
Duration |  [t]3cmResponse for |  |  |
H1.4, H2.4 | Length of time between the issue’s creation, and it being resolved for the final time. |  Ratio | Difference between GitHub Issue API’s “created_at” value for the issue and final “closed” or “merged” event. |
Mentions |  [t]3cmResponse for |  |  |
H3.1, H4.1 | Number of mentions within a repository’s release notes. |  Ratio | Scrape for issue numbers through GitHub’s Releases API. |
[t]2cmOrder of |  |  |  |
Address |  [t]3cmResponse for |  |  |
H3.2, H4.2 | Issue order after top 25% of contributor’s inter-issue resolution times. |  Interval | Issues assigned to a contributor are retrieved through the GitHub Issues API. |
Reactions |  [t]3cmResponse for |  |  |
H3.3, H4.3 | Number of reactions on a particular issue. |  Ratio | Count of reactions for an issue, from GitHub’s Reactions API. |
Labelling |  [t]3cmResponse for |  |  |
H3.4, H4.4 | Assigned priority or importance label of a particular issue. |  Interval | Labels are extracted via the GitHub Issues API, and then normalised (per repository) to a 3-degree scale, ‘low-importance’, ’regular-importance’, ‘high-importance’. |

##  5\. Data Sets

In this section we present the data sources that we will use, and how we will
use them.

###  5.1. Sources

For this study, open-source software stored on GitHub serves as our primary
and sole source for software issues. Unfortunately, GitHub imposes a
restrictive limit to how many interactions with its API a user can make.
Typically this is 5000 calls per hour (Docs, 2022a), and so we take the top
5000 most ‘starred’ repositories as our data set. Stars represent a user
liking a repository, and therefore indicate popular repositories, more likely
to have high numbers of issues due to increased testing and feature requests —
a result of more users (Borges et al., 2016).

This supposes that GitHub is used in the same way by the maintainers of those
5000 repositories, which is not the case. Many of these repositories do not
make use of the Issues feature; many of these repositories are not software
development related and possess little to no code; and other repositories are
simply a mirror of a repository developed and hosted elsewhere (Kalliamvakou
et al., 2014). Additionally, we wish to limit the scope to English language
repositories. To find the repositories that fit these conditions, we plan to
use G-Repo (Romano et al., 2021) as a means to filter out non-software
repositories, repositories making little use of the Issues feature (less than
1000 issues), and non-English repositories.

###  5.2. Language Model Transfer

The statistical language model (SLM) used to determine the surprisal of an
issue, is intended to represent a probabilistic model of what the content of
an issue looks like. Software issues are typically written in such a way that
requires domain-specific knowledge of terminologies and jargon, and are in
most cases very specific to the project they reference. As a result, a more
specific SLM, trained on a software-based corpus would see some improvement in
accurately modelling the software language used, compared to a more general
pre-trained model (Wang et al., 2021), §5.3. For this reason, it was decided
that a bespoke language model will be trained for the task.

###  5.3. Pre-processing

Issues are composed of a title and description. Both of these elements have
the possibility of individually containing pertinent information, and in some
examples do not describe the information the other holds. It is for this
reason that during the pre-processing stage, we will prepend the issue
description with the issue title. The SLM will then be trained on these title-
description combinations.

Before training a model on the issue text, we will clean the data with the
following pre-processing steps:

  1. (1)

HTML Void Elements (Standard, 2022) are translated into special tokens, e.g.,
<br> becomes [BR].
(Also code blocks, see details following.)

  2. (2)

Other HTML elements are replaced with their contents, e.g., a list element
becomes a simple string of its content.

  3. (3)

Text undergoes normalisation of its Unicode forms. Canonical Composition (NFC)
is used in accordance with the Character Model standard proposed by W3C
(Group, 2021).

  4. (4)

Punctuation and symbols are removed on word boundaries, and when isolated.

  5. (5)

Stop words are removed, and remaining words are stemmed.
(See details following.)

As singular code tokens — variable names and the like — may provide important
contextual information across multiple issues, they will be preserved in the
training data. Code blocks on the other hand risk introducing too many
globally-unique tokens into the model without introducing useful and
actionable information to the description. The reason is that the syntax of
code is entirely disparate with that of English for example, and so code
blocks will be replaced with a special token [CODE] where they are demarcated
with the <pre> and <code> HTML tags (as is typical on GitHub).

It is possible that the quality of the language model could be improved by
further transforming the text in a final step. In many natural language
processing applications, stop word removal is conducted to remove low-
information tokens, thereby increasing the average entropy. In other
applications, lemmatisation or stemming is used to reduce the occurrence of
high-information tokens. Concepts are learned better if you can reduce
“fishing” and “fishlike” to “fish”. However we cannot know for certain what
using either will achieve when modelling surprisal, and a preliminary
investigation will be conducted to see if it indeed increases the accuracy of
the model. The choice of algorithms is still to be determined, and will be
revisited in the final report.

####  5.3.1. Priority Labelling

Some GitHub repositories use the labelling system to assign priority grades or
importance to issues in their triage process. These labels are user text input
and can represent these grades in a number of different ways. For example, one
repository may use the labels ‘Low Priority’, and ‘High Priority’, where
another may use the labels ‘P1’ through ‘P5’.

In order to process these priorities, it is necessary for manual
classification to normalise these different ranges. For the purpose of this
experiment, priority is distributed among the three degrees ‘low-importance’,
‘regular-importance’, and ‘high-importance’. Tie-breaking is settled by ruling
in favour of a higher importance, using the previous examples for example;
‘low-importance’ would take ‘P1’ and ‘Low Priority’; ‘regular-importance’
would take ‘P2’ and ‘P3’; and ‘high-importance’ would take ‘P4’, ‘P5’, and
‘High Priority’.

To ensure that these have been correctly classified, we propose giving two
researchers a list of 400 randomly selected labels from the population of
approximately 9000 different labels and asking them to classify them using the
method above, or classify them as ‘unrelated’ to priority or importance. The
agreement between the researchers is then calculated using Cohen’s kappa, and
in the event that a consensus value of 0.7 is reached, we consider the task
sufficiently unambiguous, and a single researcher can then be trusted to
accurately represent the rest of the $\sim 9000$ labels. If not, a more
involved process needing agreement by more researchers is required to classify
the labels.

##  6\. Execution Plan

In this section we present the method we plan to use for the study. It is
important that all the decisions made during the experiment will be recorded,
along with intermediate outputs. For example: a record will be made listing
the 5000 most starred repositories on GitHub at the time of the experiment;
after repositories are removed when failing to meet selection criteria, those
will also be made into a record; and decisions such as how the total issue
count was calculated will be made into a record too. All the code used will be
made available in a GitHub repository for the final report.

###  6.1. Method

To test all hypotheses the following method is proposed:

  1. (1)

Using the GitHub API, query the 5000 most starred repositories on GitHub.

  2. (2)

Query the number of issues. If any repository has less than 1000 issues,
remove it from the pool.

  3. (3)

Extract all GitHub issues from each repository.

  4. (4)

Train SLM on entire issue description corpus.

  5. (5)

For each repository:

    1. (a)

Determine if it uses priority/importance labels, and normalise those labels
into three degrees of importance. Ties broken in favour of being more
important.

    2. (b)

For each issue:

      1. (i)

Determine and record surprisal of issue.

      2. (ii)

Record how many re-openings have occurred.

      3. (iii)

Record how many unique participants have interacted with the issue.

      4. (iv)

Record how many interactions have been made with the issue.

      5. (v)

Calculate and record open duration for the issue.

      6. (vi)

Record number of reactions to the issue.

      7. (vii)

Record normalised importance label, if any.

    3. (c)

Parse all release notes for the repository, incrementing the mentions of a
particular issue when it appears.

    4. (d)

For each contributor:

      1. (i)

Determine top 25% of inter-issue resolution times.

      2. (ii)

Split all issues assigned to the contributor into bands following a break of
at least the 25th percentile’s length.

      3. (iii)

Order the issues by oldest assigned time.

      4. (iv)

Record the ordinal position of each issue in its inter-break band.

  6. (6)

Split issues from GitHub into pull requests and simple issues.

  7. (7)

Generate descriptive statistics for each repository according to analysis
plan.

  8. (8)

Generate inferential statistics across entire issue data set according to
analysis plan.

StartQuery GitHub APIPre-processingCalculate DifficultiesCalculate
ImportanceTrain SLMSelect Test IssueGenerate StatisticsEnd Figure 1. Method
Flowchart

###  6.2. Statistical Language Modelling

N-gram models saw some popularity during the 2000s and 2010s, but gradually
saw declining popularity due to their perceived contextual fragility
(Rosenfeld, 2000)(2000), and competitive neural language models being
developed (Mikolov, 2012)(2012). Despite this, n-gram models still perform
admirably in Natural Language Processing applications. For our SLM, we use a
trigram model using word tokens (shingles). We chose this due to its
simplicity, familiarity, popularity, and reasonably effective results (Chen
and Goodman, 1999). More modern statistical approaches typically use a n-gram
model utilising a backoff strategy to deal with unknown tokens.

We train the SLM using the entire corpus of issues. This gives us the
advantage of never coming across an unknown token, as all the tokens that we
generate surprisal values for are contained in the training set. Any unknown,
in this case ‘unique’, tokens would otherwise be infinitely surprising, which
might not accurately represent how common it is to appear in say a larger
sample size of repositories and issues.

The loss of generality realised by the lack of testing set is insignificant
when compared to the small sample size of 5000. For this study, we are more
concerned with the formulation of surprisal, and the two correlations applying
to the repositories we have selected. Transferability can be further
investigated after this proof of concept, if successful. Conversely, because
singular repositories have a small sample size of issues, the specificity
gained by training only on a single repository introduces massive sparsity in
the model. This is another reason we include all issues into the training set.

###  6.3. Model Smoothing

Despite the lack of unknown tokens, we still apply a smoothing algorithm to
obtain a more uniform, less sparse, and more representative model of the
larger corpus of issues outside of the repositories that we have selected. The
modified Kneser-Ney algorithm described by Chen and Goodman (Chen and Goodman,
1999), §3, was chosen due to its excellent smoothing performance.

###  6.4. Model Improvement

While trigrams were chosen for the reasons previously mentioned, it is worth
understanding how n-gram order affects the language model’s calculation of
surprisal. It is also worth understanding how the use of a testing set from
the training data affects this calculation of surprisal.

These two factors contribute to the SLM and reveal important information
regarding RQ1. In order to measure the affects of these factors, we propose
the following additional experiment which compares the SLM performance against
a manual classification:

  1. (1)

A repository of relatively few issues is chosen as a model.

  2. (2)

Take a representative sample of issues from the repository.

  3. (3)

Two researchers are asked to rate the surprisal in each issue, after reading
all issues thoroughly.

    1. (a)

Issues are rated on a Likert scale.

    2. (b)

A value of 1 means that the description of an issue is not surprising.

    3. (c)

A value of 5 means that the issue contains almost completely unique
information.

    4. (d)

Surprisal judgements should be based on factors such as topic, formatting,
length, and word usage.

  4. (4)

Choose the SLM training set:

    1. (a)

Entire issue descriptions corpus.

    2. (b)

Entire issue descriptions corpus, with the chosen repository absent.

    3. (c)

Entire issue descriptions corpus, with each issue removed individually.

  5. (5)

Choose n-gram order, repeating from 1-grams to 10-grams.

  6. (6)

Measure and record surprisal of each issue (the same individually removed from
the training set if that type of training set is chosen).

  7. (7)

Generate agreement statistics between each model and the manual
classification.

##  7\. Analysis Plan

This is a correlational study of independent random variables, the following
is a design of the quantitative analysis that will be undertaken. All
calculations will be performed using a statistics package.

##### Model Analysis

In Section 6.4 we propose a method with which the surprisal of a repository-
representative sample of issues is judged manually by two researchers. The
surprisal of the issues is then measured using the surprisal calculated by the
SLM. We propose using Cohen’s kappa to measure the agreement between the two
researchers, and report how contentious this task is. We also propose using a
Kendall rank correlation to measure the agreement between the SLM- and
manually-computed surprisal values. In the event that the agreement is
statistically significant, it can be said that SLMs can calculate surprisal to
a similar degree as a human participant can, and are suitable for the
following analyses. A qualitative analysis of the issues that have
disagreement in the surprisal ratings is conducted. Both the case of inter-
researcher disagreement and researcher-SLM disagreement will be analysed in an
effort to understand what caused this disagreement.

We expect that a language model using the entire corpus of issue descriptions
and trigrams sufficiently agrees with the researcher’s judgements. However, we
will run the deeper analysis using different training sets and n-gram orders
to confirm those choices are indeed correct. A combination of each choice will
maximise the agreement with the manual computation, and this combination will
be used in the analyses going forward. In the event that even the combination
that maximises the agreement is still not statistically significant, a more
thorough inspection of the factors influencing the SLM is required.

##### Descriptive Statistics

We will present descriptive statistics of the predictor and response variables
in a summary of the complete data set. Sample size, mean, standard deviation,
maximum, and minimum values for each variable will be included

##### Inferential Statistics

Before presenting a correlation and regression analysis using a linear
regression, we graph the relationship between the surprisal and each response
variable for the issues in a repository. We will then test each hypothesis
separately with its corresponding variable, and look for statistical
significance.

A Shapiro-Wilk test is used to determine if the data shows normality (Shapiro
and Wilk, 1965). If the data is normally distributed, we can choose to use the
additional descriptive power of a Pearson correlation. If not, a Spearman
correlation can be used instead (Spearman, 1987), §1.2.a).

In order to test H1.5 and H2.5, we will perform a multiple linear regression,
this time including one statistically significant measure of difficulty into
the null model. If the F-test is not statistically significant, we can accept
the alternate hypothesis.

This test shows if a combination of our chosen difficulty factors is better in
representing difficulty as a whole, in comparison to the measure added to the
null model on its own. When performing this second regression, we expect to
see multicollinearity, a high Variance Inflation Factor for these measures
(Kutner et al., 2004), §10.5, as our belief is that they represent the same
variable: difficulty.

##### Interpretation of Results.

We will present our interpretation of our findings, a discussion on any
assumptions discovered, limitations, threats to validity, and future research.

## References

  * (1)
  * Xie et al. (2021) Xiaoyuan Xie, Yuhui Su, Songqiang Chen, Lin Chen, Jifeng Xuan, and Baowen Xu. 2021\.  MULA: A Just-In-Time Multi-labeling System for Issue Reports.  _IEEE Transactions on Reliability_ 1, 1 (2021), 1–14.
  * Docs (2022a) GitHub Docs. 2022a.  Rate limit.  https://docs.github.com/en/rest/reference/rate-limit
  * Docs (2022b) GitHub Docs. 2022b.  Issue event types.  https://docs.github.com/en/developers/webhooks-and-events/events/issue-event-types
  * Wang et al. (2021) Jun Wang, Xiaofang Zhang, and Lin Chen. 2021.  How well do pre-trained contextual language representations recommend labels for GitHub issues?  _Knowledge-Based Systems_ 232 (2021), 107476.
  * Fano (1949) Robert M Fano. 1949\.  _The transmission of information_.  Massachusetts Institute of Technology, Research Laboratory of Electronics, Cambridge, MA, United States.
  * Chen and Goodman (1999) Stanley F Chen and Joshua Goodman. 1999.  An empirical study of smoothing techniques for language modeling.  _Computer Speech & Language_ 13, 4 (1999), 359–394.
  * Leite et al. (2015) Larissa Leite, Christoph Treude, and Fernando Figueira Filho. 2015\.  UEDashboard: Awareness of unusual events in commit histories. In _Proceedings of the Joint Meeting on Foundations of Software Engineering_. ACM, New York, NY, United States, 978–981.
  * Jiang et al. (2019) Jing Jiang, Abdillah Mohamed, and Li Zhang. 2019.  What are the characteristics of reopened pull requests? a case study on open source projects in github.  _IEEE Access_ 7 (2019), 102751–102761.
  * Goyal et al. (2018) Raman Goyal, Gabriel Ferreira, Christian Kästner, and James Herbsleb. 2018.  Identifying unusual commits on GitHub.  _Journal of Software: Evolution and Process_ 30, 1 (2018), e1893.
  * Group (2021) W3C Working Group. 2021\.  Character Model for the World Wide Web: String Matching.  https://www.w3.org/TR/charmod-norm/#normalizationChoice
  * Niven (2007) Robert K. Niven. 2007\.  Combinatorial Information Theory: I. Philosophical Basis of Cross-Entropy and Entropy.  arXiv:cond-mat/0512017 [cond-mat.stat-mech]
  * Abebe et al. (2016) Surafel Lemma Abebe, Nasir Ali, and Ahmed E Hassan. 2016\.  An empirical study of software release notes.  _Empirical Software Engineering_ 21, 3 (2016), 1107–1142.
  * Borges et al. (2016) Hudson Borges, Andre Hora, and Marco Tulio Valente. 2016\.  Understanding the factors that impact the popularity of GitHub repositories. In _Proceedings of the International Conference on Software Maintenance and Evolution_. IEEE, Los Alamitos, CA, United States, 334–344.
  * Hindle et al. (2016) Abram Hindle, Earl T Barr, Mark Gabel, Zhendong Su, and Premkumar Devanbu. 2016.  On the naturalness of software.  _Commun. ACM_ 59, 5 (2016), 122–131.
  * Borges et al. (2019) Hudson Borges, Rodrigo Brito, and Marco Tulio Valente. 2019\.  Beyond Textual Issues: Understanding the Usage and Impact of GitHub Reactions. In _Proceedings of the Brazilian Symposium on Software Engineering_. ACM, New York, NY, United States, 397–406.
  * Romano et al. (2021) Simone Romano, Maria Caulo, Matteo Buompastore, Leonardo Guerra, Anas Mounsif, Michele Telesca, Maria Teresa Baldassarre, and Giuseppe Scanniello. 2021.  G-Repo: a Tool to Support MSR Studies on GitHub. In _2021 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)_. IEEE, Los Alamitos, CA, United States, 551–555.  https://doi.org/10.1109/SANER50967.2021.00064
  * Dawson et al. (2010) Maurice Dawson, Darrell Norman Burrell, Emad Rahim, and Stephen Brewster. 2010.  Integrating software assurance into the software development life cycle (SDLC).  _Journal of Information Systems Technology and Planning_ 3, 6 (2010), 49–53.
  * Armeni et al. (2017) Kristijan Armeni, Roel M Willems, and Stefan L Frank. 2017\.  Probabilistic language models in cognitive neuroscience: Promises and pitfalls.  _Neuroscience & Biobehavioral Reviews_ 83 (2017), 579–588.
  * Treude et al. (2015) Christoph Treude, Fernando Figueira Filho, and Uirá Kulesza. 2015\.  Summarizing and measuring development activity. In _Proceedings of the Joint Meeting on Foundations of Software Engineering_. ACM, New York, NY, United States, 625–636.
  * Kutner et al. (2004) Michael H Kutner, Christopher J Nachtsheim, John Neter, et al. 2004\.  _Applied linear regression models_. Vol. 4.  McGraw-Hill/Irwin, New York, NY, United States.
  * Mikolov (2012) Tomáš Mikolov. 2012\.  _Statistical Language Models Based on Neural Networks_.  Ph.D. thesis. Brno University of Technology, Faculty of Information Technology.  https://www.fit.vut.cz/study/phd-thesis/283/
  * Mohamed et al. (2018) Abdillah Mohamed, Li Zhang, Jing Jiang, and Ahmed Ktob. 2018\.  Predicting which pull requests will get reopened in GitHub. In _2018 25th Asia-Pacific Software Engineering Conference (APSEC)_. IEEE, Los Alamitos, CA, United States, 375–385.
  * Kavaler et al. (2017) David Kavaler, Sasha Sirovica, Vincent Hellendoorn, Raul Aranovich, and Vladimir Filkov. 2017\.  Perceived language complexity in GitHub issue discussions and their effect on issue resolution. In _2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, Los Alamitos, CA, United States, 72–83.
  * Shapiro and Wilk (1965) Samuel Sanford Shapiro and Martin B Wilk. 1965.  An analysis of variance test for normality (complete samples).  _Biometrika_ 52, 3/4 (1965), 591–611.
  * Spearman (1987) Charles Spearman. 1987\.  The proof and measurement of association between two things.  _The American Journal of Psychology_ 100, 3/4 (1987), 441–471.
  * Gonzalez et al. (2021) Danielle Gonzalez, Thomas Zimmermann, Patrice Godefroid, and Max Schäfer. 2021.  Anomalicious: Automated Detection of Anomalous and Potentially Malicious Commits on GitHub. In _Proceedings of the International Conference on Software Engineering: Software Engineering in Practice_. IEEE, Los Alamitos, CA, United States, 258–267.
  * Standard (2022) HTML Living Standard. 2022\.  HTML Living Standard: Void Elements.  https://html.spec.whatwg.org/#void-elements
  * Rosenfeld (2000) R. Rosenfeld. 2000\.  Two decades of statistical language modeling: where do we go from here?  _Proc. IEEE_ 88, 8 (2000), 1270–1278.  https://doi.org/10.1109/5.880083
  * Kalliamvakou et al. (2014) Eirini Kalliamvakou, Georgios Gousios, Kelly Blincoe, Leif Singer, Daniel M German, and Daniela Damian. 2014.  The promises and perils of mining github. In _Proceedings of the 11th Working Conference on Mining Software Repositories (MSR)_. ACM, New York, NY, United States, 92–101.


11institutetext: Norwegian University of Science and Technology (NTNU),
Gjøvik, Norway

# Analyzing eyebrow region for morphed image detection

Abdullah Zafar 11 Christoph Busch 11

###### Abstract

Facial images in passports are designated as primary identifiers for the
verification of travelers according to the International Civil Aviation
Organization (ICAO) [9]. Hence, it is important to ascertain the sanctity of
the facial images stored in the electronic Machine-Readable Travel Document
(eMRTD). With the introduction of automated border control (ABC) systems that
rely on face recognition for the verification of travelers [2], it is even
more crucial to have a system to ensure that the image stored in the eMRTD is
free from any alteration that can hinder or abuse the normal working of a
facial recognition system. One such attack against these systems is the face-
morphing attack. Even though many techniques exist to detect morphed images,
morphing algorithms are also improving to evade these detections. In this
work, we analyze the eyebrow region for morphed image detection. The proposed
method is based on analyzing the frequency content of the eyebrow region. The
method was evaluated on two datasets that each consisted of morphed images
created using two algorithms. The findings suggest that the proposed method
can serve as a valuable tool in morphed image detection, and can be used in
various applications where image authenticity is critical.

###### Keywords:

Morphed image detection eyebrow region analysis automatic border control (ABC)
security.

##  1 Introduction

Face morphing is a real and live threat against the ABC systems which verify a
person’s identity by comparing the live image with the facial reference stored
in the eMRTD [20]. Despite the simplicity of the solution of having the
passport holder come to the center to take photographs, it is still not
universally used due to financial reasons. Furthermore, many countries have
adopted or are in the process of adopting web-based passport/visa applications
for the ease of applicants where the user can upload a digital copy of the
image to the web portal [13]. With technological advances, it is counter-
intuitive for the general user to be asked to come to an office just to take a
photograph. These things make the detection of a morphed image even more
relevant in these times.

For simplicity, face morphing can be explained as an attack against face
recognition systems where the images of two individuals are combined to create
a morphed image that is used as a reference image. This reference image
produces a positive match against the images that were used in creating the
morphed image. One serious application of such an attack is explained in [6]
where face morphing enables an individual to travel on someone else’s
passport.

This paper will present a morphing attack detection (MAD) technique to detect
morphed images during the enrollment phase. MAD methods can be divided into
two categories i.e. single image MAD (S-MAD) and Differential MAD (D-MAD)
[10]. S-MAD involves analyzing an image to determine whether the image is a
morph or not. D-MAD involves analyzing the image and another trusted live
source for detection. The approach presented in this paper is an S-MAD
technique. However, the same technique can also be applied in a D-MAD
scenario. S-MAD is more relevant for the passport application process because
no trusted live source exists during the enrollment phase.

The approach is based on the analysis of the eyebrow region. The assumption is
that the eyebrow region has a high-frequency content due to the presence of
hairs. The idea is to analyze the possible reduction in this high-frequency
information due to the smoothening effect that results from the averaging of
two images in the creation of a morphed image. Furthermore, the eyebrow region
is interesting because of its universality and importance in the performance
of face recognition systems [15]. The rest of the paper is organized as:
Section 2 will highlight some of the related work, Section 3 explains the
methodology, Section 4 presents the results obtained, and Section 5 concludes
with final remarks and discussion.

##  2 Prior Work

Morphing-based attacks against ABC systems were first identified by Ferrara et
al. [6] where they demonstrated the hypothetical scenario of a malicious actor
who travels on his friend’s passport by means of a face morphing attack. After
that, the topic of morphed image detection piqued the interest of researchers
resulting in a number of studies presenting different kinds of morphed
detection techniques. In this section, past works are presented where texture
descriptors are used for morphing attack detection.

Ramachandra et al. [14] in 2016 proposed the first single image based morphed
attack detection system. It relied on texture descriptor differences in
bonafide and morphed images. The algorithm worked by obtaining a micro-texture
variation using Binarized Statistical Image Features (BSIF) and then making
the decision using a linear support vector machine. The same detection
technique was tested in [17] against two databases of printed-scanned images.
The reason for using print-scan images was to mimic the image quality in a
visa application process as the visa application process in many countries
requires submitting printed images that later get scanned to be saved in the
system [16]. The results showed that the detection performance of this
technique dropped compared to the digital images.

Spreeuwers et al. in [19] presented another MAD technique that was based on
local binary patterns (LBD). Experiments were conducted on multiple databases
and with different morphing algorithms to test the robustness of the proposed
method. The results obtained were comparable to the BSIF-based method on one
dataset, but the same performance could not be observed while testing on
multiple datasets.

The application of Fourier spectrum analysis on different facial
characteristics was first suggested by Ndeh de Mbah in [10]. This approach is
based on analyzing the power density of six identified facial features. The
decision of whether an image is morphed or bonafide is based on the total
score obtained from the six classifiers. Experiments were conducted on two
databases where the results varied greatly depending on the dataset used.
However, the reasoning behind the difference in results was not addressed in
the paper.

In this paper, we focus on analyzing the eyebrow region in the frequency
domain to distinguish between a morphed and bonafide image. We use bonafide
and morphed images from two different datasets containing ICAO-compliant
printed scanned bonafide images and their morphs created using two morphing
algorithms to test our approach. The proposed method is based on single-image
detection that can also be used in a differential detection scenario.

##  3 The Proposed Method

The proposed method is a texture-based detection method where the smoothness
property of the eyebrow region is studied to distinguish a morphed image from
a bonafide image. The eyebrow region due to the presence of hairs is expected
to have high-frequency content present in the Fourier domain. The study aims
to find if this high-frequency content is lost due to the smoothening effect
in a morphed image making it suitable to differentiate morphed images from
bonafide images.

The experiments were based on developing a segmentation technique to crop the
eyebrow region from a face image and then analyzing the segmented region in
the frequency domain. The different steps of the experiment pipeline are
described in the following subsections:

###  3.1 Eyebrow region segmentation

For eyebrow region segmentation, we used Dlib’s facial landmark detector to
locate the eyebrow region in a face image. The pre-trained shape predictor
provided by Dlib was used in this study [4] which is based on the dataset from
[8]. The Dlib shape predictor returns an array of length 68 containing
coordinates of different facial features including the eyebrows. Eyebrows are
marked by 10 array values with each eyebrow represented by 5 coordinates (Fig.
1(a)). These eyebrow coordinates are then used to find the limits of the
eyebrow region and crop a rectangle around the region as shown in Fig. 1. This
segmentation technique based on Dlib’s landmark detector proved to be very
effective for the ICAO-compliant images from the datasets.

(a) Sample image - Dlib landmarks

(b) Cropped eyebrow region

Figure 1: Cropping the eyebrow region

###  3.2 Pre-processing

In this step, we prepare the cropped region for the frequency domain analysis.
First, the image is converted to grayscale to be processed by the next stages.
Converting the images to grayscale is important for the system to be used in a
differential morph detection setting because some border control cameras only
provide the trusted live capture image in grayscale [18].

After converting the images to grayscale, the contrast of the cropped eyebrow
region is increased to enhance the variations in the cropped image. Contrast
enhancement is done through black clipping and white clipping. It works by
converting 1% percentage of the darkest grey pixels to black (black clipping)
and 5% of the brightest grey pixels to white (white clipping), and then the
rest of the gray pixels are scaled between the highest and the lowest values.

Since bonafide images are expected to have more variations, contrast
enhancement is expected to further enhance these variations. Contrary to this,
the morphed images because of their smoothened nature will be relatively less
affected by this step. This phenomenon is also shown in the results in section
4.3.1. Fig. 2 shows the image after going through the pre-processing step.

(a) Cropped eyebrow region

(b) Enhanced grayscale image

Figure 2: Preprocessing the cropped image

###  3.3 Fourier analysis

As explained earlier, the idea is to differentiate morphed images from
bonafide images by observing the smoothening of eyebrows in morphed images. In
a sharp image, hairs in the eyebrows can be observed as edges separate from
each other. These edges are represented by the high-frequency content in the
frequency domain.

Next, 2D Fourier transform of the preprocessed image is calculated to get the
frequency representation of the image. Since the interest here is in the
strength of the frequency content, only the magnitude of the Fourier transform
is considered in the analysis. Fig. 3 shows the averaged DFT magnitude spectra
of the eyebrow region of bonafide images and morphed (FaceFusion) images from
the FRGC dataset.

The plots are shifted to move the values associated with zero frequency to the
middle so that the frequency increases as we move away from the origin. It can
be observed from the DFT magnitude spectra that the outer circle for bonafide
images is bigger than the morphed images indicating a wider spread out of
high-frequency content in bonafide images. In our approach, we exploit this
difference in the frequency content to distinguish between morphed and
bonafide images.

(a) Bonafide

(b) Morhped

Figure 3: Averaged DFT magnitude spectra of eyebrow regions of bonafide and
morphed images

###  3.4 Calculating frequency content

Once we have the Fourier spectrum, the next step is to establish a way of
calculating the frequency content. We calculated the frequency content by
taking the sum of the complete magnitude spectrum. The normalized sum
calculated in our testing is expressed by the equation 1.

$sum=\frac{1}{MN}\sum_{n=1}^{N}\sum_{m=1}^{M}f(n,m)$ (1)

Here, f is an MxN array of the 2-dimensional DFT magnitude of the image, and M
and N are the length and width of the cropped eyebrow region, respectively.
The sum of coefficients is divided by the number of pixels in the cropped
region for generalizing because the size of the cropped region can vary among
different people and images of different resolutions.

##  4 Experimental Results

In the following subsections, the datasets used in the experimentation,
evaluation metrics, the experiment setup, and results are presented.

###  4.1 Datasets

The morphed and bonafide images used in this experimentation are taken from
[18]. The bonafide images belong to two different datasets i.e. FERET [12] and
FRGCv2 [11]. As described in [18], morphs are created by choosing the two
subjects among the same dataset. In addition, the subjects are chosen based on
their sex and whether they are wearing glasses.

622 bonafide images from the FERET dataset and 1440 images from the FRGC
dataset were used in this experiment. Both the bonafide and morph images are
post-processed by passing through a print and scan pipeline to mimic the post-
processing steps followed in a passport application process.

The morphed images are created using FaceFusion [5] and UBO Morpher [7]. More
information about the images used is provided in the table 1. Figures 5 and 4
show sample images from FERET and FRGCv2 datasets and their morphs created
using FaceFusion and UBO Morpher.

Table 1: Number of images in the datasets Dataset | Bonafide | FaceFusion | UBO Morpher
---|---|---|---
FERET | 622 | 529 | 529
FRGCv2 | 1440 | 964 | 964

(a) Subject 1

(b) Subject 2

(c) Facefusion

(d) UBO Morpher

Figure 4: Sample images from FRGC subset and their morphs

(a) Subject 1

(b) Subject 2

(c) FaceFusion

(d) UBO Morpher

Figure 5: Sample images from FERET subset and their morphs

###  4.2 Evaluation metrics

The proposed solution is tested based on the metrics established by the
ISO/IEC 30107-3 for performance assessment of presentation attack detection
mechanisms i.e. attack presentation classification error rate (APCER) and bona
fide presentation classification error rate (BPCER) [1]. APCER is a measure of
falsely accepting the morphed images as bonafide images. Whereas, the BPCER is
a measure of falsely rejecting the bonafide images by classifying them as
morphed images. Here, the detection equal error rate (D-EER) point (where
APCER=BPCER) is reported, along with the BPCER10 (where APCER = 10%) and
BPCER20 (where APCER = 5%), as described in [3].

###  4.3 Experiment setup and results

The experiments were carried out under different settings and with various
datasets to fine-tune and evaluate the reliability of the proposed method. The
final results are hereby reported in this section.

####  4.3.1 Effect of increasing contrast

For the preprocessing step, we experimented with increasing the contrast of
the cropped eyebrow region. Since image contrast can be used to enhance the
differentiation in the textures present in the image, the idea is that this
can further increase the frequency content of the bonafide images as compared
to the morphed images. Figure 6 shows the DET curves by varying the contrast
on the FRGC images. The ISO metrics are presented in table 2 where it can be
seen that the assumption is correct showing that increasing contrast helps
improve the efficiency of the system.

Figure 6: DET curves by varying contrast Table 2: Effect of enhancing contrast on morph detection Image Enhancement | D-EER (%) | BPCER10 (%) | BPCER20 (%)
---|---|---|---
No | 8.5 | 7.3 | 13.9
Yes (high contrast) | 6.5 | 4.2 | 9.6

####  4.3.2 Effect of cropping low-frequency content

Since the eyebrows are associated with the presence of high-frequency content,
we experimented with cropping the low-frequency coefficients in the frequency
spectrum. The experiments were conducted by ignoring 0%, 5%, and 10% of the
low-frequency region from the calculations. The results are presented in table
3 which show that cropping the low-frequency region slightly reduces the
performance of morphed image detection. Since the results are against the
proposed idea, this step was not incorporated in the final proposed algorithm.

Table 3: Effect of ignoring low-frequency component Crop (%) | D-EER (%) | BPCER10 (%) | BPCER20 (%)
---|---|---|---
0 | 6.5 | 4.2 | 9.6
5 | 6.6 | 3.8 | 9.9
10 | 6.7 | 4.1 | 9.8

####  4.3.3 Final results on different datasets

Table 4 shows the results by applying the proposed scheme on the two datasets
separately and then combining them. It can be seen that the proposed scheme
gives a much better result with the FRGC dataset than with the FERET dataset.
D-EER of 6.5% obtained with the FRGCv2 dataset is considerably lower than the
D-EER of 22.2% on the FERET dataset.

Table 4: Detection performance with different datasets Dataset | D-EER (%) | BPCER10 (%) | BPCER20 (%)
---|---|---|---
FRGCv2 | 6.5 | 4.2 | 9.6
FERET | 22.2 | 38.2 | 51.7
Combined | 14.2 | 17.02 | 23.2
Figure 7: DET curves obtained by applying the proposed scheme on different
datasets

On inspecting the images, it was found that the FERET images have lower
quality compared to the FRGC dataset. This can also be attributed to the fact
that the FERET images are relatively old (from 2011) compared to the FRGCv2
images (from 2014). Hence, due to the lower resolution of the images in the
FERET dataset, it is harder to differentiate between a bonafide image and a
morphed image. This claim is supported by the sizes of files from both
datasets as shown in the table 5. These results are also supported by the
findings in [18], where all MAD algorithms based on texture descriptors gave
better results for the FRGC dataset compared to FERET.

Table 5: Bonafide image sizes among different datasets (in Kb) Dataset | Min | Max | Average
---|---|---|---
FERET | 704.4 | 1218.6 | 924.7
FRGCv2 | 344.4 | 1065.8 | 722.2

####  4.3.4 Comparison with previous work

In this section, the results will be presented by comparing them with the
previous S-MAD techniques. For comparing the results with [10], experiments
are performed on the FRGC dataset by dividing the dataset into training and
testing subsets. The resulting error rates are presented in table 6. These
also include ACER which is the average classification error rate [10]. These
error rates are higher than the ones obtained in [10] showing that the
proposed scheme does not improve the detection performance. However, if we
consider the overall performance on different datasets, the proposed scheme
gives a lower D-EER of 22.2% in comparison with ACER of 38.24% in the other
paper.

Table 6: BPCER, APCER, and ACER values with FRGC dataset Subjects | Total | Rightly classified | Wrongly classified | APCER | BPCER | ACER
---|---|---|---|---|---|---
Bonafide | 720 | 680 | 40 | 5.5 | 4.9 | 5.2
Morphs | 964 | 916 | 48 |  |  |

##  5 Conclusion and Discussion

The results indicate the effectiveness of the proposed method in detecting
morphed images. The frequency spectrum analysis of the eyebrow region proves
to be a promising approach for the detection of morphed images in light of the
results. In addition to the S-MAD scenario, the proposed method can also be
applied in a D-MAD system where a reduction in error rate is expected. Even
though the detection capabilities were found to be dependent on the choice of
dataset, these results were expected given the quality of images varied
between the datasets, as explained in the results section. However, the
detection capabilities were found to be robust against two different kinds of
morphing techniques.

There are some limitations of using this approach that need to be investigated
further. 1) People with certain diseases can not have eyebrows. Our datasets
do not contain any such cases and hence the result of the segmentation method
and frequency analysis in these cases needs can be studied for improving this
method. 2) The morphed images used in the experiments are created from
automated morphing algorithms. For an attacker to conduct a successful attack,
a manually generated image is sufficient. So, further analysis can be made to
study the possibility of altering a morphed image to bypass the proposed
detection scheme.

## References

  * [1] Biometrics, I.: Iso/iec 30107-1: 2016. information technology biometric presentation attack detection. Part 1 Fram. Int. Organ. Stand (2016)
  * [2] Carlos-Roca, L.R., Torres, I.H., Tena, C.F.: Facial recognition application for border control. In: 2018 International Joint Conference on Neural Networks (IJCNN). pp. 1–7. IEEE (2018)
  * [3] Debiasi, L., Rathgeb, C., Scherhag, U., Uhl, A., Busch, C.: Prnu variance analysis for morphed face image detection. In: 2018 IEEE 9th International Conference on Biometrics Theory, Applications and Systems (BTAS). pp. 1–9. IEEE (2018)
  * [4] dlib.net/face_landmark_detection.py.html. http://dlib.net/face˙landmark˙detection.py.html, (Accessed on 04/30/2023)
  * [5] Facefusion - have fun mixing faces together! - home of the best face fusion app for iphone and ipad. http://www.wearemoment.com/FaceFusion/, (Accessed on 05/08/2023)
  * [6] Ferrara, M., Franco, A., Maltoni, D.: The magic passport. IEEE International Joint Conference on Biometrics pp. 1–7 (2014)
  * [7] Ferrara, M., Franco, A., Maltoni, D.: Decoupling texture blending and shape warping in face morphing. In: 2019 international conference of the biometrics special interest group (BIOSIG). pp. 1–5. IEEE (2019)
  * [8] ibug - resources - facial point annotations. https://ibug.doc.ic.ac.uk/resources/facial-point-annotations/, (Accessed on 04/28/2023)
  * [9] ICAO, D.: 9303-machine readable travel documents-part 9: Deployment of biometric identification and electronic storage of data in emrtds. International Civil Aviation Organization (ICAO) 123 (2015)
  * [10] Ndeh de Mbah, E.Y.: Morphed Image Detection using Local Spectrum Analysis. B.S. thesis, University of Twente (2021)
  * [11] Phillips, P.J., Flynn, P.J., Scruggs, T., Bowyer, K.W., Chang, J., Hoffman, K., Marques, J., Min, J., Worek, W.: Overview of the face recognition grand challenge. In: 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05). vol. 1, pp. 947–954. IEEE (2005)
  * [12] Phillips, P.J., Wechsler, H., Huang, J., Rauss, P.J.: The feret database and evaluation procedure for face-recognition algorithms. Image and vision computing 16(5), 295–306 (1998)
  * [13] Raja, S.V.R.R.K., Busch, C.: Face morphing attack generation & detection: A comprehensive survey
  * [14] Ramachandra, R., Raja, K., Busch, C.: Detecting morphed face images. In: 8th IEEE International Conference on Biometrics Theory, Applications and Systems, BTAS. pp. 1–7 (2016)
  * [15] Sadr, J., Jarudi, I., Sinha, P.: The role of eyebrows in face recognition. Perception 32(3), 285–293 (2003)
  * [16] Scherhag, U., Budhrani, D., Gomez-Barrero, M., Busch, C.: Detecting morphed face images using facial landmarks. In: Image and Signal Processing: 8th International Conference, ICISP 2018, Cherbourg, France, July 2-4, 2018, Proceedings 8. pp. 444–452. Springer (2018)
  * [17] Scherhag, U., Raghavendra, R., Raja, K.B., Gomez-Barrero, M., Rathgeb, C., Busch, C.: On the vulnerability of face recognition systems towards morphed face attacks. In: 2017 5th international workshop on biometrics and forensics (IWBF). pp. 1–6. IEEE (2017)
  * [18] Scherhag, U.J.: Face Morphing and Morphing Attack Detection. Ph.D. thesis (2020)
  * [19] Spreeuwers, L., Schils, M., Veldhuis, R.: Towards robust evaluation of face morphing detection. In: 2018 26th European Signal Processing Conference (EUSIPCO). pp. 1027–1031. IEEE (2018)
  * [20] Zhang, L.B., Peng, F., Long, M.: Face morphing detection using fourier spectrum of sensor pattern noise. In: 2018 IEEE international conference on multimedia and expo (ICME). pp. 1–6. IEEE (2018)