# ChatGPT as a Factual Inconsistency Evaluator for Abstractive Text
Summarization

Zheheng Luo, Qianqian Xie, Sophia Ananiadou
Department of Computer Science, The University of Manchester
{zheheng.luo, qianqian.xie<EMAIL_ADDRESS>hCorresponding
author

###### Abstract

The performance of abstractive text summarization has been greatly boosted by
pre-trained language models recently. The main concern of existing abstractive
summarization methods is the factual inconsistency problem of their generated
summary. To alleviate the problem, many efforts have focused on developing
effective factuality evaluation metrics based on natural language inference
and question answering et al. However, they have limitations of high
computational complexity and relying on annotated data. Most recently, large
language models such as ChatGPT have shown strong ability in not only natural
language understanding but also natural language inference. In this paper, we
study the factual inconsistency evaluation ability of ChatGPT under the zero-
shot setting by evaluating it on the coarse-grained and fine-grained
factuality evaluation tasks including binary natural language inference (NLI),
summary ranking, and consistency rating. Experimental results show that
ChatGPT outperforms previous SOTA evaluation metrics on 6/9 datasets across
three tasks, demonstrating its great potential for assessing factual
inconsistency in the zero-shot setting. The results also highlight the
importance of prompt design and the need for future efforts to address
ChatGPT’s limitations on evaluation bias, wrong reasoning, and hallucination.

##  1 Introduction

Recently, pre-trained language models have greatly improved the performance of
abstractive text summarization Liu and Lapata (2019); Lewis et al. (2020);
Zhang et al. (2020). However, a major concern of existing SOTA text
summarization methods is factual inconsistency, namely, the generated
summaries containing information that is not entailed by the source
article111the problem is also referred to as unfaithfulness, we use these
terms interchangeably in the following  with input documents Kryściński et al.
(2020); Maynez et al. (2020). To fill the gap, significant efforts have been
made in developing automatic evaluation metrics for assessing the factuality
of generated summaries, such as the semi-supervised evaluation method FactCC
Kryściński et al. (2020), the question-answering based evaluation method
QuestEval Scialom et al. (2021), and the natural language inference (NLI)
based method SummaC (Laban et al., 2022). However, these evaluation metrics
either have high computational complexity which requires training on a huge
amount of data or rely on multi-model combined pipelines, putting in more
uncertainties during inferences. Moreover, their correlations with human
judgement on consistency are limited(Pagnoni et al., 2021). Inspired by the
ability of pre-trained language models on natural language understanding and
generation, some efforts have been devoted to building data and computational-
efficient evaluation metrics based on pre-trained language models such as
BARTScore Yuan et al. (2021).

Most recently, large language models (LLMs), such as GPT-3 (Brown et al.,
2020), InstructGPT (Ouyang et al., 2022), PaLM (Chowdhery et al., 2022), and
BLOOM (Scao et al., 2022), have dwarfed fine-tuned models in various natural
language processing tasks by only few-shot or even zero-shot learning. Not
only perform well in natural language understanding and generation, LLMs also
have shown great potential in natural language inference and reasoning.
Specifically, equipped with explicitly designed prompts, LLMs can better solve
a range of various reasoning tasks in terms of arithmetics, symbolic, and
logic (Kojima et al., 2022; Wei et al., 2022). Moreover, the most recent
ChatGPT 222https://openai.com/blog/chatgpt in particular has been proven to
obtain strong natural language inference ability, surpassing fine-tuned BERT-
based models on several datasets (Zhong et al., 2023). As a result,
researchers have paid close attention to using large language models (LLMs) as
evaluation metrics for generated text. Kocmi and Federmann (2023a)
investigated the use of rating-based prompts in translation evaluation and
achieved better accuracy compared to other metrics across three language
pairs. Inspired by this work, Wang et al. (2023b) extends the method into a
broader natural language generation field including summarisation, where
ChatGPT shows dominating alignment with human rating on four attributes
including coherence, relevance, fluency, and consistency. However, their
experiments only use a single summarisation evaluation dataset and rather
focus on exploring ChatGPT to evaluate the overall quality of generated
summaries references. Moreover, They framed the evaluation as a marking task
and only compared the results with general text generation metrics such as
ROUGE Lin (2004) and BERTScore Zhang et al. , which have been proven to be not
effective in assessing factual consistency Maynez et al. (2020). Metrics
proposed specifically for assessing inconsistency such as FactCC, DAE (Goyal
and Durrett, 2020) SummaC have not been examined, leaving a huge gap for a
thorough exploration of using ChatGPT to assess the factual consistency in
text summarisation.

To fill the gap, in this paper, we conduct a preliminary study of how ChatGPT
can perform in both binary classification and fine-grained factual
inconsistency evaluation from three tasks including entailment inference (EI)
based inconsistency detection, ranking generated summaries based on factual
consistency, and scoring factual consistency of generated summaries. We design
different prompts on both zero-shot and zero-shot chain-of-thought Kojima et
al. (2022) to explore the factuality assessment ability of ChatGPT. We conduct
experiments on the benchmark of NLI based inconsistency task including six
large standardized inconsistency detection datasets, benchmark datasets on
other two tasks, and compare the results with SOTA evaluation methods. From
experimental results and analysis, we have the following findings:

  1. 1.

ChatGPT shows great potential for evaluation factuality of text summarisation
under the zero-shot setting and outperforms previous SOTA evaluation methods
on 6/9 datasets across three tested tasks.

  2. 2.

Though showing remarkable performance measured by numeric metrics, ChatGPT is
found to have the preference to predict a document and a claim is consistent
when the lexical similarity is high but the semantics are evidently different.
Moreover, evidence of ChatGPT conducting false inferences has been observed,
suggesting the limitation of ChatGPT’s language reasoning ability.

  3. 3.

Despite effectively instructing ChatGPT to detect inconsistency, the tested
prompts are not able to keep the output constantly sticking to the given
requirements, indicating the insufficient alignment of ChatGPT.

To the best of our knowledge, we are the first to systematically explore the
zero-shot factuality evaluation ability of ChatGPT on abstractive text
summarization. Overall, our results show a comparable or even better ability
of ChatGPT than SOTA evaluation metrics, but it has concerns on evaluation
bias and hallucination, which should be addressed to make it more reliable to
use.

##  2 Related Work

###  2.1 Factuality Evaluation in Abstractive Summarization

Existing factuality evaluation metrics generally can be classified into
unsupervised and semi-supervised methods. Unsupervised evaluation metrics
generally include information extraction (IE) based methods, natural language
inference (NLI) based methods, and question answering (QA) based methods.
Goodrich et al. (2019) proposed the model-based factuality evaluation metric
to calculate the overlap of relation tuples (subject, relation, object) that
are extracted from generated summaries and the ground truth by the information
extraction (IE) model. Nan et al. (2021) proposed the new evaluation metric
assessing the entity-level factuality consistency of generated summaries.

Besides the IE-based methods, natural language inference (NLI) is also
explored for factuality evaluation by assessing whether the generated summary
is entailed by the input document. Falke et al. (2019a) found the factuality
evaluation methods trained on the NLI datasets have a poor ability to the
assessment of text summarization. Mishra et al. (2021) further found that the
poor performance of the evaluation methods training with the NLI datasets is
caused by the short length of premises in NLI datasets. Most recently, Laban
et al. (2022) revisited the use of NLI in inconsistency detection by
calculating the factuality score based on sentence pairs, and proposed the
novel benchmark SUMMAC (Summary Consistency) with six datasets. SUMMAC is used
in our experiments.

Moreover, there is also question answering-based metrics such as FEQA Durmus
et al. (2020a), QAGS Wang et al. (2020), and QuestEval Scialom et al. (2021),
by assessing the alignment of the generated answer based on the generated
summary and the source, with the given question. Different from unsupervised
NLI-based methods, the semi-supervised methods further utilize the synthetic
data from text summarization for weakly supervised learning, such as FactCC
Kryściński et al. (2020). However, these methods are usually computationally
expensive or rely on annotated data Huang et al. (2021). Inspired by the
effectiveness of PLMs, there are efforts on developing factuality evaluation
metrics based on the likelihoods of PLMs, that are computation and data-
efficient such as BARTScore Yuan et al. (2021) and T5Score Qin et al. (2022).

###  2.2 ChatGPT for Natural Language Processing

Most recently, many efforts have explored the zero-shot ability of ChatGPT on
various natural language processing tasks. ChatGPT has been proven to exhibit
good performance on machine translation Jiao et al. (2023). On the GLUE
benchmark, Zhong et al. (2023) has found ChatGPT shows significantly better
performance on inference tasks, has comparable performance on sentiment
analysis and question-answering tasks, and has poor performance on paraphrase
and similarity tasks when compared with 4 representative BERT-based fin-tuning
methods. Qin et al. (2023) further shows ChatGPT has superior performance on
reasoning-required tasks including dialogue tasks, natural language inference
tasks, and question-answering tasks than GPT-3.5, and has worse performance on
the summarization task than GPT-3.5. ChatGPT and GPT-3.5 have comparable
performance on sentiment analysis. Bang et al. (2023) shown ChatGPT
outperforms SOTA zero-shot methods in 9/13 NLP datasets and has poor
performance on low-resource languages such as Marathi, Sundanese, and
Buginese. Yang et al. (2023) and Wang et al. (2023a) explored the query and
aspect-based text summarization and cross-lingual summarization with ChatGPT,
where it shows comparable performance with the fine-tuning-based methods. Wang
et al. (2023b) explored the ability of ChatGPT on evaluating natural language
generation (NLG) tasks such as summarization, story generation, and data-to-
text tasks. ChatGPT shows great potential as the NLG metric, whose evaluation
results have a high correlation with human judgment. However, they only
utilized one summarisation dataset and focused on exploring ChatGPT to
evaluate the relevance by comparing it with non-factuality evaluation metrics
such as ROUGE and BERTScore, leaving a huge gap for a thorough exploration of
the ability of ChatGPT to assess the factual consistency in text
summarisation.

##  3 ChatGPT as a Factual Inconsistency Evaluator

In this section, we introduce the details of three different tasks for
detecting inconsistency with ChatGPT including the prompt designing,
evaluation setting, tested datasets, and baseline models.

###  3.1 Entailment Inference

Evaluation Setting. Inconsistency evaluation of the generated summary can be
cast as a binary natural language inference classification, in which the
evaluation model is solely required to assess if the summary is consistent
with the source document rather than rating the consistent levels Laban et al.
(2022). Under this framework, two parameters are needed for the prompts:
source document and summary. We provide ChatGPT with the question including
the source document and the corresponding generated summary and ask it to
answer yes or no to infer the consistency between the source document and the
corresponding generated summary, and then we collect the decisions from the
outputs and aggregate the results.

Prompts. We experiment with two different zero-shot prompts in the NLI
setting. The first one is based on direct assessment by directly asking
ChatGPT to answer yes or no given the question. Another is based on zero-shot
Chain-of-Thought inspired by previous work (Kojima et al., 2022) of adding
"let’s think step by step" in prompt to encourage LLMs unfolding a chain-of-
thought style reasoning process, which has been proved to be effective on
several reasoning tasks. We follow the approach to create the second prompt.
The zero-shot template is shown below:

> Decide if the following summary is consistent with the corresponding
> article. Note that consistency means all information in the summary is
> supported by the article.
>
> Article: [Article]
>
> Summary: [Summary]
>
> Answer (yes or no):

The zero-shot CoT template is:

> Decide if the following summary is consistent with the corresponding
> article. Note that consistency means all information in the summary is
> supported by the article.
>
> Article: [Article]
>
> Summary: [Summary]
>
> Explain your reasoning step by step then answer (yes or no) the question:

When processing the responses, we only consider solid judgment like "the
summary is consistent with the article" as consistency, claims such as
"partially consistent" or ’mostly consistent’ are all deemed as inconsistent.
We also tried to use few-shot prompts. However, we found the performance
unstable when changing the label, order, and amount of examples, so we decide
to leave it for further exploration.

Datasets We evaluate ChatGPT’s performance on the SUMMAC benchmark (Laban et
al., 2022) which includes six largest summary inconsistency detection datasets
FactCC Kryściński et al. (2020), CoGenSumm Falke et al. (2019a), XSumFaith
Maynez et al. (2020), SummEval (Fabbri et al., 2021), FRANK Pagnoni et al.
(2021), and Polytope (Huang et al., 2020). Notably, not all the datasets in
the SUMMAC benchmark are built for binary consistency classification. For
example, in SummEval (Fabbri et al., 2021), generated summaries are marked on
consistency over a range from 1-5 points. SUMMAC standardizes the six datasets
into a binary classification format where each instance contains a triplet of
(document, summary, label). The label is either consistent or inconsistent.
Moreover, they manually created validation and test split for datasets where a
such split is not conducted and computed the inter-annotator agreement for
data with multiple annotators. The statistics of the benchmark are shown in
Table 1.

Baseline Models. We compare ChatGPT’s performance with the following methods:

  * •

NER Overlap uses the named entity recognition (NER) model to detect
inconsistency by examining if an entity in the summary is in the document
(Laban et al., 2021). The tested model considers only a subset of entity types
such as PERSON, LOCATION, ORGANIZATION, etc.

  * •

MNLI-doc fine-tunes a Roberta model (Liu et al., 2019) on the MNLI dataset
(Williams et al., 2018) and labels the document-summary pair by the predicted
probability of entailment.

  * •

FactCC (Kryściński et al., 2020) is a Roberta model fine-tuned on data
synthesized by corrupting sentences in the original documents as inconsistent
candidates.

  * •

DAE (Goyal and Durrett, 2020) is a parsing-based model evaluating
inconsistency by examining the entailment of individual dependency arcs.

  * •

FEQA (Durmus et al., 2020b) first generates question-answer pairs from
candidate summaries, then compare the answers extracted from the source
documents by asking the same questions. Then the answer sets are compared to
determine the consistency.

  * •

QuestEval (Scialom et al., 2021) extends the methods above by adding an
information recall score to a QA-based metric.

  * •

SummaC (Laban et al., 2022) builds an NLI matrix by splitting the document and
summary into sentence sets, then predicts a score for each sentence pair in
the matrix. SummaC zero-shot ($Summac_{zs}$) first obtain the maximum along
the columns then average over to get a final consistency score. SummaC
convolution ($SummaC_{Conv}$) instead trains a convolution layer to predict a
score for each column and then uses the mean output as the summary-level
score.

Detailed implementations of the above models used to compare can be found in
Laban et al. (2022). For scoring models, the threshold is selected using the
validation set and allowed to vary over different datasets.

Metric. Due to the unbalanced distribution of positive and negative samples in
the testing sets, we choose balanced accuracy (Brodersen et al., 2010) as the
main metric since it is more sensitive to predictions difference for data of
smaller proportions. Balanced accuracy is defined as following:

$bACC=\frac{1}{2}*(\frac{TP}{TP+FN}+\frac{TN}{TN+FP})$ (1)

The first term in the equation is sensitivity, which represents the recall of
true positives while the next one is specificity standing for the recall of
true negatives. We specifically counted the two sub-metrics to analyze
ChatGPT’s behavior.

Dataset |  Valid. size |  Test size |  %Positive |  Source
---|---|---|---|---
CoGenSumm |  1281 |  400 |  49.8 |  C
XSumFaith |  1250 |  1250 |  10.2 |  X
Polytope |  634 |  634 |  6.6 |  C
FactCC |  931 |  503 |  85.0 |  C
SummEval |  850 |  850 |  90.6 |  C
FRANK |  671 |  1575 |  33.2 |  C+X
Table 1:  Statistics of datasets in SUMMAC Benchmark.

###  3.2 Summary Ranking

Evaluation Setting Except binary NLI, a model’s awareness of factual
inconsistency can also be tested on how whether it can rank a consistent
summary over an inconsistent one. In this section, we introduce another
evaluation task Summary Ranking which is introduced in Falke et al. (2019a)
and has been tested in other previous work. Specifically, the model will be
asked to choose the consistent one over two candidate summaries (one is
faithful, the other one is not) given the source document.

Prompts We use a zero-shot prompt which directly asks ChatGPT to answer which
sentence out of the two candidates is more consistent with the given article
sentence.

> Decide which of the following summary is more consistent with the article
> sentence. Note that consistency means all information in the summary is
> supported by the article.
>
> Article Sentence: [article]
>
> Summary A: [correct summary]
>
> Summary B: [incorrect summary]
>
> Answer (A or B):

Dataset Here we use the dataset built by Falke et al. (2019a) which contains
373 samples, each containing an input source document from CNN/DM (Nallapati
et al., 2016) and two summary sentences covering the same content. One of the
summary sentences are consistent with the article while the other is
inconsistent.

Baseline Models We compare other evaluation models that reported their
performance on this dataset including the aforementioned FactCC (Kryściński et
al., 2020), MNLI-doc, DAE (Goyal and Durrett, 2020) and a human judgement from
Falke et al. (2019a).

Metric We report the accuracy of models successfully choosing consistent
summary over inconsistent one. Specifically, when collecting responses from
ChatGPT, we only deem claims that confirm the correct sentence is consistent
as correct. Outputs alleging both candidate sentences are consistent or
inconsistent are rendered as failures.

###  3.3 Consistency Rating

Evaluation Setting. Recently, several studies have found when given
accordingly request prompts GPT models are able to mark the quality of
generated text from different aspects (Kocmi and Federmann, 2023b; Fu et al.,
2023; Wang et al., 2023c). These scores show high correlations with human
assessment, suggesting the potential of ChatGPT in predicting fine-grained
consistency levels for summarisation. Moreover, in the experiments of the NLI
task in Section 3.1, we found that part of the output judgments is "partially
consistent" or "mostly consistent", indicating ChatGPT’s awareness of
different inconsistency degrees. Therefore, we apply the consistency rating
task on ChatGPT by asking it to mark the consistency of a summary with the
reference to its source document on a scale from 1-10 points, where 1 point
stands for total inconsistency, and 10 represents full consistency.

Prompts. Following Kocmi and Federmann (2023b)’s approach, we design a prompt
that requests ChatGPT to evaluate the consistency of a candidate summary w.r.t
the source article in a [1-10] scale:

> Score the following summary given the corresponding article with respect to
> consistency from 1 to 10. Note that consistency measures how much
> information included in the summary is present in the source article. 10
> points indicate the summary contains only statements that are entailed by
> the source document.
>
> [Summary]:
>
> [Source Article]:
>
> Marks:

The definition of consistency is added for the model to better understand the
aspect it is to rate.

Datasets. The original versions of SummEval and FRANK datasets are used on
this task given there are detailed consistency scores in their annotations. In
SummEval, 1600 summaries were labeled using a 5-point Likert scale along four
categories: coherence, consistency, fluency, and relevance by 3 expert
annotators. We average the points in the consistency aspect as the final
score. FRANK has to a binary consistency score for each sentence in a summary
labeled by annotators, then aggregates a summary-level score from 0-1,
resulting in 2250 marked summaries in total.

Baseline Models We compare other evaluation models that reported their
performance on this dataset including the aforementioned FactCC, FEQA, DAE and
QAGS (Wang et al., 2020), which is a QA-based faithfulness evaluation method.

Metrics To evaluate to what extent the examined models align with human
judgment. Two widely-used correlation measures are adopted: (1) Spearman
correlation (Zar, 2005) assesses the monotonic relationships between two
variables; (2) Pearman correlation (Mukaka, 2012) measures the linear
relationships between two sets of data; (3) Kendall’s Tau Kendall
(1938)evaluates the ordinal association between two measured quantities.

Methods | SUMMAC Benchmark Datasets |
---|---|---
CoGenSum |  XsumFaith |  Polytope |  FactCC |  SummEval |  FRANK
NER Overlap |  53.0 |  63.3 |  52.0 |  55.0 |  56.8 |  60.9
MNLI-doc |  57.6 |  57.5 |  61.0 |  61.3 |  66.6 |  63.6
FactCC-CLS |  63.1 |  57.6 |  61.0 |  75.9 |  60.1 |  59.4
DAE |  63.4 |  50.8 |  62.8 |  75.9 |  70.3 |  61.7
FEQA |  61.0 |  56.0 |  57.8 |  53.6 |  53.8 |  69.9
QuestEval |  62.6 |  62.1 |  70.3 |  66.6 |  72.5 |  82.1
SummaCZS |  70.4 |  58.4 |  62.0 |  83.8 |  78.7 |  79.0
SummaCConv |  64.7 |  66.4 |  62.7 |  89.5 |  81.7 |  81.6
ChatGPTZS |  63.3 |  64.7 |  56.9 |  74.7 |  76.5 |  80.9
ChatGPTZS-COT |  74.3 |  63.1 |  61.4 |  79.5 |  83.3 |  82.6
Table 2:  Balanced Accuracy results of inconsistency detect models on the test
set of SummaC. Results of baselinses are referenced from the paper Laban et
al. (2022).

##  4 Experiment

We conduct our experiments using the API of ChatGPT (gpt-3.5-turbo-0301) which
is trained based on InstructGPT (Ouyang et al., 2022) with reinforce learning
from human feedback (RLHF). To avoid the effects of historical dialogues, we
sent each request individually to obtain the response.

###  4.1 Entailment Inference

The full results of the entailment inference task are shown in Table 2.
Overall, ChatGPT is able to achieve comparable performance or even better
performance compared to the previous state-of-the-art evaluation models
without training on relevant tasks, demonstrating the potential of ChatGPT-
like LLMs on detecting inconsistency between two pieces of text in a zero-shot
setting. Specifically, ChatGPT with zero-shot CoT prompt produces the best
results and outperforms the previous SOTA method SummaCZS by 3.9%, 1.6% and
1.0% on CoGenSum, SummEval, and FRANK datasets correspondingly. It remains
comparable to the best models on the rest three datasets including XsumFaith
(63.1% compared to SummaCConv with 66.4%), Polytope (61.4% compared to
QuestEval with 70.3%), FactCC (79.5% compared to SummaCConv with 89.5%). In
almost all datasets, the ChatGPTZS-COT which guides the ChatGPT with the
chain-of-thought prompt has significantly better performance than ChatGPTZS,
In detail, ChatGPTZS-COT outperforms ChatGPTZS by 11.0%, 4.5%, 4.8%, 6.8% and
1.7% on the CoGenSum, Polytope, FactCC, SummEval, and FRANK datasets
correspondingly. It shows great potential to better explore the factuality
evaluation ability of ChatGPT by prompt engineering in the future.

Figure 1: The results of sensitivity and specificity of ChatGPTZS-COT.

To further investigate ChatGPT’s performance in consistent and inconsistent
instances, we break the balanced accuracy results of ChatGPTZS-COT into
sensitivity (positive recall) and specificity (negative recall), the
comparison is in Fig 1. In five out of the total six datasets, ChatGPT can
successfully retrieve more than 95% consistent summaries (high negative recall
namely specificity), while performing rather poorly on identifying all the
inconsistent ones (low positive recall namely sensitivity). Based on this
observation, we assume that during inference, ChatGPT might still rely more on
semantic similarity to make its decision on consistency detection since most
of the candidate summaries are lexically close to sentences in the source
articles, causing its vulnerability in finding these trivial modifications in
inconsistent summaries that changes the meaning of the source document. This
could be further demonstrated in ChatGPT’s reverse performance on the two
types of candidate summaries in the XSumFaith dataset which contains summaries
generated by models trained on the XSum dataset in Table 2. Previous works
(Durmus et al., 2020a) have shown that the generated summaries are highly
affected by training data and models trained on CNN/DM produce nearly
extractive summaries while the same model trained on XSum will give
significantly more abstractive ones. Abstrativeness brings the decline of
lexical similarity between the candidate summary and the source document which
might be the main reason why in XSumFaith, ChatGPT tends to predict more cases
as inconsistent.

###  4.2 Summary Ranking

Model |  Ranking Acc.
---|---
FactCC |  70.0
MNLI-doc |  78.3
Rule-based dependency |  74.8
DAE |  83.6
Human |  83.9
ChatGPT |  85.2
Table 3:  Performance of models on the summary ranking task. Results of
baselines are reported in Goyal and Durrett (2020).

The results of the summary ranking task are shown in Table 3. It shows that
ChatGPT without any in-context learning can outperform not only existing
methods but also a human assessment reported in Falke et al. (2019b). Notably,
the ranking dataset is sampled from the output of models trained on CNN/DM.
Therefore, the candidate summaries are mostly identical to some sentences in
the source document, the inconsistent ones tend to contain minor adjustments
corrupting the meaning like deleting modifiers like "half of" as shown in
Figure 2. Though we conclude from Section 4.1 that ChatGPT relies heavily on
lexical similarity to decide the consistency degree of sentences, in this
summary ranking task, we see that ChatGPT can detect the trivial semantic
differences even when given two highly similar candidates and pick out the
consistent one. For example, in the second case of Figure 2, ChatGPT can
correctly assess that sentence B is more consistent with the input article,
given the highly lexical similarity between sentence B and sentence A. In our
manual inspection, we found that ChatGPT is able to point out the
inconsistency in some cases where it failed in the entailment inference when
ranking them compared to their consistent counterparts. As shown in the first
case of Figure 2, ChatGPT failed in detecting the inconsistency of the summary
with the input article in the entailment inference task. While it can
correctly pick out the more consistent one when given two summaries with
highly lexical similarity in the summary ranking task, as shown in the second
case of Figure 2.

| FRANK | FRANK(CNN/DM) | FRANK(XSum) | SummEval
---|---|---|---|---
Metrics |  Pear. |  Spear. |  Pear. |  Spear. |  Pear. |  Spear. |  Pear. |  Spear.
$\rho$ |  $r$ |  $\rho$ |  $r$ |  $\rho$ |  $r$ |  $\rho$ |  $r$
FEQA |  0.00 |  0.01 |  -0.01 |  -0.01 |  0.02 |  0.07 |  - |  -
QAGS |  0.06 |  0.08 |  0.13 |  0.09 |  -0.02 |  0.01 |  - |  -
DAE |  0.16 |  0.14 |  0.25 |  0.24 |  0.04 |  0.28 |  0.20 |  0.27
FactCC |  0.20 |  0.30 |  0.36 |  0.33 |  0.07 |  0.25 |  0.32 |  0.34
ChatGPT |  0.70 |  0.69 |  0.50 |  0.46 |  0.34 |  0.27 |  0.49 |  0.35
Table 4:  Pearson correlation, spearman rank correlation coefficients and
p-values between human judgements and evaluation scores of different methods.
Figure 2: ChatGPT’s actions when given the same source document and an
inconsistent summary but with and without a consistent one. Red underlined
text in the article is content highly related to the candidate summaries

###  4.3 Consistency Rating

We further show the performance of all methods on the consistency rating task
in Table 4, where we compared the correlations of their rating results with
human judgement. Still, without in-context training, ChatGPT outperforms other
consistency metrics by aligning closer to human assessments. Especially in the
whole FRANK dataset, ChatGPT leads other metrics by a large margin,
emphasising its superior ability in measuring the consistency degree than the
baseline models.

In particular, when splitting the FRANK dataset into summaries from CNN/DM and
XSum, the correlations of ChatGPT show a considerable decline from CNN/DM to
XSum, which matches our analysis in the previous two parts. The difference
might come from the abstractiveness of summaries generated from models trained
on XSum, so their lower lexical similarity with the source document affects
the model’s judgement of consistency, leading to the worse performance in the
FRANK XSum dataset. However, though the abstractiveness of XSum summaries
lowers the correlations generally, ChatGPT’s pearson’s correlation is still
much higher than the single-digit results of the baselines, suggesting its
better language understanding and inference ability.

Figure 3: An example of ChatGPT fail to stick to the given definition of
consistency

###  4.4 Error Analysis

In this part, we show some example cases of ChatGPT in the three tasks to
showcase its limitations and attempt to provide a hint to understand ChatGPT’s
behavior when it was asked to detect inconsistency.

In Figure 2, we show an example from the CoGenSumm dataset where ChatGPT
failed in the entailment inference task. The model neglects the disappearance
of "half of " in the candidate summary which significantly changes the meaning
and decides the summary is consistent with the article. However, when putting
the same summary and article into the summary ranking task combined with a
consistent claim, ChatGPT successfully picks the consistent one and gives the
right reasoning of why "Summary A" is inconsistent. The first case of Figure 2
supports our assumption of ChatGPT counting on lexical similarity to determine
consistency as the high lexical overlap between inconsistent summary and the
red-underlined part in the article cheats ChatGPT. Nevertheless, when another
summary is both lexically and semantically closer to the article, ChatGPT
detects the difference and manages to answer correctly in the second case of
Figure 2.

With further investigation of failure cases, we found ChatGPT makes false
inferences as shown in Figure 4. The summary claims that "prime minister
matteo renzi" won the vote while the red underlined part in the article
clearly says the bill has passed the lower house but is held up to be approved
by both houses. However, ChatGPT determines this summary is consistent and
tries to justify it by using "the bill is approved by the lower house" as
evidence. This example, combined with the upper case in the first example,
demonstrates that ChatGPT still has a limitation on understanding and
inferencing of language. Furthermore, a CoT-style prompt is applied in this
example to encourage the model to generate a reasoning process to assist its
judgment. But ChatGPT directly produces the conclusion first and then unfolds
its inference progress afterward. According to the autoregressive training
nature of GPT, we guess the explanation is conditioned on the "consistent"
conclusion and thus cannot guide the decision. In our manual inspection,
answers with the conclusion at the first are not rare, suggesting zero-shot
CoT-style prompts might not be the optimal instruction for ChatGPT to conduct
a language inference task with reasoning progress. We suppose fined-engineered
few-shot prompts might help to guide ChatGPT’s generation and further improve
its performance and will investigate it in the future.

Moreover, there are examples that ChatGPT demonstrates limited comprehension
of given prompts. Fig 3 shows a case of the SummEval dataset in the
consistency rating task. Though the summary is short, the fact within it is
consistent with the article which ChatGPT also admits in the answer.
Therefore, all three experts mark 5 out of 5 for the summary. However, ChatGPT
then only rates the summary 1 point as it does not cover other facts in the
article which is not in the given marking rubric, showing an inadequate
understanding of giving prompts. This example demonstrates the insufficient
alignment brought by our tested prompt. Prompt engineering including human-in-
the-loop alignment optimization and few-shot in-context learning might be
helpful to better calibrate ChatGPT’s output.

Figure 4: An example of ChatGPT conducts wrong reasoning

##  5 Conclusion

In this paper, we comprehensively investigate the factual inconsistency
evaluation ability of ChatGPT in the zero-shot setting with three coarse-
grained and fine-grained factual inconsistency detection tasks. Our
experimental results empirically show the great potential of ChatGPT as a good
factual inconsistency evaluator, where it outperforms SOTA evaluation metrics
on six out of nine datasets. Although its great potential, ChatGPT is also
found to have limitations on evaluation bias, wrong reasoning, and
hallucination, which should be further addressed for its reliable use. The
experiments also show that ChatGPT’s performance can be significantly boosted
by the chain-of-thought prompt. We also analyzed the limitation of the chain-
of-thought prompt, which highlights the importance of prompt designing in
future work. The study in our paper is just the initial step in exploring the
inconsistency evaluation ability of ChatGPT, which we hope can provide useful
insights for future work in this direction.

## Limitations

Our study has the following limitations: 1) Due to the cost limitation of
using the API of ChatGPT, we only investigated the effectiveness of using
zero-shot prompts on three tasks. More effective prompts such as the few-shot
prompts can be explored in future work; 2) We only evaluated the performance
of ChatGPT on the factual inconsistency evaluation. A thorough comparison of
different large language models (LLMs) such as GPT-3.5 and GPT-4 can be
studied in future work, to help us figure out the superiors and limitations of
different LLMs.

## References

  * Bang et al. (2023) Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023.  A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity.  _arXiv preprint arXiv:2302.04023_.
  * Brodersen et al. (2010) Kay Henning Brodersen, Cheng Soon Ong, Klaas Enno Stephan, and Joachim M Buhmann. 2010.  The balanced accuracy and its posterior distribution.  In _2010 20th international conference on pattern recognition_ , pages 3121–3124. IEEE.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Durmus et al. (2020a) Esin Durmus, He He, and Mona Diab. 2020a.  FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5055–5070, Online. Association for Computational Linguistics.
  * Durmus et al. (2020b) Esin Durmus, He He, and Mona T Diab. 2020b.  Feqa: A question answering evaluation framework for faithfulness assessment in abstractive summarization.  _ArXiv_ , abs/2005.03754.
  * Fabbri et al. (2021) Alexander R Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021.  Summeval: Re-evaluating summarization evaluation.  _Transactions of the Association for Computational Linguistics_ , 9:391–409.
  * Falke et al. (2019a) Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019a.  Ranking generated summaries by correctness: An interesting but challenging application for natural language inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2214–2220, Florence, Italy. Association for Computational Linguistics.
  * Falke et al. (2019b) Tobias Falke, Leonardo F R Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019b.  Ranking generated summaries by correctness: An interesting but challenging application for natural language inference.
  * Fu et al. (2023) Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023.  Gptscore: Evaluate as you desire.  _ArXiv_ , abs/2302.04166.
  * Goodrich et al. (2019) Ben Goodrich, Vinay Rao, Peter J Liu, and Mohammad Saleh. 2019.  Assessing the factual accuracy of generated text.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 166–175.
  * Goyal and Durrett (2020) Tanya Goyal and Greg Durrett. 2020.  Evaluating factuality in generation with dependency-level entailment.  pages 3592–3603. Association for Computational Linguistics (ACL).
  * Huang et al. (2020) Dandan Huang, Leyang Cui, Sen Yang, Guangsheng Bao, Kun Wang, Jun Xie, and Yue Zhang. 2020.  What have we achieved on text summarization? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 446–469, Online. Association for Computational Linguistics.
  * Huang et al. (2021) Yichong Huang, Xiachong Feng, Xiaocheng Feng, and Bing Qin. 2021.  The factual inconsistency problem in abstractive text summarization: A survey.  _arXiv preprint arXiv:2104.14839_.
  * Jiao et al. (2023) Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023.  Is chatgpt a good translator? a preliminary study.  _arXiv preprint arXiv:2301.08745_.
  * Kendall (1938) Maurice G Kendall. 1938.  A new measure of rank correlation.  _Biometrika_ , 30(1/2):81–93.
  * Kocmi and Federmann (2023a) Tom Kocmi and Christian Federmann. 2023a.  Large language models are state-of-the-art evaluators of translation quality.  _ArXiv_ , abs/2302.14520.
  * Kocmi and Federmann (2023b) Tom Kocmi and Christian Federmann. 2023b.  Large language models are state-of-the-art evaluators of translation quality.  _ArXiv_ , abs/2302.14520.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  _ArXiv_ , abs/2205.11916.
  * Kryściński et al. (2020) Wojciech Kryściński, Bryan McCann, Caiming Xiong, and Richard Socher. 2020.  Evaluating the factual consistency of abstractive text summarization.  pages 9332–9346. Association for Computational Linguistics (ACL).
  * Laban et al. (2021) Philippe Laban, Tobias Schnabel, Paul Bennett, and Marti A. Hearst. 2021.  Keep it simple: Unsupervised simplification of multi-paragraph text.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6365–6378, Online. Association for Computational Linguistics.
  * Laban et al. (2022) Philippe Laban, Tobias Schnabel, Paul N Bennett, and Marti A Hearst. 2022.  Summac: Re-visiting nli-based models for inconsistency detection in summarization.  _Transactions of the Association for Computational Linguistics_ , 10:163–177.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019.  Text summarization with pretrained encoders.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3730–3740, Hong Kong, China. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _ArXiv_ , abs/1907.11692.
  * Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan T McDonald. 2020.  On faithfulness and factuality in abstractive summarization.  _ArXiv_ , abs/2005.00661.
  * Mishra et al. (2021) Anshuman Mishra, Dhruvesh Patel, Aparna Vijayakumar, Xiang Lorraine Li, Pavan Kapanipathi, and Kartik Talamadupula. 2021.  Looking beyond sentence-level natural language inference for question answering and text summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1322–1336, Online. Association for Computational Linguistics.
  * Mukaka (2012) Mavuto M Mukaka. 2012.  A guide to appropriate use of correlation coefficient in medical research.  _Malawi medical journal_ , 24(3):69–71.
  * Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016.  Abstractive text summarization using sequence-to-sequence rnns and beyond.  _arXiv preprint arXiv:1602.06023_.
  * Nan et al. (2021) Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, and Bing Xiang. 2021.  Entity-level factual consistency of abstractive text summarization.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2727–2733, Online. Association for Computational Linguistics.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. 2022\.  Training language models to follow instructions with human feedback.  _ArXiv_ , abs/2203.02155.
  * Pagnoni et al. (2021) Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021.  Understanding factuality in abstractive summarization with frank: A benchmark for factuality metrics.
  * Qin et al. (2023) Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023.  Is chatgpt a general-purpose natural language processing task solver?  _arXiv preprint arXiv:2302.06476_.
  * Qin et al. (2022) Yiwei Qin, Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2022.  T5score: Discriminative fine-tuning of generative evaluation metrics.  _arXiv preprint arXiv:2212.05726_.
  * Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Elizabeth-Jane Pavlick, and Suzana Ili’c et al. 2022.  Bloom: A 176b-parameter open-access multilingual language model.  _ArXiv_ , abs/2211.05100.
  * Scialom et al. (2021) Thomas Scialom, Paul-Alexis Dray, Patrick Gallinari, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, and Alex Wang. 2021.  Questeval: Summarization asks for fact-based evaluation.
  * Wang et al. (2020) Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020.  Asking and answering questions to evaluate the factual consistency of summaries.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5008–5020, Online. Association for Computational Linguistics.
  * Wang et al. (2023a) Jiaan Wang, Yunlong Liang, Fandong Meng, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2023a.  Cross-lingual summarization via chatgpt.  _arXiv preprint arXiv:2302.14229_.
  * Wang et al. (2023b) Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023b.  Is chatgpt a good nlg evaluator? a preliminary study.  _arXiv preprint arXiv:2303.04048_.
  * Wang et al. (2023c) Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023c.  Is chatgpt a good nlg evaluator? a preliminary study.  _ArXiv_ , abs/2303.04048.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, Quoc Le, and Denny Zhou. 2022.  Chain of thought prompting elicits reasoning in large language models.  _ArXiv_ , abs/2201.11903.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Yang et al. (2023) Xianjun Yang, Yan Li, Xinlu Zhang, Haifeng Chen, and Wei Cheng. 2023.  Exploring the limits of chatgpt for query or aspect-based text summarization.  _arXiv preprint arXiv:2302.08081_.
  * Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021.  Bartscore: Evaluating generated text as text generation.  _Advances in Neural Information Processing Systems_ , 34:27263–27277.
  * Zar (2005) Jerrold H Zar. 2005.  Spearman rank correlation.  _Encyclopedia of biostatistics_ , 7.
  * Zhang et al. (2020) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.  In _International Conference on Machine Learning_ , pages 11328–11339. PMLR.
  * (48) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.
  * Zhong et al. (2023) Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023.  Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert.  _ArXiv_ , abs/2302.10198.


Visualizing the outcome of dynamic analysis of Android malware with VizMal | 2020 | TRUE
ARTDroid: A Virtual-Method Hooking Framework on Android ART Runtime. | 2016 | FALSE
Mobile-sandbox: having a deeper look into android applications | 2013 | FALSE
Effectiveness of state-of-the-art dynamic analysis techniques in identifying diverse Android malware and future enhancements | 2020 | FALSE
Droid-sec: deep learning in android malware detection | 2014 | FALSE
AndroShield: automated android applications vulnerability detection, a hybrid static and dynamic analysis approach | 2019 | FALSE
Crowdroid: behavior-based malware detection system for android | 2011 | FALSE
Reevaluating android permission gaps with static and dynamic analysis | 2015 | FALSE
Android taint flow analysis for app sets | 2014 | TRUE
Leakminer: Detect information leakage on android with static taint analysis | 2012 | TRUE
Flowdroid: Precise context, flow, field, object-sensitive and lifecycle-aware taint analysis for android apps | 2014 | TRUE
Highly precise taint analysis for android applications | 2013 | TRUE
Do android taint analysis tools keep their promises? | 2018 | TRUE
Scalable and precise taint analysis for android | 2015 | TRUE
I know what leaked in your pocket: uncovering privacy leaks on Android Apps with Static Taint Analysis | 2014 | TRUE
A qualitative analysis of Android taint-analysis results | 2019 | TRUE
Apposcopy: Semantics-based detection of android malware through static analysis | 2014 | TRUE
HybriDroid: static analysis framework for Android hybrid applications | 2016 | TRUE
Static analysis of android apps: A systematic literature review | 2017 | TRUE
TaintMan: An ART-compatible dynamic taint analysis framework on unmodified and non-rooted Android devices | 2017 | TRUE
TrustDroid: Preventing the use of SmartPhones for information leaking in corporate networks through the used of static analysis taint tracking | 2012 | TRUE
Heaps’n leaks: how heap snapshots improve Android taint analysis | 2020 | TRUE
Appcaulk: Data leak prevention by injecting targeted taint tracking into android apps | 2014 | FALSE
BridgeTaint: a bi-directional dynamic taint tracking method for JavaScript bridges in android hybrid applications | 2018 | TRUE
On the effectiveness of dynamic taint analysis for protecting against private information leaks on Android-based devices | 2013 | TRUE
Leveraging historical versions of Android apps for efficient and precise taint analysis | 2018 | TRUE
Overview of information flow tracking techniques based on taint analysis for android | 2014 | TRUE
Poking the bear: Lessons learned from probing three android malware datasets | 2018 | FALSE
Binsign: fingerprinting binary functions to support automated analysis of code executables | 2017 | FALSE
Droidscope: Seamlessly reconstructing the OS and dalvik semantic views for dynamic android malware analysis | 2012 | FALSE
DroidNative: Automating and optimizing detection of Android native code malware variants | 2017 | FALSE
Don’t Pick the Cherry: An Evaluation Methodology for Android Malware Detection Methods | 2019 | FALSE
Automatic fingerprinting of vulnerable ble iot devices with static uuids from mobile apps | 2019 | FALSE
Unmasking android obfuscation tools using spatial analysis | 2018 | TRUE
Authorship attribution of android apps | 2018 | TRUE
Obfuscation fingerprinting in Android binaries | 2015 | TRUE
Android security: a survey of issues, malware penetration, and defenses | 2014 | FALSE
Hidemyapp: Hiding the presence of sensitive apps on android | 2019 | FALSE
Toward a more dependable hybrid analysis of android malware using aspect-oriented programming | 2018 | FALSE
Compiler Provenance Attribution | 2020 | FALSE
EveDroid: Event-aware Android malware detection against model degrading for IoT devices | 2019 | FALSE
Exploiting android system services through bypassing service helpers | 2016 | FALSE
Big Data fingerprinting information analytics for sustainability | 2018 | FALSE