# CyberMetric: A Benchmark Dataset for Evaluating Large Language Models
Knowledge in Cybersecurity

Norbert Tihanyi, Mohamed Amine Ferrag, Ridhi Jain, Merouane Debbah

###### Abstract

Large Language Models (LLMs) excel across various domains, from computer
vision to medical diagnostics. However, understanding the diverse landscape of
cybersecurity, encompassing cryptography, reverse engineering, and managerial
facets like risk assessment, presents a challenge, even for human experts. In
this paper, we introduce CyberMetric, a benchmark dataset comprising 10,000
questions sourced from standards, certifications, research papers, books, and
other publications in the cybersecurity domain. The questions are created
through a collaborative process, i.e., merging expert knowledge with LLMs,
including GPT-3.5 and Falcon-180B. Human experts spent over 200 hours
verifying their accuracy and relevance. Beyond assessing LLMs’ knowledge, the
dataset’s main goal is to facilitate a fair comparison between humans and
different LLMs in cybersecurity. To achieve this, we carefully selected 80
questions covering a wide range of topics within cybersecurity and involved 30
participants of diverse expertise levels, facilitating a comprehensive
comparison between human and machine intelligence in this area. The findings
revealed that LLMs outperformed humans in almost every aspect of
cybersecurity.

###### Index Terms:

Benchmrk Dataset, Large Language Models, LLM cybersecurity knowledge,
Generative AI,

##  I Introduction

The 18th century Industrial Revolution sparked a technological era,
introducing advancements like the steam engine that surpassed human and animal
labor. Innovations such as the 1830s McCormick Reaper in mechanized farming
and the 1910s assembly line by Henry Ford marked milestones, outperforming
human capabilities in efficiency. The mid-20th century witnessed the UNIVAC
[1] computer’s arrival in the 1950s, which performed complex calculations and
data processing faster than any human. By the 1970s, early chess programs
began challenging experienced human players, illustrating the evolving
potential of artificial intelligence in strategic thinking. In 1997, IBM’s
Deep Blue [2] defeating Garry Kasparov [3] marked a groundbreaking moment in
AI, showcasing machines surpassing humans in sophisticated intellectual
contests. This trend persisted with AlphaGo [4], developed by Google DeepMind,
outclassing world champion Lee Sedol in the intricate game of Go in 2016 [5].
Today, technology has advanced to the point where machines excel in tasks
requiring physical labor and sophisticated intellectual challenges like chess
and Go.

Over the past decade, advancements in Machine Learning (ML) have elevated the
field to a new level, making it possible to solve previously unattainable
tasks requiring a wider range and more complex forms of thinking. Large
Language Models (LLMs) have revolutionized Natural Language Processing (NLP),
resulting in substantial progress in automated text generation and
interactions that closely mimic human conversation. These breakthroughs are
now extending their influence into various sectors, including medicine [6],
finance [7], and notably, cybersecurity [8]. LLMs’ potential in cybersecurity
is immense, offering new possibilities in threat detection [9], policy
interpretation, and security strategy optimization. Cybersecurity, a broad and
diverse domain, encompasses cryptography, requiring a strong mathematical
foundation, creative thinking, and analytical skills for tasks like reverse
engineering and penetration testing. The field also includes risk assessment
and strategy planning, demanding management skills. Mastering this varied
spectrum is challenging due to the distinct skill sets each area demands.
Machine learning algorithms and large language models (LLMs) have performed
remarkably in numerous cybersecurity domains, often exceeding human
capabilities. Can machine intelligence surpass humans across the entire
breadth of cybersecurity? In this research, we aim to address the following
research questions:

  * •

RQ1: Do current LLMs have the ability to match or even surpass human experts
in various cybersecurity domains?

  * •

RQ2: Which currently available model ranks highest in efficiency relative to
its parameter size?

  * •

RQ3: In which domains do human experts demonstrate a notable superiority over
machines?

To address these complex research questions, three primary challenges must be
met: 1) a trustworthy, human-validated dataset must be created to evaluate
LLMs’ cybersecurity skills; 2) choosing a targeted set of questions for
assessing human experts; and 3) forming a fair comparison framework between
humans and machines is crucial. Although various datasets and surveys have
been conducted across different domains, a comprehensive study in
cybersecurity is still notably absent from existing literature.

This paper seeks to bridge this gap by introducing of CyberMetric, the first
comprehensive benchmark dataset created to assess the knowledge of LLMs across
the spectrum of cybersecurity. This includes areas ranging from cryptography
and network security to disaster recovery and identity management. The dataset
comprises $10,000$ carefully selected questions and answers, covering over
$500$ guidelines, standards, books, and research papers, which together span
more than $100,000$ pages. The creation of CyberMetric employs an innovative
semi-automated question-and-answer generation approach using different LLMs,
with further enhancement through the involvement of human experts at various
stages to ensure the dataset’s quality and relevance. Figure 1 represents the
nine domains covered in CyberMetric dataset.

Figure 1: Covered Domains in CyberMetric

The main contributions of this paper can be summarized as follows:

  1. 1.

CyberMetric, a comprehensive benchmark dataset of $10,000$ cybersecurity-
related questions designed to evaluate the understanding of various LLMs
across nine distinct domains within cybersecurity. Throughout the development
phase, substantial human effort is invested to eliminate incorrect or
inaccurate questions.

  2. 2.

An extensive survey, using 80 carefully selected questions, to compare the
performance of human experts and LLMs in the cybersecurity field. This in-
depth analysis involved 30 participants and various LLMs, broadly comparing
human expertise and machine intelligence.

  3. 3.

CyberMetric has been made accessible to the research community as a
foundational metric in cybersecurity at https://github.com/CyberMetric

CyberMetric serves a twofold purpose: firstly, as a robust dataset for
assessing the competency of LLMs in cybersecurity, and secondly, as a resource
to steer the development of LLMs specialized for this sector. CyberMetric was
used to perform a comparative study on the latest, highly-rated LLMs such as
GEMINI-pro [10], GPT-3.5 [11], GPT-4 [12], Llama2 [13], Mixtral-8x7B [14],
Zephyr-7B [15], and Falcon-180B [16], alongside evaluating the expertise of
cybersecurity professionals. These assessments not only shed light on the
strengths of LLMs in cybersecurity but also identified their limitations,
highlighting the necessity for cybersecurity-focused LLMs. Moreover, we
pinpoint the domains where humans currently still outperform LLMs, possibly
marking one of the last times in human history where this is the case. With
the rapid evolution of generative AI, these exceptions are expected to
diminish shortly. The rest of the paper is organized as follows. Section II
briefly overviews the prior literature. The details of dataset generation are
specified in Section III. The experimental setup, objectives, and results are
discussed in Section IV. We discuss our observations in Section V. Finally, we
conclude this paper in Section VI.

Figure 2: Framework for AI-driven question generation methodology,
incorporating human validation.

Ethical considerations. This study relies solely on documents available on the
internet to the public. CyberMetric includes a wide variety of standard and
diverse documents from the security field, incorporating standards, Open
Access research papers, special publications from the National Institute of
Standards and Technology (NIST), and Requests for Comments (RFC) documents.
During the collection phase, any documents that did not fall into the Public
Domain category were manually excluded from the analysis.

##  II Related work

As LLMs have evolved, the necessity for domain-specific benchmark datasets has
grown significantly. These datasets are vital for assessing LLMs’ competence
in specialized domains such as cybersecurity and providing direction in their
development and training. Curating QA datasets have gained enormous popularity
in recent years. Khot et al. [17] propose a multi-hop multiple-choice
questions (MCQs) dataset, QASC, using sentence composition over elementary and
middle school level science. OpenBookQA [18] is a similar MCQ dataset for
elementary science facts. COVID-QA [19] is a question-answering dataset
curated with articles related to COVID-19 [20]. JEC-QA [21] is a question-
answering dataset in legal domain. Event-QA [22] is a dataset for answering
event-centric questions over knowledge graphs. Several multi-lingual question-
answering datases have also been introduced, such as TYDI-QA, DuReader [23]
and DRCD [24].

Multiple other domains have also explored the concept of a QA dataset. For
instance, TriviaQA [25] stands out as a reading comprehension dataset,
encompassing over 650,000 question-answer-evidence triples. MultiMedQA [6]
stands as a benchmark in healthcare for medical question answering,
encompassing a broad spectrum that includes medical exams, medical research,
and consumer health queries. In 2022, Shah et al. introduced the Financial
Language Understanding Evaluation (FLUE) [26], comprehensive and open-source
benchmarks designed explicitly for the financial domain. In 2023, Maatouk et
al. introduced TeleQnA [27] dataset focused on telecommunications, featuring
$10,000$ questions. This dataset was developed using a similar approach,
employing LLMs to create a comprehensive collection.

Further, not only can these datasets be used in teaching and training in
respective topics, but similar techniques can also be used to extract relevant
content from specific domains, such as to find legal text relevant to
compliance requirements [28], to query project data [29], to answer questions
from software repositories [30] or to to extract relevant answers from
software forums [31, 32].

In a recent publication, Z. Liu introduced a cybersecurity dataset [33]
designed to evaluate the capabilities of LLMs. This dataset was generated
using GPT 4.0 and is entirely based on a single material: ”Computer Systems
Security: Planning for Success.” It includes around two hundred questions.
Notably, human comparison and validation were not part of the development
process. To address this gap, we present the CyberMetric dataset, comprising
$10,000$ AI-driven, cybersecurity-related questions in various domains. Human
experts dedicated over 200 hours to verifying the accuracy and relevance of
the questions, removing any that were not pertinent. Additionally, our
approach includes a thorough survey utilizing 80 carefully chosen questions to
compare the performance of human experts and various LLMs in the field.

##  III Methodology

Our methodology features a thoughtfully constructed Framework that harnesses
the synergy of human experts and LLMs for creating cybersecurity-related
questions. The framework for generating and evaluating the CyberMetric dataset
is illustrated in Figure 2. The Framework comprises five crucial phases: ➀
Data Collection, ➁ Question Generation, ➂ Question Post-processing, ➃ Question
Validation Phase, and ➄ Reference Dataset Creation Phase. Human involvement is
integral in three key stages: during question generation to eliminate invalid
and irrelevant questions; in the test phase, where cybersecurity experts with
over 10 years of experience validate and test question relevance and accuracy;
and in the survey phase, where 30 participants are recruited to answer 80
questions, enabling comparison between different LLMs and human intelligence.
Moreover, human participants’ feedback serves as reference points in the
analysis; for instance, if an LLM like GPT-3.5 achieves 87% accuracy on the 80
validated dataset but only 50% on the $10,000$ dataset, it indicates potential
issues with either the questions or the solutions. Before delving into the
analysis of the results, we outline the significant phases within the
framework in this section.

###  III-A ➀ Data Collection

The questions are generated from popular cybersecurity documents and
references available over the internet. The collected documents include open
standards, RFCs, publications, and guidelines from NIST and other
organizations. In total, we gathered 580 documents spanning more than
$100,000$ pages. All of the collected documents were in PDF format. Therefore,
we have utilized pdfminer111https://pypi.org/project/pdfminer/ to handle the
files and extract the text from them, which can then be fed into an LLM for
further processing. During this stage, we eliminated any irrelevant sections
of the text that were unrelated to cybersecurity (e.g., table of contents,
prefaces, acknowledgments, pictures, references, and appendices).

###  III-B ➁ Question Generation

During this stage, the extracted raw text is segmented into chunks to ensure
it can be effectively fed into an LLM (with each chunk being smaller than the
input token size of the LLM). Initially, we utilized the GPT-3.5 API to
generate ten questions for every $8,000$ tokens. This approach aims to
maintain a balanced representation of each publication. Creating $1,000$
questions from a ten-page document might introduce bias to the dataset. By
employing this method, we ensure that the distribution and representation of
questions align proportionately with the length of each topic. The generated
questions are reviewed by Falcon-180B to identify and correct grammatical and
semantic errors. We refer to this module as the FALCON validator, which also
filters out questions irrelevant to the topic. By applying semantic analysis
[34] to the text, we excluded questions unrelated to cybersecurity. For
instance, the question ”Which is the correct spelling: ’cybersecurity’ or
’cyber-security’?” primarily focuses on linguistic matters rather than
cybersecurity, despite containing the word cybersecurity. Throughout this
stage, non-security human validators prioritize evaluating the relevance of
the questions and ensuring correct English grammar while reviewing the
questions to ensure they are easily understandable for humans. In this phase,
a total of $11,000$ questions are generated by GPT-3.5. This extra $10\%$
margin allows for the removal of unnecessary or irrelevant questions. During
this phase, the FALCON validator eliminated 1.7% of the questions, while human
experts dedicated over $30$ hours to remove 2.3% of all irrelevant and
unrelated to cybersecurity questions. These questions comprise the
intermediate database, which consists of grammatically correct cybersecurity-
related questions. However, at this stage, neither LLMs nor human experts have
verified the accuracy of the solutions. Currently, we have $10,560$ questions
awaiting validation by human experts to identify and remove any questions
featuring multiple or incorrect solutions. All these questions will undergo
additional analysis involving various models and human experts.

###  III-C ➂ Question Post-processing

First, we employ the cutting-edge T5-base model from Google, which is trained
for grammar correction and can identify inaccuracies more efficiently than
human experts. This model identified 230 questions where the English could be
improved. For instance, the model identified inaccuracies such as ”Which of
the following elements do not apply to privacy?”. The sentence contains a
grammatical inconsistency related to subject-verb agreement. The subject
”which of the following elements” is singular because it refers to one element
among several. Therefore, the verb should also be singular to match the
subject, so the correct sentence would be: ”Which of the following elements
does not apply to privacy?” After conducting a grammatical error check, we
have again utilized the FALCON validator. However, this time, instead of
focusing on semantic checking, we aimed to eliminate non-contextual questions
that cannot be understood without external reference. Questions generated
during the creation phase, such as ”According to Figure 1” or ”As seen in
Table 6,” are nonsensical without relevant content. A thorough understanding
of English is essential to filter out these inappropriate questions. This is
where Falcon-180B comes into play, enabling the systematic identification and
correction of such issues. After fixing and removing such issues, we have used
GPT-4.0 to analyze all the remaining $10470$ questions and requested it to
highlight those questions where it is very confident that the answer is
incorrect. GPT-4.0 attained an accuracy 85% on the dataset and identified
around 1620 questions where it believes the answer is incorrect.

###  III-D ➃ Test Phase

We have reached a critical stage in our analysis. All the questions
highlighted by GPT-4 need to be reviewed by human experts to determine whether
the question is correct, as GPT-4 lacks the answer, or if GPT-4 was accurate
in flagging it as incorrect. Upon thoroughly examining the questions that
GPT-4 couldn’t answer correctly (despite providing correct reasoning in most
cases), we discovered that many questions were inaccurate or imprecise. We
categorize these issues in the following five categories:

  1. 1.

Multiple Correct Answers: Some of the curated questions in the dataset had
more than one correct answer; however, we devised our experiments to have only
one correct answer out of the provided four options.

  2. 2.

Time Relevant Questions: Since the questions and their answers are extracted
from resources from as long as ten years ago, the specified information may
not hold in the present. For instance, the response to this question, ”Which
web server dominates the market according to web surveys?” was Apache in 2015
(as per the source document’s publication date), but in 2023 it would be
Nginx.

  3. 3.

Incomplete Context: Some questions were partially formed and offered only
incomplete information, resulting in multiple potential solutions or no
solution.

  4. 4.

Incorrect Information in the Source: To our surprise, some selected sources
contained incorrect information despite being very reputed and widely
followed. We verified such incorrect statements with multiple field experts.

  5. 5.

Missing References: Some generated questions were deriving references from the
origin document, such as ”As per table 2” or “According to Figure 1”. These
questions cannot be answered without the referred content. These questions
should have been eliminated already, but we still found statements like ”In
chapter 4” or ”In the author’s opinion”.

After thoroughly reviewing and resolving question-related issues, we have
$10,170$ questions remaining. Our team dedicated over 200 expert hours during
the manual review process to rectify inaccuracies and ensure the elimination
of any false positives. The remaining 170 questions were removed from the
database through similarity checking using FALCON-180B. Questions that asked
the same or similar concepts were minimized in the database. For instance,
only one question remains from the pair ”What is the fundamental goal of
information security, according to the CIA triad?” and ”What is the primary
objective of information security, as outlined by the CIA triad?”, as they are
similar. In the last phase, an automated statistical analysis is carried out.
This is to confirm that all answers are consistent within the A, B, C, and D
options. Additionally, we have taken steps to ensure that the answers are
evenly distributed among the options. These thoroughly reviewed and finalized
questions establish the core of the CyberMetric dataset containing exactly
$10.000$ questions. In Table I, one can observe the final distribution of
questions.

TABLE I: CyberMetric Dataset: Questions Domain Distribution Domain |  Questions verified |  Number of Questions |  Creation Method
---|---|---|---
Penetration Testing / Ethical Hacking | ✔ | $1000$ | LLM & Human
Cryptography | ✔ | $1500$ | LLM & Human
Network Security / IoT Security | ✔ | $1000$ | LLM & Human
Information Security / Information Governance | ✔ | $1500$ | LLM & Human
Compliance / Disaster recovery | ✔ | $1500$ | LLM & Human
Cloud Security / Identity Management | ✔ | $1500$ | LLM & Human
NIST guidelines / RFC documents | ✔ | $2000$ | LLM & Human
CyberMetric | ✔ | $10000$ | LLM & Human

###  III-E ➂ Reference Dataset Creation

The primary aim of this study is to conduct a comprehensive assessment of LLMs
by comparing their performance to human responses when addressing security-
related queries. Asking $10,000$ questions would be logistically impractical
for human participants. While randomly selecting a subset, such as ten
questions, could serve a purpose, it introduces the risk of bias as some
questions may not adequately represent the entire spectrum of security
inquiries. To mitigate this challenge, a panel of experts in various
cybersecurity domains, each with a minimum of 10 years of experience and
holding internationally recognized certifications like CISSP, CISM, OSCP,
OSEP, and ISO 27001LA, collaboratively selected 80 questions. We refer to this
dataset as CyberMetric-80, which will be the foundation for comparing human
and machine intelligence in cybersecurity. These questions have been carefully
chosen to facilitate a fair and meaningful comparison between LLMs and human
respondents. We recruited 30 survey participants through our social networks
and contacted individuals from universities, research institutions, and Big4
consulting companies to volunteer. We have prepared a Google Forms survey
containing inquiries related to gender, age, years of experience, and the
highest level of education and requested volunteers, spanning from beginners
to experts, to complete the questionnaire. The comprehensive analysis of the
accuracy of the 30 participants and the LLMs will be discussed in Section IV.
Moreover, the CyberMetric-80 dataset can be used as a reference point. If
there’s a notable difference in LLM’s accuracy between CyberMetric and the
fully validated CyberMetric-80 dataset (e.g., 30%), it indicates that the
larger dataset lacks precision, or the solutions are wrong.

TABLE II: Comparison of various LLMs performance on CyberMetric-80.

LLM model | Company | Size | License | Run 1 | Run 2 | Run 3 | Run 4 | Mean | Std
---|---|---|---|---|---|---|---|---|---
GPT-4.0-1106-preview | OpenAI | 1.6T* | Proprietary | $97.50$ | $93.75$ | $96.25$ | $95.00$ | $95.63$ | $1.61$
Mixtral-8x7B-Instruct | Mistral AI |  $45$B | Apache 2.0 | $93.75$ | $92.50$ | $91.25$ | $92.50$ | $92.50$ | $1.02$
GEMINI-pro (Bard) | Google |  $137$B | Proprietary | $90.00$ | $91.25$ | $92.50$ | $90.00$ | $90.94$ | $1.20$
GPT-3.5-turbo-1106 | OpenAI | 175B* | Proprietary | $90.00$ | $87.50$ | $85.00$ | $87.50$ | $87.50$ | $2.04$
Falcon-180B-Chat | TII | 180B | Apache 2.0 | $82.50$ | $82.50$ | $82.50$ | $82.50$ | $82.50$ | $0.00$
Flan-T5-XXL | Google | 11B | Apache 2.0 | $81.75$ | $82.50$ | $81.75$ | $81.75$ | $81.94$ | $0.63$
Zephyr-7B-beta | HuggingFace | 7B | MIT | $81.25$ | $81.25$ | $81.25$ | $80.00$ | $80.94$ | $0.63$
Llama 2-70B | Meta | 70B | Apache 2.0 | $75.00$ | $72.50$ | $72.50$ | $75.00$ | $72.38$ | $0.14$
Mistral-7B-Instruct | Mistral AI | 7B | Apache 2.0 | $72.50$ | $72.50$ | $72.50$ | $72.50$ | $72.50$ | $0.00$
Falcon-40B-Instruct | TII | 40B | Apache 2.0 | $67.50$ | $66.25$ | $61.25$ | $61.25$ | $64.06$ | $3.28$
Llama 2-13B | Meta | 13B | Open | $55.00$ | $56.25$ | $52.50$ | $51.25$ | $53.75$ | $2.28$
Flan-T5-Base [35] | Google | 0.25B | Apache 2.0 | $51.25$ | $51.25$ | $51.25$ | $51.25$ | $51.25$ | $0.00$
Llama 2-7B | Meta | 13B | Open | $46.25$ | $46.25$ | $50.00$ | $43.75$ | $44.06$ | $2.95$
Dolly V2 12b BF16 [36] | Databricks | 12B | MIT | $33.75$ | $33.75$ | $32.5$ | $30.0$ | $32.50$ | $1.77$

* The exact model parameter size has not been officially revealed.

##  IV Experimental Results

The experiments can be classified into two main groups: the first involves
conducting thorough testing of LLMs, and the second focuses on evaluating
human performance. These two sets of results are then merged to gauge the
performance of humans versus machines in the field of cybersecurity precisely.

###  IV-A LLMs Performance on CyberMetric-80

Throughout the assessment, we test the most relevant LLM models. In our
testing configuration, each model undergoes execution and evaluation on an AWS
[37] ml.p4d.24xlarge instance equipped with 8x NVIDIA Tesla A100 GPU cards,
complemented by 382 GB of RAM, and running on Ubuntu 22.04. For large models
like Falcon-180B or Mixtral-8x7B, the minimum requirement to run on Amazon
SageMaker is the ml.p4d.24xlarge instance, which costs approximately $$40$ per
hour, resulting in a monthly cost of $$28,800$ USD. Due to variations in how
these models handle inputs, a dedicated framework was designed to enhance
query processing efficiency. Additionally, the choice of prompts played a
significant role in achieving the highest accuracy. For instance, in a 7B
model, the formulation of the prompt is critical. In contrast, in a larger and
more complex model like Falcon-180B, prompts become less relevant as the model
can comprehend most questions without further instructions. A simple ”Please
choose the best answer” suffices in such cases. The default temperature is
configured at 1.0, along with a $top\\_p$ parameter set to $0.9$ and $top\\_k$
set to $50$. We conducted four separate runs for each LLM model to assess
potential response inconsistencies. To evaluate variability, we calculated
both the standard deviation and mean values. Standard deviation is a critical
statistical tool used to measure the dispersion or variability within a
dataset. It is denoted as $\sigma$ and is calculated as:

$\sigma=\sqrt{\frac{1}{N}\sum_{i=1}^{N}(x_{i}-\mu)^{2}}$

where $N$ is the number of observations, $x_{i}$ is each observation, and
$\mu$ is the mean of these observations (in our specific scenario, N=4, and
the $x_{i}$ values represent the individual runs).

Table II displays the varying outcomes of the 14 tested LLMs, including the
four separate runs, standard deviation, and mean values. Standard deviation
calculation is key in enabling fair comparisons among different LLMs. It can
reveal the ”stability” of an LLM model. A smaller standard deviation might
indicate higher confidence in the model’s responses, suggesting consistency
across different iterations. Conversely, a large standard deviation implies
that the model’s answers vary significantly in different runs, indicating less
output reliability. It is important to mention that a lower temperature
setting, intended to introduce more freedom and randomness in the model’s
output, can substantially elevate the standard deviation. The most accurate
model we tested was GPT-4 with mean accuracy $95.63$% (c.f Table II). The two
most effective models available as open source are Mistral AI’s
Mixtral-8x7B-Instruct and TII’s Falcon-180B. Among these, the top-performing
model in terms of model size is HuggingFace’s fine-tuned Transformer
architecture known as Zephyr-7B-beta. This model comprises a modest 7 billion
parameters yet achieves an impressive mean accuracy of $80.94\%$. A thorough
examination of incorrect responses, weaknesses, and strengths will be
discussed in Section V.

###  IV-B Humans Performance on CyberMetric-80

TABLE III: Distribution of accuracy among participants.

EXPERIENCED PARTICIPANTS
---
# | E | D | A | G | R
P16 | 10+ | P.hD. | 35-50 | M | 88.75%
P29 | 10+ | P.hD. | 35-50 | F | 87.50%
P14 | 1-5 | MA/MSc | 35-50 | M | 87.50%
P24 | 10+ | BA/BSc | 35-50 | M | 86.25%
P26 | 1-5 | MA/MSc | 18-35 | M | 86.25%
P13 | 10+ | P.hD. | 50+ | M | 83.75%
P5 | 10+ | P.hD. | 35-50 | M | 82.50%
P3 | 5-10 | MA/MSc | 35-50 | M | 82.50%
P1 | 5-10 | MA/MSc | 18-35 | M | 76.25%
P25 | 5-10 | BA/BSc | 35-50 | M | 75.00%
P20 | 1-5 | Secondary | 18-35 | M | 72.50%
P30 | 5-10 | BA/BSc | 18-35 | F | 71.25%
P12 | 1-5 | MA/MSc | 18-35 | M | 71.25%
P22 | 1-5 | P.hD. | 18-35 | M | 70.00%
P9 | 1-5 | BA/BSc | 18-35 | M | 70.00%
P28 | 1-5 | Secondary | 18-35 | M | 68.75%
P2 | 1-5 | BA/BSc | 18-35 | M | 58.75%
P15 | 1-5 | MA/MSc | 18-35 | M | 58.75%
P21 | 1-5 | MA/MSc | 18-35 | F | 53.75%
Mean accuracy: |  $\approx$72.24%
BEGINNERS
# | E | D | A | G | R
P19 | 0 | BA/BSc | 18-35 | M | 63.75%
P23 | 0 | MA/MSc | 18-35 | M | 61.25%
P8 | 0 | BA/BSc | 35-50 | M | 55.00%
P27 | 0 | BA/BSc | 35-50 | M | 55.00%
P6 | 0 | MA/MSc | 35-50 | M | 51.25%
P18 | 0 | MA/MSc | 18-35 | F | 42.50%
P11 | 0 | MA/MSc | 35-50 | F | 37.50%
P4 | 0 | MA/MSc | 35-50 | F | 31.25%
P7 | 0 | BA/BSc | 35-50 | F | 21.25%
Mean accuracy: |  $\approx$46.58%
DISQUALIFIED PARTICIPANTS
P10 | 0 | MA/MSc | 35-50 | F | 87.50%
P17 | 0 | BA/BSc | 18-35 | F | 83.75%

Legend:
E: Experience D: Degree, A: Age, G: Gender, R: Result

We must first evaluate human accuracy to draw meaningful conclusions from the
LLMs’ accuracy. Responding to all 80 questions typically takes an expert
around 40 to 60 minutes; the considerable time investment indicates a
significant degree of determination and commitment. While a short
questionnaire comprising 4-5 questions might often be answered hastily or even
randomly, the extensive nature of an 80-question survey naturally serves as a
deterrent to casual or inattentive responses. The fact that participants
devoted enough time and effort to thoroughly answer such a large number of
questions strongly suggests that they were highly engaged and committed to the
task. This participation level indicates their determination to complete the
survey with due diligence. The participants also acknowledged this fact after
the survey’s completion. Altogether, $30$ participants completed our
CyberMetric-80 questionnaire. Two participants were excluded from the analysis
due to their unusually high accuracy levels despite lacking cybersecurity-
related experience. Their responses suspiciously mirrored mistakes made by the
GPT-3.5 model, raising concerns about the validity of their answers. Both
participants later confirmed this evidence themselves. Among the participants,
there were nine females and 21 males. The majority, accounting for 48.3% of
the participants, fell within the age range of $35$-$50$. Only one individual
was older than 50, while seven participants were between the ages of $18-35$.
Out of the total participants, $11$ had no prior experience in cybersecurity,
while the remaining $19$ individuals were highly skilled in cybersecurity. The
CyberMetric dataset exhibits a high diversity, presenting a significant
challenge for human experts. The highest score is $71/80\approx 88.75\%$ by an
individual holding a PhD and certifications in CISSP, OSCP, and ISO27001LA.
The median score among the $30$ participants is $56$, while the mean accuracy
is $53.83$. The mean accuracy for experienced participants (with at least 1-5
years of cybersecurity experience) is $72.24$. Beginners with no prior
cybersecurity experience can achieve scores ranging from 21.25% to 63.75%,
whereas the mean accuracy is $46.58$ on this dataset, indicating a solid
understanding of the English language, as some questions can be inferred from
the context.

##  V Discussion - Human vs Machine

This section analyzes humans’ performance and compares it with LLMs. We will
identify the most difficult questions for both humans and LLMs. Identifying
areas where LLMs encounter difficulties can facilitate the enhancement of
questionnaires where AI assistance can be provided if needed. This occurred in
our study, where we accurately identified two participants who utilized the
GPT-3.5 model, owing to the striking similarity in accuracy and the fact that
92% of the incorrect responses matched those generated by GPT-3.5.

###  V-A Participant Distirbution

Table IV (EXPERTS) and Table V (BEGINNERS) display the distribution of
participants categorized by gender and their highest degree obtained,
excluding the two participants with secondary school degrees from the
analysis.

TABLE IV: Statistical Distribution (EXPERTS) Degree | Gender | Mean Accuracy (%)
---|---|---
PhD | Female | 87.50
PhD | Male | 81.25
Msc | Male | 77.08
Bsc | Male | 72.50
BSc | Female | 71.25
MSc | Female | 53.75
TABLE V: Statistical Distribution (BEGINNERS) Degree | Gender | Mean Accuracy (%)
---|---|---
BSc | Male | 57.92
Msc | Male | 56.25
MSc | Female | 37.08
Bsc | Female | 21.25

In Table VI, the chi-square test results are presented, displaying a chi-
square statistic of 0.484 and a p-value of 0.487. This indicates no
statistically significant association between gender and the classification of
participants as either BEGINNERS or EXPERTS. The degree of freedom for this
test is 1, reflecting the comparison between two categories within a single
categorical variable. Table VII outlines the expected gender frequencies among
the BEGINNERS and EXPERTS groups under the null hypothesis that gender and
group classification are independent. The calculated expected frequencies for
females and males in both groups suggest that the observed gender distribution
across these groups does not significantly deviate from what would be expected
by chance, supporting the conclusion drawn from the chi-square statistic and
p-value presented in Table VI.

TABLE VI: Chi-square test results for the association between gender and classification as BEGINNERS or EXPERTS. Metric | Value
---|---
Chi-square Statistic | 0.484
p-value | 0.487
Degrees of Freedom | 1
TABLE VII: Expected gender frequencies among BEGINNERS and EXPERTS under the null hypothesis of independence. Group | Expected Females | Expected Males
---|---|---
Beginners | 2.7 | 6.3
Experts | 6.3 | 14.7

###  V-B Most Difficult Questions for Humans

The analysis revealed the questions that posed the greatest challenge to
participants. Table VIII highlights the top 5 questions where all participant
responses were at or below a 25% success rate.

TABLE VIII: Most challenging questions for Humans No. |  Questions
---|---
Q41 |  What is the essential requirement for the security of the Discrete Logarithm Integrated Encryption Scheme (DLIES) to be maintained?
Q42 |  Which document provides the recommendation for elliptic curve cryptography?
Q45 |  What is the purpose of implementing monitoring systems?
Q61 |  What is the primary goal of an Information Security Governance strategy?
Q66 |  According to NIST guidelines, what is a recommended practice for creating a strong password?
Figure 3: Heatmap of the Distribution of Incorrect Answers

Questions 41 and 66 posed similar challenges to both human beings and LLMs.
This similarity can be attributed to these questions being relatively new
recommendations from NIST and BSI. The heatmap showcasing the distribution of
answers for these specific questions is displayed in Figure 3. The proportion
of correct solutions in each question was notably low (the accurate responses
for questions Q41, Q42, Q61, Q66, and Q45 are C, D, B, C, and B,
respectively.)

###  V-C Most Difficult Questions for LLMs

Which domain poses the greatest challenges for LLMs? This serves as a critical
metric for conducting a meaningful comparison. Do humans encounter the same
difficulties, or are these issues entirely distinct? Here, we present some
fascinating findings, highlighting the most problematic domain for LLMs to
tackle in answering questions.

####  V-C1 Latest research direction

LLMs may encounter difficulties in providing accurate responses to questions
rooted in the most recent research findings, especially when their training
data is not up-to-date. Consider a simple example: ”What is the recommended
interval for changing passwords?” Previously, many guidelines stipulated a
mandatory 30-day password change interval for all organizations. Most models
undergo training with extensive data, and many standards and documents adhere
to this guideline, usually endorsing 30 days as the appropriate answer.
However, according to the latest guidelines, NIST SP 800-63B Section 5.1.1.2
paragraph 9 advises: “Verifiers SHOULD NOT require memorized secrets to be
changed arbitrarily (e.g., periodically). However, verifiers SHALL force a
change if there is evidence of compromise of the authenticator.”. The official
explanation, while simple, is not entirely straightforward as it challenges
the established research paradigm of password behaviors that has prevailed for
the past 30 years. Recent research indicates that users tend to select weaker
passwords when they know they must change them soon. When making these
changes, they often create a new password similar to their old one by applying
common modifications, like incrementing a number in the password. Since most
documents do not align with these updated guidelines, it is anticipated that
all LLMs will recommend users to change their passwords. CyberMetric includes
questions from the most recent recommendations, such as the latest NIST
recommendations, posing a significant challenge, even for security experts.
Only a limited number of models, such as GPT-4, can answer this question. They
may either possess internet access or utilize the Retrieval-Augmented
Generation (RAG) technique to retrieve pertinent new information from the
internet. Another question that advanced language models and humans often find
challenging concerns the latest Federal Office for Information Security (BSI)
recommendations. In January 2023, the BSI released the ”Cryptographic
Mechanisms: Recommendations and Key Lengths (BSI TR-02102-1)” guideline. This
guideline specifies that for the Discrete Logarithm Integrated Encryption
Scheme (DLIES), the prime number $p$ should be no less than 3000 bits, and the
prime number $q$ should be a minimum of 250 bits. The choice of $3000$ bits
for $p$, which is not a power of two, and the requirement for $q$ to be $256$
bits results in what appears to be an unbalanced prime scheme. Even expert
security analysts often find this question challenging, and many language
models struggle to provide correct answers when dealing with questions. In our
analysis, there was no occurrence where all the previously tested models
failed to answer this question in at least one of the four runs.

####  V-C2 Complex computations

Another instance where LLMs face difficulties is in scenarios requiring
precise calculations. Since many models are not equipped with RAG or access to
the internet and external tools, they might struggle with seemingly simple
questions. For example, a question like ”What is the result of the bitwise XOR
operation between 110101 and 101010 in binary?” can be challenging. While a
model like GPT-4.0 can easily handle this with an embedded Python tool to
ensure accurate results, models like Mixtral-8x7B-Instruct or GPT-3.5, lacking
access to proper external tools, might not always provide the correct answer.
This phenomenon exemplifies the consistent incorrect reasoning from
Mixtral-8x7B-Instruct, even following prompt modifications. As shown in
Listing V-C2, the XOR calculation is precise. Nevertheless, the model
indicates that a bitwise OR calculation occurred. GPT-4 can compose small
Python scripts, and even larger examples pose no issue.

Incorrect reasoning for XOR calculation

ANSWER B:

The bitwise XOR operation between 110101 and 101010 is 100111.

Here’s how you can calculate it:

110101

101010

——

011111 (this would be thre result of bitwise OR)

Similarly, the straightforward question, ”What is the CIDR notation equivalent
for the subnet mask 255.255.248.0?” should be easily answerable. However, most
models struggle to respond correctly to this query without external help.
Complex models still tend to produce incorrect answers in situations requiring
precise calculations, and one evident area for improvement is the utilization
of external tools to solve intricate mathematical or reasoning problems.

###  V-D Compariosn

Nonetheless, achieving a remarkable accuracy of $95.63$% (c.f TABLE II) by
GPT-4 or is an extraordinary feat. In our evaluation, GPT-4 outperformed all
existing freely accessible and monetized LLMs. This can be attributed to the
fact that GPT-4 can handle complex computations with correct reasoning. It
also generally writes code to solve prompted mathematical problems. The older
versions of OpenAI’s GPT lack this reasoning process and, thus, often produce
incorrect results [38].

Our study indicates that, in most cases, LLMs perform better than humans.
However, the questions are extracted from various sources containing vast
amounts of information, making it difficult for a human to get all the details
right. Conversely, LLMs, having been trained on vast quantities of data from
various domains, have an enhanced ability to understand a diverse range of
topics more effectively. Although LLMs have achieved notable proficiency in
complex topics, there is still significant scope for improvement, especially
in scenarios requiring external tools or support. Figure 4 illustrates a
comparison in a single chart based on CyberMetric-80, showcasing how human
experts, beginners, and various LLM models stack up against each other.

Figure 4: Human vs Machine in the Cybersecurity

In the last 5 yeras, LLMs have shown significant improvements across various
fields, including cybersecurity, leading to the assertion that no human is
competing against machines in tasks that require advanced analytical
capabilities. This trend suggests that humans and AI will no longer be
directly compared within the cybersecurity domain in challenges requiring
sophisticated thought processes. The advancements indicate a future where the
distinction between AI and human capabilities in cybersecurity becomes
increasingly irrelevant.

To assist the research community in conducting precise measurements, we have
made available the comprehensive CyberMetric dataset, containing 10,000
records, to evaluate the efficacy of LLMs. Additionally, we have created two
other datasets of varying sizes, CyberMetric-80 and CyberMetric-500,
containing $80$ and $500$ questions, respectively. These datasets are designed
for testing more resource-intensive LLMs like GPT-4, since running the full
set of $10,000$ questions multiple times can be prohibitively expensive.
Detailed performance analyses of several models on these extensive datasets
will be discussed in the subsequent section.

###  V-E The Big CyberMetric LLM test

We have conducted tests with various models on the CyberMetric-500 dataset.
The outcomes of various LLMs on the CyberMetric-500 dataset are presented in
Table IX. The mean accuracy was derived from four runs, similarly to our
approach in CyberMetric-80.

TABLE IX: Performance on CyberMetric-500 LLM model | Mean | Std
---|---|---
GPT-4-0125-preview | $94.30$ | $0.77$
GPT-3.5-turbo-1106 | $87.30$ | $0.87$
GEMINI-pro | $85.05$ | $0.88$
Falcon-180B-Chat | $77.80$ | $0.26$
Zephyr-7B-beta | $76.40$ | $0.0$

TABLE X: Comparison of various datasets based on their labeling classifications. Document Name |  Questions verified |  Document Title |  Publication Date |  Publicly Available |  Document Category |  Document token size
---|---|---|---|---|---|---
NIST SP 800-77 rev1. | ✔ | Guide to IPsec VPNs | $2022$ | ✔ | Network Security | $105\,932$
NIST SP 800-133 rev2. | ✔ |  Recommendation for Cryptographic Key Generation | 2020 | ✔ | Cryptography | $20\,214$
BSI TR-02102-1 | ✔ | Recommendation | 2023 | ✔ | Cryptography | $58\,606$
BSI TR-03184 | ✔ | Recommendation | 2023 | ✔ |  Information Security / Space system | $35\,847$

We selected five different LLMs for comparison:GPT-4, GPT-3.5, GEMINI-pro,
Falcon-180B, and Zephyr-7B each known for their robust performance. GPT-4,
GPT-3.5 and GEMINI-pro are accessible to anyone through an API, allowing for
easy testing of its capabilities. Falcon 180-B and Zephyr are available for
download and use under the Apache 2.0 license. For medium GPU capacity, we
recommend Zephyr 7B, which, despite having only 7 billion parameters, offers
impressively good performance and fast inference time. Given the
characteristics of the questions within the Cybermetric-80, we forecasted a
small drop in the overall accuracy for the CyberMetric-500 dataset. This
expectation arises due to the employment of several new Recommendations in the
compilation of the expansive dataset. Table X presents a selection of new
recommendations from BSI and NIST, which have also been incorporated into the
dataset.

##  VI Conclusion

In this research, we introduced the CyberMetric dataset, designed to evaluate
the knowledge of large language models (LLMs) within the cybersecurity domain.
Our study includes an in-depth comparison of LLM capabilities versus human
intelligence, involving 30 participants. The evaluation underscores the
strengths and limitations of LLMs in responding to security-centric questions.
These models demonstrate a remarkable ability to navigate and articulate
responses across a wide array of security scenarios and technical details.
Although LLMs offer unparalleled speed and breadth in processing information,
their reliance on historical data and the potential for embedded biases
necessitate careful validation of their outputs. Findings reveal that modern
LLMs frequently surpass cybersecurity experts in answering questions across
various areas, showcasing their advanced analytical prowess.

## Acknowledgments

We extend our heartfelt thanks to the volunteers who participated in the
intricate CyberMetric-80 survey, which necessitated considerable time and
effort. In today’s world, even finding support for a brief 5-minute survey is
challenging, so the dedication shown by participants in completing a complex,
80-question test is immensely appreciated and vital for the completion of this
research.

## References

  * [1] J. Presper Eckert, James R. Weiner, H. Frazer Welsh, and Herbert F. Mitchell.  The univac system.  In Joint AIEE-IRE Computer Conference: Review of Electronic Digital Computers, page 6–16. ACM, 1951.
  * [2] Murray Campbell, A Joseph Hoane Jr, and Feng-hsiung Hsu.  Deep blue.  Artificial intelligence, 134(1-2):57–83, 2002.
  * [3] Feng-hsiung Hsu.  Behind Deep Blue: Building the Computer That Defeated the World Chess Champion.  2022\.
  * [4] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al.  Mastering the game of go without human knowledge.  nature, 550(7676):354–359, 2017.
  * [5] Tanguy Chouard.  The Go Files: AI computer wraps up 4-1 victory against human champion.  Nature, page nature.2016.19575, March 2016.
  * [6] Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Abubakr Babiker, Nathanael Schärli, Aakanksha Chowdhery, Philip Mansfield, Dina Demner-Fushman, Blaise Agüera Y Arcas, Dale Webster, Greg S. Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomasev, Yun Liu, Alvin Rajkomar, Joelle Barral, Christopher Semturs, Alan Karthikesalingam, and Vivek Natarajan.  Large language models encode clinical knowledge.  Nature, 620(7972):172–180, August 2023.
  * [7] Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann.  BloombergGPT: A Large Language Model for Finance, March 2023.
  * [8] Mohamed Amine Ferrag, Ammar Battah, Norbert Tihanyi, Merouane Debbah, Thierry Lestable, and Lucas C. Cordeiro.  SecureFalcon: The Next Cyber Reasoning System for Cyber Security, July 2023.
  * [9] Mohamed Amine Ferrag, Mthandazo Ndhlovu, Norbert Tihanyi, Lucas C. Cordeiro, Merouane Debbah, and Thierry Lestable.  Revolutionizing Cyber Threat Detection with Large Language Models, June 2023.
  * [10] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  arXiv preprint arXiv:2312.11805, 2023.
  * [11] OpenAI.  Gpt-3.5, 2022.  https://platform.openai.com/docs/models/gpt-3-5.
  * [12] OpenAI.  Gpt-4, 2023.  https://openai.com/research/gpt-4.
  * [13] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [14] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al.  Mixtral of experts.  arXiv preprint arXiv:2401.04088, 2024.
  * [15] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al.  Zephyr: Direct distillation of lm alignment.  arXiv preprint arXiv:2310.16944, 2023.
  * [16] UAE TII.  Falcon-180b: A 180 billion token language model.  https://huggingface.co/tiiuae/falcon-180B, 2023.
  * [17] Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal.  Qasc: A dataset for question answering via sentence composition.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8082–8090, 2020.
  * [18] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  arXiv preprint arXiv:1809.02789, 2018.
  * [19] Timo Möller, Anthony Reina, Raghavan Jayakumar, and Malte Pietsch.  Covid-qa: A question answering dataset for covid-19.  In Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020, 2020.
  * [20] Fan Wu, Su Zhao, Bin Yu, Yan-Mei Chen, Wen Wang, Zhi-Gang Song, Yi Hu, Zhao-Wu Tao, Jun-Hua Tian, Yuan-Yuan Pei, et al.  A new coronavirus associated with human respiratory disease in china.  Nature, 579(7798):265–269, 2020.
  * [21] Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun.  Jec-qa: a legal-domain question answering dataset.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9701–9708, 2020.
  * [22] Tarcísio Souza Costa, Simon Gottschalk, and Elena Demidova.  Event-qa: A dataset for event-centric question answering over knowledge graphs.  In Proceedings of the 29th ACM international conference on information & knowledge management, pages 3157–3164, 2020.
  * [23] Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, et al.  Dureader: a chinese machine reading comprehension dataset from real-world applications.  arXiv preprint arXiv:1711.05073, 2017.
  * [24] Chih Chieh Shao, Trois Liu, Yuting Lai, Yiying Tseng, and Sam Tsai.  Drcd: A chinese machine reading comprehension dataset.  arXiv preprint arXiv:1806.00920, 2018.
  * [25] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer.  TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver, Canada, 2017. Association for Computational Linguistics.
  * [26] Raj Shah, Kunal Chawla, Dheeraj Eidnani, Agam Shah, Wendi Du, Sudheer Chava, Natraj Raman, Charese Smiley, Jiaao Chen, and Diyi Yang.  When FLUE Meets FLANG: Benchmarks and Large Pretrained Language Model for Financial Domain.  In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2322–2335, Abu Dhabi, UAE, 2022. Association for Computational Linguistics.
  * [27] Ali Maatouk, Fadhel Ayed, Nicola Piovesan, Antonio De Domenico, Merouane Debbah, and Zhi-Quan Luo.  TeleQnA: A Benchmark Dataset to Assess Large Language Models Telecommunications Knowledge.  2023\.
  * [28] Sallam Abualhaija, Chetan Arora, Amin Sleimi, and Lionel C Briand.  Automated question answering for improved understanding of compliance requirements: A multi-document study.  In 30th International Requirements Engineering Conference (RE), pages 39–50. IEEE, 2022.
  * [29] Jinfeng Lin, Yalin Liu, Jin Guo, Jane Cleland-Huang, William Goss, Wenchuang Liu, Sugandha Lohar, Natawut Monaikul, and Alexander Rasin.  Tiqi: A natural language interface for querying software project data.  In 32nd IEEE/ACM ASE, pages 973–977, 2017.
  * [30] Ahmad Abdellatif, Khaled Badran, and Emad Shihab.  Msrbot: Using bots to answer questions from software repositories.  Empirical Software Engineering, 25:1834–1863, 2020.
  * [31] Bowen Xu, Zhenchang Xing, Xin Xia, and David Lo.  Answerbot: Automated generation of answer summary to developers’ technical questions.  In 2017 32nd IEEE/ACM international conference on automated software engineering (ASE), pages 706–716. IEEE, 2017.
  * [32] Swapna Gottipati, David Lo, and Jing Jiang.  Finding relevant answers in software forums.  In 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011), pages 323–332. IEEE, 2011.
  * [33] Zefang Liu.  SecQA: A Concise Question-Answering Dataset for Evaluating Large Language Models in Computer Security.  2023\.
  * [34] Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing.  Sentiment analysis in the era of large language models: A reality check, 2023.
  * [35] Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al.  The flan collection: Designing data and methods for effective instruction tuning.  arXiv preprint arXiv:2301.13688, 2023.
  * [36] Mike Conover, Matt Hayes, Ankit Mathur, Xiangrui Meng, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, et al.  Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023.
  * [37] Amazon web services (aws).  https://aws.amazon.com/.  Accessed: .
  * [38] Yiannis Charalambous, Norbert Tihanyi, Ridhi Jain, Youcheng Sun, Mohamed Amine Ferrag, and Lucas C Cordeiro.  A new era in software security: Towards self-healing software via large language models and formal verification.  arXiv preprint arXiv:2305.14752, 2023.


Further author information: E-mail<EMAIL_ADDRESS>
# IMAGE-OI: an OIFITS extension and its application in OImaging to compare
image reconstruction algorithms

Ferréol Soulez  Univ. Lyon, Univ. Lyon 1, ENS de Lyon, CNRS, Centre de
Recherche Astrophysique de Lyon UMR5574, F-69230, Saint-Genis-Laval, France
Laurent Bourgès  Univ. Grenoble Alpes, CNRS, Observatoire des Sciences de
l’Univers de Grenoble (France)  Antoine Kaszczyc  Univ. Lyon, Univ. Lyon 1,
ENS de Lyon, CNRS, Centre de Recherche Astrophysique de Lyon UMR5574, F-69230,
Saint-Genis-Laval, France  Guillaume Mella  Univ. Grenoble Alpes, CNRS,
Observatoire des Sciences de l’Univers de Grenoble (France)  Martin Pratoussy
Univ. Lyon, Univ. Lyon 1, ENS de Lyon, CNRS, Centre de Recherche Astrophysique
de Lyon UMR5574, F-69230, Saint-Genis-Laval, France  Gilles Duvert  Univ.
Grenoble Alpes, CNRS, Observatoire des Sciences de l’Univers de Grenoble
(France)  Jacques Kluska  Institute of Astronomy, KU Leuven (Belgium)  Eric
Thiébaut  Univ. Lyon, Univ. Lyon 1, ENS de Lyon, CNRS, Centre de Recherche
Astrophysique de Lyon UMR5574, F-69230, Saint-Genis-Laval, France  John Young
University of Cambridge, Cambridge (UK)

###### Abstract

In interferometry, the quality of the reconstructed image depends on the
algorithm used and its parameters, and users often need to compare the results
of several algorithms to disentangle artifacts from actual features of the
astrophysical object. Such comparisons can rapidly become cumbersome, as these
software packages are very different. OImaging is a graphical interface
intended to be a common frontend to image reconstruction software packages.
With OImaging, the user can now perform multiple reconstructions within a
single interface. From a given dataset, OImaging allows benchmarking of
different image reconstruction algorithms and assessment of the reliability of
the image reconstruction process. To that end, OImaging uses the IMAGE-OI
OIFITS extension proposed to standardize communication with image
reconstruction algorithms.

###### keywords:

Image reconstruction, Optical long baseline interferometry

##  1 INTRODUCTION

Many image reconstruction algorithms (see [1] for a review) have been
developed to process optical interferometric data. Using these algorithms may
require substantial expertise (for instance you may have to learn a specific
programming language). To make image reconstruction accessible to as many
people as possible, we have developed a common graphical user interface (GUI)
to drive such imaging algorithms in a _user friendly_ way. For obvious
reasons, we also wanted to avoid, as far as possible, rewriting the image
reconstruction algorithms.

The purpose of this work is twofold: (i) proposes an IMAGE-OI OI-FITS
extension to communicate with image reconstruction software backends and (ii)
presents OImaging, a graphical user interface that uses this standard to
provide a common frontend for several image reconstruction algorithms.

##  2 Image-OI: a common interface for image reconstruction software packages

All existing algorithms take their input data in the OIFITS (either v1 or v2)
format [2, 3]. This format stores the optical interferometric data in FITS
binary tables. More generally, a FITS file [4] comprises a number of so-called
header data units (HDU), each HDU having a header part with scalar parameters
specified by FITS keywords111A FITS keyword consists of upper case latin
letters, digits, hyphen or underscore characters and has at least one
character and at most 8 characters. and a data part. The header part obeys a
specific format but it is textual and easy to read for an human. The data part
usually consists of binary data stored in various forms. We only consider FITS
_images_ (that is a multidimensional array of values of the same data type)
and FITS _binary tables_. These are sufficient for our purposes.

We are currently using an extension of the OIFITS format to build a common
software interface for image reconstruction algorithms 222further discussions
about IMAGE-OI can be found at https://github.com/JMMC-OpenDev/OI-Imaging-JRA.
This extension will soon be proposed to the OIFITS group for approval. By
adding FITS HDUs or introducing new FITS keywords, it is possible to provide
any additional data (_e.g._ the initial image) and parameters required to run
the image reconstruction process. Thus, the input file is a valid OIFITS file
(resulting from the merging of all the optical interferometric data to
process) with additional HDUs containing further data needed for the
reconstruction (for instance, the initial image) and with scalar input
settings provided as the values of FITS keywords. This is detailed in the
following subsections.

The HDUs that are specific to the interface specification described in this
paper have EXTNAME values prefixed with ’IMAGE-OI’. This is to distinguish
them from OIFITS HDUs, which use the prefix ’OI_’. At least three of these
HDUs must be added:

  * •

the image ’IMAGE-OI INITIAL IMAGE’ that contains the initial image guess
required to start the algorithm,

  * •

the binary table ’IMAGE-OI INPUT PARAM’ that contains all input parameters
needed for the reconstruction, and

  * •

the binary table ’IMAGE-OI OUTPUT PARAM’ that contains all output parameters
given by the reconstruction algorithm.

A command line tool for creating and editing these IMAGE-OI OIFITS files has
been developed, see https://github.com/jsy1001/OI-Interface-Tools.

###  2.1 Input parameters

All input parameters needed by the reconstruction algorithms must be stored in
a separate HDU. In anticipation of perhaps needing to store vector parameters
in future, this HDU shall be a binary table HDU. This binary table shall have
the EXTNAME keyword set to ’IMAGE-OI INPUT PARAM’ and shall contain all of the
non-image parameters, with the exception of the pixel size and the dimensions
of the reconstructed image which are specified by those of the initial image
(see below), itself stored in a dedicated (image) HDU. Table LABEL:tab:input-
params gives the minimal list of input parameters required by reconstruction
algorithms. Reconstruction algorithms shouldn’t modify this HDU. If the actual
value of an input parameter used by a algorithm changes, the new value must be
given in ’IMAGE-OI OUTPUT PARAM’.

Note that the definition of this ’IMAGE-OI OUTPUT PARAM’ HDU is sufficiently
versatile to also store parameters used by model fitting algorithms.

Table 1: FITS keywords used to specify the input parameters in header of the
binary table HDU with EXTNAME of ’IMAGE-OI INPUT PARAM’.  Data Selection
(mandatory)
---
Keyword | Type |  Description
TARGET | string |  Identifier of the target object to reconstruct
WAVE_MIN | real |  Minimum wavelength to select (in meters)
WAVE_MAX | real |  Maximum wavelength to select (in meters)
USE_VIS | string |  Complex visibility data to consider if any†
USE_VIS2 | logical |  Use squared visibility data if any
USE_T3 | string |  Bispectrum data to consider if any†
† value can be: ’NONE’, ’ALL’, ’AMP’ or ’PHI’.
Algorithm Settings
Keyword | Type |  Description
INIT_IMG | string |  Identifier of the initial image
MAXITER | integer |  Maximum number of iterations to run
RGL_NAME | string |  Name of the regularization method
AUTO_WGT | logical |  Automatic regularization weight
RGL_WGT | real |  Weight of the regularization
RGL_PRIO | string |  Identifier of the HDU with the prior image
FLUX | real |  Assumed total flux (1 is the default)
FLUXERR | real |  Error bar for the total flux (0 means strict constraint)
HDUPREFX | string |  Prefix that specifies the leading text to use in the HDUNAME of the final image

####  2.1.1 Data selection

To keep things simple, any sophisticated selection, merging or editing of the
data should be done by a separate tool. The image reconstruction software
applications shall assume that they receive clean input data. There are
however a few parameters devoted to the selection of data. The TARGET keyword
specifies the name of the target object to reconstruct. The value of this
keyword should match one of the identifiers in the column TARGET in the
OI_TARGET binary table of the OIFITS file. In order to restrict the types of
interferometric data used for the reconstruction, keywords USE_VIS, USE_VIS2
and USE_T3 should be set with values specifying which complex visibility data,
powerspectrum data and bispectrum data to use if any. More specifically,
keywords USE_VIS and USE_T3 take string values which are ’NONE’, ’ALL’, ’AMP’
or ’PHI’ to indicate whether all, none, only the amplitude or only the phase
of such data are to be used. Keyword USE_VIS2 has a boolean value indicating
whether to use the powerspectrum data. Not all algorithms can use all types of
data and the values of these keywords should be set (in the output file) to
reflect what was really used.

The primary objective is to consider monochromatic image reconstruction. The
interferometric data are generally available at many wavelengths. The result
will, in fact, be a gray image of the target built from the data in the
wavelength range (inclusive) specified by the FITS keywords WAVE_MIN and
WAVE_MAX. The wavelength range is given in meters.

Whatever these settings, the image reconstruction algorithms must honor the
FLAG column of the OI data. We recall that this field has a logical value
which is true when the corresponding piece of data should be discarded (in
addition to any data with NULL values) and false when it should be considered.

####  2.1.2 Algorithm settings

Any parameter needed by the reconstruction algorithm can be added as a keyword
in the ’IMAGE-OI INPUT PARAM’ HDU paying attention to the naming constraints
of FITS keywords. The algorithm settings keywords shown in table
LABEL:tab:input-params are common to several algorithms (if not all as e.g.
MAXITER). More generally, even for future keywords, algorithms should use the
same keywords for common parameters. The types of these keywords can be
logical, integer, real or string. A string keyword can be used to identify an
HDU containing additional information. For example, RGL_PRIO contains the
identifier of the HDU with the prior image needed by the reconstruction
algorithm.

###  2.2 Initial image

All reconstruction algorithms are iterative and require an initial image to
start with. The initial image is provided as a FITS image in one of the HDUs
of the input file. The primary HDU can be used to store the initial image but
in that case it has to be moved to another location in the output OIFITS file.
The pixel size and the dimensions of the initial image determine those of the
reconstructed image. The dimensions are given by the FITS keywords NAXIS1 and
NAXIS2 while the pixel size is given by the FITS keywords CDELT1 and CDELT2
(both values must have the same magnitude but may differ in sign).

It is not intended that the image reconstruction algorithm be able to deal
with any possible world coordinate system (WCS) nor with any possible
coordinate units. We therefore restrict this standard to WCS which can be
fully specified by the CRPIXi, CRVALi, CDELTi, CTYPEi, and CUNITi keywords
(where i is the axis number). Other WCS keywords like CROTAn, PCi_j, DCi_j,
should not be specified as their default values (according to the FITS
standard) are not suitable for us. The same WCS conventions hold for any
output image produced by the reconstruction algorithm. For any external
software packages to correctly display the images, the parameters of the WCS
must be completely and correctly specified. On the sub-arcsecond fields of
view of optical interferometry observations all projections are for all
purposes identical.

The conventions are that the first image axis corresponds to right ascension
(RA) and second image axis corresponds to declination (DEC) both relative to
the center of the field of view (FOV) specified by the keywords CRPIXi (in
fractional pixel units). If keywords CRPIXi are omitted, they default to the
geometric center of the FOV. The pixel size (which is specified by the
absolute values of CDELT1 and CDELT2) must be the same in both directions.
Following standard conventions for display of a celestial image, to have the
relative right ascension (RA) oriented toward East to correspond to the left
(first columns) of the image, CDELT1 should be strictly negative, while, to
have the relative declination (DEC) oriented toward North to correspond to the
top (last rows) of the image, CDELT2 should be strictly positive. By default,
the physical coordinate units are in degrees; otherwise CUNITi may be `’deg’`
for degrees or `’arcsec’` for arcseconds. Table LABEL:tab:initial-image
summarizes these rules.

The OIFITS norm specifies that RAEP0 and DECEP0 in OI_TARGET are the
coordinates of the phase center, which we assume to be the absolute world
coordinates of the reference pixel in the reconstructed image (Notes that if
the data is not phase referenced, the reconstructed image is shift invariant
and the absolute coordinates are unknown).

Table 2: Initial image HDU keywords. The HDUNAME of this HDU is specified by
the value of the INIT_IMG keyword in ’IMAGE-OI INPUT PARAM’.  Image Parameters
---
Keyword | Type |  Description
HDUNAME | string |  Unique name for the image within the FITS file
NAXIS1 | integer |  First dimension of the image
NAXIS2 | integer |  Second dimension of the image
CTYPE1 | string |  Coordinate name ’RA---TAN’ for 1st axis
CTYPE2 | string |  Coordinate name ’DEC--TAN’ for 2nd axis
CDELTi | real |  Physical increment along i-th dimension of the image (for i = 1 or 2)
CUNITi | string |  Physical units for CDELTi and CRVALi; defaults to ’deg’ if omitted
CRPIXi | real |  Index of reference pixel along i-th dimension (for i = 1 or 2); defaults to the geometric center of the field of view if omitted
CRVALi | real |  Physical coordinate of reference pixel along i-th dimension (for i = 1 or 2) and relative to the center of the field of view; defaults to 0 if omitted

###  2.3 Output parameters

It is proposed that the output format be as similar as possible as the input
format. The output file must provide the reconstructed image but also some
information for interpreting the result.

Any scalar output parameters from the image reconstruction shall be stored in
a binary table HDU with EXTNAME of ’IMAGE-OI OUTPUT PARAM’. Examples of scalar
outputs include $\chi^{2}$, any regularization parameters estimated from the
data, and any model parameters that are not image pixels (for example stellar
disk parameters in SPARCO[5]). If an algorithm automatically changes some
input parameters (e.g. UVMAX with [6]), the new value should be written in
this ’IMAGE-OI OUTPUT PARAM’ leaving ’IMAGE-OI INPUT PARAM’ untouched.

Standard output parameters are listed in Table LABEL:tab:output-params.

Table 3: FITS keywords used to specify the output parameters. These keywords
must be stored in a binary table HDU with EXTNAME of ’IMAGE-OI OUTPUT PARAM’.
Algorithm Results
---
Keyword | Type |  Description
LAST_IMG | string |  Identifier of the final image
NITER | integer |  Total iterations done in the current program run
CHISQ | real |  Reduced chi-squared
FPRIOR | real |  Regularization penalty
FLUX | real |  Total image flux
PROCSOFT | string |  Software name and version number
CONVERGE | boolean |  Set to ’T’ if the software stopped because it has converged

###  2.4 Final image

To compare the initial and the final images, they must be stored in different
HDUs, hence the FITS keyword HDUNAME is used to distinguish them (the EXTNAME
keyword should not be used in the primary HDU). As explained previously, the
dimensions, pixel size and orientation of the output image(s) are the same as
for the initial image.

As most image viewers are only capable of displaying the image stored in the
primary HDU, we suggest storing the initial image in the primary HDU of the
input file, but storing the final or current image in the primary HDU of the
output file. The idea is to have the most relevant image stored in the primary
HDU. For the image reconstruction algorithm and for any software designed to
display or analyze the results, the different images are distinguished by
their names (given by their HDUNAME keyword). The complete name HDUNAME of the
final image is set by the algorithm and must begin with ‘IMAGE-OI‘ followed by
the value of HDUPREFX given in ’IMAGE-OI INPUT PARAM’ if any. This HDUPREFX
keyword can be used by the caller to prevent identical HDUNAME conflicts.

In order to continue the iterations of a previous reconstruction run, the
image reconstruction algorithm may be started with the final image instead of
the initial one. To that end, there must be some means to specify the starting
image for the reconstruction. This is the purpose of the INIT_IMG keyword (see
Table LABEL:tab:input-params) in the input parameters HDU which indicates the
HDUNAME of the initial image. This HDUNAME must be unique within the file.

###  2.5 Model of the data

As each imaging algorithm may implement its own method for estimating the
complex visibilities given the image of the object, it is necessary that the
model of every fitted data point be computed by the algorithm itself rather
than by another tool.

The OIFITS format [2] specifies that optical interferometric data is stored in
binary tables as columns with specific names. As there is no restriction that
the tables only contain the columns specified by the standard, we propose to
store the model of the data in the same tables by adding new columns. The
additional columns have the prefix ’NS_MODEL_’ to distinguish them from the
columns defined by the OIFITS standard. The names of the new columns are
listed in Table 4. In this way it is very easy to compare the actual data and
the corresponding model values as computed from the reconstructed image and
the instrument model assumed by the reconstruction algorithm. Another
advantage of this convention is that the same format can be exploited to store
the values given by model fitting software.

Table 4: Colums inserted in OIFITS binary tables to store the values given by
the model. NWAVE is the number of wavelengths.  New columns in OI_VIS tables
---
Label | Format | Description
NS_MODEL_VISAMP | D(NWAVE) | Model of the visibility amplitude
NS_MODEL_VISAMPERR | D(NWAVE) | Model of the error in visibility amplitude (optional)
NS_MODEL_VISPHI | D(NWAVE) | Model of the visibility phase in degrees
NS_MODEL_VISPHIERR | D(NWAVE) | Model of the error in visibility phase in degrees (optional)
New columns in OI_VIS2 tables
Label | Format | Description
NS_MODEL_VIS2 | D(NWAVE) | Model of the squared visibility
NS_MODEL_VIS2ERR | D(NWAVE) | Model of the error in squared visibility (optional)
New columns in OI_T3 tables
Label | Format | Description
NS_MODEL_T3AMP | D(NWAVE) | Model of the triple-product amplitude
NS_MODEL_T3AMPERR | D(NWAVE) | Model of the error in triple-product amplitude (optional)
NS_MODEL_T3PHI | D(NWAVE) | Model of the triple-product phase in degrees
NS_MODEL_T3PHIERR | D(NWAVE) | Model of the error in triple-product phase in degrees (optional)

##  3 OImaging: a single graphical user interface for image reconstruction
algorithms

OImaging is a Java graphical user interface developed at JMMC333Jean Marie
Mariotti Center http://www.jmmc.fr/. With it, image reconstruction is carried
out in three main steps:

  * •

after loading an OI-FITS files, it creates and fills the ’IMAGE-OI INPUT
PARAM’ and ’IMAGE-OI INITIAL IMAGE’ HDUs with the values entered by the user
in the GUI input panel;

  * •

then the IMAGE-OI OI-FITS file is sent to a server (either locally or remotely
at JMMC) that launches the chosen algorithm and performs the reconstruction;

  * •

finally the reconstructed image is sent back to OImaging and is shown in the
result panel.

###  3.1 Input panel

Figure 1: OImaging Input Panel

When started OImaging opens on the input panel that is used to load the data,
select the algorithm and set the parameters.

####  3.1.1 OIFITS file loading

The data can be loaded either through the file loader or through the SAMP
protocol [7]. This protocol is for sending data directly from one software
program to another without saving it to a file. As an example, OIFITS files
can be sent from the OIdb database444http://oidb.jmmc.fr/index.html [8]
without explicitly saving them locally. If several files are selected they are
just merged all together without any kind of filtering. When loading an OIFITS
file, OImaging will check if it contains an ’IMAGE-OI INPUT PARAM’ HDU. If so,
it will fill the fields of the input panel with the stored values. Otherwise,
it will create it with default values.

####  3.1.2 Data visualization

Loaded data can be explored in the visualization sub-panel. In this sub-panel,
the user can plot the data. He can also explore all the values stored in the
OIFITS files: all the tables containing the data and in particular the values
of the image reconstruction parameters stored in the ’IMAGE-OI INPUT PARAM’.
This visualization sub-panel can also be used to display and modify all the
images stored in the IMAGE-OI OIFITS file, in particular the initial image and
the prior image.

####  3.1.3 Data selection

As explained above, any sophisticated selection, merging or editing of the
data should be done by a separate tool such as JMMC’s OIFITSExplorer. However,
basic selection tools are available in the input panel, namely the target and
the wavelength range. In addition, the user can select the types of data used
in the reconstruction from the amplitudes and the phases of the visibilities
(VISAMP and VISPHI), the squared visibilities (VIS2) and the amplitudes and
the phases of the bispectra (T3AMP and T3PHI).

####  3.1.4 Algorithm selection

The first step for performing image reconstruction is to select the algorithm.
At the time of writing, four algorithms are available: WISARD[9], BSMEM[6],
MiRA[10] and SPARCO[5]. For each algorithm a small help window can show a
brief description, the reference paper and a description of all supported
parameters.

####  3.1.5 Parameter settings

Once the algorithm has been selected, OImaging automatically updates the input
fields used to set the parameters. There are three groups of parameters:

  * •

first the parameters shared by all algorithms, namely the number of iterations
MAXITER, the regularization name RGL_NAME, the regularization weight RGL_WGT
and the error on the total flux FLUXERR;

  * •

the second group contains parameters specific to the algorithm and/or the
regularization function;

  * •

in addition, there is a text field for Manual options that will be appended to
the command line used to invoke the reconstruction software. This is a
convenient way to access unsupported features that may change the algorithm
behavior or the log contents. It is mainly intended for debugging.

####  3.1.6 Initial and prior images

All algorithms need an initial guess of the image. As explained in Sec. 2.2,
the pixel size and the dimensions of the initial image determine those of the
reconstructed image. This initial image can be specified by several means:

  * •

loading a FITS file locally,

  * •

sending the image from another software like SAO-DS9[11] or LITpro[12]. This
is particularly convenient as an initial estimate of the target morphology can
be done with the model fitting software LITpro. LITpro can then generate the
modeled image and send it to OImaging as an initial guess;

  * •

OImaging has a small widget accessible by the create image button to create an
image of a Gaussian. Depending of the Gaussian width, it can approximate
either a Dirac or a constant background. In this widget, OImaging suggests
default sizes of both the field of view and the pixel according to the size of
the telescope and the maximum baseline;

  * •

after some reconstruction iterations, previous results can also be used as
initial image; also

  * •

any loaded image can be edited to change is pixel size or its field of view by
the mean of the modify image button at the top right of the image.

Some regularization functions require a prior image, which can be viewed using
the view radio-button next to the parameter field. By default, if no prior
image is set, the initial image is also used as the prior image.

####  3.1.7 Running the algorithm

When all parameters are correctly set, the algorithm can be run by clicking on
the Run button. This button stays grayed out when there is an issue with the
parameters. In that case, the user must read the message in the box below it.
A second click on the Run button while the reconstruction is in progress
cancels the computation.

###  3.2 Result panel

Figure 2: OImaging Result Panel

The reconstructed images are shown in the result panel. This panel is composed
of two sub-panels: the visualization sub-panel and the results table.

####  3.2.1 Results table

All results are stored in the results table at the bottom of the result panel.
This table gives an overview of all the parameters stored in the result OI-
IMAGE OIFITS files: either the input parameters such as the algorithm or the
regularization weight or the output parameters computed by the algorithm such
as the $\chi^{2}$ or the value of the regularization function. The table
contains additional fields such as “Success” that show if the software stopped
with an error. It also contains RATING and USERNOTE fields to allow the user
to add additional information concerning a given reconstruction. These fields
are all stored in the resulting OIFITS file.

To help the user navigate the results, the table can be sorted and edited.
Using the Table Editor button, the table is fully customizable: any keyword
present in the OIFITS file can be set as field in the table. Reconstruction
results can be also be removed from the table by means of the Delete button.

When several results are selected, the compare button displays the
reconstructed images side by side.

####  3.2.2 Result visualization

The selected result in the panel can be explored in a visualization sub-panel
similar to the visualization sub-panel of the input panel. This visualization
sub-panel is itself composed of four sub-panels:

  * •

OIFits data sub-panel shows all the data stored in the OI-FITS tables either
as text or as graphs. As the algorithms store the model of the data as
additional columns with the prefix ’NS_MODEL_’, the modeled data and the
residuals can also be plotted in this panel.

  * •

Images sub-panel can display all the images contained in the OI-FITS file, by
default the result image but also the initial image or the prior image
depending on the HDU name selected in the menu at the top of the panel. This
Image sub-panel also has two buttons at the top: a Set as init img button that
put the displayed image in the initial image list in the input panel and a
modify image button to change the sizes of the pixels and the field of view.
In that case, the modified image does not overwrite the current image but is
rather put the initial image list in the input panel. This sub-panel contains
also a ruler button to help the user to measure distance and angle on the
image.

  * •

Execution log sub-panel shows the output log of the algorithm.

  * •

Parameters sub-panel shows the parameters set in the HDUs ’IMAGE-OI INPUT
PARAM’ and
’IMAGE-OI OUTPUT PARAM’.

####  3.2.3 Interacting with the results

In addition to the visualization sub-panel and the results table, the result
panel shows four buttons for interacting with the selected results:

  * •

Continue uses the reconstructed image as the initial image to perform MAXITER
more iterations without changing any other parameters. It is a convenient way
to perform a reconstruction for a large number of iterations checking the
image every MAXITER iterations.

  * •

Restart updates all the parameters in the input panel with the values in
’IMAGE-OI OUTPUT PARAM’ and with the same initial image.

  * •

Update parameters is equivalent to Restart but using the result image as the
initial image in the input panel.

  * •

Save OIFits files saves the selected result.

##  4 Conclusion

We have presented in this paper a new OI-FITS extension: IMAGE-OI. This new
standard has two mains purpose: (i) to standardize communicating with image
reconstruction software backends and (ii) to store in a single file along with
the reconstructed image, the data and all the parameters that where used in
the reconstruction process enabling its reproducibility.

In a second part, we presented presents OImaging, a graphical user interface
that uses this standard to provide a common frontend for several image
reconstruction algorithms. This open source software is freely available on
JMMC website555https://www.jmmc.fr/english/tools/data-analysis/oimaging/. Its
development is still active and several feature may be added in the future
(such that the automatic fit of a simple model to generate the initial image,
tools to perform batch processing with different parameters). Support for this
software is provided via the network of VLTI Expertise Centers and help can be
requested at https://apps.jmmc.fr/feedback/.

## References

  * [1] Thiébaut, É. and Young, J., “Principles of image reconstruction in optical interferometry: tutorial,” JOSA A 34(6), 904–923 (2017).
  * [2] Pauls, T. A., Young, J. S., Cotton, W. D., and Monnier, J. D., “A data exchange standard for optical (visible/IR) interferometry,” PASP 117, 1255–1262 (Nov. 2005).
  * [3] Duvert, Gilles, Young, John, and Hummel, Christian A., “Oifits 2: the 2nd version of the data exchange standard for optical interferometry,” A&A 597, A8 (2017).
  * [4] Pence, W. D., Chiappetti, L., Page, C. G., Shaw, R. A., and Stobie, E., “Definition of the Flexible Image Transport System (FITS), version 3.0,” A&A 524, A42 (Nov 2010).
  * [5] Kluska, J., Malbet, F., Berger, J.-P., Baron, F., Lazareff, B., Le Bouquin, J.-B., Monnier, J., Soulez, F., and Thiébaut, E., “Sparco: a semi-parametric approach for image reconstruction of chromatic objects-application to young stellar objects,” Astronomy & Astrophysics 564, A80 (2014).
  * [6] Baron, F. and Young, J. S., “Image reconstruction at cambridge university,” in [Optical and Infrared Interferometry ], 7013, 1288–1298, SPIE (2008).
  * [7] Taylor, M. B., Boch, T., and Taylor, J., “Samp, the simple application messaging protocol: Letting applications talk to each other,” Astronomy and Computing 11, 81–90 (2015).
  * [8] Haubois, X., Bernaud, P., Mella, G., Duvert, G., Benisty, M., Bério, P., Bourges, L., Chelli, A. E., Chesneau, O., Lacour, S., Lafrasse, S., Le Bouquin, J.-B., Mourard, D., Nardetto, N., and Olofsson, J., “A global database for optical interferometry,” in [Optical and Infrared Interferometry IV ], Rajagopal, J. K., Creech-Eakman, M. J., and Malbet, F., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 9146, 91460O (July 2014).
  * [9] Meimon, S. C., Mugnier, L. M., and Le Besnerais, G., “Reconstruction method for weak-phase optical interferometry,” Optics Letters 30(14), 1809–1811 (2005).
  * [10] Thiébaut, E., “MIRA: an effective imaging algorithm for optical interferometry,” in [Optical and Infrared Interferometry ], Schöller, M., Danchi, W. C., and Delplancke, F., eds., 7013, 479 – 490, International Society for Optics and Photonics, SPIE (2008).
  * [11] Joye, W. A. and Mandel, E., “New features of saoimage ds9,” in [Astronomical data analysis software and systems XII ], 295, 489 (2003).
  * [12] Tallon-Bosc, I., Tallon, M., Thiébaut, E., Béchet, C., Mella, G., Lafrasse, S., Chesneau, O., de Souza, A. D., Duvert, G., Mourard, D., et al., “Litpro: a model fitting software for optical interferometry,” in [Optical and Infrared Interferometry ], 7013, 491–499, SPIE (2008).