# What is a good question? Task-oriented asking with fact-level masking

Matthew Toles, Yukun Huang, Zhou Yu & Luis Gravano
Department of Computer Science
Columbia University
New York, NY 10027, USA
{m.toles, yh3310<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Asking questions is an important element of real-life collaboration on
reasoning tasks like question answering. For example, a legal assistant
chatbot may be unable to make accurate recommendations without specific
information on the user’s circumstances. However, large language models are
usually deployed to solve reasoning tasks directly without asking follow-up
questions to the user or third parties. We term this problem task-oriented
asking (TOA). Zero-shot chat models can perform TOA, but their training is
primarily based on next-token prediction rather than whether questions
contribute to successful collaboration. To enable the training and evaluation
of TOA models, we present a definition and framework for natural language
task-oriented asking, the problem of generating questions that result in
answers useful for a reasoning task. We also present fact-level masking (FLM),
a procedure for converting natural language datasets into self-supervised TOA
datasets by omitting particular critical facts. Finally, we generate a TOA
dataset from the HotpotQA dataset using FLM and evaluate several zero-shot
language models on it. Our experiments show that current zero-shot models
struggle to ask questions that retrieve useful information, as compared to
human annotators. These results demonstrate an opportunity to use FLM datasets
and the TOA framework to train and evaluate better TOA models.

##  1 Introduction

Asking and answering questions are important elements of collaboration when
performing reasoning tasks like question answering (QA). Large language models
(LLMs) have shown strong performance on question answering (Chang et al.,
2023), but little investigation has been done on their ability to ask useful
questions during collaboration. In the context of collaboration, a question is
useful if its answer helps with the downstream task, like QA. Many LLM
applications would benefit from the ability to ask these kinds of intermediate
questions.

Consider the use case of a corporate legal assistant chatbot. Providing proper
legal advice is often highly situational, even differing from case to case
within a single company. Important details such as an executive’s level of
risk tolerance may never appear in writing, so they must be requested directly
from the user. In many commercial products like tax software, static methods
such as forms and decision trees are used to request information (Yu et al.,
2020). More recently, chatbots have been used to leverage the convenience of
dialogue. However, many chatbots (Wang et al., 2019) only generate questions
from a hand-coded fixed list, which does not scale or generalize well.

Figure 1:  Overview of the TOA task, which simulates the need to formulate a
question. Conventionally (*B), the primary model performs the primary task
directly. However, in TOA (*B), the TOA model first uses the context to
generate an intermediate question. The question is presented to the oracle
model, which generates a response. The response is appended to the original
context, then finally sent to the primary model. For strong TOA models, we
expect the primary model to achieve better performance on context plus TOA
answer than on context alone.

The most recent generations of zero-shot chat LLMs are capable of generating
questions by predicting the maximum likelihood response. However, this
strategy fails to account for non-contextual factors, like what facts may be
known by each agent. For example ”Did your tax bill exceed $X?” may be much
easier for the user to recall than ”How much tax did you pay last year?” Thus
it is important to evaluate questions based on their effect on the user’s
goal, rather than their likelihood. To promote objective comparison of models’
ability to ask for missing information, we propose task-oriented asking (TOA)
as a benchmark for language model question generation.

Existing TOA datasets are few, limited in scope, and do not evaluate the
impact of TOA within the context of a complex natural language reasoning task.
To evaluate and fine-tune TOA models, we propose creating a large, challenging
natural language TOA dataset. Each example in a TOA dataset should contain an
instance of a primary task, such as QA. Ideally, the example context should be
missing some information critical to completing the task, naturally prompting
a question seeking that information. Manually creating such information-
incomplete tasks is time-consuming, so instead we propose a method for
creating these datasets through self-supervision. We begin with an existing QA
dataset that contains critical factual information in the example contexts.
Then we omit or ”mask” portions of the context, naturally creating QA
instances lacking sufficient background information. This process has the
additional benefit of specifying exactly what information is missing from each
example. We term this process fact-level masking (FLM).

These masked contexts provide an opportunity for language models to generate
questions that seek out information. To evaluate their success, it would be
useful to know what kinds of answers a generated question would elicit. One
approach would be to annotate a corpus of question-answer pairs, but this
strategy does not scale well, nor generalize to unseen questions. Instead, we
propose simulating the answering agent (the ”oracle”) with a language model
that outputs responses dynamically. This allows us to produce responses to a
wide range of generated questions without the need for annotation.

Given some response to a question, we would like to evaluate whether it
contains useful information originally absent from the context. Again one
might consider labeling a corpus of answers as ”useful” or ”not useful,” but
this too does not scale well. Whether information is useful depends on many
factors, such as whether the information appears elsewhere in the context or
whether the consuming agent has previously memorized the information. We
propose instead to evaluate the information based on whether the downstream
agent (e.g., QA model) consuming the information improves its performance. For
example, we can assess the change in QA accuracy of a model with versus
without the response. Similar to our process of producing responses, this
procedural approach robustly evaluates question-answer pairs. In this way, the
primary and oracle models form a pipeline that, together with the FLM dataset,
can be used to objectively evaluate the usefulness of an information-seeking
question.

In summary, the primary contributions of this paper are threefold:

  * •

A definition and framework for evaluating natural language task-oriented
asking (TOA)

  * •

Fact-Level Masking (FLM): A procedure for converting natural language datasets
into self-supervised TOA

  * •

FLM-HotpotQA: A TOA dataset generated with FLM and associated evaluation
pipeline for FLM-HotpotQA with baseline results

We find that although zero-shot LLMs can generate plausible-sounding
questions, these questions extract less useful information compared to human-
generated questions. This finding indicates that current state-of-the-art LLMs
are still limited in their capacity to pose insightful questions, unveiling a
need for TOA training and evaluation methods and a promising avenue for future
research.

##  2 Related Work

###  2.1 General Question Generation

Question Generation (QG), speaking generally, is the task of automatically
generating questions (Rus et al., 2008). Questions can be generated using
syntactic (Gates, 2008; Yao et al., 2012) or neural (Chen et al., 2018)
approaches. Duan et al. (2017) and Wang et al. (2020) generate questions for
data augmentation for QA tasks and pretraining, respectively, using
convolutional, recurrent, and transformer architectures. Chatbots designed for
social dialogue may ask questions to exhibit emotional intelligence, prompt
users, and drive engagement (Shum et al., 2018). Non-task-oriented question-
asking can also be used for educational purposes (Kurdi et al., 2020). During
general QG, outputs are often evaluated based on the Bleu or Rouge score of
the question, as in (Chen et al., 2018), while questions in social dialog are
evaluated by the same metrics as other dialog turns (e.g., user engagement (Yu
et al., 2016)). Our approach, in contrast, evaluates a question based on the
usefulness of the answer it prompts (Figure 1).

###  2.2 Task-Oriented Question Generation

TOA is a subset of question generation where a question is evaluated based on
whether its answer is useful to a downstream primary task. Clarifying
questions are a type of task-oriented question for disambiguating user intent,
as in (Aliannejadi et al., 2019). ClariQ (Aliannejadi et al., 2020) is a
supervised dataset of queries and clarifying questions for them. The
GuessWhat?! (De Vries et al., 2017) and CLEVR Ask (Matsumori et al., 2021)
datasets are examples of constrained iterative binary TOA tasks in the vision
domain. These datasets contain images paired with conversations by annotators
playing a guessing game. In GuessWhat?! one annotator asks questions trying to
narrow down which object in the image the other has chosen, and the other
answers yes or no. In CLEVR Ask, images and answers are generated
synthetically. Existing TOA datasets are sparse and limited in scope to
clarifying or binary questions and to the vision domain while we introduce the
first natural language TOA dataset in the QA domain.

###  2.3 Related Tasks

In task-oriented dialog (TOD), the system is designed to converse with the
user to perform a slot-filling task. Slot-filling tasks are typically
straightforward and well-defined, like booking a hotel. What information is
missing, such as the desired price range, is usually easily defined by which
slots are empty (Budzianowski et al., 2018). In such cases, template-based
systems can be sufficient for question asking in TOD, with the main challenge
being natural language understanding and dialog state tracking. Since the set
of useful questions is neither complex nor numerous, TOD systems often assume
that the user will be able to answer all system-generated questions. By
decoupling TOD from a fixed slot ontology and accounting for incomplete user
knowledge, TOA can be viewed as a generalization of the dialog planning and
natural language generation steps of TOD.

Finally, TOA is similar to the idea of tool-use, where models can consult APIs
like a calculator, search engine, or QA model to improve performance on a
downstream task. Tool-use models like Toolformer (Schick et al., 2023) call
APIs internally during generation to gather additional knowledge. Tool-use
using a QA model differs from task-oriented asking in that TOA allows for
independently training and inference of the question generation model.

##  3 Methods

###  3.1 Problem Description

The goal of task-oriented asking is for the TOA model to transfer information
from a knowledgeable agent (the oracle) to an executive agent (the primary
model) by asking a question. The primary model is a model that directly
executes some task for the user, such as a legal assistant chatbot or QA
model. The oracle is an agent capable of answering intermediate questions
related to the primary task. This could be a human user, expert, or LLM stand-
in like Flan-T5 (Chung et al., 2022). The TOA model is a computer agent,
capable of generating questions that assist the primary model in its task,
such as Alpaca (Taori et al., 2023). It takes the primary task as input and
generates a question for the oracle. The oracle response is concatenated to
the original context and then passed to the primary model, giving the primary
model access to the information requested in the question. The TOA model’s
performance is evaluated using the difference between the primary model’s
performance with and without the oracle’s answer.

Many factors affect the extent of TOA performance gains elicited by oracle
responses, including the context, the bias of the models, the possible oracle
responses, and what information has been memorized by the primary model.
Hence, TOA performance can only be assessed in the context of a particular
pipeline. Our pipeline, as described above, consists of a primary model,
$M_{1}$, tasked with performing some task, and an oracle, $\Theta$, which
responds to questions generated by the secondary TOA model, $M_{2}$. In the
next section, we present a specific $M_{2}\rightarrow\Theta\rightarrow M_{1}$
pipeline and dataset on which to evaluate it.

###  3.2 Problem Definition

Let $t$ be a natural language statement of a task. Let $f_{1},...,f_{n}$ be
natural language facts consisting the context for the task. Let example
$x=\\{t,f_{1},...,f_{n}\\}$. Let $M_{1}(x)\rightarrow y$ be a primary model
that takes $x$ as input and outputs $y$. Let $M_{2}(x)\rightarrow q$ be a TOA
model that takes $x$ as input and generates a natural language question $q$.
Let $R(M_{1},x,y)\rightarrow r$ be some reward on which $M_{1}$ is evaluated,
where more positive values are better, such as F-score, accuracy, or negative
loss. For brevity, we often omit $M_{1}$ and $y$.

We say a fact $f$ is supporting if it is believed that
$R(x\cup\\{f\\})>R(x\backslash f)$. Otherwise we say $f$ is distracting (Yang
et al., 2018). Let $\Theta(q)\rightarrow f_{r}$ be an oracle model that takes
$q$ as input and returns a response $f_{r}$. The TOA task is to create a model
$M_{2}$ that maximizes

$\displaystyle\Delta r$ $\displaystyle=R(x\cup\\{f_{r}\\})-R(x)$
$\displaystyle=R(x\cup\\{\Theta(M_{2}(x)))\\}-R(x)$

One may construct more complex versions of TOA involving multiple missing
facts, iterative asking, multiple oracles, or cost functions for different
types of questions. In this paper, we limit TOA to the costless, single-mask,
single-turn, single-oracle case. Similarly, we do not address the separate
task of determining whether a task lacks sufficient context.

##  4 Experiments

###  4.1 Dataset

We contribute FLM-HotpotQA, a version of the QA dataset HotpotQA for
evaluating task-oriented asking (Yang et al., 2018). HotpotQA is a multi-hop
QA reasoning task where each example contains both supporting and distractor
facts from Wikipedia as determined by human annotators. We choose reward
function $R$ to be the F1 score of the word overlap between the predicted
answer and the ground truth answer following the original HotpotQA. Thus
$r\in[0,1]$ and $\Delta r\in[-1,1]$.

Incomplete Example $x^{i}$Complete Example $x^{c}$Candidate Oracle Responses t |  When was the composer of ”Persian Surgery Dervishes” born?
---|---
$f^{sup}_{1}$ |  Persian Surgery Dervishes is a recording of two live solo electric organ concerts, the first held in Los Angeles on 18 April 1971 and the second in Paris on 24 May 1972, by avant-garde minimalist composer Terry Riley.
$f^{sup}_{2}$ ($f^{*}$) |  Terrence Mitchell ”Terry” Riley (born June 24, 1935) is an American composer and performing musician associated with the minimalist school of Western classical music.
$f^{dis}_{1}$ |  Thomas Christian David (December 22, 1925 - January 19, 2006) was an Austrian composer, conductor, choral conductor, and flutist.
$f^{dis}_{2}$ |  Abdolreza Razmjoo is a composer, arranger and singer Tenor of Iran Kurdish ancestry from Kermansha.
Figure 2: An example containing a primary task t, supporting facts
$f^{sup}_{1,...,n}$, and distractor facts $f^{dis}_{1...n}$. (Additional facts
not shown.) We create an incomplete example $x^{i}$ by masking one supporting
fact, $f^{*}$, chosen at random, from the facts in the complete example
$x^{c}$. Prompted with $x^{i}$, the TOA model poses a question to the oracle
which returns one oracle response $f_{r}$ from the supporting or distractor
facts.

To simplify the primary task, we create a complete example $x^{c}$ that
contains the task and every supporting fact (Figure 2). Next, we apply fact-
level masking to each example: From each complete example, we create an
incomplete example $x^{i}$ by randomly selecting one supporting fact,
$f^{*}\in x^{c}$, to be the masked fact and deleting it from the context:
$x^{i}=x^{c}\backslash f^{*}$. The masked fact, along with the distractor
facts and the other supporting facts, make up the set of possible responses,
$f_{r}$, the oracle may give. We create the response example $x^{r}$ by
appending $f_{r}$ to $x^{i}$, i.e. $x^{r}=x^{i}\cup\\{f_{r}\\}$. When missing
one supporting fact, the primary task becomes substantially more difficult,
even for strong zero-shot models like GPT-4 OpenAI (2023) (Figure 7). To
benchmark human performance, one author of this paper annotated a Test subset
of 400 task-oriented questions.

In general, we expect the complete example $x^{c}$, which contains every
supporting fact, to have the highest possible reward. Meanwhile, we say an
example $x$ is improvable if there exists at least one possible response
$f_{r}$ such that $\Delta r(f_{r})>0$. By masking facts in $x^{c}$ we can
decrease the reward on the example, producing an improvable self-supervised
example. Note that not all incomplete examples will be improvable, for
example, if:

  * •

Two facts contain redundant information

  * •

$M_{1}$ has memorized knowledge of information in $f^{*}$

  * •

$f^{*}$ is mislabeled as supporting

  * •

$x^{i}$ still allows $M_{1}$ to make a spurious correlation without $f^{*}$

It is also possible for $x^{i}$ to be improved by a response $f_{r}$ even if
$f_{r}\neq f^{*}$, if $f_{r}$ and $f^{*}$ contain similar information. 27.6%
and 28.5% of examples in our Full and Test datasets, respectively, are
improvable. We preserve unimprovable examples in the dataset to avoid bias;
the primary model may sometimes achieve the correct response through a
spurious correlation on $x^{i}$, but fail to make the spurious correlation on
$x^{r}$. Similarly, the primary model may fail on $x^{i}\cup\\{f^{*}\\}$ but
succeed on $x^{i}\cup\\{f_{r}\\}$ if $f_{r}$ contains more helpful information
than $f^{*}$.

###  4.2 Implementation Details

To generate and evaluate answers to TOA questions, we construct the following
pipeline (Figure 3). The TOA model $M_{2}$ takes an incomplete example $x^{i}$
as input to generate a task-oriented question $q$. As baselines for $M_{2}$ we
choose GPT-4 (OpenAI, 2023), ChatGPT (OpenAI, 2022), and Alpaca (Taori et al.,
2023). We select these models for their strong performance on zero-shot and
few-shot tasks. We choose a prompt template for each model by evaluating three
zero-shot and three 5-shot in-context prompts on 400 examples from the
training dataset (Appendix A.1). We also include a dummy Repeater model among
the baselines, which simply returns its input.

SupportingExample$x^{c}=$$\\{t\\}\cup
F^{sup}$FLMptMaskedExample$x^{c}=$$\\{t\\}\cup F^{sup}\backslash
f^{*}$ptTOAModel$M_{2}$ptQuestion$q$ptOracle$\Theta$ptResponse$f_{r}\in$$F^{sup}\cup
F^{dis}$ptconcatptAnsweredExample$x^{r}=$$\\{t\\}\cup F^{sup}\backslash
f^{*}\cup\\{f_{r}\\}$ptPrimary Model$M_{1}$ Figure 3: Our TOA evaluation
pipeline. From left to right, a complete example is masked via fact-level
masking to create an incomplete example. The TOA model generates a question
based on the incomplete example. We provide the question to the oracle, which
generates a response. The response is concatenated onto the incomplete example
to produce the response example. Finally, all three examples are evaluated py
the primary model.

Questions generated by $M_{2}$ are passed to the oracle $\Theta$, also a
Flan-T5-Base model, which we choose for its accessibility and strong zero-shot
performance on other QA tasks. The oracle model serves as a stand-in for a
human expert answering task-oriented questions generated by $M_{2}$. $\Theta$
returns $f_{r}$, the most likely response to $q$ from among all possible
distractor facts $F^{dis}$ present in the original HotpotQA example
($\mu=39.2$, $\sigma=11.4$), all supporting $F^{sup}$ facts ($n-1$ of which
are already present in the context, $\mu=1.43,\sigma=0.71$), and the masked
fact $f^{*}$.

To create the response example $x^{r}$, we append the oracle response to the
incomplete example. Note that by appending rather than inserting, the order of
facts may be altered as compared to $x^{c}$, even if $f_{r}=f^{*}$, which may
occasionally affect the output of the primary model.

Finally, complete examples containing $f^{*}$, response examples containing
$f_{r}$, and incomplete examples lacking any response are passed to the
primary model. If $M_{2}$ produces a question with positive $M_{1}$
improvement, then one should expect $R(x^{c})\geq R(x^{r})>R(x^{i})$. To
express reward relative to its theoretical minimum ($R(x^{i})$) and maximum
($R(x^{c})$) values, we define recovery as:

$\displaystyle\rho=100\cdot\frac{R(x^{r})-R(x^{i})}{R(x^{c})-R(x^{i})}$

and select F1 recovery as our primary evaluation metric.

##  5 Results and Discussion

###  5.1 Baseline Performance

We report F1 and exact match recovery results for the baseline TOA models on
the full HotpotQA validation set ($n=7404$, Figure 5). Of the four baseline
models, GPT-4 performs best in both F1 and exact match (EM), recovering 65.6%
and 66.9% respectively. These results, however, fall well short of complete
recovery of missing information, indicating room for improvement even in
strong zero-shot models. Other models perform substantially worse. Alpaca
achieved 39.8% F1 and 38.5% EM recovery. GPT-3.5 Turbo (OpenAI, 2022) achieved
only 30.4% F1 recovery, which is only a moderate improvement over the dummy
Repeater model. We suspect Repeater achieves its recovery (22.5%) by
exploiting a bias in the oracle towards choosing responses with high keyword
overlap with the input question.

Figure 4: F1 and exact match recovery of baseline models and human annotators.
Results shown for the Full validation set ($n=7404$) and the Test subset
($n=400$), which contains human-generated TOA questions.

(a) Alpaca

(b) GPT-3.5 Turbo

(c) GPT-4

(d) Human Annotation

Figure 5: Proportion of questions (Q) answered with a masked fact (MS) vs.
distractor (D) by oracle (left section). Proportion of answers given resulting
in positive, zero, or negative difference in primary model performance (right
section).

###  5.2 Comparison to Human Performance

Table 1: F1 and exact match recovery for full validation set and manually annotated test subset. | Full |  | Test
---|---|---|---
Model | F1 | F1 Recovery | EM | EM Recovery |  | F1 | F1 Recovery | EM | EM Recovery
Alpaca | 61.3 | 39.8 | 45.3 | 38.5 |  | 60.4 | 39.7 | 46.8 | 45.6
GPT-3.5 Turbo | 59.7 | 30.4 | 44.4 | 31.9 |  | 59.3 | 34.0 | 45.5 | 38.2
GPT-4 | 65.6 | 64.6 | 49.4 | 66.9 |  | 65.3 | 65.6 | 50.2 | 66.2
Repeater | 58.3 | 22.5 | 43.1 | 22.8 |  | 58.5 | 29.6 | 45.8 | 39.7
Human | - | - | - | - |  | 68.8 | 84.4 | 54.3 | 89.7
Masked | 54.5 | 0.0 | 39.9 | 0.0 |  | 53.0 | 0.0 | 39.0 | 0.0
Supporting | 71.7 | 100.0 | 54.1 | 100.0 |  | 71.7 | 100.0 | 56.0 | 100.0

We find that human-generated questions on the Test subset are more likely to
elicit the masked fact $f^{*}$ in the response. This leads to human
annotations performing significantly better than the best baseline models
(Figure 5). Human annotation achieved 84.4% F1 and 89.7% EM recovery, compared
to the strongest baseline, GPT-4, which achieved 65.6% F1 and 50.2% EM
recovery on the test set. This indicates room for improvement in baseline
models performing task-oriented asking. Finally, distractor fact responses to
human-generated questions were less likely than those by baselines to induce
hallucinations in the primary model (6.3% vs. 11.0% of distractors for human
generation and GPT-4, respectively).

###  5.3 Primary Model Ablation

We evaluate all available sizes of Flan-T5, GPT-3.5 Turbo, and GPT-4 as
candidate primary models using a Flan-T5-Base model as the oracle and human-
generated questions as the TOA model. Models lose between 11.7% (GPT-4) and
22.0% (Flan-T5-Large) absolute points F1 score as a result of masking a single
supporting fact (Figure 7). Models recover between 65.6% (GPT-3.5 Turbo) and
84.4% (Flan-T5-Base) of the F1 score lost during masking. Although models are
affected differently by FLM, with GPT-X models being more robust, consistency
in F1 recovery rate suggests that the choice of primary model has minimal
impact on TOA evaluation. We suspect GPT-X models are more robust than Flan-T5
since in exploration they appear to have memorized large portions of
Wikipedia, which minimizes the impact of removing Wikipedia facts from
context.

Figure 6: Supporting, answered, and masked F1 as a function of primary model
architecture.

Figure 7: F1, exact match and masked fact response rate (MFRR) as a function
of oracle size and architecture.

Table 2: Illustrative example behavior of TOA models on FLM-HotpotQA. Select
models omitted for brevity. Emphasis added.

(a) Success mode where $f^{*}\neq f_{r}$. Despite receiving a distractor
response, GPT-4 recovers the missing information contained in the masked fact.
Alpaca fails due to requesting irrelevant information.

Task |  Which film was released first: Sacred Planet or Oz the Great and Powerful?
---|---
Context |  Oz the Great and Powerful: Oz the Great and Powerful is a 2013 American fantasy adventure film directed by Sam Raimi and produced by Joe Roth, from a screenplay written by David Lindsay-Abaire and Mitchell Kapner.
Masked Fact |  Sacred Planet: Sacred Planet is a 2004 documentary directed by Jon Long and Hairul Salleh Askor.
Answer |  Sacred Planet

|  Question |  Response |  $M_{1}$ Output |
---|---|---|---|---
Alpaca |  When was Oz the Great and Powerful released? |  Snegithiye: The film, released in 2000, proved to be an average grosser at the box office but bagged positive reviews from critics. |  Oz the Great and Powerful |
GPT-4 |  When was the film Sacred Planet released? |  Sacred Planet: The film was released by Walt Disney Pictures on April 22, 2004, and grossed $1,108,356. |  Sacred Planet |  ✓

(b) Failure mode where GPT-4 requests information that is both irrelevant and
already contained in the context. The oracle responds to the question
accurately, but the new information is not useful, and the primary model
predicts ”Knoxville, Tennessee,” which is incorrect.

Task |  The Tennessee Volunteers football team plays as a member for a conference in what city?
---|---
Context |  1984 Tennessee Volunteers football team: Playing as a member of the Southeastern Conference (SEC), the team was led by head coach Johnny Majors, in his eighth year, and played their home games at Neyland Stadium in Knoxville, Tennessee.
Masked Fact |  Southeastern Conference: The conference is headquartered in Birmingham, Alabama.
Answer |  Birmingham, Alabama

|  Question |  Response |  $M_{1}$ Output |
---|---|---|---|---
GPT-4 |  In what city is Neyland Stadium located? |  1983 Tennessee Volunteers football team: Playing as a member of the Southeastern Conference (SEC), the team was led by head coach Johnny Majors, in his seventh year, and played their home games at Neyland Stadium in Knoxville, Tennessee. |  Knoxville, Tennessee |
Human |  What city is the Southeastern Conference headquartered in? |  Southeastern Conference: The conference is headquartered in Birmingham, Alabama. |  Birmingham, Alabama |  ✓

(c) Failure mode where GPT-4 only requests part of the missing information.
Both the name and the birth year of the author are missing from the context,
which the human annotator successfully requests in a single question. Although
GPT-4’s question is well-formed, it refers to the manga in question, ’I”s,’
overly verbosely as ”the manga series featuring 16-year-old high school
student Ichitaka Seto,” likely causing the oracle to respond with an
irrelevant fact. Lacking any years in the context, the primary model
hallucinates 1939.

Task |  A Japanese manga series based on a 16-year-old high school student Ichitaka Seto, is written and illustrated by someone born in what year?
---|---
Context |  I”s: The story’s main character is 16-year-old high school student Ichitaka Seto who is in love with his classmate Iori Yoshizuki, but too shy to tell her.
Masked Fact |  Masakazu Katsura: Masakazu Katsura (桂 正和 , Katsura Masakazu , born December 10, 1962) is a Japanese manga artist, known for several works of manga, including ”Wing-man”, ”Shadow Lady”, ”DNA²”, ”Video Girl Ai”, ”I”s”, and ”Zetman”.
Answer |  1962

|  Question |  Response |  $M_{1}$ Output |
---|---|---|---|---
GPT-4 |  Who is the author and illustrator of the Japanese manga series featuring 16-year-old high school student Ichitaka Seto? |  Silver Spoon (manga): Silver Spoon (Japanese: 銀の匙 , Hepburn: Gin no Saji ) is a Japanese manga series written and illustrated by Hiromu Arakawa, set in the fictional Ooezo Agricultural High School in Hokkaido. |  1939 |
Human |  When was the author of I”s born? |  Masakazu Katsura: Masakazu Katsura (桂 正和 , Katsura Masakazu , born December 10, 1962) is a Japanese manga artist, known for several works of manga, including ”Wing-man”, ”Shadow Lady”, ”DNA²”, ”Video Girl Ai”, ”I”s”, and ”Zetman”. |  1962 |  ✓

###  5.4 Oracle Ablation

We test GPT-3.5 Turbo, GPT-4, and all sizes of Flan-T5 as the Oracle on human-
generated questions. Flan-T5-Base and larger respond with the incomplete
example in more than 68% of cases (Figure 7). Furthermore, we observe
consistently strong performance by these models on F1 and exact match, with
both metrics exceeding 84% recovery in all cases. This indicates that when
prompted by well-formed and informative questions, Flan-T5 of size Base and
larger can consistently respond with appropriate answers. For the sake of
accessibility, we choose the smallest strong model, Flan-T5-Base, as our
oracle. Interestingly, although GPT-4 responds with the masked fact far less
frequently than any Flan-T5 model (GPT-4: 39.0%, Flan-T5-XXL: 74.0%), GPT-4
achieves the highest F1 recovery overall and 100% exact match recovery. This
suggests that although GPT-4 gives distractor or redundant supporting facts
most of the time, the facts it chooses still carry critical information. This
illustrates the importance of measuring information gain rather than nominal
correctness.

###  5.5 Failure Modes

We observe one failure mode associated with the oracle and three associated
with the TOA model, which prevent TOA questions from recovering missing
information. Most obviously, the oracle may return an irrelevant and unhelpful
response. In 31.5% of cases, human-generated questions induce responses other
than the masked fact. When $f^{*}\neq f_{r}$, the F1 score of the primary
model increases in only 11.1% of cases, compared to 32.5% of cases when
$f^{*}=f_{r}$ (Figure 5). When a distractor fact does cause an increase in F1,
it is often because the distractor fact contains information similar to that
in the masked fact (see Table 2(a)).

Other times, the failure mode is due to the TOA model generating poor
questions. For example, in Table 2(b), GPT-4 asks a question requesting
irrelevant information. The oracle produces an accurate response to the
question, but since the information is not useful, the primary model remains
incorrect. In other cases, such as in Table 2(c), GPT-4 only asks for part of
the necessary missing information, requesting the name of the author but not
his birth year.

##  6 Conclusion, Limitations and Future Work

In this paper, we presented a novel framework for evaluating task-oriented
questions and observed that state-of-the-art zero-shot LLMs struggle at this
task compared to humans. We attribute this deficiency to a lack of training
data and evaluation processes for TOA. To overcome these challenges, we
introduced fact-level masking and FLM-HotpotQA, a self-supervised TOA dataset,
and an associated evaluation pipeline. To conclude, our contributions
highlight the challenges faced by state-of-the-art, zero-shot LLMs, address
limitations in training data and evaluation methods for task-oriented asking,
and pave the way for future TOA development.

Exploring more challenging task-oriented asking present in normal human
interactions is a promising future research direction. Broadly speaking, scope
can increase along two axes: the task and the pipeline. One could apply FLM to
other tasks requiring longer contexts, more specific domain knowledge, or more
hops in logical reasoning. Likewise, increasingly complex tasks such as code
generation or chatbot assistants could benefit from TOA. Meanwhile, one may
employ an unconstrained generative oracle instead, or employ multiple oracles,
each lacking complete information.

Several other possible variations of TOA also have interesting opportunities.
Although our work primarily manipulated contextual information, later we will
investigate the role of information memorized by the primary agent. This
work’s modality was also restricted to natural language, but future work may
involve tabular, graph, or multi-modal data. In addition, future directions
may require iterative task-oriented asking with humans or costly APIs while
optimizing each ask. Finally, it remains to be seen how well language models
can perform on TOA when specifically trained for the task using self-
supervised FLM datasets or reinforcement learning.

#### Acknowledgments

This material is based upon work supported by the National Science Foundation
Graduate Research Fellowship under Grant No. (DGE-2036197).

## References

  * Aliannejadi et al. (2019) Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W Bruce Croft.  Asking clarifying questions in open-domain information-seeking conversations.  In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pp. 475–484, 2019.
  * Aliannejadi et al. (2020) Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeff Dalton, and Mikhail Burtsev.  ConvAI3: Generating clarifying questions for open-domain dialogue systems (ClariQ).  _arXiv preprint arXiv:2009.11352_ , 2020.
  * Budzianowski et al. (2018) Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić.  Multiwoz–a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling.  _arXiv preprint arXiv:1810.00278_ , 2018.
  * Chang et al. (2023) Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al.  A survey on evaluation of large language models.  _arXiv preprint arXiv:2307.03109_ , 2023.
  * Chen et al. (2018) Guanliang Chen, Jie Yang, Claudia Hauff, and Geert-Jan Houben.  LearningQ: a large-scale dataset for educational question generation.  In _Proceedings of the International AAAI Conference on Web and Social Media_ , volume 12, 2018.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_ , 2022.
  * De Vries et al. (2017) Harm De Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville.  Guesswhat?! Visual object discovery through multi-modal dialogue.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 5503–5512, 2017.
  * Duan et al. (2017) Nan Duan, Duyu Tang, Peng Chen, and Ming Zhou.  Question generation for question answering.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pp. 866–874, 2017.
  * Gates (2008) Donna Gates.  Generating look-back strategy questions from expository texts.  In _The Workshop on the Question Generation Shared Task and Evaluation Challenge, NSF, Arlington, VA. http://www. cs. memphis. edu/~ vrus/questiongeneration//1-Gates-QG08. pdf_ , 2008.
  * Kurdi et al. (2020) Ghader Kurdi, Jared Leo, Bijan Parsia, Uli Sattler, and Salam Al-Emari.  A systematic review of automatic question generation for educational purposes.  _International Journal of Artificial Intelligence in Education_ , 30:121–204, 2020.
  * Matsumori et al. (2021) Shoya Matsumori, Kosuke Shingyouchi, Yuki Abe, Yosuke Fukuchi, Komei Sugiura, and Michita Imai.  Unified questioner transformer for descriptive question generation in goal-oriented visual dialogue.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 1898–1907, 2021.
  * OpenAI (2022) OpenAI.  ChatGPT: Optimizing language models for dialogue, 2022.  URL https://openai.com/blog/chatgpt/.
  * OpenAI (2023) OpenAI.  GPT-4 technical report, 2023.
  * Rus et al. (2008) Vasile Rus, Zhiqiang Cai, and Art Graesser.  Question generation: Example of a multi-year evaluation campaign.  _Proceedings in the Workshop on the Question Generation Shared Task and Evaluation Challenge_ , 2008.
  * Schick et al. (2023) Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.  Toolformer: Language models can teach themselves to use tools.  _arXiv preprint arXiv:2302.04761_ , 2023.
  * Shum et al. (2018) Heung-Yeung Shum, Xiao-dong He, and Di Li.  From Eliza to XiaoIce: challenges and opportunities with social chatbots.  _Frontiers of Information Technology & Electronic Engineering_, 19:10–26, 2018.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford Alpaca: An instruction-following LLaMA model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * Wang et al. (2019) Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu.  Persuasion for good: Towards a personalized persuasive dialogue system for social good.  _arXiv preprint arXiv:1906.06725_ , 2019.
  * Wang et al. (2020) Yanmeng Wang, Wenge Rong, Jianfei Zhang, Shijie Zhou, and Zhang Xiong.  Multi-turn dialogue-oriented pretrained question generation model.  _Complex & Intelligent Systems_, 6:493–505, 2020.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning.  HotpotQA: A dataset for diverse, explainable multi-hop question answering.  _arXiv preprint arXiv:1809.09600_ , 2018.
  * Yao et al. (2012) Xuchen Yao, Gosse Bouma, and Yi Zhang.  Semantics-based question generation and implementation.  _Dialogue & Discourse_, 3(2):11–42, 2012.
  * Yu et al. (2020) Jay Yu, Kevin McCluskey, and Saikat Mukherjee.  Tax knowledge graph for a smarter and more personalized TurboTax.  _arXiv preprint arXiv:2009.06103_ , 2020.
  * Yu et al. (2016) Zhou Yu, Ziyu Xu, Alan W Black, and Alexander Rudnicky.  Strategy and policy learning for non-task-oriented conversational systems.  In _Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pp. 404–412, 2016.

##  Appendix A Appendix

###  A.1 List of Prompts

  1. 1.

Ask another question that would help you answer the following question:
{context} {q1}

  2. 2.

Some information is missing from this context. Ask a simpler question that
would help you answer it. Context: {context} Main Question: {q1} Simpler
question:

  3. 3.

What question can you ask to help you answer the final question? {context}
{q1} You can ask:

  4. 4.

Ask another question that would help you answer the following question: {in-
context examples} {context} {q1}

  5. 5.

Some information is missing from this context. Ask a simpler question that
would help you answer it. {in-context examples} Context: {context} Main
Question: {q1} Simpler question:

  6. 6.

What question can you ask to help you answer the final question? {in-context
examples} {context} {q1} You can ask:

Based on performance on $n=400$ examples from the HotpotQA train dataset we
select prompts 3, 3, and 2 for Alpaca, GPT-3.5 Turbo, and GPT-4, respectively.

###  A.2 Oracle Architecture Ablation

Table 3: Oracle architecture ablation for oracle models using Flan-T5-Base as primary model on the full validation set. | F1 | F1 Recovery | EM | EM Recovery | MFRR
---|---|---|---|---|---
Flan-T5-Small | 64.9 | 63.8 | 50.5 | 67.6 | 50.7
Flan-T5-Base | 68.8 | 84.4 | 54.3 | 89.7 | 68.5
Flan-T5-Large | 69.2 | 86.5 | 55.0 | 94.1 | 71.3
Flan-T5-XL | 69.8 | 90.1 | 55.5 | 97.1 | 74.3
Flan-T5-XXL | 70.4 | 92.9 | 56.0 | 100.0 | 74.0
GPT-3.5 Turbo | 66.2 | 70.8 | 50.7 | 69.1 | 31.5
GPT-4 | 70.9 | 95.9 | 56.0 | 100.0 | 39.0
Masked | 53.0 | 0.0 | 39.0 | 0.0 | -
Supporting | 71.7 | 100.0 | 56.0 | 100.0 | -

###  A.3 Primary Model Ablation

Table 4: Primary model architecture ablation using Flan-T5 base as oracle on the Full validation set. | F1 |  | EM
---|---|---|---
| Masked | Answered | Supporting | Recovery |  | Masked | Answered | Supporting | Recovery
Flan-T5-Small | 41.4 | 51.1 | 53.6 | 79.3 |  | 28.5 | 35.3 | 37.8 | 73.0
Flan-T5-Base | 53.0 | 68.8 | 71.7 | 84.4 |  | 39.0 | 54.3 | 56.0 | 89.7
Flan-T5-Large | 59.8 | 76.1 | 81.8 | 74.2 |  | 42.5 | 58.0 | 63.5 | 73.8
Flan-T5-XL | 62.3 | 78.9 | 82.9 | 80.5 |  | 45.8 | 60.8 | 64.8 | 78.9
Flan-T5-XXL | 65.2 | 78.9 | 82.2 | 80.6 |  | 50.5 | 62.5 | 65.8 | 78.7
GPT-3.5 Turbo | 61.4 | 69.3 | 73.5 | 65.6 |  | 32.5 | 36.8 | 41.3 | 48.6
GPT-4 | 69.0 | 77.2 | 80.7 | 70.2 |  | 43.0 | 47.3 | 51.2 | 51.5

###  A.4 Response Flow

Table 5: Flow of responses and outcomes for baseline models on the human-annotated Test subset (%). Masked and distractor response indicate how frequently questions elicit masked and distractor facts from the oracle, respectively. +, =, and - indicate whether the masked or distractor fact increased, had no effect, or decreased primary model performance. Distractor hallucination rate indicates how often distractor responses cause a decrease in performance, as a percentage of all distractor responses. | Alpaca | GPT-3.5 Turbo | GPT-4 | Repeater | Human
---|---|---|---|---|---
Masked Response | 28.5 | 33.5 | 54.5 | 32.8 | 68.5
Distractor Response | 71.5 | 66.5 | 45.5 | 67.3 | 31.5
Masked + | 12.3 | 12.8 | 19.0 | 15.0 | 22.3
Masked = | 13.3 | 16.3 | 30.5 | 14.0 | 40.8
Masked - | 3.0 | 4.5 | 5.0 | 3.8 | 5.5
Distractor + | 4.8 | 5.3 | 4.5 | 5.3 | 3.5
Distractor = | 62.0 | 54.5 | 36.0 | 51.7 | 26.0
Distractor - | 4.8 | 6.8 | 5.0 | 10.3 | 2.0
Distractor Hallucination Rate | 6.6 | 10.2 | 11.0 | 15.2 | 6.3


11institutetext: University of Palermo, DMI Department, via Archirafi 34,
90123 Palermo, Italy
11email<EMAIL_ADDRESS>
22institutetext: University of Messina, MIFT Department, V.le F. S.
D’Alcontres 31, 98166 Messina, Italy
22email<EMAIL_ADDRESS>
33institutetext: University of Messina, DICAM Department, Viale G. Palatuci
13, 98168 Messina, Italy
33email<EMAIL_ADDRESS>
# Multilayer Network Analysis: The Identification of Key Actors in a Sicilian
Mafia Operation

Annamaria Ficara 1122 0000-0001-9517-4131 Giacomo Fiumara 22
0000-0003-1528-7203 Pasquale De Meo 33 0000-0001-7421-216X Salvatore Catanese
22 0000-0002-0369-8235

###### Abstract

Recently, Social Network Analysis studies have led to an improvement and to a
generalization of existing tools to networks with multiple subsystems and
layers of connectivity. These kind of networks are usually called multilayer
networks. Multilayer networks in which each layer shares at least one node
with some other layer in the network are called multiplex networks. Being a
multiplex network does not require all nodes to exist on every layer. In this
paper, we built a criminal multiplex network which concerns an anti-mafia
operation called “Montagna” and it is based on the examination of a pre-trial
detention order issued on March 14, 2007 by the judge for preliminary
investigations of the Court of Messina (Sicily). “Montagna” focus on two Mafia
families called “Mistretta” and “Batanesi” who infiltrated several economic
activities including the public works in the north-eastern part of Sicily,
through a cartel of entrepreneurs close to the Sicilian Mafia. Originally we
derived two single-layer networks, the former capturing meetings between
suspected individuals and the latter recording phone calls. But some networked
systems can be better modeled by multilayer structures where the individual
nodes develop relationships in multiple layers. For this reason we built a
two-layer network from the single-layer ones. These two layers share 47 nodes.
We followed three different approaches to measure the importance of nodes in
multilayer networks using degree as descriptor. Our analysis can aid in the
identification of key players in criminal networks.

###### Keywords:

Social network analysis Criminal networks Multilayer networks

##  1 Introduction

During his lifetime, each individual continuously deal with multiple social
networks. He does it with no effort and this does not mean that it is a
trivial activity which can be overlooked. The connections between people
through multiple types of relational ties represent only one possible view of
a problem already known long before the field of social network analysis (SNA)
was developed. Looking only at a single type of relational tie within a single
social network risks either defining a world where different kinds of
relationships are ontologically equivalent or overlooking the invisible
relationships emerging from the interactions among different types of ties.
For a long time, these interactions have largely been studied within a single-
layer perspective and one of the most effective SNA tools to measure social
interactions has been the simple graph. A simple graph is defined as a set of
nodes, also called actors (i.e. individuals or organizations) with edges
between them, also called links or connections (i.e. relational ties such as
friendship relationships) and with no edges connecting a node to itself [1].
According to Wasserman and Faust [2], social networks contain at least three
different dimensions: a structural dimension corresponding to the social graph
(e.g. actors and their relationships); a compositional dimension describing
the actors (e.g. their personal information); and an affiliation dimension
(e.g. members of the same family or organization). These three dimensions
provide a minimal description needed to understand the full complexity of
social structures. An alternative conceptual approach to dealing with the same
set of problems is to think of multiple relationships as a set of connected
levels, or layers, forming a single multidimensional social network [1]. In
fact, a social network with nodes and/or edges can be organized into multiple
layers, where each layer represents a different kind of node or edge, a
different social context, a different community, a different online social
network (OSN), and so on. The analysis of multiple layers can provide
knowledge that is not present in each layer when layers are considered
independently of each other. Kivelä et al. [3] review and discuss many of the
relevant works on the topic. Then they try to unify the literature by
introducing a general framework for multilayer networks [4, 6, 5, 7]. Such
framework can represent the different notions of networks (e.g. single-layer
or monoplex networks [4, 8], multiplex networks [9, 10, 11], interdependent
networks, networks of networks) by simply introducing cumulative constraints
on the general model [12]. Multilayer social networks appear in a number of
different contexts, where data are characterized by different sizes, different
natures (e.g. online, offline, hybrid), and different layer semantics (e.g.
contact, communication, time, context, etc.). Many multirelational networks,
that is actors connected by multiple types of ties, have been collected during
SNA studies. These networks are often characterized by a small size, because
they were often collected through offline questionnaires or interviews and
they can be very useful in qualitatively checking the behavior and results of
new methods [1]. An interesting multirelational network about criminal
relationships is described by Bright et al. [13] who focused on eight types of
edges related to the exchange of a particular resource (e.g. drugs, money) in
a criminal network with $128$ actors. We propose a new real multilayer
criminal network derived from two simple graphs described in our earlier works
[14, 15, 16, 17], whose datasets are publicly available on Zenodo [18]. These
simple graphs capture the physical meetings and phone calls among suspects in
an anti-mafia operation called “Montagna”, concluded in 2007 by the Public
Prosecutor’s Office of Messina (Sicily). Starting from these two simple
graphs, we created an undirected and weighted multilayer network with two
layers called Meetings and Phone Calls, $154$ nodes and $439$ edges. The
Sicilian Mafia [19, 20, 21] is one of the most renowned criminal organisations
(i.e. clans, families, gangs, syndicates) whose social structure analysis
generated great scientific interest [22]. SNA has become an important tool for
the study of criminal networks [23, 24, 25] and it can be used to describe the
structure and functioning of a criminal organisation, to construct crime
prevention systems [26, 15], to identify leaders within a criminal
organisation [27] or to evaluate police interventions aimed at dismantling and
disrupting criminal networks [28, 29, 16]. To identify leaders within a
criminal network, we have to use a family of measures aimed at identifying the
most important actors in a social network [2]. The family of centrality
measures is probably the most widely applied set of SNA tools in practical
contexts. Centrality [30] is an intrinsically relational concept, because to
be central, an actor needs to have relations [1]. An actor might be important
because he is connected to a large number of different nodes or because he is
connected to other important nodes. An actor can also be considered important
because his absence would result in a loosely connected social network made of
many isolated components. Centrality is also often described in terms of the
power that an actor could receive from it (e.g. an actor strategically located
within a network will have a high control over the information flowing through
the network). In this paper, we focus on the importance of nodes in multilayer
networks. We used three different approaches (see details in Sect. 4) to
compute the nodes’ degree (i.e. the number of edges adjacent to it) and to
identify the $20$ most important nodes (i.e. the key actors) in our criminal
multilayer network.

##  2 Background

###  2.1 Single-layer networks

A single-layer network is a simple graph [2] which can be defined as a tuple
$G=(V,E)$ where $V=\\{v_{1},v_{2},...,v_{N}\\}$ is the set composed by $N$
nodes and $E=\\{l_{1},l_{2},...,l_{L}\\}$, $E\subseteq V\times V$, is the set
of edges, whose generic element $l_{k}$ represents the edge existing between a
pair of nodes $(v_{i},v_{j})$. The graph edges sometimes have weights, which
indicate the strength (or some other attribute) of each connection between the
nodes.

From a purely mathematical point of view, the information contained in both
sets $V$ and $E$ can be represented in an $A$ matrix of dimension $N\times N$
called the adjacency matrix [2], whose generic element $a_{ij}$ is defined as

$a_{ij}=\begin{cases}$1$&\text{if $v_{i}\rightarrow v_{j}\qquad
i,j=1,2,...,N$}\\\ $0$&\text{otherwise}\end{cases}$ (1)

In undirected graphs, $a_{ij}=a_{ji}$ for all $i\neq j$ and therefore the
adjacency matrix $A$ will be symmetrical: $A=A^{T}$ (where $T$ is the
transpose representation).

It is possible to define some descriptive measures able to highlight
particular characteristics of a network. For example, we define the degree of
a node $d(v_{i})$ as the number of nodes adjacent to it [2] or as the
cardinality of the set of neighbors of that node:

$d(v_{i})=|\mathcal{N}(v_{i})|=|\\{v_{j}:\exists(v_{i},v_{j})\lor\exists(v_{j},v_{i}),j\neq
i\\}|$ (2)

The degree assumes a discrete value between a minimum of $0$ when a node is
isolated (i.e. it is not connected to any other node) and a maximum of $N-1$
if the node is connected to all other nodes in the network. The degree of a
node can be calculated by adding the columns (or rows) of the adjacency matrix
$A$ [2]:

$d(v_{i})=\sum_{j=1}^{N}a_{ij}=\sum_{i=1}^{N}a_{ij}$ (3)

The concepts presented so far can be expressed using an alternative notation,
which makes use of tensors and Einstein’s notation [4, 8].

Given the canonical basis in the vector space $\mathbb{R}^{N}$,
$\xi=\\{e_{1},e_{2},...,e_{N}\\}$ where $e_{i}=(0,...,0,1,0,...,0)^{T}$ is $1$
in the ith component, and $0$ otherwise. Given a set of $N$ nodes $v_{i}$
(where $i=1,2,...,N$ and $N\in\mathbb{N}$), we associate with each node a
state that is represented by the canonical vector $e_{i}$ in the vector space
$\mathbb{R}^{N}$. A node $v_{i}$ can be related with each other and the
presence and the intensity of such relationships in the vector space is
indicated using the tensor product [31] (i.e. the Kronecker product)
$\mathbb{R}^{N}\otimes\mathbb{R}^{N}=\mathbb{R}^{N\times N}$. Thus, second-
order (i.e. rank-2) canonical tensors are defined by $E_{ij}=e_{i}\otimes
e^{T}_{j}$ (where $i,j=1,2,...,N$). The relationship tensor can consequently
be written as

$W=\sum_{i=1}^{N}\sum_{j=1}^{N}w_{ij}E_{ij}=\sum_{i=1}^{N}\sum_{j=1}^{N}w_{ij}e_{i}\otimes
e^{T}_{j}$ (4)

where the intensity of the relationship from node $v_{i}$ to node $v_{j}$ is
indicated by $w_{ij}$.

The matrix $W$ is an example of an adjacency tensor and, in the context of
single-layer networks, it is just a $N\times N$ matrix of a weighted graph
with $N$ nodes which is equivalent to the adjacency matrix $A$.

The covariant notation by Ricci and Levi-Civita [32] can be used to write an
adjacency tensor. In this case, a row vector $a\in\mathbb{R}^{N}$ can be
represented using its covariant and controvariant components which are the
vector $a_{\alpha}$ (i.e. $\alpha=1,2,...,N$) and its dual vector $a^{\alpha}$
(i.e. a column vector in the Euclidean space). A linear combination of tensors
in the canonical basis can be used to represent the adjacency tensor $W$:

$W^{\alpha}_{\beta}=\sum_{i=1}^{N}\sum_{j=1}^{N}w_{ij}e^{\alpha}(i)e_{\beta}(j)=\sum_{i=1}^{N}\sum_{j=1}^{N}w_{ij}E^{\alpha}_{\beta}(ij)$
(5)

where $e^{\alpha}(i)$ and $e_{\beta}(j)$ are respectively the $\alpha$th and
the $\beta$th components of the $i$th contravariant and $j$th covariant
canonical vectors in $\mathbb{R}^{N}$,
$E^{\alpha}_{\beta}(ij)\in\mathbb{R}^{N\times N}$ is the tensor in the
canonical basis which represents $E_{ij}$ (i.e. the tensor product of the
canonical vectors assigned to nodes $v_{i}$ and $v_{j}$).

Define the 1-vector $u^{\alpha}=(1,1,...,1)^{T}\in\mathbb{R}^{N}$ and let
$U_{\alpha}^{\beta}=u_{\alpha}u^{\beta}$ be the second-order tensor whose
elements are all equal to 1 (i.e. a so-called 1-tensor). The degree vector is
calculated adding up all the columns of the adjacency tensor defined in Eq. 5:

$k_{\beta}=W^{\alpha}_{\beta}u_{\alpha}$ (6)

It is possible to calculate the degree of node $v_{i}$ by projecting the
degree vector onto the $i$th canonical vector:

$k(i)=k_{\beta}e^{\beta}(i)$ (7)

###  2.2 Multilayer networks

Kivelä et al. [3] define a multilayer network as the most general structure
which can be used to represent any kind of network. At the base of this
structure, there is the elementary concept of graph, defined in Subsect. 2.1.
The representation of networks at multiple levels or with multiple types of
edges (or with other similar features) requires structures that have layers in
addition to nodes and edges. Moreover, the concept of aspect can be defined as
a feature of a layer representing one dimension of the layer structure (e.g.
the type of an edge or the time at which an edge is present) [12]. More
specifically, an “elementary layer” is an element of one of the possible sets
of layers from a specific aspect and the term “layer” refers to a combination
of elementary layers from all aspects.

A multilayer network can be defined as a quadruplet $M=(V_{M},E_{M},V,L)$.
$V_{M}\subseteq V\times L_{1}\times\cdot\cdot\cdot\times L_{d}$ is the set of
the node-layer combinations, that is the set of layers in which a node
$v_{i}\in V$ is present. $E_{M}\subseteq V_{M}\times V_{M}$ is the edge set
containing the set of pairs of possible combinations of nodes and elementary
layers. $V$ is the set of all nodes independently from the layer.
$L=\\{L_{a}\\}^{d}_{a}=1$ is the sequence of sets of elementary layers such
that there is one set of elementary layers $L_{a}$ for each aspect $a$. If
$d=0$, the multilayer network $M$ reduces to a single-layer network. If $d=1$,
then $M$ reduces to a multiplex network.

Using multiple layers, it is possible to represent different types of edges:
those among nodes in the same layer, called intralayer edges, and those among
nodes in different layers, called interlayer edges. For this reason, the
concepts of intralayer and interlayer adiacency tensor are introduced [4]. The
intralayer adiacency tensor $C^{\alpha}_{\beta}(\widetilde{k}\widetilde{k})$
is defined as the relationships among nodes in the same layer $\widetilde{k}$
which is indicated by the second-order tensor
$W^{\alpha}_{\beta}(\widetilde{k})$, where $\alpha,\beta=1,2,...,N$ as defined
in Eq. 5. Indices which refer to layers are distinguished from those which
correspond to nodes using the tilde symbol. The second-order interlayer
adjacency tensor $C^{\alpha}_{\beta}(\widetilde{h}\widetilde{k})$ is
introduced instead to encode information about relationships between nodes on
different layers (e.g. a node $v_{i}$ from layer $\widetilde{h}$ can be
connected to a node $v_{j}$ in an other layer $\widetilde{k}$). The interlayer
adjacency tensor $C^{\alpha}_{\beta}(\widetilde{h}\widetilde{k})$ corresponds
to the intralayer adjacency tensor $W^{\alpha}_{\beta}(\widetilde{k})$ when
the same layer $\widetilde{k}$ is represented by a couple of layers.

Following a similar approach to the one used to define the adjacency tensor
for single-layer networks (see Eq. 5), the vector
$e^{\widetilde{\gamma}}(\widetilde{k})$ (where
$\widetilde{\gamma},\widetilde{k}=1,2,...,L$) of the canonical basis in the
space $\mathbb{R}^{L}$ is introduced. In this definition, the vector
components are indicated by a greek index while the $k$th canonical vector is
indicated by a latin index. Therefore the second-order tensors, which
correspond to the canonical basis in the space $\mathbb{R}^{L\times L}$, are
constructed as

$E^{\widetilde{\gamma}}_{\widetilde{\delta}}(\widetilde{h}\widetilde{k})=e^{\widetilde{\gamma}}(\widetilde{h})e_{\widetilde{\delta}}(\widetilde{k})$
(8)

The multilayer adjacency tensor [8] can be written from Eq. 8, using a tensor
product between the adjacency tensors
$C^{\widetilde{\alpha}}_{\widetilde{\beta}}(\widetilde{h}\widetilde{k})$ and
the canonical tensors
$E^{\widetilde{\gamma}}_{\widetilde{\delta}}(\widetilde{h}\widetilde{k})$. A
fourth-order tensor is obtained as

$M^{\alpha\widetilde{\gamma}}_{\beta\widetilde{\delta}}=\sum_{\widetilde{h}=1}^{L}\sum_{\widetilde{k}=1}^{L}C^{\alpha}_{\beta}(\widetilde{h}\widetilde{k})E^{\widetilde{\gamma}}_{\widetilde{\delta}}(\widetilde{h}\widetilde{k})$
(9)

The second-order interlayer adjacency tensor
$C^{\alpha}_{\beta}(\widetilde{h}\widetilde{k})$ can be written when
$\widetilde{h}=\widetilde{k}$ as

$C^{\alpha}_{\beta}(\widetilde{h}\widetilde{k})=\sum_{i=1}^{N}\sum_{j=1}^{N}w_{ij}(\widetilde{h}\widetilde{k})E^{\alpha}_{\beta}(ij)$
(10)

where $w_{ij}(\widetilde{h}\widetilde{k})$ are just real numbers which specify
the intensity of the relationship between a node $v_{i}$ in layer
$\widetilde{h}$ and a node $v_{j}$ in an other layer $\widetilde{k}$. Then,
the fourth-order tensor of the canonical basis of the space
$\mathbb{R}^{N\times N\times L\times L}$ is defined as

$\xi^{\alpha\widetilde{\gamma}}_{\beta\widetilde{\delta}}(ij\widetilde{h}\widetilde{k})=E^{\alpha}_{\beta}(ij)E^{\widetilde{\gamma}}_{\widetilde{\delta}}(\widetilde{h}\widetilde{k})=e^{\alpha}(i)e_{\beta}(j)e^{\widetilde{\gamma}}(\widetilde{h})e_{\widetilde{\delta}}(\widetilde{k})$
(11)

Replacing in Eq. 9 the expressions obtained in Eq. 10 and Eq. 11, the
multilayer adjacency tensor can be written as

$M^{\alpha\widetilde{\gamma}}_{\beta\widetilde{\delta}}=\sum_{\widetilde{h},\widetilde{k}=1}^{L}\sum_{i,j=1}^{N}w_{ij}(\widetilde{h}\widetilde{k})\xi^{\alpha\widetilde{\gamma}}_{\beta\widetilde{\delta}}(ij\widetilde{h}\widetilde{k})$
(12)

In some cases, it is possible to aggregate multiple networks constructing a
single-layer network. This aggregation can be useful in different kind of
studies such as those on temporal or social networks. To change a multilayer
network into a weighted single-layer network, the corresponding tensor is
multiplied by the 1-tensor
$U_{\alpha\widetilde{\gamma}}^{\beta\widetilde{\delta}}$. The obtained
projected single-layer network $P^{\alpha}_{\beta}$ [4] is

$P^{\alpha}_{\beta}=M^{\alpha\widetilde{\gamma}}_{\beta\widetilde{\delta}}U_{\widetilde{\gamma}}^{\widetilde{\delta}}=\sum_{\widetilde{h}=1}^{L}\sum_{\widetilde{k}=1}^{L}C^{\alpha}_{\beta}(\widetilde{h}\widetilde{k})$
(13)

A structure similar to the projected single-layer network is the aggregate or
overlay single-layer network [4]. It is obtained ignoring the interlayer edges
and summing for each node the edges over all layers in the multilayer network.
The multilayer adjacency tensor is used to define the aggregate network
contracting the indices which correspond to the layer components as

$O^{\alpha}_{\beta}=M^{\alpha\widetilde{\gamma}}_{\beta\widetilde{\gamma}}=\sum_{\widetilde{r}=1}^{L}W^{\alpha}_{\beta}(\widetilde{r})$
(14)

In the case of an aggregate network, the degree computation is the same of a
single-layer network and it is computed like in Eq. 6.

On the contrary, the multidegree centrality vector $K^{\alpha}$ [4] is defined
using 1-tensors of the appropriate order as in the case of single-layer
networks:

$K^{\alpha}=\Big{[}M^{\alpha\widetilde{\gamma}}_{\beta\widetilde{\delta}}U_{\widetilde{\gamma}}^{\widetilde{\delta}}\Big{]}u^{\beta}=\Big{[}P^{\alpha}_{\beta}\Big{]}u^{\beta}=\Bigg{[}\sum_{\widetilde{h}=1}^{L}\sum_{\widetilde{k}=1}^{L}C^{\alpha}_{\beta}(\widetilde{h}\widetilde{k})\Bigg{]}u^{\beta}=\sum_{\widetilde{h}=1}^{L}\sum_{\widetilde{k}=1}^{L}k^{\alpha}(\widetilde{h}\widetilde{k})$
(15)

where $k^{\alpha}(\widetilde{h}\widetilde{k})$ is the degree vector defined in
Eq. 6 computed on the interlayer adjacency tensor
$C^{\alpha}_{\beta}(\widetilde{h}\widetilde{k})$.

##  3 Dataset

Our dataset (available on Zenodo [18]) derives from an anti-mafia operation
called “Montagna” concluded in 2007 by the Public Prosecutor’s Office of
Messina (Sicily) and a specialized anti-mafia police unit of the Italian
Carabinieri called R.O.S. (Special Operations Group). The investigation
focused on two Mafia families known as “Mistretta” and “Batanesi”. From 2003
to 2007, these families infiltrated several economic activities including the
public works in the north-eastern part of Sicily, through a cartel of
entrepreneurs close to the Mafia. The “Mistretta” family had also a role of
mediator between other families in Palermo and Catania and other criminal
organizations around Messina, such as the “Barcellona” and the “Caltagirone”
families. Our main data source is the pre-trial detention order issued on
March 14, 2007 by the judge for preliminary investigations of the Court of
Messina. The Court ordered the pre-trial detention for 38 individuals writing
a document of more than two hundred pages with an lot of details about the
suspects’ crimes, activities, meetings, and calls. From the analysis of this
document, we initially built two graphs: Meetings and Phone Calls, in which
nodes were uniquely associated with suspected criminals and edges specified
meetings and phone calls respectively among individuals [14, 15, 16, 17]. The
Meetings graph had $101$ nodes and $256$ edges. The Phone Calls graph had
$100$ nodes and $124$ edges. There were $47$ individuals who jointly belonged
to both graphs.

Figure 1: Multilayer network. The layered visualization is obtained using
Muxviz. The color of nodes is given by their community assignment (i.e. how
actors are clustered together) and the size by their degree.

Starting from the Meetings and Phone Calls simple graphs, we created an
undirected and weighted multilayer network with two layers called Meetings and
Phone Calls, $154$ nodes and $439$ edges (see Fig. 1). The links within layer
Meetings refer to the meetings among members of the criminal network while
edges in the layer Phone Calls represent phone communications among distinct
phone numbers they use. The weight encodes the number of meetings or phone
calls. According to the definition of the different kind of multilayer
networks in Subsect. 2.2, we can identify our network as an edge-colored
multilayer (i.e. a network with multiple types of edges) and more precisely a
multiplex network which does not require all nodes to exist on every layer.
Each layer have to share at least one node with some other layer in the
network to be multiplex. In our case the two layers share $47$ nodes.
Moreover, interlayer edges are only those between nodes and their counterparts
in another layers and no cost is associated to them.

##  4 Methodology and Results

In this paper, we perform an analysis of the nodes’ importance in the
multiplex network described in Sect 3. The concept of centrality as
“importance” is debated and strongly depends on the context. Here we used a
simple descriptor which is the degree. This measure quantifies the number of
different interaction of each node. The computation of the nodes’ degree in a
multilayer network can be done using three different approaches:

The two layers of the multilayer network are merged to obtain a single-layer
network (i.e. the aggregate network shown in Fig. 2). This process, often
called flattening, is performed creating a new network with one node for every
actor and an edge between two nodes if the corresponding actors are connected
in any of the layers. Once the aggregate network is obtained, traditional
degree (see Eq. 6 in Sect. 2) can be computed.

The traditional degree (see Eq. 6 in Sect. 2) can be applied to each layer
separately. Then, the results are compared.

Multiple layers are considered at the same time, but without treating them as
being ontologically different. Measures based on this approach explicitly
consider the difference between interlayer and intralayer edges and also make
numerical distinctions between different layers (e.g. through weights), but at
the end they typically produce single numerical values merging the
contributions of the different types of edges [4, 5] (see Eq. 15).

Figure 2: Aggregate network. The edge-colored multigraph visualization is
obtained using Muxviz. The color of nodes is given by their community
assignment (i.e. how actors are clustered together) and the size by their
degree.

Table 1 gives a summary of the 20 top nodes ranked by their degree in the
Aggregate network (i.e. according to 4), in the single layers Phone Calls and
Meetings (i.e. according to 4) and in the Multilayer network (i.e. according
to 4). The nodes’ importance given by their degree is compared with the real
roles these nodes have in the Sicilian Mafia families observed during the
“Montagna” operation. These roles have been reconstructed by us while reading
court documents of the “Montagna” operation and they are also available on
Zenodo [18].

Table 1: The 20 top ranked nodes in the Multilayer, Aggregate, Phone Calls and Meetings networks compared with their roles in the “Montagna” Operation. Node | Degree
---|---
Name |  Role |  Multilayer |  Aggregate |  Phone Calls |  Meetings
18 |  Caporegime Mistretta Family |  51 |  41 |  25 |  24
47 |  Deputy Caporegime Batanesi Family |  42 |  29 |  21 |  19
27 |  Caporegime Batanesi Family |  29 |  21 |  11 |  16
68 |  Caporegime Batanesi Family |  27 |  19 |  10 |  15
29 |  Enterpreneur |  24 |  16 |  9 |  13
61 |  Caporegime Mistretta Family |  23 |  19 |  17 |  4
45 |  Associate Batanesi Family |  20 |  14 |  6 |  12
12 |  Associate Mistretta Family |  19 |  16 |  1 |  16
11 |  Mafia activity coordinator in Messina |  18 |  15 |  4 |  12
22 |  Pharmacist |  18 |  15 |  2 |  14
51 |  Associate Batanesi Family |  17 |  11 |  4 |  11
25 |  Caporegime Mistretta Family |  16 |  13 |  1 |  13
43 |  Messaggero |  16 |  11 |  5 |  9
48 |  Associate Batanesi Family |  15 |  12 |  1 |  12
19 |  External partnership |  14 |  11 |  3 |  9
36 |  Aiding and abetting of a fugitive |  14 |  11 |  4 |  8
75 |  Associate Mistretta Family |  14 |  12 |  8 |  4
89 |  Associate Batanesi Family |  14 |  12 |  Absent node |  12
54 |  Enterpreneur |  13 |  7 |  5 |  6
5 |  Sighted with nodes 11 and 12 |  12 |  10 |  Absent node |  10
Figure 3: The structure of a Mafia Family.

In particular, a Mafia Family has a typical structure which is shown in Fig.
3. On top of the pyramid hierarchical chart is the Boss who makes all the
major decisions, controls the Mafia members and resolves any disputes. Usually
the real boss keeps a low-profile and keeps his real identity hidden. Just
below the boss is the Underboss who is the second in command. He can resolve
disputes without involving the boss himself and replaces the boss if he is old
or in danger of going to jail. In-between the boss and underboss is a role of
the Consigliere who is an advisor to the boss and makes impartial decisions
based upon fairness and for the good of the Mafia. Also in-between there is
the Messaggero who is a messenger who functions as liaison between criminal
families. He can reduce the need for sit-downs, or meetings, of the mob
hierarchy, and thus limits the public exposure of the bosses. Below the
underboss is the Caporegime (also called Captain or Capo) who manages his own
crew within the criminal family in a designated geographical location. A
Capo’s career relies heavily on how much money they can bring into the family.
How many capos there are in a given family simply depends on how big that
family is. Then, there are soldiers who report to their Caporegime. They are
street level mobsters who essentially are no more than your average type
criminals. Many soldiers can be assigned to one Capo. The final part of a
family comes in the shape of associates, who are not actual members of the
Mafia, but they work with Mafia soldiers and caporegimes on various criminal
enterprises. An associate is simply someone who works with the mob, including
anyone from a burglar or drug dealer to a pharmacist, entrepreneur, lawyer,
investment banker, police officer or politician.

Figure 4: The 20 top ranked nodes by degree in Multilayer (Blue), Aggregate
(Orange), Phone Calls (Green) and Meetings (Red) networks.

We performed the analysis of nodes’ degree for each layer separately, the
aggregate and the multilayer networks using Muxviz [33] and Python. The
multilayer framework allows to quantify the importance of a node across all
the layers. The top 20 nodes ranked by their degree are shown in Fig. 4. The
results for each layer separately (i.e. Meetings or Phone Calls), shown in the
stacked histogram, reveal that the most important actors per layer are nodes
$18$ and $47$. The result from the aggregate network, obtained by summing up
all interactions across the whole network while neglecting the layered
structure, also identify nodes $18$ and $47$ as the most central actors. The
same result is obtained for the multilayer network, i.e. considering the
layered structure. These two nodes are effectively important because they are
respectively Caporegime of the Mistretta family and deputy Caporegime of the
Batanesi family. Using the multilayer framework it was possible to identify
two key Caporegimes of the Mistretta family (i.e. Nodes $61$ and $25$). Node
$61$ was one the twenty most important nodes in the Phone Calls layer but not
in the Meetings layer. Node $25$ was one the twenty most important nodes in
the Meetings layer but not in the Phone Calls layer. So, the importance of
these nodes doesn’t emerge from the analysis of the single layers but only
from the analysis of the Aggregate network and even more of the Multilayer
one. We can also identify the Messaggero (i.e. Node $43$) who didn’t seem so
important from the analysis of the single layers. Then, we can identify some
key associates such us pharmacist or entrepreneurs needed in synthetic drug
synthesis processes or to facilitate the award of public contracts to
companies close to criminal organizations. The identification of these figures
can be very useful to define attack strategies to disrupt criminal networks
[28, 29, 16].

##  5 Conclusions

In this paper, we used a real criminal dataset obtained by parsing a two
hundred pages pre-trial detection order by the Court of Messina during an
anti-mafia operation called “Montagna” concluded in 2007. Starting from this
dataset we initially built two social networks, one for meetings and one for
phone calls among suspects. Since some suspects meet and also call each other,
it was natural to identify meetings and phone calls with the layer of a
multiplex network. First thing we did was to import the data as an undirected
and weighted network where the weight encoded the number of meetings or phone
calls.

We wanted to perform an analysis of suspects’ importance for each layer
separately, for the aggregate network and the multilayer one which allowed to
quantify the importance across the whole series of layers. The concept of
centrality as “importance” is debated and strongly depends on the context.
Here we used a simple descriptor which is the degree. This measures just
quantifies the number of different meetings or phone calls of each suspect.

We showed the top 20 characters ranked by their degree in each layer, in the
aggregate network and in the multilayer network comparing the resulting
importance with the role they had in the Sicilian Mafia families which were
the protagonists of the “Montagna” operation.

As future works, we want to apply to our network other centrality descriptors
such as multiplexity which quantifies the fraction of layers where a node
appears and PageRank which was introduced by Google’s founders and ranks nodes
by assuming that more important ones are likely to interact with other
important nodes. Then we want to compute the multiplex participation
coefficient [9] which quantifies the participation of a node to the different
communities of a network.

A further analysis of our dataset revealed the possibility to build a new
multilayer network with three layers, one for meetings, one for phone calls
and a third layer for the crimes committed by the suspects.

## References

  * [1] Dickison, M.E., Magnani, M., Rossi, L.: Multilayer Social Networks. Cambridge University Press (2016). https://doi.org/10.1017/CBO9781139941907
  * [2] Wasserman, S., Faust, K.: Social network analysis: Methods and applications, vol. 8. Cambridge University Press (1994). https://doi.org/10.1017/CBO9780511815478
  * [3] Kivelä, M., Arenas, A., Barthelemy, M., Gleeson, J.P., Moreno, Y., Porter, M.A.: Multilayer networks. Journal of Complex Networks 2(3), 203–271 (2014). https://doi.org/10.1093/comnet/cnu016
  * [4] De Domenico, M., Solé-Ribalta, A., Cozzo, E., Kivelä, M., Moreno, Y., Porter, M.A., Gómez, S., Arenas, A.: Mathematical formulation of multilayer networks. Phys. Rev. X 3, 041022 (2013). https://doi.org/10.1103/PhysRevX.3.041022
  * [5] De Domenico, M., Solé-Ribalta, A., Omodei, E., Gómez, S., Arenas, A.: Ranking in interconnected multilayer networks reveals versatile nodes. Nature Communications 6(1), 6868 (2015). https://doi.org/10.1038/ncomms7868
  * [6] Boccaletti, S., Bianconi, G., Criado, R., del Genio, C., Gómez-Gardeñes, J., Romance, M., Sendiña-Nadal, I., Wang, Z., Zanin, M.: The structure and dynamics of multilayer networks. Physics Reports 544(1), 1–122 (2014). https://doi.org/10.1016/j.physrep.2014.07.001
  * [7] Catanese, S.A.: New perspectives in criminal network analysis: multilayer networks, time evolution, and visualization. Ph.D. thesis, University of Catania (2017)
  * [8] Degani, E.: Monoplex to Multiplex networks analysis generalization: formalization, description and implementation of the commonest measures via a statistical package. BSc thesis, University of Padua (2016)
  * [9] Battiston, F., Nicosia, V., Latora, V.: Structural measures for multiplex networks. Phys. Rev. E 89, 032804 (2014). https://doi.org/10.1103/PhysRevE.89.032804
  * [10] Solé-Ribalta, A., De Domenico, M., Gómez, S., Arenas, A.: Centrality rankings in multiplex networks. In: Proceedings of the 2014 ACM Conference on Web Science. p. 149–155. WebSci ’14, Association for Computing Machinery, New York, NY, USA (2014). https://doi.org/10.1145/2615569.2615687
  * [11] Nicosia, V., Latora, V.: Measuring and modeling correlations in multiplex networks. Phys. Rev. E 92, 032805 (2015). https://doi.org/10.1103/PhysRevE.92.032805
  * [12] Tomasini, M.: An introduction to multilayer networks (2015). https://doi.org/10.13140/RG.2.2.16830.18243
  * [13] Bright, D.A., Greenhill, C., Ritter, A., Morselli, C.: Networks within networks: using multiple link types to examine network structure and identify key actors in a drug trafficking operation. Global Crime 16(3), 219–237 (2015). https://doi.org/10.1080/17440572.2015.1039164
  * [14] Ficara, A., Cavallaro, L., De Meo, P., Fiumara, G., Catanese, S., Bagdasar, O., Liotta, A.: Social Network Analysis of Sicilian Mafia Interconnections. In: Complex Networks and Their Applications VIII, pp. 440–450. Springer International Publishing (2020). https://doi.org/10.1007/978-3-030-36683-4_36
  * [15] Calderoni, F., Catanese, S., De Meo, P., Ficara, A., Fiumara, G.: Robust link prediction in criminal networks: A case study of the Sicilian Mafia. Expert Systems with Applications 161, 113666 (2020). https://doi.org/10.1016/j.eswa.2020.113666
  * [16] Cavallaro, L., Ficara, A., De Meo, P., Fiumara, G., Catanese, S., Bagdasar, O., Song, W., Liotta, A.: Disrupting resilient criminal networks through data analysis: The case of Sicilian Mafia. PLOS ONE 15(8), 1–22 (2020). https://doi.org/10.1371/journal.pone.0236476
  * [17] Cavallaro, L., Ficara, A., Curreri, F., Fiumara, G., De Meo, P., Bagdasar, O., Liotta, A.: Graph Comparison and Artificial Models for Simulating Real Criminal Networks. In: Complex Networks and Their Applications IX, pp. 286–297. Springer International Publishing (2021). https://doi.org/10.1007/978-3-030-65351-4_23
  * [18] Cavallaro, L., Ficara, A., De Meo, P., Fiumara, G., Catanese, S., Bagdasar, O., Song, W., Liotta, A.: Criminal Network: The Sicilian Mafia. “Montagna Operation”. Zenodo (2020). https://doi.org/10.5281/zenodo.3938818
  * [19] Gambetta, D.: The Sicilian Mafia: The Business of Private Protection. Harvard University Press, Cambridge (1996)
  * [20] Paoli, L.: Italian organised crime: Mafia associations and criminal enterprises. Global Crime Today: The Changing Face of Organised Crime 6(1), 19–32 (2004). https://doi.org/10.1080/1744057042000297954
  * [21] Paoli, L.: Mafia brotherhoods: Organized crime, Italian style. Oxford University Press, Oxford Scholarship Online (2008). https://doi.org/10.1093/acprof:oso/9780195157246.001.0001
  * [22] Kleemans, E.R., de Poot, C.J.: Criminal careers in organized crime and social opportunity structure. European Journal of Criminology 5(1), 69–98 (2008). https://doi.org/10.1177/1477370807084225
  * [23] Ferrara, E., De Meo, P., Catanese, S., Fiumara, G.: Visualizing criminal networks reconstructed from mobile phone records. CEUR Workshop Proceedings 1210 (2014)
  * [24] Ferrara, E., De Meo, P., Catanese, S., Fiumara, G.: Detecting criminal organizations in mobile phone networks. Expert Systems with Applications 41(13), 5733–5750 (2014). https://doi.org/10.1016/j.eswa.2014.03.024
  * [25] Agreste, S., Catanese, S., De Meo, P., Ferrara, E., Fiumara, G.: Network structure and resilience of Mafia syndicates. Information Sciences 351, 30–47 (2016). https://doi.org/10.1016/j.ins.2016.02.027
  * [26] Berlusconi, G., Calderoni, F., Parolini, N., Verani, M., Piccardi, C.: Link prediction in criminal networks: A tool for criminal intelligence analysis. PLOS ONE 11(4), 1–21 (2016). https://doi.org/10.1371/journal.pone.0154244
  * [27] Johnsen, J.W., Franke, K.: Identifying central individuals in organised criminal groups and underground marketplaces. In: Shi, Y., Fu, H., Tian, Y., Krzhizhanovskaya, V.V., Lees, M.H., Dongarra, J., Sloot, P.M.A. (eds.) Computational Science – ICCS 2018. pp. 379–386. Springer International Publishing, Cham (2018). https://doi.org/10.1007/978-3-319-93713-7_31
  * [28] Duijn, P.A.C., Kashirin, V., Sloot, P.M.A.: The relative ineffectiveness of criminal network disruption. Scientific Reports 4(1), 4238 (2014). https://doi.org/10.1038/srep04238
  * [29] Villani, S., Mosca, M., Castiello, M.: A virtuous combination of structural and skill analysis to defeat organized crime. Socio-Economic Planning Sciences 65(C), 51–65 (2019). https://doi.org/10.1016/j.seps.2018.01.00
  * [30] Ficara, A., Fiumara, G., De Meo, P., Liotta, A.: Correlations among Game of Thieves and other centrality measures in complex networks. In: Fortino, G., Liotta, A., Gravina, R., Longheu, A. (eds.) Data Science and Internet of Things. Springer International Publishing (2021). https://doi.org/10.1007/978-3-030-67197-6_3
  * [31] Abraham, R., Marsden, J.E., Ratiu, R.: Manifolds, Tensor Analysis, and Applications: 2nd Edition. Springer-Verlag, Berlin, Heidelberg (1988). https://doi.org/10.1007/978-1-4612-1029-0
  * [32] Ricci, M.M.G., Levi-Civita, T.: Méthodes de calcul différentiel absolu et leurs applications. Mathematische Annalen 54(1), 125–201 (1900). https://doi.org/10.1007/BF01454201
  * [33] De Domenico, M., Porter, M.A., Arenas, A.: MuxViz: a tool for multilayer analysis and visualization of networks. Journal of Complex Networks 3(2), 159–176 (2014). https://doi.org/10.1093/comnet/cnu038