# Self-training Large Language Models
through Knowledge Detection

Wei Jie Yeo1, Teddy Ferdinan2, Przemyslaw Kazienko2, Ranjan Satapathy3, Erik
Cambria1
1Nanyang Technological University
2Wroclaw University of
Science and Technology
3Institute of High Performance Computing (IHPC),
Agency for Science, Technology and Research (A$\textasteriskcentered$ STAR)
<EMAIL_ADDRESS>

###### Abstract

Large language models (LLMs) often necessitate extensive labeled datasets and
training compute to achieve impressive performance across downstream tasks.
This paper explores a self-training paradigm, where the LLM autonomously
curates its own labels and selectively trains on unknown data samples
identified through a reference-free consistency method. Empirical evaluations
demonstrate significant improvements in reducing hallucination in generation
across multiple subjects. Furthermore, the selective training framework
mitigates catastrophic forgetting in out-of-distribution benchmarks,
addressing a critical limitation in training LLMs. Our findings suggest that
such an approach can substantially reduce the dependency on large labeled
datasets, paving the way for more scalable and cost-effective language model
training111https://github.com/wj210/Self-Training-LLM.

##  1 Introduction

Large language models (LLMs) have revolutionized natural language processing
(NLP), enabling remarkable performance across various downstream tasks Llama-3
(2024). However, their development is heavily reliant on vast amounts of
labeled data and significant computational resources, which are not always
readily accessible Cambria et al. (2023). Self-learning is an applicable field
that can tackle such limitations and enables a low-resource training
environment. However, LLMs are known to hallucinate Huang et al. (2023) due to
the inherent biases, noise in their pre-training dataset, or just lack of
data. This makes it challenging to apply self-learning to continuously improve
the knowledge of the model.

Another key problem in LLM fine-tuning is catastrophic forgetting Luo et al.
(2023). This phenomenon occurs when the model learns new information in one
domain but simultaneously suffers from a degradation of knowledge in
previously acquired areas. A naïve solution is to exploit larger data mixing
new and old knowledge, which may not be feasible for domains with limited
resources. Alternative approaches, such as continual learning Ke et al.
(2023); Jang et al. (2021) and inference-only correction Meng et al. (2022a);
Hernandez et al. (2023) offer potential solutions. However, these methods face
other limitations such as reduced efficiency in learning new knowledge and
scalability towards large domains.

To address the aforementioned limitations, this paper explores a self-training
paradigm where the LLM autonomously curates its own labels and performs
selective training on samples filtered using a new knowledge detection. This
measure identifies instances that are annotated as "unknown", indicating the
model’s low confidence in providing accurate answers Ferdinan et al. (2024);
Liang et al. (2024). This filtering step is specifically used to curate a
preference dataset to perform knowledge correction via Direct Preference
Optimization (DPO) Rafailov et al. (2024). The rationale behind performing the
selection step is twofold. Firstly, this allows for a larger distance between
the preferred and dispreferred sample, thereby reducing noise in the training.
This helps to prevent degeneration, a common issue observed when implementing
DPO Pal et al. (2024). Secondly, training exclusively on samples related to
lack of knowledge is resource-efficient and aids in retaining previously
learned information.

Our results demonstrate that the proposed framework enhances factual accuracy
in answering questions pertaining to a specified knowledge source.
Additionally, training on the selected samples not only preserves but, in some
instances, improves performance on out-of-distribution benchmarks. Comparative
analyses with baseline approaches, which demand higher computational
resources, reveal that our approach outperforms these baselines.

##  2 Related work

Self-Training: LLMs have demonstrated the capability to annotate datasets
without the need for human-annotated labels, facilitating a low-resource
training process for other LLMs. Typically, a larger model referred to as the
teacher, generates the labels, while a smaller model, the student, is trained
on these labels in a process known as context distillation. A range of
training and inference algorithms can be used, including conventional
supervised fine-tuning (SFT) Alpaca (2023); Mukherjee et al. (2023); Li et al.
(2022); Hsieh et al. (2023), in-context learning Krishna et al. (2024) and
preference optimization Tunstall et al. (2023); Llama-3 (2024). Self-learning
methods eliminate the need for the larger LLM, which typically requires
substantially more computational resources and incurs higher API costs. Recent
studies have shown that this is achievable, given an unlabeled dataset with a
small set of examples as supplementary context Huang et al. (2022); Tian et
al. (2023); Wang et al. (2022). He et al. (2019) performs an initial step of
supervised fine-tuning on a small labeled dataset before using the trained
generator to annotate the unlabeled set, Jie et al. (2024) similarly for
rationalization tasks. Meng et al. (2022b) augments a given labeled dataset
with additional samples, but is however limited to only classification tasks.
Our work diverges from these approaches where training is only conducted
exclusively on samples labeled as unknown. Cheng et al. (2024) is similar to
our work, but only teaches the model to abstain from answering unknown
questions.

Knowledge Detection: Detecting knowledge gaps in a model has been a long-
standing area of research, with the primary goal of assessing the truthfulness
of a model’s outputs. Early works employed questions structured in cloze
format to detect knowledge prescene Petroni et al. (2019), but this approach
is limited to unambiguous and short-form questions. Subsequent research, such
as Wang et al. (2023); Dong et al. (2024) asserts the presence of knowledge
through paraphrased and perturbated queries. FactScore Min et al. (2023)
decomposes a generation into a list of atomic facts and generates an average
truthfulness score relative a knowledge source, allowing for finer analysis.
Chern et al. (2023) utilizes multiple external tools, such as Google search,
GitHub, and others to perform fact-checking. SelfCheckGPT Manakul et al.
(2023) introduces a reference-free detection technique that evaluates the
likelihood of hallucinations by examining consistency across sampled
generations from the model. This is particularly useful in the event that the
labels or knowledge source is unavailable.

##  3 Self-training

Figure 1: An overview of the self-training framework, instruction generation
(1), SFT stage (2), preference labeling (3) and knowledge filtering (4). The
four steps are implemented in sequence and the final model is assessed for
truthfulness.

This section introduces the details of our self-training framework, broken
down into four sequential steps: Instruction Generation, SFT stage, Preference
Labeling and Knowledge Filtering. As a start, we assume access to a knowledge
source as the main source of material to perform both training and
truthfulness evaluation. The full illustration is shown in Figure 1. A key
benefit of our framework is that it does not require significant human efforts
besides a few manually crafted instruction-answer examples for in-context
generation.

###  3.1 Instruction generation

We utilize Wikipedia222https://huggingface.co/datasets/wikimedia/wikipedia as
the foundation of our knowledge source given its widespread acceptance and
reliability. Note that this framework is likely to be applicable to any other
form of knowledge sources as long as they do not contain any significant noise
or ambiguous information. In order to ensure comprehensive coverage across
subjects of notable interest, we sample documents from the following topics:
{Geography, Art, Medical, History, Biology, Science, Musician, Actor,
Economics, Astronomy}. For each topic, we randomly sample 100 documents to
form the training set and 10 documents for evaluation purposes. A crucial
aspect of our approach is ensuring that the generated instructions are
relevant to the documents and prompt for answers that can be found within
them. We observe that non-instructed pre-trained language models often fail in
this regard, generating irrelevant instructions. Therefore, we employ the
instruct-tuned LLM, OpenAI’s GPT-3.5 as the instruction generator, $G_{Instr}$
to construct the instruction sets.

For each document, we generate $N$ questions and remove any duplicate
questions within each document. We split the document into chunks of length,
$L=512$, where each chunk is provided as input to $G_{Instr}$, along with few-
shot examples. We find that providing non-overlapping contexts helps reduce
duplicate instructions, and including few-shot instruction generation examples
can align the generator to produce objective instructions. Subsequently, a de-
duplication step is performed across the $N$ questions within each document.

###  3.2 SFT stage

We start with a pre-trained LLM: $G_{PLM}$ that is to be self-trained. First,
it is used to self-annotate the instruction-only training set from the first
stage, given few-shot examples, forming the SFT dataset,
$D_{SFT}=\\{(x_{i},\hat{y}_{i})\mid i=1,2,\ldots,N\\}$, $x_{i}$ refers to the
instruction and $\hat{y}_{i}$ is the self-annotated label. Providing a set of
examples is a common procedure to avoid degenerated responses since $G_{PLM}$
is not inherently familiar with the task format Tian et al. (2023); Wang et
al. (2022); Huang et al. (2022). The primary difference lies with the addition
of a second dataset, Reading Comprehension (RC), $D_{RC}$ which is similar to
$D_{SFT}$, but includes the document chunk. We employ GPT-3.5 to generate the
label, by feeding both the instruction and document into the input prompt. The
purpose of $D_{RC}$ is to train the model in generating responses by
referencing the document, which we later show to be beneficial towards
stability in the training process. We use $\frac{1}{3}$ of the instruction set
to construct $D_{RC}$, with the remainder $\frac{2}{3}$for $D_{SFT}$. Both
datasets are then combined to perform SFT on $G_{PLM}$ to form the instruct-
tuned model $G_{SFT}$.

###  3.3 Preference Labeling

Given the instruct-tuned model $G_{SFT}$, we proceed to construct the
preference dataset to implement DPO. The primary objective at this stage is to
generate a dataset that corrects the biases learned during the SFT stage.
These biases arise due to the limitations of self-generating labels, which
depend on the knowledge acquired during the pre-training phase. For each
instruction, we provide $G_{SFT}$ with two input prompts: one including the
document chunk $c$ and one without. We sample $K$ generations from each format
to form the chosen set, $Y_{c}=f^{K}(G_{SFT},x,c)$, and the rejected set,
$Y_{r}=f^{K}(G_{SFT},x)$. Additionally, we use greedy decoding to generate
$y_{c}^{*}=f^{*}(G_{SFT},x,c)$. This forms the base preference dataset,
$D_{DPO}$, which is further filtered, see Sec. 3.4. Here we denote $f^{K}$ as
the sampling operation producing $K$ outputs and $f^{*}$ as the greedy
decoding operation. We assume that when the model is given the document, the
response will be more truthful than without it. We demonstrate empirically in
subsequent experiments that this assumption is valid.

###  3.4 Knowledge Filtering

Rather than straightforwardly performing the DPO directly on $D_{DPO}$, we
perform an additional filtering procedure, to minimize the noise in the
preference dataset. This filtering procedure is implemented across each sample
in $D_{DPO}$ and involves two stages: (1) consistency filtering and (2)
knowledge filtering. The idea of consistency filtering is to compute a
consistency score $S_{L}$, measuring the consistency of the reference response
$y_{c}^{*}$ with the $K$ chosen responses $Y_{c}$, corresponding to each
instruction. In contrast, knowledge filtering evaluates whether the SFT model
$G_{SFT}$ tends to hallucinate on a given sample, measured by the knowledge
score $S_{K}$, against $Y_{r}$.

$S_{L}=\frac{1}{K}\sum_{y_{c}\in Y_{c}}S_{C}(y_{c}^{*},y_{c})$ (1)
$S_{K}=\frac{1}{K}\sum_{y_{r}\in Y_{r}}S_{C}(y_{c}^{*},y_{r})$ (2)

To measure the difference between any two responses, we use the contradiction
score $S_{C}$ computed by a separate encoder trained on a vast amount of
natural language inference (NLI) data, this is similar to the NLI component in
SelfCheckGPT Manakul et al. (2023). $S_{C}$ represents the probability of the
contradiction class between a pair of responses. We chose SelfCheckGPT because
it is a reference-free method of detecting hallucination signs and is
relatively low-cost. In contrast, reference-based methods like FactScore Min
et al. (2023) require significantly more computation due to atomic fact
decomposition, making them costly for large datasets. Additionally,
SelfCheckGPT has shown a high correlation with human assessments in
hallucination detection.

In the first stage, $D_{DPO}$ from step three undergoes a consistency
filtering to filter out low-confidence responses. Intuitively, if the average
contradiction between the sampled responses and the greedy decoded response is
high, it indicates a higher probability of hallucination in the reference
response. This approach ensures that the final model does not maximize the
probability of low-quality answers. It is worth noting that this filtering
step could be performed during the SFT stage; however, we refrain from doing
so to avoid over-filtering, as a high contradiction score may result from
unfamiliarity with the task rather than a lack of knowledge. We fix the
threshold, $\tau_{L}$ to be 0.5, filtering out samples, $S_{L}>\tau_{L}$.

The second stage, knowledge filtering, removes samples where the model is
considered knowledgeable. The objective is to prevent over-training,
particularly on samples where the model has a higher accuracy tendency. This
approach has two benefits: first, it ensures a larger discrepancy between the
chosen and rejected responses, and second, it mitigates cases of catastrophic
forgetting. The first benefit is crucial for reducing noise in the
optimization objective, DPO in Equation 5 which aims to learn the optimal
policy by maximizing the margin between the probability of the chosen and
rejected candidates. The second benefit prevents overfitting on instances
where the model is sufficiently knowledgeable and may experience knowledge
forgetting in other tasks due to continual training. Similarly, we start with
an initial threshold $\tau_{K}=0.5$ and later study the effects of gradually
increasing $\tau_{K}$. The final DPO dataset, $D^{*}$ is constructed from the
dataset filtered for consistency by excluding samples where $S_{K}<\tau_{K}$
or $S_{L}>\tau_{L}$.. The full filtering procedure is demonstrated in
Algorithm 1.

DPO Rafailov et al. (2024) is a variant of Reinforcement Learning (RL), that
allows learning an optimal policy without the need to optimize an external
reward function. This simplifies the training by fitting the optimal policy
$\pi_{\theta}$ from a fixed preference dataset.

Algorithm 1 Knowledge and Consistency Filtering

Input: $D_{DPO}$, $\tau_{L}$, $\tau_{K}$
Output: Filtered dataset, $D^{*}$

1: $D^{*}\leftarrow\emptyset$

2: for $x_{i},y^{*}_{c},Y_{c},Y_{r}$ in $D_{DPO}$ do

3: $S_{L}$ = $\frac{1}{K}\sum_{y_{c}\in Y_{C}}S_{C}(y_{c}^{*},y_{c})$ in ( 1)

4: if $S_{L}$ < $\tau_{L}$ then

5: $S_{K}$ = $\frac{1}{K}\sum_{y_{r}\in Y_{R}}S_{C}(y_{c}^{*},y_{r})$ in ( 2)

6: if $S_{K}$ > $\tau_{K}$ then

7: $y_{w}\leftarrow y_{c}^{*}$

8: $y_{l}\leftarrow\operatorname*{argmax}_{y_{r}\in
Y_{r}}S_{C}(y_{c}^{*},y_{r})$

9: $D^{*}\leftarrow D^{*}\cup\\{x_{i},y_{w},y_{l}\\}$

10: end if

11: end if

12: end for

13: return $D^{*}$

$\delta_{c}=\log\frac{\pi_{\theta}(y_{w}\mid x)}{\pi_{\text{ref}}(y_{w}\mid
x)}$ (3) $\delta_{r}=\log\frac{\pi_{\theta}(y_{l}\mid
x)}{\pi_{\text{ref}}(y_{l}\mid x)}$ (4)
$L_{DPO}(\pi_{\theta};\pi_{\text{ref}})=-\mathbb{E}_{(x,y_{w},y_{l})\sim
D}[\log\sigma(\beta(\delta_{c}-\delta_{r}))]$ (5)

$\beta$ is the regularization operator, while $\pi_{ref}$ is the reference
policy, initialized from $G_{SFT}$. $y_{w}$ and $y_{l}$ are the chosen and
rejected candidates, where $y_{w}$ is preferred over $y_{l}$. The probability
distribution of this preference, $p(y_{w}\succ y_{l})$ follows the Bradley-
Terry model Bradley and Terry (1952), where the latent reward function,
$r^{*}$ is assumed to be implicitly represented in the preference dataset.

$p(y_{w}\succ y_{l})=\sigma(r^{*}(x,y_{w})-r^{*}(x,y_{l}))$ (6)

In this work, we always set $y_{c}^{*}$ as $y_{w}$ while $y_{l}$ is selected
among the K rejected samples $Y_{r}$ in Sec. 3.3. We select the sample with
the highest contradiction score as $y_{l}$.

$y_{l}=\operatorname*{argmax}_{y_{r}\in Y_{r}}S_{C}(y_{c}^{*},y_{r})$ (7)

Based on the above formulation, we observe that performing consistency
filtering encourages $\delta_{c}$ to push the target model in the right
direction, while knowledge filtering pertains to $\delta_{c}-\delta_{r}$.

##  4 Experiments

Through the following experiments, we would like to answer the following
research questions:

  1. RQ1:

Are LLMs capable of performing self-training to improve truthfulness in
responses?

  2. RQ2:

How does conducting selective training improve truthfulness in LLMs and what
are the effects of forgetting on out-of-distribution tasks?

  3. RQ3:

How sensitive is the knowledge filtering threshold, $\tau_{K}$ with respect to
mitigating hallucinations?

###  4.1 Dataset

Train: The training dataset used is constructed using OpenAI’s GPT-3.5, we
generate 8 questions per document after chunking, for 100 documents from each
of the 10 topics. After de-duplication, we end up with 5,780 instructions to
conduct self-training. The instructions are used in constructing both the SFT
and DPO datasets.

Test: The primary test dataset comprises the held-out questions curated from
the target topics discussed in Sec. 3.1, with 10 documents for each of the 10
topics, we refer to this as Wiki-Test. We construct 2 questions from each
document, resulting in a total of 200 questions, generated using GPT-4 Achiam
et al. (2023). We manually check the questions to ensure they are aligned with
the documents. We also conducted experiments on the Open LLM
leaderboard333https://huggingface.co/open-llm-leaderboard consisting of
various NLP benchmarks that are likely not to be directly included in the
model’s training set. The purpose of these evaluations is to detect signs of
forgetting across tasks such as commonsense reasoning and general knowledge
understanding when the model is fine-tuned on data of different distributions.

###  4.2 Model

We conduct the experiments on pre-trained LLMs of different sizes, i.e.,
Tinyllama-1.1B Zhang et al. (2024), Llama2-7B and 13B Touvron et al. (2023),
to study the effect of parameter scaling on the ability to conduct effective
self-training. We choose DeBERTa-v3-large He et al. (2021) as the encoder to
compute $S_{C}$, which is pre-trained on MNLI Williams et al. (2017). We
compare our proposed approach of self-training, which performs the two stages
of filtering, with both $\tau_{L}$ and $\tau_{K}$ set to 0.5 against several
baselines. The first baseline, denoted as w/o filtering, does not perform both
filtering stages and trains the model on the full DPO dataset instead of
$D^{*}$. In this case, this refers to only performing steps 7 to 9 in
Algorithm 1. The second baseline uses GPT-3.5 to generate the chosen response
instead of the model itself, also without any filtering steps. The document is
not provided in the prompt to see if the raw knowledge of GPT-3.5 is
sufficient as a learning signal, similar to performing context distillation on
the target LLM. Lastly, we compare against an inference-type baseline, DOLA
Chuang et al. (2023), which has been shown to be effective in eliciting
truthful responses from LLM.

###  4.3 Experiment details

Figure 2: Win-Tie-Lose on main held-out questions based on Wikipedia
documents. Left pertains to TinyLlama-1.1B, middle to Llama2-7B and right
refers to 13B. Scores are evaluated based on pairwise comparison using GPT-4
as the evaluator and all approaches are compared against the respective SFT
model.

We use a learning rate of 2e-5 and 1e-5 during SFT and 1e-6 and 5e-7 for DPO
for the 1B and 7/13B variants respectively. We conduct early stopping only
during SFT and fix the total training step to be 300 for DPO, to standardize
the number of training steps across datasets of varying sizes. We exploited a
batch size of 32 and set $\beta$ to be $0.3$. Temperature was set to $1.0$ for
the sampling operation and $K=10$ responses were sampled. The primary test set
metric is LLM-Judge Zheng et al. (2024), which uses GPT-4 to conduct pairwise
ranking. Responses from both models are provided along with the document from
which the question was constructed, and GPT-4 is prompted to compare the
responses based on their truthfulness with respect to the document. On Open
LLM leaderboard, we use accuracy as the evaluation metric.

##  5 Results

To compare different approaches with pairwise evaluation, we set the SFT-ed
model, $G_{SFT}$ as the baseline and compare all approaches against it. We run
the evaluation twice for each instance, concluding with a tie if both
evaluations disagree, similar to Yuan et al. (2024).

###  5.1 Impact of Self-training on Truthfulness

RQ1 Effects of self-training on truthfulness: On Wiki-Test, we can see in
Figure 2 that it is possible for LLMs to self-train on their own outputs,
without the need for human-annotated data. This capability extends even to
models with significantly fewer parameters, such as the 1.1B parameter model.
Notably, no few-shot examples are included in the prompt when constructing the
preference dataset. Context distillation underperforms compared to self-
training for Tinyllama but achieves a higher win rate for the 7B and 13B
models, albeit suffering a higher loss rate. We hypothesize that this
discrepancy can be partly attributed to inaccuracies in GPT-3.5’s knowledge
base, which in turn causes instability in the distillation process.
Conversely, when the document is provided as context to the model, it
encourages more truthful responses, thereby correcting any errors in its
previously learned knowledge.

RQ2 Benefits of filtering: The results showed that performing selective
training on the filtered dataset produced superior results compared to
training on the entire dataset, except the 13B model. However, we will show in
later experiments that the gap can be reduced by tuning the scoring threshold,
$\tau_{K}$. Nonetheless, this resonates with our initial belief that having a
preference dataset with a larger distance between the preferred and
dispreferred labels can lead to more stable training. This is logical, as not
all samples in a dataset satisfy the property $(y_{w}\succ y_{l})$. Notably,
DOLA fails to achieve any significant improvements across all models, besides
a marginal increase in performance in the 7 and 13B models.

###  5.2 Catastrophic Forgetting

| ARC | HellaSwag | TruthfulQA | Winogrande | MMLU | Average
---|---|---|---|---|---|---
1.1B | Ours | 29.8 | 60 | 36.4 | 58 | 26.2 | 42.1
w/o filtering | 27.6 | 57.4 | 33 | 56.4 | 25.3 | 39.9
SFT | 30.2 | 59.5 | 35.5 | 58.4 | 26.2 | 41.9
7B | Ours | 40.4 | 73.5 | 40.2 | 68.4 | 43.8 | 53.3
w/o filtering | 38.4 | 71.2 | 37.2 | 66.1 | 41.9 | 50.9
SFT | 40.2 | 72.1 | 41.4 | 67.9 | 43.8 | 53.1
13B | Ours | 44 | 76.4 | 36.9 | 72.2 | 53.2 | 56.6
w/o filtering | 42.9 | 74.3 | 34.9 | 71 | 51.1 | 54.9
SFT | 43.8 | 75.2 | 37.2 | 72.5 | 53.2 | 56.4
Table 1: Performance of the three models on the Open LLM leaderboard. All
tasks are performed 0-shot except MMLU, using 5-shot. Displayed results are
the accuracy metric.

One natural concern regarding fine-tuning is the impact on out-of-distribution
benchmarks. More specifically, we want to see if continual training on
instances where the model is sufficiently knowledgeable, can induce
catastrophic forgetting effects. We use $G_{SFT}$ as the baseline and compare
the performance after DPO with and without filtering on the preference
dataset. To do so, we conducted evaluations on two benchmarks, Open LLM
leaderboard, and a dataset consisting of instructions filtered out from
$D^{*}$. The first benchmark assesses LLMs on commonsense reasoning, general
knowledge, and sentence completion. The second set refers to the samples that
are labeled as known and were thus left out in $D^{*}$ after filtering.
Ideally, this experiment seeks to study if $G_{SFT}$, after doing SFT on its
own outputs, will encounter any deterioration in its knowledge after doing
preference tuning on instances where it was deemed to be knowledgable.. Due to
the high cost of evaluating the full dataset, we randomly sample 200
instances, similar to the primary test set. The known dataset is filtered
using a value of $0.5$ for $\tau_{K}$.

RQ2 Effects of filtering on knowledge retention: Based on Table 1, we observe
that performing knowledge filtering retains the performance of the model on
out-of-distribution tasks. Performing preference tuning on the full dataset
conversely suffers a performance degradation despite being exposed to a more
diverse dataset. This is particularly true for TruthfulQA, which may be less
surprising given the results in Figure 2. Likewise in Table 3, performing
knowledge filtering is shown to suffer a lower losing rate as compared to
without. This is surprising since the evaluation is conducted on samples where
preference tuning was conducted in the case of w/o filtering. This finding
supports our initial belief that over-training on known instances can have
adverse effects on the model.

Figure 3: Percentage of losing rate on 200 randomly sampled instances
classified as known. All approaches are compared against $\pi_{SFT}$.

###  5.3 Varying Filtering Theshold

Figure 4: Effects of varying $\tau_{K}$ on the win rate. Dashed lines shows
the results without performing knowledge filtering for each model.

In the previous experiments, we fixed the knowledge filtering threshold
$\tau_{K}=0.5$. However, this value may not be the most optimal value across
different models. A more capable model should theoretically require a higher
threshold to distinguish between a known and unknown sample. This is because a
more capable model is likely to exhibit lesser variance between generating a
response based on its existing knowledge and when exposed to relevant
materials. We repeat the experiments from Sec. 5.1 while varying $\tau_{K}$.

RQ3 Effects of $\tau_{K}$: Figure 4 shows that increasing the threshold
generally results in higher win rates. We observe a steeper slope in larger
models such as the 13B model while the 1.1B models exhibits a less pronounced
effect. This yields a surprising finding: despite shrinking the dataset as
$\tau_{K}$ increases, the model does not overfit when trained for a higher
number of iterations over a smaller set of data. A plausible hypothesis is
that increasing $\tau_{K}$ allows us to identify critical instances where the
model would fail with just SFT. By implementing DPO on these samples, the
model achieves more pronounced benefits compared to samples where $G_{SFT}$
may already have an acceptable level of knowledge. Another reason could be the
noise in identifying unknown instances; the standard value used in previous
experiments may have caused a higher number of false positives. The statistics
on the size of $D^{*}$ is shown in Table 3.

###  5.4 Ablation: Preference Labeling without Context

Previously, the preferred response in the preference dataset was constructed
by providing the relevant document as supporting context. However, we would
like to see if LLMs can generate a training set with sufficient distance
between the preferred and dispreferred response to yield a stable training
process. In this scenario, to construct $D_{DPO}$, we exclude the document in
the input prompt from Sec. 3.3 and generate a single set of $K$ responses,
$Y=f^{K}(G_{SFT},x_{i})$. We treat each response as the reference response in
place of $y_{c}^{*}$ in Equation 2 and compute the averaged contradiction
score against the other responses in the set. We then select the response with
the minimal score as the preferred response, $y_{w}$, and the response with
the maximum score as the dispreferred response, $y_{l}$, for DPO in Equation
5. This approach tunes the model towards the most consistent response and away
from the least consistent response.

| Win | Lose
---|---|---
1.1B | w document | 54 | 16
w/o document | 35.2 | 22.8
7B | w document | 40.4 | 9.9
w/o document | 25.4 | 22.8
13B | w document | 36.9 | 12
w/o document | 27.5 | 28.5
Table 2: Ablation results comparing between constructing preference dataset
with and without document as context, on Wiki-Test.

Based on Table 2, including the document as a reference results in substantial
improvement in teaching the model to be more truthful. This effect is
particularly pronounced in larger models, where Llama-13B has a higher losing
rate than winning rate. One explanation is that larger models tend to be more
calibrated and thus exhibit lesser variance between the sampled paths, making
it harder to optimize for the margin when implementing DPO. Providing the
document allows for new insights in cases when the model may hallucinate when
relying on its knowledge. Nonetheless, both the 1.1B and 7B models still
produce positive results when they can only rely on their learned knowledge.
Additionally, the assessment did not include optimizing for the optimal
threshold, $\tau_{K}$ which may yield more favorable results, as observed in
Sec. 5.3. We perform additional studies on the effects of varying $K$ in Sec.
A.3.

##  6 Conclusion

In this work, we present a cost-effective approach to guiding LLMs to perform
self-training on their own output. We develop a framework that minimizes human
intervention and demonstrates that LLMs can self-correct errors through
preference-tuning. Specifically, our framework facilitates the creation of a
high-quality preference dataset by excluding low signal-to-noise ratio samples
using a knowledge detection technique. Our experiments illustrate the dual
benefits of our approach: enhancing the truthfulness of LLMs and promoting
knowledge retention post-training. Moreover, self-training offers significant
incentives such as maintaining data privacy, which is crucial for
organizations hesitant to expose sensitive information to third-party
platforms for dataset generation.

This work opens multiple avenues for future research. Given that the context
is built upon publicly accessible material that may have been exposed to the
model during pre-training, an intriguing direction would be to investigate its
impact on specialized domains such as healthcare reports or financial
statements, where human-labeled data is often scarce and private.
Additionally, our current results are based on a single iteration. Future work
could explore the potential for continual improvement by augmenting the
preference dataset with new context through successive iterations.

##  7 Limitations

Firstly, our work is constrained to a single iteration of our self-training
framework due to limited resources for generating additional materials for
continual preference tuning. Another limitation is the scope of subjects, as
our experiments are restricted to ten specified topics collected on a single
platform. In the future, we plan to extend our framework on materials which
can be collected across multiple platforms, including news reports, recent
research papers or data from third-party sources. This could potentially yield
greater improvements since the model is unlikely to be exposed to such
information.

## References

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * Alpaca (2023) Alpaca. 2023.  Introducing alpaca: A strong and performant instruction-following language model.  Accessed: 2024-06-10.
  * Bradley and Terry (1952) Ralph Allan Bradley and Milton E Terry. 1952.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  _Biometrika_ , 39(3/4):324–345.
  * Cambria et al. (2023) Erik Cambria, Rui Mao, Melvin Chen, Zhaoxia Wang, and Seng-Beng Ho. 2023.  Seven pillars for the future of artificial intelligence.  _IEEE Intelligent Systems_ , 38(6):62–69.
  * Cheng et al. (2024) Qinyuan Cheng, Tianxiang Sun, Xiangyang Liu, Wenwei Zhang, Zhangyue Yin, Shimin Li, Linyang Li, Kai Chen, and Xipeng Qiu. 2024.  Can ai assistants know what they don’t know?  _arXiv preprint arXiv:2401.13275_.
  * Chern et al. (2023) I Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, Pengfei Liu, et al. 2023.  Factool: Factuality detection in generative ai–a tool augmented framework for multi-task and multi-domain scenarios.  _arXiv preprint arXiv:2307.13528_.
  * Chuang et al. (2023) Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. 2023.  Dola: Decoding by contrasting layers improves factuality in large language models.  _arXiv preprint arXiv:2309.03883_.
  * Dong et al. (2024) Qingxiu Dong, Jingjing Xu, Lingpeng Kong, Zhifang Sui, and Lei Li. 2024.  Statistical knowledge assessment for large language models.  _Advances in Neural Information Processing Systems_ , 36.
  * Ferdinan et al. (2024) Teddy Ferdinan, Jan Kocoń, and Przemysław Kazienko. 2024.  Into the unknown: Self-learning large language models.
  * He et al. (2019) Junxian He, Jiatao Gu, Jiajun Shen, and Marc’Aurelio Ranzato. 2019.  Revisiting self-training for neural sequence generation.  _arXiv preprint arXiv:1909.13788_.
  * He et al. (2021) Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021.  Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing.  _arXiv preprint arXiv:2111.09543_.
  * Hernandez et al. (2023) Evan Hernandez, Belinda Z Li, and Jacob Andreas. 2023.  Inspecting and editing knowledge representations in language models.  _arXiv preprint arXiv:2304.00740_.
  * Hsieh et al. (2023) Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023.  Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes.  _arXiv preprint arXiv:2305.02301_.
  * Huang et al. (2022) Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022.  Large language models can self-improve.  _arXiv preprint arXiv:2210.11610_.
  * Huang et al. (2023) Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023.  A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.  _arXiv preprint arXiv:2311.05232_.
  * Jang et al. (2021) Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, and Minjoon Seo. 2021.  Towards continual knowledge learning of language models.  _arXiv preprint arXiv:2110.03215_.
  * Jie et al. (2024) Yeo Wei Jie, Ranjan Satapathy, and Erik Cambria. 2024.  Plausible extractive rationalization through semi-supervised entailment signal.  _arXiv preprint arXiv:2402.08479_.
  * Ke et al. (2023) Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. 2023.  Continual pre-training of language models.  _arXiv preprint arXiv:2302.03241_.
  * Krishna et al. (2024) Satyapriya Krishna, Jiaqi Ma, Dylan Slack, Asma Ghandeharioun, Sameer Singh, and Himabindu Lakkaraju. 2024.  Post hoc explanations of language models can improve language models.  _Advances in Neural Information Processing Systems_ , 36.
  * Li et al. (2022) Shiyang Li, Jianshu Chen, Yelong Shen, Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong Wang, Jing Qian, Baolin Peng, Yi Mao, et al. 2022.  Explanations from large language models make small reasoners better.  _arXiv preprint arXiv:2210.06726_.
  * Liang et al. (2024) Yuxin Liang, Zhuoyang Song, Hao Wang, and Jiaxing Zhang. 2024.  Learning to trust your feelings: Leveraging self-awareness in llms for hallucination mitigation.  _arXiv preprint arXiv:2401.15449_.
  * Llama-3 (2024) Llama-3. 2024.  Meta llama 3.  Accessed: 2024-06-10.
  * Luo et al. (2023) Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023.  An empirical study of catastrophic forgetting in large language models during continual fine-tuning.  _arXiv preprint arXiv:2308.08747_.
  * Manakul et al. (2023) Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023.  Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models.  _arXiv preprint arXiv:2303.08896_.
  * Meng et al. (2022a) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022a.  Locating and editing factual associations in gpt.  _Advances in Neural Information Processing Systems_ , 35:17359–17372.
  * Meng et al. (2022b) Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. 2022b.  Generating training data with language models: Towards zero-shot language understanding.  _Advances in Neural Information Processing Systems_ , 35:462–477.
  * Min et al. (2023) Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023.  Factscore: Fine-grained atomic evaluation of factual precision in long form text generation.  _arXiv preprint arXiv:2305.14251_.
  * Mukherjee et al. (2023) Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023.  Orca: Progressive learning from complex explanation traces of gpt-4.  _arXiv preprint arXiv:2306.02707_.
  * Pal et al. (2024) Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. 2024.  Smaug: Fixing failure modes of preference optimisation with dpo-positive.  _arXiv preprint arXiv:2402.13228_.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019.  Language models as knowledge bases?  _arXiv preprint arXiv:1909.01066_.
  * Rafailov et al. (2024) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024.  Direct preference optimization: Your language model is secretly a reward model.  _Advances in Neural Information Processing Systems_ , 36.
  * Tian et al. (2023) Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. 2023.  Fine-tuning language models for factuality.  _arXiv preprint arXiv:2311.08401_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Tunstall et al. (2023) Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. 2023.  Zephyr: Direct distillation of lm alignment.  _arXiv preprint arXiv:2310.16944_.
  * Wang et al. (2023) Weixuan Wang, Barry Haddow, Alexandra Birch, and Wei Peng. 2023.  Assessing the reliability of large language model knowledge.  _arXiv preprint arXiv:2310.09820_.
  * Wang et al. (2022) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022.  Self-instruct: Aligning language models with self-generated instructions.  _arXiv preprint arXiv:2212.10560_.
  * Williams et al. (2017) Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017.  A broad-coverage challenge corpus for sentence understanding through inference.  _arXiv preprint arXiv:1704.05426_.
  * Yuan et al. (2024) Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024.  Self-rewarding language models.  _arXiv preprint arXiv:2401.10020_.
  * Zhang et al. (2024) Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024.  Tinyllama: An open-source small language model.  _arXiv preprint arXiv:2401.02385_.
  * Zheng et al. (2024) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _Advances in Neural Information Processing Systems_ , 36.

##  Appendix A Appendix

###  A.1 Details on Data Generation

In step one of Figure 1, we generate instructions from a text segment of the
selected document from Wikipedia. Both GPT-3.5 and GPT-4 are prompted with the
format from Table 4. We ensure that the instruction does not explicitly
mention the document since at test time, the model is not given any reference
material. We do so by asking $G_{Instr}$ to provide a straightforward
instruction and omit instructions on instances when it failed to do so after
several tries. This results in a yield rate of $72.5\%$. More efficient
methods can be used in the future to improve the yield rate without
sacrificing too much cost in API usage or manual inspection. The second half
of Table 4 contains the prompt used to generate responses which includes the
relevant document as additional context. This includes both curating the label
for $D_{RC}$ and generating $y_{c}^{*}$ for $D_{DPO}$.

###  A.2 Dataset Statistics

The selective training framework performs a two-stage filtering process. In
general, consistency filtering does not affect the original dataset much for
larger models. We find that due to the difference in probability calibration
between models of different sizes, a value of $0.5$ may not be suitable across
all models. The dataset sizes are shown in Table 3.

An example of a generated instruction corresponding to the document is in
Table 4. To understand why knowledge filter is crucial for performing DPO, we
can observe from the example in Table 6. We show two instances, one labeled as
unknown and another as known. There is a visible difference between the two
responses in unknown, where the dispreferred response incorrectly names
"Criminal" as the lead single, while the preferred response correctly names
the title "Shameika". The instability in implementing DPO arises when the
preferred and dispreferred response contains marginal differences as shown in
the known example. This creates a noisy signal for the model since the
dispreferred response is an acceptable answer to the instruction, causing the
model to inevitably lower the probabilities of the correct sequence.

| $\tau_{L}$ | $\tau_{K}$
---|---|---
| 0.5 | 0.5 | 0.6 | 0.7 | 0.8
1.1B | 5182 | 4172 | 3459 | 2742 | 2035
7B | 5740 | 2379 | 1834 | 1401 | 1053
13B | 5754 | 2234 | 1708 | 1277 | 946
Table 3: Statistics on the size of the $D^{*}$ after knowledge filtering by
varying the value of $\tau_{K}$. The original dataset size without filtering
is 5780.

We notice a trend where a larger model tends to prune a higher number of
samples during knowledge filtering. This is expected as larger models tend to
produce responses that are more truthful and thus, fewer hallucinated cases
can be identified.

###  A.3 Approximating knowledge detection by varying $K$

Previously, we approximated the indicator of the knowledge presence of a model
by averaging across the contradiction score over a set of sampled responses,
controlled by the sampling parameter, $K$. One straightforward simplification
is to directly use the greedy decoded response without providing the reference
context, $c$ to get $y_{r}^{*}=f^{*}(G_{SFT},x)$. We can then derive the
knowledge score, by computing the contradiction score between the two greedy
decoded responses, $y_{c}^{*}$ and $y_{r}^{*}$.

Figure 5: Impact of varying $K$ to approximate the average contradiction
score. The value of $K$ affects the number of responses used to compute both
$S_{L}$ and $S_{K}$.

However, comparing against a single response may generate an inaccurate
estimate of knowledge being present in the model. In Figure 5, using a single
sample results in a lower win rate, and using more than 5 allows for a better
estimate. However, this affects larger models to a lesser extent, similar to
previous findings in Sec. 5.4 where larger models tend to be more consistent
between the sampled outputs. The results show that the standard value of
$K=10$ is generally acceptable and a lower value such as $K=5$ is sufficient
in scenarios where computational resources are limited, without sacrificing
too much on the performance.

Instruction generation |  Instruction: Propose a single question regarding the topic of {topic}, whose corresponding answer can be found in the given document.
The question must be detailed and objective, whose corresponding answer should
be non-debatable and be found in the given document.
The proposed question must not mention the existence of the document, but
should mention the topic, {topic}.
Document: {document}
Proposed question:
---|---
Reading Comprehension |  Read the document provided and use the relevant information to answer the question carefully.
It is important that you must not explicitly mention the document’s existence,
while ensuring that your response is factual and relevant according to the
document.
Ensure your answer is well-structured according to the question.
Document: {document}
Question: {instruction}
Answer:
Table 4: Prompts used for instruction generation and eliciting responses from the instruct-tuned model, $G_{SFT}$ when exposed to the document. Document |  Willard Carroll Smith II (born September 25, 1968) is an American actor, rapper and film producer. In 2008, Smith was reported to be developing a film entitled The Last Pharaoh, in which he would be starring as Taharqa. Smith later starred in the superhero movie Hancock, which grossed $227,946,274 in the United States and Canada and had a worldwide total of $624,386,746. In 2013, Smith starred in After Earth with his son Jaden. The film was a disappointment at the domestic box office and was panned critically. Calling the film "the most painful failure in my career", Smith ended up taking a year and a half break as a result.
---|---
Generated instruction |  What film did Will Smith star in with his son Jaden in 2013, and how was it received?
Table 5: Example of a document in the topic of "Actor" and a generated
instruction from GPT-3.5. Unknown
---
Instruction |  What was the lead single from Fiona Apple’s fifth studio album, "Fetch the Bolt Cutters," that won the Grammy Award for Best Rock Performance in 2021?
Preferred response |  The lead single from Fiona Apple’s fifth studio album, "Fetch the Bolt Cutters," that won the Grammy Award for Best Rock Performance in 2021 was "Shameika."
Dispreferred response |  The lead single from Fiona Apple’s fifth studio album, "Fetch the Bolt Cutters", titled "Criminal", won the Grammy Award for Best Rock Performance in 2021. The song features a catchy melody and lyrics about being arrested and charged with a crime. It has become one of Apple’s most popular tracks and is often cited as her best work.
Known
Instruction |  What film did Will Smith star in with his son Jaden in 2013, and how was it received?
Preferred response |  In 2013, Will Smith starred in the film "After Earth" alongside his son Jaden. The film was a commercial disappointment at the domestic box office and received negative reviews from critics. Smith himself described the film as "the most painful failure in my career," leading him to take a break from acting for over a year and a half following its release.
Dispreferred response |  Will Smith and his son Jaden Smith starred together in the science fiction action film "After Earth" (2013), directed by M. Night Shyamalan. The film received mixed reviews, with critics praising the visual effects but criticizing the lack of character development and predictable plot. Despite this, the movie grossed over $270 million worldwide.
Table 6: Example from samples classified as known and unknown. Generations
from the SFT model, Llama2-7B.


# Gradients of O-information: low-order descriptors of high-order dependencies

T. Scagliarini  Dipartimento Interateneo di Fisica, Universitá degli Studi
Aldo Moro, Bari and INFN, Italy  D. Nuzzi  Dipartimento Interateneo di Fisica,
Universitá degli Studi Aldo Moro, Bari and INFN, Italy  Y. Antonacci
Dipartimento di Ingegneria, Universitá di Palermo, Italy  L. Faes
Dipartimento di Ingegneria, Universitá di Palermo, Italy  F.E. Rosas  Data
Science Institute, Imperial College London, United Kingdom  Centre for
Psychedelic Research, Department of Brain Science, Imperial College London,
United Kingdom  Centre for Complexity Science, Imperial College London, United
Kingdom  Center for Eudaimonia and Human Flourishing, University of Oxford,
United Kingdom  Department of Informatics, University of Sussex, United
Kingdom  D. Marinazzo  Department of Data Analysis, Ghent University, Belgium
S. Stramaglia  Dipartimento Interateneo di Fisica, Universitá degli Studi Aldo
Moro, Bari and INFN, Italy  Center of Innovative Technologies for Signal
Detection and Processing (TIRES), Universitá degli Studi Aldo Moro, Italy

###### Abstract

O-information is an information-theoretic metric that captures the overall
balance between redundant and synergistic information shared by groups of
three or more variables. To complement the global assessment provided by this
metric, here we propose the gradients of the O-information as low-order
descriptors that can characterise how high-order effects are localised across
a system of interest. We illustrate the capabilities of the proposed framework
by revealing the role of specific spins in Ising models with frustration, and
on practical data analysis on US macroeconomic data. Our theoretical and
empirical analyses demonstrate the potential of these gradients to highlight
the contribution of variables in forming high-order informational circuits.

Network Science [1], a field encompassing approaches where complex systems are
represented by graphs, has grown tremendously in the last twenty years thanks
to the development of powerful computational techniques to tease
interdependencies out of data [2, 3]. However, despite the great success of
this endeavor, some important questions about complex systems cannot be
properly addressed by dyadic representations, requiring to take into account
higher-order interactions involving more than two elements. Such approaches
typically represent systems as hypergraphs that can be studied via topological
data analysis [4] to reveal the structure of complex systems of interest.

A complementary line of research focuses on emergent properties related to
what the system does, and characterize its high-order behavior from observed
data identifying an equivalent to hyperedges from the dynamics of the data
[5]. A prominent role in this literature is played by the framework of partial
information decomposition [6] and its subsequent developments [7], which
exploit information-theoretic tools to evidence high-order dependencies in
groups of three or more variables — and the description of their synergistic
or redundant nature. In this context, redundancy corresponds to information
which can be retrieved independently from more than one source, while synergy
correspond to statistical relationships that exist in the whole but cannot be
seen in the parts [8]: see, e.g., Ref. [9] for an application of these
principles in neuroscience. Another popular computational tool is the
O-information $\Omega$ [10], which captures the overall balance between
redundant and synergistic high-order dependencies in complex systems [11, 12],
whereas a positive (negative) $\Omega$ means that the multiplet of variables
at hand is dominated by redundant (synergistic) dependencies. The
computational complexity of $\Omega$ calculation scales more gracefully with
the number of variables than PID, making it particularly well-suited for
practical data analysis.

Crucially, the quest for high-order descriptions of complex systems comes with
important computational and conceptual costs, as these representations often
grow super-exponentially with the system size. Moreover, while coarse-grained
measures such as $\Omega$ exist, their global nature fails short not being
able to provide a local description of how high-order phenomena are
distributed across systems of interest. Hence, there is a urgent need of
intermediate approaches that can enable a compact yet meaningful
representation of informational multiplets. Indeed, the success of Network
Science rests partially on the availability of metrics that quantify the role
of specific nodes or links in the system — metrics which are not as immediate
to develop and grasp for high-order analyses, in particular when the high-
order links are statistical dependencies.

Here we address this problem by introducing a novel approach providing low-
order (i.e. univariate and pairwise) descriptors of high-order dependencies in
the system. The proposed approach is based on the gradients of the
O-information: instead of focusing on the O-information of groups of
variables, we focus on the variation of the O-information when variables are
added to the rest of the system to form these groups. This provides a more
nuanced descriptions of synergistic or redundant informational circuits, in
which the role of each variable can be disambiguated. This new framework is
operationalised by means of the definitions and derivations presented below.

The O-information $\Omega$ of a complex system described by $n$ stochastic
variables $\bm{X}^{n}=(X_{1},\dots,X_{n})$, see [10], can be calculated as:

$\Omega(\bm{X}^{n})=(n-2)H(\bm{X}^{n})+\sum_{i=1}^{n}\Big{[}H(X_{i})-H(\bm{X}^{n}_{-i})\Big{]},$
(1)

where $H$ is the Shannon entropy. In order to assess the contribution of a
given variable $X_{i}$ to the informational circuit contained in $\bm{X}^{n}$,
we propose to calculate its “gradient of O-information” given by

$\displaystyle\partial_{i}\Omega(\bm{X}^{n})=$
$\displaystyle\Omega(\bm{X}^{n})-\Omega(\bm{X}^{n}_{-i})$ $\displaystyle=$
$\displaystyle(2-n)I(X_{i};\bm{X}^{n}_{-i})+\sum_{k=1,k\neq
i}^{n}I(X_{k};\bm{X}^{n}_{-ik}),$ (2)

where $\bm{X}^{n}_{-i}$ denotes the set of all the variables in $\bm{X}^{n}$
but $X_{i}$, and $I$ is the mutual information 111Correspondingly,
$\bm{X}^{n}_{-ik}$ corresponds to all the variables in $\bm{X}^{n}$ except
$X_{i}$ and $X_{k}$.. The quantity $\partial_{i}\Omega(\bm{X}^{n})$ captures
how much the O-information changes because of adding $X_{i}$, hence giving an
account of how this variable contributes to the high-order properties of the
system. Correspondingly, $\partial_{i}\Omega(\bm{X}^{n})>0$ means that $X_{i}$
introduces mainly redundant information, while
$\partial_{i}\Omega(\bm{X}^{n})<0$ indicates that it fosters synergistic
interdependencies.

A direct calculation shows that the following bounds hold and are tight:

$-(n-2)\log|\mathscr{X}|\leq\partial_{i}\Omega(\bm{X}^{n})\leq\log|\mathscr{X}|,$
(3)

where $|\mathscr{X}|$ is the cardinality of the largest alphabet in
$\bm{X}^{n}$ (a proof of this can be found in the supplementary material). The
asymmetry between the two bounds has an important consequence: while
redundancy can be only build step by step, synergy can be established more
rapidly.

Following a similar rationale to the one that lead to Eq. (2), one can further
introduce a second-order descriptor of high-order interdependencies by
considering gradients of gradients. In particular, the second-order gradient
of a pair of variables $X_{i}$ and $X_{j}$ can be defined as

$\displaystyle\partial_{j}\partial_{i}\Omega(\bm{X}^{n})=\partial_{i}\Omega(\bm{X}^{n})-\partial_{i}\Omega(\bm{X}^{n}_{-j}).$
(4)

This second-order gradient captures how much the presence of the variable
$X_{j}$ alters the variation of O-information of the system due to the
inclusion of $X_{i}$. It is direct to verify the symmetry
$\partial_{i}\partial_{j}\Omega(\bm{X}^{n})=\partial_{j}\partial_{i}\Omega(\bm{X}^{n})$;
therefore, we simply denote this quantity as
$\partial^{2}_{ij}\Omega(\bm{X}^{n})$.

An interesting property of $\partial^{2}_{ij}\Omega(\bm{X}^{n})$ is that it
can be re-written as a ‘whole-minus-sum’ property:

$\displaystyle\partial^{2}_{ij}\Omega(\bm{X}^{n})$
$\displaystyle=\big{[}\Omega(\bm{X}^{n})-\Omega(\bm{X}^{n}_{-ij})\big{]}$
$\displaystyle-\big{[}$
$\displaystyle\Omega(\bm{X}^{n}_{-i})-\Omega(\bm{X}^{n}_{-ij})\big{]}-\big{[}\Omega(\bm{X}^{n}_{-j})-\Omega(\bm{X}^{n}_{-ij})\big{]}~{}.$
(5)

In other words, $\partial^{2}_{ij}\Omega(\bm{X}^{n})$ measures to what degree
the variation to the O-information due to the inclusions of both $X_{i}$ and
$X_{j}$ is more than the sum of the variations one obtains when including them
separately. Consequently, $\partial^{2}_{ij}\Omega(\bm{X}^{n})$ vanishes if
variables $X_{i}$ and $X_{j}$ are part of independent informational circuits.

The second-order gradient $\partial^{2}_{ij}\Omega(\bm{X}^{n})$ can be
compared with the local O-information between the variables $X_{i}$ and
$X_{j}$ (introduced in [10]) $I\left(X_{i};X_{j};\bm{X}^{n}_{-ij}\right)$,
which corresponds to the interaction information [14] between $X_{i}$, $X_{j}$
and variables $\bm{X}^{n}_{-ij}$. Interestingly, for $n=3$ the local
O-information and $\partial^{2}_{ij}\Omega(\bm{X}^{n})$ coincide, while for
$n\geq 4$ they generally differ. A key difference between these quantities is
that the local O-information treats the rest of the system (i.e.
$\bm{X}^{n}_{-ij}$) as a whole, whilst in the value of the former is actually
dependent on the specific partition that divides $\bm{X}^{n}_{-ij}$ into
parts, which gives it more sensitivity to evaluate informational circuits.

Successive gradients follow automatically, resulting in a simple chain rule.
If $\gamma$ is a subset of $\\{1,\ldots,n\\}$ of cardinality $|\gamma|$, then:

$\displaystyle\partial^{|\gamma|}_{\gamma}\Omega(\bm{X}^{n})$
$\displaystyle=\sum_{\alpha\subseteq\gamma}(-1)^{|\alpha|}\Omega(\bm{X}^{n}_{-\alpha}),$
(6)

the sum being over all the subsets $\alpha$ of $\gamma$. For example, for
triplets of variables the gradient of the O-information reads:

$\displaystyle\partial^{3}_{ijk}\Omega(\bm{X}^{n})$
$\displaystyle=\Omega(\bm{X}^{n})-\Omega(\bm{X}^{n}_{-i})-\Omega(\bm{X}^{n}_{-j})-\Omega(\bm{X}^{n}_{-k})$
$\displaystyle+$
$\displaystyle\Omega(\bm{X}^{n}_{-ij})+\Omega(\bm{X}^{n}_{-ik})+\Omega(\bm{X}^{n}_{-jk})-\Omega(\bm{X}^{n}_{-ijk}),$
(7)

and measures the irreducible contribution to the O-information by the triplet
$\\{i,j,k\\}$ which cannot be ascribed to the inclusion of pairs nor single
variables of the triplet. The potential of interpreting these quantities in a
topological manner, as is has been done with the entropy [15], is an
interesting avenue for future research.

Figure 1: Top left: the hexagonal geometry of the Ising model, where
continuous and dashed lines indicate ferromagnetic and anti-ferromagnetic
interactions, respectively. Top right: the gradients w.r.t. single spins
$\partial_{i}\Omega(\bm{s}^{7})$ are plotted versus $\beta$. For symmetry, the
curves of peripheral spins 2-7 are equal. Bottom left: the second order
gradients $\partial^{2}_{ij}\Omega(\bm{s}^{7})$ are plotted versus $\beta$.
Due to symmetry, only five non equivalent curves are plotted. Bottom right:
the local O-information $I(s_{i};s_{j};\bm{s}^{7}_{-ij})$ is plotted for the
same pairs of spins.

To illustrate the power of the proposed tools, let’s start considering a Ising
model with Hamiltonian given by

$\mathscr{H}(\bm{s}^{n})=-\sum_{i\neq j}J_{ij}s_{i}s_{j}.$ (8)

Our analysis considers a case where $n=7$, with couplings $J_{ij}=\pm 1$ as
depicted in Fig.1 (top-left panel). Informational measures can be computed
directly from the probability of a spin configuration
$\bm{s}^{7}=(s_{1},s_{2},\ldots,s_{7})$, which is given by

$p(\bm{s}^{7})=\frac{e^{-\beta\mathscr{H}(\bm{s}^{7})}}{Z}~{},$ (9)

where $Z$ is the partition function

$Z=\sum_{s_{1},\cdots,s_{7}}e^{-\beta\mathscr{H}(\bm{s}^{7})},$ (10)

and $\beta$ is the inverse temperature. An evaluation of
$\partial{i}\Omega(\bm{s}^{7})$ shows that the contribution of peripheral
spins is dominated by redundancy, whilst the central spin (the one that, when
added to the rest of the system, introduces frustration) introduces
synergistic dependencies — with the synergy peaking at a a finite temperature.
These findings confirms the relationship between synergy and frustration in
spin systems already noticed in [16], while explaining which elements are most
responsible for it. Additionally, an analysis using second-order gradients
show that all the pairwise descriptors are associated with redundancy when the
temperature is low, while for high temperatures they are synergistic for pairs
consisting of the central spin and a peripheral spin, as well as for the pairs
consisting of two neighboring spins on the periphery. Overall, these finding
add an important spatial description to the previously reported [17]
relationship between synergy and higher temperature systems. We note that
these findings cannot be retrieved by applications of the local O-information
on the same system (Fig.1, bottom-right panel).

As an econometric application, let’s consider $14$ US macroeconomic time
series taken from the Federal Reserve Economic Dataset (FRED) [18]. We
consider quarterly indicators over a period of 61 years (Apr 1959- Jan 2020)
for a total of $244$ observations: paid compensation of employees (COE),
consumer price index (CPIAUCSL), effective federal funds rate (FEDFUNDS),
government consumption expenditures and investment (GCE), gross domestic
product (GDP), gross domestic product price deflator (GDPDEF), gross private
domestic investment (GPDI), ten-year treasury bond yield (GS10), non-farm
business sector index of hours worked (HOANBS), M1 money supply (narrow money
M1SL), M2 money supply (broad money M2SL), personal consumption expenditures
(PCEC), three-month treasury bill yield (TB3MS), and unemployment rate
(UNRATE). A wide literature (see e.g. [19, 20]) have tried leveraging similar
data to address the fundamental question regarding the source of economic
fluctuations. Here we are not interested in the role played by shocks and
frictions neither in predicting business cycles; rather, our goal is to
evidence high-order dependencies in macro-indicators of the US economy. In
order to deal with stationary time-series, the proposed approach has been
applied to the logarithmic returns of the series, over which the gradients of
the O-information were calculated using the Gaussian Copula approach described
in Ref. [21]. For each gradient, significance testing is performed via
bootstrap sampling with replacement: if the $95\%$ confidence interval of that
gradient (here computed on $1000$ realizations) does not contain zero, the
gradient is declared significant. Our results show that seven indicators are
redundant with the rest of the system (see Table Gradients of O-information:
low-order descriptors of high-order dependencies) — which is consistent with
the prevalence of redundancy in real-world multivariate systems, as reflected
by latent factors being typically associated to positive O-information (see
also Ref. [22]). In contrast, GPDI was found to play a major synergistic role
with respect to the rest of the system, which may be associated with the fact
that GPDI is considered a good predictor of the productive capacity of the
economy.

[tabular=lc, table head=, table foot=, no head, late after line=
, late after first line=
,]indicatori_macroeconomici.csv

Table 1: Gradients of O-information for US macroeconomic indicators (only
statistically significant values).

Figure 2:  Left: second order gradients for pairs of economic indicators.
Right: local O-information of pairs of economic indicators. Edge values are
encoded by color (sign) and width (absolute value). Only statistically
significant edges — calculated via bootstrap resampling — are included.

These first-order analysis can be enriched by the second-order gradient.
Results show that several pairs of variables are involved in informational
circuits, as shown in Fig. 2: whilst first-order analysis shows a prevalence
of redundancy, second order gradients show a prevalence of synergy. The most
connected node for synergy is GDP, displaying significantly negative
$\partial^{2}_{ij}\Omega$ with four other nodes. When compared with the local
O-information, the proposed pairwise descriptors leads to a more sparse and
parsimonuous pattern.

Summarizing, in this work we have introduced the gradients of O-information as
measures of how much a variable — or a pair of variables — are functionally
connected with the rest of a network through redundant and synergistic
informational circuits. The use of this novel metric, together with measures
of pairwise dependencies and global measures of higher-order information like
the regular O-information, provides a more complete description of the
informational character of dynamics in complex systems. By analyzing the
dynamical interactions between US macroeconomics indicators we have shown how
these new tools are capable of revealing high-order informational circuits,
which evidenced the synergistic role of GPDI — it is matter for further
research to verify if existing econometric models are able to reproduce this
high order behavior exhibited by data. The extension of these novel measures
of high-order behavior to dynamical systems scenarios is also in order, as
that would open new avenues for investigating the function of complex
networked systems on a wide range of applications.

## References

  * [1] A. Barabasi and M. Pósfai, Network science.  Cambridge: Cambridge University Press, 2016.

* [2] S. Boccaletti, V. Latora, Y. Moreno, M. Chavez, and D.-U. Hwang, “Complex networks: Structure and dynamics,” Physics Reports, vol. 424, no. 4, pp. 175–308, 2006.
* [3] F. Battiston, G. Cencetti, I. Iacopini, V. Latora, M. Lucas, A. Patania, J.-G. Young, and G. Petri, “Networks beyond pairwise interactions: Structure and dynamics,” Physics Reports, vol. 874, pp. 1–92, 2020.
* [4] F. Battiston, E. Amico, A. Barrat, G. Bianconi, G. Ferraz de Arruda, B. Franceschiello, I. Iacopini, S. Kéfi, V. Latora, Y. Moreno, M. Murray, T. Peixoto, F. Vaccarino, and G. Petri, “The physics of higher-order interactions in complex systems,” Nat. Phys., vol. 17, pp. 1093–1098, 2021\.
* [5] F. Rosas, P. Mediano, A. Luppi, T. Varley, J. Lizier, S. Stramaglia, H. Jensen, and D. Marinazzo, “Disentangling high-order mechanisms and high-order behaviours in complex systems,” Nat. Phys., vol. 18, pp. 476–477, 2022\.
* [6] P. L. Williams and R. D. Beer, “Nonnegative decomposition of multivariate information,” 2010.
* [7] J. T. Lizier, N. Bertschinger, J. Jost, and M. Wibral, “Information decomposition of target effects from multi-source interactions: Perspectives on previous, current and future work,” Entropy, vol. 20, no. 4, 2018.
* [8] M. Wibral, V. Priesemann, J. W. Kay, J. T. Lizier, and W. A. Phillips, “Partial information decomposition as a unified approach to the specification of neural goal functions,” Brain and cognition, vol. 112, pp. 25–38, 2017.  Publisher: Elsevier.
* [9] A. Luppi, P. Mediano, F. Rosas, N. Holland, T. Fryer, J. O’Brien, J. Rowe, D. Menon, D. Bor, and E. Stamatakis, “A synergistic core for human brain evolution and cognition.,” Nat. Neurosci., 2022.
* [10] F. E. Rosas, P. A. Mediano, M. Gastpar, and H. J. Jensen, “Quantifying high-order interdependencies via multivariate extensions of the mutual information,” Physical Review E, vol. 100, no. 3, p. 032305, 2019.  Publisher: APS.
* [11] M. Gatica, R. Cofré, P. A. Mediano, F. E. Rosas, P. Orio, I. Diez, S. P. Swinnen, and J. M. Cortes, “High-order interdependencies in the aging brain,” Brain connectivity, vol. 11, 2021.
* [12] S. Stramaglia, T. Scagliarini, B. C. Daniels, and D. Marinazzo, “Quantifying dynamical high-order interdependencies from the o-information: an application to neural spiking dynamics,” Frontiers in Physiology, vol. 11, p. 1784, 2021.  Publisher: Frontiers.
* [13] Correspondingly, $\bm{X}^{n}_{-ik}$ corresponds to all the variables in $\bm{X}^{n}$ except $X_{i}$ and $X_{k}$.
* [14] W. McGill, “Multivariate information transmission,” Psychometrika, vol. 19, pp. 97–116, 1954.
* [15] P. Baudot, M. Tapia, D. Bennequin, and J.-M. Goaillard, “Topological information data analysis,” Entropy, vol. 21, no. 9, p. 869, 2019.
* [16] T. Scagliarini, D. Marinazzo, Y. Guo, S. Stramaglia, and F. E. Rosas, “Quantifying high-order interdependencies on individual patterns via the local o-information: Theory and applications to music analysis,” Phys. Rev. Research, vol. 4, p. 013184, Mar 2022.
* [17] D. Marinazzo, L. Angelini, M. Pellicoro, and S. Stramaglia, “Synergy as a warning sign of transitions: The case of the two-dimensional ising model,” Phys. Rev. E, vol. 99, p. 040101, Apr 2019.
* [18] FRED, “Federal reserve economic dataset.” https://fred.stlouisfed.org/, 2022 (accessed May 5, 2022).
* [19] F. Smets and R. Wouters, “Shocks and frictions in us business cycles: A bayesian dsge approach.,” European Central Bank, Working Paper Series, vol. 722, 2007.
* [20] A. Justiniano, G. E. Primiceri, and A. Tambalotti, “Investment shocks and business cycles,” Journal of Monetary Economics, vol. 57, no. 2, pp. 132–145, 2010.
* [21] R. A. Ince, B. L. Giordano, C. Kayser, G. A. Rousselet, J. Gross, and P. G. Schyns, “A statistical framework for neuroimaging data analysis based on mutual information estimated via a gaussian copula,” Human Brain Mapping, vol. 38, no. 3, pp. 1541–1573, 2017.
* [22] D. Marinazzo, J. Van Roozendaal, F. E. Rosas, M. Stella, R. Comolatti, N. Colenbier, S. Stramaglia, and Y. Rosseel, “An information-theoretic approach to hypergraph psychometrics,” arXiv preprint arXiv:2205.01035, 2022.

Supplementary material

##  I Bounds for first order gradients of the O-information

Here we present the proof of the bounds in equation (3) in the main
manuscript. Let us consider $n$ random variables
$\bm{X}^{n}=(X_{1},X_{2},\ldots,X_{n})$. Two popular extensions of the mutual
information are the total correlation $\text{TC}(\bm{X}^{n})$ and the dual
total correlation $\text{DTC}(\bm{X}^{n})$

$\text{TC}(\bm{X}^{n})=\sum_{i=1}^{n}H(X_{i})-H(\bm{X}^{n});\qquad\qquad\text{DTC}(\bm{X}^{n})=H(\bm{X}^{n})-\sum_{i=1}^{n}H(X_{i}\mid\bm{X}^{n}_{-i})$
(11)

where $H(\cdot)$ is the Shannon entropy. The O-information $\Omega$ of the
system is given by the difference:

$\displaystyle\Omega(\bm{X}^{n})$
$\displaystyle\equiv\text{TC}(\bm{X}^{n})-\text{DTC}(\bm{X}^{n})=(n-2)H(\bm{X}^{n})+\sum_{k=1}^{n}\Big{[}H(X_{j})-H(\bm{X}^{n}_{-j})\Big{]}.$
(12)

The gradient of the O-information (see the main manuscript) is given by:

$\displaystyle\partial_{i}\Omega(\bm{X}^{n})$
$\displaystyle=\Omega(\bm{X}^{n})-\Omega(\bm{X}^{n}_{-i})=(2-n)I(X_{i};\bm{X}^{n}_{-i})+\sum_{k=1}^{n-1}I(X_{k};\bm{X}^{n}_{-ik}).$
(13)

Analogously we can consider the gradient of the total correlation

$\displaystyle\partial_{i}\text{TC}$
$\displaystyle=\text{TC}(\bm{X}^{n})-\text{TC}(\bm{X}^{n}_{-i})=I(X_{i};\bm{X}_{-i}^{n})\geq
0,$ (14)

which, being equivalent to a mutual information, satisfies
$0\leq\partial_{i}\text{TC}\leq\log|\mathscr{X}|$, where $|\mathscr{X}|$ is
the cardinality of the largest alphabet in $\bm{X}^{n}$. On the other hand the
gradient of the dual total correlation can be written as a sum of conditional
mutual information terms:

$\displaystyle\partial_{i}\text{DTC}(\bm{X}^{n})$
$\displaystyle=\text{DTC}(\bm{X}^{n})-\text{DTC}(\bm{X}^{n}_{-i})$
$\displaystyle=\sum_{k=1}^{n-1}\Big{[}H(\bm{X}_{-i}^{n})-H(\bm{X}^{n}_{-ki})-H(\bm{X}^{n})+H(\bm{X}_{-k}^{n})\Big{]}$
$\displaystyle=\sum_{k=1}^{n-1}\Big{[}I(X_{i};\bm{X}_{-i}^{n})-I(X_{i};\bm{X}_{-ki}^{n})\Big{]}$
$\displaystyle=\sum_{k=1}^{n-1}\Big{[}H(X_{k}|\bm{X}^{n}_{-jk})-H(X_{k}|\bm{X}^{n}_{-k})\Big{]}$
(15) $\displaystyle=\sum_{k=1}^{n-1}I(X_{k};X_{j}|\bm{X}^{n}_{-jk})\geq 0,$
(16)

thus implying that
$0\leq\partial_{i}\text{DTC}(\bm{X}^{n})\leq(n-1)\log|\mathscr{X}|$.

Putting these results together one can find that

$-(n-2)\log|\mathscr{X}|\leq\partial_{i}\Omega(\bm{X}^{n})\leq\log|\mathscr{X}|.$
(17)

The reminding of the proof demonstrates these bounds and their tightness.

The upper bound is trivial, indeed we have already shown that
$\partial_{i}\text{DTC}(\bm{X}^{n})\geq 0$, hence:

$\partial_{i}\Omega(\bm{X}^{n})=\partial_{i}\text{TC}(\bm{X}^{n})-\partial_{i}\text{DTC}(\bm{X}^{n})\leq\partial_{i}\text{TC}(\bm{X})\leq\log|\mathscr{X}|.$
(18)

The tightness of the upper bound can be proven by showing that is achieved by
the $n$-COPY gate, specifically by taking $X_{1}$ as a Bernoulli variable with
$p=1/2$ and $X_{1}=X_{2}=\dots=X_{n}$. Since we have that
$I(X_{i};\bm{X}^{n}_{-i})=1$ and $I(X_{i};\bm{X}^{n}_{-ik})=1$ for
$i=1,2,\ldots,n$, using Equation (13) it follows that

$\partial_{i}\Omega(\bm{X})=(2-n)+(n-1)=1,\qquad i=1,2,\ldots,n.$ (19)

This covers the case of binary random variables, but the result can be readily
generalized to $|\mathscr{X}|>2$.

To prove the lower bound is a little more tricky: we start noting that
$\partial_{i}\Omega(\bm{X}^{n})$ can be written as a sum of conditional
interaction information terms. Indeed it has been shown in [10] that
O-information can be decomposed as a sum of interaction information terms

$\Omega(\bm{X}^{n})=\sum_{k=2}^{n-1}I(X_{k};\bm{Y}^{k-1}_{1};\bm{Y}_{k+1}^{n}),$
(20)

where we used the notation $\bm{Y}^{q}_{k}=(X_{k},\dots,X_{q})$. For
definiteness we fix $i=1$, obtaining

$\displaystyle\partial_{1}\Omega(\bm{X}^{n})$
$\displaystyle=\Omega(\bm{X}^{n})-\Omega(\bm{X}^{n}_{-1})=\sum_{k=2}^{n-1}I(X_{k};\bm{Y}^{k-1}_{1};\bm{Y}_{k+1}^{n})-\sum_{k=3}^{n-1}I(X_{k};\bm{Y}^{k-1}_{2};\bm{Y}_{k+1}^{n})$
$\displaystyle=I(X_{2};X_{1};\bm{Y}_{3}^{n})+\sum_{k=3}^{n-1}\left[I(X_{k};\bm{Y}^{k-1}_{1};\bm{Y}_{k+1}^{n})-I(X_{k};\bm{Y}^{k-1}_{2};\bm{Y}_{k+1}^{n})\right]$
$\displaystyle=\sum_{k=2}^{n-1}I(X_{k};X_{1};\bm{Y}_{k+1}^{n}|\bm{Y}_{2}^{k-1}).$
(21)

Now we notice that each term in the sum can be written as a difference of two
conditional mutual information terms (bounded between $0$ and
$\log|\mathscr{X}|$), hence each term has the following bounds

$-\log|\mathscr{X}|\leq
I(X_{k};X_{1};\bm{Y}_{k+1}^{n}|\bm{Y}_{2}^{k-1})\leq\log|\mathscr{X}|.$ (22)

This implies that

$-(n-2)\log_{2}|\mathscr{X}|\leq\partial_{1}\Omega(\bm{X}^{n}),$ (23)

thus proving the lower bound. Finally, to prove that the lower bound is tight,
we consider the $n$-XOR gate, that is $X_{1}\dots X_{n-1}$ as Bernoulli random
variables with $p=1/2$ and $X_{n}=(\sum_{j=1}^{n-1}X_{j})\ \text{mod}\ 2$.
Using Equation (13) we have $I(X_{i};\bm{X}_{-i}^{n})=1$ and
$I(X_{k};\bm{X}_{-ik}^{n})=0$ then

$\partial_{i}\Omega=(2-n),\qquad i=1,2,\ldots,n$ (24)

Then the lower bound is tight, since is achieved by the variables composing a
$n$-XOR gate.

##  II O-information of US macroeconomic indicators: triplets and quadruplets

Concerning the US economic data set, we report here the conventional
O-information analysis, taking into account triplets and quadruplets of
variables. We first obtain all the triplets which are significantly
synergistic and those which are significantly redundant. Then, for each
variable, we sum $\Omega$ over all the redundant and significant triplets
which contain that variable, obtaining $R_{\Omega}$ which is an index of
redundancy of that variable. The same is done summing over synergistic
triplets, thus leading to an index of synergy of that variable $R_{\Omega}$ :
the results are shown in figure 3top-left, where these indexes are compared
with the first order gradient as found by the proposed approach. Analogously,
for each pair of variables we sum over the triplets containing that pair, and
obtain a synergy index and a redundancy index for all pairs of variables,
depicted in figure3top-middle and compared with second order gradients. In
figure3top-right the distribution of $\Omega$ for all the significant
triplets.

In the second row of figure3 the same quantities are calculated using
quadruplets: we stress that no synergistic quadruplet is found to be
statistically significant.

These results show that, as far as the redundancy is concerned, the proposed
approach leads to a pruning of redundant pairs of variables w.r.t. the index
$R_{\Omega}$, as shown by the vertical cloud of red points around
$\partial_{ij}^{2}\Omega=0$. Moreover, the redundant pattern of $R_{\Omega}$
is quite stable going from triplets to quadruplets. On the other hand, as far
as the synergy is concerned, $\partial_{i}\Omega$ seems unrelated to
$S_{\Omega}$ calculated on triplets.

Figure 3: The first row depicts the redundancy index $R_{\Omega}$ and the
synergy index $S_{\Omega}$ in the univariate (left) and pairwise (middle)
case, see the text, plotted as a function the first order gradients and second
order gradients, respectively, of the corresponding variable or pair of
variables; in the right we depict the distribution of the O-information values
of all the significant triplets. In the second row the same analysis has been
shown for the significant O-information quadruplets. Red and blue dots
indicates $R_{\Omega}$ and $S_{\Omega}$, respectively.