Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

OM-2023: The 18th International Workshop on Ontology Matching, November 7th,
2023, Athens, Greece

<EMAIL_ADDRESS>] [orcid=0000-0001-5307-9857,
<EMAIL_ADDRESS>]

<EMAIL_ADDRESS>] [orcid=0000-0003-1707-4842<EMAIL_ADDRESS>]

[orcid=0000-0002-1990-7740<EMAIL_ADDRESS>]

[orcid=0000-0002-6780-3217<EMAIL_ADDRESS>]

[1]Work done while at IBM Research.

# Matching Table Metadata with Business Glossaries Using Large Language Models

Elita Lobo  Oktie Hassanzadeh  Nhan Pham  Nandana Mihindukulasooriya
Dharmashankar Subramanian  Horst Samulowitz  IBM Research, Yorktown Heights,
NY, United States  University of Massachusetts Amherst, MA, United States

(2023)

###### Abstract

Enterprises often own large collections of structured data in the form of
large databases or an enterprise data lake. Such data collections come with
limited metadata and strict access policies that could limit access to the
data contents and, therefore, limit the application of classic retrieval and
analysis solutions. As a result, there is a need for solutions that can
effectively utilize the available metadata. In this paper, we study the
problem of matching table metadata to a business glossary containing data
labels and descriptions. The resulting matching enables the use of an
available or curated business glossary for retrieval and analysis without or
before requesting access to the data contents. One solution to this problem is
to use manually-defined rules or similarity measures on column names and
glossary descriptions (or their vector embeddings) to find the closest match.
However, such approaches need to be tuned through manual labeling and cannot
handle many business glossaries that contain a combination of simple as well
as complex and long descriptions. In this work, we leverage the power of large
language models (LLMs) to design generic matching methods that do not require
manual tuning and can identify complex relations between column names and
glossaries. We propose methods that utilize LLMs in two ways: a) by generating
additional context for column names that can aid with matching b) by using
LLMs to directly infer if there is a relation between column names and
glossary descriptions. Our preliminary experimental results show the
effectiveness of our proposed methods.

##  1 Introduction

Large collections of structured tabular data that businesses possess can be
invaluable resources for various analytic tasks. Traditionally, such data
collections are gathered in large databases or data warehouses, along with
mechanisms of collecting and maintaining metadata with well-curated schemas,
data catalogs, or master data as a part of a master data management solution.
In practice, the overhead of maintaining accurate metadata may be
prohibitively difficult and expensive. More recently, enterprises are moving
toward collecting all their data in data lakes without any requirements or
strict enforcement of metadata availability or quality. As a result, there is
a need for solutions that can effectively use limited metadata, such as column
headers, and automatically generate useful metadata. Most organizations
maintain some business glossary [1] with a set of concepts that are relevant
to the business processes. If table columns can be annotated with business
glossary terms, it helps downstream tasks such as data discovery, data
integration, or performing advanced analytics.

The task of mapping table columns to a business glossary is similar to the
task of annotating a table column with an ontology or knowledge graph concept,
which is referred to as the Column Type Annotation (CTA) task [2]. However, to
our knowledge, prior work has not considered further restricting the task to
using only the table metadata (table name and column headers) and business
glossary containing labels and descriptions only. The problem we study in this
paper is inspired by our ongoing work on implementing an automated semantic
layer for enterprise data lakes [3], and has the following characteristics: 1)
we do not have access to data contents due to access restrictions common in
enterprise data lakes; 2) we have tabular data with no metadata other than
column headers, which is a result of large data imports from highly
heterogeneous sources or automated table extraction pipelines; and 3) there is
no or very little training data, as the process of manually labeling table
columns with business glossary terms is laborious and requires domain
expertise. Figure 1 shows a few example column headers along with their
context (other column headers in the same table) and their associated business
glossary terms.

Figure 1: Example column headers and their associated business glossary terms

In the absence of rich metadata and an ontology, the matching process can only
rely on the header labels and glossary labels and descriptions. Essentially,
the problem becomes a string or text similarity matching problem. Prior work
has studied various flavors of such matching methods for record matching in
databases [4, 5] as well as ontology alignment [6]. Such methods rely on
either syntactic matching methods, which rely on common tokens and substrings
between the terms that should be matched, or semantic matching methods, which
rely on the availability of a dictionary of terms along with lists of related
terms such as synonyms, hypernyms, and hyponyms. In our setup, we often need
to match terms with very little syntactic similarity, and we do not have
access to a dictionary that could enable semantic matching. Column headers in
tabular data are often cryptic terms, and business glossaries use terminology
very specific to a particular enterprise. More recent work has proposed the
use of vector representations of terms in the form of embeddings [7]; however,
such methods require domain-specific training data and tuning.

In this paper, we propose a novel matching solution that relies on the power
of Large Language Models (LLMs) to enable the matching of table columns with
glossaries when column headers are not very descriptive, and glossary terms do
not have a close syntactic similarity to the column headers, and little or no
training data is available. In what follows, we first discuss related work. We
then present the problem description, and then we present the details of our
solution. In section 5, we present the results of our experiments using real-
world enterprise data and business glossaries. We end the paper by outlining a
number of lessons learned and avenues for future work.

##  2 Related Work

A core problem in semantic table understanding [8] is column type annotation,
i.e., annotating table columns with a type from an ontology, which enables
many business intelligence tasks such as semantic retrieval, data exploration,
and knowledge discovery. SemTab challenge [2], which aims at benchmarking
systems dealing with the tabular data to KG matching problem, provides several
datasets in which the column type annotation task can be evaluated. In the
SemTab challenge, this task is formulated as an unsupervised task where
participating systems are not given training data.

##### Column type annotations using table data

MTab [9], JenTab [10], and DAGOBAH [11], are an example of the systems that
participated in the SemTab challenge comprising three KG matching tasks,
namely, cell to KG entity (CEA task), column to KG class (CTA task), and
column pair to KG property (CPA task). As these systems typically solve the
three tasks in a joint manner, they follow a pipeline architecture. The first
step links cell mentions to entities within the target ontology. The second
step predicts the most likely type for the query column based on the linking
results. MTab and DAGOBAH also use additional information from the graph, such
as entity relations, to improve cell linking accuracy. It is a requirement for
these systems to have cell values that can be mapped to KG entities, which
might not be the case in most industry tables.

##### Column type annotations with only using metadata

The problem setup that is studied in this paper differs from the traditional
column type annotation task. In our setup, the system that is performing the
matching between the table column and glossary concepts will only have access
to the table metadata (i.e., table name and column headers) but not the actual
table data (i.e., cell values). This problem setup has similarities with the
ontology matching methods that rely purely on string similarity measures.
String similarity measures have been studied extensively for various matching
tasks, including in ontology alignment [6]. Syntactic measures of similarity
measure how close two strings are based on measuring the overlap between
tokens or substrings in two strings or measures based on the number of
character edit operations (e.g., removal or replacement) that can transform
one string to another. Examples of such methods are edit similarity and Jaro-
Winkler [12]. While such approaches have shown very promising performance in
various matching tasks, they are inherently not capable of differentiating
between strings that are syntactically very similar but semantically
dissimilar. Classic semantic measures rely on resources such as WordNet [13]
containing related terms. The application of those methods is limited to when
such resources are available. More recently, methods that rely on vector
representations for semantic similarity have shown superior performance in
various tasks. Initial approaches relied on word2vec [14], which can handle
semantic matching between words and short phrases. More recently, word
embeddings [15, 16, 17, 18, 19] and sentence embeddings [7] have shown
promising performance in semantic textual similarity tasks. As we will show in
our experiments, business glossaries often have very similar labels and
descriptions that these sentence transformer-based approaches alone cannot
effectively differentiate between.

##  3 Preliminaries

###  3.1 Problem Setup

We assume a setting in which we have only superficial tabular metadata
corresponding to any chosen table in the form of a list of $n$ superficial
metadata fields, say $M=\\{(M_{i})_{i=1}^{n}\\}$. In practice, this list could
contain a main column name of interest that must be matched with the correct
business concept in a glossary, along with the other column names in the table
under question. We also assume access to a business glossary that consists of
$m$ glossary items $\mathcal{G}=\\{(l_{j},d_{j})\\}_{j=1}^{m}$. Here, $l_{j}$
and $d_{j}\forall j\in[1,m]$ represent respectively the _label_ and
_description_ of the $j^{th}$ glossary item. For example, the glossary could
be a list of tuples containing labels and descriptions of various business
concepts. Given such superficial metadata $M$, the task is to find its closest
glossary item match.

In this paper, we consider a relaxed version of the glossary matching problem,
where the task is to select $k$ glossary items for any given metadata $M$ such
that it maximizes the probability of Hit@k for any given $M$. A Hit@k for a
given metadata represents a Boolean variable that takes the value _one_ if the
selected $k$ glossary items contain the closest match of the metadata and
_zero_ otherwise. Finally, we also assume that we have a human feedback bank
available in the form of $l$ tuples
$\mathcal{H}=\\{(M_{k},G_{k})_{k=1}^{l}\\}$ where $M_{k}$ represents some
metadata and $G_{k}\in\mathcal{G}$ represents the correct glossary match. We
will use the human feedback bank $\mathcal{H}$ to construct task
demonstrations for the In-Context Learning approach described in the next
section.

###  3.2 Large Language Models

Recent work [20, 21, 22, 23, 24] has demonstrated that Large Language Models
(LLMs) perform extraordinarily well on instruction-based tasks as long as
these tasks can be represented in natural language. LLMs are transformer
models with billions of parameters, trained on large data corpora and fine-
tuned on instructions-based tasks, including classification tasks, generation
tasks, and question-answering tasks. An LLM takes as input a prompt containing
the description of the task along with additional context represented using
natural language and outputs the results of the task in natural language. In
this work, we leverage LLMs, specifically, Flan-t5-models [25] to obtain more
accurate metadata for business glossary matching. Since LLMs have been trained
on large data corpora, they can identify complex relations and patterns
between different objects in natural language and, thereby, can be used to
obtain more accurate matching.

###  3.3 In-Context Learning

Fine-tuning LLMs for new tasks or datasets is often very computationally
expensive and requires large amounts of data, which is often not feasible. A
common approach to evade this problem is by appending one (one-shot) or
multiple (multi-shot) demonstrations of the task in natural language to the
prompt. This is commonly known as the One-shot or Multi-shot In-Context
Learning (ICL) or In-Context Prompting method [26]. Figure 2 shows an example
of multi-shot in-context learning on a classification task. In-context
learning is known to have worked well for several new problems in the prior
literature. This work uses the human feedback bank to generate relevant
demonstrations for in-context learning. We conjecture that ICL, with good
demonstrations, can improve the performance of the glossary matching task
without additional fine-tuning.

Figure 2: Example prompt for tuned and pre-tuned Flan-t5 models used in MDG-
MICL (2-shots) method.

##  4 Methodology

We propose two different classes of methods a) Metadata Description to
Glossary Matching (MDGM) b) Direct Metadata to Glossary Matching (DMGM) that
retrieve a set of $k$ glossary items for any given metadata such that it
contains the glossary match with high probability. In MDGM methods, we use the
Large Language Models to obtain a metadata description and use the description
to retrieve $k$ glossary items that are most similar to the given description
in some latent space. On the other hand, in the DMGM methods, we use LLMs to
directly match metadata to business glossaries. More specifically, we treat
the metadata to glossary matching problem as either a Boolean classification
or a Multi-class classification problem. We design special prompts to LLMs to
_directly_ infer which glossary items are potential descriptions of the
metadata and choose the top-$k$ glossary items most likely to be the
description of the given metadata. Although MDGM methods seem like an indirect
approach to glossary matching, they can be useful when the glossary constantly
changes during test time, and direct inference over large glossaries is
expensive. We show in section 5.2 that MDGM methods tend to outperform DMGM
methods.

###  4.1 Column Description to Glossary Matching

We now describe various techniques we propose for generating descriptions of
metadata for the metadata to business glossary matching problem.

####  4.1.1 Metadata Description Generation via Multi-Shot In-Context Learning
(MDG-MICL)

Since LLMs are trained on large data corpora, they can generate good
descriptions of any concept they may have seen during the training period. In
this method, we leverage this knowledge of LLM to generate descriptions of the
given metadata. We design a special prompt instructing the LLM to generate a
metadata description. Further, we use ICL to improve the quality and control
the format of the description generated by the LLM. Figure 2 shows an example
of the ICL prompt used for this task with tuned Flan-t5-xl and Flan-t5-xxl
models. To construct demonstrations for ICL, we proceed as follows. Using the
Sentence-BERT (SBERT) sentence embeddings [7], we first generate sentence
embeddings of the metadata and all the descriptions corresponding to the
glossary items in the human-feedback bank, $\mathcal{H}$. We use the cosine
similarity metric to find $e$ glossary item descriptions from $\mathcal{H}$
closest to the metadata in the SBERT sentence embedding space. We construct
demonstrations from these $e$ glossary items in $\mathcal{H}$, and append them
to the prompt, in response to which the LLM generates a description. We obtain
the final description by appending the table name and metadata to the LLM-
generated description. We embed this description using SBERT and obtain the
top-$k$ glossary items by computing the cosine similarity metric between this
embedding and the sentence embeddings of all the glossary item descriptions.
The procedure of computing top-$k$ glossary items from the glossary set
$\mathcal{G}$ for a given metadata description using the SBERT sentence
embeddings and cosine similarity metric is used in several subsequent methods;
for the sake of brevity, we will here on refer to this procedure as the SBERT
$k-$nearest neighbors method.

####  4.1.2 Metadata Description Generation via Classification (MDG-Cl)

As discussed in section 3, LLMs can also identify complex relations between
various concepts in natural language. Therefore, in this method, we leverage
LLMs to directly select the best description from the set of glossary
descriptions in $\mathcal{G}$ using a classification-based technique.

Specifically, for a given metadata and glossary set $\mathcal{G}$, we
construct a binary classification prompt against each glossary item in the set
$\mathcal{G}$, that queries the LLM on whether the given glossary item is a
potential description of the metadata. Figure 2 shows an example of the
classification prompt used for this task. Note that when the glossary set
$\mathcal{G}$ is too large, it may result in high inference costs. We can
mitigate these high costs by first shortlisting the top $k_{1},(k_{1}>k)$
glossary item from the glossary set $\mathcal{G}$ using the SBERT $k-$nearest
neighbors method before proceeding with the classification prompts.

Figure 3: Example prompt for tuned Flan-t5-xl and pre-tuned Flan-t5-xxl models
used in MDG-Cl method.

The final description of the metadata corresponds to the glossary description
for which the classification response was positive with the highest log
probability score of the classification task, appended to the table name and
the metadata itself. If no such glossary item exists, we use the metadata
itself as the description. Once the metadata description is generated, we
select the top-$k$ glossary items from the glossary set $\mathcal{G}$ using
SBERT $k-$nearest neighbors described in MDG-MICL method.

####  4.1.3 Metadata Description Generation via Multiple Choice Question
Answering (MDG-MCQA)

An alternative way of generating descriptions using LLMs is by using a
Multiple Choice Question Answer (MCQA) prompt, as shown in Figure 4, that
instructs the LLM to choose the best description of the metadata amongst the
descriptions of selected glossary items. Although this may seem
counterintuitive, we observe in our experiments that using the description of
the selected glossary item to find the top-$k$ glossary items instead of
simply returning the glossary item corresponding to the selected description
results in a higher Hit@5 rate. Similar to previous methods, we shortlist the
top $k_{1}(k_{1}>k)$ glossary items that are closest to the metadata in
sentence-embedding space and use them as choices in our Multiple Choice
Question Answer prompt. We also add "None of the above" to the list of choices
in the MCQA prompt. Finally, we append the metadata and the table name to the
description of the LLM-selected glossary item and use it to find the top-$k$
glossary items using the SBERT $k-$nearest neighbors method. Note that when
the LLM selects the “None of the above" option, we use the metadata itself as
the description.

Figure 4: Prompt for Flan-t5-xl and pre-tuned Flan-t5-xxl models used in MDG-
MCQA method.

###  4.2 Direct Metadata to Glossary Matching

####  4.2.1 Direct Inference via Classification (DI-Cl)

This method is a variant of the MDG-Cl method that uses LLM to directly select
the top-$k$ glossary items for any given metadata without needing to generate
a description of the metadata. Similar to previous methods, we shortlist the
top $k_{1}\,(k_{1}>k)$ glossary items closest to the metadata in the sentence-
embedding space. For each of these glossary items, we construct binary
classification prompts that query the LLM on whether the description of the
glossary item matches the metadata. An example prompt is shown in Figure 5.
Among all glossary items with positive responses, the glossary items with
top-$k$ highest log probability scores are selected. It is important to note
that this method may return less than $k$ glossary items. Such missing items
are assumed to be incorrect matches while computing the Hit@k metric.

Figure 5: Example prompt for all pre-tuned and tuned Flan-t5 models used in
DI-Cl method.

####  4.2.2 Direct Inference via Multiple Choice Question Answering (DI-MCQA)

We consider another variation of the MDG-MCQA method that computes a single
best match of the given metadata without generating a description of the
metadata. In this method, we follow the same procedure as MDG-MCQA to
construct Multi-Choice Classification prompts, as shown in Figure 6, and
return the glossary item corresponding to the description selected by the LLM.
Since this method always returns a single glossary item, we will assume that
the $k-1$ missing glossary items are incorrect matches while computing the
Hit@k metric.

Figure 6: Example prompt for Flan-t5-xl and pre-tuned Flan-t5-xxl models used
in DI-MCQA method.

##  5 Experiments

In this section, we empirically evaluate and compare the performance of all
the MDGM and DMGM methods proposed in section 4. Specifically, we investigate
the following two questions a) Which of the proposed methods: MDG-MICL
(0-shot, 1-shot, 2-shots), MDG-Cl, MDG-MCQA, DI-Cl, and DI-MCQA, best leverage
LLM in solving the metadata to glossary matching task, and b) is it possible
to obtain more accurate matching, i.e., higher Hit@5 and Hit@1 using LLMs than
using basic similarity-score based matching methods? Our preliminary results
show that LLMs are indeed effective in improving glossary matching accuracy.

###  5.1 Experimental setup

In all our experiments, the metadata consists of the column name of interest
and the other column names in the table. The glossary is a list of tuples
where each tuple consists of a label and a description of the label. Multiple
column names may be matched to the same label. Furthermore, the label of the
matched glossary item for each column name may not coincide with the column
name. We evaluate our methods on the Flan-T5-XL, Flan-T5-XXL [27, 25], and a
Flan-T5-XL model that we fine-tune on the training dataset using the
supervised fine-tuning method for LLMs [28]. For each method and LLM model, we
experiment with 4-6 different prompt templates. However, due to lack of space,
we only provide examples of the prompt templates that achieved the highest
hit@5 rate (2,3,4,5,6).

We use the all-mpnet-base-v2 SBERT model from the sentence-transformers
library[7] in all experiments. We evaluate each method based on their Hit@5
and Hit@1 rates on the test dataset, i.e., the empirical mean of Hit@5 and
Hit@1 computed on the test dataset. These measures were chosen to reflect our
goal of having the correct glossary item as the top or within the top 5
glossary items returned to the user on a GUI [3]. We compare these scores
against ones produced by a baseline, which computes the top-$k$ glossary items
based on the cosine similarity of the sentence embedding between the column
name and the descriptions in the glossary.

##### MDE Dataset

The MDE Dataset is an IBM-internal benchmark developed by annotating the
column names of the “Customer Insight” example database of “IBM InfoSphere
Warehouse Pack" [29] with theevaluateglossary terms from the IBM Knowledge to
fine-tune Financial Services (IBM KAFS) [30] glossary. The Customer Insight
database consists of 26 tables with 688 columns. The column names contain
cryptic codes and abbreviations to reflect realistic tables commonly seen in
client engagements. The IBM KAFS glossary contains 9,137 business terms with
their labels and descriptions. Out of 688 columns, 488 have suitable matching
terms in the glossary, and the rest are annotated as null mappings and ignored
in the evaluation. We split the mappings into train, test, and demonstration
splits with 208, 212, and 68 columns, respectively. These splits contain
tuples of the form $(column\\_name,other\\_column\\_names,glossary\\_item)$.
The training, test, and demonstration splits are used for fine-tuning the LLM
model, evaluating the method, and as a proxy for human feedback.

###  5.2 Experimental Results

Table 1 shows the Hit@5 and Hit@1 rates achieved by different methods on the
MDE dataset with different LLM models. As expected, the Hit@5 rate increases
with the number of demonstrations in the MDG-MICL method. This result suggests
that it may be beneficial to use in-context learning whenever demonstrations
are readily available. However, we do not observe a similar trend for _Hit@1_
rate. We believe this is due to the difficulty of matching metadata to a
single glossary item when multiple glossary items have similar descriptions.
Overall, MDG-MICL achieves the highest Hit@5 and Hit@1 scores and
significantly outperforms the baseline method when two demonstrations are
provided in the prompt. Meanwhile, we observe that the DI-Cl and DI-MCQA
methods achieve the worst Hit@5 rates. We conjecture that this may be due to
the underlying biases of LLMs towards certain class labels in classification
and question-answering tasks as observed in prior works [31, 32]. These biases
can be corrected using various calibration techniques [31, 32], which we leave
for future work. It is also important to note that the DI-MCQA method selects
a single best glossary match and, thus, more likely fails to select the
correct item when multiple glossary items have similar descriptions. These
preliminary results indicate that LLMs alone may not improve the matching
accuracy. Finally, although MDG-Cl and MDG-MCQA use the same classification
prompt and Multiple Choice Question Answer prompts as DI-Cl and DI-MCQA, they
achieve higher Hit@5 and Hit@1 than the latter methods. We believe that this
is because these methods select glossary items whose descriptions are similar
to the closest glossary match and, thus, tend to perform better.

Methods | Flan-t5-xl | Flan-t5-xxl | Tuned Flan-t5-xl
---|---|---|---
Hit@1 |  Hit@5 |  Hit@1 |  Hit@5 |  Hit@1 |  Hit@5
Baseline Method |  0.4575 |  0.7406 |  0.4575 |  0.7406 |  0.4575 |  0.7406
MDG-MICL (0-shot) |  0.5 |  0.7877 |  0.4858 |  0.7689 |  0.4434 |  0.7594
MDG-MICL (1-shot) |  0.4764 |  0.7972 |  0.4953 |  0.8113 |  0.467 |  0.75
MDG-MICL (2-shots) |  0.5047 |  0.8133 |  0.4858 |  0.816 |  0.5 |  0.7877
MDG-Cl |  0.434 |  0.7453 |  0.5 |  0.7642 |  0.4575 |  0.7547
MDG-MCQA |  0.5708 |  0.7547 |  0.5755 |  0.7453 |  0.5753 |  0.7642
DI-Cl |  0.3585 |  0.4717 |  0.4811 |  0.6934 |  0.3632 |  0.4953
DI-MCQA |  0.5519 |  0.5519 |  0.5708 |  0.5708 |  0.5519 |  0.5519
Table 1:  This table shows the Hit@5 and Hit@1 scores achieved by the
Baseline, MDG-MICL (0,1,2)-shots, MDG-Cl, MDG-MCQA, DI-Cl and DI-MCQA methods
on Flan-T5-XL, Flan-T5-XXL and Tuned Flan-T5-XL models. The highlighted
numbers indicate that the corresponding methods have outperformed the baseline
method in terms of the Hit@5/Hit@1 value. MDG-MICL consistently achieves the
highest Hit@5 and Hit@1, whereas DI-Cl and DI-MCQA have the worst Hit@5.

##  6 Discussion and Future work

This paper proposes two different classes of methods, i.e., MDGM and DMGM,
that leverage LLMs for solving the metadata to glossary matching problem. MDGM
methods use LLMs to generate good metadata descriptions, which we couple with
similarity-based metrics for more refined matches. This class of methods is
necessary when the glossary is likely to change during test time frequently,
and repeated inferences are expensive. The second class of methods (DMGM)
utilizes LLMs to infer which glossary items are potential matches directly.
These methods are helpful when the metadata is too complex, and there is a
significant difference between the description of the metadata and that of its
closest glossary match. Although we have shown that many of these methods can
potentially obtain more accurate glossary matching, we can further improve
them in several ways. Our experiments show that DMGM methods perform poorly
compared to MDGM methods. This may be due to the undesirable biases towards
specific class labels that LLMs learned during training. One approach to
mitigating these biases is using various calibration techniques [31, 32, 33].
Providing several positive and negative demonstrations in the classification
and multiple-choice question-answer prompts may also help mitigate LLMs’
default biases [26]. We can further improve the MDGM methods that generate
descriptions of metadata by constraining LLMs to sample words mainly from the
glossary or providing LLMs with the top-$k$ glossary items and prompting LLMs
to generate descriptions similar to those of the glossary items. This can be
achieved by using the Constrained Beam Search algorithm [34] or simply
manipulating the output distribution of LLMs before sampling such that it
assigns higher weights to words from the glossary. It may also be helpful to
use various prompt-tuning and prompt-editing methods [35] to further improve
the efficiency of the prompts used with LLMs. Although these directions remain
intriguing, they warrant more in-depth empirical study, which we leave for
future work.

## References

  * Sarkar [2015] P. Sarkar, Business Glossary for DaaS, 2015, pp. 103–122.
  * Jiménez-Ruiz et al. [2020] E. Jiménez-Ruiz, O. Hassanzadeh, V. Efthymiou, J. Chen, K. Srinivas,  Semtab 2019: Resources to benchmark tabular data to knowledge graph matching systems,  in: European Semantic Web Conference, Springer, 2020, pp. 514–530.
  * Weidele et al. [2023] D. K. I. Weidele, G. Rossiello, G. Bramble, A. N. Valente, S. Bagchi, M. F. M. Chowdhury, M. Martino, N. Mihindukulasooriya, H. Ananthakrishnan, H. Strobelt, H. Patel, A. M. Gliozzo, O. Cornec, A. Gupta, S. Mehta, M. Kesarwani, H. Samulowitz, O. Hassanzadeh, A. Agarwal, L. Amini, Conversational GUI for Semantic Automation Layer, 2023\. Under Submission to ISWC 2023, Industry Track (will be on arxiv soon).
  * Chandel et al. [2007] A. Chandel, O. Hassanzadeh, N. Koudas, M. Sadoghi, D. Srivastava,  Benchmarking declarative approximate selection predicates,  in: SIGMOD, 2007, pp. 353–364.
  * Hassanzadeh et al. [2007] O. Hassanzadeh, M. Sadoghi, R. J. Miller,  Accuracy of approximate string joins using grams,  in: QDB, 2007.
  * Cheatham and Hitzler [2013] M. Cheatham, P. Hitzler,  String similarity metrics for ontology alignment,  in: The Semantic Web - ISWC 2013 - 12th International Semantic Web Conference, Sydney, NSW, Australia, October 21-25, 2013, Proceedings, Part II, volume 8219, Springer, 2013, pp. 294–309.
  * Reimers and Gurevych [2019] N. Reimers, I. Gurevych,  Sentence-bert: Sentence embeddings using siamese bert-networks,  in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 2019\.
  * Pujara et al. [2021] J. Pujara, P. Szekely, H. Sun, M. Chen,  From tables to knowledge: Recent advances in table understanding,  in: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021, pp. 4060–4061.
  * Nguyen et al. [2021] P. Nguyen, I. Yamada, N. Kertkeidkachorn, R. Ichise, H. Takeda,  Semtab 2021: Tabular data annotation with mtab tool,  in: SemTab at (ISWC 2021), 2021, pp. 92–101.
  * Abdelmageed and Schindler [2021] N. Abdelmageed, S. Schindler,  JenTab meets semtab 2021’s new challenges.,  in: SemTab@ ISWC, 2021.
  * Huynh et al. [2021] V.-P. Huynh, J. Liu, Y. Chabot, F. Deuzé, T. Labbé, P. Monnin, R. Troncy,  DAGOBAH: Table and graph contexts for efficient semantic annotation of tabular data,  in: SemTab@ ISWC, 2021.
  * Winkler [1990] W. E. Winkler,  String comparator metrics and enhanced decision rules in the fellegi-sunter model of record linkage,  in: Proceedings of the Section on Survey Research, 1990.
  * Lin and Sandkuhl [2008] F. Lin, K. Sandkuhl,  A survey of exploiting wordnet in ontology matching,  in: Artificial Intelligence in Theory and Practice II, Springer US, 2008, pp. 341–350.
  * Liao et al. [2021] J. Liao, Y. Huang, H. Wang, M. Li,  Matching ontologies with word2vec model based on cosine similarity,  in: Proceedings of the International Conference on Artificial Intelligence and Computer Vision (AICV2021), Springer International Publishing, 2021.
  * Wang et al. [2018] L. L. Wang, C. Bhagavatula, M. Neumann, K. Lo, C. Wilhelm, W. Ammar,  Ontology alignment in the biomedical domain using entity definitions and context,  in: Proceedings of the BioNLP 2018 workshop, Association for Computational Linguistics, Melbourne, Australia, 2018, pp. 47–55.
  * Chen et al. [2021] J. Chen, E. Jiménez-Ruiz, I. Horrocks, D. Antonyrajah, A. Hadian, J. Lee,  Augmenting ontology alignment by semantic embedding and distant supervision,  in: The Semantic Web, Springer International Publishing, Cham, 2021, pp. 392–408.
  * Zhang et al. [2014] Y. Zhang, X. Wang, S. Lai, S. He, K. Liu, J. Zhao, X. Lv,  Ontology matching with word embeddings,  in: M. Sun, Y. Liu, J. Zhao (Eds.), Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data, Springer International Publishing, 2014\.
  * Kolyvakis et al. [2018] P. Kolyvakis, A. Kalousis, D. Kiritsis,  DeepAlignment: Unsupervised ontology matching with refined word vectors,  in: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), Association for Computational Linguistics, New Orleans, Louisiana, 2018, pp. 787–798.
  * Tounsi Dhouib et al. [2019] M. Tounsi Dhouib, C. Faron Zucker, A. G. B. Tettamanzi,  An ontology alignment approach combining word embedding and the radius measure,  in: Semantic Systems. The Power of AI and Knowledge Graphs, Springer International Publishing, Cham, 2019, pp. 191–197.
  * Ouyang et al. [2022] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. E. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, R. J. Lowe,  Training language models to follow instructions with human feedback,  ArXiv (2022).
  * Wei et al. [2022] J. Wei, M. P. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, Q. V. Le,  Finetuned language models are zero-shot learners,  2022\.
  * Zhou et al. [2023] Y. Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, J. Ba,  Large language models are human-level prompt engineers,  in: ICLR, 2023.
  * Jang et al. [2022] J. Jang, S. Ye, M. Seo,  Can large language models truly follow your instructions?,  in: NeurIPS ML Safety Workshop, 2022\.
  * Zhao et al. [2023] W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, Y. Du, C. Yang, Y. Chen, Z. Chen, J. Jiang, R. Ren, Y. Li, X. Tang, Z. Liu, P. Liu, J. Nie, J. rong Wen,  A survey of large language models,  ArXiv abs/2303.18223 (2023).
  * Longpre et al. [2023] S. Longpre, L. Hou, T. Vu, A. Webson, H. W. Chung, Y. Tay, D. Zhou, Q. V. Le, B. Zoph, J. Wei, et al.,  The flan collection: Designing data and methods for effective instruction tuning,  arXiv preprint arXiv:2301.13688 (2023).
  * Min et al. [2022] S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, L. Zettlemoyer,  Rethinking the role of demonstrations: What makes in-context learning work?,  in: EMNLP, 2022.
  * Chung et al. [2022] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma, et al.,  Scaling instruction-finetuned language models,  arXiv preprint arXiv:2210.11416 (2022).
  * Ouyang et al. [2022] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Gray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, R. Lowe,  Training language models to follow instructions with human feedback,  in: Advances in Neural Information Processing Systems, 2022.
  * Announcement [2010] I. U. S. Announcement, Ibm infosphere warehouse pack for customer insight v8.2, 2010.
  * kaf [2023] IBM knowledge accelerator for financial services, 2023\.
  * Xu et al. [2023] B. Xu, Q. Wang, Z. Mao, Y. Lyu, Q. She, Y. Zhang,  $k$NN prompting: Beyond-context learning with calibration-free nearest neighbor inference,  in: ICLR, 2023.
  * Zhao et al. [2023] T. Zhao, M. Wei, J. S. Preston, H. Poon, Llm calibration and automatic hallucination detection via pareto optimal self-supervision, 2023\. arXiv:2306.16564.
  * Han et al. [2023] Z. Han, Y. Hao, L. Dong, Y. Sun, F. Wei,  Prototypical calibration for few-shot learning of language models,  in: ICLR, 2023.
  * Hokamp and Liu [2017] C. Hokamp, Q. Liu,  Lexically constrained decoding for sequence generation using grid beam search,  in: ACL, 2017, pp. 1535–1546.
  * Zhang et al. [2023] T. Zhang, X. Wang, D. Zhou, D. Schuurmans, J. E. Gonzalez,  TEMPERA: Test-time prompt editing via reinforcement learning,  in: The Eleventh International Conference on Learning Representations, 2023.


2021 ACML 2021

#  Bayesian neural network unit priors
and generalized Weibull-tail property

Mariia Vladimirova<EMAIL_ADDRESS>
Julyan Arbel<EMAIL_ADDRESS>
Stéphane Girard<EMAIL_ADDRESS>
Univ. Grenoble Alpes  Inria  CNRS  LJK  38000 Grenoble  France

###### Abstract

The connection between Bayesian neural networks and Gaussian processes gained
a lot of attention in the last few years. Hidden units are proven to follow a
Gaussian process limit when the layer width tends to infinity. Recent work has
suggested that finite Bayesian neural networks may outperform their infinite
counterparts because they adapt their internal representations flexibly. To
establish solid ground for future research on finite-width neural networks,
our goal is to study the prior induced on hidden units. Our main result is an
accurate description of hidden units tails which shows that unit priors become
heavier-tailed going deeper, thanks to the introduced notion of generalized
Weibull-tail. This finding sheds light on the behavior of hidden units of
finite Bayesian neural networks.

###### keywords:

generalized Weibull-tail, sub-Weibull, Bayesian neural networks

##  1 Introduction

Theoretical insights and the development of a comprehensive theory are often
the driving force underlying the development of new and improved methods.
Neural networks are powerful models, but they still lack comprehensive
theoretical support. The Bayesian approach applied to neural networks is
considered one of the best-suited frameworks to obtain theoretical
explanations and improve the models.

Infinite-width Bayesian neural networks are well-studied. Induced priors in
Bayesian neural networks with Gaussian weights are Gaussian processes when the
number of hidden units per layer tends to infinity (Neal, 1996; Matthews et
al., 2018; Lee et al., 2018; Garriga-Alonso et al., 2019). Stable
distributions also lead to stable processes, which are generalizations of
Gaussian ones (Favaro et al., 2020).

Since in some cases finite models perform better (Lee et al., 2018; Garriga-
Alonso et al., 2019; Lee et al., 2020), there is a need for theoretical
justifications. One of the ways is to study induced priors. By analyzing the
priors over representations, Aitchison (2020) suggests that finite Bayesian
neural networks may generalize better than their infinite counterparts because
of their ability to learn representations. Another idea is to find the induced
priors in the functional space (Wilson and Izmailov, 2020).

By bounding moments of distributions induced on units, Vladimirova et al.
(2019) proved that hidden units have heavier-tailed upper bounds and follow
sub-Weibull distributions with an increasing tail parameter depending on the
layer depth. Those bounds are optimal as they are achieved for shallow
Bayesian neural networks; however, they are not accurate.

Recently, Zavatone-Veth and Pehlevan (2021); Noci et al. (2021) showed that
there exists a precise description of induced unit priors through Meijer
G-functions. These are full descriptions of priors at the unit level. The
results are in accordance with the heavy-tailed nature and asymptotic
expansions in a wide regime, but it has restrictions. First, the setting is
simplified: linear or ReLU activation functions and Gaussian priors on
weights. While Wilson and Izmailov (2020) argue that vague Gaussian priors in
the parameter space induce useful function-space priors, in some cases,
heavier-tailed priors can perform better (Fortuin et al., 2021). Second, it is
hard to work with Meijer G-functions due to their complexity. Our goal is to
obtain more general characterizations for hidden units.

We introduce a new concept for describing distributional properties of tails
by extending the existing Weibull-tail characterization. A random variable $X$
is called Weibull-tail (Gardes et al., 2011) with tail parameter $\beta>0$,
which is denoted by $X\sim\text{WT}_{\mathbb{R}_{+}}(\beta)$, if its
cumulative distribution function $F_{X}$ satisfies

$\overline{F}_{X}(x)=1-F_{X}(x)=\mathrm{e}^{-x^{\beta}l(x)},\,\,\,\text{for
}x>0,$ (1)

where $l(x)$ is a slowly-varying function, i.e. it is a positive function such
that for all $t>0$ $\lim_{x\to\infty}\frac{l(tx)}{l(x)}=1$. We note that the
Weibull-tail property only considers the right tail of distributions. Here we
adapt the Weibull-tail characterization to the whole space $\mathbb{R}$ by
taking into consideration the left tail as well. Additionally, we introduce
generalized Weibull-tail random variables with tail parameter $\beta>0$ which
have Weibull-tail upper and lower bounds for both tails (Definition 2.1). Such
a characterization is easily interpretable and stable under basic operations,
even for dependent random variables. The family of generalized Weibull-tail
distributions covers a large variety of fundamental distributions such as
Gaussian ($\beta=2$), gamma ($\beta=1$), Weibull ($\beta>0$), to name a few,
and turns out to be a key tool to describe distributional tails.

Contributions. We make the following contributions:

  * •

We introduce a new notion of tail characteristics called generalized Weibull-
tail, a version of Weibull-tail characteristics on $\mathbb{R}_{+}$ extended
to variables on $\mathbb{R}$ (Section 2). The additional advantage of this
notion is stability under basic operations such as multiplication by a
constant and summation.

  * •

With the results on generalized Weibull-tail characterization and dependence,
we obtain an accurate characterization of the heavy-tailed nature of hidden
units in Bayesian neural networks (Section 3). We establish these results
under possibly heavy-tailed priors and relatively mild assumptions on the non-
linearity. The conclusions of Vladimirova et al. (2019); Zavatone-Veth and
Pehlevan (2021); Noci et al. (2021) which consider only Gaussian priors,
mostly follow as corollaries of the obtained characterization. The comparison
of different characterizations and related works are deferred to Sections 4
and 5.

##  2 Generalized Weibull-tail random variables

The study of the distributional tail behavior arises in many applied
probability models of different areas, such as hydrology (Strupczewski et al.,
2011), finance (Rachev, 2003) and insurance risk theory (McNeil et al., 2015).
Since exact distributions are not available in most cases, deriving asymptotic
relationships for their tail probabilities becomes essential. In this context,
an important role is played by so-called Weibull-tail distributions satisfying
Equation (1) (Gardes et al., 2011; Gardes and Girard, 2016).

A majority of works focuses on right tails of distributions and develops a
theory only applicable to right tails, while it is essential to study both
right and left tails of distributions. We extend the notion of Weibull-tail on
$\mathbb{R}_{+}$ to $\mathbb{R}$ and introduce generalized Weibull-tail random
variables on $\mathbb{R}$ in the following definition. They are stable (under
basic operations) extensions of Weibull-tail random variables on $\mathbb{R}$
(Appendix B).

######  Definition 2.1 (Generalized Weibull-tail on $\mathbb{R}$).

A random variable $X$ is generalized Weibull-tail on $\mathbb{R}$ with tail
parameter $\beta>0$ if both its right and left tails are upper and lower
bounded by some Weibull-tail functions with tail parameter $\beta$:

$\displaystyle\mathrm{e}^{-x^{\beta}l_{1}^{r}(x)}\leq$
$\displaystyle\overline{F}_{X}(x)\leq\mathrm{e}^{-x^{\beta}l_{2}^{r}(x)},\quad\
\ \ \text{for }x>0\text{ and $x$ large enough},$
$\displaystyle\mathrm{e}^{-|x|^{\beta}l_{1}^{l}(|x|)}\leq$ $\displaystyle
F_{X}(x)\leq\mathrm{e}^{-|x|^{\beta}l_{2}^{l}(|x|)},\quad\text{for }x<0\text{
and $-x$ large enough},$

where $l_{1}^{r}$, $l_{2}^{r}$, $l_{1}^{l}$ and $l_{2}^{l}$ are slowly-varying
functions. We note $X\sim\mathrm{GWT}_{\mathbb{R}}(\beta)$.

This family includes widely used distributions such as Gaussian ($\beta=2$),
Laplace ($\beta=1$) and generalized Gaussian distributions. These
distributions are also symmetric (around 0), so their left and right tails are
equal. Thus, it naturally leads to obtain tail characteristics for symmetric
distributions by considering random variables whose absolute value is Weibull-
tail on $\mathbb{R}_{+}$. See Appendix B for details. While Weibull-tail
random variables are also generalized Weibull-tail, the opposite is not always
true. Consider slowly-varying functions $l_{1}\equiv 1$ and $l_{2}\equiv 2$.
Then function $l(\cdot)=1+\cos^{2}(\ln(\cdot))$ satisfies $l_{1}\leq l\leq
l_{2}$ but is not slowly varying. However, for any $\beta\geq 2$, function
$\overline{F}_{X}(x)=\mathrm{e}^{-x^{\beta}l(x)}$ is the survival function (it
is decreasing) of some random variable $X$ and it satisfies
$\mathrm{e}^{-x^{\beta}l_{2}(x)}\leq\overline{F}_{X}(x)\leq\mathrm{e}^{-x^{\beta}l_{1}(x)}$.
Therefore, $X$ is $\text{GWT}(\beta)$ but not $\text{WT}(\beta)$.

Next, we aim to obtain a tail characterization for the sum of generalized
Weibull-tail variables on $\mathbb{R}$, where we allow random variables to be
dependent. Further, we show that under the following assumption of positive
dependence, the sum has a tail parameter equal to the minimum among the
considered ones (Theorem 2).

######  Definition 2.2 (Positive dependence condition).

Random variables $X_{1},\dots,X_{N}$ satisfy the positive dependence (PD)
condition if the following inequalities hold for all $z\in\mathbb{R}$ and some
constant $C>0$:

$\displaystyle\mathbb{P}\left(X_{1}\geq 0,\ldots,X_{N-1}\geq 0|X_{N}\geq
z\right)\geq C,\quad\mathbb{P}\left(X_{1}\leq 0,\ldots,X_{N-1}\leq 0|X_{N}\leq
z\right)\geq C.$

######  Remark 1.

The choice of zeros and $z$ in the PD condition is arbitrary: one can choose
any $z_{1},\dots,z_{N}$ instead such that $z_{1}+\cdots+z_{N}=z$. The choice
of the $N$-th variable within $X_{1},\dots,X_{N}$ is also arbitrary. Besides,
if random variables $X_{1},\dots,X_{N}$ are independent with non-zero right
and left tails, then they satisfy the PD condition and the constant $C$ is
equal to the minimum between $\mathbb{P}(X_{1}\leq
0)\ldots\mathbb{P}(X_{N-1}\leq 0)$ and $\mathbb{P}(X_{1}\geq
0)\ldots\mathbb{P}(X_{N-1}\geq 0)$.

There is a great variety of dependent distributions that obey this dependence
property including pre-activations in Bayesian neural networks (see Lemma 3.1
for details). Note that the positive orthant dependence condition (POD, see
Nelsen, 2007) implies the PD condition.

######  Theorem 2 (Sum of GWTR variables).

Let $X_{1},\dots,X_{N}\in\mathrm{GWT}_{\mathbb{R}}$ with tail parameters
$\beta_{1},\dots,\beta_{N}$. If $X_{1},\dots,X_{N}$ satisfy the PD condition
of Definition 2.2, then,
$X_{1}+\dots+X_{N}\in\mathrm{GWT}_{\mathbb{R}}(\beta)$ with
$\beta=\min\\{\beta_{1},\dots,\beta_{N}\\}$.

All proofs are deferred in Appendix. The intuition of the PD condition is to
prevent the tail of a sum from becoming lighter due to a negative connection.
The simplest example of such negative dependence comes with counter-
monotonicity where $(X,Y)$ is such that $Y=-X$. Another less trivial example
is $Y=X\mathbb{I}(|X|\leq m)-X\mathbb{I}(|X|>m)$ for some $m>0$: the sum
$X+Y=2X\mathbb{I}(|X|\leq m)$ is a version of $X$ truncated to the compact set
$[-m,m]$. In both cases, it is easy to see that the PD condition does not
hold. We conclude this section with a result on the product of independent
generalized Weibull-tail random variables.

######  Theorem 3 (Product of $\text{GWT}_{\mathbb{R}}$ variables).

Let $X_{1},\dots,X_{N}\in\mathrm{GWT}_{\mathbb{R}}$ be independent symmetric
with tail parameters $\beta_{1},\dots,\beta_{N}$. Then, the product
$X_{1}\ldots X_{N}\in\mathrm{GWT}_{\mathbb{R}}(\beta)$ with $\beta$ such that
$\frac{1}{\beta}=\frac{1}{\beta_{1}}+\cdots+\frac{1}{\beta_{N}}$.

Now the obtained results can be applied to Bayesian neural networks, showing
that hidden units are generalized Weibull-tail on $\mathbb{R}$.

##  3 Bayesian neural networks induced priors

Neural networks are hierarchical models made of layers: an input, several
hidden layers, and an output. Each layer following the input layer consists of
units which are linear combinations of previous layer units transformed by a
nonlinear function, often referred to as the nonlinearity or activation
function denoted by $\phi$. Given an input $\mathbf{x}\in\mathbb{R}^{H_{0}}$,
the $\ell$-th hidden layer consists of a vector whose size is called the width
of the layer, denoted by $H_{\ell}$. The coefficients of linear combinations
are called weights, denoted by $w_{ij}^{(\ell)}$ for
$i=1,\dots,H_{\ell-1},j=1,\dots,H_{\ell}$. In Bayesian neural networks,
weights are assigned some prior distribution (Neal, 1996). The pre-activations
and post-activations of layer $\ell$ are respectively defined as

$g_{j}^{(\ell)}=\sum_{i=1}^{H_{\ell-1}}w_{ij}^{(\ell)}h_{i}^{(\ell-1)},\quad
h_{j}^{(\ell)}=\phi(g_{j}^{(\ell)})$ (2)

where $h_{i}^{(0)}$ are elements of input vector $\mathbf{x}$, so
$h_{i}^{(0)}$ are deterministic numerical object features.

The main ingredient for Theorem 2 is the positive dependence condition of
Definition 2.2. The product of hidden units and weights satisfies the positive
dependence condition:

###### Lemma 3.1

Let $X_{1},\dots,X_{N}$ be some possibly dependent random variables and
$W_{1},\dots,W_{N}$ be symmetric, mutually independent and independent from
$X_{1},\dots,X_{N}$, then random variables $X_{1}W_{1},\dots,X_{N}W_{N}$
satisfy the PD condition.

Along with Theorem 2, the previous lemma implies that neural network hidden
units are generalized Weibull-tail with tail parameter depending on those of
the weights.

######  Theorem 1 (Hidden units are GWT).

Consider a Bayesian neural network as described in Equation (2) with ReLU
activation function. Let $\ell$-th layer weights be independent symmetric
generalized Weibull-tail on $\mathbb{R}$ with tail parameter
$\beta^{(\ell)}_{w}$. Then, $\ell$-th layer pre-activations are generalized
Weibull-tail on $\mathbb{R}$ with tail parameter $\beta^{(\ell)}$ such that
$\frac{1}{\beta^{(\ell)}}=\frac{1}{\beta^{(1)}_{w}}+\dots+\frac{1}{\beta^{(\ell)}_{w}}$.

Note that the most popular case of weight prior, iid Gaussian (Neal, 1996),
corresponds to $\text{GWT}_{\mathbb{R}}(2)$ weights. This leads to units of
layer $\ell$ which are $\text{GWT}_{\mathbb{R}}(\frac{2}{\ell})$.

To illustrate this theorem, we have built neural networks of 4 hidden layers,
with 4 hidden units on each layer. We used a fixed input $\mathbf{x}$ of size
$10^{4}$, which can be thought of as an image of dimension $100\times 100$.
This input was sampled once for all with standard Gaussian entries. In order
to obtain samples from the prior distribution of the neural network units, we
have sampled the weights from independent centered Gaussians from which units
were obtained by forward evaluation with the ReLU non-linearity. This process
was iterated $n=10^{6}$ times. Note that for a $\mathrm{GWT}_{\mathbb{R}}$
random variable, $\mathbb{P}(X\geq x)=\mathrm{e}^{-x^{\beta}l(x)}$ so the tail
parameter can be expressed as:

$\beta=\frac{\log(-\log\mathbb{P}(X\geq x))}{\log x}-\frac{\log l(x)}{\log
x}.$ (3)

In Figure 1, we plot $\log(-\log\mathbb{P}(X\geq x))$ as a function of $\log
x$. We see that the obtained tail parameters approximations are increasing for
the increasing layer number and visually correspond to the theoretical tail
parameter.

[capbesideposition=left,top,capbesidewidth=.45]figure[]

Figure 1: Solid lines: approximations of tail parameters $\beta^{(\ell)}$
based on Equation (3) where $X$ are hidden units of layers $\ell=1,2,3,4$
corresponding theoretically to generalized Weibull-tail with tail parameters
$\beta^{(\ell)}=2,1,\nicefrac{{2}}{{3}},\nicefrac{{1}}{{2}}$, under the
independent Gaussian weights assumption. Dashed lines: linear regressions with
coefficients equal to the theoretical tail parameters
$\beta^{(\ell)}=\nicefrac{{2}}{{\ell}}$ and manually selected biases to
approach the solid lines for visual comparison.

##  4 Comparison of different characterizations

###  4.1 Generalized Weibull-tail vs sub-Weibull

Some of the commonly used techniques to study the tail behavior is to consider
probability tail bounds such as sub-Gaussian, sub-exponential, or their
generalization to sub-Weibull distributions (Vladimirova et al., 2020;
Kuchibhotla and Chakrabortty, 2018). A non-negative random variable $X$ is
called sub-Weibull with tail parameter $\theta>0$ if its survival function is
upper-bounded by that of a Weibull distribution:

$\displaystyle\overline{F}_{X}(x)\leq
a\mathrm{e}^{-bx^{1/\theta}},\,\,\,\text{for }x>0\,\,\,\text{and some }a,b>0.$
(4)

This property ensures the existence of the moment generating function as well
as bounds on moments. In contrast, the Weibull-tail property characterizes the
survival or density functions without a hand on moments. While tail parameters
in Equation (1) and (4) of generalized Weibull-tail and sub-Weibull properties
respectively are different, there exist connections. Notice that for any
constants $a,b,\beta>0$, function $l(x)=b-\frac{\log a}{x^{\beta}}\geq 0$ is
slowly-varying for $x$ large enough and
$a\mathrm{e}^{-bx^{\beta}}=\mathrm{e}^{-x^{\beta}l(x)}$. It means that if a
random variable $X$ is sub-Weibull with parameter $\theta=1/\beta>0$,
satisfying Equation (4), then the survival function of $X$ is upper-bounded by
a Weibull-tail function with tail parameter $\beta$ and slowly-varying
function $l(x)=1$, satisfying Equation (1). If random variable $X$ is
generalized Weibull-tail with tail parameter $\beta$, then from the last item
of Proposition 1, for $a_{1},a_{2}>0$ we have

$a_{1}\mathrm{e}^{-x^{\beta_{1}}}\leq\overline{F}_{X}(x)=\mathrm{e}^{-x^{\beta}l(x)}\leq
a_{2}\mathrm{e}^{-x^{\beta_{2}}},$

or $\text{GWT}_{\mathbb{R}_{+}}(\beta)\subset\text{SubW}(1/\beta_{2})$ and
$\text{GWT}_{\mathbb{R}_{+}}(\beta)\not\subset\text{SubW}(1/\beta_{1})$ for
$x$ large enough and $\forall(\beta_{1},\beta_{2})$ such that
$0<\beta_{2}<\beta<\beta_{1}$, as illustrated on Figure 2.

$\text{SubW}\left(\frac{1}{\beta_{1}}\right)$$\text{SubW}\left(\frac{1}{\beta_{2}}\right)$$\text{GWT}\left(\beta\right)$

$\text{SubW}\left(\frac{1}{\beta_{1}}\right)$$\text{GWT}\left(\beta\right)$$\text{SubW}\left(\frac{1}{\beta_{2}}\right)$

Figure 2: Relation between sub-Weibull and generalized Weibull-tail
characteristics.

It was recently shown in Vladimirova et al. (2019) that hidden units of
Bayesian neural networks with iid Gaussian priors are sub-Weibull with tail
parameter proportional to the hidden layer number, that is
$\theta=\frac{\ell}{2}$. It means that the unit distributions of hidden layer
$\ell$ can be upper-bounded by some Weibull distributions
$a\mathrm{e}^{-x^{2/\ell}}$ for all $\ell$. For larger tail parameter
$\theta$, Weibull distribution $a\mathrm{e}^{-x^{1/\theta}}$ is heavier-tailed
but being sub-Weibull does not guarantee the heaviness of the tails. However,
this upper bound is optimal in the sense that it is achieved for neural
networks with one hidden unit per layer.

From Theorem 1, for neural networks with independent Gaussian weights, hidden
units of $\ell$-th layer are generalized Weibull-tail with tail parameter
$\beta=1/\theta=2/\ell$ so they have upper and lower bounds of the form
$\mathrm{e}^{-x^{2/\ell}l(x)}$ up to a constant where $l$ is some slowly-
varying function. Therefore, it proves that hidden units are heavier-tailed as
going deeper for any finite numbers of hidden units per layer.

###  4.2 Meijer G-functions description

In Springer and Thompson (1970) it was shown that the probability density
function of the product of independent normal variables could be expressed
through a Meijer G-function. It resulted in an accurate description of induced
unit priors given Gaussian priors on weights and linear or ReLU activation
function (Zavatone-Veth and Pehlevan, 2021; Noci et al., 2021). It is a full
description of function-space priors but under strong assumptions, requiring
Gaussian priors on weights and linear or ReLU activation functions, and with
convoluted expressions. In contrast, we provide results for many
distributions, including heavy-tailed ones, and our results can be extended to
smooth activation functions, such as PReLU, ELU, Softplus.

##  5 Future applications

#### Cold posterior effect and priors.

It was recently empirically found that Gaussian priors led to the cold
posterior effect in which a tempered “cold” posterior, obtained by
exponentiating the posterior to some power largely greater than one, performs
better than an untempered one (Wenzel et al., 2020). The performed Bayesian
inference is considered sub-optimal due to the need for cold posteriors, and
the model is deemed misspecified. From that angle, Wenzel et al. (2020)
suggested that Gaussian priors might not be a good choice for Bayesian neural
networks. In some works, data augmentation is argued to be the main reason for
this effect (Izmailov et al., 2021; Nabarro et al., 2021) as the increased
amount of observed data naturally leads to higher posterior contraction
(Izmailov et al., 2021). At the same time, even considering the data
augmentation for some models, the cold posterior effect is still present. In
addition, Aitchison (2021) demonstrates that the problem might originate in
the wrong likelihood of the models and that modifying only the likelihood
based on data curation mitigates the cold posterior effect. Nabarro et al.
(2021) hypothesize that using an appropriate prior incorporating knowledge of
data augmentation might provide a solution. Moreover, heavy-tailed priors have
been shown to mitigate the cold posterior effect (Fortuin et al., 2021).
According to Theorem 1, heavier-tailed priors lead to even heavier-tailed
induced priors in function-space. Thus, the heavy-tail property of
distributions in function-space might be a highly beneficial feature. Fortuin
et al. (2021) also proposed correlated priors for convolutional neural
networks since trained weights are empirically strongly correlated. Correlated
priors improve overall performance but do not alleviate the cold posterior
effect. Our theory can be extended to correlated weight priors. This direction
is promising for further uncovering the effect of weight prior on function-
space prior.

#### Edge of Chaos.

An active line of research studies the propagation of deterministic inputs in
neural networks (Poole et al., 2016; Schoenholz et al., 2017; Hayou et al.,
2019). The main idea is to explore the covariance between pre-activations for
two given different data points. Poole et al. (2016) and Schoenholz et al.
(2017) obtained recurrence relations under the assumption of Gaussian
initialization and Gaussian pre-activations. They conclude that there is a
critical line, so-called Edge of Chaos, separating signal propagation into two
regions. The first one is an ordered phase in which all inputs end up
asymptotically correlated. The second is a chaotic phase in which all inputs
end up asymptotically independent. To propagate the information deeper in a
neural network, one should choose Gaussian prior variances corresponding to
the separating line. Hayou et al. (2019) show that the smoothness of the
activation function also plays an important role. Since this line of works
considers Gaussian priors not only on the weights but also on the pre-
activations, it is closely related to a wide regime where the number of hidden
units per layer tends to infinity. Given that hidden units are heavier-tailed
with depth, we speculate that future research will focus on finding better
approximations of the pre-activation functions in recurrence relations
obtained for finite-width neural networks.

##  6 Conclusion

We extend the theory on induced distributions in Bayesian neural networks and
establish an accurate and easily interpretable characterization of hidden
units tails. The obtained results confirm the heavy-tailed nature of hidden
units for different weight priors.

## References

  * Aitchison (2020) Laurence Aitchison.  Why bigger is not always better: on finite and infinite neural networks.  In _International Conference on Machine Learning_ , 2020.
  * Aitchison (2021) Laurence Aitchison.  A statistical theory of cold posteriors in deep neural networks.  In _International Conference on Learning Representations_ , 2021.
  * Bingham et al. (1989) Nicholas H Bingham, Charles M Goldie, and Jef L Teugels.  _Regular variation_.  Number 27. Cambridge university press, 1989.
  * Favaro et al. (2020) Stefano Favaro, Sandra Fortini, and Peluchetti Stefano.  Stable behaviour of infinitely wide deep neural networks.  In _International Conference on Artificial Intelligence and Statistics_ , 2020.
  * Fortuin et al. (2021) Vincent Fortuin, Adrià Garriga-Alonso, Florian Wenzel, Gunnar Rätsch, Richard Turner, Mark van der Wilk, and Laurence Aitchison.  Bayesian neural network priors revisited.  _arXiv preprint arXiv:2102.06571_ , 2021.
  * Gardes and Girard (2016) Laurent Gardes and Stéphane Girard.  On the estimation of the functional Weibull tail-coefficient.  _Journal of Multivariate Analysis_ , 146:29–45, 2016.
  * Gardes et al. (2011) Laurent Gardes, Stéphane Girard, and Armelle Guillou.  Weibull tail-distributions revisited: a new look at some tail estimators.  _Journal of Statistical Planning and Inference_ , 141(1):429–444, 2011.
  * Garriga-Alonso et al. (2019) Adrià Garriga-Alonso, Carl Edward Rasmussen, and Laurence Aitchison.  Deep convolutional networks as shallow Gaussian processes.  In _International Conference on Learning Representations_ , 2019.
  * Hayou et al. (2019) Soufiane Hayou, Arnaud Doucet, and Judith Rousseau.  On the impact of the activation function on deep neural networks training.  In _International Conference on Machine Learning_ , 2019.
  * Izmailov et al. (2021) Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Wilson.  What are Bayesian neural network posteriors really like?  In _International Conference on Machine Learning_ , 2021.
  * Kuchibhotla and Chakrabortty (2018) Arun Kumar Kuchibhotla and Abhishek Chakrabortty.  Moving beyond sub-Gaussianity in high-dimensional statistics: Applications in covariance estimation and linear regression.  _arXiv preprint arXiv:1804.02605_ , 2018.
  * Lee et al. (2018) Jaehoon Lee, Jascha Sohl-Dickstein, Jeffrey Pennington, Roman Novak, Sam Schoenholz, and Yasaman Bahri.  Deep neural networks as Gaussian processes.  In _International Conference on Learning Representations_ , 2018.
  * Lee et al. (2020) Jaehoon Lee, Samuel Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein.  Finite versus infinite neural networks: an empirical study.  In _International Conference on Neural Information Processing Systems_ , 2020.
  * Matthews et al. (2018) Alexander G de G Matthews, Mark Rowland, Jiri Hron, Richard E Turner, and Zoubin Ghahramani.  Gaussian process behaviour in wide deep neural networks.  In _International Conference on Learning Representations_ , 2018.
  * McNeil et al. (2015) Alexander J McNeil, Rüdiger Frey, and Paul Embrechts.  _Quantitative risk management: concepts, techniques and tools_.  Princeton university press, 2015.
  * Nabarro et al. (2021) Seth Nabarro, Stoil Ganev, Adrià Garriga-Alonso, Vincent Fortuin, Mark van der Wilk, and Laurence Aitchison.  Data augmentation in Bayesian neural networks and the cold posterior effect.  _arXiv preprint arXiv:2106.05586_ , 2021.
  * Neal (1996) Radford M Neal.  _Bayesian learning for neural networks_.  Springer Science & Business Media, 1996.
  * Nelsen (2007) Roger B Nelsen.  _An introduction to copulas_.  Springer Science & Business Media, 2007.
  * Noci et al. (2021) Lorenzo Noci, Gregor Bachmann, Kevin Roth, Sebastian Nowozin, and Thomas Hofmann.  Precise characterization of the prior predictive distribution of deep ReLU networks.  _arXiv preprint arXiv:2106.06615_ , 2021.
  * Poole et al. (2016) Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli.  Exponential expressivity in deep neural networks through transient chaos.  In _International Conference on Neural Information Processing Systems_ , 2016.
  * Rachev (2003) Svetlozar Todorov Rachev.  _Handbook of Heavy Tailed Distributions in Finance: Handbooks in Finance, Book 1_.  Elsevier, 2003.
  * Schoenholz et al. (2017) Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein.  Deep information propagation.  In _International Conference on Learning Representations_ , 2017.
  * Springer and Thompson (1970) Melvin Dale Springer and William E. Thompson.  The distribution of products of beta, gamma and Gaussian random variables.  _SIAM Journal on Applied Mathematics_ , 18(4):721–737, 1970.
  * Strupczewski et al. (2011) Witold G Strupczewski, Krzysztof Kochanek, Iwona Markiewicz, Ewa Bogdanowicz, Stanislaw Weglarczyk, and Vijay P Singh.  On the tails of distributions of annual peak flow.  _Hydrology Research_ , 42(2-3):171–192, 2011\.
  * Vladimirova et al. (2019) Mariia Vladimirova, Jakob Verbeek, Pablo Mesejo, and Julyan Arbel.  Understanding priors in Bayesian neural networks at the unit level.  In _International Conference on Machine Learning_ , 2019.
  * Vladimirova et al. (2020) Mariia Vladimirova, Stéphane Girard, Hien Nguyen, and Julyan Arbel.  Sub-Weibull distributions: Generalizing sub-Gaussian and sub-exponential properties to heavier tailed distributions.  _Stat_ , 9(1):e318, 2020.
  * Wenzel et al. (2020) Florian Wenzel, Kevin Roth, Bastiaan Veeling, Jakub Swiatkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin.  How good is the Bayes posterior in deep neural networks really?  In _International Conference on Machine Learning_ , 2020.
  * Wilson and Izmailov (2020) Andrew Gordon Wilson and Pavel Izmailov.  Bayesian deep learning and a probabilistic perspective of generalization.  _International Conference on Neural Information Processing Systems_ , 2020.
  * Zavatone-Veth and Pehlevan (2021) Jacob A Zavatone-Veth and Cengiz Pehlevan.  Exact priors of finite neural networks.  _arXiv preprint arXiv:2104.11734_ , 2021.

##  Appendix A Slowly and regularly-varying functions theory

The set of regular-varying functions with index $\rho\in\mathbb{R}$ is denoted
by $\mathcal{RV}_{\rho}$. Note that for $\rho=0$, the set $\mathcal{RV}_{0}$
boils down to the set of slowly-varying functions. In particular, any function
$r\in\mathcal{RV}_{\rho}$ can be written $r(x)=x^{\rho}l(x)$, where $l$ is
slowly-varying.

######  Definition A.1 (Regularly-varying function).

Let $r$ be a positive function. Then $r\in\mathcal{RV}_{\rho}$ if for all
$t>0$ $\lim_{x\to\infty}\frac{r(tx)}{r(x)}=t^{\rho}$.

######  Proposition 1.

(Bingham et al., 1989, Proposition 1.3.6) Let $l,l_{1},\ldots,l_{k}$ be
slowly-varying functions. Then:

  1. 1.

$(\log l(x))/\log x\to 0$ as $x\to\infty$.

  2. 2.

$l^{\alpha}$ varies slowly for every $\alpha\in\mathbb{R}$.

  3. 3.

$l_{1}l_{2}$, $l_{1}+l_{2}$, and (if $l_{2}(x)\to\infty$ as $x\to\infty$)
$l_{1}\circ l_{2}$ vary slowly.

  4. 4.

If $f(x_{1},\dots,x_{k})$ is a rational function with positive coefficients,
$f(l_{1},\dots,l_{k})$ varies slowly.

  5. 5.

For any $\alpha>0$, $x^{\alpha}l(x)\to\infty,\ x^{-\alpha}l(x)\to 0\
(x\to\infty)$.

###### Lemma A.1

If $l_{1}(x)$ is slowly-varying, then $l_{2}(x)=l_{1}(x^{a})$ is slowly-
varying for $a>0$.

###### Lemma A.2

If $l_{1},l_{2}$ vary slowly, so does $\max\\{l_{1},l_{2}\\}$ and
$\min\\{l_{1},l_{2}\\}$.

###### Lemma A.3

Let $r_{1}$ and $r_{2}$ be regularly-varying functions with parameters
$\beta_{1}>0$ and $\beta_{2}>0$. Then, the function $r$ such that
$\mathrm{e}^{-r}=\mathrm{e}^{-r_{1}}+\mathrm{e}^{-r_{2}}$, is regularly-
varying with parameter $\beta=\min\\{\beta_{1},\beta_{2}\\}$.

######  Proof A.2.

Let us express function $r$ from the statement:
$r(x)=-\log\varphi(x)=-\log\left(\mathrm{e}^{-r_{1}(x)}+\mathrm{e}^{-r_{2}(x)}\right)$.
If $\beta_{1}\not=\beta_{2}$, without loss of generality, let us assume that
$\beta_{1}<\beta_{2}$, then

$\varphi(x)=\mathrm{e}^{-r_{1}(x)}\left(1+\mathrm{e}^{-r_{2}(x)+r_{1}(x)}\right)=\mathrm{e}^{-r_{1}(x)}\left(1+\mathrm{e}^{r_{2}(x)\left(-1+\frac{r_{1}(x)}{r_{2}(x)}\right)}\right).$

Notice that $\frac{r_{1}}{r_{2}}\in\mathcal{RV}_{\beta_{1}-\beta_{2}}$. For
$\beta_{1}<\beta_{2}$, $\frac{r_{1}(x)}{r_{2}(x)}\to 0$ when $x\to\infty$. The
expression in the exponent
$r_{2}(x)\left(-1+\frac{r_{1}(x)}{r_{2}(x)}\right)\simeq-
r_{2}(x)\to-\infty$111We say functions $f\simeq g$ if and only if $f/g\to 1$.,
so the exponent
$\mathrm{e}^{r_{2}(x)\left(-1+\frac{r_{1}(x)}{r_{2}(x)}\right)}\to 0$. Then
$\varphi(x)\simeq\mathrm{e}^{-r_{1}(x)}$ and $r(x)=-\log\varphi(x)\simeq
r_{1}(x)$. It means that for the case $\beta_{1}<\beta_{2}$,
$r\in\mathcal{RV}_{\beta_{1}}$.

Let us consider the case with equal parameters $\beta_{1}=\beta_{2}=\beta$,
then $r_{1}(x)=x^{\beta}l_{1}(x)$, $r_{2}(x)=x^{\beta}l_{2}(x)$ with slowly-
varying $l_{1}$ and $l_{2}$. With $l=\min\\{l_{1},l_{2}\\}$, we can write

$\varphi(x)=\mathrm{e}^{-x^{\beta}l(x)}\left(1+\mathrm{e}^{-x^{\beta}|l_{2}(x)-l_{1}(x)|}\right).$

Consider the logarithm of the latter expression

$-\log\varphi(x)=x^{\beta}\left(l(x)+x^{-\beta}\log\left(1+\mathrm{e}^{-x^{\beta}|l_{2}(x)-l_{1}(x)|}\right)\right).$

Since the function $l$ is slowly-varying by Lemma A.2 and
$0\leq\mathrm{e}^{-x^{\beta}|l_{2}(x)-l_{1}(x)|}\leq 1$, then
$r(x)=-\log\varphi(x)\simeq x^{\beta}l(x)\in\mathcal{RV}_{\beta}$.

##  Appendix B Weibull-tail properties on $\mathbb{R}$

Let us firstly introduce a notion of generalized Weibull-tail random variable
which has an additional property of stability:

######  Definition B.1 (Generalized Weibull-tail on $\mathbb{R}_{+}$).

A random variable $X$ is called generalized Weibull-tail with tail parameter
$\beta>0$ if its survival function $\overline{F}_{X}$ is bounded by Weibull-
tail functions of tail parameter $\beta$ with possibly different slowly-
varying functions $l_{1}$ and $l_{2}$:

$\mathrm{e}^{-x^{\beta}l_{1}(x)}\leq\overline{F}_{X}(x)\leq\mathrm{e}^{-x^{\beta}l_{2}(x)},\,\,\,\text{for
}x>0.$ (5)

We note $X\sim\text{GWT}_{\mathbb{R}_{+}}(\beta)$.

Now we define a random variable whose both right and left tails are Weibull-
tail on $\mathbb{R}_{+}$.

######  Definition B.2 (Weibull-tail on $\mathbb{R}$).

A random variable $X$ on $\mathbb{R}$ is Weibull-tail on $\mathbb{R}$ with
tail parameter $\beta>0$ if both its right and left tails are Weibull-tail
with tail parameter $\beta$:

$\overline{F}_{X}(x)=1-F_{X}(x)=\mathrm{e}^{-x^{\beta}l_{1}(x)},\quad x>0$
$F_{X}(x)=\mathrm{e}^{-|x|^{\beta}l_{2}(|x|)},\quad x<0,$

where $l_{1}$ and $l_{2}$ are slowly-varying functions. We note
$X\sim\text{WT}_{\mathbb{R}}(\beta)$.

###### Lemma B.1

  1. (i)

If random variable $X$ on $\mathbb{R}$ is $\text{WT}_{\mathbb{R}}(\beta)$,
then $|X|$ is $\text{WT}_{\mathbb{R}_{+}}(\beta)$.

  2. (ii)

If $X$ is asymmetric but both tails are Weibull-tail and $|X|$ is
$\text{WT}_{\mathbb{R}_{+}}(\beta)$, then one of the tails (right or left) is
$\text{WT}_{\mathbb{R}_{+}}(\beta)$ and the other one is
$\text{WT}_{\mathbb{R}_{+}}(\beta^{\prime})$ where $\beta^{\prime}\geq\beta$.

  3. (iii)

For symmetric distributions, $X$ is $\text{WT}_{\mathbb{R}}(\beta)$ if and
only if $|X|$ is $\text{WT}_{\mathbb{R}_{+}}(\beta)$.

######  Proof B.3.

  1. (i)

For $x>0$, the cumulative distribution function of $|X|$ is the following

$\displaystyle F_{|X|}(x)=F_{X}(x)-F_{X}(-x).$

Then, $\overline{F}_{|X|}(x)$ can be expressed as a sum of the right and left
tails:

$\displaystyle\overline{F}_{|X|}(x)=\overline{F}_{X}(x)+F_{X}(-x).$

If $X$ is $\text{WT}_{\mathbb{R}}(\beta)$, then $|X|$ is
$\text{WT}_{\mathbb{R}_{+}}(\beta)$ as a consequence of Definition B.2 and
Lemma A.3.

  2. (ii)

Let $|X|$ be $\text{WT}_{\mathbb{R}_{+}}(\beta)$ and $X$ has Weibull left and
right tails with different tail parameters. Without loss of generality, assume
that ${F}_{X}(-x)$ is Weibull-tail with tail parameter $\beta^{\prime}<\beta$.
According to Lemma A.3, the sum survival function $\overline{F}_{|X|}(x)$ will
be Weibull-tail with tail parameter
$\beta^{\prime}=\min\\{\beta^{\prime},\beta\\}$. We obtained a contradiction
and ${F}_{X}(-x)$ must have tail parameter greater or equal $\beta$. If both
tail parameters are greater than $\beta$, then the tails sum have the tail
parameter equal to the minimum tail parameter among them which is greater than
$\beta$. It means that at least one tail must have tail parameter $\beta$.

  3. (iii)

For symmetric distributions $\overline{F}_{X}(x)={F}_{X}(-x)$ for any $x$,
then $\frac{1}{2}\overline{F}_{|X|}(x)=\overline{F}_{X}(x)={F}_{X}(-x)$ and we
have the equality.

###### Lemma B.2

  1. (i)

If a random variable $X$ is $\text{GWT}_{\mathbb{R}}(\beta)$, then $|X|$ is
$\text{GWT}_{\mathbb{R}_{+}}(\beta)$.

  2. (ii)

For symmetric distributions, $X$ is $\text{GWT}_{\mathbb{R}}(\beta)$ if and
only if $|X|$ is $\text{GWT}_{\mathbb{R}_{+}}(\beta)$.

######  Proof B.4.

Similarly as in Lemma B.1, we obtain
$\overline{F}_{|X|}(x)=\overline{F}_{X}(x)+F_{X}(-x)$ for $x\geq 0$.

  1. (i)

If $X$ is $\text{GWT}_{\mathbb{R}}(\beta)$, then right and left tails are
upper and lower-bounded by some Weibull-tail functions. Then, the sum of the
right and left tails is upper and lower-bounded by these Weibull-tail
functions and $|X|\sim\text{GWT}_{\mathbb{R}_{+}}(\beta)$.

  2. (ii)

For symmetric distributions we have $\overline{F}_{X}(x)={F}_{X}(-x)$ for all
$x$, then $\frac{1}{2}\overline{F}_{|X|}(x)=\overline{F}_{X}(x)={F}_{X}(-x)$
and we have the equality.

######  Lemma B.3 (Power and multiplication by a constant)

If $X\sim\text{GWT}_{\mathbb{R}}(\beta)$ and the distribution of $X$ is
symmetric, then
$a|X|^{b}\sim\text{GWT}_{\mathbb{R}_{+}}\left(\frac{\beta}{b}\right)$ for
$a,b>0$.

######  Proof B.5.

According to Lemma B.1, $|X|\sim\text{GWT}_{\mathbb{R}_{+}}(\beta)$. For
$a,b>0$, the tail of $Y=a|X|^{b}$ is

$\mathbb{P}\Bigl{(}a|X|^{b}\geq
y\Bigr{)}=\mathbb{P}\left(|X|^{b}\geq\frac{y}{a}\right)=\mathbb{P}\Bigl{(}|X|\geq\left(\frac{y}{a}\right)^{1/b}\Bigr{)}.$

Since $|X|$ is generalized Weibull-tail on $\mathbb{R}_{+}$ with tail
parameter $\beta$, $\mathrm{e}^{-x^{\beta}l_{1}(x)}\leq\mathbb{P}\left(|X|\geq
x\right)\leq\mathrm{e}^{-x^{\beta}l_{2}(x)}$, where $l_{1}$ and $l_{2}$ are
slowly-varying functions, it implies

$\mathrm{e}^{-y^{\beta/b}\tilde{l}_{1}(y)}\leq\mathbb{P}\Bigl{(}a|X|^{b}\geq
y\Bigr{)}\leq\mathrm{e}^{-y^{\beta/b}\tilde{l}_{2}(y)},$

where $\tilde{l}_{i}(y)=\frac{l_{i}((y/a)^{1/b})}{a^{\beta/b}}$, $i=1,2$ are
slowly-varying functions by Lemma A.1. It leads to the statement of the lemma.

######  Theorem 2 (Sum of GWTR variables)

Let $X_{1},\dots,X_{N}\in\mathrm{GWT}_{\mathbb{R}}$ with tail parameters
$\beta_{1},\dots,\beta_{N}$. If $X_{1},\dots,X_{N}$ satisfy the PD condition
of Definition 2.2, then,
$X_{1}+\dots+X_{N}\in\mathrm{GWT}_{\mathbb{R}}(\beta)$ with
$\beta=\min\\{\beta_{1},\dots,\beta_{N}\\}$.

######  Proof B.6.

Let us start with $N=2$. For any random variables $X$ and $Y$, the following
upper bound holds:

$\mathbb{P}(X+Y\geq
z)\leq\mathbb{P}(X\geq\nicefrac{{z}}{{2}})+\mathbb{P}(Y\geq\nicefrac{{z}}{{2}})\leq
2\max\\{\mathbb{P}(X\geq\nicefrac{{z}}{{2}}),\mathbb{P}(Y\geq\nicefrac{{z}}{{2}})\\}.$

The PD condition leads to a lower bound for the sum:

$\mathbb{P}(X+Y\geq z)\geq\mathbb{P}(X\geq 0,Y\geq z)=\mathbb{P}(X\geq 0|Y\geq
z)\mathbb{P}(Y\geq z)\geq C\,\mathbb{P}(Y\geq z),$

where constant $C>0$. Thus, the sum survival function
$\overline{F}_{Z}(z)=\mathbb{P}(X+Y\geq z)$ has the following bounds for the
right tail:

$C\,\overline{F}_{Y}(z)\leq\overline{F}_{Z}(z)\leq
2\max\\{\overline{F}_{X}(\nicefrac{{z}}{{2}}),\overline{F}_{Y}(\nicefrac{{z}}{{2}})\\},$

where $\overline{F}_{X}$ and $\overline{F}_{Y}$ are the survival functions of
$X$ and $Y$.

Let $X$ and $Y$ be generalized Weibull-tail on $\mathbb{R}$ of parameters
$\beta_{x}$ and $\beta_{y}$, then for $z>0$ large enough

$C\,\mathrm{e}^{-z^{\beta_{y}}l_{1}^{r}(z)}\leq\overline{F}_{Z}(z)\leq
2\mathrm{e}^{-z^{\min\\{\beta_{x},\beta_{y}\\}}l_{2}^{r}(z)},$

where $l_{1}^{r}=l_{1}^{y}$ is the slowly-varying function appearing in the
right tail lower bound of generalized Weibull-tail $Y$ and
$l_{2}^{r}(z)=\min\\{\frac{1}{2^{\beta_{x}}}l_{2}^{x}(\nicefrac{{z}}{{2}}),\frac{1}{2^{\beta_{y}}}l_{2}^{y}(\nicefrac{{z}}{{2}})\\}$
is the minimum among slowly-varying functions, where $l_{2}^{x}$ and
$l_{2}^{y}$ are slowly-varying functions in the right tails upper bounds of
$X$ and $Y$. According to Lemma A.2, $l_{2}^{r}(z)$ is also slowly-varying.
Similarly, we can get bounds for the left tail. Therefore, $X+Y$ is
generalized Weibull-tail on $\mathbb{R}$ with tail parameter
$\beta=\min\\{\beta_{x},\beta_{y}\\}$.

Similarly as above when $N=2$, bounds for the right tail of the sum
$Z=X_{1}+\dots+X_{N}$ with survival function
$\overline{F}_{Z}=\mathbb{P}(Z\geq z)$ are:

$C\overline{F}_{N}(z)\leq\overline{F}_{Z}(z)\leq
N\max\\{\overline{F}_{1}(\nicefrac{{z}}{{N}}),\dots,\overline{F}_{N}(\nicefrac{{z}}{{N}})\\},$

where $\overline{F}_{i}$ is the survival functions of $X_{i}$ and constant
$C>0$. The rest of the proof is identical to the one of the case $N=2$.

The case when distributions have only right tails (or only left tails), can be
considered as a particular case of the last theorem: a sum of non-negative
generalized Weibull-tail random variables is non-negative generalized Weibull-
tail with tail parameter equal to the minimum among those of the terms.

######  Theorem 3 (Product of independent $\text{GWT}_{\mathbb{R}}$ variables)

Let $X_{1},\dots,X_{N}\in\mathrm{GWT}_{\mathbb{R}}$ be independent symmetric
with tail parameters $\beta_{1},\dots,\beta_{N}$. Then, the product
$X_{1}\ldots X_{N}\in\mathrm{GWT}_{\mathbb{R}}(\beta)$ with $\beta$ such that
$\frac{1}{\beta}=\frac{1}{\beta_{1}}+\cdots+\frac{1}{\beta_{N}}$.

######  Proof B.7.

Consider two independent symmetric generalized Weibull-tail random variables
with tail parameters $\beta_{x}=\frac{1}{\theta_{x}}$ and
$\beta_{y}=\frac{1}{\theta_{y}}$,
$X\sim\text{GWT}_{\mathbb{R}}\left(\frac{1}{\theta_{x}}\right)$ and
$Y\sim\text{GWT}_{\mathbb{R}}\left(\frac{1}{\theta_{y}}\right)$. From Lemma
B.1 and since random variables $X$ and $Y$ are symmetric, it is equivalent to
$|X|\sim\text{GWT}_{\mathbb{R}_{+}}\left(\frac{1}{\theta_{x}}\right)$ and
$|Y|\sim\text{GWT}_{\mathbb{R}_{+}}\left(\frac{1}{\theta_{y}}\right)$.

The product of independent symmetric distributions is symmetric since
$Z=XY=^{d}(-X)Y=^{d}-Z$. From Lemma B.1, $Z\sim\text{GWT}_{\mathbb{R}}(\beta)$
if and only if $|Z|\sim\text{GWT}_{\mathbb{R}_{+}}(\beta)$.

Our goal is to show that for some slowly-varying functions $l_{1}$ and
$l_{2}$, there exist upper and lower bounds for the survival function of $|Z|$
and $z$ large enough as follows:

$\displaystyle\mathrm{e}^{-z^{\frac{1}{\theta_{x}+\theta_{y}}}l_{1}(z)}\leq\overline{F}_{|Z|}(z)=\mathbb{P}(|XY|\geq
z)\leq\mathrm{e}^{-z^{\frac{1}{\theta_{x}+\theta_{y}}}l_{2}(z)}.$ (6)

  1. (i)

Upper bound. First, notice that from the concavity of the logarithm, we have
$\ln(pu+(1-p)v)\geq p\ln u+(1-p)\ln v$ for any $u,v>0$ and $p\in(0,1)$. Then
$pu+(1-p)v\geq u^{p}v^{1-p}$. The change of variables $x=u^{p}$, $y=v^{1-p}$
implies $px^{\nicefrac{{1}}{{p}}}+(1-p)y^{\nicefrac{{1}}{{(1-p)}}}\geq xy$.
From the latter equation, an upper bound of the product tail is

$\mathbb{P}(|XY|\geq
z)\leq\mathbb{P}\left(p|X|^{\nicefrac{{1}}{{p}}}+(1-p)|Y|^{\nicefrac{{1}}{{(1-p)}}}\geq
z\right).$ (7)

Lemma B.3 implies that
$p|X|^{\nicefrac{{1}}{{p}}}\sim\text{GWT}_{\mathbb{R}_{+}}\left(\frac{p}{\theta_{x}}\right)$
and
$(1-p)|Y|^{\nicefrac{{1}}{{1-p}}}\sim\text{GWT}_{\mathbb{R}_{+}}\left(\frac{1-p}{\theta_{y}}\right)$.
Taking $p=\frac{\theta_{x}}{\theta_{x}+\theta_{y}}$ and
$1-p=\frac{\theta_{y}}{\theta_{x}+\theta_{y}}$, yields a sum of two
independent non-negative generalized Weibull-tail random variables with tail
parameter $\frac{1}{\theta_{x}+\theta_{y}}$ on the right-hand side of Equation
(7). By Theorem 2, this sum is generalized Weibull-tail with the same tail
parameter $\frac{1}{\theta_{x}+\theta_{y}}$. It means that there exists a
slowly-varying function $l_{2}$ such that the tail of product absolute value
$|XY|$ is upper-bounded by

$\mathbb{P}(|XY|\geq
z)\leq\mathrm{e}^{-z^{\frac{1}{\theta_{x}+\theta_{y}}}l_{2}(z)}.$ (8)

  2. (ii)

Lower bound. By independence of $|X|$ and $|Y|$ we have

$\mathbb{P}(|XY|\geq z)\geq\mathbb{P}\left(|X|\geq
z^{\frac{\theta_{x}}{\theta_{x}+\theta_{y}}}\right)\mathbb{P}\left(|Y|\geq
z^{\frac{\theta_{y}}{\theta_{x}+\theta_{y}}}\right).$

Since $|X|$ and $|Y|$ are generalized Weibull-tail on $\mathbb{R}_{+}$, we can
define function
$l_{1}(z)=l_{1}^{x}(z^{\theta_{x}/{(\theta_{x}+\theta_{y})}})+l_{1}^{y}(z^{\theta_{y}/{(\theta_{x}+\theta_{y})}})$
with $l_{1}^{x}$ and $l_{1}^{y}$ being slowly-varying functions in the lower
bounds of generalized Weibull-tail $|X|$ and $|Y|$. Then, $l_{1}$ is slowly-
varying by Lemma A.1 and we have

$\mathbb{P}(|XY|\geq
z)\geq\mathrm{e}^{-z^{\frac{1}{\theta_{x}+\theta_{y}}}l_{1}(z)}.$ (9)

Combining together Equations (8) and (9) and Definition B.1 with Lemma B.1
implies the statement of the theorem.

##  Appendix C Bayesian neural network properties

Proofs of Section 3.

###### Lemma 3.1

Let $X_{1},\dots,X_{N}$ be some possibly dependent random variables and
$W_{1},\dots,W_{N}$ be symmetric, mutually independent and independent from
$X_{1},\dots,X_{N}$, then random variables $X_{1}W_{1},\dots,X_{N}W_{N}$
satisfy the PD condition.

######  Proof C.1.

The joint probability for the right tail
$\mathbb{P}\left(\bigcap_{i=1}^{N}W_{i}X_{i}\geq z_{i}\right)$ can be
expressed as

$\int_{-\infty}^{\infty}\dots\int_{-\infty}^{\infty}\mathbb{P}\left(\bigcap_{i=1}^{N}W_{i}x_{i}\geq
z_{i}\Bigl{.}\Bigr{|}\bigcap_{j=1}^{N}X_{j}=x_{j}\right)f(x_{1},\dots,x_{N})\,\mathrm{d}x_{1}\dots\mathrm{d}x_{N}.$
(10)

Independence between $W_{i}$ and $X_{j}$ yields

$\mathbb{P}\left(\bigcap_{i=1}^{N}W_{i}x_{i}\geq
z_{i}\Bigl{.}\Bigr{|}\bigcap_{j=1}^{N}X_{j}=x_{j}\right)=\mathbb{P}\left(\bigcap_{i=1}^{N}W_{i}x_{i}\geq
z_{i}\right)=\prod_{i=1}^{N}\mathbb{P}\left(W_{i}x_{i}\geq z_{i}\right),$

where the last equality is due to the mutual independence of weights
$W_{1},\dots,W_{N}$. Let $z_{1}=\dots=z_{N-1}=0$ and $z_{N}=z$. If $x_{i}=0$,
the probability $\mathbb{P}\left(W_{i}x_{i}\geq 0\right)=1$. If $x_{i}\not=0$,
then, due to the symmetry of $W_{i}$, the probability
$\mathbb{P}\left(W_{i}x_{i}\geq 0\right)=\frac{1}{2}$. Thus, the following
lower bound holds:

$\mathbb{P}\left(W_{i}x_{i}\geq 0\right)\geq\frac{1}{2}\quad\text{for all
}i\in\\{1,\dots,N-1\\}.$

Notice that

$\displaystyle\int_{-\infty}^{\infty}\dots\int_{-\infty}^{\infty}\mathbb{P}\left(W_{N}x_{N}\geq
z\right)f(x_{1},\dots,x_{N})\,\mathrm{d}x_{1}\dots\mathrm{d}x_{N}=\mathbb{P}\left(W_{N}X_{N}\geq
z\right).$

Substituting the latter equations into Equation (10) leads to the lower bound:

$\mathbb{P}\left(\bigcap_{i=1}^{N-1}W_{i}X_{i}\geq 0,W_{N}X_{N}\geq
z\right)\geq\frac{1}{2^{N-1}}\mathbb{P}\left(W_{N}X_{N}\geq z\right).$

By the conditional probability definition, we have

$\mathbb{P}\left(\bigcap_{i=1}^{N-1}W_{i}X_{i}\geq
0\Bigl{.}\Bigr{|}W_{N}X_{N}\geq
z\right)=\frac{\mathbb{P}\left(\bigcap_{i=1}^{N}W_{i}X_{i}\geq
0\right)}{\mathbb{P}\left(W_{N}X_{N}\geq z\right)}\geq\frac{1}{2^{N-1}}.$

The proof for the left tail is identical.

###### Theorem 1

Consider a Bayesian neural network as described in Equation (2) with ReLU
activation function. Let $\ell$-th layer weights be independent symmetric
generalized Weibull-tail on $\mathbb{R}$ with tail parameter
$\beta^{(\ell)}_{w}$. Then, $\ell$-th layer pre-activations are generalized
Weibull-tail on $\mathbb{R}$ with tail parameter $\beta^{(\ell)}$ such that
$\frac{1}{\beta^{(\ell)}}=\frac{1}{\beta^{(1)}_{w}}+\dots+\frac{1}{\beta^{(\ell)}_{w}}$.

######  Proof C.2.

The goal is to show that
$h_{j}^{(\ell)}\sim\text{GWT}_{\mathbb{R}_{+}}\left(\beta^{(\ell)}\right)$
where
$\frac{1}{\beta^{(\ell)}}=\frac{1}{\beta^{(1)}_{w}}+\dots+\frac{1}{\beta^{(\ell)}_{w}}$.
We proceed by induction on the layer depth $\ell$.

  1. (i)

First hidden layer

For $\ell=1$, by Lemma 3.1, products $w_{ij}^{(1)}h_{i}^{(0)}$ for
$i=1,\dots,H_{1}$ satisfy the positive dependence condition of Definition 2.2,
thus the sum $g_{j}^{(1)}=\sum_{i}w_{ij}^{(1)}h_{i}^{(0)}$ is generalized
Weibull-tail on $\mathbb{R}$ with tail parameter
$\beta^{(1)}=\beta^{(1)}_{w}$. Since ReLU function does not change the right
tail and post-activations lie completely on $\mathbb{R}_{+}$, we have
$h_{j}^{(1)}\sim\text{GWT}_{\mathbb{R}_{+}}(\beta^{(1)})$.

  2. (ii)

Step of induction

Let
$h_{j}^{(\ell-1)}\sim\text{GWT}_{\mathbb{R}_{+}}\left(\beta^{(\ell-1)}\right)$
where
$\frac{1}{\beta^{(\ell-1)}}=\frac{1}{\beta^{(1)}_{w}}+\dots+\frac{1}{\beta^{(\ell-1)}_{w}}$.
Since $h_{j}^{(\ell-1)}$ is non-negative and $w_{ij}^{(\ell)}$ are symmetric,
their product is symmetric by symmetry of the weights:
$h_{j}^{(\ell-1)}w_{ij}^{(\ell)}$ has the same distribution as
$h_{j}^{(\ell-1)}\left(-w_{ij}^{(\ell)}\right)=-h_{j}^{(\ell-1)}w_{ij}^{(\ell)}$.
By Theorem 3 and independence of random variables $h_{j}^{(\ell-1)}$ and
$w_{ij}^{(\ell)}$, we obtain that
$h_{j}^{(\ell-1)}w_{ij}^{(\ell)}\sim\text{GWT}_{\mathbb{R}}(\beta^{(\ell)})$
where
$\frac{1}{\beta^{(\ell)}}=\frac{1}{\beta^{(\ell-1)}}+\frac{1}{\beta^{(\ell)}_{w}}=\frac{1}{\beta^{(1)}_{w}}+\dots+\frac{1}{\beta^{(\ell-1)}_{w}}+\frac{1}{\beta^{(\ell)}_{w}}$.

According to Lemma 3.1, $w_{ij}^{(\ell)}h_{i}^{(\ell-1)}$,
$i=1,\dots,H_{\ell}$ satisfy the dependence condition of Definition 2.2. Thus,
the sum $g_{j}^{(\ell)}=\sum_{i=1}^{H_{\ell}}w_{ij}^{(\ell)}h_{i}^{(\ell-1)}$
is symmetric generalized Weibull-tail with the same tail parameter. The ReLU
activation does not impact the right tail, therefore,
$h_{j}^{(\ell)}\sim\text{GWT}_{\mathbb{R}_{+}}(\beta^{(\ell)})$.