11institutetext: Institute for AI in Medicine (IKIM), University Medicine
Essen 22institutetext: University of Duisburg-Essen, Germany 33institutetext:
University of Marburg, Germany 44institutetext: University of Mannheim,
Germany
44email<EMAIL_ADDRESS>
44email<EMAIL_ADDRESS>
# A Second Look on BASS – Boosting Abstractive Summarization with Unified
Semantic Graphs

Osman Alperen Koraş and Jörg Schlötterer and Christin Seifert 1122
0009-0006-6490-3139 3344 0000-0002-3678-0390 33 0000-0002-6776-3868

###### Abstract

We present a detailed replication study of the BASS framework, an abstractive
summarization system based on the notion of Unified Semantic Graphs. Our
investigation includes challenges in replicating key components and an
ablation study to systematically isolate error sources rooted in replicating
novel components. Our findings reveal discrepancies in performance compared to
the original work. We highlight the significance of paying careful attention
even to reasonably omitted details for replicating advanced frameworks like
BASS, and emphasize key practices for writing replicable papers.

###### Keywords:

Replication Abstractive Summarization Graph-Enhanced Transformer

##  1 Introduction

The goal of automatic text summarization is to generate a fluent, concise,
informative, and faithful summary of source documents [10]. Extractive
summarization systems select salient phrases from the source document and
concatenate them to form the summary. In contrast, abstractive summarization
systems freely generate text conditioned on an intermediate representation of
the source document [10]. Consequently, the capabilities of abstractive
summarization systems depend on the richness of this intermediate
representation.

Many state-of-the-art abstractive summarization systems are based on Pre-
trained Language Models (PLM), such as BERT [4], PEGASUS [34], or T5 [25]. And
the success of transformers [27] across many domains shows that they are
capable of generating rich representations for a wide range of signals,
including vision [6], audio [5] and graphs [33]. The Graphormer [33] is one of
many Attentive Graph Neural Networks [28, 3, 33], which have been successfully
adapted for transformers to leverage graphs in abstractive summarization
systems [20, 24, 36, 15, 7, 32, 16, 17, 11], with the aim to complement or
guide the rich representation of transformers with explicitly structured data
to improve accuracy and faithfulness of the generated summaries.

One of the graph-enhanced transformer models is BASS [30], which is of
specific interest because i) it introduces a compressed dependency graph
structure based on the idea of semantic units and ii) the authors report
competitive performance in abstractive summarization while being only half the
size (201M parameters for BASS vs. 406M parameters for BART [19] and PEGASUS).

Because the original paper [30] is not accompanied by source code, we conduct
a replication study of the BASS framework. Our results contribute to the
broader discourse surrounding reproducibility concerns of Machine Learning
[12, 14] and in particular NLP research, sometimes even referred to as the
“reproducibility crisis” [1, 2]. Belz et. al report that fewer than 15% scores
of their study were reproducible, and that “worryingly small differences in
code have been found to result in big differences in performance” [1]. Even
for performance scores reproduced under the same conditions, they discovered
that almost 60% of reproduced scores were worse than the original score.
Consequently, results from different works have to be compared with caution,
even if similar components are employed, drawing attention to the importance
of generating or replicating own baselines for meaningful comparisons and
drawing conclusions. Concretely, the contributions of this paper are the
following:

  1. 1.

We conduct a replication study of the BASS framework and publish our
implementation111https://github.com/osmalpkoras/bass-replication , including
source code for the graph construction component provided by the authors of
the original paper.

  2. 2.

We conduct an ablation study to examine BASS’ architectural adaptations to
incorporate the graph information into transformers and find we can not
replicate the performance improvement on the summarization task.

  3. 3.

We detail our replication for each framework component, and summarize the
specific and general challenges we faced during replication.

##  2 Replication Methodology

Our initial goal was to implement the BASS framework (cf. Fig. 1) in Python
one component at a time, solely from information available to the community,
i.e., the paper. We started by implementing the pre-processing
\raisebox{-.9pt} {1}⃝ and graph construction \raisebox{-.9pt} {2}⃝, but
quickly identified missing information and uncertainties (see Sec. 5).

When key information was missing on a component, we inquired the authors via
email for missing details and source code. When met with uncertainties, we
contacted the authors only when we were not certain to have faithfully
replicated the component. We exchanged multiple emails with over 20 questions
out of which roughly three quarters have been answered. On average, they
responded to questions within 10 days. In the end, the authors provided a Java
implementation for the graph construction \raisebox{-.9pt} {2}⃝ and snippets
for pre-processing \raisebox{-.9pt} {1}⃝. But we were not provided with
information about the batch size for example, despite multiple inquiries.

With the additional information, we found some inconsistencies between author
information, paper details and source code (see Sec. 3 and Appendix 0.A). We
let details provided in the paper take precedence over implementation details
of the Java source code, and let the source code take precedence over our
correspondence with the authors. For details that still remained unclear, we
made a best guess.

A summary of our replication is shown in Tab. 1, where we indicate per
component which information sources we used, and whether uncertainties
remained. We implemented the pre-processing \raisebox{-.9pt} {1}⃝ in Java to
use the authors’ source code for graph construction \raisebox{-.9pt} {2}⃝a,
but additionally replicated the graph construction \raisebox{-.9pt} {2}⃝b in
Python. All other components \raisebox{-.9pt} {3}⃝ – \raisebox{-.9pt} {9}⃝ are
replicated in Python only. To switch between programming languages, we save
the pre-processing and graph construction output as needed for subsequent
computation in Python.

Figure 1: Illustration of the BASS framework. The pre-processing and graph construction is done on the input document (left). The resulting graph information is used for token-to-node alignment \raisebox{-.9pt} {5}⃝, the graph encoder \raisebox{-.9pt} {6}⃝ and the respective cross-attention module \raisebox{-.9pt} {7}⃝ in the decoder. Table 1: Overview of the completeness of information on components (cf. Fig. 1) based on community-available information, i.e., the paper, and which information Sources we actually used. Paper shows whether key information was missing, making a replication impossible ($\times$), whether minor details were omitted (\raisebox{-.9pt} {}⃝) or whether all required information on a component is complete (✓). Complete shows whether we are certain to have faithfully replicated a component (✓) or if uncertainty remained (\raisebox{-.9pt} {}⃝). n.a.: not applicable to graph construction \raisebox{-.9pt} {2}⃝a, as we use the provided source code as it is. Component | Paper | Sources | Complete
---|---|---|---
\raisebox{-.9pt} {1}⃝ Pre-processing | $\times$ | Paper, Source Code, Authors | ✓
\raisebox{-.9pt} {2}⃝a Graph Construction (provided) | n.a. | Source Code | n.a.
\raisebox{-.9pt} {2}⃝b Graph Construction (replicated) | $\times$ | Paper, Source Code, Authors | \raisebox{-.9pt} {}⃝
\raisebox{-.9pt} {3}⃝ Graph Augmentation | ✓ | Paper | ✓
\raisebox{-.9pt} {4}⃝ Tokenizer & Text Encoder | \raisebox{-.9pt} {}⃝ | Paper, Authors | ✓
\raisebox{-.9pt} {5}⃝ Token To Node Alignment | \raisebox{-.9pt} {}⃝ | Paper | ✓
\raisebox{-.9pt} {6}⃝ – \raisebox{-.9pt} {9}⃝ Model Architecture | \raisebox{-.9pt} {}⃝ | Paper, Authors | \raisebox{-.9pt} {}⃝

##  3 Replicating the BASS framework

BASS is an abstractive summarization framework (cf. Fig. 1), which uses i)
dependency parse trees to generate Unified Semantic Graphs (USG) for documents
to compress and relate information across the input document, and ii) a model
architecture, which incorporates the graph information. As this work focuses
on the replication of the framework, we refer to the original work [30] for
it’s details and only elaborate on necessary complementary information in the
following.

##### Pre-processing \raisebox{-.9pt} {1}⃝.

An input document is passed to a linguistic parser for POS tagging, co-
reference resolution and dependency parsing. We used the latest CoreNLP
library [22] (v4.5.2) and had the authors confirm a configuration we found in
their source code, as no details were given in the paper, that is:

⬇

”annotators”: ”tokenize, ssplit, pos, lemma, ner, parse, coref”,

”coref.algorithm”: ”neural”, ”depparse.extradependencies”: ”MAXIMAL”

We were initially unable to pre-process many documents (over 20%) due to
endless runtimes or out-of-memory errors of the CoreNLP parser. Upon inquiry,
the authors’ confirmed to have used the pre-processing strategy indicated in
their source code, so we chunk source documents into blocks of sentences with
approx. 500 words and pre-process them separately. The resulting graphs per
chunk are concatenated into a single document-level graph consisting of
multiple disjoint sub-graphs. In particular, nodes across different sub-graphs
(e.g., those referring to the same entity) are neither merged nor connected.
We additionally set a maximum runtime (of up to 10 hours) and RAM consumption
(of up to 90GB) for pre-processing a single chunk. The output is one
dependency parse tree per sentence, co-reference chains across each chunk and
a POS tag for each word.

##### Graph Construction \raisebox{-.9pt} {2}⃝.

To construct USGs, the dependency trees of all sentences are viewed as
directed graphs with one node representing a single word, paired with its POS
tag and an edge representing the respective dependency relation. Nodes are
merged in multiple steps as follows: i) the nodes of entity mentions from co-
reference resolution are merged into a single entity node, ii) nodes are
removed or merged based on linguistic rules reasoning over the dependency and
POS annotations, iii) entity nodes of mentions in the same co-reference chain
are merged, iv) non-entity nodes representing the same phrase (exact lexical
match) are merged.

We noticed strictly complying with the paper [30] yields a different algorithm
than the one provided with the Java source code (Appendix 0.A). Therefore, we
consider two USG variants in this work: i) USGsrc generated by original Java
implementation \raisebox{-.9pt} {2}⃝a and ii) USGppr generated by our paper-
compliant replication of the graph construction \raisebox{-.9pt} {2}⃝b in
Python. Since the linguistic rules in step iii) are omitted from the paper, we
adopt these from the Java code.

##### Graph Augmentation \raisebox{-.9pt} {3}⃝.

Following the paper, the USG is augmented by adding self-loops and reverse
edges. All nodes are connected with their two-hop neighbours and a supernode
connecting to all nodes is added. The output of this step is two matrices, the
graph construction matrix $C$ (solid orange line) and the adjacency matrix $A$
(dashed orange line). See Appendix 0.B for more details.

##### Text Encoder and Tokenizer \raisebox{-.9pt} {4}⃝.

Following our correspondence with the authors, we use the pre-trained RoBERTa-
base model [37] for the text encoder with the tokenizer RobertaTokenizerFast
from Hugging Face’s transformer library ([29], v4.26.1). The context length is
extended to $N=1024$ by randomly initializing the extended part.

##### Aligning Text to Graph Embeddings \raisebox{-.9pt} {5}⃝.

We align text embeddings with graph embeddings by multiplying the text encoder
output with the graph construction matrix from step \raisebox{-.9pt} {3}⃝. We
refer to Appendix 0.B for more details.

##### Model Architecture \raisebox{-.9pt} {6}⃝ – \raisebox{-.9pt} {9}⃝.

The model architecture builds upon a standard transformer encoder-decoder
architecture for abstractive summarization, complemented with three additional
components: i) a graph encoder \raisebox{-.9pt} {6}⃝, which is a standard two-
layered transformer encoder using the adjacency matrix of the graph as
attention mask, ii) a corresponding multi-head attention module
\raisebox{-.9pt} {7}⃝ in a six-layered decoder attending over the graph
encoder output and iii) a fully connected linear layer \raisebox{-.9pt} {8}⃝
to fuse graph and text information, followed by a Residual Dropout [27] layer
\raisebox{-.9pt} {9}⃝.

The Causal Self Attention, i.e., masked self attention attending to left
context only, and Feed-Forward [27] layers are also followed by a Residual
Dropout layer, not indicated in the figure. In contrast to regular cross-
attention modules, the attention weights in \raisebox{-.9pt} {7}⃝ are
propagated using PageRank and the augmented adjacency matrix to leverage the
graph structure. We refer to Appendix 0.C for more details on our
implementation. In the end, our model ended up having approx. 205M trainable
parameters, which is around 2% larger than reported in the BASS paper [30]
(201M parameters), implying architectural differences we could not entirely
resolve.

##  4 Evaluation

We conduct a replication study by training one model each for the author-
provided graphs USGsrc and the paper-compliant graphs USGppr (cf. Sec. 3) to
better isolate potential errors rooted in replicating the model architecture
from potential errors rooted in replicating the graph construction. Since
replication studies are known to be challenging due to an overwhelming number
of error sources, we follow up with an ablation study where we generate our
own baselines (not using graphs) to measure the impact of architectural
adaptations.

##### Datasets.

We follow [30] and select the BigPatent [26] dataset for Single Document
Summarization. However, we were only able to pre-process $99.79\%$ documents,
resulting in 1,204,631 documents for training and 66,962 documents each for
validation and test. Using both graph construction methods (cf. Sec. 3) we
generate two sets of graphs USGsrc and USGppr and obtain two respective
training datasets. Running the pre-processing and graph construction
\raisebox{-.9pt} {2}⃝a on the Big Patent dataset took about four days on our
cluster with 900 CPU cores (Intel(R) Xeon(R) Silver 4216 CPU) with an
aggregated runtime of 3,360 days $\pm$ 1 day.

##### Models.

We use the following baselines for our studies (cf. [30, Tab.3] for original
results): TransS2S, which is a standard text-only transformer encoder-decoder
model [27], RoBERTaS2S, which differs from TransS2S by using RoBERTa-base for
the encoder and decoder, and BASS, which is the original model we replicate.

For our replication study, we train the following models: BASSours/src, which
is our replicated BASS model trained with USGsrc, and BASSours/ppr, which is a
full replication of the original paper differing from BASSours/src only by
it’s use of USGppr graphs.

For the ablation study, we additionally train two transformer based encoder-
decoder architectures on the BigPatent dataset without any graphs: RTS2S,
which is a text-only model consisting of the text encoder \raisebox{-.9pt}
{4}⃝ and a standard 6-layered decoder without any graph components (i.e.,
omitting \raisebox{-.9pt} {3}⃝,\raisebox{-.9pt} {5}⃝ – \raisebox{-.9pt} {8}⃝).
And exRTS2S, which extends RTS2S by the graph encoder and the decoder
components \raisebox{-.9pt} {6}⃝ – \raisebox{-.9pt} {8}⃝ without informative
graph structure (\raisebox{-.9pt} {3}⃝,\raisebox{-.9pt} {5}⃝), and by
replacing Graph-prop Attention \raisebox{-.9pt} {7}⃝ with normal Context
Attention.

RTS2S is most similar to TransS2S and RoBERTaS2S, using the encoder of
RoBERTaS2S and the decoder of TransS2S. We therefore expect this model to
perform somewhere in-between. exRTS2S is most similar to BASSours and differs
only in the lack of graph structure: every token is considered a graph node
connected to every other node. Hence layer \raisebox{-.9pt} {5}⃝ is skipped
and no graph structure is injected into the attention mechanism in the graph-
encoder \raisebox{-.9pt} {6}⃝ and no attention is graph-propagated in the
cross-attention module \raisebox{-.9pt} {7}⃝.

We want to measure the impact of all architectural adaptations proposed for
BASS with RTS2S as our baseline. With exRTS2S as a baseline, we further
isolate the impact of the USGsrc structure from the impact of increasing model
size. We choose USGsrc assuming the graph construction method \raisebox{-.9pt}
{2}⃝a reproduces the graphs from the original work.

##### Training Details.

We use the same training and hyper-parameter setup as the original work [30,
§5.2] for all models, which uses the learning rate schedule of Liu and Lapata
[21]. Each model is trained once for 300,000 steps. Since we were not able to
find out the batch size used in the original work, we use a batch size of 48
per step, which is the largest possible value on our hardware (6 RTX A6000
GPUs with a total RAM of 288GB – the original authors reported the use of
8xV100, of which the latest version totals 256GB). Training the models for
10,000 steps took about 4 hours $\pm$ 30 minutes at average on our machines
(cf. Tab. 2).

##### Evaluation.

We evaluate the models on the test set and apply beam search [31] with trigram
blocking [23] for decoding using a beam size of $5$ and a length penalty of
$0.9$. We enforce a maximum decoding length of 1024 and report ROUGE scores
[13] R-1, R-2, sentence-level R-L, summary-level R-Lsum and $F_{1}$ BERTScore
[35]222lang="en" model_type="roberta-large" rescale_with_baseline=True BS. We
exclude summaries, for which no eos token has been generated during decoding
from evaluation and use paired bootstrap resampling [9] with $p=0.05$ for
significance testing.

###  4.1 Experimental Results

##### Replication.

Comparing our replication with the original (BASS), we score more than 4
points lower than reported originally,4 performing even worse than the
RoBERTaS2S baseline. Since the training batch size of the original work
remains unknown, we investigated if our models might be undertrained by
extending the training of BASSours/src for another 300,000 steps333We did not
train further, because we already doubled the computational budget used for
the original paper. (cf. Tab. 2) and are indeed able to raise the scores but
only by about $0.7\pm 0.2$ points.

Seeing how the graph construction implementation \raisebox{-.9pt} {2}⃝a
provided to us differs algorithmically from our paper-compliant implementation
\raisebox{-.9pt} {2}⃝b, we also compare USGsrc and USGppr graphs (cf. Tab. 3).
The replicated graphs are slightly larger in size. They also cover at least
30% more tokens and while we expected BASSours/ppr to perform better for this
reason, we actually observe mixed results: a small increase in the BERTscore,
and a decrease in ROUGE scores.

Table 2: Evaluation of our models on the BigPatent dataset. The baselines are all taken from prior work [30] and best scores are in bold. All scores are pairwise significantly different from each other, except those indicated by $\dagger$. | R-1 | R-2 | R-L | R-Lsum | BS | params | time
---|---|---|---|---|---|---|---
Baselines and Paper
TransS2S [30] | 34.93 | 9.86 | 29.92444While the BASS paper reports sentence-level R-L scores, they systematically match better with our summary-level R-Lsum scores, which may indicate that prior results are actually R-Lsum scores. Hence we place the scores reported by BASS between columns and compare them with our R-Lsum scores. | 9.42 | n.a. | n.a.
RoBERTaS2S [30] | 43.62 | 18.62 | 37.864 | 18.18 | n.a. | n.a.
BASS [30] | 45.04 | 20.32 | 39.214 | 20.13 | 201M | n.a.
Replication Study
BASSours/src | 40.52 | 14.97 | 26.64 | 34.67 | 16.15 | 204.6M | 5d 15h
BASSours/src 600k | 41.23 | 15.62 | 27.18 | 35.37 | 17.03 | 204.6M | 11d 11h
BASSours/ppr | 39.30 | 14.65 | 26.31 | 33.74 | 16.51 | 204.6M | 5d 16h
Ablation Study
RTS2S | 39.75 | 14.84† | 26.48 | 34.04 | 16.34 | 172.4M | 4d 9h
exRTS2S | 39.62 | 14.86† | 26.56 | 33.91 | 16.63 | 204.6M | 5d 7h
Table 3: Comparison of USGsrc and USGppr structures for the BigPatent dataset $D=\left\\{d_{i}\right\\}_{i\in I}$, with $d_{i}$ denoting a tokenized input document. We consider the subsets $D_{T}=\left\\{d_{i}\mid\left|T-t(d_{i})\right|\leq 20\right\\}$ for token count $t(d_{i})$ and $T\in\left\\{400,600,800,1000\right\\}$. Let $\overline{t}_{D_{T}}$ denote the average token count of documents $d\in D_{T}$ and $\left|D_{T}\right|$ the cardinality of $D_{T}$. We report the average node count $\overline{n}$, average edge count $\overline{e}$ as well as the average count of tokens $\overline{t}_{c}$ covered by the graphs generated for $d\in D_{T}$. The bottom row shows the increase in respective quantities for USGppr w.r.t USGsrc $\overline{t}_{D_{T}}$ |  | 400 |  |  | 600 |  |  | 800 |  |  | 1000 |
---|---|---|---|---|---|---|---|---|---|---|---|---
$\left|D_{T}\right|$ |  | 152 |  |  | 966 |  |  | 3045 |  |  | 6087 |
| $\overline{n}$ | $\overline{e}$ | $\overline{t}_{c}$ | $\overline{n}$ | $\overline{e}$ | $\overline{t}_{c}$ | $\overline{n}$ | $\overline{e}$ | $\overline{t}_{c}$ | $\overline{n}$ | $\overline{e}$ | $\overline{t}_{c}$
USGsrc | 117 | 142 | 232 | 168 | 211 | 340 | 227 | 281 | 463 | 278 | 349 | 569
USGppr | 129 | 156 | 301 | 185 | 233 | 453 | 240 | 308 | 606 | 293 | 385 | 759
Increase | 10% | 10% | 30% | 10% | 10% | 33% | 6% | 10% | 31% | 5% | 10% | 33%

##### Ablation.

We observe slight but mostly significant differences in model performances
(cf. Tab. 2, Ablation Study), with no clear winner. However, our ablation
models perform in-between TransS2S and RoBERTaS2S, as expected. The
introduction of graph components (exRTS2S vs. RTS2S) mostly improves the
BERTScore with mixed results in terms of ROUGE. Further comparing exRTS2S with
our replicated models shows the impact of using USGs: using USGppr slightly
hurts performance overall, while using USGsrc slightly improves ROUGE scores
while hurting BERTScore.

###  4.2 Discussion

##### Replication.

Our replicated BASS models substantially fall short in performance, even below
baselines of the original work (cf. RoBERTaS2S in Tab.2). Since this is true
even for our model BASSours/src which is trained on USGsrc graphs, we mainly
attribute this to the model architecture, assuming that the provided graph
construction \raisebox{-.9pt} {2}⃝a is the same one used in the original work.
By further extending the training, we find that our models might be
undertrained, indicating that the original work might have used a larger
effective batch size to achieve their results.

Since the discrepancy between the replicated and the original performance of
BASS can be attributed to the model architecture, the impact of using USGppr
over USGsrc graphs on downstream performance being minor does not surprise,
despite substantial qualitative differences in graph structures. However, the
authors’ source code not complying with the paper, possibly having undergone
some changes (see Sec. 3 & 5), sheds doubts on whether the provided graph
construction \raisebox{-.9pt} {2}⃝a reproduces the graphs of the original work
and whether the mismatch in graph structures indicates a failed replication of
the original work.

##### Ablation.

Contrary to [30] we did not find substantial performance gains, neither in
model adaptations (increased model size, cf. Tab. 2, RTS2S vs. exRTS2S), nor
in USGs (additional structured information, cf. Tab. 2, exRTS2S vs.
BASSours/src and BASSours/ppr). This is shown by the minor impact the model
adaptations and USGs have on our baselines. Our replicated USGppr graphs even
consistently hurt the performance overall. Nonetheless the comparison with
previous baselines (TransS2S and RoBERTaS2S) shows that RTS2S performs
reasonably well. We therefore ascribe the lack of substantial gains in our
replicated models solely to BASS’ model adaptations and graph information not
being as effective as expected.

##  5 Replication Challenges and Recommendations

In this section, we reflect on the replication process and highlight the main
challenges we encountered, hoping to sensitize readers to the underlying
issues that compromise the replicability of research papers. We conclude by
recommending key practices for writing replicable papers, that would have
significantly helped us with the replication.

##### Self-Explanatory Details.

Some details are omitted from papers usually for good reasons: being straight-
forward, well-known or trivial. However, our experience showed that the leeway
in implementation choices for omitted details (e.g. in every step of the graph
construction, or for aligning the tokenizations of CoreNLP’s tokenizer
\raisebox{-.9pt} {1}⃝ and RoBERTa’s tokenizer \raisebox{-.9pt} {4}⃝, but also
for the pre-processing strategy) entails ambiguities and consequently an
avalanche of potential error sources and mitigation strategies to resolve
them. Hence, these omitted details can make the difference between an accurate
replication and an endless errand to fix errors. Although we got many details
confirmed by the authors, studied the source code and strictly followed the
paper, some uncertainties remained (cf. Tab. 1). In the end, we were neither
able to pre-process the entire dataset considered in this work (see Sec. 4)
nor achieve the same model size as the original authors (see Sec. 3).

##### Missing Third Party Information.

One problem was missing version information and configuration of third-party
components, i.e., of the CoreNLP pipeline, the RoBERTa model and the
tokenizers. We were able to resolve most of these issues through
correspondence with the authors for this paper.

##### Missing Key Information.

Overall, we encountered many details that required additional information or
clarification beyond the paper. However, not all missing details fundamentally
obstructed our replication: our first attempt to replicate the paper (see Sec.
2) failed primarily due to the omission of the linguistic rules reasoning over
the dependency trees for creating USGs. Nevertheless, following our
correspondence with the authors and access to source code, we identified and
resolved many misunderstandings.

##### Algorithmic Complexity and Error-Proneness.

A thorough analysis of the original source code was necessary to fix (or not
to fix) our replicated graph construction algorithm due to the many errors we
encountered during runtime, often rooted in erroneous annotation results, such
as wrong POS tag annotation, co-reference resolution, or even dependency
graphs being rooted in punctuation tokens or sentences mistakenly being split
at decimal points. To our surprise, the provided graph construction slightly
differs from the description in the paper. This is likely because the source
code has been used in other projects, as noted by the authors, and
consequently might have undergone some changes before or after the paper’s
publication, emphasizing the importance of version control systems. The
complexity of algorithms, whereas, can be lessened using tools during
development to analyze and reduce cognitive complexity in software.

##### Recommendations.

We found the mathematical and algorithmic descriptions (notation, equations,
pseudo-code) most helpful along the way, allowing us to consolidate many
misconceptions. Therefore we emphasize the importance of i) providing a clear
and complete technical context, ii) a clear and (given the context) complete
notation, iii) technical and mathematical precision particularly for
describing how different components (novel or not) interleave, and iv)
commented pseudo-code. We feel the latter can often replace a detailed
description of an algorithm, while being shorter and less ambiguous.

We also strongly encourage to use technical terms coined by prior work
wherever applicable, such as “Residual Dropout” [27] for layer
\raisebox{-.9pt} {9}⃝, instead of short descriptions of well-known components.
The latter can be inaccurate and leave readers questioning potential
differences and misunderstandings in case of failed replications, while
undermining the development of a well-defined and well-known terminology of a
research domain.

##  6 Conclusion

We started implementing the BASS framework based on the paper, but found that
most components were not sufficiently described (cf. Tab. 1). Some
uncertainties persisted even after our correspondence with the authors, and
the examination of the provided source code (cf. Sec. 2), partly because some
inquiries (e.g. for the training batch size) had been left pending. On one
hand, the provided graph construction \raisebox{-.9pt} {2}⃝a did not align
with the paper (cf. Sec. 3). On the other hand, our model’s parameter size was
larger by approx 3.6M parameters than reported in the original work (cf. Sec.
3). Therefore, it is unsurprising we could neither replicate prior results of
BASS on the BigPatent dataset, nor clear performance improvements on the
summarization task as a result of the novel adaptations proposed for BASS (cf.
Sec. 4.2). Assuming the graph construction method \raisebox{-.9pt} {2}⃝a
provided by the original authors’ reproduces the same USGs as in the original
work, our results indicate the poor performance can be ascribed to the model
architecture, which might, in addition, be undertrained. However, some doubts
remain on whether the provided graph construction method even reproduces the
original USGs.

Moreover, we found the pre-processing \raisebox{-.9pt} {1}⃝ and in extension
the graph construction \raisebox{-.9pt} {2}⃝ to be very error-prone and time
consuming. Parsing one document of the BigPatent dataset with approx. 1,000
tokens took us about 3.5 minutes, not accounting for the 2,811 documents
(approx. 0.2%) we had to exclude for not being parseable in less than 10
hours. Additionally, erroneous dependency annotations make it difficult to
construct USGs, leading to fractured graphs, isolated nodes or deletion of
salient information. Based on our experiences, we suggest investigating the
use of simpler semantic dependency parsing methods [8] which reportedly are
more accurate, or to move away from systems that construct graphs from
semantic annotations based on manually hand-crafted rules.

Overall, the replication was complicated by missing third-party information,
the ambiguity of self-explanatory details, and omission of some key
information (the latter requiring us to contact the authors for a faithful
replication), despite the fact that the original paper is very detailed and
comprehensive, representative of the high quality of the venue it was
published on (ACL’21). However, our experience shows that the way information
is detailed is just as vital as being comprehensive. Furthermore, as this
lesson is learned only after attempting a replication, it may lead reviewers,
who lack similar experiences, to overrated reproducibility assessments. We
have therefore emphasized key issues and practices for replicable papers and
recommend supplementing reproducibility as well as reviewer checklists with a
corresponding section to address these problems.

####  6.0.1 Acknowledgements

We thank the authors [30] for their correspondence, their source code and for
giving us their consent to share it. Funding for this work was provided by the
German State Ministry of Culture and Science NRW, for research under the
Cancer Research Center Cologne Essen (CCCE) foundation. The funding was not
provided specifically for this project.

## References

  * [1] Belz, A., Agarwal, S., Shimorina, A., Reiter, E.: A systematic review of reproducibility research in natural language processing. In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. pp. 381–393. Association for Computational Linguistics, Online (Apr 2021). https://doi.org/10.18653/v1/2021.eacl-main.29
  * [2] Belz, A., Thomson, C., Reiter, E., Mille, S.: Non-repeatable experiments and non-reproducible results: The reproducibility crisis in human evaluation in NLP. In: Findings of the Association for Computational Linguistics: ACL 2023. pp. 3676–3687. Association for Computational Linguistics, Toronto, Canada (Jul 2023). https://doi.org/10.18653/v1/2023.findings-acl.226, https://aclanthology.org/2023.findings-acl.226
  * [3] Brody, S., Alon, U., Yahav, E.: How attentive are graph attention networks? ArXiv abs/2105.14491 (2021)
  * [4] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp. 4171–4186. Association for Computational Linguistics, Minneapolis, Minnesota (Jun 2019). https://doi.org/10.18653/v1/N19-1423
  * [5] Dong, L., Xu, S., Xu, B.: Speech-transformer: A no-recurrence sequence-to-sequence model for speech recognition. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 5884–5888 (2018). https://doi.org/10.1109/ICASSP.2018.8462506
  * [6] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. ArXiv abs/2010.11929 (2020)
  * [7] Dou, Z.Y., Liu, P., Hayashi, H., Jiang, Z., Neubig, G.: GSum: A general framework for guided neural abstractive summarization. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 4830–4842. Association for Computational Linguistics, Online (Jun 2021). https://doi.org/10.18653/v1/2021.naacl-main.384
  * [8] Dozat, T., Manning, C.D.: Simpler but more accurate semantic dependency parsing. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 484–490. Association for Computational Linguistics, Melbourne, Australia (Jul 2018). https://doi.org/10.18653/v1/P18-2077
  * [9] Dror, R., Baumer, G., Shlomov, S., Reichart, R.: The hitchhiker’s guide to testing statistical significance in natural language processing. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1383–1392. Association for Computational Linguistics, Melbourne, Australia (Jul 2018). https://doi.org/10.18653/v1/P18-1128
  * [10] El-Kassas, W.S., Salama, C.R., Rafea, A.A., Mohamed, H.K.: Automatic text summarization: A comprehensive survey. Expert Systems with Applications 165, 113679 (2021). https://doi.org/10.1016/j.eswa.2020.113679
  * [11] Fan, A., Gardent, C., Braud, C., Bordes, A.: Using local knowledge graph construction to scale Seq2Seq models to multi-document inputs. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 4186–4196. Association for Computational Linguistics, Hong Kong, China (Nov 2019). https://doi.org/10.18653/v1/D19-1428
  * [12] Gibney, E.: Could machine learning fuel a reproducibility crisis in science? Nature 608, 250 – 251 (2022), https://api.semanticscholar.org/CorpusID:251102207
  * [13] Google LLC: rouge-score, https://pypi.org/project/rouge-score
  * [14] Gundersen, O.E., Coakley, K., Kirkpatrick, C.R.: Sources of irreproducibility in machine learning: A review. ArXiv abs/2204.07610 (2022), https://api.semanticscholar.org/CorpusID:248227686
  * [15] Hu, J., Li, J., Chen, Z., Shen, Y., Song, Y., Wan, X., Chang, T.H.: Word graph guided summarization for radiology findings. In: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. pp. 4980–4990. Association for Computational Linguistics, Online (Aug 2021). https://doi.org/10.18653/v1/2021.findings-acl.441
  * [16] Huang, L., Wu, L., Wang, L.: Knowledge graph-augmented abstractive summarization with semantic-driven cloze reward. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 5094–5107. Association for Computational Linguistics, Online (Jul 2020). https://doi.org/10.18653/v1/2020.acl-main.457
  * [17] Jin, H., Wang, T., Wan, X.: Semsum: Semantic dependency guided neural abstractive summarization. Proceedings of the AAAI Conference on Artificial Intelligence 34(05), 8026–8033 (Apr 2020). https://doi.org/10.1609/aaai.v34i05.6312, https://ojs.aaai.org/index.php/AAAI/article/view/6312
  * [18] Klicpera, J., Bojchevski, A., Günnemann, S.: Predict then propagate: Graph neural networks meet personalized pagerank. In: International Conference on Learning Representations (2018)
  * [19] Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., Zettlemoyer, L.: BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 7871–7880. Association for Computational Linguistics, Online (Jul 2020). https://doi.org/10.18653/v1/2020.acl-main.703
  * [20] Li, H., Peng, Q., Mou, X., Wang, Y., Zeng, Z., Bashir, M.F.: Abstractive financial news summarization via transformer-bilstm encoder and graph attention-based decoder. IEEE/ACM Transactions on Audio, Speech, and Language Processing 31, 3190–3205 (2023). https://doi.org/10.1109/TASLP.2023.3304473
  * [21] Liu, Y., Lapata, M.: Text summarization with pretrained encoders. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 3730–3740. Association for Computational Linguistics, Hong Kong, China (Nov 2019). https://doi.org/10.18653/v1/D19-1387
  * [22] Manning, C.D., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S.J., McClosky, D.: The Stanford CoreNLP natural language processing toolkit. In: Association for Computational Linguistics (ACL) System Demonstrations. pp. 55–60 (2014), http://www.aclweb.org/anthology/P/P14/P14-5010
  * [23] Paulus, R., Xiong, C., Socher, R.: A deep reinforced model for abstractive summarization (2017)
  * [24] Qi, P., Huang, Z., Sun, Y., Luo, H.: A knowledge graph-based abstractive model integrating semantic and structural information for summarizing chinese meetings. In: 2022 IEEE 25th International Conference on Computer Supported Cooperative Work in Design (CSCWD). pp. 746–751 (2022). https://doi.org/10.1109/CSCWD54268.2022.9776298
  * [25] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21(1) (jan 2020)
  * [26] Sharma, E., Li, C., Wang, L.: BIGPATENT: A large-scale dataset for abstractive and coherent summarization. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 2204–2213. Association for Computational Linguistics, Florence, Italy (Jul 2019). https://doi.org/10.18653/v1/P19-1212
  * [27] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. p. 6000–6010. NIPS’17, Curran Associates Inc., Red Hook, NY, USA (2017)
  * [28] Velickovic, P., Cucurull, G., Casanova, A., Romero, A., Lio’, P., Bengio, Y.: Graph attention networks. ArXiv abs/1710.10903 (2017)
  * [29] Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., Rush, A.: Transformers: State-of-the-art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. pp. 38–45. Association for Computational Linguistics, Online (Oct 2020). https://doi.org/10.18653/v1/2020.emnlp-demos.6
  * [30] Wu, W., Li, W., Xiao, X., Liu, J., Cao, Z., Li, S., Wu, H., Wang, H.: BASS: Boosting abstractive summarization with unified semantic graph. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 6052–6067. Association for Computational Linguistics, Online (Aug 2021). https://doi.org/10.18653/v1/2021.acl-long.472
  * [31] Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Kaiser, L., Gouws, S., Kato, Y., Kudo, T., Kazawa, H., Stevens, K., Kurian, G., Patil, N., Wang, W., Young, C., Smith, J., Riesa, J., Rudnick, A., Vinyals, O., Corrado, G., Hughes, M., Dean, J.: Google’s neural machine translation system: Bridging the gap between human and machine translation. CoRR abs/1609.08144 (2016), http://arxiv.org/abs/1609.08144
  * [32] Xu, J., Gan, Z., Cheng, Y., Liu, J.: Discourse-aware neural extractive text summarization. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 5021–5031. Association for Computational Linguistics, Online (Jul 2020). https://doi.org/10.18653/v1/2020.acl-main.451
  * [33] Ying, C., Cai, T., Luo, S., Zheng, S., Ke, G., He, D., Shen, Y., Liu, T.Y.: Do transformers really perform bad for graph representation? In: Neural Information Processing Systems (2021)
  * [34] Zhang, J., Zhao, Y., Saleh, M., Liu, P.J.: Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In: Proceedings of the 37th International Conference on Machine Learning. ICML’20, JMLR.org (2020)
  * [35] Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: Bertscore: Evaluating text generation with bert. In: International Conference on Learning Representations (2020), https://openreview.net/forum?id=SkeHuCVFDr
  * [36] Zhu, C., Hinthorn, W., Xu, R., Zeng, Q., Zeng, M., Huang, X., Jiang, M.: Enhancing factual consistency of abstractive summarization. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 718–733. Association for Computational Linguistics, Online (Jun 2021). https://doi.org/10.18653/v1/2021.naacl-main.58
  * [37] Zhuang, L., Wayne, L., Ya, S., Jun, Z.: A robustly optimized BERT pre-training approach with post-training. In: Proceedings of the 20th Chinese National Conference on Computational Linguistics. pp. 1218–1227. Chinese Information Processing Society of China, Huhhot, China (Aug 2021)

##  Appendix 0.A Discrepancies between USGsrc and USGppr

In the following, we point out algorithmic differences between the two graph
constructions methods \raisebox{-.9pt} {2}⃝a and \raisebox{-.9pt} {2}⃝b (cf.
Tab. 1). These differences arise from strictly complying with the paper for
the replicated method \raisebox{-.9pt} {2}⃝b. For the sake of simplicity, we
align our comparison with the pseudo-code in the appendix of the original work
[30].

##### REMOVE_PUNCTUATION

While \raisebox{-.9pt} {2}⃝b removes all tokens whose dependency relation
equals punct or whose POS tag is an element of $P=${., ,, :, !, ?, (, )},
\raisebox{-.9pt} {2}⃝a only removes tokens whose POS tag is an element of $P$.
In \raisebox{-.9pt} {2}⃝a, punctuation is removed recursively as part of the
MERGE_NODES routine.

##### MERGE_COREF_PHRASE

\raisebox{-.9pt} {2}⃝b merges all tokens of a co-reference mention into a
single node, while \raisebox{-.9pt} {2}⃝a i) uses merging rules not mentioned
in the paper and also ii) immediately merges the resulting nodes in the same
co-reference chain into a single node. \raisebox{-.9pt} {2}⃝b, on the other
hand, merges nodes in the same co-reference chain only in the MERGE_PHRASES
step.

##### MERGE_NODES

While \raisebox{-.9pt} {2}⃝a and \raisebox{-.9pt} {2}⃝b use exactly the same
rules to merge nodes, \raisebox{-.9pt} {2}⃝a traverses the dependency trees
pre-order depth-first. We intuitively chose to traverse post-order depth-first
for \raisebox{-.9pt} {2}⃝b without looking at the provided source code, as
working through the tree bottom up from leaf to root nodes generally complies
better with the intention to merge nodes (which includes rules to delete
nodes). For example, if \raisebox{-.9pt} {2}⃝a deletes a child, all
descendants are detached from the tree and never visited by the algorithm.

##### MERGE_PHRASES

\raisebox{-.9pt} {2}⃝b merges the nodes of mentions in the same co-reference
chain into a single node and later merges all nodes, whose phrases are equal
(exact lexical match). \raisebox{-.9pt} {2}⃝a, on the other hand, only does
the latter.

##  Appendix 0.B Aligning Graph and Text Embeddings

The Unified Semantic Graph imposes a graph structure on the token embeddings
returned by the text encoder. The text encoder output $t$ must therefore be
mapped to the graph encoder input $g$, before passing it to the graph encoder.
For this, we match tokens with nodes based on text characters from left to
right, as the CoreNLP tokenizer \raisebox{-.9pt} {1}⃝ is different from
RoBERTa’s tokenizer \raisebox{-.9pt} {4}⃝.

Let $G:=(V,E)$ be the augmented Unified Semantic Graph \raisebox{-.9pt} {3}⃝
with nodes $V:=\left\\{v_{i}\right\\}_{i=0}^{N_{V}}$ and edges
$E:=\left\\{e_{ij}\right\\}\subset V\times V$ and let
$S=\left\\{s_{v_{i}}\right\\}_{i=0}^{N_{V}}$ with
$s_{v_{j}}=\left\\{c_{i}\right\\}_{i\in I_{v_{j}}}$ be the set of characters
of the input document $D=\left\\{c_{i}\right\\}_{i=0}^{N_{D}}$ being
represented by node $v_{j}$. As a result of merging nodes across a document,
$s_{v_{j}}$ may consist of multiple disconnected character sequences.

We map nodes $v_{j}$ to tokens $t_{v_{j}}\subset T$, where $t_{v_{j}}$ is the
subset of input tokens $T=\left\\{t_{i}\right\\}_{i=0}^{N_{T}}$ associated
with at least one character $c_{i}\in s_{v_{j}}$. This gives us the graph
construction matrix $C=(c_{ij})\in\mathbb{R}^{N_{V}\times N_{T}}$ and the
adjacency matrix $A=(a_{ij})\in\mathbb{R}^{N_{V}\times N_{V}}$ with

$c_{ij}=\begin{cases}1,&\text{if }t_{j}\in t_{v_{i}}\\\
0,&\text{otherwise}\end{cases}$

$a_{ij}=\begin{cases}1,&\text{if }e_{ij}\in E\\\
0,&\text{otherwise.}\end{cases}$

Let $d_{model}$ be the dimension of token embeddings and
$t\in\mathbb{R}^{N_{T}\times d_{model}}$ be the output of the text encoder.
The graph encoder input $g$ is then given by $g:=C^{\prime}t$ where
$C^{\prime}$ is the degree normalized graph construction matrix $C$.
Multiplicating $t$ with $C^{\prime}$ is equal to calculating the
representation of node $v_{j}$ by averaging over the tokens $t_{v_{j}}$. The
matrix $g$ is then passed to the graph encoder alongside the node padding and
the adjacency matrix $A$ as attention mask.

##  Appendix 0.C Graph Propagation

The paper suggests propagating attention weights in the cross attention module
for the graph encoder using PageRank [18] and the adjacency matrix $A$ given
by the augmented Unified Semantic Graph. For this, we compute the graph
propagation matrix
$P=\omega^{p}\hat{A}+(1-\omega)(\sum_{i=0}^{p-1}\omega^{i}\hat{A}^{i})$, where
$p$ is the number of aggregation steps, $\omega$ is the teleport probability
and $\hat{A}$ is the degree normalized adjacency matrix $A$, including self
loops and reverse edges, supernode edges and shortcut edges. The graph
propagated attention weights are then computed as $\alpha^{\prime}=\alpha
P^{T}$, where $\alpha=(\alpha_{ij})\in\mathbb{R}^{N_{V}\times N_{V}}$ are the
attention weights of a single head in the multi-headed cross-attention module
for the graph encoder given by
$\alpha_{ij}=(y_{i}W_{Q})(v_{j}W_{K})^{T}/\sqrt{d_{k}}$, with query and key
projection weights $W_{Q},W_{K}$, the $i$-th token representation as query
$y_{i}$ and the $j$-th node representation as key $v_{j}$. Here, $d_{k}$
denotes the query and key dimensions.


The proof of \thex\ref{teor2classinprestacks} also gives us a recipe for the characteristic morphism $z\:F\to \t{\O}$ of a discrete opfibration $\phi\:G\to F$ in $\m{\C\op}{\Cat}$ with small fibres.
\sq[l][6][6][\h[-3]\v[2]\opn{comma}]{G}{1}{F}{\t{\O}}{}{\phi}{\t{\omega}}{z}
We obtain that $z$ is the 2-natural transformation whose component on $C\in \C$ is the functor $z_C$ that sends $X\in F(C)$ to
\begin{fun}
z_C(X) & \: & {\left(\slice{\C}{C}\right)}\op & \too & \Set \\[1ex]
&& (D\ar{f}C) & \mto & {\left(\phi_D\right)}_{F(f)(X)}\\[0.6ex]
&&(f \al{g} f\c g)&\mto& G(g)
\end{fun}
Given $\nu\:X\to X'$ in $F(C)$, we have that $z_C(\nu)$ is the natural transformation whose component on $f\:D\to C$ is the function
$${F(f)(\nu)}\stb\:{\left(\phi_D\right)}_{F(f)(X)}\to {\left(\phi_D\right)}_{F(f)(X')}$$
that calculates the codomain of the liftings along $\phi_D$ of $F(f)(\nu)$.

It is interesting to compare our result with what happens in dimension 1. The characteristic morphism for a subobject $G\ito F$ in $1$-dimensional presheaves has component on $C$ that sends $X\in F(C)$ to
$$\Union{D\in \C}{\set{D\ar{f}C}{F(f)(X)\in G(D)}}.$$
While in dimension 1 the fibre on $F(f)(X)$ can only be either empty or a singleton, in dimension 2 we need to handle the general sets formed by such fibres.
\end{rem}

\section{A 2-classifier in stacks}\label{sectiontwoclassinstacks}

In this section, we restrict our good 2-classifier in prestacks (\thex\ref{teor2classinprestacks}) to a good 2-classifier in stacks that classifies all discrete opfibrations with small fibres (\thex\ref{teor2classinstacks}). We follow the strategy described in the general \thex\ref{teorfactorization} to restrict a good 2-classifier to a nice sub-2-category. The idea is to select, out of all the presheaves on slices involved in the definition of $\t{\O}$, the sheaves with respect to the Grothendieck topology induced on the slices. This restriction of $\t{\O}$ is tight enough to give a stack $\O_J$, but at the same time loose enough to still host the classification process of prestacks.

Our result solves a problem posed by Hofmann and Streicher in~[7]. Indeed, in a different context, they considered the same natural idea to restrict their analogue of $\t{\O}$ by taking sheaves on slices. However, this did not work for them, as it does not give a sheaf. Our results show that such a restriction yields nonetheless a stack and a good 2-classifier in stacks.

As explained in \remx\ref{remourstacks}, we take strictly functorial stacks with the respect to a subcanonical topology $J$. Throughout this section, we consider the full sub-2-category $\St[J]{\C}$ of $\m{\C\op}{\Cat}$ on stacks. Call $i\:\St[J]{\C}\ffto \m{\C\op}{\Cat}$ the injective on objects and fully faithful 2-functor of inclusion.

We want to show that $i$ satisfies all the assumptions of \thex\ref{teorfactorization}.

The following proposition does not seem to appear in the literature.

\begin{prop}\label{proplimitsinstacks}
The 2-category $\St+[J]{\C}$ of pseudofunctorial stacks has all bilimits and all flexible limits, calculated in $\Psm{\C\op}{\Cat}$ and hence pointwise.

$\St[J]{\C}$ has all flexible limits \pteor{thus all comma objects and the terminal object} and all pullbacks along discrete opfibrations, calculated in $\m{\C\op}{\Cat}$ and hence pointwise.
\end{prop}
\begin{proof}
By \thex\ref{stacksarebireflective}, $\St+[J]{\C}$ is a bireflective sub-2-category of $\Psm{\C\op}{\Cat}$. So by \remx\ref{remlimitsinpseudofunctors} it has all bilimits, calculated in $\Psm{\C\op}{\Cat}$. Consider then a flexible weight $W$ and a 2-diagram $F$ in $\St+[J]{\C}$. Then the flexible limit of $F$ weighted by $W$ exists in $\Psm{\C\op}{\Cat}$, by \remx\ref{remlimitsinpseudofunctors}. In particular, by flexibility of $W$, it satisfies the universal property of a bilimit of a 2-diagram that factors through $\St+[J]{\C}$. And it is then a pseudofunctorial stack. It follows that it is the flexible limit of $F$ weighted by $W$ in $\St+[J]{\C}$, since fully faithful 2-functors reflect 2-limits.

Consider now a 2-diagram $F$ in $\St[J]{\C}$ and a flexible weight $W$. The flexible limit of $F$ weighted by $W$ exists in $\m{\C\op}{\Cat}$, calculated pointwise. Then it is also the flexible limit in $\Psm{\C\op}{\Cat}$, as the latter is as well calculated pointwise. So it is a stack, as the 2-diagram in $\Psm{\C\op}{\Cat}$ factors through $\St+[J]{\C}$. Whence we have produced the flexible limit of $F$ weighted by $W$ in $\St[J]{\C}$.

Finally, consider $p\: E\to B$ and $z\: F\to B$ in $\St[J]{\C}$, with $p$ a discrete opfibration. The pullback of $p$ and $z$ exists in $\m{\C\op}{\Cat}$, calculated pointwise. Then it is also the bi-iso-comma object of $p$ and $z$ in $\Psm{\C\op}{\Cat}$, by \remx\ref{remlimitsinpseudofunctors}. We conclude that it is a stack and hence the pullback of $p$ and $z$ in $\St[J]{\C}$, by the argument above.
\end{proof}

\begin{prop}\label{propdiscopfibinstacks}
A morphism in $\St[J]{\C}$ is a discrete opfibration if and only if its underlying morphism in $\m{\C\op}{\Cat}$ is so. In particular, $i\:\St[J]{\C}\ffto \m{\C\op}{\Cat}$ preserves discrete opfibrations.
\end{prop}
\begin{proof}
The ``if" part is clear by the fact that $\St[J]{\C}\ffto \m{\C\op}{\Cat}$ is fully faithful. The ``only if" part follows from \prox\ref{propdiscopfibinpresheaves}, since $J$ is subcanonical.
\end{proof}

\begin{rem}\label{remstacksformnicesub}
Putting together \prox\ref{proplimitsinstacks} and \prox\ref{propdiscopfibinstacks}, we have thus proved that $i\:\St[J]{\C}\cont \m{\C\op}{\Cat}$ is a nice sub-2-category (\defx\ref{defnice}).
\end{rem}

\begin{defne}
We say that a discrete opfibration $\phi$ in $\St[J]{\C}$ \dfn{has small fibres} if $i(\phi)$ has small fibres. Notice that this is in line with \remx\ref{rempropPinsub2cat}.
\end{defne}

\begin{prop}\label{propffinprestacks}
Let $l\:F\to B$ be a morphism in $\m{\C\op}{\Cat}$ \pteor{that is, a 2-natural transformation $l$}. $l$ is a fully faithful morphism if and only if for every $C\in \C$ the component $l_C$ of $l$ on $C$ is a fully faithful functor.

$l$ is chronic \pteor{\defx\ref{defchronic}} if and only if for every $C\in \C$ the component $l_C$ of $l$ on $C$ is an injective on objects and fully faithful functor.
\end{prop}
\begin{proof}
The proof is straightforward.
\end{proof}

\begin{cons}
We want to produce the object $\O_\M$ of \thex\ref{teorfactorization} in our case with $i\:\St[J]{\C}\cont \m{\C\op}{\Cat}$. That is, a stack, which we will call $\O_J$, that is a nice restriction of the good 2-classifier $\t{\O}$ in prestacks.
Recall that, in dimension 1, the subobject classifier in sheaves is given by taking closed sieves. We produce a 2-categorical notion of closed sieve.

We have already said in \conx\ref{constwoclassinprestacks} that discrete opfibrations over representables generalize the concept of sieve to dimension 2; we call them 2-sieves. Thanks to \prox\ref{propstrictification} (indexed Grothendieck construction, explored in our joint work with Caviglia~[3]), we can equivalently consider presheaves on slice categories. We now need to generalize closedness of a sieve to dimension 2. The indexed Grothendieck construction can be restricted to a bijection between 1-dimensional sieves on $C\in \C$ and presheaves ${\left(\slice{\C}{C}\right)}\op\to \2$. It can be shown that closed 1-sieves correspond with sheaves ${\left(\slice{\C}{C}\right)}\op\to \2$, and that the closure of a sieve corresponds to the sheafification of the corresponding presheaves. So we define \dfn{closed 2-sieves} to be the sheaves ${\left(\slice{\C}{C}\right)}\op\to \Set$ (with respect to the Grothendieck topology induced by $J$ on the slices, that we call again $J$). And we use them to restrict our good 2-classifier $\omega\:1\to\t{\O}$ in prestacks to a good 2-classifier $\omega_J\:1\to\O_J$ in stacks (\thex\ref{teor2classinstacks}).
\end{cons}

\begin{rem}\label{remclosed2sieves}
The ``maximal" 2-sieve $\id{\y{C}}$, associated with $\t{\omega}_C=\Delta1\:{\left(\slice{\C}{C}\right)}\op \to \Set$ (see \remx\ref{remhofmannstricherawodey}), is a closed 2-sieve.

Closed 2-sieves are stable under pullbacks. Indeed if $F\:{\slice{\C}{C}}\op\to \Set$ is a sheaf and $f\:D\to C$ is a morphism in $\C$ then also $F\c (f\c =)\:{\slice{\C}{D}}\op\to \Set$ is a sheaf.
\end{rem}

\begin{prop}\label{propOJisastack}
The 2-functor
\begin{fun}
{\O_J} & \: & \C\op \hphantom{c}& \too & \hphantom{c}\Cat \\[1.3ex]
&& C\hphantom{C} & \mto & \Sh[J]{\slice{\C}{C}}\\[0.3ex]
&& (C\al{f} D) & \mto & -\c {(f\c =)}\op\v[0.5]
\end{fun}
is a stack with respect to the Grothendieck topology $J$.\v

Moreover, the inclusions $\Sh[J]{\slice{\C}{C}}\ffto \m{{\left(\slice{\C}{C}\right)}\op}{\Set}$ form a chronic arrow \linebreak[4]${\ell\:i(\O_J)\ffto\t{\O}}$ in $\m{\C\op}{\Cat}$. And $\t{\omega}\:1\to \t{\O}$ factors through $\ell$; call $\omega_J\:1\to \O_J$ the resulting morphism.
\end{prop}
\begin{proof}
The second part of the statement is clear after \prox\ref{propffinprestacks} and \remx\ref{remclosed2sieves}.

We prove that $\O_J$ is a stack (recall from \defx\ref{defstack} the definition). So let $C\in \C$ and $S\in J(C)$ a covering sieve on $C$.

We first prove the uniqueness of gluings of morphisms. Let $M,N\in \Sh[J]{\slice{\C}{C}}$ and let $\alpha,\beta\:M\aR{}N$ two natural transformations such that $f\st \alpha=f\st \beta$ for every $(D\ar{f}C)\in S$. We show that $\alpha=\beta$. Given $(D\ar{f}C)\in S$,
$$\alpha_f=(f\st \alpha)_{\id{D}}=(f\st \beta)_{\id{D}}=\beta_f.$$
Let now $g\:E\to \C$ in $\C$ and consider $g\st S\in J(g)=J(E)$. Since $M$ is a sheaf,
$$M(g)\iso \Match[M]{g\st S}$$
where the right hand side denotes the set of matching families for $M$ with respect to the covering sieve $g\st S$. And this holds analogously for $N$. We produce a commutative square
\begin{cd}[4][4.5]
M(g) \arrow[d,iso,"{}"]\arrow[r,"{\alpha_g}"]\& N(g)\arrow[d,iso,"{}"] \\
\Match[M]{g\st S}\arrow[r,"{[\alpha_g]}"'] \& \Match[N]{g\st S}
\end{cd}
and analogously for $\beta$. We define $[\alpha_g]$ to send a matching family $(h\in g\st S)\am{m} (x_h\in M(g\c h))$ to the matching family $(h\in g\st S)\mto (\alpha_{g\c h}(x_h)\in N(g\c h))$. The latter is indeed a matching family by naturality of $\alpha$. The square above commutes since, for every $m\in \Match[M]{g\st S}$, calling $X$ the amalgamation of $m$, we have that $\alpha_g(X)$ is an amalgamation of $[\alpha_g](m)$. Notice now that $[\alpha_g]=[\beta_g]$, as for every $h\in g\st S$ we have that $g\c h\in S$ and hence $\alpha_{g\c h}=\beta_{g\c h}$. Thus $\alpha_g=\beta_g$.

We prove that we have the gluings of morphisms. Let $M,N\in \Sh[J]{\slice{\C}{C}}$ and consider a matching family
$$(D\ar{f} C)\in S \h[3]\mto \h[3] (\alpha_f\:f\st M\aR{}f\st N) \text{ in } \Sh[J]{\slice{\C}{D}}$$
So that for every $D'\ar{l}D\ar{f}C$ with $f\in S$ it holds that $l\st \alpha_f=\alpha_{f\c l}$. We produce a natural transformation $\lambda\:M\aR{}N$ such that $f\st\lambda=\alpha_f$ for every $(D\ar{f}C)\in S$. Given $(D\ar{f}C)\in S$, we would like to define $\lambda_f\deq {(\alpha_f)}_{\id{D}}$. Let $g\:E\to C$ in $\C$. We define $\lambda_g$ to be the composite
$$M(g)\iso \Match[M]{g\st S} \ar{[\lambda_g]}\Match[N]{g\st S}\iso N(g)$$
where $[\lambda_g]$ sends a matching family $(h\in g\st S)\am{m} (x_h\in M(g\c h))$ to the matching family $(h\in g\st S)\mto ((\alpha_{g\c h})_{\id{}}(x_h)\in N(g\c h))$. The latter is indeed a matching family by naturality of $\alpha_{g\c h}$. It is then straightforward to prove that $\lambda$ is natural. Given $(D\ar{f}C)\in S$, we have that $\lambda_f={(\alpha_f)}_{\id{D}}$, since $\id{D}\in f\st S$ and hence the amalgamation of any matching family on $f\st S$ is just the datum on $\id{D}$. So $f\st \lambda=\alpha_f$. Indeed for every $l\:D'\to D$ in $\C$
$$(f\st \lambda)_l=\lambda_{f\c l}={(\alpha_{f\c l})}_{\id{E}}={(l\st \alpha_f)}_{\id{E}}={(\alpha_f)}_l.$$

It remains to prove that we have the gluing of objects. So consider a descent datum
$$(D\ar{f} C)\in S \h[3]\mto \h[3] M_f\in\Sh[J]{\slice{\C}{D}}$$
with $\phi^{f,h}\:h\st M_f\iso M_{f\c h}$ such that the cocycle condition
\begin{cd}[5][5]
k\st h\st M_f \arrow[d,"", iso] \arrow[r,"k\st \phi^{f,h}"] \&   k\st M_{f\c h} \arrow[d,"\phi^{f\c h,k}"] \\
(h\c k)\st M_f \arrow[r,"\phi^{f,h\c k}"']\& {M_{f\c h\c k}.}
\end{cd}
holds for every $D''\ar{k} D' \ar{h}  D\ar{f} C$ with $f\in S$. We produce $M\in \Sh[J]{\slice{\C}{C}}$ and for every $(D\ar{f}C)\in S$ isomorphisms $\psi^f\:f\st M\iso M_f$ such that
\begin{cd}[5][5]
h\st f\st M \arrow[d,"",iso] \arrow[r,"h\st \psi^{f}"] \&   {h\st M_f} \arrow[d,"\phi^{f,h}"] \\
(f\c h)\st M\arrow[r,"\psi^{f\c h}"']\& {M_{f\c h}.}
\end{cd}
for every $D'\ar{h} D \ar{f} C$ with $f\in S$. We construct the presheaf \v[1]
\begin{fun}
Z & \: & {\left(\slice{\C}{C}\right)}\op & \too & \Set \\[1.3ex]
&&\parbox{2.05cm}{\trslice*+[3.7][3.7]{D}{C}{D'}{f\in S}{h}{f'}}&\mtoo&\h[-3]\fib[n][2.7]{M_f(\id{D})}{Z(\underline{h})}{M_{f'}(\id{D'})}\\[1.3ex]
&&\p{D\ar{g\notin S}C}&\mtoo& \hphantom{C}\emp
\end{fun}
where $Z(\underline{h})$ is the composite
$$M_f(\id{D})\ar{M_f(\underline{h})} M_f(h)=(h\st M_f)(\id{D'})\aisoo[\phi^{f,h}_{\id{D'}}] M_{f\c h}(\id{D'}).\v[-1]$$
$Z$ is indeed a functor, by the cocycle condition. Moreover, it is straightforward to show that $f\st Z\iso M_f$ for every $(D\ar{f}C)\in S$. However, $Z$ is not a sheaf. So we define $M\deq Z^{++}$, where $Z^+$ is the plus construction of $Z$ and hence $Z^{++}$ is the sheafification of $Z$. It is straightforward to check that $(f\st Z)^+\iso f\st (Z^+)$, by the explicit plus construction. Thus, using that $f\st Z\iso M_f$, we define $\psi^f$ to be the composite
$$f\st (Z^{++})\iso (f\st Z^+)^+\iso (f\st Z)^{++}\iso (M_f)^{++}\iso M_f,$$
where the last isomorphism is given by the fact that $M_f$ is a sheaf. It is then straightforward to show that the isomorphisms $\psi^f$ satisfy the required condition.
\end{proof}

\begin{rem}\label{remrepresentablesinstacks}
Representables form a fully faithful dense generator $\yy\:\C\to \St[J]{\C}$ of the kind described in \conx\ref{consdenseoffullsubcat}. We want to apply \thex\ref{teorfactorization} on such a dense generator. So we need to factorize the characteristic morphisms in $\m{\C\op}{\Cat}$ of discrete opfibrations with small fibres in $\St[J]{\C}$ over representables.
\end{rem}

\begin{prop}\label{propfactorizationofcharmorinstacks}
For every $\psi\:H\to \y{C}$ a discrete opfibration in $\St[J]{\C}$ with small fibres, with $C\in \C$, every characteristic morphism of $i(\psi)$ with respect to $\t{\omega}$ factors through $\ell\:i(\O_J)\ffto \t{\O}$.
\end{prop}
\begin{proof}
It suffices to prove that the characteristic morphism $z$ for $i(\phi)$ with respect to $\t{\omega}$ produced in \remx\ref{remconcreterecipeforclasmorph} (and \thex\ref{teor2classinprestacks}) factors through $\ell$. Indeed such factorization only depends on the isomorphism class of $z$, by the Yoneda lemma, as any presheaf isomorphic to a sheaf is a sheaf. By \remx\ref{remconcreterecipeforclasmorph}, $z$ is the 2-natural transformation $\y{C}\to \t{\O}$ that corresponds with the functor
\begin{fun}
z_C(\id{C}) & \: & {\left(\slice{\C}{C}\right)}\op & \too & \Set \\[1ex]
&& (D\ar{f}C) & \mto & {\left(\phi_D\right)}_{f}\\[0.6ex]
&&(f \al{g} f\c g)&\mto& H(g)\v
\end{fun}
So it suffices to prove that such functor is a sheaf. Let $f\:D\to C$ in \C and $R$ a covering sieve on $f\:D\to C$, i.e.\ $R\in J(D)$. Consider then a matching family
$$\trslicematchN{D'}{D}{C}{g}{}{f}{4}{4} \!\in R \quad \amm{m} \quad X_g\in {(\phi_{D'})}_{f\c g}$$
on $R$ for $z_C(\id{C})$. We need to show that there is a unique $X\in {(\phi_D)}_f$ such that $H(g)(X)=X_g$ for every $\p{D'\ar{g}D}\in R$. Notice that $m$ is also a matching family on $R\in J(D)$ for $H$, as ${(\phi_{D'})}_{f\c g}\subseteq H(D')$ and the action of $z_C(\id{C})$ on morphisms is given by the action of $H$. Since $\y{C}\:\C\op\to \Set$, also $H\:\C\op\to \Set$. As $H$ is a stack, it then needs to be a sheaf. So there is a unique $X\in H(D)$ such that $H(g)(X)=X_g$ for every $\p{D'\ar{g}D}\in R$. It remains to prove that $\phi_D(X)=f$. Since $\y{C}$ is separated, it suffices to prove that $\phi_D(X)\c g=f\c g$ for every $\p{D'\ar{g}D}\in R$. But by naturality of $\phi$
$$\phi_D(X)\c g=\phi_{D'}(H(g)(X))=\phi_{D'}(X_g)=f\c g.$$
We thus conclude that $z_C(\id{C})$ is a sheaf.
\end{proof}

We can now apply \thex\ref{teorfactorization} (based on our theorems of reduction of the study of a 2-classifier to dense generators) to guarantee that we have produced a good 2-classifier in stacks that classifies all discrete opfibrations with small fibres. The following theorem is original.

\begin{teor}\label{teor2classinstacks}
The 2-natural transformation $\omega_J$ from $1$ to
\begin{fun}
{\O_J} & \: & \C\op \hphantom{c}& \too & \hphantom{c}\Cat \\[1.3ex]
&& C\hphantom{C} & \mto & \Sh[J]{\slice{\C}{C}}\\[0.3ex]
&& (C\al{f} D) & \mto & -\c {(f\c =)}\op\v[0.5]
\end{fun}
that picks the constant at 1 sheaf on every component is a good $2$-classifier in $\St[J]{\C}$ that classifies all discrete opfibrations with small fibres.
\end{teor}
\begin{proof}
By \thex\ref{teorfactorization}, the restriction $\omega_J$ of the good 2-classifier $\t{\omega}$ in $\m{\C\op}{\Cat}$ along $i\:\St[J]{\C}\ffto \m{\C\op}{\Cat}$ is a good 2-classifier in $\St[J]{\C}$ with respect to the property of having small fibres. We can apply \thex\ref{teorfactorization} thanks to \remx\ref{remstacksformnicesub}, \thex\ref{teor2classinprestacks}, \prox\ref{propOJisastack}, \remx\ref{remrepresentablesinstacks} and \prox\ref{propfactorizationofcharmorinstacks}.
\end{proof}

\begin{rem}
We can extract from \thex\ref{teorfactorization} and \remx\ref{remconcreterecipeforclasmorph} a recipe for the characteristic morphism $z_J\:F\to \O_J$ of a discrete opfibration $\phi\:G\to F$ in $\St[J]{\C}$ with small fibres.
\sq[l][6][6][\h[-3]\v[2]\opn{comma}]{G}{1}{F}{\O_J}{}{\phi}{\omega_J}{z_J}
Recall that we denote as $i$ the inclusion $\St[J]{\C}\cont \m{\C\op}{\Cat}$. We obtain that $z_J$ corresponds to the 2-natural transformation $i(z_J)\:i(F)\to i(\O_J)$ whose component on $C\in \C$ is the functor $i(z_J)_C$ that sends $X\in i(F)(C)$ to the sheaf
\begin{fun}
i(z_J)_C(X) & \: & {\left(\slice{\C}{C}\right)}\op & \too & \Set \\[1ex]
&& (D\ar{f}C) & \mto & {\left(i(\phi)_D\right)}_{i(F)(f)(X)}\\[0.6ex]
&&(f \al{g} f\c g)&\mto& i(G)(g)\\[0.275ex]
\end{fun}
\end{rem}

\subsection*{Acknowledgements}

I sincerely thank my PhD supervisor Nicola Gambino for the numerous helpful discussions on the subject of this paper and useful comments on earlier versions of this work. Part of this research has been conducted while visiting the University of Manchester.

\bibliographystyle{abbrv}
\begin{thebibliography}{10}

[1]
\newblock {On Hofmann-Streicher universes}.
\newblock {\em arXiv: \url{https://arxiv.org/abs/2205.10917}}, 2023.

[2]
G.~J. Bird, G.~M. Kelly, A.~J. Power, and R.~H. Street.
\newblock Flexible limits for 2-categories.
\newblock {\em J. Pure Appl. Algebra}, 61(1):1--27, 1989.

[3]
E.~Caviglia and L.~Mesiti.
\newblock {Indexed Grothendieck construction}.
\newblock {\em arXiv: \url{https://arxiv.org/abs/2307.16076}}, 2023.

[4]
M.~E. Descotte, E.~J. Dubuc, and M.~Szyld.
\newblock Sigma limits in 2-categories and flat pseudofunctors.
\newblock {\em Adv. Math.}, 333:266--313, 2018.

[5]
\newblock M{\'e}thode de la descente.
\newblock {\em Bull. Soc. Math. Fr., Suppl., M{\'e}m.}, 2:115, 1964.

[6]
J.~W. Gray.
\newblock {\em Formal category theory: {Adjointness} for 2-categories}, volume
391 of {\em Lect. Notes Math.}
\newblock Springer, 1974.

[7]
M.~Hofmann and T.~Streicher.
\newblock {Lifting Grothendieck universes}.
\newblock {\em preprint:
\url{https://www2.mathematik.tu-darmstadt.de/~streicher/NOTES/lift.pdf}},

[8]
G.~M. Kelly.
\newblock Basic concepts of enriched category theory.
\newblock {\em Repr. Theory Appl. Categ.}, 2005(10):1--136, 2005.

[9]
\newblock Codescent objects and coherence.
\newblock 175(1):223--241, 2002.

[10]
S.~Lack and M.~Shulman.
\newblock Enhanced 2-categories and limits for lax morphisms.
\newblock {\em Adv. Math.}, 229(1):294--356, 2012.

[11]
\newblock Colimits in 2-dimensional slices.
\newblock {\em arXiv: \url{https://arxiv.org/abs/2305.01494}}, 2023.

[12]
\newblock {The 2-Set-enriched Grothendieck construction and the lax normal
conical 2-limits}.
\newblock {\em arXiv: \url{https://arxiv.org/abs/2302.04566}}, 2023.

[13]
\newblock A general coherence result.
\newblock 57(2):165--173, 1989.

[14]
\newblock {Fibrations and Yoneda's Lemma in a 2-category}.
\newblock {Category Sem., Proc., Sydney 1972/1973, Lect. Notes Math. 420,
104-133, Springer}, 1974.

[15]
\newblock Limits indexed by category-valued 2-functors.
\newblock {\em J. Pure Appl. Algebra}, 8:149--181, 1976.

[16]
\newblock Characterization of bicategories of stacks.
\newblock Category theory. {Applications} to algebra, logic and topology,
{Proc}. int. {Conf}., {Gummersbach} 1981, {Lect}. {Notes} {Math}. 962,
282-291 (1982)., 1982.

[17]
{The Univalent Foundations Program}.
\newblock {\em Homotopy Type Theory: Univalent Foundations of Mathematics}.
\newblock \url{https://homotopytypetheory.org/book}, Institute for Advanced
Study, 2013.

[18]
\newblock Univalent foundations of mathematics.
\newblock In {\em Logic, language, information and computation. 18th
international workshop, WoLLIC 2011, Philadelphia, PA, USA, May 18--20, 2011.
Proceedings}, page~4. Berlin: Springer, 2011.

[19]
\newblock {Yoneda structures from 2-toposes}.
\newblock {\em {Appl. Categ. Struct.}}, 15(3):259--323, 2007.

\end{thebibliography}

\end{document}