# Edisum: Summarizing and Explaining Wikipedia Edits at Scale

Marija Šakota1 Isaac Johnson2 Guosheng Feng1 Robert West1
1EPFL 2Wikimedia Foundation
{marija.sakota, guosheng.feng<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

An edit summary is a succinct comment written by a Wikipedia editor explaining
the nature of, and reasons for, an edit to a Wikipedia page. Edit summaries
are crucial for maintaining the encyclopedia: they are the first thing seen by
content moderators and they help them decide whether to accept or reject an
edit. Additionally, edit summaries constitute a valuable data source for
researchers. Unfortunately, as we show, for many edits, summaries are either
missing or incomplete. To overcome this problem and help editors write useful
edit summaries, we propose a model for recommending edit summaries generated
by a language model trained to produce good edit summaries given the
representation of an edit diff. This is a challenging task for multiple
reasons, including mixed-quality training data, the need to understand not
only what was changed in the article but also why it was changed, and
efficiency requirements imposed by the scale of Wikipedia. We address these
challenges by curating a mix of human and synthetically generated training
data and fine-tuning a generative language model sufficiently small to be used
on Wikipedia at scale. Our model performs on par with human editors.
Commercial large language models are able to solve this task better than human
editors, but would be too expensive to run on Wikipedia at scale. More
broadly, this paper showcases how language modeling technology can be used to
support humans in maintaining one of the largest and most visible projects on
the Web.

##  1 Introduction

Wikipedia is the largest online encyclopedia, housing 60 million articles in
over 300 languages, with the English Wikipedia alone featuring 6.7 million
entries. It is edited collaboratively, meaning that anyone can be an editor to
most of the articles, resulting in massive numbers of edits performed
continuously; e.g., on English Wikipedia alone, over 3 million edits are
performed each month (Wikipedia, 2024c). When performing an edit, the editor
can leave an _edit summary_ (example in Fig. 1), a short comment explaining
the content of the edit and, sometimes, a reason why the edit was performed.
It is often the first source of information about an edit that editors see
when browsing edit histories for content moderation or other purposes and is
an opportunity for an editor to justify their changes.

Figure 1: An example of an edit diff. The + and – signs denote the text that
was added and removed, respectively. The edit summary is the text highlighted
in green in the screenshot.

Edit summaries are also valuable to researchers. They provide important
insights into editor roles and actions on Wikipedia (Geiger & Ribes, 2010;
Arazy et al., 2016; Wattenberg et al., 2007). They are used for building
datasets for various purposes, such as the detection of low-quality Wikipedia
content (Asthana et al., 2021) or detecting conflicts (Sumi et al., 2011).
Edits and edit summaries have also been used to build datasets for iterative
text generation, due to their incremental nature (Schick et al., 2022;
Faltings et al., 2021).

Despite being a valuable asset, edit summaries have a number of drawbacks that
prevent them from being used more efficiently. Many users leave the edit
summary blank when performing an edit. Even when provided, summaries can be
misleading—and not necessarily deliberately (as opposed to vandalism). Some
editors also use canned edit summaries (Wikipedia, 2024b), to quickly insert
commonly used summaries in the current Wikipedia space. For instance, these
can be edit summaries such as “Added links” or “Fixed typo”. They are not
intentionally misleading, but frequently do not reflect the content of the
edit precisely. Although it would be hard to mitigate the effects of vandalism
on edit summaries, our analyses show that a large fraction of edits would
benefit from a more specific, tailored edit summary. This is currently an
unexplored area within research, with no previous attempts to automatically
generate Wikipedia edit summaries. Given their performance on text generating
tasks, generative language models arise as a promising solution.

Generating Wikipedia edit summaries is a tricky problem for several reasons.
Although blank edit summaries are easy to detect, there are no established
heuristics for determining whether an edit summary is a good description of
its edit or not. This can lead to mixed-quality data for training a model,
and, consequently, poor performance at deployment time. Furthermore, edit
summaries should ideally explain why the edit was performed, along with what
was changed, which often requires external context. From an engineering
perspective, it is also not trivial to design an appropriate prompt for this
task, in particular because most generative models work with a small context
size. Finally, a generative model used for this task at scale cannot be a
large language model (LLM). Even though LLMs are promising candidates for
automatic edit summary generation in theory, deploying an LLM-based system
would require immense expenses, especially for platforms like Wikipedia, where
vast numbers of users would rely on the model every day. Furthermore, use of
open-source technology is a guiding principle for Wikimedia (Wikimedia,
2024a).

In this paper, we perform a detailed qualitative analysis of edit summaries,
uncovering some of the drawbacks of human-written summaries. We show that this
task can be, at a small scale, solved with LLMs. Based on these results, we
carefully select a high-quality subset of edits and edit summaries. For a
subset of the edits lacking summaries, we synthetically generate edit
summaries using an LLM. Due to efficiency and input-size constraints, we then
fine-tune a range of smaller generative language models with longer context
size, built on LongT5 (Guo et al., 2022), which we call _Edisum_. We finetune
on a mix of editor-provided and synthetic data, using a representation of edit
diffs as inputs. This approach balances providing sufficient context for most
edits while remaining suitable for large-scale application on platforms like
Wikipedia.

Results. We compare our solution to two baselines, human editors111We envision
that our model could provide a recommended edit summary to human editors,
simplifying the process of writing it and encouraging uptake. and the far more
resource-heavy GPT-4, via both automatic and human evaluation. Our results
indicate that while GPT-4 is effective (but expensive), Edisum trained on
synthetic data matches human editors’ performance, offering an ideal solution
for a large-scale application on Wikipedia.

Contributions. In short, our contributions are the following:

(i) We perform a comprehensive qualitative analysis of the existing Wikipedia
edit summaries, which shows that many existing edit summaries have hard-to-
detect flaws.

(ii) We show that edit summary generation is, at small scale, solvable using
high-performance LLMs (GPT-4).

(iii) We show that Edisum, which is, to the best of our knowledge, the first
solution to automate the generation of highly-contextual Wikipedia edit
summaries at large scale, achieves performance similar to the human editors

(iv) We release the dataset consisting of cleaned edit summaries and
synthetically generated data, which can be used for future research. The code
can be found at https://github.com/epfl-dlab/edisum.

##  2 Related work

Wikipedia edit summaries. Edit summaries play an important role on Wikipedia
in helping patrollers quickly monitor edits for vandalism or otherwise
problematic edits (Wikimedia, 2024b). They are simpler and easier to scan than
the edit diffs, and thus are important for enabling fast patrolling of content
on Wikipedia (Morgan, 2019). Despite this importance, we are not aware of work
that focuses on helping editors to improve edit summaries.

Edit summaries have been used extensively, however, to understand and model
behavior on Wikipedia. Researchers who utilize edit summaries occasionally
comment on anecdotal patterns in usage, but descriptive statistics are
minimal. Panciera et al. (2009) describe the usage of links to Wikipedia
policy pages in edit summaries, showing that the likelihood of invoking a
policy increases with editor experience. Wattenberg et al. (2007) convert edit
summaries into colors to visualize how different editors approach tasks on
Wikipedia and Geiger & Ribes (2011) describe the importance of edit summaries
in tracing activity on Wikipedia for understanding bots and vandalism, while
Stvilia et al. (2008) point out that edit summaries are often blank or
misleading, rendering them less useful. Multiple works (Yang et al., 2017;
Pavalanathan et al., 2018; Asthana et al., 2021) construct datasets of edits
for training models by filtering edits based on certain keywords in the edit
summaries. Notably, Yang et al. (2017) classify edits based on their
intention, including labels commonly found in edit summaries, such as
“clarification”. In contrast to their multi-label classification method, we
opt for a more flexible generative language model approach.

Synthetic data generation. Early approaches using generative models to produce
synthetic data focused on finetuning a pretrained model which is then used as
a generator (Anaby-Tavor et al., 2020; Papanikolaou & Pierleoni, 2020;
Mohapatra et al., 2020; Kumar et al., 2020). This requires an existing dataset
for finetuning the generator. Recently, the focus has shifted on unsupervised
methods for synthetic data generation using pretrained language models (PLMs).
These methods do not require lengthy and expensive labeling. One such example
is the work by Wang et al. (2021), in which they generate synthetic labels by
using only unlabeled examples sent to the LLM. There have been several
attempts to generate data for different natural language processing (NLP)
tasks by carefully designing prompts to the PLMs. This includes work by Ye et
al. (2022) and Gao et al. (2022) in which they evaluate this procedure on text
classification, question answering, and natural language inference tasks.
Similarly, Meng et al. (2022) do this for GLUE (Wang et al., 2018) tasks.
There have been successful attempts to use synthetic data generated in this
way for intent classification (Sahu et al., 2022), and question answering (Li
et al., 2022) as well.

There are also examples of synthetic data generation for more tailored
purposes. Shao et al. (2023) use the synthetic data as demonstrations to
improve the propmting of LLMs. Additionally, synthetic data has been used to
solve tasks that LLMs cannot directly solve, such as closed information
extraction (Josifoski et al., 2023). Our task is not a standard NLP task, such
as text classification or summarization, but can still be seen as a text
generation task. As such, it is likely that LLMs can solve it with careful
prompting, enabling synthetic data generation for training a more efficient
system suitable for large-scale use.

##  3 Qualitative analysis of Wikipedia edit summaries

Metric |  Summary (what) |  Explain (why) |  Misleading |  Inappropriate |  Generate-able (what) |  Generate-able (why)
---|---|---|---|---|---|---
Description |  Attempts to describe what the edit did. For example, “added links” |  Attempts to describe why the edit was made. For example, “Edited for brevity and easier reading”. |  Overly vague or misleading per English Wikipedia guidance. For example, “updated” without explaining what was updated is too vague. |  Could be perceived as inappropriate or uncivil per English Wikipedia guidance. |  Could a language model feasibly describe the “what” of this edit based solely on the edit diff. |  Could a language model feasibly describe the “why” of this edit based solely on the edit diff.
% Agreement |  0.89 |  0.8 |  0.77 |  0.98 |  0.97 |  0.8
Cohen’s Kappa |  0.65 |  0.57 |  0.50 |  -0.01 |  0.39 |  0.32
Overall (n=100) |  0.75 - 0.86 |  0.26 - 0.46 |  0.23 - 0.46 |  0.00 - 0.02 |  0.96 - 0.99 |  0.08 - 0.28
IP editors (n=25) |  0.76 - 0.88 |  0.20 - 0.44 |  0.40 - 0.64 |  0.00 - 0.08 |  0.92 - 0.96 |  0.04 - 0.16
Newcomers (n=25) |  0.76 - 0.84 |  0.36 - 0.48 |  0.24 - 0.52 |  0.00 - 0.00 |  0.92 - 1.00 |  0.12 - 0.20
Mid-experienced (n=25) |  0.76 - 0.88 |  0.28 - 0.52 |  0.16 - 0.36 |  0.00 - 0.00 |  1.00 - 1.00 |  0.08 - 0.28
Experienced (n=25) |  0.72 - 0.84 |  0.20 - 0.40 |  0.12 - 0.32 |  0.00 - 0.00 |  1.00 - 1.00 |  0.08 - 0.48

Table 1: Statistics on agreement for qualitative coding for each facet and the
proportion of how many edit summaries met each criteria. Ranges are a lower
bound (both of the coders marked an edit) and an upper bound (at least one of
the coders marked an edit). The majority of summaries are expressing only what
was done in the edit, which we also expect a language model to do. A
significant portion of edits is of low quality, i.e., misleading.

Given the dearth of data on the nature and quality of edit summaries on
Wikipedia, we perform qualitative coding to guide our modeling decisions.
Specifically, we analyze a sample of 100 random edits made in August 2023 to
English Wikipedia. To ensure good coverage, the sample is stratified among a
diverse set of editor expertise levels. Two of the authors each coded all 100
summaries and we report the results in Table 1. Because there were only two
coders, we report the range for each category instead of the majority label.
The lower bound represents the case in which both annotators marked the edit
as belonging to that category, while for the upper bound, at least one marked
it. Edit summaries were coded by following criteria set by the English
Wikipedia community (Wikimedia, 2024b) (see Table 1). For more details on the
annotation process, see Appendix A.

Overall, we see a relatively high annotator agreement. Lower Cohen’s kappa for
some categories indicates that these judgements can be difficult and highly
subjective. The vast majority ($\sim$80%) of current edit summaries are
oriented towards the “what” and only to a much lesser degree (30–40%) the
“why” of the edit. This aligns with the raters’ judgement of what a language
model can generate from the edit diff alone (see columns ”Generate-able
(what)“ and ”Generate-able (why)“ in Table 1). Accurately describing the “why”
would require outside context that a model would not have access to, such as
information about a source being added or events happening in the world.

A sizeable minority ($\sim$35%) of edit summaries were labeled as
“misleading”. This was generally due to overly vague summaries or summaries
that only mention part of the edit. This makes training a model on this data
challenging. Almost no edit summaries are inappropriate, likely because highly
inappropriate edit summaries would be deleted (Wikipedia, 2024d) by
administrators and not appear in our dataset. This suggests that it is
unlikely for a model trained on edit summaries to learn to suggest
inappropriate summaries and thus we do no further filtering of edit summaries
for inappropriate language.

##  4 Method

###  4.1 Synthetic data generation

From the analysis in Sec. 3, we notice there is a considerable number of lower
quality edits, which are not easily detectable. At the same time, as LLMs
perform well for a wide variety of tasks, often in a few-shot setting, we
expect them to generate a good quality edit summary after some prompt tuning
for majority of the edits, including what was done, but also why the edit was
performed when obvious from the context. Our initial exploration on GPT-4 and
GPT-3.5 confirms these assumptions. Our idea is not to just prompt LLMs to
solve the task, but to rather generate synthetic data which will be used to
tune a more efficient model.

LLM. After experimenting with available OpenAI models (OpenAI, 2024b), we opt
for gpt-3.5-turbo model with 4k token context as a good compromise between
price and quality of the results. This model is optimized for the dialogue
setting. We prompt it by sending the explanation of what an edit summary is as
a system prompt, while the demonstrations are presented as alternating
dialogue turns by the user (edit diff) and the model (edit summary).

Generating useful synthetic training data requires an LLM that can already
solve the task of automated edit summary generation—the very task we set out
to solve—, which might, at a first glance, seem to defeat the purpose of this
paper. We hence emphasize that LLMs do not fit the bill for the kind of
solution we seek, due to their size and the prohibitive maintenance and
deployment costs, especially in a high-throughput regime as faced by
Wikipedia, where hundreds of thousands of edit summaries need to be generated
every day. For instance, using 100 random edits to calculate the average
prompt length, running GPT-4 with 8k-token context, just for English
Wikipedia, would cost above $1300 per day, assuming the current rate of 3
million edits per month (OpenAI, 2024a). These are considerable expenses and
do not scale to over 300 languages of Wikipedia. Using GPT-3.5 with 4k-token
context, the cost would be around $25 per day. Despite the lower cost of this
model, the OpenAI models are not open-sourced, so using them would be against
the principles of Wikipedia.

Prompt construction. We settle on the five-shot setting, instructing the LLM
to only explain what was done in the edit, as the reason why the edit was
performed is often too difficult to infer from the context. Nonetheless, we
observe that LLMs often generate the reason organically where it is
appropriate.222For instance, for the edit
https://en.wikipedia.org/w/index.php?diff=1172890678 GPT-4 will generate
“Removed unnecessary quotation marks around the name Claudia.”, hinting that
the edit was performed because the quotation marks were unnecessary. The
examples of edits with good summaries, represented with the edit diff between
the revision immediately before vs. after the edit, are used as demonstrations
(see Fig. 1). The edit diff is much shorter than the full revisions, which
makes it easier to fit our prompt into the length constraints imposed by the
LLM. Additionally, the edit diff provides rich information about what was
performed during the edit, omitting a large amount of text that was irrelevant
for the edit. For more details on the prompt tuning and quality check of
generated data, see Appendix C.

###  4.2 Data cleaning and collection

We filter Wikipedia data for training the models with two aspects in mind.
First, _edit summaries for certain types of edits are trivial_. For example,
HotCat, a tool that many editors use to change categories on a page,
automatically generates reasonable summaries via heuristics (e.g. “added
Category:Shoegazing musical groups using
HotCat”,333https://en.wikipedia.org/w/index.php?diff=1033805631). Based on
this, we focus on edits altering the text of the article, where heuristics
struggle and a language model would be well-suited. Second, _existing edit
summaries are of mixed quality_ , which is reflected in the qualitative coding
described in Section 3. This is most salient in IP editors, and, to a lesser
degree, new editors. In this context, we exclude the following edits:

(i) Edits which did not change, insert, or remove at least one sentence in the
article.

(ii) Edits with auto-generated summaries by Mediawiki software (Wikipedia,
2024a).

(iii) Edits made by bots, which often have very good edit summaries, but it is
not useful to have a language model learn edit summaries that have already
been hard-coded into a bot.

(iv) Reverted edits, as many of them are vandalism and unlikely to have a
useful summary.

(v) Edits that made the revert to previous edits, as these often talk about
reason why the revert was performed. These reasons are usually external and
are not easy to infer from the edit diff. As such, they are difficult to be
generated by a language model.

(vi) Edits with blank edit summaries, as all edits should have a basic edit
summary. Many edits have an indicator of which section of the article they
affected, which we removed from all edits as well, so it does not affect our
checking of whether the summary is blank.

We also annotate the edit summaries with the various metadata (e.g. length) to
enable further filtering or balancing of our edit summary sample (see Appendix
B).

###  4.3 Model

Since 4.6% of our data requires input size longer than 512 tokens used by
standard small generative models (Chung et al., 2022; Lewis et al., 2019), as
the model to finetune, we use LongT5 (Guo et al., 2022), which has the ability
to work with longer context windows. We denote each finetuned model as
Edisum[$S\%$], where $S$ is a percentage of synthetic data in the training
set. We intentionally use a very small model, because of limitations of
Wikipedia’s infrastructure. In particular, Wikipedia does not have access to
many GPUs on which we could deploy big models (Wikitech, 2024), meaning that
we have to focus on the ones that can run effectively on CPUs. Note that this
task requires a model running virtually in real-time, as edit summaries should
be created when edit is performed, and cannot be precalculated to decrease the
latency. Models of similar size have already been successfully implemented in
Wikipedia applications. For details on implementation, see Appendix D.

Because this is an unexplored area, with no previous attempts to automatize
the generation of Wikipedia edit summaries, there is no apparent baseline to
compare against. We thus directly compare our method to the actual ground-
truth data: edit summaries written by Wikipedia editors. In addition to that,
we evaluate how close our model is to LLMs. We evaluate GPT-4, today’s highest
performing LLM from OpenAI, and GPT-3.5, which we used to generate synthetic
data. We ran both LLMs on 500 randomly chosen edits from the test data. In the
experiments, GPT-4 and GPT-3.5 were prompted with the same prompt used for
synthetic data generation, and with the generation parameters from Table 3.

##  5 Experimental setup

###  5.1 Data

We use edits made in August 2023 to articles on English Wikipedia. This
includes over 500K edits without a summary, from which we randomly take 100k
edits for synthetic data generation. After the initial filtering from Sec.
4.2, we are left with around 600K edits. We additionally limit the data by
filtering edits with summaries longer than 200 and shorter than 5 characters.
We leave out edits from the editors who have made less than 30 edits and keep
at most 3 edits with the same summary to enforce diversity. This leaves us
with around 127K samples. For experiments, we combine data obtained in both
ways: from existing Wikipedia edit summaries, and by generating synthetic
data. We use in total 100K samples for training, and 10K for validation. The
rest is used for testing ($\sim$17K samples).

We run experiments with 5 different proportions of synthetic data in the
training set (0%, 25%, 50%, 75%, 100%), by choosing the synthetic and human
editor’s data randomly from the collected datasets. As input to the model, we
use the edit diff between the two revisions of the article in question to keep
the input short while preserving the most important information. We extract
the difference and represent the input in the same manner as in Sec. 4.1. To
separate the text from the old and the new revision, we use <old_text> and
<new_text> prefixes. Each sentence is separated by <sent_sep> prefix. We
filter out data points with inputs longer than 1,024 tokens for convenience
(only 2.3% of our data is longer than that). As the output, we use the (human-
or synthetically generated) edit summary.

###  5.2 Evaluation

We perform a twofold evaluation: (1) a cheap and fast-to-conduct automatic
evaluation that uses human-written edit summaries as the ground truth against
which auto-generated summaries are compared; and (2) an expensive and slower-
to-conduct human evaluation, where human raters compare auto-generated to
human-written edit summaries. In the former case, the best a model can do is
reproduce a human-written summary, whereas in the latter case, a model can in
principle outperform humans on this task.

Automatic evaluation. For automatic evaluation, we use MoverScore (Zhao et
al., 2019), designed for measuring the semantic similarity between two texts.
It takes values from 0 to 1 (larger is better), and correlates better with
human judgement than token-matching metrics such as BLEU (Papineni et al.,
2002) or ROUGE (Lin, 2004). This is especially important in settings similar
to ours, where many good outputs with different phrasing may be equally
appropriate. To evaluate a single Edisum model or a single LLM (GPT-4 or
GPT-3.5), for each edit, we take edit diff, generate the automatic summary
with it, and calculate the MoverScore by comparing it to the existing summary.
We obtain the measure of quality for the current Edisum model by averaging
this measure over the whole dataset.

Human evaluation. Although data cleaning increases the overall quality of the
edit summaries we consider, some of them are still misleading or incorrect, as
we do not have a good heuristic to detect this. Yet, MoverScores are obtained
by comparing to those existing edit summaries, which can result in scores that
have little to no meaning. To surpass this limitation, we perform a human
evaluation. We compare two of our models with the summaries written by human
editors and GPT-4 (highest performing model from automatic evaluation). We
choose the best-performing Edisum model according to the MoverScore (cf. Sec.
6.1), Edisum trained with 100% synthetic data (Edisum[100%]). To inspect the
effect of synthetic data on training, we also evaluate the version with no
synthetic data (Edisum[0%]).

We randomly select 100 samples from the testing dataset to perform this
evaluation, from which we discard one sample without a good edit summary
option. Each sample corresponds to a Wikipedia edit, and is associated with a
web page of the edit diff between two revisions444e.g.,
https://en.wikipedia.org/w/index.php?title=Albert_Einstein&diff=prev&oldid=1177682587;
see Fig. 1 for a visual example. For each sample, annotators are presented
with four edit summaries in random order, to prevent bias: ground truth,
Edisum[100%], Edisum[0%], and GPT-4 summary. They are asked to choose the best
and the worst summary out of the four. The task can be seen as ranking with
ties, where the two summaries that were not chosen as neither are tied for the
second place. Since this is not a simple task, to ensure high-quality results,
instead of relying on the crowdsourcing platforms, we recruited 3 MSc students
to perform the annotation. Conflicts were resolved by one of the authors of
the paper. To measure the agreement between the annotators, we report Kendall
rank correlation coefficient (Kendall’s $\tau$) between each pair of them. As
our annotation task can be seen as a ranking task, we choose this as a
suitable measure. For more details on annotation task, see Appendix F.

##  6 Results

(a) Results of Edisum evaluation with MoverScore. Error bars are 95%
confidence intervals (CIs). GPT-4 and GPT-3.5 perform better than Edisum, with
the average MoverScore of 0.724 and 0.722, respectively. We do not show the
performance of GPT-4 and GPT-3.5 credibly on y-axis for convenience, as their
performance is susbstantially higher than for the other models. Note that both
of these are shown as a dot on the plot, as there is no notion of the
percentage of synthetic data in the training set for these models.

(b) Average aggregated scores of human evaluation. Each method was scored with
1 point for winning, 0 points for losing, and 0.5 for neither winning nor
losing. Error bars are 95% confidence intervals (CIs).

(c) Results of human evaluation. _Left:_ % of time summaries from each method
are chosen as the best. _Right:_ % of time summaries from each method are
chosen as the worst. Error bars are 95% confidence intervals (CIs).

Figure 2: Results of automatic and human evaluation.

###  6.1 Automatic evaluation

In Fig. 2(a), we present the results of automatic evaluation. Performance of
all Edisum models is decent, according to the MoverScore. The performance of
Edisum[0%] is lower than the models with some fraction of synthetic data, in
particular Edisum[75%] and Edisum[100%], for which this difference is also
statistically significant. This confirms our assumption that synthetic data
can be a useful asset when tackling the task of edit summary generation. One
might be surprised that a fully-synthetic training set results in higher score
when comparing to the existing data than the training set with only existing
data, but this is not unexpected. Existing data has more structural variety
and features various Wikipedia tags, which can be hard for a language model to
pick up. Synthetic data might not have the same surface form as the existing
data, but it expresses the key information about the edit while maintaining
simpler structure, making it easier to train on.

When it comes to LLMs, as anticipated, the results show that they effectively
solve this task, achieving scores higher than for any of the Edisum models.
The difference between GPT-4 and GPT-3.5 is small. We suspect this happens
because we did not tune the prompt or generation parameters specifically to
GPT-4. Further tuning could only improve the results, confirming the
usefulness of these models. These models, however, are not suitable for use in
a setting like Wikipedia, where large number of editors are performing
thousands of edits every day (See Sec. 4.1 for cost estimation and
limitations). This is why it is essential to have a smaller model that can do
a decent enough job. For trickier edits, we could still reach out to an LLM.
This approach would tremendously lower the costs of running such a system.

###  6.2 Human evaluation

Recall that in the human evaluation, for each edit, raters were asked to pick
the best and the worst one out of four summaries, each generated by one of
four methods: human editors, Edisum[0%], Edisum[100%], and GPT-4. As
described, the four candidate summaries for each edit were evaluated by three
independent raters. Inter-rater agreement, measured in terms of Kendall’s
$\tau$, was 0.588, 0.556, and 0.562 for the three pairs of raters, indicating
a relatively strong positive agreement among the raters. To compute a single
performance score per method, we awarded a method a score of 1 if it was
chosen as the best one, a score of 0 as the worst one, and a score of 0.5 if
it was not chosen as neither. In Fig. 2(b), we report the average score
obtained by each method. From the plot, we observe that GPT-4 scores best and
Edisum[0%] scores worst, while the average scores of Edisum[100%] and human
editors are nearly identical and not statistically significantly different.

To investigate further, Fig. 2(c) analyzes wins and losses separately. The
left and right subfigure show the percentage of edits for which each method
was chosen as the best and worst, respectively. In line with Fig. 2(b), GPT-4
is chosen the most often as the best model and the least often as the worst,
while Edisum[0%] is the opposite. Most importantly, the human editors and
Edisum[100%] are tied on a middle ground, with the editors being chosen
slightly more often as the best, but also as the worst, compared to
Edisum[100%]. Integrating Fig. 2(b) and 2(c), we conclude that Edisum[100%]
performs equally well as human editors, but with less variance: it achieves
similar average ranking scores as the human editors (Fig. 2(b)), while taking
extreme positions less often than it (Fig. 2(c)).

Overall, these results confirm the conclusions from the automatic evaluation.
The positive effects of synthetic training data are even more evident here.
Similarly, GPT-4 is again observed to generate edit summaries of the highest
quality. However, as noted in Sec. 4.1 and Sec. 6.1, running such a system on
a daily basis on a platform as big as Wikipedia for all the edits would not be
feasible today. Our “distilled” Edisum[100%] model, which aims to mimic GPT’s
high-quality summaries, offers a fertile middle ground, performing as well as
humans while being much smaller and cheaper to run.

###  6.3 Error analysis

| What | Why
---|---|---
Method | Correct | No change | Not specific | Unclear | Unexhaustive | Unrelated | Correct | Incorrect | Missing
_Human editors_ |  |  |
Win | 0.65 ± 0.07 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.05 ± 0.03 | 0.25 ± 0.06 | 0.05 ± 0.03 | 0.70 ± 0.06 | 0.00 ± 0.00 | 0.30 ± 0.06
Lose | 0.15 ± 0.06 | 0.00 ± 0.00 | 0.15 ± 0.05 | 0.39 ± 0.06 | 0.23 ± 0.06 | 0.08 ± 0.03 | 0.23 ± 0.06 | 0.08 ± 0.03 | 0.69 ± 0.06
Neither | 0.59 ± 0.06 | 0.00 ± 0.00 | 0.35 ± 0.07 | 0.06 ± 0.03 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.53 ± 0.07 | 0.00 ± 0.00 | 0.47 ± 0.07
_GPT-4_ |  |  |
Win | 0.92 ± 0.03 | 0.00 ± 0.00 | 0.04 ± 0.03 | 0.00 ± 0.00 | 0.04 ± 0.03 | 0.00 ± 0.00 | 0.36 ± 0.07 | 0.00 ± 0.00 | 0.64 ± 0.06
Neither | 0.40 ± 0.07 | 0.00 ± 0.00 | 0.04 ± 0.03 | 0.00 ± 0.00 | 0.32 ± 0.06 | 0.24 ± 0.06 | 0.48 ± 0.06 | 0.12 ± 0.04 | 0.40 ± 0.06
_Edisum[100%]_ |  |  |
Win | 0.63 ± 0.07 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.28 ± 0.06 | 0.09 ± 0.04 | 0.45 ± 0.06 | 0.00 ± 0.00 | 0.54 ± 0.07
Lose | 0.00 ± 0.00 | 0.14 ± 0.05 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.14 ± 0.05 | 0.71 ± 0.06 | 0.14 ± 0.05 | 0.00 ± 0.00 | 0.86 ± 0.04
Neither | 0.34 ± 0.06 | 0.00 ± 0.00 | 0.09 ± 0.04 | 0.06 ± 0.03 | 0.19 ± 0.05 | 0.31 ± 0.06 | 0.28 ± 0.06 | 0.16 ± 0.05 | 0.56 ± 0.07

Table 2: Error analysis results.

To further examine the difference in performance between GPT-4 and other
methods, we manually inspect 150 edit summaries from two perspectives: “why”
(description of why the edit was performed) and “what” (description of what
was done). The samples were chosen to cover all the cases (win, lose or
neither) for all three methods. For details on the annotation procedure and
taxonomy, see Appendix G. The results are presented in Table 2.

When observing results for “why” meta-category, we notice, as expected, that
human written summaries express the correct reason why the edit was performed
more often than the ones generated by GPT-4 or Edisum. However, both methods
frequently express the, usually correct, reason. This reflects the edits for
which reason can be inferred from the context. When it comes to the results
for “what” category, the performance gap between GPT-4 and other methods is
still visible. Specifically for Edisum, we can attribute the drop in
performance to its size. Edisum is a very small model (223M parameters),
incapable of fully capturing patterns present in more complex tasks, like edit
summary generation. The distribution of errors for GPT-4 and Edisum for
summaries that were not chosen as neither the best or the worst is very
similar, with the most errors being attributed to the unrelated or
unexhaustive summaries. On the other hand, human editor’s summaries from this
category tend to be less specific or unclear. This holds also for the
summaries which were chosen as the worst ones. Summaries that won were most
often not exhaustive enough. Overall, the analysis confirms that Edisum is a
suitable option to use at scale.

##  7 Future work

As a new language modeling task, this paper establishes what LLMs can
accomplish while also demonstrating the feasibility of using much smaller
language models to perform the task. Alongside these new benchmarks, this
research points towards areas for improvement in future modeling efforts. For
example, while we find that models trained on synthetic data outperform those
trained on existing edit summaries on Wikipedia, this approach likely has
limitations in learning editor community norms such as common abbreviations.
We limited our training samples to edit summaries by editors with at least 30
edits based on our qualitative analysis of existing edit summaries, but future
work could explore additional strategies for producing a high-quality, diverse
dataset of existing edit summaries. Kocetkov et al. (2022) found significant
improvements from applying near de-duplication to their code dataset and we
suspect that many edits are quite similar with minor differences and a similar
pipeline might bring improvements to this task as well. The other major avenue
of exploration, especially for smaller models, is how to represent an edit
diff to the model. We chose a structured approach that extracted the changed
sentences but this does reduce the context provided to the model, which might
be beneficial for generating appropriate summaries for some edits. Future work
could explore alternative approaches to determine the right balance of
simplicity and efficiency to providing sufficient context for the model.

##  8 Conclusion

In this paper, we investigate the quality of Wikipedia edit summaries, i.e.,
short comments that editors write when performing changes in Wikipedia. These
summaries serve a wide range of purposes in Wikipedia, but also for general
research community. We find that a non-negligible number of them is of bad
quality or missing. At the same time, we show that GPT-4 is able to solve this
task better than human editors at small scale. To assist editors, we train a
small language model that can, unlike GPT-4, effectively generate edit
summaries on a large scale while matching the performance of human editors.

## References

  * Anaby-Tavor et al. (2020) Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling.  Do not have enough data? deep learning to the rescue!  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pp. 7383–7390, 2020.
  * Arazy et al. (2016) Ofer Arazy, Johannes Daxenberger, Hila Lifshitz-Assaf, Oded Nov, and Iryna Gurevych.  Turbulent stability of emergent roles: The dualistic nature of self-organizing knowledge coproduction.  _Information Systems Research_ , 27(4):792–812, 2016.
  * Asthana et al. (2021) Sumit Asthana, Sabrina Tobar Thommel, Aaron Lee Halfaker, and Nikola Banovic.  Automatically labeling low quality content on wikipedia by leveraging patterns in editing behaviors.  _Proceedings of the ACM on Human-Computer Interaction_ , 5(CSCW2):1–23, 2021.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei.  Scaling instruction-finetuned language models, 2022.
  * Faltings et al. (2021) Felix Faltings, Michel Galley, Gerold Hintz, Chris Brockett, Chris Quirk, Jianfeng Gao, and Bill Dolan.  Text editing by command.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 5259–5274, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.414.  URL https://aclanthology.org/2021.naacl-main.414.
  * Gao et al. (2022) Jiahui Gao, Renjie Pi, Yong Lin, Hang Xu, Jiacheng Ye, Zhiyong Wu, Xiaodan Liang, Zhenguo Li, and Lingpeng Kong.  Zerogen+: Self-guided high-quality data generation in efficient zero-shot learning, 2022.  URL https://arxiv.org/abs/2205.12679.
  * Geiger & Ribes (2010) R. Stuart Geiger and David Ribes.  The work of sustaining order in wikipedia: The banning of a vandal.  In _Proceedings of the 2010 ACM Conference on Computer Supported Cooperative Work_ , CSCW ’10, pp. 117–126, New York, NY, USA, 2010. Association for Computing Machinery.  ISBN 9781605587950.  doi: 10.1145/1718918.1718941.  URL https://doi.org/10.1145/1718918.1718941.
  * Geiger & Ribes (2011) R Stuart Geiger and David Ribes.  Trace ethnography: Following coordination through documentary practices.  In _2011 44th Hawaii international conference on system sciences_ , pp. 1–10. IEEE, 2011.
  * Guo et al. (2022) Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang.  Longt5: Efficient text-to-text transformer for long sequences, 2022.
  * Josifoski et al. (2023) Martin Josifoski, Marija Sakota, Maxime Peyrard, and Robert West.  Exploiting asymmetry for synthetic training data generation: Synthie and the case of information extraction, 2023.
  * Kocetkov et al. (2022) Denis Kocetkov, Raymond Li, LI Jia, Chenghao Mou, Yacine Jernite, Margaret Mitchell, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, et al.  The stack: 3 tb of permissively licensed source code.  _Transactions on Machine Learning Research_ , 2022.
  * Kumar et al. (2020) Varun Kumar, Ashutosh Choudhary, and Eunah Cho.  Data augmentation using pre-trained transformer models.  _arXiv preprint arXiv:2003.02245_ , 2020.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension, 2019.
  * Li et al. (2022) Junlong Li, Zhuosheng Zhang, and Hai Zhao.  Self-prompting large language models for open-domain qa, 2022.  URL https://arxiv.org/abs/2212.08635.
  * Lin (2004) Chin-Yew Lin.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pp. 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics.  URL https://aclanthology.org/W04-1013.
  * Meng et al. (2022) Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han.  Generating training data with language models: Towards zero-shot language understanding, 2022.  URL https://arxiv.org/abs/2202.04538.
  * Mohapatra et al. (2020) Biswesh Mohapatra, Gaurav Pandey, Danish Contractor, and Sachindra Joshi.  Simulated chats for building dialog systems: Learning to generate conversations from instructions.  _arXiv preprint arXiv:2010.10216_ , 2020.
  * Morgan (2019) Jonathan Morgan.  Patrolling on wikipedia, 2019.  URL https://meta.wikimedia.org/wiki/Research:Patrolling_on_Wikipedia.
  * OpenAI (2024a) OpenAI.  OpenAI model pricing.  https://openai.com/pricing, 2024a.  Accessed: 2024-03-12.
  * OpenAI (2024b) OpenAI.  OpenAI models documentation.  https://platform.openai.com/docs/models, 2024b.  Accessed: 2024-03-12.
  * Panciera et al. (2009) Katherine Panciera, Aaron Halfaker, and Loren Terveen.  Wikipedians are born, not made: a study of power editors on wikipedia.  In _Proceedings of the 2009 ACM International Conference on Supporting Group Work_ , pp. 51–60, 2009.
  * Papanikolaou & Pierleoni (2020) Yannis Papanikolaou and Andrea Pierleoni.  Dare: Data augmented relation extraction with gpt-2.  _arXiv preprint arXiv:2004.13845_ , 2020.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: A method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics_ , ACL ’02, pp. 311–318, USA, 2002. Association for Computational Linguistics.  doi: 10.3115/1073083.1073135.  URL https://doi.org/10.3115/1073083.1073135.
  * Pavalanathan et al. (2018) Umashanthi Pavalanathan, Xiaochuang Han, and Jacob Eisenstein.  Mind your pov: Convergence of articles and editors towards wikipedia’s neutrality norm.  _Proceedings of the ACM on Human-Computer Interaction_ , 2(CSCW):1–23, 2018.
  * Sahu et al. (2022) Gaurav Sahu, Pau Rodriguez, Issam H. Laradji, Parmida Atighehchian, David Vazquez, and Dzmitry Bahdanau.  Data augmentation for intent classification with off-the-shelf large language models, 2022.  URL https://arxiv.org/abs/2204.01959.
  * Schick et al. (2022) Timo Schick, Jane Dwivedi-Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebastian Riedel.  Peer: A collaborative language model, 2022.
  * Shao et al. (2023) Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen.  Synthetic prompting: Generating chain-of-thought demonstrations for large language models, 2023.  URL https://arxiv.org/abs/2302.00618.
  * Stvilia et al. (2008) Besiki Stvilia, Michael B Twidale, Linda C Smith, and Les Gasser.  Information quality work organization in wikipedia.  _Journal of the American society for information science and technology_ , 59(6):983–1001, 2008.
  * Sumi et al. (2011) Robert Sumi, Taha Yasseri, Andr´s Rung, Andr´s Kornai, and J´nos Kertesz.  Edit wars in wikipedia.  In _2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing_. IEEE, oct 2011.  doi: 10.1109/passat/socialcom.2011.47.  URL https://doi.org/10.1109%2Fpassat%2Fsocialcom.2011.47.
  * Veselovsky et al. (2023) Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West.  Artificial artificial artificial intelligence: Crowd workers widely use large language models for text production tasks, 2023.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  _arXiv preprint arXiv:1804.07461_ , 2018.
  * Wang et al. (2021) Zirui Wang, Adams Wei Yu, Orhan Firat, and Yuan Cao.  Towards zero-label language learning.  _CoRR_ , abs/2109.09193, 2021.  URL https://arxiv.org/abs/2109.09193.
  * Wattenberg et al. (2007) Martin Wattenberg, Fernanda B Viégas, and Katherine Hollenbach.  Visualizing activity on wikipedia with chromograms.  In _Human-Computer Interaction–INTERACT 2007: 11th IFIP TC 13 International Conference, Rio de Janeiro, Brazil, September 10-14, 2007, Proceedings, Part II 11_ , pp. 272–287. Springer, 2007.
  * Wikimedia (2024a) Wikimedia.  Wikimedia foundation guiding principles.  https://foundation.wikimedia.org/wiki/Resolution:Wikimedia_Foundation_Guiding_Principles, 2024a.  Accessed: 2024-03-12.
  * Wikimedia (2024b) Wikimedia.  Edit summary.  https://meta.wikimedia.org/wiki/Help:Edit_summary, 2024b.  Accessed: 2024-03-12.
  * Wikipedia (2024a) Wikipedia.  Automatic edit summaries.  https://en.wikipedia.org/wiki/Help:Automatic_edit_summaries, 2024a.  Accessed: 2024-03-12.
  * Wikipedia (2024b) Wikipedia.  Canned edit summaries.  https://en.wikipedia.org/wiki/Wikipedia:Canned_edit_summaries, 2024b.  Accessed: 2024-03-12.
  * Wikipedia (2024c) Wikipedia.  Wikipedia statistics on number of edits performed.  https://stats.wikimedia.org/#/en.wikipedia.org/contributing/edits/normal|bar|2-year|editor_type~anonymous*group-bot*name-bot*user+(page_type)~content|monthly, 2024c.  Accessed: 2024-03-12.
  * Wikipedia (2024d) Wikipedia.  Wikipedia revision deletion.  https://en.wikipedia.org/wiki/Wikipedia:Revision_deletion, 2024d.  Accessed: 2024-03-12.
  * Wikitech (2024) Wikitech.  Wikipedia: Access to gpus.  https://wikitech.wikimedia.org/wiki/Machine_Learning/AMD_GPU#Do_we_have_Nvidia_GPUs, 2024.  Accessed: 2024-03-12.
  * Yang et al. (2017) Diyi Yang, Aaron Halfaker, Robert Kraut, and Eduard Hovy.  Identifying semantic edit intentions from revisions in wikipedia.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pp. 2000–2010, 2017.
  * Ye et al. (2022) Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong.  Zerogen: Efficient zero-shot learning via dataset generation, 2022.  URL https://arxiv.org/abs/2202.07922.
  * Zhao et al. (2019) Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger.  MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 563–578, Hong Kong, China, November 2019.  doi: 10.18653/v1/D19-1053.  URL https://aclanthology.org/D19-1053.

##  Appendix A Qualitative analysis annotation process

Data sample. The qualitative analysis was performed on 100 samples, as
annotation of edits is a lengthy process. To ensure a diverse enough group of
edits, we stratify the sample based on the experience of editors. More
precisely, we divide editors in four categories: IP editors (anonymous
editors), newcomers (editors with < 10 edits), mid-experienced editors (10 -
1000 edits), and experienced editors (1000+ edits). We exclude edits by
bots,555Anecdotally, many bots actually have very good edit summaries
generated by their code but also their edits are usually straightforward to
describe. edits without summaries, and revert-related edits to focus on good-
faith contributions to English Wikipedia. This means that our sample likely
lacks highly-inappropriate comments, as they would be removed by the editors.
Notably, 46% of the edits from August 2023 do not have a summary. The
proportion varies greatly by editor type: 74% of edits for IP editors, 46% of
edits for newcomers, 58% for medium-experienced editors, and 38% of edits for
experienced editors. This highlights the value of better support for
generating edit summaries.

Annotation. Annotation was done by the two authors of the paper. A discussion
was held after the first ten summaries to ensure there was agreement on the
codebook before completing the sample.

##  Appendix B Data annotation

We annotated the cleaned edit data as follows:

(i) Frequency with which the edit summary appears in our dataset. This enables
us to sample the dataset to be more diverse. To check frequency, we lower-case
all characters and replace any links with a generic link character before
calculating frequency.

(ii) Editor’s edit count. This enables us to up-sample edits from more
experienced editors, who are expected to be more likely to write correct edit
summaries.

(iii) Summary length. While very short summaries are okay (e.g., “ce” is often
used to stand for “copy-edit” and indicate small grammar or spelling changes),
summaries are limited to 500 characters and the English Wikipedia community
suggests to avoid unnecessarily long
summaries.666https://en.wikipedia.org/wiki/Help:Edit_summary

(iv) User frequency in the dataset. A small number of editors make a large
proportion of edits on Wikipedia, and while they may write reasonable edit
summaries, we want to learn from a diverse sample of Wikipedians.

(v) Semi-automated edits. If an edit is made through a tool that enables very
quick editing or has several preset edit summaries, we flag this, as these
edit summaries are unlikely to be strongly contextualized to the specific
edit.

##  Appendix C Synthetic data generation process

Figure 3: System instruction and the example of a demonstration used for
synthetic data generation.

Prompt choice. Experimentation process for choosing the prompt is done of 10
samples of edit diffs. For each one of them, we generate an edit summary with
different prompts, and after manual inspection, we settle on the prompt that
is used for synthetic data generation. We experiment with different
instructions and different numbers of demonstrations, as well as their
content.

For the instruction, as already mentioned in Sec. 4.1, we only focused on
asking the LLM to explain what was performed in the edit. We also explained
the format of the edit summary and the input, and gave a few guidelines to
follow. For the full instructions, see Fig. 3.

For the demonstrations, as explained in Sec. 4.1 we provide the LLM with the
edit diff between the two revisions immediately before vs. immediately after
the edit. We extract this diff using the
mwedittypes777https://pypi.org/project/mwedittypes/ library. From the output
of this library, we can extract sentences that were added and removed in the
editing process. We group all the removed sentences into “old text” and all
the added sentences into “new text”. On 100 randomly chosen and manually
inspected edit diff outputs using this library, in 4 cases, these sentences
are not ordered by the way they are appearing in the revision of the Wikipedia
page. Because of that, we order the sentences in “old text” and “new text”
alphabetically, to avoid confusion. We then represent the diff by
concatenating both of those, separating them by stating “old text:” and “new
text:” before each group. For an example of a demonstration, see Fig. 3.

When choosing the demonstrations, we make sure to include both longer and
shorter edits in terms of content, and also edits with summaries of various
length. We include both demonstrations for edits that add and remove content.
We tried out different numbers of demonstrations ranging from 2 to 10 (2, 3,
5, 6, and 10 demonstrations), using same demonstrations for each edit summary
generated. We settled on five demonstrations, which we found to provide us
with sufficient information to generate high-quality data, while keeping the
length of the input shorter, and consequently, the cost of the generation
process smaller. The same five demonstrations are used for all the generated
samples.

parameter | max_tokens | temperature | top-p | frequency_penalty | presence_penalty | stop | n | best_of
---|---|---|---|---|---|---|---|---
value | 1000 | 0 | 1 | 0.2 | 0 | "\n" | 1 | 1

Table 3: Generation parameters used with gpt-3.5-turbo to generate synthetic
data. These parameters were also used when testing GPT-4 and GPT-3.5
performance on the testing dataset.

Generation parameters and process. We decide to only work with edits that have
textual changes and exclude the ones with changes in the Wiki markup, such as
category modifications or templates. We do this because this is where we
expect the language model to give us the biggest gains, as this is where the
biggest variety of different edits are performed. We experimented with
different generation parameters for the OpenAI models. In particular, we tried
out different values of temperature, top-$p$, and frequency penalty. We make
the decision on the best parameters manually, based on the same 10 samples we
used for prompt construction. The set of parameters for the best-performing
setup is displayed in Table 3.

Quality check. To verify that the quality of generated synthetic data is
satisfying, we perform a quality check on 100 random edits, by comparing the
data generated by GPT-3.5 with the existing one. We find that synthetic data
has satisfying quality more often (87% vs. 78% of the time). On top of that,
in 30% of the cases in which both summaries were seen as suitable, the
generated one was chosen as better 30% of the time (vs 4% for the existing
summaries).

##  Appendix D Implementation hyperparamters and details

We used the long-t5-local-base888https://huggingface.co/google/long-t5-local-
base as our base model which we then finetune on the collected training data
(see above). The models were trained using the Adam optimizer with learning
rate $3\times 10^{-4}$, 0.1 gradient clipping on the Euclidean norm, and
weight decay 0.05. They were trained for 10 epochs with batch size 2 and a
polynomial learning-rate scheduler with 1,000 warm-up steps and a final
learning rate of $3\times 10^{-5}$. Training was performed on a single NVIDIA
Titan X Maxwell 12GB GPU, taking around 30 hours for each model.

##  Appendix E Examples of ground truth and generated edit summaries

edit |  human editor |  Edisum [0%] |  Edisum [100%] |  GPT-4
---|---|---|---|---
https://en.wikipedia.org/w/index.php?diff=1172472704 |  added the range |  added location |  Added location information |  Added information about the location and altitude of Boveys Lookout and the orientation of the Burton Range.
https://en.wikipedia.org/w/index.php?diff=1169223741 |  in 1982 |  Added date and citation. |  Added the year of the speech |  Added the year of Fidel Castro’s speech.
https://en.wikipedia.org/w/index.php?diff=1169701032 |  added artist |  Added a wikilink and a link |  Added a new name to the list of artists |  Added Stefan Brüggemann to the list of artists whose works are included.
https://en.wikipedia.org/w/index.php?diff=1170976899 |  lede; infobox |  infobox (inapplicable); lede; -cats |  Clarified the nature of Denis- Auguste Affre’s cause and added information about his title |  Updated the description of Denis-Auguste Affre’s religious status and clarified the stage of his canonization process.
https://en.wikipedia.org/w/index.php?diff=1170699096 |  fixed link to aias awards, it was at 5th, not 7th |  5th |  Corrected the award number from 7th to 5th |  Corrected the year of the Annual Interactive Achievement Awards nomination.
https://en.wikipedia.org/w/index.php?diff=1169516533 |  Amended lists so they are split at 100 m |  ""at least", not "at least"" |  Clarified the range of heights in the list |  Added information about two different height categories for the ranking of buildings and structures in Greater Manchester.
https://en.wikipedia.org/w/index.php?diff=1173169263 |  UPD Romelu Lukaku |  updated Lukaku |  Removed the end of the season |  Removed incorrect future date for Romelu Lukaku’s tenure at Inter.
https://en.wikipedia.org/w/index.php?diff=1171641782 |  "This \"however\" doesn’t make sense here" |  "rmv deprecated, ambiguous "however" per MOS:RELTIME" |  Removed unnecessary word |  Removed the word "However," from the beginning of the sentence.
https://en.wikipedia.org/w/index.php?diff=1168487383 |  Per feedback given in GA review, elaborated on David King Udall and Ella Stewart Udall in the lead and in Ella Udall’s first mention; David Udall was already identified |  Added link; improved phrasing and punctuation |  Added additional information about the individuals’ professions |  Added professional details for David King Udall, Ella Stewart Udall, and Mary Ann Linton Morgan Udall.

Table 4: Examples of summaries written by human editors and generated by
Edisum and GPT-4.

In Table 4, we present some of the existing edit summaries, as well as the
ones generated with two of our models, Edisum[100%] and Edisum[0%], and GPT-4.

##  Appendix F Human evaluation setup

Annotation task. As mentioned in Sec. 5.2, human evaluation was done on 100
samples from the testing dataset, each associated with four edit summaries
from different methods, a different Wikipedia edit, and presented with the
corresponding web page. The sample size is relatively small as grading these
edit summaries is a long and tedious process – each annotator has to manually
assess the edit diff, and sometimes even the whole revisions of the article,
in order to understand what was done in it. From the web page, we remove the
element showing the actual human edit summary to make sure that the annotators
are not aware of the existing edit summary. The web page also shows the
“current” version of the article, right after the edit, in case the annotators
need more context to give their judgement.

Annotators were asked to choose the best and the worst summary out of the four
according to the following guidelines:

_A good edit summary should_:

(1) Summarize what was done in the edit

(2) Cover all the changes performed (either explicitly or by adding something
like “and misc”)

(3) Be specific; e.g., a summary “I made some changes” is not specific

(4) Explain why the change was made, if it is unclear from the change itself

_A good edit summary should not_:

(1) Use uncommon abbreviations

(2) Be too long: it is not supposed to be a paragraph, but a sentence-long
summary

(3) Attack other editors’ work or be aggressive

Annotators were provided with examples of good and bad edit summaries, with
explanations what makes them good or bad. They were also given Wikipedia’s
manual of style999https://en.wikipedia.org/wiki/Wikipedia:Manual_of_Style to
get familiar with tags that often appear in edit summaries. Finally, to ensure
the high-quality of the results, we train them on a few selected samples,
teaching them what to look for in the edit summary and making sure they
understood the guidelines and the assignment.

Annotators. As mentioned in Sec. 5.2, we recruit 3 MSc students as annotators
for our task. We opt for this option over crowdsourcing platforms for several
reasons, all of which ensure high-quality annotations. This task is not
trivial for a person not familiar with Wikipedia’s norms and rules. For
instance, large fraction of edit summaries references Wikipedia’s manual of
style which might look like irrelevant words to a layperson. By recruiting
students, we had more control over the quality of annotators we are taking.
All of the annotators were MSc students in computer science, familiar with
Wikipedia, but not with our work. On top of that, we had a more
straightforward way to train MSc students for this task, as this might be a
tricky thing to do with crowdsourcing platforms. Finally, another increasing
concern that comes with the use of crowdsourcing platforms is the usage of
LLMs by the crowd workers, who today frequently “outsource” text-processing
tasks to LLMs to facilitate their work (Veselovsky et al., 2023).

The students were paid the equivalent of US$25 per hour for their work.
Conflicts were resolved by one of the authors of the paper. To measure the
agreement between the annotators, we report Kendall rank correlation
coefficient (Kendall’s $\tau$) between each pair of them. Kendall’s $\tau$ is
a statistical measure used to assess the degree of association or correlation
between two sets of rankings or ordinal data. As our task is a version of a
ranking task, we opt to use this metric. For each pair of annotators, and for
each sample with four summaries that are rated, we calculate Kendall’s $\tau$.
To get the value of Kendall’s $\tau$ for all 99 samples, we take the average.
This way, we report three numbers, a value for each pair of annotators.

##  Appendix G Annotation procedure for error analysis

Data. We annotate in total 150 samples, 50 for each method (GPT-4,
Edisum[100%] and human editors). For each method, samples were chosen to have
as similar as possible fractions of samples that won, lost or were not chosen
as either. For GPT-4, there was only 1 sample chosen as the worst, so we do
not include this category in the analysis.

Annotation procedure. Samples were annotated according to two meta-categories:
“what” (content of the edit) and “why” (reason for the edit). Taxonomies for
each of the meta-categories ware derived after manual inspection of the subset
of the outputs. For “why”, we settle on three simple categories: missing,
correct, and incorrect. For “what”, we derived the following categories:

  1. 1.

Correct: edit summary is fully correct and exhaustive

  2. 2.

No change: the summary indicates that no change was performed

  3. 3.

Not specific: the summary is not describing exact changes that were performed

  4. 4.

Unclear: the summary seems to be pointing to the correct modifications, but is
hard to understand without looking at the diff

  5. 5.

Unexhaustive: the summary does not cover all changes performed

  6. 6.

Unrelated: the summary describes unrelated edit

Annotation was done by one of the paper authors. The annotator chose one of
the categories from the taxonomy for each of the presented edit summaries. To
confirm the validity of the results, another author annotated 30 random
samples. We calculated the agreement between the two annotators on those 30
samples using Cohen’s kappa. For “what” meta-category, Cohen’s kappa is 0.60,
while for “why” it is 0.67. Both of these numbers indicate high overlap
between the annotators.


11institutetext: Athinoula A. Martinos Center for Biomedical Imaging,
Massachusetts General Hospital and Harvard Medical School, Charlestown, MA,
United States 22institutetext: Department of Pharmacology and Physiology,
University of Rochester School of Medicine, Rochester, NY, United States
33institutetext: Wellcome Centre for Integrative Neuroimaging, FMRIB Centre,
Nuffield Department of Clinical Neurosciences, University of Oxford, UK
44institutetext: McLean Hospital, Belmont, MA, United States
44email<EMAIL_ADDRESS>
# Constrained self-supervised method with temporal ensembling for fiber bundle
detection on anatomic tracing data

Vaanathi Sundaresan 11 Julia F. Lehman 22 Sean Fitzgibbon 33 Saad Jbabdi 33
Suzanne N. Haber 22 4 4 Anastasia Yendiki 11

###### Abstract

Anatomic tracing data provides detailed information on brain circuitry
essential for addressing some of the common errors in diffusion MRI
tractography. However, automated detection of fiber bundles on tracing data is
challenging due to sectioning distortions, presence of noise and artifacts and
intensity/contrast variations. In this work, we propose a deep learning method
with a self-supervised loss function that takes anatomy-based constraints into
account for accurate segmentation of fiber bundles on the tracer sections from
macaque brains. Also, given the limited availability of manual labels, we use
a semi-supervised training technique for efficiently using unlabeled data to
improve the performance, and location constraints for further reduction of
false positives. Evaluation of our method on unseen sections from a different
macaque yields promising results with a true positive rate of $\sim$0.90. The
code for our method is available at
https://github.com/v-sundaresan/fiberbundle_seg_tracing

###### Keywords:

Anatomic tracing fiber bundle detection self-supervised contrastive loss.

##  1 Introduction

Diffusion MRI (dMRI) allows us to probe the macroscopic organization and the
microscopic features of white matter (WM) pathways in vivo, and to study their
role in psychiatric and neurological disorders [1, 2]. However, dMRI can only
provide indirect measurements of axonal orientations based on water diffusion,
and only at the mm scale. As a result, dMRI tractography sometimes fails,
particularly in areas of complex fiber configurations, such as branching,
fanning, or sharp turns [2, 3, 4]. In contrast, anatomic tracing in non-human
primates enables us to follow the trajectory of individual axons. As the
fibers travel from an injection site, split into different fiber bundles and
reach their terminal fields, they provide in-depth knowledge of how the brain
is actually wired [5, 6, 7, 8]. Example tracer data from an injection site at
the frontopolar cortex of a macaque monkey are shown in Fig. 1. Anatomic
tracing has been used to visualize tracts that are challenging for dMRI
tractography [3, 7, 9]. For instance, WM fibers from prefrontal cortex travel
through the gray matter of the striatum in small fascicles before entering the
internal capsule (IC) [7, 9]. From the IC, these fibers also enter anterior
commissure perpendicular to the its main fiber tract orientation [9]. These
discontinuities and tortuous trajectories of fibers confound dMRI
tractography, but can be visualized clearly with anatomic tracing. However,
the manual charting of fiber bundles on histological slides is extremely time
consuming and labor-intensive, limiting the availability of annotated tracer
data for large-scale validation studies.

Figure 1: Photomicrographs showing coronal sections (1-3); (1a, 2a, 3a)
terminal fields at different cortical locations. In the rostrocaudal
direction, a fiber bundle stalk (1b) branches into two fiber bundles in
prefrontal white matter (2b) and travels laterally in the external capsule
(3b) and medially in the corpus callosum (3c). Manual chartings of dense and
moderate bundles shown in green and orange respectively.

The goal of this work is to accelerate this process by developing an accurate,
automated method for detecting fiber bundles in histological sections.
Existing work on automated fiber bundle segmentation on tracer data is quite
scarce, and has been done primarily in marmoset brains [10]. In contrast with
these new world primates, the cortex of old world primates, (i.e. macaques) is
evolutionarily much closer to humans and, as such the trajectory of fibers are
more similar [5, 11]. In general, automated fiber bundle detection on anatomic
tracing data is challenging due to various confounding structures (e.g.,
terminal fields), sectioning distortions, localized and speckle artifacts and
varying background intensities/textures. In this work, we propose a deep
learning-based method for fully automated, accurate fiber bundle detection for
the first time on tracer data from macaque brains, using only a few manually
labeled sections. We use a multi-tasking model with anatomy-constrained self-
supervised loss and utilise continuity priors to ensure accurate detection and
to avoid false positives (FPs). So far, various semi/self-supervised and
ensembling techniques have been shown to work well on noisy data and limited
labels with uncertainties [12, 13, 14, 15, 16]. Given the shortage of manually
annotated tracer data, we successfully adapt a semi-supervised training
technique to improve fiber bundle detection. We also evaluate the robustness
of fiber bundle detection by validating our method on unseen sections from a
different macaque brain.

In addition to segmenting fiber bundles, our automated tool can also provide
further quantification of fibers (e.g., density, volume). It is publicly
available and we plan to deploy it for quantitative analyses of tracer data,
and for large-scale validation studies where the accuracy of tractography
algorithms will be evaluated across multiple seed areas.

##  2 Method

For training our method, we used (1) an encoder-decoder architecture for
segmenting fiber bundles, while simultaneously discriminating fiber bundles
from background using self-supervised contrastive loss, and (2) a temporal
ensembling framework to efficiently use sections without manual charting from
different brains.

###  2.1 Self-supervised, temporal ensembling framework

Anatomy-constrained self-supervised learning: We used a 2D U-Net [17] to build
a multi-tasking model as shown in Fig. 2, since U-Net is one of the most
successful architectures for medical image segmentation tasks [18]. The multi-
tasking model consists of a U-Net backbone ($F_{Seg}$) for segmenting the
fiber regions/bundles and an auxiliary classification arm ($F_{Class}$) for
discriminating fiber patches from background patches. We provided randomly
sampled RGB patches of size 256 $\times$ 256 $\times$ 3 as input. $F_{Class}$
is connected to the bottleneck of the encoder of $F_{Seg}$, where the feature
maps are passed through a downsampling module followed by two fully connected
layers (fc1024, fc256) and an output layer with 2 nodes (fiber bundle vs
background). The downsampling module consists of 2 max-pooling layers, each
followed by two 3 $\times$ 3 convolution layers to extract high-level global
features in the patches. We used focal loss (eqn. 1) for training $F_{Seg}$,
since it handles class imbalance well [19]. The focal loss is given by:

$\vspace{-1em}FL(p_{t})=-\alpha_{t}(1-p_{t})^{\gamma}log(p_{t}),\quad
p_{t}=\begin{cases}p,&\text{if}~{}y=1\\\ (1-p),&\text{otherwise}\end{cases}$
(1)

where $\alpha$ and $\gamma$ are weighing and focusing parameters,
respectively, and $p\in$[0,1] is the predicted probability for the fiber
bundle class. As mentioned earlier, the manual charting of fiber bundles does
not include all fiber regions and might not be precise along the boundaries.
Moreover, we have texture variations and noise in the background. Therefore,
we used a self-supervised technique for training $F_{Class}$ that learns
intrinsic texture/intensity variations in addition to the fiber features from
the manual charting alone. We used a contrastive loss function based on SimCLR
[12], where augmented data from each sample constitute the positive example to
the sample while the rest were treated as negatives for the loss calculation.
In SimCLR, random cropping and color distortions were shown to perform well.
In our case we adapted the learning method by choosing augmentations better
suited to our problem: (i) random cropping of patches closer to the input
patch ($<$ 20$\mu$m), constrained within the white matter (by iterative
sampling of patches until mean intensity criterion is satisfied), (ii) noise
injection + Gaussian blurring (with randomly chosen $\sigma\in$ [0.05, 0.3]).
The self-supervised loss with the above augmentations has two advantages: (1)
effective separation between fiber and non-fiber background patches and (2)
identification of fiber patches correctly even in the presence of artifacts,
aided by the shared weights in the encoder of $F_{Seg}$. We used the
contrastive loss [12] (eqn. 2) between positive pairs of patches $(i,j)$ of
$F_{Class}$, given by:

$\vspace{-0.5em}CL(i,j)=\frac{exp(sim(f_{i},f_{j})/\tau)}{\sum_{k=1}^{2N}I_{k\neq
i}~{}exp(sim(f_{i},f_{k})/\tau)},\quad sim(x,y)=\frac{x^{T}y}{||x||\hskip
2.5pt||y||}$ (2)

where $f$ is the output of $F_{Class}$, $sim(.)$ is the cosine similarity
function, $I_{k\neq i}=1$ if $k\neq i$, else 0 is the indicator function and
$\tau$ is the temperature parameter.

Figure 2: Network architecture used for fiber segmentation, and the use of
location priors from rostral sections for false positive reduction.

Temporal ensembling (TE) training:  Only $\sim$6% of sections were manually
charted, which would be insufficient for this challenging detection problem.
Hence, after initially pretraining the model for $Np$ epochs using the
manually charted samples alone, we used the additional unlabeled samples for
training both $F_{Seg}$ and $F_{Class}$, using the temporal ensembling
technique [15], where predictions from the previous $r$ epochs
($[P_{N-r},...,P_{N-1}]$) were averaged and thresholded to obtain the target
label for the current epoch $N$ (we empirically set $r$ = 3). We used focal
loss for pretraining the encoder-decoder of $F_{Seg}$, since contrastive loss
was calculated at patch-level in $F_{Class}$. For the first 3 epochs after
pretraining, predictions from the pretrained model $P_{N_{p}}$ were used for
label generation. Averaging predictions reduced segmentation noise and aided
in adapting the model to data from different brains.

Inference on test brain sections:  We obtained the predictions by applying the
segmentation part $F_{Seg}$ of the model on the whole coronal sections (or
patches of size 1024 $\times$ 1024 in the case of sections with dimensions
larger than 1024 voxels).

###  2.2 Continuity prior for false positive removal

We used the spatial continuity of fibers from the injection site to remove
obvious FPs (e.g., in cortical regions). We downsampled the sections by a
factor of 10 and aligned the sections along the ventricles (or along the
lateral edges of the brain for sections without ventricles) to roughly form 3D
histological volumes. We applied a triplanar U-Net architecture used in [20]
to obtain a 3D priormap containing a crude segmentation of main dense fiber
bundles and later upsampled it to the original dimensions. For each section,
we computed the average of the segmented fiber bundle masks from the two
nearest neighboring priormap sections (in rostral and/or caudal directions, if
available). We removed any detected fiber bundle region in the current
section, whose distance from the averaged bundles of prior sections was
$>$0.2mm.
Postprocessing:  We further reduced noisy regions and FPs by automated
rejection of the predicted regions with area $<$2mm2 and those near the brain
outline.

##  3 Experiments

Dataset used:  We used digitized, coronal histological sections from 12
macaques, with a slice thickness 50$\mu$m and in-plane resolution of
0.4$\mu$m. We considered every 8th section, resulting in a slice gap of
400$\mu$m (refer to [6, 8, 21] for more details on tracer injection,
immunocytochemistry and histological processing). Manual charting of fiber
bundles labeled under ‘dense’ and ‘moderate’ bundles (examples shown in Fig.
1) had been done previously by an expert neuroanatomist under dark-field
illumination with a 4.0 or 6.4x objective, using Neurolucida software (MBF
Bioscience). Manually charted region masks were registered with the tracing
data using similarity (affine) transform with 6 DOF. Manual chartings were
available for 2 macaques (61 sections). Dataset 1 (DS1) consists of 465
sections, including 25 charted sections (out of 61) from one macaque and 440
unlabeled sections from 10 macaques for training. Dataset 2 (DS2) consists of
36 charted sections from the other annotated macaque for testing. Both
datasets were downsampled in-plane by a factor of 4 for training and testing.

Implementation details: For training, we used the Adam optimizer [22]
($\epsilon=10^{-3}$), batch size = 8, pretraining epochs ($N_{p}$) = 100 and
trained with TE for 100 epochs with a patience value of 25 epochs for early
stopping (converged at $\sim$90 epochs). For focal loss, we used $\alpha=$
0.25; $\gamma=$ 2, and for contrastive loss, we used $\tau=$ 0.5. The
hyperparameters were chosen empirically. For $F_{Seg}$, we augmented data
using translation (offset $\in$ [-50, 50] voxels), rotation ($\theta\in$
[-20o, 20o]), horizontal/vertical flipping and scaling ($s\in$ [0.9, 1.2]).
The model was implemented using PyTorch 1.10.0 on Nvidia GeForce RTX 3090,
taking $\sim$10 mins/epoch for $\sim$22,000 samples (training:validation =
90:10).

Experimental setup and evaluation metrics:  We performed 5-fold cross-
validation on 465 sections (440 unlabeled + 25 labeled) from DS1 with a
training-validation-testing split ratio of 80-13-5 sections (for each fold,
only manually charted sections were used for testing). We then trained the
model on the DS1 and tested it on the unseen dataset DS2 (sections from a
macaque different from the training one). We also performed an ablation study
of the method on DS2. We studied the detection performance for dense and
moderate bundles with the addition of individual components of the method: (i)
$F_{Seg}$ with cross-entropy loss (CE loss), (ii) $F_{Seg}$ with focal loss,
(iii) $F_{Seg}$ with addition of $F_{Class}$ with contrastive loss (focal loss
+ ss_con loss), (iv) $F_{Seg}$ and $F_{Class}$ with TE (focal loss + ss_con
loss + TE). We used the same postprocessing for all cases (i-iv), since our
main aim is to study the effect of addition of $F_{Seg}$, ss_con loss and TE,
rather than postprocessing. For evaluation, we used the following metrics: (1)
True positive rate (TPR): number of true positive bundles / total number of
true bundles charted manually, (2) Average number of FPs ($FP_{avg}$): number
of false positive bundles / number of test sections and (3) Fiber density
ratio ($fib\\_dens$): ratio between fiber voxels (obtained from fiber binary
map) and the total bundle area. We obtained the fiber binary map by
considering fibers within the bounding box of the bundle, enhancing the
contrast using contrast-limited adaptive histogram equalization [23] and
thresholding at the 95${}^{\text{th}}$ percentile of intensity values (sample
fiber maps shown in Fig. 4). We calculated the difference in the ratio
($\delta_{fib\\_dens}$) between the manual charting and the detected bundles.

##  4 Results and discussion

Cross-validation (CV) on DS1: On performing 5-fold CV on DS1, we obtained
better performance in the detection of dense bundles than moderate ones due to
the contrast, increased fiber density and texture differences of the former
with respect to the background. Fig. 3(a) shows FROC curves for dense and
moderate fiber bundle detection. We obtained a TPR of 0.92/0.84 for
dense/moderate bundles at 3.7 FPs/section at the elbow point (shown in dotted
lines) for a threshold value of 0.4.

Figure 3: Results of cross-validation on DS1. (a) FROC curves for fiber bundle detection. (b) boxplots of TPR and $\delta_{fib\\_dens}$ at $FP_{avg}$=2 FPs/section, after applying continuity constraints and postprocessing. Table 1: Results of 5-fold cross-validation and ablation study. CE loss - Cross-entropy loss, Focal loss - $F_{Seg}$ with focal loss, ss_con loss - $F_{Class}$ with self-supervised contrastive loss, TE - temporal ensembling. (*) indicates significant improvements in the results compared to the previous row, determined using paired two-tailed T-tests. The best performance in the ablation study is highlighted in bold. $\uparrow$/$\downarrow$ indicate that higher/lower values lead to better results. |  TPR $\uparrow$ |  $|\mathbf{\delta_{fib\\_dens}}|$ (%) $\downarrow$ |  $\mathbf{FP_{avg}}$ $\downarrow$
---|---|---|---
|  Dense |  Moderate |  Dense |  Moderate |
5-fold cross-validation |  0.90$\pm$0.20 |  0.82$\pm$0.17 |  1.5$\pm$0.45 |  3.5$\pm$1.67 |  2.0
Ablation study
CE loss |  0.76$\pm$0.33 |  0.65$\pm$0.29 |  4.1$\pm$0.39 |  5.8$\pm$1.09 |  7.5
Focal loss |  *0.85$\pm$0.31 |  *0.71$\pm$0.34 |  *3.0$\pm$0.48 |  *4.6$\pm$1.01 |  *4.0
Focal loss + ss_con loss |  *0.88$\pm$0.20 |  *0.78$\pm$0.23 |  2.8$\pm$0.21 |  *4.0$\pm$0.86 |  3.5
Focal loss + ss_con loss + TE |  *0.89$\pm$0.21 |  0.79$\pm$0.30 |  *2.0$\pm$0.21 |  *3.7$\pm$0.80 |  *2.5
Figure 4: Two sample results of the ablation study, with the profile of fibers
within detected bundles. (a, b) Sections with ROIs enlarged (white dotted
box); (i – iv) Ablation study results on the ROIs - TP, FP and FN bundles
shown in yellow, red and blue outlines respectively (the proposed method
highlighted in green box (iv)). Further enlarged ROIs (orange dotted box)
containing fibers in the original RGB, grayscale and fiber binary maps.

Fig. 3(b) shows the boxplots of TPR and $\delta_{fib\\_dens}$ values after
postprocessing (performance values reported in Table 1). We observed a
significant reduction of $FP_{avg}$ ($p$ $<$ 0.05) after postprocessing,
mainly due to continuity constraints, for much lower changes in TPR values.
Typically, fiber density ratios $fib\\_dens$ ranged between $\sim$6-20% and
$\sim$2-10% for dense and moderate bundles respectively. We obtained mean
$\delta_{fib\\_dens}$ = -1.5% and -3.5% for dense and moderate bundles,
respectively. While % values closer to 0 are better, the negative values
indicate more fibers in the predicted regions than manual charting in most
cases, indicating that predicted regions recover most of the fibers within the
bundle.

Ablation study results on DS2:  We trained the method on dataset DS1 for
ablation study cases (i-iv), tested on DS2 (from different brain) and used a
threshold of 0.4 to obtain binary maps. We used the same postprocessing for
all cases (i-iv) of the study. Table 1 reports the results of the study and
Fig. 4 shows sample results of the ablation study for a dense bundle in the
prefrontal white matter (a) and a moderate bundle in the IC (b). Among all the
methods, experiments using focal loss (ii - iv) gave significantly better
performance than the CE loss (case i), showing that focal loss was better at
handling the heavy class imbalance. Also, using the self-supervised
contrastive loss (ss_con loss) significantly improved TP regions (especially
moderate fiber bundles) and reduced $FP_{avg}$ due to the better
discrimination between subtle variations in the background intensity and
texture. We also observed a significant reduction in $\delta_{fib\\_dens}$ for
focal loss + ss_con loss (case iii) in moderate fiber bundles (where the
fibers are sparser than dense bundles). This shows that contrastive loss
function not only reduced FPs, but also improved the segmentation of predicted
regions. Using TE (case iv) further improved the detection, especially
increasing the TPR of dense bundles and reducing $FP_{avg}$. We observed that
the value of $r$ (number of prior epochs to predict the target labels) in TE
played a crucial role in the reduction of prediction noise. We set $r$ = 3
because it significantly reduced $FP_{avg}$ over $r$ = 1 ($p<$ 0.01), but
provided $FP_{avg}$ values not significantly different from those with higher
$r$ = 5 ($p=$ 0.52).

The main source of FPs included terminal fields (shown in fig 1), artifacts
such as glare or dust particles, and other structures with similar intensity
profiles. Use of continuity priors and ss_con loss was highly useful in
removing these spurious regions. Currently, inclusion of such priors in the
training framework was not possible due to the lack of sufficient number of
manual chartings for consecutive sections. Hence, a future direction of this
work could explore the possibility of integrating the priors within the
training framework for further reduction of FPs, and improving the method to
reduce the variation (indicated by standard deviation) in our results. Another
area for further study is the quantification of fiber-level characteristics
(e.g., fiber density and orientation).

##  5 Conclusions

In this work, we proposed an end-to-end automated, anatomy-constrained self
supervised learning tool for accurate detection of fiber bundles on macaque
tracer data. With only $\sim$6% of training data manually charted, we achieved
TPR of 0.90/0.80 for dense/moderate fiber bundles on different macaque brain
sections. Our tool could be used for generating voxel visitation maps to
analyse the precise route of axon bundles and their densities along fiber
trajectories for voxel-level validation of dMRI tractography across multiple
seed regions. The code for our method is available at
https://github.com/v-sundaresan/fiberbundle_seg_tracing.

## Acknowledgements

This work was supported by the National Institute of Mental Health
(R01-MH045573, P50-MH106435), the National Institute of Neurological Disorders
and Stroke (R01-NS119911), and the National Institute of Biomedical Imaging
and Bioengineering (R01-EB021265).

## References

  * [1] Yendiki, A., Aggarwal, M., Axer, M., Howard, A.F., van Walsum, A.M.v.C., Haber, S.N.: Post mortem mapping of connectional anatomy for the validation of diffusion mri. NeuroImage p. 119146 (2022)
  * [2] Grisot, G., Haber, S.N., Yendiki, A.: Diffusion mri and anatomic tracing in the same brain reveal common failure modes of tractography. Neuroimage 239, 118300 (2021)
  * [3] Maffei, C., Girard, G., Schilling, K.G., Aydogan, D.B., Adluru, N., Zhylka, A., Wu, Y., Mancini, M., Hamamci, A., Sarica, A., et al.: Insights from the irontract challenge: optimal methods for mapping brain pathways from multi-shell diffusion mri. NeuroImage 257, 119327 (2022)
  * [4] Schilling, K.G., Gao, Y., Stepniewska, I., Janve, V., Landman, B.A., Anderson, A.W.: Anatomical accuracy of standard-practice tractography algorithms in the motor system-a histological validation in the squirrel monkey brain. Magnetic resonance imaging 55, 7–25 (2019)
  * [5] Haber, S.N., Liu, H., Seidlitz, J., Bullmore, E.: Prefrontal connectomics: from anatomy to human imaging. Neuropsychopharmacology 47(1), 20–40 (2022)
  * [6] Lehman, J.F., Greenberg, B.D., McIntyre, C.C., Rasmussen, S.A., Haber, S.N.: Rules ventral prefrontal cortical axons use to reach their targets: implications for diffusion tensor imaging tractography and deep brain stimulation for psychiatric illness. Journal of Neuroscience 31(28), 10392–10402 (2011)
  * [7] Safadi, Z., Grisot, G., Jbabdi, S., Behrens, T.E., Heilbronner, S.R., McLaughlin, N.C., Mandeville, J., Versace, A., Phillips, M.L., Lehman, J.F., et al.: Functional segmentation of the anterior limb of the internal capsule: linking white matter abnormalities to specific connections. Journal of Neuroscience 38(8), 2106–2117 (2018)
  * [8] Haynes, W.I., Haber, S.N.: The organization of prefrontal-subthalamic inputs in primates provides an anatomical substrate for both functional specificity and integration: implications for basal ganglia models and deep brain stimulation. Journal of Neuroscience 33(11), 4804–4814 (2013)
  * [9] Jbabdi, S., Lehman, J.F., Haber, S.N., Behrens, T.E.: Human and monkey ventral prefrontal fibers use the same organizational principles to reach their targets: tracing versus tractography. Journal of Neuroscience 33(7), 3190–3201 (2013)
  * [10] Woodward, A., Gong, R., Abe, H., Nakae, K., Hata, J., Skibbe, H., Yamaguchi, Y., Ishii, S., Okano, H., Yamamori, T., et al.: The nanozoomer artificial intelligence connectomics pipeline for tracer injection studies of the marmoset brain. Brain Structure and Function 225(4), 1225–1243 (2020)
  * [11] Preuss, T.M., Wise, S.P.: Evolution of prefrontal cortex. Neuropsychopharmacology 47(1), 3–19 (2022)
  * [12] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International conference on machine learning. pp. 1597–1607. PMLR (2020)
  * [13] Wu, H., Wang, Z., Song, Y., Yang, L., Qin, J.: Cross-patch dense contrastive learning for semi-supervised segmentation of cellular nuclei in histopathologic images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11666–11675, New Orleans (2022)
  * [14] Lai, Z., Wang, C., Hu, Z., Dugger, B.N., Cheung, S.C., Chuah, C.N.: A semisupervised learning for segmentation of gigapixel histopathology images from brain tissues. In: 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). pp. 1920-1923. IEEE (2021)
  * [15] Perone, C.S., Ballester, P., Barros, R.C., Cohen-Adad, J.: Unsupervised domain adaptation for medical imaging segmentation with self-ensembling. NeuroImage 194, 1–11 (2019)
  * [16] Huang, R., Noble, J.A., Namburete, A.I.: Omni-supervised learning: scaling up to large unlabelled medical datasets. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 572–580, Springer, Granada (2018)
  * [17] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. pp. 234–241, Springer, Munich (2015)
  * [18] Panayides, A. S., Amini, A., Filipovic, N. D., Sharma, A., Tsaftaris, S. A., Young, A., Foran, D., Do, N., Golemati, S., Kurc, T., et al.: AI in medical imaging informatics: current challenges and future directions. IEEE Journal of Biomedical and Health Informatics 24(7), pp. 1837-1857, IEEE (2020)
  * [19] Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision. pp. 2980–2988, Venice (2017)
  * [20] Sundaresan, V., Zamboni, G., Rothwell, P.M., Jenkinson, M., Griffanti, L.: Triplanar ensemble u-net model for white matter hyperintensities segmentation on MR images. Medical image analysis 73, 102184 (2021)
  * [21] Haber, S.N., Kim, K.S., Mailly, P., Calzavara, R.: Reward-related cortical inputs define a large striatal region in primates that interface with associative cortical connections, providing a substrate for incentive-based learning. Journal of Neuroscience 26(32), 8368–8376 (2006)
  * [22] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [23] Zuiderveld, K.: Contrast limited adaptive histogram equalization. Graphics gems pp. 474–485 (1994)