# Forgotten Knowledge: Examining the Citational Amnesia in NLP

First Author
Affiliation / Address line 1
Affiliation / Address line 2
Affiliation / Address line 3
email@domain
&Second Author
Affiliation / Address line 1
Affiliation / Address line 2
Affiliation / Address line 3
email@domain

###### Abstract

Citing papers is the primary method through which modern scientific writing
discusses and builds on past work. Yet there is little work looking at broad
temporal patterns of citation: e.g., the degree to which we cite recently
published papers vs. those published much earlier. This work, systematically
and empirically examines: How far back in time do we tend to go to cite
papers? How has that changed over time, and what factors impact this
citational attention/amnesia? We chose NLP as our domain of interest, and
analyze $\sim$75K NLP papers from the ACL Anthology to show and quantify
several key trends in citation. Notably, $\sim$62% of cited papers are from
the immediate five years prior to publication, while only $\sim$17% are for
papers more than ten years old. Furthermore, we show that the median age and
age diversity of cited papers was steadily increasing from 1990 to 2014, but
since then the trend has reversed, and dramatically so. Our work provides a
basis for broader self-reflection: e.g., how much we are seduced by the latest
technological fads at the expense of ignoring important work from the more
distant past, and what kind of work leaves a lasting long-term impact.

##  1 Introduction

Study the past if you would define the future.
— Confucius
The goal of scientific research is to obtain generalized knowledge and create
a better future for humanity. To do this we innovate on ideas and knowledge
from the past. Thus, a central characteristic of the scientific method and
modern scientific writing is to discuss other work: to build on ideas, to
critique or reject earlier conclusions, to borrow ideas from other fields, and
to situate the proposed work. Even when proposing something that others might
consider dramatically novel, it is widely believed that these new ideas have
been made possible because of a number of older ideas. Citation (referring to
another paper in a prescribed format) is the primary mechanism to point the
reader to these prior pieces of work and also to assign credit for shaping
current work. Thus, we argue that examining citation patterns across time can
lead to crucial insights into what we value, what we have forgotten, and what
we should do for the future. In this work, we systematically and empirically
examine how far back we often cite papers, how such citation patterns change
over time, and what factors affect our citation behaviors by analyzing
temporal citation patterns of scientific papers.

Of particular interest is the extent to which older work is being forgotten —
citational amnesia. Historically, this was impacted by various factors around
access and invention. For example, the invention of the printing press in the
year 1440 allowed a much larger number of people access scientific writing a.
The era of the internet and digitization of scientific literature that began
in the 1990s also greatly increased the ease with which one could access past
work GoogleGiantsPaper. However, other factors such as the birth of paradigm-
changing technologies may also impact citation patterns; ushering in a trend
of citing very new work or citing work from previously ignored fields of work.
Such dramatic changes are largely seen as beneficial, however, strong
tailwinds may also lead to a myopic focus on recent papers and those from only
some areas, at the expense of good relevant work from other time periods and
other areas.

We choose as our domain of interest, papers on Natural Language Processing
(NLP), specifically those in the ACL Anthology. This choice is motivated by
the fact that NLP (and other related fields of Artificial Intelligence) are in
a period of dramatic change. There are notable and frequent gains on benchmark
datasets. NLP technology is becoming increasingly ubiquitous in society; new
sub-fields of NLP such as Computational Social Science, Ethics and NLP, and
Sustainable NLP are all emerging at an accelerated rate. The incredibly short
research-to-production cycle and move-fast-and-break-things attitude in NLP
(and Machine Learning more broadly) have also led to considerable adverse
outcomes to various sections of society, especially those with least power.
Thus, going beyond the most recent fads, and reflecting on the lessons and
ideas from a broad spectrum of past research (from various fields of study) is
arguably especially important now. That said, we believe it is in the interest
of all fields of study to reflect on the questions addressed in this paper.

We compiled a dataset of [blah]K NLP papers that were published between 1990
and 2021, along with their meta-information such as number of citations they
received in each of the years since they were published. We will refer to this
dataset as the Age of Citations (AoC) dataset. We use AoC to answer a series
of specific research questions on what we value, what we have forgotten, what
factors impact this citational attention/amnesia, what citation patterns are
associated with different types of papers, and how these citation patterns
have changed over time.111Note that this work will not attempt to
automatically identify individual pieces of good work that ought to be cited
more. That requires the judgment of an expert in the field and different
stakeholders may find different pieces of work important. The answers to these
questions not only shed light on various temporal citation patterns in NLP,
they also enable further enquiry and future work into bigger questions such as
what kind of work leaves lasting long-term impact. All of the data and code
associated with the project will be made freely available on the project
homepage.

##  2 Related Work

In the broad area of Scientometrics (study of quantitative aspects of
scientific literature), citations and their networks have been studied from
several perspectives, including: paper quality (buela2010analysis), field of
study (costas2009scaling), novelty, length of paper (antoniou2015bibliometric;
falagas2013impact), impact factor (callaham2002journal), venue of publication
(callaham2002journal; WahleRMG22), language of publication
(lira2013influence), and number of authors (della2008multi;
bosquet2013academics), collaboration (nomaler2013more), self-citation
(costas2010self), as well as author’s reputation (collet2014does), affiliation
(sin2011international; lou2015does), geographic location nielsen2021global;
lee2010author; pasterkamp2007citation; paris1998region; nishioka2022does;
OurEMNLPpaper, gender, race and age (ayres2000determinants;
leimu2005determines; chatterjee2021gender; llorens2021gender).

However, there has been relatively little work exploring the temporal patterns
of citation. verstak2014shoulders analyzed [blah] papers to show that the
percentage of older papers being cited steadily increased from 1990 to 2013
(the year up to which they analyzed papers), for seven of the nice fields of
study explored. (They treated papers that were published more than ten years
before a particular citation as old papers.) For Computer Science papers
published in 2013, on average, 28% of the cited papers were published more
than ten years before. This represented an increase of 39% from 1990. They
attributed this increasing trend in citing old papers to the ease of access of
scientific literature on the world wide web, as well as the then relatively
new scientific-literature-aggregating services such as Google Scholar.

PAROLO2015734 analyzed bout 25 million papers from Clinical Medicine,
Molecular Biology, Physics, and Chemistry published until 2014 to show that
typically the number of citations a paper receives per year increases in the
years after publication, reaches a peak, and then decays exponentially.
Interestingly they showed that this rate of decay was increasing in the more
recent papers of their study. They attribute this quicker decay (or more
“forgetting” of recent papers) to the substantial increase in the number of
publications; a lot more papers are being published, and due to the limited
attention span of subsequent researchers, on average, papers are being
forgotten faster.

Past work on NLP papers and their citations includes work on gender bias
schluter2018glass; vogel-jurafsky-2012-said; mohammad-2020-gender,
geographical bias OurEMNLPpaper, and on broad general trends such as average
number of citations over time and by type of paper Mohammad20c; Mohammad20b;
WahleRMG22. However, there is no work on the temporal trends of citations in
NLP papers. Further, it is unclear whether even the broader trends in the
Sciences until about 2014 (discovered by verstak2014shoulders and
PAROLO2015734), still hold true. Our work explores temporal citations in much
greater detail than prior work, asking new questions (discussed in the
Introduction), and focusing on NLP papers up till 2021.

##  3 Dataset

As of January 2022, the ACL Anthology (AA) had 71,568
papers.222https://aclanthology.org/ We used the AA Citation Corpus released by
X. This dataset contains information about the paper title, names of authors,
year of publication, and venue of publication for each of these papers from
the repository. This dataset does not contain information about the cited
papers. Therefore, we augmented this dataset further to include the references
for each paper. This information is gathered using the Semantic Scholar
API.333 https://www.semanticscholar.org/

Not all cited papers are present in the AA Citation Corpus, and thereby we do
not have meta-information about these papers. We further use the unique
Semantic Scholar ID (SSID) of each paper to extract information like paper
title, year of publication, and venue of publication. In this way, we were
able to gather all the information about the cited papers which are required
for our next steps.

##  4 Age of Citation

We use the AoC dataset to answer a series of questions on how research papers
are cited and different trends across years. We start with a look at the
average number of papers cited by a AA paper, followed by analyzing the
distribution and possible factors impacting it.

Figure 1: R1: Average number of papers referenced by an AA paper = 20.6. Plots
showing the trend of the average number of papers referenced by an AA paper
across different years. Figure 2: R1: Plots showing the trend of the average
number of papers referenced by an AA paper across different years for
different venues of publication.

Q1. What is the average number of unique references in the AA papers? How does
this number vary by publication type, such as workshop, conference, and
journal? Has this average stayed roughly the same or has it changed markedly
over the years?

Ans. We calculated the average number of unique references for all papers in
the AoC dataset, as well as for each publication type (workshops, conferences,
and journals). We then binned all papers by publication year, computed the
mean and median for each bin for each year.

| Mean | Median
---|---|---
Journal | 23.24 | 15
Conference | 21.11 | 19
Workshop | 19.07 | 17
Overall | 20.63 | 18

Table 1:  Mean and median of the number of unique references in an AA paper.

### Results

The scores are shown in Table 1. Figure LABEL:fig:r1_new_count_ref_combined
shows how the mean has changed across the years.444The numbers of AA papers
published each year until 1990 were rather low, and so in Figure
LABEL:fig:r1_new_count_ref_combined, we only show the trajectory from 1990.
However, note that the numbers generally increase even from 1965 to 1990. The
graph shows a general upward trend. The trend seems roughly linear until the
mid 2000s, at which point we see that the slope of the trend line increases
markedly. Even just considering the last 7 years, there has been a 41.74%
increase in referenced papers in 2021 compared to 2014.

Similar overall trends can be observed when papers are grouped by publication
type. Not surprisingly, the longer journal articles cite markedly more papers
than conference and workshop papers. The plot for conferences and workshops is
relatively smooth compared to journal articles. This is because the number of
papers for each year in journals is far less. For example, in the year 2015,
only 139 papers were published in journals, whereas 1709 and 983 papers were
published in conferences and workshops respectively.

### Discussion

The steady increase in the number of unique references from 1965 is likely
because of the increasing number of relevant papers as the field develops and
grows. However, it is interesting that this growth has not plateaued even
after 55 years. By the late-2000s, with the advent of widely accessible
electronic proceedings, *ACL venues started experimenting with more generous
page limits: relaxing it from a strict 8 pages to first allowing one or two
additional pages for references to eventually allowing unlimited pages for
references.555In 2008, EMNLP became the first major NLP conference allow an
extra page for references; this was followed by ACL in 2009. Other factors
that may have contributed to more papers being referred to (cited) within a
paper, include: an additional page for incorporating reviewer comments,
allowing Appendices, and the inclusion of an increasing number of experiments
per paper over time.

Figure 3: R2: Overall Distribution of AoC for all papers in AA. For a given
age in the x-axis, the corresponding value in the y-axis represents the
percentage of total references having that age. Figure 4: R2: Distribution of
AoC for all papers in AA based on venue of publication. For a given age in the
x-axis, the corresponding value in the y-axis represents the percentage of
total references from the venue having that age.

Q2. On average, how far back in time do we go to cite papers? As in, what is
the average age of cited papers? What is the distribution of this age across
all citations? How do these vary by publication type?

Ans. If a paper $x$ cites a paper $y_{i}$, then the age of the citation (AoC)
is taken to be the difference between the year of publication (YoP) of $x$ and
$y_{i}$:

$\textrm{\it AoC}(x,y_{i})=\textrm{\it YoP}(x)-\textrm{\it YoP}(y_{i})$ (1)

We calculated the AoC for each of the citations in the AoC dataset. For each
paper, we also calculated the mean $AoC$ of all papers cited by it:

$mAoC(x)=\frac{1}{N}\sum_{i=1}^{N}AoC(x,y_{i})$ (2)

here $N$ refers to the number of papers cited by $x$.

### Results

The average mAoC for all the papers in the AoC dataset is 7.02. The scores
were 8.16 for journal articles, 6.93 for conference papers, and 7.01 for
workshop papers. Figure LABEL:fig:r2_new_aoc_distribution_combined shows the
distribution of $AoC$s in the dataset across the years after the publication
of the cited paper (overall, and across publication types). For example, the
y-axis point for year 0 corresponds to the average of the percentage of
citations papers received in the same year as it they were published. The
y-axis point for year 1 corresponds to the average of percentage of citations
the papers received in the year after they were published. And so on.

Observe that the majority of the citations are for papers published one year
prior, (AoC = 1). This is true for conference and workshop subsets as well,
but in journal papers, the most frequent citations are for papers published
two years prior. Overall though all the arcs have a similar shape, rising
sharply from the number in year 0 to the peak value and then dropping off at
an exponential rate in the years after the peak is reached. For the full set
of citations, this exponential decay from the peak has a half life of about 4
years. Roughly speaking, the line plot for journals is shifted to the right by
a year compared to the line plots for conferences and workshops. It also has a
lower peak value and its citations for the years after the peak are at a
higher percentage than those for conferences and workshops. Additionally,
citations in workshop papers have the highest percentage of current year
citations (age 0), whereas citations in journal article have the lowest
percentage of current year citations.

Analogous to Figure LABEL:fig:r2_new_aoc_distribution_combined, Figure
LABEL:fig:r2_new_aoc_distribution_bins presents the distribution of AoCs,
albeit broken down by the total citations received by a paper. It is worth
noting that the distribution leans more towards the right for papers with a
higher number of citations. This shows that papers with a higher citation
count continue to receive significant citations even far ahead in the future,
which is intuitive.

### Discussion

Overall, we observe that papers are cited most in years immediately after
publication, and their chances of citation fall exponentially after that. The
slight right-shift for the journal article citations is likely, at least in
part, because journal submissions have a long turn-around time from the first
submission to the date of publication (usually between 6 and 18 months). A
list of the oldest papers cited by AA papers is available on the project’s
GitHub repository.

Figure 5: R2: Trend showing AoC over the years. Blue line indicates the mean.
[change the y-axis label to mean age of citation –Janvijay] [is it possible to
do a truncated figure? for instance, skip the range og 15 to 25 in the y-axis.
This will make these boxes easier to be read - currently these numbers are too
hard to be separated with each other –diyi] Figure 6: R2: Gini coefficient
trend for the overall distribution of AoC across years. [change the caption it
is not $\boldsymbol{AoC}$ but $\boldsymbol{mAoC}$ –Janvijay] Figure 7: R3:
Trend of distribution of AoC for different venues across years.  [to be
shifted to appendix –Janvijay] Figure 8: R3: Plot of Gini coefficient for
different venues across time.

Q3. What is the trend in the variation of $\boldsymbol{AoC}$ over time and how
does this variation differ across different publication venues in NLP?

Ans. To answer this question, we split the papers into bins corresponding to
the year of publication, and then examined the distribution of $mAoC$ in each
bin. We define a new metric called the Citation Age Diversity (CAD) Index,
which measures the diversity in the $mAoC$ for a set of papers. In simpler
terms, a higher CAD Index indicates that $mAoCs$ covers a broader range,
implying that the cited papers span a wider time period of publication. This
metric offers valuable insights into the temporal spread of scholarly
influence and the long-term impact of research. Precisely, the CAD Index for a
bin of papers $b$, is defined using the Gini Coefficient as follows:

$CAD(b)=1-\sum_{i=1}^{N}\sum_{j=1}^{N}\frac{\left|mAoC(b_{i})-mAoC(b_{j})\right|}{2N^{2}\bar{b}}$
(3)

here, $b_{i}$ corresponds to $i^{th}$ paper within bin $b$, $N$ denotes the
total number of papers in bin $b$ and $\bar{b}$ represents the mean of $mAoC$
of papers’ associated with bin $b$. A CAD Index close to 0 indicates minimum
temporal diversity in citations (citing papers from just one year), whereas a
CAD Index of 1 indicates maximum temporal diversity in citations (citing
papers uniformly from past years). In addition to CAD Index, we also compute
median $mAoC$ of each such yearly bin. The results for both CAD Index and
median $mAoC$ have roughly identical trends across the years. We discuss the
CAD Index analysis below. (The discussion of the median $mAoC$ results is in
the Appendix LABEL:sec:supplementary_1.)

### Results

Figure LABEL:fig:r3_new_gini_combined shows the CAD Index across years (higher
CAD Index indicates high diversity), and across different publication types.
The CAD Index plot of Figure LABEL:fig:r3_new_gini_combined shows that the
temporal diversity of citations had an increasing trend from 1990 to 2014, but
the period from 1998 to 2004, and 2014 to 2021 (dramatically so) were periods
of decline in temporal diversity (decreasing CAD Index scores). These
intervals coincide with the year intervals in which we observed a decreasing
trend in median $mAoC$ of published papers (discussed in the Appendix). This
suggests that the increase or decrease in diversity is largely because of the
decreased or increased focus on papers from recent years, respectively.

The CAD Index plots by publication type all have similar trends, with journal
paper submissions consistently having markedly higher scores (indicating
markedly higher temporal diversity) across the years studied. However, they
also seem to be most impacted by the trend since 2014 to cite very recent
papers. (CAD Index not only goes back to the 1990 level, but also undershoots
beyond it.)

### Discussion

Overall, we find that all the gains in temporal diversity of citations from
1990 to 2014 (a period of 35 years), have been negated in the 7 years from
2014. This change is driven largely by the deep neural revolution in the early
2010’s and strengthened further by the substantial impact of transformers on
NLP and Machine Learning.

Q4. What percentage of cited papers are old papers? How has this varied across
years and publication venues?

Ans. Just as verstak2014shoulders, we define a cited paper as older if it was
published at least ten years prior to the citing paper. We then divided all AA
papers into groups based on the year in which they were published. For each AA
paper, we determined the number of citations to older papers.

### Results

Figure LABEL:fig:r4_new_old_refs shows the percentage of older papers cited by
papers published in different years. Observe that this percentage increased
steadily from 1990 to 1999, before decreasing until 2002. After 2002, the
trend of citing older papers picked up again; reaching an all time high of
$\sim$30% by 2014. However, since 2014, the percentage of citations to older
papers has dropped dramatically, falling by 12.5% and reaching a historical
low of $\sim$17.5% in 2021. Similar patterns are observed for different
publication types. However, we note that a greater (usually around 5% more)
percentage of a journal paper’s citations are to older papers, than in
conference and workshop papers.

### Discussion

These results confirm that the trends in diversity discussed in Q2 are aligned
with the trends in citing older papers. This dramatic drop in citing older
papers since 2014 can largely be attributed to the explosion of paper count
and the paradigm shift in the field of NLP brought on by deep learning and
transformers.

Figure 9: R7: Plots showing the distribution of $mAoC$ for different bi-grams
appearing in the title of citing paper.  [update the y-axis label, improve the
font-size –Janvijay]

Q5. What is the $\boldsymbol{mAoC}$ distribution for different areas within
NLP? Relative to each other, which areas tend to cite more older papers and
which areas have a strong bias towards recent papers?

Ans. The ACL Anthology does not include metadata for sub-areas within NLP.
Further, a paper may be associated with more than one area and the distinction
between areas can often be fuzzy. Thus, we follow a rather simple approach
used earlier in mohammad-2020-gender: using paper title word bigrams as
indicators of topics relevant to the paper. A paper with machine translation
is very likely to be relevant to the area of machine translation. Using title
bigrams for this analysis also allows for a finer analysis within areas. For
example, two bigrams pertaining to finer subareas within the same area can be
examined separately. (Papers in different sub-areas of an area need not be
similar in terms of the age of the papers they cite.)

We first compiled a list of the top 60 most frequent bigrams from the titles
of AA papers. Next, for each of these bigrams, we created a bin containing all
AA papers that had that bigram in their title.666A single paper may be
included in multiple bins. For each paper included in any of these bins, we
computed $mAoC$. Finally, we plotted the distribution of $mAoC$ values for the
papers in each bin, as shown in Figure 9. Note that, for the purpose of
improving the visibility of the plot, only selected $mAoC$ distributions are
depicted in the figure 9. We then examined the distribution of $mAoC$ for each
of these bins.

### Results

Figure 9 shows the $mAoC$ violin plots for each of the bins pertaining to the
title bigrams (in decreasing order of median $mAoC$). Observe that papers with
the title bigrams word alignment, parallel corpus/corpora, Penn Treebank,
sense disambiguation and word sense (common in the word sense disambiguation
area), speech tagging, coreference resolution, named entity and entity
recognition (common in the named entity recognition area), and dependency
parsing have some of the highest median $mAoC$ (cite more older papers). In
contrast, papers with the title bigrams glove vector, BERT pre, deep
bidirectional, and bidirectional transformers (which correspond to new
technologies) and papers with title bigrams reading comprehension, shared
task, question answering, language inference, language models, and social
media (which correspond to NLP subareas or domains) have some of the lowest
median $mAoC$ (cite more recent papers).

### Discussion

The above results suggest that not all NLP subfields are equal in terms of the
age of cited papers. In fact, some papers cited markedly more newer papers
than others. This could be due to factors such as early adoption or greater
applicability of the latest developments, the relative newness of the area
itself (possibly enabled by new inventions such as social media), etc.

Figure 10: Table illustrates the distribution of the most frequent bigrams
across different year intervals in all cited papers. As shown, 11.41% of
citations in the 2016-2022 period had the word "neural" in the cited paper’s
title.  [update figure –Janvijay] Figure 11: Table illustrates the
distribution of the most frequent unigrams across different year intervals in
all cited papers. Follows the same convention as Figure 19.

Q6. What topics are more pronounced in cited papers across different periods
of time?

Ans. To address this question, we partitioned the research papers into those
published between: 1990–1999, 2000–2009, 2010–2015, and 2016–2021.7772010–2021
period was split into two because of the large number of papers published in
this period and as it allows for a finer examination. For papers from each
period: we first extracted all unigrams and bigrams from the titles of the
cited papers. Next, for the top 100 most frequent unigrams and bigrams, we
calculated the percentage of all citations that had the respective ngram in
the cited paper’s title — the ngram citation percentage.

### Results

Upon examining various bigram citation percentages, we found that bigrams
pertaining to areas such as tree-adjoining grammars have been in decline since
the 1990s (cited less as with every subsequent interval). Bigrams pertaining
to areas such as conditional random fields and coreference resolution gained
momentum in the middle periods (2000–2016) but have since lost popularity
post-2016. On the other hand, techniques such as domain adaptation have
consistently gained momentum since the 2010s. Post-2016 keywords related to
deep learning technologies such as convolutional neural nets, deep bi-
directional, deep learning, deep neural, Global vectors, and jointly learning
experienced a substantial surge in popularity. Additionally, certain areas
such as cross-lingual and entity recognition consistently gained momentum
since the 1990s.

Upon examining various unigram citation percentages, we found that deep-
learning-related terms such as attention, bert, deep, neural, embeddings, and
recurrent saw a substantial increase in citation post-2016. Furthermore, we
observed that since the 1990s, there has been a growing trend in NLP papers
towards citing research on the social aspects of language processing, as
evidenced by the increasing popularity of keywords such as social and
sentiment.

Figures 11 and 10 in the Appendix show a number of unigrams and bigrams with
the most notable changes in the ngram citation percentage across the chosen
time intervals.

Q7. Do well-cited papers cite more old papers and have more $\boldsymbol{AoC}$
diversity?

### Ans.

We introduce three hypotheses to explore the correlation between temporal
citation patterns of target papers and the number of citations the target
papers themselves get in the future.

  1. [label=H0.]

  2. 1.

The degree of citation has no correlation with temporal citation patterns of
papers.

  3. 2.

Highly cited papers have more temporal citation diversity than less cited
papers.

  4. 3.

Highly cited papers have less temporal citation diversity than less cited
papers.

Without an empirical experiment, it is difficult to know which hypothesis is
true. H1 seemed likely, however, there were reasons to suspect H2 and H3 also.
Perhaps cite more widely is correlated with other factors such the quality of
work and thus correlates with higher citations (supporting H2). Or, perhaps,
early work in a new area receives lots of subsequent citations and work in a
new area often tends to have limited citation diversity as there is no long
history of publications in the area (supporting H3).

On, Nov 30, 2022, we used the Semantic Scholar API to extract the number of
citations for each of the papers in the AoC dataset. We divided the AoC papers
into nine different bins as per the number of citations: 0, 1–9, 10–49, 50–99,
100–499, 500–999, 1000–1999, 2000–4999, or 5000+ citations. For each bin, we
calculated the mean of $mAoC$ and CAD Index. We also computed the Spearman’s
Rank Correlation between the CAD Index of the citation bins and the mean of
the citation range of each of these bins.

### Results

Figure LABEL:fig:r7_diversity shows the $mAoC$ and CAD Index for each bin (a)
for the full AoC dataset, and (b) for the subset of papers published between
1990 and 2000. (Figures LABEL:fig:r7_diversity_2000 and
LABEL:fig:r7_diversity_2010 in the Appendix show plots for papers from two
additional time periods.) On the full dataset (Figure
LABEL:fig:r7_diversity_overall), we observe a clear pattern that the CAD Index
decreases with increasing citation bin (with the exception of papers in the
1K–2K and 2K–5K bins). The mean $mAoC$ follows similar trend w.r.t. the CAD
Index.

These results show that, for the full dataset, the higher citation count
papers tend to have less temporal citation diversity than lower-citation count
papers. However, on the 1990s subset (Figure LABEL:fig:r7_diversity_1990), the
CAD Index decreased till the citation count < 50 and increased markedly after
that. This shows that during the 1990s, the highly cited papers also cited
papers more widely in time. Plots for the 2000s and 2010s (Figure
LABEL:fig:r7_diversity_appendix) follow a similar trend as the overall plot
(Figure LABEL:fig:r7_diversity_overall), indicating that trend of highly cited
papers having less temporally diverse citations started around the year 2000.

1990–99 | 2000-09 | 2010–15 | 1965–2021 (All)
---|---|---|---
0.16 | -1.00∗ | -0.97∗ | 0.72∗

Table 2:  Correlation between the mean of citation bins and CAD Index for the
bins for various time periods. The ∗ indicates that the correlation is
statistically significant (p-value $<0.05$).

The Spearman’s rank Correlation Coefficients between the mean number of
citations for a bin and the mean $mAoC$ of the citation bins are shown in
Table 2.888We did not compute correlations for the 2016–2021 period because
those papers have had only a few years to accumulate citations and reach the
larger citation bins. Observe that for the 1990’s papers there is essentially
no correlation, but there are strong correlations for the 2000s, 2010s, and
the full dataset papers.

Similar to Figure LABEL:fig:r7_diversity_overall, in Figure
LABEL:fig:r7_diversity_area_appendix (in the Appendix) we show how mean mAoC
and CAD Index of AA papers published between 1965 and 2021 but when broken
down by research topics. This examination across various research topics
consistently shows a trend: the higher the citations, the lower the age
diversity of citations. This may be because “mainstream” work in an area tends
to cite lots of other very recent work and brings in proportionately fewer
ideas from the past. In contrast, “non-mainstream” work tends to incorporate
proportionally more ideas from outside, yet receives fewer citations as there
may be less future work in that space to cite it.

### Discussion

Papers may receive high citations for a number of reasons; and those that
receive high citations are not necessarily model research papers. While they
may have some aspects that are appreciated by the community (leading to high
citations), they also have flaws. High-citation papers (by definition) are
more visible to the broader research community and are likely to influence
early researchers more. Thus their strong recency focus in citations is a
cause of concern. Multiple anecdotal incidents in the community have suggested
how early researchers often consider papers that were published more than two
or three years back as "old papers". This goes hand-in-hand with a feeling
that they should not cite old papers and therefore, do not need to read them.
The lack of temporal citation diversity in recent highly cited papers may be
perpetuating such harmful beliefs.

—————-old stuff after this—————

Q1. What is the average age of the papers that we cite? What is the
distribution of age of citations?

A. In order to gain insights into the distribution of citations in the AA
citation dataset, we examined the age of citations associated with all papers
present in the dataset. The age of the cited paper $y_{i}$, within a given
paper $x$ is determined as the difference between the publication year of $x$
and the publication year of $y_{i}$. Specifically, this age $AoC(x,y_{i})$ can
be expressed in terms of year of publication, $YoP$, as follows:

$AoC(x,y_{i})=YoP(x)-YoP(y_{i})$

### Discussion

In Figure 12, we plotted the distribution of age of all citations in the
dataset. The figure provides a visual representation of the distribution of
citations (as percentages of total citations) across the age of cited papers.
Upon examination of the figure, several key observations can be made. Firstly,
it can be seen that the majority of papers have an age of 1 year. This
suggests that the majority of citations in the dataset are made to papers that
have been recently published. Secondly, it can be observed that cited papers
with an age of 0 are greater in number than cited papers with an age greater
than 5. This implies that papers that have just been published tend to receive
more citations in comparison to older papers. Additionally, the age of cited
papers roughly follows an exponential decay, with a half-life of nearly 4
years.

This analysis has revealed intriguing patterns in the age of cited papers, at
aggregate. The findings from this analysis motivated us to further investigate
the dynamics of citation patterns by delving deeper into the evolution of age
of citations over time, which we examine in our next research question. A
deeper understanding of the evolution of $AoC$ can provide valuable insights
into how the popularity of papers changes over time and provide a deeper
understanding of the dynamics of citation patterns in the field of NLP.

Figure 12: R1: Overall Distribution of AoC for all papers in AA. For a given
age in the x-axis, the corresponding value in the y-axis represents the
percentage of total references having that age. [for all figures, make sure
the font size is big enough. currently they are too small to view. –diyi]

Q2. How has $\boldsymbol{AoC}$ changed over the years?

A. To gain a deeper understanding of the trend of age of citations, $AoC$s, we
examined the recency and diversity of $AoC$ associated with papers published
in different years. To do this, we first defined a metric, $mAoC(x)$, as the
mean of $AoC$ of all papers cited by a given paper $x$. Specifically, we
define $mAoC$ as:

$mAoC(x)=\frac{1}{N}\sum_{i=1}^{N}AoC(x,y_{i})$

here $y_{1},y_{2},y_{3}\ldots,y_{N}$ are $N$ papers cited by paper $x$. Next,
we computed $mAoC$ for all papers in the AA citation dataset and then plotted
the distribution of $mAoC$ bucketed across different years, as shown by violin
plots in Figure 13. For example, if a paper $x$ was published in year $t$,
then $mAoC(x)$ will be a data point for plotting the distribution of year $t$.
The median $mAoC$ for a given year reflects the recency of citations, with a
lower median $mAoC$ indicating that papers published in that year have cited
relatively recent papers. Further, for each year, we also computed the gini-
coefficient of all $mAoC$ belonging to that year, as shown in Figure 14. This
figure captures how the diversity of $mAoC$ has varied across years, with a
higher gini-coefficient indicating low diversity.

Figure 13: R2: Trend showing AoC over the years. Blue line indicates the mean.
[change the y-axis label to mean age of citation –Janvijay] Figure 14: R2:
Gini coefficient trend for the overall distribution of AoC across years.
[change the caption it is not $\boldsymbol{AoC}$ but $\boldsymbol{mAoC}$
–Janvijay]

### Discussion

We investigate the recency of citations across different years using figure
13. Our analysis reveals several noteworthy observations. From 2014 onward,
the median $mAoC$ has consistently decreased each year. Specifically, the
median $mAoC$ in 2021 is nearly 2.5 years less than that of 2014. A similar
trend of decreasing median $mAoC$ can also be observed between the years 1998
and 2004. This reveals a consistent trend of decreasing $mAoC$ in both the
intervals of 1998-2004 and 2014-2021, suggesting that papers published in
these years tend to cite relatively recent papers compared to previous years.

In addition to examining the median, we also studied the mean of $mAoC$, as
represented by the blue line in the figure 13. The mean follows a similar
trend as the median, with slight variations. In particular, it is consistently
higher than the median, indicating that the data is skewed to the right, with
a few papers having very large $mAoC$ that significantly affect the mean.

[is it possible to do a truncated figure 2? for instance, skip the range og 15
to 25 in the y-axis. This will make these boxes easier to be read - currently
these numbers are too hard to be separated with each other –diyi] In addition
to studying the recency of citations, we also investigate the diversity of
citations using Figure 14. Our analysis of the gini coefficient reveals an
upward trend from 2014 to 2021 as well as from 1999 to 2002. These intervals
nearly overlap with the year intervals in which we observed a decreasing trend
in median $mAoC$ of published papers. This suggests that the diversification
in $mAoC$ has been on a decreasing trend in the year intervals 1999-2002 and
2014-2021. In other words, papers published in these years tend to cite papers
from a narrower range of publication years compared to papers published in
other years. Overall, these observed variations in $mAoC$ raises the question
of identifying the factors responsible for it. In our next research question,
we aim to investigate the potential factors that may contribute to trends in
$mAoC$.

Q3. What are the factors that may impact AoC? (Ans. includes change in
research paradigms, number of papers published at various times, topic of
citing paper, topic of cited paper, number of papers being cited,…)

A.

Q4. How many citations by AA papers total? What is the average number of
papers cited by an AA paper? How has that changed with time (AoC(t))?

A. To commence the understading of the how has citation pattern changed with
time and how far we go back in time to unearth the technoligcal findings in
NLP research, we start with quantifying the number of references mentioned in
the AA papers. In order to measure this, we aggregate the number of papers
referenced by an AA paper and group them based on years. We then compute the
average for each year and plot a line graph indicating it. This number for a
year denotes the average number of papers referenced by an AA paper published
in that year.

### Discussion

The AA citation dataset includes papers published between 1950 and 2022.
Figure LABEL:fig:r4_count_ref illustrates the average number of papers
referenced by an AA paper versus the year of publication. The graph generally
follows an upward trend, but there are abrupt up-and-down fluctuations in the
graph until 1990. This is due to the limited number of papers published during
this period, causing the average number of references to be highly
inconsistent. After 1990, there is a fairly considerable number of papers
published in the AA, and the graph post-1990 follows a more consistent upward
trend. Notably, the number of references per paper has continuously increased
over time. This increase can be attributed to various factors such as page
limits, the availability of more relevant work in the field, and others.
Before X, the ACL imposed a limit of 8 pages for references, limiting the
number of references that could be included in a paper. However, after X, this
limit was lifted, resulting in a significant increase in the number of
references in published papers. Additionally, as research in the field
continued to grow, the number of relevant works also increased, further
contributing to the rise in referenced papers. Compared to 2010, the average
number of referenced papers has doubled by 2021. Recent years, especially the
last 5-6 years, has seen a significant increase in referenced papers, with the
advent of Neural Networks. The current year has seen a X% increase in the
number of referenced papers compared to 2014. The graph still follows an
upward trend with no signs of plateauing in recent years.

Q5. What is the distribution of AoC for journal papers, conference papers, and
workshop papers, respectively? Are they markedly different? What are the
factors that impact AoC of journal articles, conference papers, and workshop
papers, respectively?

A. The venue of publication is believed to be somewhat correlated with the
quality and impact of a publication. There are also common trends in papers
published in different venues, which can be attributed to various factors such
as the scope of research, contribution to the field, page limits, and others.
In the ACL Anthology, there are many different venues, and for the sake of
simplicity and ease of visualization, we aggregated them into three broader
venues: Journals, Conferences, and Workshops. In this analysis, we study the
distribution of the Age of Citation (AoC) for each of these venues, the trend
of diversification, and the change in the number of research papers cited by a
paper published in these venues.

Firstly, we follow a similar analysis done in Research Question 1. We compute
the $AoC(x,y_{i})$ as specified there for each of the three venues. We
aggregate the information for each venue which is shown in Figure
LABEL:fig:r4_count_ref.

Secondly, to understand the impact of the number of papers cited by a paper,
for each year we compute the average number of papers cited by a paper
published in that year across different venues. Figure
LABEL:fig:r5_count_ref_venue shows the line plot of the average number of
papers cited by a paper versus the year of publication.

Figure 15: R5: Distribution of AoC for all papers in AA based on venue of
publication. For a given age in the x-axis, the corresponding value in the
y-axis represents the percentage of total references from the venue having
that age.

### Discussion

Figure 15 shows three line plots for the distribution of Age of Citation (AoC)
for the papers published in each of the three broader venues: Journals,
Conferences, and Workshops. From the figure, we can make the following
observations: Firstly, each of the line plots shows a similar pattern where
the majority of papers have an age of 1-2 years. This indicates that papers
published recently are cited much greater than others. Secondly, the line plot
for journals seems to be right-shifted by a year when compared to the line
plots for conferences and workshops. One possible reason for this could be the
longer review period of journal papers. Because of this, the peak for the
journal line plot is achieved for the age of 2 years, whereas the peak for the
other venues is achieved for the age of one year. Lastly, workshops have a
greater number of papers with an age of 0 (same as the year of publication).
This indicates that workshops and conferences tend to cite more recent papers
compared to journals.

Figure LABEL:fig:r5_count_ref_venue shows the line plot indicating the average
number of papers referenced by an AA paper versus the year of publication for
each of the venues. In general, the graph follow an upward trend for all of
them. The plot for conferences and workshops follows a smooth trend, whereas
journals have quite frequent ups and downs. This is because the number of
papers for each year in journals is far less . For example, in the year 2015,
only 139 papers were published in journals, whereas 1709 and 983 papers were
published in conferences and workshops respectively.

Q6. How have the AoC distributions of journals, conferences, and workshops
changed with time? What has impacted these at specific times to cause a marked
change?

A. Similar to Question 2, we will delve into the distribution of Age of
Citation (AoC) for different venues across time. This distribution will
provide details into different parameters like mean, median, interquartile
distribution, and many more. This also helps to distinguish patterns across
different venues and also compare them across different years. We plot two
graphs to analyze this question.

Firstly, we plot a violin plot of the distribution of the average age of the
citation per paper aggregated across different years shown in Figure 16. We do
this in two steps. In the first step, we compute the mean age of all the
references mentioned in the paper. In the second step, we group all the papers
based on their year of publication and thus, finally, we get a list of the
average age of citation for each year. We do this for all three venues and
plot them in the same graph for effective comparison and analysis.

Secondly, to compute the Gini Coefficient, we require a finite distribution of
buckets. Therefore, we consider papers having a maximum citation age of 20
years. For each paper, we create a list of ages. We extend this list to
include similar details for all the other papers published in that year. We
finally compute the Gini coefficient to understand the diversification among
the cited papers (in terms of age) present in these venues. Figure
LABEL:fig:r5_count_ref_venue shows the plot of the Gini Coefficient for each
year starting from 1990 till 2021 and for each venue.

Figure 16: R5: Trend of distribution of AoC for different venues across years.
Figure 17: R6: Plot of Gini coefficient for different venues across time.

### Discussion

Figure 16 shows the violin plot of the distribution of the average age of the
citation per paper aggregated across different years. From the graph, we can
make the following four key observations: Firstly, journals have a higher
median than conferences and workshops. This is observed for all the years
under consideration. Additionally, we see that journals have an offset from
zero, indicating that journals cite older papers in general. Secondly, the
distributions for conferences and workshops are roughly similar. Before the
2000s, conferences have a higher median than workshops, but after that, the
medians for both of them follow a roughly similar pattern. This showcases
greater diversification and a tendency to cite older papers in workshops as
well. Thirdly, overall, we can see that the median (indicated by the white dot
in Figure 16) kept increasing until 2014 and decreased considerably since
then. This decrease in median indicates a tendency to cite more recent papers,
which can be attributed to the fast-paced outlook of this neural era. Lastly,
the range of distribution of the average Age of Citation (AoC) for journals is
comparatively less when compared to conferences and workshops.

A higher Gini Coefficient indicates an unfair distribution of papers, where
only certain years are cited. On the other hand, a lower coefficient indicates
that citing papers has a roughly equal distribution of papers from different
years. For our scenario, we consider papers having a maximum age of 20 years.
In Figure X, we plot the Gini-coefficient for these age distributions, to show
how the spread of citations has changed over the years. A decreasing value of
the Gini-coefficient indicates that the spread is becoming closer to equal (a
proxy for research fairness and equitable distribution). From the plot 17, we
can see two segments for all the venues. In the first segment from 1990 to
2014, the Gini coefficient of diversification drops almost linearly for all
venues. Specifically, for conferences, the average of Gini coefficients drops
from 0.5108 to 0.3693 from 1990 to 2014. This is encouraging as a decrease in
the Gini coefficient indicates diversification in research. This result shows
that there was a trend to cite papers from all the years roughly comparably.
However, in the second segment from 2014 to 2021, the Gini coefficient of
diversification increases almost linearly for all venues. For conferences, the
average of Gini coefficients increases from 0.3693 to 0.5766 from 2014 to
2021. This steep increase in the trend showcases a myopic view of the age of
citation distribution where most of the references belong to a certain time
period. In recent times, there has been a huge polarisation towards neural
networks which has resulted in papers citing only recent papers and
specifically those involving neural methods. Also, a notable point is that the
first segment spanned across 25 years but the second segment is able to
achieve a similar Gini coefficient in just 8 years.

Figure 18: R7: Plots showing the distribution of $mAoC$ for different bi-grams
appearing in the title of citing paper.  [update the y-axis label, improve the
font-size –Janvijay]

Q7. What is the AoC distribution for different areas of NLP?

A. In order to study the distribution of $AoC$ across different areas of NLP,
we computed the $mAoC$ for all papers in the AA citation dataset. We then
plotted the distribution of $mAoC$ bucketed across different areas using box
plots, as shown in Figure 18. To ensure a representative sample of papers from
various areas of NLP, we utilized word bigrams from the title of the papers.
Our approach is similar to the one used by Mohammad (2020) where we considered
the top-60 bigrams based on the number of papers sampled for each bigram. Each
bigram then loosely represents an area of NLP, allowing us to sample papers
from a diverse range of sub-fields within the field. This approach enables us
to obtain a comprehensive understanding of the citation patterns across
different areas of NLP research.

### Discussion

From Figure 18, we make several noteworthy observations. Firstly, papers that
include deep-learning terminologies, such as BERT, GLoVe, pre-training, neural
network, and neural machine, in their title tend to have a lower median $mAoC$
compared to papers that do not include them. Moreover, box plots corresponding
to such “areas” also exhibit a reduced inter-quartile range in comparison to
other “areas”. This suggests that papers that utilize these deep-learning
techniques tend to cite more recent papers on average. Additionally, it is
interesting that certain subfields within NLP, such as word-sense
disambiguation, speech tagging, coreference resolution, and named entity
recognition, demonstrate a higher median $mAoC$ compared to other areas, such
as social media analysis, reading comprehension, text classification, and
language modeling. This suggests that not all subfields have experienced an
equal increase in median $mAoC$, and thus the phenomenon recency of citations
is not uniform across all subfields. The observation from this question
prompted an investigation into identifying the prevalent bigrams and unigrams
in the cited papers. In the subsequent research question, we delve into this
area by exploring the prevalence of bigrams and unigrams in cited papers and
examining if any temporal variations exist in these patterns across different
year intervals.

Figure 19: Table illustrates the distribution of the most frequent bigrams
across different year intervals in all cited papers. As shown, 11.41% of
citations in the 2016-2022 period had the word "neural" in the cited paper’s
title.  [update figure –Janvijay] Figure 20: Table illustrates the
distribution of the most frequent unigrams across different year intervals in
all cited papers. Follows the same convention as Figure 19.

Q8. For the papers published in the 1990s, what are the most common content
unigrams and bigrams in the titles of papers? Same for the 2000s, 2010–2015,
2016–2021? Which unigrams and bigrams are were popular in the 1990s list but
not in the others? Same for 2000s, 2010–2015, 2016–2021?

A. To address this question, we divided the research papers in the AA Citation
dataset into four year intervals: 1990-1999, 2000-2009, 2010-2015, and
2016-2021. Subsequently, we extracted all unigrams and bigrams from the papers
cited by the papers in each interval. For the purpose of analysis, we
considered the top-x most frequently cited unigrams and bigrams across all
intervals. Figure 20 and 19 present the distribution of these top-x unigrams
and bigrams, respectively, across the different intervals. Specifically, the
number associated with a year interval and unigram/bigram represents the
percentage of all cited papers that had the respective unigram/bigram in the
title and were cited in that particular year interval. We also highlight
unigrams/bigrams that have undergone drastic changes. A single * indicates
that the relative gain from the minimum to the maximum across all year
intervals is more than 1500% for unigrams and 3000% for bigrams. A double *
denotes that certain bigrams/unigrams were not cited at all in some year
interval.

### Discussion

Our analysis of the bigrams and unigrams highlighted with * in Figure 20 and
19 revealed several interesting observations. From the figure 19, we observed
that older methods such as tree-adjoining grammars have been in decline since
the 1990s. Additionally, bigrams such as conditional random fields,
coreference resolution and error rate gained momentum between 2000-2016 but
have since lost popularity post-2016. On the other hand, techniques such as
domain adaptation have consistently gained momentum since the 2010s. It is
also noteworthy that with the advent of deep learning in NLP, post-2016,
keywords related to deep learning technologies such as convolutional neural
nets, Deep bi-directional, Deep learning, Deep neural, Global vectors, Glove
global, jointly learning, have experienced a drastic surge in popularity since
2016. Additionally, certain areas such as cross-lingual and entity recognition
have consistently gained momentum since the 1990s.

We make similar observations from the figure 20. Firstly, keywords related to
deep learning, such as Attention, Bert, Deep, Neural, embeddings, and
recurrent, have seen a significant increase in popularity post-2016.
Furthermore, we observe that since the 1990s, there has been a growing trend
in the NLP community towards research on the social aspects of language
processing, as evidenced by the increasing popularity of keywords such as
social and sentiment.

##  5 Conclusion

## Limitations

##  Appendix A Example Appendix

This is a section in the appendix.