# Paragraph-level Citation Recommendation
based on Topic Sentences as Queries

Zoran Medić  Jan Šnajder
University of Zagreb, Faculty of Electrical Engineering and Computing
Text Analysis and Knowledge Engineering Lab
<EMAIL_ADDRESS>
###### Abstract

Citation recommendation (CR) models may help authors find relevant articles at
various stages of the paper writing process. Most research has dealt with
either global CR, which produces general recommendations suitable for the
initial writing stage, or local CR, which produces specific recommendations
more fitting for the final writing stages. We propose the task of _paragraph-
level CR_ as a middle ground between the two approaches, where the paragraph’s
topic sentence is taken as input and recommendations for citing within the
paragraph are produced at the output. We propose a model for this task, fine-
tune it using the quadruplet loss on the dataset of ACL papers, and show
improvements over the baselines.

##  1 Introduction

When writing papers, scientists should position their work within the
landscape of existing work and cite relevant articles. However, keeping track
of related work is becoming increasingly taxing in the era of information
overload (Johnson et al., 2018). Citation recommendation (CR) models may help
the authors by producing a list of articles to be cited for a given query. In
the task of _global citation recommendation_ (GCR), (Bhagavatula et al., 2018;
Ali et al., 2021, 2022) the queries are typically composed of the citing
article’s title and abstract. In contrast, in _local citation recommendation_
(LCR) (Ebesu and Fang, 2017; Medić and Šnajder, 2020a; Gu et al., 2022)
queries are citation contexts from the citing article’s body. Consequently,
while GCR recommendations tend to be diverse and general, LCR recommendations
are typically more specific and tailored to the citation context. These
different types of recommendations respond to different information needs in
the process of writing a scientific article. GCR recommendations are perhaps
more beneficial in the initial stages of research when the authors have a
rough research idea and look for previous work on that topic. In contrast, LCR
recommendations are better suited for the final stages of the writing process
when the authors are already familiar with prior work and only want to
complement the text with citations at specific places. However, between the
initial and the final stages, there are typically many intermediate stages
where authors draft and revise their manuscripts. Gauging the usability of GCR
and LCR for these intermediate stages is difficult, where queries are likely
to be more specific than for GCR but less specific than for LCR.

In this work, we consider a CR approach targeting the intermediate stages of
the writing process, particularly the writing of the related work (RW)
section. When writing the RW section, authors typically know enough about the
topic to identify and name the main strands of RW but cannot yet produce
detailed citation contexts. We introduce the Paragraph-level Citation
Recommendation (PCR) task, which takes a paragraph’s topic sentence as a query
and produces a list of articles for citing in that specific paragraph. As
topic sentences are typically more general than citation contexts appearing
later in the paragraph, they can be seen as proxies for queries the author
might pose in the intermediate stages of writing a RW section.

We develop the PCR model by fine-tuning SciNCL (Ostendorff et al., 2022), a
state-of-the-art transformer-based article encoder. We fine-tune SciNCL on a
dataset derived using CORWA (Li et al., 2022), a dataset of sentences from the
related work sections annotated with discourse labels. The proposed model
performs better than a baseline GCR model on this task, demonstrating the
potential of paragraph-level citation recommendation. Our contributions are
(1) PCR, a novel task of producing citation recommendations for an article’s
paragraph based on the paragraph’s topic sentence and (2) a model for PCR
fine-tuned and adapted from the existing GCR state-of-the-art model.

##  2 Related Work

State-of-the-art CR models all rely on deep learning. Typically, the models
encode text separately from the metadata information such as authors, venues,
or citation graphs (Bhagavatula et al., 2018; Medić and Šnajder, 2020a; Ali et
al., 2021). In recent work on GCR and LCR, the text is typically encoded with
transformer-based modules (Vaswani et al., 2017), which produce article
embeddings that can be used in the different recommendation and classification
tasks (Cohan et al., 2020; Ostendorff et al., 2022). For an overview of the
approaches to CR tasks, we refer the reader to (Medić and Šnajder, 2020b). We
follow the recent line of work that uses transformer-based encoders to produce
query and candidate article embeddings. However, our work differs from
previous work in that our model produces recommendations at the paragraph
level.

Recently introduced large language models (Brown et al., 2020; Zhang et al.,
2022; Scao et al., 2022; Taylor et al., 2022) that generate text when prompted
could be adapted for CR tasks. CR queries could be used to construct prompts
that require the model to output a list of recommended articles or even
generate an article’s text with citations inserted. However, such models tend
to generate texts containing false information (Ji et al., 2022). In this
work, we focus on the task of retrieving relevant articles given a query,
leaving the generation of citing text for future work.

##  3 Paragraph-Level CR

#### Task.

As with GCR and LCR, we define PCR as a ranking task: given a query
$\mathit{q}$ and a pool of articles
$\mathit{P}=\\{\mathit{a}_{(i)}\\}_{i=1}^{N}$, the model has to rank the
articles in $\mathit{P}$ based on the recommendation score for citing in
$\mathit{q}$. To construct a query $\mathit{q}$, we consider a citing article
$\mathit{a_{q}}$, consisting of a title and an abstract, a paragraph
$\mathit{p_{q}}$ from $\mathit{a_{q}}$, and sentences
$[\mathit{s_{1},...,s_{m}}]$ that make $\mathit{p_{q}}$. We form $\mathit{q}$
by concatenating $\mathit{a_{q}}$’s title and abstract with the sentence
$\mathit{s_{1}}$ from $\mathit{p_{q}}$, i.e., paragraph’s topic sentence. In
this setup, the model performs successfully if it ranks high the articles that
are cited in the rest of $\mathit{p_{q}}$ (i.e., in
$[\mathit{s_{2},...,s_{m}}]$).

#### Dataset.

To identify topic sentences in paragraphs, we used CORWA (Li et al., 2022), a
dataset derived from S2ORC (Lo et al., 2020), a large dataset of over 81M
articles from various scientific fields. CORWA contains discourse-labeled
sentences from the RW sections of ACL articles.111Association for
Computational Linguistics, https://aclanthology.org/ Albeit citations appear
in paragraphs from other sections as well, we focused on RW sections as they
provide citations to work topically relevant to the citing article and
typically group them into paragraphs by topic. CORWA’s discourse labels
reflect the sentences’ specific roles and information sources. We rely on the
Transition label, assigned to the sentences that “serve as topic introductions
or transitions from one topic to another” (Li et al., 2022). Suppose the first
sentence in a paragraph is labeled as Transition. In that case, we considered
it a topic sentence and treated the citations in the rest of the paragraph as
relevant for a query constructed using that sentence. Filtering CORWA based on
this criterion left us with 790 annotated RW paragraphs that start with a
topic sentence and cite at least one article. To enlarge the dataset, we used
the discourse classifier from (Li et al., 2022) and applied it to all the ACL
articles with full text available in S2ORC. We then added the silver-labeled
paragraphs that started with a “Transition”-labeled sentence to the gold-
labeled dataset, resulting in a total of 10,481 paragraphs.

To construct the training and evaluation sets, we split the paragraphs
published before, in, and after 2017 into the train, validation, and test set,
respectively. This resulted in 7,773, 560, and 2,148 paragraphs in the train,
validation, and test set, respectively. For the pool of articles for test set
evaluation, we extracted from S2ORC all articles with an ACL ID and all the
articles cited in any ACL article. We also removed all the articles with empty
title or abstract fields and ensured that only articles older than the query
make up the pool. The resulting pool contained 94,129 articles.

##  4 Model

We based the PCR model on SciNCL (Ostendorff et al., 2022), a transformer-
based scientific articles encoder that obtained state-of-the-art results on
benchmarks for evaluating scientific article representations (Cohan et al.,
2020; Medić and Šnajder, 2022). The model takes an article’s title and
abstract as input and produces an article embedding. It has been trained using
triplet loss with triplets of the form $\mathit{(a^{q},a^{+},a^{-})}$, where
$\mathit{a^{q}}$ is the query (the citing article), $\mathit{a^{+}}$ is a
positive item (a cited article), and $\mathit{a^{-}}$ is a negative item (an
article not cited in the citing article). Given embedding $\mathbf{e}$ for
input $a$, the triplet loss is defined as
$\mathcal{L}_{t}=\max(0,\|\mathbf{e^{q}}-\mathbf{e^{+}}\|-\|\mathbf{e^{q}}-\mathbf{e^{-}}\|+m)$,
where $\|\cdot\|$ is the L2 norm and $m$ is a margin between the two norms. We
adapted SciNCL to PCR by modifying the query construction and the loss
function. We built queries by concatenating the paragraph’s topic sentence to
the citing article’s title and abstract and separating it from the rest with a
special [TS] token. As in SciNCL, we took the final layer’s [CLS]’s
representation as the query’s embedding. We used the same encoder to embed the
pool articles, with their title and abstract as input.

Unlike Ostendorff et al. (2022), who minimize the triplet loss, we minimized
the quadruplet loss. The quadruplet loss has been used for LCR (Zhang and Ma,
2021), where it has been found to outperform models trained with triplet loss.
The quadruplet has two positive items, $\mathit{a^{+}_{1}}$ and
$\mathit{a^{+}_{2}}$, and the loss is modified accordingly. Given a quadruplet
$(a^{q},a^{+}_{1},a^{+}_{2},a^{-})$, we minimized the following loss:

$\displaystyle\mathcal{L}_{q}$
$\displaystyle=\sum_{i}\max(0,\|\mathbf{e^{q}}-\mathbf{e^{+}_{i}}\|-\|\mathbf{e^{q}}-\mathbf{e^{-}}\|+m)$
$\displaystyle+\sum_{i}\max(0,\|\mathbf{e^{+}_{1}}-\mathbf{e^{+}_{2}}\|-\|\mathbf{e^{+}_{i}}-\mathbf{e^{-}}\|+m)$

Compared to $\mathcal{L}_{t}$, loss $\mathcal{L}_{q}$ includes an additional
term for promoting similarity between the query and the second positive item
(the first term), as well as two additional terms that force the embeddings of
the positive items to be close to each other and far from the negative item
(the second term).

Previous work has shown the importance of choosing informative and hard
negative examples for training text encoders (Li et al., 2019; Ostendorff et
al., 2022). To select the challenging negative items, we sampled them from
three pools: (1) articles cited in the RW section of the citing article but in
a paragraph different from the query’s, (2) articles cited in the citing
article but in a section other than RW, and (3) articles cited in the cited
articles but not in the citing article. We sampled 10 quadruplets for each
paragraph: three quadruplets each from the first and the second pool, and four
quadruplets from the third pool.

We fine-tuned the SciNCL checkpoint from the Hugging Face Hub (Wolf et al.,
2020) for five epochs and tracked R-precision on the validation set. To avoid
overfitting, we only fine-tuned the last two layers of SciNCL. We used the
AdamW optimizer (Loshchilov and Hutter, 2018) with $\beta_{1}$ = 0.99,
$\beta_{2}$ = 0.999, weight decay set to 0.01, and learning rate of 1e-5 with
a linear scheduler for the first 10% of learning steps. The value of $m$ in
$\mathcal{L}_{q}$ was set to 0.5 for all the loss terms. We embedded all the
test set queries and the pool articles using the trained model and performed
the nearest neighbor search using Faiss (Johnson et al., 2019). We call our
model qPCR (quadruplet-based Paragraph Citation Recommender).222Our code with
links to data and the model is available at:
https://github.com/zoranmedic/qpcr

##  5 Experiments

We compared the performance of qPCR against three baselines: (1) SciNCL with
the citing article’s title and abstract as input (SciNCL-base), (2) SciNCL
with topic sentence concatenated with title and abstract as input (SciNCL-ts),
and (3) SciNCL fine-tuned with ACL’s RW triplets (SciNCL-aclrw). For the last
model, we fine-tuned SciNCL with triplets of cited and non-cited pairs sampled
from the articles cited in the RW sections of ACL articles. This baseline
helps us analyze whether the performance gap between qPCR and SciNCL-ts is due
to domain adaptation (as SciNCL was trained on a multi-domain corpus) or the
inclusion of a topic sentence. We also report the results of fine-tuning
SciNCL with triplets instead of quadruplets (SciNCL-tri).

We report R-precision, R@5, R@10, and MRR. Among these, R-precision seems the
most appropriate for the PCR task, as the number of cited articles varies
across paragraphs.333The average number of articles cited per paragraph is
3.5. However, since authors often do not cite all the relevant articles, but
only a few, we also report R@5 and R@10 as more lenient metrics that do not
penalize false positives as much as R-precision. We report an average across
three fine-tuning runs for each model we fine-tuned.

Model | R-prec | R@5 | R@10 | MRR
---|---|---|---|---
SciNCL-base | 6.92 | 8.47 | 12.70 | 17.32
SciNCL-ts | 7.62 | 9.86 | 14.58 | 19.23
SciNCL-aclrw | 7.61 | 9.87 | 14.77 | 19.21
SciNCL-tri | 7.63 | 9.57 | 14.58 | 19.30
qPCR | 8.00 | 10.21 | 15.19 | 20.05
Table 1: Performance of PCR models on test set queries. Values are in the
0–100 range. The best results are shown in bold.

#### Results.

Table 1 shows the results. The first observation is that SciNCL-ts yields
better results than SciNCL-base, demonstrating the potential of using topic
sentences in CR queries. SciNCL-tri does not perform better than SciNCL-ts,
while qPCR obtains the best results across all the reported metrics. The
increase in performance from SciNCL-tri to qPCR indicates the benefits of
training with a more informative signal derived from quadruplets instead of
triplets. Another observation is that domain adaptation of SciNCL on ACL RW
triplets does not increase performance compared to SciNCL-ts, except for R@10.

Overall, the improvement of qPCR over SciNCL-ts, the best-performing baseline
model that does not use topic sentences in training, is not that large:
ranging from 0.35 for R@5 to 0.82 for MRR. We believe this small difference is
indicative of the difficulty of the task. It also reflects an issue with using
citations as relevance signals when articles are represented using text only,
as text is often not the only citing criteria (Mammola et al., 2022).

#### Qualitative Analysis.

Figure 1: Average ranks of cited articles per publication year for the ranking
obtained with qPCR.

To gain insights into PCR’s difficulty and qPCR’s performance, we manually
compared some of the qPCR and SciNCL-ts ranking lists. Recommendations
provided by both models often seem relevant to the given paragraph but are
counted as false positives as they were not cited in the paragraph (cf.
Appendix for some examples). While this is not surprising, as citing all
articles relevant to a specific context is not feasible, it suggests that
better evaluation datasets are needed for developing CR models.

We also find that both models perform worse when the cited article is older
than the citing article. Fig. 1 shows the average qPCR ranking of cited
articles across their publication years, revealing that older articles tend to
have a lower ranking. In contrast, new ones are closer to the top of the
ranking list. This is also confirmed by the positive Pearson correlation
coefficient of 0.15 (p$<$0.05) between the difference in cited and citing
article’s publication years ($\Delta t)$ and the cited article’s rank. We
hypothesized this might be due to the vocabulary mismatch between the citing
and cited articles that developed over time (cf. Appendix for some examples).
Indeed, the Pearson correlation coefficient between $\Delta t$ and the Jaccard
index between the query as defined in PCR and the cited article (as a measure
of word overlap) is $-0.24$ (p$<$0.05), i.e., a larger gap in publication
years is accompanied by a smaller term overlap, supporting (though not
proving) the vocabulary mismatch hypothesis.

Another observation is that qPCR more often than SciNCL-ts recommends articles
with titles more topically aligned with a paragraph’s topic sentence (cf.
Appendix for some examples). To verify this, we calculated the L2 distance
between the embeddings of topic sentences and the titles of recommended
articles using CoSentBert (Mysore et al., 2022), a variant of Sentence-BERT
(Reimers and Gurevych, 2019) trained on the scientific domain. The average
distance between topic sentences and titles recommended by qPCR and SciNCL-ts
is 14.16 and 14.38, respectively. In addition, on average, qPCR’s attention
heads across all layers assign higher weights to tokens from the topic
sentences than SciNCL-ts. Specifically, the average sum of the topic
sentence’s token weights in the case of qPCR and SciNCL-ts is 0.25 and 0.12,
respectively. These findings confirm that qPCR leverages topic sentence
information from the query. We leave the investigation of what constitutes an
optimal PCR query for future work.

##  6 Conclusion

We introduced paragraph-level citation recommendation (PCR), a task of
recommending articles for citing in a paragraph based on the paragraph’s topic
sentence. The proposed qPCR model outperformed other sensible baselines and
demonstrated improvements of training with the quadruplet loss over standard
triplet loss. Interesting future work directions include generative PCR
models, which use qPCR’s recommendations to generate paragraphs, and the
construction of evaluation datasets based on stronger relevance signals than
citations.

## Limitations

qPCR was trained only with paragraphs from RW sections of ACL articles. This
raises the question of how well the model would perform in different article
sections and scientific domains. Furthermore, as our dataset relies on CORWA’s
classifier for discourse labels, errors made by the classifier are propagated
to our dataset and can affect the model’s performance. Additionally, the
articles’ texts are processed using parsers and tools for sentence
segmentation that could result in errors and produce both incoherent titles
and abstracts and topic sentences identified by CORWA’s classifier. Regarding
the language, our dataset contains only the articles written in English, which
limits the applicability of our model to other languages.

## Ethics Statement

Both the dataset we use and the model we fine-tune are publicly available. Our
experiments’ dataset is constructed from CORWA (Li et al., 2022), derived from
S2ORC (Lo et al., 2020), a publicly available dataset of scientific articles
released under CC BY-NC 2.0 license. Silver-labeled part of the dataset in our
experiments was obtained by running CORWA classifier on articles from S2ORC.
SciNCL, the model that we fine-tune and use in our experiments, is available
at Hugging Face Hub (Wolf et al., 2020) and released under MIT Licence.

As we use citations from S2ORC as labels when training qPCR, the bias inherent
to citing patterns in articles from S2ORC might be passed onto the model
trained on these articles. For example, training the model on articles that
cite the work from specific authors or institutions when more articles fit
into the context might lead to the model preferring articles from particular
authors or institutions over others. Many other subjective factors influencing
citing behavior might be passed onto the model trained on such data. However,
despite the potential biases that might appear in the model we trained, we
believe introducing a new task in our work benefits the scientific community.
It could stimulate more work on the task of citation recommendation and,
alongside it, the work on debiasing citation recommendation systems in various
scenarios.

## Acknowledgments

The first author has been supported by a grant from the Croatian Science
Foundation (HRZZ-DOK-2018-09).

## References

  * Ali et al. (2022) Zafar Ali, Guilin Qi, Pavlos Kefalas, Shah Khusro, Inayat Khan, and Khan Muhammad. 2022.  Spr-smn: scientific paper recommendation employing specter with memory network.  _Scientometrics_ , 127(11):6763–6785.
  * Ali et al. (2021) Zafar Ali, Guilin Qi, Khan Muhammad, Pavlos Kefalas, and Shah Khusro. 2021.  Global citation recommendation employing generative adversarial network.  _Expert Systems with Applications_ , 180:114888.
  * Bhagavatula et al. (2018) Chandra Bhagavatula, Sergey Feldman, Russell Power, and Waleed Ammar. 2018.  Content-based citation recommendation.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 238–251, New Orleans, Louisiana. Association for Computational Linguistics.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Cohan et al. (2020) Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S Weld. 2020.  Specter: Document-level representation learning using citation-informed transformers.  _arXiv preprint arXiv:2004.07180_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ebesu and Fang (2017) Travis Ebesu and Yi Fang. 2017.  Neural citation network for context-aware citation recommendation.  In _Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval_ , pages 1093–1096.
  * Gu et al. (2022) Nianlong Gu, Yingqiang Gao, and Richard HR Hahnloser. 2022.  Local citation recommendation with hierarchical-attention text encoder and scibert-based reranking.  In _European Conference on Information Retrieval_ , pages 274–288. Springer.
  * Ji et al. (2022) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2022.  Survey of hallucination in natural language generation.  _ACM Computing Surveys_.
  * Johnson et al. (2019) Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019.  Billion-scale similarity search with GPUs.  _IEEE Transactions on Big Data_ , 7(3):535–547.
  * Johnson et al. (2018) Rob Johnson, Anthony Watkinson, and Michael Mabe. 2018.  The stm report: An overview of scientific and scholarly publishing.
  * Li et al. (2019) Jia Li, Chongyang Tao, Yansong Feng, Dongyan Zhao, Rui Yan, et al. 2019.  Sampling Matters! An Empirical Study of Negative Sampling Strategies for Learning of Matching Models in Retrieval-Based Dialogue Systems.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1291–1296.
  * Li et al. (2022) Xiangci Li, Biswadip Mandal, and Jessica Ouyang. 2022.  CORWA: A citation-oriented related work annotation dataset.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5426–5440, Seattle, United States. Association for Computational Linguistics.
  * Lo et al. (2020) Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel S Weld. 2020.  S2orc: The semantic scholar open research corpus.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4969–4983.
  * Loshchilov and Hutter (2018) Ilya Loshchilov and Frank Hutter. 2018.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_.
  * Mammola et al. (2022) Stefano Mammola, Elena Piano, Alberto Doretto, Enrico Caprio, and Dan Chamberlain. 2022.  Measuring the influence of non-scientific features on citations.  _Scientometrics_ , pages 1–15.
  * Medić and Šnajder (2020a) Zoran Medić and Jan Šnajder. 2020a.  Improved local citation recommendation based on context enhanced with global information.  In _Proceedings of the First Workshop on Scholarly Document Processing_ , pages 97–103, Online. Association for Computational Linguistics.
  * Medić and Šnajder (2020b) Zoran Medić and Jan Šnajder. 2020b.  A survey of citation recommendation tasks and methods.  _Journal of computing and information technology_ , 28(3):183–205.
  * Medić and Šnajder (2022) Zoran Medić and Jan Šnajder. 2022.  Large-scale evaluation of transformer-based article encoders on the task of citation recommendation.  In _Proceedings of the Third Workshop on Scholarly Document Processing_ , pages 19–31, Gyeongju, Republic of Korea. Association for Computational Linguistics.
  * Mysore et al. (2022) Sheshera Mysore, Arman Cohan, and Tom Hope. 2022.  Multi-vector models with textual guidance for fine-grained scientific document similarity.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4453–4470, Seattle, United States. Association for Computational Linguistics.
  * Ostendorff et al. (2022) Malte Ostendorff, Nils Rethmeier, Isabelle Augenstein, Bela Gipp, and Georg Rehm. 2022.  Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings.  In _The 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022)_ , Abu Dhabi. Association for Computational Linguistics.  7-11 December 2022. Accepted for publication.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022.  Bloom: A 176b-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_.
  * Taylor et al. (2022) Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022\.  Galactica: A large language model for science.  _arXiv preprint arXiv:2211.09085_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_.
  * Zhang and Ma (2021) Yang Zhang and Qiang Ma. 2021.  Recommending multiple positive citations for manuscript via content-dependent modeling and multi-positive triplet.  In _IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology_ , pages 629–634.

##  Appendix A Appendix

Here we provide examples of paragraphs in which older articles were cited and
of those for which many of the recommendations by either SciNCL-ts or qPCR
seem relevant but were considered as false positives since they were not cited
in the given paragraph. Examples of paragraphs with older cited articles are
given in Tables 2 and 3. Examples of cases with seemingly relevant false
positives are given in Tables 4, 5, and 6. Tables 7 and 8 showcase examples of
paragraphs for which qPCR produces recommendations with titles that are more
topically relevant for the topic sentence than those of SciNCL-ts.

Title: CommonsenseQA: A Question Answering Challenge Targeting Commonsense
Knowledge
---
Year: 2018
Abstract: When answering a question, people often draw upon their rich world
knowledge in addition to some task-specific context. Recent work has focused
primarily on answering questions based on some relevant document or content,
and required very little general background. To investigate question answering
with prior knowledge, we present CommonsenseQA: a difficult new dataset for
commonsense question answering. To capture common sense beyond associations,
each question discriminates between three target concepts that all share the
same relationship to a single source drawn from ConceptNet (Speer et al.,
2017). This constraint encourages crowd workers to author multiple-choice
questions with complex semantics, in which all candidates relate to the
subject in a similar way. We create 9,500 questions through this procedure and
demonstrate the dataset’s difficulty with a large number of strong baselines.
Our best baseline, the OpenAI GPT (Radford et al., 2018), obtains 54.8%
accuracy, well below human performance, which is 95.3%.
Paragraph: Machine common sense, or the knowledge of and ability to reason
about an open ended world, has long been acknowledged as a critical component
for natural language understanding. Early work sought programs that could
reason about an environment in natural language (McCarthy, 1959) , or leverage
a world-model for deeper language understanding (Winograd, 1972) . Many
commonsense representations and inference procedures have been explored
(McCarthy and Hayes, 1969; Kowalski and Sergot, 1986 ) and large-scale
commonsense knowledge-bases have been developed (Lenat, 1995; Speer et al.,
2017) . However, evaluating the degree of common sense possessed by a machine
remains difficult.
Rank |  SciNCL-ts |  Rank |  qPCR
1 |  Philosophers are Mortal: Inferring the Truth of Unseen Facts |  1 |  Philosophers are Mortal: Inferring the Truth of Unseen Facts
2 |  Acquiring comparative commonsense knowledge from the web |  2 |  NewsQA: A Machine Comprehension Dataset
3 |  SemEval-2012 Task 7: Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning |  3 |  MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text
4 |  Learner: a system for acquiring commonsense knowledge by analogy |  4 |  Learner: a system for acquiring commonsense knowledge by analogy
5 |  Predicting Answer Location Using Shallow Semantic Analogical Reasoning in a Factoid Question Answering System |  5 |  Deep Questions without Deep Understanding
6 |  Planning, Executing, and Evaluating the Winograd Schema Challenge |  6 |  Open Mind Common Sense: Knowledge Acquisition from the General Public
7 |  NewsQA: A Machine Comprehension Dataset |  7 |  Mind Commons : An Inquisitive Approach to Learning Common Sense
8 |  Deep Questions without Deep Understanding |  8 |  SemEval-2012 Task 7: Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning
9 |  WebBrain: Joint Neural Learning of Large-Scale Commonsense Knowledge |  9 |  MovieQA: Understanding Stories in Movies through Question-Answering
10 |  A Neural Network for Factoid Question Answering over Paragraphs |  10 |  Acquiring comparative commonsense knowledge from the web
>1000 |  Programs with common sense |  >1000 |  Programs with common sense
Cited abstract: This paper discusses programs to manipulate in a suitable
formal language (most likely a part of the predicate calculus) common
instrumental statements. The basic program will draw immediate conclusions
from a list of premises. These conclusions will be either declarative or
imperative sentences. When an imperative sentence is deduced the program takes
a corresponding action. These actions may include printing sentences, moving
sentences on lists, and reinitiating the basic deduction process on these
lists.
Table 2: An example of a paragraph that cites an article that is older than
the citing. Citation marker in the paragraph is printed in olive, as well as
the title of the cited article. Although an experienced researcher might
conclude from the title and abstract alone that the cited article might be
suitable for citing in recent articles on commonsense reasoning, the
vocabulary mismatch between the articles might introduce difficulties for CR
models. Title: BanditSum: Extractive Summarization as a Contextual Bandit
---
Year: 2018
Abstract: In this work, we propose a novel method for training neural networks
to perform single-document extractive summarization without heuristically-
generated extractive labels. We call our approach BanditSum as it treats
extractive summarization as a contextual bandit (CB) problem, where the model
receives a document to summarize (the context), and chooses a sequence of
sentences to include in the summary (the action). A policy gradient
reinforcement learning algorithm is used to train the model to select
sequences of sentences that maximize ROUGE score. We perform a series of
experiments demonstrating that BanditSum is able to achieve ROUGE scores that
are better than or comparable to the state-of-the-art for extractive
summarization, and converges using significantly fewer update steps than
competing approaches. In addition, we show empirically that BanditSum performs
significantly better than competing approaches when good summary sentences
appear late in the source document.
Paragraph: Extractive summarization has been widely studied in the past.
Recently, neural network-based methods have been gaining popularity over
classical methods (Luhn, 1958; Gong and Liu, 2001; Conroy and O’leary, 2001;
Mihalcea and Tarau, 2004; Wong et al., 2008) , as they have demonstrated
stronger performance on large corpora. Central to the neural network-based
models is the encoderdecoder structure. These models typically use either a
convolution neural network (Kalchbrenner et al., 2014; Kim, 2014; Yin and Pei,
2015; Cao et al., 2015) , a recurrent neural network (Chung et al., 2014;
Cheng and Lapata, 2016; Nallapati et al., 2017) , or a combination of the two
(Narayan et al., 2018; Wu and Hu, 2018) to create sentence and document
representations, using word embeddings (Mikolov et al., 2013; Pennington et
al., 2014) to represent words at the input level. These vectors are then fed
into a decoder network to generate the output summary. The use of
reinforcement learning (RL) in extractive summarization was first explored by
Ryang and Abekawa (2012) , who proposed to use the TD($\lambda$) algorithm to
learn a value function for sentence selection. Rioux et al. (2014) improved
this framework by replacing the learning agent with another TD($\lambda$)
algorithm. However, the performance of their methods was limited by the use of
shallow function approximators, which required performing a fresh round of
reinforcement learning for every new document to be summarized. The more
recent work of Paulus et al. (2018) and Wu and Hu (2018) use reinforcement
learning in a sequential labeling setting to train abstractive and extractive
summarizers, respectively, while Chen and Bansal (2018) combines both
approaches, applying abstractive summarization to a set of sentences extracted
by a pointer network (Vinyals et al., 2015) trained via REINFORCE. However,
pre-training with a maximum likelihood objective is required in all of these
models.
Rank |  SciNCL-ts |  Rank |  qPCR
1 |  Classify or Select: Neural Architectures for Extractive Document Summarization |  1 |  Extractive Multi-Document Summarization with Integer Linear Programming and Support Vector Regression
2 |  Neural Summarization by Extracting Sentences and Words |  2 |  Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
3 |  A Neural Attention Model for Abstractive Sentence Summarization |  3 |  Neural Summarization by Extracting Sentences and Words
4 |  Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond |  4 |  Multi-Document Summarization Using A* Search and Discriminative Learning
5 |  A Neural Attention Model for Sentence Summarization |  5 |  SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents
6 |  SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents |  6 |  Discovery of Topically Coherent Sentences for Extractive Summarization
7 |  Framework of Automatic Text Summarization Using Reinforcement Learning |  7 |  Extractive Multi-Document Summaries Should Explicitly Not Contain Document Specific Content
8 |  Towards the Use of Deep Reinforcement Learning with Global Policy For Query-based Extractive Summarisation |  8 |  Multi-document abstractive summarization using ILP based multi-sentence compression
9 |  From neural sentence summarization to headline generation: a coarse-to-fine approach |  9 |  A Hybrid Hierarchical Model for Multi-Document Summarization
10 |  Using Supervised Bigram-based ILP for Extractive Summarization |  10 |  Ultra-Summarization: A Statistical Approach to Generating Highly Condensed Non-Extractive Summaries
>1000 |  The automatic creation of literature abstracts |  >1000 |  The automatic creation of literature abstracts
Cited abstract: Excerpts of technical papers and magazine articles that serve
the purposes of conventional abstracts have been created entirely by automatic
means. In the exploratory research described, the complete text of an article
in machine-readable form is scanned by an IBM 704 data-processing machine and
analyzed in accordance with a standard program. Statistical information
derived from word frequency and distribution is used by the machine to compute
a relative measure of significance, first for individual words and then for
sentences. Sentences scoring highest in significance are extracted and printed
out to become the "auto-abstract."
Table 3: Another example of an article that cites an older work. Similarly as
in Table 2, an experienced researcher might find the cited article fairly
relevant as a classical method for extractive summarization, but CR models
might struggle with vocabulary mismatch (e.g., “model” in citing is a
“machine” in the cited article, “document” is an “article”, and “summary” is
an “excerpt”). Title: MOROCO: The Moldavian and Romanian Dialectal Corpus
---
Year: 2019
Abstract: In this work, we introduce the MOldavian and ROmanian Dialectal
COrpus (MOROCO), which is freely available for download at this https URL. The
corpus contains 33564 samples of text (with over 10 million tokens) collected
from the news domain. The samples belong to one of the following six topics:
culture, finance, politics, science, sports and tech. The data set is divided
into 21719 samples for training, 5921 samples for validation and another 5924
samples for testing. For each sample, we provide corresponding dialectal and
category labels. This allows us to perform empirical studies on several
classification tasks such as (i) binary discrimination of Moldavian versus
Romanian text samples, (ii) intra-dialect multi-class categorization by topic
and (iii) cross-dialect multi-class categorization by topic. We perform
experiments using a shallow approach based on string kernels, as well as a
novel deep approach based on character-level convolutional neural networks
containing Squeeze-and-Excitation blocks. We also present and analyze the most
discriminative features of our best performing model, before and after named
entity removal.
Paragraph: Other languages. The Nordic Dialect Corpus (Johannessen et al.,
2009 ) contains about 466K spoken words from Denmark, Faroe Islands, Iceland,
Norway and Sweden. The authors transcribed each dialect by the standard
official orthography of the corresponding country. Francom et al. (2014)
introduced the ACTIV-ES corpus, which represents a cross-dialectal record of
the informal language use of Spanish speakers from Argentina, Mexico and
Spain. The data set is composed of 430 TV or movie subtitle files. The DSL
corpus collection (Tan et al., 2014) comprises news data from various corpora
to emulate the diverse news content across different languages. The collection
is comprised of six language vari- ety groups. For each language, the
collection contains 18K training sentences, 2K validation sentences and 1K
test sentences. The ArchiMob corpus (Samardžić et al., 2016) contains
manuallyannotated transcripts of Swiss German speech collected from four
different regions: Basel, Bern, Lucerne and Zurich. The data set was used in
the 2017 and 2018 VarDial Evaluation Campaigns (Zampieri et al., 2017
(Zampieri et al., , 2018 . Kumar et al. (2018) constructed a corpus of five
Indian dialects consisting of 307K sentences. The samples were collected by
scanning, passing through an OCR engine and proofreading printed stories,
novels and essays from books, magazines or newspapers. Romanian. To our
knowledge, the only empirical study on Romanian dialect identification was
conducted by Ciobanu and Dinu (2016) . In their work, Ciobanu and Dinu (2016)
used only a short list of 108 parallel words in a binary classification task
in order to discriminate between Daco-Romanian words versus Aromanian, Istro-
Romanian and Megleno-Romanian words. Different from Ciobanu and Dinu (2016) ,
we conduct a large scale study on 33K documents that contain a total of about
10 million tokens.
Rank |  SciNCL-ts |  Rank |  qPCR
1 |  German Dialect Identification Using Classifier Ensembles |  1 |  Exploiting Convolutional Neural Networks for Phonotactic Based Dialect Identification
2 |  Classifying English Documents by National Dialect |  2 |  German Dialect Identification Using Classifier Ensembles
3 |  Exploiting Convolutional Neural Networks for Phonotactic Based Dialect Identification |  3 |  Native Language Identification with String Kernels
4 |  A Character-level Convolutional Neural Network for Distinguishing Similar Languages and Dialects |  4 |  Discriminating between Indo-Aryan Languages Using SVM Ensembles
5 |  CLUZH at VarDial GDI 2017: Testing a Variety of Machine Learning Tools for the Classification of Swiss German Dialects |  5 |  UnibucKernel Reloaded: First Place in Arabic Dialect Identification for the Second Year in a Row
6 |  The GW/LT3 VarDial 2016 Shared Task System for Dialects and Similar Languages Detection |  6 |  Convolutional Neural Networks and Language Embeddings for End-to-End Dialect Recognition
7 |  Sentence Level Dialect Identification in Arabic |  7 |  A Character-level Convolutional Neural Network for Distinguishing Similar Languages and Dialects
8 |  Word-Based Dialect Identification with Georeferenced Rules |  8 |  Cross-corpus Native Language Identification via Statistical Embedding
9 |  UnibucKernel Reloaded: First Place in Arabic Dialect Identification for the Second Year in a Row |  9 |  An Exploration of Language Identification Techniques for the Dutch Folktale Database
10 |  Cross-corpus Native Language Identification via Statistical Embedding |  10 |  Feature Extraction for Native Language Identification Using Language Modeling
Table 4: An example of recommendations for the paragraph describing work on
dialect identification in other languages. None of the recommendations of
SciNCL-ts and qPCR was cited in the paragraph, although some of the articles
could fit into the paragraph’s narrative.  Title: Introducing two Vietnamese
Datasets for Evaluating Semantic Models of (Dis-)Similarity and Relatedness
---
Year: 2018
Abstract: We present two novel datasets for the low-resource language
Vietnamese to assess models of semantic similarity: ViCon comprises pairs of
synonyms and antonyms across word classes, thus offering data to distinguish
between similarity and dissimilarity. ViSim-400 provides degrees of similarity
across five semantic relations, as rated by human judges. The two datasets are
verified through standard co-occurrence and neural network models, showing
results comparable to the respective English datasets.
Paragraph: For other languages, only a few gold standard sets with scored word
pairs exist. Among others, Gurevych (2005) replicated Rubenstein and
Goodenough’s experiments after translating the original 65 word pairs into
German. In later work, Gurevych (2006) used the same experimental setup to
increase the number of word pairs to 350. Leviant and Reichart (2015)
translated two prominent evaluation sets, WordSim-353 (association) and
SimLex-999 (similarity) from English to Italian, German and Russian, and
collected the scores for each dataset from the respective native speakers via
crowdflower 6 .
Rank |  SciNCL-ts |  Rank |  qPCR
1 |  Comparing Semantic Relatedness between Word Pairs in Portuguese Using Wikipedia |  1 |  Comparing Semantic Relatedness between Word Pairs in Portuguese Using Wikipedia
2 |  Gathering Information About Word Similarity from Neighbor Sentences |  2 |  A Framework for the Construction of Monolingual and Cross-lingual Word Similarity Datasets
3 |  Duluth : Measuring Degrees of Relational Similarity with the Gloss Vector Measure of Semantic Relatedness |  3 |  Human and Machine Judgements for Russian Semantic Relatedness
4 |  A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches |  4 |  SemEval-2014 Task 3: Cross-Level Semantic Similarity
5 |  ExB Themis: Extensive Feature Extraction from Word Alignments for Semantic Textual Similarity |  5 |  An English-Chinese Cross-lingual Word Semantic Similarity Measure Exploring Attributes and Relations
6 |  Lemon and Tea Are Not Similar: Measuring Word-to-Word Similarity by Combining Different Methods |  6 |  ExB Themis: Extensive Feature Extraction from Word Alignments for Semantic Textual Similarity
7 |  Measuring Semantic Relatedness Using People and WordNet |  7 |  HHU at SemEval-2016 Task 1: Multiple Approaches to Measuring Semantic Textual Similarity
8 |  Measuring Semantic Relatedness using Multilingual Representations |  8 |  A critique of word similarity as a method for evaluating distributional semantic models
9 |  Comparing Wikipedia and German Wordnet by Evaluating Semantic Relatedness on Multiple Datasets |  9 |  Gathering Information About Word Similarity from Neighbor Sentences
10 |  A New Set of Norms for Semantic Relatedness Measures |  10 |  A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches
Table 5: An example of recommendations for the paragraph describing work in
other languages on the construction of word pairs similarity datasets. None of
the recommendations of SciNCL-ts and qPCR were cited in the paragraph,
although some of the articles could fit into the paragraph’s narrative. Title:
Part-of-Speech Tagging on an Endangered Language: a Parallel Griko-Italian
Resource
---
Year: 2018
Abstract: Most work on part-of-speech (POS) tagging is focused on high
resource languages, or examines low-resource and active learning settings
through simulated studies. We evaluate POS tagging techniques on an actual
endangered language, Griko. We present a resource that contains 114 narratives
in Griko, along with sentence-level translations in Italian, and provides gold
annotations for the test set. Based on a previously collected small corpus, we
investigate several traditional methods, as well as methods that take
advantage of monolingual data or project cross-lingual POS tags. We show that
the combination of a semi-supervised method with cross-lingual transfer is
more appropriate for this extremely challenging setting, with the best tagger
achieving an accuracy of 72.9%. With an applied active learning scheme, which
we use to collect sentence-level annotations over the test set, we achieve
improvements of more than 21 percentage points.
Paragraph: Most of the previous approaches are rarely tested on under-
represented languages, with research on POS tagging for endangered languages
being sporadic. Ptaszynski and Momouchi (2012) , for example, presented an
HMM-based POS tagger for the extremely endangered Ainu language, based on
dictionaries, 12 narratives (yukar), using one annotated story (200 words) for
evaluation. To our knowledge, no other previous work has extensively tested
several approaches on an actual endangered language.
Rank |  SciNCL-ts |  Rank |  qPCR
1 |  CombiTagger: A System for Developing Combined Taggers |  1 |  Distantly Supervised POS Tagging of Low-Resource Languages under Extreme Data Sparsity: The Case of Hittite
2 |  Distantly Supervised POS Tagging of Low-Resource Languages under Extreme Data Sparsity: The Case of Hittite |  2 |  What Can We Get From 1000 Tokens? A Case Study of Multilingual POS Tagging For Resource-Poor Languages
3 |  TagMiner: A Semisupervised Associative POS Tagger Effective for Resource Poor Languages |  3 |  Unsupervised adaptation of supervised part-of-speech taggers for closely related languages
4 |  Improving the PoS tagging accuracy of Icelandic text |  4 |  TagMiner: A Semisupervised Associative POS Tagger Effective for Resource Poor Languages
5 |  Wiki-ly Supervised Part-of-Speech Tagging |  5 |  Part-of-Speech Tag Disambiguation by Cross-Linguistic Majority Vote
6 |  Coupling an annotated corpus and a lexicon for state-of-the-art POS tagging |  6 |  POS Tagging for Historical Texts with Sparse Training Data
7 |  Something Borrowed, Something Blue: Rule-based Combination of POS Taggers |  7 |  Unsupervised and Lightly Supervised Part-of-Speech Tagging Using Recurrent Neural Networks
8 |  Coupling an annotated corpus and a morphosyntactic lexicon for state-ofthe-art POS tagging with less human effort |  8 |  Wiki-ly Supervised Part-of-Speech Tagging
9 |  What Can We Get From 1000 Tokens? A Case Study of Multilingual POS Tagging For Resource-Poor Languages |  9 |  Improving Accuracy in Word Class Tagging through the Combination of Machine Learning Systems
10 |  Ensemble system for Part-of-Speech tagging |  10 |  Mac-Morpho Revisited: Towards Robust Part-of-Speech Tagging
Table 6: An example of recommendations for the paragraph describing work on
POS tagging for endangered languages. Similar to Table 5, none of the
recommendations of SciNCL-ts and qPCR were cited in the paragraph, although
some of the articles could fit into the paragraph’s narrative. Title:
Dependency Parsing for Spoken Dialog Systems
---
Year: 2019
Abstract: Dependency parsing of conversational input can play an important
role in language understanding for dialog systems by identifying the
relationships between entities extracted from user utterances. Additionally,
effective dependency parsing can elucidate differences in language structure
and usage for discourse analysis of human-human versus human-machine dialogs.
However, models trained on datasets based on news articles and web data do not
perform well on spoken human-machine dialog, and currently available
annotation schemes do not adapt well to dialog data. Therefore, we propose the
Spoken Conversation Universal Dependencies (SCUD) annotation scheme that
extends the Universal Dependencies (UD) (Nivre et al., 2016) guidelines to
spoken human-machine dialogs. We also provide ConvBank, a conversation dataset
between humans and an open-domain conversational dialog system with SCUD
annotation. Finally, to demonstrate the utility of the dataset, we train a
dependency parser on the ConvBank dataset. We demonstrate that by pre-training
a dependency parser on a set of larger public datasets and fine-tuning on
ConvBank data, we achieved the best result, 85.05% unlabeled and 77.82%
labeled attachment accuracy.
Paragraph: Previous work has demonstrated that expanding the UD annotation
scheme can result in successful parsers for other domains. For example, Liu et
al. (2018) expand the UD scheme to train a parser for Twitter data. Thus we
take a similar approach in expanding the UD scheme to encompass issues common
to automated speech transcription.
Rank |  SciNCL-ts |  Rank |  qPCR
1 |  Adding Syntactic Annotations to Transcripts of Parent-Child Dialogs |  1 |  Discourse parsing for multi-party chat dialogues
2 |  Discourse parsing for multi-party chat dialogues |  2 |  Annotation for and Robust Parsing of Discourse Structure on Unrestricted Texts
3 |  A Dependency Treebank for English |  3 |  A Dependency Parser for Tweets
4 |  Discovering Conversational Dependencies between Messages in Dialogs |  4 |  An Iterative Approach for Joint Dependency Parsing and Semantic Role Labeling
5 |  Annotating Spoken Dialogs: From Speech Segments to Dialog Acts and Frame Semantics |  5 |  Multilingual Dependency Learning: A Huge Feature Engineering Method to Semantic Dependency Parsing
6 |  DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset |  6 |  SemEval-2012 Task 5: Chinese Semantic Dependency Parsing
7 |  The ICSI Meeting Recorder Dialog Act (MRDA) Corpus |  7 |  Parsing Tweets into Universal Dependencies
8 |  The Tübingen Treebanks for Spoken German, English, and Japanese |  8 |  Hybrid Learning of Dependency Structures from Heterogeneous Linguistic Resources
9 |  Annotation for and Robust Parsing of Discourse Structure on Unrestricted Texts |  9 |  Universal Dependency Parsing with a General Transition-Based
10 |  Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech |  10 |  Dependency Parsing: Past, Present, and Future
Table 7: An example of recommendations demonstrating high topical relevancy of
recommended titles for the topic sentence in the case of qPCR. While SciNCL-
ts’s recommendations are more focused on dialog systems, as that is the main
topic of the citing article, qPCR’s recommendations are more about universal
dependencies, which are mentioned in the topic sentence. Title: Cross-lingual
Argumentation Mining: Machine Translation (and a bit of Projection) is All You
Need!
---
Year: 2018
Abstract: Argumentation mining (AM) requires the identification of complex
discourse structures and has lately been applied with success monolingually.
In this work, we show that the existing resources are, however, not adequate
for assessing cross-lingual AM, due to their heterogeneity or lack of
complexity. We therefore create suitable parallel corpora by (human and
machine) translating a popular AM dataset consisting of persuasive student
essays into German, French, Spanish, and Chinese. We then compare (i)
annotation projection and (ii) bilingual word embeddings based direct transfer
strategies for cross-lingual AM, finding that the former performs considerably
better and almost eliminates the loss from cross-lingual transfer. Moreover,
we find that annotation projection works equally well when using either costly
human or cheap machine translations. Our code and data are available at
http://github.com/UKPLab/coling2018-xling_argument_mining.
Paragraph: Cross-lingual Word Embeddings are the (modern) basis of the direct
transfer method. As with monolingual embeddings, there exists a veritable zoo
of different approaches, but they often perform very similarly in applications
(Upadhyay et al., 2016) and seemingly very different approaches are oftentimes
also equivalent on a theoretical level (Ruder et al., 2017) .
Rank |  SciNCL-ts |  Rank |  qPCR
1 |  A Two-Phase Approach Towards Identifying Argument Structure in Natural Language |  1 |  Inducing Multilingual Text Analysis Tools Using Bidirectional Recurrent Neural Networks
2 |  Argumentative Writing Support by means of Natural Language Processing |  2 |  Cross-Lingual Induction and Transfer of Verb Classes Based on Word Vector Space Specialisation
3 |  Baselines and test data for cross-lingual inference |  3 |  Cross-lingual sentiment transfer with limited resources
4 |  Using Discourse Structure Improves Machine Translation Evaluation |  4 |  Transferring Coreference Resolvers with Posterior Regularization
5 |  A Shared Task on Argumentation Mining in Newspaper Editorials |  5 |  Leveraging Monolingual Data for Crosslingual Compositional Word Representations
6 |  Neural End-to-End Learning for Computational Argumentation Mining |  6 |  Learning Crosslingual Word Embeddings without Bilingual Corpora
7 |  Exploiting Debate Portals for Semi-Supervised Argumentation Mining in User-Generated Web Discourse |  7 |  One-Shot Neural Cross-Lingual Transfer for Paradigm Completion
8 |  Discourse Structure in Machine Translation Evaluation |  8 |  Evaluating Unsupervised Dutch Word Embeddings as a Linguistic Resource
9 |  The Far Reach of Multiword Expressions in Educational Technology |  9 |  Bilingual Word Embeddings from Parallel and Non-parallel Corpora for Cross-Language Text Classification
10 |  Discourse-level features for statistical machine translation |  10 |  A Strong Baseline for Learning Cross-Lingual Word Embeddings from Sentence Alignments
Table 8: Another example of recommendations that demonstrate high topical
relevancy of recommended titles for the topic sentence in the case of qPCR.
SciNCL-ts’s recommendations dominantly cover topics of argumentation mining
and machine translation, while qPCR’s recommendations are about the cross-
lingual transfer, i.e., focused on the topic of the topic sentence.

###  A.1 Model Training Details

SciNCL, the model that we compare with and fine-tune, is a transformer-based
model whose architecture resembles BERT (Devlin et al., 2019). The model we
used contains 12 layers with hidden state embeddings of size 768. We fine-
tuned all the models on an NVIDIA GeForce RTX 3090 GPU with 24GB of RAM. We
used PyTorch version 1.12.1, Transformers version 4.20.1, CUDA 11.4, and Faiss
1.7.2. In total, all experiments and evaluations amounted to around 125 GPU
hours.