# Information retrieval for label noise document ranking by bag sampling and
group-wise loss

Chunyu Li<EMAIL_ADDRESS>tongji universitychina ,  Jiajia Ding
<EMAIL_ADDRESS>Edinburgh universitychina ,  xing hu
<EMAIL_ADDRESS>University College Londonchina and  fan wang
<EMAIL_ADDRESS>tongji universitychina

(2018)

###### Abstract.

Long Document retrieval (DR) has always been a tremendous challenge for
reading comprehension and information retrieval. The pre-training model has
achieved good results in the retrieval stage and Ranking for long documents in
recent years. However, there is still some crucial problem in long document
ranking, such as:data label noises, long document representations, negative
data Unbalanced sampling, etc. To eliminate the noise of labeled data and to
be able to sample the long documents in the search reasonably negatively, we
propose the bag sampling method and the group-wise Localized Constrastive
Esimation(LCE) method. We use the head middle tail passage for the long
document to encode the long document, and in the retrieval, stage Use dense
retrieval to generate the candidate’s data. The retrieval data is divided into
multiple bags at the ranking stage, and negative samples are selected in each
bag. After sampling, two losses are combined. The first loss is LCE. To fit
bag sampling well, after query and document are encoding, the global features
of each group are extracted by convolutional layer and max-pooling to improve
the model’s resistance to the impact of labeling noise, finally, calculate the
LCE group-wise loss. Notably, our model shows excellent performance on the MS
MARCO Long document ranking leaderboard.

Large-scale Information Retrieval, neural networks, Learning To Ranking

††copyright: acmcopyright††journalyear: 2018††doi:
10.1145/1122445.1122456††conference: ; ; ††price: 15.00††isbn:
978-1-4503-XXXX-X/18/06††ccs: Information systems Language models††ccs:
Learning to rank††ccs: Question and answer

##  1\. Introduction

In recent years, with the development of deep learning, more and more reading
comprehension data sets have been released. The more classic ones are SQuAD
and SQuAD2.0. Due to the harsh requirements of the industry on data sets,
large-scale reading comprehension data sets Start publishing. Reading
comprehension data set MS MARCO (Microsoft Machine Reading Comprehension)
based on large-scale real scene data. The data set is based on actual search
queries in the Bing search engine and Cortana intelligent assistant and
contains 1 million queries, 8 million documents, and 180,000 manually edited
answers.

Retrieving and sorting long documents in all data sets is a long document
retrieval and sorting task. The search and sorting of long documents are
different from the sorting task of short documents, and the difficulty brought
by it is as follows. 1) How to encode long documents? Long documents usually
contain long words. At the same time, the topic of long documents is difficult
to express, which brings challenges to search. 2) Whether it is a task of
super long documents or short documents sorting, how to better Reasonable
connection of search and sorting. Information retrieval is divided into full
ranking and two-stage ranking. The full ranking can effectively avoid the gap
caused by two stages.

Through experiments and related theoretical research, it can be concluded that
the pipeline using first retrieval + re-ranking can effectively improve the
ranking index, but in the actual data set and model, two problems will arise
(1): The data set usually has extreme labels -bias For example, through the
analysis of the MSMARCO data set and other data sets are shown in fig 1, some
of the top-ranked samples can answer the questions accurately, but there is
only one marked as a positive sample. Random sampling ng hard will also cause
noise Introduced. (2) Experiments have proved that by re-ranking the top data
obtained by dense retrieval, good results can be achieved. At the same time,
the problem faced is how to effectively inherit the first retrieval ranking
information when selecting training samples. The most common method is to
normalize the first retrieval score and add query-document as additional
features. This paper proposes a method of bag negative sample selection,
combined with group-wise loss calculation. On the one hand, it can alleviate
the positive and negative sampling boundary problem caused by label bias. At
the same time, the group-wise loss formed by the bag can inherit the sorting
of first retrieval information.

##  2\. Related Work

###  2.1. Full Ranking And Re-ranking Model

In the field of information retrieval, there have been many machine learning
ranking models (Learning to Rank) used to solve document ranking problems in
the early days, including LambdaRank(Burges, 2010), AdaRank(Xu and Li, 2007),
etc. These models rely on many manually constructed features. With the
popularity of deep learning technology in machine learning, researchers have
proposed many neural ranking models, such as DSSM (Shen et al., 2014), KNRM
(Dai et al., 2018), and so on. These models map the representation of the
problem and the document into a continuous vector space and then calculate
their similarity through a neural network, thus avoiding tedious manual
feature construction.

In recent years, pre-trained models have successfully brought significant
progress to the NLP field. An important goal of pre-training models such as
GPT(Radford et al., 2005), XLNET(Yang et al., 2019), and BERT(Devlin et al.,
2018) is to pre-train through a large amount of unlabeled data. Through large-
scale, The label data and the training model can be greatly improved by only a
small number of labeled data sets in the subsequent finetuning stage. BERT is
the first to start the pre-training model. It uses a two-way transformer
coding structure to perform MLM and NSP tasks to obtain contextual language
representations and sentence-pair representations. Applying BERT directly to
downstream IR tasks can achieve better results. In addition to now using BERT
for IR tasks, there have also been many pre-training models specifically for
IR tasks.

In order to coordinate efficiency and contextualization in IR, the author
proposes a ColBERT ranking model based on contextualized post-interaction.
ColBERT proposed a new post-interaction paradigm for estimating the
correlation between query q and document d. Compared with the existing bert-
based model, the results show that ColBERT is more than 170 times faster and
requires FLOPs /query is 14000 times less, and has little impact on quality,
and is better than all non-bert baselines.

BM25 has always been a strong baseline. However, for semantic recall and
sorting, the accuracy of BM25 is greatly reduced. DPR(Karpukhin et al., 2020)
is a search framework based on vector representation. The article proves that
retrieval based on dense vectors is superior to the traditional sparse
retrieval in answering open-domain questions. Experiments show that a simple
dual encoder method can achieve amazing results. More complex model frameworks
or similarity measurement functions may not provide additional gains. DPR
improves the results of retrieval performance and breaks records in multiple
open domain question answering benchmark tests

ANCE proposes Approximate nearest neighbor Negative Contrastive Learning(Xiong
et al., 2020), a learning mechanism that selects hard training negatives
globally from the entire corpus, using an asynchronously updated ANN index.
Demonstrate the effectiveness of ANCE on web search, question answering, and
in a commercial search environment.

###  2.2. Long Document Representation And Ranking

Although BERT is present, due to the design of the BERT model itself, it faces
some challenges when processing texts longer than 512 tokens. Therefore, it is
difficult to apply BERT to the document ranking task directly. Specifically,
it is mainly reflected in the following two aspects 1)During the training
phase, we still don’t know what form of content to provide to the model. The
critical issue is that the data set at the document level provides relevant
judgments for document ranking. However, the assessment of ”relevance” comes
from the document that contains ”relevant content”, but we cannot accurately
judge whether the correct answer is the subject of the document and how it is
distributed in the document. 2) In the inference phase, if the document is too
long to be fully inputted to BERT, how to deal with the long document becomes
the primary problem. At the same time, in the training phase and the inference
phase, different processing methods will produce different gaps.

Birch(Yilmaz et al., 2019) used data without length problems for training and
then migrated these correlation matching models to the target domain/task,
thereby completely avoiding training problems In the prediction phase,
estimating the relevance of a document is converted to evaluating the
relevance of a single sentence, and then the result scores are aggregated.
Birch needs to calculate the document relevance, inferring each sentence in
the document, and then top-n Combination of the score and original document
score (i.e., first stage score) Among them is the score based on the highest
scoring sentence of BERT, and the inference for each sentence is the same. In
other words, the relevance score of the document comes from the original
candidate document score (for example, from BM25) and the group of the
highest-scoring sentence in the document determined by the BERT model.

As an extension of BERT–MaxP(Dai and Callan, 2019), PCGM(Wu et al., 2020)
considers whether grading the paragraph-level relevance judgments during
training can get a more effective ranking. In response to this problem, the
author marked the paragraph-level cumulative income in the Chinese news
corpus, which is defined as the amount of relevant information that readers
will accumulate after reading the document to the specified paragraph. This
work uses natural paragraphs as paragraphs. Therefore, by definition, the
cumulative gain at the document level is the incremental gain at the highest
paragraph level. Based on these human annotations, the author found the
following conclusion: On average, highly relevant documents are longer than
other types of documents, whether in paragraphs or words. The higher the
cumulative gain at the document level, the more paragraphs the user needs to
read before reaching the cumulative growth at the document level through the
incremental increase at the paragraph level

Document-level sorting based on paragraph representation aggregation PARADE(Li
et al., 2020) is a series of models that can divide the long text into
multiple paragraphs and summarize each paragraph’s [CLS] representation.
Specifically, PARADE divides the long text into a fixed number of fixed-length
paragraphs. When the text contains fewer paragraphs, the paragraphs will be
filled and blocked during the presentation aggregation process. When the text
contains more paragraphs, the first and last paragraphs are always kept, and
the remaining paragraphs will be randomly sampled. Successive paragraphs
partially overlap to minimize the chance of separating relevant information
from its context.

###  2.3. Negative Sampling and Loss Function For Ranking Model

The data space for a retrieval task has various data distribution in degrees
of text/semantic/social matches, and it is important to design a training data
set for an embedding model to learn efficiently and effectively on such
space(Huang et al., 2020). Much research effort has addressed sampling
strategies for hard negative and hard positive. Positive sampling techniques
have also been proposed for sentence, audio(Logeswaran and Lee, 2018)(Oord et
al., 2018). In supervised learning, the hard negative sample can help guide a
model to correct mistakes more quickly(Oh Song et al., 2016), (Robinson et
al., 2020) introduce a simple hard negative sampling method overcoming an
apparent roadblock: negative mining in metric learning uses pair-wise
similarity information as a core component, while contrastive learning is
unsupervised. The loss function is an important module in the Ranking model,
and There is always a three-loss function in the Ranking model point-wise,
pair-wise and list-wise. Point-wise sorting takes each item in the training
set as a sample to obtain the rank function. The main solution is to convert
the classification problem into a classification or regression problem for a
single item. Pair-wise sorting uses two different items in the same query as a
sample. The main idea is to convert the rank problem into a binary
classification problem. Listwise sorting regards the entire item sequence as a
sample and is realized by directly optimizing the evaluation method of
information retrieval and defining the loss function. These loss functions
need according to different applications to choose.(Liu, 2011)

##  3\. THE PROPOSED MODEL

In this section, in response to the above problems, bag-sampling and group-
wise loss for long document ranking are proposed. The structure of the model
is shown in the figure. First, it introduces how to model the long document,
and secondly, for the selection of negative samples, use a bag. The method,
under the condition of dividing the difficulty level, minimizes the noise
caused by fewer labeled samples. Finally, to be able to inherit the scores
provided by the first retrieval and maximize the effect of bag sampling, use
LCE group-wise loss to calculate the loss score of each step group. The model
architecture, as shown in Fig 1 and Fig 2

Figure 1. bert question answer

bag sampling analysis

Figure 2. model architecture

bag sampling analysis

###  3.1. Long Document Model

This paper designs the following long text expressions based on the
statistical distribution of long text lengths and related answer
distributions. The maximum size of bert is 512. Through analysis, it can be
seen that some answers exist at the end and the center of the sentence.
Usually, the central theme of long documents is the title, the beginning of
the article, and the conclusion of the end. Inspired by maxp, this article
will design the head-tail passage coding, maxp is used to divide the long
document into four parts, this article divides the document into two necessary
parts of the head and tail, and the remaining two Passage Randomly select two
passages of 512 lengths in the center, shown as in fig 2. The final results in
equation(1) (2)

(1) $Score_{document}=[S_{head},S_{middle-1},S_{middle-2},S_{tail}]$

(2) $Score(passage_{i})=cls(BERT(concat(q,passage_{i})))$

###  3.2. BAG Sampling for Reranking

Typically, when deep learning is sampling, it uses the same batch of data for
the sample and calculates the gradient. That is, for the same query q, the
recall strategy is used to randomly select the document d+ related to q from
the recalled document $D^{r}etrieval$, and more Unrelated documents d-, as
shown in equation.

(3) $\begin{split}Batch={(q,d^{+},d^{-})|q\in rand(Q^{N}),d^{+},d^{-}\in
rand(D^{retrieval})}\end{split}$

ANCE uses global negative sampling to dynamically sample negative samples and
sample some hard negative samples simultaneously. It is challenging to select
negative samples, and for incompletely labeled data sets (such as MSMARCO),
the correct labeled answer for each question is only One. We selected a
question as an example as shown in Fig 3 From the clustering graph, it can be
concluded that the query document after the bert encoder will show apparent
category effects. Since there is only one correctly labeled answer, there will
be documents and topics similar to the correct answer near the right answer.
It is more difficult to distinguish, and in the case of a small labeled data
set, the documents with similar valid documents are likely to be accurate but
unlabeled data. This article will carry out bag sampling for the interval
section, which can effectively distinguish the samples at the stage The set is
N ($d_{1},d_{2},d_{3},d_{4},d_{5}.....d_{N}$ ), the sample set is divided into
M bags, (1, 2, 3, 4, 5….M), randomly select i samples in each pack for
calculation:

(4)
$Batch_{bs}={(q,B(d^{+}_{1},d^{-}_{5}),B(d^{-}_{6},d^{-}_{8})..B(d^{-}_{N-1},d^{-}_{N}))}$

(5) $Batch_{bs}={q\in rand(Q^{N}),B\in M,d^{+},d^{-}\in rand(B_{N})}$

Figure 3. bag sampling analysis

bag sampling analysis

###  3.3. LCE for Group Wise Loss

####  3.3.1. Local Contrastive Loss

Inspired by LCE, this article uses the first stage recall topN data set, uses
the ANCE method, and uses local negative sampling to calculate the correlation
between a single accurate sample and the document. The specific sampling
method is as shown in section 3.2 to select the bag sampling data. After
selecting all positive and negative samples, extract the group of positive
samples to the top, and calculate contractual loss, as shown in equation 5

(6) $Score(q,d)=V^{T}_{p}CLS(BERT(concat(q,d)))$

with which we define the contrastive loss for each query q as:

(7) $\boldsymbol{L_{q,d}}=-\log\frac{exp(score(q,d^{+}_{q}))}{\sum_{d\in
G_{q}}exp(score(q,d^{-}_{q}))}$

####  3.3.2. Local Contrastive Group Wise Loss

To make bag negative sampling more useful in reranking, this article proposes
LCGW(local contraste group wise ) loss for the first time. After encoding by
bert, the expression of each group is obtained according to Fig 2.
$[(r(q,d+),r(q,d-),r(q,d-)),(r(q,d-),r(q,d-),r(q,d-))]$, To obtain the global
characteristics of each group, we use a layer of convolutional network to
extract the overall signal characteristics of each group, then max-pool
selects the strong high level matching signal characteristics, and finally
gets Score of each group. Through group high-level feature extraction, we can
get that the feature representations in each group are similar. This kind of
global feature representation helps to remove the influence caused by
insufficient labeling.

(8) $Input_{Group-i}=concat(BERT(q,d_{1})...BERT(q,d_{i}))$

(9) $\boldsymbol{Encoder_{Group-i}}=Conv(Input_{Group-i})$

##  4\. EXPERIMENTS

Our main training and inference dependency is PyTorch, HuggingFace
transformer,in the initial recall stage, we use ANCE to recall the candidate
set

###  4.1. Document collection and query Datasets

There is insufficient labeling, and large-scale, long document data sets are
relatively scarce. MSMRACO is a pretty typical data set, with 300W long
documents and 30w training annotation queries. Due to the massive amount of
data, it is easy to appear in the literature above. The labeling is not
sufficient, and labeling noise is generated. As an indicator, we select the
commonly used MRR as the evaluation indicator of the test data. In the initial
stage of training, the 30w query data is trained and recalled through ANCE,
and the number of recalls is top100, and the sampling method proposed in this
article is used for sampling.

###  4.2. Parameter For Model and Results

We use topk k=100 retrieval document for Re-ranking, group size 10 and each
bag size s 10, sample 4 documents from bag, the results as shown in Table 1,
our model achieves the best results on the dev dataset, and it drops slightly
on the test

Table 1. RESULTS ANCE BAG SAMPLING GROUP-WISE LOSS(ANCE BS+GL) LEADER BOARDER MSMARCO-Document-Ranking 2021/04/27 Model | MRR@100(dev) | MRR@100(eval)
---|---|---
DML | 0.470 | 0.416
LCEB Model | 0.479 | 0.419
PROP step400K(v2) | 0.479 | 0.423
ANCE + LongP (ensemble) | 0.481 | 0.420
ANCE BS+GL | 0.489 | 0.421

##  5\. CONCLUSION

This paper proposes the Long document ranking model, which use max-passage and
bag sampling method, to avoid hard negative and positive affect, the model
adopt group-wise loss . Of course, there are still some shortcomings. We will
test the model’s effect on more datasets in the future and explore the
influence of more sampling mechanisms.

##  6\. Acknowledgments

## References

  * (1)
  * Burges (2010) Christopher JC Burges. 2010\.  From ranknet to lambdarank to lambdamart: An overview.  _Learning_ 11, 23-581 (2010), 81.
  * Dai and Callan (2019) Zhuyun Dai and Jamie Callan. 2019.  Deeper text understanding for IR with contextual neural language modeling. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 985–988.
  * Dai et al. (2018) Zhuyun Dai, Chenyan Xiong, Jamie Callan, and Zhiyuan Liu. 2018\.  Convolutional neural networks for soft-matching n-grams in ad-hoc search. In _Proceedings of the eleventh ACM international conference on web search and data mining_. 126–134.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Huang et al. (2020) Jui-Ting Huang, Ashish Sharma, Shuying Sun, Li Xia, David Zhang, Philip Pronin, Janani Padmanabhan, Giuseppe Ottaviano, and Linjun Yang. 2020.  Embedding-based retrieval in facebook search. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 2553–2561.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020\.  Dense passage retrieval for open-domain question answering.  _arXiv preprint arXiv:2004.04906_ (2020).
  * Li et al. (2020) Canjia Li, Andrew Yates, Sean MacAvaney, Ben He, and Yingfei Sun. 2020.  PARADE: Passage representation aggregation for document reranking.  _arXiv preprint arXiv:2008.09093_ (2020).
  * Liu (2011) Tie-Yan Liu. 2011\.  Learning to rank for information retrieval.  (2011).
  * Logeswaran and Lee (2018) Lajanugen Logeswaran and Honglak Lee. 2018.  An efficient framework for learning sentence representations.  _arXiv preprint arXiv:1803.02893_ (2018).
  * Oh Song et al. (2016) Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. 2016\.  Deep metric learning via lifted structured feature embedding. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 4004–4012.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ (2018).
  * Radford et al. (2005) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2005.  Improving language understanding by generative pre-training.  (2005).
  * Robinson et al. (2020) Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. 2020.  Contrastive learning with hard negative samples.  _arXiv preprint arXiv:2010.04592_ (2020).
  * Shen et al. (2014) Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. 2014.  Learning semantic representations using convolutional neural networks for web search. In _Proceedings of the 23rd international conference on world wide web_. 373–374.
  * Wu et al. (2020) Zhijing Wu, Jiaxin Mao, Yiqun Liu, Jingtao Zhan, Yukun Zheng, Min Zhang, and Shaoping Ma. 2020.  Leveraging passage-level cumulative gain for document ranking. In _Proceedings of The Web Conference 2020_. 2421–2431.
  * Xiong et al. (2020) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020\.  Approximate nearest neighbor negative contrastive learning for dense text retrieval.  _arXiv preprint arXiv:2007.00808_ (2020).
  * Xu and Li (2007) Jun Xu and Hang Li. 2007\.  Adarank: a boosting algorithm for information retrieval. In _Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval_. 391–398.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019\.  XLNet: Generalized Autoregressive Pretraining for Language Understanding.  _Advances in Neural Information Processing Systems_ 32 (2019), 5753–5763.
  * Yilmaz et al. (2019) Zeynep Akkalyoncu Yilmaz, Wei Yang, Haotian Zhang, and Jimmy Lin. 2019.  Cross-domain modeling of sentence-level evidence for document retrieval. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. 3481–3487.

## References

  * (1)
  * Burges (2010) Christopher JC Burges. 2010\.  From ranknet to lambdarank to lambdamart: An overview.  _Learning_ 11, 23-581 (2010), 81.
  * Dai and Callan (2019) Zhuyun Dai and Jamie Callan. 2019.  Deeper text understanding for IR with contextual neural language modeling. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 985–988.
  * Dai et al. (2018) Zhuyun Dai, Chenyan Xiong, Jamie Callan, and Zhiyuan Liu. 2018\.  Convolutional neural networks for soft-matching n-grams in ad-hoc search. In _Proceedings of the eleventh ACM international conference on web search and data mining_. 126–134.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Huang et al. (2020) Jui-Ting Huang, Ashish Sharma, Shuying Sun, Li Xia, David Zhang, Philip Pronin, Janani Padmanabhan, Giuseppe Ottaviano, and Linjun Yang. 2020.  Embedding-based retrieval in facebook search. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 2553–2561.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020\.  Dense passage retrieval for open-domain question answering.  _arXiv preprint arXiv:2004.04906_ (2020).
  * Li et al. (2020) Canjia Li, Andrew Yates, Sean MacAvaney, Ben He, and Yingfei Sun. 2020.  PARADE: Passage representation aggregation for document reranking.  _arXiv preprint arXiv:2008.09093_ (2020).
  * Liu (2011) Tie-Yan Liu. 2011\.  Learning to rank for information retrieval.  (2011).
  * Logeswaran and Lee (2018) Lajanugen Logeswaran and Honglak Lee. 2018.  An efficient framework for learning sentence representations.  _arXiv preprint arXiv:1803.02893_ (2018).
  * Oh Song et al. (2016) Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. 2016\.  Deep metric learning via lifted structured feature embedding. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 4004–4012.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ (2018).
  * Radford et al. (2005) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2005.  Improving language understanding by generative pre-training.  (2005).
  * Robinson et al. (2020) Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. 2020.  Contrastive learning with hard negative samples.  _arXiv preprint arXiv:2010.04592_ (2020).
  * Shen et al. (2014) Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. 2014.  Learning semantic representations using convolutional neural networks for web search. In _Proceedings of the 23rd international conference on world wide web_. 373–374.
  * Wu et al. (2020) Zhijing Wu, Jiaxin Mao, Yiqun Liu, Jingtao Zhan, Yukun Zheng, Min Zhang, and Shaoping Ma. 2020.  Leveraging passage-level cumulative gain for document ranking. In _Proceedings of The Web Conference 2020_. 2421–2431.
  * Xiong et al. (2020) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020\.  Approximate nearest neighbor negative contrastive learning for dense text retrieval.  _arXiv preprint arXiv:2007.00808_ (2020).
  * Xu and Li (2007) Jun Xu and Hang Li. 2007\.  Adarank: a boosting algorithm for information retrieval. In _Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval_. 391–398.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019\.  XLNet: Generalized Autoregressive Pretraining for Language Understanding.  _Advances in Neural Information Processing Systems_ 32 (2019), 5753–5763.
  * Yilmaz et al. (2019) Zeynep Akkalyoncu Yilmaz, Wei Yang, Haotian Zhang, and Jimmy Lin. 2019.  Cross-domain modeling of sentence-level evidence for document retrieval. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. 3481–3487.


# Improving Motion Forecasting for Autonomous Driving with the Cycle
Consistency Loss

Titas Chakraborty
Carnegie Mellon University
<EMAIL_ADDRESS>
Akshay Bhagat
Motional
<EMAIL_ADDRESS>
Henggang Cui
Motional
<EMAIL_ADDRESS>
Work done while interning at Motional.

###### Abstract

Robust motion forecasting of the dynamic scene is a critical component of an
autonomous vehicle. It is a challenging problem due to the heterogeneity in
the scene and the inherent uncertainties in the problem. To improve the
accuracy of motion forecasting, in this work, we identify a new consistency
constraint in this task, that is _an agent’s future trajectory should be
coherent with its history observations and visa versa_. To leverage this
property, we propose a novel cycle consistency training scheme and define a
novel cycle loss to encourage this consistency. In particular, we reverse the
predicted future trajectory backward in time and feed it back into the
prediction model to predict the history and compute the loss as an additional
cycle loss term. Through our experiments on the Argoverse dataset, we
demonstrate that cycle loss can improve the performance of competitive motion
forecasting models.

##  1 Introduction

Motion forecasting in autonomous vehicles is a challenging research problem to
solve for the autonomous driving industry. Motion forecasting involves
predicting the future trajectories of the agents in a scene given their
history trajectories and the scene context, usually in the form of a lane
graph. The problem is multi-modal since, given a history, there can be
multiple possible futures. For example, at an intersection, an agent can take
different possible maneuvers (e.g., straight, right turn, left turn) and
follow different speed profiles.

Motion forecasting approaches are highly diverse in their input
representations. Several methods, such as [4, 1], found success representing
the input scene as an image and using convolutional neural networks to
generate a scene representation. More recently, vectorized representations
used in [17, 7, 11, 10] have found success due to computationally efficient
sparse representation and ability to model long-range dependencies. [9, 4, 5,
6, 15] represent lanes as nodes in a graph to make use of the connectivity
information. There are several output representations and loss functions in
use as well. [14, 9] predict an unconstrained regressed output and use a
simple regression loss. [4, 5, 6] predict an unconstrained heatmap with a
cross entropy loss, sample the trajectory endpoint from the heatmap, and they
complete the whole trajectory conditioned on the endpoint.

There are a number of consistencies and symmetries inherent to the motion
forecasting problem. Recently, there has been an effort to integrate this
information, either as an explicit loss or in the model structure to improve
the model performance. Certain symmetries such as rotation, translation and
scale symmetries have been incorporated into the loss function using data
augmentation in various methods [9, 14, 15]. [17] incorporates rotation and
translation invariance explicitly into the model structure and input
representation. [14] enforces temporal and spatial consistency constraints
into the loss function, which is shown to improve prediction accuracy.

Figure 1: Cycle prediction. In the backward prediction pass, we reverse the
trajectories as well as the direction of the road graph, and let the model
predict backward in time.

In this work, we propose a novel consistency-based loss called cycle loss,
which is illustrated in Figure 1. The key motivation of our method is that an
agent’s future trajectory should be coherent with its past observations, and
this coherency should still hold even if we reverse the sequence backward in
time. Motivated by this observation, we propose to add an auxiliary task and
loss term in the existing training scheme of the motion forecasting model.
After the model predicts the future trajectory using the history input, we
reverse the predicted future trajectory backward in time and reverse the
direction of the road graph, and we feed them back into the prediction model
to let it predict the history. We compute the loss of this auxiliary task as
an additional cycle loss term. Similar to the temporal consistency loss
proposed in [14], our cycle loss method is generic and can be applied to any
motion forecasting model.

We summarize our contributions as follows:

  1. 1.

We propose cycle loss, a novel training scheme and consistency loss for motion
forecasting. This loss explicitly ensures that the future trajectories
predicted by the model are coherent with the history observations. We also
introduce the concept of ground truth trajectory mixing, which is necessary
for cycle loss to work.

  2. 2.

We conduct extensive experimentation on the Argoverse dataset [2]. We justify
the design choices needed for cycle loss to work through ablations and
demonstrate that cycle loss can improve the performance of competitive motion
forecasting models.

##  2 Related Work

Attention-based motion forecasting models: Attention is used widely in motion
forecasting models to fuse diverse features, especially agent and lane
features. [9] proposed a novel form of graph convolution to aggregate short
and long-range lane features and used self and cross attention to fuse agent
and lane features. [4, 5, 6] also use self and cross-attention between agent
and lane features. More recently, several transformer-based papers such as
[10, 7, 11] incorporate temporal attention as well. Some works, such as [17],
introduce a novel formulation of attention to enforce spatial and rotational
invariance. Other works, such as [15], model the scene using two graph layers
and use attention to fuse embeddings of the two layers. In our experiments, we
applied our proposed cycle loss method to two attention-based motion
forecasting models, GOHOME [5] and Autobots [7]. We selected them because of
their popularity and competitive performance.

Consistency-Based Losses: There are several inherent constraints in the motion
forecasting problem that have been discussed in the literature. [14]
introduced temporal and spatial consistency losses. Temporal consistency
enforces that inputs shifted by a small time interval produce similar output
trajectories. Spatial consistency enforces that inputs perturbed by a small
amount of noise produce similar output trajectories. The loss introduced by
[13] is the closest to our work in motion prediction of autonomous vehicles.
They pass learning embeddings for the future timesteps through a temporal flow
network to reconstruct the input. However, since they use a separate model for
reconstruction, it is not truly a consistency loss; but rather a method to
enrich the learned embeddings. In our work, we use the same model to predict
forward and backward in time, making our method a consistency loss on the
model. Another difference between our cycle loss method and [13] is that we
use the predicted trajectories to do backward prediction, while [13] uses
feature embeddings. [12] uses a similar approach to ours for human trajectory
prediction. They pretrain two models for forward and backward prediction and
refine the predictions for both models by jointly training both models with
cycle loss. This makes their training procedure lengthy as they have to train
three times. Our approach differs since we use the same model for forward and
backward prediction making our training end-to-end and therefore much faster
than [12]. This also makes cycle loss a true consistency loss on the model
which is not the case for [12]. The element of ground truth trajectory mixing
is crucial for our method to work.

##  3 Methods

###  3.1 Problem Statement

The general motion forecasting problem involves predicting the future
trajectories of agents given the history trajectories and map context (usually
represented as a lane graph). Consider that the number of agents in the scene
is $A$, of which $P$ are target agents whose trajectories need to be predicted
by the model, and $B$ are background agents which are used to provide scene
context for the predictions of the target agents. Let the history length be
$H$ and the future length to be predicted be $F$. We are given the history
trajectory of the target agent $i$ as
$\textbf{x}^{i}=\\{x_{1}^{i},\dots,x_{H}^{i}\\}$ and the history trajectory of
background agent $i$ as $\textbf{b}^{i}=\\{b_{1}^{i},\dots,b_{H}^{i}\\}$ where
each element is a 2D array with $x$ and $y$ coordinates. The map context
$\mathcal{M}$ is represented as a directional lane graph, where each node in
the graph is a lane segment with a start point and an end point. We denote the
ground-truth future positions of target agent $i$ as
$\textbf{y}^{i}=\\{y_{1}^{i},\dots,y_{F}^{i}\\}$ and background agent $i$ as
$\textbf{b}^{i}_{f}=\\{y_{f1}^{i},\dots,y_{fF}^{i}\\}$. Our task is to predict
$K$ future trajectories for each target agent, where the $k$-th future
trajectory is denoted as
$\textbf{y}^{pik}=\\{y_{1}^{pik},\dots,y_{F}^{pik}\\}$.

###  3.2 Cycle Consistency Training

Figure 2: Cycle consistency training architecture

The overall training scheme with cycle loss is illustrated in Figure 2. We
conduct two passes of the model, a normal forward prediction pass that is the
same as the regular motion forecasting training and a backward prediction pass
that reverses the trajectories as well as the lane graph backward in time.

In the forward prediction pass, we use the input agent histories, x and b,
along with the map context $\mathcal{M}$, to predict the target agent future
trajectories $\textbf{y}^{pk}$. We use the standard winner-takes-all
multimodal prediction loss [3] as the forward loss, which has a mode
classification term and a trajectory regression term. The trajectory
regression term is computed on the prediction trajectory that is closest to
the ground-truth measured by the final displacement error (FDE).

In the backward prediction pass, we reverse the future trajectories of the
agents as well as the directions of the road graph, and feed them back to the
model to predict the history trajectory. When the model predicts $K>1$ future
trajectories for the target agent, we pick the trajectory that is closest to
the ground-truth as measured by the final displacement error (FDE), denoted as
$\textbf{y}^{p}$. And we denote the reversed future trajectory as
$\textbf{y}^{rp}$ where $\textbf{y}^{rpi}=\\{y_{F}^{pi},\dots,y_{1}^{pi}\\}$,
which is the backward pass input for the target agent. For the background
agents that the model doesn’t predict for, we use their ground-truth future
trajectories to reverse and get $\textbf{b}^{r}_{f}$ where
$\textbf{b}^{ri}_{f}=\\{y_{fF}^{i},\dots,y_{f1}^{i}\\}$. We also reverse the
map context to obtain $\mathcal{M}^{r}$. For the map context, we reverse the
connection directions of the lane graph, and we reverse the directions of all
the lane segments in the graph.

In the simpler case when $F=H$, we use $\textbf{x}_{rev}=\textbf{y}^{rp}$ and
$\textbf{b}_{rev}=\textbf{b}_{f}^{r}$, along with the reversed map context
$\mathcal{M}^{r}$, to predict backward in time the target agent history
trajectories $\textbf{y}^{pk}_{rev}$. The predicted history trajectory and the
original agent history x are then used to compute our cycle loss. Similar to
the forward pass, we use the winner-takes-all approach to compute the cycle
loss. We pick the history trajectory that is closest to the original history
as measured by the final displacement error (FDE) to calculate the loss.

$Cycle\;Loss\;=\frac{1}{HP}\sum_{i=1}^{i=P}\min_{k\in\\{1,\dots
K\\}}\sum_{j=1}^{j=H}||y^{pik}_{rev,H-j}-x_{j}^{i}||_{2}$

###  3.3 Trajectory Truncation or Extension for $F\neq H$

In many datasets (such as Argoverse), the length of the future prediction
trajectory $F$ is longer than the history length $H$, that is, $F>H$. For
those datasets, we need to truncate the future trajectory to length $H$ when
passing as the input to the backward pass. We provide ablations for different
choices of truncation parameters in Section 4.4.1.

Similarly, in datasets where $F<H$, we need to extend the future trajectory
with a part of the agent history to make it length $H$.

###  3.4 Ground Truth Trajectory Mixing

If we use purely the predicted target agent trajectories as the input to the
backward pass, there is a possibility that this encourages the model to
predict future trajectories that are easier to regress backward but are less
accurate. For example, the model might predict purely straight line modes
instead of accounting for right or left turns since straight line modes have
only velocity uncertainty as opposed to other modes that have maneuver
uncertainty as well.

One straightforward way to mitigate this issue is to reduce the weight of the
cycle loss term, but this will limit the improvement offered by cycle loss. In
this work, we find a better approach to addressing this issue is to mix the
predicted future trajectories with the ground-truth future trajectories with a
probability $p$ to generate a mixed trajectory, and use the mixed trajectory
as input to the backward pass. This ensures that if the predicted future
trajectory is too far from the ground truth, the mixed trajectory will be
completely infeasible and will result in poor backward predictions and a high
cycle loss.

Mathematically, as we described in Section 3.2, after reversing the predicted
target agent future trajectories, we obtain
$\textbf{y}^{rp}\in\mathcal{R}^{P\times F\times 2}$. We also flip the ground
truth target agent future trajectories to obtain $\textbf{y}^{r}$ where
$\textbf{y}^{ri}=\\{y_{F}^{i},\dots,y_{1}^{i}\\}$. Here,
$\textbf{y}^{r}\in\mathcal{R}^{P\times F\times 2}$. We generate a binary mask
which we denote as $GTM\in\mathcal{R}^{P\times F\times 2}$ where each element
of $GTM$ is generated independently and is $1$ with probability $p$ and $0$
with probability $1-p$. Thus, with ground truth trajectory mixing,

$\textbf{x}_{rev}=GTM\otimes\textbf{y}^{rp}+(1-GTM)\otimes\textbf{y}^{r}$

###  3.5 Implementation Details

####  3.5.1 GOHOME Implementation

Figure 3: Block diagram for our GOHOME-Mod model

We implemented our cycle consistency training scheme on GOHOME [5], which is a
goal-based motion forecasting model using attentions. It first predicts the
endpoint of the trajectory and then completes the whole trajectory conditioned
on the goal. Since the original GOHOME model code is not open-sourced, we
implemented our own variant of the GOHOME model, and we refer to it as GOHOME-
Mod for the rest of the paper.

The architecture of our GOHOME-Mod model is illustrated in Figure 3. We use 1D
CNN + GRU as our encoder in both agent and lane encoders to encode agent
trajectories and lane centerlines. We use cross-attention block Agents2Lanes
to generate agent-aware lane features and use cross-attention block
Lanes2Agents to generate lane-aware agent features. We then apply self-
attention block Agents2Agents to produce inter-aware agent features and use a
$3$-layer MLP decoder to predict the trajectory endpoints of the target
agents. Finally, we use a trajectory regressor module to complete the
trajectory conditioned on the predicted endpoint and the agent feature
embeddings. Similar to the other goal-based prediction approaches such as [4,
5, 6, 14], we train the trajectory regressor conditioned on the ground truth
trajectory endpoints. For all attention blocks, we use multi-head attention
with $8$ attention heads.

We add our proposed cycle loss as an additional loss term in addition to its
original forward losses of GOHOME-Mod. We refer to the model with cycle
consistency loss as GOHOME-Mod + CL.

####  3.5.2 Autobots Implementation

We also applied our cycle loss method to the official open-source
implementation of Autobots [7], which is a transformer-based motion prediction
model. We used the single agent prediction version of the model (Autobots-Ego)
since it has better performance on the Argoverse dataset. Same as the GOHOME-
Mod + CL model, we added the cycle loss as an additional loss term in addition
to its original forward losses. We refer to the model with cycle consistency
loss as Autobots + CL.

##  4 Evaluation

###  4.1 Datasets

We use the Argoverse 1.1 dataset [2], which is a widely used dataset for
motion forecasting to evaluate our method. The Argoverse 1.1 dataset consists
of real-world driving scenarios with 205,942 training samples, 39,472
validation samples, and 78,143 test samples. For training and validation
samples, the dataset provides a $2$ second history and a $3$ second future
trajectories sampled at $10$ Hz. For the test samples, only the $2$ second
history is provided. The dataset is collected from two cities, Miami and
Pittsburgh. In addition to the agent trajectories, the map information is also
provided in the form of a lane graph. It represents lanes as lane centerlines,
which is a sequence of points, with lane attributes including turn direction,
presence of traffic control, and presence of intersections. To construct the
reversed lane features required by cycle consistency, we reverse the turn
directions and reverse the order of points in each centerline. Argoverse is a
single-agent dataset that only requires the models to predict the trajectories
for one single target agent in each frame. Therefore, using our terminology
described in Section 3.1, for the Argoverse dataset, $H=20$, $F=30$ and $P=1$.

We use the minADE, minFDE, MR, and DAC metrics, which are standard metrics
used by the Argoverse leaderboard [2]. minFDEK is the minimum displacement
between the last waypoint of the ground truth trajectory and the last
predicted waypoint of top K predicted trajectories. minADEK is the minimum
average displacement between all waypoints of the ground truth trajectory and
all predicted waypoints of top K predicted trajectories. MR-K represents the
rate at which minFDEK exceeds $2$ meters over the entire dataset. DACK
represents the percentage of the top K predicted trajectories that do not
leave the drivable area at any point. For minADE, minFDE, and MR, lower
numbers are better. For DAC, higher numbers are better.

###  4.2 Experimental Details

For the GOHOME-Mod model, we train the model for $200$ epochs with an initial
learning rate of $10^{-3}$ decayed by $0.5$ every $60$ epochs.

For the Autobots model, we train the model for $150$ epochs with an initial
learning rate of $7.5\times 10^{-4}$ decayed by $0.5$ every $20$ epochs.

Unless otherwise specified, for both models, we use the first $20$ predicted
future timesteps to reverse and pass them into the model in the backward
prediction pass, and we use a probability of $p=0.5$ for ground truth
trajectory mixing. We set the weight of cycle loss to $2$ for GOHOME-Mod and
to $1$ for Autobots.

###  4.3 Argoverse Test Results

Model | minFDE6 | minADE6 | DAC6 | MR6
---|---|---|---|---
THOMAS [6] | 1.4388 | 0.9423 | 0.9781 | 0.1038
TNT [16] | 1.4457 | 0.9097 | 0.9889 | 0.1656
GOHOME [5] | 1.4503 | 0.9425 | 0.9811 | 0.1048
GOHOME-Mod (ours) | 1.4368 | 0.9426 | 0.9787 | 0.1763
GOHOME-Mod + CL (ours) | 1.3896 | 0.8949 | 0.9833 | 0.1682
Table 1: Results on Argoverse Test Set. The baseline numbers are from the
leaderboard.

Table 1 shows the results of our models on the Argoverse test set compared to
some standard baselines, including THOMAS [6], TNT [16], and the original
GOHOME [5] model. The numbers for the baseline models are directly copied from
the leaderboard. We can see that, by adding cycle loss, our GOHOME-Mod + CL
model outperforms all metrics of its corresponding baseline GOHOME-Mod and
improves minFDE6 by a significant amount. This result demonstrates the
effectiveness of our cycle loss method. The result also shows our GOHOME-Mod
implementation archives similar performance as the reported numbers of the
original GOHOME model on the Argoverse leaderboard, indicating that our re-
implementation is a competitive model.

###  4.4 Ablation Studies

We performed the following ablation studies on the Argoverse validation set.

####  4.4.1 Importance of Predicted and Ground-Truth Positions

Model | Future input | History target | ValFDE6 | ValADE6
---|---|---|---|---
GOHOME-Mod | - | - | 1.163 | 0.767
GOHOME-Mod + CL | 30-50 | 1-30 | 1.146 | 0.7543
GOHOME-Mod + CL | 20-40 | 1-20 | 1.104 | 0.7369
Table 2: Ablation study for the predicted and matched positions used for cycle
loss. A position of a-b indicates the positions from the ath timestep to the
bth timestep are used for calculating cycle loss. Trajectories in Argoverse
$1$ have a total of $50$ timesteps (20 history + 30 future).

In Argoverse, the length of the future trajectory is longer than the length of
the history ($H=20$ and $F=30$). As a result, in the backward prediction pass,
we need to pick which part of the future trajectory we feed as the backward
input. In this ablation study, we studied different choices of this parameter,
as summarized in Table 2.

The result shows that the model performs the best when we feed in the first
$20$ future waypoints as the backward pass input and use them to predict the
$20$ history waypoints. The losses for the remaining part of the backward
prediction trajectories are masked out. We believe the reason why this setting
yields the best performance is because the uncertainty in the last $20$
positions is larger compared to the uncertainty in the first $20$ positions,
making it tough for the model to obey cycle consistency.

####  4.4.2 Importance of Ground Truth Trajectory Mixing

Model | Mixing probability ($p$) | ValFDE6
---|---|---
GOHOME-Mod | - | 1.163
GOHOME-Mod + CL | 0 (All Ground Truth Positions) | 1.133
GOHOME-Mod + CL | 1 (All Predicted Positions) | 1.184
GOHOME-Mod + CL | 0.5 | 1.104
Table 3: Ablation study for the mixing probability of ground truth positions
and predicted positions. The first $20$ predicted waypoints are used to feed
into the model, and the $20$ input positions are used for calculating cycle
loss, as is found optimal in Section 4.4.1

In Table 3, we studied different parameters of ground-truth mixing (described
in Section 3.4). As we can see from the results, mixing ground truth future
trajectories with the predicted future trajectories gives us the best FDE on
the validation set. An interesting fact to note is that, when we use all
ground truth futures as the backward prediction inputs, it can also reduce the
FDE by an amount of $3$ cm over the baseline. Since using all ground truth
futures essentially corresponds to augmenting the dataset using the time-
reversed data, the result suggests that time-reversing can also be used as an
augmentation strategy to enhance the training of the motion forecasting
models.

We believe the main reason why time flipping augmentation helps in motion
forecasting is that it reduces the imbalances of complex maneuvers in the
dataset. For example, if the dataset has more left turns than right turns,
time flipping augmentation can balance the number of those maneuvers.
Similarly, a car might increase its speed or decrease its speed in the future
trajectory. If the number of instances of the car speeding up are higher than
the car slowing down, that creates an imbalance which time flipping
augmentation can again correct.

Meanwhile, when we use all predicted futures, its performance is worse than
not applying cycle loss at all. This highlights the importance of ground truth
trajectory mixing, which ensures that the predicted trajectory does not stray
off the ground truth in a bid to minimize the cycle loss.

###  4.5 Additional Results on Autobots

In addition to GOHOME-Mod, we also implemented cycle loss on Autobots [7]. The
design choices made were the same as in the GOHOME-Mod model, except that the
weight of cycle loss was fixed at $1$ based on our parameter search.

Model | minFDE6
---|---
Autobots [7] | 1.114
Autobots + CL | 1.084
Table 4: Autobots Results on Argoverse Validation Set

As the results in Table 4 show, the addition of cycle loss improves the
prediction performance of the open-sourced Autobots model as well. This result
demonstrates that our cycle loss method is generic.

##  5 Limitations

There are several limitations to our proposed consistency loss method. First,
similar to the temporal consistency loss proposed in [14], the cycle loss
method incurs overhead at the training time, as it requires two passes of the
model during training, nearly doubling the training time. Second, some of the
cycle loss parameters depend on the model architecture and dataset. For
example, we find the optimal cycle loss weight is different in GOHOME-Mod and
in Autobots. Third, cycle loss requires the model to be end-to-end
differentiable. Some models [5, 6, 4, 8] rely on sampling a probability
distribution as an intermediate step, and more sophisticated designs will be
needed in order to apply cycle loss to those models.

##  6 Conclusion

In this work, we propose cycle loss, a novel consistency-based training scheme
and loss for motion forecasting. We demonstrated its effectiveness on two
competitive motion forecasting models on the Argoverse dataset. The result
shows that cycle loss is able to improve the prediction performance of those
models by a significant margin.

## References

  * Chai et al. [2020] Yuning Chai, Benjamin Sapp, Mayank Bansal, and Dragomir Anguelov.  Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction.  In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, _Proceedings of the Conference on Robot Learning_ , volume 100 of _Proceedings of Machine Learning Research_ , pages 86–99. PMLR, 30 Oct–01 Nov 2020.
  * Chang et al. [2019] Ming-Fang Chang, John W Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, and James Hays.  Argoverse: 3d tracking and forecasting with rich maps.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Cui et al. [2019] Henggang Cui, Vladan Radosavljevic, Fang-Chieh Chou, Tsung-Han Lin, Thi Nguyen, Tzu-Kuo Huang, Jeff Schneider, and Nemanja Djuric.  Multimodal trajectory predictions for autonomous driving using deep convolutional networks.  In _2019 International Conference on Robotics and Automation (ICRA)_ , pages 2090–2096. IEEE, 2019.
  * Gilles et al. [2021] Thomas Gilles, Stefano Sabatini, Dzmitry Tsishkou, Bogdan Stanciulescu, and Fabien Moutarde.  Home: Heatmap output for future motion estimation.  In _2021 IEEE International Intelligent Transportation Systems Conference (ITSC)_ , pages 500–507, 2021.  doi: 10.1109/ITSC48978.2021.9564944.
  * Gilles et al. [2022a] Thomas Gilles, Stefano Sabatini, Dzmitry Tsishkou, Bogdan Stanciulescu, and Fabien Moutarde.  Gohome: Graph-oriented heatmap output for future motion estimation.  In _2022 International Conference on Robotics and Automation (ICRA)_ , pages 9107–9114, 2022a.  doi: 10.1109/ICRA46639.2022.9812253.
  * Gilles et al. [2022b] Thomas Gilles, Stefano Sabatini, Dzmitry Tsishkou, Bogdan Stanciulescu, and Fabien Moutarde.  THOMAS: Trajectory heatmap output with learned multi-agent sampling.  In _International Conference on Learning Representations_ , 2022b.  URL https://openreview.net/forum?id=QDdJhACYrlX.
  * Girgis et al. [2022] Roger Girgis, Florian Golemo, Felipe Codevilla, Martin Weiss, Jim Aldon D’Souza, Samira Ebrahimi Kahou, Felix Heide, and Christopher Pal.  Latent variable sequential set transformers for joint multi-agent motion prediction.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=Dup_dDqkZC5.
  * Gu et al. [2021] Junru Gu, Chen Sun, and Hang Zhao.  Densetnt: End-to-end trajectory prediction from dense goal sets.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 15303–15312, 2021.
  * Liang et al. [2020] Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun.  Learning lane graph representations for motion forecasting.  In _ECCV_ , 2020.
  * Nayakanti et al. [2022] Nigamaa Nayakanti, Rami Al-Rfou, Aurick Zhou, Kratarth Goel, Khaled S. Refaat, and Benjamin Sapp.  Wayformer: Motion forecasting via simple & efficient attention networks.  2022\.  doi: 10.48550/ARXIV.2207.05844.  URL https://arxiv.org/abs/2207.05844.
  * Ngiam et al. [2022] Jiquan Ngiam, Vijay Vasudevan, Benjamin Caine, Zhengdong Zhang, Hao-Tien Lewis Chiang, Jeffrey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, David J Weiss, Ben Sapp, Zhifeng Chen, and Jonathon Shlens.  Scene transformer: A unified architecture for predicting future trajectories of multiple agents.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=Wm3EA5OlHsG.
  * Sun et al. [2020] Hao Sun, Zhiqun Zhao, and Zhihai He.  Reciprocal learning networks for human trajectory prediction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * Wang et al. [2022] Yuting Wang, Hangning Zhou, Zhigang Zhang, Chen Feng, Huadong Lin, Chaofei Gao, Yizhi Tang, Zhenting Zhao, Shiyu Zhang, Jie Guo, Xuefeng Wang, Ziyao Xu, and Chi Zhang.  Tenet: Transformer encoding network for effective temporal flow on motion prediction.  2022\.  URL https://arxiv.org/abs/2207.00170.
  * Ye et al. [2022] Maosheng Ye, Jiamiao Xu, Xunnong Xu, Tongyi Cao, and Qifeng Chen.  Dcms: Motion forecasting with dual consistency and multi-pseudo-target supervision.  2022\.
  * Zhang et al. [2021] Lu Zhang, Peiliang Li, Jing Chen, and Shaojie Shen.  Trajectory prediction with graph-based dual-scale context fusion.  _arXiv preprint arXiv:2111.01592_ , 2021.
  * Zhao et al. [2020] Hang Zhao, Jiyang Gao, Tian Lan, Chen Sun, Benjamin Sapp, Balakrishnan Varadarajan, Yue Shen, Yi Shen, Yuning Chai, Cordelia Schmid, Congcong Li, and Dragomir Anguelov.  Tnt: Target-driven trajectory prediction.  In _CoRL_ , 2020.
  * Zhou et al. [2022] Zikang Zhou, Luyao Ye, Jianping Wang, Kui Wu, and Kejie Lu.  Hivt: Hierarchical vector transformer for multi-agent motion prediction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 8823–8833, June 2022.


# Beyond Summarization:
Designing AI Support for Real-World Expository Writing Tasks

Zejiang Shen†, Tal August□, Pao Siangliulue□, Kyle Lo□, Jonathan Bragg□
Jeff Hammerbacher□, Doug Downey□,⋄, Joseph Chee Chang□, David Sontag†
†Massachusetts Institute of Technology, □Allen Institute for AI, ⋄Northwestern
University zjshen<EMAIL_ADDRESS>tala, paos, kylel, jbragg,
jeffhammerbacher, dougd<EMAIL_ADDRESS>
###### Abstract.

Large language models have introduced exciting new opportunities and
challenges in designing and developing new AI-assisted writing support tools.
Recent work has shown that leveraging this new technology can transform
writing in many scenarios such as ideation during creative writing, editing
support, and summarization. However, AI-supported _expository writing_
—including real-world tasks like scholars writing literature reviews or
doctors writing progress notes—is relatively understudied.

In this position paper, we argue that developing AI supports for expository
writing has unique and exciting research challenges and can lead to high real-
world impacts. We characterize expository writing as evidence-based and
knowledge-generating: it contains summaries of external documents as well as
new information or knowledge. It can be seen as the product of authors’
sensemaking process over a set of source documents, and the interplay between
reading, reflection, and writing opens up new opportunities for designing AI
support. We sketch three components for AI support design and discuss
considerations for future research.

AI-Assisted Writing, Summarization, Expert Writing, Augmented Writing,
Expository Writing.

Figure 1. Dissecting an expository writing piece. In this example, the writing
is a paragraph of a paper’s related work section (Kojima et al., 2022). While
it contains summaries of relevant previous work, it also elects to group
several papers together as they describe similar concepts, or reconcile
different findings under a shared framework. Finally, it contrasts the current
paper with the previous work. The writing is based on evidence from existing
work, but also presents new knowledge about the relationship between existing
work as well as the novelty of the current paper, which are the final product
of the authors’ sensemaking process during the writing.

##  1\. Introduction

The advent of large language models (LLMs) (Brown et al., 2020; Smith et al.,
2022; Rae et al., 2021) has brought about a dramatic change in the design
space of AI-assisted writing.111In the remainder of this work, unless
otherwise specified, AI-assisted writing refers to the use of LLMs to support
writing. The language understanding capabilities and high-quality text
generation of LLMs promise to semi-automate cognitively-demanding writing
tasks, i.e., help produce outlines or even generate long and grammatically
correct paragraphs based on a short natural language input
prompt.222https://platform.openai.com/examples As a result, there is growing
interest from both the research and commercial communities in exploring new
designs for intelligent writing support systems, including supporting creative
story writing (Lee et al., 2022; Singh et al., 2022; Chung et al., 2022), blog
posts or email composition,333For example, commercial tools like copy.ai,
https://copy.ai and Lex, https://lex.page/. personal knowledge
management,444Including apps like mem, https://get.mem.ai and Notion-AI,
https://www.notion.so/. and so on.

While prior work has explored many exciting applications for LLMs, we argue
that expository writing is a task that is understudied in existing work on AI
augmented writing. We define expository writing pieces as articles that
summarize facts and produce new knowledge or information. For example, this
could be researchers collecting and reading multiple papers to write a survey
paper (Palani et al., 2023), or doctors reviewing clinical notes to devise a
treatment plan (Kind and Smith, 2008). In these cases, authors not only
summarize the source documents but also add information or bring new insights
that do not exist in the source, e.g., organizing the relevant papers or
synthesizing the patient’s symptoms and test results to create a differential
diagnosis and possible treatment options.

Compared to other types of writing that do not involve external documents,
e.g., creative story or argumentative writing (Lee et al., 2022; Yang et al.,
2022), expository writing requires authors to comprehend the source, generate
new insights, and faithfully reference and represent extracted information in
the final article. The interaction between reading and writing brings new
challenges for developing AI support, and the design is largely unexplored for
incorporating the latest generations of language models. While summarization
is often a necessary component, expository writing is also distinguished from
document summarization tasks in terms of its goal to bring about new
information that does not exist in the source.

Expository writing can be seen as a sensemaking process (Russell et al.,
1993), and different types of sub-tasks are involved: typically, authors start
with iteratively exploring and reading multiple relevant documents to
_identify and extract key evidence_ , then they _organize the evidence into
useful schema_ and further _synthesize into coherent writing_ to communicate
new knowledge or information (Pirolli, 2007). Therefore, the role of AI may
vary during the process of writing, and we argue it is important to design
different types and levels of AI support to maximally help the authors while
minimally influencing or shaping their opinions. In one approach, in the early
stages of writing, the writers would initiate and drive the work and AI should
only provide limited supporting functions. As ideas manifest and authors have
a better sense of the writing content, AI could assume more responsibilities
with authors “supervising” the model’s work. For example, when starting
writing a survey paper, the authors come up with the query to find relevant
papers first, and AI helps execute the search and discover related documents;
after authors read the retrieved papers and come up with ideas for the
writing, AI can help generate the writing text based on authors’ ideas, and
the writers only need to proofread model output. AI discovery helps the user
learn better, and editing support makes the writing more efficient and
polished; most importantly, writers are in full control of the thinking and
the ideas included in the produced articles.

Expository writing occurs frequently in many real-world tasks, and we argue
that real-world expository writing is a high value open problem for AI
augmented writing support, with many challenges and opportunities in the
space. Many realistic expository writing tasks require domain experts, e.g.,
describing key events in lawsuits (Shen et al., 2022), explaining scientific
concepts or ideas (King et al., 2022), and briefing on patients’ conditions
and treatments (Adams et al., 2021). Successful augmented writing systems for
these tasks stand to both reduce the expensive expert hours required to
perform the writing, and to improve the quality of the output (e.g. Bell et
al. (2020) reports that 1 in 5 patients find a mistake in their clinical notes
written by doctors or nurses). Often, supporting such real-world tasks also
allows us to draw upon existing rich repositories of example data and
established evaluation protocols, which can further the development of future
AI augmented writing systems. In the following sections, we formally define
the expository task and sketch components of the design that assist the
writing process rather than just the final article.

##  2\. Characterizing Expository Writing

In contrast to other writing tasks, expository writing has two unique
characteristics: it is _evidence-driven_ and _knowledge-generating_ as
illustrated in Figure 1. Formally, given a set of _source documents_ and a
collection of writing objectives, expository writing aims to compose a _piece_
that synthesizes the information from the source and produces _new
information_ in accordance with the objectives. There are different ways for
authors to synthesize the source, including but not limited to the _selection_
, _grouping_ , _contrasting_ , or _reconciliation_ of multiple pieces of
similar or different (even conflicting) statements, and the authors’ synthesis
brings information not present in the source documents. For example, a doctor
might _select and group_ a set of relevant observations from clinical notes,
and _reconcile_ with a possible condition to achieve the goal of _devising_ a
treatment plan. The writing objectives guide both the reading and synthesis
process, and they can also be updated given new insights generated during the
course of writing. Unlike a summarization task, here, it is not necessary to
involve all source documents in the final writing: in fact, the choice to not
include a document also constitutes new information in the writing product
(i.e., by indicating the document’s relative importance to the goal).

##  3\. Designing AI Supports for Expository Writing

To optimally involve LLMs to help expository writing, we argue that there are
three components: 1) supporting the reading and evidence-seeking for correct
and efficient understanding of the source, 2) assisting information synthesis
and the production of new knowledge and ideas, and 3) facilitating text
composition to communicate relevant evidence and insights. Expository writing
pieces aim at bringing about new knowledge or perspectives, but it takes
significant effort for authors to comprehend the source and convert the
thinking into writing. By reducing the cognitive load and the interaction
costs for information extraction and sensemaking during the reading, it can
help authors focus on the reasoning and idea generation (Kang et al., 2022;
Palani et al., 2023). On the other hand, while LLMs have demonstrated strong
performance in document understanding and text generation, they currently
suffer from hallucination (Ji et al., 2022) and are limited in reasoning
(Brown et al., 2020; Smith et al., 2022) and long-form generation
capabilities. As such, it is sub-optimal to use LLMs to generate the whole
writing piece altogether and post-edit (Cheng et al., 2022). A modular design
that provides varying AI support at different stages of writing can be most
helpful, and we detail the components as follows.

### Augmenting Reading and Collecting Evidence

Reading relevant documents to gather evidence is a crucial early step for
expository writing. While there have been significant research efforts on
document discovery and comprehension (e.g., for scientific documents (Cohan et
al., 2020; Rachatasumrit et al., 2022; et al., 2022; Kang et al., 2023; Head
et al., 2020)), AI tools for supporting _reading for writing_ is relatively
sparse (Han et al., 2022; Kang et al., 2022; Rachatasumrit et al., 2021).
Experts typically expend significant manual effort reading through many source
documents to identify key relevant information to help them synthesize new
knowledge. This is referred as “establishing the working memory” (Hayes, 2000;
Baddeley, 1992), a key step during the cognitive process of writing. One
existing approach to facilitate reading is using language models to
automatically generate summaries for long documents (Brown et al., 2020; Zhang
et al., 2023), but they can also be prone to hallucination (Ji et al., 2022).
More importantly, Ziegler et al. (2019) shows that recent instruction-tuned
models are merely “smart copiers” when performing summarization, and overly
trusting the model outputs risks biasing authors’ views and may lead to
authors collecting inaccurate evidence.

As such, authors’ reading and understanding the source still plays a central
role, and designing for _reading for expository writing_ should focus on the
_augmentation_ of reading and extraction of key information from source
documents (Kang et al., 2022), but at the same time providing support for
verifying the extracted evidence. For example, the system can automatically
generate a list of the key facts relevant to an author’s goal from a set of
source documents, and guide the authors to examine and check the facts’
correctness by linking each one back to relevant parts of the original source
documents.

### Supporting Information Synthesis

As an important middle step between reading and writing, the writer typically
takes a long time to inspect and reason over the evidence collected and
produce new ideas to be written, e.g., finding the similarities and
differences between papers and reconciling them under a shared framework as
illustrated in Figure 1. Synthesizing a large of collection of information
gathered across source documents can be challenging, and different interfaces
and techniques have been developed to support this step (Han et al., 2022;
Rachatasumrit et al., 2021; Cattan et al., 2021). Language models can be
involved to improve the efficiency and accuracy of these approaches by, e.g.,
automatic grouping based on the textual representation (Reimers and Gurevych,
2019) or enabling semantic search. Beyond better organizing evidence to
reflect an author’s mental model, a system could also leverage LLMs for
inspiration during synthesis (Gero et al., 2021). One example is proposing
connections between evidences and suggesting relevant/different topics to
discover, though it is important to also provide mechanisms so that authors
can verify and ground model-generated ideas with relevant evidence. The
involvement of AI models is not intended to be a replacement for authors’
thinking and contemplation but enhancement for producing new ideas, and
discovering otherwise unseen connections or findings.

### Facilitating Text Composition

The translation of ideas into language has been considered to be equally (if
not more) cognitively demanding and challenging as the other components: as
Flower and Hayes (1981) puts, it needs “juggle all the special demands of
written English” and “the process can interfere with … planning”. The latest
language models have demonstrated strong capabilities in generating fluent and
high-quality text, and some studies suggest the generations are even on par
with freelance writers for certain summarization tasks (Zhang et al., 2023).
It is promising to incorporate language models to support text composition,
but several design choices need to be considered to reduce the risk of
generation errors and biasing authors’ opinions (Mirowski et al., 2022). For
example, it may be important for language models in this case to be evidence-
aware and explicit—the generation should only be invoked when the authors call
it, and relevant evidence and a short prompt about the writing intent needs to
be provided. An instance of model generation could be relatively short, e.g.,
one or two sentences, with the authors calling it in multiple turns. Compared
to generating long completions at one time, e.g., a whole paragraph, we
speculate that this could decrease the possibility of erroneous and biased
generation, and decrease the cost of verification by the authors.

##  4\. Conclusion

Expository writing is genre of evidence-driven and knowledge-generating
writing that takes place in many real-world settings. In this paper, we argue
that the unique characteristics in expository writing open up new
opportunities for designing AI support. We sketch the components of the design
and highlight considerations and challenges for future implementation.

## Acknowledgments

We thank the anonymous reviewers for their insightful comments and
suggestions. We appreciate the helpful discussion and advice from Oren
Etzioni, Arvind Satyanarayan, Dennis Wei, Prasanna Sattigeri, Subhro Das,
Barbara Lam, Lauren Yu, and Ruochen Zhang.

## References

  * (1)
  * Adams et al. (2021) Griffin Adams, Emily Alsentzer, Mert Ketenci, Jason Zucker, and Noémie Elhadad. 2021.  What’s in a Summary? Laying the Groundwork for Advances in Hospital-Course Summarization. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, Online, 4794–4811.  https://doi.org/10.18653/v1/2021.naacl-main.382
  * Baddeley (1992) Alan Baddeley. 1992\.  Working memory.  _Science_ 255, 5044 (1992), 556–559.
  * Bell et al. (2020) Sigall K Bell, Tom Delbanco, Joann G Elmore, Patricia S Fitzgerald, Alan Fossa, Kendall Harcourt, Suzanne G Leveille, Thomas H Payne, Rebecca A Stametz, Jan Walker, et al. 2020\.  Frequency and types of patient-reported errors in electronic health record ambulatory care notes.  _JAMA network open_ 3, 6 (2020), e205867–e205867.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners. In _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877–1901.
  * Cattan et al. (2021) Arie Cattan, Sophie Johnson, Daniel Weld, Ido Dagan, Iz Beltagy, Doug Downey, and Tom Hope. 2021.  Scico: Hierarchical cross-document coreference for scientific concepts.  _arXiv preprint arXiv:2104.08809_ (2021).
  * Cheng et al. (2022) Ruijia Cheng, Alison Smith-Renner, Ke Zhang, Joel R Tetreault, and Alejandro Jaimes. 2022\.  Mapping the design space of human-ai interaction in text summarization.  _arXiv preprint arXiv:2206.14863_ (2022).
  * Chung et al. (2022) John Joon Young Chung, Wooseok Kim, Kang Min Yoo, Hwaran Lee, Eytan Adar, and Minsuk Chang. 2022.  TaleBrush: sketching stories with generative pretrained language models. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_. 1–19.
  * Cohan et al. (2020) Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S. Weld. 2020.  SPECTER: Document-level Representation Learning using Citation-informed Transformers. In _Annual Meeting of the Association for Computational Linguistics_.
  * et al. (2022) Joseph Chee Chang et al. 2022\.  CiteSee: Augmenting Citations in Scientific Papers with Persistent and Personalized Historical Context.  _ArXiv_ abs/2022.99999 (2022).
  * Flower and Hayes (1981) Linda Flower and John R Hayes. 1981.  A cognitive process theory of writing.  _College composition and communication_ 32, 4 (1981), 365–387.
  * Gero et al. (2021) K. Gero, Vivian Liu, and Lydia B. Chilton. 2021.  Sparks: Inspiration for Science Writing using Language Models.  _Designing Interactive Systems Conference_ (2021).
  * Han et al. (2022) Han L. Han, Junhang Yu, Raphael Bournet, Alexandre Ciorascu, Wendy E. Mackay, and Michel Beaudouin-Lafon. 2022.  Passages: Interacting with Text Across Documents.  _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ (2022).
  * Hayes (2000) John R Hayes. 2000\.  A new framework for understanding cognition and affect in writing.  _Perspectives on writing: Research, theory, and practice_ (2000), 6–44.
  * Head et al. (2020) Andrew Head, Kyle Lo, Dongyeop Kang, Raymond Fok, Sam Skjonsberg, Daniel S. Weld, and Marti A. Hearst. 2020.  Augmenting Scientific Papers with Just-in-Time, Position-Sensitive Definitions of Terms and Symbols.  _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ (2020).
  * Ji et al. (2022) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2022.  Survey of hallucination in natural language generation.  _Comput. Surveys_ (2022).
  * Kang et al. (2022) Hyeonsu B Kang, Joseph Chee Chang, Yongsung Kim, and Aniket Kittur. 2022.  Threddy: An Interactive System for Personalized Thread-based Exploration and Organization of Scientific Literature.  _arXiv preprint arXiv:2208.03455_ (2022).
  * Kang et al. (2023) Hyeonsu B Kang, Nouran Soliman, Matt Latzke, Joseph Chee Chang, and Joseph Chee Chang. 2023.  ComLittee: Literature Discovery with Personal Elected Author Committees.  _ArXiv_ abs/2302.06780 (2023).
  * Kind and Smith (2008) Amy JH Kind and Maureen A Smith. 2008.  Documentation of mandated discharge summary components in transitions from acute to subacute care.  _Advances in patient safety: new directions and alternative approaches (Vol. 2: culture and redesign)_ (2008).
  * King et al. (2022) Daniel King, Zejiang Shen, Nishant Subramani, Daniel S. Weld, Iz Beltagy, and Doug Downey. 2022\.  Don’t Say What You Don’t Know: Improving the Consistency of Abstractive Summarization by Constraining Beam Search.  _ArXiv_ abs/2203.08436 (2022).
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022\.  Large language models are zero-shot reasoners.  _arXiv preprint arXiv:2205.11916_ (2022).
  * Lee et al. (2022) Mina Lee, Percy Liang, and Qian Yang. 2022.  Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_. 1–19.
  * Mirowski et al. (2022) Piotr Mirowski, Kory W Mathewson, Jaylen Pittman, and Richard Evans. 2022.  Co-writing screenplays and theatre scripts with language models: An evaluation by industry professionals.  _arXiv preprint arXiv:2209.14958_ (2022).
  * Palani et al. (2023) Srishti Palani, Aakanksha Naik, Doug Downey, Amy X Zhang, Jonathan Bragg, and Joseph Chee Chang. 2023\.  Relatedly: Scaffolding Literature Reviews with Existing Related Work Sections.  _arXiv preprint arXiv:2302.06754_ (2023).
  * Pirolli (2007) Peter Pirolli. 2007\.  The Sensemaking Process and Leverage Points for Analyst Technology as Identified Through Cognitive Task Analysis.
  * Rachatasumrit et al. (2022) Napol Rachatasumrit, Jonathan Bragg, Amy X. Zhang, and Daniel S. Weld. 2022.  CiteRead: Integrating Localized Citation Contexts into Scientific Paper Reading.  _27th International Conference on Intelligent User Interfaces_ (2022).
  * Rachatasumrit et al. (2021) Napol Rachatasumrit, Gonzalo A. Ramos, Jina Suh, Rachel Ng, and Christopher Meek. 2021\.  ForSense: Accelerating Online Research Through Sensemaking Integration and Machine Research Support.  _26th International Conference on Intelligent User Interfaces_ (2021).
  * Rae et al. (2021) Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John F. J. Mellor, Irina Higgins, Antonia Creswell, Nathan McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, L. Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, N. K. Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Tobias Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew G. Johnson, Blake A. Hechtman, Laura Weidinger, Iason Gabriel, William S. Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem W. Ayoub, Jeff Stanway, L. L. Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. 2021.  Scaling Language Models: Methods, Analysis & Insights from Training Gopher.  _ArXiv_ abs/2112.11446 (2021).
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_ (2019).
  * Russell et al. (1993) Daniel M Russell, Mark J Stefik, Peter Pirolli, and Stuart K Card. 1993.  The cost structure of sensemaking. In _Proceedings of the INTERACT’93 and CHI’93 conference on Human factors in computing systems_. 269–276.
  * Shen et al. (2022) Zejiang Shen, Kyle Lo, Lauren Yu, Nathan Dahlberg, Margo Schlanger, and Doug Downey. 2022\.  Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple Granularities.  _arXiv preprint arXiv:2206.10883_ (2022).
  * Singh et al. (2022) Nikhil Singh, Guillermo Bernal, Daria Savchenko, and Elena L Glassman. 2022.  Where to hide a stolen elephant: Leaps in creative writing with multimodal machine intelligence.  _ACM Transactions on Computer-Human Interaction_ (2022).
  * Smith et al. (2022) Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Anand Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. 2022\.  Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model.  _ArXiv_ abs/2201.11990 (2022).
  * Yang et al. (2022) Kevin Yang, Nanyun Peng, Yuandong Tian, and Dan Klein. 2022\.  Re3: Generating longer stories with recursive reprompting and revision.  _arXiv preprint arXiv:2210.06774_ (2022).
  * Zhang et al. (2023) Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. 2023.  Benchmarking Large Language Models for News Summarization.  _CoRR_ (2023).  arXiv:2301.13848 [cs.CL] http://arxiv.org/abs/2301.13848v1
  * Ziegler et al. (2019) Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019\.  Fine-tuning language models from human preferences.  _arXiv preprint arXiv:1909.08593_ (2019).