# MuLTI: Efficient Video-and-Language Understanding with Text-Guided MultiWay-
Sampler and Multiple Choice Modeling

Jiaqi Xu, Bo Liu, Yunkuo Chen, Mengli Cheng, Xing Shi

###### Abstract

Video-and-language understanding has a variety of applications in the
industry, such as video question answering, text-video retrieval, and multi-
label classification. Existing video-and-language understanding methods
generally adopt heavy multi-modal encoders and feature fusion modules, which
consume high computational costs. Specially, they have difficulty dealing with
dense video frames or long text prevalent in industrial applications.

This paper proposes MuLTI, a highly accurate and efficient video-and-language
understanding model that achieves efficient and effective feature fusion and
rapid adaptation to downstream tasks. Specifically, we design a Text-Guided
MultiWay-Sampler based on adapt-pooling residual mapping and self-attention
modules to sample long sequences and fuse multi-modal features, which reduces
the computational costs and addresses performance degradation caused by
previous samplers. Therefore, MuLTI can handle longer sequences with limited
computational costs. Then, to further enhance the model’s performance and fill
in the lack of pretraining tasks in the video question answering, we propose a
new pretraining task named Multiple Choice Modeling. This task bridges the gap
between pretraining and downstream tasks and improves the model’s ability to
align video and text features. Benefiting from the efficient feature fusion
module and the new pretraining task, MuLTI achieves state-of-the-art
performance on multiple datasets. Implementation and pretrained models will be
released.

## Introduction

Figure 1:  Comparison of different models. Previous works such as (a) and (b)
cannot easily handle long sequences. Previous works such as (c) use randomly
initialized query vectors for sampler and condense video features, which is
sub-optimal solution.

Video-and-language understanding has a wide range of applications such as
video question answering (videoQA), text-video retrieval and multi-label
classification (Diba et al. 2019). Existing methods have made significant
progress in video-and-language understanding. However, they still suffer from
two challenges: Balancing computational efficiency and performance when
dealing with long sequences and the domain gap between pretraining and
downstream tasks.

The video-text model generally consists of three modules: text encoder, video
encoder, and feature fusion module. The latter two usually cause high
computational costs.

Feature fusion modules face efficiency and effectiveness challenges. Previous
studies (Fu et al. 2021; Huang et al. 2022) concatenate video-text encoder
outputs for transformer encoders processing, with complexity growing with
sequence length squared. Other studies (Lei et al. 2021b; Li et al. 2021; Yang
et al. 2022; Lei et al. 2021a) reduce computation by condensing video features
via mean pooling or class tokens before feature fusion, risking loss of
critical details. Flamingo (Alayrac et al. 2022) employs samplers and random
queries for efficient video feature condensation, though this approach is
suboptimal and may compromise feature integrity. In summary, balancing
computational costs and the model’s accuracy in the feature fusion module is
still challenging. Following (Miech et al. 2019; Sun et al. 2019; Li et al.
2020; Zhu and Yang 2020; Miech et al. 2020), we explore strategies for
selectively freezing encoder components to lower visual encoder training
costs.

Aligning pretraining with downstream tasks is challenging. Previous
pretraining frameworks generally apply four typical pretraining tasks: Masked
Frame Modeling (MVM) tasks (Lei et al. 2021a; Ma, Lou, and Ouyang 2021; Fu et
al. 2021; Huang et al. 2022) for video encoder optimization, Masked Language
Modeling (MLM) tasks (Devlin et al. 2018; Sun et al. 2019; Zhu and Yang 2020;
Luo et al. 2020; Li et al. 2020; Lei et al. 2021b) for text encoder
optimization, Video Text Matching (VTM) and Video Text Comparison (VTC) tasks
(Li et al. 2020; Luo et al. 2020; Fu et al. 2021; Li et al. 2021) for joint
optimization of video and text encoders. Although the above methods have
proven effective in learning video and text representations, there are still
significant domain gaps between pretraining and downstream tasks, especially
in videoQA. Only the VTC task is consistent with text-video retrieval among
the above pretraining tasks. In summary, narrowing the domain gap between the
pretraining and downstream tasks is still challenging.

Addressing these challenges, we introduce MuLTI, featuring a Text-Guided
MultiWay-Sampler for sequence condensation and multi-modal fusion. Existing
methods typically use a learnable query vector to sample the video feature
through self-attention modules (Alayrac et al. 2022). A randomly initialized
query vector can discard vital original feature information, causing
performance drops. We design an lightweight Adapt-Pooling method in Text-
Guided MultiWay-Sampler to obtain the condensed features by calculating the
importance of each sequence block. Then, we add the condensed features to the
sampled features and use short text features to sample and fuse long video
features. We share the self-attention and reserve different feed forward
networks for different modalities in the sampler.

Figure 1 shows that previous models (a)(Fu et al. 2021; Huang et al. 2022) and
(b)(Li et al. 2021) consume substantial video memory with their lengthy
concatenated feature fusion. Both (b) and (c)(Alayrac et al. 2022) compress
video features, a common choice due to their greater length compared to text.
However, excessive compression can impair performance because of the rich
information in video features. In contrast, we design MuLTI like (d) and
introduces the Text-Guided MultiWay-Sampler to efficiently condense text
features for fusion. Since text is more concise, we use the streamlined text
to direct video feature sampling, resulting in enhanced performance.

Pretraining on large-scale video-text datasets could improve the performance
of video-text models significantly. However, there are still domain gaps
between the existing pretraining tasks and downstream tasks, specifically in
videoQA. The difficulty of introducing videoQA to pretraining tasks is
constructing suitable question-answer pairs. To reduce the domain gap between
the pretraining task and the downstream task in videoQA, we introduce a new
pretraining task named Multiple Choice Modeling (MCM). The MCM can bridge the
task gap between pretraining and downstream tasks by constructing multiple-
choice question answering tasks on large-scale video-text datasets. It asks
the model to find text descriptions that match the video most from a randomly
constructed collection, which enhances the representation ability of the video
and text encoders and the alignment between video and text features.

The contributions can be summarized as follows:

(1) We propose MuLTI, a highly accurate and memory-efficient video-and-
language framework, which achieves efficient and effective feature fusion
through the feature sampling and attention modules.

(2) We propose a Text-Guided MultiWay-Sampler to sample long sequence features
and facilitate the interactions between video and text features, reducing
memory cost and improving performance.

(3) We design a new pretraining task called Multiple Choice Modeling (MCM) to
bridge the task gap between pretraining and downstream tasks. Experimental
results on seven English tasks and one Chinese multi-label classification task
demonstrate the effectiveness of MuLTI.

Although we designed MuLTI for industrial scenarios with long sequences, MuLTI
still handles short sequences well and achieves state-of-the-art performance.

## Related Work

Figure 2:  (a) shows the framework of MuLTI. MuLTI contains a video encoder, a
text encoder, and a Text-Guided MultiWay-Sampler. Text-Guided MultiWay-Sampler
is used to condense the extracted features and feature fusion. (b) shows the
framework of the Text-Guided MultiWay-Sampler. The adapt-pooling feature
provides origin information. We share the self-attention module and reserve
different feed forward networks for different modalities in the sampler to
accommodate modalities.

Video-and-Language Structure. Clover (Huang et al. 2022) and VIOLET (Fu et al.
2021) directly concatenate video and text features, using an encoder to manage
their complex interactions, with complexity tied to the concatenated sequence
length squared. ALPRO (Li et al. 2021) similarly uses an encoder for fusing
features but applies mean pooling on video features before concatenation,
risking loss of crucial details. AllInOne (Wang et al. 2022a) reduces memory
demands by merging text with image features frame-by-frame but still faces
high computational loads with extensive OCR transcripts. Flamingo (Alayrac et
al. 2022) attempts cost-cutting by condensing video features using samplers
and random queries, which isn’t ideal. To tackle the above problems, we design
a Text-Guided MultiWay-Sampler based on adapt-pooling residual mapping and
self-attention modules to sample long sequence features and fuse multi-modal
features.

Video-and-Language Pretraining. Four typical pretraining tasks are applied in
previous pretraining framework: Masked Frame Modeling (MVM) tasks (Lei et al.
2021a; Ma, Lou, and Ouyang 2021; Fu et al. 2021; Huang et al. 2022), Masked
Language Modeling (MLM) tasks (Devlin et al. 2018; Sun et al. 2019; Zhu and
Yang 2020; Luo et al. 2020; Li et al. 2020; Lei et al. 2021b; Fu et al. 2021),
Video Text Matching (VTM) and Video Text Comparison (VTC) tasks (Li et al.
2020; Luo et al. 2020; Fu et al. 2021; Li et al. 2021). MVM is used for video
encoder optimization, MLM is used for text encoder optimization, VTM and VTC
are used for joint optimization of video and text encoders. In (Ge et al.
2022), Multiple Choice Questions (MCQ) is proposed to learn fine-grained video
and text features. However, MCQ is trained by contrastive loss and does not
correlate well with videoQA. In summary, downstream task gaps persist between
pretraining and downstream tasks, particularly in videoQA. To address this, we
enhance MuLTI with MCM, bridging the pretraining and downstream tasks.

## Methodology

### MuLTI’s Architecture

Figure 2 (a) gives an overview of MuLTI’s architecture. Details for each
component are as follows.

Video and Text Encoders: Unless specified, a 12-layer VIT-B/16224 (Radford et
al. 2021) is used as video encoder. We sparsely sample $N_{v}$ frames from the
input video. The VIT-B/16224 model divides each frame into $K$ non-overlapping
patches. The per-video features is $\tilde{\vec{v}}\in\mathbb{R}^{N_{v}\times
K\times d}$, where $d$ is the feature dimension. The output of the video
encoder is a video features sequence: $\\{\vec{v}_{1},...,\vec{v}_{N_{v}}\\}$,
with $\vec{v}_{i}\in\mathbb{R}^{K\times d}$. Experiments revealed the class
token is unnecessary and was removed to save computation. Unless specified, a
12-layer bert (Devlin et al. 2018) is used as the text encoder. Assuming the
length of input text is $N_{t}$, the output of the text encoder is a text
features sequence $\tilde{\vec{t}}\in\mathbb{R}^{N_{t}\times d}$:
$\\{\vec{t}_{\mathrm{cls}},\vec{t}_{1},...,\vec{t}_{N_{t}}\\}$, with
$\vec{t}_{i}\in\mathbb{R}^{d}$. The $\vec{t}_{\mathrm{cls}}$ is the output of
the text [CLS] token. Following (Miech et al. 2019; Sun et al. 2019; Li et al.
2020), we explore training strategies for partially freezing encoder layers.

Text-Guided MultiWay-Sampler: The multi-modal fusion core is the Text-Guided
MultiWay-Sampler, adapted from the transformer decoder, shown in Figure 2 (b).
The Text-Guided MultiWay-Sampler is designed to condense text features and
fuse different modal features efficiently. Following (Alayrac et al. 2022), we
initialize a random learnable query to condense features via sampling. The
expression $Sampler(z,q)$ represents the sampling of feature $z$ using the
query vector $q$ through the sampler.

(i) Why we need Adapt-Pooling? Learnable queries can compress features well,
but starting with random vectors may reduce their effectiveness. Random
initialization may lose key details in original features, weakening the
model’s ability to capture and retain the essence of the data. Therefore, we
design an attention-based lightweight Adapt-Pooling method to condense long
sequence features. The Adapt-Pooling structure is shown on the left side of
the Figure 2 (b). The formula is shown below, $AdaPool(z)$ is the output of
the Adapt-Pooling, with $W^{reduce}\in\mathbb{R}^{d\times N_{s}}$, $d$ the
hidden dimension of the transformer, $N_{s}$ the length of condensed features,
$*.T$ the transposition of the matrix.

$AdaPool(z)=Softmax((W^{reduce}z).T)\times z$ (1)

The $Softmax((W^{reduce}z).T)$ yields an importance weight matrix of shape
$[N_{s},N_{i}]$, with each element signifying the relative importance of the
corresponding block within the sequence, and $N_{i}$ representing the length
of the input features. Adapt-Pooling selectively highlights key input
segments, condensing features while preserving its critical attributes. This
integration enriches the feature set with distilled information and ensures
full data utilization, boosting the model’s capacity and robustness.

(ii) Why we condense text features? The video features are often redundant,
whereas text features are denser and more meaningful (He et al. 2022).
Language guidance is key to distilling valuable information from videos. Both
(Li et al. 2021) and (Alayrac et al. 2022) condense the video features.
Excessive compression harms model performance; using condensed text to sample
video features improves results. Before fusion, learnable time embeddings
enhance image features for temporal modeling. The short text features are used
to sample the video features to fuse multi-modal features. In our Text-Guided
MultiWay-Sampler, we use shared self-attention modules but distinct FFNs for
each modality to handle multi-modal features efficiently. The fuse feature is
shown as follow, with $q$ the query embedding of text features, $z_{out}$ the
fused feature:

$z_{out}=Sampler(\tilde{\vec{v}},Sampler(\tilde{\vec{t}},q)+AdaPool(\tilde{\vec{t}}))$
(2)

A work similar to ours is Token Learner (Ryoo et al. 2021), which uses spatial
attention in model to extract 8 or 16 representative vectors from an image.
The difference is that we use Adapt-Pooling and self-attention to condense
features for multi-modal fusion. The sampler extracts complex information via
self-attention, while Adapt-Pooling provides fast, simple features through
residual mapping.

(iii) Why Text-Guided MultiWay-Sampler is efficient ? Our feature fusion
module outperforms flatten-based methods and transformer encoders in
efficiency, as simple analysis shows: we assume VIT-B/16224 is used as video
encoder, each frame will be flattened into a sequence of length 196. Let the
number of queries be $N^{q}_{t}$ for text, the length of the video features be
$N_{v}\times 196$, and the length of the text features be $N_{t}$. Thus the
complexity of the flatten method will be $O((N_{t}+N_{v}\times 196)^{2})$.
After applying the Text-Guided MultiWay-Sampler, the complexity is
$O(N^{q}_{t}\times N_{v}\times 196+N^{q}_{t}\times N_{t})$. As $N^{q}_{t}$ are
generally much smaller than $N_{v}$ and $N_{t}$. our method is much more
efficient than other methods.

Method | FLOPs | Params | FPS
---|---|---|---
MuLTI-S | 99G | 203M | 20.74
MuLTI-B | 346G | 247M | 10.13
VIOLET (Fu et al. 2021) | 249G | 198M | 9.05
ALPRO (Li et al. 2021) | 432G | 235M | 9.97
MuLTI-L | 1509G | 746M | 3.12
FrozenBiLM (Yang et al. 2022) | 1733G | 1224M | 2.54
Table 1:  Comparison among models with 16 frames. Text length is 512. FPS is
based on 1 NVIDIA V100 16GB GPU.

MuLTI for different scenes. In this section, we built scalable models for
resource-varied scenes. We replace the video encoder from VIT-B/16 to VIT-L/14
and the text encoder from bert-base to bert-large. Then, we obtain MuLTI-L. In
addition, we replace the video encoder from VIT-B/16 to VIT-B/32 and reduce
the text encoder from 12 layers to 6 layers. Then, we obtain MuLTI-S. The
floating point of operations (FLOPs), parameters (Params) and frames per
second (FPS) of different models are shown in Table 1.

### Pretraining for MuLTI

Multiple Choice Modeling: Despite MLM and VTM’s success in learning video-text
representations, a significant gap remains between pretraining and downstream
tasks like videoQA. The difficulty of introducing videoQA into the pretraining
task is constructing suitable question-answer pairs. Inspired by multiple
choice videoQA, we find the text descriptions paired with videos are the
correct natural answers. Therefore, we introduce Multiple Choice Modeling, a
new pretraining task that bridges the task gap between pretraining and
downstream tasks. Specifically, it is constructed as follows, which is a four-
choice question.

"[CLS]<Question> ? [SEP] Option 1: <Answer 1>. [SEP] Option 2: <Answer 2>.
[SEP] Option 3: <Answer 3>. [SEP] Option 4: <Answer 4>."

We randomly place the correct descriptions in <Answer 1>, <Answer 2>, <Answer
3>, <Answer 4>, and obtain answers other than the correct descriptions through
the text corpus. The <Question> also has various choices, such as “What does
this picture describe?”, “What does this video describe?” and so on.

As shown in Figure 2 (a), typical MLM, VTM, and VTC tasks correspond to the
red arrows and red squares in the image. The MCM corresponds to the image’s
blue arrows and blue squares, and the MCM does not conflict with the other
pretraining tasks. The MCM is seamlessly integrated with other pretraining
tasks and does not require additional manual annotations or data
preprocessing. It utilizes video encoders to extract visual features and text
encoders for generating textual representations, followed by a Text-Guided
MultiWay-Sampler for feature fusion. The MCM head evaluates the given options’
relevance to the video, optimizing alignment using cross-entropy loss. The MCM
task, choosing the best description from options, mirrors essential videoQA
cognition, enhancing the model’s cross-modal reasoning and alignment. MCM
directly improves the model’s ability to match text with corresponding videos,
enhancing performance in text-video retrieval tasks.

Pretraining Objectives: We also employ the MLM, VTM and VTC, considering their
effectiveness. The MLM randomly masks input tokens with 15% probability and
replaces them with [MASK], which are predicted based on video and text. The
VTC treats matching video text pairs as positive pairs and other video text
pairs in the batch as negative pairs. The VTM is slightly different from VTC,
where the multi-modal features are fused before used for classification. The
overall pretraining objective of MuLTI is:

$L=L_{mlm}+L_{vtc}+L_{vtm}+L_{mcm}$ (3)

|  | MSRQ | MSVQ | TGIF. | TGIF. | TGIF | MSRVTT | DiDeMo
---|---|---|---|---|---|---|---|---
|  | Act. | Tran. | Fra. | Ret | Ret
Method | #PT | Acc.$\uparrow$ | Acc.$\uparrow$ | Acc.$\uparrow$ | Acc.$\uparrow$ | Acc.$\uparrow$ | R1 / R5 / R10 $\uparrow$ | G-M $\uparrow$ | R1 / R5 / R10 $\uparrow$ | G-M $\uparrow$
CLIP4CLIP | 400M | - | - | - | - | - | 43.1 / 70.4 / 80.8 | 62.6 | 43.4 / 70.2 / 80.6 | 62.6
QB-Norm | 400M | - | - | - | - | - | 47.2 / 73.0 / 83.0* | 65.9* | 43.3 / 71.4 / 80.8* | 63.0*
CAMoE | 400M | - | - | - | - | - | 47.3 / 74.2 / 84.5* | 66.7* | 43.8 / 71.4 / 79.9* | 63.0*
TS2-Net | 400M | - | - | - | - | - | 54.0 / 79.3 / 87.4* | 72.1* | 47.4 / 74.1 / 82.4* | 66.1*
ALPRO | 5.5M | 42.1 | 45.9 | - | - | - | 33.9 / 60.7 / 73.2 | 53.2 | 35.9 / 67.5 / 78.8 | 57.6
VIOLET | 185.5M | 43.9 | 47.9 | 92.5 | 95.7 | 68.9 | 34.5 / 63.0 / 73.4 | 54.2 | 32.6 / 62.8 / 74.7 | 53.5
AllInOne | 102.5M | 44.3 | 47.9 | 92.7 | 94.3 | 64.2 | 37.9 / 68.1 / 77.1 | 58.4 | 32.7 / 61.4 / 73.5 | 52.8
Clover | 5.5M | 44.1 | 52.4 | 95.0 | 98.2 | 71.6 | 40.5 / 69.8 / 79.4 | 60.7 | 50.1 / 76.7 / 85.6 | 69.0
Flamingo | 2139M | 47.4 | 52.3 | - | - | - | - | - | - | -
FrozenBiLM | 10M | 47.0 | 54.4 | - | - | 68.6 | - | - | - | -
MuLTI-S | 5.5M | 45.6 | 50.0 | 97.3 | 98.9 | 71.2 | 41.3 / 70.6 / 79.7 | 61.5 | 42.6 / 71.4 / 80.0 | 62.5
45.8 / 73.5 / 82.0* | 65.1* | 47.9 / 73.0 / 82.6* | 66.1*
MuLTI-B | 5.5M | 46.6 | 53.0 | 97.3 | 99.1 | 73.5 | 45.1 / 72.4 / 81.8 | 64.4 | 45.2 / 74.6 / 82.2 | 65.2
49.4 / 75.9 / 84.0* | 68.0* | 48.3 / 75.4 / 83.5* | 67.2*
MuLTI-L | 5.5M | 47.8 | 54.7 | 97.9 | 99.0 | 75.6 | 48.7 / 75.9 / 83.9 | 67.7 | 50.5 / 78.5 / 86.2 | 69.9
54.7 / 77.7 / 86.0* | 71.5* |  56.5 / 80.2 / 87.0* | 73.3*
Table 2:  Comparisons with existing methods. #PT means number of pretrain
datasets. Acc. (%) denotes the performance of videoQA. R@k denotes recall (%)
with k retrieval efforts. G-M denotes the geometric mean of R@1, R@5, R@10.
The datasets commonly used are WebVid2M (Bain et al. 2021), WebVid10M (Bain et
al. 2021), WIT (Radford et al. 2021), HowTo100M (Miech et al. 2019), YT-
Temporal-180M (Zellers et al. 2021), Conceptual Captions (Sharma et al. 2018).
* indicates that DSL (Cheng et al. 2021) or QB-Norm (Bogolin et al. 2021) is
used for post-processing.

## Experiments

### Implementation Details

Pretraining datasets: We pretrained the model using two large datasets. One is
WebVid-2M (Bain et al. 2021), which contains 2.5M video-text pairs. Because
pretraining the video-text model using image-text pairs also improves the
model’s performance (Lei et al. 2021b), the CC-3M(Sharma et al. 2018) is also
used as a pretrained dataset containing 3M image-text pairs.

Method | #PT | OCR | Multi-Label
---|---|---|---
VIOLET ${\ddagger}$ | - | ✘ | 55.22
ALPRO ${\ddagger}$ | - | ✘ | 58.53
MuLTI-S | - | ✘ | 63.97
MuLTI-S | - | ✔ | 66.13
MuLTI-B | - | ✘ | 64.60
MuLTI-B | - | ✔ | 67.86
Table 3:  Comparisons on multi-label classification in mAP (%). ${\ddagger}$
means the methods are reproduced in our framework.  Figure 3:  Comparisons
with existing methods on Memory-Usage with different numbers of frames. Text
length is 512.

We implement MuLTI in PyTorch (Paszke et al. 2019). In detail, the video
encoder is initialized with pretrained weights from CLIP (Radford et al.
2021). Text encoder is initialized with a 12-layer of the BERTbase model
(Devlin et al. 2018). Then, a 4-layer Text-Guided MultiWay-Sampler is used to
condense text features and fuse multi-modal features. The length of query
embedding is set to 16. MuLTI pretraining spanned 10 epochs on eight NVIDIA
A100 GPUs, a 256 batch size totaling 200k iterations. Optimization used AdamW
with a $1e^{-4}$ learning rate, 0.05 weight decay, and a warm-up scheduler. We
uniformly sample 16 frames for each video and scale them to $224\times 224$.

### Downstream Tasks and Datasets

Video Question Answering. We evaluate MuLTI on five widely used videoQA tasks.
(1) MSRQ (MSRVTT-QA) (Xu et al. 2017, 2016) is a open-ended videoQA task
includes 10k videos and 243k question-answer pairs. (2) MSVQ (MSVD-QA) (Xu et
al. 2017; Chen and Dolan 2011) is a open-ended videoQA task includes 1970
videos and 50k question-answer pairs. (3) TGIF-QA (Jang et al. 2017) contains
three datasets: TGIF-Action and TGIF-Transition for multiple-choice videoQA
tasks, and TGIF-Frame for open-ended videoQA tasks.

Text-Video Retrieval. (1) MSRR (MSRVTT-Ret) contains 10K videos with 200K
annotations. Following (Fu et al. 2021), we use 9k videos for training and 1k
videos for testing. (2) DiDeMo (DiDeMo-Ret) consists of 10K videos with 40K
annotations. Following (Lei et al. 2021b), we concatenate all annotations from
the video into a title.

Method | Base | TGMS | PB | MCM | MSRQ | MSVQ | MSRR
---|---|---|---|---|---|---|---
Acc.$\uparrow$ | Acc.$\uparrow$ | R1$\uparrow$ | R5$\uparrow$ | R10$\uparrow$ |  G-Mean $\uparrow$
MuLTI-B | ✔ | ✘ | ✘ | ✘ | 44.84 | 48.35 | 38.90 | 69.50 | 78.50 | 59.64
✔ | ✔ | ✘ | ✘ | 45.54 | 49.86 | 38.80 | 70.30 | 80.10 | 60.22
✔ | ✔ | ✔ | ✘ | 46.28 | 51.93 | 44.30 | 72.40 | 81.90 | 64.04
✔ | ✔ | ✔ | ✔ | 46.61 | 53.03 | 45.10 | 72.40 | 81.80 | 64.40
Table 4: Evaluations of the proposed methods. TGMS: Text-Guided MultiWay-
Sampler. PB (Pretraining Baseline): Pretraining model with MLM, VTM and VTC.
MCM: Multiple Choice Modeling. Acc. (%) is used to measure the performance of
videoQA. R@k denotes recall (%) with k retrieval efforts. G-Mean denotes the
geometric mean of R@1, R@5, R@10. Figure 4:  Comparisons of different text
length and number of frames on Memory-Usage. The F means Flatten, the D means
Decoder, the E means Encoder, the S means Sampler. The number in parentheses
represents the number of frames.  Figure 5:  A visualization of the cross-
attention map from the Text-Guided MultiWay-Sampler.

Multi-Label Classification. Video labels are crucial for the ranking models
used in online advertising111https://algo.qq.com/index.html. We create a short
video dataset from our app, which includes 486k videos with captions and 21696
labels. Multiple professional editors cross check the labels. We used a high-
performing text detector from
ICDAR222https://rrc.cvc.uab.es/?ch=4&com=evaluation&task=4 for OCR
transcripts. The OCR transcripts are truncated to 512. The examples for multi-
label classification can be found in the appendix.

### Performance of Proposed Methods

Table 2 compares MuLTI with CLIP4CLIP (Luo et al. 2021), QB-Norm (Bogolin et
al. 2021), CAMoE (Cheng et al. 2021), TS2-Net (Liu et al. 2022), ALPRO (Li et
al. 2021), VIOLET (Fu et al. 2021), AllInOne (Wang et al. 2022a), Clover
(Huang et al. 2022), Flamingo (Alayrac et al. 2022) and FrozenBiLM (Yang et
al. 2022).

In videoQA tasks, MuLTI surpasses all baseline models on MSRQ, MSVQ, TGIF-
Action, TGIF-Transition and TGIF-Frames. Since MuTLI does not use speech data
as input, it is compared with FrozenBiLM(Yang et al. 2022) without using
speech data. In general, MuLTI achieves state-of-the-art performance in
various QA tasks.

In text-video retrieval tasks, we finetune MuLTI using the MSRVTT and DiDeMo
datasets. Our results demonstrate that MuLTI is highly competitive in both
benchmarks, particularly in the DiDeMo dataset. These findings highlight the
effectiveness of MuLTI for text-video retrieval.

Methods | MSRQ | MSVQ | Memory Usage
---|---|---|---
Class Token | 44.54 | 47.90 | 7081
Mean Pooling | 44.40 | 47.07 | 6941
Max Pooling | 44.41 | 46.93 | 6963
Flatten + Encoder | 44.84 | 48.35 | 15791
TGMS | 45.54 | 49.86 | 10551
Table 5:  Ablation studies on feature retention methods. The number of sparse frames is set to 6 for Flatten method. TGMS means Text-Guided MultiWay-Sample.  Method | CV | CT | SS | AP | MSRQ | MSVQ
---|---|---|---|---|---|---
| ✘ | ✘ | ✘ | ✘ | 45.13 | 49.19
| ✔ | ✘ | ✘ | ✘ | 44.76 | 48.10
| ✔ | ✘ | ✔ | ✔ | 45.14 | 48.92
Flatten | ✔ | ✔ | ✘ | ✘ | 44.57 | 48.50
Decoder | ✘ | ✔ | ✘ | ✘ | 45.08 | 49.38
| ✘ | ✔ | ✔ | ✘ | 45.16 | 49.80
| ✘ | ✔ | ✘ | ✔ | 45.48 | 49.54
| ✘ | ✔ | ✔ | ✔ | 45.54 | 49.86
Table 6: An ablation study on feature compression methods. CV means Condensed
Video, CT means Condensed Text, SS means Shared-Sampler, AP means Adapt-
Pooling.

For multi-label classification, we compare MuLTI with VIOLET and ALPRO but
exclude FrozenBiLM due to its impractical size for industry deployment. VIOLET
and ALPRO do not use OCR transcripts as they would lead to out-of-memory on
V100 GPUs. We also report MuLTI’s OCR-less performance in Table 3 for a fair
comparison; MuLTI significantly surpasses both VIOLET and ALPRO. As shown in
Figure 3, MuLTI maintains a video memory cost less than half of ALPRO’s and
VIOLET’s when frame count rises during training, because its efficient fusion
modules minimizes memory cost increases.

PB | MVM | MCM | MSRQ | MSVQ | MSRR
---|---|---|---|---|---
✔ | ✘ | ✘ | 46.28 | 51.93 | 64.04
✔ | ✔ | ✘ | 45.87 | 50.16 | 63.41
✔ | ✔ | ✔ | 46.11 | 51.65 | 63.71
✔ | ✘ | ✔ | 46.61 | 53.03 | 64.40
Table 7:  Ablation studies on the Multiple Choice Modeling. PB means Pretraining Baseline.  Frozen / Total | MSRQ | MSVQ | Memory Usage
---|---|---|---
VE | TE
12/12 | 12/12 | 44.06 | 46.83 | 6109
12/12 | 0/12 | 44.07 | 47.12 | 7439
6/12 | 0/12 | 45.10 | 47.57 | 18219
9/12 | 0/12 | 45.59 | 47.52 | 11541
9/12 | 3/12 | 45.50 | 49.63 | 11131
9/12 | 6/12 | 45.54 | 49.86 | 10551
9/12 | 9/12 | 45.04 | 49.14 | 10283
Table 8:  Ablation studies on frozen layers. VE refers to video encoder and TE refers to text encoder. Frozen/Total refers to the number of frozen and total layers.  Adapter | ATT | MSRQ | MSVQ | MSRR | DiDeMo
---|---|---|---|---|---
✘ | ✘ | 45.54 | 49.86 | 60.22 | 51.68
✔ | ✘ | 45.61 | 50.48 | 60.54 | 52.08
✔ | ✔ | 45.71 | 50.63 | 61.16 | 52.42
Table 9:  Ablation studies on the Attention-Adapter. ATT means Attention.

Finally, we evaluate our main technical contributions in Table 4. Compared
with baseline models, our main technical contributions improve performance on
all datasets. The Text-Guided MultiWay-Sampler boosts MuLTI’s multi-modal
fusion ability, pinpointing key details in surplus video features. MCM
advances the model’s alignment ability and narrows the gap between pretraining
and downstream tasks.

### The Importance of Text-Guided MultiWay-Sampler

Why we condense text features? We compare performance of different aggregation
methods (i.e. Class Token, Mean Pooling, Max Pooling and Flatten) in Table 5.
Results show that Flatten outperforms other aggregation methods but requires
substantial video memory. Above section reveals the decoder uses less memory
than the encoder for long sequences, prompting its use in feature fusion. The
decoder handles datasets like MSRQ well. However, the cost is still high when
processing long text and video like our multi-label datasets. The specific
memory cost is shown in Figure 4. Following (Alayrac et al. 2022), we use a
decoder-based sampler for feature condensation Table 6 compares different
condensation methods, showing text compression’s superiority. As shown in
Figure 5, the visual part most relevant to the problem is given more weight.

The importance of Shared-Sampler. The sampler and feature fusion module, using
the same decoder structure, can share weights without compromising
performance, simplifying model optimization (Wang et al. 2022b). We share the
sampler and decoder’s self-attention but keep separate FFNs for each modality,
cutting parameters while maintaining performance. Compared with the Flatten
Method, the Shared-Sampler improves accuracies on MSRQ and MSVQ by 0.32% and
1.45%, respectively.

The importance of Adapt-Pooling. As shown in Table 6, the sampler leads to
worse performance when condensing text and video features. The sampler’s
random query vector carries the risk of losing original key features; we
design a lightweight aggregation module, Adapt Pooling, to preserve the
original features. As shown in Table 6, the Adapt-Pooling improves accuracy on
MSRQ and MSVQ. Additionally, we explored various combination methods (i.e.
add, concatenate, and multiply), and noted slight performance differences. We
achieved an accuracy of 45.51% using concatenate and 45.45% using multiply on
MSRQ.

To verify these techniques’ robustness, we applied them to condense video
features, which also improved performance.

### The Importance of Multiple Choice Modeling

MCM aims to bridge the gap between pretraining and downstream tasks by
integrating videoQA into pretraining, enhancing the model’s focus on video and
sentence subjects for better multimodal feature extraction.

We use the classical MLM, VTM, and VTC tasks to pretrain the model as a
baseline. Due to video content corruption caused by MVM, the MVM task
conflicts with other tasks (Lei et al. 2021a). In our initial attempts to
include MVM for pretraining, we observed a degradation in performance as shown
in Table 7. Thus, we have decided not to use MVM for pretraining. To confirm
MCM’s robustness, we also added MCM for pretraining based on the usage of MVM.
The results show MCM still substantially enhances model’s performance.
Compared to the model pretrained with baseline, MCM explicitly improves the
model’s performance on the videoQA task by narrowing the task gap between
pretraining and downstream tasks. MCM’s promotion of multi-modal feature
alignment enhances the model’s retrieval task performance. As shown in Table
7, the models pretrained with MCM outperformed the baseline in both videoQA
and retrieval tasks, demonstrating its effectiveness.

### Ablation Experiment on Training Strategies

Analysis of Frozen Layers. In this section, we systematically evaluate the
effect of the number of frozen layers. The results on videoQA are demonstrated
in Table 8. It indicates that unfreezing the top layers of video and text
encoders can improve performance on both datasets.

Analysis of Attention-Adapter. Analyzing frozen layers reveals that unfreezing
excess layers reduces accuracy due to overfitting from excessive parameter
adjustments. Following (Yang et al. 2022), we add adapters to the encoders in
the shallow layers. Table 9 shows that while adapters perform effectively,
their capability is constrained by the basic FFN module. By integrating a
lightweight attention module (Hu et al. 2017), the model focuses better on
informative tokens.

## Conclusion

We present MuLTI, a high-performing video-language framework with a novel
Text-Guided MultiWay-Sampler for improved sampling efficiency and a
pretraining task to better align with downstream tasks. MuLTI achieves state-
of-the-art performance on seven video-language benchmarks.

## References

  * Alayrac et al. (2022) Alayrac, J.-B.; Donahue, J.; Luc, P.; Miech, A.; Barr, I.; Hasson, Y.; Lenc, K.; Mensch, A.; Millican, K.; Reynolds, M.; Ring, R.; Rutherford, E.; Cabi, S.; Han, T.; Gong, Z.; Samangooei, S.; Monteiro, M.; Menick, J.; Borgeaud, S.; Brock, A.; Nematzadeh, A.; Sharifzadeh, S.; Binkowski, M.; Barreira, R.; Vinyals, O.; Zisserman, A.; and Simonyan, K. 2022.  Flamingo: a Visual Language Model for Few-Shot Learning.  _ArXiv_ , abs/2204.14198.
  * Bain et al. (2021) Bain, M.; Nagrani, A.; Varol, G.; and Zisserman, A. 2021.  Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval.  _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 1708–1718.
  * Bogolin et al. (2021) Bogolin, S.-V.; Croitoru, I.; Jin, H.; Liu, Y.; and Albanie, S. 2021.  Cross Modal Retrieval with Querybank Normalisation.  _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 5184–5195.
  * Chen and Dolan (2011) Chen, D.; and Dolan, W. B. 2011.  Collecting highly parallel data for paraphrase evaluation.  In _Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies_ , 190–200.
  * Cheng et al. (2021) Cheng, X.; Lin, H.; Wu, X.; Yang, F.; and Shen, D. 2021.  Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss.  _ArXiv_ , abs/2109.04290.
  * Devlin et al. (2018) Devlin, J.; Chang, M.; Lee, K.; and Toutanova, K. 2018.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  _CoRR_ , abs/1810.04805.
  * Diba et al. (2019) Diba, A.; Fayyaz, M.; Sharma, V.; Paluri, M.; Gall, J.; Stiefelhagen, R.; and Gool, L. V. 2019.  Large Scale Holistic Video Understanding.  In _European Conference on Computer Vision_.
  * Fu et al. (2021) Fu, T.-J.; Li, L.; Gan, Z.; Lin, K.; Wang, W. Y.; Wang, L.; and Liu, Z. 2021.  VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling.  _ArXiv_ , abs/2111.12681.
  * Ge et al. (2022) Ge, Y.; Ge, Y.; Liu, X.; Li, D.; Shan, Y.; Qie, X.; and Luo, P. 2022.  BridgeFormer: Bridging Video-text Retrieval with Multiple Choice Questions.  _ArXiv_ , abs/2201.04850.
  * He et al. (2022) He, K.; Chen, X.; Xie, S.; Li, Y.; Doll’ar, P.; and Girshick, R. B. 2022.  Masked Autoencoders Are Scalable Vision Learners.  _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 15979–15988.
  * Hu et al. (2017) Hu, J.; Shen, L.; Albanie, S.; Sun, G.; and Wu, E. 2017.  Squeeze-and-Excitation Networks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 42: 2011–2023.
  * Huang et al. (2022) Huang, J.; Li, Y.; Feng, J.; Sun, X.; and Ji, R. 2022.  Clover: Towards A Unified Video-Language Alignment and Fusion Model.  _ArXiv_ , abs/2207.07885.
  * Jang et al. (2017) Jang, Y.; Song, Y.; Yu, Y.; Kim, Y.; and Kim, G. 2017.  TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering.  _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 1359–1367.
  * Lei et al. (2021a) Lei, C.; Luo, S.; Liu, Y.; He, W.; Wang, J.; Wang, G.; Tang, H.; Miao, C.; and Li, H. 2021a.  Understanding Chinese Video and Language via Contrastive Multimodal Pre-Training.  _Proceedings of the 29th ACM International Conference on Multimedia_.
  * Lei et al. (2021b) Lei, J.; Li, L.; Zhou, L.; Gan, Z.; Berg, T. L.; Bansal, M.; and Liu, J. 2021b.  Less is more: Clipbert for video-and-language learning via sparse sampling.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 7331–7341.
  * Li et al. (2021) Li, D.; Li, J.; Li, H.; Niebles, J. C.; and Hoi, S. C. H. 2021.  Align and Prompt: Video-and-Language Pre-training with Entity Prompts.  _ArXiv_ , abs/2112.09583.
  * Li et al. (2020) Li, L.; Chen, Y.-C.; Cheng, Y.; Gan, Z.; Yu, L.; and Liu, J. 2020.  Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training.  _ArXiv_ , abs/2005.00200.
  * Liu et al. (2022) Liu, Y.; Xiong, P.; Xu, L.; Cao, S.; and Jin, Q. 2022.  TS2-Net: Token Shift and Selection Transformer for Text-Video Retrieval.  In _Proceedings of the European Conference on Computer Vision (ECCV)_.
  * Luo et al. (2020) Luo, H.; Ji, L.; Shi, B.; Huang, H.; Duan, N.; Li, T.; Li, J.; Bharti, T.; and Zhou, M. 2020.  Univl: A unified video and language pre-training model for multimodal understanding and generation.  _arXiv preprint arXiv:2002.06353_.
  * Luo et al. (2021) Luo, H.; Ji, L.; Zhong, M.; Chen, Y.; Lei, W.; Duan, N.; and Li, T. 2021.  CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval.  _Neurocomputing_ , 508: 293–304.
  * Ma, Lou, and Ouyang (2021) Ma, Z.; Lou, M.; and Ouyang, X. 2021.  Top1 Solution of QQ Browser 2021 Ai Algorithm Competition Track 1 : Multimodal Video Similarity.  _ArXiv_ , abs/2111.01677.
  * Miech et al. (2020) Miech, A.; Alayrac, J.-B.; Smaira, L.; Laptev, I.; Sivic, J.; and Zisserman, A. 2020.  End-to-End Learning of Visual Representations From Uncurated Instructional Videos.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 9876–9886.
  * Miech et al. (2019) Miech, A.; Zhukov, D.; Alayrac, J.-B.; Tapaswi, M.; Laptev, I.; and Sivic, J. 2019.  HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips.  _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2630–2640.
  * Paszke et al. (2019) Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. 2019.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32: 8026–8037.
  * Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021.  Learning Transferable Visual Models From Natural Language Supervision.  In _ICML_.
  * Ryoo et al. (2021) Ryoo, M. S.; Piergiovanni, A. J.; Arnab, A.; Dehghani, M.; and Angelova, A. 2021.  TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?  _ArXiv_ , abs/2106.11297.
  * Sharma et al. (2018) Sharma, P.; Ding, N.; Goodman, S.; and Soricut, R. 2018.  Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning.  In _ACL_.
  * Sun et al. (2019) Sun, C.; Myers, A.; Vondrick, C.; Murphy, K. P.; and Schmid, C. 2019.  VideoBERT: A Joint Model for Video and Language Representation Learning.  _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 7463–7472.
  * Wang et al. (2022a) Wang, A.; Ge, Y.; Yan, R.; Ge, Y.; Lin, X.; Cai, G.; Wu, J.; Shan, Y.; Qie, X.; and Shou, M. Z. 2022a.  All in One: Exploring Unified Video-Language Pre-training.  _ArXiv_ , abs/2203.07303.
  * Wang et al. (2022b) Wang, W.; Bao, H.; Dong, L.; Bjorck, J.; Peng, Z.; qiang liu; Aggarwal, K.; Mohammed, O. K.; Singhal, S.; Som, S.; and Wei, F. 2022b.  Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks.  _ArXiv_ , abs/2208.10442.
  * Xu et al. (2017) Xu, D.; Zhao, Z.; Xiao, J.; Wu, F.; Zhang, H.; He, X.; and Zhuang, Y. 2017.  Video question answering via gradually refined attention over appearance and motion.  In _Proceedings of the ACM international conference on Multimedia_ , 1645–1653.
  * Xu et al. (2016) Xu, J.; Mei, T.; Yao, T.; and Rui, Y. 2016.  Msr-vtt: A large video description dataset for bridging video and language.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 5288–5296.
  * Yang et al. (2022) Yang, A.; Miech, A.; Sivic, J.; Laptev, I.; and Schmid, C. 2022.  Zero-Shot Video Question Answering via Frozen Bidirectional Language Models.  _ArXiv_ , abs/2206.08155.
  * Zellers et al. (2021) Zellers, R.; Lu, X.; Hessel, J.; Yu, Y.; Park, J. S.; Cao, J.; Farhadi, A.; and Choi, Y. 2021.  MERLOT: Multimodal Neural Script Knowledge Models.  In _Neural Information Processing Systems_.
  * Zhu and Yang (2020) Zhu, L.; and Yang, Y. 2020.  ActBERT: Learning Global-Local Video-Text Representations.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 8743–8752.


# A Massive Scale Semantic Similarity Dataset of Historical English

Emily Silcock1, Melissa Dell${}^{1,2^{\ast}}$
1Harvard University; Cambridge, MA, USA.
2National Bureau of Economic Research; Cambridge, MA, USA.
∗Corresponding author<EMAIL_ADDRESS>
###### Abstract

A diversity of tasks use language models trained on semantic similarity data.
While there are a variety of datasets that capture semantic similarity, they
are either constructed from modern web data or are relatively small datasets
created in the past decade by human annotators. This study utilizes a novel
source, newly digitized articles from off-copyright, local U.S. newspapers, to
assemble a massive-scale semantic similarity dataset spanning 70 years from
1920 to 1989 and containing nearly 400M positive semantic similarity pairs.
Historically, around half of articles in U.S. local newspapers came from
newswires like the Associated Press. While local papers reproduced articles
from the newswire, they wrote their own headlines, which form abstractive
summaries of the associated articles. We associate articles and their
headlines by exploiting document layouts and language understanding. We then
use deep neural methods to detect which articles are from the same underlying
source, in the presence of substantial noise and abridgement. The headlines of
reproduced articles form positive semantic similarity pairs. The resulting
publicly available HEADLINES dataset is significantly larger than most
existing semantic similarity datasets and covers a much longer span of time.
It will facilitate the application of contrastively trained semantic
similarity models to a variety of tasks, including the study of semantic
change across space and time.

##  1 Introduction

Transformer language models contrastively trained on large-scale semantic
similarity datasets are integral to a variety of applications in natural
language processing (NLP). Contrastive training is often motivated by the
anisotropic geometry of pre-trained transformer language models like BERT and
GPT [7], which complicates working with their hidden representations.
Representations of low frequency words are pushed outwards, the sparsity of
low frequency words violates convexity, and the distance between embeddings is
correlated with lexical similarity. This leads to poor alignment between
semantically similar texts and poor performance when individual term
representations from the transformer model are pooled to create a
representation for longer texts, since pooling assumes convexity. Contrastive
training reduces anisotropy, increasing uniformity of the distribution of
features on the hypersphere and improving alignment of features from positive
pairs [26]. This in turn significantly improves pooled representations of
sentences (or longer documents) [23].

A variety of semantic similarity datasets have been used for contrastive
training. Many of these datasets are relatively small, and the bulk of the
larger datasets are created from recent web texts; e.g., positive pairs are
drawn from the texts in an online comment thread or from questions marked as
duplicates in a forum. To fill important gaps in the suite of existing
datasets, this study develops HEADLINES (Historical Enormous-Scale Abstractive
DupLIcate News Summaries), a massive dataset containing nearly 400M high
quality semantic similarity pairs drawn from 70 years of off copyright U.S.
newspapers. Historically, around half of content in the many thousands of
local newspapers across the U.S. was taken from centralized sources such as
the Associated Press [11]. Local newspapers reprinted wire articles but wrote
their own headlines, which form abstractive summaries of the articles.
Headlines written by different papers to describe the same wire article form
positive semantic similarity pairs.

To construct HEADLINES, we digitize front pages of off-copyright newspaper
page scans, localizing and OCRing individual content regions like headlines
and articles. The headlines, bylines, and article texts that form full
articles span multiple bounding boxes - often arranged with complex layouts -
and we associate them using a model that combines layout information and
language understanding [20]. Then, we use neural methods from [24] to
accurately predict which articles come from the same underlying source, in the
presence of noise and abridgement.

Figure 1: Semantic similarity pairs by year.

HEADLINES allows us to leverage the collective writings of many thousands of
local editors across the U.S., spanning much of the 20th century, to create a
massive, high-quality dataset that captures semantic similarity across time.
The dataset contains 393,635,650 positive semantic similarity pairs, spanning
the period between 1920 and 1989 and covering 49 of the 50 states (Figure 1).
The dataset could be significantly expanded by extending the pipeline to
interior newspaper pages, which also published extensive syndicated content.
We do not do so at this time, as the dataset is already massive.

Relative to existing datasets, HEADLINES is very large, covering a vast array
of topics. This makes it useful generally speaking for semantic similarity
pre-training. It also covers a long period of time, making it a rich training
data source for the study of historical texts and semantic change. It captures
semantic similarity directly, as the positive pairs summarize the same
underlying texts.

This study is organized as follows. Section 2 describes HEADLINES, and Section
3 discusses how it relates to existing datasets. Section 4 outlines the
methods used for dataset construction, and Section 5 concludes.

##  2 Dataset Description

HEADLINES contains 393,635,650 positive headline pairs from off-copyright
newspapers. Figure 2 shows examples of semantic similarity pairs, which
include noise from the OCR. We do not post-process the headlines with
spellchecking, to provide users maximal flexibility in choosing how aggressive
they wish to be with spellchecking.

Figure 2: Semantic similarity examples, showing article image crops and OCR’ed
headlines.

Figure 1 plots the number of headline pairs by year, illustrating coverage
across time from 1920 to 1989. Content declines sharply in the late 1970s, due
to a major copyright law change effective January 1, 1978.

Figure 3 plots the distribution of content by state, illustrating national
coverage. Newspapers covered a vast array of topics, leading to strong
coverage across the vocabulary.

Figure 3: Geographic variation in source of headlines

Content are drawn from off-copyright newspapers. Earlier in the 20th century,
it was rare for papers outside those with the widest circulation to include a
copyright notice (or to renew copyrights if published with notice), required
formalities at the time. Individual pieces of syndicated content in off-
copyright papers could be copyrighted: most frequently comics, occasionally
ads - these are detected by our layout model and we do not OCR either - and
occasional runs of syndicated fiction. News wire articles were not copyrighted
[21], as renewing a copyright on yesterday’s news would have been costly with
no commercial value, and in any case our dataset exploits locally written
headlines. Headlines are a small share of overall text content.

HEADLINES has a Creative Commons CC-BY license, to encourage widespread use,
and is available on Huggingface.111https://huggingface.co/datasets/dell-
research-harvard/headlines-semantic-similarity

##  3 Existing Semantic Similarity Datasets

There is a dense literature on semantic similarity, with datasets covering
diverse types of textual similarity and varying greatly in size. The focus of
HEADLINES on semantic similarity in historical texts sets it apart from other
widely used datasets. It also dwarfs the size of most existing datasets,
aggregating the collective work of 20th century newspapers editors, from towns
across the U.S. Its paired headlines summarize the same text, rather than
being related by other forms of similarity frequently captured by datasets,
such as being in the same conversation thread or answering a corresopnding
question.

Amongst the most closely related semantic similarity datasets to HEADLINES are
Microsoft COCO [4] and Flickr 30k [28].222Dataset sizes are drawn, when
applicable, from a table documenting the training of Sentence BERT [23]. MS
COCO used Amazon’s Mechanical Turk to collect five captions for each image in
the dataset, resulting in around 828,000 positive caption pairs. In Flickr,
around 152,000 crowd-sourced captions describing around 32,000 images yield
317,695 positive semantic similarity pairs. Like HEADLINES, positive pairs in
these datasets describe the same underlying content, but are describing an
image rather than providing an abstractive summary of a longer text. The
nearly 400M semantic similarity pairs in HEADLINES significantly exceed the
size of these datasets, which are used for a much broader variety of tasks
including object detection and semantic segmentation. In future work,
HEADLINES could be expanded to include caption pairs describing the same
underlying photo wire image, as local papers frequently wrote their own
captions. We have developed a pipeline for detecting reproduced images [1]
that would yield hundreds of millions of positive semantic similarity pairs
describing images rather than texts. We do not deploy it for this iteration of
HEADLINES, since the dataset already contains nearly 400M positive pairs and
processing this content is costly.

Another related class of semantic similarity datasets consists of duplicate
questions from web platforms. These datasets include questions tagged by users
as duplicates from WikiAnswers (77.4 million positive pairs) [8], duplicate
stack exchange questions (around 304,000 duplicate title pairs) [2], and
duplicate quora questions (around 400,000 duplicate pairs) [15]. These
datasets are created from recent web texts, and hence cannot be used to study
semantic change across long spans of time. They also do not contain systematic
information about the geographic location of the author.

Online comment threads have also been used to train semantic similarity
models. For example, the massive scale Reddit Comments [14] draws positive
semantic similarity pairs from Reddit conversation threads between 2016 and
2018, providing 726.5 million positive pairs. Semantic similarity between
comments in an online thread reflects conversational similarity, to the extent
the thread stays on topic, rather than abstractive similarity, making
HEADLINES a complement.

While other datasets exploit abstractive summaries, to our knowledge there are
not large-scale datasets with abstractive summaries of the same underlying
texts. Semantic Scholar (S2ORC), a database of academic articles, has been
used to create semantic similarity pairs of the titles and abstracts of papers
that cite each other. For instance, Sentence BERT [23] trains on 52.6 million
positive citation abstract pairs constructed from S2ORC, capturing topic
similarity.

Finally, question-answer and natural-language inference datasets are widely
used for semantic similarity training. These include 65 million question-
answer pairs from PAQ [19], 9 million MS MARCO question-answer triples [5], 3
million question answer pairs for GOOAQ [17], 1.2 million title-answer pairs
from Yahoo Answers [29], 582,000 question-answer pairs from SearchQA [6],
100,000 question-answer pairs from Natural Questions [18], 87,000 question-
answer pairs from SQuAD [22], and 277,000 NLI pairs from SNLI [3] and MultiNLI
[27]. These datasets capture the semantic similarity between questions and
answers or hypotheses and premises - rather than abstractive similarity -
again making them complements to HEADLINES.

##  4 Dataset Construction

We digitized front pages from off-copyright newspapers spanning 1920-1989. We
recognize layouts using Mask RCNN [13] and use Tesseract to OCR the texts.

After digitization, we associate multiple bounding boxes comprising the same
article. We associate the (potentially multiple) headline bounding boxes with
the (potentially multiple) article bounding boxes and byline boxes using a
combination of layout information and language understanding.

(a) The green article is not-contiguous and it is unclear if the second part
continues the purple, green, or even blue articles if the text is not used.

(b) The green article is nested inside the blue article, which causes rule-
based and image-based models to either associate the green and blue articles
together, or miss the second half of the blue article.

(c) The smallest rectangle that contains the green article also contains the
purple and red articles.

Figure 4:  Articles that are misassociated with rule-based or image-based
methods

A rule-based approach using the document layouts gets many of the associations
correct, but misses some difficult cases where article bounding boxes are
arranged in complex layouts, with Figure 4 providing representative examples.
Language understanding can be used to associate such articles but must be
robust to noise, from errors in layout detection (e.g. from cropping part of a
content bounding box or adding part of the line below), from errors in
Tesseract’s rule-based line detection algorithm, and from OCR character
recognition errors.

(a) Article bounding boxes that are under the same headline and at the bottom
of the page are used to create training data.

(b) At inference time, all bounding boxes that are directly under the same
headline are associated with each other.

(c) Any other article with a headline directly above it is not compared,
leaving only a few orphans that are left to be associated.

(d) This orphan (green bounding box) is compared with the bounding box above
it. In this case, it is a separate article without a headline so it is not
associated.

(e) Orphans are also compared with bounding boxes to the right. This orphan
(green bounding box) is associated with the bounding box directly above.

(f) The final orphan is compared with two columns to the right, as sometimes
articles skip columns. If further columns to the right exist, they are not
compared.

Figure 5: Illustration of article association pipeline

Hand annotating a sufficiently large training dataset would have been too
costly. Instead, we devise a set of rules that - while recall is relatively
low - have nearly perfect precision, as measured on an evaluation set. The
algorithm exploits the positioning of article bounding boxes relative to
headline boxes (as in Figure 5), first grouping an article bounding box with a
headline bounding box if the rules are met and then associating all article
bounding boxes grouped with the same headline together. Since precision is
above 0.99, the rule can be used to generate silver-quality training data.

To train a classifier on these data to predict whether article box B follows
box A, we embed the first and last 64 tokens of the texts in boxes B and A,
respectively, with a RoBERTa base model [20]. The pair is positive when B
follows A. The training set includes 12,769 positive associated pairs, with
training details described in the supplementary materials.

At inference time, we first associate texts using the rule-based approach,
described in Figure 5, which has extremely high precision. To improve recall,
we then apply the RoBERTa cross-encoder to remaining article boxes that could
plausibly be associated, given their coordinates. Texts cannot be followed by
a text that appears to the left by a sufficiently wide margin, as layouts -
while sometimes complex - proceed from left to right. Hence, these
combinations are not considered.

Table 1 shows recall, precision and F1 for associated articles when the neural
model is used in combination with the rule based approach. The F1 of 93.7 is
high, and precision is extremely high. Errors typically occur when there is an
error in the layout analysis model or when contents are very similar, for
instance, grouping multiple obituaries into a single article.

| (1) | (2) | (3)
---|---|---|---
| F1 | Recall | Precision
Full Article Association | 93.7 | 88.3 | 99.7

Table 1: This table evaluates the full article association model, showing F1,
recall, and precision, respectively.

Accurately detecting reproduced content can be challenging, as articles were
often heavily abridged by local papers to fit within their space constraints
and errors in OCR or article association can add significant noise. Table 2
shows examples of reproduced articles.

|  SAN FRANCISCO (# — Presi-dent Eisenhower was_ reported ready to open the U. N.’s 10th anniversary session today with an important policy declaration. This word came from informed sources as the Big Four foreign ministers prepared for a private huddle tonight to plan the meeting of their chiefs of government in Geneva July 18. The President was scheduled to speak at 6 p.m., EDT. Some diplomats believed his speech would have special significance, coming as it does just a month before the top level talks. The first direct contact between Soviet Foreign Minister V. M. Molotov and the Western leaders was established last night at a dinner given by Colombian Ambassador Eduardo Zuleta Angel There was no information as tc whether any serious discussion: were held, but the two sides dic have a chance to sense the at-’}mosphere which will prevail to-’| night..| Diplomatic sources reported th« Russians likely would press for : ’Ideclaration of some sort by the | 10th anniversary meeting |  |  SAN FRANCISCO #—President Eisenhower was reported ready to open the U_N.’s 10th anniversary session today with an important policy declaration. This word came from informed sources as the Big Four foreign ministers prepared for a private huddle tonight to plan the meeting of their chiefs of government in Geneva July 18. Some diplomats believed Eisenhower’s speech would have special significance, coming as it does just ’a month before the top-level talks.| The first direct contact between (Soviet Foreign Minister V. M.| Molotov and the Western leaders | was established last night “at a’dinner given by Colombian Am| bassador Eduardo Zulcta Angel. | Diplomatic sources reported the ’Russians likely would press for a declaration of some sort by the 110th anniversary meeting. |  |  "SY MAA MARKRKRELSUN SAN FRANCISCO #®—President Eisenhower was reported ready to open the U.N.’s 10th anniversary session today with an importan! policy declaration, This word came from informed sources as the Big Four foreign ministers prepared for a private huddle tonight to plan the meeting of their chiefs of government in Geneva July 18. Some diplomats believed Eisen.hower’s speech would have special! significance, coming as it does just a month before the top-level talks. Mulles to Bo Host The first direct contact between Soviet Foreign Minister V. M. Molotov and the Western leaders was established last night at a dinner given by Colombian Ambassador Eduardo Zuleta Angel. Diplomatic sources reported the Russians likely would press for a ’declaration of some sort by the 10th anniversary meeting.
---|---|---|---|---|---
Table 2: Examples of reproduced articles. Additions are highlighted, and OCR
errors are underlined.

We use the model developed by [24], who show that a contrastively trained
neural MPNet bi-encoder - combined with single linkage clustering of article
representations - accurately and cheaply detects reproduced content. This bi-
encoder is contrastively trained on a hand-labeled dataset to create similar
representations of articles from the same wire source and dissimilar
representations of articles from different underlying sources, using S-BERT’s
online contrastive loss [12] implementation.

On a labeled sample of all front page articles appearing in our newspaper
corpus for three days in the 1930s and 1970s, the neural bi-encoder methods
achieve an adjusted rand index of 91.5, compared to 73.7 for an optimal local
sensitive hashing specification chosen on the validation set (another full-day
labeled sample) [24]. The pipeline can be scaled to 10 million articles on a
single GPU in a matter of hours, using a FAISS [16] backend. Errors typically
consist of articles about the same story from different news wire services
(e.g. the Associated Press and the United Press) or updates to a story as new
events unfolded. Both types of errors will plausibly still lead to informative
semantic similarity pairs.

We run clustering on the article embeddings by year over all years in our
sample. In post-processing, we use a simple set of rules exploiting the dates
of articles within clusters to remove content like weather forecasts and legal
notices, that are highly formulaic and sometimes cluster together when they
contain very similar content (e.g. a similar 5 day forecast) but did not
actually come from the same underlying source. We remove all headline pairs
that are below a Levenshtein edit distance, normalized by the min length in
the pair, of 0.1, to remove pairs that are exact duplicates up to OCR noise.
More details are provided in the supplementary materials.

The digitization of newspaper scans was performed using Azure F-Series CPU
nodes. Full article association and detection of reproduced content were
performed on an A6000 GPU card. All training was conducted on an A6000 GPU
card.

##  5 Conclusion

HEADLINES provides a massive-scale semantic similarity dataset, consisting of
nearly 400M positive semantic similarity pairs. It combines the collective
work of thousands of local newspaper editors across much of the 20th century,
providing rich data that can be useful more generally for contrastively
training large language models and specifically for studying semantic change.
HEADLINES forms part of a broader agenda to create more resources for
deploying large language models on historical texts.

## Supplementary Materials

##  S-1 Methods to Associate Articles

To detect whether article bounding box B follows box A, we train a cross-
encoder using a RoBERTa base model [20]. Figure 5, Panel A in the main text
describes how positive pairs are created. For hard negatives, we used article
boxes under the same headline in reverse reading order (right to left). For
standard negatives, we took pairs of articles on the same page, where B was
above and to the left of A, as articles do not read from right to left. One
twelfth of our training data were positive pairs, another twelfth were hard
negative pairs and the remainder were standard negative pairs. This
outperformed a more balanced training sample.

We used a Bayesian search algorithm [9] to find optimal hyperparameters on one
tenth of our training data (limited compute prevented us from running this
search with the full dataset), which led to a learning rate of 1.7e-5, with a
batch size of 64 and 29.57% warm up. We trained for 26 epochs with an AdamW
optimizer, and optimize a binary cross-entropy loss.

##  S-2 Methods to Detect Reproduced Content

To detect reproduced content, we use the contrastively trained bi-encoder
model developed by [24], which is trained to learn similar representations for
reproduced articles and dissimilar representations for non-reproduced
articles. This model is based on an S-BERT MPNET model [23, 25] and is fine-
tuned on a hand-labelled dataset of articles from the same underlying wire
source, using S-BERT’s online contrastive loss [12] implementation, with a 0.2
margin and cosine similarity as the distance metric. The learning rate is 2e-5
with 100% warm up and a batch size of 32. It uses an AdamW optimizer, and the
model is trained for 16 epochs.

To create clusters from the bi-encoder embeddings, we use highly scalable
single-linkage clustering, with a cosine similarity threshold of 0.94. We
build a graph using articles as nodes, and add edges if the cosine similarity
is above this threshold. As edge weights we use the negative exponential of
the difference in dates (in days) between the two articles. We then apply
Leiden community detection to the graph to control false positive edges that
can otherwise merge disparate groups of articles.

We further remove clusters that have over 50 articles and contain articles
with greater than five different dates. We also remove clusters that contain
over 50 articles, when the number of articles is more than double the number
of unique newspapers from which these articles are sourced. This removes
clusters of content that are correctly clustered in the sense of being based
on the same underlying source, but are not useful for the HEADLINES dataset.
For example, an advertisement (misclassified as an article due to an article-
like appearance) might be repeated by the same newspaper on multiple different
dates and would be removed by these rules, or weather forecasts can be very
near duplicates across space and time, forming large clusters.

##  S-3 Dataset Information

###  S-3.1 Dataset URL

HEADLINES can be found at https://huggingface.co/datasets/dell-research-
harvard/headlines-semantic-similarity.

This dataset has structured metadata following schema.org, and is readily
discoverable.333See https://search.google.com/test/rich-
results/result?id=_HKjxIv-LaF_8ElAarsM_g for full metadata.

###  S-3.2 DOI

The DOI for this dataset is: 10.57967/hf/0751.

###  S-3.3 License

HEADLINES has a Creative Commons CC-BY license.

###  S-3.4 Dataset usage

The dataset is hosted on huggingface, in json format. Each year in the dataset
is divided into a distinct file (eg. 1952_headlines.json).

The data is presented in the form of clusters, rather than pairs to eliminate
duplication of text data and minimize the storage size of the datasets.

An example from HEADLINES looks like:

    {
        "headline": "FRENCH AND BRITISH BATTLESHIPS IN MEXICAN WATERS",
        "group_id": 4
        "date": "May-14-1920",
        "state": "kansas",
    }

The data fields are:

  * •

`headline`: headline text.

  * •

`date`: the date of publication of the newspaper article, as a string in the
form mmm-DD-YYYY.

  * •

`state`: state of the newspaper that published the headline.

  * •

`group_id`: a number that is shared with all other headlines for the same
article. This number is unique across all year files.

The whole dataset can be easily downloaded using the `datasets` library:

    from datasets import load_dataset
    dataset_dict = load_dataset("dell-research-harvard/headlines-semantic-similarity")

Specific files can be downloaded by specifying them:

    from datasets import load_dataset
    load_dataset(
        "dell-research-harvard/headlines-semantic-similarity",
        data_files=["1929_headlines.json", "1989_headlines.json"]
    )

###  S-3.5 Author statement

We bear all responsibility in case of violation of rights.

###  S-3.6 Maintenance Plan

We have chosen to host HEADLINES on huggingface as this ensures long-term
access and preservation of the dataset.

###  S-3.7 Dataset documentation and intended uses

We follow the datasheets for datasets template [10].

####  S-3.7.1 Motivation

##### For what purpose was the dataset created?

Was there a specific task in mind? Was there a specific gap that needed to be
filled? Please provide a description.

Transformer language models contrastively trained on large-scale semantic
similarity datasets are integral to a variety of applications in natural
language processing (NLP). A variety of semantic similarity datasets have been
used for this purpose, with positive text pairs related to each other in some
way. Many of these datasets are relatively small, and the bulk of the larger
datasets are created from recent web texts; e.g. positives are drawn from the
texts in an online comment thread or duplicate questions in a forum. Relative
to existing datasets, HEADLINES is very large, covering a vast array of
topics. This makes it useful generally speaking for semantic similarity pre-
training. It also covers a long period of time, making it a rich training data
source for the study of historical texts and semantic change. It captures
semantic similarity directly, as the positive pairs summarize the same
underlying texts.

##### Who created this dataset (e.g., which team, research group) and on
behalf of which entity (e.g., company, institution, organization)?

HEADLINES was created by Melissa Dell and Emily Silcock, at Harvard
University.

##### Who funded the creation of the dataset?

If there is an associated grant, please provide the name of the grantor and
the grant name and number.

The creation of the dataset was funded by the Harvard Data Science Initiative,
Harvard Catalyst, and compute credits provided by Microsoft Azure to the
Harvard Data Science Initiative.

##### Any other comments?

None.

####  S-3.7.2 Composition

##### What do the instances that comprise the dataset represent (e.g.,
documents, photos, people, countries)?

Are there multiple types of instances (e.g., movies, users, and ratings;
people and interactions between them; nodes and edges)? Please provide a
description.

HEADLINES comprises instances of newspaper headlines and relationships between
them. Specifically, each headline includes information on the text of the
headline, the date of publication, and the state it was published in.
Headlines have relationships between them if they are semantic similarity
pairs, that is, if they two different headlines for the same newspaper
article.

##### How many instances are there in total (of each type, if appropriate)?

HEADLINES contains 34,867,488 different headlines and 396,001,930 positive
relationships between headlines.

##### Does the dataset contain all possible instances or is it a sample (not
necessarily random) of instances from a larger set?

If the dataset is a sample, then what is the larger set? Is the sample
representative of the larger set (e.g., geographic coverage)? If so, please
describe how this representativeness was validated/verified. If it is not
representative of the larger set, please describe why not (e.g., to cover a
more diverse range of instances, because instances were withheld or
unavailable).

Many local newspapers were not preserved, and newspapers with the widest
circulation tended to renew their copyrights, so cannot be included.

##### What data does each instance consist of? “Raw” data (e.g., unprocessed
text or images) or features?

In either case, please provide a description.

Each data instance consists of raw data. Specifically, an example from
HEADLINES is:

    {
        "headline": "FRENCH AND BRITISH BATTLESHIPS IN MEXICAN WATERS",
        "group_id": 4
        "date": "May-14-1920",
        "state": "kansas",
    }

The data fields are:

  * •

`headline`: headline text.

  * •

`date`: the date of publication of the newspaper article, as a string in the
form mmm-DD-YYYY.

  * •

`state`: state of the newspaper that published the headline.

  * •

`group_id`: a number that is shared with all other headlines for the same
article. This number is unique across all year files.

##### Is there a label or target associated with each instance?

If so, please provide a description.

Each instance contains a `group_id` as mentioned directly above. This is a
number that is shared by all other instances that are positive semantic
similarity pairs.

##### Is any information missing from individual instances?

If so, please provide a description, explaining why this information is
missing (e.g., because it was unavailable). This does not include
intentionally removed information, but might include, e.g., redacted text.

In some cases, the state of publication is missing, due to incomplete
metadata.

##### Are relationships between individual instances made explicit (e.g.,
users’ movie ratings, social network links)?

If so, please describe how these relationships are made explicit.

Relationships between instances are made explicit in the `group_id` variable,
as detailed above.

##### Are there recommended data splits (e.g., training,
development/validation, testing)?

If so, please provide a description of these splits, explaining the rationale
behind them.

There are no recommended splits.

##### Are there any errors, sources of noise, or redundancies in the dataset?

If so, please provide a description.

The data is sourced from OCR’d text of historical newspapers. Therefore some
of the headline texts contain OCR errors.

##### Is the dataset self-contained, or does it link to or otherwise rely on
external resources (e.g., websites, tweets, other datasets)?

If it links to or relies on external resources, a) are there guarantees that
they will exist, and remain constant, over time; b) are there official
archival versions of the complete dataset (i.e., including the external
resources as they existed at the time the dataset was created); c) are there
any restrictions (e.g., licenses, fees) associated with any of the external
resources that might apply to a future user? Please provide descriptions of
all external resources and any restrictions associated with them, as well as
links or other access points, as appropriate.

The data is self-contained.

##### Does the dataset contain data that might be considered confidential
(e.g., data that is protected by legal privilege or by doctor-patient
confidentiality, data that includes the content of individuals non-public
communications)?

If so, please provide a description.

The dataset does not contain information that might be viewed as confidential.

##### Does the dataset contain data that, if viewed directly, might be
offensive, insulting, threatening, or might otherwise cause anxiety?

If so, please describe why.

The headlines in the dataset reflect diverse attitudes and values from the
period in which they were written, 1920-1989, and contain content that may be
considered offensive for a variety of reasons.

##### Does the dataset relate to people?

If not, you may skip the remaining questions in this section.

Many news articles are about people.

##### Does the dataset identify any subpopulations (e.g., by age, gender)?

If so, please describe how these subpopulations are identified and provide a
description of their respective distributions within the dataset.

The dataset does not specifically identify any subpopulations.

##### Is it possible to identify individuals (i.e., one or more natural
persons), either directly or indirectly (i.e., in combination with other data)
from the dataset?

If so, please describe how.

If an individual appeared in the news during this period, then headline text
may contain their name, age, and information about their actions.

##### Does the dataset contain data that might be considered sensitive in any
way (e.g., data that reveals racial or ethnic origins, sexual orientations,
religious beliefs, political opinions or union memberships, or locations;
financial or health data; biometric or genetic data; forms of government
identification, such as social security numbers; criminal history)?

If so, please provide a description.

All information that it contains is already publicly available in the
newspapers used to create the headline pairs.

##### Any other comments?

None.

####  S-3.7.3 Collection Process

##### How was the data associated with each instance acquired?

Was the data directly observable (e.g., raw text, movie ratings), reported by
subjects (e.g., survey responses), or indirectly inferred/derived from other
data (e.g., part-of-speech tags, model-based guesses for age or language)? If
data was reported by subjects or indirectly inferred/derived from other data,
was the data validated/verified? If so, please describe how.

To create HEADLINES, we digitized front pages from off-copyright newspapers
spanning 1920-1989. Historically, around half of articles in U.S. local
newspapers came from newswires like the Associated Press. While local papers
reproduced articles from the newswire, they wrote their own headlines, which
form abstractive summaries of the associated articles. We associate articles
and their headlines by exploiting document layouts and language understanding.
We then use deep neural methods to detect which articles are from the same
underlying source, in the presence of substantial noise and abridgement. The
headlines of reproduced articles form positive semantic similarity pairs.

##### What mechanisms or procedures were used to collect the data (e.g.,
hardware apparatus or sensor, manual human curation, software program,
software API)?

How were these mechanisms or procedures validated?

These methods are described in detail in the main text and supplementary
materials of this paper.

##### If the dataset is a sample from a larger set, what was the sampling
strategy (e.g., deterministic, probabilistic with specific sampling
probabilities)?

The dataset was not sampled from a larger set.

##### Who was involved in the data collection process (e.g., students,
crowdworkers, contractors) and how were they compensated (e.g., how much were
crowdworkers paid)?

We used student annotators to create the validation sets for associating
bounding boxes, and the training and validation sets for clustering duplicated
articles. They were paid $15 per hour, a rate set by a Harvard economics
department program providing research assistantships for undergraduates.

##### Over what timeframe was the data collected? Does this timeframe match
the creation timeframe of the data associated with the instances (e.g., recent
crawl of old news articles)?

If not, please describe the timeframe in which the data associated with the
instances was created.

The headlines were written between 1920 and 1989. Semantic similarity pairs
were computed in 2023.

##### Were any ethical review processes conducted (e.g., by an institutional
review board)?

If so, please provide a description of these review processes, including the
outcomes, as well as a link or other access point to any supporting
documentation.

No, this dataset uses entirely public information and hence does not fall
under the domain of Harvard’s institutional review board.

##### Does the dataset relate to people?

If not, you may skip the remaining questions in this section.

Historical newspapers contain a variety of information about people.

##### Did you collect the data from the individuals in question directly, or
obtain it via third parties or other sources (e.g., websites)?

The data were obtained from off-copyright historical newspapers.

##### Were the individuals in question notified about the data collection?

If so, please describe (or show with screenshots or other information) how
notice was provided, and provide a link or other access point to, or otherwise
reproduce, the exact language of the notification itself.

Individuals were not notified; the data came from publicly available
newspapers.

##### Did the individuals in question consent to the collection and use of
their data?

If so, please describe (or show with screenshots or other information) how
consent was requested and provided, and provide a link or other access point
to, or otherwise reproduce, the exact language to which the individuals
consented.

The dataset was created from publicly available historical newspapers.

##### If consent was obtained, were the consenting individuals provided with a
mechanism to revoke their consent in the future or for certain uses?

If so, please provide a description, as well as a link or other access point
to the mechanism (if appropriate).

Not applicable.

##### Has an analysis of the potential impact of the dataset and its use on
data subjects (e.g., a data protection impact analysis) been conducted?

If so, please provide a description of this analysis, including the outcomes,
as well as a link or other access point to any supporting documentation.

No.

##### Any other comments?

None.

####  S-3.7.4 Preprocessing/cleaning/labeling

##### Was any preprocessing/cleaning/labeling of the data done (e.g.,
discretization or bucketing, tokenization, part-of-speech tagging, SIFT
feature extraction, removal of instances, processing of missing values)?

If so, please provide a description. If not, you may skip the remainder of the
questions in this section.

See the description in the main text.

##### Was the “raw” data saved in addition to the preprocessed/cleaned/labeled
data (e.g., to support unanticipated future uses)?

If so, please provide a link or other access point to the “raw” data.

No.

##### Is the software used to preprocess/clean/label the instances available?

If so, please provide a link or other access point.

No specific software was used to clean the instances.

##### Any other comments?

None.

####  S-3.7.5 Uses

##### Has the dataset been used for any tasks already?

If so, please provide a description.

No.

##### Is there a repository that links to any or all papers or systems that
use the dataset?

If so, please provide a link or other access point.

No.

##### What (other) tasks could the dataset be used for?

The dataset can be used for training models for semantic similarity, studying
language change over time and studying difference in language across space.

##### Is there anything about the composition of the dataset or the way it was
collected and preprocessed/cleaned/labeled that might impact future uses?

For example, is there anything that a future user might need to know to avoid
uses that could result in unfair treatment of individuals or groups (e.g.,
stereotyping, quality of service issues) or other undesirable harms (e.g.,
financial harms, legal risks) If so, please provide a description. Is there
anything a future user could do to mitigate these undesirable harms?

The dataset contains historical news headlines, which will reflect current
affairs and events of the time period in which they were created, 1920-1989,
as well as the biases of this period.

##### Are there tasks for which the dataset should not be used?

If so, please provide a description.

It is intended for training semantic similarity models and studying semantic
variation across space and time.

##### Any other comments?

None

####  S-3.7.6 Distribution

##### Will the dataset be distributed to third parties outside of the entity
(e.g., company, institution, organization) on behalf of which the dataset was
created?

If so, please provide a description.

Yes. The dataset is available for public use.

##### How will the dataset will be distributed (e.g., tarball on website, API,
GitHub)

Does the dataset have a digital object identifier (DOI)?

The dataset is hosted on huggingface. Its DOI is 10.57967/hf/0751.

##### When will the dataset be distributed?

The dataset was distributed on 7th June 2023.

##### Will the dataset be distributed under a copyright or other intellectual
property (IP) license, and/or under applicable terms of use (ToU)?

If so, please describe this license and/or ToU, and provide a link or other
access point to, or otherwise reproduce, any relevant licensing terms or ToU,
as well as any fees associated with these restrictions.

The dataset is distributed under a Creative Commons CC-BY license. The terms
of this license can be viewed at https://creativecommons.org/licenses/by/2.0/

##### Have any third parties imposed IP-based or other restrictions on the
data associated with the instances?

If so, please describe these restrictions, and provide a link or other access
point to, or otherwise reproduce, any relevant licensing terms, as well as any
fees associated with these restrictions.

There are no third party IP-based or other restrictions on the data.

##### Do any export controls or other regulatory restrictions apply to the
dataset or to individual instances?

If so, please describe these restrictions, and provide a link or other access
point to, or otherwise reproduce, any supporting documentation.

No export controls or other regulatory restrictions apply to the dataset or to
individual instances.

##### Any other comments?

None.

####  S-3.7.7 Maintenance

##### Who will be supporting/hosting/maintaining the dataset?

The dataset is hosted on huggingface.

##### How can the owner/curator/manager of the dataset be contacted (e.g.,
email address)?

The recommended method of contact is using the huggingface ‘community’
capacity. Additionally, Melissa Dell can be contacted at
<EMAIL_ADDRESS>
##### Is there an erratum?

If so, please provide a link or other access point.

There is no erratum.

##### Will the dataset be updated (e.g., to correct labeling errors, add new
instances, delete instances)?

If so, please describe how often, by whom, and how updates will be
communicated to users (e.g., mailing list, GitHub)?

We have no plans to update the dataset. If we do, we will notify users via the
huggingface Dataset Card.

##### If the dataset relates to people, are there applicable limits on the
retention of the data associated with the instances (e.g., were individuals in
question told that their data would be retained for a fixed period of time and
then deleted)?

If so, please describe these limits and explain how they will be enforced.

There are no applicable limits on the retention of data.

##### Will older versions of the dataset continue to be
supported/hosted/maintained?

If so, please describe how. If not, please describe how its obsolescence will
be communicated to users.

We have no plans to update the dataset. If we do, older versions of the
dataset will not continue to be hosted. We will notify users via the
huggingface Dataset Card.

##### If others want to extend/augment/build on/contribute to the dataset, is
there a mechanism for them to do so?

If so, please provide a description. Will these contributions be
validated/verified? If so, please describe how. If not, why not? Is there a
process for communicating/distributing these contributions to other users? If
so, please provide a description.

Others can contribute to the dataset using the huggingface ‘community’
capacity. This allows for anyone to ask questions, make comments and submit
pull requests. We will validate these pull requests. A record of public
contributions will be maintained on huggingface, allowing communication to
other users.

##### Any other comments?

None.

## References

  * [1] Arora, A., Yang, X., Jheng, S. Y., and Dell, M. Linking representations with multimodal contrastive learning.  arXiv preprint arXiv:2304.03464 (2023).
  * [2] BERT, S. stackexchange, 2021.
  * [3] Bowman, S. R., Angeli, G., Potts, C., and Manning, C. D. A large annotated corpus for learning natural language inference.  arXiv preprint arXiv:1508.05326 (2015).
  * [4] Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollár, P., and Zitnick, C. L. Microsoft coco captions: Data collection and evaluation server.  arXiv preprint arXiv:1504.00325 (2015).
  * [5] Craswell, N., Mitra, B., Yilmaz, E., Campos, D., and Lin, J. Ms marco: Benchmarking ranking models in the large-data regime.  In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (2021), pp. 1566–1576.
  * [6] Dunn, M., Sagun, L., Higgins, M., Guney, V. U., Cirik, V., and Cho, K. Searchqa: A new q&a dataset augmented with context from a search engine.  arXiv preprint arXiv:1704.05179 (2017).
  * [7] Ethayarajh, K. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings.  arXiv preprint arXiv:1909.00512 (2019).
  * [8] Fader, A., Zettlemoyer, L., and Etzioni, O. Open question answering over curated and extracted knowledge bases.  In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining (2014), pp. 1156–1165.
  * [9] Falkner, S., Klein, A., and Hutter, F. BOHB: Robust and efficient hyperparameter optimization at scale.  In Proceedings of the 35th International Conference on Machine Learning (10–15 Jul 2018), J. Dy and A. Krause, Eds., vol. 80 of Proceedings of Machine Learning Research, PMLR, pp. 1437–1446.
  * [10] Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., au2, H. D. I., and Crawford, K. Datasheets for datasets, 2021.
  * [11] Guarneri, J. Newsprint Metropolis.  University of Chicago Press, 2017.
  * [12] Hadsell, R., Chopra, S., and LeCun, Y. Dimensionality reduction by learning an invariant mapping.  In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06) (2006), vol. 2, IEEE, pp. 1735–1742.
  * [13] He, K., Gkioxari, G., Dollár, P., and Girshick, R. Mask r-cnn.  Proceedings of the IEEE international conference on computer vision (2017), 2961–2969.
  * [14] Henderson, M., Budzianowski, P., Casanueva, I., Coope, S., Gerz, D., Kumar, G., Mrkšić, N., Spithourakis, G., Su, P.-H., Vulić, I., et al. A repository of conversational datasets.  In Proceedings of the First Workshop on NLP for Conversational AI (2019), pp. 1–10.
  * [15] Iyer, S., Dandekar, N., and Csernai, K. First quora dataset release: Question pairs, 2017.
  * [16] Johnson, J., Douze, M., and Jégou, H. Billion-scale similarity search with gpus.  IEEE Transactions on Big Data 7, 3 (2019), 535–547.
  * [17] Khashabi, D., Ng, A., Khot, T., Sabharwal, A., Hajishirzi, H., and Callison-Burch, C. Gooaq: Open question answering with diverse answer types.  arXiv preprint arXiv:2104.08727 (2021).
  * [18] Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. Natural questions: a benchmark for question answering research.  Transactions of the Association for Computational Linguistics 7 (2019), 453–466.
  * [19] Lewis, P., Wu, Y., Liu, L., Minervini, P., Küttler, H., Piktus, A., Stenetorp, P., and Riedel, S. Paq: 65 million probably-asked questions and what you can do with them.  Transactions of the Association for Computational Linguistics 9 (2021), 1098–1115.
  * [20] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692 (2019).
  * [21] Ockerbloom, J. M. Everybody’s library questions: Newspaper copyrights, notices, and renewals, 2019.
  * [22] Rajpurkar, P., Jia, R., and Liang, P. Know what you don’t know: Unanswerable questions for squad.  arXiv preprint arXiv:1806.03822 (2018).
  * [23] Reimers, N., and Gurevych, I. Sentence-bert: Sentence embeddings using siamese bert-networks.  arXiv preprint arXiv:1908.10084 (2019).
  * [24] Silcock, E., D’Amico-Wong, L., Yang, J., and Dell, M. Noise-robust de-duplication at scale.  International Conference on Learning Representations (2023).
  * [25] Song, K., Tan, X., Qin, T., Lu, J., and Liu, T.-Y. Mpnet: Masked and permuted pre-training for language understanding.  Advances in Neural Information Processing Systems 33 (2020), 16857–16867.
  * [26] Wang, T., and Isola, P. Understanding contrastive representation learning through alignment and uniformity on the hypersphere.  International Conference on Machine Learning 119 (2020), 9929–9939.
  * [27] Williams, A., Nangia, N., and Bowman, S. R. A broad-coverage challenge corpus for sentence understanding through inference.  arXiv preprint arXiv:1704.05426 (2017).
  * [28] Young, P., Lai, A., Hodosh, M., and Hockenmaier, J. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.  Transactions of the Association for Computational Linguistics 2 (2014), 67–78.
  * [29] Zhang, X., Zhao, J., and LeCun, Y. Character-level convolutional networks for text classification.  Advances in neural information processing systems 28 (2015).