# FigGen: Text to Scientific Figure Generation

Juan A. Rodriguez1, David Vazquez1, Issam Laradji1, Marco Pedersoli2 & Pau
Rodriguez3,*
1ServiceNow Research, 2LIVIA, ÉTS, Montréal, 3Apple Machine Learning Research.
*Work done at ServiceNow

###### Abstract

The generative modeling landscape has experienced tremendous growth in recent
years, particularly in generating natural images and art. Recent techniques
have shown impressive potential in creating complex visual compositions while
delivering impressive realism and quality. However, state-of-the-art methods
have been focusing on the narrow domain of natural images, while other
distributions remain unexplored. In this paper, we introduce the problem of
text-to-figure generation, that is creating scientific figures of papers from
text descriptions. We present FigGen, a diffusion-based approach for text-to-
figure as well as the main challenges of the proposed task. Code and models
are available at https://github.com/joanrod/figure-diffusion.

##  1 Introduction

Scientific figure generation is an important aspect of research, as it helps
to communicate findings in a concise and accessible way. The automatic
generation of figures presents numerous advantages for researchers, such as
savings in time and effort by utilizing the generated figures as a starting
point, instead of investing resources in designing figures from scratch.
Making visually appealing and understandable diagrams would allow
accessibility for a wider audience. Furthermore, exploring the generative
capabilities of models in the domain of discrete graphics would be of high
interest.

Generating figures can be a challenging task, as it involves representing
complex relationships between discrete components such as boxes, arrows, and
text, to name a few. Unlike natural images, concepts inside figures may have
diverse representations and require a fine-grained understanding. For
instance, generating a diagram of a neural network presents an ill-posed
problem with high variance, as it can be represented by a simple box or an
unfolded representation of its internal structure. Human understanding of
figures largely relies on the text rendered within the image, as well as the
support of text explanations from the paper written in technical language.

By training a generative model on a large dataset of paper-figure pairs, we
aim to capture the relationships between the components of a figure and the
corresponding text in the paper. Dealing with variable lengths and highly
technical text descriptions, different diagram styles, image aspect ratios,
and text rendering fonts, sizes, and orientations are some of the challenges
of this problem. Inspired by impressive results in text-to-image, we explore
diffusion models to generate scientific figures. Our contributions are i)
introduce the task of text-to-figure generation and ii) propose FigGen, a
latent diffusion model that generates scientific figures from text captions.

Related work. Deep learning has emerged as a powerful tool for conditional
image generation (Ramesh et al., 2022; Saharia et al., 2022; Balaji et al.,
2022), thanks to advances in techniques such as GANs (Goodfellow et al., 2014;
Karras et al., 2019; 2021) and Diffusion (Sohl-Dickstein et al., 2015; Ho et
al., 2020; Song et al., 2020b). In the domain of scientific figures, Rodriguez
et al. (2023) presented Paper2Fig100k, a large dataset of paper-figure pairs.
In this work, we aim to explore diffusion models applied to the task of text-
to-figure generation and analyze its challenges.

##  2 Method and Experiments

We train a latent diffusion model (Rombach et al., 2021) from scratch. First,
we learn an image autoencoder that projects images into a compressed latent
representation. The image encoder uses a KL loss and OCR perceptual loss
(Rodriguez et al., 2023). The text encoder used for conditioning is learned
end-to-end during the training of the diffusion model. The diffusion model
interacts directly in the latent space and performs a forward schedule of data
corruption while simultaneously learning to revert the process through a time
and text conditional denoising U-Net (Ronneberger et al., 2015) (see Appendix
A.1 for details). We use Paper2Fig100k, composed of figure-text pairs from
research papers. It consists of $81,194$ samples for training and $21,259$ for
validation.

Experimental results. During generation, we use DDIM (Song et al., 2020a)
sampler with $200$ steps and generate $~{}12,000$ samples for each model to
compute FID, IS, KID (Regenwetter et al., 2023), and OCR-
SIM111https://github.com/joanrod/ocr-vqgan. We use classifier-free guidance
(CFG) to test super-conditioning (Ho & Salimans, 2022). Table 1 presents
results of different text encoders, and Figure 1 shows generated samples of
FigGen${}_{\text{Base}}$. We find that the large text encoder offers the best
results and that we can improve conditional generation by increasing the CFG
scale. Although qualitative samples do not present sufficient quality to solve
the task, FigGen has learned interesting relationships between texts and
figures such as the difference between plots and architectures (see also
Appendix A.3).

Figure 1: Samples generated by our model using captions from Paper2Fig100k
test set.

Model | Text encoder | Parameters | CFG | FID$\downarrow$ | IS$\uparrow$ | KID$\downarrow$ | OCR-SIM$\downarrow$
---|---|---|---|---|---|---|---
FigGen${}_{\text{Base}}$ | Bert (8 layers) | 866M | 1.0 | 302.46 | 1.04 | 0.32 | 5.97
FigGen${}_{\text{Base}}$ | Bert (8 layers) | 866M | 5.0 | 282.32 | 1.09 | 0.29 | 5.89
FigGen${}_{\text{Base}}$ | Bert (8 layers) | 866M | 10.0 | 284.12 | 1.08 | 0.29 | 5.83
FigGen${}_{\text{Mid}}$ | Bert (32 layers) | 942M | 1.0 | 308.50 | 1.03 | 0.32 | 5.95
FigGen${}_{\text{Mid}}$ | Bert (32 layers) | 942M | 5.0 | 298.98 | 1.06 | 0.31 | 5.91
FigGen${}_{\text{Mid}}$ | Bert (32 layers) | 942M | 10.0 | 301.10 | 1.06 | 0.31 | 5.86
FigGen${}_{\text{Large}}$ | Bert (128 layers) | 1.2B | 1.0 | 302.99 | 1.04 | 0.32 | 6.08
FigGen${}_{\text{Large}}$ | Bert (128 layers) | 1.2B | 5.0 | 281.25 | 1.09 | 0.29 | 5.74
FigGen${}_{\text{Large}}$ | Bert (128 layers) | 1.2B | 10.0 | 288.02 | 1.09 | 0.29 | 5.76

Table 1: Main quantitative results of our text to figure generation models.

##  3 Conclusion

In this paper, we introduce the task of text-to-figure generation and define
FigGen, a latent diffusion model that we train on the Paper2Fig100k dataset.
Our experiments show that FigGen is able to learn relationships between
figures and texts and generate images that fit the distribution. However,
these generations are not ready to be useful for researchers. One of the main
challenges to solve is the variability in text and images, and how to better
align both modalities. Also, future work must design validation metrics and
loss functions for generative models of discrete objects.

Ethics statement. A central concern of this work is fake paper generation. To
address this, we consider building classifiers or using watermarks for the
detection of fake content. However, further research is needed to elucidate
how these systems should be made public.

URM Statement. The authors acknowledge that at least one key author of this
work meets the URM criteria of ICLR 2023 Tiny Papers Track.

## References

  * Balaji et al. (2022) Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al.  ediffi: Text-to-image diffusion models with an ensemble of expert denoisers.  _arXiv preprint arXiv:2211.01324_ , 2022.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Esser et al. (2020) Patrick Esser, Robin Rombach, and Björn Ommer.  Taming transformers for high-resolution image synthesis.  _arXiv_ , abs/2012.09841, 2020.
  * Goodfellow et al. (2014) Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  _arXiv_ , abs/1406.2661, 2014.
  * Ho & Salimans (2022) Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ , 2022.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ , 33:6840–6851, 2020.
  * Karras et al. (2019) Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  _arXiv_ , abs/1912.04958, 2019.
  * Karras et al. (2021) Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  _Advances in Neural Information Processing Systems_ , 34:852–863, 2021.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  _arXiv_ , abs/2103.00020, 2021.
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv_ , abs/2204.06125, 2022.
  * Regenwetter et al. (2023) Lyle Regenwetter, Akash Srivastava, Dan Gutfreund, and Faez Ahmed.  Beyond statistical similarity: Rethinking metrics for deep generative models in engineering design.  _arXiv preprint arXiv:2302.02913_ , 2023.
  * Rodriguez et al. (2023) Juan A Rodriguez, David Vazquez, Issam Laradji, Marco Pedersoli, and Pau Rodriguez.  Ocr-vqgan: Taming text-within-image generation.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pp. 3689–3698, 2023.
  * Rombach et al. (2021) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  _arXiv_ , abs/2112.10752, 2021.
  * Ronneberger et al. (2015) Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pp. 234–241. Springer, 2015.
  * Saharia et al. (2022) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding.  _arXiv_ , abs/2205.11487, 2022.
  * Sohl-Dickstein et al. (2015) Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  _arXiv_ , abs/1503.03585, 2015.
  * Song et al. (2020a) Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020a.
  * Song et al. (2020b) Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ , 2020b.
  * Zhang et al. (2018) Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  _arXiv_ , abs/1801.03924, 2018.

##  Appendix A Appendix

###  A.1 Model details

Image encoder. The first stage, the image autoencoder, is devoted to learning
a projection from the pixel space to a compressed latent representation that
makes the diffusion model train faster. The image encoder needs to also learn
to project the latents back to the pixel space without losing important
details about the figure (e.g., text rendering quality). To this end, we
define a convolutional encoder and decoder with a bottleneck, that downsamples
images with a factor $f=8$. The encoder is trained to minimize a KL loss with
a Gaussian distribution as well as a VGG perceptual (Zhang et al., 2018) loss
and an OCR perceptual loss. We follow the adversarial procedure proposed in
VQGAN (Esser et al., 2020), which increases reconstruction quality.

Text encoder. We find that using a general-purpose text encoder (e.g., CLIP
(Radford et al., 2021)) is not well-suited for our task, because text encoders
trained on natural texts and images exhibit a domain gap with respect to the
technical descriptions present in papers. We define Bert (Devlin et al., 2018)
transformer that is trained from scratch during the diffusion process. We use
an embedding channel of size $512$, which is also the embedding size of the
cross-attention layers for conditioning the U-Net. We explore varying the
number of layers of the transformer in the set ${8,32,and128}$.

Latent diffusion model. Table 2 describes the U-Net network architecture. We
perform diffusion in a perceptually equivalent latent representation of
images, that is compressed to an input size of $64x64x4$, which makes the
diffusion model faster. We define $1000$ steps of diffusion and a linear noise
schedule.

Input (latent) shape | 64 x 64 x 4
---|---
Number of channels | 256
Number of residual blocks | 3
Self-attention resolutions | [64, 32, 16]
Channel mult. | [1, 2, 4, 4]
Dropout | 0
Table 2: Base diffusion U-Net architecture. Input (image) shape | 384 x 384 x 3
---|---
Embed dimension | 4
Number of channels | 128
Number of residual blocks | 2
Channel mult. | [1, 2, 4, 4]
Discriminator weight | 0.5
VGG perceptual loss weight | 0.2
OCR perceptual loss weight | 0.8
KL loss weight | 1e-6
Dropout | 0
Table 3: Image autoencoder architecture.

###  A.2 Training details

Our models are trained in Paper2Fig100k. During our experiments, we find that
a challenging problem is how to deal with extremely varying aspect ratios that
exist between images in the dataset (e.g., figures tend to be larger in width
than in height). Cropping figures would result in a loss of crucial
information. Therefore we opt for applying white padding to images and only
consider images with an aspect ratio between $0.5$ and $2$, to avoid having
most of the pixel information be the padded pixels. The final dataset consists
of $37,613$ samples for training and $9,506$ for validation. Images are
processed at size $512x512$ and downsampled to $64x64$ by our image
autoencoder.

For training the image autoencoder we use Adam optimizer with an effective
batch size of 4 samples, and a learning rate of $4.5e-6$, using 4 NVIDIA V100
12GB GPUs. For training stability, we warm up the model during $50k$
iterations without using the discriminator (Esser et al., 2020; Rombach et
al., 2021). For training latent diffusion models, we use Adam optimizer, using
an effective batch size of 32 and a learning rate of $1e-4$. For training the
models on Paper2Fig100k we use 8 NVIDIA A100 80GB GPU.

###  A.3 Additional generation results

Figures 2 show additional generated samples of FigGen when tuning the
classifier-free guidance (CFG) (Ho & Salimans, 2022) parameter. We observe
improvement in figure quality when increasing the CFG scale which is also
shown quantitatively. Figure 3 presents more generations of FigGen. Note the
variability in text length between samples, as well as the technical level of
the captions, which makes it difficult for the model to properly generate
understandable figures. However, high-level concepts are correctly captured,
such as captions that describe charts and plots, algorithms, or cases where we
aim to display neural network architectures.

Figure 2: Samples generated from FigGen${}_{\textbf{Base}}$. We display two
generated samples for three prompts Paper2Fig100k test set. Each row displays
the classifier-free guidance (CFG) scale, showing that we can super-condition
the generations to make the samples align better with the prompt. Figure 3:
Generated samples from the test set prompts above as input. Samples generated
using FigGen${}_{\textbf{Base}}$.


# On Dataset Transferability in Active Learning for Transformers

Fran Jelenić Josip Jukić Nina Drobac Jan Šnajder
University of Zagreb, Faculty of Electrical Engineering and Computing, Croatia
Text Analysis and Knowledge Engineering Lab
{fran.jelenic, josip.jukic, nina.drobac<EMAIL_ADDRESS>
###### Abstract

Active learning (AL) aims to reduce labeling costs by querying the examples
most beneficial for model learning. While the effectiveness of AL for fine-
tuning transformer-based pre-trained language models (PLMs) has been
demonstrated, it is less clear to what extent the AL gains obtained with one
model transfer to others. We consider the problem of transferability of
actively acquired datasets in text classification and investigate whether AL
gains persist when a dataset built using AL coupled with a specific PLM is
used to train a different PLM. We link the AL dataset transferability to the
similarity of instances queried by the different PLMs and show that AL methods
with similar acquisition sequences produce highly transferable datasets
regardless of the models used. Additionally, we show that the similarity of
acquisition sequences is influenced more by the choice of the AL method than
the choice of the model.

##  1 Introduction

Pre-trained language models (PLMs) – large over-parameterized models based on
the transformer architecture Vaswani et al. (2017) and trained on large
corpora – are the leading paradigm in modern NLP, yielding state-of-the-art
results on a wide range of NLP tasks. However, large models require large
amounts of data. _Active learning_ (AL; Settles, 2009) addresses the data
bottleneck problem by improving data labeling efficiency. It employs human-in-
the-loop labeling with the model iteratively selecting data points most
informative for labeling. Recent work has demonstrated the effectiveness of AL
for fine-tuning PLMs Dor et al. (2020); Grießhaber et al. (2020); Margatina et
al. (2022); Yuan et al. (2020); Shelmanov et al. (2021).

While AL may considerably reduce model development costs, it also potentially
limits the scope of use of the actively acquired datasets. Since data sampling
in AL is guided by the inductive bias of the acquisition model, the dataset
will typically not represent the original population’s distribution Attenberg
and Provost (2011). This is troublesome if one wishes to use the actively
acquired dataset to train a different model (consumer model) from the one used
for AL (acquisition model). If the two models’ inductive biases differ, the AL
gains can cancel or even revert: the consumer model may perform worse when
trained on the actively acquired dataset than on a randomly sampled one.
However, the robustness of the actively acquired dataset to the choice of the
consumer model is obviously highly desirable, as the acquisition model may
become unavailable or dated. The latter is common in NLP, where new and better
models are being developed faster than new datasets. However, most AL studies
use the same acquisition and consumer models, and dataset transferability is
seldom mentioned in AL literature. A notable exception is the work of Lowell
et al. (2018), who showed the unreliability of dataset transfer on standard
NLP tasks.

In this work, we examine the problem of AL dataset transferability for
transformer-based PLMs and conduct a preliminary empirical study on text
classification datasets. We first probe whether AL gains persist between
different transformer-based PLMs, considering several AL methods and datasets.
Observing that on most datasets, the transfer works in some cases but fails in
others, we investigate the mechanisms underlying transferability. We
hypothesize a link between AL dataset transferability and how the acquisition
and consumer models sample instances. To probe this, we introduce _acquisition
sequence mismatch_ (ASM) to characterize to what extent the two models differ
in how they sample instances throughout AL iterations. We investigate how ASM
affects dataset transferability and how ASM is affected by other AL variables.
We show that, while it is generally reasonable to transfer actively acquired
datasets between transformer-based PLMs, AL methods that retain low ASM
produce more transferable datasets. We also show that the choice of the AL
method affects ASM more than the choice of models.

To summarize our contributions: we (1) conduct an empirical study on the
transferability of actively acquired datasets between transformer-based PLMs,
(2) propose a measure to quantify the mismatch in the acquisition sequences of
AL models and link this to dataset transferability, and (3) analyze what
design choices affect this mismatch. We provide code for the
experiments111https://github.com/fjelenic/al-transfer with the hope that our
results will encourage NLP practitioners to use AL when fine-tuning PLMs and
motivate further research into the AL dataset’s transferability.

##  2 Related Work

Although AL has been extensively studied for shallow and standard neural
models (without pre-training), research on combining AL and PLMs lags behind.
The initial studies showed promise, with AL methods outperforming random
sampling for text classification Dor et al. (2020); Grießhaber et al. (2020).
The field is gradually gaining traction with studies demonstrating AL
effectiveness even with simple uncertainty-based methods Gonsior et al.
(2022); Schröder et al. (2022). Moreover, PLMs open up new possibilities, such
as complementing AL with model adaptation using unlabeled data Yuan et al.
(2020); Margatina et al. (2022).

While there is much research on AL for standard scenarios where the
acquisition and consumer models are the same, there is little research on AL
dataset transfer. Prabhu et al. (2019) demonstrated that combining uncertainty
AL strategies with deep models produces sampled datasets with good sampling
properties that have a large overlap with support vectors of SVM trained on
the entire dataset. Likewise, Farquhar et al. (2021) showed that deep neural
models benefit from the sample bias induced by the acquisition model (the
opposite is true for shallow models). However, the jury is still out on the
effects of sample bias on the consumer model. The most prominent empirical
study on AL transfer with neural models Lowell et al. (2018) predates PLMs.
Tsvigun et al. (2022) focused on alleviating the effects of acquisition-
consumer mismatch in PLMs by using lightweight distilled models for
acquisition and larger versions of the models as consumer models. Even though
the study focuses on improving the transferability of actively acquired
datasets, the reasons behind the successful transfer are yet to be explored.
An older study of AL dataset transferability for text classification and
shallow models by Tomanek and Morik (2011) showed that transfer works in most
cases but that neither sample nor model similarity explains transferability.
Our study explores these characteristics for acquisition-consumer pairings of
different PLMs.

##  3 Experimental Setup

Our study used four datasets, three models, and three AL methods (cf. Appendix
B for details). The datasets we used are Subjectivity (subj; Pang and Lee,
2004), CoLA (cola; Warstadt et al., 2018), AG-News (agn; Zhang et al., 2015),
and TREC (trec; Li and Roth, 2002)). The three transformer models we used are
BERT Devlin et al. (2018), RoBERTa Liu et al. (2019), and ELECTRA Clark et al.
(2020). The AL methods we considered are entropy (ent; Settles, 2009), core-
set (cs; Sener and Savarese, 2017), and BADGE (ba; Ash et al., 2019)). This
gives 108 AL configurations (72 transfer and 36 no-transfer configurations).
Furthermore, we ran each configuration with 20 different warm-start sets to
account for stochasticity. The AL acquisition was simulated until the budget
of 1500 labeled data points was exhausted (model performance for all datasets
reached a plateau), labeling 50 data points per step.

We assessed dataset transferability using the difference in the area under the
$F_{1}$ curve of the model trained on the actively acquired dataset and the
same model trained on a randomly sampled dataset ($\Delta\mathrm{AUC}$). We
deem the AL dataset transfer successful if $\Delta\mathrm{AUC}$ is not
significantly less than zero and unsuccessful otherwise. We chose
$\Delta\mathrm{AUC}$ to make the notion of transferability independent of when
the AL acquisition terminates. On the other hand, as terminating the AL after
acquiring too few labeled data is unrealistic, we also report
$\Delta\mathrm{AUC}_{10}$, which is $\Delta\mathrm{AUC}$ calculated with an
offset of 10 iterations (500 labeled instances) of the AL loop. Comparing
$\Delta\mathrm{AUC}_{10}$ to $\Delta\mathrm{AUC}$ provides insights into how
transferability changes through time.

##  4 Results

###  4.1 Dataset transferability

We grouped the 108 AL configurations into three groups based on the sign of
the mean $\Delta\mathrm{AUC}$ value and the p-value of the difference between
AUC scores of transfer and random sampling:222We used either the paired t-test
or Wilcoxon signed-rank test, depending on the results of Lilliefors’ test for
normality. negative ($\Delta\mathrm{AUC}$ $<$ 0 and p$<$.05), neutral
(p$\geq$.05), and positive ($\Delta\mathrm{AUC}$ $\geq$ 0 and p$<$.05)
transfer. The no-transfer AL configurations (where the acquisition and
consumer models are the same) are generally successful (25 positive, 9
neutral, and 2 negative configurations as per $\Delta\mathrm{AUC}$; 33
positive, 2 neutral, and 1 negative configuration as per
$\Delta\mathrm{AUC}_{10}$). The grouping of the remaining 72 configurations
with AL dataset transfer is given in Table 1. We observe that the dataset, the
acquisition-consumer model pairing, and the AL method all affect transfer
success.

|  $\Delta$- |  $\Delta$0 |  $\Delta$+ |  |  $\Delta$${}_{10}^{-}$ |  $\Delta$${}_{10}^{0}$ |  $\Delta$${}_{10}^{+}$ |  | $\Sigma$
---|---|---|---|---|---|---|---|---|---
subj | $0$ | $0$ | $18$ |  | $0$ | $0$ | $18$ |  | $18$
cola | $2$ | $8$ | $8$ |  | $2$ | $7$ | $9$ |  | $18$
agn | $7$ | $4$ | $7$ |  | $3$ | $2$ | $13$ |  | $18$
trec | $8$ | $3$ | $7$ |  | $0$ | $2$ | $16$ |  | $18$
R$\rightarrow$B | $2$ | $2$ | $8$ |  | $0$ | $1$ | $11$ |  | $12$
E$\rightarrow$B | $2$ | $2$ | $8$ |  | $0$ | $2$ | $10$ |  | $12$
B$\rightarrow$R | $2$ | $4$ | $6$ |  | $0$ | $1$ | $11$ |  | $12$
E$\rightarrow$R | $2$ | $4$ | $6$ |  | $1$ | $2$ | $9$ |  | $12$
B$\rightarrow$E | $5$ | $1$ | $6$ |  | $2$ | $2$ | $8$ |  | $12$
R$\rightarrow$E | $4$ | $2$ | $6$ |  | $2$ | $3$ | $7$ |  | $12$
ent | $11$ | $3$ | $10$ |  | $3$ | $2$ | $19$ |  | $24$
cs | $4$ | $10$ | $10$ |  | $2$ | $6$ | $16$ |  | $24$
ba | $2$ | $2$ | $20$ |  | $0$ | $3$ | $21$ |  | $24$
$\Sigma$ | $17$ | $15$ | $40$ |  | $5$ | $11$ | $56$ |  |
Table 1: Breakdown of datasets, acquisition$\to$consumer model pairs (denoted
by initial letters), and AL methods by transferability: negative ($-$),
neutral ($0$), and positive ($+$) transfer. $\Delta\mathrm{AUC}$ is shown as
$\Delta$.

Evidently, transferability differs across datasets: the transfer is always
positive on subj (which is the simplest task we considered in terms of the
number of labels, the balance of classes, and the MDL task complexity measure;
cf. Appendix B), while most neutral transfers occur on cola. A more
interesting picture emerges from the different acquisition-consumer model
pairings and AL methods. Most negative transfers are transfers to ELECTRA,
while most neutral transfers are those to RoBERTa (perhaps due to it being
optimized for robustness). On the other hand, transfer to BERT is positive in
most cases, perhaps because BERT’s pre-training regime is most similar to that
of the other two models. Among the AL methods, entropy mostly makes the
transfer negative, most neutral transfers occur with core-set, and BADGE is
the best choice for ensuring positive transferability. However, when looking
at the later steps of the AL loop, differences between entropy and BADGE
vanish, while the core-set lags slightly behind. Thus, $\Delta\mathrm{AUC}$
tends to increase throughout the AL process, suggesting that increasing the
amount of sampled data lowers the risk of unsuccessful transfer (cf. Appendix
C for additional $F_{1}$ scores analysis).

###  4.2 Acquisition sequence mismatch

We hypothesize there is a link between dataset transferability and the
sequence in which data points are acquired for labeling by AL. In particular,
we posit that dataset transferability will be successful when the acquisition
sequence of the acquisition model does not differ from what the acquisition
sequence of a consumer model would be if that model had access to the original
dataset. We introduce the _acquisition sequence mismatch_ (ASM) to measure the
differences in acquisition sequences. To compute the ASM between two
acquisition sequences, we pair the corresponding batches of the two sequences
and average their pairwise differences. To measure the difference between a
pair of batches, we take the average of the distances of best-matched examples
between the batches. To account for the fact that AL methods may choose
numerically different yet semantically similar data points, we measure the
similarity of acquired instances in representation space. We use GloVe
embeddings Pennington et al. (2014) as a common representation space
independent of the choice of acquisition and consumer models and compute the
cosine distance between averaged word embeddings. Lastly, we use the Hungarian
algorithm Kuhn (1955) to construct a bipartite graph between two batches with
distance-weighted edges to find the best-matching examples. Formally, we
define ASM as follows:

$\frac{1}{T}\displaystyle\sum_{t=1}^{T}\frac{1}{|B_{t}|}\displaystyle\min_{S(B_{A}^{t}),S(B_{B}^{t})}\left(\sum_{i=1}^{|B_{t}|}d(x_{A}^{i},x_{B}^{i})\right)$
(1)

where $T$ is the length of the sequence (the number of steps of the AL loop),
$S(B^{t})$ is the set of all of the permutations of instances in the selected
batch at step $t$, and $d(x_{A}^{i},x_{B}^{i})$ is the cosine distance between
instance representations from sequences $A$ and $B$ for a batch at position
$i$ of a given batch permutation. Intuitively, ASM assumes that both batches
cater to the same informational need of the model, so it calculates how much
the instances that should carry out the same role in the batch differ.

(a) Subjectivity

(b) CoLA

(c) AG-News

(d) TREC

Figure 1: Distributions of ASM values for combinations of AL methods and
acquisition+consumer model pairs (denoted by initial letters).

Given a dataset, we hypothesize ASM may be affected by both the choice of the
models and the choice of the AL method. Figure 1 shows that the distributions
of ASM values are more alike when grouped by the AL methods than when grouped
by the model pairings. To verify this observation, we conducted two Kruskal-
Wallis H-tests for each dataset: in the first, populations were determined by
the AL method, and we concluded that there was a significant difference in ASM
(p$<$.05); in the second, the populations were determined by the model
pairing, and there was no significant difference in ASM (p$>$.05). This
suggests that the choice of AL method affects ASM more than the choice of
acquisition-consumer model pairing.

###  4.3 Acquisition mismatch analysis

(a) Subjectivity

(b) CoLA

(c) AG-News

(d) TREC

Figure 2: The mismatch between acquired batches (top) and ADM at each step of
the AL loop (bottom) for different AL methods.

We found a statistically significant negative correlation between
$\Delta\mathrm{AUC}$ and ASM for each dataset.333Spearman correlation
coefficients are $-0.11$ for subj, $-0.19$ for cola, $-0.27$ for agn, and
$-0.38$ for trec, all significant with p$<$.05. This supports our hypothesis
that the lower the mismatch between acquisition sequences of the two models,
the higher the transferability of a dataset from one model to the other.
Besides ASM, we use another measure for analyzing dataset transferability: the
difference between the dataset acquired with AL using the acquisition model
and the dataset acquired with AL using the consumer model. We call this
measure the _acquired dataset mismatch_ (ADM). Essentially, ADM computes the
mismatch between samples similarly to ASM but between entire datasets obtained
after the last sampling step.

Above we showed that the choice of the AL method affects the ASM. Figure 2
shows that BADGE gives smaller ASM than the other two methods, whereas core-
set gives larger ASM than the other two methods.444Verified using three one-
sided Wilcoxon signed-rank tests with p$<$.05 corrected for FWER. However, the
intriguing effect emerges when comparing the difference in batches through
time and differences in the entire acquired datasets through time. In the
early steps, BADGE gives the highest similarity of acquired datasets among the
considered methods, which leads to it having the lowest ASM. However, in later
steps, entropy dominates the similarity of acquired datasets.555Verified using
three one-sided Wilcoxon signed-rank tests on ADM with p$<$.05 corrected for
FWER. It seems as if entropy acquired similar datasets for different models by
taking those models through different sequences of the population
distribution. This effect is seen in Table 1, where entropy is the worst
method when using $\Delta\mathrm{AUC}$ to measure transfer success while
managing to parry BADGE when using $\Delta\mathrm{AUC}_{10}$. The difference
in transferability between entropy and BADGE completely vanishes when looking
at the last step of the AL loop (cf. Appendix, Table 3). It is clear that
entropy can produce transferable datasets, but it requires more time to do so.

We speculate that the effect of BADGE having the lowest ASM yet entropy
achieving the lowest ADM could emerge due to the interaction between the AL
method and the model’s decision boundary. Namely, uncertainty AL methods
sample data points on the decision boundary with high overlap with support
vectors of the SVM trained on the whole dataset, as pointed out by Prabhu et
al. (2019). Since BADGE combines uncertainty and diversity, i.e., it samples
data points the model is uncertain about for diverse reasons, it samples along
the entire decision boundary at each step, and since decision boundaries of
the models are roughly the same, so are the sampled data points. Entropy, on
the other hand, relies solely on uncertainty. Due to its greedy nature,
entropy tends to sample similar points because if one data point has high
uncertainty, data points similar to it are also going to have high uncertainty
Zhdanov (2019). This may manifest as sampling local patches of space on the
decision boundary. Therefore, entropy may take more time to define the
boundary than BADGE because it is forming the boundary from patches of space
with the highest uncertainty at a given AL step rather than holistically
sampling along the boundary at each step. Since the shape of the decision
boundary is more similar between different models than the local interactions
along the boundary, entropy has a higher batch mismatch in the early steps.
However, once more data is labeled and the boundary becomes stable, both
entropy and BADGE start to have a low batch mismatch, as seen in Figure 2.
Since entropy is deterministic and never strays from the decision boundary, it
ends up having a lower ADM than BADGE. Lastly, we believe that the core-set
method has the highest ASM and ADM because it selects data based on diversity
in the model’s representation space, which is more model-specific than the
decision boundary. Further exploring the described interaction is a compelling
direction for future work.

It may be that AL methods with different acquisition sequences end up
acquiring a similar dataset and have high transferability, as in the case of
entropy, an uncertainty-based acquisition function. It is also possible that
acquired datasets differ between models but that the transfer remains
successful because it taps into some other essential aspect of a transferable
dataset, as is the case with core-set, a diversity-based acquisition function.
However, the best strategy to ensure dataset transferability appears to be a
mixture of uncertainty and diversity, as provided by BADGE. This appears to
minimize ASM between models, making datasets transferable regardless of the
number of AL steps.

##  5 Conclusion

We presented an empirical study on the transferability of actively acquired
text classification datasets for transformer-based PLMs. Our results indicate
no significant risk in transferring datasets, especially for larger amounts of
data. We also showed that transfer is largely successful when preserving the
sequence and similarity of acquired instances between the models, which is
what methods combining uncertainty and diversity acquisition functions seem to
do. Transferability appears to differ considerably across datasets, so future
work should examine what dataset characteristics are predictive of transfer
success.

## Limitations

Our study revealed considerable differences in transferability and other
measures we considered across different datasets. Nonetheless, the study
focused on the differences in transferability arising from the choice of the
models and the AL methods rather than the dataset. To eliminate confounding
due to datasets, we grouped the results by datasets and analyzed each group
separately. Despite this, the scope of our results is limited by the fact that
all datasets used are in English and possibly contain their own biases.

Even though we showed that it could still be useful to transfer actively
acquired datasets between transformer-based PLMs, it is important to keep in
mind that actively acquired datasets are not representative of the original
data distribution due to the sampling bias introduced by active learning.

## Acknowledgments

This research was supported by the AIDWAS KK.01.2.1.02.0285 grant. We thank
the anonymous reviewers for their insightful comments and suggestions.

## References

  * Ash et al. (2019) Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. 2019.  Deep batch active learning by diverse, uncertain gradient lower bounds.  _arXiv preprint arXiv:1906.03671_.
  * Attenberg and Provost (2011) Josh Attenberg and Foster Provost. 2011.  Inactive learning? difficulties employing active learning in practice.  _ACM SIGKDD Explorations Newsletter_ , 12(2):36–41.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020.  ELECTRA: Pre-training text encoders as discriminators rather than generators.  _arXiv preprint arXiv:2003.10555_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Dor et al. (2020) Liat Ein Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, and Noam Slonim. 2020.  Active learning for BERT: an empirical study.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7949–7962.
  * Farquhar et al. (2021) Sebastian Farquhar, Yarin Gal, and Tom Rainforth. 2021.  On statistical bias in active learning: How and when to fix it.  _arXiv preprint arXiv:2101.11665_.
  * Gonsior et al. (2022) Julius Gonsior, Christian Falkenberg, Silvio Magino, Anja Reusch, Maik Thiele, and Wolfgang Lehner. 2022.  To softmax, or not to softmax: that is the question when applying active learning for transformer models.  _arXiv preprint arXiv:2210.03005_.
  * Grießhaber et al. (2020) Daniel Grießhaber, Johannes Maucher, and Ngoc Thang Vu. 2020.  Fine-tuning BERT for low-resource natural language understanding via active learning.  _arXiv preprint arXiv:2012.02462_.
  * Kuhn (1955) Harold W Kuhn. 1955.  The hungarian method for the assignment problem.  _Naval research logistics quarterly_ , 2(1-2):83–97.
  * Li and Roth (2002) Xin Li and Dan Roth. 2002.  Learning question classifiers.  In _COLING 2002: The 19th International Conference on Computational Linguistics_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized BERT pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Lowell et al. (2018) David Lowell, Zachary C Lipton, and Byron C Wallace. 2018.  Practical obstacles to deploying active learning.  _arXiv preprint arXiv:1807.04801_.
  * Margatina et al. (2022) Katerina Margatina, Loïc Barrault, and Nikolaos Aletras. 2022.  On the importance of effectively adapting pretrained language models for active learning.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 825–836.
  * Pang and Lee (2004) Bo Pang and Lillian Lee. 2004.  A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts.  In _Proceedings of the ACL_.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014.  GloVe: Global vectors for word representation.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543.
  * Perez et al. (2021) Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021.  Rissanen data analysis: Examining dataset characteristics via description length.  In _International Conference on Machine Learning_ , pages 8500–8513. PMLR.
  * Prabhu et al. (2019) Ameya Prabhu, Charles Dognin, and Maneesh Singh. 2019.  Sampling bias in deep active classification: An empirical study.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4058–4068.
  * Schröder et al. (2022) Christopher Schröder, Andreas Niekler, and Martin Potthast. 2022.  Revisiting uncertainty-based query strategies for active learning with transformers.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 2194–2203.
  * Sener and Savarese (2017) Ozan Sener and Silvio Savarese. 2017.  Active learning for convolutional neural networks: A core-set approach.  _arXiv preprint arXiv:1708.00489_.
  * Settles (2009) Burr Settles. 2009.  Active learning literature survey.  Computer sciences technical report.
  * Shelmanov et al. (2021) Artem Shelmanov, Dmitri Puzyrev, Lyubov Kupriyanova, Denis Belyakov, Daniil Larionov, Nikita Khromov, Olga Kozlova, Ekaterina Artemova, Dmitry V Dylov, and Alexander Panchenko. 2021.  Active learning for sequence tagging with deep pre-trained models and bayesian uncertainty estimates.  _arXiv preprint arXiv:2101.08133_.
  * Tomanek and Morik (2011) Katrin Tomanek and Katherina Morik. 2011.  Inspecting sample reusability for active learning.  In _Active Learning and Experimental Design workshop In conjunction with AISTATS 2010_ , pages 169–181. JMLR Workshop and Conference Proceedings.
  * Tsvigun et al. (2022) Akim Tsvigun, Artem Shelmanov, Gleb Kuzmin, Leonid Sanochkin, Daniil Larionov, Gleb Gusev, Manvel Avetisian, and Leonid Zhukov. 2022.  Towards computationally feasible deep active learning.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 1198–1218.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Warstadt et al. (2018) Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2018.  Neural network acceptability judgments.  _arXiv preprint arXiv:1805.12471_.
  * Yuan et al. (2020) Michelle Yuan, Hsuan-Tien Lin, and Jordan Boyd-Graber. 2020.  Cold-start active learning through self-supervised language modeling.  _arXiv preprint arXiv:2010.09535_.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  _Advances in neural information processing systems_ , 28.
  * Zhdanov (2019) Fedor Zhdanov. 2019.  Diverse mini-batch active learning.  _arXiv preprint arXiv:1901.05954_.

##  Appendix A Reproducibility

We conducted our experiments on 4× AMD Ryzen Threadripper 3970X 32-Core
Processors and 4× NVIDIA GeForce RTX 3090 GPUs with 24GB of RAM, which took
roughly one week. We used PyTorch version 1.12.1, Transformers version 4.21.3,
and CUDA 11.4.

##  Appendix B Experimental design choices

###  B.1 Datasets

| Train | Test | # Labels | NLE | MDL
---|---|---|---|---|---
subj | $8000$ | $2000$ | $2$ | $1.00$ | $0.30$
cola | $8551$ | $1043$ | $2$ | $0.88$ | $1.00$
agn | $20000^{*}$ | $7600$ | $4$ | $1.00$ | $0.56$
trec | $5452$ | $500$ | $6$ | $0.92$ | $0.34$
Table 2: Dataset statistics. We report train and test set sizes, number of
labels, normalized label entropy (information entropy of label distribution
normalized by the entropy of uniform distribution with the same number of
variables), and MDL normalized by the MDL of the dataset with the largest
value (cola). Train set of agn was subsampled from the original train set of
120000 instances.

The datasets used in this paper are standard benchmarks in NLP for text
classification. We chose these datasets to represent different attributes: the
number of labels (binary or multi-class classification) and the balancing of
the labels (balanced and imbalanced classes). The diversity of the dataset
characteristics can give an insight into the impact of these attributes on
dataset transferability. We present dataset statistics in Table 2. There we
also show minimum description length (MDL) Perez et al. (2021) of each
dataset, which can be interpreted as the complexity of the task.
Subjectivity: Movie-review data with reviews labeled as either subjective or
objective. This is a balanced dataset with binary labels.
CoLA: The Corpus of Linguistic Acceptability is a dataset containing sentences
labeled as grammatical or not. This is an imbalanced dataset with binary
labels.
AG-News: Corpus of news articles annotated by the article’s topic (World,
Sports, Business, Sci/Tech). The dataset was created by subsampling the corpus
to the size of 20,000 examples. This is a balanced dataset with four classes.
TREC: The dataset contains questions labeled with the type of subject of the
question. This is an imbalanced dataset with six classes.

###  B.2 Models

We picked the models that share the common architecture; they are all
transformer-based PLMs but differ in pre-training data and pre-training
objectives. This choice of models enables us to analyze the impact of
different pre-training design choices on dataset transferability. All models
were trained using ADAM optimizer with a learning rate of $2\cdot 10^{-5}$ and
batch size of 64 for five epochs for both acquisition and evaluation phases.
BERT: One of the first and most popular transformer-based pre-trained language
models. The model was pre-trained using a generative masked language modeling
objective. This model has 12 layers, a hidden state size of 768, and 12 heads
with 110M parameters in total.
RoBERTa: A model with the same architecture and pre-training objective as BERT
but trained on more data and with optimized hyperparameters to make the model
more robust. This model has 12 layers, a hidden state size of 768, and 12
heads with 125M parameters in total.
ELECTRA: It uses the same architecture and pre-training data as BERT but with
discriminative instead of generative pre-training objectives. Instead of
masking some tokens in text and having to guess the identity of masked tokens
as BERT does, the generative pre-training objective corrupts some tokens by
replacing them with plausible alternatives, and then the model has to decide
for each token whether it is the original token or the replaced one. This
model has 12 layers, a hidden state size of 768, and 12 heads with 110M
parameters in total.

###  B.3 AL methods

AL methods used to select the most informative data points are divided into
two types of heuristics: uncertainty and diversity. Methods using uncertainty
as a heuristic select data based on some measure of the model’s uncertainty.
The intuition behind the uncertainty methods is that the more uncertain the
model is about a data point, the more it can learn from knowing its label. In
comparison, diversity-based methods try to represent the input space (which is
not always the same as the input population) as accurately as possible with as
few data points as possible. AL methods can combine those two heuristics to
select a group of data points the model is uncertain about for different
reasons.

The choice of the AL methods used in this experiment was motivated by the type
of heuristic (uncertainty vs. diversity) they used for sampling. These methods
allow us to analyze the impact of the choice of heuristic on the success of
dataset transfer in AL.
Entropy: An uncertainty-based method that selects data points with maximal
information entropy of their posterior class distribution.
Core-set: This diversity-based method selects data points that best cover the
representation space.
BADGE: A method that combines uncertainty and diversity by using $k$-means++
algorithm on the would-be gradients of the models’ last layer for the data
points if their most probable labels were their actual labels.

##  Appendix C Experiment runs

This section presents more results from our experiment to complement the
already presented results. Table 3 shows transferability for different
combinations in the fashion of Table 1. However, instead of measuring
transferability with $\Delta\mathrm{AUC}$ this table uses the $F_{1}$ score at
the end of the AL loop (1500 labeled instances). To illustrate the success of
regular AL (without the transfer), we present Table 4. That table shows the
same information as Table 1 and Table 3 but for situations where acquisition
and consumer models are the same. Lastly, we present the learning curves of
all of the runs of the experiment in Figure 3 for Subjectivity, Figure 4 for
CoLA, Figure 5 for AG-News, and Figure 6 for TREC dataset.

| $F_{1}^{-}$ | $F_{1}^{0}$ |  $F_{1}^{+}$ |
---|---|---|---|---
subj | $0$ | $1$ |  $17$ | $18$
cola | $3$ | $11$ |  $4$ | $18$
agn | $1$ | $4$ |  $13$ | $18$
trec | $0$ | $2$ |  $16$ | $18$
R$\rightarrow$B | $0$ | $3$ |  $9$ | $12$
E$\rightarrow$B | $0$ | $4$ |  $8$ | $12$
B$\rightarrow$R | $1$ | $1$ |  $10$ | $12$
E$\rightarrow$R | $1$ | $2$ |  $9$ | $12$
B$\rightarrow$E | $1$ | $3$ |  $8$ | $12$
R$\rightarrow$E | $1$ | $5$ |  $6$ | $12$
ent | $1$ | $5$ |  $18$ | $24$
cs | $2$ | $8$ |  $14$ | $24$
ba | $1$ | $5$ |  $18$ | $24$
| $4$ | $18$ |  $50$ |
Table 3: Breakdown of datasets, acquisition$\to$consumer model pairs (denoted by initial letters), and AL methods by transferability measured via $F_{1}$ score at the end of the AL loop: negative ($-$), neutral ($0$), positive ($+$) transfer. | $\mathrm{AUC}^{-}$ | $\mathrm{AUC}^{0}$ | $\mathrm{AUC}^{+}$ | $\mathrm{AUC}_{10}^{-}$ | $\mathrm{AUC}_{10}^{0}$ | $\mathrm{AUC}_{10}^{+}$ | $F_{1}^{-}$ | $F_{1}^{0}$ |  $F_{1}^{+}$ |
---|---|---|---|---|---|---|---|---|---|---
subj | $0$ | $0$ | $9$ | $0$ | $0$ | $9$ | $0$ | $0$ |  $9$ | $9$
cola | $0$ | $3$ | $6$ | $1$ | $2$ | $6$ | $1$ | $4$ |  $4$ | $9$
agn | $1$ | $3$ | $5$ | $0$ | $0$ | $9$ | $0$ | $0$ |  $9$ | $9$
trec | $1$ | $3$ | $5$ | $0$ | $0$ | $9$ | $0$ | $0$ |  $9$ | $9$
BERT | $1$ | $2$ | $9$ | $0$ | $0$ | $12$ | $0$ | $0$ |  $12$ | $12$
RoBERTa | $0$ | $4$ | $8$ | $1$ | $2$ | $9$ | $0$ | $3$ |  $9$ | $12$
ELECTRA | $1$ | $3$ | $8$ | $0$ | $0$ | $12$ | $1$ | $1$ |  $10$ | $12$
ent | $1$ | $4$ | $7$ | $0$ | $1$ | $11$ | $0$ | $1$ |  $11$ | $12$
cs | $1$ | $4$ | $7$ | $1$ | $0$ | $11$ | $0$ | $2$ |  $10$ | $12$
ba | $0$ | $1$ | $11$ | $0$ | $1$ | $11$ | $1$ | $1$ |  $10$ | $12$
| $2$ | $9$ | $25$ | $1$ | $2$ | $33$ | $1$ | $4$ |  $31$ |
Table 4: Breakdown of datasets, models, and AL methods in the groups based on
the performance of regular AL: negative ($-$), neutral ($0$), positive ($+$)
AL.  Figure 3: Learning curves for the Subjectivity dataset. The figure shows
the mean $F_{1}$ score of 20 runs with confidence intervals of $\pm$ standard
deviation. Figure 4: Learning curves for the CoLA dataset. The figure shows
the mean $F_{1}$ score of 20 runs with confidence intervals of $\pm$ standard
deviation. $F_{1}$ curves for RoBERTa as consumer model with entropy as AL
method have high variance because entropy tends to favor minority class
heavily, and the model starts to classify with minority class more often than
it should, so the $F_{1}$ on the test set drastically drops. These drops
happen one to two times per seed during the AL loop before the method balances
out the labels again. Figure 5: Learning curves for the AG-News dataset. The
figure shows the mean $F_{1}$ score of 20 runs with confidence intervals of
$\pm$ standard deviation. Figure 6: Learning curves for the TREC dataset. The
figure shows the mean $F_{1}$ score of 20 runs with confidence intervals of
$\pm$ standard deviation.