11institutetext: Wellcome/EPSRC center for Interventional and Surgical
Sciences (WEISS), University College London (UCL), London, UK 22institutetext:
Dept of Computer Science, UCL, London, UK 33institutetext: Dept of Obstetrics
and Gynecology, University Hospital of Rennes, Rennes, France 44institutetext:
Univ Rennes, INSERM, LTSI - UMR 1099, F35000, Rennes, France 55institutetext:
Fetal Medicine Unit, Elizabeth Garrett Anderson Wing, UCLH NHS Foundation
Trust, London, UK 66institutetext: Elizabeth Garrett Anderson Institute for
Women’s Health, UCL, London, UK 77institutetext: Dept of Obstetrics and
Gynecology, UCLH, London, UK
77email<EMAIL_ADDRESS>
# Measuring proximity to standard planes during fetal brain ultrasound
scanning††thanks: This work was supported in whole, or in part, by the
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS)
[203145/Z/16/Z], the Department of Science, Innovation and Technology (DSIT)
and the Royal Academy of Engineering under the Chair in Emerging Technologies
programme.

Chiara Di Vece 1122 Antonio Cirigliano 11 Meala Le Lous 113344 Raffaele
Napolitano 5566 Anna L. David 1177 Donald Peebles 1177 Pierre Jannin 44
Francisco Vasconcelos 1122 Danail Stoyanov 1122

###### Abstract

This paper introduces a novel pipeline designed to bring ultrasound (US) plane
pose estimation closer to clinical use for more effective navigation to the
standard planes in the fetal brain. We propose a semi-supervised segmentation
model utilizing both labeled SPs and unlabeled 3D US volume slices. Our model
enables reliable segmentation across a diverse set of fetal brain images.
Furthermore, the model incorporates a classification mechanism to identify the
fetal brain precisely. Our model not only filters out frames lacking the brain
but also generates masks for those containing it, enhancing the relevance of
plane pose regression in clinical settings. We focus on fetal brain navigation
from two-dimensional (2D) ultrasound (US) video analysis and combine this
model with a US plane pose regression network to provide sensorless proximity
detection to SPs and non-SPs planes; we emphasize the importance of proximity
detection to SPs for guiding sonographers, offering a substantial advantage
over traditional methods by allowing earlier and more precise adjustments
during scanning. We demonstrate the practical applicability of our approach
through validation on real fetal scan videos obtained from sonographers of
varying expertise levels. Our findings demonstrate the potential of our
approach to complement existing fetal US technologies and advance prenatal
diagnostic practices.

###### Keywords:

Ultrasound Fetal brain Semantic segmentation Semi-supervised learning.

##  1 Introduction

Fetal US is instrumental in the early detection of fetal growth abnormalities,
potentially averting adverse outcomes for both mother and child [1]. The
efficacy of US screenings largely hinges on the operator’s skill in navigating
to and reproducing standard imaging planes, known as SPs, which demands
spatial awareness, precise hand-eye coordination, and thorough anatomical
knowledge. Novices often find the requisite multitasking overwhelming,
struggling to manipulate the transducer while interpreting the 2D images as
representations of three-dimensional (3D) structures [2, 3]. Furthermore, the
variability in examination outcomes due to differing levels of operator
experience can cause measurement and diagnostic discrepancies, challenging the
reliability of fetal US diagnostics [4]. International guidelines underscore
the critical role of US in diagnosing fetal growth disorders, highlighting the
need for standardized scanning protocols and quality control measures [5].

ai offers promising solutions to address these challenges in different ways.
Images of the various SPs can be reliably recognized in real-time with
classifier networks [6], and standardized biometry measurements on those
planes can also be automated [7, 8, 9]. Based on these image recognition
capabilities, the freehand US scanning process can be simplified to reduce
inter-operator variability by automatically extracting SPs from 3D US volumes
[10, 11]. Most of these techniques assume that a good quality US scan has
already been acquired in the correct position and only assist in image
analysis.

A range of other approaches aims at providing spatial localization of the US
acquisition plane so that less experienced sonographers can more effectively
navigate to SPs. Some rely on additional sensors attached to the US probe [12,
13], while others are purely image-based [14, 15, 16], leveraging 3D US brain
volumes as training data. The latter methods are attractive as they can
potentially be deployed on conventional 2D US probes typically used in
clinical practice, but they still face some challenges in translation. They
have only been substantially validated on slices of 3D US volumes, which is
still not the final intended usage. Bringing them closer to translation, Yeung
et al. [17] have fine-tuned their model on 2D US video in a self-supervised
way. Additionally, while these methods output a six-dimensional (6D) pose of a
US plane, it is still poorly understood how to communicate this information to
a sonographer for effective navigation.

Figure 1: Comparison of the proposed approach with SPs classification
approaches.

This paper proposes a pipeline that aims at bringing US plane pose estimation
closer to a practical use case of assisted navigation, focusing on fetal brain
navigation from 2D US video analysis. Our envisaged system (Figure 1) detects
when the fetal brain is being visualized and provides a proximity signal to a
specific target SP (we consider transventricular (TV) SP in this work). The
contributions can be summarised as follows: 1) We propose a fetal brain
segmentation and classification model (SS-Seg+Class) that works reliably for
arbitrary sections of the brain. We achieve this in a semi-supervised manner,
using a set of labeled SP images together with unlabeled slices from 3D US
volumes; 2) We combine this segmentation model with a US plane pose estimation
network, improving accuracy with respect to an end-to-end baseline; 3) We
manually annotate the canonical pose of TV SP on training 3D US volumes and
use them to estimate relative proximity to these planes at inference time; 4)
We validate our pipeline on recorded video of real fetal scans from
sonographers with varying expertise and demonstrate that our plane proximity
predictions indicate a tendency for alignment with expert assessment of SP
quality.

##  2 Methods

In this work, we focus on guidance toward the TV SP in the fetal brain. Our
proposed pipeline involves 3 main steps: 1) detection and segmentation of the
fetal brain on a US image; 2) plane pose regression on masked fetal brain
images; 3) measurement of proximity to target SP. The method is trained in a
semi-supervised manner using a combination of labeled images of SP together
with unlabeled images from 3D US volumes.

###  2.1 Semi-supervised fetal brain segmentation

Our segmentation and classification model (Figure 2) is based on U-Net [18].

Figure 2: Components of our semi-supervised learning model. Labeled (brain and
non-brain) and unlabeled brain US images are fed into a UNet. The encoder
extracts the features from the images; the classification branch takes the
encoder output and classifies the images as containing brain or not, whereas
the decoder predicts the masks for both labeled and unlabeled brain images.

The encoder, ’timm-efficientnet-b0’ [19] pre-trained on ImageNet [20], takes
as input an image with dimensions CxHxW, where C=3 is the number of channels,
H=320 is the height, and W=320 is the width. The model has two decoder heads.
The segmentation head is a conventional U-Net decoder with skip connections;
the classification head contains a Global Average Pooling, a Fully Connected
Layer, and a sigmoid activation for binary classification (“brain”, “not-
brain”).

The model is trained with 3 losses. The classification head is trained with
Binary Cross Entropy ($\mathcal{L}_{classification}$). The segmentation head
is trained on labeled data with an average of Dice and Binary Cross Entropy
($\mathcal{L}_{Seg,labeled}=(\mathcal{L}_{Dice}(M^{\prime}_{L},M)+\mathcal{L}_{BCE}(M^{\prime}_{L},M))/2$).
The segmentation head is also trained on unlabeled volume slices with
$\mathcal{L}_{Seg,unlabeled}$, calculated as the mean square error (MSE)
between the segmentations of images acquired at the same pose from the
different volumes:
$\mathcal{L}_{Seg,unlabeled}=\frac{2}{n(n-1)}(\sum_{i=1}^{n-1}\sum_{j=i+1}^{n}\left\|M^{\prime}_{U,i}-M^{\prime}_{U,j}\right\|_{2})$,
where $n$ is the number of volumes in the training and validation sets ($n=3$
and $n=2$, respectively), $M^{\prime}_{U,i}$ and $M^{\prime}_{U,j}$ are the
predicted masks for the unlabeled images. The total loss is then:

$\mathcal{L}_{Total,SS-
Seg+Class}=\mathcal{L}_{Seg,labeled}+\alpha\mathcal{L}_{Seg,unlabeled}+\mathcal{L}_{classification}$
(1)

where $\alpha=0.5$ is a balancing coefficient that controls the relative
weight of the loss for unlabeled versus labeled images.

###  2.2 Plane pose regression

We use an 18-layer residual convolutional neural network (CNN) (ResNet-18)
[21] as a backbone for feature extraction with the pre-trained ImageNet
weights. We modified the network by re-initializing the fully connected layer
based on the representation’s dimension (nine parameters) and adding a
regression head to output the rotation and translation representations
directly. The network receives a masked US image $I$ (128$\times$128) obtained
by slicing the volume and applying the mask generated by the SS-Seg+Class
network, and its 6D pose with respect to the center of the fetal brain US
volume
$\theta_{GT}~{}=~{}(t_{x},t_{y},t_{z},\alpha_{x},\alpha_{y},\alpha_{z})$. We
use this information as the ground truth label for network training and
validation. The CNN learns to predict the 6D pose
$\theta_{Pred}=(t^{\prime}_{x},t^{\prime}_{y},t^{\prime}_{z},\alpha^{\prime}_{x},\alpha^{\prime}_{y},\alpha^{\prime}_{z})$.

For both translation and rotation we used as loss the MSE between predicted
($\textbf{t}^{\prime}$,$\textbf{R}^{\prime}$) and ground truth (t,R) values:
$\mathcal{L}_{Translation}=\frac{1}{N}\sum_{t=1}^{N}\left\|\textbf{t}^{\prime}-\textbf{t}\right\|_{2}$,
$\mathcal{L}_{Rotation}=\frac{1}{N}\sum_{t=1}^{N}\left\|\textbf{R}^{\prime}-\textbf{R}\right\|_{2}$
where $N$ denotes the total number of images $I$ within one training epoch,
$\textbf{t}^{\prime}$ denotes the predicted translation component and t the
label. R is the 3$\times$3 rotation matrix obtained from the ground truth
rotation vector $\textbf{r}~{}=~{}(r_{x},r_{y},r_{z})$ and
$\textbf{R}^{\prime}$ is the 3$\times$3 rotation matrix obtained from the 6
parameters $r_{1},...,r_{6}$ as the output of the networks. The total loss
function is then computed as:
$\mathcal{L}_{Total}~{}=~{}\mathcal{L}_{Rotation}+\lambda\mathcal{L}_{Translation}$.
where $\lambda$ is a balancing coefficient. To compute the proximity of the US
video frame to a labeled TV SP during inference, we use euclidean distance for
translation and the angle between the normal vectors of the frame
($\vec{N_{f}}$) and the SP ($\vec{N_{SP}}$) for rotation
($\alpha=\arccos\left((\vec{N_{f}}\cdot\vec{N_{SP}})/(\|\vec{N_{f}}\|\|\vec{N_{SP}}\|)\right)$),
explicitly excluding in-plane rotation components as they do not alter the
plane appearance.

##  3 Experiments and Results

All our models are implemented in _PyTorch_ and trained using a single Tesla®
V100-DGXS-32GB GPU of an NVIDIA® DGX station.

####  3.0.1 Datasets

We use 6 fetal brain US volumes from [22] (singleton pregnancy with no
abnormal findings) acquired at the axial TV SP from 20-25 weeks gestational
age (GA) pregnancies. The volumes are aligned in translation, rotation, and
scale. All volumes were processed to be isotropic with a voxel size of
0.5$\times$0.5$\times$0.5 $mm$ and an average size of
249$\times$174$\times$155 $mm$ (_coronal_ $\times$_axial_ $\times$_sagittal_).
We slice these volumes with random rotations and offsets from the volume
center within a fixed range to avoid slices with poor overlap with the volume.
We acquired 22029 images for each volume. We train the segmentation model
using sets of slices acquired in the same pose for different registered
volumes. The pose regression model is trained with slices paired with their
ground truth pose. To estimate proximity to the target plane, an obstetrician
annotated the pose of the TV SP using a 3D interface implemented in Unity.

In addition to volume slices, we use a US dataset of 346 US images acquired
from 42 fetuses during the second trimester with labeled segmentation masks of
the brain, abdomen, and femur initially used in the AutoFB work [7]. The
dataset includes 135 brain images and 211 non-brain images (abdomen and
femur).

Finally, we test our complete pipeline on routine second-trimester fetal US
scans recorded at anonymous hospital using a VolusonTM E8® US machine (GE
Healthcare, Chicago, Illinois). The study was approved by anonymous regulatory
body. The full videos used in our pipeline are available111*Link to videos.

Figure 3: Inference pipeline on frames from US videos. The frames are
classified as containing or not the brain; if the brain is detected, the mask
is generated, and the frame with the mask applied is fed into a pose
regression network to regress the 6D pose of the frame and compute the
distance to the SP.

####  3.0.2 Brain segmentation and classification

The labeled images were split into 205 for training, 53 for validation, and 97
for testing, with a balanced distribution of “brain” and “not brain” classes
in each set. Images are cropped and resized to 320x320. On the other hand, the
unlabeled volume slices are split into 6 folds (1 for each volume), and we
train 6 different segmentation models in a leave-one-out manner with 3
training and 2 validation volumes (combining a fixed labeled training set with
leave-one-out unlabeled folds). We do this to enable evaluating the downstream
pose regression model using Leave One Out Cross-Validation (LOOCV) without any
data leakage.

Each segmentation model is trained in 3 stages. First, we train a model S-Seg
in a fully supervised manner on the fixed labeled training set for 50 epochs
with a learning rate $lr=0.003$. Flipping, rotation, gaussian noise, elastic
transform, brightness and contrast change augmentations are added. Then we add
the unlabeled training set and obtain the model SS-Seg by training 150 more
epochs, with $lr=0.008$, using a StepLR scheduler with a step size of 50
epochs and a reduction rate $\gamma=0.5$. Finally, the classification head is
added, and we obtain a model SS-Seg+Class by further fine-tuning for 150
epochs with $lr=0.00003$. We use a batch size of 8 during the entire process.

We compare our results against AutoFB [7], which performs 3-class segmentation
of the femur, brain, and abdomen. It was trained in a fully supervised manner
on the same labeled training set as our model for 600 epochs with early
stopping (patience of 50 epochs). Table 1 reports the mean Intersection over
Union (IoU) (mIoU) for the labeled test set. We also report mean IoU between
pairs of slices at the same pose from the validation sets to measure
segmentation consistency in arbitrary slices. For SS-Seg and SS-Seg+Class, we
report the average performance of our 6 trained models, while for S-Seg and
AutoFB, we only report one model trained on the fixed labeled training set.

Table 1: Mean Intersection over Union across images obtained for the experiments performed with the LOOCV for AutoFB [7], S-S, SS-S, and SS-S+Class. mIoU | AutoFB [7] | S-Seg | SS-Seg | SS-Seg$+$Class
---|---|---|---|---
Test Labeled | 0.9170 | 0.9537 | 0.9144 | 0.9482
Validation Unlabeled | 0.7338 | 0.5660 | 0.7975 | 0.8278

####  3.0.3 Plane pose regression

For both training and inference with our pose regression model, we start by
taking brain segmentation predictions, dilating them with a 30x30 kernel, and
masking out the irrelevant portions of the image. We train 6 pose regression
models on masked images (one for each segmentation model) to perform LOOCV. We
use 20% of the training data for validation. The network was trained for 200
epochs with a batch size of 64 using Adam optimizer, with a learning rate
$lr=0.0001$. We report the performance of our model with and without
segmentation masking in Table 2.

Table 2: Average translation and rotation errors of our pose regression model
with and without segmentation masking.

| transmedian | transmean | transmin | transmax | rotmedian | rotmean | rotmin | rotmax
---|---|---|---|---|---|---|---|---
No masks [*] | 2.97 | 4.11 | 0.05 | 47.48 | 6.63 | 8.96 | 0.16 | 126.52
With masks | 2.49 | 3.35 | 0.02 | 42.53 | 5.62 | 6.96 | 0.09 | 109.32

####  3.0.4 Ultrasound video

(a) Distance to SPs

(b) Quality scores

Figure 4: (a) Distances to TV SP and SonoNet’s predictions. (b) Obstetrician
assessment of the TV SP per operator, with SP re-acquisitions for operators 4
and 5 due to required structures not being visible and the plane being too
oblique.

The SS-Seg+Class network was applied to the US videos described in Section
3.0.1. We extract frames from the US videos at a frame rate of 10 Hz to have a
manageable number of frames for analysis, reducing computational load while
retaining sufficient temporal resolution to capture the relevant US imagery.
Upon extraction, the frames were fed to the SS-Seg+Class network to detect the
fetal brain and isolate the area of interest from the surrounding anatomical
structures. The brain images identified and masked were then fed to the plane
pose regression network for obtaining the 6D poses. For each frame, we compute
the distance of the plane to the TV SP, previously annotated on the US
volumes. The obstetrician was asked to score the frozen TV SP individually for
quality control using the scoring system described by Salomon et al. [23]. The
reviewer was blinded to whether the images being scored were from the novice
or the expert groups. In parallel, we fed the entire video through a pre-
trained SonoNet in its PyTorch
implementation222https://github.com/baumgach/SonoNet-weights Figure 4 shows
the translation and rotation distances to the TV SP, along with SonoNet’s
predicted labels. Besides, critical moments in the video, such as when the
sonographer freezes and unfreezes the image, are highlighted, along with the
appearance of the frozen SP and the score assigned.

##  4 Discussion

Our SS-Seg+Class model shows robustness on both labeled and unlabeled data.
Our model outperforms the others in segmenting SPs and non-SPs with improved
mean IoU scores across images of 0.9482 on labeled images and 0.8272 on
unlabeled images thanks to the inter-patient loss function (Table 1). The
improved metrics with the use of masked images highlight the advancements in
plane pose regression (Table 2). The adaptation of a model trained on static
3D US data to generalize to 2D US videos underscores the innovative potential
of our approach. The predicted distances to the SP, along with SonoNet’s
predicted labels, offer insights into the model’s precision in identifying and
segmenting fetal brain structures (Figure 4(a)). The distance predictions
indicate a tendency for alignment with the expert assessment of SP quality
(Figure 4(b)). However, there are exceptions where the error in prediction
arises from factors such as zooming (Op. 2) or in-plane translation, that is
when the image is not well centered but still of good quality (Op. 4). These
observations suggest that while the model shows potential in assisting
sonographers to identify high-quality SPs with a degree of accuracy, these
cases highlight areas for improvement. Notably, these findings are derived
from a limited sample size, which underscores the need for further large-scale
studies to establish the model’s effectiveness and reliability conclusively.
Additionally, the model’s current training relies on a modest volume count,
suggesting that expanding the training data with additional volumes or video
data and integrating temporal models could further improve pose regression
quality. Besides, the model could be applied to other brain SPs or other
anatomical regions; nonetheless, extending generalized whole-fetus guidance
may encounter obstacles due to fetal pose variability. Additionally, achieving
real-time operation is critical for the practical application of this model in
clinical settings. The feasibility of real-time operation with the current
architecture remains a question for further investigation.

Envisioning a future where sonographers receive information on the distance to
SPs, we recognize the need for usability studies to assess whether more
sophisticated navigation information can effectively aid operators in real-
time.

##  5 Conclusion

This paper presents a novel approach to fetal US analysis, combining semi-
supervised segmentation and classification of brain images with sensorless
proximity detection to the TV SP. While traditional methods focus on SP
identification, we emphasize the utility of recognizing the probe’s nearing
SPs, facilitating earlier and more precise adjustments during scanning. Future
work will target increased robustness to zoom variations and off-center
anatomies typical in scans, as well as improving detection in cases of
abnormal fetal development.

## References

  * [1] Hadlock, F.P., Harrist, R.B., Sharman, R.S., Deter, R.L., Park, S.K.: Estimation of fetal weight with the use of head, body, and femur measurements: A prospective study. American Journal of Obstetrics and Gynecology 151(3), 333–337 (1985)
  * [2] Sarris, I., Ioannou, C., Dighe, M., Mitidieri, A., Oberto, M., Qingqing, W., Shah, J., Sohoni, S., Al Zidjali, W., Hoch, L., Altman, D.G., Papageorghiou, A.T.: Standardization of fetal ultrasound biometry measurements: Improving the quality and consistency of measurements. Ultrasound in Obstetrics and Gynecology 38(6), 681–687 (2011)
  * [3] Bahner, D.P., Blickendorf, J.M., Bockbrader, M., Adkins, E., Vira, A., Boulger, C., Panchal, A.R.: Language of Transducer Manipulation: Codifying Terms for Effective Teaching. Journal of Ultrasound in Medicine 35(1), 183–188 (2016)
  * [4] Sharma, H., Drukker, L., Chatelain, P., Droste, R., Papageorghiou, A.T., Noble, J.A.: Knowledge representation and learning of operator clinical workflow from full-length routine fetal ultrasound scan videos. Medical Image Analysis 69 (2021)
  * [5] Salomon, L.J., Alfirevic, Z., Da Silva Costa, F., Deter, R.L., Figueras, F., Ghi, T., Glanc, P., Khalil, A., Lee, W., Napolitano, R., Papageorghiou, A., Sotiradis, A., Stirnemann, J., Toi, A., Yeo, G.: ISUOG Practice Guidelines: ultrasound assessment of fetal biometry and growth. Ultrasound in Obstetrics and Gynecology 53(6) (2019)
  * [6] Baumgartner, C.F., Kamnitsas, K., Matthew, J., Fletcher, T.P., Smith, S., Koch, L.M., Kainz, B., Rueckert, D.: SonoNet: Real-Time Detection and Localisation of Fetal Standard Scan Planes in Freehand Ultrasound. IEEE Transactions on Medical Imaging 36(11), 2204–2215 (2017)
  * [7] Bano, S., Dromey, B., Vasconcelos, F., Napolitano, R., David, A.L., Peebles, D.M., Stoyanov, D.: AutoFB: Automating Fetal Biometry Estimation from Standard Ultrasound Planes. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 12907 LNCS, 228–238 (2021)
  * [8] Avisdris, N., Joskowicz, L., Dromey, B., David, A.L., Peebles, D.M., Stoyanov, D., Ben Bashat, D., Bano, S.: Biometrynet: Landmark-based fetal biometry estimation from standard ultrasound planes. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 279–289. Springer (2022)
  * [9] Gao, Y., Beriwal, S., Craik, R., Papageorghiou, A.T., Noble, J.A.: Label Efficient Localization of Fetal Brain Biometry Planes in Ultrasound Through Metric Learning. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 12437 LNCS, 126–135 (2020)
  * [10] Rahmatullah, B., Papageorghiou, A., Noble, J.A.: Automated selection of standardized planes from ultrasound volume. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). pp. 35–42. Springer (2011)
  * [11] Cuingnet, R., Somphone, O., Mory, B., Prevost, R., Yaqub, M., Napolitano, R., Papageorghiou, A., Roundhill, D., Noble, J., Ardon, R.: Where is my baby? a fast fetal head auto-alignment in 3d-ultrasound. In: 2013 IEEE 10th International Symposium on Biomedical Imaging. pp. 768–771 (2013)
  * [12] Droste, R., Drukker, L., Papageorghiou, A.T., Noble, J.A.: Automatic Probe Movement Guidance for Freehand Obstetric Ultrasound. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) pp. 583–592 (2020)
  * [13] Birlo, M., Edwards, P.J.E., Yoo, S., Dromey, B., Vasconcelos, F., Clarkson, M.J., Stoyanov, D.: CAL-Tutor: A HoloLens 2 Application for Training in Obstetric Sonography and User Motion Data Recording. Journal of Imaging 9(1), 6 (2022)
  * [14] Di Vece, C., Dromey, B., Vasconcelos, F., David, A.L., Peebles, D., Stoyanov, D.: Deep learning-based plane pose regression in obstetric ultrasound. International Journal of Computer Assisted Radiology and Surgery 17(5), 833–839 (2022)
  * [15] Di Vece, C., Le Lous, M., Dromey, B., Vasconcelos, F., David, A.L., Peebles, D., Stoyanov, D.: Ultrasound plane pose regression: assessing generalized pose coordinates in the fetal brain. IEEE Transactions on Medical Robotics and Bionics (2023)
  * [16] Yeung, P.H., Aliasi, M., Papageorghiou, A.T., Haak, M., Xie, W., Namburete, A.I.: Learning to map 2D ultrasound images into 3D space with minimal human annotation. Medical Image Analysis 70 (2021)
  * [17] Yeung, P.H., Aliasi, M., Haak, M., Xie, W., Namburete, A.I.L.: Adaptive 3D Localization of 2D Freehand Ultrasound Brain Images. In: LNCS. vol. 13434, pp. 207–217. Springer, Cham (2022)
  * [18] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 9351, 234–241 (2015)
  * [19] Tan, M., Le, Q.V.: EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
  * [20] Deng, J., Dong, W., Socher, R., Li, L.J., Kai Li, Li Fei-Fei: ImageNet: A large-scale hierarchical image database pp. 248–255 (3 2010)
  * [21] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. pp. 770–778 (2016)
  * [22] Pistorius, L.R., Stoutenbeek, P., Groenendaal, F., De Vries, L., Manten, G., Mulder, E., Visser, G.: Grade and symmetry of normal fetal cortical development: A longitudinal two- and three-dimensional ultrasound study. Ultrasound in Obstetrics and Gynecology 36(6), 700–708 (2010)
  * [23] Salomon, L., Winer, N., Bernard, J., Ville, Y.: A score-based method for quality control of fetal images at routine second-trimester ultrasound examination. Prenatal Diagnosis: Published in Affiliation With the International Society for Prenatal Diagnosis 28(9), 822–827 (2008)

  *[SPs]: standard plane
  *[2D]: two-dimensional
  *[US]: ultrasound
  *[3D]: three-dimensional
  *[6D]: six-dimensional
  *[SP]: standard plane
  *[TV]: transventricular
  *[MSE]: mean square error
  *[CNN]: convolutional neural network
  *[GA]: gestational age
  *[LOOCV]: Leave One Out Cross-Validation
  *[IoU]: Intersection over Union


have biases encoded in their evaluation and opinions. Similarly, in terms of
transparency, new algorithms require transparency, but the human process of
decision making also requires transparency, which can be improved in existing
processes as well. Therefore, the development of NLP assistance and the
subsequent policy decisions should be made in consideration of the trade-off
between risks, advantages, and benefits in the existing systems, and the
risks, advantages, and benefits introduced by the new systems. We outline
several areas of concern in NLP for peer review from an ethics standpoint.

Bias:

Deploying NLP assistance for peer review bears the risk of reinforcing and
augmenting existing biases in the process (Shah et al.,, 2020). Unfair
decisions will not magically go away, and authors will always disagree with
rejections (Goldberg et al., 2023b, , Section 4.2). Groups of people or areas
of research might feel excluded and this might lead to allocational harm
(i.e., high-stakes impact of unfair decisions). NLP assistance models can pick
up on idiolects of people who frequently published in the past. Thus, in the
process of designing new systems, special attention needs to be paid to
measuring and mitigating potential biases, old and new.

Transparency:

Given the importance of peer review outcomes for scientific process and
research careers, a high level of transparency is desired. Transparency of
machine-assisted peer review can be increased by explicitly stating what types
of NLP assistance are deployed in a given reviewing campaign, publishing
detailed descriptions of the models used (e.g., on blogs) and model cards.
Useful information includes: how the tools are built, deployed, what the tools
are capable of—what they can and cannot do, how are the decisions made whether
to use these tools and to what extent, and how is data privacy addressed.
Using open-source models promotes scientific openness and reproducibility—yet
increases the risks of malicious behavior by adversarial parties who can
exploit vulnerabilities in these tools. Related to transparency is
explainability: while explainable NLP is a major research direction, there are
hardly any works that explicitly target explainability in NLP for peer
reviewing (Kim et al.,, 2023)—an important area for future study.

Agency:

With NLP in the loop, who is responsible for the outcome? If an NLP assistance
module produces a faulty prediction, is it the fault of the developers who
created it, the conference organizers who deployed it, or the users who failed
to check the predictions? Given the ever-growing capabilities of NLP systems,
reviewers, authors, and editors alike may be inclined to outsource their
responsibility on the review to the “machine”, e.g., by submitting texts and
assessments automatically generated by LLMs with little further involvement.
Such cases need to be explicitly addressed by the community policy—for
example, it can be made clear that the user is responsible for the ultimate
content and quality of the work. Parallel to that, it must be ensured that the
mechanisms for reporting and recourse can adequately account for the
consequences of NLP-assisted submission, matching, reviewing, discussion, etc.

Privacy:

The sharing of peer-reviewing data and the use of models to assist in the peer
review process could potentially be abused to infer personal information about
the reviewers, threatening the integrity of the process. Following the example
by Jecmen et al., (2020), suppose the models used to assign reviewers to
papers, the data pertaining to publication history of each reviewer, and all
submissions are public. If the models and optimization procedure are
deterministic, anyone can re-run these models to reconstruct the assignment of
reviewers to papers, thereby compromising the anonymity of the review process.
Ding et al., (2020, 2022) provide further examples. Additional concerns
include reidentification (also called deanonymization) by analyzing the
writing patterns from publicly available reviews and matching them to the
publicly available papers. In addition to the careful legal treatment of
reviewing data and its derivatives (Section 7), privacy-preserving techniques
such as differential privacy (Dwork and Roth,, 2014; Klymenko et al.,, 2022)
and anonymization (Li et al.,, 2007) should be explored to minimize risks to
the participants of the peer-reviewing process when sharing data. A disclaimer
prohibiting the use of data and models for author profiling and
reidentification should be included with the release of the respective
resources.

We stress that it is not possible to foresee all potential consequences of NLP
assistance for peer review. Given the high-stakes nature of the process, once
NLP systems are deployed, the users will likely adjust their behavior over
time to gain further benefits from the system—intentionally or not. We note
that this is not unique to NLP automation; deployment of any new policy can
have a similar effect. This highlights the importance of closely monitoring
the changes in the reviewing process in response to new policies and tools.

##  10 Conclusion: A Call for Action

Peer review is a critical part of modern science. As science accelerates, peer
review faces new challenges related to logistics, cost, bias, low-quality
reviewing and strategic behavior. In this white paper, we have discussed the
potential ways in which natural language processing can help make the peer-
reviewing process more efficient and robust, while taking into account
potential legal, methodological, and ethical challenges. We believe that this
research area has great potential to contribute to the study and advancement
of the scientific process in the age of AI. The natural language processing
and machine learning communities are uniquely positioned to study AI for peer
review, as we practice peer review ourselves (and thus can reap the benefits
of AI assistance), establish our own reviewing policies, and have the
capability to develop tools that can alleviate some of the pressing issues
with the process. We thus finish the paper with a call for action—the steps
that individual readers of this work can take to help advance NLP for peer
review.

In their role as scientists, we invite the readers to _participate in the
discussion_. The goal of machine-assisted peer review is to make the lives of
researchers better—and this can not be done without taking the voice of the
community into account. Openly available reviewing data is scarce. If you can,
consider donating your data for research use—naturally, given that the data is
collected according to adequate standards of licensing, personal data and
privacy protection. NLP assistance for peer review seeks to help experts.
Experts are hard to recruit on crowdworking platforms and to capture in public
polls. When possible, _participate in the controlled experiments_ , _respond
to surveys_ , and give feedback on the deployed assistance tools.

We invite NLP and AI researchers to _contribute work_. Peer review offers a
plethora of applications for any area of NLP—from summarization to ethics to
applied user studies. To help researchers get started, our companion
repository (https://github.com/OAfzal/nlp-for-peer-review) compiles a set of
relevant datasets pertaining to peer review. It will be regularly updated and
welcomes new contributions. We call NLP and AI researchers working in the
peer-reviewing domain to _observe good scientific practice_. This includes
responsible handling of peer-reviewing data, methodological clarity and rigor,
attention to the ethical implications of the work, and honesty in
communicating the results and outlining the risks. Finally, _help build the
community_. NLP for peer review is an emerging topic that attracts interest
both within and outside NLP. Promote relevant existing work, participate in
related shared tasks, join non-AI meetings on related topics,181818Such as the
International Congress on Peer Review and Scientific Publication,
https://peerreviewcongress.org. and consider organizing new meetings that will
help grow and consolidate the community.

We call on policymakers for peer-reviewed venues to consider the opportunities
of NLP assistance in peer review, but also to account for the risks. Fostering
a culture of responsible tool use requires new policies. We believe four
aspects to be of particular importance. First, machine-assisted quality
control in science calls for _transparency_. It must be clear who decides to
use what tool and for which purpose, how these tools are developed, and what
risks they bear. Second, it is important for the community to have trust in
the tools they are using. Thus, _explainable_ methods for reviewing support
should be given preference. Next, deploying NLP assistance in live reviewing
systems calls for a high degree of quality control: new features should be
rigorously tested, and processes must be in place to revert the use of new
tools. This calls for careful, _incremental deployment_ of NLP assistance for
peer review: selecting new features carefully, observing their impact and the
way they affect the community, and evaluating them before introducing the next
bit of technology. Finally, deployments of NLP assistance (as well as any
other systematic change to the process) call for careful _monitoring_ , and
all participants of peer review—authors, reviewers, meta-reviewers and program
chairs—should be given clear _feedback channels_ that will help inform further
policy changes.

Finally, we invite the readers affiliated with research-sponsoring
organizations or serving as their reviewers to prioritize research questions
and investments that will _enable progress in NLP and AI research for
scholarly peer review_. As science grows, peer review in many disciplines and
research communities faces challenges. Inefficient peer review diverts the
public resources from producing new research and training the next generation
of scientists. Unreliable peer review, in turn, threatens the integrity of the
scientific process and undermines public trust in scientific work. We believe
that the NLP and AI research communities should be at the forefront of
developing and using tools to support peer review, and should be among the
beneficiaries of the recent progress in AI that they have enabled.

## Contributions

Coordination: Ilia Kuznetsov;

Main text: Osama Mohammed Afzal, Koen Dercksen, Nils Dycke, Alexander
Goldberg, Tom Hope, Dirk Hovy, Jonathan K. Kummerfeld, Ilia Kuznetsov, Anne
Lauscher, Kevin Leyton-Brown, Sheng Lu, Mausam, Margot Mieskes, Aurélie
Névéol, Danish Pruthi, Lizhen Qu, Anna Rogers, Roy Schwartz, Nihar Shah, Noah
A. Smith, Thamar Solorio, Jingyan Wang, Xiaodan Zhu.

Repository: Osama Mohammed Afzal, Nils Dycke, Ilia Kuznetsov, Sheng Lu,
Aurélie Névéol, Nihar Shah.

Review and feedback: Nils Dycke, Iryna Gurevych, Ilia Kuznetsov, Anne
Lauscher, Kevin Leyton-Brown, Margot Mieskes, Anna Rogers, Nihar Shah, Noah A.
Smith, Jingyan Wang.

Organisation: Iryna Gurevych, Anna Rogers, Nihar Shah.

## Acknowledgments

The work of Nihar B. Shah and Alexander Goldberg was supported by NSF 1942124
and ONR N000142212181. The work of Ilia Kuznetsov, Nils Dycke, Sheng Lu and
Iryna Gurevych was supported by the LOEWE Distinguished Chair “Ubiquitous
Knowledge Processing” (LOEWE initiative, Hesse, Germany) and co-funded by the
European Union (ERC, InterText, 101054961), the German Research Foundation
(DFG) as part of the PEER project (grant GU 798/28-1) and by the German
Federal Ministry of Education and Research and the Hessian Ministry of Higher
Education, Research, Science and the Arts within their joint support of the
National Research Center for Applied Cybersecurity ATHENE. The work of Kevin
Leyton-Brown was funded by an NSERC Discovery Grant and a CIFAR Canada AI
Research Chair (Alberta Machine Intelligence Institute). The work of Anne
Lauscher is funded under the Excellence Strategy of the German Federal
Government and the States. Mausam was funded by IBM-IITD AI Horizons network,
grants from Google, and a Jai Gupta chair fellowship. The authors would like
to thank Bahar Mehmani and Dennis Zyska for their valuable input and feedback.
Finally, the authors would like to thank David Kunz, because “why wouldn’t
they? Nobody is complaining.”

## References

  * AbuRa’ed et al., (2017) AbuRa’ed, A., Chiruzzo, L., and Saggion, H. (2017).  What sentence are you referring to and why? Identifying cited sentences in scientific literature.  In Mitkov, R. and Angelova, G., editors, Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, pages 9–17, Varna, Bulgaria. INCOMA Ltd.
  * AJE, (2013) AJE (2013).  Peer review: How we found 15 million hours of lost time.  https://www.aje.com/arc/peer-review-process-15-million-hours-lost-time/.  Online; accessed 01-March-2024.
  * Ali et al., (2020) Ali, Z., Kefalas, P., Muhammad, K., Ali, B., and Imran, M. (2020).  Deep learning in citation recommendation models survey.  Expert Systems with Applications, 162.
  * Altmäe et al., (2023) Altmäe, S., Sola-Leyva, A., and Salumets, A. (2023).  Artificial intelligence in scientific writing: A friend or a foe?  Reproductive BioMedicine Online, 47(1):3–9.
  * Amado Olivo and Kerzendorf, (2023) Amado Olivo, V. and Kerzendorf, W. (2023).  Development of a global registry for peer review in astrophysics.  In Peer Review Under Review, page 13.
  * Belouadi et al., (2023) Belouadi, J., Lauscher, A., and Eger, S. (2023).  AutomaTikZ: Text-guided synthesis of scientific vector graphics with TikZ.  In The Twelfth International Conference on Learning Representations (ICLR).
  * Beltagy et al., (2021) Beltagy, I., Cohan, A., Feigenblat, G., Freitag, D., Ghosal, T., Hall, K., Herrmannova, D., Knoth, P., Lo, K., Mayr, P., Patton, R. M., Shmueli-Scheuer, M., de Waard, A., Wang, K., and Wang, L. L., editors (2021).  Proceedings of the Second Workshop on Scholarly Document Processing. Association for Computational Linguistics.
  * Belz, (2022) Belz, A. (2022).  A metrological perspective on reproducibility in NLP.  Computational Linguistics, 48(4):1125–1135.
  * Belz et al., (2023) Belz, A., Thomson, C., Reiter, E., and Mille, S. (2023).  Non-repeatable experiments and non-reproducible results: The reproducibility crisis in human evaluation in NLP.  In Rogers, A., Boyd-Graber, J., and Okazaki, N., editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 3676–3687, Toronto, Canada. Association for Computational Linguistics.
  * Bender and Friedman, (2018) Bender, E. M. and Friedman, B. (2018).  Data statements for natural language processing: Toward mitigating system bias and enabling better science.  Transactions of the Association for Computational Linguistics, 6:587–604.
  * (11) Beygelzimer, A., Dauphin, Y., Liang, P., and Wortman Vaughan, J. (2021a).  Introducing the NeurIPS 2021 Paper Checklist.  https://neuripsconf.medium.com/introducing-the-neurips-2021-paper-checklist-3220d6df500b.  Online; accessed 17-April-2024.
  * (12) Beygelzimer, A., Dauphin, Y., Liang, P., and Wortman Vaughan, J. (2021b).  The NeurIPS 2021 consistency experiment.  https://blog.neurips.cc/2021/12/08/the-neurips-2021-consistency-experiment/.  Online; accessed 18-April-2024.
  * Bhagavatula et al., (2018) Bhagavatula, C., Feldman, S., Power, R., and Ammar, W. (2018).  Content-based citation recommendation.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 238–251.
  * Bharti et al., (2023) Bharti, P., Navlakha, M., Agarwal, M., and Ekbal, A. (2023).  PolitePEER: does peer review hurt? A dataset to gauge politeness intensity in the peer reviews.  Language Resources and Evaluation, pages 1–23.
  * Biswas et al., (2023) Biswas, S., Dobaria, D., and Cohen, H. L. (2023).  ChatGPT and the future of journal reviews: A feasibility study.  The Yale Journal of Biology and Medicine, 96:415–420.
  * Blecher et al., (2023) Blecher, L., Cucurull, G., Scialom, T., and Stojnic, R. (2023).  Nougat: Neural optical understanding for academic documents.  In The Twelfth International Conference on Learning Representations (ICLR).
  * Bornmann, (2011) Bornmann, L. (2011).  Scientific peer review.  Annual Review of Information Science and Technology, 45(1):197–245.
  * Cachola et al., (2020) Cachola, I., Lo, K., Cohan, A., and Weld, D. (2020).  TLDR: Extreme summarization of scientific documents.  In Cohn, T., He, Y., and Liu, Y., editors, Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4766–4777. Association for Computational Linguistics.
  * Çano and Bojar, (2020) Çano, E. and Bojar, O. (2020).  Two huge title and keyword generation corpora of research articles.  In Calzolari, N., Béchet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Moreno, A., Odijk, J., and Piperidis, S., editors, Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6663–6671, Marseille, France. European Language Resources Association.
  * Caragea et al., (2014) Caragea, C., Bulgarov, F. A., Godea, A., and Das Gollapalli, S. (2014).  Citation-enhanced keyphrase extraction from research papers: A supervised approach.  In Moschitti, A., Pang, B., and Daelemans, W., editors, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1435–1446, Doha, Qatar. Association for Computational Linguistics.
  * Chakraborty et al., (2020) Chakraborty, S., Goyal, P., and Mukherjee, A. (2020).  Aspect-based sentiment analysis of scientific reviews.  In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, JCDL ’20, page 207–216, New York, NY, USA. Association for Computing Machinery.
  * Chandrasekaran et al., (2020) Chandrasekaran, M. K., de Waard, A., Feigenblat, G., Freitag, D., Ghosal, T., Hovy, E., Knoth, P., Konopnicki, D., Mayr, P., Patton, R. M., and Shmueli-Scheuer, M., editors (2020).  Proceedings of the First Workshop on Scholarly Document Processing. Association for Computational Linguistics.
  * Charlin and Zemel, (2013) Charlin, L. and Zemel, R. (2013).  The Toronto Paper Matching System: An automated paper-reviewer assignment system.  In Proceedings of the 30th International Conference on Machine Learning, volume 28, Atlanta, Georgia, USA. Journal of Machine Learning Research Workshop and Conference Proceedings.
  * Cheng et al., (2020) Cheng, L., Bing, L., Yu, Q., Lu, W., and Si, L. (2020).  APE: Argument pair extraction from peer review and rebuttal via multi-task learning.  In Webber, B., Cohn, T., He, Y., and Liu, Y., editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7000–7011. Association for Computational Linguistics.
  * Chouldechova, (2017) Chouldechova, A. (2017).  Fair prediction with disparate impact: A study of bias in recidivism prediction instruments.  Big Data, 5(2):153–163.
  * Church, (2020) Church, K. W. (2020).  Emerging trends: Reviewing the reviewers (again).  Natural Language Engineering, 26(2):245–257.
  * Cohan et al., (2022) Cohan, A., Feigenblat, G., Freitag, D., Ghosal, T., Herrmannova, D., Knoth, P., Lo, K., Mayr, P., Shmueli-Scheuer, M., de Waard, A., and Wang, L. L., editors (2022).  Proceedings of the Third Workshop on Scholarly Document Processing, volume 29, Gyeongju, Republic of Korea. Association for Computational Linguistics.
  * Cohan et al., (2020) Cohan, A., Feldman, S., Beltagy, I., Downey, D., and Weld, D. S. (2020).  Specter: Document-level representation learning using citation-informed transformers.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2270–2282.
  * D’Arcy et al., (2024) D’Arcy, M., Hope, T., Birnbaum, L., and Downey, D. (2024).  MARG: Multi-agent review generation for scientific papers.  arXiv preprint arXiv:2401.04259.
  * D’Arcy et al., (2023) D’Arcy, M., Ross, A., Bransom, E., Kuehl, B., Bragg, J., Hope, T., and Downey, D. (2023).  ARIES: A corpus of scientific paper edits made in response to peer reviews.  arXiv preprint arXiv:2306.12587.
  * Daumé III, (2013) Daumé III, H. (2013).  Some NAACL 2013 statistics on author response, review quality, etc.  https://nlpers.blogspot.com/2015/06/some-naacl-2013-statistics-on-author.html.  Online; accessed 21-April-2024.
  * De Kock and Vlachos, (2022) De Kock, C. and Vlachos, A. (2022).  How to disagree well: Investigating the dispute tactics used on Wikipedia.  In Goldberg, Y., Kozareva, Z., and Zhang, Y., editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3824–3837, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Ding et al., (2022) Ding, W., Kamath, G., Wang, W., and Shah, N. B. (2022).  Calibration with privacy in peer review.  In ISIT, pages 1635–1640. IEEE.
  * Ding et al., (2020) Ding, W., Shah, N. B., and Wang, W. (2020).  On the privacy-utility tradeoff in peer-review data analysis.  In AAAI Privacy-Preserving Artificial Intelligence (PPAI-21) workshop.
  * Dodge et al., (2019) Dodge, J., Gururangan, S., Card, D., Schwartz, R., and Smith, N. A. (2019).  Show Your Work: Improved reporting of experimental results.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2185–2194, Hong Kong, China. Association for Computational Linguistics.
  * Drori and Te’eni, (2024) Drori, I. and Te’eni, D. (2024).  Human-in-the-loop AI reviewing: Feasibility, opportunities, and risks.  Journal of the Association for Information Systems, 25(1):98–109.
  * Du et al., (2022) Du, W., Raheja, V., Kumar, D., Kim, Z. M., Lopez, M., and Kang, D. (2022).  Understanding iterative revision from human-written text.  In Muresan, S., Nakov, P., and Villavicencio, A., editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3573–3590, Dublin, Ireland. Association for Computational Linguistics.
  * Dwork and Roth, (2014) Dwork, C. and Roth, A. (2014).  The algorithmic foundations of differential privacy.  Foundations and Trends in Theoretical Computer Science, 9(3-4):211–407.
  * Dycke et al., (2022) Dycke, N., Kuznetsov, I., and Gurevych, I. (2022).  Yes-yes-yes: Proactive data collection for ACL rolling review and beyond.  In Goldberg, Y., Kozareva, Z., and Zhang, Y., editors, Findings of the Association for Computational Linguistics: EMNLP 2022, pages 300–318, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * (40) Dycke, N., Kuznetsov, I., and Gurevych, I. (2023a).  NLPeer: A unified resource for the computational study of peer review.  In Rogers, A., Boyd-Graber, J., and Okazaki, N., editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5049–5073, Toronto, Canada. Association for Computational Linguistics.
  * (41) Dycke, N., Kuznetsov, I., and Gurevych, I. (2023b).  Overview of PragTag-2023: Low-resource multi-domain pragmatic tagging of peer reviews.  In Alshomary, M., Chen, C.-C., Muresan, S., Park, J., and Romberg, J., editors, Proceedings of the 10th Workshop on Argument Mining, pages 187–196, Singapore. Association for Computational Linguistics.
  * Else and Van Noorden, (2021) Else, H. and Van Noorden, R. (2021).  The fight against fake-paper factories that churn out sham science.  Nature, 591(7851):516–519.
  * Ernst and Kienbacher, (1991) Ernst, E. and Kienbacher, T. (1991).  Chauvinism.  Nature, 352(6336):560–560.
  * Färber and Jatowt, (2020) Färber, M. and Jatowt, A. (2020).  Citation recommendation: Approaches and datasets.  International Journal on Digital Libraries, 21(4):375–405.
  * Fiez et al., (2020) Fiez, T., Shah, N., and Ratliff, L. (2020).  A SUPER* algorithm to optimize paper bidding in peer review.  In Conference on Uncertainty in Artificial Intelligence, pages 580–589. Proceedings of Machine Learning Research.
  * Fok et al., (2023) Fok, R., Kambhamettu, H., Soldaini, L., Bragg, J., Lo, K., Hearst, M., Head, A., and Weld, D. S. (2023).  Scim: Intelligent skimming support for scientific papers.  In Proceedings of the 28th International Conference on Intelligent User Interfaces, pages 476–490.
  * Frachtenberg and Koster, (2020) Frachtenberg, E. and Koster, N. (2020).  A survey of accepted authors in computer systems conferences.  PeerJ Computer Science, 6.
  * Fu et al., (2022) Fu, T., Wang, W. Y., McDuff, D., and Song, Y. (2022).  DOC2PPT: Automatic presentation slides generation from scientific documents.  In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 - March 1, 2022, pages 634–642. AAAI Press.
  * Gao et al., (2019) Gao, Y., Eger, S., Kuznetsov, I., Gurevych, I., and Miyao, Y. (2019).  Does my rebuttal matter? Insights from a major NLP conference.  In Burstein, J., Doran, C., and Solorio, T., editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1274–1290, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ge et al., (2013) Ge, H., Welling, M., and Ghahramani, Z. (2013).  A Bayesian model for calibrating conference review scores.  Manuscript.  Available online http://mlg.eng.cam.ac.uk/hong/unpublished/nips-review-model.pdf Last accessed: April 4, 2021.
  * Ghosal et al., (2022) Ghosal, T., Varanasi, K. K., and Kordoni, V. (2022).  HedgePeer: A dataset for uncertainty detection in peer reviews.  In Proceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries, pages 1–5.
  * Ginther et al., (2011) Ginther, D. K., Schaffer, W. T., Schnell, J., Masimore, B., Liu, F., Haak, L. L., and Kington, R. (2011).  Race, ethnicity, and NIH research awards.  Science, 333(6045):1015–1019.
  * Glockner et al., (2024) Glockner, M., Staliūnaitė, I., Thorne, J., Vallejo, G., Vlachos, A., and Gurevych, I. (2024).  Ambifc: Fact-checking ambiguous claims with evidence.  Transactions of the Association for Computational Linguistics, 12:1–18.
  * (54) Goldberg, A., Fanti, G., and Shah, N. B. (2023a).  Batching of tasks by users of pseudonymous forums: Anonymity compromise and protection.  Proceedings of the ACM on Measurement and Analysis of Computing Systems, 7(1).
  * (55) Goldberg, A., Stelmakh, I., Cho, K., Oh, A., Agarwal, A., Belgrave, D., and Shah, N. B. (2023b).  Peer reviews of peer reviews: A randomized controlled trial and other experiments.  arXiv preprint arXiv:2311.09497.
  * Goldberg, (1968) Goldberg, P. (1968).  Are women prejudiced against women?  Trans-action, 5(5):28–30.
  * GSPR, (2018) GSPR (2018).  Global State of Peer Review 2018.  Wellington: Publons.
  * Guo et al., (2023) Guo, Y., Shang, G., Rennard, V., Vazirgiannis, M., and Clavel, C. (2023).  Automatic analysis of substantiation in scientific peer reviews.  In Bouamor, H., Pino, J., and Bali, K., editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10198–10216, Singapore. Association for Computational Linguistics.
  * Head et al., (2022) Head, A., Xie, A., and Hearst, M. A. (2022).  Math augmentation: How authors enhance the readability of formulas using novel visual design practices.  In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pages 1–18.
  * Hofstra et al., (2020) Hofstra, B., Kulkarni, V. V., Galvez, S. M.-N., He, B., Jurafsky, D., and McFarland, D. A. (2020).  The diversity–innovation paradox in science.  Proceedings of the National Academy of Sciences, 117(17):9284–9291.
  * Hu and Wan, (2013) Hu, Y. and Wan, X. (2013).  Ppsgen: learning to generate presentation slides for academic papers.  In Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence, pages 2099–2105.
  * Hua et al., (2019) Hua, X., Nikolov, M., Badugu, N., and Wang, L. (2019).  Argument mining for understanding peer reviews.  In Burstein, J., Doran, C., and Solorio, T., editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2131–2137, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Jecmen et al., (2024) Jecmen, S., Shah, N. B., Fang, F., and Akoglu, L. (2024).  On the detection of reviewer-author collusion rings from paper bidding.  arXiv preprint arXiv:2402.07860.
  * Jecmen et al., (2022) Jecmen, S., Shah, N. B., Fang, F., and Conitzer, V. (2022).  Tradeoffs in preventing manipulation in paper bidding for reviewer assignment.  arXiv preprint arXiv:2207.11315.
  * Jecmen et al., (2023) Jecmen, S., Yoon, M., Conitzer, V., Shah, N. B., and Fang, F. (2023).  A dataset on malicious paper bidding in peer review.  In Proceedings of the ACM Web Conference 2023, pages 3816–3826.
  * Jecmen et al., (2020) Jecmen, S., Zhang, H., Liu, R., Shah, N., Conitzer, V., and Fang, F. (2020).  Mitigating manipulation in peer review via randomized reviewer assignments.  Advances in Neural Information Processing Systems, 33:12533–12545.
  * Jiang et al., (2022) Jiang, C., Xu, W., and Stevens, S. (2022).  arXivEdits: Understanding the human revision process in scientific writing.  In Goldberg, Y., Kozareva, Z., and Zhang, Y., editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9420–9435, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Jurgens et al., (2018) Jurgens, D., Kumar, S., Hoover, R., McFarland, D., and Jurafsky, D. (2018).  Measuring the evolution of a scientific field through citation frames.  Transactions of the Association for Computational Linguistics, 6:391–406.
  * Kamocki et al., (2018) Kamocki, P., Mapelli, V., and Choukri, K. (2018).  Data management plan (DMP) for language data under the new general data protection regulation (GDPR).  In Calzolari, N., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Hasida, K., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Moreno, A., Odijk, J., Piperidis, S., and Tokunaga, T., editors, Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), pages 135–139, Miyazaki, Japan. European Language Resources Association.
  * Kamocki and Witt, (2020) Kamocki, P. and Witt, A. (2020).  Privacy by design and language resources.  In Calzolari, N., Béchet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Moreno, A., Odijk, J., and Piperidis, S., editors, Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3423–3427, Marseille, France. European Language Resources Association.
  * Kang et al., (2018) Kang, D., Ammar, W., Dalvi, B., van Zuylen, M., Kohlmeier, S., Hovy, E., and Schwartz, R. (2018).  A dataset of peer reviews (PeerRead): Collection, insights and NLP applications.  In Walker, M., Ji, H., and Stent, A., editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1647–1661, New Orleans, Louisiana. Association for Computational Linguistics.
  * Kennard et al., (2022) Kennard, N., O’Gorman, T., Das, R., Sharma, A., Bagchi, C., Clinton, M., Yelugam, P. K., Zamani, H., and McCallum, A. (2022).  DISAPERE: A dataset for discourse structure in peer review discussions.  In Carpuat, M., de Marneffe, M.-C., and Meza Ruiz, I. V., editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1234–1249, Seattle, United States. Association for Computational Linguistics.
  * Kim et al., (2023) Kim, J. S., Chen, V., Pruthi, D., Shah, N. B., and Talwalkar, A. (2023).  Assisting human decisions in document matching.  Transactions on Machine Learning Research.
  * Kleinberg et al., (2016) Kleinberg, J., Mullainathan, S., and Raghavan, M. (2016).  Inherent trade-offs in the fair determination of risk scores.  arXiv preprint arXiv:1609.05807.
  * Klymenko et al., (2022) Klymenko, O., Meisenbacher, S., and Matthes, F. (2022).  Differential privacy in natural language processing the story so far.  In Feyisetan, O., Ghanavati, S., Thaine, P., Habernal, I., and Mireshghallah, F., editors, Proceedings of the Fourth Workshop on Privacy in Natural Language Processing, pages 1–11, Seattle, United States. Association for Computational Linguistics.
  * Kobren et al., (2019) Kobren, A., Saha, B., and McCallum, A. (2019).  Paper matching with local fairness constraints.  In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1247–1257.
  * Kumar et al., (2021) Kumar, A., Ghosal, T., and Ekbal, A. (2021).  A deep neural architecture for decision-aware meta-review generation.  In 2021 ACM/IEEE Joint Conference on Digital Libraries (JCDL), pages 222–225.
  * Künzli et al., (2022) Künzli, N., Berger, A., Czabanowska, K., Lucas, R., Madarasova Geckova, A., Mantwill, S., and von Dem Knesebeck, O. (2022).  «I Do Not Have Time» - is this the end of peer review in public health sciences?  Public Health Reviews, 43.
  * Kuznetsov et al., (2022) Kuznetsov, I., Buchmann, J., Eichler, M., and Gurevych, I. (2022).  Revise and Resubmit: An intertextual model of text-based collaboration in peer review.  Computational Linguistics, 48(4):949–986.
  * Landhuis, (2016) Landhuis, E. (2016).  Scientific literature: Information overload.  Nature, 535(7612):457–458.
  * Lauscher et al., (2018) Lauscher, A., Glavaš, G., Ponzetto, S. P., and Eckert, K. (2018).  Investigating the role of argumentation in the rhetorical analysis of scientific publications with neural multi-task learning models.  In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J., editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3326–3338, Brussels, Belgium. Association for Computational Linguistics.
  * Lauscher et al., (2022) Lauscher, A., Ko, B., Kuehl, B., Johnson, S., Cohan, A., Jurgens, D., and Lo, K. (2022).  MultiCite: Modeling realistic citations requires moving beyond the single-sentence single-label setting.  In Carpuat, M., de Marneffe, M.-C., and Meza Ruiz, I. V., editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1875–1889, Seattle, United States. Association for Computational Linguistics.
  * Lawrence and Cortes, (2014) Lawrence, N. and Cortes, C. (2014).  The NIPS Experiment.  http://inverseprobability.com/2014/12/16/the-nips-experiment.  Online; accessed 17-April-2024.
  * Lee, (2015) Lee, C. J. (2015).  Commensuration bias in peer review.  Philosophy of Science, 82(5):1272–1283.
  * Lee et al., (2013) Lee, C. J., Sugimoto, C. R., Zhang, G., and Cronin, B. (2013).  Bias in peer review.  Journal of the American Society for Information Science and Technology, 64(1):2–17.
  * Leyton-Brown et al., (2024) Leyton-Brown, K., Mausam, Nandwani, Y., Zarkoob, H., Cameron, C., Newman, N., and Raghu, D. (2024).  Matching papers and reviewers at large conferences.  Artificial Intelligence, 331:104119.
  * Li et al., (2020) Li, J., Sato, A., Shimura, K., and Fukumoto, F. (2020).  Multi-task peer-review score prediction.  In Chandrasekaran, M. K., de Waard, A., Feigenblat, G., Freitag, D., Ghosal, T., Hovy, E., Knoth, P., Konopnicki, D., Mayr, P., Patton, R. M., and Shmueli-Scheuer, M., editors, Proceedings of the First Workshop on Scholarly Document Processing, pages 121–126. Association for Computational Linguistics.
  * Li and Nenkova, (2015) Li, J. J. and Nenkova, A. (2015).  Fast and accurate prediction of sentence specificity.  In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI’15, page 2281–2287. AAAI Press.
  * Li et al., (2023) Li, M., Hovy, E., and Lau, J. (2023).  Summarizing multiple documents with conversational structure for meta-review generation.  In Bouamor, H., Pino, J., and Bali, K., editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7089–7112, Singapore. Association for Computational Linguistics.
  * Li et al., (2007) Li, N., Li, T., and Venkatasubramanian, S. (2007).  t-Closeness: Privacy beyond k-anonymity and l-diversity.  In Chirkova, R., Dogac, A., Özsu, M. T., and Sellis, T. K., editors, Proceedings of the 23rd International Conference on Data Engineering, pages 106–115. IEEE Computer Society.
  * Liang et al., (2023) Liang, W., Zhang, Y., Cao, H., Wang, B., Ding, D., Yang, X., Vodrahalli, K., He, S., Smith, D., Yin, Y., et al. (2023).  Can large language models provide useful feedback on research papers? A large-scale empirical analysis.  arXiv preprint arXiv:2310.01783.
  * Littman, (2021) Littman, M. L. (2021).  Collusion rings threaten the integrity of computer science research.  Communications of the ACM, 64(6):43–44.
  * Liu et al., (2023) Liu, R., Jecmen, S., Conitzer, V., Fang, F., and Shah, N. B. (2023).  Testing for reviewer anchoring in peer review: A randomized controlled trial.  arXiv preprint arXiv:2307.05443.
  * Liu and Shah, (2023) Liu, R. and Shah, N. B. (2023).  ReviewerGPT? An exploratory study on using large language models for paper reviewing.  arXiv preprint arXiv:2306.00622.
  * Lo et al., (2023) Lo, K., Shen, Z., Newman, B., Chang, J., Authur, R., Bransom, E., Candra, S., Chandrasekhar, Y., Huff, R., Kuehl, B., Singh, A., Wilhelm, C., Zamarron, A., Hearst, M. A., Weld, D., Downey, D., and Soldaini, L. (2023).  PaperMage: A unified toolkit for processing, representing, and manipulating visually-rich scientific documents.  In Feng, Y. and Lefever, E., editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 495–507, Singapore. Association for Computational Linguistics.
  * Lo et al., (2020) Lo, K., Wang, L. L., Neumann, M., Kinney, R., and Weld, D. (2020).  S2ORC: The semantic scholar open research corpus.  In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J., editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4969–4983. Association for Computational Linguistics.
  * Malički and Mehmani, (2024) Malički, M. and Mehmani, B. (2024).  Structured peer review: Pilot results from 23 Elsevier journals.  bioRxiv preprint bioRxiv:10.1101/2024.02.01.578440.
  * Manzoor and Shah, (2021) Manzoor, E. and Shah, N. B. (2021).  Uncovering latent biases in text: Method and application to peer review.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 4767–4775.
  * Mao et al., (2022) Mao, Y., Zhong, M., and Han, J. (2022).  CiteSum: Citation text-guided scientific extreme summarization and domain adaptation with limited supervision.  In Goldberg, Y., Kozareva, Z., and Zhang, Y., editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10922–10935, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Meir et al., (2021) Meir, R., Lang, J., Lesca, J., Mattei, N., and Kaminsky, N. (2021).  A market-inspired bidding scheme for peer review paper assignment.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 4776–4784.
  * Mita et al., (2022) Mita, M., Sakaguchi, K., Hagiwara, M., Mizumoto, T., Suzuki, J., and Inui, K. (2022).  Towards automated document revision: Grammatical error correction, fluency edits, and beyond.  arXiv preprint arXiv:2205.11484.
  * Murray et al., (2016) Murray, D. L., Morris, D., Lavoie, C., Leavitt, P. R., MacIsaac, H., Masson, M. E., and Villard, M.-A. (2016).  Bias in research grant evaluation has dire consequences for small universities.  PLOS One, 11(6):e0155876.
  * Narimatsu et al., (2021) Narimatsu, H., Koyama, K., Dohsaka, K., Higashinaka, R., Minami, Y., and Taira, H. (2021).  Task definition and integration for scientific-document writing support.  In Beltagy, I., Cohan, A., Feigenblat, G., Freitag, D., Ghosal, T., Hall, K., Herrmannova, D., Knoth, P., Lo, K., Mayr, P., Patton, R. M., Shmueli-Scheuer, M., de Waard, A., Wang, K., and Wang, L. L., editors, Proceedings of the Second Workshop on Scholarly Document Processing, pages 18–26. Association for Computational Linguistics.
  * Neubig et al., (2021) Neubig, G., Wieting, J., McCarthy, A., Stent, A., Schluter, N., and Cohn, T. (2021).  ACL reviewer matching code.  https://github.com/acl-org/reviewer-paper-matching.  Online; accessed 21-April-2024.
  * Névéol et al., (2017) Névéol, A., Fort, K., and Hwa, R. (2017).  Report on EMNLP Reviewer Survey.  https://hal.science/hal-01660886/file/EMNLPReviewerSurvey2017.pdf.  Online; accessed 21-April-2024.
  * Noothigattu et al., (2021) Noothigattu, R., Shah, N., and Procaccia, A. (2021).  Loss functions, axioms, and peer review.  Journal of Artificial Intelligence Research, 70:1481–1515.
  * Nuijten and Polanin, (2020) Nuijten, M. B. and Polanin, J. R. (2020).  “statcheck”: Automatically detect statistical reporting inconsistencies to increase reproducibility of meta-analyses.  Research synthesis methods, 11(5):574–579.
  * Parno et al., (2017) Parno, B., Erlingsson, U., and Enck, W. (2017).  Report on the IEEE S&P 2017 submission and review process and its experiments.  https://www.ieee-security.org/TC/Reports/2017/SP2017-PCChairReport.pdf.  Online; accessed 21-April-2024.
  * Payan and Zick, (2021) Payan, J. and Zick, Y. (2021).  I will have order! Optimizing orders for fair reviewer assignment.  arXiv preprint arXiv:2108.02126.
  * Peters and Ceci, (1982) Peters, D. P. and Ceci, S. J. (1982).  Peer-review practices of psychological journals: The fate of published articles, submitted again.  Behavioral and Brain Sciences, 5(2):187–195.
  * Piniewski et al., (2024) Piniewski, M., Jarić, I., Koutsoyiannis, D., and Kundzewicz, Z. W. (2024).  Emerging plagiarism in peer-review evaluation reports: a tip of the iceberg?  Scientometrics, pages 1–10.
  * Plank and van Dalen, (2019) Plank, B. and van Dalen, R. (2019).  CiteTracked: A longitudinal dataset of peer reviews and citations.  In Proceedings of the 4th Joint Workshop on Bibliometric-enhanced Information Retrieval and Natural Language Processing for Digital Libraries (BIRNDL 2019) co-located with the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2019), pages 116–122. CEUR Workshop Proceedings.
  * Pride and Knoth, (2020) Pride, D. and Knoth, P. (2020).  An authoritative approach to citation classification.  In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, JCDL ’20, page 337–340, New York, NY, USA. Association for Computing Machinery.
  * Priya et al., (2024) Priya, P., Firdaus, M., and Ekbal, A. (2024).  Computational politeness in natural language processing: A survey.  ACM Computing Surveys.
  * ProPublica, (2016) ProPublica (2016).  Machine bias.  https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing.  Accessed: 2024-05-03.
  * Purkayastha et al., (2023) Purkayastha, S., Lauscher, A., and Gurevych, I. (2023).  Exploring Jiu-Jitsu argumentation for writing peer review rebuttals.  In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14479–14495. Association for Computational Linguistics.
  * Rastogi et al., (2024) Rastogi, C., Song, X., Jin, Z., Stelmakh, I., Daumé III, H., Zhang, K., and Shah, N. B. (2024).  A randomized controlled trial on anonymizing reviewers to each other in peer review discussions.  arXiv preprint arXiv:2403.01015.
  * Rastogi et al., (2022) Rastogi, C., Stelmakh, I., Shen, X., Meila, M., Echenique, F., Chawla, S., and Shah, N. (2022).  To ArXiv or not to ArXiv: A study quantifying pros and cons of posting preprints online.  arXiv preprint arXiv:2203.17259.
  * Régner et al., (2019) Régner, I., Thinus-Blanc, C., Netter, A., Schmader, T., and Huguet, P. (2019).  Committees with implicit biases promote fewer women when they do not believe gender bias exists.  Nature Human Behaviour, 3(11):1171–1179.
  * Robertson, (2023) Robertson, Z. (2023).  GPT-4 is slightly helpful for peer-review assistance: A pilot study.  arXiv preprint arXiv:2307.05492.
  * Rogers and Augenstein, (2020) Rogers, A. and Augenstein, I. (2020).  What can we do to improve peer review in NLP?  In Cohn, T., He, Y., and Liu, Y., editors, Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1256–1262. Association for Computational Linguistics.
  * Rogers et al., (2021) Rogers, A., Baldwin, T., and Leins, K. (2021).  ‘Just what do you think you’re doing, Dave?’ A checklist for responsible data use in NLP.  In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t., editors, Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4821–4833, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * (123) Rogers, A., Boyd-Graber, J., and Okazaki, N. (2023a).  ACL 2023 policy on AI writing assistance.  https://2023.aclweb.org/blog/ACL-2023-policy.  Online; accessed 17-April-2024.
  * (124) Rogers, A., Karpinska, M., Boyd-Graber, J., and Okazaki, N. (2023b).  Program chairs’ report on peer review at ACL 2023.  In Rogers, A., Boyd-Graber, J., and Okazaki, N., editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 40–75, Toronto, Canada. Association for Computational Linguistics.
  * Roos et al., (2012) Roos, M., Rothe, J., Rudolph, J., Scheuermann, B., and Stoyan, D. (2012).  A statistical approach to calibrating the scores of biased reviewers: The linear vs. the nonlinear model.  In Multidisciplinary Workshop on Advances in Preference Handling.
  * Ross et al., (2006) Ross, J. S., Gross, C. P., Desai, M. M., Hong, Y., Grant, A. O., Daniels, S. R., Hachinski, V. C., Gibbons, R. J., Gardner, T. J., and Krumholz, H. M. (2006).  Effect of Blinded Peer Review on Abstract Acceptance.  Journal of the American Medical Association, 295(14):1675–1680.
  * Ross-Hellauer, (2017) Ross-Hellauer, T. (2017).  What is open peer review? A systematic review [version 2; peer review: 4 approved].  F1000Research, 6(588).
  * Santu et al., (2024) Santu, S. K. K., Sinha, S. K., Bansal, N., Knipper, A., Sarkar, S., Salvador, J., Mahajan, Y., Guttikonda, S., Akter, M., Freestone, M., and Jr, M. C. W. (2024).  Prompting LLMs to compose meta-review drafts from peer-review narratives of scholarly manuscripts.  arXiv preprint arXiv:2402.15589.
  * Saveski et al., (2023) Saveski, M., Jecmen, S., Shah, N., and Ugander, J. (2023).  Counterfactual evaluation of peer-review assignment policies.  In Oh, A., Neumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S., editors, Advances in Neural Information Processing Systems, volume 36, pages 58765–58786. Curran Associates, Inc.
  * Schintler et al., (2023) Schintler, L. A., McNeely, C. L., and Witte, J. (2023).  A critical examination of the ethics of AI-mediated peer review.  arXiv preprint arXiv:2309.12356.
  * Schuff et al., (2023) Schuff, H., Vanderlyn, L., Adel, H., and Vu, N. T. (2023).  How to do human evaluation: A brief introduction to user studies in NLP.  Natural Language Engineering, 29(5):1199–1222.
  * Shah et al., (2020) Shah, D. S., Schwartz, H. A., and Hovy, D. (2020).  Predictive biases in naural language processing models: A conceptual framework and overview.  In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J., editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5248–5264. Association for Computational Linguistics.
  * Shah, (2022) Shah, N. B. (2022).  Challenges, experiments, and computational solutions in peer review.  Communications of the ACM, 65(6):76–87.
  * Shah, (2023) Shah, N. B. (2023).  The role of author identities in peer review.  PLOS One, 18(6):e0286206.
  * Shah et al., (2018) Shah, N. B., Tabibian, B., Muandet, K., Guyon, I., and Von Luxburg, U. (2018).  Design and analysis of the NIPS 2016 review process.  Journal of Machine Learning Research, 19(49):1–34.
  * Shen et al., (2022) Shen, C., Cheng, L., Zhou, R., Bing, L., You, Y., and Si, L. (2022).  MReD: A meta-review dataset for structure-controllable text generation.  In Muresan, S., Nakov, P., and Villavicencio, A., editors, Findings of the Association for Computational Linguistics: ACL 2022, pages 2521–2535, Dublin, Ireland. Association for Computational Linguistics.
  * Smith, (2010) Smith, R. (2010).  Classical peer review: An empty gun.  Breast Cancer Research, 12(4):S13.
  * (138) Stelmakh, I., Rastogi, C., Shah, N. B., Singh, A., and Daumé III, H. (2023a).  A large scale randomized controlled trial on herding in peer-review discussions.  PLOS One, 18(7):e0287443.
  * (139) Stelmakh, I., Shah, N., and Singh, A. (2021a).  PeerReview4All: Fair and accurate reviewer assignment in peer review.  Journal of Machine Learning Research, 22(163):1–66.
  * (140) Stelmakh, I., Shah, N. B., and Singh, A. (2021b).  Catch me if I can: Detecting strategic behaviour in peer assessment.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 4794–4802.
  * (141) Stelmakh, I., Shah, N. B., Singh, A., and Daumé, H. (2021c).  Prior and prejudice: The novice reviewers’ bias against resubmissions in conference peer review.  Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1).
  * (142) Stelmakh, I., Shah, N. B., Singh, A., and Daumé III, H. (2021d).  A novice-reviewer experiment to address scarcity of qualified reviewers in large conferences.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 4785–4793.
  * (143) Stelmakh, I., Wieting, J., Neubig, G., and Shah, N. B. (2023b).  A gold standard dataset for the reviewer assignment problem.  arXiv preprint arXiv:2303.16750.
  * Strauss et al., (2023) Strauss, D., Gran-Ruaz, S., Osman, M., Williams, M. T., and Faber, S. C. (2023).  Racism and censorship in the editorial and peer review process.  Frontiers in Psychology, 14.
  * Sun et al., (2024) Sun, L., Chan, A., Chang, Y. S., and Dow, S. P. (2024).  ReviewFlow: Intelligent scaffolding to support academic peer reviewing.  arXiv preprint arXiv:2402.03530.
  * Syed et al., (2018) Syed, S., Völske, M., Potthast, M., Lipka, N., Stein, B., and Schütze, H. (2018).  Task proposal: The TL;DR challenge.  In Krahmer, E., Gatt, A., and Goudbeek, M., editors, Proceedings of the 11th International Conference on Natural Language Generation, pages 318–321, Tilburg University, The Netherlands. Association for Computational Linguistics.
  * Teplitskiy et al., (2019) Teplitskiy, M., Ranub, H., Grayb, G. S., Meniettid, M., Guinan, E. C., and Lakhani, K. R. (2019).  Social influence among experts: Field experimental evidence from peer review.
  * Teufel et al., (2006) Teufel, S., Siddharthan, A., and Tidhar, D. (2006).  Automatic classification of citation function.  In Jurafsky, D. and Gaussier, E., editors, Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 103–110, Sydney, Australia. Association for Computational Linguistics.
  * Thorn Jakobsen and Rogers, (2022) Thorn Jakobsen, T. and Rogers, A. (2022).  What factors should paper-reviewer assignments rely on? community perspectives on issues and ideals in conference peer-review.  In Carpuat, M., de Marneffe, M.-C., and Meza Ruiz, I. V., editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4810–4823, Seattle, United States. Association for Computational Linguistics.
  * Tomkins et al., (2017) Tomkins, A., Zhang, M., and Heavlin, W. D. (2017).  Reviewer bias in single- versus double-blind peer review.  Proceedings of the National Academy of Sciences, 114(48):12708–12713.
  * van Rooyen et al., (1999) van Rooyen, S., Black, N., and Godlee, F. (1999).  Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts.  Journal of Clinical Epidemiology, 52 7:625–9.
  * Vervloet et al., (2012) Vervloet, M., Linn, A. J., van Weert, J. C., De Bakker, D. H., Bouvy, M. L., and Van Dijk, L. (2012).  The effectiveness of interventions using electronic reminders to improve adherence to chronic medication: a systematic review of the literature.  Journal of the American Medical Informatics Association, 19(5):696–704.
  * Vijaykumar, (2020) Vijaykumar, T. N. (2020).  Potential organized fraud in ACM/IEEE computer architecture conferences.  https://medium.com/@tnvijayk/potential-organized-fraud-in-acm-ieee-computer-architecture-conferences-ccd61169370d.  Online; accessed 17-April-2024.
  * Wadden et al., (2020) Wadden, D., Lin, S., Lo, K., Wang, L. L., van Zuylen, M., Cohan, A., and Hajishirzi, H. (2020).  Fact or fiction: Verifying scientific claims.  In Webber, B., Cohn, T., He, Y., and Liu, Y., editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534–7550. Association for Computational Linguistics.
  * Wadden et al., (2022) Wadden, D., Lo, K., Kuehl, B., Cohan, A., Beltagy, I., Wang, L. L., and Hajishirzi, H. (2022).  SciFact-Open: Towards open-domain scientific claim verification.  In Goldberg, Y., Kozareva, Z., and Zhang, Y., editors, Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4719–4734, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Wang and Shah, (2019) Wang, J. and Shah, N. B. (2019).  Your 2 is my 1, your 3 is my 9: Handling arbitrary miscalibrations in ratings.  In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 864–872.
  * Wang et al., (2021) Wang, J., Stelmakh, I., Wei, Y., and Shah, N. (2021).  Debiasing evaluations that are biased by evaluations.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10120–10128.
  * Wang et al., (2024) Wang, Q., Downey, D., Ji, H., and Hope, T. (2024).  Scimon: Scientific inspiration machines optimized for novelty.  arXiv preprint arXiv:2305.14259.
  * Wieting et al., (2019) Wieting, J., Gimpel, K., Neubig, G., and Berg-Kirkpatrick, T. (2019).  Simple and effective paraphrastic similarity from parallel translations.  In Korhonen, A., Traum, D., and Màrquez, L., editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4602–4608, Florence, Italy. Association for Computational Linguistics.
  * Wolfram et al., (2020) Wolfram, D., Wang, P., Hembree, A., and Park, H. (2020).  Open peer review: promoting transparency in open science.  Scientometrics, 125(2):1033–1051.
  * Xiao et al., (2023) Xiao, Z., Zhang, S., Lai, V., and Liao, Q. V. (2023).  Evaluating evaluation metrics: A framework for analyzing NLG evaluation metrics using measurement theory.  In Bouamor, H., Pino, J., and Bali, K., editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10967–10982, Singapore. Association for Computational Linguistics.
  * Xu and Wan, (2022) Xu, S. and Wan, X. (2022).  PosterBot: A system for generating posters of scientific papers with neural models.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 13233–13235.
  * Yamamoto et al., (2021) Yamamoto, S., Lauscher, A., Ponzetto, S. P., Glavaš, G., and Morishima, S. (2021).  Visual summary identification from scientific publications via self-supervised learning.  Frontiers in Research Metrics and Analytics, 6.
  * Zeng et al., (2023) Zeng, Q., Sidhu, M., Chan, H. P., Wang, L., and Ji, H. (2023).  Meta-review generation with checklist-guided iterative introspection.  arXiv preprint arXiv:2305.14647.
  * Zyska et al., (2023) Zyska, D., Dycke, N., Buchmann, J., Kuznetsov, I., and Gurevych, I. (2023).  CARE: Collaborative AI-assisted reading environment.  In Bollegala, D., Huang, R., and Ritter, A., editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 291–303, Toronto, Canada. Association for Computational Linguistics.


Triangulation and Persistence: Algebra 101

The second author was supported by an individual NSERC
Discovery grant.

Paul Biran, Octav Cornea and Jun Zhang

Paul Biran, Department of Mathematics, ETH-Zürich,
Rämistrasse 101, 8092 Zürich, Switzerland

Octav Cornea, Department of Mathematics and Statistics,
University of Montreal, C.P. 6128 Succ.  Centre-Ville Montreal, QC
H3C 3J7, Canada

Jun Zhang, Centre de Recherches Mathématiques, University of Montreal, C.P. 6128 Succ.  Centre-Ville Montreal, QC
H3C 3J7, Canada<EMAIL_ADDRESS>
[2010]55N31 53D12 (Primary); 35A27 (Secondary)

This paper lays the foundations of triangulated persistence
categories (TPC), which brings together persistence modules with the
theory of triangulated categories. As a result we introduce several
measurements and metrics on the set of objects of some triangulated
categories. We also provide  examples of TPC's coming from
algebra, algebraic topology, microlocal sheaf theory and symplectic topology.

[1]
Tomohiro Asano and Yuichi Ike.
Persistence-like distance on Tamarkin's category and symplectic
displacement energy.
J. Symp. Geom, 18(3):613–649, 2020.

[2]
Serguei Barannikov.
The framed Morse complex and its invariants.
In Singularities and bifurcations, volume 21 of Adv.
Soviet Math., pages 93–115. Amer. Math. Soc., Providence, RI, 1994.

[3]
Ulrich Bauer and Michael Lesnick.
Induced matchings and the algebraic stability of persistence
J. Comput. Geom., 6(2):162–191, 2015.

[4]
Paul Biran and Octav Cornea.
Bounds on the Lagrangian spectral metric in cotangent bundles.
arXiv: 2008.04756.

[5]
Paul Biran and Octav Cornea.
A Lagrangian pictionary.
To appear in the Kyoto Math. Journal.

[6]
Paul Biran and Octav Cornea.
Lagrangian cobordism. I.
J. Amer. Math. Soc., 26(2):295–340, 2013.

[7]
Paul Biran and Octav Cornea.
Lagrangian cobordism and Fukaya categories.
Geom. Funct. Anal., 24(6):1731–1830, 2014.

[8]
Paul Biran, Octav Cornea, and Egor Shelukhin.
Lagrangian shadows and triangulated categories.
To appear in Astérisque. Can be found at

[9]
Alexey I Bondal and Mikhail M Kapranov.
Enhanced triangulated categories.
Mathematics of the USSR-Sbornik, 70(1):93, 1991.

[10]
Peter Bubenik and Nikola Milićević.
Homological algebra for persistence modules.
Foundations of Computational Mathematics, pages 1–46, 2019.

[11]
François Charest and Chris Woodward.
Floer trajectories and stabilizing divisors.
J. Fixed Point Theory Appl., 19(2):1165–1236, 2017.

[12]
Frédéric Chazal, Vin de Silva, Marc Glisse, and Steve Oudot.
The structure and stability of persistence modules.
SpringerBriefs in Mathematics. Springer, [Cham], 2016.

[13]
Octav Cornea and François Lalonde.
Cluster homology: an overview of the construction and results.
Electron. Res. Announc. Amer. Math. Soc., 12:1–12, 2006.

[14]
Octav Cornea, Gregory Lupton, John Oprea, and Daniel Tanré.
Lusternik-Schnirelmann category, volume 103 of Mathematical Surveys and Monographs.
American Mathematical Society, Providence, RI, 2003.

[15]
Octav Cornea and Egor Shelukhin.
Lagrangian cobordism and metric invariants.
Journal of Differential Geometry, 112:1–45, 2019.

[16]
Octavian Cornea.
Cone-length and lusternik-schnirelmann category.
Topology, 33:95–111, 1994.

[17]
Octavian Cornea.
There is just one rational cone-length.
Trans. Amer. Math. Soc., 344(2):835–848, 1994.

[18]
William Crawley-Boevey.
Decomposition of pointwise finite-dimensional persistence modules.
J. Algebra Appl., 14(5):1550066, 8, 2015.

[19]
George Dimitrov, Fabian Haiden, Ludmil Katzarkov, and Maxim Kontsevich.
Dynamical systems and categories, volume 621 of Contemp.
Math., pages 133–170.
Amer. Math. Soc., Providence, RI, 2014.

[20]
Vladimir Drinfeld.
DG quotients of DG categories.
J. Algebra, 272(2):643–691, 2004.

[21]
Herbert Edelsbrunner.
A short course in computational geometry and topology.
SpringerBriefs in Applied Sciences and Technology. Springer, Cham,

[22]
Yu-Wei Fan and Simion Filip.
Asymptotic shifting numbers in triangulated categories.
arXiv: 2008.06159.

[23]
Yves Félix, Stephen Halperin, and Jean-Claude Thomas.
Rational homotopy theory, volume 205 of Graduate Texts in
Springer-Verlag, New York, 2001.

[24]
Kenji Fukaya, Paul Seidel, and Ivan Smith.
Exact Lagrangian submanifolds in simply-connected cotangent
Invent. Math., 172(1):1–27, 2008.

[25]
Kenji Fukaya, Paul Seidel, and Ivan Smith.
The symplectic geometry of cotangent bundles from a categorical
viewpoint, volume 757 of Lecture Notes in Phys., pages 1–26.
Springer, Berlin, 2009.

[26]
Phillip Griffiths and Joseph Harris.
Principles of algebraic geometry.
Wiley-Interscience [John Wiley & Sons], New York, 1978.
Pure and Applied Mathematics.

[27]
Misha Gromov.
Metric structures for Riemannian and non-Riemannian spaces.
Modern Birkhäuser Classics. Birkhäuser Boston, Inc., Boston,
MA, english edition, 2007.
Based on the 1981 French original, With appendices by M. Katz, P.
Pansu and S. Semmes, Translated from the French by Sean Michael Bates.

[28]
Stéphane Guillermou, Masaki Kashiwara, and Pierre Schapira.
Sheaf quantization of Hamiltonian isotopies and applications to
nondisplaceability problems.
Duke Math. J., 161(2):201–245, 2012.

[29]
Stéphane Guillermou and Pierre Schapira.
Microlocal theory of sheaves and Tamarkin's non displaceability
In Homological mirror symmetry and tropical geometry, volume 15
of Lect. Notes Unione Mat. Ital., pages 43–85. Springer, Cham, 2014.

[30]
Felix Hensel.
Stability conditions and Lagrangian cobordisms.
J. Symplectic Geom., 18(2):463–536, 2020.

[31]
Masaki Kashiwara and Pierre Schapira.
Sheaves on manifolds, volume 292 of Grundlehren der
Mathematischen Wissenschaften [Fundamental Principles of Mathematical
Springer-Verlag, Berlin, 1990.
With a chapter in French by Christian Houzel.

[32]
Pierre Lelong.
Intégration sur un ensemble analytique complexe.
Bull. Soc. Math. France, 85:239–262, 1957.

[33]
Dimitri Orlov.
Triangulated categories of singularities, and equivalences between
Landau-Ginzburg models.
Mat. Sb., 197(12):117–132, 2006.

[34]
Dmitri Orlov.
Remarks on generators and dimensions of triangulated categories.
Mosc. Math. J., 9(1):153–159, back matter, 2009.

[35]
Leonid Polterovich, Daniel Rosen, Karina Samvelyan, and Jun Zhang.
Topological persistence in geometry and analysis, volume 74 of
University Lecture Series.
AMS, Providence, RI, 2020.

[36]
Dieter Puppe.
On the formal structure of stable homotopy theory.
Colloquium on algebraic topology, Aarhus Universitet
Matematisk Institute, pages 65–71, 1962.

[37]
Paul Seidel.
Fukaya categories and Picard-Lefschetz theory.
Zurich Lectures in Advanced Mathematics. European Mathematical
Society (EMS), Zürich, 2008.

[38]
Nick Sheridan.
On the homological mirror symmetry conjecture for pairs of pants.
J. Differential Geom., 89(2):271–367, 2011.

[39]
Steve Smale.
On the topology of algorithms. I.
J. Complexity, 3(2):81–89, 1987.

[40]
Dennis Sullivan.
Infinitesimal computations in topology.
Inst. Hautes Études Sci. Publ. Math., (47):269–331 (1978),

[41]
Dmitry Tamarkin.
Microlocal condition for non-displaceability.
In Algebraic and analytic microlocal analysis, volume 269 of
Springer Proc. Math. Stat., pages 99–223. Springer, Cham, 2018.

[42]
Michael Usher.
Boundary depth in Floer theory and its applications to
Hamiltonian dynamics and coisotropic submanifolds.
Israel J. Math., 184:1–57, 2011.

[43]
Michael Usher.
Hofer's metrics and boundary depth.
Ann. Sci. Éc. Norm. Supér. (4), 46(1):57–128 (2013),

[44]
Michael Usher and Jun Zhang.
Persistent homology and Floer-Novikov theory.
Geom. Topol., 20(6):3333–3430, 2016.

[45]
Jean-Louis Verdier.
Catégories dérivées: quelques résultats (état 0).
In Cohomologie étale, volume 569 of Lecture Notes in
Math., pages 262–311. Springer, Berlin, 1977.

[46]
Charles A. Weibel.
An introduction to homological algebra, volume 38 of Cambridge Studies in Advanced Mathematics.
Cambridge University Press, Cambridge, 1994.

[47]
Jun Zhang.
Quantitative Tamarkin Theory.
CRM Short Courses; Virtual Series on Symplectic Geometry. Springer
International Publishing, 2020.

[48]
Afra Zomorodian and Gunnar Carlsson.
Computing persistent homology.
Discrete Comput. Geom., 33(2):249–274, 2005.