# Taming Transformers for Realistic Lidar Point Cloud Generation

Hamed Haghighi1 Amir Samadi1 Mehrdad Dianati2 Valentina Donzella1 Kurt
Debattista1

###### Abstract

Diffusion Models (DMs) have achieved State-Of-The-Art (SOTA) results in the
Lidar point cloud generation task, benefiting from their stable training and
iterative refinement during sampling. However, DMs often fail to realistically
model Lidar raydrop noise due to their inherent denoising process. To retain
the strength of iterative sampling while enhancing the generation of raydrop
noise, we introduce LidarGRIT, a generative model that uses auto-regressive
transformers to iteratively sample the range images in the latent space rather
than image space. Furthermore, LidarGRIT utilises VQ-VAE to separately decode
range images and raydrop masks. Our results show that LidarGRIT achieves
superior performance compared to SOTA models on KITTI-360 and KITTI odometry
datasets. Code available at:https://github.com/hamedhaghighi/LidarGRIT.

††1H. Haghighi, A.Samadi, K. Debattista and V. Donzella are with WMG,
University of Warwick, Coventry, U.K. (Corresponding author:
Hamed.Haghighi@warwick.ac.uk)††2M. Dianati is with the School of Electronics,
Electrical Engineering and Computer Science at Queen’s University of Belfast
and WMG at the University of Warwick

##  1 Introduction

Light detection and ranging (Lidar) is a critical sensor in autonomous
vehicles, providing highly precise 3D environmental scanning. However,
realistic simulation of the Lidar sensor poses challenges, involving
cumbersome tasks such as creating 3D object models along with running
computationally demanding physics-based algorithms. As an alternative, data-
driven simulation models, particularly deep generative models have gained
traction owing to their exceptional capacity to model high-dimensional data.
Initially proposed for generating photo-realistic RGB images, deep generative
models have been adapted for Lidar point cloud generation, progressing from
early GAN-based frameworks [1] to the best-performing diffusion models [14].

Diffusion models (DMs) for Lidar point cloud generation excel mainly due to
their stable training and iterative refinement during the sampling process.
While they demonstrate proficiency in capturing the 3D shape of point clouds,
they face challenges in generating realistic Lidar raydrop noise, resulting in
range images that appear unrealistic (refer to Figure 1a). This issue arises
from the inherent denoising nature of DMs.

(a)

(b)

(c)

Figure 1: (a) The range image generated by diffusion model (R2DM [9]) exhibits
less realistic raydrop noise compared to our provided sample and the real one.
(b) we propose to sample range image in the latent space via Auto-Regressive
(AR) transformer [12]. (c) We then generate the raydrop mask and clean range
image separately in the image space via VQ-VAE [2] decoder.

We introduce a novel Lidar Generative Range Image Transformer (LidarGRIT)
model to incorporate both progressive generation and accurate raydrop noise
synthesis. Our LidarGRIT works with the range image representation of Lidar
point cloud, chosen for efficient processing and compatibility with image
generative models. The generation process of the LidarGRIT consists of an
iterative sampling in the latent space via Auto-Regressive (AR) transformer
[12] (refer to Figure 1b), and decoding the sampled tokens to range images
using an adapted Vector Quantised Variational Auto-Encoder (VQ-VAE [2]) model
(refer to Figure 1c). We disentangle the generation of the range image from
the raydrop noise mask in the VQ-VAE decoder, inspired by Dusty [8], and use a
separate loss function to reconstruct clean range images and raydrop masks
during training. Furthermore, we realised that large VQ-VAE models, primarily
designed for high-resolution RGB images, tend to overfit when applied to
relatively low-resolution range images. To address this, we propose geometric
preservation, aiming to encourage the VQ-VAE to capture input geometry and
provide more expressive latent tokens. We compare our LidarGRIT model to SOTA
models on KITTI-360 and KITTI odometry datasets. Our model outperforms on
nearly all metrics, specifically excelling in the image-based metrics, SWD
[5]. The contributions of this paper can be summarised as follows:

  * •

Introduction of LidarGRIT, a novel Lidar point cloud generative model
consisting of a two-step generation process: iterative token index sampling
through an AR transformer and single-pass range image decoding via an adapted
VQ-VAE.

  * •

Proposal of two novel techniques to enhance the generation quality:
incorporating a separate raydrop estimation loss and enforcing geometry
perseverance to increase VQ-VAE generalisability.

  * •

Comprehensive evaluation of our LidarGRIT generation by comparing it with SOTA
generative models using KITTI-360 and KITTI odometry datasets.

##  2 Related-Work

Caccia et al. [1] were among the first researchers to apply deep generative
models to Lidar point clouds. They converted Lidar point clouds into range
images and adapted the Deep Convolutional GAN (DCGAN) [11] for point cloud
generation. Building on this, Dusty [8, 10] was proposed, which integrates
raydrop synthesis into the GAN training process. Another notable model,
UltraLiDAR [13], adopts a VQ-VAE framework to learn a discrete and compact
Lidar representation for point cloud restoration and generation. With the
recent achievement of DMs, LidarGen [14] and R2DM [9] models were proposed,
relying on the score-based and denoising DMs frameworks, respectively.

Our approach draws inspiration from Dusty [8] framework in the disentanglement
of raydrop and range image generation, however, we differentiate by employing
a non-adversarial and more stable training using VQ-VAE [2], treating the
raydrop estimation as a binary classification problem. Our LidarGRIT shares
similarities with UltraLiDAR [13] in its two-stage sampling using VQ-VAE and
AR transformer. However, rather than using voxelised Birds-Eye-View (BEV), we
represent point clouds with range images that provide a lossless, more
compact, and more computationally efficient format. Moreover, we focus on the
raydrop noise generation and assess the point cloud generation on both image
and point cloud representation.

(a) Adapted VQ-VAE model

(b) AR transformer model

Figure 2: Overview of the training process.

##  3 Method

The designing process of LidarGRIT involves three steps. First, we represent
the Lidar point clouds as range images (Section 3.1). Next, we tokenise range
images using the VQ-VAE encoder and decode them separately to obtain clean
range image along with the raydrop noise mask (Section 3.2). Finally, we
capture the token interactions using the AR transformer (Section 3.3).

###  3.1 Data Representation

We employ different transformations to create range images for the KITTI-360
and KITTI-odometry datasets. For KITTI-360 generation, we use spherical
projection, wherein each point in Cartesian coordinates
$(x,y,z)\in\mathbb{R}^{3}$ is transformed into its spherical coordinates
$(r,\theta,\phi)$ as:
$r=\sqrt{x^{2}+y^{2}+z^{2}},\theta=atan(y,x),\phi=atan(z,\sqrt{x^{2}+y^{2}})$.
We then quantise $\theta$ and $\phi$ into H and W bins with equal bin width,
where H and W denote the vertical and horizontal angular resolutions of the
Lidar sensor. This yields an image of size $0pt\times 0pt$ with each pixel
containing the range of its associated point. Regarding KITTI-odometry, we
employ scan unfolding representation due to the sensor’s non-linear vertical
spacing [8]. We partition the ordered sequence into $0pt$ sub-sequences, with
each sub-sequence indicating one elevation angle. Throughout the paper, we
denote the input range image as $\mathbf{x}\in\mathbb{R}^{0pt\times 0pt}$ and
ground-truth raydrop mask as $\mathbf{x}_{m}\in\\{0,1\\}^{0pt\times 0pt}$.

###  3.2 Adapting VQ-VAE

We use and adapt VQ-VAE [2] to auto-encode the range image and raydrop mask
for three purposes: tokenisation, downsampling and raydrop noise generation.
The VQ-VAE model consists of an encoder $E$, a quantiser $Q$ and a decoder
$G$. VQ-VAE encoder $E$ downsamples the input range image $\mathbf{x}$ into
latent image
$\mathbf{\hat{z}}=E(\mathbf{x})\in\mathbb{R}^{\text{h}\times\text{w}\times
n_{z}}$. Quantiser $Q$ generates tokens
$\mathbf{z}_{q}\in\mathbb{R}^{\text{h}\times\text{w}\times n_{z}}$ using a
learnable codebook $Z=\\{z_{k}\\}_{k=1}^{K}\subset\mathbb{R}^{n_{z}}$.
Finally, VQ-VAE decoder $G$ generates a clean range image
$\mathbf{\hat{x}}_{r}\in\mathbb{R}^{0pt\times 0pt}$ and raydrop mask logits
$\mathbf{\hat{x}_{\pi}}\in\mathbb{R}^{0pt\times 0pt}$ based on the quantised
latent image $\mathbf{z}_{q}$ as
$[\mathbf{\hat{x}}_{r},\mathbf{\hat{x}_{\pi}}]=G(\mathbf{z}_{q})$. The mask
logits are then converted to the binary mask
$\mathbf{\hat{x}}_{m}\in\\{0,1\\}^{0pt\times 0pt}$ via sigmoid function and
thresholding:

$\mathbf{\hat{x}}_{m}=\begin{cases}1&\text{sigmoid}(\mathbf{\hat{x}_{\pi}})\geq
0.5\\\ 0&\text{sigmoid}(\mathbf{\hat{x}_{\pi}})<0.5.\end{cases}$ (1)

The final generated range image can be obtained as
$\mathbf{\hat{x}}=\mathbf{\hat{x}}_{m}\odot\mathbf{\hat{x}}_{r}$.

####  3.2.1 Training–Raydrop Loss

We separate the training objectives for range image and raydrop mask
generation. To encourage VQ-VAE decoder $G$ to approximate the input range
image, we use masked absolute error loss:

$\mathcal{L}_{\text{rec}}(E,G)=\mathbb{E}_{\mathbf{x}}\big{[}\frac{1}{\text{H}\times\text{W}}\|\mathbf{x}_{m}\odot(\mathbf{x}-\mathbf{\hat{x}}_{r})\|_{1}\big{]}$
(2)

This induces the decoder’s range image channel to focus only on estimating the
range of existing points. On the other hand, the mask channel is enforced to
estimate raydrop noise via raydrop loss:

$\begin{split}\mathcal{L}_{\text{RL}}(E,G)&=\mathbb{E}_{\mathbf{x}}\big{[}\text{Avg}[\mathbf{x}_{m}\odot\log(\text{sigmoid}(\mathbf{\hat{x}_{\pi}}))\\\
&+(1-\mathbf{x}_{m})\odot\log(1-\text{sigmoid}(\mathbf{\hat{x}_{\pi}}))]\big{]},\end{split}$
(3)

Where Avg[$\cdot$] function calculates the average of the input across all its
elements. To align the encoder and codebook embeddings, we train VQ-VAE with
so-called commitment loss:

$\mathcal{L}_{\text{com}}(E,Z)=\mathbb{E}_{\mathbf{x}}\big{[}\|\text{sg}[\mathbf{\hat{z}}]-\mathbf{z}_{q}\|_{2}^{2}\\\
+\|\text{sg}[\mathbf{z}_{q}]-\mathbf{\hat{z}}\|_{2}^{2}\big{]}.$

The sg[$\cdot$] operator denotes the straight-through gradient estimator,
ensuring that the quantisation process remains differentiable. Total training
loss for the adapted VQ-VAE can be calculated as:

$\mathcal{L}_{\text{VQ-
VAE}}=\mathcal{L}_{\text{rec}}(E,G)+\lambda\mathcal{L}_{\text{RL}}(E,G)\\\
+\mathcal{L}_{\text{com}}(E,Z).$ (4)

By tuning $\lambda$, we can establish a trade-off between the realism of range
image generation and that of raydrop mask. We set $\lambda=0.1$ in this study.
We show the training process of the adapted VQ-VAE in Figure 2a.

####  3.2.2 Training–Geometric Perseverance

We observed that the VQ-VAE models, primarily designed for high-resolution RGB
images, are prone to overfitting when dealing with relatively low-dimensional
range images. This often results in less expressive latent codes. To mitigate
the issue, we randomly distort the input images with geometric transformations
$\mathbf{\mathcal{F}}$ and push the VQ-VAE to reconstruct the distorted image.
This encourages the VQ-VAE to prioritise and preserve the input image
geometry. In practice, we randomly replace the input range image $\mathbf{x}$
and raydrop mask $\mathbf{x}_{m}$ with their respective transformed versions
$\mathbf{\mathcal{F}}(\mathbf{x})$, $\mathbf{\mathcal{F}}(\mathbf{x}_{m})$
during the calculation of the training losses (Equation 2 and 3). Our choice
of geometric transformations includes affine transformation as well as
horizontal and vertical flips.

Figure 3: Qualitative comparison on KITTI-360 [6] and KITTI odometry datasets
[4].

###  3.3 Auto-regressive Transformer

With trained VQ-VAE, we can encode the range images into token indices
$\mathbf{s}\in\big{\\{}0,1,2,...,|Z|-1\big{\\}}^{\text{h}\times\text{w}}$ and
use AR transformers to model interactions between tokens. We train the
transformer by enforcing auto-regressive modelling. We estimate the likelihood
of each token index $\mathbf{s}_{i}$ based on the previous indices
$\mathbf{s}_{<i}$ denoted as $p(\mathbf{s}_{i}|\mathbf{s}_{<i})$. The
likelihood of the entire sequence can be calculated using chain rule as
$p(\mathbf{s})=\prod_{i}p(\mathbf{s}_{i}|\mathbf{s}_{<i})$. So training
objective for the AR transformer can be achieved by negative log-likelihood of
$p(\mathbf{s})$:

$\mathcal{L}_{\text{T}}=\mathbb{E}_{\mathbf{x}}\left[-\log
p(\mathbf{s})\right].$ (5)

We visualise the training process of the AR transformer in Figure 2b.

##  4 Evaluations

We detail our experimental settings in Section 4.1, conduct a comparison to
SOTA models in Section 4.2 and carry out an ablation study in Section 4.3.

###  4.1 Experimental Settings

Datasets We evaluate our LidarGRIT generations on two datasets: KITTI-360 [6]
and KITTI odometry [3]. We follow the Zyrizanov et al [14] for partitioning
the sequences into train and test splits as well as determining the range
image size ($64\times 1024$). Similarly for KITTI odometry, we follow the
setup explained by Nakashima et al. [8] for dataset splitting and subsampling
the range images to the size of $64\times 256$.

Metrics We evaluate our generated point clouds against SOTA models using three
representations: range image, BEV and point cloud. For range image
representation, we employ Sliced Wasserstein Distance (SWD) [5]. BEV
representation is assessed using Maximum-Mean Discrepancy (MMD) and Jensen-
Shannon Divergence (JSD) following Zyrizanov et al. [14]. Point cloud
evaluation includes Frechet Point Distance (FPD) akin to R2DM [9], along with
minimum-Matching Distance (MD) and JSD similar to Nakashima et al. [8]. For
calculating MD and JSD metrics, we subsample 512 points from $0pt\times 0pt$
points using farthest point sampling. We create sets of 5k randomly selected
generated and real samples for comparison in all metrics.

Table 1: Quantitative comparison on KITTI-360 dataset.

| Image |  BEV |  | Point cloud
---|---|---|---|---
Method | SWD$\times 10^{2}\downarrow$ |  MMD$\times 10^{4}\downarrow$ | JSD$\times 10^{2}\downarrow$ | FPD $\downarrow$
LidarGen [14] | 33.93 |  2.19 | 5.70 | 43.27
UltraLiDAR [13] | N/A |  2.23 | 10.52 | N/A
R2DM [9] | 20.82 |  4.00 | 4.55 | 10.84
LidarGRIT (Ours) | 10.29 |  2.16 | 3.93 | 12.54

Table 2: Quantitative comparison on KITTI odometry dataset.

| Image |  Point cloud |  |
---|---|---|---|---
Method | SWD$\times 10^{2}\downarrow$ |  MD $\times 10^{3}\downarrow$ | JSD$\times 10^{2}\downarrow$ | FPD $\downarrow$
LidarGAN [1] | 82.29 |  6.70 | 15.98 | 700
Dusty [8] | 52.81 |  2.07 | 5.10 | 389
LidarGRIT (Ours) | 15.15 |  1.65 | 2.06 | 116

###  4.2 Comparison to State-of-the-art

We quantitatively compare our LidarGRIT to the SOTA models on KITTI-360 and
KITTI odometry generation tasks in Table 1 and 2, respectively. For KITTI-360
generation, we select the best-performing DMs, R2DM [9] and LidarGen [14],
alongside the transformer-based model UltraLiDAR [13] (notably, the numbers
for UltraLiDAR are derived from an unofficial implementation [7] since the
public implementation is not available). For KITTI odometry generation, we
pick the top-performing GAN-based models Dusty [8] and LidarGAN [1].
As shown in the tables, LidarGRIT achieves the best performance on nearly all
metrics. Specifically, LidarGRIT obtains significantly superior results on the
image-based metric SWD compared with DMs. This can largely be attributed to
the more realistic generation of raydrop noise. On the other hand, LidarGRIT
substantially outperforms GAN-based models on point cloud representation
because of more precise 3D shape modelling via the AR transformer. The results
are further evident in the qualitative comparison depicted in Figure 3.
Noteworthy, in the KITTI-360 generation, the adapted VQ-VAE and transformer
contain 34.77M and 10.26M parameters respectively, which is is comparable to
SOTA models such as UltraLiDAR [13] which contains 40.3M parameters.

Table 3: Ablation study on KITTI odometry dataset.

|  | Image | Point cloud |  |
---|---|---|---|---|---
GP | RL | SWD$\times 10^{2}\downarrow$ | MD $\times 10^{3}\downarrow$ | JSD$\times 10^{2}\downarrow$ | FPD $\downarrow$
_ | _ | 61.73 | 1.73 | 2.82 | 212
_ | $\checkmark$ | 17.68 | 1.66 | 1.99 | 122
$\checkmark$ | $\checkmark$ | 15.15 | 1.65 | 2.06 | 116

###  4.3 Ablation Study

We evaluate the importance of the two proposed techniques in our VQ-VAE model:
raydrop loss (RL) and geometric preservation (GP) in Table 3. In the baseline
scenario (first row), we exclude the raydrop logits channel such that VQ-VAE
directly approximates input noisy range images; moreover, we disable the GP
technique during training. We introduce the RL in the second row and then
incorporate both RL and GP techniques into the baseline in the third row. As
shown, the RL hugely reduces both image-based and point cloud-based metrics
due to more accurate raydrop generation. Furthermore, the GP technique
improves performance on most metrics up to 14% by increasing the VQ-VAE
generalisability.

##  5 Conclusion

This paper introduced LidarGRIT, a novel Lidar point cloud generative model.
The proposed LidarGRIT outperforms SOTA models on KITTI-360 and KITTI odometry
datasets. This work also highlighted the importance of the proposed raydrop
loss and geometric perseverance in the adapted VQ-VAE model, leading to
higher-quality generated samples.

## References

  * Caccia et al. [2018] Lucas Caccia, Herke van Hoof, Aaron C. Courville, and Joelle Pineau.  Deep generative modeling of lidar data.  _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 5034–5040, 2018.
  * Esser et al. [2021] Patrick Esser, Robin Rombach, and Björn Ommer.  Taming transformers for high-resolution image synthesis.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021_ , pages 12873–12883. Computer Vision Foundation / IEEE, 2021.
  * Geiger et al. [2013a] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun.  Vision meets robotics: The KITTI dataset.  _International Journal of Robotics Research_ , 32(11):1231–1237, 2013a.
  * Geiger et al. [2013b] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The kitti dataset.  _International Journal of Robotics Research (IJRR)_ , 2013b.
  * Karras et al. [2017] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  _ArXiv_ , abs/1710.10196, 2017.
  * Liao et al. [2022] Yiyi Liao, Jun Xie, and Andreas Geiger.  KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d.  _Pattern Analysis and Machine Intelligence (PAMI)_ , 2022.
  * myc634 [2023] myc634.  Unofficial implementation of ultralidar.  https://github.com/myc634/UltraLiDAR_nusc_waymo, 2023.
  * Nakashima and Kurazume [2021] Kazuto Nakashima and Ryo Kurazume.  Learning to drop points for lidar scan synthesis.  _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 222–229, 2021.
  * Nakashima and Kurazume [2024] Kazuto Nakashima and Ryo Kurazume.  Lidar data synthesis with denoising diffusion probabilistic models.  abs/2309.09256, 2024.
  * Nakashima et al. [2022] Kazuto Nakashima, Yumi Iwashita, and Ryo Kurazume.  Generative range imaging for learning scene priors of 3d lidar data, 2022.
  * Radford et al. [2015] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks.  _CoRR_ , abs/1511.06434, 2015.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , page 6000–6010, Red Hook, NY, USA, 2017. Curran Associates Inc.
  * Xiong et al. [2023] Yuwen Xiong, Wei-Chiu Ma, Jingkang Wang, and Raquel Urtasun.  Learning compact representations for lidar completion and generation.  In _CVPR_ , 2023.
  * Zyrianov et al. [2022] Vlas Zyrianov, Xiyue Zhu, and Shenlong Wang.  Generate realistic lidar point clouds.  In _Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXIII_ , page 17–35, Berlin, Heidelberg, 2022. Springer-Verlag.


# The Zilber–Pink conjecture for products of curves with highly degenerate
reduction

Netan Dogra

###### Abstract.

We give a proof of the Zilber–Pink conjecture for $n$-fold self-products of a
curve $X$ inside the self-product of its Jacobian, when $X$ has appropriate
bad reduction, $\mathrm{Jac}(X)$ has endomorphism algebra $\mathbb{Z}$ and $n$
is sufficiently small. The strategy of proof follows the work of Katz,
Rabinoff and Zureick-Brown on explicit Manin–Mumford bounds.

Let $X$ be a smooth projective irreducible genus $g$ curve over $\mathbb{C}$,
and $n>0$. Let

$\iota:X\to J$

denote the Abel–Jacobi morphism relative to a point $b\in X(\mathbb{C})$. We
define the set of linearly dependent points in $X^{n}$ to be

$X^{n}(\mathbb{C})_{\mathrm{rk}<n}:=\\{(x_{1},\ldots,x_{n})\in
X^{n}(\mathbb{C}):\mathrm{rk}\langle\iota(x_{1}),\ldots,\iota(x_{n})\rangle<n\\}.$

Equivalently, this may be defined as the set of $n$-tuples $(x_{i})$ for which
there exists a non-constant function $f\in\mathbb{C}(X)^{\times}$ with
$|\mathrm{Div}(f)|\subset\\{x_{i}\\}\cup\\{b\\}$ (where for a divisor $D$ we
write $|D|$ for its support). The image of $X^{n}(\mathbb{C})_{\mathrm{rk}<n}$
in $J^{n}(\mathbb{C})$ is exactly the intersection of $\iota(X)^{n}$ with the
union of all subgroups of $J^{n}$ of the form

(1) $\mathrm{Ker}(\sum m_{i}\pi_{i*}),$

for $(m_{i})\in\mathbb{Z}^{n}-\\{(0,\ldots,0)\\}$ and $\pi_{i}:J^{n}\to J$ the
$i$th projection.

The following is a special case of the Zilber–Pink conjecture, in this
generality due to Bombieri–Masser–Zannier, Pink and Zilber (independently)
[BMZ07], [Pin05], [Zil02].

######  Conjecture 1.

Let $X$ be a smooth projective genus $g$ curve over $\mathbb{C}$, and $n>0$.
Then $X^{n}(\mathbb{C})_{\mathrm{rk}<n}$ is not Zariski dense in $X^{n}$.

The general form of the Zilber–Pink conjecture for abelian varieties is as
follows (note that we will say nothing about this general form in the
remainder of the paper).

######  Conjecture 2.

Let $A$ be an abelian variety of dimension $g$ over $\mathbb{C}$. Let
$V\subset A$ be an irreducible subvariety of dimension $d$. Then

$V(\mathbb{C})\cap\cup_{B\subset A}B(\mathbb{C})$

is not Zariski dense in $V$, where the union is over all algebraic subgroups
$B\subset A$ of codimension at least $d+1$.

If $\mathrm{End}(J)=\mathbb{Z}$, then Conjecture 1 is equal to Conjecture 2
for $(A,V)$ equal to Conjecture 2 with $(A,V)=(J^{n},X^{n})$, since all
subgroups of $J^{n}$ will be of the form (1). Otherwise conjecture 1 is
strictly weaker, as it does not involve the subgroups of $J^{n}$ arising from
extra endomorphisms. Cases of conjecture 2 when $A$ has complex multiplication
are proved by Rémond in [R0́9], and when $A$ is a product of elliptic curves
by Hubschmid and Viada [HV19]. In [BD22], Barroero and Dill showed that to
prove Conjecture 1, it is enough to prove it over number fields. In [HP16],
Habegger and Pila show how, using the Pila–Zanier strategy, the conjecture
would follow from suitable Galois lower bounds on ‘optimal singletons’.

###  0.1. Statement of results

In [Dog22, Proposition 1], a version of Conjecture 1 was proved when $n<g$ and
with $\mathbb{C}$ replaced by $\mathbb{Q}_{p}$ (this was a simple special case
of a more general result for families of curves [Dog22, Theorem 1]). This was
deduced from the Chabauty–Coleman method together with classical functional
transcendence results. In the present paper we will show that the same
strategy can be enriched to prove the following theorem, which provides a
proof of infinitely many new instances of Conjecture 1. To explain the result,
recall that the dual graph of a stable curve $C$ is the graph $\Gamma$ with
$V(\Gamma)$ equal to the set of irreducible components of $C$, and $E(\Gamma)$
the set of singular points of of $X$. Since all singularities are double
points, each singular point $e$ lies on at most two irreducible components of
$X$, and these are the vertices it connects. In particular, loops and multiple
edges are allowed. For $S$ a subset of $V(\Gamma)$, we let $E(S)$ denote the
set of edges which have an element of $S$ as an end point. Finally, for a
vertex $v$ of the dual graph we let $g(v)$ denote the genus of the
corresponding irreducible component.

######  Theorem 1.

Let $X/\mathcal{O}_{\overline{\mathbb{Q}}_{p}}$ be a stable genus $g$ curve
with smooth generic fibre $X_{\overline{\mathbb{Q}}_{p}}$. Let $\Gamma$ denote
the dual graph of $X_{\overline{\mathbb{F}}_{p}}$ in the sense above. Suppose
that

$n\leq\min\\{g-E(S)-2\sum_{v\in S}^{n}g(v):S\subset V(\Gamma),\\#S\leq n\\}.$

Then Conjecture 1 holds for $X_{\mathbb{C}}$ and $n$.

######  Remark 1.

Although the conditions on the reduction type may seem quite restrictive, for
any $n$ and any $p$ there will be a $g_{n}$ such that, for all $g\geq g_{n}$,
there are infinitely many curves $X$ (over $\overline{\mathbb{Q}}$) which
satisfy the conditions at some prime above $p$. Namely, for any $n$, for
sufficiently large $g$ the conditions on the dual graph will be satisfied on a
nonempty closed subset of $\overline{M}_{g}-M_{g}$, and we can simply choose a
lift of an $\overline{\mathbb{F}}_{p}$ point of this subset to an
$\overline{\mathbb{Z}}$ point whose generic fibre lies in $M_{g}$. For
example, when $n=2$, we can find curves of genus at least 8 which satisfy the
conditions of the theorem.

###  0.2. Katz–Rabinoff–Zureick-Brown’s Manin–Mumford bound

In [KRZB16], Katz, Rabinoff and Zureick-Brown proved Theorem 1 in the case
where $V$ is a curve and $A$ is its Jacobian. In this case the result is
simply the Manin–Mumford conjecture: the intersection of
$\iota(X(\mathbb{C}))$ with the set of torsion points of $J(\mathbb{C})$ is
finite. Moreover they gave an explicit bound for the number of torsion points
on $X(\mathbb{C})$ (see [KRZB16, Theorem 1.2 and 1.3]). Their proof used
Coleman’s description of the de Rham cohomology of a curve with semistable
reduction [CI99] [CI10] to characterise the set of torsion points on $X$ as
the zero set of a finite number of nontrivial rigid analytic functions on
affinoid curves. Since a nontrivial rigid analytic function on an affinoid
curve has only finitely many zeroes, it follows immediately that the set of
torsion points is finite.

### Acknowledgements

The idea of applying the Katz–Rabinoff–Zureick-Brown strategy to the
Zilber–Pink conjecture was inspired by ongoing work with Arnab Saha, on
Zilber–Pink in the setting of good reduction. This research was supported by a
Royal Society University Research Fellowship.

##  1\. The proof for $\mathbb{Q}_{p}$-points

To explain the proof of Theorem 1, we briefly recall the proof of the
following theorem.

######  Proposition 1.

[[Dog22]] The set $X(\mathbb{Q}_{p})^{n}_{\mathrm{rk}<n}$ is not Zariski dense
in $X^{n}$.

###  1.1. Coleman integration

Proposition 1 may be proved using Coleman’s theory of $p$-adic abelian
integrals [Col85] [Bes02], together with some results from functional
transcendence. Coleman’s theory defines functions

$\int_{b}:X(\mathbb{Q}_{p})\to H^{0}(X,\Omega_{X|\mathbb{Q}_{p}})^{*}$

with the property that if $\sum n_{i}[z_{i}-b]=0$ in $J(\mathbb{Q}_{p})$ then
$\sum n_{i}\int^{z_{i}}_{b}\omega=0$ for all $\omega\in H^{0}(X,\Omega)$.
Hence $X(\mathbb{Q}_{p})^{n}_{\mathrm{rk}<n}$ maps into the subspace

$Z_{n}(H^{0}(X,\Omega_{X|\mathbb{Q}_{p}})^{*}).$

Here we adopt the convention that, for a $K$-vector space $V$ and $n\leq\dim
V$, $Z_{n}(V)$ denotes the subspace of rank $<n$ tuples in $V^{\oplus n}$, or
equivalently the kernel of the map

$V^{\oplus n}\to\wedge^{n}V$

sending $(v_{1},\ldots,v_{n})$ to $v_{1}\wedge\ldots\wedge v_{n}$. This
subspace is codimension $\dim V$ [Eis95], and in our case of interest is given
by the vanishing of the determinants of all $n\times n$ minors of the matrix
$(\int^{z_{i}}_{b}\omega_{j})$ for $\omega_{j}$ ranging over a basis of
$H^{0}(X,\Omega)$.

For future use, we will introduce the notation that for a subspace $W$ of
$H^{0}(X,\Omega_{X|\overline{\mathbb{Q}}_{p}})$, we denote by

$\log_{W}:X(\overline{\mathbb{Q}}_{p})\to W^{*}$

the restriction of the functional $\int_{b}$ to $W$.

For $\omega\in H^{0}(X,\Omega)$, the function
$\int_{b}\omega:X(\overline{\mathbb{Q}}_{p})\to\overline{\mathbb{Q}}_{p}$ is
not rigid analytic, however for all $z\in X(\overline{\mathbb{F}}_{p})$, its
restriction to the tube $]z[$ of $z$ is a rigid analytic function. When we
speak of $\overline{\mathbb{F}}_{p}$ points we are implicitly choosing a
regular model $X/\mathbb{Z}_{p}$ (for the purpose of this discussion of the
proof of Proposition 1 – which is only motivational – the reader may assume
that $X$ has good reduction). Recall that the tube of $z$ is isomorphic to an
open ball $B(0,1)=\\{z:|z|<1\\}$. In particular, the ring of functions on
$B(0,1)$ is not Noetherian and a rigid analytic function on $B(0,1)$ may have
infinitely many zeroes. However, the $\mathbb{Q}_{p}$ points of $]z[$, for
$z\in X(\mathbb{F}_{p})$, are contained in a closed disk isomorphic to
$D(0,\frac{1}{p})=\\{z:|z|\leq\frac{1}{p}\\}$.

In particular, since affinoid algebras are Noetherian, the common
$\mathbb{Q}_{p}$-zeroes of a finite set of rigid analytic functions
$(f_{1},\ldots,f_{m})$ on $](z_{1},\ldots,z_{n})[$ is the
$\mathbb{Q}_{p}$-points of an affinoid subspace (namely
$V(f_{1},\ldots,f_{m})\cap D(0,1/p)^{n}$, via the isomorphism
$](z_{1},\ldots,z_{n})[\simeq B(0,1)^{n}$ above). To check non-Zariski density
of this set of points, it is enough to check non–Zariski density of each
irreducible component of this affinoid. Given an irreducible rigid analytic
subspace $Y$ of $X^{n}$, to check that $Y$ is not Zariski density it is enough
to check it at the formal completion of an arbitrary point.

This allows us to reduce Proposition 1 to a question amenable to classical
functional transcendence. Namely to prove non–Zariski density of
$X(\mathbb{Q}_{p})^{n}_{\mathrm{rk}<n}$, it is enough to prove it for
$X(\mathbb{Q}_{p})^{n}_{\mathrm{rk}<n}$ $\cap](z_{1},\ldots,z_{n})[$ for a
fixed $(z_{1},\ldots,z_{n})\in X(\mathbb{F}_{p})^{n}$. By the theory of
abelian integrals above, it is enough to prove that the pre-image of
$Z_{n}(H^{0}(X,\Omega)^{*})$ in $](z_{1},\ldots,z_{n})[$ under the map
$(\int_{b})^{n}$ is not Zariski dense in $X^{n}_{\mathbb{Q}_{p}}$. Since this
is contained in the common zeroes of a finite set of rigid analytic functions
on an affinoid rigid analytic space, to prove non–Zariski density it is enough
to prove it at at the formal completion of an arbitrary point
$(x_{1},\ldots,x_{n})\in
X(\mathbb{Q}_{p})^{n}_{\mathrm{rk}<n}\cap](z_{1},\ldots,z_{n})[$.

###  1.2. The Ax–Schanuel Theorem

To study the Zariski closure of the formal completion, we use the following
Theorem due to Ax [Ax72].

######  Theorem 2 (Ax–Schanuel for abelian varieties).

Let $A$ be an abelian variety over a field $K$ of characteristic zero. Let
$\widehat{A\times\mathrm{Lie}(A)}$ denote the formal completion of
$A\times\mathrm{Lie}(A)$ at the identity $(e,0)$, and let
$\pi:\widehat{A\times\mathrm{Lie}(A)}\to\widehat{A}$ denote the projection.
Let $\Delta$ denote the graph of the exponential
$\widehat{\mathrm{Lie}(A)}\to\widehat{A}$. Let $V\subset
A\times\mathrm{Lie}(A)$ be an irreducible subvariety. Let $W\subset$. Suppose

$\dim W>\max\\{0,\dim V-g\\}.$

Then $\pi(W)$ is contained in the formal completion of a proper subgroup of
$A$.

###  1.3. Conclusion of proof

The conclusion of the proof of Proposition 1 is very simple. Recall that we
have reduced to the following problem. We have a point $(x_{1},\ldots,x_{n})$
in $X(\mathbb{Q}_{p})^{n}_{\mathrm{rk}<n}\cap](z_{1},\ldots,z_{n})[$ which
maps to $P=(\int^{x_{i}}_{b})\in Z_{n}(H^{0}(X,\Omega)^{*})$. We then wanted
to show that the pre-image of $\widehat{Z_{n}}(H^{0}(X,\Omega)^{*})_{P}$ in
$\widehat{X}^{n}_{(x_{1},\ldots,x_{n})}$ under the formal completion of
$(\int_{b})^{n}$ is not Zariski dense. This is easily translated into the
setting of the Ax–Schanuel theorem: we take
$Z_{n,P}\subset(H^{0}(X,\Omega)^{*})^{n}$ to be the translation of
$Z_{n}(H^{0}(X,\Omega)^{*})$ under translation by $-P$, and view $X^{n}$ as a
subvariety of $J^{n}$ via the Abel–Jacobi morphism relative to the basepoint
$(x_{1},\ldots,x_{n})$. Then the pre-image of
$\widehat{Z_{n}}(H^{0}(X,\Omega)^{*})$ in $\widehat{X}^{n}$ is simply the
projection to $\widehat{X}^{n}$ of the intersection of $\Delta$ with
$\widehat{X}^{n}\times\widehat{Z_{n}}(H^{0}(X,\Omega)^{*})$, which is not
Zariski dense by Theorem 2.

##  2\. The Katz–Rabinoff–Zureick-Brown strategy

Clearly the above strategy does not tell us about
$X(\overline{\mathbb{Q}}_{p})^{n}_{\mathrm{rk}<n}$, for two reasons. Firstly,
there are infinitely many residue disks, since there are infinitely many
$\overline{\mathbb{F}}_{p}$-points of the special fibre. Secondly, at each
$\overline{\mathbb{F}}_{p}$-point $z$, the $\overline{\mathbb{Q}}_{p}$ points
lying above it are not contained in an affinoid open inside $]z[$, and hence
the common zeroes of a finite set of functions may have infinitely many
irreducible components.

We now explain how the Katz–Rabinoff–Zureick-Brown strategy can be used to get
around these issues, in the case of highly degenerate reduction. This allows
us to deduce Theorem 1 from Ax–Schanuel in the same manner as Proposition 1.
We first recall some things from [KRZB16]. As our definitions are slightly
different from theirs we will go through the details (although all the ideas
are taken from [KRZB16]). Let $X/\mathcal{O}$ be a stable curve with smooth
generic fibre of genus $g$. Let $\Gamma$ denote the dual graph of
$\mathcal{X}_{\overline{\mathbb{F}}_{p}}$. For $v\in\Gamma$, $X_{v}$ denote
the corresponding irreducible components. Let $\mathcal{S}\subset
X_{\overline{\mathbb{F}}_{p}}$ denote the singular locus. Let $n_{v}$ denote
the number of geometric points of $X_{v}$ which lie on the singular locus of
$\mathcal{X}_{\overline{\mathbb{F}}_{v}}$, and let $g_{v}$ denote the genus of
$X_{v}$. Let $U_{v}\subset X^{\mathrm{an}}$ be the pre-image of
$X_{v}-\mathcal{S}$ under the reduction map. For each $v$, we fix an affinoid
strict neighbourhood $U_{v}^{+}\supset U_{v}$ with the following properties:

  1. (1)

$U_{v}^{+}\subset]X_{v}[$.

  2. (2)

For each edge $e$ with end vertices $v_{1}$ and $v_{2}$,
$]e[\cap(U_{v_{1}}^{+}\cup U_{v_{2}}^{+})$ is a disjoint union of two annuli.
That is, the strict neighbourhoods $U_{v}^{+}$ do not ‘meet’ on $]e[$.

Let $H^{1}_{\mathrm{dR}}(U_{v}^{+}/\overline{\mathbb{Q}}_{p})$ denote the
overconvergent de Rham cohomology of $U_{v}^{+}$. Explicitly (since we assume
that $X$ is not smooth, and hence $X_{v}-\mathcal{S}$ is affine), we have

$H^{1}_{\mathrm{dR}}(U_{v}^{+}/\overline{\mathbb{Q}}_{p})\simeq\Omega^{\dagger}_{U_{v}^{+}|\overline{\mathbb{Q}}_{p}}/d\mathcal{O}^{\dagger}_{U_{v}^{+}|\overline{\mathbb{Q}}_{p}}$

where

$\mathcal{O}^{\dagger}_{U_{v}^{+}|\overline{\mathbb{Q}}_{p}}=\varinjlim
j^{*}\mathcal{O}(U^{\prime})$

is the limit over strict neighbourhoods $j:U_{v}\hookrightarrow U^{\prime}$ of
$U_{v}$ in $X$, and similarly for
$\Omega^{\dagger}_{U_{v}^{+}|\overline{\mathbb{Q}}_{p}}$.

Then $H^{1}_{\mathrm{dR}}(U_{v}^{+}/\overline{\mathbb{Q}}_{p})$ is a
$\overline{\mathbb{Q}}_{p}$-vector space of dimension $2g_{v}+n_{v}$. In
particular, if $g<2g_{v}+n_{v}$, then there is a nontrivial kernel of the
restriction map

$H^{0}(X_{\overline{\mathbb{Q}}_{p}},\Omega_{X|\overline{\mathbb{Q}}_{p}})\to
H^{1}_{\mathrm{dR}}(U_{v}^{+}/\overline{\mathbb{Q}}_{p}).$

For an edge $e$, we define $U_{e}^{+}\subset]e[$ to be an affinoid open inside
$]e[$ with the property that $(U_{v}^{+})_{v\in V(\Gamma)\sqcup E(\Gamma)}$ is
an admissible covering of $X^{\mathrm{an}}$:

$\cup_{v\in V(\Gamma)\sqcup E(\Gamma)}U_{v}^{+}=X^{\mathrm{an}}.$

######  Remark 2.

The notation $U_{e}^{+}$ is a little odd, as $U_{e}^{+}$ is smaller than the
tube of $]e[$, but it is designed so that $U_{v}^{+}$ makes sense whether $v$
is a vertex or an edge. It might be more elegant to take the approach of
blowing up the stable model once along each singular point, giving a dual
graph $\Gamma^{\prime}$ which is obtained by ‘subdividing’ each edge $e$ of
$\Gamma$ into two edges, with a new vertex corresponding to $e$ (this is the
approach taken in [KRZB16, Proof of Theorem 5.5]). However the admissible
covering of $X^{\mathrm{an}}$ one obtains from this is ultimately the same.

We may also form the de Rham cohomology of $U_{e}^{+}$, defined simply as

$H^{1}_{\mathrm{dR}}(U_{e}/\overline{\mathbb{Q}}_{p}):=\Omega_{U_{e}|\overline{\mathbb{Q}}_{p}}/d\mathcal{O}(U_{e}).$

This is a $\overline{\mathbb{Q}}_{p}$-vector space of dimension $1$. If
$\omega\in H^{0}(X,\Omega_{X|\overline{\mathbb{Q}}_{p}})$ vanishes upon
pulling back to $H^{1}_{\mathrm{dR}}(U_{v}^{+}/\overline{\mathbb{Q}}_{p})$,
then it also vanishes upon pulling back to
$H^{1}_{\mathrm{dR}}(U_{e}^{+}/\overline{\mathbb{Q}}_{p})$ for all edges $e$
with $v$ as an endpoint (this follows from the fact that the image of $\omega$
in $H^{1}_{\mathrm{dR}}(U_{e}^{+}/\overline{\mathbb{Q}}_{p})\simeq K$ may be
identified with the residue of $\omega|_{U_{v}^{+}}$ at $e$). We deduce the
following.

######  Proposition 2.

For $X$ as in the statement of Theorem 1, for each $n$-tuple
$(v_{1},\ldots,v_{n})$ of elements of $V(\Gamma)\sqcup E(\Gamma)$, there
exists a subspace $W$ of $H^{0}(X,\Omega)$ of dimension at least $n$ such that

  1. (1)

$X(\overline{\mathbb{Q}}_{p})^{n}_{\mathrm{rk}<n}\cap
U_{v_{1}}^{+}\times\ldots\times U_{v_{n}}^{+}$ maps to $Z_{n}(W^{*})$ under
the map $\log_{W}^{n}$.

  2. (2)

For all $\omega\in W$, the restriction of $\int\omega$ to $U_{v}^{+}$ is a
rigid analytic function on $U_{v}^{+}$.

###  2.1. Completion of proof

Via an isomorphism $\overline{\mathbb{Q}}_{p}\simeq\mathbb{C}$ it is enough to
prove that $X(\overline{\mathbb{Q}}_{p})^{n}_{\mathrm{rk}<n}$ is not Zariski
dense in $X^{n}_{\overline{\mathbb{Q}}_{p}}$. Furthermore it is enough to fix
a tuple $(v_{1},\ldots,v_{n})$ of elements of $V(\Gamma)\sqcup E(\Gamma)$ and
prove Zariski non-density of
$X(\overline{\mathbb{Q}}_{p})^{n}_{\mathrm{rk}<n}\cap
U_{v_{1}}^{+}\times\ldots\times U_{v_{n}}^{+}$. From Proposition 2 we have a
subspace $W$ of $H^{0}(X_{\overline{\mathbb{Q}}_{p}},\Omega)$ of dimension at
least $n$ such that $\log_{W}$ is given by a rigid analytic function on the
affinoid $U_{v_{i}}^{+}$ for all $i$. In particular, since the common zeroes
of a finite set of functions on an affinoid variety have finitely many
irreducible components, to prove non–Zariski density of
$X(\overline{\mathbb{Q}}_{p})^{n}_{\mathrm{rk}<n}\cap
U_{v_{1}}^{+}\times\ldots\times U_{v_{n}}^{+}$, it is enough to prove it at
the formal completion of an arbitrary point $(x_{1},\ldots,x_{n})\in
X(\overline{\mathbb{Q}}_{p})^{n}_{\mathrm{rk}<n}\cap
U_{v_{1}}^{+}\times\ldots\times U_{v_{n}}^{+}$. At this point we can evoke
Ax–Schanuel: since $Z_{n}(W^{*})$ is codimension $g>n$ in $(W^{*})^{n}$, if
the pre-image of $\widehat{Z}_{n}(W^{*})$ under $\log_{W}^{n}$ contains a
positive dimensional irreducible component, then that irreducible component
must map to a translate of a proper subgroup of $J^{n}$. In particular, this
irreducible component is not Zariski dense in $X^{n}$, completing the proof.

######  Remark 3.

The proof actually gives a slightly stronger result than is stated in Theorem
1. Namely, it is enough to suppose that $X$ has the property that, for all
subsets $S$ of $V(\Gamma)$ of size at most $n$, the kernel of

$H^{0}(X,\Omega)\to\mathrm{Image}(\oplus_{v\in
S}H^{1}_{\mathrm{dR}}(U_{v}^{+}/\overline{\mathbb{Q}}_{p})\to
H^{1}_{\mathrm{dR}}(X/\overline{\mathbb{Q}}_{p}))$

has dimension at least $n$. This is implied by the conditions of Theorem 1 but
in some cases will be a strictly weaker condition, for example a genus 4 curve
whose special fibre as in Figure 1 will satisfy Conjecture 1, by the argument
above.

Figure 1. Dual graph of the special fibre of a genus 4 curve satisfying
Conjecture 1 when $n=2$

## References

  * [Ax72] J. Ax.  Some topics in differential algebraic geometry I: Analytic subgroups of algebraic groups.  American Journal of Mathematics, 94(4):1195–1204, 1972.
  * [BD22] Fabrizio Barroero and Gabriel A. Dill.  On the Zilber-Pink conjecture for complex abelian varieties.  Ann. Sci. Éc. Norm. Supér. (4), 55(1):261–282, 2022.
  * [Bes02] Amnon Besser.  Coleman integration using the Tannakian formalism.  Math. Ann., 322(1):19–48, 2002.
  * [BMZ07] E. Bombieri, D. Masser, and U. Zannier.  Anomalous subvarieties—structure theorems and applications.  Int. Math. Res. Not. IMRN, (19):Art. ID rnm057, 33, 2007.
  * [CI99] Robert Coleman and Adrian Iovita.  The Frobenius and monodromy operators for curves and abelian varieties.  Duke Math. J., 97(1):171–215, 1999.
  * [CI10] Robert Coleman and Adrian Iovita.  Hidden structures on semistable curves.  Astérisque, (331):179–254, 2010.
  * [Col85] Robert F. Coleman.  Torsion points on curves and $p$-adic abelian integrals.  Ann. of Math. (2), 121(1):111–168, 1985.
  * [Dog22] Netan Dogra.  p-adic integrals and linearly dependent points on families of curves i.  arXiv preprint arXiv:2206.04304, 2022.
  * [Eis95] David Eisenbud.  Commutative algebra, volume 150 of Graduate Texts in Mathematics.  Springer-Verlag, New York, 1995.  With a view toward algebraic geometry.
  * [HP16] Philipp Habegger and Jonathan Pila.  O-minimality and certain atypical intersections.  Ann. Sci. Éc. Norm. Supér. (4), 49(4):813–858, 2016.
  * [HV19] Patrik Hubschmid and Evelina Viada.  An addendum to the elliptic torsion anomalous conjecture in codimension 2.  Rend. Semin. Mat. Univ. Padova, 141:209–220, 2019.
  * [KRZB16] Eric Katz, Joseph Rabinoff, and David Zureick-Brown.  Uniform bounds for the number of rational points on curves of small Mordell-Weil rank.  Duke Math. J., 165(16):3189–3240, 2016.
  * [Pin05] Richard Pink.  A combination of the conjectures of Mordell-Lang and André-Oort.  In Geometric methods in algebra and number theory, volume 235 of Progr. Math., pages 251–282. Birkhäuser Boston, Boston, MA, 2005.
  * [R0́9] Gaël Rémond.  Intersection de sous-groupes et de sous-variétés. III.  Comment. Math. Helv., 84(4):835–863, 2009.
  * [Zil02] Boris Zilber.  Exponential sums equations and the Schanuel conjecture.  J. London Math. Soc. (2), 65(1):27–44, 2002.


Current privacy research on large language models (LLMs) primarily focuses on the issue of extracting memorized training data. At the same time, models’ inference capabilities have increased drastically. This raises the key question of whether current LLMs could violate individuals’ privacy by inferring personal attributes from text given at inference time. In this work, we present the first comprehensive study on the capabilities of pretrained LLMs to infer personal attributes from text. We construct a dataset consisting of real Reddit profiles, and show that current LLMs can infer a wide range of personal attributes (e.g., location, income, sex), achieving up to $85\%$ top-1 and $95.8\%$ top-3 accuracy at a fraction of the cost ($100\times$) and time ($240\times$) required by humans. As people increasingly interact with LLM-powered chatbots across all aspects of life, we also explore the emerging threat of privacy-invasive chatbots trying to extract personal information through seemingly benign questions. Finally, we show that common mitigations, i.e., text anonymization and model alignment, are currently ineffective at protecting user privacy against LLM inference. Our findings highlight that current LLMs can infer personal data at a previously unattainable scale. In the absence of working defenses, we advocate for a broader discussion around LLM privacy implications beyond memorization, striving for a wider privacy protection.

§ INTRODUCTION

The recent advances in capabilities [OpenAI, 2023, Anthropic, 2023, Touvron et al., 2023] of large pre-trained language models (LLMs), together with increased availability, have sparked an active discourse about privacy concerns related to their usage [Carlini et al., 2021, Carlini et al., 2023].
An undesired side effect of using large parts of the internet for training is that models memorize vast amounts of potentially sensitive training data, possibly leaking them to third parties [Carlini et al., 2021].
While particularly relevant in recent generative models, the issue of memorization is not inherently exclusive to LLMs and has been demonstrated in earlier models such as LSTMs [Carlini et al., 2019].
However, as we show in this work, the privacy risks associated with current state-of-the-art LLMs extend beyond this established understanding.

This Work: Privacy Violations through LLM Inference In particular, we find that with increased capabilities, LLMs are able to automatically infer a wide range of personal author attributes from large collections of unstructured text (e.g., public forum or social network posts) given to them at inference time. Combined with the increased proliferation of LLMs, this drastically lowers the costs associated with privacy-infringing inferences. In turn, this allows an adversary to scale far beyond what previously would have been possible with expensive human profilers.
For instance, as illustrated in <ref>, imagine a user leaving the following seemingly harmless comment on a pseudonymized online platform (e.g., Reddit) under a post about daily work commutes:

“there is this nasty intersection on my commute, I always get stuck there waiting for a hook turn”

Although the user had no intent of revealing their location, current LLMs are able to pick up on small cues left in their comment. Prompting GPT-4, it correctly deduces that the user comes from Melbourne, noting that “a "hook turn" is a traffic maneuver particularly used in Melbourne.”.
In <ref>, we show two more examples (derived from <ref>) how LLMs' strong language understanding capabilities enable such inferences across various personal attributes and texts.

/csteps/inner ysep=2.5pt

Adversarial inference of personal attributes from text. We assume the adversary has access to a dataset of user-written texts (e.g., by scraping an online forum). Given a text, the adversary creates a model prompt using a fixed adversarial template 1. They then leverage a pre-trained LLM in 2 to automatically infer personal user attributes 3, a task that previously required humans. current models are able to pick up on subtle clues in text and language (<ref>), providing accurate inferences on real data.
Finally, in 4, the model uses its inference to output a formatted user profile.

In this work, we demonstrate that by scraping the entirety of a user's online posts and feeding them to a pre-trained LLM, malicious actors can infer private information never intended to be disclosed by the users.
It is known that half of the US population can be uniquely identified by a small number of attributes such as location, gender, and date of birth [Sweeney, 2002]. LLMs that can infer some of these attributes from unstructured excerpts found on the internet could be used to identify the actual person using additional publicly available information (e.g., voter records in the USA).
This would allow a malicious actor to link highly personal information inferred from posts (e.g., mental health status) to an actual person and use it for undesirable or illegal activities like targeted political campaigns, automated profiling, or stalking.

For this, we investigate the capabilities of $9$ widely used state-of-the-art LLMs (e.g., GPT-4, Claude 2, Llama 2) to infer $8$ personal attributes, showing that they achieve already ${\sim85}\%$ top-1 and ${\sim95.8}\%$ top-3 accuracy on real-world data. Despite these models achieving near-expert human performance, they come at a fraction of the cost, requiring $100\times$ less financial and $240\times$ lower time investment than human labelers—making such privacy violations at scale possible for the first time.

Emerging Frontiers
All risks discussed so far focus on LLMs being used to analyze already existing texts. However, a new form of online communication is emerging, as millions of people start to interact with thousands of custom chatbots on a range of platforms [ChAI, 2022, Poe, 2023, 15]. Our findings indicate that this can create unprecedented risks for user privacy. In particular, we demonstrate that malicious chatbots can steer conversations, provoking seemingly benign responses containing sufficient information for the chatbot to infer and uncover private information.

Potential Mititgations
Beyond attacks, we also investigate two directions from which one could try to mitigate this issue: from the client side, a first defense against LLM-based attribute inference would be removing personal attributes using existing text anonymization tools. Such an approach was recently implemented specifically for LLMs [Lakera, 2023]. However, we find that even when anonymizing text with state-of-the-art tools for detecting personal information, LLMs can still infer many personal attributes, including location and age. As we show in <ref>, LLMs often pick up on more subtle language clues and context (e.g., region-specific slang or phrases) not removed by such anonymizers. With current anonymization tools being insufficient, we advocate for stronger text anonymization methods to keep up with LLMs' rapidly increasing capabilities.

From a provider perspective, alignment is currently the most promising approach to restricting LLMs from generating harmful content. However, research in this area has primarily focused on avoiding unsafe, offensive, or biased generations [OpenAI, 2023, Touvron et al., 2023] and has not considered the potential privacy impact of model inferences. Our findings in <ref> confirm this, showing that most models currently do not filter privacy invasive prompts. We believe better alignment for privacy protection is a promising direction for future research.

Main contributions

Our key contributions are:

* The first formalization of the privacy threats resulting from inference capabilities of LLMs.

* A comprehensive experimental evaluation of LLMs' ability to infer personal attributes from real-world data both with high accuracy and low cost, even when the text is anonymized using commercial tools.

* A release of our code, prompts, and synthetic chatlogs at <https://github.com/eth-sri/llmprivacy>. Additionally, we release a dataset of 525 human-labeled synthetic examples to further the research in this area.

Responsible Disclosure
Prior to publishing this work, we contacted OpenAI, Anthropic, Meta, and Google, giving access to all our data, resulting in an active discussion on the impact of privacy-invasive LLM inferences. We refer to <ref> for a further discussion of ethical considerations.

§ RELATED WORK

*Privacy Leakage in LLMs

With the rise of large language models in popularity, a growing number of works have addressed the issue of training data memorization [Carlini et al., 2021, Kim et al., 2023, Lukas et al., 2023, Ippolito et al., 2023]. Memorization refers to the exact repetition of training data sequences during inference in response to a specific input prompt, often the corresponding prefix. Carlini et al., 2023 empirically demonstrated a log-linear relationship between memorization, model size, and training data repetitions, a worrisome trend given the rapidly growing model and dataset sizes.
As pointed out by Ippolito et al., 2023, however, verbatim memorization does not capture the full extent of privacy risks posed by LLMs.
Memorized samples can often be recovered approximately, and privacy notions are strongly context-dependent [Brown et al., 2022].
Yet, the threat of memorization is bounded to points in the model's training data. This is in stark contrast to inference-based privacy violations, which can happen on any data presented to the model. While acknowledged as a potential threat in recent literature [Bubeck et al., 2023], there is, to our knowledge, no existing study of the privacy risks of pre-trained LLMs inferences to user privacy.

*Risks of Large Language Models
Besides privacy violations (inference or otherwise), unrestricted LLMs can exhibit a wide range of safety risks. Current research in model risks and mitigations focuses mainly on mitigating harmful (e.g., "How do I create a bomb?"), unfairly biased, or otherwise toxic answers [OpenAI, 2023, Touvron et al., 2023]. The most popular provider-side mitigations currently used are all forms of "model alignment," most commonly achieved by finetuning a raw language model to align with a human-preference model that penalizes harmful generations. However, recent findings by Zou et al., 2023 show that such alignments can be broken in an automated fashion, fueling the debate for better alignment methods.

*Personal data and PII
Legal definitions of personal data vary between jurisdictions. Within the EU, the General Data Protection Regulation (GDPR)  [EU, 2016] defines personal data in Article 4 as "any information relating to an identified or identifiable natural person" explicitly including location data and a persons economic, cultural or social identity. The Personal Identifiable Information (PII) definitions applied under U.S. jurisdiction are less rigorous but, similarly to GDPR, acknowledge the existence of sensitive data such as race, sexual orientation, or religion. All of the attributes investigated in <ref> (e.g., location, income) fall under the personal data definitions of these legislatures as they could be used with additional information to identify individuals.

*Author Profiling

Author profiling, the process of extracting specific author attributes from written texts, has been a long-standing area of research in Natural Language Processing (NLP) [Estival et al., 2007, Rangel et al., 2013, Rangel et al., 2017].
However, current approaches focus predominantly on specific attributes (often gender and age), using specific feature extraction methods [Rangel et al., 2018].
As pointed out in Villegas et al., 2014, one significant challenge slowing the progress in this field is a lack of available datasets. The primary source of labeled author profiling datasets is the yearly PAN competition [Rangel et al., 2013], primarily focusing on Twitter texts and a few select attributes.
At the same time, the significant growth of available (unlabeled) online raises concerns about what other kinds of personal data malicious actors could infer from user-written texts. Our work addresses the gap between current author profiling work on specific textual domains/attributes and emerging LLMs trained on vast datasets showing strong language understanding capabilities across domains.

§ THREAT MODELS

In this section, we formalize the privacy threats presented in <ref> by introducing a set of adversaries $\mathcal{A}_{i \in \{1,2\}}$ with varying access to a pre-trained LLM $\mathcal{M}$. We first formalize the free text inference setting via an adversary $\mathcal{A}_1$ that infers personal attributes from unstructured free-form texts, such as online posts.
We show in our evaluation (<ref>) that an $\mathcal{A}_1$ adversary is both practical (i.e., high accuracy) and feasible (i.e., lower cost) on real-world data.
Considering the rapid development of LLM-based systems and proliferation of LLM-based chatbots, we additionally formalize the emerging setting of an adversary $\mathcal{A}_2$ controlling an LLM with which users interact.

§.§ Free Text Inference

Free text inference: The adversary creates a prompt from user texts, using an LLM do infer personal attributes.

The free text inference setting formalizes how an adversary can extract and infer information from unstructured texts. For this, we assume that an adversary $\mathcal{A}_1$ has access to a dataset $D$ consisting of texts written by individuals $u_i \in \mathbb{U}_D$. Such a dataset could be obtained by, e.g., scraping a large online forum or social media site. However, $D$ is not restricted to public-facing data—it could also come from (il)legally obtained records of internal communications or messenger chat logs [Yang, 2019]. Given $D$, the $A_1$ adversary's goal is to infer personal attributes of individuals contained in $D$.

Formally, let $(u,t) \in D$ be a pair of a user $u$ and text $t$ written by them. As shown in <ref>, we are interested in $\mathcal{A}_1$'s capability of extracting (attribute, value) tuples that match the author correctly. In particular, we write $u^a$ to refer to the value of attribute $a$ of user $u$. In <ref>, we have $u^{LOC}=$ Melbourne, $u^{AGE}=$ $47$,  $u^{SEX}=$ Female.
Given $t$,  $\mathcal{A}_1$ first creates a prompt $P_{\mathcal{A}_1}(t)=(\texttt{S},\texttt{P})$. For this, $P_{\mathcal{A}_1}$ is a function that takes in the text $t$ and produces both a system prompt $\texttt{S}$ and a prompt  which is given to the model . While this formulation is general, for the rest of this work, we restrict the prompt $\texttt{P}$ to $\texttt{P} =(\texttt{Prefix}~F_{\mathcal{A}_1}(t)~\texttt{Suffix})$ where $F_{\mathcal{A}_1}$ is a string formatting function. By having a fixed prefix and suffix, we exclude cases where an adversary could encode additional information via $P$ (e.g., vector-database queries).
The model $\mathcal{M}$ responds to this prompt with $\mathcal{M}(P_{\mathcal{A}_1}(t)) = \{(a_j,v_j)\}_{1 \leq j \leq k}$ the set of tuples it could infer from the text. For our experiments in <ref>, we additionally ask the model to provide its reasoning behind each inference.

It is important to note that across all settings $\mathcal{M}$ is a pre-trained LLM. In particular, the adversary $\mathcal{A}_i$ is no longer limited by the resource-intensive task of collecting a large training dataset and training a model on it. Using pre-trained “off-the-shelf" LLMs reduces such initial investments significantly, lowering the entry barrier for adversaries and enabling scaling. We explore this tradeoff further in <ref> by showing that on a restricted set of ACS [Ding et al., 2021] attributes, LLMs achieve strong 0-shot attribute inference performances, even compared to specifically finetuned classifiers.

In <ref>, we present our main experiments on real-world free text inference. We show that LLMs are already close to and sometimes even surpass the capabilities of human labelers on real-world data (<ref>). Several instances where human labelers required additional information could be correctly inferred by models based on text alone. Importantly, as we show in <ref>, we find that the models' strong inferential capabilities allow them to correctly infer personal attributes from, e.g., the specific language (such as local phrases) or subtle context that persists even under state-of-the-art text anonymization. Furthermore, such inferences become increasingly cheaper, allowing adversaries to scale beyond what would previously have been achievable with human experts.

§.§ Adversarial Interaction

With a rapidly increasing number of LLM-based chatbots and millions of people already using them daily, an emerging threat beyond free text inference is an active malicious deployment of LLMs. In such a setting, a seemingly benign chatbot steers a conversation with the user in a way that leads them to produce text that allows the model to learn private and potentially sensitive information.
This naturally extends over the passive setup of free text inference, as it enables the model to actively influence the interaction with the user, mining for private information. We formalize this setting below.

Illustration of the adversarial interaction. The user is unaware of $T_h$ given by the adversary. The model steers the conversation in each round to refine prior information.

Assume the user has only black-box access to the LLM, where, crucially, the system prompt  is only accessible by the adversary $\mathcal{A}_2$. Let $T_p$ be the public task of the LLM, e.g., “being a helpful travel assistant". Additionally, let $T_h$ be a potentially malicious hidden task of the LLM, in our case, trying to extract private information from the user. The system prompt of the LLM is a combination of both tasks, i.e., $\texttt{S} = (T_p,\, T_h)$. Each round $i$ of conversation between the user and the LLM consists of:
(1) a user message $m_i$,
(2) a hidden model response $r_i^h$ only available to the model hosting entity (e.g., PII inferences from prior responses),
and (3) a public model response $r_i^p$ revealed to the user.
For such an attack to succeed, besides fulfilling $T_h$, $T_h$ must also remain hidden from the user throughout the interaction.

In <ref>, we instantiate the $\mathcal{A}_2$ adversary using a free-conversational chatbot, mimicking the setup of popular platforms such as Character.AI [ChAI, 2022], with the hidden task of inferring personal attributes of the user. Our simulated experiment demonstrates that such a setup is already achievable with current LLMs, raising serious concerns about user privacy on such platforms.

§ A DATASET FOR LLM-BASED AUTHOR PROFILING

As mentioned in <ref>, a key issue in evaluating author profiling capabilities is the lack of available datasets [Villegas et al., 2014]. While there are commonly used datasets for LLM privacy research, such as the Enron-Email dataset [Klimt & Yang, 2004], these generally do not come with ground truth attribute labels. We found only one commonly used source of ground-truth labeled datasets in English: the yearly PAN competition datasets, which for author profiling consist of a set of texts (often tweets) with ground-truth labels for $1$ to $3$ attributes, commonly gender and age.
This is a substantial limitation when compared to the broad personal data definitions presented in <ref>.
We provide an evaluation for the (latest) author profiling dataset (PAN 2018) in <ref>—showing how GPT-4 outperforms all prior approaches by a significant margin.

*Key Requirements
To investigate LLMs' real-world attribute inference capabilities, we state two key requirements that a dataset should satisfy: (1) The texts must accurately reflect commonly used online language. As users interact with LLMs primarily in an online setting and given the volume of online texts, they are inherently at the highest risk of being subject to LLM inferences. (2) A diverse set of personal attributes associated with each text.
Data protection regulations (<ref>) are deliberately formulated to protect a wide range of personal attributes, which is not reflected by existing datasets, that focus on one or two common attributes. This is particularly relevant as the increasing capabilities of LLMs will enable the inference of more personal information from texts.

*The PersonalReddit (PR) Dataset

To fulfill these requirements, we constructed PersonalReddit (PR), a dataset consisting of 520 randomly sampled public Reddit profiles consisting of $5814$ comments between $2012$ and early $2016$. We restricted comments to a set of $381$ subreddits (see <ref>) likely to contain personal attributes.
Inspired by datasets created by the American Census Bureau (ACS), we selected the following eight attribute categories: age , education , sex , occupation , relationship state , location , place of birth , income .
We created ground truth labels by manually annotating attributes for all selected profiles. To ensure personal data is handled responsibly, labeling was not outsourced but only conducted by authors of the paper (referred to as labelers). We give a detailed overview of the labeling guidelines in <ref> and aggregate dataset statistics in <ref>. Labelers were asked to extract attribute values from each profile, providing perceived certainty and hardness scores on a 1-5 scale. We provide qualitative examples for each level in <ref>. For hardness scores $4$-$5$, labelers could use internet search engines (excluding LLMs).
While perceived hardness increases with the score for humans, samples of hardness $4$ often require extra information (internet search) but less reasoning than hardness $3$.
Further, labelers could view subreddit names not included in our LLM evaluation prompts in <ref>. This had two advantages: (1) It enabled labelers to create better ground-truth labels, often inferring meaningful information from the subreddit. (2) It allowed us to test LLM inference capabilities in an information-limited setting, assessing their ability to infer attributes from texts without meta information. The labeling procedure took roughly $112$ man-hours. To address potential memorization, we provide an extensive decontamination study of our dataset in <ref>, showing that no memorization besides very few common URLs and quotes occurred. Due to the personal data contained in the dataset, we do not plan to make it public. Instead, we provide 525 human-verified synthetic examples, detailed in <ref>.

§ EVALUATION OF PRIVACY VIOLATING LLM INFERENCES

Accuracies of 9 state-of-the-art LLMs on the PersonalReddit dataset. GPT-4 achieves the highest total top-1 accuracy of $84.6\%$. Note that Human-Labeled* had additional information.

*Free Text Inference on PersonalReddit

In our main experiment, we evaluate the capabilities of $9$ state-of-the-art LLMs at inferring personal author attributes on our PersonalReddit (PR) dataset. We select all attribute labels from PR with a certainty rating of at least 3 (quite certain). This resulted in 1066 (down from 1184) individual labels across all 520 profiles.
Using the prompt template presented <ref>, we then jointly predicted all attributes (per profile). For each attribute, we ask the models for their top 3 guesses in order (presenting all options for categoricals, see <ref>).

We present our main findings in <ref>, showing the total number of correct inferences per model and target attribute. First, we observe that GPT-4 performed the best across all models with a top-1 accuracy of $84.6\%$ across attributes. In <ref>, we show that this number rises to $95.1\%$ when looking at top-3 predictions—almost matching human labels. This is especially remarkable as humans, unlike the models, were (1) able to see corresponding subreddits in which a comment occurred and (2) had unlimited access to traditional search engines. In total, PR contains 51 labels, which humans could only infer using subreddits (e.g., subreddits like /r/Houston)—many of which GPT-4 inferred from text alone.
Further, we can observe a clear trend when comparing model sizes and attribute inference performance. While Llama-2 7B achieves a total accuracy of $51\%$, Llama-2 70B is already at $66\%$. This trend also persists across model families (assuming common estimates of model sizes), a fact especially worrying considering the already strong performance of GPT-4.

*Individual attributes

We further show the individual attribute accuracy of GPT-4 in <ref> (for other models we refer to <ref>). We first observe that each attribute is predicted with an accuracy of at least $60\%$, with gender and place of birth achieving almost $97\%$ and $92\%$, respectively.
GPT-4 shows its lowest performance on ; however, this is also the attribute with the lowest number of samples (only $40$) available. Further, when looking at the top-2 accuracy (given in <ref>), we find a significant jump to $87\%$, indicating that humans and the model are not generally misaligned. For example, we find that GPT-4 prefers predicting "Low Income ($<30k$)" instead of "No income" as the first guess, potentially a result of model alignment. We particularly highlight the $86\%$ accuracy of  predictions, which are in a non-restricted free text format. As we will show in <ref>, this performance remains strong even when removing all direct location references with state-of-the-art anonymizers.

Accuracies for each hardness level for one representative model of each family. We observe a clear decrease in accuracy with increasing hardness scores.

Our last experiment demonstrates that our human-labeled hardness scores and overall model performance are well aligned. In particular, we show in <ref>, for one representative model of each family, their accuracy across each hardness level (we provide full results in <ref>). For all models, we can observe a decrease in accuracy with increasing hardness scores, indicating that models and human labelers generally agree on which examples are harder. We also observe that the decrease from $3$ to $4$ is less clear than for other scores, notably with GPT-4 achieving a higher accuracy on hardness $4$ than $3$. Referring back to <ref>, this can be explained by examples in $4$ often requiring humans to search for additional information (e.g., by mentioning a specific local drink) but not strong reasoning capabilities as in $3$. Therefore, hardness $4$ favors models that can accurately retrieve information across various topics. We will observe a similar behavior on anonymized text in <ref>.

*Adversarial Interaction

Shortened conversation between our bots. We give the full conversation in <ref>.

In <ref>, we have formalized the emerging threat of active adversarial chatbots that inconspicuously steer their conversations to learn private user information. A practical evaluation of this threat with real persons would raise serious ethical concerns. Therefore, we simulated the experiment, demonstrating that it is already possible to build such malicious chatbots.
Similar to popular platforms like CharacterAi [ChAI, 2022], we set the public task $T_p$ to be an engaging conversational partner while now additionally setting $T_h$ to “extract the user's place of living, age, and sex”. In each conversation round, we extracted $r_i^h$ with a summary of what the bot knows, including the reasoning for their next public response $r_i^p$. We show an example of one such round in <ref>. To simulate this interaction, we construct user-bots grounded in a synthetic profile (including age, sex, etc.), as well as real hardness $5$ examples from PublicReddit. User bots are specifically instructed to not reveal any of the private information.
We instantiate all models with GPT-4, running 224 interactions on 20 different user profiles. Across all runs, the adversary achieves a top-1 accuracy of $59.2\%$ (location $60.3\%$, age: $49.6\%$, sex: $67.9\%$). While simulated, these numbers are similar to GPT-4's performance on PersonalReddit, indicating an alignment between our user bot and real data. We include full examples of simulated interactions in <ref>, showing that already now adversarial chatbots are an emerging privacy threat.

§ EVALUATION OF CURRENT MITIGATIONS

To evaluate the effectiveness of current mitigations, we investigate (1) the impact of industry-standard text anonymization procedures on Free Text Inference and (2) the impact of model alignment with respect to privacy-invasive prompts.

*Client-Side Anonymization

Shortcomings of current anonymizers. In 1, direct location references get removed, GPT-4 can still infer the location using information left in the text 2.

We instantiate our text anonymizer with an industry-standard state-of-the-art tool provided by AzureLanguageService [Aahill, 2023]. We deliberately do not use a PII-Remover as such tools commonly remove only highly sensitive plaintext information (e.g., spelled-out banking details). Across our test cases, our anonymizer is a superset of the Azure PII-Remover.
We present an example of an anonymized comment in <ref> alongside a complete overview of all anonymized entities (replaced by '*') in <ref>. Notably, we removed all locations, addresses, persons (and types of persons such as "husband"), organizations, events, dates, ages, numbers, and currencies detected by the tool with a certainty higher than $0.4$. As not all of our attributes were supported by AzureLanguageService, we only evaluate anonymization performance on the ones included, i.e., location, age, occupation, place of birth, and income.

After anonymizing all comments in the PR dataset, we tested GPT-4's inference performance on the anonymized dataset. Showing the corresponding plots in <ref>, we find that while GPT-4's accuracy across all attributes slightly decreases, the decrease is much smaller than one would desire from anonymized text. For instance, the location prediction accuracy drops from ${\sim 86\%}$ to still ${\sim 55\%}$, considerably higher than expected from a text where all mentions of locations have been explicitly removed. We observe similar behaviors for age, income, and place of birth—all of which should also have been removed.

GPT-4 accuracy on anonymized text. Despite removing direct mentions of personal attributes, many can still be inferred accurately.

We next investigate how well anonymization performs across hardness levels. As we can see, current anonymization techniques primarily work on texts that contain personal attributes directly. We observe a $41.1\%$ decrease in accuracy for hardness $1$. However, with increasing hardness, the impact of anonymization drops rapidly from $19\%$ at hardness $2$ to just $7\%$ at hardness 5. As mentioned in <ref>, we see a relative increase in effectiveness at hardness $4$ due to the examples commonly being less reasoning and more lookup-based (e.g., the name of a local event would now be anonymized making a look-up much harder).

Our findings expand on early investigations by Bubeck et al., 2023, which show that GPT-4 outperforms current state-of-the-art tools at PII detection. In particular, we show that personal attributes are often not explicitly stated in real texts but still can be inferred from context not covered by current anonymization tools. Based on this, we see both the need for stronger anonymizers capable of keeping up with LLMs as well as the chance of leveraging the strong natural language understanding of these LLMs to achieve such goals.

*Provider-Side Alignment

At the same time, our experiments show that current models are not aligned against privacy-invasive prompts. This is to be expected as much of the alignment research so far focused primarily on preventing directly harmful and offensive content [Bai et al., 2022, Touvron et al., 2023, OpenAI, 2023].

In <ref> we present the average percentage of rejected prompts, grouped by model-provider. The clear standout with $10.7\%$ of rejected prompts are Google's PALM-2 models—-however, upon closer inspection, a sizeable chunk of rejected prompts were on comments that contained sensitive topics (e.g., domestic violence), which may have triggered another safety filter. As mentioned in <ref>, we believe that improved alignment methods can help mitigate some of the impact of privacy-invasive prompting.

§ CONCLUSION

In this work, we presented the first comprehensive study on the capabilities of pretrained LLMs to infer personal attributes from text. We showed that models already achieve near-human performance on a wide range of personal attributes at only a fraction of the cost and time—making inference-based privacy violations at scale possible for the first time. Further, we showed that currently existing mitigations, such as anonymization and model alignment, are insufficient for appropriately protecting user privacy against automated LLM inference. We hope these findings lead to improvements in both approaches, ultimately resulting in better privacy protections. Additionally, we introduced and formalized the emerging threat of privacy-invasive chatbots. Overall, we believe our findings will open a new discussion around LLM privacy implications that no longer solely focuses on memorizing training data.

§ ETHICS STATEMENT

Before publishing this work, we contacted all model providers ahead of time to make them aware of this issue. Additionally, we ensured that the personal data contained in the PersonalReddit dataset is protected by (1) Not outsourcing the labeling to contract workers and (2) Not publishing the resulting dataset but instead offering the community a set of synthetically created samples on which further research can be conducted non-invasively. All examples shown in the paper are synthetic to protect individuals' privacy. However, we ensured that their core content is closely aligned with samples in PersonalReddit to not be misleading to readers. We are aware that the results indicate that LLMs can be used to automatically profile individuals from large collections of unstructured texts, impacting their personal data and privacy rights. Especially worrisome is the fact that current anonymization methods do not work as well as one would hope in these cases. However, these actions were already possible before this work, and we firmly believe that raising awareness of this issue is a critical first step in mitigating larger privacy impacts.

§ REPRODUCIBILITY

We release all our code and scripts used alongside the work. We do not intend to release the PublicReddit dataset publicly, instead we release a large set of synthetic examples that can be used for further investigations of privacy-invasive LLM inferences. As most tested models are only accessible behind API, ensuring their versioning is partially outside of our control. We provide a full overview of our experimental setup in <ref>.

[noa, 2023]
Training Data Extraction Challenge, September 2023.
URL <https://github.com/google-research/lm-extraction-benchmark>.
original-date: 2022-08-22T06:19:08Z.

[Aahill, 2023]
What is Azure AI Language - Azure AI services, July 2023.

[Anthropic, 2023]
Model-Card-Claude-2.pdf, September 2023.

[Bai et al., 2022]
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion,
Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon,
et al.
Constitutional ai: Harmlessness from ai feedback.
arXiv preprint arXiv:2212.08073, 2022.

[Brown et al., 2022]
Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and
Florian Tramèr.
What does it mean for a language model to preserve privacy?, 2022.
URL https://arxiv. org/abs/2202.05520, 2022.

[Bubeck et al., 2023]
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric
Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg,
Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang.
Sparks of Artificial General Intelligence: Early experiments
with GPT-4, April 2023.
URL <http://arxiv.org/abs/2303.12712>.
arXiv:2303.12712 [cs].

[Carlini et al., 2019]
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song.
The Secret Sharer: Evaluating and Testing Unintended
Memorization in Neural Networks.

[Carlini et al., 2021]
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel
Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar
Erlingsson, Alina Oprea, and Colin Raffel.
Extracting Training Data from Large Language Models, June

[Carlini et al., 2023]
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian
Tramer, and Chiyuan Zhang.
Quantifying Memorization Across Neural Language Models, March

[ChAI, 2022]
https://beta.character.ai/, 2022.

[Ding et al., 2021]
Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt.
Retiring adult: New datasets for fair machine learning.
Advances in neural information processing systems,
34:0 6478–6490, 2021.

[Estival et al., 2007]
Dominique Estival, Tanja Gaustad, Son Bao Pham, Will Radford, and Ben
Author profiling for english emails.
In Proceedings of the 10th Conference of the Pacific
Association for Computational Linguistics, volume 263, pp.  272. Citeseer,

[EU, 2016]
European Union EU.
General data protection regulation, 2016.
URL <https://gdpr-info.eu/>.

[Hegselmann et al., 2023]
Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi
Jiang, and David Sontag.
TabLLM: Few-shot Classification of Tabular Data with
Large Language Models, March 2023.
URL <http://arxiv.org/abs/2210.10723>.
arXiv:2210.10723 [cs].

[15]
The Model Hub.

[Ippolito et al., 2023]
Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew
Jagielski, Katherine Lee, Christopher A. Choquette-Choo, and Nicholas
Preventing Verbatim Memorization in Language Models Gives a
False Sense of Privacy, September 2023.

[Kim et al., 2023]
Siwon Kim, Sangdoo Yun, Hwaran Lee, Martin Gubri, Sungroh Yoon, and Seong Joon
Propile: Probing privacy leakage in large language models.
arXiv preprint arXiv:2307.01881, 2023.

[Klimt & Yang, 2004]
Bryan Klimt and Yiming Yang.
The enron corpus: A new dataset for email classification research.
In European conference on machine learning, pp.  217–226.
Springer, 2004.

[Lakera, 2023]
An Overview of Lakera Guard – Bringing
Enterprise-Grade Security to LLMs with One Line of Code.
https://www.lakera.ai/insights/lakera-guard-overview, 2023.

[Lukas et al., 2023]
Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and
Santiago Zanella-Béguelin.
Analyzing Leakage of Personally Identifiable Information in
Language Models, April 2023.
URL <http://arxiv.org/abs/2302.00539>.
arXiv:2302.00539 [cs].

[OpenAI, 2023]
Gpt-4 technical report.
ArXiv, abs/2303.08774, 2023.

[Poe, 2023]
Poe - fast, helpful ai chat.
https://poe.com/, 2023.

[Rangel et al., 2013]
Francisco Rangel, Paolo Rosso, Moshe Koppel, Efstathios Stamatatos, and Giacomo
Overview of the Author Proﬁling Task at PAN 2013.

[Rangel et al., 2017]
Francisco Rangel, Paolo Rosso, Martin Potthast, and Benno Stein.
Overview of the 5th Author Proﬁling Task at PAN 2017:
Gender and Language Variety Identiﬁcation in Twitter.

[Rangel et al., 2018]
Francisco Rangel, Paolo Rosso, Manuel Montes-y Gómez, Martin Potthast, and
Benno Stein.
Overview of the 6th Author Proﬁling Task at PAN 2018:
Multimodal Gender Identiﬁcation in Twitter.

[Sweeney, 2002]
Latanya Sweeney.
k-anonymity: A model for protecting privacy.
International journal of uncertainty, fuzziness and
knowledge-based systems, 100 (05):0 557–570, 2002.

[Touvron et al., 2023]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine
Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale,
Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem
Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,
Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar
Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa,
Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux,
Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier
Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew
Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan
Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang,
Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan
Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien
Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.
Llama 2: Open Foundation and Fine-Tuned Chat Models, July

[Villegas et al., 2014]
María Paula Villegas, María José Garciarena Ucelay, Marcelo Luis
Errecalde, and Leticia Cagnina.
A spanish text corpus for the author profiling task.
In XX Congreso Argentino de Ciencias de la Computación
(Buenos Aires, 2014), 2014.

[Yang, 2019]
Yuan Yang.
Hundreds of millions of Chinese chat logs leak online, March 2019.

[Zou et al., 2023]
Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson.
Universal and transferable adversarial attacks on aligned language
arXiv preprint arXiv:2307.15043, 2023.