vskip=0pt

The Coleman–Oort conjecture: reduction to three key cases111The main theorem
of this paper is based on notes written in 2011, which were long left
unfinished because I was (and still am) unable to push this result further. My
interest in this was rekindled by an invitation of Rachel Pries and Andrew
Sutherland to talk about the Coleman–Oort conjecture in the VaNTAGe seminar,
for which I should like to thank them.

by

Ben Moonen

Abstract. We show that the Coleman–Oort conjecture can be reduced to three
particular cases. As an application we extend a result of Lu and Zuo, to the
effect that for $g\geq 8$ the Coleman–Oort conjecture is true on the
hyperelliptic locus.

AMS 2010 Mathematics Subject Classification: 11G15, 14G35, 14H40

## Introduction

It has been conjectured by Coleman ([2], Conjecture 6) that for $g$
sufficiently large, there are only finitely many complete nonsingular complex
curves $C$ of genus $g$, up to isomorphism, such that $\mathrm{Jac}(C)$ is a
CM abelian variety. By the André–Oort Conjecture, now a theorem of Tsimerman
[11] (building on contributions of many others), Coleman’s conjecture is
equivalent to the following one.

######  Coleman–Oort Conjecture.

Let $\mathsf{A}_{g}$ be the coarse moduli space of $g$-dimensional principally
polarized complex abelian varieties. If $g$ is sufficiently large, there are
no special subvarieties $S\subset\mathsf{A}_{g}$ of positive dimension that
are contained in the Torelli locus $\mathsf{T}_{g}\subset\mathsf{A}_{g}$ and
that meet the open Torelli locus $\mathsf{T}_{g}^{\circ}$.

In this conjecture we should take $g$ to be at least $8$, as for all smaller
genera there are counterexamples. We refer to [10] for further discussion.

Over the last years, the Coleman–Oort conjecture has attracted a lot of
attention, and many mathematicians have contributed to a better understanding
of it. Our modest goal here is to point out that the conjecture may be reduced
to a couple of key cases. This is inspired by the following result, which
combines Hain’s results in [6] with some refinements due to de Jong and Zhang
[3].

######  Theorem (Hain, de Jong–Zhang).

Let $S\subset\mathsf{A}_{g}$ with $g\geq 4$ be a special subvariety of
positive dimension such that $S\subset\mathsf{T}_{g}$ and $S$ meets
$\mathsf{T}_{g}^{\circ}$. Assume the generic Mumford–Tate group on $S$ has a
$\mathbb{Q}$-simple adjoint group. Then at least one of the following is true:

  1. (H1)

$S$ is a ball quotient,

  2. (H2)

all irreducible components of the intersection of $S$ with
$\mathsf{T}_{g}^{\mathrm{dec}}=\mathsf{T}_{g}\setminus\mathsf{T}_{g}^{\circ}$
have codimension $\leq 2$ in $S$,

  3. (H3)

the Baily–Borel compactification $S\subset S^{*}$ has a non-empty boundary and
at least one irreducible component of the boundary has codimension $\leq 2$ in
$S^{*}$.

Interesting further results were obtained by Andreatta [1].

De Jong and Zhang in their paper remark that it is complicated to list all
Shimura varieties satisfying one of these conditions. However, this problem is
greatly simplified if one observes that for the Coleman–Oort conjecture it
suffices to consider special subvarieties $S\subset\mathsf{A}_{g}$ that are
_minimal_ , in the sense that they do not contain smaller special subvarieties
of positive dimension. Analyzing conditions (H1)–(H3) under this minimality
assumption gives the following result.

######  Main Theorem.

Let $g$ be an integer with $g\geq 8$. Assume there are infinitely many
nonsingular complex curves of genus $g$ whose Jacobians are CM abelian
varieties. Then there exists a special subvariety $S\subset\mathsf{T}_{g}$ of
positive dimension such that $S\cap\mathsf{T}_{g}^{\circ}\neq\emptyset$, and
such that at least one of the following is true:

  1. (1)

$\dim(S)=1$,

  2. (2)

$\dim(S)=2$ and $S$ is complete,

  3. (3)

$S$ is a complete ball quotient.

If $S$ is given by a Shimura datum $(G,X)$ then $S$ is complete if and only if
$\mathrm{rk}_{\mathbb{Q}}(G^{\mathrm{ad}})=0$.

An easy corollary is the following sharpening of the main result of the paper
[9] by Lu and Zuo.

######  Corollary.

For $g\geq 8$, there are finitely many nonsingular complex hyperelliptic
curves $C$ of genus $g$ for which $\mathrm{Jac}(C)$ is a CM abelian variety.

In the final section we show there exist minimal complete ball quotients of
arbitrarily high dimension.

## 1\. Minimal special subvarieties

### 1.1.

By a Shimura datum we mean a pair $(G,X)$ consisting of a reductive
$\mathbb{Q}$-group $G$ and a $G(\mathbb{R})$-conjugacy class of homomorphisms
$\mathbb{S}\to G_{\mathbb{R}}$ that satisfies conditions (2.1.1.1–3) in
Section 2.1 of Deligne’s paper [4]. We always assume that $G$ is the generic
Mumford–Tate group on $X$, i.e., there is no $\mathbb{Q}$-subgroup
$G^{\prime}\subsetneq G$ such that all $h\in X$ factor through
$G^{\prime}_{\mathbb{R}}$.

For $h\in X$, let $M_{h}\subset G$ be its Mumford-Tate group, i.e., the
smallest algebraic subgroup $M\subset G$ (over $\mathbb{Q}$) such that $h$
factors through $M_{\mathbb{R}}$. Let $Y\subset X$ be the
$M_{h}(\mathbb{R})$-orbit of $h$. The pair $(M_{h},Y)$ is again a Shimura
datum, and $Y$ is a finite union of totally geodesic submanifolds of $X$. The
submanifolds of $X$ that arise in this manner are called the _Hodge loci_ in
$X$.

Let $\mathrm{Sh}_{K}(G,X)$ be the complex Shimura variety
$G(\mathbb{Q})\backslash X\times G(\mathbb{A}_{\mathrm{f}})/K$. An irreducible
algebraic subvariety $S\subset\mathrm{Sh}_{K}(G,X)$ is called a _special
subvariety_ if there exists a Hodge locus $Y^{+}\subset X$ and a class $\gamma
K$ in $G(\mathbb{A}_{\mathrm{f}})/K$ such that $S(\mathbb{C})$ is the image of
$Y^{+}\times\\{\gamma K\\}\subset X\times G(\mathbb{A}_{\mathrm{f}})/K$ in
$\mathrm{Sh}_{K}(G,X)$.

If $S\subset\mathrm{Sh}_{K}(G,X)$ is a special subvariety and $[h,\gamma K]$
is a Hodge-generic point in $S$, then up to isomorphism the associated Shimura
datum $(M_{h},Y)$ only depends on $S$. This justifies calling $(M_{h},Y)$
_the_ Shimura datum corresponding to $S$.

######  Definition 1.2.

  1. (i)

Let $S$ be a special subvariety of a Shimura variety $\mathrm{Sh}_{K}(G,X)$.
We say that $S$ is _minimal_ if $\dim(S)>0$ and $S$ contains no special
subvarieties $S^{\prime}\subsetneq S$ of positive dimension.

  2. (ii)

We say that a Shimura datum $(G,X)$ is _minimal_ if $\dim_{\mathbb{C}}(X)>0$,
and if for every $h\in X$ its $M_{h}(\mathbb{R})$-orbit is either a point or
all of $X$.

######  Remarks 1.3.

  1. (i)

If $S\subset\mathrm{Sh}_{K}(G,X)$ is a special subvariety and $(M,Y)$ is the
corresponding Shimura datum, $S$ is minimal if and only if $(M,Y)$ is.

  2. (ii)

A Shimura datum $(G,X)$ is minimal if and only if the corresponding adjoint
datum $(G^{\mathrm{ad}},X^{\mathrm{ad}})$ is minimal.

######  Lemma 1.4.

If an adjoint Shimura datum $(G^{\mathrm{ad}},X^{\mathrm{ad}})$ is minimal,
$G^{\mathrm{ad}}$ is a $\mathbb{Q}$-simple group.

###### Proof.

Suppose not. Then there is a decomposition of Shimura data
$(G^{\mathrm{ad}},X^{\mathrm{ad}})=(H_{1},Y_{1})\times(H_{2},Y_{2})$ with
$Y_{1}$ and $Y_{2}$ of positive dimension. Choose a special point $y_{2}\in
Y_{2}$ and a Hodge-generic $y_{1}\in Y_{1}$. The $M_{h}(\mathbb{R})$-orbit of
$(y_{1},y_{2})$ is $Y_{1}\times\\{y_{2}\\}$, but this contradicts the
minimality of $(G,X)$. ∎

### 1.5.

Suppose $(G^{\mathrm{ad}},X^{\mathrm{ad}})$ is an adjoint Shimura datum such
that $G^{\mathrm{ad}}$ is $\mathbb{Q}$-simple. As explained in [4], Section
2.3.4, we then have $G^{\mathrm{ad}}=\mathrm{Res}_{E/\mathbb{Q}}\;H$ where $E$
is a totally real number field and $H$ is an absolutely simple adjoint group
over $E$. The set $\Sigma$ of real embeddings of $E$ decomposes as
$\Sigma=\Sigma_{\mathrm{c}}\coprod\Sigma_{\mathrm{nc}}$ where
$\Sigma_{\mathrm{c}}$ (resp. $\Sigma_{\mathrm{nc}}$) is the set of embeddings
$\sigma\colon E\to\mathbb{R}$ for which
$H_{\sigma}:=H\otimes_{E,\sigma}\mathbb{R}$ is compact (resp. noncompact). The
$\mathbb{Q}$-rank of $G^{\mathrm{ad}}$ is equal to the $E$-rank of $H$.

######  Proposition 1.6.

Let $S\subset\mathsf{A}_{g}$ be a minimal special subvariety such that either
$\dim(S)=2$, or $S$ is a ball quotient with $\dim(S)>1$, or $S$ is a quotient
of the $3$-dimensional hermitian symmetric space of type $\mathrm{BD\,I}$.
Then $S$ is complete.

The argument in fact works in many more cases but these are the only cases we
need.

###### Proof.

Suppose $S$ is not complete. Let $(G,X)$ be corresponding Shimura datum. By
Lemma 1.4, $G^{\mathrm{ad}}$ is $\mathbb{Q}$-simple. With notation
$G^{\mathrm{ad}}=\mathrm{Res}_{E/\mathbb{Q}}\;H$ as above, the assumption that
$S$ is not complete means that
$\mathrm{rk}_{\mathbb{Q}}(G^{\mathrm{ad}})=\mathrm{rk}_{E}(H)$ is positive.
Therefore, all factors $H_{\sigma}$ are non-compact.

First assume $S$ is a ball quotient. Then $G^{\mathrm{ad}}_{\mathbb{R}}$ has
only one noncompact factor, so we must have $E=\mathbb{Q}$ and
$G^{\mathrm{ad}}_{\mathbb{R}}\cong\mathrm{PGU}(n,1)$ with $n=\dim(S)$. There
then exist:

  * •

a division algebra $\Delta$ with an involution $x\mapsto\bar{x}$ of the second
kind, such that the centre $K=Z(D)$ is an imaginary quadratic field,

  * •

a right $\Delta$-module $V$ of finite type,

  * •

a hermitian form $\phi\colon V\times V\to\Delta$ with respect to the given
involution,

such that $G^{\mathrm{ad}}\cong\mathrm{PGU}(V,\phi)$. Because
$G^{\mathrm{ad}}$ has positive rank, the Witt index of $\phi$ is positive. The
rank of $G^{\mathrm{ad}}_{\mathbb{R}}$ is then at least the degree of
$\Delta$. But we know that $\mathrm{rk}(G^{\mathrm{ad}}_{\mathbb{R}})=1$, so
it follows that $\Delta=K$ is an imaginary quadratic field. The domain
$X^{\mathrm{ad}}$ can be identified with the space of all negative lines in
the $\mathbb{C}$-vector space $V\otimes_{\mathbb{Q}}\mathbb{R}$ with respect
to $\phi$. The assertion now follows from the remark that this Shimura datum
$(G^{\mathrm{ad}},X^{\mathrm{ad}})$ is minimal only if $n=1$. Indeed, if $n>1$
we can choose an $n$-dimensional $K$-subspace $W\subsetneq V$ such that
$\phi|_{W}$ is nondegenerate and again has Witt index $1$. If $L\subset
W\otimes_{\mathbb{Q}}\mathbb{R}$ is a very general negative line and $M_{h}$
is the Mumford–Tate group of the corresponding point $h\in X^{\mathrm{ad}}$,
the $M_{h}(\mathbb{R})$-orbit of $h$ has dimension $n-1$, in contradiction
with the minimality of $S$.

The case where $S$ is a quotient of the $3$-dimensional space of type
$\mathrm{BD\,I}$ is similar. The assumption that $S$ is not complete implies
that $E=\mathbb{Q}$ and $G^{\mathrm{ad}}$ is absolutely imple. In this case,
$G^{\mathrm{ad}}=\mathrm{SO}(V,\phi)$ where $V$ is a $5$-dimensional
$\mathbb{Q}$-vector space and $\phi$ is a symmetric bilinear form of Witt
index $>0$ (as otherwise $S$ would be complete), of signature $(1,4)$ or
$(4,1)$. Further, $X^{\mathrm{ad}}$ is biholomorphic to the $3$-dimensional
complex manifold of isotropic lines $\mathbb{C}\cdot v\subset V_{\mathbb{C}}$
such that $\phi(v,\bar{v})\neq 0$. Choose a $3$-dimensional subspace $W\subset
V$ such that $\phi$ is indefinite on $W_{\mathbb{R}}$. If $L\subset
W_{\mathbb{C}}$ is a very general isotropic line, the Mumford–Tate group
$M_{h}$ of the corresponding point $h\in X^{\mathrm{ad}}$ is isomorphic to
$\mathrm{SO}(W,\phi|_{W})$, and the $M_{h}(\mathbb{R})$-orbit of $h$ is
$1$-dimensional. By minimality of $S$ this case is therefore excluded.

Finally, suppose $\dim(S)=2$. As before, we have
$G^{\mathrm{ad}}=\mathrm{Res}_{E/\mathbb{Q}}\;H$ and if $S$ is not complete,
all factors $H_{\sigma}$ are noncompact. The case where $S$ is a ball quotient
has already been dealt with. The only other possibility is that
$[E:\mathbb{Q}]=2$ and that $H$ is of Lie type $A_{1}$. Because
$\mathrm{rk}_{E}(H)>0$ we then have $H=\mathrm{PGL}_{2,E}$ and
$(G^{\mathrm{ad}},X^{\mathrm{ad}})$ is (the adjoint datum of) the Hilbert
modular Shimura datum in genus $2$, which is well-known not to be minimal. ∎

## 2\. Special subvarieties with large Baily–Borel boundary

### 2.1.

As in [4], section 1.2, we consider pairs $(H,Y)$ consisting of a simple
adjoint group $H$ over $\mathbb{R}$ and an $H(\mathbb{R})$-conjugacy class of
non-trivial homomorphisms $\mathbb{S}\to H$ such that the adjoint
representation gives $\mathrm{Lie}(H)$ a Hodge structure of type
$(-1,1)+(0,0)+(1,-1)$, and such that $\mathrm{Inn}\bigl{(}h(i)\bigr{)}$ is a
Cartan involution.

As explained in loc. cit., $Y$ corresponds to a special vertex in the Dynkin
diagram of $H_{\mathbb{C}}$. Below we list, for $H$ of Lie type A–D, the
possibilities for the Satake diagram of $H$ and the special vertex (circled).
This merges the information contained in [4], Table 1.3.9, with the
classification results in [7], Chapter 10.

The proper boundary components of $Y$ in its Baily-Borel compactification
$Y\subset Y^{*}$ are given by the maximal $\mathbb{R}$-parabolic subgroups
$P\subset H$. The conjugacy class of $P$ corresponds to a
$\mathrm{Gal}(\mathbb{C}/\mathbb{R})$-orbit $I_{P}$ of white vertices in the
Satake diagram $\Delta$. If $I_{P}$ contains the special vertex, the boundary
component $Y_{P}\subset Y^{*}$ corresponding to the conjugacy class of $P$ is
a finite set of points. Otherwise, consider the connected component of
$\Delta\setminus I_{P}$ that contains the special vertex; the dimension of
$Y_{P}$ is then the dimension (as a complex manifold) of the corresponding
hermitian symmetric space. We list those cases in which $Y_{P}$ has
codimension at most $2$ in $Y^{*}$.

Type $\bm{\mathrm{A}_{\ell}(p,q)}$; $\bm{\ell=p+q-1\geq 1}$. Special vertex:
$p$ or $q$; $\dim(Y)=pq$.

$\cdots$$\cdots$$\vdots$$\scriptstyle 1$$\scriptstyle 2$$\scriptstyle
p$$\scriptstyle p+1$$\scriptstyle\ell$$\scriptstyle\ell-1$$\scriptstyle
q$$\scriptstyle q-1$$1\leq p<q\leq\ell$$\cdots$$\cdots$$\scriptstyle
1$$\scriptstyle 2$$\scriptstyle p-1$$\scriptstyle
p$$\scriptstyle\ell$$\scriptstyle\ell-1$$\scriptstyle p+1$$p=q$, $\ell=2p-1$

The $\mathrm{Gal}(\mathbb{C}/\mathbb{R})$-orbit $\\{i,\ell+1-i\\}$ for $1\leq
i\leq p$ gives a boundary component of type $\mathrm{A}_{\ell-2i}(p-i,q-i)$.
For $p=q$ we additionally have the Galois-orbit $\\{p\\}$, which gives a
$0$-dimensional boundary component.

Boundary components of codimension $1$: only for $p=q=1$, in which case
$\dim(Y)=1$ and the boundary component is $0$-dimensional.

Boundary components of codimension $2$: only for $p=1$ and $q=2$, in which
case $\dim(Y)=2$ and we have a $0$-dimensional boundary component.

Type $\bm{\mathrm{B}_{\ell}}$; $\bm{\ell\geq 2}$. Special vertex: $1$; trivial
$\mathrm{Gal}(\mathbb{C}/\mathbb{R})$-action; $\dim(Y)=2\ell-1$.

$\scriptstyle 1$$\scriptstyle 2$$\ell=2$$\scriptstyle 1$$\scriptstyle
2$$\scriptstyle 3$$\ell=3$$\cdots$$\scriptstyle 1$$\scriptstyle
2$$\scriptstyle
3$$\scriptstyle\ell-2$$\scriptstyle\ell-1$$\scriptstyle\ell$$\ell\geq 4$

Boundary components of codimension $2$: only for $\ell=2$, in which case
$\dim(Y)=3$ and we have a $1$-dimensional boundary component corresponding to
the Galois-orbit $\\{2\\}$.

Type $\bm{\mathrm{C}_{\ell}}$; $\bm{\ell\geq 3}$. Special vertex: $\ell$;
trivial $\mathrm{Gal}(\mathbb{C}/\mathbb{R})$-action;
$\dim(Y)=\ell(\ell+1)/2$.

$\cdots$$\scriptstyle 1$$\scriptstyle
2$$\scriptstyle\ell-2$$\scriptstyle\ell-1$$\scriptstyle\ell$

There are no cases that give a boundary component of codimension $\leq 2$.

Type $\bm{\mathrm{D}_{\ell}^{\mathbb{R}}}$; $\bm{\ell\geq 4}$. Special vertex:
$1$; trivial $\mathrm{Gal}(\mathbb{C}/\mathbb{R})$-action; $\dim(Y)=2\ell-2$.

$\cdots$$\scriptstyle 1$$\scriptstyle 2$$\scriptstyle
3$$\scriptstyle\ell-3$$\scriptstyle\ell-2$$\scriptstyle\ell-1$$\scriptstyle\ell$

There are no cases that give a boundary component of codimension $\leq 2$.

Type $\bm{\mathrm{D}_{\ell}^{\mathbb{H}}}$; $\bm{\ell\geq 5}$. Special vertex:
$\ell$; $\dim(Y)=\ell(\ell-1)/2$.

$\cdots$$\scriptstyle 1$$\scriptstyle 2$$\scriptstyle
3$$\scriptstyle\ell-3$$\scriptstyle\ell-2$$\scriptstyle\ell-1$$\scriptstyle\ell$$\ell$
odd$\cdots$$\scriptstyle 1$$\scriptstyle 2$$\scriptstyle
3$$\scriptstyle\ell-3$$\scriptstyle\ell-2$$\scriptstyle\ell-1$$\scriptstyle\ell$$\ell$
even

There are no cases that give a boundary component of codimension $\leq 2$.

### 2.2.

Let $(G,X)$ be an adjoint Shimura datum of abelian type such that the group
$G$ is $\mathbb{Q}$-simple. As in 1.5 we have
$G=\mathrm{Res}_{E/\mathbb{Q}}\,H$ for some totally real number field $E$ and
an absolutely simple adjoint group $H$ over $E$. The assumption that $(G,X)$
is of abelian type implies that $H$ is of Lie type A, B, C or D. Let
$\Sigma=\Sigma_{\mathrm{c}}\coprod\Sigma_{\mathrm{nc}}$ be the set of real
embeddings of $E$. Then $X=\prod_{\sigma\in\Sigma_{\mathrm{nc}}}\,Y_{\sigma}$,
where $Y_{\sigma}$ is an $H_{\sigma}(\mathbb{R})$-conjugacy class of
homomorphisms $\mathbb{S}\to H_{\sigma}$.

Suppose $\mathrm{rk}_{\mathbb{Q}}(G)=\mathrm{rk}_{E}(H)$ is positive. All
factors $H_{\sigma}$ are then non-compact and $\dim(Y_{\sigma})>0$ for every
$\sigma\in\Sigma$. Each pair $(H_{\sigma},Y_{\sigma})$ is of the type
considered in 2.1. Note that all $(H_{\sigma},Y_{\sigma})$ are of the same
type $\mathrm{T}_{\ell}$ with
$\mathrm{T}\in\\{\mathrm{A},\mathrm{B},\mathrm{C},\mathrm{D}\\}$ and $\ell\geq
1$, but that for type $\mathrm{A}_{\ell}$ the parameters $(p,q)$ may depend on
$\sigma$. For $K\subset G(\mathbb{A}_{\mathrm{f}})$ a compact open subgroup,
the boundary components of $\mathrm{Sh}_{K}(G,X)$ in the Baily-Borel
compactification are given by the maximal $\mathbb{Q}$-parabolic subgroups of
$G$. Any such is of the form $\mathrm{Res}_{E/\mathbb{Q}}\,P$ with $P\subset
H$ a maximal $E$-parabolic subgroup. For each $\sigma\in\Sigma$ the parabolic
subgroup $P_{\sigma}\subset H_{\sigma}$ gives rise to a boundary component
$Y_{P,\sigma}\subset Y_{\sigma}^{*}$, and the rational boundary component of
the Baily-Borel compactification $X\subset X^{*}$ that corresponds to $P$ is
$\prod_{\sigma\in\Sigma}\,Y_{P,\sigma}$. Inspecting the tables in 2.1, we find
that the Baily-Borel compactification of $X$ (equivalently: of
$\mathrm{Sh}_{K}(G,X)$) can have a non-empty proper boundary component of
codimension $\leq 2$ only in the following cases:

  1. (B1)

$[E:\mathbb{Q}]\leq 2$ and $H=\mathrm{PGL}_{2,F}$;

  2. (B2)

$E=\mathbb{Q}$ and $H=G$ is of type $\mathrm{A}_{2}(1,2)$;

  3. (B3)

$E=\mathbb{Q}$ and $H=G$ is of type $\mathrm{B}_{2}$.

## 3\. Proofs of the main results

### 3.1.

Let $\mathsf{T}_{g}\subset\mathsf{A}_{g}$ be the Torelli locus. Let
$\mathsf{T}_{g}^{\circ}\subset\mathsf{T}_{g}$ be the open Torelli locus, i.e.,
the locus of Jacobians of smooth curves. The complement
$\mathsf{T}_{g}^{\mathrm{dec}}\subset\mathsf{T}_{g}$ is the locus of
decomposable Jacobians. We are interested in closed irreducible subvarieties
$S\subset\mathsf{A}_{g}$ that satisfy the following condition:

(ST)

$S$ is a special subvariety of positive dimension, with
$S\subset\mathsf{T}_{g}$ and $S\cap\mathsf{T}_{g}^{\circ}\neq\emptyset$

(ST for Special subvariety in the Torelli locus.)

######  Lemma 3.2.

Let $S\subset\mathsf{A}_{g}$ be a special subvariety that satisfies (ST) and
such there is no $S^{\prime}\subsetneq S$ that again satisfies (ST). Then $S$
is minimal.

###### Proof.

If $S$ is not minimal, there exists a special subvariety $S^{\prime}\subsetneq
S$ of positive dimension. Every Hecke translate of $S^{\prime}$ is again a
special subvariety of $S$. These Hecke translates lie dense in $S$, so there
certainly exists one that is not fully contained in the boundary of
$\mathsf{T}_{g}$; but this contradicts the assumption. ∎

### 3.3.

_Proof of the Main Theorem._ Suppose there exist infinitely many curves
(complete, nonsingular) of genus $g$ whose Jacobians are CM abelian varieties.
By Tsimerman’s theorem (formerly the André–Oort conjecture), there then exists
a special subvariety $S\subset\mathsf{A}_{g}$ that satisfies condition (ST).
By Lemma 3.2 we may assume $S$ is minimal. As $G^{\mathrm{ad}}$ is then
$\mathbb{Q}$-simple, at least one of the conditions (H1)–(H3) in the theorem
of Hain and de Jong–Zhang holds. (See the introduction.) We check what happens
in these cases:

Case (H1): By Proposition 1.6, either $\dim(S)=1$ or $S$ is a complete ball
quotient, so either (1) or (3) in the theorem holds.

Case (H2): The irreducible components of $S\cap\mathsf{T}_{g}^{\mathrm{dec}}$
are again special subvarieties, so by minimality of $S$ these are points;
hence $\dim(S)\leq 2$. By Proposition 1.6, either (1) or (2) in the theorem
holds.

Case (H3): This means that $S$ is not complete and we are in one of the cases
(B1), (B2) or (B3) at the end of the previous section. In case (B1) we have
$\dim(S)\leq 2$, whereas in case (B2) $S$ is a ball quotient; these cases have
already been covered. Case (B3), finally, is excluded by Proposition 1.6. ∎

### 3.4.

_Proof of the Corollary._ Suppose there are infinitely many hyperelliptic
curves of genus $g$ with a CM Jacobian. This gives us a special subvariety
$S\subset\mathsf{A}_{g}$ satisfying (ST) such that $S$ is contained in the
closure of the hyperelliptic locus
$\mathsf{HT}_{g}^{\circ}\subset\mathsf{T}_{g}^{\circ}$. We may assume $S$ to
be minimal, in which case it is of one of the three types (1)–(3) in the Main
Theorem. First assume $S$ is complete. Because $\mathsf{HT}_{g}^{\circ}$ is
affine, $S^{\mathrm{dec}}=S\cap\mathsf{T}_{g}^{\mathrm{dec}}$ has codimension
$1$ in $S$. But $S^{\mathrm{dec}}$ is again a special subvariety, so by
minimality we conclude that $\dim(S)=1$. Now use that the case $\dim(S)=1$ has
been excluded by Lu and Zuo in their paper [9]. ∎

## 4\. The existence of minimal complete ball quotients

As a complex ball (equivalently: a complex hyperbolic space) has totally
geodesic complex submanifolds in every codimension, it is tempting to guess
that ball quotients of dimension $>1$ are not minimal. We here show that this
is not true: there exist minimal complete ball quotients of arbitrarily large
dimension.

### 4.1.

Let $p$ be a prime number. Let $K$ be an imaginary quadratic field, and let
$\Delta$ be a division algebra of degree $p$ over $K$, equipped with an
involution $\tau$ of the second kind. We choose these data such that there
exists an isomorphism
$\alpha\colon\Delta\otimes_{\mathbb{Q}}\mathbb{R}\xrightarrow{\leavevmode\nobreak\
\sim\leavevmode\nobreak\ }M_{p}(\mathbb{C})$ via which $\tau$ corresponds to
the adjoint involution of the hermitian form
$x_{1}\bar{y}_{1}+\cdots+x_{p-1}\bar{y}_{p-1}-x_{p}\bar{y}_{p}$. Let
$G=\mathrm{GU}(\Delta,\tau)$ be the group of unitary similitudes; see for
instance [8], §23. The isomorphism $\alpha$ induces an isomorphism
$G_{\mathbb{R}}\cong\mathrm{GU}(p-1,1)$, via which we identify the two groups.
Let $h\colon\mathbb{S}\to G_{\mathbb{R}}$ be the homomorphism that sends
$z\in\mathbb{C}^{*}=\mathbb{S}(\mathbb{R})$ to the diagonal matrix
$\mathrm{diag}(z,\ldots,z,\bar{z})$, and let
$X\subset\mathrm{Hom}(\mathbb{S},G_{\mathbb{R}})$ be the
$G(\mathbb{R})$-conjugacy class of $h$. The pair $(G,X)$ is a Shimura datum of
Hodge type. We identify $X$ with the complex hyperbolic space
$\mathbf{H}^{p-1}_{\mathbb{C}}\subset\mathbb{P}^{p-1}(\mathbb{C})$ through the
map that sends ${}^{g}h$, for $g\in G(\mathbb{R})$, to the negative line in
$\mathbb{C}^{p}$ spanned by the vector $g\cdot(0,\ldots,0,1)$. If
$S\subset\mathsf{A}_{g}$ is a special subvariety with Shimura datum isomorphic
to $(G,X)$ then $S$ is complete, as
$\mathrm{rk}_{\mathbb{Q}}(G^{\mathrm{ad}})=0$.

######  Proposition 4.2.

The Shimura datum $(G,X)$ is minimal.

###### Proof.

Suppose $(G,X)$ is not minimal, which means that there exists a sub-Shimura
datum $(G^{\prime},X^{\prime})\subsetneq(G,X)$ with $\dim(X^{\prime})>0$. Let
$X^{\prime,+}$ be a connected component of $X^{\prime}$, which is a totally
geodesic complex submanifold of $X$. There is a unique linear subspace
$H\subset\mathbb{C}^{p}$ with $1<\dim(H)<p$ such that
$X^{\prime,+}=X\cap\mathbb{P}(H)$; see [5], Section 3.1.11. Let
$G^{\prime}(\mathbb{R})^{+}$ be the (topological) identity component of
$G^{\prime}(\mathbb{R})$. The action of an element $g\in
G^{\prime}(\mathbb{R})^{+}\subset G(\mathbb{R})$ on $X$ maps
$X^{\prime,+}\subset X$ into itself.

The algebra $\Delta\otimes_{\mathbb{Q}}\mathbb{R}$ acts on $\mathbb{C}^{p}$.
Let $\mathrm{Stab}(H)\subset(\Delta\otimes_{\mathbb{Q}}\mathbb{R})$ be the
stabilizer of the subspace $H$. We have
$G(\mathbb{R})\subset(\Delta\otimes_{\mathbb{Q}}\mathbb{R})^{*}$ and this
restricts to an inclusion
$G^{\prime}(\mathbb{R})^{+}\subset\mathrm{Stab}(H)^{*}$. Because $G^{\prime}$
is not abelian, we can choose two elements $g_{1}$, $g_{2}\in
G^{\prime}(\mathbb{Q})^{+}$ that do not commute. Let
$K(g_{1},g_{2})\subset\Delta$ be the $K$-subalgebra generated by these
elements. Because the degree of $\Delta$ is prime we have
$K(g_{1},g_{2})=\Delta$. On the other hand,
$K(g_{1},g_{2})\subset\Delta\cap\mathrm{Stab}(H)$, where the intersection is
take inside $\Delta\otimes_{\mathbb{Q}}\mathbb{R}$. It follows that
$\Delta\otimes_{\mathbb{Q}}\mathbb{R}=\mathrm{Stab}(H)$, which is clearly
impossible. ∎

## References

  * [1] F. Andreatta, Coleman-Oort’s conjecture for degenerate irreducible curves. Israel J. Math. 187 (2012), 231–285.
  * [2] R. Coleman, Torsion points on curves. In: Galois representations and arithmetic algebraic geometry (Kyoto, 1985/Tokyo, 1986), Adv. Stud. Pure Math. 12, North-Holland, 1987; pp. 235–247.
  * [3] A.J. de Jong, S-W. Zhang, Generic abelian varieties with real multiplication are not Jacobians. In: Diophantine geometry, CRM Series 4, Ed. Norm., 2007; pp. 165–172.
  * [4] P. Deligne, Variétés de Shimura: interprétation modulaire, et techniques de construction de modèles canoniques. In: Automorphic forms, representations and $L$-functions (Corvallis, 1977), Proc. Sympos. Pure Math. 33, AMS, 1979; Part 2, pp. 247–289.
  * [5] W. Goldman, Complex hyperbolic geometry. Oxford Mathematical Monographs. The Clarendon Press, 1999.
  * [6] R. Hain, Locally symmetric families of curves and Jacobians. In: Moduli of curves and abelian varieties, Aspects Math. E33, Vieweg, 1999; pp. 91–108.
  * [7] S. Helgason, Differential geometry, Lie groups, and symmetric spaces. Pure and Applied Math. 80, Academic Press, 1978.
  * [8] M-A. Knus et al., The book of involutions. AMS Colloquium Publ. 44, AMS, 1998.
  * [9] X. Lu, K. Zuo, The Oort conjecture on Shimura curves in the Torelli locus of hyperelliptic curves. J. Math. Pures Appl. (9) 108 (2017), no. 4, 532–552.
  * [10] B. Moonen, F. Oort, The Torelli locus and special subvarieties. In: Handbook of moduli. Adv. Lect. Math. 25, Int. Press, 2013; Vol. II; pp. 549–594,
  * [11] J. Tsimerman, The André–Oort conjecture for $\mathcal{A}_{g}$. Ann. of Math. (2) 187 (2018), no. 2, 379–390.

<EMAIL_ADDRESS>
Radboud University Nijmegen, IMAPP, Nijmegen, The Netherlands


# Three term rational function progressions in finite fields

Guo-Dong Hong  Department of Mathematics, California Institute of Technology,
Pasadena, CA 91125, USA<EMAIL_ADDRESS>and  Zi Li Lim  Department of
Mathematics, UCLA, Los Angeles, CA 90095, USA<EMAIL_ADDRESS>
###### Abstract.

Let $F(t),G(t)\in\mathbb{Q}(t)$ be rational functions such that $F(t),G(t)$
and the constant function $1$ are linearly independent over $\mathbb{Q}$, we
prove an asymptotic formula for the number of the three term rational function
progressions of the form $x,x+F(y),x+G(y)$ in subsets of $\mathbb{F}_{p}$. The
main new ingredient is an algebraic geometry version of PET induction that
bypasses Weyl’s differencing. This answers a question of Bourgain and Chang.

##  1\. Introduction

###  1.1. Introduction

Finding patterns in a dense set is a central theme of number theory and
combinatorics. Let $\mathcal{A}$ be a discrete interval $[N]=\\{1,2,...,N\\}$
or a finite field, and $P_{1},P_{2},...,P_{m}\in\mathbb{Z}[t]$ be polynomials,
define $r_{P_{1},P_{2},...,P_{m}}(\mathcal{A})$ to be the size of the largest
subset of $\mathcal{A}$ that does not contain any nontrivial progression of
the form

(1) $x,x+P_{1}(y),x+P_{2}(y),...,x+P_{m}(y).$

Szemerédi [25] proved that any subset of the integers of positive upper
density contains an arbitrarily long nontrivial arithmetic progression, or
equivalently,

(2) $r_{t,2t,...,(k-1)t}([N])=o_{k}(N)$

for all $k\geq 2$ via combinatorial arguments.

Bergelson and Leibman [1] generalized Szemerédi’s theorem to polynomial
progressions. Let $P_{1},P_{2},...,P_{m}\in\mathbb{Z}[t]$ be polynomials with
zero constant terms, Bergelson and Leibman proved that

(3) $r_{P_{1},P_{2},...P_{m}}([N])=o_{P_{1},P_{2},...,P_{m}}(N)$

via ergodic theory. The zero constant term assumption is a natural and
convenient condition to avoid local obstructions.

Both Szemerédi’s and Bergelson-Leibman’s proofs are qualitative in nature. For
quantitative bounds, before Szemerédi, Roth [23] already showed that

(4) $r_{t,2t}([N])=O\Bigl{(}\frac{N}{\log\log N}\Bigr{)}$

via Fourier analysis, but his arguments do not work for finding longer
arithmetic progressions. In Gowers’ celebrated papers MR1844079,MR1631259, he
obtained quantitative bounds for the sets that do not contain arbitrarily long
arithmetic progressions, more precisely, for all $k\geq 2$, there exists a
constant $c_{k}>0$ such that

(5) $r_{t,2t,...,(k-1)t}([N])=O_{k}\Bigl{(}\frac{N}{(\log\log
N)^{c_{k}}}\Bigr{)}$

via introducing higher order Fourier analysis. For quantitative bounds for
general polynomial progressions, the general theory is not complete yet, but
see Prendiville’s [22], Peluse and Prendiville’s [20], Peluse’s [15], Peluse
and Prendiville’s [19], Bloom and Maynard’s [2] and Peluse, Sah and Sawhney’s
[21] results for current state of the art.

Now, we turn our attention to finding progressions in the finite fields, where
new phenomena emerge. Here and throughout, let $p$ be a prime and
$\mathbb{F}_{p}$ be the finite field with $p$ elements. One has the bound
$r_{P_{1},P_{2},...,P_{m}}(\mathbb{F}_{p})\leq r_{P_{1},P_{2},...,P_{m}}([p])$
a priori, but a much stronger bound is valid in many cases. The case of two
term polynomial progressions in finite fields is more straightforward and can
be handled by a Fourier analytic calculation and Weil’s bound. Hence, the
first nontrivial polynomial progression in finite fields is the three term
progression $x,x+y,x+y^{2}$. Bourgain and Chang [4] proved that
$r_{t,t^{2}}(\mathbb{F}_{p})\ll p^{1-1/15}$. In fact, Bourgain and Chang
proved an asymptotic formula: if $A$ is a subset of $\mathbb{F}_{p}$, then

(6) $\\#\\{(x,y)\in\mathbb{F}_{p}^{2}:x,x+y,x+y^{2}\in
A\\}=\frac{|A|^{3}}{p}+O\bigl{(}p^{2/5}|A|^{3/2}\bigr{)}.$

In the same paper, Bourgain and Chang posed the following three questions:

  1. (1)

Let $P_{1},P_{2}\in\mathbb{Z}[t]$ be linearly independent polynomials with
zero constant terms, is an asymptotic formula for the number of the
progressions $x,x+P_{1}(y),x+P_{2}(y)$ in subsets of $\mathbb{F}_{p}$ valid?

  2. (2)

More generally, let $P_{1},P_{2},...,P_{m}\in\mathbb{Z}[t]$ be linearly
independent polynomials with zero constant terms, is an asymptotic formula for
the number of the progressions $x,x+P_{1}(y),x+P_{2}(y),...,x+P_{m}(y)$ in
subsets of $\mathbb{F}_{p}$ valid?

  3. (3)

In a different direction, can one replace the polynomials $P_{1},P_{2}$ in the
first question $(1)$ by rational functions?

The first question $(1)$ is answered by Peluse [18] and independently by Dong,
Li and Sawin [5]. Peluse’s theorem states that, let
$P_{1},P_{2}\in\mathbb{Z}[t]$ be polynomials with zero constant terms such
that $P_{1},P_{2}$ are linearly independent over $\mathbb{Q}$, then

(7) $\\#\\{(x,y)\in\mathbb{F}_{p}^{2}:x,x+P_{1}(y),x+P_{2}(y)\in
A\\}=\frac{|A|^{3}}{p}+O_{P_{1},P_{2}}\bigl{(}p^{7/16}|A|^{3/2}\bigr{)}.$

for all subsets $A$ of $\mathbb{F}_{p}$. This implies the bound
$r_{P_{1},P_{2}}(\mathbb{F}_{p})\ll_{P_{1},P_{2}}p^{1-1/24}$. Independently,
Dong, Li and Sawin proved a similar result with the error term
$O_{P_{1},P_{2}}\bigl{(}p^{3/8}|A|^{3/2}\bigr{)}$, implying
$r_{P_{1},P_{2}}(\mathbb{F}_{p})\ll_{P_{1},P_{2}}p^{1-1/12}$. As a side note,
the linear independence assumption for $P_{1},P_{2}$ is crucial, since one
cannot hope for power saving bounds for progressions involving linearly
dependent polynomials, even the simplest case $x,x+y,x+2y$. Both Peluse’s and
Dong-Li-Sawin’s methods applied deep results from algebraic geometry; Peluse’s
method used Deligne’s bound while Dong-Li-Sawin’s method used the Deligne-Katz
theory.

The second question $(2)$ is also answered by Peluse [17]. By introducing
degree lowering method, Peluse proved that, let
$P_{1},P_{2},...,P_{m}\in\mathbb{Z}[t]$ be polynomial with zero constant terms
such that $P_{1},P_{2},...,P_{m}$ are linearly independent over $\mathbb{Q}$,
there exists a constant $\gamma>0$ depending on $P_{1},P_{2},...,P_{m}$ such
that

(8)
$\\#\\{(x,y)\in\mathbb{F}_{p}^{2}:x,x+P_{1}(y),x+P_{2}(y),...,x+P_{m}(y)\in
A\\}=\frac{|A|^{m+1}}{p^{m-1}}+O_{P_{1},P_{2},...,P_{m}}\bigl{(}p^{2-(m+1)\gamma}\bigr{)}$

for all subsets $A$ of $\mathbb{F}_{p}$. This implies the bound
$r_{P_{1},P_{2},...,P_{m}}(\mathbb{F}_{p})\ll_{P_{1},P_{2},...,P_{m}}p^{1-\gamma}$.
The degree lowering method also provides a great simplification of the
solution to the first question $(1)$ of Bourgain and Chang. Moreover, the
degree lowering method only relies on Weil’s bound for curves, as opposed to
the heavy algebraic geometry machinery utilized in the methods
MR3874848,MR4179774 for the three term polynomial progressions.

For the third question $(3)$ of Bourgain and Chang, replacing the polynomials
by rational functions presents new phenomena and difficulties. Unlike the case
of polynomials, there is no obvious analogue of rational function progressions
in integer setting, hence there is no any general bound a priori, not even a
qualitative one. The only known example is the special progression
$x,x+y,x+1/y$ considered by Bourgain and Chang [4], but their proof relies on
the arithmetic geometry theorem that is specific to Kloosterman sums [6]. All
the previous methods [4, 18, 5, 17] do not apply to the case of rational
function progressions.

In this paper, we answered this question affirmatively. To state our results,
we introduce some notation first. We use $\mathbb{E}^{*}_{x,y}$ to denote
averaging in $x,y$ variables excluding the poles of the (rational) functions
being averaged. For example, if $F(t),G(t)\in\mathbb{Q}(t)$ are rational
functions and $f_{0},f_{1},f_{2}:\mathbb{F}_{p}\longrightarrow\mathbb{C}$ are
functions, then

$\mathbb{E}^{*}_{x,y}f_{0}(x)f_{1}(x+F(y))f_{2}(x+G(y))=\frac{1}{p^{2}}\sum_{x,y\in\mathbb{F}_{p}\;\text{and}\;y\neq\text{poles}}f_{0}(x)f_{1}(x+F(y))f_{2}(x+G(y)).$

The notation $\lVert\cdot\rVert_{2}$ is the $L^{2}$-norm with respect to the
uniform probability measure on $\mathbb{F}_{p}$, that is, $\lVert
f\rVert_{2}=\bigl{(}\frac{1}{p}\sum_{x\in\mathbb{F}_{p}}|f(x)|^{2}\bigr{)}^{1/2}$.
Our main theorem is an asymptotic formula for the counting operator for three
term rational function progressions.

######  Theorem 1.1.

Let $F(t),G(t)\in\mathbb{Q}(t)$ be rational functions over $\mathbb{Q}$ such
that $F(t),G(t)$ and the constant function $1$ are linearly independent over
$\mathbb{Q}$. Then, we have the asymptotic formula

$\mathbb{E}^{*}_{x,y}f_{0}(x)f_{1}(x+F(y))f_{2}(x+G(y))=\prod_{i=0}^{2}\mathbb{E}_{z}f_{i}(z)+O_{F,G}\bigl{(}p^{-1/10}\lVert
f_{0}\rVert_{2}\lVert f_{1}\rVert_{2}\lVert f_{2}\rVert_{2}\bigr{)}$

for all functions $f_{0},f_{1},f_{2}:\mathbb{F}_{p}\longrightarrow\mathbb{C}$.

######  Remark 1.2.

In this paper we will frequently need to evaluate a $\mathbb{Q}$-coefficients
rational function over $\mathbb{F}_{p}$. Here, we use the natural
interpretation. Let $F(t)\in\mathbb{Q}(t)$ be a rational function and write
$F(t)=a(t)/b(t)$ with $a(t),b(t)\in\mathbb{Z}[t]$ and $\gcd(a(t),b(t))=1$. For
any $y\in\mathbb{F}_{p}$ such that $b(y)\neq 0$, define
$F(y)=a(y)\cdot\bigl{(}b(y)\bigr{)}^{-1}$ where $\bigl{(}b(y)\bigr{)}^{-1}$ is
the multiplicative inverse of $b(y)$ in $\mathbb{F}_{p}^{\times}$.

######  Remark 1.3.

As already observed in the case of polynomial progressions, the linear
independence assumption is crucial. The linear independence assumption for
$F(t),G(t)$ and the constant function $1$ is reasonably natural; it includes
the case that $F,G$ are linearly independent polynomials with zero constant
terms. It also includes the formulation of this question in Peluse’s survey
[16, Question 3.13].

Let $f_{0},f_{1},f_{2}$ in Theorem 1.1 be the characteristic function of a
subset $A$ of $\mathbb{F}_{p}$, we deduce that

(9) $\begin{split}&\\#\\{(x,y)\in\mathbb{F}_{p}^{2}:y\;\text{is not a pole
of}\;F\;\text{or}\;G\;\text{and}\;x,x+F(y),x+G(y)\in A\\}\\\
&=\frac{|A|^{3}}{p}+O_{F,G}\bigl{(}p^{2/5}|A|^{3/2}\bigr{)},\end{split}$

hence we conclude the following theorem.

######  Theorem 1.4.

Let $F(t),G(t)\in\mathbb{Q}(t)$ be rational functions over $\mathbb{Q}$ such
that $F(t),G(t)$ and the constant function $1$ are linearly independent over
$\mathbb{Q}$. There exists a constant $C_{F,G}>0$ such that if $A$ is a subset
of $\mathbb{F}_{p}$ of size $|A|\geq C_{F,G}\;p^{1-1/15}$, then

$\\#\\{(x,y)\in\mathbb{F}_{p}^{2}:y\;\text{is not a pole
of}\;F\;\text{or}\;G\;\text{and}\;x,x+F(y),x+G(y)\in
A\\}\gg_{F,G}\frac{|A|^{3}}{p}.$

This implies the bound $r_{F,G}(\mathbb{F}_{p})\ll_{F,G}p^{1-1/15}$. Our
approach to Theorem 1.1 also provides a new solution to the first question
$(1)$ of Bourgain and Chang. Compared to previous methods, the exponent in the
bound $r_{F,G}(\mathbb{F}_{p})\ll_{F,G}p^{1-1/15}$ is slightly better than the
exponent $1-1/24$ in Peluse’s method, but slightly worse than the exponent
$1-1/12$ in Dong-Li-Sawin’s method. Moreover, our approach has less dependence
on algebraic geometry, hence for the special progressions $x,x+y,x+y^{2}$ and
$x,x+y,x+1/y$, our proof would be an “elementary” algebra proof.

The main new difficulty for the rational function progression problem is that
differentiating a rational function does not reduce its complexity. If one
wishes to apply the degree lowering method [17], then one requires Gowers norm
control for the counting operator a priori. However, Gowers norm control is
done by PET induction via Weyl’s differencing, which is unhelpful for rational
functions. Applying Peluse’s previous method for three term polynomial
progressions [18] or Dong-Li-Sawin’s method [5] also does not work, since both
methods would reduce the problem to estimating multidimensional exponential
sums on varieties with rational function phases, which is highly nontrivial
and unknown.

The main novelty in this paper is the development of an algebraic geometry
version of PET induction that bypasses Weyl’s differencing. If we wanted to
prove the power saving bounds directly, then we would need to estimate
multidimensional exponential sums on varieties with rational function phases,
which is out of reach of the current arithmetic geometry methods. However, if
we only want to obtain the Gowers norm control, then it suffices to estimate
the dimension of certain varieties, which is in the scope of classical
algebraic geometry. We would decompose the varieties of interest into generic
parts and special subvarieties. The generic parts can be dealt with complex
analytic techniques, while the special subvarieties, which have extra
constraints and special structures, can be handled by mod $p$ point counting.
After establishing Gowers norm control, we can immediately run the degree
lowering arguments. In fact, since we obtain a $U^{2}$-Gowers norm control, we
could perform a Fourier level set decomposition of the functions to run the
degree lowering arguments more efficiently, as opposed to using the regularity
lemma.

A natural problem is to generalize algebraic geometry PET induction to longer
rational function progressions, which is a subject we intend to return in the
future. Given that all the previous methods generalize to finite fields
$\mathbb{F}_{q}$ that may not have prime cardinality, we believe our approach
also extend to all finite fields. Furthermore, it is natural to wonder about
the quantitative bounds for polynomial progressions involving linearly
dependent polynomials. This problem is still widely open; see Kuca’s work [11]
and Leng’s works leng2022quantitative,leng2023partition for current status.

This paper is organized as follow. In Section 2, we define the notation used
in this paper. The Section 3 is about the framework of the algebraic geometry
PET induction and the Section 4 is about the key dimension estimates required
for the algebraic geometry PET induction. In Section 5, we put everything
together to finish the proof. The main new ideas of this paper are in Sections
3 and 4.

###  1.2. Acknowledgments

The first author would like to thank his advisor David Conlon for his support
and discussions. The second author would like to thank his advisor Terence Tao
for inspirations, advice and support. The second author also thanks James Leng
for helpful discussions.

##  2\. Convention

###  2.1. Notation

Throughout, let $p$ be a prime and $\mathbb{F}_{p}$ be the finite field with
$p$ elements. For any finite set $\mathcal{A}$ and function
$f:\mathcal{A}\longrightarrow\mathbb{C}$, let
$\mathbb{E}_{x\in\mathcal{A}}f(x)=\frac{1}{|\mathcal{A}|}\sum_{x\in\mathcal{A}}f(x)$
be the average of $f$ over $\mathcal{A}$. When the summation variables are in
$\mathbb{F}_{p}$, we would often omit the subscript $\mathbb{F}_{p}$ in the
summation notation and write $\mathbb{E}_{x}$ instead. We use $\mathbb{E}^{*}$
to denote averaging excluding the poles of the (rational) functions being
averaged. For example, if $F(t),G(t)\in\mathbb{Q}(t)$ are rational functions
and $f_{0},f_{1},f_{2}:\mathbb{F}_{p}\longrightarrow\mathbb{C}$ are functions,
then

$\mathbb{E}^{*}_{x,y}f_{0}(x)f_{1}(x+F(y))f_{2}(x+G(y))=\frac{1}{p^{2}}\sum_{x,y\in\mathbb{F}_{p}\;\text{and}\;y\neq\text{poles}}f_{0}(x)f_{1}(x+F(y))f_{2}(x+G(y)).$

Similarly, we would denote the summation excluding the poles by $\sum^{*}$.
Furthermore, we would often omit the subscript $F,G$ in Vinogradov’s notation
$\ll,\gg$ and Landau’s notation $O,o$ since the implied constants would
usually depend on the rational functions $F,G$.

###  2.2. Fourier Analysis Notation

For any function $f:\mathbb{F}_{p}\longrightarrow\mathbb{C}$, denote the
$L^{r}$-norm with respect to the uniform probability measure on
$\mathbb{F}_{p}$ by $\lVert f\rVert_{r}=\bigl{(}\frac{1}{p}\sum_{x}\lvert
f(x)\rvert^{r}\bigr{)}^{1/r}$ and the $l^{r}$-norm with respect to the
counting measure on $\mathbb{F}_{p}$ by $\lVert
f\rVert_{l^{r}}=\bigl{(}\sum_{x}\lvert f(x)\rvert^{r}\bigr{)}^{1/r}$. Denote
$e_{p}(x)=e^{2\pi ix/p}$ and Fourier transform
$\widehat{f}(\xi)=\mathbb{E}_{x}f(x)e_{p}(-x\xi)$. In this convention,
Parseval’s identity reads $\lVert f\rVert_{2}=\lVert\widehat{f}\rVert_{l^{2}}$
and Fourier inversion reads $f(x)=\sum_{\xi}\widehat{f}(\xi)e_{p}(\xi x)$.

###  2.3. Algebraic Geometry Terminology

By (classical) varieties, we mean any classical affine, quasi-affine,
projective or quasi-projective varieties. We do not require the varieties to
be irreducible, hence our affine varieties and projective varieties may be
algebraic sets in some references. By abuse of terminology, we also refer to
schemes that are separated and of finite type over $\mathrm{Spec}\;k$ for some
field $k$ as a variety. We would often use
$Y(\mathbb{C}),Z(\mathbb{F}_{p}),...$ to denote classical varieties and
$Y,Z,...$ to denote schemes. If $X$ is a scheme over some field $k$, we also
write $X(k)$ as $k$-valued points $\mathrm{Hom}_{k}(\mathrm{Spec}\;k,X)$.

If $X$ is a variety or scheme, denote the dimension of $X$ as a variety (or
scheme) by $\dim_{\mathrm{Var}}X$ (or $\dim_{\mathrm{Sch}}X$), which is the
supremum of the length of the chains of irreducible closed subsets in the
underlying topological space.

##  3\. Algebraic Geometry PET Induction

In this section, we establish the framework of algebraic geometry PET
induction, bounding the counting operator for three term rational function
progressions in terms of $U^{2}$-Gowers norm (equivalently, the $l^{4}$-norm
of the Fourier transform) and the size of the $\mathbb{F}_{p}$-points of
certain varieties. This allows us to bypass Weyl’s differencing, which is
unhelpful for rational functions, to obtain Gowers norm control.

Let $F(t),G(t)\in\mathbb{Q}(t)$ be rational functions over $\mathbb{Q}$.
Define the _Roth variety_ associated to $F(t),G(t)$ to be the (quasi-affine)
variety cut out by the following system of equations in eight variables
$y_{1},y_{2},y_{3},y_{4},y_{5},y_{6},y_{7},y_{8}$.

(10) $\displaystyle
F(y_{1})-G(y_{1})-F(y_{2})+G(y_{2})-F(y_{3})+G(y_{3})+F(y_{4})-G(y_{4})=0$
(11) $\displaystyle
F(y_{5})-G(y_{5})-F(y_{6})+G(y_{6})-F(y_{7})+G(y_{7})+F(y_{8})-G(y_{8})=0$
(12) $\displaystyle F(y_{1})-F(y_{3})-F(y_{5})+F(y_{7})=0$ (13) $\displaystyle
F(y_{2})-F(y_{4})-F(y_{6})+F(y_{8})=0$ (14) $\displaystyle
G(y_{3})-G(y_{4})-G(y_{7})+G(y_{8})=0$

The Roth variety could be regarded as a classical variety or a scheme over
various fields, for example, the complex numbers $\mathbb{C}$ or finite fields
$\mathbb{F}_{p}$.

######  Theorem 3.1.

Let $F(t),G(t)\in\mathbb{Q}(t)$ be rational functions over $\mathbb{Q}$ and
$Y(\mathbb{F}_{p})$ be $\mathbb{F}_{p}$-points of the Roth variety associated
to $F(t),G(t)$. For all functions
$f_{0},f_{1},f_{2}:\mathbb{F}_{p}\longrightarrow\mathbb{C}$, we have the
following $U^{2}$-Gowers norm control:

$\lvert\mathbb{E}^{*}_{x,y}f_{0}(x)f_{1}(x+F(y))f_{2}(x+G(y))\rvert\leq\lVert
f_{0}\rVert_{2}^{1/2}\lVert\widehat{f_{0}}\rVert_{l^{4}}^{1/2}\lVert
f_{1}\rVert_{2}\lVert
f_{2}\rVert_{2}\left(\frac{1}{p^{3}}|Y(\mathbb{F}_{p})|\right)^{1/8}.$

###### Proof.

Fourier expand the functions $f_{0},f_{1},f_{2}$ to obtain

(15)
$\begin{split}&\lvert\mathbb{E}^{*}_{x,y}f_{0}(x)f_{1}(x+F(y))f_{2}(x+G(y))\rvert\\\
&=\Bigl{\lvert}\sum_{n_{0},n_{1},n_{2}}\widehat{f_{0}}(n_{0})\widehat{f_{1}}(n_{1})\widehat{f_{2}}(n_{2})\mathbb{E}_{x}e_{p}(n_{0}x+n_{1}x+n_{2}x)\mathbb{E}^{*}_{y}e_{p}(n_{1}F(y)+n_{2}G(y))\Bigr{\rvert}.\end{split}$

For convenience, denote
$K(n_{1},n_{2})=\mathbb{E}^{*}_{y}e_{p}(n_{1}F(y)+n_{2}G(y))$. By
orthogonality of the characters, we have

(16)
$\begin{split}&\Bigl{\lvert}\sum_{n_{0},n_{1},n_{2}}\widehat{f_{0}}(n_{0})\widehat{f_{1}}(n_{1})\widehat{f_{2}}(n_{2})\mathbb{E}_{x}e_{p}(n_{0}x+n_{1}x+n_{2}x)\mathbb{E}^{*}_{y}e_{p}(n_{1}F(y)+n_{2}G(y))\Bigr{\rvert}\\\
&=\Bigl{\lvert}\sum_{n_{1},n_{2}}\widehat{f_{0}}(-n_{1}-n_{2})\widehat{f_{1}}(n_{1})\widehat{f_{2}}(n_{2})K(n_{1},n_{2})\Bigr{\rvert}.\end{split}$

Rearrange, apply Cauchy–Schwarz inequality in $n_{2}$ and apply Parseval’s
identity to obtain

(17)
$\begin{split}&\Bigl{\lvert}\sum_{n_{1},n_{2}}\widehat{f_{0}}(-n_{1}-n_{2})\widehat{f_{1}}(n_{1})\widehat{f_{2}}(n_{2})K(n_{1},n_{2})\Bigr{\rvert}\\\
&=\Bigl{\lvert}\sum_{n_{2}}\widehat{f_{2}}(n_{2})\sum_{n_{1}}\widehat{f_{0}}(-n_{1}-n_{2})\widehat{f_{1}}(n_{1})K(n_{1},n_{2})\Bigr{\rvert}\\\
&\leq\lVert
f_{2}\rVert_{2}\Bigl{(}\sum_{n_{2},n_{1},m_{1}}\widehat{f_{0}}(-n_{1}-n_{2})\overline{\widehat{f_{0}}}(-m_{1}-n_{2})\widehat{f_{1}}(n_{1})\overline{\widehat{f_{1}}}(m_{1})K(n_{1},n_{2})\overline{K}(m_{1},n_{2})\Bigr{)}^{1/2}.\end{split}$

Once again, rearrange, apply Cauchy–Schwarz inequality in $n_{1},m_{1}$ and
apply Parseval’s identity to obtain

(18) $\begin{split}&\lVert
f_{2}\rVert_{2}\Bigl{(}\sum_{n_{2},n_{1},m_{1}}\widehat{f_{0}}(-n_{1}-n_{2})\overline{\widehat{f_{0}}}(-m_{1}-n_{2})\widehat{f_{1}}(n_{1})\overline{\widehat{f_{1}}}(m_{1})K(n_{1},n_{2})\overline{K}(m_{1},n_{2})\Bigr{)}^{1/2}\\\
&=\lVert
f_{2}\rVert_{2}\Bigl{(}\sum_{n_{1},m_{1}}\widehat{f_{1}}(n_{1})\overline{\widehat{f_{1}}}(m_{1})\sum_{n_{2}}\widehat{f_{0}}(-n_{1}-n_{2})\overline{\widehat{f_{0}}}(-m_{1}-n_{2})K(n_{1},n_{2})\overline{K}(m_{1},n_{2})\Bigr{)}^{1/2}\\\
&\leq\lVert f_{2}\rVert_{2}\lVert
f_{1}\rVert_{2}\Bigl{(}\sum_{n_{1},m_{1},n_{2},m_{2}}\widehat{f_{0}}(-n_{1}-n_{2})\overline{\widehat{f_{0}}}(-m_{1}-n_{2})\overline{\widehat{f_{0}}}(-n_{1}-m_{2})\widehat{f_{0}}(-m_{1}-m_{2})\\\
&\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad
K(n_{1},n_{2})\overline{K}(m_{1},n_{2})\overline{K}(n_{1},m_{2})K(m_{1},m_{2})\Bigr{)}^{1/4}.\end{split}$

By the change of variables $n_{1}\leftrightarrow n_{1}-n_{2}$,
$m_{1}\leftrightarrow m_{1}-n_{2}$, the last line in the expression (18) is
equal to

(19) $\begin{split}&\lVert f_{2}\rVert_{2}\lVert
f_{1}\rVert_{2}\Bigl{(}\sum_{n_{1},m_{1},n_{2},m_{2}}\widehat{f_{0}}(-n_{1})\overline{\widehat{f_{0}}}(-m_{1})\overline{\widehat{f_{0}}}(-n_{1}+n_{2}-m_{2})\widehat{f_{0}}(-m_{1}+n_{2}-m_{2})\\\
&\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad
K(n_{1}-n_{2},n_{2})\overline{K}(m_{1}-n_{2},n_{2})\overline{K}(n_{1}-n_{2},m_{2})K(m_{1}-n_{2},m_{2})\Bigr{)}^{1/4}.\end{split}$

By the change of variables $a\leftrightarrow n_{2}-m_{2},b\leftrightarrow
n_{2}$, the expression (19) is equal to

(20) $\begin{split}&\lVert f_{2}\rVert_{2}\lVert
f_{1}\rVert_{2}\Bigl{(}\sum_{n_{1},m_{1},a,b}\widehat{f_{0}}(-n_{1})\overline{\widehat{f_{0}}}(-m_{1})\overline{\widehat{f_{0}}}(-n_{1}+a)\widehat{f_{0}}(-m_{1}+a)\\\
&\quad\quad\quad\quad\quad\quad\quad\quad
K(n_{1}-b,b)\overline{K}(m_{1}-b,b)\overline{K}(n_{1}-b,b-a)K(m_{1}-b,b-a)\Bigr{)}^{1/4}.\end{split}$

Rearranging and applying Cauchy–Schwarz inequality in $n_{1},m_{1},a$, the
expression (20) is equal to

(21) $\begin{split}&\lVert f_{2}\rVert_{2}\lVert
f_{1}\rVert_{2}\Bigl{(}\sum_{n_{1},m_{1},a}\widehat{f_{0}}(-n_{1})\overline{\widehat{f_{0}}}(-m_{1})\overline{\widehat{f_{0}}}(-n_{1}+a)\widehat{f_{0}}(-m_{1}+a)\\\
&\quad\quad\quad\quad\quad\quad\quad\quad\sum_{b}K(n_{1}-b,b)\overline{K}(m_{1}-b,b)\overline{K}(n_{1}-b,b-a)K(m_{1}-b,b-a)\Bigr{)}^{1/4}\\\
&\leq\lVert f_{2}\rVert_{2}\lVert
f_{1}\rVert_{2}\Bigl{(}\sum_{n_{1},m_{1},a}\lvert\widehat{f_{0}}(-n_{1})\overline{\widehat{f_{0}}}(-m_{1})\overline{\widehat{f_{0}}}(-n_{1}+a)\widehat{f_{0}}(-m_{1}+a)\rvert^{2}\Bigr{)}^{1/8}\\\
&\quad\Bigl{(}\sum_{n_{1},m_{1},a,b,b^{\prime}}K(n_{1}-b,b)\overline{K}(m_{1}-b,b)\overline{K}(n_{1}-b,b-a)K(m_{1}-b,b-a)\\\
&\quad\quad\quad\quad\quad\quad\overline{K}(n_{1}-b^{\prime},b^{\prime})K(m_{1}-b^{\prime},b^{\prime})K(n_{1}-b^{\prime},b^{\prime}-a)\overline{K}(m_{1}-b^{\prime},b^{\prime}-a)\Bigr{)}^{1/8}.\end{split}$

We estimate the two sums above separately. First, note that

(22)
$\begin{split}&\sum_{n_{1},m_{1},a}\lvert\widehat{f_{0}}(-n_{1})\overline{\widehat{f_{0}}}(-m_{1})\overline{\widehat{f_{0}}}(-n_{1}+a)\widehat{f_{0}}(-m_{1}+a)\rvert^{2}\\\
&=\sum_{n_{1},m_{1}}\lvert\widehat{f_{0}}(-n_{1})\widehat{f_{0}}(-m_{1})\rvert^{2}\sum_{a}\lvert\widehat{f_{0}}(-n_{1}+a)\widehat{f_{0}}(-m_{1}+a)\rvert^{2}\\\
&\leq\sum_{n_{1},m_{1}}\lvert\widehat{f_{0}}(-n_{1})\widehat{f_{0}}(-m_{1})\rvert^{2}\lVert\widehat{f_{0}}\rVert_{l^{4}}^{4}\\\
&=\lVert
f_{0}\rVert_{2}^{4}\lVert\widehat{f_{0}}\rVert_{l^{4}}^{4}\end{split}$

where the second to last line follows from the Cauchy–Schwarz inequality.
Next, note that

(23)
$\begin{split}&\sum_{n_{1},m_{1},a,b,b^{\prime}}K(n_{1}-b,b)\overline{K}(m_{1}-b,b)\overline{K}(n_{1}-b,b-a)K(m_{1}-b,b-a)\\\
&\quad\quad\quad\quad\quad\quad\overline{K}(n_{1}-b^{\prime},b^{\prime})K(m_{1}-b^{\prime},b^{\prime})K(n_{1}-b^{\prime},b^{\prime}-a)\overline{K}(m_{1}-b^{\prime},b^{\prime}-a)\\\
&=\frac{1}{p^{8}}\sum_{\begin{subarray}{c}n_{1},m_{1},a\\\
b,b^{\prime}\end{subarray}}\quad\sideset{}{{}^{*}}{\sum}_{\begin{subarray}{c}y_{1},y_{2},y_{3},y_{4}\\\
y_{5},y_{6},y_{7},y_{8}\end{subarray}}\\\ &\quad\quad\quad
e_{p}((n_{1}-b)F(y_{1})+bG(y_{1}))\;e_{p}(-(m_{1}-b)F(y_{2})-bG(y_{2}))\\\
&\quad\quad\quad
e_{p}(-(n_{1}-b)F(y_{3})-(b-a)G(y_{3}))\;e_{p}((m_{1}-b)F(y_{4})+(b-a)G(y_{4}))\\\
&\quad\quad\quad
e_{p}(-(n_{1}-b^{\prime})F(y_{5})-b^{\prime}G(y_{5}))\;e_{p}((m_{1}-b^{\prime})F(y_{6})+b^{\prime}G(y_{6}))\\\
&\quad\quad\quad
e_{p}((n_{1}-b^{\prime})F(y_{7})+(b^{\prime}-a)G(y_{7}))\;e_{p}(-(m_{1}-b^{\prime})F(y_{8})-(b^{\prime}-a)G(y_{8})).\end{split}$

This is equal to

(24)
$\begin{split}&\frac{1}{p^{8}}\quad\sideset{}{{}^{*}}{\sum}_{\begin{subarray}{c}y_{1},y_{2},y_{3},y_{4}\\\
y_{5},y_{6},y_{7},y_{8}\end{subarray}}\quad\sum_{\begin{subarray}{c}n_{1},m_{1},a\\\
b,b^{\prime}\end{subarray}}\\\ &\quad\quad\quad
e_{p}(b(-F(y_{1})+G(y_{1})+F(y_{2})-G(y_{2})+F(y_{3})-G(y_{3})-F(y_{4})+G(y_{4})))\\\
&\quad\quad\quad
e_{p}(b^{\prime}(F(y_{5})-G(y_{5})-F(y_{6})+G(y_{6})-F(y_{7})+G(y_{7})+F(y_{8})-G(y_{8})))\\\
&\quad\quad\quad e_{p}(n_{1}(F(y_{1})-F(y_{3})-F(y_{5})+F(y_{7})))\\\
&\quad\quad\quad e_{p}(m_{1}(-F(y_{2})+F(y_{4})+F(y_{6})-F(y_{8})))\\\
&\quad\quad\quad e_{p}(a(G(y_{3})-G(y_{4})-G(y_{7})+G(y_{8})))\\\
&=\frac{1}{p^{8}}\quad\sum_{(y_{1},y_{2},y_{3},y_{4},y_{5},y_{6},y_{7},y_{8})\in
Y(\mathbb{F}_{p})}p^{5}\\\ &=\frac{1}{p^{3}}|Y(\mathbb{F}_{p})|\end{split}$

where the second to last equality follows from the orthogonality of the
characters. Combining inequalities (21),(22) and (24) completes the proof. ∎

######  Remark 3.2.

In the proof of Theorem 3.1, if we were to only apply Cauchy–Schwarz
inequality two times and stop at (20), then we could formulate an alternative
approach to Theroem 1.1, conditionally on the estimates of multidimensional
exponential sums on varieties with rational function phases, which is highly
nontrivial and out of reach of the current arithmetic geometry methods.

##  4\. Dimension Estimates

In Section 3, we establish a Gowers norm control, conditionally on estimating
the size of the $\mathbb{F}_{p}$-points of the Roth variety. We will prove the
required estimates in Theorem 4.1, completing the algebraic geometry PET
induction. The proof of Theorem 4.1 is quite long and technical, hence we
would like to briefly describe the strategy first.

We would decompose the Roth variety $Y$ into five pieces
$Y=Y_{\mathrm{gen}}\cup Y_{\mathrm{low}}\cup Z_{\mathrm{good}}\cup
Z_{\mathrm{bad}}\cup Z_{\mathrm{low}}$ and estimate via different techniques.
The variety $Y_{\mathrm{gen}}$ is a generic part of $Y$, which can be handled
by complex analytic techniques. The variety $Z_{\mathrm{good}}$ is a generic
component of a special subvariety of $Y$, which can be handled by complex
analytic techniques too. On the other hand, the variety $Z_{\mathrm{bad}}$ is
a special subvariety with extra constraints and special structures, hence we
have extra inputs for mod $p$ point counting. Finally, the varieties
$Y_{\mathrm{low}}$ and $Z_{\mathrm{low}}$ are of lower dimension, therefore we
can control them more easily.

######  Theorem 4.1.

Let $F(t),G(t)\in\mathbb{Q}(t)$ be rational functions over $\mathbb{Q}$ such
that $F(t),G(t)$ and the constant function $1$ are linearly independent over
$\mathbb{Q}$, and $Y(\mathbb{F}_{p})$ be the $\mathbb{F}_{p}$-points of the
Roth variety associated to $F(t),G(t)$. Then, we have the following point
counting estimates:

$|Y(\mathbb{F}_{p})|\ll p^{3}.$

###### Proof.

We would decompose $Y(\mathbb{F}_{p})$ into various parts and estimate via
different techniques accordingly. First, note that the first five columns of
the Jacobian matrix of the defining equations (10), (11), (12), (13), (14) of
the Roth variety are

$\begin{bmatrix}F^{\prime}(y_{1})-G^{\prime}(y_{1})&-F^{\prime}(y_{2})+G^{\prime}(y_{2})&-F^{\prime}(y_{3})+G^{\prime}(y_{3})&F^{\prime}(y_{4})-G^{\prime}(y_{4})&0\\\
0&0&0&0&F^{\prime}(y_{5})-G^{\prime}(y_{5})\\\
F^{\prime}(y_{1})&0&-F^{\prime}(y_{3})&0&-F^{\prime}(y_{5})\\\
0&F^{\prime}(y_{2})&0&-F^{\prime}(y_{4})&0\\\
0&0&G^{\prime}(y_{3})&-G^{\prime}(y_{4})&0\end{bmatrix}.$

Let $D(y_{1},y_{2},y_{3},y_{4},y_{5})$ be the determinant of the matrix, that
is,

$\bigl{(}G^{\prime}(y_{5})-F^{\prime}(y_{5})\bigr{)}\begin{vmatrix}F^{\prime}(y_{1})-G^{\prime}(y_{1})&-F^{\prime}(y_{2})+G^{\prime}(y_{2})&-F^{\prime}(y_{3})+G^{\prime}(y_{3})&F^{\prime}(y_{4})-G^{\prime}(y_{4})\\\
F^{\prime}(y_{1})&0&-F^{\prime}(y_{3})&0\\\
0&F^{\prime}(y_{2})&0&-F^{\prime}(y_{4})\\\
0&0&G^{\prime}(y_{3})&-G^{\prime}(y_{4})\end{vmatrix}$

which is equal to

$\bigl{(}G^{\prime}(y_{5})-F^{\prime}(y_{5})\bigr{)}\begin{vmatrix}-G^{\prime}(y_{1})&G^{\prime}(y_{2})&0&0\\\
F^{\prime}(y_{1})&0&-F^{\prime}(y_{3})&0\\\
0&F^{\prime}(y_{2})&0&-F^{\prime}(y_{4})\\\
0&0&G^{\prime}(y_{3})&-G^{\prime}(y_{4})\end{vmatrix}.$

Hence, we have

(25)
$D(y_{1},y_{2},y_{3},y_{4},y_{5})=(G^{\prime}(y_{5})-F^{\prime}(y_{5}))\Bigl{[}G^{\prime}(y_{1})F^{\prime}(y_{2})F^{\prime}(y_{3})G^{\prime}(y_{4})-F^{\prime}(y_{1})G^{\prime}(y_{2})G^{\prime}(y_{3})F^{\prime}(y_{4})\Bigr{]}.$

Define

(26)
$Y_{\mathrm{gen}}=Y\cap\\{(y_{1},y_{2},y_{3},y_{4},y_{5},y_{6},y_{7},y_{8})\in\mathbb{A}^{8}:D(y_{1},y_{2},y_{3},y_{4},y_{5})\neq
0\\}$

and

(27)
$Y_{\mathrm{sp}}=Y\cap\\{(y_{1},y_{2},y_{3},y_{4},y_{5},y_{6},y_{7},y_{8})\in\mathbb{A}^{8}:D(y_{1},y_{2},y_{3},y_{4},y_{5})=0\\}.$

Note that we have $Y(\mathbb{F}_{p})=Y_{\mathrm{gen}}(\mathbb{F}_{p})\cup
Y_{\mathrm{sp}}(\mathbb{F}_{p})$. Informally, the variety $Y_{\mathrm{gen}}$
is a generic part of $Y$ which lies in the smooth component of $Y$, and
$Y_{\mathrm{sp}}$ is a special subvariety of $Y$ cut out by
$D(y_{1},y_{2},y_{3},y_{4},y_{5})$.

The associated complex analytic space of $Y_{\mathrm{gen}}(\mathbb{C})$ is a
$3$-dimensional complex manifold since the Jacobian matrices have full rank
and we can apply the implicit function theorem. By Noether normalization,
there exists a finite surjective map
$Y_{\mathrm{gen}}(\mathbb{C})\longrightarrow\mathbb{A}^{d}(\mathbb{C})$
between varieties, where $d$ is the dimension of
$Y_{\mathrm{gen}}(\mathbb{C})$ as a variety and $\mathbb{A}^{d}$ is the
$d$-dimensional affine space. Regard the map
$Y_{\mathrm{gen}}(\mathbb{C})\longrightarrow\mathbb{A}^{d}(\mathbb{C})$ as a
finite holomorphic map between complex manifolds. By the first theorem in [9,
Chapter 5, Section 4.1], we deduce that the dimension of
$Y_{\mathrm{gen}}(\mathbb{C})$ as a variety is $3$. (We compare the dimension
of a variety over $\mathbb{C}$ and the dimension of its associated complex
analytic space via ad hoc argument here, for general dimension theory of
complex analytic spaces, see [10, Appendix B] and [9, Chapter 5, Section 1].)

Consider an affine $\mathbb{Z}$-model of $Y_{\mathrm{gen}}(\mathbb{C})$ by
clearing out the denominators in the defining equations of the Roth variety
and adding one auxiliary variable $z$ and one equation to require that the
denominators are nonzero. To be more specific, write $F(t)=a(t)/b(t)$ and
$G(t)=c(t)/d(t)$ with $a(t),b(t),c(t),d(t)\in\mathbb{Z}[t]$, let
$I\subset\mathbb{Z}[y_{1},y_{2},y_{3},y_{4},y_{5},y_{6},y_{7},y_{8},z]$ be the
ideal generated by the following polynomials:

(28)
$\displaystyle\Bigl{(}\prod_{i=1}^{4}b(y_{i})d(y_{i})\Bigr{)}\Bigl{(}F(y_{1})-G(y_{1})-F(y_{2})+G(y_{2})-F(y_{3})+G(y_{3})+F(y_{4})-G(y_{4})\Bigr{)}$
(29)
$\displaystyle\Bigl{(}\prod_{i=5}^{8}b(y_{i})d(y_{i})\Bigr{)}\Bigl{(}F(y_{5})-G(y_{5})-F(y_{6})+G(y_{6})-F(y_{7})+G(y_{7})+F(y_{8})-G(y_{8})\Bigr{)}$
(30)
$\displaystyle\Bigl{(}b(y_{1})b(y_{3})b(y_{5})b(y_{7})\Bigr{)}\Bigl{(}F(y_{1})-F(y_{3})-F(y_{5})+F(y_{7})\Bigr{)}$
(31)
$\displaystyle\Bigl{(}b(y_{2})b(y_{4})b(y_{6})b(y_{8})\Bigr{)}\Bigl{(}F(y_{2})-F(y_{4})-F(y_{6})+F(y_{8})\Bigr{)}$
(32)
$\displaystyle\Bigl{(}d(y_{3})d(y_{4})d(y_{7})d(y_{8})\Bigr{)}\Bigl{(}G(y_{3})-G(y_{4})-G(y_{7})+G(y_{8})\Bigr{)}$
(33) $\displaystyle
z\Bigl{(}\prod_{i=1}^{8}b(y_{i})d(y_{i})\Bigr{)}\biggl{[}\Bigl{(}\prod_{i=1}^{5}b(y_{i})d(y_{i})\Bigr{)}^{2}D(y_{1},y_{2},y_{3},y_{4},y_{5})\biggr{]}-1.$

Define affine $\mathbb{Z}$-model
$Y_{\mathrm{gen}}=\mathrm{Spec}\;\mathbb{Z}[y_{1},y_{2},y_{3},y_{4},y_{5},y_{6},y_{7},y_{8},z]/I$.
Note that the $\mathbb{C}$-points of
$Y_{\mathrm{gen}}\times_{\mathbb{Z}}\mathbb{C}$ is isomorphic to
$Y_{\mathrm{gen}}(\mathbb{C})$, hence we have
$\dim_{\mathrm{Sch}}Y_{\mathrm{gen}}\times_{\mathbb{Z}}\mathbb{C}=\dim_{\mathrm{Var}}Y_{\mathrm{gen}}(\mathbb{C})=3$.
By [10, Chapter 2, Exercise 3.20.(f)], we have the invariance of dimension
under base change, i.e.,
$\dim_{\mathrm{Sch}}Y_{\mathrm{gen}}\times_{\mathbb{Z}}\mathbb{Q}=\dim_{\mathrm{Sch}}Y_{\mathrm{gen}}\times_{\mathbb{Z}}\mathbb{C}=3$.
(Note that the integral assumption in [10, Chapter 2, Exercise 3.20.(f)] can
be removed if one does not require the scheme after base change is
equidimensional.)

Note that the structure map
$Y_{\mathrm{gen}}=\mathrm{Spec}\;\mathbb{Z}[y_{1},y_{2},y_{3},y_{4},y_{5},y_{6},y_{7},y_{8},z]/I\longrightarrow\mathrm{Spec}\;\mathbb{Z}$
is of finite type and the base $\mathrm{Spec}\;\mathbb{Z}$ is irreducible, by
[24, Section 05F6, Lemma 37.30.1], we deduce that the dimension of the generic
fiber is the same as mod $p$ fibers for all but finitely many $p$, that is,
$\dim_{\mathrm{Sch}}Y_{\mathrm{gen}}\times_{\mathbb{Z}}\mathbb{Q}=\dim_{\mathrm{Sch}}Y_{\mathrm{gen}}\times_{\mathbb{Z}}\mathbb{F}_{p}=3$
for all but finitely many $p$. Once again, by [10, Chapter 2, Exercise
3.20.(f)], we have
$\dim_{\mathrm{Sch}}Y_{\mathrm{gen}}\times_{\mathbb{Z}}\mathbb{F}_{p}=\dim_{\mathrm{Sch}}Y_{\mathrm{gen}}\times_{\mathbb{Z}}\overline{\mathbb{F}}_{p}$.
Hence, we have $\dim_{\mathrm{Var}}Y(\overline{\mathbb{F}}_{p})=3$ for all but
finitely many $p$. Now, the Lang-Weil bound [12] (also see [26, Lemma 1])
implies that $Y_{\mathrm{gen}}(\mathbb{F}_{p})\ll p^{3}$.

Next, we would like to further decompose $Y_{\mathrm{sp}}(\mathbb{F}_{p})$,
let $\widetilde{D}(y_{1},y_{2},y_{3},y_{4})$ denote the function

(34)
$\frac{D(y_{1},y_{2},y_{3},y_{4},y_{5})}{F^{\prime}(y_{1})F^{\prime}(y_{2})F^{\prime}(y_{3})F^{\prime}(y_{4})\bigl{(}G^{\prime}(y_{5})-F^{\prime}(y_{5})\bigr{)}}=\frac{G^{\prime}(y_{1})G^{\prime}(y_{4})}{F^{\prime}(y_{1})F^{\prime}(y_{4})}-\frac{G^{\prime}(y_{2})G^{\prime}(y_{3})}{F^{\prime}(y_{2})F^{\prime}(y_{3})}.$

Define

(35)
$Y_{\mathrm{low}}=Y_{\mathrm{sp}}\cap\\{(y_{1},y_{2},y_{3},y_{4},y_{5},y_{6},y_{7},y_{8})\in\mathbb{A}^{8}:F^{\prime}(y_{1})F^{\prime}(y_{2})F^{\prime}(y_{3})F^{\prime}(y_{4})\bigl{(}G^{\prime}(y_{5})-F^{\prime}(y_{5})\bigr{)}=0\\}$

and

(36)
$Z=Y_{\mathrm{sp}}\cap\\{(y_{1},y_{2},y_{3},y_{4},y_{5},y_{6},y_{7},y_{8})\in\mathbb{A}^{8}:\widetilde{D}(y_{1},y_{2},y_{3},y_{4})=0\\}.$

Note that we have
$Y_{\mathrm{sp}}(\mathbb{F}_{p})=Y_{\mathrm{low}}(\mathbb{F}_{p})\cup
Z(\mathbb{F}_{p})$. Informally, the variety $Y_{\mathrm{low}}(\mathbb{F}_{p})$
has strong constraint
$F^{\prime}(y_{1})F^{\prime}(y_{2})F^{\prime}(y_{3})F^{\prime}(y_{4})\bigl{(}G^{\prime}(y_{5})-F^{\prime}(y_{5})\bigr{)}=0$,
hence this lower dimensional contribution can be bounded more easily later. We
would like to further decompose $Z(\mathbb{F}_{p})$. To do so, consider the
Jacobian matrix of the following equations:

(37) $\displaystyle
F(y_{1})-G(y_{1})-F(y_{2})+G(y_{2})-F(y_{3})+G(y_{3})+F(y_{4})-G(y_{4})=0$
(38) $\displaystyle\widetilde{D}(y_{1},y_{2},y_{3},y_{4})=0,$

which is

$\begin{bmatrix}F^{\prime}(y_{1})-G^{\prime}(y_{1})&-F^{\prime}(y_{2})+G^{\prime}(y_{2})&-F^{\prime}(y_{3})+G^{\prime}(y_{3})&F^{\prime}(y_{4})-G^{\prime}(y_{4})\\\
\frac{G^{\prime}(y_{4})}{F^{\prime}(y_{4})}\Bigl{(}\frac{G^{\prime}(y_{1})}{F^{\prime}(y_{1})}\Bigr{)}^{\prime}&-\frac{G^{\prime}(y_{3})}{F^{\prime}(y_{3})}\Bigl{(}\frac{G^{\prime}(y_{2})}{F^{\prime}(y_{2})}\Bigr{)}^{\prime}&-\frac{G^{\prime}(y_{2})}{F^{\prime}(y_{2})}\Bigl{(}\frac{G^{\prime}(y_{3})}{F^{\prime}(y_{3})}\Bigr{)}^{\prime}&\frac{G^{\prime}(y_{1})}{F^{\prime}(y_{1})}\Bigl{(}\frac{G^{\prime}(y_{4})}{F^{\prime}(y_{4})}\Bigr{)}^{\prime}\end{bmatrix}.$

Let $E(y_{1},y_{4})$ be the determinant of the minor matrix of the Jacobian
matrix consisting of the first column and fourth column, i.e.

$E(y_{1},y_{4})=\begin{vmatrix}F^{\prime}(y_{1})-G^{\prime}(y_{1})&F^{\prime}(y_{4})-G^{\prime}(y_{4})\\\
\frac{G^{\prime}(y_{4})}{F^{\prime}(y_{4})}\Bigl{(}\frac{G^{\prime}(y_{1})}{F^{\prime}(y_{1})}\Bigr{)}^{\prime}&\frac{G^{\prime}(y_{1})}{F^{\prime}(y_{1})}\Bigl{(}\frac{G^{\prime}(y_{4})}{F^{\prime}(y_{4})}\Bigr{)}^{\prime}\end{vmatrix}$

which is

(39)
$\bigl{(}F^{\prime}(y_{1})-G^{\prime}(y_{1})\bigr{)}\frac{G^{\prime}(y_{1})}{F^{\prime}(y_{1})}\Bigl{(}\frac{G^{\prime}(y_{4})}{F^{\prime}(y_{4})}\Bigr{)}^{\prime}-\bigl{(}F^{\prime}(y_{4})-G^{\prime}(y_{4})\bigr{)}\frac{G^{\prime}(y_{4})}{F^{\prime}(y_{4})}\Bigl{(}\frac{G^{\prime}(y_{1})}{F^{\prime}(y_{1})}\Bigr{)}^{\prime}.$

Also, let $\widetilde{E}(y_{1},y_{4})$ denote the function

(40)
$\frac{E(y_{1},y_{4})}{\Bigl{(}\frac{G^{\prime}(y_{1})}{F^{\prime}(y_{1})}\Bigr{)}^{\prime}\Bigl{(}\frac{G^{\prime}(y_{4})}{F^{\prime}(y_{4})}\Bigr{)}^{\prime}}=\frac{\bigl{(}F^{\prime}(y_{1})-G^{\prime}(y_{1})\bigr{)}\frac{G^{\prime}(y_{1})}{F^{\prime}(y_{1})}}{\Bigl{(}\frac{G^{\prime}(y_{1})}{F^{\prime}(y_{1})}\Bigr{)}^{\prime}}-\frac{\bigl{(}F^{\prime}(y_{4})-G^{\prime}(y_{4})\bigr{)}\frac{G^{\prime}(y_{4})}{F^{\prime}(y_{4})}}{\Bigl{(}\frac{G^{\prime}(y_{4})}{F^{\prime}(y_{4})}\Bigr{)}^{\prime}}.$

Define

(41)
$Z_{good}=Z\cap\\{(y_{1},y_{2},y_{3},y_{4},y_{5},y_{6},y_{7},y_{8})\in\mathbb{A}^{8}:E(y_{1},y_{4})\neq
0\\},$ (42)
$Z_{bad}=Z\cap\\{(y_{1},y_{2},y_{3},y_{4},y_{5},y_{6},y_{7},y_{8})\in\mathbb{A}^{8}:\widetilde{E}(y_{1},y_{4})=0\\}$

and

(43)
$Z_{low}=Z\cap\Bigl{\\{}(y_{1},y_{2},y_{3},y_{4},y_{5},y_{6},y_{7},y_{8})\in\mathbb{A}^{8}:\Bigl{(}\frac{G^{\prime}(y_{1})}{F^{\prime}(y_{1})}\Bigr{)}^{\prime}\Bigl{(}\frac{G^{\prime}(y_{4})}{F^{\prime}(y_{4})}\Bigr{)}^{\prime}=0\Bigr{\\}}.$

Note that we have $Z(\mathbb{F}_{p})=Z_{\mathrm{good}}(\mathbb{F}_{p})\cup
Z_{\mathrm{bad}}(\mathbb{F}_{p})\cup Z_{\mathrm{low}}(\mathbb{F}_{p})$.
Informally, the variety $Z_{\mathrm{good}}(\mathbb{F}_{p})$ is a generic part
of $Z(\mathbb{F}_{p})$ which satisfies good property, the variety
$Z_{\mathrm{bad}}(\mathbb{F}_{p})$ is a special subvariety cut out by
$\widetilde{E}(y_{1},y_{4})=0$, and $Z_{\mathrm{low}}(\mathbb{F}_{p})$ is a
lower dimensional contribution. It remains to bound
$|Z_{\mathrm{good}}(\mathbb{F}_{p})|,|Z_{\mathrm{bad}}(\mathbb{F}_{p})|,|Z_{\mathrm{low}}(\mathbb{F}_{p})|$
and $|Y_{\mathrm{low}}(\mathbb{F}_{p})|$.

We start from bounding $|Z_{\mathrm{good}}(\mathbb{F}_{p})|$. Consider the
auxiliary variety $X\subset\mathbb{A}^{4}$ defined by the following equations

(44) $\displaystyle
F(y_{1})-G(y_{1})-F(y_{2})+G(y_{2})-F(y_{3})+G(y_{3})+F(y_{4})-G(y_{4})=0$
(45) $\displaystyle\widetilde{D}(y_{1},y_{2},y_{3},y_{4})=0$ (46)
$\displaystyle E(y_{1},y_{4})\neq 0.$

The associated complex analytic space of $X(\mathbb{C})$ is a $2$-dimensional
complex manifold since the condition $E(y_{1},y_{4})\neq 0$ guarantees that
the Jacobian matrices of the equations (44), (45) have full rank and we can
apply the implicit function theorem. Similar to the dimension comparison
between $Y_{\mathrm{gen}}(\mathbb{C})$ and its associated complex analytic
space, this implies that the dimension of $X(\mathbb{C})$ as a variety is $2$.
Moreover, similar to the previous arguments regarding $Y_{\mathrm{gen}}$, we
conclude that $X(\mathbb{F}_{p})\ll p^{2}$.

For all points $(y_{1},y_{2},y_{3},y_{4},y_{5},y_{6},y_{7},y_{8})\in
Z_{\mathrm{good}}(\mathbb{F}_{p})$, the coordinates
$(y_{1},y_{2},y_{3},y_{4})$ lie on $X(\mathbb{F}_{p})$, hence there are at
most $O(p^{2})$ choices of $y_{1},y_{2},y_{3},y_{4}$. Besides, there are at
most $p$ choices of $y_{5}$, hence there are at most $O(p^{3})$ choices of
$y_{1},y_{2},y_{3},y_{4},y_{5}$. For any given
$y_{1},y_{2},y_{3},y_{4},y_{5}$, there are only $O(1)$ choices of $y_{7}$ by
equation (12). For any given $y_{1},y_{2},y_{3},y_{4},y_{5},y_{7}$, there are
only $O(1)$ choices of $y_{8}$ by equation (14). For any given
$y_{1},y_{2},y_{3},y_{4},y_{5},y_{7},y_{8}$, there are only $O(1)$ choices of
$y_{6}$ by equation (13). Therefore, we have
$Z_{\mathrm{good}}(\mathbb{F}_{p})\ll p^{3}$.

Next, we turn our attention to bounding $|Z_{\mathrm{bad}}(\mathbb{F}_{p})|$.
We would first show that the equation $\widetilde{E}(y_{1},y_{4})=0$ is a
nontrivial constraint by showing

(47)
$\frac{\bigl{(}F^{\prime}(y_{1})-G^{\prime}(y_{1})\bigr{)}\frac{G^{\prime}(y_{1})}{F^{\prime}(y_{1})}}{\Bigl{(}\frac{G^{\prime}(y_{1})}{F^{\prime}(y_{1})}\Bigr{)}^{\prime}}$

is nonconstant. Suppose not, we have

(48)
$c\Bigl{(}\frac{G^{\prime}(y_{1})}{F^{\prime}(y_{1})}\Bigr{)}^{\prime}=\bigl{(}F^{\prime}(y_{1})-G^{\prime}(y_{1})\bigr{)}\frac{G^{\prime}(y_{1})}{F^{\prime}(y_{1})}$

for some constant $c$. If $c=0$, then we have
$F^{\prime}(y_{1})-G^{\prime}(y_{1})=0$ or
$G^{\prime}(y_{1})/F^{\prime}(y_{1})=0$, which both contradict the linear
independence of $F,G$ and the constant function $1$. If $c\neq 0$, then we
have

(49)
$\Bigl{(}\frac{G^{\prime}(y_{1})}{F^{\prime}(y_{1})}\Bigr{)}^{\prime}-\frac{F^{\prime}(y_{1})-G^{\prime}(y_{1})}{c}\frac{G^{\prime}(y_{1})}{F^{\prime}(y_{1})}=0.$

Multiply integrating factor to equation (49), we have

(50)
$\biggl{[}\exp\Bigl{(}-\frac{F(y_{1})-G(y_{1})}{c}\Bigr{)}\frac{G^{\prime}(y_{1})}{F^{\prime}(y_{1})}\biggr{]}^{\prime}=0,$

integrate both side of (50), we deduce

(51)
$\exp\Bigl{(}-\frac{F(y_{1})-G(y_{1})}{c}\Bigr{)}\frac{G^{\prime}(y_{1})}{F^{\prime}(y_{1})}=\widetilde{c}$

for some constant $\widetilde{c}$. Since $G^{\prime}(y_{1})/F^{\prime}(y_{1})$
and $\bigl{(}F(y_{1})-G(y_{1})\bigr{)}/c$ are rational functions, the equality
(51) can only happen when $G^{\prime}(y_{1})/F^{\prime}(y_{1})=0$ or
$\bigl{(}F(y_{1})-G(y_{1})\bigr{)}/c$ is a constant, which both contradict the
linear independence of $F,G$ and the constant function $1$. Hence, the
rational function (47) is not a constant.

Now, for all points $(y_{1},y_{2},y_{3},y_{4},y_{5},y_{6},y_{7},y_{8})\in
Z_{\mathrm{bad}}(\mathbb{F}_{p})$, the coordinates $y_{2},y_{4},y_{5}$ have at
most $p^{3}$ choices. For any given $y_{2},y_{4},y_{5}$, there are at most
$O(1)$ choices of $y_{1}$ by equation $\widetilde{E}(y_{1},y_{4})=0$. For any
given $y_{1},y_{2},y_{4},y_{5}$, there are at most $O(1)$ choices of $y_{3}$
by equation (10). For any given $y_{1},y_{2},y_{3},y_{4},y_{5}$, there are at
most $O(1)$ choices of $y_{7}$ by equation (12). For any given
$y_{1},y_{2},y_{3},y_{4},y_{5},y_{7}$, there are at most $O(1)$ choices of
$y_{8}$ by equation (14). For any given
$y_{1},y_{2},y_{3},y_{4},y_{5},y_{7},y_{8}$, there are at most $O(1)$ choices
of $y_{6}$ by equation (13). Therefore, we have
$|Z_{\mathrm{bad}}(\mathbb{F}_{p})|\ll p^{3}$.

Finally, we estimate $|Y_{\mathrm{low}}(\mathbb{F}_{p})|$ and
$|Z_{\mathrm{low}}(\mathbb{F}_{p})|$. These varieties have strong constraints
which allow us to bound their cardinality more easily. We first decompose
$Y_{\mathrm{low}}(\mathbb{F}_{p})$. Define

(52)
$Y_{\mathrm{low}}^{(1)}=Y_{\mathrm{low}}\cap\\{(y_{1},y_{2},y_{3},y_{4},y_{5},y_{6},y_{7},y_{8})\in\mathbb{A}^{8}:F^{\prime}(y_{1})=0\\},$
(53)
$Y_{\mathrm{low}}^{(2)}=Y_{\mathrm{low}}\cap\\{(y_{1},y_{2},y_{3},y_{4},y_{5},y_{6},y_{7},y_{8})\in\mathbb{A}^{8}:F^{\prime}(y_{2})=0\\},$
(54)
$Y_{\mathrm{low}}^{(3)}=Y_{\mathrm{low}}\cap\\{(y_{1},y_{2},y_{3},y_{4},y_{5},y_{6},y_{7},y_{8})\in\mathbb{A}^{8}:F^{\prime}(y_{3})=0\\},$
(55)
$Y_{\mathrm{low}}^{(4)}=Y_{\mathrm{low}}\cap\\{(y_{1},y_{2},y_{3},y_{4},y_{5},y_{6},y_{7},y_{8})\in\mathbb{A}^{8}:F^{\prime}(y_{4})=0\\}$

and

(56)
$Y_{\mathrm{low}}^{(5)}=Y_{\mathrm{low}}\cap\\{(y_{1},y_{2},y_{3},y_{4},y_{5},y_{6},y_{7},y_{8})\in\mathbb{A}^{8}:\bigl{(}G^{\prime}(y_{5})-F^{\prime}(y_{5})\bigr{)}=0\\}.$

Note that we have
$Y_{\mathrm{low}}(\mathbb{F}_{p})=Y_{\mathrm{low}}^{(1)}(\mathbb{F}_{p})\cup
Y_{\mathrm{low}}^{(2)}(\mathbb{F}_{p})\cup
Y_{\mathrm{low}}^{(3)}(\mathbb{F}_{p})\cup
Y_{\mathrm{low}}^{(4)}(\mathbb{F}_{p})\cup
Y_{\mathrm{low}}^{(5)}(\mathbb{F}_{p})$. For all
$(y_{1},y_{2},y_{3},y_{4},y_{5},y_{6},y_{7},y_{8})\in
Y_{\mathrm{low}}^{(1)}(\mathbb{F}_{p})$, the coordinates $y_{2},y_{4},y_{5}$
have at most $p^{3}$ choices. For any given $y_{2},y_{4},y_{5}$, there are at
most $O(1)$ choices of $y_{1}$ by equation $F^{\prime}(y_{1})=0$. For any
given $y_{1},y_{2},y_{4},y_{5}$, there are at most $O(1)$ choices of $y_{3}$
by equation (10). For any given $y_{1},y_{2},y_{3},y_{4},y_{5}$, there are at
most $O(1)$ choices of $y_{7}$ by equation (12). For any given
$y_{1},y_{2},y_{3},y_{4},y_{5},y_{7}$, there are at most $O(1)$ choices of
$y_{8}$ by equation (14). For any given
$y_{1},y_{2},y_{3},y_{4},y_{5},y_{7},y_{8}$, there are at most $O(1)$ choices
of $y_{6}$ by equation (13). Therefore, we have
$|Y_{\mathrm{low}}^{(1)}(\mathbb{F}_{p})|\ll p^{3}$. Similarly, we have
$|Y_{\mathrm{low}}^{(2)}(\mathbb{F}_{p})|,|Y_{\mathrm{low}}^{(3)}(\mathbb{F}_{p})|,|Y_{\mathrm{low}}^{(4)}(\mathbb{F}_{p})|,|Y_{\mathrm{low}}^{(5)}(\mathbb{F}_{p})|\ll
p^{3}$ and hence $|Y_{\mathrm{low}}(\mathbb{F}_{p})|\ll p^{3}$. A similar
decomposing and counting argument would show that
$|Z_{\mathrm{low}}(\mathbb{F}_{p})|\ll p^{3}$ too. This completes the proof. ∎

##  5\. Final Arguments

In this section, we would put everything together to complete the proof of
Theorem 1.1. After establishing Gowers norm control, we could immediately run
the degree lowering arguments [17]. In fact, since we obtain a $U^{2}$-Gowers
norm control, we would perform a Fourier level set decomposition of the
functions to run the degree lowering arguments more efficiently, as opposed to
using the regularity lemma.

We first start with the case of two term rational function progressions, which
serves as the base case of the induction. As observed by Peluse [17], one
should consider the counting operator twisted by the characters for the
purpose of induction. The proof of Proposition 5.1 is essentially the same as
[17, Lemma 3.2], except that we apply Bombieri’s bound instead of Weil’s
bound. Denote the characteristic function by $1_{\xi=0}$, that is,
$1_{\xi=0}=1$ when $\xi=0$ and $1_{\xi=0}=0$ when $\xi\neq 0$.

######  Proposition 5.1.

Let $F(t),R(t)\in\mathbb{Q}(t)$ be rational functions over $\mathbb{Q}$ such
that $F(t),R(t)$ and the constant function $1$ are linearly independent over
$\mathbb{Q}$. Then, we have the asymptotic formula

$\mathbb{E}^{*}_{x,y}f_{0}(x)f_{1}(x+F(y))e_{p}(\xi
R(y))=1_{\xi=0}\prod_{i=0}^{1}\mathbb{E}_{z}f_{i}(z)+O_{F,R}\bigl{(}p^{-1/2}\lVert
f_{0}\rVert_{2}\lVert f_{1}\rVert_{2}\bigr{)}$

for all functions $f_{0},f_{1}:\mathbb{F}_{p}\longrightarrow\mathbb{C}$ and
$\xi\in\mathbb{F}_{p}$.

###### Proof.

Fourier expand the functions $f_{0},f_{1}$ to obtain

(57) $\begin{split}&\mathbb{E}^{*}_{x,y}f_{0}(x)f_{1}(x+F(y))e_{p}(\xi
R(y))\\\
&=\mathbb{E}^{*}_{x,y}\sum_{n_{0},n_{1}}\widehat{f_{0}}(n_{0})\widehat{f_{1}}(n_{1})e_{p}(n_{0}x+n_{1}x)e_{p}(n_{1}F(y)+\xi
R(y))\\\
&=\sum_{n_{1}}\widehat{f_{0}}(-n_{1})\widehat{f_{1}}(n_{1})\mathbb{E}^{*}_{y}e_{p}(n_{1}F(y)+\xi
R(y)).\end{split}$

By linear independence of $F,R$ and the constant function $1$, the function
$n_{1}F(y)+\xi R(y)$ is nonconstant unless $n_{1}=\xi=0$. When $n_{1}\neq 0$
or $\xi\neq 0$, we have Bombieri’s bound on single variable exponential sums
in finite fields with rational function phases [3] (also see [4, Proposition
2.2]), that is,

(58) $\mathbb{E}^{*}_{y}e_{p}(n_{1}F(y)+\xi R(y))\ll p^{-1/2}.$

Combining Bombieri’s bound, the Cauchy–Schwarz inequality and Parseval’s
identity, we have

(59) $\sum_{n_{1}\neq
0}\widehat{f_{0}}(-n_{1})\widehat{f_{1}}(n_{1})\mathbb{E}^{*}_{y}e_{p}(n_{1}F(y)+\xi
R(y))\ll p^{-1/2}\lVert f_{0}\rVert_{2}\lVert f_{1}\rVert_{2}.$

Note that the main term comes from the term with $n_{1}=0$, that is,

(60) $\widehat{f_{0}}(0)\widehat{f_{1}}(0)\mathbb{E}^{*}_{y}e_{p}(\xi
R(y))=1_{\xi=0}\prod_{i=0}^{1}\mathbb{E}_{z}f_{i}(z)+O\bigl{(}p^{-1/2}\lVert
f_{0}\rVert_{2}\lVert f_{1}\rVert_{2}\bigr{)}.$

This completes the proof. ∎

Next, we show that the counting operator can be controlled by the dual
function.

######  Proposition 5.2.

Let $F(t),G(t)\in\mathbb{Q}(t)$ be rational functions over $\mathbb{Q}$ and
$f_{0},f_{1},f_{2}:\mathbb{F}_{p}\longrightarrow\mathbb{C}$ be functions.
Define the dual function $\mathcal{D}=\mathcal{D}_{f_{1},f_{2}}$ by

$\mathcal{D}(x)=\mathbb{E}^{*}_{y}f_{1}(x+F(y))f_{2}(x+G(y)).$

Then, we have

$\lvert\mathbb{E}^{*}_{x,y}f_{0}(x)f_{1}(x+F(y))f_{2}(x+G(y))\rvert\leq\lVert\widehat{f_{0}}\rVert_{l^{1}}\lVert\widehat{\mathcal{D}}\rVert_{l^{\infty}}.$

###### Proof.

By the definition of dual function and Parseval’s identity, we have

(61)
$\lvert\mathbb{E}^{*}_{x,y}f_{0}(x)f_{1}(x+F(y))f_{2}(x+G(y))\rvert=\lvert\mathbb{E}_{x}f_{0}(x)\mathcal{D}(x)\rvert=\Bigl{\lvert}\sum_{m}\widehat{f_{0}}(-m)\widehat{\mathcal{D}}(m)\Bigr{\rvert},$

which is bounded by
$\lVert\widehat{f_{0}}\rVert_{l^{1}}\lVert\widehat{\mathcal{D}}\rVert_{l^{\infty}}$
by applying Hölder’s inequality. ∎

The following proposition says that the dual function is controlled by shorter
progressions.

######  Proposition 5.3.

Let $F(t),G(t)\in\mathbb{Q}(t)$ be rational functions over $\mathbb{Q}$ such
that $F(t),G(t)$ and the constant function $1$ are linearly independent over
$\mathbb{Q}$, and $f_{1},f_{2}:\mathbb{F}_{p}\longrightarrow\mathbb{C}$ be
functions with $\mathbb{E}_{z}f_{2}(z)=0$. Define the dual function
$\mathcal{D}=\mathcal{D}_{f_{1},f_{2}}$ by

$\mathcal{D}(x)=\mathbb{E}^{*}_{y}f_{1}(x+F(y))f_{2}(x+G(y)).$

Then, we have

$\lVert\widehat{\mathcal{D}}\rVert_{l^{\infty}}\ll p^{-1/2}\lVert
f_{1}\rVert_{2}\lVert f_{2}\rVert_{2}.$

###### Proof.

Note that

(62)
$\begin{split}\widehat{\mathcal{D}}(\xi)=&\mathbb{E}_{x}\mathbb{E}^{*}_{y}f_{1}(x+F(y))f_{2}(x+G(y))e_{p}(-x\xi)\\\
=&\mathbb{E}^{*}_{x,y}f_{1}(x)e_{p}(-x\xi)f_{2}(x+G(y)-F(y))e_{p}(\xi
F(y)).\end{split}$

To finish the proof, apply Proposition 5.1 and note that the main term is $0$
due to the assumption $\mathbb{E}_{z}f_{2}(z)=0$. ∎

Finally, we have all ingredients to finish the proof of Theorem 1.1.

###### Proof of Theorem 1.1.

We will use convenient notation $\Lambda_{F,G}(f_{0},f_{1},f_{2})$ to denote
the counting operator
$\mathbb{E}^{*}_{x,y}f_{0}(x)f_{1}(x+F(y))f_{2}(x+G(y))$. First, we start with
a preliminary reduction. Let $f_{2}^{\prime}=f_{2}-\mathbb{E}_{z}f_{2}(z)$ be
the balanced version of the function $f_{2}$, so that the mean of
$f_{2}^{\prime}$ is $0$. Note that we have

(63)
$\begin{split}\Lambda_{F,G}(f_{0},f_{1},f_{2})&=\Lambda_{F,G}(f_{0},f_{1},\mathbb{E}_{z}f_{2}(z))+\Lambda_{F,G}(f_{0},f_{1},f_{2}^{\prime})\\\
&=\Bigl{(}\mathbb{E}_{z}f_{2}(z)\Bigr{)}\mathbb{E}^{*}_{x,y}f_{0}(x)f_{1}(x+F(y))+\Lambda_{F,G}(f_{0},f_{1},f_{2}^{\prime})\\\
&=\prod_{i=0}^{2}\mathbb{E}_{z}f_{i}(z)+O\bigl{(}p^{-1/2}\lVert
f_{0}\rVert_{2}\lVert f_{1}\rVert_{2}\lVert
f_{2}\rVert_{2}\bigr{)}+\Lambda_{F,G}(f_{0},f_{1},f_{2}^{\prime}),\end{split}$

where the last equality follows from Proposition 5.1. It remains to bound
$\Lambda_{F,G}(f_{0},f_{1},f_{2}^{\prime})$.

Next, we would construct a Fourier level set decomposition of $f_{0}$. Let
$\varepsilon>0$ be a parameter to be chosen later, define

(64)
$g(\xi)=\begin{cases}\widehat{f_{0}}(\xi),&\lvert\widehat{f_{0}}(\xi)\rvert>\frac{\lVert
f_{0}\rVert_{2}}{\varepsilon p^{1/2}}\\\ 0,&\text{otherwise}\end{cases}$

and

(65)
$h(\xi)=\begin{cases}\widehat{f_{0}}(\xi),&\lvert\widehat{f_{0}}(\xi)\rvert\leq\frac{\lVert
f_{0}\rVert_{2}}{\varepsilon p^{1/2}}\\\ 0,&\text{otherwise}\end{cases}.$

Note that we have $\widehat{f_{0}}=g+h$. By Parseval’s identity, we have
$l^{2}$-bounds $\lVert g\rVert_{l^{2}}\leq\lVert f_{0}\rVert_{2}$ and $\lVert
h\rVert_{l^{2}}\leq\lVert f_{0}\rVert_{2}$. Moreover, we have good
$l^{1}$-bound for $g$, that is,

(66) $\lVert g\rVert_{l^{1}}=\sum_{\xi}\lvert g(\xi)\rvert\leq\sum_{\xi}\lvert
g(\xi)\rvert\cdot\lvert\widehat{f_{0}}(\xi)\rvert\biggl{(}\frac{\lVert
f_{0}\rVert_{2}}{\varepsilon p^{1/2}}\biggr{)}^{-1}\leq\varepsilon
p^{1/2}\lVert f_{0}\rVert_{2}.$

Furthermore, we have good $l^{4}$-bound for $h$, that is,

(67) $\lVert h\rVert_{l^{4}}\leq\lVert h\rVert_{l^{\infty}}^{1/2}\lVert
h\rVert_{l^{2}}^{1/2}\leq\biggl{(}\frac{\lVert f_{0}\rVert_{2}}{\varepsilon
p^{1/2}}\biggr{)}^{1/2}\lVert
h\rVert_{l^{2}}^{1/2}\leq\varepsilon^{-1/2}p^{-1/4}\lVert f_{0}\rVert_{2}.$

Decompose $f_{0}=\check{g}+\check{h}$, where $\check{g},\check{h}$ are the
inverse Fourier transform of $g,h$ respectively.

To finish the proof, we would use dual function control to bound
$\Lambda_{F,G}(\check{g},f_{1},f_{2}^{\prime})$ and algebraic geometry PET
induction to bound $\Lambda_{F,G}(\check{h},f_{1},f_{2}^{\prime})$. By
Proposition 5.2, Proposition 5.3 and the $l^{1}$-bound for $g$, we have

(68) $\lvert\Lambda_{F,G}(\check{g},f_{1},f_{2}^{\prime})\rvert\leq\lVert
g\rVert_{l^{1}}\lVert\widehat{\mathcal{D}}\rVert_{l^{\infty}}\ll
p^{-1/2}\lVert g\rVert_{l^{1}}\lVert f_{1}\rVert_{2}\lVert
f_{2}\rVert_{2}\leq\varepsilon\lVert f_{0}\rVert_{2}\lVert
f_{1}\rVert_{2}\lVert f_{2}\rVert_{2}.$

By Theorem 3.1, Theorem 4.1 and the $l^{4}$-bound for $h$, we have

(69)
$\lvert\Lambda_{F,G}(\check{h},f_{1},f_{2}^{\prime})\rvert\ll\lVert\check{h}\rVert_{2}^{1/2}\lVert
h\rVert_{l^{4}}^{1/2}\lVert f_{1}\rVert_{2}\lVert
f_{2}\rVert_{2}\leq\varepsilon^{-1/4}p^{-1/8}\lVert f_{0}\rVert_{2}\lVert
f_{1}\rVert_{2}\lVert f_{2}\rVert_{2}.$

Hence, we have

(70)
$\begin{split}\Lambda_{F,G}(f_{0},f_{1},f_{2}^{\prime})&=\Lambda_{F,G}(\check{g},f_{1},f_{2}^{\prime})+\Lambda_{F,G}(\check{h},f_{1},f_{2}^{\prime})\\\
&\ll\varepsilon\lVert f_{0}\rVert_{2}\lVert f_{1}\rVert_{2}\lVert
f_{2}\rVert_{2}+\varepsilon^{-1/4}p^{-1/8}\lVert f_{0}\rVert_{2}\lVert
f_{1}\rVert_{2}\lVert f_{2}\rVert_{2}.\end{split}$

Choosing $\varepsilon=p^{-1/10}$ will optimize the result and complete the
proof. ∎

## References

  * [1] V. Bergelson and A. Leibman  “Polynomial extensions of van der Waerden’s and Szemerédi’s theorems”  In _J. Amer. Math. Soc._ 9.3, 1996, pp. 725–753  DOI: 10.1090/S0894-0347-96-00194-4
  * [2] Thomas F. Bloom and James Maynard  “A new upper bound for sets with no square differences”  In _Compositio Mathematica_ 158.8, 2022, pp. 1777–1798  DOI: 10.1112/S0010437X22007679
  * [3] Enrico Bombieri  “On exponential sums in finite fields”  In _Amer. J. Math._ 88, 1966, pp. 71–105  DOI: 10.2307/2373048
  * [4] J. Bourgain and M.-C. Chang  “Nonlinear Roth type theorems in finite fields”  In _Israel J. Math._ 221.2, 2017, pp. 853–867  DOI: 10.1007/s11856-017-1577-9
  * [5] Dong Dong, Xiaochun Li and Will Sawin  “Improved estimates for polynomial Roth type theorems in finite fields”  In _J. Anal. Math._ 141.2, 2020, pp. 689–705  DOI: 10.1007/s11854-020-0113-8
  * [6] Étienne Fouvry, Emmanuel Kowalski and Philippe Michel  “A study in sums of products”  In _Philos. Trans. Roy. Soc. A_ 373.2040, 2015, pp. 2014030926  DOI: 10.1098/rsta.2014.0309
  * [7] W.. Gowers  “A new proof of Szemerédi’s theorem”  In _Geom. Funct. Anal._ 11.3, 2001, pp. 465–588  DOI: 10.1007/s00039-001-0332-9
  * [8] W.. Gowers  “A new proof of Szemerédi’s theorem for arithmetic progressions of length four”  In _Geom. Funct. Anal._ 8.3, 1998, pp. 529–551  DOI: 10.1007/s000390050065
  * [9] Hans Grauert and Reinhold Remmert  “Coherent analytic sheaves” 265, Grundlehren der mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences]  Springer-Verlag, Berlin, 1984, pp. xviii+249  DOI: 10.1007/978-3-642-69582-7
  * [10] Robin Hartshorne  “Algebraic geometry” No. 52, Graduate Texts in Mathematics  Springer-Verlag, New York-Heidelberg, 1977, pp. xvi+496
  * [11] Borys Kuca  “Further bounds in the polynomial Szemerédi theorem over finite fields”  In _Acta Arith._ 198.1, 2021, pp. 77–108  DOI: 10.4064/aa200218-9-6
  * [12] Serge Lang and André Weil  “Number of points of varieties in finite fields”  In _Amer. J. Math._ 76, 1954, pp. 819–827  DOI: 10.2307/2372655
  * [13] James Leng  “A Quantitative Bound For Szemerédi’s Theorem for a Complexity One Polynomial Progression over $\mathbb{Z}/N\mathbb{Z}$”, 2022  arXiv:2205.05540 [math.NT]
  * [14] James Leng  “The partition rank vs. analytic rank problem for cyclic groups I. Equidistribution for periodic nilsequences”, 2023  arXiv:2306.13820 [math.NT]
  * [15] Sarah Peluse  “Bounds for sets with no polynomial progressions”  In _Forum of Mathematics, Pi_ 8 Cambridge University Press, 2020, pp. e16  DOI: 10.1017/fmp.2020.11
  * [16] Sarah Peluse  “Finite field models in arithmetic combinatorics – twenty years on”, 2023  arXiv:2312.08100 [math.NT]
  * [17] Sarah Peluse  “On the polynomial Szemerédi theorem in finite fields”  In _Duke Math. J._ 168.5, 2019, pp. 749–774  DOI: 10.1215/00127094-2018-0051
  * [18] Sarah Peluse  “Three-term polynomial progressions in subsets of finite fields”  In _Israel J. Math._ 228.1, 2018, pp. 379–405  DOI: 10.1007/s11856-018-1768-z
  * [19] Sarah Peluse and Sean Prendiville  “A polylogarithmic bound in the nonlinear Roth theorem”  In _Int. Math. Res. Not. IMRN_ , 2022, pp. 5658–5684  DOI: 10.1093/imrn/rnaa261
  * [20] Sarah Peluse and Sean Prendiville  “Quantitative bounds in the nonlinear Roth theorem”, 2022  arXiv:1903.02592 [math.NT]
  * [21] Sarah Peluse, Ashwin Sah and Mehtaab Sawhney  “Effective bounds for Roth’s theorem with shifted square common difference”, 2023  arXiv:2309.08359 [math.NT]
  * [22] Sean Prendiville  “Quantitative bounds in the polynomial Szemerédi theorem: the homogeneous case”  In _Discrete Analysis_ , 2017  DOI: 10.19086/da.1282
  * [23] K.. Roth  “On Certain Sets of Integers”  In _Journal of the London Mathematical Society_ s1-28.1, 1953, pp. 104–109  DOI: https://doi.org/10.1112/jlms/s1-28.1.104
  * [24] The Stacks project authors  “The Stacks project”, https://stacks.math.columbia.edu, 2023
  * [25] E. Szemerédi  “On sets of integers containing k elements in arithmetic progression”  In _Acta Arithmetica_ 27.1, 1975, pp. 199–245  URL: http://eudml.org/doc/205339
  * [26] Terence Tao  “The Lang-Weil Bound”, https://terrytao.wordpress.com/2012/08/31/the-lang-weil-bound/, 2023