# Milnor and Tjurina numbers for an isolated complete intersection singularity

A.J. Parameswaran , Mohit Upmanyu

###### Abstract

This paper aims to prove that given a isolated complete intersection
singularity, the Milnor number will be bounded by a bound depending only on
Tjurina number and dimension of the singularity. The proof uses
A$\mathfrak{m}$AC (introduced in [PU22]) and as with such methods, the bound
is purely existential.

###### Acknowledgements

This work was supported by the Department of Atomic Energy, Government of
India [project no. 12 - R&D - TFR - 5.01 - 0500].

##  1 Introduction

Throughout this paper we fix $A=\mathbb{C}[\\![x_{1},x_{2},...,x_{n+k}]\\!]$
and $\mathfrak{m}$ denote its maximal ideal.

######  Theorem 1.1 ([Liu18]).

Let $X\subset\operatorname{Spec}(A)$ be an isolated hypersurface singularity
of dimension $n$ (i.e k = 1) with the Tjurina number $\tau$. Then
$\mu(X)\leq(n+1)\tau$.

In [DG18], $\mu\leq\frac{4}{3}\tau$ was proposed for the case of plane curves,
which was later proven in [ACABMH20] and [HG19] separately for the case of
irreducible curves. In [Alm21], the bound for all curves was proven. Moreover,
in [Alm21], it is proven that if the Durfee conjecture holds for a
hypersurface $X$ of dimension 2 then, $\mu\leq\frac{3}{2}\tau$.

For complete intersection singularities, a bound of $\mu\leq\frac{4}{3}\tau$
was proved for certain families of curve in [Alm21]. We prove an existential
bound for the case of complete intersection as follows.

######  Theorem 1 (3.5).

Let $X\subset\operatorname{Spec}(A)$ be an isolated complete intersection
singularity of dimension n with Tjurina number $\tau$. Then there
abstractly111We use the word abstractly to indicate that what we give is an
existential argument. More precisely, what we prove is that the invariant is
not infinite so we conclude that a bound exists. exists a constant $c>0$ such
that, $\mu(X)\leq c$. This $c$ depends only on $\tau$ and $n$.

Here $\mu(X)$ refers to the Milnor number, for a definition see [Loo84, 5.11].

##  2 Preliminaries

Let $R$ be a noetherian ring of dimension $d$ with a maximal ideal
$\mathfrak{m}$. Let $I$ be an $\mathfrak{m}$-primary ideal. We will use
$\operatorname{len}(M)$ to denote length of a module $M$.

######  Definition 2.1.

Define Samuel function $\chi_{R}^{I}(t):=\operatorname{len}(R/I^{t+1})$. If
the maximal ideal is clear from the context and $I=\mathfrak{m}$, then we will
drop the $\mathfrak{m}$ in the notation. We take the definition of
multiplicity as defined in [Mat89, Formula 14.1].

$e(R,I)=\lim_{t\to\infty}d!\frac{\operatorname{len}(R/I^{t+1})}{t^{d}}$

We will use the following fundamental result of dimension theory.

######  Lemma 2.2.

Let $R$ be a noetherian ring with Krull dimension $d$ with maximal ideal
$\mathfrak{m}$. Let $I$ be an $\mathfrak{m}$-primary ideal. Then for $t$
sufficiently large, $\chi^{I}_{R}(t)=p(t)$, where $p(t)$ is a rational
polynomial of degree $d$ (called Samuel polynomial). Furthermore, the
coefficient of $x^{d}$ is $e(R,I)/d!$.

###### Proof.

For the degree of polynomial see [Mat89, Theorem 13.4]. For the multiplicity
as coefficient see [Mat89, Section 14]∎

######  Lemma 2.3.

Let $R$ be a noetherian ring of dimension $d$ with a maximal ideal
$\mathfrak{m}$ such that, $R/\mathfrak{m}$ is infinite. Let $\chi^{I}_{R}(t)$
be the Samuel function of $R$ w.r.t. $I$ which is an $\mathfrak{m}$-primary
ideal. Then

$\chi^{I}_{R}(t)\geq{t+d\choose d}$ (1)

Let $e$ be the multiplicity of $R$ w.r.t $\mathfrak{m}$. Then we can obtain
better bounds as follows.
For $t\leq e-1$,

$\chi^{I}_{R}(t)\geq{t+d+1\choose d+1}$ (2)

For $t\geq e-1$,

$\chi^{I}_{R}(t)\geq\sum_{i=0}^{d}(-1)^{i}{e\choose i+1}{t+d-i\choose
d-i}=e{t+d\choose d}+\text{lower order terms (of $t$)}$ (3)

Note that ${t+d\choose d}$ is the Samuel polynomial of a regular local ring of
dimension $d$

###### Proof.

In [PU22, lemma 2.6], this lemma is proven for $I=\mathfrak{m}$, The same
proof generalizes directly for any $\mathfrak{m}$-primary ideal $I$. ∎

######  Definition 2.4.

Let $\mathbb{A}^{k}=\operatorname{Spec}\mathbb{C}[x_{1},...,x_{k}]$. Let
$f=(f_{1},...,f_{k}):SpecA\to\mathbb{A}^{k}$ and let $X:=f^{-1}(0)$ considered
as a closed subscheme of $\operatorname{Spec}(A)$. Define
$\Sigma^{\alpha}_{X}$ as the scheme of “all points $p\in X$ such that,
$\dim(T_{p}(X))>\alpha$”, whose scheme structure is given as follows.

$\Sigma^{\alpha}_{X}:=X\cap\operatorname{Spec}\frac{A}{J_{{n+k}-\alpha}(f)}=\operatorname{Spec}\frac{A}{I(X)+J_{{n+k}-\alpha}(f)}$

where $J_{{n+k}-\alpha}(f)$ is the ideal generated by
${n+k}-\alpha\times{n+k}-\alpha$ minors of the Jacobian matrix
$(\frac{\partial f_{i}}{\partial x_{j}})$.

As seen in [Loo84, 4.D], $J_{{n+k}-\alpha}(f)$ corresponds to the Fitting
ideal of the module
$\Omega_{f}:=Coker(A^{k}=(f^{*}(\Omega_{\mathbb{A}^{k}}))\xrightarrow{Jac(f_{i})}A^{n+k}(=\Omega_{\operatorname{Spec}A})$,
so the scheme $\Sigma^{\alpha}_{X}$ does not depend on the choice of
generators of the ideal.

We define the critical locus of the map $f$ as the scheme of “all points
$p\in\operatorname{Spec}A$ such that, $rank(Df_{p}(X))<k$”, whose scheme
structure is given as follows.

$C_{f}:=\operatorname{Spec}\frac{A}{J_{k}(f)}$

######  Definition 2.5.

Let $f=(f_{1},...,f_{k}):SpecA\to\mathbb{A}^{k}$. Let $X$ be the closed
subscheme of $\operatorname{Spec}(A)$ defined as $f^{-1}(0)$. Then $X$ is
called an isolated compete intersection singularity (icis) if
$\dim(\Sigma^{n}_{X})=0$.

Following [Loo84, 6.B], if $X$ is an icis then we define $T^{1}_{X}$ as the
cokernel of $T(\operatorname{Spec}A)|_{X}\to f^{*}T(\mathbb{A}^{k})|_{X}$.
Since the support of $T^{1}_{X}$ is on singular locus which is of dimension 0.
So we define the Tjurina number $\tau(X)$ as the
$\operatorname{len}(T^{1}_{X})$.

######  Remark 2.6.

Note that as defined above, $(f_{1},...,f_{k})$ define a complete intersection
iff $(f_{1},...,f_{k})$ form a regular sequence. This we will take as a
convention.

Here we define algebraic $\mathfrak{m}$-adically closed subsets introduced in
[PU22], on which the theory is built.

######  Definition 2.7.

A subset $P\subset A^{k}$ (Cartesian product) is said to be an algebraic
$\mathfrak{m}$-adically closed (A$\mathfrak{m}$AC) class, if
$P=\underleftarrow{\lim}P_{i}$ where $P_{i}$ are constructible subsets of
$(A/\mathfrak{m}^{i})^{k}$ (in Zariski topology).

If $U$ is a set complement of an A$\mathfrak{m}$AC then we refer to it as an
algebraic $\mathfrak{m}$-adically open (A$\mathfrak{m}$AO) class.

Finally, when we say ”Let $P=\underleftarrow{\lim}P_{i}$ be an
A$\mathfrak{m}$AC class in $A^{k}$” , It will be assumed that
$P_{i}\subset(A/\mathfrak{m}^{i})^{k}$ and $P\subset A^{k}$

We restate some properties of A$\mathfrak{m}$AC proved in [PU22]

######  Lemma 2.8.

[PU22][lemma 3.6] Let $P^{1}=\underleftarrow{\lim}P^{1}_{i}$,
$P^{2}=\underleftarrow{\lim}P^{2}_{i}$,…,
$P^{j}=\underleftarrow{\lim}P^{j}_{i}$ ,… be an infinite sequence of
A$\mathfrak{m}$AC classes in $A^{k}$.

  1. 1.

A$\mathfrak{m}$AC classes are closed under finite union. ($P^{1}\cup
P^{2}=\underleftarrow{\lim}P^{1}_{i}\cup P^{2}_{i}$)

  2. 2.

A$\mathfrak{m}$AC classes are closed under finite intersections. ($P^{1}\cap
P^{2}=\underleftarrow{\lim}P^{1}_{i}\cap P^{2}_{i}$)

  3. 3.

A$\mathfrak{m}$AC classes are closed under countable intersections.
($\bigcap_{j}P^{j}=\underleftarrow{\lim}\bigcap_{j\leq i}P^{j}_{i}$ )

######  Lemma 2.9.

Let $P^{1}=\underleftarrow{\lim}P^{1}_{i}$,
$P^{2}=\underleftarrow{\lim}P^{2}_{i}$,…,
$P^{j}=\underleftarrow{\lim}P^{j}_{i}$ ,… be an infinite sequence of
A$\mathfrak{m}$AC classes in $A^{k}$, Then countable intersection of $P_{i}$
is non-empty iff each finite intersection of $P_{i}$ is non-empty.

###### Proof.

By previous lemma $\bigcap_{j}P^{j}=\underleftarrow{\lim}\bigcap_{j\leq
i}P^{j}_{i}$

Now by non emptyness lemma [PU22][lemma 3.4]

$\bigcap_{j}P^{j}\neq\emptyset\iff\underleftarrow{\lim}\bigcap_{j\leq
i}P^{j}_{i}\neq\emptyset\forall i\iff\bigcap_{j\leq
i}P^{j}\neq\emptyset\forall i$

∎

##  3 Main Theorem

Before we get to the Theorem we would need to prove the following lemma.

######  Lemma 3.1.

Let $f=(f_{1},...,f_{k}):\operatorname{Spec}A\to\mathbb{A}^{k}$. Let
$X:=f^{-1}(0)\subset\operatorname{Spec}(A)$ be an isolated complete
intersection singularity. Then $\mu(X)\leq$ the multiplicity of critical locus
w.r.t. $\langle f_{1},...,f_{k}\rangle$

###### Proof.

We use formula from [Loo84, 5.11.a], to conclude that

$\mu(X)\leq\operatorname{len}(\frac{A}{J_{k}(f)+\langle
g_{1},...,g_{k-1}\rangle})$

Here $g_{1},...,g_{k-1}$ can be any linear combinations of $f_{1},...,f_{k}$.

Now By [Loo84, 4.4] $\mathcal{O}_{C_{f}}$ is finite and Cohen-Macaulay over
$\mathbb{C}[\\![f_{1},...,f_{k}]\\!]$ of dimension $k-1$. So for $k-1$ general
linear combinations $g_{1},...,g_{k-1}$, $\mathcal{O}_{C_{f}}$ is finite and
Cohen-Macaulay over $\mathbb{C}[\\![g_{1},...,g_{k-1}]\\!]$. Now the
projective dimension of $\mathcal{O}_{C_{f}}$ over
$\mathbb{C}[\\![g_{1},...,g_{k-1}]\\!]$ is
$\dim\mathbb{C}[\\![g_{1},...,g_{k-1}]\\!]-depth(\mathcal{O}_{C_{f}})$ =
$k-1-\dim\mathcal{O}_{C_{f}}$ = $0$. So $\mathcal{O}_{C_{f}}$ is a free module
over $\mathbb{C}[\\![g_{1},...,g_{k-1}]\\!]$

We can also assume $e(\mathcal{O}_{C_{f}},\langle f_{1},...,f_{k}\rangle)$ =
$e(\mathcal{O}_{C_{f}},\langle g_{1},...,g_{k-1}\rangle)$, using [Mat89,
Theorem 14.14].

Since $\mathcal{O}_{C_{f}}$ is a free module over
$\mathbb{C}[\\![g_{1},...,g_{k-1}]\\!]$.

$e(\mathcal{O}_{C_{f}},\langle
g_{1},...,g_{k-1}\rangle)=rank(\mathcal{O}_{C_{f}})=\operatorname{len}\frac{\mathcal{O}_{C_{f}}}{\langle
g_{1},...,g_{k-1}\rangle}=\operatorname{len}(\frac{A}{J_{k}(f)+\langle
g_{1},...,g_{k-1}\rangle})$

.

We can finally combine all the above to get

$\mu(X)\leq\operatorname{len}(\frac{A}{J_{k}(f)+\langle
g_{1},...,g_{k-1}\rangle})=e(\mathcal{O}_{C_{f}},\langle
f_{1},...,f_{k}\rangle)$

∎

Using the observation that $\mu(X)\leq$ the multiplicity of critical locus
w.r.t. $\langle f_{1},...,f_{\tau}\rangle$, bounding $\mu$ reduces to bounding
$e(\mathcal{O}_{C_{f}},\langle f_{1},...,f_{k}\rangle)$. For bounding
$e(\mathcal{O}_{C_{f}},\langle f_{1},...,f_{k}\rangle)$ we require the
following A$\mathfrak{m}$AC classes. (The proof of the fact that they are
A$\mathfrak{m}$AC is given in section 4)

######  Example 3.2.

  1. 1.

$T(r)=\\{(f_{1},...,f_{k})\in A^{k}\ |\ (f_{1},...,f_{k})\text{ define an icis
and }\tau(f_{1},...,f_{k})=r\\}$ (for proof see 4.1)

  2. 2.

$D(d):=\\{(f_{1},...,f_{k})\in A^{k}\ |\ \dim(\Sigma^{n}_{X})\geq d\\}$ (For
definition of $\Sigma^{n}_{X}$ see 2.4.) (for proof see 4.2)

######  Remark 3.3.

One notes that $(f_{1},...,f_{k})\in D(1)$ iff $(f_{1},...,f_{k})$ do not
define an icis. The fact that we have a characterization of not icis as an
A$\mathfrak{m}$AC will prove very useful.

  3. 3.

$\displaystyle\begin{split}C(e):=\\{(f_{1},...,f_{k})\in A^{k}\ |&\
\text{either }\dim(\Sigma^{n}_{X})\geq 1\text{ or }(f_{1},...,f_{k})\text{
define an icis and }\\\ &e(\mathcal{O}_{C_{f}},\langle
f_{1},...,f_{k}\rangle)\geq e\\}\text{ (for proof see \ref{critical
AmAC})}\end{split}$

######  Theorem 3.4.

Let $f=(f_{1},...,f_{k}):\operatorname{Spec}A\to\mathbb{A}^{k}$. Let
$X:=f^{-1}(0)\subset\operatorname{Spec}(A)$ be an isolated complete
intersection singularity with Tjurina number $\tau$. Then there abstractly
exists a constant $c>0$ such that, the multiplicity of $C_{f}$ w.r.t. $\langle
f_{1},...,f_{k}\rangle$ $\leq c$. This $c$ depends only on $\tau$ and $n$

###### Proof.

Note That $D(1)=\bigcap_{e}C(e)$.

For this proof we fix $k=\tau$. Since Tjurina number is the dimension of base
of a mini-versal deformation [Loo84], we see that all possible icis with
Tjurina number $\tau$ must occur within codimension $k=\tau$. As a consequence
$T(\tau)$ is a collection of all possible icis which satisfy the hypothesis of
the Theorem (albeit with repetition up to isomorphism). (Using this we have
shown the bound does not depend on $k$)

Assume for a contradiction that the multiplicity of $C_{f}$ w.r.t. $\langle
f_{1},...,f_{\tau}\rangle$ is not bounded on the set $T(\tau)$.

$\displaystyle\begin{split}&T(\tau)\cap C(e)\neq\emptyset\quad\forall e\text{
sufficiently larger than }\tau\\\
\implies&T(\tau)\cap\bigcap_{e}C(e)\neq\emptyset\quad\text{ (by Lemma
\ref{countable intersection lemma})}\\\ \implies&T(\tau)\cap
D(1)\neq\emptyset\end{split}$

This contradicts the fact that any $(f_{1},...,f_{\tau})\in T(\tau)$ can only
define icis. ∎

######  Theorem 3.5.

Let $X\subset\operatorname{Spec}(A)$ be an isolated complete intersection
singularity of dimension n with Tjurina number $\tau$. Then there abstractly
exists a constant $c>0$ such that, $\mu(X)\leq c$. This $c$ depends only on
$\tau$ and $n$.

###### Proof.

We use 3.1 to conclude $\mu\leq$ the multiplicity of critical locus w.r.t.
$\langle f_{1},...,f_{\tau}\rangle$, so the result follows from 3.4 ∎

##  4 Proof of Examples

Here we give proves of the examples stated and used in Theorem 3.4

######  Lemma 4.1.

$T(r)$ (defined in 3.2.1) is both A$\mathfrak{m}$AC and A$\mathfrak{m}$AO

###### Proof.

We look at $B_{r+2}=(A/\mathfrak{m}^{r+2})^{k}$ regarded as an affine space.
Define $\pi:A^{k}\to B_{r+2}$ to be the canonical quotient map.

Using semi continuity of Tjurina number,
$\overline{T}:=\\{(\overline{f}_{1},...,\overline{f}_{k})\in B_{r+2}\ |\
\operatorname{len}(T^{1}_{X}/\mathfrak{m}^{r+1}T^{1}_{X})=r\\}$ is a Zariski
constructible subset.

We will prove that $T(r)=\pi^{-1}(\overline{T})$

First, let $(f_{1},...,f_{k})\in T(r)$. Since
$\operatorname{len}(T^{1}_{X})=r$, $\mathfrak{m}^{r}T^{1}_{X}=0$, so
$T^{1}_{X}=T^{1}_{X}/\mathfrak{m}^{r+1}T^{1}_{X}$, so
$\pi(f_{1},...,f_{k})\in\overline{T}$

Let $(f_{1},...,f_{k})\in\pi^{-1}(\overline{T})$.

$\displaystyle\begin{split}&\operatorname{len}(\frac{T^{1}_{X}}{\mathfrak{m}^{r+1}T^{1}_{X}})=r\\\
\implies&\mathfrak{m}^{r}(\frac{T^{1}_{X}}{\mathfrak{m}^{r+1}T^{1}_{X}})=0\\\
\implies&\frac{\mathfrak{m}^{r}T^{1}_{X}}{\mathfrak{m}^{r+1}T^{1}_{X}}=0\\\
\implies&\mathfrak{m}^{r}T^{1}_{X}=\mathfrak{m}^{r+1}T^{1}_{X}\\\
\implies&\mathfrak{m}^{r}T^{1}_{X}=0\text{ (By Nakyama lemma)}\\\
\implies&T^{1}_{X}=T^{1}_{X}/\mathfrak{m}^{r+1}T^{1}_{X}\\\ \end{split}$

So we can conclude $\operatorname{len}(T^{1}_{X})=r$, hence
$(f_{1},...,f_{k})\in T(r)$. ∎

######  Lemma 4.2.

$D(d)$ (defined in 3.2.3) is an A$\mathfrak{m}$AC.

###### Proof.

We will prove that $D(d)=\underleftarrow{\lim}D_{i}(d)$, where $D_{i}(d)$ is
defined as follows.

$D_{r}(d):=\\{(\overline{f}_{1},...,\overline{f}_{k})\in(A/\mathfrak{m}^{r})^{k}\
|\
\operatorname{len}(\mathcal{O}_{\Sigma^{n}_{X}}/\mathfrak{m}^{t+1})\geq{t+d\choose
d},\ \forall\ t\leq r-2\\}$

One sees that, $D_{r}(d)$ is a Zariski constructible subset. (One argument can
be seen in [PU22][lemma 2.9].)

$D(d)\subset\underleftarrow{\lim}D_{i}(d)$ because if
$\dim(\Sigma^{n}_{X})\geq r$ then
$(\overline{f}_{1},...,\overline{f}_{k})=(f_{1}+\mathfrak{m}^{r},...,f_{k}+\mathfrak{m}^{r})$
satisfies the condition of $D_{r}(d)$ by lemma 2.3(1).

Let $(f_{1},...,f_{k})\in A$ and $(f_{1},...,f_{k})\not\in D(d)$, then
$\operatorname{len}(\mathcal{O}_{\Sigma^{n}_{X}}/\mathfrak{m}^{t+1})$ is a
polynomial of degree $<d$ (lemma 2.2). So for large $t$,
$\operatorname{len}(\mathcal{O}_{\Sigma^{n}_{X}}/\mathfrak{m}^{t+1})<{t+d\choose
d}$ which is a higher degree polynomial. So
$(f_{1},...,f_{k})\not\in\underleftarrow{\lim}D_{i}(d)$ ∎

######  Lemma 4.3.

$C(e)$ (defined in 3.2.2) is an A$\mathfrak{m}$AC

###### Proof.

$\displaystyle\begin{split}C_{r}:=\\{(\overline{f}_{1},...,\overline{f}_{k})\in(A/\mathfrak{m}^{r})^{k}\
&|\ \operatorname{len}(\frac{\mathcal{O}_{C_{F}}}{\mathfrak{m}^{e{t+k-1\choose
k-1}}+\langle
f_{1},...,f_{k}\rangle^{t+1}})\geq\sum_{i=0}^{d}(-1)^{i}{e\choose
i+1}{t+k-1-i\choose k-1-i},\\\ &\forall\ t\text{ such that, }e-1\leq t\text{
and }e{t+k-1\choose k-1}\leq r-2\\}\end{split}$

One sees that, $C_{r}$ is a Zariski constructible subset. (Similar to the
previous Lemma)

We will prove that $C(e)=\underleftarrow{\lim}C_{i}\cup D(1)$.

$C(e)\subset\underleftarrow{\lim}C_{i}\cup D(1)$ because if
$(f_{1},...,f_{k})\in C(e)$ and $(f_{1},...,f_{k})\not\in D(1)$ then
$(f_{1},...,f_{k})$ is an icis, and so
$(\overline{f}_{1},...,\overline{f}_{k})=(f_{1},...,f_{k})+\mathfrak{m}^{r}$
satisfies the condition of $C_{i}$ by lemma 2.3(3).

First Note that from definition of $C(e)$ is $D(1)\subset C(e)$

Let $(f_{1},...,f_{k})\in A^{k}$, and
$(f_{1},...,f_{k})\in\underleftarrow{\lim}C(e)-D(1)$ then $(f_{1},...,f_{k})$
defines an icis. Then $\operatorname{len}(\frac{C_{f}}{\langle
f_{1},...,f_{k}\rangle^{t+1}})$ makes sense and is a polynomial of degree $>d$
(in this case it is not an icis and is in $D(1)$) or if degree $=d$ the
highest coefficient $\geq e/d!$ (lemma 2.2 and definition of multiplicity). So
Critical locus $C_{f}$ has multiplicity $\geq e$. So $(f_{1},...,f_{k})\in
C(e)$ ∎

## References

  * [ACABMH20] Maria Alberich-Carraminana, Patricio Almiron, Guillem Blanco, and Alejandro Melle-Hernandez.  The minimal tjurina number of irreducible germs of plane curve singularities, 2020.
  * [Alm21] Patricio Almiron.  On the quotient of milnor and tjurina numbers for two-dimensional isolated hypersurface singularities, 2021.
  * [DG18] Alexandru Dimca and Gert-Martin Greuel.  On 1-forms on isolated complete intersection curve singularities.  J. Singul., 18:114–118, 2018.
  * [HG19] Marcelo E. Hernandes and Yohann Genzmer.  On the saito’s basis and the Tjurina number for plane branches, 2019\.
  * [Liu18] Yongqiang Liu.  Milnor and Tjurina numbers for a hypersurface germ with isolated singularity, 2018.
  * [Loo84] E. J. N. Looijenga.  Isolated singular points on complete intersections, volume 77 of London Mathematical Society Lecture Note Series.  Cambridge University Press, Cambridge, 1984.
  * [Mat89] Hideyuki Matsumura.  Commutative ring theory, volume 8 of Cambridge Studies in Advanced Mathematics.  Cambridge University Press, Cambridge, second edition, 1989.  Translated from the Japanese by M. Reid.
  * [PU22] A. J. Parameswaran and Mohit Upmanyu.  Generalization of gurjar’s hyperplane section theorem to arbitrary analytic varieties and a$\mathbb{m}$ac classes, 2022.


# Competitive Retrieval: Going Beyond the Single Query

Haya Nachimovsky<EMAIL_ADDRESS>TechnionIsrael ,  Moshe
Tennenholtz<EMAIL_ADDRESS>TechnionIsrael ,  Fiana Raiber
<EMAIL_ADDRESS>Yahoo ResearchIsrael and  Oren Kurland
<EMAIL_ADDRESS>TechnionIsrael

###### Abstract.

Previous work on the competitive retrieval setting focused on a single-query
setting: document authors manipulate their documents so as to improve their
future ranking for a given query. We study a competitive setting where authors
opt to improve their document’s ranking for multiple queries. We use game
theoretic analysis to prove that equilibrium does not necessarily exist. We
then empirically show that it is more difficult for authors to improve their
documents’ rankings for multiple queries with a neural ranker than with a
state-of-the-art feature-based ranker. We also present an effective approach
for predicting the document most highly ranked in the next induced ranking.

##  1\. Introduction

In the Web search setting, publishers (authors) of documents are sometimes
incentivized to have their documents highly ranked by search engines for some
queries. The basic motivation is the fact that documents at top ranks often
attract most user engagement (Joachims et al., 2005). As a result of their
ranking incentives, publishers might modify their documents in response to
rankings induced by a search engine so as to improve their future rankings.
The search setting then becomes competitive (Kurland and Tennenholtz, 2022)
with corpus dynamics driven in part by ranking-incentivized document
modifications. These modifications are often referred to as search engine
optimization (SEO) (Gyöngyi and Garcia-Molina, 2005).

Lately, there has been a growing body of work on studying ranking-incentivized
document modifications, and more generally, on competitive retrieval (Ben-
Basat et al., 2017; Raifer et al., 2017; Goren et al., 2018, 2020, 2021; Liu
et al., 2022; Song et al., 2022; Wang et al., 2022; Wu et al., 2022; Chen et
al., 2023a, b; Liu et al., 2023; Vasilisky et al., 2023; Wu et al., 2023).
This renewed interest in the often dubbed adversarial retrieval setting
(Castillo and Davison, 2010) can be attributed in part to a large volume of
work on adversarial machine learning in the era of neural network models
(Szegedy et al., 2014; Huang et al., 2017; Jia and Liang, 2017; Papernot et
al., 2017; Xie et al., 2017; Kreuk et al., 2018; Jia et al., 2019; Zhang et
al., 2019).

The focus of some recent work on competitive retrieval is on algorithmic
attacks on ranking functions; e.g., substituting terms with their synonyms
(Goren et al., 2021; Liu et al., 2022; Song et al., 2022; Wang et al., 2022;
Wu et al., 2022; Chen et al., 2023a, b; Liu et al., 2023). There is also an
emerging thread of work on analyzing, and improving, the robustness of ranking
functions to ranking-incentivized document modifications (Ben-Basat et al.,
2017; Goren et al., 2018; Chen et al., 2023a; Vasilisky et al., 2023; Wu et
al., 2023). Another type of work, which is our focus in this paper, is
analyzing human strategies of ranking-incentivized document manipulations
(Raifer et al., 2017) and their resultant effect on the corpus (Ben-Basat et
al., 2017; Raifer et al., 2017; Goren et al., 2021). For example, it was shown
theoretically and empirically that a prominent (worthwhile) strategy of
publishers is to mimic content in documents that were highly ranked in the
past for queries of interest (Raifer et al., 2017), which results in an
herding effect (Goren et al., 2021).

Almost all recent work on competitive retrieval has focused on a single-query
setting; i.e., assuming a publisher modifies her document to improve its
ranking for a single query111We discuss in Section 2 the two exceptions:
algorithmic attacks for topically-related queries (Liu et al., 2023) and the
corpus effect of modifying documents for two queries (topics) (Ben-Basat et
al., 2017). Algorithmic attacks are outside the scope of this paper.. However,
publishers often opt to have their document highly ranked for multiple
queries; e.g., those representing a topic in the document. Accordingly, in
this paper we present the first — to the best of our knowledge — theoretical
and empirical study of ranking-incentivized document-modification strategies
of (human) publishers opting to promote their documents for multiple queries.

Using game theoretic analysis we show that when publishers modify their
documents to improve their rankings for multiple queries, there is not
necessarily an equilibrium. This result implies instability: a potentially
endless document modification race which can have negative effects on users;
e.g., documents at top ranks consistently change not due to pure editorial
considerations but rather due to ranking incentives. This theoretical result
is in contrast to recent findings about the existence of an equilibrium when
documents are modified for a single query (Raifer et al., 2017).

We then fully characterize the conditions for an equilibrium in the multiple-
queries setting for a basic family of ranking functions. Furthermore, we show
that best response dynamics (Nisan et al., 2007) does not necessarily converge
to an equilibrium in case it exists. Best response dynamics is a process where
each publisher modifies her document to attain the best possible ranking given
the documents written by all other publishers. The implication is that
reaching a steady state in case it is achievable might call for an external
intervention; e.g., of the search engine.

We next present an empirical study of document manipulation strategies applied
by publishers that opt for improved ranking for multiple queries representing
the same information need (topic). The study is based on ranking competitions
we organized between students. Our competitions were inspired by those
organized for the single-query setting (Raifer et al., 2017; Goren et al.,
2018, 2021). The competitions we report differ from previous competitions
(Raifer et al., 2017; Goren et al., 2018, 2021) not only by the virtue of
having publishers modify their documents for multiple queries rather than a
single one, but also by two additional aspects. First, our competitions are
the first to employ not only a feature-based learning-to-rank (LTR) method
(Liu, 2011) as the ranking function, but also a neural ranker (Lin et al.,
2020). Second, in some of our competitions, we encouraged the participants to
use generative AI tools (e.g., GPT (OpenAI, 2023)) to help modify their
documents. The competitions were approved by ethics committees. The data of
the competitions and our code will be made public upon publication of this
paper.

Analysis of the ranking competitions revealed that, as is the case for the
single-query setting (Raifer et al., 2017), publishers tend to mimic content
from documents that were previously highly ranked. This trend was more
prominent when publishers were not allowed to use generative AI tools. We also
found that the neural ranker’s rankings for different queries representing the
same information need were more diverse than those of the LTR method.
Accordingly, it was much harder for publishers to have their documents highly
ranked for multiple queries with the neural ranker.

Finally, as in work on the single-query setting (Raifer et al., 2017), we
pursue the task of predicting which document among those not ranked first in
the last ranking will become the most highly ranked assuming that the top
ranked document is going to change. As it turns out, utilizing information
induced from rankings for other queries representing the same information need
can help improve prediction effectiveness for the query at hand. This is yet
another difference between the multiple-queries setting and the single-query
setting.

Our contributions can be summarized as follows:

  * •

A theoretical and empirical study of document manipulations intended to
improve ranking for multiple queries.

  * •

A full game theoretic characterization of when equilibrium of document
manipulation strategies exists.

  * •

Showing that best response dynamics might not converge even when an
equilibrium exists.

  * •

Organizing novel ranking competitions where publishers compete for multiple
queries. Additional novel aspects are using a neural ranker and allowing
publishers to use generative AI tools for document modification.

  * •

A method for predicting which documents among those not ranked first might be
ranked first in the next ranking; and, demonstrating the merit of using
information induced from other queries to this end.

##  2\. Related Work

The work most related to ours is based on a game theoretic and empirical
analysis of ranking incentivized document manipulations applied for a single
query (Raifer et al., 2017). In contrast to our multiple queries settings, an
equilibrium in the single-query setting always exists. Raifer et al. (Raifer
et al., 2017) and Goren et al. (Goren et al., 2021) found using ranking
competitions that publishers tend to mimic content from documents highly
ranked in the past for a query: a strategy justified by Raifer et al. (Raifer
et al., 2017) using a game theoretic analysis. We found a similar pattern in
ranking competitions organized for the multiple-queries setting. We use both a
neural ranker and a feature-based ranker while Raifer et al. (Raifer et al.,
2017) used only the latter. In contrast to our work, Raifer et al. (Raifer et
al., 2017) did not allow the use of generative AI tools for document
manipulation in their ranking competitions.

Several algorithmic attacks on neural retrieval methods have been recently
reported (Goren et al., 2021; Liu et al., 2022; Song et al., 2022; Wang et
al., 2022; Wu et al., 2022; Chen et al., 2023a, b; Liu et al., 2023). The
attacks are almost always for a single query, except for an attack for
topically related queries (Liu et al., 2023). In contrast, in our game
theoretic analysis, the document manipulations are changes of the relative
emphasis of a document on different queries. This type of strategic
manipulation has not been studied in previous work to the best of our
knowledge. Furthermore, we are not aware of any other studies, as ours, of
human document-modification strategies for multiple queries with or without
generative AI tools.

The suboptimality of the probability ranking principle (Robertson, 1977) in
competitive retrieval settings was demonstrated using a game theoretic
approach (Ben-Basat et al., 2017). Publishers can write either a single topic
document or a two-topics document with equal emphasis for the topics. Since
the resulting ranking game has a finite number of players (publishers) and a
finite number of actions a player can play (i.e., documents they can write),
by Nash’s theorem (Nash, 1950), a mixed-strategies equilibrium exists. That
is, documents are stochastically created by applying a distribution over a
document set. The setting is less realistic than the one we address here where
publishers can write a document with different emphasis (represented as a real
number) on different queries. Our resulting games have an infinite number of
actions a publisher can take, and we show that there is not necessarily an
equilibrium.

##  3\. Game Theoretic Analysis

To analyze the search setting when publishers modify their documents to have
them highly ranked for multiple queries, we use game theory. We consider
publishers as players in a ranking game: the documents they publish are their
strategies and the search engine’s ranking function is the mediator (Kurland
and Tennenholtz, 2022).

As noted above, some previous work used game theory to analyze the single-
query search setting where publishers opt to promote their documents for a
single query (Raifer et al., 2017); single-peaked ranking functions were used
(Raifer et al., 2017). Several sparse retrieval methods are single peaked;
e.g., cosine between tf.idf vectors and negative KL divergence over
multinomial distributions (Zhai and Lafferty, 2006). Furthermore, considering
the common technique of stuffing query terms in the document to improve its
ranking (Gyöngyi and Garcia-Molina, 2005), the retrieval function could also
be viewed as single peaked with respect to query term occurrence. That is,
moderately increasing query term occurrence usually results in monotonic
increase of retrieval scores. However, as from a certain point, further
increasing query term occurrence can result in increased penalty to retrieval
scores when document quality measures are used (Bendersky et al., 2011b).
There is also some empirical evidence in prior work (Raifer et al., 2017) and
ours (Section 5) that publishers might view the undisclosed ranking function
as single peaked. When considering incomplete information about the ranker
(unknown peak), the ranking game has a minimax regret equilibrium where
publishers mimic content of documents that were previously the highest ranked
(Raifer et al., 2017).

Turning to our multiple-queries setting, we assume that a document has split
emphasis on various queries; e.g., different passages match different queries
to varying degrees. We assume as Raifer et al. (Raifer et al., 2017) a single
peaked ranking function, and focus as Basat et al. (Ben-Basat et al., 2015) on
ranking games where the retrieval function is disclosed (i.e., its peak is
known). We show that the resulting situation in the multiple-queries setting
with a disclosed ranking function is considerably more subtle than that in the
single query setting where it is trivial to show that an equilibrium exists.
Specifically, we prove that there is not necessarily an equilibrium in the
multiple-query setting and fully characterize the cases when it exists.

###  3.1. Game Definition

Let $G=\langle n,m,p\rangle$ be a ranking game defined as follows: $n$
publishers (players) write and modify documents in a corpus $D$; a query set
$Q$ contains $m$ queries: $\\{q_{1},\ldots,q_{m}\\}$. Each player writes a
single document $d\in D\stackrel{{\scriptstyle
def}}{{=}}\\{(d^{1},\ldots,d^{m})\in[0,1]^{m}:\sum_{j=1}^{m}d^{j}\leq 1\\}$,
where $d^{j}$ ($\in[0,1]$) is the relative emphasis of the document $d$ on
information relevant to query $q_{j}$.222In reference to work on focused
retrieval (Geva et al., 2012), $d^{j}$ can be defined as the portion of the
text in $d$ that is marked as relevant to query $q_{j}$.

Let $f:[0,1]\rightarrow[0,1]$ be a single peak function: there exists a unique
$p\in[0,1]$ such that $f$ is monotonically increasing in $[0,p]$ and
monotonically decreasing in $[p,1]$. The retrieval function $r:D\times
Q\rightarrow[0,1]$ assigns document $d~{}(\in D$) the retrieval score
$r(d,q_{j})\stackrel{{\scriptstyle def}}{{=}}f(d^{j})$ with respect to query
$q_{j}$ ($j\in\\{1,\ldots,m\\}$); documents are ranked in descending order of
scores with ties broken arbitrarily.

We will refer to document $d_{i}$ written by player $i$ as the (pure) strategy
of player $i$ in the game333There is also a notion of mixed strategies where
players apply a distribution over pure strategies. Herein, we focus on pure
strategies.. That is, the strategy $d_{i}$ is defined by the choice of
relative emphasis on each query in the document $d_{i}$. A strategy profile
$s\stackrel{{\scriptstyle def}}{{=}}(d_{1},\ldots,d_{n})$ is the tuple of pure
strategies of the players in the game; i.e., the set of documents in the
corpus.

The utility of a player $i$ who wrote document $d_{i}$, denoted $U_{i}(s)$, is
the number of queries for which $d_{i}$ is ranked first; $d_{i}$’s rank for a
query depends on the strategies of all other players (i.e., their documents).
To summarize, $G$ is a strategic game where $n$ players (publishers) opt to
maximize their documents’ ranks for $m$ queries.

A strategy profile $s$ is a (pure) Nash equilibrium if there is no incentive
for any player $i$ to change her strategy given the strategies
$\\{d_{j}\\}_{j\neq i}$ of all other players; i.e., a change will result in
reduced utility.

###  3.2. Game Analysis

Our goal is to find a full characterization of the existence (or lack thereof)
of pure Nash equilibria for $G$, given $n$, $m$ and $p$.

It is easy to see that for a small enough $p$, there exists a pure Nash
equilibrium where the documents published by all players have the same
emphasis on all queries:444Recall that the retrieval function has a single
peak at $p$.

###### Lemma 3.1.

If $p\leq\frac{1}{m}$, then the profile where all players write
$d=(p,\ldots,p)$ is a pure Nash equilibrium.

To extend our analysis for any $p$ ($\in[0,1]$), we start by analyzing the
case of two players ($n=2$). This case has an interesting property, because in
any pure Nash equilibrium, the two players have the same utility. They can
attain this utility, for example, by using the same strategy (i.e., writing
the same document):

###### Lemma 3.2.

Let $n=2$ and $m>n$. If $s=(d_{1},d_{2})$ is a pure Nash equilibrium, then
$U_{1}(s)=U_{2}(s)=\frac{m}{2}$.

###### Proof.

Observe that for any profile $s=(d_{1},d_{2})$, $U_{1}(s)+U_{2}(s)=m$. Assume
by contradiction that $U_{1}(s)\neq U_{2}(s)$, w.l.o.g. $U_{1}(s)<U_{2}(s)$;
i.e., $U_{1}(s)<\frac{m}{2}<U_{2}(s)$. If player 1 adopts player 2’s strategy,
i.e., writes $d_{1}^{\prime}=d_{2}$ then
$U_{1}(d_{1}^{\prime},d_{2})=U_{2}(d_{1}^{\prime},d_{2})=\frac{m}{2}>U_{1}(d_{1},d_{2})$;
hence, $s=(d_{1},d_{2})$ is not a pure Nash equilibrium — a contradiction.

∎

We use Lemma 3.2 to fully characterize in Theorem 3.3 the existence of pure
Nash equilibrium for $G$ when $n=2$ (i.e., two players).

###### Theorem 3.3.

If $n=2$ and $m>n$, then the game $G$ has a pure Nash equilibrium iff
$p\leq\frac{1}{m-1}$.

###### Proof.

We only provide a proof sketch due to space considerations and as it is
technical. Say a strategy profile $(d_{1},d_{2})$ is a pure Nash equilibrium;
let $d_{2}=(d_{2}^{1},\ldots d_{2}^{m})$, and assume w.l.o.g. that
$d_{2}^{1}\geq d_{2}^{2}\geq\ldots\geq d_{2}^{m}$. We show that there cannot
be two queries $q_{i}$ and $q_{j}$ such that $d_{2}^{i}<p$ and $d_{2}^{j}<p$.
This allows to show that if there is an equilibrium, then
$p\leq\frac{1}{m-1}$. If $p\leq\frac{1}{m}$, then by Lemma 3.1 the profile
where both players publish $d=(p,\ldots,p)$ is a pure Nash equilibrium. We
show that for $\frac{1}{m}<p\leq\frac{1}{m-1}$ the profile where
$d_{1}=d_{2}=(p,\ldots,p,1-p\cdot(m-1))$ is a pure Nash equilibrium. ∎

To extend the analysis to a game with more than two players, we start by
considering the case of $m>n$:

###### Theorem 3.4.

The game $G=\langle n,m,p\rangle$ with $n<m$ has a pure Nash equilibrium iff
$p\leq\frac{1}{\left\lceil\frac{2\cdot m}{n}-1\right\rceil}$.

###### Proof.

The central idea in the proof is to show that in any pure Nash equilibrium the
following properties must hold: (i) the highest ranked document $d$ for each
query $q_{j}$ has $d^{j}\geq p$; and, (ii) for each player there is at most
one query for which her document is ranked first solely. The proofs of these
properties are mostly technical and are omitted due to space considerations.
For any $p>\frac{1}{\left\lceil\frac{2\cdot m}{n}-1\right\rceil}$, these
properties cannot hold, and therefore there is no pure Nash equilibrium.
Conversely, we show that if $p\leq\frac{1}{\left\lceil\frac{2\cdot
m}{n}-1\right\rceil}$ then we can construct a pure Nash equilibrium as
follows.

Given some strategy profile $s=(d_{1},\ldots,d_{n})$, let $h_{j}(s)$ denote
the number of documents assigned the same highest retrieval score for query
$q_{j}$ when $s$ is played. The strategy profile $s_{\mathrm{alg}}$ obtained
by the following algorithm is a pure Nash equilibrium (proof omitted due to
space considerations):

Algorithm 1 Equilibrium construction for $G=\langle n,m,p\rangle$ with $n<m$

Initialize $d_{i}^{j}=0$ for all $i,j$.

$k=\left\lfloor\frac{1}{p}\right\rfloor$

for $t=1$ to $k$ do

for $i=1$ to $n$ do

$j^{*}=\arg\min_{j\in\\{1,\ldots,m\\}}h_{j}(d_{1},\ldots,d_{n})$

$d_{i}^{j^{*}}=p$

end for

end for

∎

In the case $n\geq m$, it is easy to see that a pure Nash equilibrium exists
for every $p$.

Specifically, w.l.o.g. each (disjoint) group of
$\left\lfloor\frac{n}{m}\right\rfloor$ players is assigned to an arbitrarily
different query $q_{j}$ among $q_{1},\ldots,q_{m}$; each player in this group
writes a document $d$ with $d^{j}=p$ and $d^{k}=0$ for $k\neq j$. Together
with Theorem 3.4, we arrive to the final result:

###### Corollary 3.5.

G=$\langle n,m,p\rangle$ has a pure Nash equilibrium iff
$p\leq\frac{1}{\max{\\{\left\lceil\frac{2\cdot m}{n}-1\right\rceil,1\\}}}$.

###  3.3. Best response dynamics

The best response of a player to the strategies played by all other players
(i.e., their documents) is the strategy (document) that maximizes her utility.
Iteratively finding best responses until convergence (if exists) — a process
known as best response dynamics — is a standard mechanism for reaching Nash
equilibria, specifically in potential games (Nisan et al., 2007). In these
games, the best response dynamics is guaranteed to converge to an equilibrium.
We now show that in our ranking games the best response dynamics does not
necessarily converge. Hence, these are not potential games. An important
implication is that in order to achieve stability (specifically, equilibrium),
the game might need external intervention.

Consider the game $G=\langle n,m,p\rangle$ for $n=2,m=3$ and $p=\frac{1}{2}$.
We show that the best response dynamics does not converge in this game,
although by Theorem 3.3 an equilibrium exists. Let $s_{t}$ denote the strategy
profile played at round $t$ by two players. The following sequence of
strategies is a best response dynamics given an initial strategy profile
$s_{0}=\left((0.3,0.4,0),(0.2,0.3,0.5)\right)$:
$s_{1}=\left((0.3,0.4,0),(0.4,0.5,0.1)\right)$,
$s_{2}=\left((0.5,0.3,0.2),(0.4,0.5,0.1)\right)$,
$s_{3}=\left((0.5,0.3,0.2),(0,0.4,0.3)\right)$. For example, since player
$1$’s document at round $0$ is $(0.3,0.4,0)$ then player 2’s document at round
$1$, $(0.4,0.5,0.1)$, attains the maximal possible utility (3) as it outranks
player $1$’s document for all three queries. Player $1$’s best possible
utility given player $2$’s document from round 1, $(0.4,0.5,0.1)$, is 2, which
is obtained by writing, for example, a document $(0.5,0.3,0.2)$ at round $2$,
and so forth. Since $s_{0}$ and $s_{3}$ are symmetric, the dynamics does not
converge.

##  4\. Ranking Competitions

Our next goal is to empirically analyze ranking competitions in which
publishers (document authors) modify documents with respect to multiple
queries ($m>1$). To the best of our knowledge, there are no datasets that
support this type of analysis. There are two publicly available datasets
(Raifer et al., 2017; Goren et al., 2020) that resulted from ranking
competitions. However, in these competitions, documents were modified with
respect to a single query. Thus, we created a new dataset by organizing our
own ranking competitions inspired by those organized in (Raifer et al., 2017;
Goren et al., 2020). International and institutional ethics committees
approved the competitions. The participants signed a consent form and could
withdraw at any time.

In addition to having publishers compete for multiple queries rather than a
single one, we introduced two novel aspects in our competitions. The first is
the ranking function. While in previous competitions only a feature-based
learning-to-rank (LTR) method was used (Raifer et al., 2017; Goren et al.,
2020), we also used a neural (NEU) ranker (BERT-based (Nogueira and Cho,
2019)). The ranking functions are described below. The second aspect is the
use of AI tools to modify documents in some of the competitions. In total, we
organized four ranking competitions which differ by the ranking function
applied and/or whether generative AI tools were allowed: LTR$\wedge$AI,
LTR$\wedge\urcorner$AI, NEU$\wedge$AI, and NEU$\wedge\urcorner$AI. Table 1
provides an overview of the competitions.

Table 1.  Overview of the four ranking competitions. Competition | Ranking Function | AI Tools
---|---|---
LTR$\wedge$AI | LambdaMART | ✓
LTR$\wedge\urcorner$AI | LambdaMART | ✗
NEU$\wedge$AI | BERT | ✓
NEU$\wedge\urcorner$AI | BERT | ✗

Each competition lasted for ten rounds; in each round, $30$ games were held,
each with respect to a different backstory (topic) from the UQV dataset
(Bailey et al., 2016).555The dataset includes query variants for TREC’s
$2013$-$2014$ Web track topics ($201$-$300$). Each topic was represented using
$m=3$ queries: the query selected to be the focus of the backstory and two
additional query variants selected from all those provided for the backstory.
The same $30$ topics were used in all four competitions. Refer to Appendix A.1
for further details about the topic and query selection procedures.

Before the start of the game, the participants were provided with three
queries representing the topic of the game, the topic description (backstory)
and an example of a relevant document. This document was generated using
GPT-3.5, as described in Appendix A.1. Given the provided information, the
participants were asked to write and submit their own document. In each round
of the game, the participants were presented with the documents submitted in
the previous round and the rankings induced over these documents for each of
the three queries. They were encouraged to further modify their document to
improve its ranking in the next round. All the documents were in plain text
format, limited to $150$ words. Refer to Appendix A.2 for detailed competition
guidelines.

A group of $84$ undergraduate and graduate students enrolled in an information
retrieval course participated in our competitions. Each student participated
in three (repeated) games and submitted one document per game in a round. We
made sure that the three games were selected from at least two different
competitions and that there was no overlap between the topics of these games.
Each game included exactly five participants: a combination of students and
bots. Two or three students participated in every game together with two bots
that used GPT-3.5 to create documents, as explained in Appendix A.3. If a game
had only four participants, i.e., two students and two bots, we added a third
static bot that published the initial example document in every round. To
prevent any potential bias, the students’ identities were anonymized, and they
were not informed about the use of bots. Unless otherwise mentioned, our
analysis is performed for the students’ documents only.

To encourage students to participate and modify their documents so as to
attain high rankings in the competitions, we offered bonus points for their
course grades. We note that the students could have earned a perfect grade in
the course without participating in the competitions. The bonus per game was
awarded based on the median reciprocal rank of a student’s document across the
three queries in the game. For example, if a student’s document was ranked
first, second, and third across the queries, they would earn half a point for
that game. The points were accumulated over the student’s three games per
round and over the ten rounds.

As noted above, we experimented with two ranking functions in our
competitions: LTR and NEU. For our learning-to-rank (LTR) approach, as in
previous work (Raifer et al., 2017; Goren et al., 2020; Vasilisky et al.,
2023), we use LambdaMART (Wu et al., 2010) via the RankLib
toolkit666https://sourceforge.net/p/lemur/wiki/RankLib/. Unless otherwise
specified, the implementation details of this model, including the training
procedure and setting of hyper-parameter values, are the same as those of
Vasilisky et al. (2023). The model was trained on the Combined dataset
(Vasilisky et al., 2023), which includes data from the two previous ranking
competitions (Raifer et al., 2017; Goren et al., 2020). Each document was
represented using a nine-dimensional feature vector. Some of the features are
estimates of the similarity between the query and document, and the rest are
query-independent document quality measures shown to be effective for Web
retrieval (Bendersky et al., 2011a). The features are: (i) TF: sum of query
term frequencies in the document, (ii) NormTF: TF normalized by the document’s
length, (iii) BM25: BM25 similarity between the query and document, (iv) LMIR:
language-model-based similarity between the query and document777Unigram
language models with Dirichlet smoothing ($\mu=1000$) are used., (v) BERT:
BERT-based similarity between the query and document888We used BERT-Large
fine-tuned for passage ranking on MS MARCO (Nogueira and Cho, 2019)., (vi)
LEN: document’s length, (vii) FracStop: percentage of terms in the document
that are stopwords999The NLTK stopword list was used in our experiments:
www.nltk.org/nltk_data/., (viii) StopCover: percentage of stopwords on a
stopword list that appear in the document, and (ix) ENT: entropy of the
document’s term distribution. For our neural ranking function (NEU), we used
BERT as in feature (v) above. Thus, BERT served both as a feature in LTR and
as a standalone ranking function. We note that the ranking functions were not
disclosed to the students, who also did not know that four ranking
competitions were held.

Our dataset includes a total of $2520$ documents submitted by students ($847$
of which are unique), $2400$ documents generated by bots ($1833$ of which are
unique), and $30$ documents that served as the initial document per topic. We
will make the dataset publicly available upon the publication of this work.

Each student document was judged by three crowd workers in CloudResearch’s
connect platform (Hartman et al., 2023) for binary relevance to the topic and
by five workers for content quality, with $0.75$ and $0.49$ inter-annotator
agreement rates (free-marginal multi-rater Kappa), respectively. All workers
were native English speakers. More than $97\%$ of all student documents were
judged relevant by at least two out of three annotators. This high relevance
percentage is in accordance with previous reports on competitions held for a
single query (Raifer et al., 2017; Goren et al., 2021), albeit a bit elevated.
Although students were not instructed to write relevant documents, the
documents were short and students modified them for multiple queries
representing the same topic. We note that our focus in this paper is on
ranking-incentivized manipulations and not on ranking effectiveness.

Five annotators judged each student document for content quality using the
categories: valid, keyword stuffed and spam (useless). When using AI tools,
$89\%$ and $71\%$ of the documents were judged to be valid by at least three
and at least four out of the five annotators, respectively. When not using AI
tools, the percentages were $85\%$ and $61\%$, respectively. This attests to
the merit of using AI tools for rank promotion while maintaining content
quality. Only $9\%$ and $6\%$ of the documents created with no AI tools and
with AI tools, respectively, were judged as keyword stuffed by at least three
annotators. Almost none of the documents were judged as spam. The distribution
of quality judgments was not different between the LTR and NEU competitions.

##  5\. Empirical Analysis

###  5.1. Analysis of Strategies

Feature Values. Inspired by work on the single-query setting (Raifer et al.,
2017), we examine the document modification strategies in our multiple-queries
setting: we analyze the changes in the documents’ feature values along
competition rounds. Recall that three document rankings were induced per
topic, each for one of the three queries representing the topic. In this
analysis, we examine each query (document ranking) separately. We focus on
changes in documents that lost (i.e., were not ranked first) for at least four
consecutive rounds before reaching the first rank ($\mathbf{L}$). We compare
the average feature values of the winners (i.e., the highest ranked documents)
per query ($\mathbf{W_{q}}$) with the average feature values of two types of
losing documents ($L$): those whose feature value was lower than or equal to
that of the winning document four rounds prior to their win ($\mathbf{L\leq
W_{q}}$), and those whose feature value was higher ($\mathbf{L>W_{q}}$). We
also present for reference the average feature values of the (at most three)
winning documents per topic ($\mathbf{W_{t}}$). Figure 1 presents the results
for representative query-independent and query-dependent features of LTR.
Figures for other features exhibit the same patterns and are omitted as they
convey no additional insight.

LEN

StopCover

ENT

(a) Query-Independent Features

TF

Okapi

BERT

(b) Query-Dependent Features

Figure 1. Averaged feature values of documents that lost in at least four
consecutive rounds before winning. The documents are grouped based on whether
their feature values four rounds before the win were lower than or equal to
($L\leq W_{q}$) or higher than ($L>W_{q}$) the values of the winners for the
given query. $W_{q}$ and $W_{t}$: the averaged feature values of the
corresponding winner per query and all the winners per the same topic,
respectively. x-axis: the (negative) number of rounds leading up to a win.
Note: in NEU$\wedge$AI and NEU$\wedge\urcorner$AI, when BERT is used as a
standalone ranking function, the feature values of the losing documents are
always lower than those of the winning document; therefore, $L>W_{q}$ is not
shown.

We see in Figure 1 that regardless of the ranking function (LTR or NEU), the
use of generative AI tools (AI or $\urcorner$AI) and the initial feature
values ($L\leq W_{q}$ or $L>W_{q}$), with a few exceptions, the features of
the losing documents tend to gradually converge to those of the winners. This
finding is in line with results for the setting where publishers modified
their documents for a single query (Raifer et al., 2017). Contrary to Raifer
et al. (2017), we see that the feature values of the winning documents changed
throughout the competition rounds. This might be attributed to the fact that
documents in our competitions were modified for three different queries. For
example, say a document was ranked first for some query. The student who wrote
the document may have continued to modify it to improve its ranking for the
other two queries, and thus affected its feature values. We delve deeper into
this point in Section 5.2.

We also note that changes of winners’ feature values conceptually echo our
game theoretic findings in Section 3: in the single-query setting an
equilibrium is guaranteed in contrast to the multiple-queries setting. Hence,
the single-query setting is more likely to reach a stable state than the
multiple-queries setting101010We hasten to point out that the game theoretic
analysis was performed for single peak functions. The rankers used in the
competition are not single peaked.

We observe in Figure 1(a) a general upward trend for the query-independent
features: LEN, StopCover, and ENT; the trends are more pronounced for
LTR$\wedge$AI and LTR$\wedge\urcorner$AI. Hence, the documents became longer
(LEN) with increased content diversity (ENT) and stopwords occurrence
(StopCover). Interestingly, Raifer et al. (2017) found a downward trend for
ENT when students modified documents for a single query. We therefore
hypothesize that when competing for a single query there is an attempt to
focus the content for this query, while competing for multiple queries results
in increased diversity so as to better cover the query set.

We see in Figure 1(b) that the query-dependent feature values for $L\leq
W_{q}$ (where the initial feature value of the losing document was lower than
or equal to that of the winner) increased along the rounds, whereas the values
for $L>W_{q}$ (where the initial feature value of the losing document was
higher than that of the winner) somewhat fluctuated and even decreased at
times. This implies that in the former case, the students gradually increased
the number of query term occurrences in their documents, but in the latter
case, the students were careful not to add too many query terms, possibly to
avoid keyword stuffing which they were warned about. Furthermore, this finding
potentially suggests that from the point of view of students, the ranking
function might be single peaked with respect to query term occurrence and the
features based on it.

Finally, we can see that the differences between $W_{q}$ and $W_{t}$ are
smaller for LTR than for NEU. This finding might be attributed to the fact
that for LTR, a winning document was more likely to win for multiple queries,
resulting in fewer unique winning documents per topic. We present further
support to this claim in Section 5.2.

Prompts. To gain additional insights about the document modification
strategies, we asked the students to share the prompts they used for the
generative AI tools. We collected $44$ prompts from rounds $8$ to $10$ of the
competitions. The prompts typically included instructions to create a document
that would be ranked high for the specified queries. All prompts, except for
three, had no mention of the reward mechanism used in the competitions. As an
example, one of the three prompts included the text: “Since the score is given
by the median ranking among 3 queries, we can focus on maximizing the rankings
on two queries.”

Out of the $44$ prompts, $29$ included example documents: the initial example
document ($2$ prompts), the student’s document from the previous round ($15$
prompts), at least one of the winning documents from the previous round ($24$
prompts), or even all the documents in the previous round ($5$ prompts). This
further indicates that the students considered winning documents a valuable
source of information about the undisclosed ranking functions.

###  5.2. Ranking Functions

Thus far, we studied the document modification strategies in our competitions.
We now shift our focus to examining the effect of the ranking functions used.
We first analyze the differences between rankings induced by a function for
the three queries representing a topic. We computed for each ranking function
the RBO similarity (with $p=0.9$) (Webber et al., 2010) between the document
rankings induced for each pair of queries per topic. We then averaged these
values over the pairs per topic and over topics. Figure 2 presents the results
for each round of the four competitions. We see that the similarities for
NEU$\wedge$AI and NEU$\wedge\urcorner$AI are considerably and consistently
lower than those of LTR$\wedge$AI and LTR$\wedge\urcorner$AI, suggesting that
NEU (BERT) is much more sensitive to the queries used than LTR (LambdaMART).
Previous research demonstrated BERT’s sensitivity to document modifications in
competitions held for a single query (Vasilisky et al., 2023).

Figure 2.  Average RBO similarity between all pairs of rankings induced per
topic, averaged over topics per round.

We further study the effect of the ranking functions on the competitions in
Table 2 by examining documents that won for at least one of the queries per
topic. We present the percentage of these documents that won for exactly one,
two or three queries. Additionally, we present the average ranks of these
documents in rankings induced for queries for which these documents did not
win. We can see that for LTR, at least $65\%$ of the winners won all three
queries in a game, while for NEU, less than $9\%$ did so. Furthermore, the
average rank of documents for queries they did not win was, on average, higher
when NEU was used than when LTR was used.111111The rank of the highest-ranked
document is $1$. Increased rank means lower ranking. These findings
demonstrate that the ranking function used affects the ability of a document
to win for multiple queries per topic. That is, a document that won for one of
the queries is more likely to win for additional queries for the same topic in
competitions that employed LTR compared to those that employed NEU.

Table 2.  Docs: percentage of winning documents that won for exactly $x$ queries per topic. Rank: for documents that won for $x$ queries per topic, the average ranks with respect to the $3-x$ queries for which they did not win. |  LTR$\wedge$AI |  LTR$\wedge\urcorner$AI |  NEU$\wedge$AI |  NEU$\wedge\urcorner$AI
---|---|---|---|---
| Docs | Rank | Docs | Rank | Docs | Rank | Docs | Rank
$x=1$ | 17.9% | 2.2 | 17.7% | 2.4 | 62.9% | 3.4 | 65.7% | 3.5
$x=2$ | 17.0% | 2.1 | 15.2% | 2.3 | 28.5% | 3.3 | 27.6% | 3.2
$x=3$ | 65.1% | - | 67.0% | - | 8.6% | - | 6.7% | -

###  5.3. Generative AI Tools

We allowed the use of generative AI tools in some of our competitions. We next
compare the documents created by students who were allowed to use such tools
(AI) with those who were not allowed ($\urcorner$AI). We use the cosine of
tf.idf vectors to compute per topic the similarity between each document
submitted by a student in a specific round and (i) documents in the previous
round that won for at least one of the queries and (ii) all the other
documents from the previous round. We averaged the similarities over the
documents and grouped them by whether or not the students were allowed to use
AI tools. We see in Figure 3 that there were noticeable differences in the
similarity values of AI and $\urcorner$AI. Specifically, the documents created
without the help of AI tools in LTR$\wedge\urcorner$AI and
NEU$\wedge\urcorner$AI were much more similar to documents submitted in the
previous round, whether they were winners or not, than those created with the
help of such tools in LTR$\wedge$AI and NEU$\wedge$AI. This finding indicates
that using AI tools may lead to the creation of more diverse content. We also
observe higher similarities with the winning documents than with the losing
ones for both AI and $\urcorner$AI, further supporting our earlier findings
that the documents, whether they were directly modified by students or
generated using AI tools, tended to converge to the winners.

Figure 3.  Average similarity between all documents in a round per topic and
(i) Winners: documents that won at least one query for the topic in the
previous round and (ii) Losers: all other documents in the previous round.

##  6\. Predicting Winners

In this section, we address the challenge of predicting which document among
all those that did not rank first in round $l-1$ will be ranked the highest
(win) in round $l$ if the winner indeed changes. The documents in our
competitions were modified with respect to $m=3$ queries in the query set $Q$
for a topic; yet, the prediction is performed separately for each query $q\in
Q$. A similar challenge was previously tackled for competitions held for a
single query (Raifer et al., 2017).

For prediction, we represent each document as a $27$-dimensional feature
vector. The features we use can be divided into seven sets. The first two
sets, Micro ($12$ features) and Macro ($3$ features), were proposed by Raifer
et al. (2017). These features model properties of the document in round $l$
($D_{l}$), the document published by the same author in round $l-1$
($D_{l-1}$), and the winning document in round $l-1$ for the query $q$
($W^{q}_{l-1}$). The features in Micro quantify the changes in $D_{l}$ with
respect to $W^{q}_{l-1}$ by counting term additions and deletions. These
features are defined for three groups of unique terms: query terms, stopwords,
and terms that are neither query terms nor stopwords. For each group of terms,
four features were defined: the number of terms in $W^{q}_{l-1}$ that were
either (i) added to or (ii) deleted from $D_{l-1}$, and the number of terms
that were not in $W^{q}_{l-1}$ and were either (iii) deleted from or (iv)
added to $D_{l-1}$. The features in Macro are the similarities between
$D_{l}$, $D_{l-1}$, and $W^{q}_{l-1}$; the similarity between a pair of
documents was computed using the cosine of their tf.idf vector
representations.

The features just discussed only consider the winner for the given query
($W^{q}_{l-1}$), disregarding the rankings induced for the other two queries
and their corresponding winners. Accordingly, we introduce an additional set
of features, TopicMacro, that consider the group of all winners for the three
queries in $Q$ ($W^{Q}_{l-1}$). We compute the similarities between $D_{l}$
and $W^{Q}_{l-1}$ and between $D_{l-1}$ and $W^{Q}_{l-1}$. We use cosine to
compute the similarities and represent the documents in $W^{Q}_{l-1}$ via the
centroid of their tf.idf vectors.

In the following two feature sets, we quantify properties of $D_{l-1}$’s ranks
in the three rankings induced for all three queries per topic. The first set,
TopicRank, includes three features: the minimum, median, and maximum rank of
$D_{l-1}$ across the three queries in $Q$. The second set, QueryRank, includes
three binary features that are set to $1$ if $D_{l-1}$’s minimum, median, and
maximum rank for the three queries is equal to its rank for the given query
$q$; otherwise, they are set to $0$.

The next feature set, PrevChange, includes three features that quantify past
changes (improvements) in the document’s ranks. The underlying assumption is
that if a publisher improved her document’s rank in round $l-1$, she might do
so again in round $l$. To this end, we also examine the publisher’s document
in round $l-2$ ($D_{l-2}$). For each of the three queries, we compute the
difference between the ranks of $D_{l-1}$ and $D_{l-2}$, and scale it by the
maximal possible rank change given $D_{l-2}$’s rank in round $l-2$. We use the
minimum, median, and maximum values across the three queries.

Finally, we found in Section 5.1 that the documents tended to become longer
throughout the four competition rounds leading to a win; accordingly, the
feature in the final set, Len, is $D_{l}$’s length.

###  6.1. Setup

Our dataset includes documents submitted by both students and bots. We
discarded queries for which a bot generated the winning document. In addition,
we discarded queries for which the winners in round $l-1$ ($W^{q}_{l-1}$) and
round $l$ ($W^{q}_{l}$) were generated by the same publisher. This was done
because we aimed to predict which of the losing documents would win in the
following round. As a result, the number of queries per competition round
ranged from $3$ to $39$.

We experimented with four different classifiers with the features defined
above via the scikit-learn library (Pedregosa et al., 2011): logistic
regression (LReg), linear SVM (LSVM), polynomial SVM (PSVM) and random forests
(RForest). We applied min-max normalization to feature values per query. The
document assigned the highest prediction score per query by a classifier was
deemed the winner; the remaining documents were considered losers.

The features in PrevChange can only be computed when data for at least two
previous rounds is available; therefore, our experiments were conducted for
rounds $3$ to $10$. We used leave-one-out cross-validation over rounds to
train our models and select parameter values. We used the queries from one of
the rounds for testing and those from the remaining seven for training and
validation. We repeatedly trained a model on data from six of the seven rounds
with different parameter configurations and validated the effectiveness over
the seventh round. We selected the parameter values that yielded the highest
prediction effectiveness on average across the seven validation rounds. Then,
we trained a final model using data from all seven rounds and applied it to
the held-out test round (fold). We repeated this entire procedure for each
test fold. The models were trained separately for each of the four
competitions.

We use Acc (percentage of correctly classified winners and losers) and F1
(harmonic mean of precision and recall), averaged over queries and test folds,
to measure prediction effectiveness; the former served to select parameter
values. Statistically significant prediction differences were determined
across queries using the two-tailed paired t-tests with $p\leq 0.05$. LReg,
LSVM and PSVM were trained with L1 regularization; the regularization
parameter was selected from $\\{1,10,50,100\\}$. The degree of the polynomial
kernel in PSVM was selected from $\\{2,3,4,5\\}$. For RForest, the number of
trees and leaves were set to values in $\\{10,50,100,500\\}$ and
$\\{10,20,30\\}$, respectively. All other parameters were set to default
values (Pedregosa et al., 2011).

Table 3.  Prediction effectiveness. All differences between our classifiers (LReg, LSVM, PSVM, RForest) and the baselines are statistically significant. Bold: best result in a row. |  | Rand | QMaj | TMaj | AllW | AllL | Raifer | LReg | LSVM | PSVM | RForest
---|---|---|---|---|---|---|---|---|---|---|---
LTR$\wedge$AI | Acc | 0.654 | 0.435 | 0.503 | 0.466 | 0.534 | 0.696 | 0.822 | 0.853 | 0.895 | 0.843
LTR$\wedge$AI | F1 | 0.622 | 0.397 | 0.497 | 0.635 | 0.000 | 0.678 | 0.810 | 0.843 | 0.885 | 0.832
LTR$\wedge\urcorner$AI | Acc | 0.707 | 0.398 | 0.504 | 0.480 | 0.520 | 0.772 | 0.886 | 0.870 | 0.846 | 0.837
LTR$\wedge\urcorner$AI | F1 | 0.697 | 0.368 | 0.497 | 0.648 | 0.000 | 0.763 | 0.879 | 0.863 | 0.839 | 0.831
NEU$\wedge$AI | Acc | 0.669 | 0.532 | 0.468 | 0.464 | 0.536 | 0.766 | 0.815 | 0.798 | 0.806 | 0.855
NEU$\wedge$AI | F1 | 0.643 | 0.495 | 0.588 | 0.634 | 0.000 | 0.747 | 0.800 | 0.782 | 0.790 | 0.842
NEU$\wedge\urcorner$AI | Acc | 0.556 | 0.481 | 0.500 | 0.491 | 0.509 | 0.722 | 0.750 | 0.750 | 0.796 | 0.750
NEU$\wedge\urcorner$AI | F1 | 0.547 | 0.471 | 0.592 | 0.658 | 0.000 | 0.717 | 0.742 | 0.742 | 0.790 | 0.743

###  6.2. Results

We compare in Table 3 the prediction effectiveness of our four classifiers
(LReg, LSVM, PSVM, and RForest) with that of five baselines121212Except for
TopicMajority, all the baselines were also used by Raifer et al. (2017).: (i)
Random (Rand) : the winner is randomly selected, (ii) QueryMajority (QMaj) :
the winner is the document whose publisher had the highest number of past wins
for the given query; ties were broken randomly, (iii) TopicMajority (TMaj):
the winner is the document whose publisher had the highest number of past wins
for all three queries of the given topic; ties were broken randomly, (iv)
AllWinners (AllW): all the documents are predicted to be winners (only one
document is correctly classified per query), (v) AllLosers (AllL): all the
documents are predicted to be losers (all but one of the documents are
correctly classified per query). We also report the effectiveness of the
classifiers when using only the two feature sets proposed by Raifer et al.
(2017) (Micro and Macro). For this baseline, we show the results only for the
classifier (LReg, LSVM, PSVM, or RForest) with the highest Acc per
competition.

The documents in our competitions were modified with respect to three queries.
We see in Table 3 that when using only the two feature sets proposed by Raifer
et al. (2017), which use information only about the query at hand, the
performance surpasses that of all baselines. This attests to the effectiveness
of these feature sets, whether the documents are modified with respect to a
single query as in (Raifer et al., 2017) or with respect to multiple queries
as in our setting. Furthermore, we see that when adding features that quantify
various properties of all three queries and the corresponding rankings, all
our classifiers consistently and statistically significantly outperform all
the baselines, including that of Raifer et al. (2017). This attests to the
merits of using information about other queries for the same topic to predict
for a given query.

To further examine the effectiveness of the different feature sets, we
performed ablation tests, removing one of the sets each time. We anzlye the
results for PSVM, for which the best overall performance was attained for two
of the four competitions in Table 3. Actual numbers are omitted as they convey
no additional insight and due to space considerations. Except for Macro,
removing each feature set results in at least one statistically significant
drop in performance, attesting to the complementary nature of our features.
Comparing Macro and TopicMacro, the drop for the latter is more substantial.
These findings suggest that these two feature sets might be somewhat
correlated and that TopicMacro, which considers the winners for all queries of
the topic in the previous round, is somewhat more informative in our setting.

##  7\. Conclusions

We presented a novel theoretical and empirical study of the competitive
retrieval setting where document authors modify documents to improve their
ranking for multiple queries. We showed using game theory that in contrast to
past work on document modifications for a single query, in the multiple-
queries setting there is not necessarily an equilibrium and characterized the
cases when it exists. We organized novel ranking competitions where publishers
modified documents for multiple queries representing the same topic. We used
both neural and non-neural rankers, and allowed the use of generative AI tools
in some competitions. We found that publishers use content from documents
highly ranked in the past, but to a somewhat reduced extent when they apply AI
tools. Finally, we addressed the task of predicting which document will be
promoted to the highest rank in the next ranking round. We demonstrated the
merits of using information induced from other queries representing the same
topic to this end.

##  Appendix A Ranking Competitions

###  A.1. Topics, Queries, and Initial Documents

We selected from the UQV dataset (Bailey et al., 2016) topics with commercial
intent that were more likely to incite competition131313The selected topics
are $201$, $203$, $204$, $209$, $210$, $211$, $212$, $218$, $226$, $228$,
$233$, $235$, $244$, $245$, $246$, $249$, $250$, $252$, $255$, $258$, $261$,
$262$, $268$, $272$, $274$, $281$, $283$, $289$, $291$, and $296$.. For each
topic, three queries were used. The first query was the focus of the
backstory, while the other two were selected from all the available query
variants as follows. To ensure that the variants were of high quality, we
retrieved $1000$ documents per variant from the ClueWeb12 category B corpus
using LMIR via the Indri toolkit141414www.lemurproject.org/indri. We filtered
out variants with average precision (AP) lower than $0.05$. In addition, to
ensure that the queries were diverse enough, we computed the Jacquard
similarity between the terms of each pair of queries and the RBO similarity
(with $p=0.9$) (Webber et al., 2010) between the corresponding retrieved
document lists. These two scores were combined using Reciprocal Rank Fusion
(RRF) with $k=60$ (Cormack et al., 2009). Finally, from the remaining query
variants, we iteratively selected the following query with the lowest average
similarity to the already selected queries.

We provided the students with an example of a relevant document for each
topic. We generated these documents using GPT-3.5 with a prompt per topic that
included the backstory and the three selected queries. We continued generating
the document until it was deemed relevant by four annotators. We used the same
document per topic across all four competitions, resulting in $30$ documents.

###  A.2. Additional Guidelines

The students were instructed to create high quality plain text documents
without using any formatting tags, hyperlinks, or special characters. They
were advised to avoid using slang or informal language and were warned against
keyword stuffing. We mentioned that using such techniques would result in a
penalty, although no actual penalty was imposed. Copying parts of other
students’ documents or Web pages was allowed as long as the document also
included original content. Documents that were copied completely from other
documents received no reward. We noticed after the third round that some
students were submitting the initial example document without making any
modifications. As a result, we prohibited submitting the initial documents
starting from the fourth round. Students who still used these documents did
not receive points. We emphasized that the goal was to promote documents in
rankings regardless of their relevance to the information need expressed by
the three queries.

###  A.3. Bots

We used bots in our competitions to increase the number of players and to make
the games more dynamic. The bots generated documents using GPT-3.5. The
prompts instructed the bots to create documents with a maximum length of $150$
terms, but some of the generated documents still exceeded this limit. In such
cases, we removed sentences from the end of a document until its length was
between $140$ and $150$ terms. If no sentence could be removed without making
the document shorter than $140$ terms, we truncated the last sentence to $150$
terms.

We provided our bots with the backstory, the three queries representing the
backstory, and the content and ranks of the bot’s document in the previous
round. In addition, we provided information about other submitted documents
that varied across bots based on two parameters. The first parameter
determined which of the documents submitted in a round were accessible to the
bot. We tested bots that had access to all submitted documents (including
their ranks), the document with the highest median rank across the three
queries (excluding the actual ranks), and the bot’s document only. The second
parameter determined the number of rounds for which the above information was
made available. We tested bots that had information about the three previous
rounds and those that had information about the previous round only. We used
six bot configurations in our competitions, each corresponding to a different
combination of the two parameters. Each bot was assigned to ten different
topics and competed in all ten rounds of the game.

We found that the documents generated by bots were generally ranked lower than
those written by students. The average rank for the bots’ documents was $3.2$,
while the students’ documents had an average rank of $2.6$. Moreover, the
bots’ documents were less likely to win in our competitions. Specifically,
$67.9\%$ of the winning documents were written by students, compared to only
$23.5\%$ that were generated by bots.151515The percentages do not add up to
$100$ as we excluded from this analysis the static bots that submitted the
initial document in all competition rounds.

## References

  * (1)
  * Bailey et al. (2016) Peter Bailey, Alistair Moffat, Falk Scholer, and Paul Thomas. 2016.  UQV100: A Test Collection with Query Variability. In _Proceedings of SIGIR_. 465–474.
  * Ben-Basat et al. (2015) Ran Ben-Basat, Moshe Tennenholtz, and Oren Kurland. 2015.  The Probability Ranking Principle is Not Optimal in Adversarial Retrieval Settings. In _Proceedings of ICTIR_. 51–60.
  * Ben-Basat et al. (2017) Ran Ben-Basat, Moshe Tennenholtz, and Oren Kurland. 2017.  A Game Theoretic Analysis of the Adversarial Retrieval Setting.  _J. Artif. Intell. Res._ 60 (2017), 1127–1164.
  * Bendersky et al. (2011a) Michael Bendersky, W. Bruce Croft, and Yanlei Diao. 2011a.  Quality-biased ranking of web documents. In _Proceedings of WSDM_. 95–104.
  * Bendersky et al. (2011b) Michael Bendersky, Donald Metzler, and W. Bruce Croft. 2011b.  Parameterized concept weighting in verbose queries. In _Proceedings of SIGIR_. 605–614.
  * Castillo and Davison (2010) Carlos Castillo and Brian D. Davison. 2010.  Adversarial Web Search.  _Foundations and Trends in Information Retrieval_ 4, 5 (2010), 377–486.
  * Chen et al. (2023a) Xuanang Chen, Ben He, Le Sun, and Yingfei Sun. 2023a.  Defense of Adversarial Ranking Attack in Text Retrieval: Benchmark and Baseline via Detection.  _CoRR_ abs/2307.16816 (2023).
  * Chen et al. (2023b) Xuanang Chen, Ben He, Zheng Ye, Le Sun, and Yingfei Sun. 2023b.  Towards Imperceptible Document Manipulations against Neural Ranking Models. In _Findings of the Association for Computational Linguistics: ACL_. 6648–6664.
  * Cormack et al. (2009) Gordon V. Cormack, Charles L. A. Clarke, and Stefan Büttcher. 2009.  Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In _Proceedings of SIGIR_. 758–759.
  * Geva et al. (2012) Shlomo Geva, Jaap Kamps, and Ralf Schenkel (Eds.). 2012.  _Focused Retrieval of Content and Structure, 10th International Workshop of the Initiative for the Evaluation of XML Retrieval, INEX 2011, Saarbrücken, Germany, December 12-14, 2011, Revised Selected Papers_.
  * Goren et al. (2018) Gregory Goren, Oren Kurland, Moshe Tennenholtz, and Fiana Raiber. 2018.  Ranking Robustness Under Adversarial Document Manipulations. In _Proceedings of SIGIR_. 395–404.
  * Goren et al. (2020) Gregory Goren, Oren Kurland, Moshe Tennenholtz, and Fiana Raiber. 2020.  Ranking-Incentivized Quality Preserving Content Modification. In _Proceedings of SIGIR_. 259–268.
  * Goren et al. (2021) Gregory Goren, Oren Kurland, Moshe Tennenholtz, and Fiana Raiber. 2021.  Driving the Herd: Search Engines as Content Influencers. In _Proceedings of SIGIR_. 586–595.
  * Gyöngyi and Garcia-Molina (2005) Zoltán Gyöngyi and Hector Garcia-Molina. 2005.  Web Spam Taxonomy. In _Proceedings of AIRWeb 2005, First International Workshop on Adversarial Information Retrieval on the Web_. 39–47.
  * Hartman et al. (2023) Rachel Hartman, Aaron Moss, Shalom Jaffe, Cheskie Rosenzweig, Leib Litman, and Jonathan Robinson. 2023.  Introducing Connect by CloudResearch: Advancing Online Participant Recruitment in the Digital Age.
  * Huang et al. (2017) Sandy Huang, Nicolas Papernot, Ian J. Goodfellow, Yan Duan, and Pieter Abbeel. 2017.  Adversarial Attacks on Neural Network Policies. In _Proceedings of ICLR_.
  * Jia and Liang (2017) Robin Jia and Percy Liang. 2017.  Adversarial Examples for Evaluating Reading Comprehension Systems. In _Proc. of EMNLP_. 2021–2031.
  * Jia et al. (2019) Robin Jia, Aditi Raghunathan, Kerem Göksel, and Percy Liang. 2019.  Certified Robustness to Adversarial Word Substitutions. In _Proceedings of EMNLP-IJCNLP_. 4127–4140.
  * Joachims et al. (2005) Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. 2005.  Accurately interpreting clickthrough data as implicit feedback. In _Proceedings of SIGIR_. 154–161.
  * Kreuk et al. (2018) Felix Kreuk, Yossi Adi, Moustapha Cissé, and Joseph Keshet. 2018.  Fooling End-To-End Speaker Verification With Adversarial Examples. In _Proceedings of ICASSP_. 1962–1966.
  * Kurland and Tennenholtz (2022) Oren Kurland and Moshe Tennenholtz. 2022.  Competitive Search. In _Proceedings of SIGIR_. 2838–2849.
  * Lin et al. (2020) Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2020.  Pretrained Transformers for Text Ranking: BERT and Beyond.  _CoRR_ abs/2010.06467 (2020).
  * Liu et al. (2022) Jiawei Liu, Yangyang Kang, Di Tang, Kaisong Song, Changlong Sun, Xiaofeng Wang, Wei Lu, and Xiaozhong Liu. 2022.  Order-Disorder: Imitation Adversarial Attacks for Black-box Neural Ranking Models. In _Proceedings of SIGSAC_. 2025–2039.
  * Liu (2011) Tie-Yan Liu. 2011.  _Learning to Rank for Information Retrieval_.  Springer. I–XVII, 1–285 pages.
  * Liu et al. (2023) Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Wei Chen, Yixing Fan, and Xueqi Cheng. 2023.  Topic-oriented Adversarial Attacks against Black-box Neural Ranking Models. In _Proceedings of SIGIR_. 1700–1709.
  * Nash (1950) John F. Nash. 1950.  Equilibrium Points in N-Person Games.  _Proc. of the National Academy of Sciences of the United States of America_ 36.1 (1950), 48–49.
  * Nisan et al. (2007) Noam Nisan, Tim Roughgarden, Éva Tardos, and Vijay V. Vazirani (Eds.). 2007.  _Algorithmic Game Theory_.  Cambridge University Press.  Section 19.3.
  * Nogueira and Cho (2019) Rodrigo Nogueira and Kyunghyun Cho. 2019.  Passage Re-ranking with BERT.  _arXiv preprint arXiv:1901.04085_ (2019).
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  _CoRR_ abs/2303.08774 (2023).
  * Papernot et al. (2017) Nicolas Papernot, Patrick D. McDaniel, Ian J. Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. 2017.  Practical Black-Box Attacks against Machine Learning. In _Proceedings of AsiaCCS_. 506–519.
  * Pedregosa et al. (2011) Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake VanderPlas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Edouard Duchesnay. 2011.  Scikit-learn: Machine Learning in Python.  _Journal of Machine Learning Research_ 12 (2011), 2825–2830.
  * Raifer et al. (2017) Nimrod Raifer, Fiana Raiber, Moshe Tennenholtz, and Oren Kurland. 2017.  Information Retrieval Meets Game Theory: The Ranking Competition Between Documents’ Authors. In _Proceedings of SIGIR_. 465–474.
  * Robertson (1977) Stephen E. Robertson. 1977.  The Probability Ranking Principle in IR.  _Journal of Documentation_ (1977), 294–304.  Reprinted in K. Sparck Jones and P. Willett (eds), Readings in Information Retrieval, pp. 281–286, 1997.
  * Song et al. (2022) Junshuai Song, Jiangshan Zhang, Jifeng Zhu, Mengyun Tang, and Yong Yang. 2022.  TRAttack: Text Rewriting Attack Against Text Retrieval. In _Proceedings of RepL4NLP@ACL_. 191–203.
  * Szegedy et al. (2014) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. 2014.  Intriguing properties of neural networks. In _Proceedings of ICLR_.
  * Vasilisky et al. (2023) Ziv Vasilisky, Oren Kurland, Moshe Tennenholtz, and Fiana Raiber. 2023.  Content-Based Relevance Estimation in Retrieval Settings with Ranking-Incentivized Document Manipulations. In _Proceedings of SIGIR_. 205–2014.
  * Wang et al. (2022) Yumeng Wang, Lijun Lyu, and Avishek Anand. 2022.  BERT Rankers are Brittle: A Study using Adversarial Document Perturbations. In _Proceedings of ICTIR_. 115–120.
  * Webber et al. (2010) William Webber, Alistair Moffat, and Justin Zobel. 2010.  A similarity measure for indefinite rankings.  _ACM Trans. Inf. Syst._ 28, 4 (2010), 20:1–20:38.
  * Wu et al. (2022) Chen Wu, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, and Xueqi Cheng. 2022.  PRADA: Practical Black-Box Adversarial Attacks against Neural Ranking Models.  arXiv:2204.01321
  * Wu et al. (2023) Chen Wu, Ruqing Zhang, Jiafeng Guo, Yixing Fan, and Xueqi Cheng. 2023.  Are Neural Ranking Models Robust?  _ACM Trans. Inf. Syst._ 41, 2 (2023), 29:1–29:36.
  * Wu et al. (2010) Qiang Wu, Christopher J. C. Burges, Krysta Marie Svore, and Jianfeng Gao. 2010.  Adapting boosting for information retrieval measures.  _Information Retrieval_ 13, 3 (2010), 254–270.
  * Xie et al. (2017) Cihang Xie, Jianyu Wang, Zhishuai Zhang, Yuyin Zhou, Lingxi Xie, and Alan L. Yuille. 2017.  Adversarial Examples for Semantic Segmentation and Object Detection. In _Proceedings of ICCV 2017_. 1378–1387.
  * Zhai and Lafferty (2006) ChengXiang Zhai and John D. Lafferty. 2006.  A risk minimization framework for information retrieval.  _Inf. Process. Manag._ 42, 1 (2006), 31–55.
  * Zhang et al. (2019) Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. 2019.  Theoretically Principled Trade-off between Robustness and Accuracy. In _Proceedings of ICML_. 7472–7482.