Hypergeometric Orthogonal Polynomials with respect to Newtonian Bases

Hypergeometric Orthogonal Polynomials
with respect to Newtonian Bases\star\star$\star$This paper is a contribution
to the Special Issue on Orthogonal Polynomials, Special Functions and
Applications. The full collection is available at
http://www.emis.de/journals/SIGMA/OPSFA2015.html

Luc VINET † and Alexei ZHEDANOV ‡

L. Vinet and A. Zhedanov

† Centre de recherches mathématiques, Université de Montréal,
† P.O. Box 6128, Centre-ville Station, Montréal (Québec), H3C 3J7 Canada
<EMAIL_ADDRESS>
‡ Institute for Physics and Technology, 83114 Donetsk, Ukraine
<EMAIL_ADDRESS>
Received February 08, 2016, in final form May 07, 2016; Published online May
14, 2016

We introduce the notion of “hypergeometric” polynomials with respect to
Newtonian bases. These polynomials are eigenfunctions
($LP_{n}(x)=\lambda_{n}P_{n}(x)$) of some abstract operator $L$ which is
2-diagonal in the Newtonian basis $\varphi_{n}(x)$:
$L\varphi_{n}(x)=\lambda_{n}\varphi_{n}(x)+\tau_{n}(x)\varphi_{n-1}(x)$ with
some coefficients $\lambda_{n}$, $\tau_{n}$. We find the necessary and
sufficient conditions for the polynomials $P_{n}(x)$ to be orthogonal. For the
special cases where the sets $\lambda_{n}$ correspond to the classical grids,
we find the complete solution to these conditions and observe that it leads to
the most general Askey–Wilson polynomials and their special and degenerate
classes.

abstract hypergeometric operator; orthogonal polynomials; classical orthogonal
polynomials

42C05; 42C15

##  1 Introduction

It is well known that all “classical” orthogonal polynomials $P_{n}(x)$ from
the Askey tableau [7] have rather simple expressions of the form

$\displaystyle P_{n}(x)=\sum_{s=0}^{n}W_{ns}\varphi_{s}(x),$ (1.1)

where $\varphi_{n}(x)$ are the Newtonian basis (interpolated) polynomials
defined as

$\displaystyle\varphi_{0}=1,\qquad\varphi_{n}(x)=(x-a_{0})(x-a_{1})\cdots(x-a_{n-1}),\qquad
n=1,2,\dots$

with the real numbers $a_{i}$ as the interpolation nodes $i=0,1,2,\dots$.

The expansion coefficients $W_{ns}$ in these formulas satisfy the two-term
recurrence relation

$\displaystyle W_{n,s+1}=R_{ns}W_{ns},$ (1.2)

where $R_{ns}$ are simple rational functions either of $n$ or of $q^{n}$. This
property allows to obtain the coefficients $W_{ns}$ explicitly and this leads
to the hypergeometric expressions for the classical orthogonal polynomials.

The recurrence relation (1.2) follows from the fact that the classical
orthogonal polynomials $P_{n}(x)$ satisfy the eigenvalue equation

$\displaystyle LP_{n}(x)=\lambda_{n}P_{n}(x),$ (1.3)

where $L$ is either a differential or difference operator of second order.

It turns out that in all these cases the operator $L$ acts on the Newtonian
basis according to

$\displaystyle
L\varphi_{n}(x)=\lambda_{n}\varphi_{n}(x)+\tau_{n}\varphi_{n-1}(x),$ (1.4)

where $\lambda_{n}$ are the same eigenvalues as in (1.3) and $\tau_{n}$ are
some coefficients.

From (1.3) and (1.4) follows the recurrence relation (1.2) for these
coefficients which takes the form

$\displaystyle\frac{W_{n,s+1}}{W_{ns}}=\frac{\lambda_{n}-\lambda_{s}}{\tau_{s+1}},$

whence

$\displaystyle
W_{ns}=W_{n,0}\frac{(\lambda_{n}-\lambda_{0})(\lambda_{n}-\lambda_{1})\cdots(\lambda_{n}-\lambda_{s-1})}{\tau_{1}\tau_{2}\cdots\tau_{s}},\qquad
s=1,2,\dots,n.$ (1.5)

The coefficient $W_{n0}$ can be chosen arbitrarily. One possible choice is
$W_{n0}=1$ for all $n$. This corresponds to the “hypergeometric-like” form of
the polynomial $P_{n}(x)$. Another choice is the monic form
$P_{n}(x)=x^{n}+O(x^{n-1})$. In this case

$\displaystyle
W_{n0}=\frac{\tau_{1}\tau_{2}\cdots\tau_{n}}{(\lambda_{n}-\lambda_{0})(\lambda_{n}-\lambda_{1})\cdots(\lambda_{n}-\lambda_{n-1})}.$

It is then convenient to present the expansion coefficients as follows

$\displaystyle
W_{n,n-k}=\frac{\tau_{n}\tau_{n-1}\cdots\tau_{n-k+1}}{(\lambda_{n}-\lambda_{n-1})(\lambda_{n}-\lambda_{n-2})\cdots(\lambda_{n}-\lambda_{n-k})},\qquad
k=1,2,\dots,n,$ (1.6)

and we thus have the following expression for the polynomial $P_{n}(x)$:

$\displaystyle
P_{n}(x)=\varphi_{n}(x)+\sum_{k=1}^{n}{\frac{\tau_{n}\tau_{n-1}\cdots\tau_{n-k+1}}{(\lambda_{n}-\lambda_{n-1})(\lambda_{n}-\lambda_{n-2})\cdots(\lambda_{n}-\lambda_{n-k})}\varphi_{n-k}(x)}.$

One can propose an “inverse” problem: assume that the operator $L$ is given in
an abstract form by its action (1.4) with unknown coefficients $\lambda_{n}$,
$\tau_{n}$. The interpolation points $a_{n}$ are unknown as well.

The only restrictions for these coefficients are:

  1. (i)

the spectrum is nondegenerate $\lambda_{n}\neq\lambda_{m}$ for $n\neq m$;

  2. (ii)

the initial conditions for the coefficients $\lambda_{n}$, $\tau_{n}$ are

$\displaystyle\lambda_{0}=\tau_{0}=a_{0}=0.$

  3. (iii)

In the infinite case $\tau_{n}\neq 0$ for all $n=1,2,\dots$.

In the finite case $\tau_{n}\neq 0$ for all $n=1,2,\dots,N$ and
$\tau_{N+1}=0$.

(For the finite case see the discussion in the next section.)

These conditions are quite natural. Indeed, $\tau_{0}=0$ follows from the
truncation condition of the action of the operator $L$ on the constant
$\varphi_{0}(x)=1$. Condition $\lambda_{0}$ can always be achieved by the
appropriate addition of a constant to the operator $L$. Moreover, we will
assume that $a_{0}=0$. Indeed, if $a_{0}\neq 0$, we only need to choose
$\tilde{\varphi}_{n}(x)=\varphi_{n}(x-a_{0})$; the Newtonian polynomials
$\tilde{\varphi}_{n}(x)$ have the same properties with respect to the operator
$L$ and the polynomials $P_{n}(x-a_{0})$ and hence we can always assume that
$a_{0}=0$.

Note that the operator $L$ defined by the abstract relation (1.4) can be
considered in the context of the so-called umbral calculus [11] in which case
concrete operators (like the derivative operator $\partial_{x}$) are replaced
with their abstract symbols when acting on specific bases (For instance the
abstract umbral derivative operator $\cal D$ can be defined by its symbols
$\mu_{n}$ from the abstract relation ${\cal D}x^{n}=\mu_{n}x^{n-1}$; for the
ordinary derivative operator one has $\mu_{n}=n$).

Then it is obvious that the eigenvalue equation (1.3) generates a unique
system of monic polynomials $P_{n}(x)$ having explicit expansion coefficients
(1.6). However, in general, for an arbitrary choice of the coefficients
$\lambda_{n}$, $\tau_{n}$ and $a_{n}$, the polynomials $P_{n}(x)$ will not be
orthogonal.

The main problem is to find the coefficients $\lambda_{n}$, $\tau_{n}$,
$a_{n}$ such that the eigenpolynomials $P_{n}(x)$ are orthogonal with respect
to a nondegenerate linear functional $\sigma$

$\displaystyle\langle\sigma,P_{n}(x)P_{m}(x)\rangle=0,\qquad n\neq m,$

where $\sigma$ can be determined in terms of the moments

$\displaystyle\langle\sigma,x^{n}\rangle=c_{n},\qquad n=0,1,2,\dots.$

The linear functional $\sigma$ is called nondegenerate if the conditions
$\Delta_{n}\neq 0$, $n=0,1,2,\dots$ hold, where
$\Delta_{n}=|c_{i+k}|_{i,k=0}^{n}$ are the Hankel determinants constructed
from the moments.

Equivalently, the polynomials $P_{n}(x)$ are orthogonal if and only if they
satisfy the three-term recurrence relation

$\displaystyle P_{n+1}(x)+b_{n}P_{n}(x)+u_{n+1}P_{n-1}(x)=xP_{n}(x),\qquad
n=1,2,\dots$ (1.7)

with some coefficients $b_{n}$, $u_{n}$.

The linear functional $\sigma$ is nondegenerate if and only if $u_{n}\neq 0$,
$n=1,2,\dots$ [2]. In a special situation when the coefficients $b_{n}$ are
real and $u_{n}>0$, $n=1,2,\dots$ the polynomials are orthogonal with respect
to a positive measure $d\mu(x)$ on the real axis:

$\displaystyle\int_{a}^{b}P_{n}(x)P_{m}(x)d\mu(x)=h_{n}\delta_{nm},$ (1.8)

where $h_{0}=1$ and $h_{n}=u_{1}u_{2}\cdots u_{n}$, $n=1,2,\dots$. The
integration limits $a$, $b$ in (1.8) may be either finite or infinite.

The simplest case of the monomial basis $\varphi_{n}(x)=x^{n}$ was considered
in [20]. In this instance, a full classification was found. This scheme
describes all admissible operators $L$ and corresponding orthogonal
polynomials $P_{n}(x)$. In a nutshell, the analysis performed in [20] led to
the following list of of possibilities:

  1. (i)

$P_{n}(x)$ are the little $q$-Jacobi polynomials and their special and
degenerate cases. The operator $L$ is up to a scaling factor, the basic
hypergeometric operator;

  2. (ii)

$P_{n}(x)$ are the ordinary Jacobi polynomials and their degenerate cases. The
operator $L$ coincides with the ordinary hypergeometric operator;

  3. (iii)

$P_{n}(x)$ are the little $-1$ Jacobi polynomials. The operator $L$ coincides
with the Dunkl type differential operator of the first order [19].

In the present paper we consider the general Newtonian basis and derive the
necessary and sufficient conditions that the coefficients $\lambda_{n}$,
$\tau_{n}$ and $a_{n}$ must satisfy for the associated polynomials to be
orthogonal. Provided the eigenvalues $\lambda_{n}$ are taken to be of the
“classical” form, we also show that this approach leads to all the polynomials
of the Askey scheme (including their $q=-1$ limits). By “classical” we mean
the Askey–Wilson grid $\lambda_{n}=C_{1}q^{n}+C_{2}q^{-n}+C_{0}$ and their
degenerate forms: the quadratic grid $\lambda_{n}=C_{1}n^{2}+C_{2}n+C_{0}$ and
the Bannai–Ito grid $\lambda_{n}=(-1)^{n}(C_{1}n+C_{2})+C_{0}$ with arbitrary
constants $C_{0}$, $C_{1}$, $C_{2}$. We propose also a new classification
scheme of the Askey tableau elements based only on the knowledge of the
coefficients $\lambda_{n}$ and $a_{n}$. This classification method is purely
algebraic and does not depend on any particular choice of the operator $L$
(differential or difference). This provides a natural ground for the hierarchy
of the polynomials in the Askey tableau.

Note that the study of orthogonal polynomials $P_{n}(x)$ having an expansion
(1.1) with respect to the Newtonian basis $\varphi_{n}(x)$ was initiated in
the pioneering paper of Geronimus [6]. For orthogonal polynomials of several
variables, the Newtonian interpolation scheme plays an important role as well
[9, 10].

We here focus exclusively on the infinite-dimensional situation. It should be
said that the finite-dimensional problem has already been considered in [12,
13, 14, 15, 16]. Indeed for finite $N$, the polynomials of hypergeometric type
with respect to a Newtonian basis are displayed in [17].

Note also that in the special case when all interpolation nodes $a_{k}$ are
distinct $a_{i}\neq a_{k}$, $i\neq k$ the abstract hypergeometric polynomials
have a specific duality property. Indeed, let us define the polynomials
$P_{n}^{*}(x)$, $n=0,1,\dots$ by the formula

$\displaystyle
P_{n}^{*}(x)=\sum_{s=0}^{n}\frac{(a_{n}-a_{0})(a_{n}-a_{1})\cdots(a_{n}-a_{s-1})\tilde{\phi}_{s}(x)}{\tau_{1}\tau_{2}\cdots\tau_{s}},$

where $\tilde{\phi}_{n}(x)$ is the (dual) Newtonian basis defined as

$\displaystyle\tilde{\phi}_{n}(x)=(x-\lambda_{0})(x-\lambda_{1})\cdots(x-\lambda_{n-1}).$

Then from (1.1) and (1.5) we have (assuming that $W_{n0}=1$)

$\displaystyle P_{n}(a_{k})=P_{k}^{*}(\lambda_{n}).$ (1.9)

Property (1.9) corresponds to the duality property of polynomials proposed by
Leonard [8].

In the case when both polynomial systems $P_{n}(x)$ and $P_{n}^{*}(x)$ are
assumed to be orthogonal, it is possible to give a full classification of all
such families of polynomials. For the finite-dimensional case this was done by
Leonard in [8]; Bannai and Ito in [1] then extended this result by including
the infinite-dimensional case.

The paper is organized as follows. In Section 2 we derive the necessary and
sufficient conditions for the polynomials $P_{n}(x)$ to be orthogonal. These
conditions are presented in the form of a system of equations for the matrix
$Q_{n}^{(k)}$ of reduced moments.

In Section 3 we show that for the “classical” expressions of the eigenvalues
$\lambda_{n}$, the remaining coefficients $\tau_{n}$ and $a_{n}$ can be found
explicitly. This leads to all entries of the Askey tableau.

In Section 4 we propose a new classification scheme of the polynomials from
the Askey tableau. This classification scheme differs from the known ones and
is based only on the knowledge of the explicit expression of the coefficients
$\lambda_{n}$ and $a_{n}$.

##  2 Necessary and sufficient conditions for orthogonality

One way to derive the necessary and sufficient conditions for the polynomials
$P_{n}(x)$ to be orthogonal is to exploit the three-term recurrence relation
(1.7).

Assume that the polynomials $P_{n}(x)$ are orthogonal which means that these
polynomials satisfy (1.7).

Using formulas (1.1) and (1.7), we obtain a system of equations for the
coefficients $W_{nk}$:

$\displaystyle W_{n+1,k}+(b_{n}-a_{k})W_{n,k}+u_{n}W_{n-1,k}=W_{n,k-1},\qquad
k=0,2,\dots,n+1.$ (2.1)

For $k=n+1$, condition (2.1) becomes trivial: $1=1$. For $k=n$ and $k=n-1$, we
obtain the following explicit expressions for the recurrence coefficients
$u_{n}$, $b_{n}$ in terms of the coefficients $\lambda_{n}$, $\tau_{n}$,
$a_{n}$

$\displaystyle
b_{n}=a_{n}+\frac{\tau_{n}}{\lambda_{n}-\lambda_{n-1}}-\frac{\tau_{n+1}}{\lambda_{n+1}-\lambda_{n}},$
(2.2a) $\displaystyle
u_{n}=\frac{\tau_{n}(a_{n-1}-b_{n})}{\lambda_{n}-\lambda_{n-1}}+\frac{\tau_{n}\tau_{n-1}}{(\lambda_{n}-\lambda_{n-1})(\lambda_{n}-\lambda_{n-2})}-\frac{\tau_{n}\tau_{n+1}}{(\lambda_{n+1}-\lambda_{n})(\lambda_{n+1}-\lambda_{n-1})},$
(2.2b)

where we have used formulas (1.6).

From (2.2b) it is seen that $u_{N+1}=0$ if $\tau_{N+1}=0$. This leads to the
degeneration of the corresponding orthogonal polynomials. This justifies our
condition $\tau_{n}\neq 0$ for all $n=1,2,\dots$. Nevertheless when
$\tau_{n}\neq 0$ for $n=1,2,\dots,N$ but $\tau_{N+1}=0$ we have the special
case of a finite system of polynomials orthogonal on the set of points
$a_{0},a_{1},\dots,a_{N}$ (in this case we should assume that all points
$a_{i}$ are distinct, of course). Indeed, the condition $\tau_{N+1}=0$ means
that

$\displaystyle P_{N+1}(x)=\varphi_{N+1}(x)=(x-a_{0})(x-a_{1})\cdots(x-a_{N}).$

From the standard theory of orthogonal polynomials [2] we have that the
polynomials $P_{n}(x)$, $n=0,1,\dots,N$ satisfy the orthogonality relations

$\displaystyle\sum_{s=0}^{N}P_{n}(a_{s})P_{m}(a_{s})w_{s}=h_{n}\delta_{nm},$

where the weights $w_{s}$ are expressed as

$\displaystyle w_{s}=\frac{u_{1}u_{2}\cdots
u_{N}}{P^{\prime}_{N+1}(a_{s})P_{N}(a_{s})}.$

Thus the condition $\tau_{N+1}=0$ leads to the special case of a finite set of
polynomials $P_{n}(x)$ orthogonal on a set $a_{0},a_{1},\dots,a_{N}$ of
distinct points of the real axis. We will discuss this special case in Section
5.

The equations (2.1) corresponding to $k=n-2,n-3,\dots,0$ give an ensemble of
restrictions upon the coefficients $\lambda_{n}$, $\tau_{n}$, $a_{n}$. Instead
of solving this system of restrictions, we shall follow a method which was
successfully applied in [20] to find necessary and sufficient conditions for
the orthogonality of the polynomials $P_{n}(x)$.

It was shown in [20] that the polynomials $P_{n}(x)$ corresponding to the
operator $L$ are orthogonal if and only if the operator $L$ is symmetric on
the space of polynomials. In more details, this means that for any two
polynomials $f(x)$ and $g(x)$ the condition

$\displaystyle\langle\sigma,f(x)Lg(x)\rangle=\langle\sigma,g(x)Lf(x)\rangle$
(2.3)

must hold.

Note that when $L$ is a differential operator, condition (2.3) is well known
[4]. When $L$ is a higher-order difference operator, condition (2.3) was
derived in [3]. Terwilliger in [16] considers this condition from the
algebraic point of view in the finite-dimensional case.

Condition (2.3) is equivalent to the set of conditions

$\displaystyle\langle\sigma,\varphi_{k}(x)L\varphi_{n}(x)\rangle=\langle\sigma,\varphi_{n}(x)L\varphi_{k}(x)\rangle,\qquad
n,k=0,1,2,\dots.$ (2.4)

Taking into account relation (1.4), we can present (2.4) in the form

$\displaystyle(\lambda_{n}-\lambda_{k})\langle\sigma,\varphi_{k}\varphi_{n}\rangle+\tau_{n}\langle\sigma,\varphi_{k}\varphi_{n-1}\rangle-\tau_{k}\langle\sigma,\varphi_{k-1}\varphi_{n}\rangle=0.$
(2.5)

We shall take the relations (2.5) as the set of necessary and sufficient
conditions for the polynomials $P_{n}(x)$ to be orthogonal.

It is convenient to introduce the generalized moments $\psi_{n}^{(k)}$

$\displaystyle\psi_{n}^{(k)}\equiv\langle\sigma,\varphi_{k}\varphi_{n}\rangle,\qquad
n,k=0,1,2,\dots.$ (2.6)

By definition, these moments possess the symmetry property

$\displaystyle\psi_{n}^{(k)}=\psi_{k}^{(n)},\qquad n,k=0,1,2,\dots.$ (2.7)

The orthogonal polynomials $P_{n}(x)$ can then be presented in determinant
form in terms of the generalized moments

$\displaystyle
P_{n}(x;t)=\frac{1}{H_{n}(t)}\left|\begin{matrix}\psi_{0}^{(0)}&\psi_{1}^{(0)}&\dots&\psi_{n}^{(0)}\\\
\psi_{0}^{(1)}&\psi_{1}^{(1)}&\dots&\psi_{n}^{(1)}\\\
\dots&\dots&\dots&\dots\\\
\psi_{0}^{(n-1)}&\psi_{1}^{(n-1)}&\dots&\psi_{n}^{(n-1)}\\\
\varphi_{0}(x)&\varphi_{1}(x)&\dots&\varphi_{n}(x)\end{matrix}\right|,$

where

$\displaystyle H_{0}=1,\qquad
H_{n}=\left|\begin{matrix}\psi_{0}^{(0)}&\psi_{1}^{(0)}&\dots&\psi_{n-1}^{(0)}\\\
\psi_{0}^{(1)}&\psi_{1}^{(1)}&\dots&\psi_{n-1}^{(1)}\\\
\dots&\dots&\dots&\dots\\\
\psi_{0}^{(n-1)}&\psi_{1}^{(n-1)}&\dots&\psi_{n-1}^{(n-1)}\end{matrix}\right|,\qquad
n=1,2,\dots.$

Note that the nondegeneracy condition of the linear functional $\sigma$ is
equivalent to the condition

$\displaystyle H_{n}\neq 0,\qquad n=0,1,\dots.$

The relations (2.5) can be presented in the form

$\displaystyle(\lambda_{n}-\lambda_{k})\psi_{n}^{(k)}+\tau_{n}\psi_{n-1}^{(k)}-\tau_{k}\psi_{n}^{(k-1)}=0.$
(2.8)

Moreover, from the obvious identity

$\displaystyle\varphi_{k+1}(x)\varphi_{n}(x)=\varphi_{k}(x)\varphi_{n+1}(x)+(a_{n}-a_{k})\varphi_{k}(x)\varphi_{n}(x),$

we obtain the relation

$\displaystyle\psi_{n}^{(k+1)}=\psi_{n+1}^{(k)}+(a_{n}-a_{k})\psi_{n}^{(k)}.$
(2.9)

Consider relation (2.8) for $k=0$. Due to the conditions
$\lambda_{0}=\tau_{0}=0$ we have

$\displaystyle\lambda_{n}\psi_{n}^{(0)}=-\tau_{n}\psi_{n-1}^{(0)}.$ (2.10)

We already know that $\lambda_{n}\neq 0$ for $n=1,2,\dots$. We now show that
similarly $\tau_{m}\neq 0$ for $n=1,2,\dots$. Indeed, assume that $\tau_{j}=0$
for some positive integer $j$. Then from (2.10) we obtain that necessarily
$\psi_{n}^{(0)}=0$ for $n=j,j+1,j+2,\dots$. From (2.9) we have that
$\psi_{n}^{(k)}=0$ for $n\geq j$ and for all $k=0,1,2,\dots$. But then
$H_{n}=0$ for $n\geq j+1$ which would imply that the functional $\sigma$ is
degenerate. We thus see that $\tau_{n}\neq 0$ for $n>0$. Moreover, from the
same relation (2.10), it is seen that $\psi_{0}^{(0)}\neq 0$, $n=0,1,2,\dots$.

We can put $\psi_{0}^{(0)}=1$ (this is merely the standard normalization
condition for the functional $\sigma$). Then from (2.10) we find the explicit
expression for $\psi_{n}^{(0)}$:

$\displaystyle\psi_{n}^{(0)}=(-1)^{n}\frac{\tau_{1}\tau_{2}\cdots\tau_{n}}{\lambda_{1}\lambda_{2}\cdots\lambda_{n}}.$
(2.11)

From relation (2.9), step-by-step, we can find the explicit expressions for
all the generalized moments
$\psi_{n}^{(1)},\psi_{n}^{(2)},\dots,\psi_{n}^{(k)},\dots$. For example

$\displaystyle\psi_{n}^{(1)}=\left(a_{n}-\frac{\tau_{n+1}}{\lambda_{n+1}}\right)\psi_{n}^{(0)}.$

It is convenient to present $\psi_{n}^{(k)}$ as follows

$\displaystyle\psi_{n}^{(k)}=Q_{n}^{(k)}\psi_{n}^{(0)}\psi_{0}^{(k)},$ (2.12)

where $Q_{n}^{(0)}=Q_{0}^{(k)}=1$.

Obviously, the reduced moments $Q_{n}^{(k)}$ are symmetric with respect to the
variables $n$, $k$

$\displaystyle Q_{n}^{(k)}=Q_{k}^{(n)}.$

Moreover, relations (2.9) and (2.8) become

$\displaystyle\zeta_{k}Q_{n}^{(k+1)}=\zeta_{n}Q_{n+1}^{(k)}+(a_{k}-a_{n})Q_{n}^{(k)}$
(2.13a)
$\displaystyle\lambda_{n}(Q_{n}^{(k)}-Q_{n-1}^{(k)})=\lambda_{k}(Q_{n}^{(k)}-Q_{n}^{(k-1)}),$
(2.13b)

where

$\displaystyle\zeta_{n}=\frac{\tau_{n+1}}{\lambda_{n+1}},\qquad
n=0,1,2,\dots.$ (2.14)

Relation (2.13a) allows to determine unambiguously the quantities
$Q_{n}^{(k)}$, $k=1,2,\dots$ iteratively from the initial value
$Q_{n}^{(0)}=0$.

The first two quantities are

$\displaystyle Q_{n}^{(1)}=\frac{\zeta_{n}-a_{n}}{\zeta_{0}}$

and

$\displaystyle
Q_{n}^{(2)}=\frac{(a_{1}-a_{n})(\zeta_{n}-a_{n})}{\zeta_{0}\zeta_{1}}+\frac{\zeta_{n}(\zeta_{n+1}-a_{n+1})}{\zeta_{0}\zeta_{1}}.$

Relations (2.13b) give then additional conditions upon the quantities
$Q_{n}^{(k)}$. These conditions are equivalent to the set of necessary and
sufficient conditions (2.5) for the orthogonality of the polynomials
$P_{n}(x)$. Indeed, we already derived conditions (2.13) from the necessary
and sufficient conditions (2.8). Conversely, assume that $Q_{n}^{(k)}$ is a
solution of conditions (2.13) for $k,n=0,1,2,\dots$ with the initial
conditions $Q_{n}^{(0)}=Q_{0}^{(k)}=1$ and with the symmetry condition
$Q_{n}^{(k)}=Q_{k}^{(n)}$. Then one can construct the moments $\psi_{n}^{(k)}$
from (2.12) and (2.11). These moments will satisfy relations (2.9) and (2.7)
which guarantee that the moments $\psi_{n}^{(k)}$ are compatible with their
definition (2.6). Moreover, the moments $\psi_{n}^{(k)}$ will satisfy the
condition (2.8) for all $n,k=0,1,2,\dots$. The latter condition is equivalent
to the necessary and sufficient conditions (2.4). Hence relations (2.13)
(together with the symmetry and initial conditions) are equivalent to the
necessary and sufficient conditions (2.4).

For example, the corresponding condition coming from (2.13b) for $k=1$ looks
like

$\displaystyle\lambda_{n}y_{n-1}+(\lambda_{1}-\lambda_{n})y_{n}+\tau_{1}=0,$
(2.15)

where we have introduced the new variable

$\displaystyle y_{n}=a_{n}-\zeta_{n}$ (2.16)

instead of $\zeta_{n}$.

Similarly, for $k=2$ we obtain the relation

$\displaystyle(y_{n+1}-y_{n})(\lambda_{n}-\lambda_{2})a_{n}-\lambda_{n}(y_{n}-y_{n-1})a_{n-1}$
$\displaystyle\qquad{}=\lambda_{n}y_{n}(y_{n+1}-y_{n-1})+a_{1}\lambda_{n}(y_{n-1}-y_{n})-\lambda_{2}y_{n}(y_{n+1}-y_{1}).$
(2.17)

It is interesting to note that the reduced moments $Q_{n}^{(k)}$ satisfy
3-term recurrence relations resembling the recurrence relations of the
orthogonal polynomials.

Indeed, in (2.13a) we can shift $k\to k-1$ and then substitute the expression
for $Q_{n}^{(k-1)}$ given in (2.13b). We then obtain the recurrence relation

$\displaystyle\frac{\tau_{n+1}}{\lambda_{n+1}}(\lambda_{k}-\lambda_{n+1})Q_{n+1}^{(k)}+\left(\tau_{n+1}-\tau_{k}+(\lambda_{n}-\lambda_{k})(a_{n}-a_{k-1})\right)Q_{n}^{(k)}$
$\displaystyle\qquad{}+\lambda_{n}(a_{k-1}-a_{n})Q_{n-1}^{(k)}=0,$ (2.18)

where the superscript $k$ is the same and the subscript $n$ takes the values
$n-1$, $n$, $n+1$.

Similarly, in (2.13b) we can shift $k\to k+1$ and then substitute the
expression for $Q_{n}^{(k+1)}$. We obtain another recurrence relation:

$\displaystyle\frac{\tau_{n+1}}{\lambda_{n+1}}(\lambda_{n}-\lambda_{k+1})Q_{n+1}^{(k)}+\left(\tau_{k+1}-\tau_{n}+(\lambda_{n}-\lambda_{k+1})(a_{k}-a_{n})\right)Q_{n}^{(k)}$
$\displaystyle\qquad{}+\lambda_{n}(a_{n-1}-a_{k})Q_{n-1}^{(k)}=0.$ (2.19)

Relations (2.18) and (2.19) are not independent due to relations (2.13).

##  3 General solution for prescribed classical eigenvalues

###  3.1 The case of linear $\boldsymbol{\lambda_{n}}$

The general solution of the necessary and sufficient conditions (2.13) is
rather complicated. Instead we present here a special solution of this
problem, starting with prescribed dependence of $\lambda_{n}$ on $n$. Let us
consider first the simplest case of linear dependence: $\lambda_{n}=\alpha
n+\beta$. Clearly, the scaling transformation
$\lambda_{n}\to\kappa\lambda_{n}$, $\tau_{n}\to\kappa\tau_{n}$ leaves the
problem invariant, so we can choose $\alpha=1$. Moreover, the initial
condition $\lambda_{0}=0$ leads to $\beta=0$. Hence we may take
$\lambda_{n}=n$ without loss of generality.

Consider equation (2.15) with $\lambda_{n}=n$. The general solution of this
equation is elementary

$\displaystyle y_{n}=\gamma n-\tau_{1},$ (3.1)

where $\gamma$ is an arbitrary constant.

Substituting expression (3.1) into equation (2.17) we obtain the equation for
the grid $a_{n}$

$\displaystyle\frac{a_{n-1}}{(n-1)(n-2)}-\frac{a_{n}}{n(n-1)}=\frac{a_{1}}{(n-1)(n-2)}.$

The solution of this equation is immediate:

$\displaystyle a_{n}=a_{1}n+\alpha n(n-1)$ (3.2)

with an arbitrary constant $\alpha$.

From (2.16) we obtain the solution for $\tau_{n}$:

$\displaystyle\tau_{n+1}=(n+1)\left(\tau_{1}+(a_{1}-\gamma)n+\alpha
n(n-1)\right).$ (3.3)

We have thus found the general solution of the problem for the special case
$\lambda_{n}=n$. Instead of checking the compatibility of all conditions
(2.13) with the obtained solution it is sufficient to notice that this
solution describes orthogonal polynomials that are already known.

Indeed, if $\alpha=a_{1}=0$, then $a_{n}=0$, $n=0,1,2,\dots$ and we obtain the
monomial basis $\phi_{n}(x)=x^{n}$. For $\tau_{n}$ we have from (3.3):

$\displaystyle\tau_{n}=n\left(\tau_{1}-\gamma(n-1)\right).$

The corresponding operator $L$ is the second-order differential operator

$\displaystyle L=-\gamma x\partial_{x}^{2}+(x-\tau_{1})\partial_{x}.$ (3.4)

It is well known that the operator (3.4) has Laguerre polynomials as
eigenfunctions [7].

If $\alpha=0$ but $a_{1}\neq 0$, then the grid is linear:

$\displaystyle a_{n}=a_{1}n.$

The coefficient $\tau_{n}$ is a quadratic function in $n$:

$\displaystyle\tau_{n}=n(\tau_{1}+(a_{1}-\gamma)(n-1)).$

This case corresponds to all classical polynomials on the uniform grid:
Krawtchouk, Meixner, Meixner–Pollaczek and Charlier polynomials.

Finally, if $\alpha\neq 0$ we have the quadratic grid (3.2). This corresponds
to the dual Hahn polynomials (including the continuous dual Hahn polynomials)
[7].

The analysis of the case where $\lambda_{n}$ is linear in $n$ expounds all
known classical orthogonal polynomials with such a spectrum. The only
exception are the Hermite polynomials, because the eigenvalue equation for
these polynomials in the monomial basis [7] $x^{n}$ contains the terms $x^{n}$
and $x^{n-2}$ instead of $x^{n}$ and $x^{n-1}$ as required in our approach.

###  3.2 The case of quadratic $\boldsymbol{\lambda_{n}}$

Consider the case when $\lambda_{n}$ is a quadratic polynomial in $n$. Because
$\lambda_{0}=0$ it is sufficient to take

$\displaystyle\lambda_{n}=n(n+\alpha)$ (3.5)

with some real parameter $\alpha$. The only restriction is
$\alpha\neq-1,-2,-3,\dots$ because otherwise $\lambda_{j}=0$ for some positive
integer $j$ and that is forbidden by hypothesis since we are assuming that the
spectrum is non degenerate.

Consider first equation (2.15). Substituting expression (3.5), we get

$\displaystyle\frac{(n+\alpha)y_{n-1}}{n-1}-\frac{(n+\alpha+1)y_{n}}{n}+\tau_{1}\left(\frac{1}{n-1}-\frac{1}{n}\right)=0$
(3.6)

(with $y_{n}$ again as in (2.16)). The solution of (3.6) is immediately found
to be:

$\displaystyle y_{n}=\frac{\gamma n-\tau_{1}}{n+\alpha+1},$ (3.7)

where $\gamma$ is an arbitrary constant.

Equation (2.17) now reads

$\displaystyle\kappa(na_{n-1}-(n-2)a_{n}-a_{1}n)=0,$

where

$\displaystyle\kappa=(\alpha+2)(\tau_{1}+\gamma(\alpha+1)).$

Note that $\kappa\neq 0$. Indeed, $\alpha=-2$ means that $\lambda_{2}=0$ which
is forbidden since the spectrum must be non degenerate. If
$\tau_{1}+\gamma(\alpha+1)=0$, it is easy to show from equation (2.13a) that
all moments are degenerate
$Q_{n}^{(1)}=Q_{n}^{(2)}=\cdots=\operatorname{const}$ which is again forbidden
for degenaracy reasons. We thus conclude that $\kappa\neq 0$ and hence have
the equation

$\displaystyle na_{n-1}-(n-2)a_{n}-a_{1}n=0$ (3.8)

which has the general solution

$\displaystyle a_{n}=a_{1}n+\beta n(n-1)$ (3.9)

with an arbitrary constant $\beta$. (The value of $a_{1}$ is arbitrary as
well, it can be considered as the initial condition for $a_{n}$). We thus
obtained a quadratic grid for $a_{n}$.

Finally, from (2.16) and (3.7), we find the explicit expression for
$\tau_{n}$:

$\displaystyle\tau_{n+1}=\lambda_{n+1}a_{n}+(n+1)(\tau_{1}-\gamma n),$

where $\lambda_{n}$ and $a_{n}$ are given by (3.5) and (3.9).

In general, $\tau_{n}$ is a polynomials of 4-th degree in $n$. In this case
the grid $a_{n}$ is quadratic and the corresponding polynomials are the
Wilson–Racah polynomials. When $\beta=0$ but $a_{1}\neq 0$, then the grid
$a_{n}$ is linear, $\tau_{n}$ is a polynomial of third degree and the
corresponding polynomials are the Hahn polynomials. Finally, when
$a_{1}=\beta=0$, then $a_{n}=0$ for all $n$. The basis $\varphi_{n}(x)=x^{n}$
is the monomial one and the corresponding polynomials $P_{n}(x)$ are the
Jacobi polynomials.

###  3.3 The Askey–Wilson grid

Consider the eigenvalues

$\displaystyle\lambda_{n}=\big{(}1-q^{n}\big{)}\big{(}\alpha-q^{-n}\big{)}$
(3.10)

with an arbitrary real parameter $\alpha$. These eigenvalues correspond to the
Askey–Wilson polynomials.

In this case, the general solution to (2.15) for $y_{n}$ is

$\displaystyle
y_{n}=\frac{\tau_{1}(q-1)^{-1}q^{-n}+\gamma(q^{-n}-1)}{\alpha-q^{-n-1}}$
(3.11)

with $\gamma$ an arbitrary parameter.

Substituting then expression (3.11) into (2.17), we obtain the equation for
the unknown grid $a_{n}$:

$\displaystyle\big{(}q^{n-1}-q\big{)}a_{n}-\big{(}q^{n}-1\big{)}a_{n-1}=a_{1}\big{(}1-q^{n}\big{)}.$
(3.12)

This is a simple first order difference equation which has for general
solution

$\displaystyle
a_{n}=\big{(}1-q^{-n}\big{)}\big{(}a_{1}q(q-1)^{-1}+\nu\big{(}q^{n-1}-1\big{)}\big{)}$
(3.13)

with $\nu$ an arbitrary parameter.

Expression (3.13) corresponds to the grid of the Askey–Wilson polynomials.
Indeed, one checks directly that the recurrence coefficients $b_{n}$, $u_{n}$
constructed with the help of formulas (2.2) correspond to the most general
Askey–Wilson polynomials. There are 4 free parameters: $\tau_{1}$, $a_{1}$ and
the integration constants $\gamma$, $\nu$. These parameters correspond to the
4 parameters of the Askey–Wilson polynomials.

###  3.4 Bannai–Ito grid

Finally, consider the case of the Bannai–Ito eigenvalues [18]

$\displaystyle\lambda_{n}=(-1)^{n}(n+\alpha)-\alpha$ (3.14)

with an arbitrary parameter $\alpha$. Note that for real $\alpha$ the even and
the odd eigenvalues form two equidistant lattices each with step 2. These
sublattices are separated by the gap $\Delta=2\alpha+1$. For the special case
$\alpha=1/2$, these two sublattices can be combined into one equidistant
lattice $\lambda_{n}=\pm 2n$, $n=0,1,2,\dots$.

The eigenvalues (3.14) can be obtained from a specific limit of the AW
eigenvalues (3.10) (see, e.g., [18]). It is instructive however to start from
(3.14) to see how the analysis proposed here applies directly to the BI
lattice.

The first equation, namely, (2.15) can be split into two equations according
to the parity of $n$

$\displaystyle 2ny_{2n-1}-(2n+2\alpha+1)y_{2n}+\tau_{1}=0,$ (3.15)
$\displaystyle 2ny_{2n+1}-(2n+2\alpha+1)y_{2n}+\tau_{1}=0.$ (3.16)

The solution of these equations is readily seen to be

$\displaystyle y_{2n}=\frac{2\gamma n+\tau_{1}}{2n+2\alpha+1},\qquad
y_{2n+1}=\gamma,$ (3.17)

where $\gamma$ is an arbitrary parameter.

Substituting (3.17) for $y_{n}$ into (2.17), we obtain the following equation
for the unknown grid $a_{n}$:

$\displaystyle\kappa\left((n-1)a_{2n}+na_{2n-1}-a_{1}n\right)=0,\qquad\kappa(a_{2n+1}+a_{2n}-a_{1})=0,$
(3.18)

where

$\displaystyle\kappa=\gamma(2\alpha+1)-\tau_{1}.$

As in the previous cases the constant $\kappa$ should be nonzero to avoid
degeneracies. The solution of the equations (3.18) is elementary

$\displaystyle a_{2n}=2\nu n,\qquad a_{2n+1}=a_{1}-2\nu n,\qquad
n=0,1,2,\dots,$

where $\nu$ is an arbitrary parameter.

The coefficients $\tau_{n}$ are then expressed as

$\displaystyle\tau_{n}=\lambda_{n}(a_{n-1}-y_{n-1}),\qquad n=1,2,\dots.$
(3.19)

Using the expressions for $a_{n}$, $\lambda_{n}$, $\tau_{n}$, we can calculate
the recurrence coefficients $b_{n}$, $u_{n}$ with the help of formulas (2.2).
These coefficients coincide with the recurrence coefficients of the Bannai–Ito
polynomials [18] for an arbitrary choice of the parameters $a_{1}$,
$\tau_{1}$, $\gamma$, $\nu$.

We have thus considered all possible choices for the classical spectrum
$\lambda_{n}$; in all cases the orthogonal polynomials that are found coincide
with the corresponding polynomials from the Askey tableau. It is quite natural
to conjecture that there are no other (“non-classical”) expressions for
$\lambda_{n}$ which give rise to orthogonal polynomials. This means that only
the polynomials from the Askey tableau do satisfy the conditions of the
Newtonian “hypergeometric” polynomials. This conjecture remains however an
open problem.

##  4 Algebraic classification of the elements of the Askey tableau

All the elements of the Askey tableau can be characterized naturally within
the unified algebraic “abstract” hypergeometric scheme offered here. We
observed that the broad features of the spectrum $\lambda_{n}$ provide a first
categorization. Indeed, we already know that all classical polynomials from
the “abstract” hypergeometric scheme can be classified by the eigenvalues
$\lambda_{n}$. There are 3 main classes: one corresponding to a generic $q$
(Askey–Wilson class) and two associated to the special cases, $q=1$
(Racah–Wilson class) and $q=-1$ (Bannai–Ito class). Within each of these
classes, the interpolation grids $a_{n}$ further split into 3 different types.
This gives 9 basic classes of classical hypergeometric polynomials.

Consider for example the general Askey–Wilson class. The grid $a_{n}$ may be
taken to be the most general – this gives the generic AW polynomials. If the
grid is exponential (this corresponds to $\nu=0$), we obtain the big
$q$-Jacobi polynomials. Finally, if the grid degenerates to a single point
(i.e. when $\nu=a_{1}=0$), we obtain the little $q$-Jacobi polynomials.

For the Racah–Wilson class ($q=1$), we have 3 subclasses according to the
particular expression (3.8) of the interpolation grid $a_{n}$. If both
parameters $a_{1}$ and $\beta$ are nonzero, we are then dealing with the
general Racah–Wilson polynomials. If $\beta=0$ but $a_{1}\neq 0$, the
interpolation grid becomes linear and we have the Hahn polynomials (including
the case of the continuous Hahn polynomials). Finally, if $a_{1}=\beta=0$, the
grid is then completely degenerate $a_{n}=0$, the corresponding basis is the
monomial one $\phi_{n}(x)=x^{n}$ and we have the Jacobi polynomials.

For the Bannai–Ito class we have again 3 subcategories depending on the
expression of $a_{n}$. If both parameters $a_{1}$ and $\nu$ are nonzero, then
the grid $a_{n}$ is the general Bannai–Ito grid and this leads to the
Bannai–Ito polynomials. If $\nu=0$ but $a_{1}\neq 0$, we have a degenerate
grid which consists of only two distinct points: $a_{2n}=0$ and
$a_{2n+1}=a_{1}$; this case corresponds to the big $-1$ Jacobi polynomials.
Finally, if $a_{1}=\nu=0$ then the grid becomes completely degenerate
$a_{n}=0$ and the corresponding polynomials are the little $-1$ Jacobi
polynomials.

One can consider further subdivisions inside these 9 classes. For example, in
the Racah–Wilson class one may consider the case of linear eigenvalues
$\lambda_{n}$; one then obtains the classical orthogonal polynomials of the
Hahn class and their degenerate types (Meixner, Krawtchouk, Charlier and
Laguerre). Similarly, if one takes the case of the exponential AW-grid
corresponding to $\alpha=0$ in (3.10), one is led to the $q$-Hahn polynomials
and their degenerate and special cases.

##  5 Conclusions

In summary, we derived the necessary and sufficient conditions for the
orthogonality of the “hypergeometric” polynomials with respect to the Newton
interpolation basis. These conditions are presented in the form of a system of
equations (2.13) involving the reduced moments $Q_{n}^{(k)}$. When the
eigenvalues $\lambda_{n}$ are given by “classical” expressions (i.e.,
trigonometric, quadratic or Bannai–Ito grid), we have showed that the
corresponding polynomials $P_{n}(x)$ coincide with the entries of the Askey
tableau. This gives a simple and natural new classification of all polynomials
of the Askey tableau.

Let us remark that the monomial basis (which generates Jacobi, Laguerre and
Hermite polynomials) is a degeneration of the Newton interpolation basis. As
pointed out by a referee, let us mention that the little $q$-Jacobi
polynomials can also be viewed as a special case of the big $q$-Jacobi
polynomials (see, e.g., formula (III.7) in [5]) and hence also associated with
to the same Newtonian basis as the latter.

An unresolved issue is to prove that the general problem with all unknown
coefficients $\lambda$, $\tau_{n}$, $a_{n}$ gives in all likelihood the same
solution (i.e., the polynomials of the Askey tableau). Another interesting
question is to find the explicit expression of the moments $Q_{n}^{(k)}$ for
all polynomial families of the Askey tableau. Indeed, formulas (2.18) and
(2.19) can be considered as three-term recurrence relations for these moments;
one could expect that they are related to the recurrence relations of the
corresponding orthogonal polynomials.

The problem considered here admits several possible generalizations. We
mention only one of them. Fix some positive integer $j$ and assume that the
operator $L$ satisfy the following more general property in the Newtonian
basis $\varphi_{n}(x)$:

$\displaystyle L\varphi_{n}(x)=\sum_{s=0}^{j}\tau_{ns}\varphi_{n-s}(x)$ (5.1)

with some coefficients $\tau_{ns}$ such that $\tau_{n0}=\lambda_{n}$. Then the
eigenvalue problem (1.3) generates again a set of unique monic polynomials
$P_{n}(x)$. The issues examined in the present paper correspond to the
simplest nontrivial case $j=1$. For $j=2,3,\dots$ we can consider similarly
the orthogonality of the corresponding polynomials $P_{n}(x)$. Of course, all
“classical” orthogonal polynomials are obtained as special cases for every
$j$. Indeed, let $L$ be the operator associated to the classical polynomials
$P_{n}(x)$, then the operator
$\tilde{L}=\xi_{j}L^{j}+\xi_{j-1}L^{j-1}+\cdots+\xi_{1}L+\xi_{0}$ (i.e., any
polynomial of the operator $L$ of degree $j$) will trivially have the same
orthogonal polynomials as solutions of the eigenvalue problem

$\displaystyle\tilde{L}P_{n}(x)=\tilde{\lambda}_{n}P_{n}(x).$

The nontrivial question is the existence of non-classical orthogonal
polynomials satisfying (1.3) with “higher-order” operators (5.1). Such non-
classical polynomials appear, e.g., in the solution of the Krall problem that
describes all the orthogonal polynomials satisfying higher-order differential
equations [4]. We can therefore expect the appearance of generalized Krall
polynomials (e.g., those introduced by Durán [3]) as solutions of this
problem.

### Acknowledgments

The authors thank the referees and the editors for valuable remarks and
suggestions. AZ thanks to the Centre de Recherches Mathématiques (Université
de Montréal) for hospitality. The authors would like to thank S. Tsujimoto for
stimulating discussions. The research of LV is supported in part by a research
grant from the Natural Sciences and Engineering Research Council (NSERC) of
Canada.

## References

  * [1] Bannai E., Ito T., Algebraic combinatorics. I, The Benjamin/Cummings Publishing Co., Inc., Menlo Park, CA, 1984, association schemes.
  * [2] Chihara T.S., An introduction to orthogonal polynomials, Mathematics and its Applications, Vol. 13, Gordon and Breach Science Publishers, New York – London – Paris, 1978.
  * [3] Durán A.J., Orthogonal polynomials satisfying higher-order difference equations, Constr. Approx. 36 (2012), 459–486.
  * [4] Everitt W.N., Kwon K.H., Littlejohn L.L., Wellman R., Orthogonal polynomial solutions of linear ordinary differential equations, J. Comput. Appl. Math. 133 (2001), 85–109.
  * [5] Gasper G., Rahman M., Basic hypergeometric series, Encyclopedia of Mathematics and its Applications, Vol. 96, 2nd ed., Cambridge University Press, Cambridge, 2004.
  * [6] Geronimus J., The orthogonality of some systems of polynomials, Duke Math. J. 14 (1947), 503–510.
  * [7] Koekoek R., Lesky P.A., Swarttouw R.F., Hypergeometric orthogonal polynomials and their $q$-analogues, Springer Monographs in Mathematics, Springer-Verlag, Berlin, 2010.
  * [8] Leonard D.A., Orthogonal polynomials, duality and association schemes, SIAM J. Math. Anal. 13 (1982), 656–663.
  * [9] Okounkov A., On Newton interpolation of symmetric functions: a characterization of interpolation Macdonald polynomials, Adv. in Appl. Math. 20 (1998), 395–428.
  * [10] Rains E.M., ${\rm BC}_{n}$-symmetric polynomials, Transform. Groups 10 (2005), 63–132, math.QA/0112035.
  * [11] Roman S., The theory of the umbral calculus. I, J. Math. Anal. Appl. 87 (1982), 58–115.
  * [12] Terwilliger P., Two linear transformations each tridiagonal with respect to an eigenbasis of the other, Linear Algebra Appl. 330 (2001), 149–203, math.RA/0406555.
  * [13] Terwilliger P., Leonard pairs from 24 points of view, Rocky Mountain J. Math. 32 (2002), 827–888, math.RA/0406577.
  * [14] Terwilliger P., Leonard pairs and the $q$-Racah polynomials, Linear Algebra Appl. 387 (2004), 235–276, math.QA/0306301.
  * [15] Terwilliger P., Two linear transformations each tridiagonal with respect to an eigenbasis of the other; the TD-D canonical form and the LB-UB canonical form, J. Algebra 291 (2005), 1–45, math.RA/0304077.
  * [16] Terwilliger P., Two linear transformations each tridiagonal with respect to an eigenbasis of the other: comments on the split decomposition, J. Comput. Appl. Math. 178 (2005), 437–452, math.RA/0306290.
  * [17] Terwilliger P., Two linear transformations each tridiagonal with respect to an eigenbasis of the other; comments on the parameter array, Des. Codes Cryptogr. 34 (2005), 307–332, math.RA/0306291.
  * [18] Tsujimoto S., Vinet L., Zhedanov A., Dunkl shift operators and Bannai–Ito polynomials, Adv. Math. 229 (2012), 2123–2158, arXiv:1106.3512.
  * [19] Vinet L., Zhedanov A., A ‘missing’ family of classical orthogonal polynomials, J. Phys. A: Math. Theor. 44 (2011), 085201, 16 pages, arXiv:1011.1669.
  * [20] Zhedanov A., Abstract “hypergeometric” orthogonal polynomials, arXiv:1401.6754.


# Language Models Learn Rare Phenomena from Less Rare Phenomena:
The Case of the Missing AANNs

Kanishka Misra Kyle Mahowald
Department of Linguistics
The University of Texas at Austin
<EMAIL_ADDRESS>
###### Abstract

Language models learn rare syntactic phenomena, but it has been argued that
they rely on rote memorization, as opposed to grammatical generalization.
Training on a corpus of human-scale in size (100M words), we iteratively
trained transformer language models on systematically manipulated corpora and
then evaluated their learning of a particular rare grammatical phenomenon: the
English Article+Adjective+Numeral+Noun (AANN) construction (“a beautiful five
days”). We first compared how well this construction was learned on the
default corpus relative to a counterfactual corpus in which the AANN sentences
were removed. AANNs were still learned better than systematically perturbed
variants of the construction. Using additional counterfactual corpora, we
suggest that this learning occurs through generalization from related
constructions (e.g., “a few days”). An additional experiment showed that this
learning is enhanced when there is more variability in the input. Taken
together, our results provide an existence proof that models learn rare
grammatical phenomena by generalization from less rare phenomena. Code
available at https://github.com/kanishkamisra/aannalysis.

Language Models Learn Rare Phenomena from Less Rare Phenomena:
The Case of the Missing AANNs

Kanishka Misra Kyle Mahowald Department of Linguistics The University of Texas
at Austin<EMAIL_ADDRESS>

Figure 1: A sampling of some of our key experimental manipulations and
resulting performance changes. We iteratively train LMs on varied input
corpora and measure the learning of the rare aann (“a beautiful five days”) or
related constructions. E.g., we train on the default BabyLM corpus, a corpus
in which we remove all aanns, a corpus in which we replace all aanns with a
corrupted version (“beautiful a five days”) and measure learning of the
corrupted version, and a corpus in which we remove aanns and also remove
related constructions like “a few weeks is”.

##  1 Introduction

###  1.1 Motivation and Prior Work

Humans come to learn and use rare grammatical structures, even if they have
encountered those structures only rarely or even not at all (Pullum and
Scholz, 2002; Pearl, 2022). For instance, humans accept the grammaticality of
the PiPP construction (“surprising though it may be…”) even where the preposed
element crosses a finite close boundary (“surprising though I know it may be
that…”) (Pullum, 2017) and even though they may plausibly have never
encountered such a sentence in their linguistic experience.111Potts (2023)
found that the finite-clause-crossing PiPP occurred in only 56 sentences out
of a corpus of over 7 billion sentences, which is sufficiently rare that an
adult English speaker may never have encountered this construction! How people
come to know an utterance is grammatical has occupied a central place in
linguistics. Specifically, mastery of never-before-encountered grammatical
structures has been taken to mean that people are endowed with innate
linguistic knowledge (Chomsky, 1986, 1957, 1965).

Recent evidence, though, suggests that Large Language Models (LLMs) can learn
complex grammar (Wilcox et al., 2018; Futrell et al., 2019; Linzen et al.,
2016; Mahowald et al., ) even from human-scale amounts of input (Warstadt et
al., 2023; Eldan and Li, 2023; Huebner et al., 2021). This raises the
possibility that input data, along with an appropriately sophisticated or
weakly biased statistical learning mechanism, is sufficient for learning rare
constructions by allowing for models to emergently learn appropriate
grammatical abstraction (Baroni, 2022; Piantadosi, 2023).

But modern LLMs often have access to much more training input than people do
and thus might memorize rare constructions in a way that humans can’t
(Warstadt and Bowman, 2022; Warstadt et al., 2023). The possibility that LLMs
are “stochastic parrots” (Bender et al., 2021) that rely heavily on
memorization, much more so than humans, is a common criticism of using LLMs to
study human language (e.g., Chomsky et al., 2023).

There are different levels of memorization, though, requiring different levels
of abstraction. Consider the relatively rare AANN construction: “a beautiful
five days in Texas” (Solt, 2007; Keenan, 2013; Dalrymple and King, 2019),
which is interestingly distinct from the default “five beautiful days in
Texas”. A model that strictly memorizes this construction might come to know
that “a beautiful five days in Texas” is grammatical but have no idea that “a
beautiful four days in Texas” is grammatical if it never appeared in its
training. A model that generalizes just a bit more might know that “a
beautiful five days in New York” is also grammatical by analogizing that a
U.S. state can fill the slot. One that generalizes even more might find “a
beautiful three hours at the mall” to be acceptable. Knowing that “an
astonishing 200 pages” is acceptable requires generalization beyond mere
lexical items. And knowing that “a blue five pencils” is not acceptable
(because the adjective is “stubbornly distributive”, Schwarzschild 2011)
requires yet more generalization. Even for an idealized learner, it is
difficult to precisely formulate how many examples are required to get the
right kinds of generalizations for even this one construction.

There is increasing evidence that LLMs can generate novel linguistic
utterances (McCoy et al., 2023), and also make subtle judgments on relatively
rare constructions like these (Weissweiler et al., 2022; Potts, 2023),
including the AANN construction (Mahowald, 2023). If they do so by memorizing
examples verbatim from an unrealistically large input corpus, that would not
be particularly informative for human processing. But, if they do learn rare
grammatical phenomena from smaller amounts of data and can generalize beyond
just those verbatim instances, that would raise the question of how they do
it. For instance, in the context of the PiPP construction, Potts (2023)
speculates that the comparative construction (e.g., “They are happier than we
are.”) “may be the key to all of this [i.e., learning the PiPP]” because such
constructions are “incredibly common” yet share abstract structure with the
PiPP. If LLMs learn rare grammatical structures in part by learning and
generalizing structures from much more common constructions, that would be
powerful evidence for abstraction in LLMs and raise the possibility that even
quite general learners could learn very rare phenomena without strong innate
priors.

To that end, our goal in this paper is to study a rare grammatical phenomenon
in LMs that are trained on controlled input corpora and that are (a) of human
realistic scale, and (b) systematically manipulated with respect to the target
constructions as well as key related constructions. Our hypothesis is that
generalization abilities of LMs on such rare phenomena come from abstractions
and structures learned from more frequent phenomena—and that the more frequent
phenomena are the “keys to all of this.”

As a case study, we focus on the aforementioned aann construction, although we
highlight how the methods used here could serve as a blueprint for work on
other rare grammatical phenomena. Our method is to train different
instantiations of a standard transformer model on the 100M-word BabyLM corpus,
which we systematically manipulate—via removal and replacement—to shed light
on the extent to which frequent and related phenomena encountered during
training can give rise to abstractions that allow for generalizations in LMs.
To test for generalization, we subjected our different LMs to a series of
acceptability tests on sentences which do not appear in the training corpus
and which specifically target the special properties of the aann construction.
This approach of training on a systematically manipulated corpus has been
fruitfully used to debias models (Lu et al., 2020; Maudslay et al., 2019),
study whether LMs rely on lexical knowledge in learning syntactic rules (Wei
et al., 2021), explore the effect of permuting words on pretrained models
(Sinha et al., 2021), and test whether LMs can learn languages judged to be
hard for humans (Kallini et al., 2024), among others. Our experiments yielded
three main findings, summarized below.

###  1.2 Summary of findings

First, we find BabyLM-trained LMs to successfully generalize to novel
instances of the aann construction. Performance unsurprisingly drops for LMs
that were trained without being exposed to even a single aann during training,
but perhaps surprisingly, not by all that much—they are substantially above
chance. This suggests that certain items present in the training data might
give rise to LMs’ non-trivial performance in judging acceptability of aanns.
This finding is further strengthened by the fact that LMs trained on
counterfactual variants of the aann—e.g., anan and naan, obtained by shuffling
word order and are far less likely to share structure with training data
items—are unable to generalize to those constructions as well as they do to
aanns.

Next, we investigated what might enable LMs’ learning of the aann, by further
systematically manipulating their training data to hold out utterances
conforming to specific linguistic and statistical phenomena. Through our
manipulations, we find LMs become increasingly worse at predicting novel
instances of the aann as more frequent, non-AANN-but-AANN-related phenomena
are held out. For example, phenomena such as the treatment of measure noun
phrases as singular (e.g., a few days is all we need)—similar to how aanns
treat a plural NP as singular—end up making unseen aanns less likely by 31% on
average. Importantly, these results could not be explained simply by loss of
data—LMs that were trained with these phenomena left in but without an
equivalently large chunk of the training data removed were almost as good as
LMs that never saw aanns. This further strengthens the conclusion that the
hypothesized linguistic phenomena did indeed affect generalization of the
targeted construction. Notably, LMs are largely affected by these
manipulations when they do not see any aanns during training, highlighting the
expected non-trivial role of encountering some instances of aanns to show
stronger generalization.

Finally, we characterized the aforementioned interplay between the properties
of the encountered aanns and the LMs generalizations on novel instances. Here
we found LMs that observed aanns with more variability on the adjective,
numeral, and noun slots to show better generalization than did LMs that saw
more restricted-but-repeating instances of aanns. This importantly mimicked
analogous findings of inductive inference in humans across linguistics
(Goldberg, 1995, 2005; Suttle and Goldberg, 2011; Baayen, 2009; O’Donnell,
2015) and cognitive psychology studies (Osherson et al., 1990; Xu and
Tenenbaum, 2007).

Taken together, these results provide an existence proof that a weakly biased
but sophisticated general-purpose statistical learner can in fact learn rare
and complex linguistic phenomena, and that this can arise as a result of the
presence of key related constructions during training.

##  2 General Methods

Here we describe methods that we use to characterize learning of the targeted
rare construction, the aann. As mentioned in Section 1, we take these to be a
general blueprint which can be used to study other rare constructions.

###  2.1 Data and Model

#### Corpus

Throughout the paper, we use the BabyLM-strict corpus (Warstadt et al., 2023)
as our main training set. We use this primarily because of its tractable size
(100M tokens), which allows us to (1) detect and control the instances of the
target construction as well as related linguistic phenomena; and (2) train a
whoppingly large amount of LMs in sufficient time. Since much of our
experiments involve ablating constructions present in this corpus, a majority
of the LMs are trained on ablated versions of BabyLM.

#### Construction Detection

We use regexes over a POS-tagged version of the BabyLM training data to detect
aann instances.222We use spaCy for getting POS-tags, see Appendix 0.B for the
exact details of the detection pipeline. We detect 2,301 aanns in the BabyLM
corpus, which accounts for about 0.02% of all utterances, making construction
quite rare for learners trained on this corpus.

#### Model

Our LMs are instances of the OPT LM (Zhang et al., 2022), which is an
autoregressive transformer architecture. Our LMs have 12 layers and attention
heads, use a vocabulary size of 16,384, and are trained for a maximum of 20
epochs using the transformers library (Wolf et al., 2020). The results we
report for a given LM in our experiments are averaged over three different
runs (with different random seeds). We list all other hyper-parameters and
architectural details in Appendix 0.A.

###  2.2 Acceptability data

To test our LMs on their knowledge of the aann, we use data from Mahowald
(2023), which consists of 12,960 templatically generated sentences that
contain aanns. Out of these, 3,420 contain acceptability ratings provided by
190 human participants, ranging from 1 (unacceptable) to 10 (acceptable). We
use 7 as the threshold for clear acceptability, in that we only keep instances
where human participants rated the acceptability of the construction in
context to be greater than 7. We additionally discarded instances where the
aanns appear in the BabyLM training set ($n=4$), as testing on these would not
shed light on the LMs’ generalization behavior. This leaves us with 2,027
items.

For each aann instance in the dataset, Mahowald (2023) has also made available
its corresponding corrupted versions, which focus on (1) adjective-numeral
order; (2) presence of the article; (3) presence of the adjective; and (4)
presence of the numeral. A hypothetical example of these corruptions is shown
in Table 1 under the “aann” column. A model that has knowledge of the aann
should find the well-formed instance to be more likely than each of its
corrupted versions. Below we describe methods to measure likelihood and assess
accuracy on these tests.

###  2.3 Measurements

#### Scoring

We use the Syntactic Log-odds Ratio (SLOR) proposed by Pauls and Klein (2012);
Lau et al. (2017), to score sentences in our tests. Given a sentence
containing a prefix followed by our target construction $\mathcal{C}{}$ and an
optional suffix, slor is computed as the log of the ratio between the
probability of the construction given the prefix as estimated by the LM, and
that estimated by a unigram model, normalized by the length of the
construction. Given a language model $m$ and a unigram estimator $u$:

$\displaystyle\textsc{slor}{}_{\text{prefix}}(\mathcal{C}{})=\frac{1}{\mid\mathcal{C}{}\mid}\log\frac{p_{m}(\mathcal{C}{}\mid\text{prefix})}{p_{u}(\mathcal{C}{})}$
(1)

Context | aann | anan | naan
---|---|---|---
Well-formed |  a whopping ninety LMs |  a ninety whopping LMs |  ninety whopping a LMs
Corruptions
Order-swap |  a ninety whopping LMs |  a whopping ninety LMs |  whopping ninety a LMs
No article |  whopping ninety LMs |  ninety whopping LMs |  ninety whopping LMs
No modifier |  a ninety LMs |  a ninety LMs |  ninety a LMs
No numeral |  a whopping LMs |  a whopping LMs |  whopping a LMs

Table 1: Well-formed and corrupted examples of the aann construction and its
counterfactual versions (anan and naan). Corruption types are taken from
Mahowald (2023). Note that in the actual data, these constructions are
embedded in full sentences. Figure 2: Accuracies on tests for aann and its
counterfactuals (anan and naan), achieved by LMs trained on BabyLM with
various aann-manipulations (aann as is, no aann, anan, naan). The dashed line
represents chance performance (6.25%), the dot-dashed line represents
accuracies for slors computed using 2- and 4-gram LMs trained on BabyLM.
Accuracies for GPT-2-XL (Radford et al., 2019) and Llama-2-7B (Touvron et al.,
2023) are computed using log-probabilities, since unigram frequencies were
unavailable for these LMs’ corpora.

Importantly, we train the unigram estimator for a given corpus using the same
tokenizer used to train our autoregressive LMs on that corpus. We use slor in
lieu of the usual normalized log-probability measure to ensure that the
comparison between two models cannot be explained simply by the difference in
unigram frequencies brought about by our manipulations. All log-probabilities
were computed using minicons (Misra, 2022).

#### Accuracy

Using slor values computed over test instances, we calculate the accuracy of a
model as:

$\displaystyle\frac{1}{N}\sum_{i=1}^{N}\mathbbm{1}\left[\bigcap\limits_{k}^{4}\left\\{\frac{\textsc{slor}{}_{\text{prefix}_{i}}(\mathcal{C}{}_{i})}{\textsc{slor}{}_{\text{prefix}_{i}}(\mathcal{C}{}^{k}_{i})}>1\right\\}\right],$
(2)

where, $\mathbbm{1}$ is the indicator function, and $k$ enumerates all four of
our corruptions, and $N$ enumerates the test instances. That is, an instance
within our test set is considered to be correct iff. the score on the well-
formed construction, $\mathcal{C}_{i}$, is greater than the score for _all_
corrupted instances, $\mathcal{C}^{k}_{i}$. The accuracy, then, is the
proportion of correct instances within the test set. Since this involves four
pairwise comparisons, chance performance is 6.25%.

###  2.4 Ablations

Common to subsequent experiments (Section 4 and Section 5) is the fact that we
hold out certain parts of the BabyLM corpus—parts that conform to a certain
linguistic or statistical hypothesis. This creates a gap between the
experience of LMs trained on these ablated versions of the corpus, and that of
the LM trained on the full BabyLM data. To circumvent this issue, we up-sample
non-hypothesis-conforming utterances in BabyLM after performing our ablations,
in a manner such that the LM still ends up encountering the exact same amount
of tokens in such cases. As a result, all our LMs see the same quantity of
tokens, but differ in content.

##  3 Experiment 1: LMs learn about aanns without having seen a single
instance

###  3.1 LMs learn about aanns…

To investigate the extent to which LMs trained on BabyLM learn the aann
construction, we measure their accuracy on our tests described in Section 2.2.
From Figure 2, we observe that the BabyLM-trained LMs obtain accuracies around
70%, which is substantially above chance. This suggests that LMs can
reasonably acquire generalizations to aanns from exposure to positive evidence
that makes up only 0.02% of their training experience.

For comparison to larger, state-of-the-art LMs, we test Llama-2-7B (Touvron et
al., 2023) and GPT-2 XL (Radford et al., 2019) on the aanns, and find them to
achieve 83% and 78%, respectively.333Qualitatively, performance is on these
models is worse than data reported in Mahowald (2023), which used GPT-3.
Because log-probabilities are not available from GPT-3, a direct comparison is
not possible. Similarly, as a comparison to shallower LMs, we tested on 2-and
4-gram LMs trained on BabyLM and found both of them to achieve 0% accuracy,
eliminating the possibility that the observed results are due to surface level
statistics.

###  3.2 …without having seen a single instance…

Given that BabyLM-trained LMs learn the aann construction, how well would an
LM generalize to aanns without having seen a single positive instance? To this
end, we compare accuracies in the previous experiment to that obtained by LMs
trained on BabyLM with our 2,301 detected aanns removed (i.e., no aann).

From Figure 2, we find LMs trained with the no aann condition to achieve an
average accuracy of 54%, which is a noticeable drop compared to the 70%
obtained by the LMs trained on the complete BabyLM corpus, but importantly
47.75 points above chance performance (and, as we show below, well above
comparable baselines with other constructions). This is a non-trivial result,
since it suggests that LMs are able to learn the acceptability of a
construction’s instances, without having seen a single positive occurrence,
which indicates that there exist some systematic patterns in the corpus that
are driving this generalization behavior.

###  3.3 …more strongly than they learn counterfactual aann variants

To further contextualize the above results, we consider two counterfactual
cases, where we replaced aanns in BabyLMs with instances involving the same
lexical items, but in a word order that violates English grammar: (1) anan
(e.g., a 90 whopping LMs); and (2) naan (e.g., 90 whopping a pages). This
allows us to test if the results before are genuinely because LMs recognize
the nuances of the aann construction, or whether they can learn alternative
variants of the construction, which likely share fewer properties with other
more frequent utterances in the data. If LMs are able to learn these
constructions just as well as the the LMs in the previous experiments learned
aanns, then the generalization claims from the previous experiments would be
weakened.

To test for such possibilities, we create counterfactual versions of the
Mahowald (2023) stimuli, where we apply analogous corruptions to the
counterfactual variants of aann, such that they are violated in a similar
manner as in the aann tests. Examples for the three types of instances in our
tests can be found in Table 1. We then evaluate the previous two LMs (trained
on BabyLM with and without seeing any aanns) with LMs trained on BabyLM with
these counterfactual variants on judging the acceptability of aanns, anans,
and naans. Figure 2 shows these results.

From Figure 2, we make two high-level observations. First, and most
importantly, LMs that see anans and naans do not learn those constructions as
well as they learn aanns—especially the LM that saw no aanns (54% aann
accuracy compared to 36% naan accuracy obtained by the naan-trained LM).
Second, these LMs end up learning aanns more strongly relative to how well
they learn constructions that they observe in lieu of the aann—e.g., naan
trained LM achieves around 54% accuracy on aanns even though naans appeared
frequently in the data set and no aanns did. This, combined with the results
of the previous two sub-experiments suggests strongly that LMs pick up on cues
from other—likely related—constructions encountered during training in order
to assign non-trivial probability to unseen instances of aanns.

Figure 3: slors on aanns from Mahowald (2023) for our LMs (left) and a 4-gram
baseline (right) trained on BabyLM and its ablated versions. Our LMs show a
range of hypothesized effects, whereas the 4-gram baselines only show an
effect of the adjective/numeral balanced condition. slor for unablated BabyLM-
trained LM shown with dotted line.

Phenomenon/Manipulation | Example/Desc. | Freq.
---|---|---
AANN | a fine eighteen months | 2,301
DT ANN |  | the usual forty dollars fine
---
14,347
| A few/couple/dozen/etc. NNS
---
a few plums | 55,226
| Measure NNS with SG verbs
---
and/or indef. articles
6 months is a long time | 62,597
A/An + ADJ/NUM balancing |  | enforce freq. balance
---
571,874
Random removal (control) | randomized ablation | 571,874

Table 2: Manipulated Phenomena, their examples/descriptions, and their
frequency in the BabyLM corpus.

##  4 Experiment 2: Keys to Learning aanns

Our previous experiments reveal that, keeping everything else the same, LMs
learn the aann construction far more accurately than they do its
counterfactual variants. Furthermore, we also see strong aann acceptability
judgments in LMs that have never encountered a single instance. This suggests
that there are instances in the training data that contribute to the learning
of the construction.

What might these be? Below we enumerate four hypotheses, each of which tackles
subtly different aspects of the aann construction, then measure the effect of
these phenomena, by separately holding them out during training and computing
the average slor of the well-formed instances of the aann tests. The effect of
a particular phenomenon on the acceptability of aanns can therefore be
measured by comparing slors before and after holding out instances of that
phenomenon. Methods for detecting the hypothesized phenomena can be found in
Appendix 0.B. As control, we additionally also hold out a random set of
utterances, which do not conform to the target phenomena of interest. Note
again that for all these cases, we ensure the LMs see the same number of
tokens during training, by up-sampling other, non-aann containing sentences.
Table 2 lists the hypotheses we consider, along with an example of their
utterance and frequency of occurrence, in the BabyLM corpus.

#### The presence of “the ANN”

Phrases like “the beautiful five days” are common in corpora, which are not as
unusual because “the” regularly takes plural nouns. We hypothesize that the
acceptability of these structures affects the acceptability of aanns, since an
LM might analogize from the general observation that ‘a’ or ‘an’ can
substitute ‘the’ (e.g., a ball vs. the ball). Therefore, we consider all cases
wheere a determiner precedes the contiguous sequence of article, numeral,
plural noun.

#### A few/couple/dozen/etc. NNS

Another related phenomenon that is more common relative to the aann
construction involves phrases such as “a few days” or “a couple bottles”. To
an LM learner, they might provide evidence that measure noun phrases with
plural nouns can be attached to an indefinite article (a/an), as is the case
in aanns.

#### Measure NNS treated as singular

We consider yet another phenomenon involving phrases that treat measure nouns
as singular, this time in terms of agreement, e.g., “Five miles is a long way
to go”, and “1,000 pages is a lot for a dissertation.” These cases might
provide further evidence to the model that measure noun phrases with plural
nouns can be treated as a singular unit, thereby affecting the acceptability
of the aann. These are less prevalent compared to the cases involving a
few/couple/dozen NNS, but still far more frequent than the aann, therefore, we
combine the two as a general case of treating measure NPs as singular.

#### Balancing the frequencies of A/An + ADJ/NUM

A more surface-level reason why “a beautiful five days” might be more natural
to LMs than is “a five beautiful days”, could be that adjectives are more
likely to follow indefinite articles than are numerals. For instance,
adjectives are $\approx$14.6 times more likely to follow indefinite articles
in the BabyLM corpus than are numerals. To measure this effect, we hold out
instances such that adjectives and numerals are equally likely to follow an
indefinite article. This ends up being the largest portion of the data that we
hold out.

#### Control: Random removal

A potential confound in the above ablations could be that the slor values of
the aanns go down merely due to loss of content—this could be despite the fact
that we add back additional tokens from BabyLM (such that all LMs see the
exact same amount of tokens). To account for this, we additionally consider a
control condition where we remove as many tokens as in the largest ablation
(i.e., the A/An + ADJ/NUM case) such that none of the above phenomena are
taken out. Insofar as the LMs rely on the aforementioned phenomena, results on
LMs trained with this ablation should be closer to the original BabyLM-trained
LM’s results.

###  4.1 Analysis and Results

In our experiments, we individually ablate out each of the aforementioned
phenomena under two settings: (1) aanns are removed during training in
addition to the phenomena; and when possible, (2) aanns are seen during
training. (1) is a stricter setting, since here the LMs see neither the target
phenomenon nor a single instance of the aann. Comparing average slors obtained
in this condition to that obtained for the no aann can shed light on the the
extent to which the target phenomenon is critical in allowing LMs to assign
non-trivial probabilities on unseen aanns, zero-shot. On the other hand, (2)
still allows for the LMs to perform lexical generalization (Kim et al., 2022)
from seen aann instances—where they may arrive at strong acceptability
measures on the test aanns by performing slot filling, without necessarily
relying on the hypothesized phenomena (characterization of which is equally
important).

Figure 3 shows the average slors obtained across various ablations under the
two settings. As a baseline, we compare our results to 4-gram LMs444Trained
using KenLM (Heafield, 2011) trained on corresponding ablations of the BabyLM
corpus. We observe that holding out all our hypothesized phenomena has non-
trivial effects on our LMs’ ratings of unseen well-formed aanns, and that
these effects are different for when aanns are seen during training, or are
held out. When aanns are held out along with the phenomena, we see substantial
drops in the average slor values assigned by LMs on unseen aanns relative to
that assigned by LMs in the no aann condition. Specifically, balancing the
frequency of adjectives and numerals following an article has the greatest
effect, followed by the two cases where measure nouns are treated as singular,
followed by the removal of all cases that involve determiner + adjective +
numeral + noun sequences. This suggests that in the absence of even a single
aann during training, these phenomena are critical for LMs to assign non-
trivial likelihoods to aanns. Importantly, simply ablating large amounts of
data cannot explain these results, since LMs trained on our controlled
condition show higher slor values than in all our hypothesis-informed
ablations. These patterns are absent in 4-gram LMs, suggesting that they do
not arise as a result of shallow, surface statistics, with the slight
exception of differences observed in the case of the article+adjective/numeral
ablation. Overall, this is a noteworthy finding, since it indicates that LMs
can demonstrate a completely novel phenomenon (aann) by relying on other
related—and more frequent—phenomena.

When aanns are seen during training, however, we observe LMs’ results on
unseen aanns to show more similar slor values with respect to the the LMs
trained on the unablated BabyLM corpus, although they are still significantly
reduced in some cases (e.g., singular measure nouns).

We conclude that when LMs see evidence of the aann construction, they do learn
from it. But key related phenomena where measure nouns are treated as singular
do seem to show some notable effects even when AANNs are present, suggesting
that these enable additional learning even when aanns are present.

##  5 Experiment 3: The Role of Variability

Results from the previous experiment highlight the importance of seen aanns in
order for LMs to generalize to unseen instances. What properties of these seen
instances make LMs generalize? More broadly, there is a longstanding question
as to how the nature of the instances of a construction provided during
learning affect its (partial) productivity (Goldberg, 2005, 2019). In the
context of aanns, we consider the role of variability on the open slots of the
construction as a factor that might play a role in LMs’ productivity on unseen
instances. The idea that slot-variability could affect learnability is
motivated by theoretical claims in usage-based linguistics (Bybee, 1995), as
well as existing research on novel constructions (Suttle and Goldberg, 2011),
morphological productivity (Baayen, 2009; O’Donnell, 2015), and inductive
inferences in cognitive psychology (Osherson et al., 1990; Xu and Tenenbaum,
2007). The idea is that encountering a slot with a wide variety of lexical
items serves as a cue that the slot is flexible.

We hypothesize that instances of aanns that provide natural evidence of
greater open-slot variability—i.e. evidence that many different adjectives,
numerals, and nouns can go in their respective positions in the aann
construction—would lead LMs to assign greater likelihood to unseen aanns. On
the other hand, LMs that only encounter a restricted set of instances might
overfit, and be more conservative in extending the coverage of possible aanns
to novel combinations of the slot-fillers. To test this, we divided our set of
2,301 aann-containing utterances in the BabyLM corpus into two equal
subsets—one that contained aanns which were individually highly frequent but
restricted in the types of adjective/numeral/nouns, and the other where the
aanns were individually less frequent, but showed more variability in those
slots. We then trained LMs on the BabyLM corpus containing utterances
involving either of these two cases. Figure 4 shows the resulting average
slors obtained from this experiment, along with those obtained by LMs trained
on unablated BabyLM and the no aann conditions.

From Figure 4, we see that LMs that only saw aanns that were highly variable
in their open-slots demonstrated slors that were comparable and sometimes
greater than those obtained by LMs that saw all aanns. By contrast, LMs that
only saw aanns with low variability were as good as LMs that never saw any
aanns. Therefore, LMs were sensitive to the nature of range of fillers that
went into the construction’s open slots, showing relatively greater
productivity when they observed evidence that the slots were highly variable.
This is compatible with our hypothesis that slot-variability might affect the
extent to which LMs “permit” productive uses of a construction.

Figure 4: slors on aanns from Mahowald (2023) for LMs trained on BabyLM with
low and high variability in the observed instances of aann. slor for unablated
BabyLM-trained LM shown with dotted line.

##  6 Conclusion

There is increasing interest in computational linguistics in how language
models can handle what has been variously called the “long tail” of language
(Prange et al., 2021), “extremely rare constructions” (Potts, 2023),
“exceptions to syntactic rules” (Leong and Linzen, 2023), “rare linguistic
phenomena” (Weissweiler et al., 2024), inter alia. Studies of such phenomena
are important first because LLMs (and LMs and statistical models in general)
are known to be extremely sensitive to frequency and to perform far better in
data-rich environments and, second, because the human ability to generalize to
rare phenomena is central to knowledge of language.

We found that LMs trained on a human-scale of data can learn a rare
construction like the aann. We found that this learning occurs even without
veridical instances of the aann construction in the training data, and that it
is mediated by occurrences of other related constructions in training. As
such, these results join a growing body of work showing the ability of LLMs to
learn constructions (Tayyar Madabushi et al., 2020; Tseng et al., 2022; Li et
al., 2022; Veenboer and Bloem, 2023).

##  7 Limitations

In future work, it would be valuable to extend this method to a wider range of
constructions. But scaling this approach up is not straightforward since it
requires identifying and extracting idiosyncratic constructions, and—more
onerously—developing testable hypotheses about what makes them learnable from
limited amounts of data. While this is a limitation, it also calls for more
synergistic collaborations between theoretical and computational linguists.

Another limitation is that our method requires repeated training of LMs from
scratch which can be computationally expensive. Alternate methods could be to
ablate knowledge of particular hypotheses using representational editing
methods such as AlterRep (Ravfogel et al., 2021), etc.

Unlike Weissweiler et al. (2022), we do not test the ability to interpret
these constructions for downstream tasks. Instead, we focus on linguistic
form. Extending these results to semantic tasks would be informative.

##  8 Acknowledgments

(KM)2 acknowledge funding from NSF Grant 2104995 awarded to Kyle Mahowald. For
helpful conversations, we thank Adele Goldberg, Leonie Weissweiler, the
computational linguistics research group at UT Austin, the syntax-semantics
research group at UT Austin, the audience at the Texas Linguistics Society
meeting. We thank Chris Potts for his paper on the PiPP construction (Potts,
2023) which inspired the “keys to all of this” idea.

## References

  * Baayen (2009) R Harald Baayen. 2009.  43\. corpus linguistics in morphology: morphological productivity.  _Corpus linguistics. An international handbook_ , pages 900–919.
  * Baroni (2022) Marco Baroni. 2022.  On the proper role of linguistically oriented deep net analysis in linguistic theorising.  In _Algebraic structures in natural language_ , pages 1–16. CRC Press.
  * Bender et al. (2021) Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the dangers of stochastic parrots: Can language models be too big?  In _Proceedings of the 2021 ACM conference on fairness, accountability, and transparency_ , pages 610–623.
  * Bybee (1995) Joan Bybee. 1995.  Regular morphology and the lexicon.  _Language and cognitive processes_ , 10(5):425–455.
  * Chomsky (1957) N. Chomsky. 1957.  _Syntactic Structures_.  The Hague: Mouton.
  * Chomsky (1965) N. Chomsky. 1965.  _Aspects of the Theory of Syntax_.  MIT Press, Cambridge, MA.
  * Chomsky (1986) N. Chomsky. 1986.  _Knowledge of language: Its nature, origin, and use_.  Praeger Publishers.
  * Chomsky et al. (2023) Noam Chomsky, Ian Roberts, and Jeffrey Watumull. 2023.  Noam Chomsky: The false promise of ChatGPT.  _The New York Times_.
  * Dalrymple and King (2019) Mary Dalrymple and Tracy Holloway King. 2019.  An amazing four doctoral dissertations.  _Argumentum_ , 15(2019).  Publisher: Debreceni Egyetemi Kiado.
  * Eldan and Li (2023) Ronen Eldan and Yuanzhi Li. 2023.  Tinystories: How small can language models be and still speak coherent english?  _arXiv preprint arXiv:2305.07759_.
  * Futrell et al. (2019) Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019.  Neural language models as psycholinguistic subjects: Representations of syntactic state.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 32–42, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Goldberg (1995) Adele E Goldberg. 1995.  _Constructions: A construction grammar approach to argument structure_.  University of Chicago Press.
  * Goldberg (2005) Adele E Goldberg. 2005.  _Constructions at Work: The Nature of Generalization in Language_.  Oxford University Press.
  * Goldberg (2019) Adele E Goldberg. 2019.  _Explain me this: Creativity, competition, and the partial productivity of constructions_.  Princeton University Press.
  * Heafield (2011) Kenneth Heafield. 2011.  KenLM: Faster and smaller language model queries.  In _Proceedings of the Sixth Workshop on Statistical Machine Translation_ , pages 187–197, Edinburgh, Scotland. Association for Computational Linguistics.
  * Honnibal et al. (2020) Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020.  spaCy: Industrial-strength natural language processing in python.
  * Huebner et al. (2021) Philip A. Huebner, Elior Sulem, Fisher Cynthia, and Dan Roth. 2021.  BabyBERTa: Learning more grammar with small-scale child-directed language.  In _Proceedings of the 25th Conference on Computational Natural Language Learning_ , pages 624–646, Online. Association for Computational Linguistics.
  * Kallini et al. (2024) Julie Kallini, Isabel Papadimitriou, Richard Futrell, Kyle Mahowald, and Christopher Potts. 2024.  Mission: Impossible language models.  _arXiv preprint arXiv:2401.06416_.
  * Kayne (2007) Richard S Kayne. 2007.  On the syntax of quantity in english.  _Linguistic theory and south Asian languages: Essays in honour of Ka Jayaseelan_ , 102:73.
  * Keenan (2013) Caitlin Keenan. 2013.  “A pleasant three days in Philadelphia”: Arguments for a pseudopartitive analysis.  _University of Pennsylvania Working Papers in Linguistics_ , 19(1):11.
  * Kim et al. (2022) Najoung Kim, Tal Linzen, and Paul Smolensky. 2022.  Uncontrolled lexical exposure leads to overestimation of compositional generalization in pretrained models.  _arXiv preprint arXiv:2212.10769_.
  * Lau et al. (2017) Jey Han Lau, Alexander Clark, and Shalom Lappin. 2017.  Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge.  _Cognitive science_ , 41(5):1202–1241.
  * Leong and Linzen (2023) Cara Su-Yi Leong and Tal Linzen. 2023.  Language models can learn exceptions to syntactic rules.  In _Proceedings of the Society for Computation in Linguistics 2023_ , pages 133–144, Amherst, MA. Association for Computational Linguistics.
  * Li et al. (2022) Bai Li, Zining Zhu, Guillaume Thomas, Frank Rudzicz, and Yang Xu. 2022.  Neural reality of argument structure constructions.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 7410–7423, Dublin, Ireland. Association for Computational Linguistics.
  * Linzen et al. (2016) Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016.  Assessing the ability of LSTMs to learn syntax-sensitive dependencies.  _Transactions of the Association for Computational Linguistics_ , 4:521–535.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _arXiv preprint arXiv:1907.11692_.
  * Lu et al. (2020) Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. 2020.  Gender bias in neural natural language processing.  _Logic, language, and security: essays dedicated to Andre Scedrov on the occasion of his 65th birthday_ , pages 189–202.
  * Mahowald (2023) Kyle Mahowald. 2023.  A discerning several thousand judgments: GPT-3 rates the article + adjective + numeral + noun construction.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 265–273, Dubrovnik, Croatia. Association for Computational Linguistics.
  * (29) Kyle Mahowald, Anna A Ivanova, Idan A Blank, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko.  Dissociating language and thought in large language models.  _Trends in Cognitive Sciences_.
  * Maudslay et al. (2019) Rowan Hall Maudslay, Hila Gonen, Ryan Cotterell, and Simone Teufel. 2019.  It’s all in the name: Mitigating gender bias with name-based counterfactual data substitution.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5267–5275, Hong Kong, China. Association for Computational Linguistics.
  * McCoy et al. (2023) R. Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz. 2023.  How much do language models copy from their training data? evaluating linguistic novelty in text generation using RAVEN.  _Transactions of the Association for Computational Linguistics_ , 11:652–670.
  * Misra (2022) Kanishka Misra. 2022.  minicons: Enabling flexible behavioral and representational analyses of transformer language models.  _arXiv preprint arXiv:2203.13112_.
  * O’Donnell (2015) Timothy J O’Donnell. 2015.  _Productivity and reuse in language: A theory of linguistic computation and storage_.  MIT Press.
  * Osherson et al. (1990) Daniel N Osherson, Edward E Smith, Ormond Wilkie, Alejandro Lopez, and Eldar Shafir. 1990.  Category-based Induction.  _Psychological Review_ , 97(2):185.
  * Pauls and Klein (2012) Adam Pauls and Dan Klein. 2012.  Large-scale syntactic language modeling with treelets.  In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 959–968, Jeju Island, Korea. Association for Computational Linguistics.
  * Pearl (2022) Lisa Pearl. 2022.  Poverty of the stimulus without tears.  _Language Learning and Development_ , 18(4):415–454.
  * Piantadosi (2023) Steven Piantadosi. 2023.  Modern language models refute chomsky’s approach to language.  _Lingbuzz Preprint, lingbuzz_ , 7180.
  * Potts (2023) Christopher Potts. 2023.  Characterizing English Preposing in PP constructions.  Ms., Stanford University.
  * Prange et al. (2021) Jakob Prange, Nathan Schneider, and Vivek Srikumar. 2021.  Supertagging the long tail with tree-structured decoding of complex categories.  _Transactions of the Association for Computational Linguistics_ , 9:243–260.
  * Pullum (2017) Geoffrey K Pullum. 2017.  Theory, data, and the epistemology of syntax.  In _Grammatische Variation. Empirische Zugänge und theoretische Modellierung_ , pages 283–298. de Gruyter.
  * Pullum and Scholz (2002) Geoffrey K Pullum and Barbara C Scholz. 2002.  Empirical assessment of stimulus poverty arguments.  _The Linguistic Review_ , 19(1-2):9–50.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI Blog_ , 1(8).
  * Ravfogel et al. (2021) Shauli Ravfogel, Grusha Prasad, Tal Linzen, and Yoav Goldberg. 2021.  Counterfactual interventions reveal the causal effect of relative clause representations on agreement prediction.  In _Proceedings of the 25th Conference on Computational Natural Language Learning_ , pages 194–209, Online. Association for Computational Linguistics.
  * Schwarzschild (2011) Roger Schwarzschild. 2011.  Stubborn distributivity, multiparticipant nouns and the count/mass distinction.  In _Proceedings of NELS_ , volume 39, pages 661–678. Graduate Linguistics Students Association, University of Massachusetts.  Issue: 2.
  * Sinha et al. (2021) Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. 2021.  Masked language modeling and the distributional hypothesis: Order word matters pre-training for little.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 2888–2913, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Solt (2007) Stephanie Solt. 2007.  Two types of modified cardinals.  In _International Conference on Adjectives. Lille_.
  * Suttle and Goldberg (2011) Laura Suttle and Adele E Goldberg. 2011.  The partial productivity of constructions as induction.
  * Tayyar Madabushi et al. (2020) Harish Tayyar Madabushi, Laurence Romain, Dagmar Divjak, and Petar Milin. 2020.  CxGBERT: BERT meets construction grammar.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 4020–4032, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Tseng et al. (2022) Yu-Hsiang Tseng, Cing-Fang Shih, Pin-Er Chen, Hsin-Yu Chou, Mao-Chang Ku, and Shu-Kai Hsieh. 2022.  CxLM: A construction and context-aware language model.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 6361–6369, Marseille, France. European Language Resources Association.
  * Veenboer and Bloem (2023) Tim Veenboer and Jelke Bloem. 2023.  Using collostructional analysis to evaluate BERT’s representation of linguistic constructions.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 12937–12951, Toronto, Canada. Association for Computational Linguistics.
  * Warstadt and Bowman (2022) Alex Warstadt and Samuel R Bowman. 2022.  What artificial neural networks can tell us about human language acquisition.  In _Algebraic Structures in Natural Language_ , pages 17–60. CRC Press.
  * Warstadt et al. (2023) Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell. 2023.  Findings of the BabyLM challenge: Sample-efficient pretraining on developmentally plausible corpora.  In _Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning_ , pages 1–34, Singapore. Association for Computational Linguistics.
  * Wei et al. (2021) Jason Wei, Dan Garrette, Tal Linzen, and Ellie Pavlick. 2021.  Frequency effects on syntactic rule learning in transformers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 932–948, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Weissweiler et al. (2022) Leonie Weissweiler, Valentin Hofmann, Abdullatif Köksal, and Hinrich Schütze. 2022.  The better your syntax, the better your semantics? probing pretrained language models for the English comparative correlative.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 10859–10882, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Weissweiler et al. (2024) Leonie Weissweiler, Abdullatif Köksal, and Hinrich Schütze. 2024.  Hybrid human-llm corpus construction and llm evaluation for rare linguistic phenomena.  _arXiv preprint arXiv:2403.06965_.
  * Wilcox et al. (2018) Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. 2018.  What do RNN language models learn about filler–gap dependencies? In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 211–221, Brussels, Belgium. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Xu and Tenenbaum (2007) Fei Xu and Joshua B Tenenbaum. 2007.  Word learning as bayesian inference.  _Psychological review_ , 114(2):245.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022.  OPT: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_.

##  Appendix 0.A LM training details

As mentioned in the main text (see Section 2), we use the OPT architecture
(Zhang et al., 2022) to train our LMs on all versions of the BabyLM corpus.
This was the best performing autoregressive LM in the BabyLM Competition
(Warstadt et al., 2023). For each instance of the BabyLM (ablated or
otherwise), we tune the learning rate555We searched the following set: {1e-4,
3e-4, 1e-3, 3e-3} based on the validation set, and use the best learning rate
as a result of the tuning to train an additional two language models using
different seeds. As a result, for each ablation of the BabyLM corpus, we run 6
LM training experiments, which amounts to a whopping 90 LMs for all our
experiments. Table 3 contains further details of the training.

(Hyper)parameter | Value
---|---
Architecture | OPT (Zhang et al., 2022)
Embed size | 768
FFN dimension | 3,072
Num. layers | 12
Attention heads | 12
Vocab size | 16,384
Max. seq. length | 256
Batch size | 32
Warmup steps | 32,000
Epochs | 20
Total parameters | 97M
Training size | 100M tokens
Compute | 1x NVIDIA A40
Training time | 21 hours
Table 3: LM Training details

##  Appendix 0.B Detecting aanns and related phenomena

In this section, we briefly describe our methods to extract constructions and
phenomena relevant to this paper from the BabyLM corpus (Warstadt et al.,
2023). Our methods primarily rely on: 1) the surface form of the sentences in
the corpus; 2) their corresponding part-of-speech (POS) tag sequences; and in
a few cases, 3) their depedency parses. For the latter two, we used spacy
(Honnibal et al., 2020), specifically, its en_web_trf model, which is based on
the RoBERTa-base LM (Liu et al., 2019). Next we describe how we used these
artifacts to detect our target constructions:

###  0.B.1 AANNs

To detect AANNs we primarily rely on POS-tagged sequences, and construct a
regex pattern over them666In reality this was constructed over several
iterations, taking into account many different possible realizations of the
construction in free text. which is able to robustly detect aanns:

⬇

pattern =
r’\b(DT)(?:(?:\s(RB))*\s(JJ|JJR|JJS)(?:\s(CC))*)+(\s(CD|JJ|JJR|JJS|NN|CD\sCD)(?:\s(TO|CC)\s(CD))*)(\s(NNS|NNPS|(NN\sNNS)|((NN|NNS)␣IN␣NNS)))+’

where we restrict the determiner (DT) to be either ‘a’, ‘an’, or ‘another’.
This regex permits multiple adjectives (an exhilarating and marvelous three
months) optional adverbs (an excruciatingly painful two semesters), multi-word
noun phrases with plural head-nouns (a refreshing two glasses of aperol
spritz), numeral-expressions involving subordinate clauses (a measly three to
five days), among other potential edge cases. We additionally use the
following adjectives as proxies for numerals, as per the guidelines of Kayne
(2007) and Solt (2007):

⬇

numeral_proxies = [’few’, ’dozen’, ’couple’, ’several’, ’many’, ’more’]

For instance, the following examples would all count as instances of the aann:

  * (1)

    * a.

a beautiful few days.

    * b.

an amazing dozen eggs.

    * c.

a pictorial several pages.

    * d.

a great many days.

    * e.

an awful last couple of days.

    * f.

a few more inches.

Once detected, we map the found constructions to their respective positions
within the aann format, which allows us to measure metrics such as slot
variability, etc.

###  0.B.2 DT ANNs

We follow the exact same procedure as the one for aanns, but no longer
restrict the determiner position to only be an indefinite determiner.

###  0.B.3 A few/couple/dozen NOUNs

An important phenomenon that we consider to be related to the aann involves
cases such as: “that only lasted a few days” and “could you bring me a couple
liters?”, etc., where the plural nouns are attached to an indefinite article.
To detect such cases, we consider the following two dependency configurations,
where we have an indefinite determiner (a, an, another) with either a det
relation with the plural noun (NNS or NNPS) or a quantmod relation with a noun
which has a nummod with the plural noun. In the former case, we usually have
an amod relation between the noun and the adjective.

[column sep=0.6cm, row sep=.1ex] …& DT & JJ & NNS & …

…& a & few & days & …

24det 34amod

[column sep=0.6cm, row sep=.1ex] …& DT & NN & NNS & …

…& a & couple & days & …

23quantmod 34nummod

###  0.B.4 Measure NNS with Singular Verbs

Similar to the previous case, another phenomenon which might be related to the
aann constructions is when measure noun-phrases with plural nouns are treated
as singular via their agreement with a verb—e.g., “five dollars is plenty!” To
detect such cases, we again rely on the following dependency configuration,
where we have a plural noun (NNS or NNPS) attached to a cardinal number (CD)
via the nummod dependency relation, and at the same time also attached to
singular verbs via the nsubj dependency relation (i.e., are subjects of
singular verbs).

[column sep=0.6cm] …& CD & NNS & VB & …

…& five & dollars & is & …

23nummod 34nsubj

##  Appendix 0.C A/An + ADJ/NUM frequency balancing

A corpus analysis of BabyLM, along with its POS-tagged version suggests that
the sequence “a/an/another (JJ|JJR|JJS)” occurs 613,985 times while
“a/an/another CD” occurs only 42,111 times – this suggests that adjectives are
approximately 14.6 more likely to follow an indefinite article than are
numerals. We therefore balance these values by removing 571,874 instances
where adjectives follow an indefinite article. This constitutes the largest-
sized ablation in this work.