# Heat expansion and zeta

Alain Connes  A. Connes: Collège de France
3, rue d’Ulm
Paris, F-75005 France
I.H.E.S. 35 Route de Chartres Bures sur Yvette, F-91440<EMAIL_ADDRESS>
###### Abstract.

We compute the full asymptotic expansion of the heat kernel ${\rm
Tr}(\exp(-tD^{2}))$ where $D$ is, assuming RH, the self-adjoint operator whose
spectrum is formed of the imaginary parts of non-trivial zeros of the Riemann
zeta function. The coefficients of the expansion are explicit expressions
involving Bernoulli and Euler numbers. We relate the divergent terms with the
heat kernel expansion of the Dirac square root of the prolate wave operator
investigated in our joint work with Henri Moscovici.

###### Key words and phrases:

Heat expansion, Riemann zeta function, Bernoulli numbers, Euler numbers

###### 2020 Mathematics Subject Classification:

Primary 11M06, 41A60; Secondary 34B24

Dedicated to Fedor Sukochev

###### Contents

  1. 1 Introduction
  2. 2 Explicit formula
  3. 3 Archimedean contribution
  4. 4 Contribution of the primes
  5. 5 Counting function

##  1\. Introduction

We compute in closed form, using Bernoulli and Euler numbers, the full
asymptotic expansion of the heat trace ${\rm Tr}(\exp(-tD^{2}))$ where $D$ is,
assuming RH, the self-adjoint operator whose spectrum is formed of the
imaginary parts of non-trivial zeros of the Riemann zeta function. The heat
expansion, i.e. the asymptotic expansion for $t\to 0$, of the trace ${\rm
Tr}(\exp(-tD^{2}))$ where $D$ is a self-adjoint operator, plays a key role in
the theory of spectral asymptotics. It is a powerful tool in mathematical
physics, differential geometry, and spectral theory, describing how solutions
to the heat equation on a manifold or more general spaces behave, especially
in the context of the Laplace operator or more general elliptic operators. The
expansion typically takes the form of an asymptotic series, with coefficients
that reflect geometric and topological properties of the space. These
coefficients are often referred to as Seeley-DeWitt or heat kernel
coefficients. This type of asymptotic expansion also plays a key role in
noncommutative geometry as a way to extend curvature invariants to
noncommutative spaces such as noncommutative tori, and Fedor Sukochev has
greatly contributed to this research together with his collaborators with whom
he forms a very active research group.

Computing the coefficients of the asymptotic expansion in closed form is a
challenging problem, and there are only a few well-understood cases where all
the coefficients can be explicitly calculated. Some of these examples include:

1\. Flat Spaces: In Euclidean spaces $\left(\mathbb{R}^{n}\right)$ or flat
tori, the heat kernel and its coefficients can be computed exactly because the
geometry is simple and fully understood. The heat kernel in these cases is
given by explicit formulas involving exponential functions.

2\. Spheres: For the standard sphere $\left(S^{n}\right.$ ) in various
dimensions, the heat kernel coefficients can be computed due to the high
symmetry of the space and the explicit knowledge of the eigenvalues and
eigenfunctions of the Laplacian.

3\. Compact Lie Groups: For compact Lie groups equipped with a biinvariant
metric, the heat kernel can be expressed in terms of the sum over the group’s
representations. This allows for the explicit calculation the heat kernel
coefficients due to the algebraic structure of the group and the
representation theory.

4\. Certain Symmetric Spaces: Symmetric spaces of compact type can sometimes
allow for the explicit computation of heat kernel coefficients, again due to
the symmetry and the structure of the space allowing for the representation of
the heat kernel in a manageable form.

5\. Hyperbolic Spaces: For constant negative curvature spaces (hyperbolic
spaces), the heat kernel can be expressed in terms of special functions, and
in some cases, the coefficients can be computed explicitly.

In general, the ability to compute all heat kernel coefficients in closed form
is rare and typically relies on having a deep understanding of the spectral
properties of the operator and the geometry of the underlying space. Most of
these examples rely on spaces with high degrees of symmetry or simplicity.
In this paper we assume $\mathrm{RH}$ and compute in closed form the
asymptotic expansion of a putative self-adjoint operator whose spectrum is
formed of the imaginary parts of nontrivial zeros of the Riemann zeta function
as follows:

######  Theorem 1.1.

Assume RH and let $D$ be the self-adjoint operator whose spectrum is formed of
the imaginary parts of non-trivial zeros of the Riemann zeta function. One
then has the asymptotic expansion for $t\to 0$

${\rm
Tr}(\exp(-tD^{2}))\sim\frac{\log\left(\frac{1}{t}\right)}{4\sqrt{\pi}\sqrt{t}}-\frac{(\log
4\pi+\frac{1}{2}\gamma)}{2\sqrt{\pi}\sqrt{t}}+2\exp(t/4)+\sum a_{n}t^{n/2}$
(1)

where $a_{0}=-\frac{1}{4}$ and for $k>0$, using Bernouilli numbers $B_{j}$ and
Euler numbers $E(k)$,

$a_{2k-1}=\frac{\Gamma(k)\left(2^{2k-1}-1\right)B_{2k}}{2\sqrt{\pi}(2k)!},\ \
a_{2k}=-\frac{1}{4}\,\Gamma(k+\frac{1}{2})\,\frac{E(2k)}{\sqrt{\pi}(2k)!}.$

The Euler numbers are defined as

$E(2n):=\sum_{k=1}^{2n}\left(-\frac{1}{2}\right)^{k}\sum_{j=0}^{2k}(-1)^{j}\binom{2k}{j}(k-j)^{2n}$
(2)

One has the asymptotic behavior when $k\to\infty$

$\frac{E(2k)}{(2k)!}\sim(-1)^{k}2^{2k}\frac{4}{\pi}\pi^{-2k}$

which shows that the asymptotic expansion (1) is by no means convergent since
its general coefficient $a_{n}$ diverges like a factorial.
Figure 1 shows the graph of ${\rm Tr}(\exp(-D^{2}/a))$ as a function of
$a\in{\mathbb{R}}_{+}$.

Figure 1. Graph of ${\rm Tr}(\exp(-D^{2}/a))$.

The proof of Theorem 1.1 is based on the Riemann-Weil explicit formulas which
we recall in section 2. It is divided in two parts, we handle the contribution
of the archimedean place in section 3 and the contribution of the primes in
section 4.
The first two terms of (1) are the only divergent ones and their form is
already a great puzzle at the level of the geometry thus revealed at the
spectral level. In our joint work with H. Moscovici [2], Theorem 5.1, we found
a Dirac type square root $D$ of the prolate wave operator whose counting
function for the imaginary eigenvalues behaves as follows

$N(E)=\frac{E}{2\pi}\left(\log\frac{E}{2\pi}-1\right)+O(\log E).$ (3)

In section 5, we describe the link between the heat kernel asymptotic
expansion and the counting function $N(E):=\\#({\rm Spec}\,D\cap[0,E])$ and
show in Theorem 5.1 that the knowledge of the behavior of $N(E)$ in the form
(3) suffices to secure the first terms of the heat kernel asymptotic expansion
in the form of (1). This indicates that Theorem 5.1 of [2] gives a first
approximation for the geometry underlying zeta zeros.

##  2\. Explicit formula

Following [1], let $f(x)$ be a smooth function on $\mathbb{R}_{+}^{*}$, such
that there is $\delta>0$ such that $f(x)=O\left(x^{\delta}\right)$ as
$x\rightarrow 0+$ and $f(x)=O\left(x^{-1-\delta}\right)$ as
$x\rightarrow+\infty$. Let $\widetilde{f}(s)$ be the Mellin transform

$\tilde{f}(s):=\int_{0}^{\infty}f(x)x^{s-1}dx.$ (4)

which is an analytic function of $s$ for $-\delta<\Re(s)<1+\delta$. Then, with
$f^{\sharp}(x):=x^{-1}f(x^{-1})$ the explicit formula takes the form

$\sum_{\rho}\tilde{f}(\rho)=\int_{0}^{\infty}f(x)dx+\int_{0}^{\infty}f^{\sharp}(x)dx-\sum_{v}{\mathcal{W}}_{v}(f),$
(5)

where $v$ runs over all places $\\{{\mathbb{R}},2,3,5,\ldots\\}$ of
${\mathbb{Q}}$, the sum on the left hand side is over all complex zeros $\rho$
of the Riemann zeta function, and for $v=p$

${\mathcal{W}}_{p}(f)=(\log
p)\sum_{m=1}^{\infty}\left(f(p^{m})+f^{\sharp}(p^{m})\right).$ (6)

The archimedean distribution is defined as

${\mathcal{W}}_{\mathbb{R}}(f):=(\log
4\pi+\gamma)f(1)+\int_{1}^{\infty}\left(f(x)+f^{\sharp}(x)-\frac{2}{x}f(1)\right)\frac{dx}{x-x^{-1}}.$
(7)

One then has

${\mathcal{W}}_{\mathbb{R}}(f)=(\log\pi)f(1)-\frac{1}{2\pi
i}\int_{1/2+iw}\Re\left(\frac{\Gamma^{\prime}}{\Gamma}\left(\frac{w}{2}\right)\right)\tilde{f}(w)dw.$
(8)

With the above notations, let $F(x):=x^{1/2}f(x)$, one then has

$F(x^{-1})=x^{-1/2}f(x^{-1})=x^{1/2}f^{\sharp}(x)$

and the archimedean contribution (8) gives, using $d^{*}x:=dx/x$,

$W_{\mathbb{R}}(F)=(\log
4\pi+\gamma)F(1)+\int_{1}^{\infty}\left(F(x)+F(x^{-1})-2x^{-1/2}F(1)\right)\frac{x^{1/2}}{x-x^{-1}}d^{*}x.$
(9)

We consider the Fourier transform for the duality between
${\mathbb{R}}_{+}^{*}$ and ${\mathbb{R}}$ in the form

${\widehat{F}}(s):=\int F(u)u^{-is}d^{*}u$ (10)

so that for $F(x):=x^{1/2}f(x)$ one has, using (4),
${\widehat{F}}(s)=\tilde{f}(\frac{1}{2}-is)$. Thus assuming RH and letting
$Z:=\\{\rho\in{\mathbb{R}}\mid\zeta(\frac{1}{2}+i\rho)=0\\}$, we can rewrite
(5) as

$\sum_{Z}{\widehat{F}}(\rho)={\widehat{F}}(i/2)+{\widehat{F}}(-i/2)-W_{\mathbb{R}}(F)-\sum_{p}W_{p}(F),$
(11)

where for each prime $p$, $W_{p}(F)={\mathcal{W}}_{p}(f)$ which gives

$W_{p}(F)=(\log p)\sum_{m=1}^{\infty}p^{-m/2}\left(F(p^{m})+F(p^{-m})\right)$
(12)

##  3\. Archimedean contribution

We start from the Riemann-Weil explicit formula in the form (11) and first
ignore the contributions of the finite places. With
$Z:=\\{\rho\in{\mathbb{R}}\mid\zeta(\frac{1}{2}+i\rho)=0\\}$ as above we are
thus dealing with the first three terms in the right hand side of (11).
We choose $F_{t}$ such that ${\widehat{F}_{t}}(s)=\exp(-ts^{2})$. One has

$F_{t}(e^{y})=\frac{e^{-\frac{y^{2}}{4t}}}{2\sqrt{\pi}\sqrt{t}},\ \
F_{t}(e^{-y})=F_{t}(e^{y})$ (13)

Thus one gets with $\psi(t):=\sum_{p}W_{p}(F_{t})$, (see (25) below),

$\sum_{Z}\exp(-t\rho^{2})={\widehat{F}_{t}}(i/2)+{\widehat{F}_{t}}(-i/2)-W_{\mathbb{R}}(F_{t})-\psi(t)=2\exp(t/4)-W_{\mathbb{R}}(F_{t})-\psi(t)$
(14)

We let $x=e^{u}$ and get, using (9),

$W_{\mathbb{R}}(F_{t}):=(\log
4\pi+\gamma)F_{t}(1)+\int_{0}^{\infty}\left(F_{t}(e^{u})+F_{t}(e^{-u})-2e^{-u/2}F_{t}(1)\right)\frac{e^{u/2}}{e^{u}-e^{-u}}du.$
(15)

Our task now is to compute the full asymptotic expansion of (15) when $t\to
0$. One has

$\int_{0}^{a}\left(\frac{1}{e^{u}-e^{-u}}-\frac{1}{2u}\right)\,du+\int_{a}^{\infty}\frac{1}{e^{u}-e^{-u}}\,du=-\frac{1}{2}\log\left(\frac{a}{2}\right)$

which taking $a=2$ gives the equality

$\int_{0}^{2}\left(\frac{1}{e^{u}-e^{-u}}-\frac{1}{2u}\right)\,du+\int_{2}^{\infty}\frac{1}{e^{u}-e^{-u}}\,du=0$
(16)

One has for $F(e^{u})$ even function of $u$,

$I:=\int_{0}^{\infty}\left(F(e^{u})+F(e^{-u})-2e^{-u/2}F(1)\right)\frac{e^{u/2}}{e^{u}-e^{-u}}du=$
$=\int_{0}^{2}\left(2F(e^{u})-2e^{-u/2}F(1)\right)\frac{e^{u/2}}{e^{u}-e^{-u}}du+\int_{2}^{\infty}\left(2F(e^{u})-2e^{-u/2}F(1)\right)\frac{e^{u/2}}{e^{u}-e^{-u}}du$

Using (16) one gets

$\int_{2}^{\infty}\left(-2e^{-u/2}F(1)\right)\frac{e^{u/2}}{e^{u}-e^{-u}}du=\int_{0}^{2}\left(\frac{2F(1)}{e^{u}-e^{-u}}-\frac{2F(1)}{2u}\right)\,du$

Let

$I^{\prime}=\int_{2}^{\infty}\left(2F(e^{u})\right)\frac{e^{u/2}}{e^{u}-e^{-u}}du$

Thus one obtains,

$I=\int_{0}^{2}\left(\left(2F(e^{u})-2e^{-u/2}F(1)\right)\frac{e^{u/2}}{e^{u}-e^{-u}}+\left(\frac{2F(1)}{e^{u}-e^{-u}}-\frac{2F(1)}{2u}\right)\right)du+I^{\prime}=$
$\int_{0}^{2}\left(2F(e^{u})\frac{e^{u/2}}{e^{u}-e^{-u}}-\frac{2F(1)}{2u}\right)du+I^{\prime}=\int_{0}^{2}\frac{F(e^{u})-F(1)}{u}du+\int_{0}^{2}2F(e^{u})r(u)du+I^{\prime}$

where

$r(u):=\frac{e^{u/2}}{e^{u}-e^{-u}}-\frac{1}{2u}$

One has moreover

$I^{\prime}=\int_{2}^{\infty}2F(e^{u})\frac{e^{u/2}}{e^{u}-e^{-u}}du=\int_{2}^{\infty}2F(e^{u})\left(\frac{1}{2u}+r(u)\right)du$

and we obtain

$I=\int_{0}^{2}\frac{F(e^{u})-F(1)}{u}du+\int_{0}^{\infty}2F(e^{u})r(u)du+\int_{2}^{\infty}\frac{F(e^{u})}{u}du$

which gives the formula, for $F(e^{u})$ even function of $u$,

$W_{\mathbb{R}}(F)=(\log
4\pi+\gamma)F(1)+\int_{0}^{2}\frac{F(e^{u})-F(1)}{u}du+\int_{0}^{\infty}2F(e^{u})r(u)du+\int_{2}^{\infty}\frac{F(e^{u})}{u}du$
(17)

We apply this formula to $F=F_{t}$ as in (13). The last term gives

$\int_{2}^{\infty}\frac{F_{t}(e^{u})}{u}du=\frac{1}{2\sqrt{\pi}\sqrt{t}}\int_{2}^{\infty}e^{-\frac{u^{2}}{4t}}\frac{du}{u}=\frac{1}{2\sqrt{\pi}\sqrt{t}}\int_{t^{-1/2}}^{\infty}e^{-v^{2}}\frac{dv}{v}=\frac{1}{4\sqrt{\pi}\sqrt{t}}\Gamma(0,1/t)$

where

$\Gamma(0,a)=\int_{a}^{\infty}e^{-t}\frac{dt}{t}\sim e^{-a}/a,\ \ \text{for}\
a\to\infty$ (18)

since

$\int_{a}^{\infty}e^{-t}\frac{dt}{t}=e^{-a}/a\int_{0}^{\infty}e^{-x}\frac{dx}{1+x/a}.$

Thus when $t\to 0$, the last term in (17) is
$\sim\frac{\sqrt{t}}{4\sqrt{\pi}}\exp(-1/t)=O(t^{\infty})$.
We now consider the second term in (17). One has

$\int_{0}^{a}\frac{1-\exp(-u)}{u}\,du=\log(a)+\Gamma(0,a)+\gamma$

This gives, with $u:=\frac{y^{2}}{4t}$ that

$\int_{0}^{2}\frac{(e^{-\frac{y^{2}}{4t}}-1)}{y}dy=-\frac{1}{2}\int_{0}^{1/t}\frac{1-\exp(-u)}{u}\,du=-\frac{1}{2}\left(\log(\frac{1}{t})+\Gamma(0,\frac{1}{t})+\gamma\right)$

Thus, up to a term which is infinitely flat when $t\to 0$,

$\int_{0}^{2}\frac{F_{t}(e^{y})-F_{t}(1)}{y}dy=\frac{-\log\left(\frac{1}{t}\right)-\gamma}{4\sqrt{\pi}\sqrt{t}}+O(t^{\infty})$
(19)

and the sum of the first two terms in (17) gives

$(\log
4\pi+\gamma)F_{t}(1)+\int_{0}^{2}\frac{F_{t}(e^{u})-F_{t}(1)}{u}du=\frac{-\log\left(\frac{1}{t}\right)-\gamma}{4\sqrt{\pi}\sqrt{t}}+(\log
4\pi+\gamma)\frac{1}{2\sqrt{\pi}\sqrt{t}}+O(t^{\infty})$ (20)

It remains to deal with the third term in (17), i.e.
$\int_{0}^{\infty}2F(e^{u})r(u)du$.

######  Lemma 3.1.

One has the power series expansion, converging for $|u|<\pi$,

$r(u)=\frac{e^{u/2}}{e^{u}-e^{-u}}-\frac{1}{2u}=\sum_{0}^{\infty}b_{n}\,u^{n},$
(21)

where $b_{0}=\frac{1}{4}$ and

$b_{2k-1}=-\frac{\left(1-2^{1-2k}\right)B_{2k}}{2(2k)!},\ \
b_{2k}=\frac{1}{4}\,2^{-2k}\,\frac{E(2k)}{(2k)!}$

in terms of the Bernoulli numbers $B_{n}$ and the Euler numbers of (2).

###### Proof.

One has

$\frac{e^{u/2}}{e^{u}-e^{-u}}=\frac{e^{u/2}}{(e^{u/2}-e^{-u/2})(e^{u/2}+e^{-u/2})}=\frac{1}{2}\left(\frac{1}{(e^{u/2}+e^{-u/2})}+\frac{1}{(e^{u/2}-e^{-u/2})}\right)$

One has the Taylor expansion

$\frac{2}{e^{u/2}+e^{-u/2}}=1+\sum_{1}^{\infty}\frac{E(2n)}{(2n)!}\left(\frac{u}{2}\right)^{2n}$

using Euler numbers defined in (2). Using Bernoulli numbers one has

$\frac{2}{e^{u/2}-e^{-u/2}}=\frac{2}{u}-\sum_{1}^{\infty}\frac{2\left(2^{2k-1}-1\right)B_{2k}}{(2k)!}\left(\frac{u}{2}\right)^{2k-1}$

One has

$B_{2n}\sim(-1)^{n+1}4\sqrt{\pi n}\left(\frac{n}{\pi e}\right)^{2n},\ \
B_{2n}/(2n)!\sim(-1)^{n+1}2\left(2\pi\right)^{-2n}$

and

$E_{2n}\sim(-1)^{n}8\sqrt{\frac{n}{\pi}}\left(\frac{4n}{\pi e}\right)^{2n},\ \
2^{-2k}\,\frac{E(2k)}{(2k)!}\sim(-1)^{k}\frac{4}{\pi}\pi^{-2k}$

which checks the convergence of the series (21) for $|u|<\pi$ as expected from
the first poles of $r(u)$ at $u=\pm i\pi$.∎

######  Lemma 3.2.

One has the asymptotic expansion for $t\to 0$

$\int_{0}^{\infty}2F_{t}(e^{u})r(u)du\simeq-\sum a_{n}t^{n/2},\ \
a_{n}=-\frac{2^{n}\Gamma\left(\frac{n+1}{2}\right)}{\sqrt{\pi}}b_{n}$ (22)

###### Proof.

We use the equality

$\int_{0}^{\infty}2F_{t}(e^{y})y^{n}dy=\frac{2^{n}t^{n/2}\Gamma\left(\frac{n+1}{2}\right)}{\sqrt{\pi}}$
(23)

which gives the terms of the expansion (26).
In order to show that we get an asymptotic expansion we need to estimate the
remainder in the Taylor expansion (21). To avoid negative powers of $u$ we let

$R(u):=2u\,r(u)=\frac{u\exp\left(\frac{u}{2}\right)}{\sinh(u)}-1$

and we use the Taylor formula with integral remainder for $R(u)$ at $u=0$
which gives

$r(u)=\frac{e^{u/2}}{e^{u}-e^{-u}}-\frac{1}{2u}=\sum_{0}^{k}b_{n}\,u^{n}+\frac{1}{2u\,k!}\int_{0}^{u}R^{(k+1)}(v)(u-v)^{k}dv$
(24)

By construction the function $R(u)$ is smooth and moreover its derivatives
fulfill

$R^{(n)}(u)\sim(-2)^{-(n-1)}(2n-u)e^{-u/2}$

and are thus bounded on $[0,\infty)$, $|R^{(n)}(v)|\leq c_{n}$ which gives the
inequality

$\left|\int_{0}^{u}R^{(k+1)}(v)(u-v)^{k}dv\right|\leq
c_{k+1}\frac{u^{k+1}}{k+1}$

Thus the third term in (17) gives

$\int_{0}^{\infty}2F_{t}(e^{u})r(u)du=\sum_{0}^{k}b_{n}\int_{0}^{\infty}2F_{t}(e^{y})y^{n}dy+\rho_{k}(t)$

where

$\rho_{k}(t)=\frac{1}{k!}\int_{0}^{\infty}F_{t}(e^{u})\left(\frac{1}{u}\int_{0}^{u}R^{(k+1)}(v)(u-v)^{k}dv\right)du$

so that

$|\rho_{k}(t)|\leq\frac{c_{k+1}}{(k+1)!}\int_{0}^{\infty}F_{t}(e^{u})u^{k}dy=\frac{c_{k+1}\Gamma\left(\frac{k+1}{2}\right)}{\sqrt{\pi}(k+1)!}2^{k-1}t^{k/2}$

which gives the required bound for the remainder. ∎

##  4\. Contribution of the primes

We also need to control the contribution of the finite places and it is given,
using the von Mangoldt function $\Lambda(n)$ as the sum

$\psi(t)=2\sum_{2}^{\infty}\Lambda(n)n^{-1/2}F_{t}(n)=\sum_{2}^{\infty}\Lambda(n)n^{-1/2}\,\frac{e^{-\frac{(\log
n)^{2}}{4t}}}{\sqrt{\pi}\sqrt{t}}$ (25)

One has for any integer $n\geq 2$ and $t\leq t_{0}=\frac{\log 6}{8}\sim
0.22397$, the inequality

$e^{-\frac{(\log n)^{2}}{4t}}=n^{-\frac{\log n}{4t}}\leq
4n^{-2}e^{-\frac{(\log 2)^{2}}{4t}}$

and one thus obtains, since $\Lambda(n)n^{-1/2}\leq 1$ the estimate for $t\leq
t_{0}$

$|\psi(t)|\leq 4\left(\frac{\pi^{2}}{6}-1\right)\frac{e^{-\frac{(\log
2)^{2}}{4t}}}{\sqrt{\pi}\sqrt{t}}$

which shows that it does not contribute to the asymptotic expansion when $t\to
0$. One can compute the discrepancy between ${\rm Tr}(\exp(-tD^{2}))$ and the
first terms of the approximation (1) (using up to the linear term in $t$),
i.e. the terms

$\frac{\sqrt{\frac{1}{t}}\log\left(\frac{1}{t}\right)}{4\sqrt{\pi}}-\frac{\gamma\sqrt{\frac{1}{t}}}{4\sqrt{\pi}}-\frac{\sqrt{\frac{1}{t}}\log(4\pi)}{2\sqrt{\pi}}+\frac{7}{4}+\frac{\sqrt{t}}{24\sqrt{\pi}}+\frac{9\,t}{16}$

and for instance for $t=10^{-4}$ the difference gives $-2.5\times 10^{-9}$.
The graph of the difference in terms of $a:=1/t\in[100,10000]$ is plotted in
Figure 2

Figure 2. Graph of discrepancy.

##  5\. Counting function

The next theorem provides the link between the asymptotic form of the counting
function (compare with Theorem 5.1 of [2]) and the first terms of the heat
expansion.

######  Theorem 5.1.

Let $D$ be an operator with discrete real spectrum invariant under
$x\mapsto-x$ and such that the counting function $N(E):=\\#({\rm
Spec}\,D\cap[0,E])$ fulfills
$N(E)=\frac{E}{2\pi}\left(\log\frac{E}{2\pi}-1\right)+O(\log E)$, then one has
for $t\to 0$,

${\rm
Tr}(\exp(-tD^{2}))=\frac{\log\left(\frac{1}{t}\right)}{4\sqrt{\pi}\sqrt{t}}-\frac{(\log
4\pi+\frac{1}{2}\gamma)}{2\sqrt{\pi}\sqrt{t}}+O\left(\log(\frac{1}{t})\right)$
(26)

###### Proof.

Up to a bounded term one has, using
$\partial_{x}(e^{-tx^{2}})=-2txe^{-tx^{2}}$,

${\rm Tr}(\exp(-tD^{2}))\sim
2\int_{1}^{\infty}\partial_{E}N(E)\exp(-tE^{2})dE\sim
4\int_{1}^{\infty}N(E)tE\exp(-tE^{2})dE$

Let

$n(x):=\frac{2x\left(\log\left(\frac{x}{2\pi}\right)-1\right)}{2\pi}$

In order to replace $2N(E)$ by $n(E)$ we estimate the remainder due to the
term in $O(\log E)$. We compute (integrating by parts) the integral

$R(t)=\int_{1}^{\infty}\log(x)\left(2txe^{-tx^{2}}\right)\,dx=\frac{\Gamma(0,t)}{2}=\frac{1}{2}(-\log(t)-\gamma)+O(t)$

and this suffices to give an $O(\log\frac{1}{t})$ bound for the remainder.
It remains to compute the integral

$\displaystyle J(t)$
$\displaystyle:=\int_{1}^{\infty}n(x)\left(2txe^{-tx^{2}}\right)\,dx$
$\displaystyle=\frac{e^{-t}}{2\pi\sqrt{t}}\left(e^{t}G_{2,3}^{3,0}\left(t\left|\begin{array}[]{c}1,1\\\
0,0,\frac{3}{2}\\\
\end{array}\right.\right)-(1+\log(2\pi))\left(\sqrt{\pi}\,e^{t}\,\text{erfc}\left(\sqrt{t}\right)+2\sqrt{t}\right)\right)$

where the Meijer G-function in this special case is the integral

$G_{2,3}^{3,0}\left(t\left|\begin{array}[]{c}1,1\\\ 0,0,\frac{3}{2}\\\
\end{array}\right.\right)=\frac{1}{2\pi
i}\int\frac{\Gamma(s)\Gamma(s)\Gamma(s+\frac{3}{2})}{\Gamma(1+s)\Gamma(1+s)}t^{-s}ds=\frac{1}{2\pi
i}\int\frac{\Gamma(s+\frac{3}{2})}{s^{2}}t^{-s}ds$

The contribution of this function to the integral comes from the term in
$x\log x$ in $n(x)$

$I(t):=\frac{1}{\pi}\int_{1}^{\infty}x\log x\left(2txe^{-tx^{2}}\right)\,dx$
(27)

One obtains the asymptotic expansion of $J(t)$ when $t\to 0$ in the form

$J(t)=\frac{-\log(t)-2-2\log(2\pi)+\Gamma^{\prime}/\Gamma\left(\frac{3}{2}\right)}{4\sqrt{\pi}\sqrt{t}}+O(t)$

To evaluate $\Gamma^{\prime}/\Gamma\left(\frac{3}{2}\right)$ we use the
Legendre duplication formula

$\Gamma(z)\Gamma(z+\frac{1}{2})=2^{1-2z}\sqrt{\pi}\,\Gamma(2z)$

which gives

$\Gamma^{\prime}/\Gamma(z)+\Gamma^{\prime}/\Gamma(z+\frac{1}{2})=-2\log
2+2\,\Gamma^{\prime}/\Gamma(2z)$

and thus, taking $z=1$, and using $\Gamma^{\prime}/\Gamma(2)=1-\gamma$,
$\Gamma^{\prime}/\Gamma(1)=-\gamma$

$\Gamma^{\prime}/\Gamma\left(\frac{3}{2}\right)=-2\log
2+2\,\Gamma^{\prime}/\Gamma(2)-\Gamma^{\prime}/\Gamma(1)=-2\log 2+2-\gamma$
(28)

We thus get

$J(t)=\frac{-\log(t)-2-2\log(2\pi)+(-2\log
2+2-\gamma)}{4\sqrt{\pi}\sqrt{t}}+O(t)=$
$=\frac{\log\left(\frac{1}{t}\right)}{4\sqrt{\pi}\sqrt{t}}-\frac{(\log
4\pi+\frac{1}{2}\gamma)}{2\sqrt{\pi}\sqrt{t}}+O(t)$

∎

Let us give a direct way to compute the integral (27) up to an $O(t)$ term for
$t\to 0$. We consider

$\int_{0}^{\infty}x^{z}e^{-tx^{2}}dx=\frac{1}{2}\,t^{-\frac{z}{2}-\frac{1}{2}}\Gamma(\frac{1+z}{2})$

and we take the derivative $\partial_{z}$ of both sides at $z=2$ to obtain,
using $\Gamma(\frac{3}{2})=\frac{\sqrt{\pi}}{2}$ and (28),

$\int_{0}^{\infty}x^{2}\log(x)e^{-tx^{2}}dx=\frac{\sqrt{\pi}(-2\log
2+2-\gamma)}{8t^{3/2}}-\frac{\sqrt{\pi}\log(t)}{8t^{3/2}}$

Multiplying the result by $\frac{2t}{\pi}$ one obtains

$I(t)=\frac{-2\log 2+2-\gamma-\log(t)}{4\sqrt{\pi}\sqrt{t}}+O(t)$

The remaining term to obtain $J(t)$ is, up to a term in $O(t)$

$-\frac{(1+\log(2\pi)}{\pi}\int_{0}^{\infty}x\left(2txe^{-tx^{2}}\right)\,dx=-\frac{1+\log(2\pi)}{2\sqrt{\pi}\sqrt{t}}$

which gives

$J(t)=\frac{\log\left(\frac{1}{t}\right)}{4\sqrt{\pi}\sqrt{t}}-\frac{(\log
4\pi+\frac{1}{2}\gamma)}{2\sqrt{\pi}\sqrt{t}}+O(t).$

## References

  * [1] E. Bombieri, _The Riemann hypothesis_. The millennium prize problems, 107–124, Clay Math. Inst., Cambridge, MA, 2006.
  * [2] A. Connes, H. Moscovici, The UV prolate spectrum matches the zeros of zeta. Proc. Natl. Acad. Sci. USA 119 (2022), no. 22.


# Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions
about Code

Jaromir Savelka1 , Arav Agarwal1 , Christopher Bogart1 and Majd Sakr1
1School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA
https://orcid.org/0000-0002-3674-5456 https://orcid.org/0000-0001-9848-1663
https://orcid.org/0000-0001-8581-115X https://orcid.org/0000-0001-5150-8259

###### Abstract

We analyzed effectiveness of three generative pre-trained transformer (GPT)
models in answering multiple-choice question (MCQ) assessments, often
involving short snippets of code, from introductory and intermediate
programming courses at the postsecondary level. This emerging technology stirs
countless discussions of its potential uses (e.g., exercise generation, code
explanation) as well as misuses in programming education (e.g., cheating).
However, the capabilities of GPT models and their limitations to reason about
and/or analyze code in educational settings have been under-explored. We
evaluated several OpenAI’s GPT models on formative and summative MCQ
assessments from three Python courses (530 questions). We found that MCQs
containing code snippets are not answered as successfully as those that only
contain natural language. While questions requiring to fill-in a blank in the
code or completing a natural language statement about the snippet are handled
rather successfully, MCQs that require analysis and/or reasoning about the
code (e.g., what is true/false about the snippet, or what is its output)
appear to be the most challenging. These findings can be leveraged by
educators to adapt their instructional practices and assessments in
programming courses, so that GPT becomes a valuable assistant for a learner as
opposed to a source of confusion and/or potential hindrance in the learning
process.

##  1 INTRODUCTION

This paper analyzes the effectiveness of generative pre-trained transformers
(GPT), specifically `text-davinci-*` models, to handle multiple-choice
question (MCQ) assessments, often involving small snippets of code, from
introductory and intermediate programming courses. We manually collected a
sizeable data set of 530 MCQs from three existing Python courses. Using a
combination of simple pattern matching and manual curation, we organized the
questions into meaningful categories according to their type (e.g., true/false
questions, or questions asking about an output of the provided code snippet).
We analyzed the performance of the GPT models across the categories to
determine if questions of a certain type are handled more successfully than
questions of other types. We also benchmark the older InstructGPT `text-
davinci-001` model against the more recent GPT-3.5 `text-davinci-002` and
`text-davinci-003` models to gauge the rate of improvement that has been
achieved over the past several years.

There has been a burst of public attention to GPT models’ potential impact on
education as the result of the recent release of OpenAI’s ChatGPT111ChatGPT.
https://chat.openai.com/ [Accessed 2023-01-26]. For example, the tool has been
blocked by New York City public schools [Elsen-Rooney, 2023] because it may
enable student plagiarism and provide inappropriate or incorrect content.
Universities have also been reacting, adjusting assignments [Huang, 2023] and
seeking out tools like GPTZero that detect text generated by AI tools [Bowman,
2023]. OpenAI have released a similar tool themselves. However, reliability of
these tools has not been thoroughly tested.

Programming instructors as well as CS educators in general have been
sensitized to this development even earlier. Large language models, such as
GPT, can generate computer program code (i.e., perform computer program
synthesis) with a high degree of success. They can also explain computer
program code in natural language terms. Recently, a number of computer program
code generation tools have been released. Among these, the most prominent ones
are OpenAI’s Codex [Chen et al., 2021], DeepMind’s AlphaCode [Li et al.,
2022], and Amazon’s CodeWhisperer [Ankur and Atul, 2022]. GitHub’s
Copilot222GitHub Copilot: Your AI pair programmer. Available at:
https://github.com/features/copilot [Accessed 2023-01-20] (a version of Codex)
conveniently integrates with IDEs, such as Visual Studio Code, and hence has
attracted much attention. Microsoft dubs Copilot as “Your AI pair programmer”
(a reference to pair programming [Beck, 2000, McDowell et al., 2002]). Since
it is available for free to students and educators, it is inevitable that
learners will use it to complete their course assignments and assessments.
Similarly, there are no technical or cost barriers to using ChatGPT which can
be, among many other things, leveraged to generate answers to MCQ questions.

To investigate how GPT models handle the MCQ assessments of various types in a
programming education context, we analyzed the following research questions:

  * •

Is there a difference between how successfully the GPT models handle questions
that contain only natural language and those that also contain snippets of
computer code?

  * •

Are there particular types of MCQs that are more challenging for the GPT
models compared to other types of MCQs?

By carrying out this work, we provide the following contributions to the CS
education research community. To the best of our knowledge, this is the first
comprehensive study that:

  * •

Evaluates the performance of GPT models on MCQ-style assessments that involve
code snippets, across different types of such questions.

  * •

Lays a systematic foundation for discussions about suitable uses of GPT models
in programming classes by providing quantitative analysis of the models
capabilities and limitations in handling of computer code.

3.4cm(.5cm,-5.3cm) 1 3.4cm(11.75cm,-2.15cm) 2 3.4cm(11.25cm,-5.6cm) 3
3.4cm(3.55cm,-1.15cm) 4

Figure 1: The upper-left screenshot depicts a conversation with ChatGPT when
asked to explain a code snippet line by line. It correctly explains the
bahavior (1). The lower-right shows a conversation with ChatGPT when asked to
generate the code snippet with the same behavior. The generated code is
correct (2). The upper-right screenshot depicts a conversation with ChatGPT
when asked a straightforward MCQ about a code it can correctly explain line by
line as well as correctly generate. The answer is wrong (3)—compare the actual
output of the code snippet which is shown in the lower-left corner (4).

##  2 Motivating Example

Consider the below Python script that asks a user to input a value which is
expected to be a number. The entered value of type `str` is cast to an `int`
and divided by the length of the raw input (`str`). Note that the code defends
against the possibility of a `ZeroDivisionError` which cannot really occur, as
explained below. However, this likely confuses GPT models when answering
questions about this snippet.

try: value = input("Enter a value: ") print(int(value) / len(value))except
ZeroDivisionError: print("Very bad input...")

If a user enters 22, then the output of the script would be 11.0 (i.e., 22 /
2). As shown in Figure 1, if one provides ChatGPT (one of the state-of-the-art
GPT-3.5 models) with the code snippet and asks, “what would be the output if
the user enters 0,” (letting ChatGPT choose from “A. 0.0” or “B. Very bad
input…”), the provided answer is “B. Very bad input…” Of course, this is an
incorrect answer because the length of the string `"0"` is 1 and, hence, the
output is 0.0 (as shown in Figure 1).

A human learner making this error would likely be suspected of having several
crucial misconceptions. Firstly, selecting the “B. Very bad input…” option
would be somewhat more understandable if the `value` variable were not placed
within the `len()` function call. In that case, one could assume that the
learner simply failed to recognize that the output of the `input()` function
call is a `str` and assumed it was an `int` instead. However, applying the
`len()` function to an `int` would result in a `TypeError` being raised.
Hence, the only input that could theoretically raise a `ZeroDivisionError`
would be an empty string. However, even that input would not result in that
particular error because it would fail on an attempt to cast the `value`
variable to `int` (`ValueError`) that would occur prior to the division.
Overall, a human learner selecting the “B. Very bad input…” answer over the
correct “A. 0.0” would clearly demonstrate a lack of understanding of the
workings of this simple snippet of code.

Figure 1 shows the output of ChatGPT when asked to explain the code snippet
line by line. Interestingly, the explanation is correct, including the line
where the division takes place. With respect to the statement on that line, it
declares that: “[it] takes the input value and first converts it to an integer
using the `int()` function, then divides it by the length of the input value
using the `len()` function.” Furthermore, Figure 1 also shows the output of
ChatGPT when asked to generate Python code with the same functionality as the
provided code snippet. From the natural language description, ChatGPT
generates correct Python code with the specified behavior.

In this example, a GPT model is capable of correctly explaining the behavior
(execution) of a computer program on a local level (i.e., line by line). It is
equally capable of generating the computer program from a natural language
description. Yet, it fails spectacularly in answering simple questions about
the very same program. This is quite likely in stark contrast with a typical
human learner. A learner capable of independently writing the program from the
natural language description as well as correctly explaining its execution
line by line, would quite likely be in a position to answer such questions
with ease.

##  3 RELATED WORK

In prior work, we evaluated the capability of a GPT model (`text-
davinci-003`), to pass a diverse set of assessment instruments, including
MCQs, in the realistic context of full-fledged programming courses [Savelka et
al., 2023]. We found that the current GPT models are not capable of passing
the full spectrum of assessments typically involved in a Python programming
course (below 70% on even entry-level modules); but a straightforward
application of these models could enable a learner to obtain a non-trivial
portion of the overall available score (over 55%) in introductory and
intermediate courses alike. We observed that an important limitation of the
GPT models is their apparent struggle with activities that require chains of
reasoning steps, and that there appeared to be a difference in success rate
between MCQs that contain a code snippet and those that do not [Savelka et
al., 2023]. In this paper, we further explore this phenomenon, focusing on
discovery of more fine-grained properties of MCQs that are challenging for the
GPT models to handle.

To the best of our knowledge, there is no other study of GPT’s performance on
MCQs from the programming domain. There is work evaluating the performance on
MCQ data sets from other domains; in many cases the tool does better than
random chance; sometimes even well enough to pass a test. For example,
Robinson et al. apply InstructGPT [Ouyang et al., 2022] and Codex to
OpenBookQA [Mihaylov et al., 2018], StoryCloze [Mostafazadeh et al., 2016],
and RACE-m [Lai et al., 2017] data sets which focus on multi-hop reasoning,
recall, and reading comprehension, reporting 77.4-89.2% accuracy [Robinson et
al., 2022]. In some cases, GPT can generate code when applied to programming
assignments in higher education courses. Drori and Verma used Codex to write
Python programs to solve 60 computational linear algebra MCQs, reporting 100%
accuracy [Drori and Verma, 2021]. Others have used GPT models to solve various
MCQ-based exams, including the United States Medical Licensing Examination
(USMLE), with accuracy around 50% [Kung et al., 2022, Gilson et al., 2022,
Liévin et al., 2022], the Multistate Bar Examination (MBE) [Bommarito II and
Katz, 2022], and the American Institute of Certified Public Accountants’
(AICPA) Regulation (REG) exam [Bommarito et al., 2023].

Although, programming-related MCQs have not been studied directly, some
researchers in adjacent fields have studied reasoning about similarly formal
topics. Although, GPT can often answer questions _about_ systems and rules, it
is especially challenged by tasks that involve _applying_ them and reasoning
about their implications in novel examples. Hendryks et al. created data set
that includes a wide variety of MCQs across STEM, humanities and arts, with
GPT-3 performing at levels above 50% for subjects such as marketing and
foreign policy, but below 30% for topics like formal logic [Hendrycks et al.,
2020]. They found that the model performed particularly poorly in quantitative
subjects. For example, in Elementary Mathematics they note that GPT can answer
questions _about_ arithmetic order of operations (e.g. that multiplications
are performed before additions), it cannot correctly answer questions that
require _applying_ this concept. They also note that GPT performance is not
necessarily correlated with how advanced the topic is for humans, doing better
at College Mathematics than Elementary Mathematics. Finally, they noted that
GPT does poorly on tests of legal and moral reasoning [Hendrycks et al.,
2020].

Lu et al. studied GPT models’ performance on a large data set consisting of
21,208 MCQs on topics in natural science, social science, and language [Lu et
al., 2022]. They prompted the models to produce an explanation along with its
answer and reported 1-3% improvement in accuracy (74.04%). In this work, we do
not adopt the approach and, hence, leave space for future work as it appears
quite promising and definitely applicable in the context of programming MCQs.

There is a growing body of related work on GPT models’ capabilities in solving
programming tasks by generating code. Finnie-Ansley et al. evaluated Codex on
23 programming tasks used as summative assessments in a CS1 programming course
[Finnie-Ansley et al., 2022]. Denny et al. focused on the effects of prompt
engineering when applying Copilot to a set of 166 exercises from the publicly
available CodeCheck repository [Denny et al., 2022]. Outside of the
educational context, there have been studies exploring GPT’s capabilities on
competitive and interview programming tasks. Chen et al. released the
HumanEval data set where Codex achieved 28.8% success rate on the first
attempt and 72.3% when allowed 100 attempts [Chen et al., 2021]. Li et al.
report Deepmind’s AlphaCode performance on Codeforces
competitions,333Codeforces. Available at: https://codeforces.com/contests
[Accessed 2023-01-22] achieving a 54.3% ranking amongst 5,000 participants [Li
et al., 2022]. Karmakar et al. reported 96% pass rate for Codex on a data set
of 115 programming problems from HackerRank444HackerRank. Available at:
https://www.hackerrank.com/ [Accessed 2023-01-22] [Karmakar et al., 2022].
Nguyen and Nadi reported Copilot’s effectiveness on LeetCode555LeetCode.
Available at: https://leetcode.com/ [Accessed 2023-01-22] problems, achieving
42% accuracy [Nguyen and Nadi, 2022].

Program code does more than control computer execution; it also, some argue
primarily, serves as communication among developers [Knuth, 1984]. Since GPT
is a text prediction model trained on code in the context of human discussions
about it, the model’s representation of code is likely to capture code’s
_design intent_ more strongly than code’s _formal properties_. For example,
work from multiple studies suggest that models that interpret code depend
heavily on function names and input variables [Mohammadkhani et al., 2022,
Yang et al., 2022]. Although, models like GPT are not trained to simulate code
execution, they can in many cases generate code based on natural language
description of the code’s intent. Researchers have reported varying success at
generating code in response to programming assignments, ranging from Codex’s
100% success generating Python computational linear algebra programs [Drori
and Verma, 2021], to 78.3% on some CS1 programming problems [Finnie-Ansley et
al., 2022], to 79% on the CodeCheck666CodeCheck: Python Exercises. Available
at: https://horstmann.com/codecheck/python-questions.html [Accessed
2022-01-22] repository of Python programming problems [Denny et al., 2022].

Researchers have identified distinct cognitive processes involved in
programming. Characterizing the kinds of learning necessary to teach
programming, Robins et al. claim for example that the _knowledge_ of how
programming constructs work is cognitively different from the _strategy_ or
plan for how to build a program; and that programming _comprehension_ and
_generation_ are distinct mental processes that must be taught. Programming
skill is a blend of related cognitive processes; it is not surprising that a
generative model would not mimic all these processes equally well [Robins et
al., 2003].

GPT’s ability to answer questions intended as educational assessments
naturally raises the question of its use for cheating. Biderman and Raff noted
that GPT solutions can evade plagiarism detection by code similarity tools
such as MOSS [Biderman and Raff, 2022]. On the other hand, Wermelinger notes
that while Copilot-generated solutions can typically pass some tests, they do
not pass enough to get a passing grade on a typical assignment; he concludes
that Copilot can be a useful springboard towards solving CS1 problems, but
outside of very common stereotyped beginners’ exercises, learners’ substantial
contribution is still required [Wermelinger, 2023]. Becker et al. include a
broader discussion of the opportunities and challenges posed by code
generating tools [Becker et al., 2022].

##  4 DATA SET

We manually collected MCQ assessment exercises from three Python programming
courses. _Python Essentials - Part 1 (Basics)_ 777OpenEDG: Python Essentials -
Part 1 (Basics). Available at: https://edube.org/study/pe1 [Accessed
2023-01-15] (PE1) aims to guide a learner from a state of complete programming
illiteracy to a level of programming knowledge which allows them to design,
write, debug, and run programs encoded in the Python language. The course
consists of four content units and one completion (summary) test. The units
include (i) introduction to Python and computer programming, (ii) data types
variables, basic I/O, operations and basic operators, (iii) boolean values,
conditional loops, lists, logical and bitwise operators, and (iv) functions,
tuples, dictionaries, data processing and exceptions.

_Python Essentials - Part 2 (Intermediate)_ (PE2)888OpenEDG: Python Essentials
- Part 2 (Intermediate). Available at: https://edube.org/study/pe2 [Accessed
2023-01-15] is focused on more advanced aspects of Python programming,
including modules, packages, exceptions, file processing, object-oriented
programming. Similarly to PE1, the course is organized into four content units
and one completion (summary) test. The course units are (i) modules, packages,
and pip, (ii) strings, string and list methods, and exceptions, (iii) object-
oriented programming, and (iv) miscellaneous.

Finally, _Practical Programming with Python_ 999Sail(): Social and Interactive
Learning Platform. Available at: https://sailplatform.org/courses. [Accessed
2023-03-03] (PPP) emphasizes hands-on experience with fundamental Python
constructs and exposure to software development tools, practices, and real-
world applications. The course consists of eight units which include (i)
Python basics and introduction to functions, (ii) control flow, strings, input
and output, (iii) Python data structures, (iv) object-oriented programming,
(v) software development, (vi) data manipulation, (vii) web scraping and
office document processing, and (viii) data analysis.

In PE1 and PE2, formative assessments are called quizzes while summative
assessments are called tests. The tests determine if learners pass the courses
whereas quizzes are meant as practice. The MCQs often include small snippets
of code for learners to reason about. From the two courses, we collected 297
questions (179 have code snippets). PPP uses MCQ-style inline activities as
formative assessment and tests as summative assessment. From this course, we
collected 233 MCQs (144 with code snippets). Table 1 has additional details.

Table 1: Descriptive statistics of the created dataset. Each row provides information about the MCQs each of the courses employ. Each column reports on the distribution of the code content of each MCQ set in each course. Course | Units | MCQ | MCQ | Course
---|---|---|---|---
| (topics) | (plain) | (+code) | Overall
PE1 | 4 | 53 | 96 | 149
PE2 | 4 | 65 | 83 | 148
PPP | 8 | 89 | 144 | 233
Type Overall | 16 | 207 | 323 | 530

We used simple pattern matching combined with manual curation as the second
step to organize the MCQs into several categories. The first distinction was
made between MCQs _with code_ and MCQs with _no code_. For an MCQ, to be
considered as _with code_ one of the following two had to be true:

  * •

Within the body of the question there had to be at least one line fully
dedicated to computer code.

  * •

The choices were computer code expressions.

Inline mentions of names of functions or variables were not considered as
sufficient for an MCQ to be considered _with code_.

Figure 2: Distribution of MCQs into categories. Note that the MCQs asking
about the output of a code snippet as well as MCQs focused on filling-in the
blanks in a snippet are not present in the MCQs with no code. This is to be
expected given the nature of those questions. The MCQs with code are quite
dominated by questions that ask about the output of a code snippet as well as
with questions of other type. Otherwise, the distribution is relatively
uniform.

The second distinction was made along the following lines, focusing on the
overall syntax of what the question writer asks the student to do:

  * •

True/False
The learner is asked to assess the truthfulness of a single statement. For
example:

> Developers that write code individually are not expected to apply code
> standards.
> A. True
> _B. False_

> Evaluate the following expression and determine whether it is True or False.
>
>     >     2 + 2 != 2 * 2
>
>
> A. True
> _B. False_

  * •

Identify True/False Statement
The learner is asked to pick one or more answer choices that are either true
or false. Note that this is different from the True/False questions (previous
category). For example:

> Which of the following statements is false?
> A. The pandas module provides some CSV-related methods.
> B. Python has a built-in XML package with several modules for XML parsing.
> _C. JSON data format has syntax to represent all Python data structure
> types._
> D. Python has a built-in `csv` module containing methods for reading and
> writing into CSV files.

> Take a look at the snippet and choose one of the following statements which
> is true:
>
>     >      nums = []
>      vals = nums[:]
>      vals.append(1)
>
>
> A. `nums` is longer than ‘vals‘
> _B._ `vals` _is longer than_ `nums`
> C. `nums` and `vals` are of the same length

  * •

Finish Statement.
The learner is asked to complete a statement. For example:

> The ‘**‘ operator:
> A. performs duplicated multiplication
> B. does not exist
> _C. performs exponentiation_

> Right-sided binding means that the following expression:
>
>     >     1 ** 2 ** 3
>
>
> will be evaluated:
> _A. from right to left_
> B. in random order
> C. from left to right

  * •

Output
The learner is asked to identify the choice that corresponds to the output of
a given snippet of code. This category is applicable only to questions _with
code_. For example:

> What is the output of the following snippet if the user enters two lines
> containing `2` _and_ `4` _respectively?_
>
>     >     x = int(input())
>     y = int(input())
>     print(x + y)
>
>
> A. 2
> B. 24
> _C. 6_

> What is the output of the following snippet?
>
>     >     my_list_1 = [1, 2, 3]
>     my_list_2 = []
>     for v in my_list_1:
>     my_list_2.insert(0, v)
>     print(my_list_2)
>
>
> A. `[1, 2, 3]`
> B. `[1, 1, 1]`
> C. `[3, 3, 3]`
> _D._ `[3, 2, 1]`

  * •

Fill-in Blanks
The learner is asked to fill in a code snippet by selecting the appropriate
choice as an answer. This category is applicable only to questions _with
code_. For example:

> Fill in the blank of the `is_negative` function definition shown below, so
> that the function returns `True` when the argument provided to `num` is a
> negative number and returns `False` otherwise.
>
>     >     def is_negative(num):
>         return _________________
>
>
> A. `not (num > 0)`
> B. `num > 0`
> C. `num <= 0`
> _D._ `num < 0`

> The following code snippet should open the `myfile` file and assign the
> lines to the `all_lines` variable. Which of the options below should be used
> to fill in the blanks?
>
>     >     with __________________________
>         all_lines = file.readlines()
>
>
> _A._ _open("myfile",’r’) as file:_
> B. `"myfile" in open as file:`
> C. `with open "myfile" as file:`

  * •

Other
Any MCQ that does not fall into any of the above categories. For example:

> How many times will the code snippet below print ‘X‘.?
>
>     >     for i in range(1, 7):
>         for j in range(2, 6):
>         print(’X’)
>
>
> _A. 24_
> B. 28
> C. 35

Notice that the above example is closely related to the questions asking for
the _output_ of the snippet. However, there is a subtle difference since this
questions does not ask what is the output directly.

> Given the piece of code presented in the code snippet below, what is the
> value of `palindromes[1]`?
>
>     >     palindromes = [’pop’, ’noon’, ’madam’]
>
>
> A. `’pop’`
> _B._ `’noon’`
> C. `’p’`
> D. `’madam’`
> E. `’o’`

Figure 2 shows the distribution of the MCQs into the individual categories.
The MCQs asking about the output of a code snippet as well as MCQs focused on
filling-in the blanks in a snippet are not present in the MCQs with _no code_.
This is to be expected given the nature of those questions. The MCQs with code
are quite dominated by questions that ask about the _output_ of a code snippet
as well as with questions of _other_ type. Otherwise, the distribution is
relatively uniform. The _fill-in_ questions are rare. The distribution of the
_no code_ questions is close to uniform.

##  5 MODELS

The original GPT model [Radford et al., 2018] is a 12-layer decoder-only
transformer [Vaswani et al., 2017] with masked self-attention heads. Its core
capability is fine-tuning on a downstream task. The GPT-2 model [Radford et
al., 2019] largely follows the details of the original GPT model with a few
modifications, such as layer normalization moved to the input of each sub-
block, additional layer-normalization after the first self-attention block,
and a modified initialization. Compared to the original model it displays
remarkable multi-task learning capabilities [Radford et al., 2019]. The next
generation of GPT models [Brown et al., 2020] uses almost the same
architecture as GPT-2. The only difference is that it uses alternating dense
and locally banded sparse attention patterns in the layers of the transformer.
The main focus of Brown et al. was to study the dependence of performance and
model size where eight differently sized models were trained (from 125 million
to 175 billion parameters). The largest of the models is commonly referred to
as GPT-3. The interesting property of these models is that they appear to be
very strong zero- and few-shot learners. This ability appears to improve with
the increasing size of the model [Brown et al., 2020].

We are primarily interested in the performance of `text-davinci-003`, one of
the most advanced GPT models offered by OpenAI. The `text-davinci-003` model
builds on top of previous `text-davinci-002`, which in turn is based on `code-
davinci-002` (focused on code-completion tasks). To gauge the rate of
improvement over the several recent years, we compare the performance of
`text-davinci-003` to `text-davinci-002` as well as to the previous
generation’s InstructGPT model (`text-davinci-001`).101010OpenAI: Model index
for researchers. Available at: https://beta.openai.com/docs/model-index-for-
researchers/instructgpt-models [Accessed 2023-01-15] Recently, OpenAI has also
released `gpt-3.5-turbo` which reportedly matches the performance of the
`text-davinci-003` for tenth of the cost.

We set the `temperature` to 0.0, which corresponds to no randomness. The
higher the `temperature` the more creative the output but it can also be less
factual. We set `max_tokens` to 500 (a token roughly corresponds to a word).
This parameter controls the maximum length of the output. We set `top_p` to 1,
as is recommended when `temperature` is set to 0.0. This parameter is related
to `temperature` and also influences creativeness of the output. We set
`frequency_penalty` to 0, which allows repetition by ensuring no penalty is
applied to repetitions. Finally, we set `presence_penalty` to 0, ensuring no
penalty is applied to tokens appearing multiple times in the output.

##  6 EXPERIMENTAL DESIGN

To test the performance of the three `text-davinci-*` models, we submit MCQs
one by one using the `openai` Python library111111GitHub: OpenAI Python
Library. Available at: https://github.com/openai/openai-python [Accessed
2023-01-16] which is a wrapper for the OpenAI’s REST API. We embed each
question in the prompt template shown in Figure 3. The text of the prompt’s
preamble is inspired by OpenAI’s QA example.121212OpenAI: Q&A. Available at:
https://platform.openai.com/examples/default-qa [Accessed 2023-03-04] The
{{question}} token is replaced with the question text. The {{choices}} token
is replaced with the candidate answers where each one is placed on a single
line preceded by a capital letter. Each model returns one or more of the
choices as the prompt completion, which is then compared to the reference
answer. For PE1 and PE2, we let partially correct answers be incorrect,
following the course creators’ assessment guidelines. In PPP, there is always
exactly one correct answer.

As the baseline, we use a simple model that selects the answer with the
highest Jaccard similarity to the question. In case of a tie the longest
answer is selected. Jaccard similarity is one of the simplest measures of text
similarity. Hence, it is an ideal candidate for a baseline as it allows to
detect what ratios of the questions within their respective categories could
be solved employing this simple, yet sensible, heuristic. Such MCQs likely
pose very little challenge for GPT models.

We report the proportions of the correct answers (i.e., the accuracy) for each
model per MCQ category. We specifically focus on the differences in
performance of the `text-davinci-003` model on MCQs that contain code snippets
(_with code_) compared to MCQs that do not (_no code_). We are also interested
in the difference between the performance on completion-based MCQs (_Finish
Statement_ and _Fill-in Blanks_) compared to the rest. This is because these
question types are not too far off from the pre-training objective and, hence,
the expectation is that the models’ performance should be higher on these
types. To test statistical significance we use a simple two-independent
proportions test which is a statistical hypothesis test used to determine
whether two proportions are different from each other.

I am a highly intelligent bot that can easily handle answering multiple-choice
questions on introductory Python topics. Given a question and choices I can
always pick the right ones. Question: {{question}} Choices: {{choices}} The
correct answer: 3.4cm(5.7cm,-2.75cm) 1 3.4cm(3.7cm,-2.45cm) 2
3.4cm(1.95cm,-1.4cm) 3

Figure 3: MCQ Prompt Template. The text of the preamble (1) is inspired by
OpenAI’s QA example. The {{question}} token (2) is replaced with the question
text. The {{choices}} token (3) is replaced with the candidate answers where
each one is placed on a single line preceded by a capital letter.

##  7 RESULTS

Table 2: Results of the experiments. The Jaccard column reports the performance of the baseline. The text-davinci-001, text-davinci-002, and text-davinci-003 columns report the performance of the different GPT3 models. Results of the No Code and With Code sections are summarized in the Total rows. The Overall row at the bottom reports the average performance of the models across all the types of MCQs. Question Type | Jaccard | text-davinci-001 | text-davinci-002 | text-davinci-003
---|---|---|---|---
No Code
True/False | 11/25 | 13/25 | 19/25 | 20/25
| (44.0%) | (52.0%) | (76.0%) | (80.0%)
Identify True/False Statement | 8/44 | 12/44 | 22/44 | 27/44
| (18.2%) | (27.3%) | (50.0%) | (61.4%)
Finish Statement | 12/53 | 40/53 | 46/53 | 48/53
| (22.6%) | (75.5%) | (86.8%) | (90.6%)
Other | 9/47 | 27/50 | 43/50 | 39/50
| (19.1%) | (53.2%) | (86.0%) | (74.0%)
Total | 40/172 | 92/172 | 130/172 | 134/172
| (23.2%) | (53.5%) | (75.6%) | (77.9%)
With Code
True/False | 9/23 | 12/23 | 10/23 | 10/23
| (39.1%) | (52.2%) | (43.5%) | (43.5%)
Identify True/False Statement | 4/26 | 10/26 | 15/26 | 11/26
| (15.4%) | (38.5%) | (57.7%) | (42.3%)
Output | 22/110 | 28/110 | 58/110 | 53/110
| (20.0%) | (25.4%) | (52.7%) | (48.2%)
Fill-in | 2/13 | 5/13 | 10/13 | 11/13
| (15.4%) | (38.5%) | (76.9%) | (84.6%)
Finish Statement | 8/27 | 10/27 | 22/27 | 22/27
| (29.6%) | (37.0%) | (81.5%) | (81.5%)
Other | 39/159 | 42/159 | 97/159 | 106/159
| (24.5%) | (26.4%) | (61.1%) | (66.7%)
Total | 84/358 | 107/358 | 212/358 | 213/358
| (23.4%) | (29.9%) | (59.2%) | (59.5%)
Overall | 124/530 | 199/530 | 342/530 | 347/530
| (23.4%) | (37.5%) | (64.5%) | (65.5%)

Table 2 reports the results of our experiments. Firstly, as one would expect
all three GPT models clearly outperform the simple Jaccard similarity
baseline. The `text-davinci-003` model appears to perform the best (65.5%
overall) with a small margin over the `text-davinci-002` (64.5% overall). The
performance of the `text-davinci-001` appears to be much lower compared to the
other two models. This is to be expected. While the `text-davinci-002` is a
direct predecessor of the `text-davinci-003` (hence, the small difference) the
`text-davinci-001` is quite removed from the two. The major breakthrough in
OpenAI GPT-3’s capabilities in handling computer code was Codex (`code-
davinci-002`) [Chen et al., 2021] which is the direct predecessor of `text-
davinci-002`.131313OpenAI: Model index for researchers. Available at:
https://beta.openai.com/docs/model-index-for-researchers/instructgpt-models
[Accessed 2023-01-15]

There appears to be a clear difference between the performance of the most
capable `text-davinci-003` on the MCQs that contain code snippets (59.5%
overall) compared to those that do not (77.9% overall). This difference is
statistically significant ($p<0.0001$). This is to be expected as the
combination of code and natural language likely constitutes (on average) more
complex input than natural language alone. Additionally, it is quite possible
that in our particular context the questions with code are (on average) more
difficult than questions with no code.

There also appears to be clear difference between the performance of `text-
davinci-003` on the completion-oriented MCQs (87.1%) and the rest (60.1%).
This difference is statistically significant ($p<0.0001$). Since GPT models
are primarily focused on prompt completion, be it text or computer code, this
finding is also as expected.

##  8 DISCUSSION

Our experimental results suggest that there, indeed, is a difference between
how successfully the GPT models handle questions that contain only natural
language and those that also contain snippets of computer code (RQ1).
Tentatively, we can conclude that inclusion of a code snippet within an MCQ
makes the question more challenging for GPT models to handle. This conclusion
is supported by universally lower performance on MCQs with code across all the
subtypes, i.e., _True/False_ , _Identify True/False Statement_ , _Finish
Statement_ , and _Other_. The root cause for this discrepancy is likely one or
more of the following: (i) GPT models are somewhat more limited with respect
to handling computer programs compared to natural language; (ii) GPT models
struggle with the combination of different types of expressions (i.e., natural
language and code); and/or (iii) the questions with code snippets are
inherently more difficult.

While the greater difficulty of the questions with code might certainly be a
factor it appears that the GPT models sometimes struggle to answer questions
with code that one might judge as simple. For example, consider the following
MCQ:

> The following statement:
>
>
>     assert var == 0
>
>
> A. is erroneous
> _B. will stop the program when var != 0_
> C. has no effect
> D. will stop the program when var == 0

The answer of `text-davinci-003` to this question was “D. will stop the
program when var == 0”. Hence, it appears there are certain limitations in the
capabilities of the GPT models to answer questions about code. This is
somewhat surprising if one considers the well documented capabilities of the
models when it comes to generation or explanation of computer programs.

The results also show that certain types of MCQs are more challenging than
others for the GPT models (RQ2). The questions that involve generation of
natural language and/or code appear to be handled with much more success than
other types of questions. This is to be expected as GPT models are primarily
focused on prompt completion. On the other hand, it leads to somewhat
paradoxical situations such as the one illustrated in the motivating example
(Section 2). The models are capable of generating code based on a natural
language description, as well as generating natural language explaining
execution of the code line-by-line. Yet, somehow these capabilities do not
seem to extend to the realm of answering pointed specific questions about the
code (often quite simple ones).

We hypothesize that the above described paradox might be related to the
phenomenon described by [Détienne and Bott, 2002]. They point out that program
code serves two simultaneous purposes: it is both a narrative description of a
programmer’s intent, and an artifact that controls computer execution.
Accordingly, human programmers maintain, and synchronize, at least two kinds
of mental models of code, a _functional_ model that captures the purpose the
program is supposed to play in the world, and a _structural_ model that allows
mental simulation of data and control flow.

Since GPT models are trained on large corpora that include texts in natural
language as well as program code with comments and documentation, they may
acquire robust representations of the functional relationship between code and
the intent it expresses. The training corpora likely do not contain code with
outputs or trace logs of its execution. Thus, models may lack the required
data to build a representation of a structural model of code’s function. This
is not to say that including the mentioned resources into the training corpora
would necessarily result in the acquisition of such a model. This is because
an effective use of the model may require the ability to simulate execution of
the code, or its part. The current large language models, including GPT, do
not have this capability. Note that there is an active area of research in
augmenting large language models with reasoning skills and providing them with
the ability to use tools, such as the Python interpreter [Mialon et al.,
2023].

Arguably, building up these connected mental models of code’s purpose and
operation should be a key part of what CS education teaches. The particular
limitations of GPT models provide a useful lens into what kind of mental model
we are evaluating in typical higher education programming assessments. It may
be that _True/False_ and _Identify True/False Statements_ MCQs more likely
require mental simulation of the code execution. An experiment to validate our
hypothesis might be to classify MCQs according to their focus on (a)
predicting actual behavior, or (b) inferring intent, and measure if and how
the GPT models’ performance correlates with this classification.

There are ongoing debates as to the changes the emergence of GPT-based tools
such as ChatGPT or GitHub Copilot will inflict on the software development
profession as well as programming education. Firstly, it appears inevitable
that the tools will become an integral and accepted part in the software
development process. Therefore, future programmers will likely need to write
less code. On the other hand, they will need to be able to validate the auto-
generated code, spot deficiencies, and correct them efficiently. Hence,
programming education might need to de-prioritize teaching learners how to
write code and start emphasizing skills such as requirements formulation,
debugging, trade-off analysis, and critical thinking.

Finally, the GPT-based tools present numerous opportunities to improve current
instructional and assessment practices in programming classes. Our experiments
suggest that GPT models are capable of explaining code in plain and easily
understandable terms. Similarly, they are capable of generating and completing
program code. A judicious use of these capabilities might result in numerous
novel tools and instructional approaches for novices and advanced learners
alike. However, there are also potential threats. An improper or misinformed
use of the tools may result in an academic integrity violation (AIV) incident
(i.e., cheating). Similarly, over-reliance on GPT-based tools may rather
hinder than improve the learning process.

##  9 CONCLUSIONS AND FUTURE WORK

We evaluated `text-davinci-*` GPT models on a sizeable set of 530 MCQs, many
of which contained code snippets, from three Python programming courses. The
overall accuracy of the most capable `text-davinci-003` model was measured at
65.5% (compared to the 23.4% Jaccard similarity baseline). While such
performance is impressive there appear to be some noticeable limitations.
First of all, it appears that the MCQs containing code snippets were somewhat
more challenging (59.5%) for the model than those with no code (77.9%). In
addition, MCQs that ask to complete a sentence or fill-in a blank appear to be
handled much more successfully (87.1%) compared to other types of questions
(60.1%). Therefore, GPT models’ capabilities seem limited when it comes to
handling MCQs about computer code requiring reasoning beyond mere completion
(56.6%).

While our study of GPT models’ performance on diverse types of MCQs yielded
numerous valuable insights, it is subject to countless limitations and leaves
much room for improvement. Hence, we suggest several directions for future
work: (i) further analyze the effects of prompt-tuning (ii) and/or iterative
prompt-construction; (iii) examine the performance of GPT models on other
domains, e.g., competitive mathematics; (iv) develop a systematic framework to
comprehensively assess the capabilities and limitations of GPT models; and (v)
study possibilities of effective integration of GPT-based tools, e.g., ChatGPT
or Copilot, into programming education.

## REFERENCES

  * Ankur and Atul, 2022 Ankur, D. and Atul, D. (2022).  Introducing Amazon CodeWhisperer, the ML-powered coding companion.  AWS Machine Learning Blog.  June 24, 2022. https://aws.amazon.com/blogs/machine-learning/introducing-amazon-codewhisperer-the-ml-powered-coding-companion/.
  * Beck, 2000 Beck, K. (2000).  Extreme programming explained: embrace change.  Addison-Wesley professional.
  * Becker et al., 2022 Becker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., and Santos, E. A. (2022).  Programming is hard–or at least it used to be: Educational opportunities and challenges of ai code generation.  arXiv preprint arXiv:abs/2212.01020.
  * Biderman and Raff, 2022 Biderman, S. R. and Raff, E. (2022).  Fooling moss detection with pretrained language models.  Proceedings of the 31st ACM International Conference on Information & Knowledge Management.
  * Bommarito et al., 2023 Bommarito, J., Bommarito, M., Katz, D. M., and Katz, J. (2023).  GPT as knowledge worker: A zero-shot evaluation of (AI) CPA capabilities.  arXiv preprint arXiv:abs/2301.04408.
  * Bommarito II and Katz, 2022 Bommarito II, M. and Katz, D. M. (2022).  GPT takes the bar exam.  arXiv preprint arXiv:abs/2212.14402.
  * Bowman, 2023 Bowman, E. (2023).  A college student created an app that can tell whether ai wrote an essay.  NPR Technology.  January 9, 2023. https://www.npr.org/2023/01/09/1147549845.
  * Brown et al., 2020 Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020).  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901.
  * Chen et al., 2021 Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. (2021).  Evaluating large language models trained on code.  arXiv preprint arXiv:abs/2107.03374.
  * Denny et al., 2022 Denny, P., Kumar, V., and Giacaman, N. (2022).  Conversing with Copilot: Exploring prompt engineering for solving cs1 problems using natural language.  arXiv preprint arXiv:abs/2210.15157.
  * Drori and Verma, 2021 Drori, I. and Verma, N. (2021).  Solving linear algebra by program synthesis.  arXiv preprint arXiv:2111.08171.
  * Détienne and Bott, 2002 Détienne, F. and Bott, F. (2002).  Software design–cognitive aspects.  Springer Verlag.
  * Elsen-Rooney, 2023 Elsen-Rooney, M. (2023).  NYC education department blocks ChatGPT on school devices, networks.  Chalkbeat New York.  January 3, 2023.
  * Finnie-Ansley et al., 2022 Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., and Prather, J. (2022).  The robots are coming: Exploring the implications of OpenAI Codex on introductory programming.  In Australasian Computing Education Conference, ACE ’22, page 10–19, New York, NY, USA. Association for Computing Machinery.
  * Gilson et al., 2022 Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L. S., Taylor, R. A., and Chartash, D. (2022).  How well does chatgpt do when taking the medical licensing exams? the implications of large language models for medical education and knowledge assessment.  In medRxiv.  https://doi.org/10.1101/2022.12.23.22283901.
  * Hendrycks et al., 2020 Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2020).  Measuring massive multitask language understanding.  arXiv preprint arXiv:abs/2009.03300.
  * Huang, 2023 Huang, K. (2023).  Alarmed by A.I. chatbots, universities start revamping how they teach.  New York Times.  January 16, 2023.
  * Karmakar et al., 2022 Karmakar, A., Prenner, J. A., D’Ambros, M., and Robbes, R. (2022).  Codex hacks HackerRank: Memorization issues and a framework for code synthesis evaluation.  ArXiv, abs/2212.02684.
  * Knuth, 1984 Knuth, D. E. (1984).  Literate programming.  The computer journal, 27(2):97–111.
  * Kung et al., 2022 Kung, T. H., Cheatham, M., Medinilla, A., Sillos, C., De Leon, L., Elepano, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2022).  Performance of ChatGPT on USMLE: Potential for ai-assisted medical education using large language models.  medRxiv preprint.  https://doi.org/10.1101/2022.12.19.22283643.
  * Lai et al., 2017 Lai, G., Xie, Q., Liu, H., Yang, Y., and Hovy, E. (2017).  Race: Large-scale reading comprehension dataset from examinations.  arXiv preprint arXiv:abs/1704.04683.
  * Li et al., 2022 Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., Hubert, T., Choy, P., de Masson d’Autume, C., Babuschkin, I., Chen, X., Huang, P.-S., Welbl, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D. J., Robson, E. S., Kohli, P., de Freitas, N., Kavukcuoglu, K., and Vinyals, O. (2022).  Competition-level code generation with AlphaCode.  Science, 378(6624):1092–1097.
  * Liévin et al., 2022 Liévin, V., Hother, C. E., and Winther, O. (2022).  Can large language models reason about medical questions?  ArXiv preprint arXiv:abs/2207.08143.
  * Lu et al., 2022 Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. (2022).  Learn to explain: Multimodal reasoning via thought chains for science question answering.
  * McDowell et al., 2002 McDowell, C., Werner, L., Bullock, H., and Fernald, J. (2002).  The effects of pair-programming on performance in an introductory programming course.  In Proceedings of the 33rd SIGCSE technical symposium on Computer science education, pages 38–42.
  * Mialon et al., 2023 Mialon, G., Dessì, R., Lomeli, M., Nalmpantis, C., Pasunuru, R., Raileanu, R., Rozière, B., Schick, T., Dwivedi-Yu, J., Celikyilmaz, A., et al. (2023).  Augmented language models: a survey.  arXiv preprint arXiv:2302.07842.
  * Mihaylov et al., 2018 Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. (2018).  Can a suit of armor conduct electricity? A new dataset for open book question answering.  arXiv preprint arXiv:abs/1809.02789.
  * Mohammadkhani et al., 2022 Mohammadkhani, A. H., Tantithamthavorn, C. K., and Hemmati, H. (2022).  Explainable AI for pre-trained code models: What do they learn? When they do not work?  ArXiv preprint, arXiv:abs/2211.12821.
  * Mostafazadeh et al., 2016 Mostafazadeh, N., Chambers, N., He, X., Parikh, D., Batra, D., Vanderwende, L., Kohli, P., and Allen, J. (2016).  A corpus and cloze evaluation for deeper understanding of commonsense stories.  In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839–849.
  * Nguyen and Nadi, 2022 Nguyen, N. and Nadi, S. (2022).  An empirical evaluation of GitHub Copilot’s code suggestions.  In 2022 IEEE/ACM 19th International Conference on Mining Software Repositories (MSR), pages 1–5.
  * Ouyang et al., 2022 Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022).  Training language models to follow instructions with human feedback.  arXiv preprint arXiv:abs/2203.02155.
  * Radford et al., 2018 Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. (2018).  Improving language understanding by generative pre-training.
  * Radford et al., 2019 Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019).  Language models are unsupervised multitask learners.
  * Robins et al., 2003 Robins, A., Rountree, J., and Rountree, N. (2003).  Learning and Teaching Programming: A Review and Discussion.  Computer Science Education, 13(2):137–172.
  * Robinson et al., 2022 Robinson, J., Rytting, C. M., and Wingate, D. (2022).  Leveraging large language models for multiple choice question answering.  arXiv preprint arXiv:abs/2210.12353.
  * Savelka et al., 2023 Savelka, J., Agarwal, A., Bogart, C., Song, Y., and Sakr, M. (2023).  Can generative pre-trained transformers (gpt) pass assessments in higher education programming courses?  In Proceedings of the 28th Annual ACM Conference on Innovation and Technology in Computer Science Education.
  * Vaswani et al., 2017 Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017).  Attention is all you need.  Advances in neural information processing systems, 30.
  * Wermelinger, 2023 Wermelinger, M. (2023).  Using GitHub Copilot to solve simple programming problems.  Proceedings of the 54th ACM Technical Symposium on Computing Science Education (SIGCSE).
  * Yang et al., 2022 Yang, G., Zhou, Y., Yang, W., Yue, T., Chen, X., and Chen, T. (2022).  How important are good method names in neural code generation? A model robustness perspective.  ArXiv, abs/2211.15844.