# Phase retrieval on circles and lines

I. Chalendar  Isabelle CHALENDAR, Université Gustave Eiffel, LAMA, (UMR 8050),
UPEM, UPEC, CNRS, F-77454, Marne-la-Vallée (France)  isabelle.chalendar@univ-
eiffel.fr and  J. R. Partington  Jonathan R. PARTINGTON, School of
Mathematics, University of Leeds, Leeds LS2 9JT, Yorkshire, U.K.
<EMAIL_ADDRESS>
###### Abstract.

Let $f$ and $g$ be analytic functions on the open unit disc ${\mathbb{D}}$
such that $|f|=|g|$ on a set $A$. We first prove that there exists $c$ in the
unit circle ${\mathbb{T}}$ such that $f=cg$ when $A$ is the union of two lines
in ${\mathbb{D}}$ intersecting at an angle that is an irrational multiple of
$\pi$. The same conclusion is valid when $f$ and $g$ are in the Nevanlinna
class and $A$ is the union of the unit circle and an interior circle,
tangential or not. We also provide sequential versions of the previous results
and analyse the case $A=r{\mathbb{T}}$. Finally we examine the situation when
there is equality on two distinct circles in the disc, proving a result or
counterexample for each possible configuration.

###### Key words and phrases:

Hardy space, phase retrieval, inner function, outer function

###### 2010 Mathematics Subject Classification:

30D05, 30H10, 94A12

##  1\. Background

A fundamental question of phase retrieval is the following: For which
measurable sets $A\subset\overline{\mathbb{D}}$ is an analytic function $f\in
H^{1}$ determined uniquely to within a unimodular constant by the values of
$|f|$ on $A$? Here ${\mathbb{D}}$ denotes the unit disc and
$H^{1}=H^{1}({\mathbb{D}})$ the associated Hardy class of analytic functions.
Such questions arise in signal processing, crystallography, quantum mechanics,
microscopy, and many other applications.

In [7] Pohl, Li and Boche worked with the class $H^{1}_{R}$ of functions in
$H^{1}=H^{1}({\mathbb{D}})$ such that the inner factor is a Blaschke product
(no singular part). They showed (their Theorem 3) that any such function $f$
is determined to within a unimodular constant by the values of $|f|$ on
${\mathbb{T}}$ and $r{\mathbb{T}}$ for any $0<r<1$.

Then in [3] Jaming, Kellay and Perez solved a phase retrieval problem in
$L^{2}({\mathbb{R}},e^{2c|x|}\,dx)$ for $c>0$. This is isometric by the
Fourier transform to the Hardy space of a strip
${\mathcal{S}}=\\{z\in{\mathbb{C}}:|\mathop{\rm Im}\nolimits z|<c\\}$ (similar
calculations occur in [8]).

By conformal mapping the phase retrieval problem can then be reformulated on
$H^{2}({\mathbb{D}})$ in terms of finding an expression for all pairs $F,G\in
H^{2}$ such that $|F|=|G|$ on $(-1,1)$. For example, with the notation $F^{*}$
for the function defined by $F^{*}(z)=\overline{F(\bar{z})}$, they showed that
for $F,G\in H^{2}$ one has $|F|=|G|$ on $(-1,1)$ if and only if there exist
$u,v\in\mathop{\rm Hol}({\mathbb{D}})$ (the space of all holomorphic functions
on ${\mathbb{D}}$) such that $F=uv$ and $G=uv^{*}$.

Earlier results along similar lines are due to McDonald [5], who studied the
problem of determining an entire function $f$ when $|f(x)|$ is known for every
real $x$.

This paper deals with several cases, for which it provides positive and
negative results: intersecting curves, disjoint circles, disjoint lines, and a
full analysis of the two-circle situation. We also show that many results have
sequential counterparts.

##  2\. Results

###  2.1. Intersecting curves

The following result generalizes [3, Lem. 4.5], which proves the results for
equality on two lines $(-1,1)$ and $e^{i\theta}(-1,1)$.

######  Theorem 2.1.

Let $f,g:{\mathbb{D}}\to{\mathbb{C}}$ be analytic functions and $L_{1}$,
$L_{2}$ two line segments in ${\mathbb{D}}$, intersecting at an angle
$\theta\in(0,\pi/2)\setminus\pi{\mathbb{Q}}$, such that $|f(z)|=|g(z)|$ for
all $z\in L_{1}\cup L_{2}$. Then $f=\beta g$ for some constant
$\beta\in{\mathbb{T}}$.

###### Proof.

Clearly by considering the functions on a small disc
$\\{z\in{\mathbb{C}}:|z-a|\leq r\\}$ where $a$ is the intersection point of
$L_{1}$ and $L_{2}$, and by translating and rescaling as necessary, we may
suppose that $f$ and $g$ lie in the disc algebra and the two lines intersect
at $0$.

We may also suppose that the functions are non-vanishing except possibly at
$0$.

If they do vanish at $0$, then the orders of the zeros are equal, since for
some $N$ $\lim_{z\to 0}|f(z)|/|z|^{N}$ exists and is nonzero, and the same for
$g$. So by dividing out any the zeros at $0$, and restricting to a smaller
disc if necessary, we may suppose that $f$ and $g$ are invertible functions in
the disc algebra.

Now $h:=f/g$ is also in the disc algebra and it has modulus $1$ on both lines
$L_{1}$ and $L_{2}$. Thus $h$ maps $L_{1}$ and $L_{2}$ into the unit circle in
the complex plane.

Suppose that for some $k\geq 1$ we have

$h(z)=c_{0}+c_{k}z^{k}+O(z^{k+1})$

near $0$, with $|c_{0}|=1$ and $c_{k}\neq 0$; then near to $0$ it magnifies
angles by a factor of $k$, and so if $h$ is non-constant, then $k\theta$ must
be an integer multiple of $\pi$. ∎

######  Example 2.2.

Take $L_{1}=[-1,1]$ and $L_{2}=i[-1,1]$. Then

$f(z)=\frac{z^{2}-2i}{z^{2}+2i}\qquad\hbox{and}\qquad
g(z)=\frac{z^{2}-3i}{z^{2}+3i}$

are both unimodular on $L_{1}\cup L_{2}$. Similar examples can be constructed
for other rational multiples of $\pi$.

######  Corollary 2.3.

The conclusions of Theorem 2.1 also hold if $L_{1}$ and $L_{2}$ are two
$C^{1}$ curves intersecting at an angle
$\theta\in(0,\pi/2]\setminus\pi{\mathbb{Q}}$.

###### Proof.

The conformality argument works with minor changes. ∎

We may prove sequential versions of the above results with the aid of the
following easy lemma.

######  Lemma 2.4.

Let $h$ be a function holomorphic on a neighbourhood of $0$,
$\theta\in[0,2\pi]$ and $(a_{n})$ a real sequence such that $a_{n}\to 0$ and
$h(a_{n}e^{i\theta})$ is real for all $n$. Then all the derivatives of the
function $z\mapsto h(ze^{-i\theta})$ at $0$ are real.

###### Proof.

Clearly we may assume without loss of generality that $\theta=0$. Then
$h^{\prime}(0)=\lim_{n\to\infty}h(a_{n})/a_{n}\in{\mathbb{R}}$. The map
$h_{1}:z\mapsto h(z)/z-h^{\prime}(0)$ is also holomorphic around $0$, and
satisfies $h_{1}(a_{n})\in{\mathbb{R}}$. Then, in a same way, we get
$h_{1}^{\prime}(0)=h^{\prime\prime}(0)\in{\mathbb{R}}$. Iterating this
reasoning, we see that all the derivatives of $h$ at $0$ must be real. ∎

######  Corollary 2.5.

Let $f,g:{\mathbb{D}}\to{\mathbb{C}}$ be analytic functions and $L_{1}$,
$L_{2}$ two line segments in ${\mathbb{D}}$, intersecting at a point $p$, at
an angle $\theta\in(0,\pi/2)\setminus\pi{\mathbb{Q}}$, such that there are
sequences $(a_{n})\subset L_{1}$ and $(b_{n})\subset L_{2}$ tending to $p$
such that $|f(a_{n})|=|g(a_{n})|$ and $|f(b_{n})|=|g(b_{n})|$ for all $n$.
Then $f=\beta g$ for some constant $\beta\in{\mathbb{T}}$.

###### Proof.

We use the argument of the proof of Theorem 2.1 up to the point where we have
that $h(a_{n})$ and $h(b_{n})$ lie in ${\mathbb{T}}$ for all $n$. By composing
with suitable bilinear transformations we can construct a function $\tilde{h}$
holomorphic near $0$ with sequences $c_{n}\to 0$ and $d_{n}\to 0$ such that
the $c_{n}$ are real and the $d_{n}$ have argument $\theta$, but with
$\tilde{h}(c_{n})$ and $\tilde{h}(d_{n})$ both real for each $n$. By Lemma
2.4, we conclude that all the derivatives of $z\mapsto\tilde{h}(z)$ and
$z\mapsto\tilde{h}(ze^{i\theta})$ are real. By looking at the Taylor series,
we see that either $\tilde{h}$ is constant (and thus $h$ is constant) or
$e^{in\theta}=1$ for some $n>0$. ∎

Clearly, a similar generalization of Corollary 2.3 can be derived.

###  2.2. Disjoint circles

We can now prove a more general version of the main result, Theorem 3, from
[7], removing the restriction on the singular factor.

######  Theorem 2.6.

Let $f,g\in H^{1}({\mathbb{D}})$ satisfy $|f|=|g|$ a.e. on ${\mathbb{T}}$ and
$r{\mathbb{T}}$ for some $0<r<1$. Then $f=\lambda g$ for some
$\lambda\in{\mathbb{T}}$.

###### Proof.

By dividing out by the common outer factor $u$ given by

$u(z)=\exp\left(\frac{1}{2\pi}\int_{0}^{2\pi}\frac{e^{it}+z}{e^{it}-z}\log|f(e^{it})|\,dt\right)$

we may suppose without loss of generality that $f$ and $g$ are inner
functions.

Now, if there are any zeros of $f$ or $g$ on $r{\mathbb{T}}$ they are at the
same points, and finite in number. By dividing out a finite Blaschke product
we may suppose without loss of generality that $f$ and $g$ have no zeros on
$r{\mathbb{T}}$, so indeed $f/g$ is analytic on an annulus
$\\{z:r-\delta<|z|<r+\delta\\}$ for some $\delta>0$ and unimodular on
$r{\mathbb{T}}$.

Indeed, $f/g$ has finitely many poles in $\\{z:|z|<r+\delta\\}$ so,
multiplying by a function $h$ such that $h(z/r)$ is a finite Blaschke product
$B_{1}(z)$, $fh/g$ is a function $k$ such that $k(z/r)$ is also a finite
Blaschke product $B_{2}(z)$ since it is analytic in a neighbourhood of the
disc $r{\mathbb{T}}$.

Our conclusion is that $f/g$ is a rational function $F(z)=B_{2}(rz)/B_{1}(rz)$
in $r{\mathbb{D}}$; by analytic continuation (and removing any isolated
singularities) $f=Fg$ throughout the whole disc. But if $F$ is non-constant
then it is not unimodular on ${\mathbb{T}}$ as the zeros and poles are not
placed at inverse points. ∎

The result above may easily be extended to the Nevanlinna class
${\mathcal{N}}$ of holomorphic functions expressible as the ratio of two
$H^{1}$ functions.

With the aid of the following lemma, which gives a slight extension of [4,
Lem. 1.3], we can prove a stronger result.

######  Lemma 2.7.

Let $F$ be a function meromorphic on an open set containing the unit circle
${\mathbb{T}}$ and suppose that $|F(z)|=1$ for infinitely many
$z\in{\mathbb{T}}$. Then $|F(z)|=1$ for all $z\in{\mathbb{T}}$.

###### Proof.

This result uses a modification of an argument introduced in [1, Thm. 1.4].
Namely, we restrict $F$ to a continuous map from ${\mathbb{T}}$ to the Riemann
sphere ${\mathbb{C}}\cup\\{\infty\\}$, which we continue to denote by $F$.
Then $X=F^{-1}({\mathbb{T}})$ is a closed subset of ${\mathbb{T}}$.

If $X$ is an infinite proper subset of ${\mathbb{T}}$, we claim that $X$
contains a point that is a limit of two sequences, one in $X$ and one in
${\mathbb{T}}\setminus X$. This is clear if ${\mathbb{T}}\setminus X$ consists
of a finite number of intervals. If there are infinitely many intervals, then
their endpoints lie in $X$ and accumulate at a point $z_{0}\in X$ with the
required property.

This point satisfies $|g(z_{0})|=1$, and there exist two sequences
$(u_{n})\subset X$ and $(v_{n})\subset{\mathbb{T}}\backslash X$ which tend to
$z_{0}$, such that $|F(u_{n})|=1$ and $|F(v_{n})|\neq 1$ (by passing to a
subsequence we may avoid any poles of $F$).

By composing $F$ with conformal mappings between ${\mathbb{T}}$ and
${\mathbb{R}}\cup\\{\infty\\}$ we obtain a function $h$ holomorphic around
$0$, and two sequences $(a_{n}),(b_{n})\subset{\mathbb{R}}$ such that
$a_{n}\to 0$, $b_{n}\to 0$, $h(a_{n})\in{\mathbb{R}}$ and
$h(b_{n})\not\in{\mathbb{R}}$. By Lemma 2.4 we have that all the derivatives
of $h$ at $0$ must be real, so $h(b_{n})\in{\mathbb{R}}$ (since
$b_{n}\in{\mathbb{R}}$). This contradiction shows that $X$ cannot be an
infinite proper subset of ${\mathbb{T}}$. ∎

######  Theorem 2.8.

Let $f,g\in H^{1}({\mathbb{D}})$ satisfy $|f|=|g|$ a.e. on ${\mathbb{T}}$ and
on an infinite subset $X\subset r{\mathbb{T}}$ for some $0<r<1$. Then
$f=\lambda g$ for some $\lambda\in{\mathbb{T}}$.

###### Proof.

Let $f$ and $g$ have inner–outer factorizations $f=u_{1}v_{1}$,
$g=u_{2}v_{2}$, where $u_{1}$ and $u_{2}$ are inner and $v_{1}$ are $v_{2}$
outer with $v_{1}(0)>0$ and $v_{2}(0)>0$. Since $|f|=|g|$ on ${\mathbb{T}}$ we
have $v_{1}=v_{2}$. We may therefore suppose without loss of generality that
$f$ and $g$ are inner.

Note that the function $F:=f/g$ is meromorphic in the disc and $|F(z)|=1$ for
an infinite subset of ${\mathbb{T}}$.

By Lemma 2.7 $|F|=1$ on ${\mathbb{T}}$ and so $|f|=|g|$ on $r{\mathbb{T}}$.
The result now follows from Theorem 2.6.

∎

######  Corollary 2.9.

Let $f,g\in H^{1}({\mathbb{D}})$ (or more generally in the Nevanlinna class)
satisfy $|f|=|g|$ a.e. on ${\mathbb{T}}$ and on an infinite subset
$X\subset\Psi_{\omega,\alpha}(r{\mathbb{T}})$ for some $0<r<1$, where
$\omega\in{\mathbb{T}}$, $\alpha\in{\mathbb{D}}$, and
$\Psi_{\omega,\alpha}(z):=\omega\frac{\alpha-z}{1-\overline{\alpha}z}$. Then
$f=\lambda g$ for some $\lambda\in{\mathbb{T}}$.

###### Proof.

We apply Theorem 2.8 to $f\circ\Psi_{\omega,\alpha}$ and
$g\circ\Psi_{\omega,\alpha}$, which implies that
$f\circ\Psi_{\omega,\alpha}=g\circ\Psi_{\omega,\alpha}$, and thus $f=g$ since
$\Psi_{\omega,\alpha}$ is an automorphism of the open unit disc. ∎

Note that every circle contained in ${\mathbb{D}}$ can be written as
$\Psi_{\omega,\alpha}(r{\mathbb{T}})$ for a suitable choice of
$\omega\in{\mathbb{T}}$, $\alpha\in{\mathbb{D}}$, and $0<r<1$.

######  Remark 2.10.

It is clearly not enough to suppose that $|f|=|g|$ a.e. on ${\mathbb{T}}$ and
on some finite subset $X\subset r{\mathbb{T}}$. For example, we can let $f$
and $g$ be two different inner functions constructed in the following way:

$f=\Psi_{\alpha}\circ(Bu)\mbox{ and }g=\Psi_{\alpha}\circ(Bv),$

where $B$ is the finite Blaschke product associated with the finite set $X$,
$u$ and $v$ are inner functions with $u\neq cv$ for any $c\in{\mathbb{T}}$ and
$\Psi_{\alpha}(z):=\frac{\alpha-z}{1-\overline{\alpha}z}$ for an arbitrary
$\alpha\in{\mathbb{D}}$. Since $f$ and $g$ are inner, their radial limits are
of modulus one almost everywhere on ${\mathbb{T}}$, $f(z)=\alpha=g(z)$ for all
$z\in X$. The choice of $u$ and $v$ implies that there is no
$c\in{\mathbb{T}}$ such that $f=cg$.

If the inner parts of $f$ and $g$ are finite Blaschke products $B_{1}$ and
$B_{2}$, then, depending on their degrees, $|f|=|g|$ a.e. on ${\mathbb{T}}$
and at only finitely many points on $r{\mathbb{T}}$ is a sufficient condition
for Theorem 2.8 to hold.

######  Theorem 2.11.

Let $B_{1}$ and $B_{2}$ be Blaschke products of degrees $M$ and $N$,
respectively and take $0<r<1$. If $|B_{1}(z)|=|B_{2}(z)|$ for more than
$2N+2M-1$ distinct points on $r{\mathbb{T}}$, then $B_{2}$ is a constant
multiple of $B_{1}$.

###### Proof.

Note that on $r{\mathbb{T}}$ we have $\bar{z}=r^{2}/z$; so for an elementary
Blaschke factor with $a\in{\mathbb{D}}$ we have

$\frac{z-a}{1-\bar{a}z}\frac{\bar{z}-\bar{a}}{1-a\bar{z}}=\frac{(z-a)(r^{2}-\bar{a}z)}{(1-\bar{a}z)(z-r^{2}a)}$

and thus the equation

$B_{1}(z)\overline{B_{1}(z)}=B_{2}(z)\overline{B_{2}(z)}$

can be rewritten as $R_{1}(z)=R_{2}(z)$, where $R_{1}$ and $R_{2}$ are
rational functions of degrees $2M$ and $2N$ respectively. If
$\alpha_{1},\cdots,\alpha_{M}$ are the zeroes of $B_{1}$ and
$\beta_{1},\cdots,\beta_{N}$ are the zeroes of $B_{2}$, we get

$R_{1}(z)=\left(\frac{z-\alpha_{1}}{1-\overline{\alpha_{1}}z}\right)\cdots\left(\frac{z-\alpha_{M}}{1-\overline{\alpha_{M}}z}\right)\left(\frac{r^{2}-\overline{\alpha_{1}}z}{z-\alpha_{1}r^{2}}\right)\cdots\left(\frac{r^{2}-\overline{\alpha_{M}}z}{z-\alpha_{M}r^{2}}\right)$

and

$R_{2}(z)=\left(\frac{z-\beta_{1}}{1-\overline{\beta_{1}}z}\right)\cdots\left(\frac{z-\beta_{N}}{1-\overline{\beta_{N}}z}\right)\left(\frac{r^{2}-\overline{\beta_{1}}z}{z-\beta_{1}r^{2}}\right)\cdots\left(\frac{r^{2}-\overline{\beta_{N}}z}{z-\beta_{N}r^{2}}\right).$

Denote by $P_{1}(z)$ (resp. $P_{2}(z)$) the numerator of $R_{1}(z)$ (resp.
$R_{2}(z)$) and $Q_{1}(z)$ (resp. $Q_{2}(z)$) the denominator of $R_{1}(z)$
(resp. $R_{2}(z)$). This reduces to a polynomial equation of degree at most
$2N+2M-1$, noting that the coefficient of $z^{2N+2M}$ of the polynomial
$P_{1}Q_{2}$ is
$(-1)^{N+M}\prod_{i=1}^{M}\overline{\alpha_{i}}\prod_{j=1}^{N}\overline{\beta_{j}}$,
which coincide with the coefficient of $z^{2N+2M}$ of the polynomial
$P_{2}Q_{1}$. Therefore if $P_{1}Q_{2}-P_{2}Q_{1}$it has more than $2N+2M-1$
roots then it is identically zero. That is, $|B_{1}|=|B_{2}|$ on the whole of
$r{\mathbb{T}}$ and thus the result follows from the previous discussions. ∎

In general we have the following parametrization of functions of equal modulus
on $r{\mathbb{T}}$.

######  Theorem 2.12.

Suppose that $f,g\in\mathop{\rm Hol}({\mathbb{D}})$ and $|f|=|g|$ on
$r{\mathbb{T}}$ for some $0<r<1$. Then there exist finite Blaschke products
$B_{1},B_{2}$ such that

$B_{1}(z/r)f(z)=B_{2}(z/r)g(z).$

###### Proof.

We have that $f/g$ is meromorphic on a neighbourhood of
$r\overline{\mathbb{D}}$ and $|f/g|=1$ on $r{\mathbb{T}}$. By choosing a
suitable finite Blaschke product $B_{1}$ and writing
$\tilde{B}_{1}(z)=B_{1}(z/r)$ we have that $\tilde{B}_{1}f/g$ is still
unimodular on $r{\mathbb{T}}$ and holomorphic on a neighbourhood of
$r\overline{\mathbb{D}}$. Thus it has the form $\tilde{B}_{2}$, where
$\tilde{B}_{2}(z)=B_{2}(z/r)$ for some finite Blaschke product $B_{2}$. ∎

###  2.3. Disjoint lines

In [7] it is claimed that a similar result to Theorem 2.6 can be proved for
the corresponding space $H^{1}_{R}({\mathbb{C}}^{+})$ of analytic functions in
the upper half-plane ${\mathbb{C}}^{+}$, considering the values on
${\mathbb{R}}$ and $i+{\mathbb{R}}$, although no proof is given. The result
cannot be deduced directly from the disc result as this strip
$\mathbb{S}=\\{s\in{\mathbb{C}}:0<\mathop{\rm Re}\nolimits s<1\\}$ is not
conformally equivalent to the annulus.

We note also that there are non-constant meromorphic functions $F$ defined on
${\mathbb{C}}^{+}$ such that $|F|=1$ on both ${\mathbb{R}}$ and
$i+{\mathbb{R}}$. One example is

$F(s)=\frac{i-\exp(\pi s)}{i+\exp(\pi s)},$

which provides a conformal map of the strip $\mathbb{S}$ onto the unit disc.
However the function does not extend to a quotient of $H^{p}$ functions, since
its zeros $\\{i(\frac{1}{2}+2n):n\geq 0\\}$ do not form a Blaschke sequence.

By means of the Weierstrass factorization theorem, $F$ can be modified to give
distinct analytic functions $G$ and $H$ on ${\mathbb{C}}^{+}$ such that
$|G|=|H|$ on ${\mathbb{R}}$ and $1+i{\mathbb{R}}$.

In fact, the result we require does hold for the half-plane. It will be
convenient to work with the right half-plane ${\mathbb{C}}_{+}$.

######  Theorem 2.13.

Suppose that $f,g\in H^{1}({\mathbb{C}}_{+})$ and $|f|=|g|$ a.e. on
$i{\mathbb{R}}$, while $|f|=|g|$ on $1+i{\mathbb{R}}$. Then $f=cg$ for some
unimodular constant $c$.

###### Proof.

We may assume without loss of generality that $f,g$ are inner.

Now, the functions $F$ and $G$ defined by $F(s)=f(1+s)$ and $G(s)=g(1+s)$
satisfy $|F|=|G|$ on $i{\mathbb{R}}$; and these functions are also in
$H^{1}({\mathbb{C}}_{+})$, and holomorphic in a neighbourhood of
$i{\mathbb{R}}$. Let us consider the inner–outer factorization $F=uh$, say.

Clearly, the inner factor $u$ has no zeros accumulating at a point on
$i{\mathbb{R}}$. Moreover, the outer factor $h$ is continuous on the closed
half-plane ${\mathbb{C}}_{+}$ (see, for example, the arguments in [6, Sec.
4.3.8]), which means that the inner factor $u$ is also continuous except
possibly at the discrete set consisting of the zeros of $uh$ on
$i{\mathbb{R}}$. On factoring out a zero at $iy_{0}$, by considering $uh/b$,
where $b(s)=\dfrac{s-iy_{0}}{1+s}$ (an outer function) we see that $u$ is also
continuous at the zeros of $uh$. Thus $u$ has no singular part except possibly
an exponential factor $e^{-\alpha s}$ for some $\alpha\geq 0$. We conclude
that $u(s)=B_{1}(s)e^{-\alpha s}$ where $B_{1}$ is a Blaschke product whose
zeros do not accumulate at any point in $\overline{{\mathbb{C}}_{+}}$. The
function $B_{1}$, which is a product of factors of the form
$\dfrac{s-a_{n}}{s+\overline{a_{n}}}$, therefore has a meromorphic extension
to ${\mathbb{C}}$.

So suppose that $B_{1}$ has a zero at $w$. Then the Blaschke factor of $f$,
say, $b_{1}$, has a zero at $w+1$, and hence a pole at $-\overline{w}-1$. Thus
$B_{1}$ has a pole at $-\overline{w}-2$ and hence a zero at $w+2$.

Repeating this argument, we find that $B_{1}$ has zeros at $w_{n}:=w+2n$,
$n\in{\mathbb{N}}$. But these do not satisfy the Blaschke condition
$\sum_{n=0}^{\infty}\frac{\mathop{\rm Re}\nolimits
w_{n}}{1+|w_{n}|^{2}}<\infty$, and that is a contradiction.

So $B_{1}$ is a unimodular constant and we may apply a similar argument to the
inner function $v$. We conclude that $f(s)/g(s)=c\exp(\lambda(s-1))$, for a
unimodular constant $c$, which, since $f$ and $g$ are inner, means that
$|\exp(\lambda(iy-1))|=1$ a.e., and $\lambda=0$. ∎

The translation of the previous theorem into the open unit disc via a standard
conformal map between the unit disc and the right half-plane is the following.

######  Corollary 2.14.

Suppose that $f,g\in H^{1}({\mathbb{D}})$ and $|f|=|g|$ a.e. on
${\mathbb{T}}$, while $|f|=|g|$ on $\\{z\in{\mathbb{C}}:|z-r|=1-r\\}$ for some
$r\in(0,1)$. Then $f=cg$ for some unimodular constant $c$.

######  Remark 2.15.

Corollary 2.14 also holds when $f$ and $g$ are in thee Nevalinna class
$\mathcal{N}$ and when $|f|=|g|$ on ${\mathbb{T}}$ and on a sequence included
in $\\{z\in{\mathbb{C}}:|z-r|=1-r\\}$.

###  2.4. Two circles contained in the unit disc

Suppose now that $f$ and $g$ are holomorphic on ${\mathbb{D}}$ and $|f|=|g|$
on two distinct circles $C_{1},C_{2}$ contained in ${\mathbb{D}}$, which bound
open discs $D_{1}$ and $D_{2}$ respectively. There are five cases to consider
(we can of course swap the roles of $C_{1}$ and $C_{2}$ if we wish):

  1. (1)

$C_{1}$ and $C_{2}$ are “internally” disjoint with $C_{2}\subset D_{1}$;

  2. (2)

$C_{1}$ and $C_{2}$ are “externally” disjoint with
$C_{2}\subset{\mathbb{D}}\setminus\overline{D_{1}}$;

  3. (3)

$C_{1}$ and $C_{2}$ are “internally” tangential with
$C_{2}\subset\overline{D_{1}}$;

  4. (4)

$C_{1}$ and $C_{2}$ are “externally” tangential with
$C_{2}\subset{\mathbb{D}}\setminus D_{1}$;

  5. (5)

$C_{1}$ and $C_{2}$ intersect in two points, at which they make an angle
$\theta\in(0,\pi/2]$.

The five cases for two circles contained in ${\mathbb{D}}$.

######  Theorem 2.16.

Suppose that $C_{1}$ and $C_{2}$ satisfy one of the five conditions listed
above and that $|f|=|g|$ on $C_{1}\cup C_{2}$. Then in cases (1)–(4) $f=cg$
for some $c$ with $|c|=1$. The same holds in case (5) if $\theta$ is an
irrational multiple of $\pi$, but need not hold if $\theta$ is a rational
multiple of $\pi$.

###### Proof.

For parts (1)–(4) we may suppose without loss of generality, by composing with
an automorphism, that $C_{1}$ is the circle $r{\mathbb{T}}$ centred at $0$
with radius $r$ for some $0<r<1$. For (1) we now consider $f_{r}$ and $g_{r}$
defined by $f_{r}(z)=f(z/r)$ and $g_{r}(z)=g(z/r)$. Then (1) follows easily
from Corollary 2.9 and the comment following it.

For (2) with $C_{1}=r{\mathbb{T}}$ we note from Theorem 2.12 that $f(z)/g(z)$
is a rational function of the form $B_{2}(z/r)/B_{1}(z/r)$ and hence
meromorphic on ${\mathbb{C}}\cup\\{\infty\\}$. With the change of variable
$w=z/r$ we have a meromorphic function $B_{2}(w)/B_{1}(w)$ that is unimodular
on a smaller circle contained in ${\mathbb{D}}$. By composition with an
automorphism we may supposed that the smaller circle is centred at $0$. But
now the fact that $f/g$ is a constant follows an argument similar to that used
in the proof of Theorem 2.6 (specifically, that poles and zeros occur in pairs
of inverse points with respect to one circle, which means that they cannot
occur at inverse points with respect to the other circle).

(3) is easily derived from Corollary 2.14.

For (4) we may again suppose that $C_{1}=r{\mathbb{T}}$ and that
$f(z)/g(z)=B_{2}(z/r)/B_{1}(z/r)$. Now we take $w=z/r$ again, reducing to the
case when a function of the form $B_{2}(w)/B_{1}(w)$ is of modulus 1 on a
circle internally tangential to ${\mathbb{T}}$. By means of a conformal
mapping we can transform this to the half-plane, producing a rational function
that has absolute value 1 on the lines $i{\mathbb{R}}$ and $1+i{\mathbb{R}}$.
Finally, an argument based on the fact that zeros and poles must occur at
pairs of inverse points $a$ and $-\overline{a}$ with respect to
$i{\mathbb{R}}$ as well as inverse points $a$ and $-\overline{a}+2$ with
respect to $1+i{\mathbb{R}}$ (cf. the proof of Theorem 2.13) leads to a
contradiction unless $f/g$ is constant.

For (5), suppose that the two intersection points of $C_{1}$ and $C_{2}$ are
$a$ and $b$. By changing the variable to $w=1/(z-a)$ we transform the circles
into straight lines, which meet at $1/(b-a)$, still at an angle $\theta$. The
result for irrational $\theta/\pi$ now follows from Theorem 2.1, applied to a
small disc centred at $1/(b-a)$.

We may similarly construct counterexamples for rational $\theta/\pi$. Let
$C_{1}$ and $C_{2}$ be circles of radius $1/3$ centered at $\pm
1/(3\sqrt{2})$. Then they meet at $\pm a$, where $a=i/(3\sqrt{2})$. The angle
between them is a right angle. The transformation $w=\dfrac{z+a}{z-a}$ sends
the circles to two perpendicular lines, meeting at $0$, from which examples
similar to those in Example 2.2 can be constructed easily. ∎

######  Example 2.17.

There are limitations to the “inverse points” argument above if two circles
are not concentric. For example, let
$C_{1}=\\{z\in{\mathbb{C}}:|z-3/5|=1/5\\}$ and
$C_{2}=\\{z\in{\mathbb{C}}:|z+3/5|=1/5\\}$. Then the points
$z_{\pm}:=\pm\sqrt{8}/5$ are inverse with respect to both circles, and one can
construct rational functions of the form $c\dfrac{z-z_{+}}{z-z_{-}}$ that have
constant absolute values on both circles. However, they do not have the same
absolute values on $C_{1}$ and $C_{2}$, so do not provide a counterexample to
the above result.

Finally, we note that Theorem 2.16 has sequential counterparts, which may be
proved using the methods of Subsection 2.1. We omit the details.

## References

  * [1] I. Chalendar, L. Oger and J.R. Partington, Linear isometries of $\mathop{\rm Hol}({\mathbb{D}})$. Preprint, 2024. https://arxiv.org/abs/2402.14671
  * [2] P.L. Duren, Theory of $H^{p}$ spaces. Pure and Applied Mathematics, Vol. 38 Academic Press, New York–London 1970.
  * [3] P. Jaming, K. Kellay, R. Perez III, Phase retrieval for wide band signals. J. Fourier Anal. Appl. 26 (2020), no. 4, Paper No. 54, 21 pp.
  * [4] H. Kamowitz, The spectra of composition operators on $H^{p}$. J. Functional Analysis 18 (1975), 132–150.
  * [5] J.N. McDonald, Phase retrieval and magnitude retrieval of entire functions. J. Fourier Anal. Appl. 10 (2004), no.3, 259–267.
  * [6] N.K. Nikolski. Operators, functions, and systems: an easy reading. Volume I: Hardy, Hankel, and Toeplitz. Transl. from the French by Andreas Hartmann. English. Vol. 92. Math. Surv. Monogr. Providence, RI: American Mathematical Society (AMS), 2002.
  * [7] V. Pohl, N. Li and H. Boche, Phase retrieval in spaces of analytic functions on the unit disk, IEEE Proc. International Conference on Sampling Theory and Applications (SampTA), 2017.
  * [8] M. Smith, The spectral theory of Toeplitz operators applied to approximation problems in Hilbert spaces. Constr. Approx. 22 (2005), no. 1, 47–65.


# DocuMint: Docstring Generation for Python using Small Language Models

Bibek Poudel*, Adam Cook*, Sekou Traore*, Shelah Ameli*  Min H. Kao Department
of Electrical Engineering and Computer ScienceUniversity of Tennessee,
KnoxvilleTNUSA<EMAIL_ADDRESS>
(2018)

###### Abstract.

Effective communication, specifically through documentation, is the beating
heart of collaboration among contributors in software development. Recent
advancements in Language Models (LMs) have enabled the introduction of a new
type of actor in that ecosystem: LM-powered assistants capable of code
generation, optimization, and maintenance. Our study investigates the efficacy
of Small Language Models (SLMs) in generating high-quality docstrings by
assessing accuracy, conciseness, and clarity, benchmarking performance
quantitatively through mathematical formulas and qualitatively through human
evaluation using Likert scale. Further, we introduce DocuMint, as a large-
scale supervised fine-tuning dataset with $100,000$ samples. In quantitative
experiments, Llama 3 $8$B achieved the best performance across all metrics,
with conciseness and clarity scores of $0.605$ and $64.88$, respectively.
However, under human evaluation, CodeGemma $7$B achieved the highest overall
score with an average of $8.3$ out of $10$ across all metrics. Fine-tuning the
CodeGemma $2$B model using the DocuMint dataset led to significant
improvements in performance across all metrics, with gains of up to $22.5\%$
in conciseness. The fine-tuned model and the dataset can be found in
HuggingFace111https://huggingface.co/documint and the code can be found in the
repository222https://github.com/Docu-Mint/DocuMint.

Docstring generation, LLMs for documentation, Fine-tune LLMs for docstring
generation.

††copyright: acmlicensed††journalyear: 2018††doi: XXXXXXX.XXXXXXX††isbn:
978-1-4503-XXXX-X/18/06††ccs: Software and its engineering Automatic
programming††ccs: Software and its engineering Software prototyping

##  1\. Introduction

Code is at the frontier of Artificial Intelligence (AI). The emergence of
Large Language Models for Code (Code LLMs), including foundation models such
as CodeLlama (Roziere et al., 2023), Code-
Gemma (Google, 2024), and StarCoder (Lozhkov et al., 2024), along with
products like GitHub Copilot (git, 2024) and GhostWriter (rep, 2024), has
paved the way for even more advanced software development tools. The latest
saga in this odyssey is the rise of CodeLLM-enabled AI agents such as Devin
(cog, 2024), Devika (dev, 2024), and OpenDevin (ope, 2024), which can automate
entire software development pipelines, ushering us into an era where software
is co-developed by humans and AI. Already in $2022$, GitHub reported that on
average, $46\%$ of all code written across all programming languages was
assisted by GitHub Copilot (GitHub, 2023). While the formal syntax, limited
vocabulary, and deterministic outcomes in code provide an ideal (structured)
environment for CodeLLMs to thrive (Zhang et al., 2023), at the same time,
their usefulness is challenged by the need to understand context, interpret
natural language instructions, and convey the intent for generated code. As
the CodeLLM landscape evolves and software development increasingly involves
collaboration between human and AI, it becomes increasingly important for both
parties to convey their objective and comprehend the code generated by their
counterparts.

def example_function(param1, param2):
"""
This is an example of a docstring
for a Python function.

Docstrings are enclosed in triple
quotes and appear immediately
after the function definition.
"""

# Function implementation

Figure 1. An example of docstring for a python function. Docstrings serve as
documentation and help developers understand how to use the respective code.

Given the impressive capabilities of LLMs and CodeLLMs, there is growing
interest in smaller, more efficient models called Small Language Models (SLMs)
(Microsoft Research, 2024), such as Phi (Abdin et al., 2024), Gemma (Google,
2024) and smaller variants of Llama (Meta, 2023; AI@Meta, 2024). SLMs are
significantly more cost-effective in terms of training and inference (latency,
memory, throughput, and energy consumption), and they are small enough for
considerations on running locally; about $7$B parameters for consumer-level
GPUs and about $2$B parameters for comsumer CPUs. As language models continue
to shape the landscape of software development, high-quality documentation
becomes increasingly crucial for facilitating effective communication and
collaboration between humans and AI. Given that developers already spend
$58\%$ of their time on comprehension of unfamiliar code (Xia et al., 2017),
writing clear and consistent documentation helps helps reduce this time while
enabling both human developers and AI to convey the intent behind the code and
understand each other’s contributions better. This not only enhances the
developer experience but also improves overall software quality.

Although CodeLLMs are capable of generating docstrings (as docstrings are a
part of their pre-training and fine-tuning data), to date, limited research
has focused on evaluating the quality of the generated docstrings. Existing
benchmarks, such as HumanEval (Chen et al., 2021) and MBPP (Austin et al.,
2021), assess code generation performance, leaving a gap in measuring the
quality of the accompanying documentation. Furthermore, traditional natural
language generation metrics, such as Bilingual Evaluation Understudy (BLEU),
fall short in evaluating docstring quality due to their emphasis on n-gram
overlap between generated and reference sentences, disregarding the semantic
meaning and context. In this work, we address these limitations by introducing
DocuMint, a large-scale dataset and methodology for evaluating the quality of
docstrings generated by SLMs. We make three key contributions:

  * •

First, we breakdown the “quality” assessment of docstrings into three
categories: accuracy, conciseness, and clarity with well defined metric for
each category.

  * •

Second, we benchmark the performance of leading code generation SLMs in
generating docstrings using both the mathematical metrics and human
evaluation.

  * •

Third, we introduce DocuMint, a supervised fine-tuning dataset consisting of
$100,000$ samples.

To the best of our knowledge, DocuMint is the first comprehensive work
designed to evaluate and improve the quality of docstrings generated by SLMs,
providing a large-scale dataset, benchmarks, a fine-tuned model, and an
evaluation methodology.

##  2\. Related Work

There exist two mainstream code-related evaluations for LLMs: code generation
and code comprehension.

Code Generation: The standard code-related evaluation benchmarks for language
models, such as HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), and
SWE-Bench (Jimenez et al., 2023), assess their abilities to solve problems
through generated code. HumanEval presents carefully crafted programming
problems and evaluates whether the generated code solutions pass on hidden
test cases. MBPP tests the ability of LLMs to translate instructions into
functional code. More recently, SWE-Bench tests their abilities to tackle
real-world software issues sourced from GitHub, directly assessing a model’s
proficiency in understanding and resolving software problems.

Code Comprehension: Most of the prior research in this field has focused on
using LLMs to comprehend human-written or AI-generated code, including code
summarization (Cai et al., 2024; Wang et al., 2024) and explanations (Sarsa et
al., 2022). These tasks leverage the abilities of LLMs to reason in a Chain-
of-thought (Wei et al., 2022) or in a step-by-step (Kojima et al., 2022)
manner. Code summarization focuses on generating succinct, human-readable
summaries of code snippets, whereas step-by-step explanations are more
prominent in educational settings (Brusilovsky et al., 2023).

To date, limited research has focused on the evaluation of CodeLLMs in
docstring generation, and prominent datasets for this task are scarce. Early
works on code documentation generation, such as (Barone and Sennrich, 2017),
introduced datasets of parallel code and natural language descriptions of
$150,370$ Python function declarations, bodies, and docstrings scraped from
open-source repositories. However, these efforts were limited by the size and
diversity of the available data, and they were conducted before the emergence
of language models, which questions their relevance. Similarly, line-by-line
comment generation (Hu et al., 2018) and contextual function/method call and
usage information (McBurney and McMillan, 2014) have been explored, but these
works also predate the rise of LLMs. More recently, RepoAgent (Luo et al.,
2024) has demonstrated the potential of LLMs for repository-level code
documentation generation, generating and proactively maintaining high-quality
documentation for entire projects.

While these existing works provide valuable insights, they differ from our
approach, which is to evaluate the abilities of CodeLLMs themselves to
generate documentation alongside the code they produce. We evaluate the
efficacy of language models in documentation generation as a software
development tool, i.e., given a code function, is the CodeLLM able to describe
its core functionality, input/output parameters, and intended usage in an
accurate, concise, and clear manner?

##  3\. Preliminaries

In this section, we introduce the various training phases of a language model
and the tasks that it is trained to perform.

###  3.1. Training

Creating a helpful Language Model (LM) involves three essential steps: pre-
training, fine-tuning, and alignment.

Pre-training: Pre-training involves training a LM on vast amounts of unlabeled
text data using self-supervised learning techniques such as masked language
modeling or next token prediction. This process allows the model to capture
the intrinsic patterns and statistical regularities present in natural
language. Pre-training is computationally intensive and requires clusters of
GPUs to process the internet-scale data efficiently, resulting in a ”base
model” that serves as a foundation for further fine-tuning.

Fine-tuning: Pre-training alone does not enable the model to understand or
follow specific instructions. The utility of LMs is significantly improved by
fine-tuning and depending on the nature of the task, fine tuning could be
Supervised Fine Tuning (SFT) or Instruction Tuning (IT). SFT adapts a pre-
trained base model to perform specific downstream tasks by training it on
labeled data relevant to the task. Whereas IT is employed to turn the base
model into a useful assistant capable of understanding and responding to user
instructions expressed in natural language.

Alignment: Also known as human preference fine-tuning, this step aligns the
behavior of an LM with human preferences, enhancing its friendliness,
helpfulness, and safety. This process involves collecting human feedback on
the model’s outputs and using this feedback as a reward signal to guide the
model’s behavior. By optimizing the model to generate outputs that align with
human preferences, this fine-tuning helps to mitigate undesirable behaviors
and ensures more socially appropriate and beneficial responses.

###  3.2. Tasks

LMs can be useful for various tasks, such as text/code infilling and
completion.

Infilling: is a specialized task designed for code-generation models, where
the objective is to generate code snippets or comments that best fit within a
given prefix and suffix. This task is particularly relevant for code
assistants, which are trained to provide code suggestions based on the
surrounding context at the current cursor position. Infilling models learn to
capture the syntactic and semantic patterns present in code, allowing them to
generate coherent and contextually appropriate code fragments.

Completion: refers to the fundamental capability of LMs to generate fluent and
coherent text by predicting the next token in a sequence based on the
preceding context. Completion allows LLMs to generate human-like text that
follows the style, tone, and subject matter of the given prompt. This is the
most common task of a LM.

##  4\. Methodology

In this section, we provide an overview of the selected state-of-the-art
models, the data used for inference and bench-marking them, the data
extraction process for fine-tuning, and the applied fine-tuning technique.

###  4.1. Selected Models

Models were selected from the EvalPlus Leaderboard (Liu et al., 2023) at full
precision, with their rank on the leaderboard as of April $25$,$2024$ shown in
Table1.

Model | Rank
---|---
DeepSeek Coder $6.7$B Instruct | $14$
Meta Llama3 $8$B Instruct | $38$
CodeGemma $7$B Instruct | $43$
StarCoder2 $7$B | $66$
CodeGemma $2$B | $83$

Table 1. Selected models and their rank.

Since the selected models are CodeLLMs, use instruct variants whenever
available.

###  4.2. Benchmarking Data

For benchmarking the selected models, we extract seven python functions from
three widely established NLP datasets (Chang et al., 2023; Zhao et al., 2023;
Roziere et al., 2023; Nijkamp et al., 2022; Li et al., 2022; Wei et al., 2022;
Kaddour et al., 2023) and perform model inference.

The Mostly Basic Python Problems MBPP dataset (Austin et al., 2021) consists
of around $1,000$ crowd-sourced Python programming problems, designed to be
solvable by entry level programmers, covering programming fundamentals,
standard library functionality, and so on. Each problem consists of a task
description, code solution and $3$ automated test cases. The dataset is split
into a train and test set, we select functions from the test set.

The Hand-Written Evaluation Set HumanEval dataset (Chen et al., 2021) consists
of $164$ hand written problems with a function signature, a docstring
describing the task, reference solution, and tests. These problems are our
medium level functions, they contain functions and problems that are notably
more complex than the MBPP dataset but less algorithmic intensive when
compared to our selected APPS functions.

The Automated Programming Progress Standard APPS dataset (Hendrycks et al.,
2021) comprises 10,000 coding examples, evenly distributed between train and
test sets. These examples were sourced from various open-access coding
websites like Codeforces and Kattis. Each problem within the dataset includes
a question description, solutions, input/output details, difficulty level, and
source URL, with problems categorized into Introductory (simple and easy),
Interview (technical interview questions), and Competition Level (advanced
programming competitions).

We focused on functions categorized under the ’interview level’ section of the
test set. From each problem, we extracted solutions contained within functions
defined by the ’def’ keyword, excluding class objects from our inference
dataset for analysis.

###  4.3. Fine Tuning Data

As the human annotation process is expensive and relatively slow, we leverage
existing code from the Free and open-source software (FOSS) ecosystem. We
utilize World of Code (WoC) (Ma et al., 2019, 2021), an efficient and flexible
project analysis framework that provides an abstracted interface to the
intricacies of this ecosystem, enabling a reliable and straightforward
approach to research. Our goal is to ensure quantity, quality, and diversity
of data for fine-tuning.

We query the repositories in WoC and filter them based on metrics such as
number of contributors $(>50)$, commits $(>5k)$, stars $(>35k)$, and forks
$(>10k)$. This ensures that we focus on well-established and actively
maintained projects. To extract functions from the filtered repositories, we
employ a parser that utilizes the GitPython and abstract syntax tree modules
to navigate through repository files and identify function definitions. The
parser extracts $100,000$ Python functions along with their respective
docstrings. The data was formatted into a JSON file with the Alpaca
instruction format (Taori et al., 2023) shown below.

{
"instruction": "<function definition>",
"response": "<ideal docstring>"
}

###  4.4. Fine Tuning

The full model fine-tuning for language models, even for modest sizes, is
memory and compute-intensive; hence, we resort to parameter-efficient fine-
tuning, updating only a small subset of model parameters, among which Low-Rank
Adaptation (LoRA)(Hu et al., 2021) is a popular technique. We perform
supervised instruction fine-tuning on the Gemma-2B base model, which has been
trained to predict the next token on internet text without any instructions,
using LoRA. This fine-tuning process involves using examples that demonstrate
how the model should respond to specific instructions. The total fine-tuning
time was $48$ GPU hours, with $78,446,592$ LoRA parameters at full precision
($32$ bit) and $185,040,896$ total training tokens. The full set of parameters
used in fine-tuning are provided in Table2.

Category | Parameter | Value
---|---|---
CodeGemma | vocabulary size | $256000$
max pos. embeddings | $8192$
Config | num_hidden_layers | $18$
| num_heads | $8$
Fine | max_seq_len | $128$
LoRA rank | $64$
LoRA alpha | $128$
LoRA dropout | $0.1$
Tuning | batch size | $8$
| grad accumulation steps | $16$
| epochs | $4$
| optimizer |  $8$ bit adam
| Learning rate (initial) |  $2$e$-4$

Table 2. Detailed configuration of the CodeGemma model and hyper-parameters
used in fine-tuning.

##  5\. Experiments

In this section, we define our experiments, describe the metrics used to
assess the generated docstrings, provide summary statistics of the human
participants, and detail the experiment setup.

Our research objectives are contained in the three experiments described
below.

Experiment $1$ quantitatively evaluates the docstrings generated by SLMs using
the three metrics: Accuracy, Conciseness, and Clarity, as described in Section
5.1.

Experiment $2$ qualitatively evaluates a subset of the docstrings generated by
SLMs using a human questionnaire where participants with prior Python
programming experience score the Accuracy, Conciseness, and Clarity of
generated docstrings on a Likert scale.

Experiment $3$ evaluates the docstrings generated by a SLM that has been fine-
tuned on data extracted from World of Code and quantitatively compares
(similar to Experiment $1$) the performance against the original model.

Model | Accuracy | Conciseness | Clarity
---|---|---|---
CodeGemma 7B | $0.609$ | $0.569$ | $76.49$
DeepSeek Coder $6.7$B | $\bf{0.679}$ | $0.734$ | $\bf{64.44}$
Llama3 8B | $0.668$ | $\bf{0.605}$ | $\bf{64.88}$
StarCoder$2$ $7$B | $0.626$ | $0.510$ | $69.74$

Table 3. Experiment 1 results i.e., comparative evaluation of various CodeLLMs
in generating docstring as measured by Accuracy, Conciseness, and Clarity.
Instruct variants used for CodeGemma, DeepSeek, and Llama3. Overall, Llama 3
performs the best across all three metrics. DeepSeek achieves the best
Accuracy score, CodeGemma achieves the best Conciseness score, and both
DeepSeek and Llama 3 share the best Clarity score. System Prompt You are a
helpful AI assistant that specializes in generating high-quality docstrings
for Python code functions. Your task is to create docstrings that are:

Accurate: Cover functionality, parameters, return values, and exceptions.

Concise: Brief and to the point, focusing on essential information.

Clear: Use simple language and avoid ambiguity.

Generate docstring in this format: """<generated docstring>""". Figure 2.
System prompt that is attached before the function definition when querying
the model for inference.

###  5.1. Experiment Metrics

Classic NLP metrics such as Bilingual Evaluation Understudy (BLEU) are often
used to evaluate machine translation systems by comparing the machine-
generated translations to human-generated reference translations. However,
they are not reliable in the context of code (Evtikhiev et al., 2023). Two
dramatically different code snippets can be semantically equivalent, and since
the same vocabulary is often used with docstrings attached to the code, we
refrain from using BLEU. We consider the primary concern when it comes to LLM-
generated docstring is accuracy (correctness); to further get a comprehensive
measurement, we consider conciseness, and clarity.

Accuracy: measures the correctness in the description of the function logic
and coverage of the generated docstring on associated code elements such as
input and output variables. We measure accuracy using BERTScore (Zhang et al.,
2019) given by

$\text{Accuracy}=\frac{v_{g}\cdot
v_{e}}{\sqrt{v_{g}^{2}}\cdot\sqrt{v_{e}^{2}}},$

where $v_{g}$ and $v_{e}$ are the Bidirectional Encoder Representations from
Transformers (Devlin et al., 2018) encodings of the SLM generated docstring
and an “expert” docstring generate by the Claude-3 Opus (Anthropic, 2024) LLM,
respectively. A higher accuracy score indicates a greater similarity between
the generated docstring and expert docstring and vice versa.

Conciseness: measures the ability of the generated docstring to convey
information succinctly without unnecessary verbosity. We measure conciseness
using a compression ratio defined as:

$\text{Conciseness}=\frac{\text{Text}_{C}}{\text{Text}_{O}},$

where ${\text{Text}_{C}}$ and ${\text{Text}_{O}}$ are the sizes of the
compressed text and the original text respectively. The compression ratio is
bounded between $0$ and $1$ with a lower ratio indicating more conciseness. An
ideal score is between $0.5$ and $0.6$ i.e., if the score is too low, it
indicates that the generated docstrings are extremely short.

Clarity: measures the readability of the generated docstring i.e., uses simple
language and is unambiguous. We measure clarity using the Flesch-Kincaid
reading score:

$\text{Clarity}=206.835-(1.015*\frac{w}{l})-(84.6*\frac{s}{w})),$

where w is the number of words generated, l is the number of sentences or
lines, and s is the number of syllables. A score between $70-100$ indicates
that the text is very easy to read, at $50-70$ text slowly becomes more
difficult to read (high school level), and $0-50$ is regarded as difficult to
read and best understood by university graduates. Ideally, for docstrings a
score of $50-70$ is desired.

###  5.2. Summary statistics of the Human Population

We selected participants to perform a ‘qualitative’ measure of the docstrings
generated by each of the four Small Language Models (SLMs). Each participant
evaluated five questions from the inference dataset, with three persons per
SLM. A sample of the questionnaire is included in appendix A.1 for reference.

Figure 3. Survey participants’ frequency in reading docstrings. About 42$\%$
read docstrings several times a week, $\sim$ 17$\%$ read docstrings once a
week and several times a month each, while a quarter of them reads docstring
once a month.

The selected population of this study comprised of human programmers, with a
median age of $25.5$ years. The cumulative years of Python Programming
experience is 47 years, averaging 4 years per participant. We had a diverse
group of 8 males and 4 females. Figure 3 highlights the habits of the
population with reading and engaging with docstrings.

###  5.3. Experiment Setup

The data and model exploration phase was conducted using Lightning AI with
A-$10$G GPUs. For inference by zero-shot prompting in experiment $1$, we
employed an Intel $14900$K CPU, an Nvidia RTX-A$5000$ GPU, and $32$GB RAM.
Fine-tuning experiments were performed using an Intel $12900$K CPU, an Nvidia
RTX-$3090$ GPU, and $64$GB RAM. All models used in this study were obtained
through HuggingFace.

##  6\. Experiment Results

In this section, we present the results of our quantitative benchmarks,
qualitative human questionnaires, and fine-tuning experiments on the selected
SLMs.

###  6.1. Experiment 1 Results

Table 3 shows the quantitative measurements of accuracy, conciseness, and
clarity for the docstrings generated by SLMs, all of which have roughly 7
billion parameters. Accuracy scores were led by DeepSeek at $0.679$, with
Llama 3 in second place at $0.668$. This indicates that the docstrings
generated by DeepSeek and Llama 3 were the most similar to the control model
(Claude 3 Opus, whose parameter count is in the hundreds of billions). On the
other hand, CodeGemma scored the lowest accuracy at $0.609$, indicating the
most dissimilarity with the control model.

For conciseness, CodeGemma had the best score at $0.569$, while StarCoder2,
with a score of $0.510$, generated extremely short docstrings that may not
convey enough information. Surprisingly, DeepSeek Coder exceeded the ideal
threshold of $0.5-0.6$ with a score of $0.734$, implying that the generated
docstrings were verbose. Further, DeepSeek, Llama3, and StarCoder2 have
clarity scores within the desired range of $50-70$, with StarCoder2 at the
upper threshold at $69.74$. However, CodeGemma exceeds the ideal threshold
with a score of $76.49$. This means that while CodeGemma is deemed ”easy to
read,” it uses much simpler language and syntax and may not be considered of
professional quality.

Figure 4. The average weighted scores from the human questionnaire. Instruct
variants used for CodeGemma, DeepSeek, and Llama3. CodeGemma performs the best
overall surpassing all other models in terms of accuracy and concisenes.
Llama3 achieves the best clarity score. StarCoder2 was the second-best-
performing model from a human perspective.

###  6.2. Experiment 2 Results

Figure 4 shows the average accuracy, conciseness, clarity, and overall scores
from the human questionnaires. For accuracy, CodeGemma achieves the overall
best performance with 8.2 out of 10. Surprisingly, Starcoder2 performed well
with an accuracy of 7.7 from a human perspective despite having the worst
results in the math benchmarks (Section 3). Llama3 and DeepSeek Coder have the
lowest accuracy scores, which varies from the results achieved with the
quantitative benchmarks. This suggests that humans consider CodeGemma and
DeepSeek Coder as more reliable in generating correct docstrings compared to
the other two models.

For conciseness, CodeGemma takes the lead with a high score of 9.2 out of 10,
indicating its ability to generate more succinct and compact responses, this
is consistent with the quantitative metric where CodeGemma also had the best
conciseness score. Llama3 and Starcoder2 show similar levels of conciseness,
while DeepSeek Coder generates the least concise score (5.8) among the four
SLMs. Whereas for Clarity, humans rank Llama3 as being the clearest with a
clarity score of $\sim$ 9, with CodeGemma coming closely behind with 7.8
clarity score demonstrating comparable performance. DeepSeek Coder and
Starcoder2 exhibit lower clarity scores, suggesting that humans perceived
their generated responses to be less understandable compared to the top two
models.

Overall, CodeGemma and Starcoder2 stand out as the best-performing models,
with CodeGemma having an overall score of 7.9 on the Likert scale. These two
models consistently score higher across the three metrics compared to Llama3
and DeepSeek Coder. The results indicate that CodeGemma and Starcoder2 are
more effective in generating accurate, concise, and clear responses.

###  6.3. Experiment 3 Results

Table 4 presents the comparative evaluations between the base CodeGemma 2B
model and its fine-tuned version. The fine-tuned model shows notable
improvements across all metrics: accuracy increased by $12.7\%$, conciseness
by $22.5\%$, and clarity by three reading levels. Despite the fine-tuned model
being a 2B parameter model, it achieves competitive performance compared to
the larger Llama3 $8$B model shown in Table 3. The fine-tuned CodeGemma 2B
model has $14\%$ lower accuracy than Llama3 $8$B, but its conciseness falls
within a desirable range, and its clarity is only slightly below the
threshold. These results validate the effectiveness of our fine-tuning process
using the data obtained from World of Code. The fine-tuning loss curve is
shown in Figure 5.

Model | Accuracy | Conciseness | Clarity
---|---|---|---
CodeGemma $2$B | $0.516$ | $0.425$ | $91.69$
CodeGemma $2$B | $\bf{0.582}$ | $\bf{0.521}$ | $\bf{58.75}$
Fine-tuned |  |  |

Table 4. Experiment 3 results i.e., comparative evaluations between CodeGemma
2B base model and it’s fine-tuned version. The fine-tuned model displays
improvements in all metrics compared to the original model.
Figure 5. The loss curve during fine-tuning of the CodeGemma-$2$B model. The
loss decreases sharply during the first epoch after which the curve saturates
with marginal decrements per epoch.

##  7\. Conclusion and Future Work

In this project, we introduced DocuMint, a dataset and methodology for
evaluating the ability of Small Language Models (SLMs) to generate useful
Python docstrings. Our experiments indicate that SLMs are capable of
generating docstrings that are accurate, concise, and clear to a reasonable
degree. Some of the results are surprising. First, Llama3 8B, a general-
purpose SLM, performed the best overall in calculated metrics, despite the
fact that its training data only contains a small fraction of code samples
compared to code-specific models (although the exact number is not available,
it was reported that Llama3 had four times more code samples than the dataset
used to train Llama 2, which likely improved its abilities). Second, CodeGemma
7B was perceived as the most accurate by humans, despite having lower accuracy
scores in calculated metrics. Further, to the performance of SLMs in
generating docstrings, we fine-tuned CodeGemma 2B model on data extracted from
open source repositories using World of Code which leads to improvements
across all metrics (upto $35\%$ improvement in docstring clarity).

In the future, there are many interesting directions to extend this work.
Expanding DocuMint with more diverse code samples and human-annotated
reference docstrings would increase the quality of fine-tuning dataset.
Developing better automated metrics to measure docstring quality that better
align with human judgement could be another crucial area for future research.
To facilitate future efforts, we have release the DocuMint dataset, the fine-
tuned CodeGemma $2$B model, and the code we used in this work in public
repositories on GitHub and HuggingFace.

##  8\. Reflections and Lessons Learned

Insights into the FOSS Ecosystem: The process of analyzing the Free and Open
Source Software (FOSS) ecosystem through World of Code provided some crucial
insights. Initially, we encountered a significant reduction in usable
data—from 148,000 extracted functions to only 66,000 actually usable-due to
syntactic errors that affected the correct parsing by the AST module. This
outlines the challenge that the most popular projects are not always the best
written.

Subjectivity of Docstring Quality: Docstring quality in projects varies widely
and tends to be subjective, based on what developers deem important to
document. For our fine-tuning data, the goal was to find an ideal balance in
docstring clarity, conciseness, and accuracy. However, there is no single
strategy that helps us achieve this across repositories.

Challenges Encountered During Sampling: The sampling process itself presented
numerous challenges:

  * •

Syntactic errors in repositories hindered AST parsing.

  * •

Difficulty in identifying Python-specific files in multi-language
repositories.

  * •

Computational and time expenses associated with parsing and analyzing large
codebases.

  * •

Discrepancies in repository sizes posed challenges in maintaining uniformity
in sampling.

  * •

Subjective differences in the overall quality of the docstrings.

  * •

Difficulty in avoiding repetitions in such a large dataset (many
implementations of the same algorithms might have the same docstrings).

  * •

Finding a sufficiently diverse set of data.

Further, the human questionnaire survey that we conducted for Experiment $2$
also provided some qualitative insights behind the numerical scores. While the
participants had praises for the SLMs output, they also highlighted issues
that reduced the quality of the generated docstrings. The most common feedback
for all models are listed below:

  * •

Inconsistencies in spelling, description, and examples.

  * •

Incorrect references to arguments and return type.

  * •

Raises errors and exceptions not present in code.

  * •

Weak explanations for nested loops.

  * •

Too long Docstrings.

##  9\. Contributions

All authors contributed equally to the work.

Bibek Poudel: Researched model precision and parameter efficient fine tuning.
Introduced HuggingFace and Lightning AI. Secured required compute. Led the
fine-tuning effort and technical report writing. Assisted in model inference
and evaluation metrics setup.

Adam Cook: Researched available Small Language Models to implement for
generating docstrings; developed a Python file that created an output text
file containing docstrings generated from an input SLM and JSON data file;
calculated the metrics depicted in Table 2.

Sekou Traore: Educated the group on tokenization and data related tasks.
Developed a data strategy for fine tuning, familiarized with World of Code and
performed data operations such as scraping, filtering, and json formatting as
required for fine-tuning.

Shelah Ameli: Initial research on available compute within the ISAAC cluster.
Studied standard datasets such as MBPP, HumanEval, and APPS; scraped and
filtered a curated dataset for inference. Provided a JSON structure file for
setting up inference experiment. Facilitated and compiled the human benchmark
experiments.

###### Acknowledgements.

The authors would like to thank Dr. Audris Mockus for his guidance on the
project and help with World of Code. The authors would also like to thank the
Fluidic City Lab for providing the compute resources.

## References

  * (1)
  * dev (2024) 2024\.  devika.  https://github.com/stitionai/devika.
  * git (2024) 2024\.  GitHub Copilot.  https://github.com/features/copilot.
  * cog (2024) 2024\.  Introducing Devin.  https://www.cognition-labs.com/introducing-devin.
  * ope (2024) 2024\.  OpenDevin.  https://github.com/OpenDevin/OpenDevin.
  * rep (2024) 2024\.  Replit AI.  https://replit.com/ai.
  * Abdin et al. (2024) Marah Abdin, Sam Ade Jacobs, and et al. 2024.  Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone.  _arXiv preprint arXiv:2404.14219_ (2024).  https://arxiv.org/abs/2404.14219 Accessed 2024 April.
  * AI@Meta (2024) AI@Meta. 2024.  Llama 3 Model Card.  (2024).  https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md
  * Anthropic (2024) Anthropic. 2024.  Introducing the next generation of Claude.  https://www.anthropic.com/news/claude-3-family Accessed: 2024-05-13.
  * Austin et al. (2021) Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021\.  Program synthesis with large language models.  _arXiv preprint arXiv:2108.07732_ (2021).
  * Barone and Sennrich (2017) Antonio Valerio Miceli Barone and Rico Sennrich. 2017.  A parallel corpus of python functions and documentation strings for automated code documentation and code generation.  _arXiv preprint arXiv:1707.02275_ (2017).
  * Brusilovsky et al. (2023) Peter Brusilovsky, Arun-Balajiee Lekshmi-Narayanan, Priti Oli, Jeevan Chapagain, Mohammad Hassany, Rabin Banjade, and Vasile Rus. 2023.  Explaining code examples in introductory programming courses: Llm vs humans.  _arXiv preprint arXiv:2403.05538_ (2023).
  * Cai et al. (2024) Yufan Cai, Yun Lin, Chenyan Liu, Jinglian Wu, Yifan Zhang, Yiming Liu, Yeyun Gong, and Jin Song Dong. 2024.  On-the-Fly Adapting Code Summarization on Trainable Cost-Effective Language Models.  _Advances in Neural Information Processing Systems_ 36 (2024).
  * Chang et al. (2023) Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2023\.  A survey on evaluation of large language models.  _ACM Transactions on Intelligent Systems and Technology_ (2023).
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021\.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_ (2021).
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Evtikhiev et al. (2023) Mikhail Evtikhiev, Egor Bogomolov, Yaroslav Sokolov, and Timofey Bryksin. 2023.  Out of the bleu: how should we assess quality of the code generation models?  _Journal of Systems and Software_ 203 (2023), 111741.
  * GitHub (2023) GitHub. 2023.  GitHub Copilot for Business is now available.  https://github.blog/2023-02-14-github-copilot-for-business-is-now-available/.
  * Google (2024) Google. 2024.  CodeGemma: Open Code Models Based on Gemma.  https://storage.googleapis.com/deepmind-media/gemma/codegemma_report.pdf.
  * Google (2024) Google. 2024.  Gemma: Open Models for Developers.  https://blog.google/technology/developers/gemma-open-models/.
  * Hendrycks et al. (2021) Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. 2021\.  Measuring coding challenge competence with apps.  _arXiv preprint arXiv:2105.09938_ (2021).
  * Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_ (2021).
  * Hu et al. (2018) Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. 2018.  Deep code comment generation. In _Proceedings of the 26th conference on program comprehension_. 200–210.
  * Jimenez et al. (2023) Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023.  Swe-bench: Can language models resolve real-world github issues?  _arXiv preprint arXiv:2310.06770_ (2023).
  * Kaddour et al. (2023) Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. 2023.  Challenges and applications of large language models.  _arXiv preprint arXiv:2307.10169_ (2023).
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  _Advances in neural information processing systems_ 35 (2022), 22199–22213.
  * Li et al. (2022) Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022\.  Competition-level code generation with alphacode.  _Science_ 378, 6624 (2022), 1092–1097.
  * Liu et al. (2023) Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023.  Is your code generated by ChatGPT really correct.  _Rigorous evaluation of large language models for code generation. CoRR, abs/2305.01210_ (2023).
  * Lozhkov et al. (2024) Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. 2024\.  StarCoder 2 and The Stack v2: The Next Generation.  _arXiv preprint arXiv:2402.19173_ (2024).
  * Luo et al. (2024) Qinyu Luo, Yining Ye, Shihao Liang, Zhong Zhang, Yujia Qin, Yaxi Lu, Yesai Wu, Xin Cong, Yankai Lin, Yingli Zhang, et al. 2024\.  RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generation.  _arXiv preprint arXiv:2402.16667_ (2024).
  * Ma et al. (2019) Yuxing Ma, Chris Bogart, Sadika Amreen, Russell Zaretzki, and Audris Mockus. 2019.  World of code: an infrastructure for mining the universe of open source VCS data. In _2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR)_. IEEE, 143–154.
  * Ma et al. (2021) Yuxing Ma, Tapajit Dey, Chris Bogart, Sadika Amreen, Marat Valiev, Adam Tutko, David Kennard, Russell Zaretzki, and Audris Mockus. 2021.  World of code: enabling a research workflow for mining and analyzing the universe of open source VCS data.  _Empirical Software Engineering_ 26 (2021), 1–42.
  * McBurney and McMillan (2014) Paul W McBurney and Collin McMillan. 2014.  Automatic documentation generation via source code summarization of method context. In _Proceedings of the 22nd International Conference on Program Comprehension_. 279–290.
  * Meta (2023) Meta. 2023.  LLaMA.  https://llama.meta.com.
  * Microsoft Research (2024) Microsoft Research. 2024.  Phi-2: The Surprising Power of Small Language Models.  https://www.microsoft.com/en-us/research/blog/
phi-2-the-surprising-power-of-
small-language-models/.

  * Nijkamp et al. (2022) Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022.  Codegen: An open large language model for code with multi-turn program synthesis.  _arXiv preprint arXiv:2203.13474_ (2022).
  * Roziere et al. (2023) Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023\.  Code llama: Open foundation models for code.  _arXiv preprint arXiv:2308.12950_ (2023).
  * Sarsa et al. (2022) Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022.  Automatic generation of programming exercises and code explanations using large language models. In _Proceedings of the 2022 ACM Conference on International Computing Education Research-Volume 1_. 27–43.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023.  Alpaca: A strong, replicable instruction-following model.  _Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/alpaca. html_ 3, 6 (2023), 7.
  * Wang et al. (2024) Yuki Wang, Gonzalo Gonzalez-Pumariega, Yash Sharma, and Sanjiban Choudhury. 2024.  Demo2code: From summarizing demonstrations to synthesizing code via extended chain-of-thought.  _Advances in Neural Information Processing Systems_ 36 (2024).
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022\.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in neural information processing systems_ 35 (2022), 24824–24837.
  * Xia et al. (2017) Xin Xia, Lingfeng Bao, David Lo, Zhenchang Xing, Ahmed E Hassan, and Shanping Li. 2017.  Measuring program comprehension: A large-scale field study with professionals.  _IEEE Transactions on Software Engineering_ 44, 10 (2017), 951–976.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019.  Bertscore: Evaluating text generation with bert.  _arXiv preprint arXiv:1904.09675_ (2019).
  * Zhang et al. (2023) Ziyin Zhang, Chaoyu Chen, Bingchang Liu, Cong Liao, Zi Gong, Hang Yu, Jianguo Li, and Rui Wang. 2023.  Unifying the perspectives of nlp and software engineering: A survey on language models for code.  _arXiv preprint arXiv:2311.07989_ (2023).
  * Zhao et al. (2023) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023\.  A survey of large language models.  _arXiv preprint arXiv:2303.18223_ (2023).

##  Appendix A Appendix

###  A.1. Experiment 2 Human Questionnaire Sample

See pages 1 of figures/appendix.pdf See pages 2-3 of figures/appendix.pdf