# TOI-4641b: An Aligned Warm Jupiter Orbiting a Bright (V=7.5) Rapidly
Rotating F-star

Allyson Bieryla*  |,1,2 George Zhou*  |,2 Juliana García-Mejía*  |,1,3,4 Tyler
R. Fairnington*  |,1 David W. Latham*  |,1 Brad Carter*  |,2 Jiayin Dong*
|,5,6 Chelsea X. Huang*  |,2 Simon J. Murphy*  |,2 Avi Shporer*  |,4 Karen A.
Collins*  |,1 Samuel N. Quinn*  |,1 Mark E. Everett*  |,7 Lars A. Buchhave*
|,8 René Tronsgaard*  |,8 David Charbonneau*  |,1 Marshall C. Johnson*  |,9
Gilbert A. Esquerdo*  |,1 Michael Calkins,1 Perry Berlind,1 Jon M. Jenkins*
|,10 George R. Ricker*  |,4 Sara Seager*  |,4,11,12 Joshua N. Winn*  |,13
Thomas Barclay*  |,14 Ismael Mireles*  |,15 Martin Paegert*  |,1 and Joseph D.
Twicken*  |16

1Center for Astrophysics | Harvard & Smithsonian, 60 Garden Street, Cambridge, MA 02138, USA
2University of Southern Queensland, Centre for Astrophysics, West Street,
Toowoomba, QLD 4350 Australia
351 Pegasi b Fellow
4Department of Physics and Kavli Institute for Astrophysics and Space
Research, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
5Flatiron Research Fellow
6Center for Computational Astrophysics, Flatiron Institute, 162 Fifth Avenue,
New York, NY 10010, USA
7NSF’s National Optical-Infrared Astronomy Research Laboratory, 950 N. Cherry
Ave., Tucson, AZ 85719, USA
8DTU Space, Technical University of Denmark, Elektrovej 328, DK-2800 Kgs.
Lyngby, Denmark
9Department of Astronomy, The Ohio State University, 4055 McPherson
Laboratory, 140 West 18th Ave., Columbus, OH 43210 USA
10NASA Ames Research Center, Moffett Field, CA 94035, USA
11Department of Earth, Atmospheric and Planetary Sciences, Massachusetts
Institute of Technology, 77 Massachusetts Ave, Cambridge, MA 02139, USA
12Department of Aeronautics and Astronautics, Massachusetts Institute of
Technology, 77 Massachusetts Avenue, Cambridge, MA 02139, USA
13Department of Astrophysical Sciences, Princeton University, 4 Ivy Lane,
Princeton, NJ 08544, USA
14NASA Goddard Space Flight Center, 8800 Greenbelt Road, Greenbelt, MD 20771,
USA
15Department of Physics and Astronomy, University of New Mexico, 210 Yale Blvd
NE, Albuquerque, NM 87106, USA
16SETI Institute, Mountain View, CA 94043, USA
E-mail<EMAIL_ADDRESS>
(Accepted 2023 December 05. Received 2023 December 05; in original form 2023
September 17)

###### Abstract

We report the discovery of TOI-4641b, a warm Jupiter transiting a rapidly
rotating F-type star with a stellar effective temperature of 6560 K. The
planet has a radius of 0.73 $R_{\rm Jup}$, a mass smaller than 3.87 $M_{\rm
Jup}$ $(3\sigma)$, and a period of 22.09 days. It is orbiting a bright star
(V=7.5 mag) on a circular orbit with a radius and mass of 1.73 $R_{\odot}$ and
1.41 $M_{\odot}$. Follow-up ground-based photometry was obtained using the
Tierras Observatory. Two transits were also observed with the Tillinghast
Reflector Echelle Spectrograph (TRES), revealing the star to have a low
projected spin-orbit angle ($\lambda$=$1.41^{+0.76}_{-0.76}$$\degr$). Such
obliquity measurements for stars with warm Jupiters are relatively few, and
may shed light on the formation of warm Jupiters. Among the known planets
orbiting hot and rapidly-rotating stars, TOI-4641b is one of the longest-
period planets to be thoroughly characterized. Unlike hot Jupiters around hot
stars which are more often misaligned, the warm Jupiter TOI-4641b is found in
a well-aligned orbit. Future exploration of this parameter space can add one
more dimension to the star-planet orbital obliquity distribution that has been
well-sampled for hot Jupiters.

###### keywords:

exoplanets - techniques: radial velocities - techniques: spectroscopic -
techniques: photometric - methods: observational

††pubyear: 2023††pagerange: TOI-4641b: An Aligned Warm Jupiter Orbiting a
Bright (V=7.5) Rapidly Rotating F-star–TOI-4641b: An Aligned Warm Jupiter
Orbiting a Bright (V=7.5) Rapidly Rotating F-star

##  1 Introduction

Many hot Jupiters are thought to have formed at least a few astronomical units
away from their stars and migrated inward via dynamical interactions. One
possible migration mechanism is high-eccentricity tidal migration, whereby
their orbits gradually circularize with each periastron passage due to
dissipative tidal interactions with their host stars (e.g. Rasio & Ford,
1996). They may instead have formed in-situ (e.g. Batygin, 2012) or undergone
disk-driven migration soon after their formation (e.g. Lin et al., 1996). Gas
giants in somewhat longer period orbits, where tidal effects are negligible,
provide useful laboratories for the study of the migration processes that
produce hot Jupiters (Dawson & Johnson, 2018).

Spin-orbit angle, the angle between the orbital normal and the spin-axis of
the host star, serves as a fossil record of past dynamical interactions
experienced by the system. In particular, warm Jupiters, planets with
$a/R_{\star}>10$ and $R_{p}>8\,R_{e}$, offer an opportunity to study the
primordial obliquities of their host stars (Albrecht et al., 2022) without
having to account for planet-star tidal interactions that may have modified
the orbital architecture.

There exists clear trends in the hot Jupiter spin-orbit obliquity distribution
as a function of stellar temperature (e.g. Albrecht et al., 2012). The
dependence between the observed obliquity distribution and stellar temperature
for longer period planets is less clear due to observational biases that make
such observations more difficult. Close-in giant planets around cool stars
($T_{\rm eff}$ $<6250$ K) are generally observed to have well-aligned orbits
(Albrecht et al., 2022). In contrast, more distantly orbiting giant planets
about cool stars with $a/R_{\star}>10$ show a wider range of spin-orbit
angles, as in systems like WASP-8 (Queloz et al., 2010; Bourrier et al.,
2017), Kepler-420 (Santerne et al., 2014), and HD 80606 (Winn et al., 2009;
Pont et al., 2009). Such a dependence is expected if the close-in planet
obliquity distribution is strongly shaped by planet-star tidal interactions.

Short-period Jovian planets around early-type stars exhibit a wide
distribution of orbital obliquities. Few long-period planets around hot stars
have had their spin-orbit angles mapped. Kepler-448b is the only Jovian planet
around an early-type star with an spin-orbit angle measured spectroscopically
found to reside in a well-aligned orbit (Bourrier et al., 2015; Johnson et
al., 2017).

The lack of well characterized long period planets around early-type stars
prohibits informative tests on the mechanisms thought to induce misalignments
in planet orbits. Early-type stars with radiative envelopes experience weaker
planet-star tidal interactions and offer the opportunity of exploring the
primordial period obliquity relationship for giant planets. Each mechanism
that induces planet-star misalignments have their own expected dependencies on
orbital distance. Warps in the protoplanet disk lead to preferentially longer
period planets being found in misaligned orbits (Heller, 1993; Wijnen et al.,
2017). The spin-axis of stars hotter than the Kraft break (Kraft, 1967) may
also evolve over time by itself. Gravity-wave instabilities induced at the
radiative-convective boundary of early-type stars may result in their outer
envelopes changing in spin-axis over time (Rogers et al., 2012), leading to a
wide range of spin-orbit obliquites for planets around such stars. Such
instabilities are not dependent on the planetary system, and as such no
period-spin-orbit obliquity dependencies should be expected.

In this paper we report the planetary confirmation of TOI-4641b, a warm
Jupiter on a 22-day orbit that is well aligned with the equatorial plane of a
rapidly rotating F-star. In Section 2, we describe the photometric data from
_TESS_ and the Tierras Observatory, high-resolution speckle imaging, and TRES
spectroscopic observations to measure the stellar obliquity. Section 3
describes the global modeling of the system and Section 4 describes the
stellar variability. We conclude with a discussion in Section 5.

##  2 Observations

###  2.1 Photometric Observations

####  2.1.1 TESS Photometry

The Transiting Exoplanet Survey Satellite (TESS; Ricker et al. 2015) is an
all-sky survey searching for transiting exoplanets around nearby bright host
stars. The satellite uses four cameras to stare at 24 $\times$ 96 degree
sectors of the sky for approximately 27 days at a time. TOI-4641 (TIC
436873727) was observed by TESS during Sector 18 of the primary mission and
then again in Sectors 42, 43, 44, and 58 of the extended mission, in all cases
with 2-minute cadence. The data were processed by the NASA Science Processing
Operations Center pipeline (SPOC; Jenkins et al. 2016) and the light curves
were downloaded from the Mikulski Archive for Space Telescopes
(MAST)111https://mast.stsci.edu/portal/Mashup/Clients/Mast/Portal.html using
the Lightkurve package (Lightkurve Collaboration et al., 2018). The Presearch
Data Conditioning Simple Aperture Photometry (PDCSAP; (Stumpe et al., 2012,
2014; Smith et al., 2012)) light curves were employed in our analysis and are
plotted in Figure 1. Sector 44 was excluded from our planet analysis because
the transit occurred at the edge of a gap in the lightcurve.

Figure 1: Per-sector Normalized TESS PDCSAP light curves for TOI-4641. The
target star was observed over five TESS sectors. Because the Sector-44 dataset
covers only one part of one transit, this dataset was not included in our
global modeling of the planetary properties. Significant stellar variability
is seen within each sector of the TESS observations, as discussed in Section
4.

A candidate exoplanet orbiting TOI-4641 with a period of 22.1d was identified
in light curves including data through Sector 43 in both SPOC (Jenkins et al.,
2016) and QLP (Huang et al., 2020; Kunimoto et al., 2022) pipelines. The SPOC
performed a transit search with an adaptive, noise-compensating matched filter
(Jenkins, 2002; Jenkins et al., 2010, 2020), producing a Threshold Crossing
Event (TCE) for which an initial limb-darkened transit model was fitted (Li et
al., 2019) and a suite of diagnostic tests were conducted to help assess the
planetary nature of the signal (Twicken et al., 2018). The QLP performed its
transit search with the Box Least Squares Algorithm (Kovács et al., 2002). The
transit signature passed all SPOC data validation diagnostic tests, and the
TESS Science Office issued an alert (Guerrero et al., 2021) for TOI 4641.01 on
19 November 2021. The difference image centroid offsets localized the transit
source for TOI 4641.01 within $2.4\pm 2.5$ arcsec; all TIC v8 (Stassun et al.,
2019) objects other than TOI-4641 were excluded as potential sources of the
transit signature.

####  2.1.2 Ground-based Photometry

To check the field for nearby eclipsing binaries that could potentially be
contaminating the _TESS_ photometry and to confirm the transit was on the
target star, we used the TESS Transit Finder, a customized version of the
Tapir software package (Jensen, 2013), to schedule ground-based transit
observations. Follow-up observations were done using the Tierras Observatory
(Garcia-Mejia et al., 2020) at the Fred Lawrence Whipple Observatory atop
Mount Hopkins in Arizona, USA. Tierras is a 1.3m telescope with a Teledyne e2v
4K $\times$ 4K NIR-optimized deep-depletion CCD. It has a 0.48 $\times$ 0.25
degree field-of-view and a $0.43$ arcsecond per pixel scale. The camera was
designed to have a custom narrow (40 nm full width at half maximum) bandpass
filter centered around 863.5 nm to minimize precipitable water vapor errors
known to limit ground-based photometry of M dwarfs.

The data were reduced using a custom pipeline based on similar procedures as
outlined in Irwin et al. (2015) and aperture photometry was performed using
AstroImageJ (AIJ, Collins et al., 2017). Two partial transits were observed on
UT December 17, 2022 (egress) and UT January 30, 2023 (ingress). A full
transit was observed on UT January 8, 2023. The transits on the nights UT
December 17, 2022 and UT January 8, 2023 coincided with the spectroscopic
observations described in more detail in Section 2.2.1. We extracted the
photometry using an aperture radius of 12 pixels (5.16") from the data on UT
December 17, 2022 to determine that the target star was the source of the
transit events and we excluded any star more distant than 5 arseconds as the
source of the dips observed by _TESS_. That light curve is shown in Figure 2
along with the phase folded _TESS_ light curve. We note $\sim 6$ hour
variation in the ground-based light curve at the $\sim 1$ mmag level. The
light curve variability is discussed in more detail in Section 4. Data from
the nights of UT January 8 and January 30, 2023 were not of sufficient quality
to add any value to the fit due to poor observing conditions and experimental
exposure times leading to occasional saturation and were not included in the
global analysis. The photometric analysis files from AIJ are all available
publicly on the Exoplanet Follow-up Observing Program
(ExoFOP)222https://exofop.ipac.caltech.edu/tess/ website.

Figure 2: Top: Phase folded TESS light curves of 5 transits from Sectors 18,
42, 43, and 58. The best fit model from the global modeling analysis is
illustrated as a red line. Bottom: Follow-up observations from the Tierras
Observatory, capturing the egress event on UT December 17, 2022.

####  2.1.3 High-resolution Imaging

We observed TOI-4641 on UT February 5, 2023 using NESSI (Scott et al., 2018),
a speckle imager at the WIYN 3.5m telescope. The observations consisted of
taking speckle sequence data in two filters with central wavelengths of 562
and 832nm. We reduced these data using the standard speckle pipeline (Howell
et al., 2011) to obtain reconstructed images of the focal plane as well as a
contrast curve centered on the target star and extending out to a radius of
1.2 arcseconds. No secondary sources were detected surrounding TOI-4641. The
background limit plots are shown in Figure 3.

Figure 3: Contrast curves based on NESSI speckle imaging of TOI-4641 in two
filters centered at 562 nm (blue curve) and 832 nm (red curve). Speckle
imaging analysis was confined to the inner $1.2\arcsec$ of the full $4.6\times
4.6\arcsec$ field-of-view. Speckle reconstructed images centered on the star
are inset at the top of the figure. We find no companions to TOI-4641 in these
high resolution imaging observations.

###  2.2 Spectroscopic Observations

####  2.2.1 Radial Velocity Observations

A single observation of TOI-4641 was obtained on UT November 23, 2021 with the
NRES (Siverd et al., 2018) spectrograph located at the Las Cumbres Observatory
(LCOGT; Brown et al., 2013) node at McDonald Observatory, Texas, USA. Two
successive 900 second observations were stacked and reduced via the BANZAI-
NRES pipeline (McCully et al., 2022) to estimate a $v\sin I_{\star}$ of
approximately 90 km s-1, indicating that TOI-4641b was a favorable target for
further investigation into the projected spin-orbit angle.

Spectroscopic observations were obtained with the Tillinghast Reflector
Echelle Spectrograph (TRES; Fűrész (2008)) as part of the TESS Follow-up
Observing Program (TFOP) SubGroup 2 (SG2) Reconnaissance Spectroscopy program.
TRES is an optical (3900-9100 Å) fiber-fed echelle spectrograph on the 1.5m
Tillinghast Reflector at the Fred Lawrence Whipple Observatory (FLWO) in
Arizona, USA. The spectrograph has a resolving power of R $\sim$ 44,000.

TOI-4641 was first observed on UT November 30, 2021 with an exposure time of
180 seconds leading to a signal-to-noise per resolution element (SNRe) of 113.
The quick-look classification showed that the host star was rapidly rotating.
Despite the rapid rotation which would make precise radial velocities very
challenging, a second TRES spectrum (SNRe of 131) at opposite orbital
quadrature was obtained on UT December 21, 2021 to check for a large velocity
variation that would be indicative of a stellar companion. The velocity offset
between the two spectra was $\sim 275$ m s-1 based on the standard TRES
pipeline which derives velocities using a single order of the spectrum
centered on the Mg b features as described in Buchhave et al. (2010). An
additional 13 TRES observations were obtained between UT December 10-31, 2022
with longer exposures than the original reconnaissance spectra in an attempt
to gain high enough SNRe to detect an orbital solution. The average SNRe of
the new observations was 195.

In an attempt to get the best velocity precision, we used a least squares
deconvolution technique (Zhou et al., 2016) based on the methods of Donati et
al. (1997) and Collier Cameron et al. (2010) to extract radial velocities. Due
to the rapid rotation of the host star, we were unable to detect an orbital
signal of the planet. We also tried a multi-spectral order analysis of the
available spectra. Each spectrum was cross-correlated order-by-order against
the highest SNRe observed spectrum to derive relative multi-order velocities.
While we again were unable to detect a clear orbital signal, we were able to
determine a 3$\sigma$ upper limit planetary mass of 3.87 $M_{\rm Jup}$. The
least squares deconvolution velocities are presented in Table 1, and were
adopted for further analysis in Section 3. In addition, we also modeled the
line profile determined from each spectrum to derive rotational and
macroturbulent broadening velocities for the host star. The line profile is
modeled as per Gray & Corbally (1994), with the macroturbulent broadening
component described via a radial-tangential model. We find a best-fit
rotational broadening velocity of $86.3\pm 1.0$ km s-1, and a macroturbulent
velocity of 4.00${}^{+0.62}_{-0.66}$ km s-1for TOI-4641.

####  2.2.2 Transit Spectroscopic Observation

As a planet transits across its host star’s stellar disk, a portion of the
stellar blue- and red-shifted light is blocked which produces a shift in the
radial velocity measurements. This effect, known as the Rossiter-McLaughlin
(RM; Rossiter (1924), McLaughlin (1924)) effect, allows us to measure the
projected spin-orbit angle of a transiting system.

To determine the projected spin-orbit angle, spectra were obtained during two
transits on UT December 17, 2022 and UT January 8, 2023 using the TRES
spectrograph. Spectra were acquired in the standard way by obtaining a set of
three 300 second exposures surrounded on either side by Thorium-Argon
calibration spectra. The three spectra were then combined using cosmic ray
rejection and run through the standard TRES pipeline as described in Section
2.2.1. We observed a partial ingress on the night of UT December 17, 2022
collecting 18 sets of 3 spectra and a full transit on UT January 8, 2023 of 39
sets of 3 spectra with an average SNRe of 147 and 195, respectively.

To extract the planetary Doppler shadow, we derived the line broadening
profiles for each observation in an analysis similar to that described above.
We performed a least squares deconvolution between the observed spectra and a
synthetic ATLAS9 stellar template (Castelli & Kurucz, 2004), generated at the
atmospheric parameters of the host star, with no rotational broadening
incorporated. We then modeled the differences between each derived line
profile and the median combined line profile measured over the transit night
as part of our global modeling analysis as per Zhou et al. (2019). Briefly,
for each observation, we calculated the integrated line profile of the portion
of the star blocked by the planet. This incorporates the effects of local limb
darkening, macroturbulent broadening, and rotation. This is modeled
simultaneously with the system parameters as part of the global modeling
process (Section 3). The line profile residuals for each transit observation,
and for the combined observations, are shown in Figure 4. The dark trail from
bottom left to top right represents the shadow of the planet during the
transit in the line profile residuals.

Table 1: TRES Recon Radial Velocities Time | Velocity | Uncertainty
---|---|---
[$\rm{BJD_{TDB}}$] | [ km s-1] | [ km s-1]
2459548.73116 | 24.12 | 0.26
2459560.66830 | 24.35 | 0.22
2459923.66941 | 24.34 | 0.29
2459924.68547 | 24.58 | 0.31
2459925.70482 | 24.53 | 0.21
2459928.73172 | 24.31 | 0.27
2459929.73860 | 24.19 | 0.32
2459933.80715 | 24.56 | 0.31
2459934.80076 | 24.49 | 0.28
2459935.67590 | 24.58 | 0.30
2459936.82834 | 24.53 | 0.22
2459937.70062 | 24.44 | 0.27
2459938.74970 | 24.52 | 0.30
2459939.81605 | 24.51 | 0.26
2459944.75598 | 24.34 | 0.30
2022-12-17 | 2023-01-08 | Joint modeling
---|---|---
|  |
Figure 4: TRES Doppler spectroscopic results. The color scale represents the
fractional variation in the spectral line profile. Each plot shows the
planetary signal after the average rotational profile is subtracted (top), the
best-fit model (middle), and the residuals after subtracting the planetary
signal (bottom). Left: Doppler spectroscopy egress transit event on December
17, 2022. Middle: Doppler spectroscopy observations from the full transit
event on January 8, 2023 Right: The combined Doppler spectroscopy result from
the two TRES transit observations. The vertical lines are the $v\sin i$
boundaries, while the horizontal lines mark the ingress and egress timings.
Table 2: Stellar parameters for TOI-4641.

Stellar Parameters | Value | Source
---|---|---
Catalog Information
TIC ID | 436873727 | TESS TOI Catalog
TOI ID | 4641 | TESS TOI Catalog
Gaia DR3 ID | 114340658009875072 | GAIA DR3
2MASS ID | J02501388+2520010 | 2MASS
TYC ID | 1785-00801-1 | Tycho
Coordinates and Proper Motion
Right Ascension (h:m:s) | 02:50:13.90 (J2000) | GAIA DR3
Declination (d:m:s) | 25:20:01.00 (J2000) | GAIA DR3
Parallax (mas) | $11.409\pm 0.026$ | GAIA DR3
$\mu$R.A (mas yr-1) | $61.447\pm 0.027$ | GAIA DR3
$\mu$Dec. (mas yr-1) | $-20.020\pm 0.024$ | GAIA DR3
Magnitudes
$TESS$ (mag) | $7.1323\pm 0.0061$ | TESS TOI Catalog
$G$ (mag) | 7.42334 $\pm$ 0.00031 | GAIA DR3
$B_{p}$ (mag) | 7.60660 $\pm$ 0.00093 | GAIA DR3
$R_{p}$ (mag) | 7.09227 $\pm$ 0.00057 | GAIA DR3
$B$ (mag) | 7.895 $\pm$ 0.027 | Tycho
$V$ (mag) | 7.51 $\pm$ 0.030 | Tycho
$J$ (mag) | 6.746 $\pm$ 0.020 | 2MASS
$H$ (mag) | 6.639 $\pm$ 0.023 | 2MASS
$K$ (mag) | 6.579 $\pm$ 0.029 | 2MASS
$WISE_{3.4\mu}$ (mag) | 6.552 $\pm$ 0.081 | WISE
$WISE_{4.6\mu}$ (mag) | 6.527 $\pm$ 0.024 | WISE
$WISE_{12\mu}$ (mag) | 6.588 $\pm$ 0.016 | WISE
$WISE_{22\mu}$ (mag) | 6.426 $\pm$ 0.085 | WISE

Note: TESS TOI Primary Mission Catalog; (Guerrero et al., 2021), Tycho; (Høg
et al., 2000); GAIA DR3; (Gaia Collaboration et al., 2022a), 2MASS; (Cutri et
al., 2003), WISE; (Cutri et al., 2021a)

####  2.2.3 Stellar Parameters from TRES spectra

We used the Stellar Parameter Classification (SPC; Buchhave et al. 2012) tool
to derive stellar parameters using the TRES spectra. SPC cross-correlates an
observed spectrum against a library grid of synthetic spectra calculated using
the Kurucz (1992) atmospheric models. A $\sim 310$ Å region of the spectrum
surrounding the Mg b lines is used to derive effective temperature, $T_{\rm
eff}$, surface gravity, $\log g$, rotational velocity, $v\sin i$, and
metallicity, $\left[{\rm m}/{\rm H}\right]$. Metallicity is derived using all
available metal lines rather than just the Fe lines and therefore reported as
$\left[{\rm m}/{\rm H}\right]$ but is closely related to $\left[{\rm Fe}/{\rm
H}\right]$ values.

We recently developed a quality flag (QF) metric - Excellent, Good, Fair, and
Poor - for SPC results based on the known limitations of SPC. Each spectrum is
run through an algorithm (shown in Figure 5) to determine the reliability of
the result using stellar effective temperature, rotational velocity, SNRe, and
the cross-correlation function (CCF) peak value as quality indicators. As a
service to the community, SPC stellar parameters for all TRES spectra of
_TESS_ targets are uploaded to the ExoFOP website when the QF is Excellent or
Good. If the QF is determined to be Fair, as in the case of TOI-4641 due to
the high stellar rotational velocity, the stellar parameters are not
considered reliable and only the rotational velocity of the star is uploaded.
Stellar parameters are not uploaded when a QF of Poor is determined.

SPC reports a $v\sin i$ of $91.60\pm 0.50$ km s-1 for TOI-4641 but because SPC
does not solve for macroturbulence and the Least Squares Deconvolution
analysis does, we chose to use the Least Squares Deconvolution reported value
as determined in Section 2.2.1 ($86.3\pm 1.0$ km s-1 ) as a prior for our
global analysis. The SPC uncertainty reported is the floor error and should be
inflated due to the star’s rapid rotation.

Figure 5: SPC quality flag decision tree assigns a quality flag - Excellent,
Good, Fair, and Poor - based on the known limitations of SPC. The quality flag
reliability of the result uses the stellar effective temperature, rotational
velocity, SNRe, and the cross-correlation function (CCF) peak value as quality
indicators.

###  2.3 SED analysis

We used all available broadband photometry, including _Hipparcos_ $B$ and $V$
bands (Perryman et al., 1997), _Gaia_ DR3 $G$, $Bp$, $Rp$ (Gaia Collaboration
et al., 2022b), 2MASS $J$, $H$, $K$ (Skrutskie et al., 2006), and WISE
$W_{1}$, $W_{2}$, $W_{3}$, $W_{4}$ bands (Cutri et al., 2021b), as well as
_Gaia_ DR3 parallaxes to model the spectral energy distribution of TOI-4641.
The spectral energy distribution is modeled simultaneously with the
photometric and spectroscopic observations of the system, such that the
stellar properties derived are jointly constrained by the transit and the
photometric properties of the star. At each iteration of the global model
(Section 3), we compute the interpolated isochrone magnitudes for each tested
stellar mass, age, and metallicity. We then compute the log likelihood between
the isochrone magnitudes and the observed values and associated uncertainties
for each band. We adopt the MIST isochrones (Dotter, 2016), interpolated via
the minimint package (Koposov, 2020), for our stellar models. The best fitting
model, as per Table 2, is shown in Figure 6.

Figure 6: Spectral energy distribution of the target star TOI-4641. Magnitudes
from _Gaia_ $G$, $B_{p}$, $R_{p}$, _Tycho_ $B$, $V$, 2MASS $J$, $H$, $Ks$ are
included in the global modeling of the system and are shown in dark blue.
Magnitudes from WISE $W1$, $W2$, $W3$, and $W4$ are shown in light grey but
not included in the model.

##  3 Global Modeling

To determine the stellar and planetary parameters of the system holistically,
we performed a joint analysis of all available photometric, spectroscopic, and
catalog observations. This included photometric transit observations from TESS
and Tierras (Section 2.1.1 and 2.1.2), two spectroscopic transit observations
from TRES (Section 2.2.2), the TRES out-of-transit radial velocities (Section
2.2.1), and available catalog observations (Section 2.3). Free parameters
largely describing the transit include the orbital period $P$, reference time
of transit center $T_{0}$, planet-to-star radius ratio $R_{p}/R_{\star}$,
line-of-sight inclination $i$, and orbital eccentricity parameters
$\sqrt{e}\cos\omega$ and $\sqrt{e}\sin\omega$. In addition, the radial
velocity orbit is modeled by including the free parameter describing the mass
of the planetary companion $M_{p}$. The spectroscopic transit is modeled as
per Zhou et al. (2019), with free parameters including the projected spin-
orbit angle $\lambda$, rotational broadening $v\sin I_{\star}$, and
macroturbulent broadening velocity. Simultaneous with the transit models, we
also interpolate the stellar isochrones as per Section 2.3. At each step, we
model the spectral energy distribution to constrain the stellar parameters.
Free parameters describing the stellar isochrone and spectral energy
distribution modeling include stellar mass $M_{\star}$, age, metallicity
[m/H], and parallax.

A number of parameters are constrained by informed priors in the global
modeling. Parallax is tightly constrained by a Gaussian prior about its _Gaia_
DR3 value and associated uncertainties. Rotational and macroturbulent
broadening velocities are constrained by Gaussian priors about their
spectroscopically determined values. Stellar metallicity is constrained by a
Gaussian prior about the SPC-determined value. Uniform priors about reasonable
physical parameter spaces are adopted for all other free parameters. The
adopted priors are noted in Table 3. Gaussian priors are noted as
$\mathcal{G}(\mu,\sigma)$, while uniform priors and their adopted ranges are
noted by $\mathcal{U}(\mathrm{min},\mathrm{max})$.

The photometric transits are modeled as per Mandel & Agol (2002) via the
batman package (Kreidberg, 2015). Limb darkening parameters are interpolated
and fixed to their values as per Claret & Bloemen (2011), Claret (2017), and
Eastman et al. (2013). The spectroscopic transit is modeled as per Section
2.2.2, via a disk integration of the portion of the stellar surface occulted
by the planet, incorporating the effect of local macroturbulence and
rotational broadening.

The best fit parameters are presented in Table 2 and Table 3. A number of
additional parameters are derived from the posterior chains and reported in
Table 3 for completeness. These are marked as ‘inferred’ in the table. Stellar
parameters for luminosity, effective temperature, surface gravity, and age are
re-derived from MIST isochrones interpolations for each given link in the MCMC
chain subsequent to the global modeling. Planet properties, including radius,
orbital semi-major axis, eccentricity, and transit impact parameter are
subsequently derived from the posteriors. In addition to the global model, we
also propagate the posteriors and follow Masuda & Winn (2020) to derive the 3D
spin-orbit obliquity of the system. We make use of the rotational period (see
Section 4), line broadening, and projected spin-orbit angle to derive a 3D
orbital obliquity of $2.4\pm 1.3^{\circ}$.

Table 3: Best-fit Stellar and Planetary Properties for TOI-4641

Parameters | Description (Units) | Prior Values | Best Fit
---|---|---|---
Stellar Parameters:
$M_{\star}$ | Stellar Mass ($M_{\odot}$) | $\mathcal{U}(1.2,3.5)$ | $1.41^{+0.068}_{-0.059}$
$R_{\star}$ | Stellar Radius ($R_{\odot}$) | $\mathcal{U}(1.2,3.5)$ | $1.72^{+0.041}_{-0.043}$
$L_{\star}$ | Stellar Luminosity ($L_{\odot}$) | Inferred | $4.95^{+0.94}_{-1.21}$
$T_{\rm eff}$ | Effective Temperature (K) | Inferred | $6560^{+300}_{-340}$
$\log g$ | Surface Gravity (cgs) | Inferred | $4.11^{+0.69}_{-0.69}$
$\mathrm{[m/H]}$ | Metallicity (dex) | $\mathcal{G}(-0.2,0.2)$ | $-0.09^{+0.11}_{-0.18}$
$v\sin I_{\star}$ | Projected Rotational Velocity ( km s-1) | $\mathcal{G}(86.3,1.0)$ | 86.3${}^{+1.00}_{-0.99}$
$v_{\mathrm{macro}}$ | Macroturbulent Velocity ( km s-1) | $\mathcal{G}(5.5,7.7)$ | 4.00${}^{+0.62}_{-0.66}$
Parallax | Parallax (mas) | $\mathcal{G}(11.409,0.026)$ | $11.409\pm\,0.025$
Age | Age (Gyr) | Inferred | 2.69${}^{+0.81}_{-1.14}$
Distance | Distance (pc) | Inferred | 87.60${}^{+0.20}_{-0.20}$
$I_{\star}$ | Stellar Inclination (deg) | Derived |  $>83$ $(3\sigma)$
$u_{\mathrm{1,TESS}}$ | Limb darkening coefficient | Fixed | 0.197
$u_{\mathrm{2,TESS}}$ | Limb darkening coefficient | Fixed | 0.317
$u_{\mathrm{1,Tierras}}$ | Limb darkening coefficient | Fixed | 0.209
$u_{\mathrm{2,Tierras}}$ | Limb darkening coefficient | Fixed | 0.314
Planetary Parameters:
$P$ | Orbital Period (days) | $\mathcal{U}(22.09,22.10)$ | 22.093410${}^{+0.000051}_{-0.000047}$
$T_{o}$ | Epoch (BJD) | $\mathcal{U}(245910.80,245910.85)$ | 2459510.82759${}^{+0.00091}_{-0.00100}$
$M_{p}$ | Planet Mass ($M_{\rm Jup}$) | $\mathcal{U}(0,105)$ | $<3.87\,(3\sigma)$
$R_{p}$ | Planet Radius ($R_{\rm Jup}$) | Inferred | 0.730${}^{+0.026}_{-0.028}$
$R_{p}/R_{\star}$ | Radius of planet to star ratio | $\mathcal{U}(0.01,0.1)$ | 0.042${}^{+0.0009}_{-0.0009}$
$a/R_{\star}$ | Semi-major axis to star radius ratio | Inferred | $21.53^{+0.51}_{-0.52}$
$a$ | Semi-major axis (AU) | Inferred | 0.173${}^{+0.015}_{-0.015}$
$e$ | eccentricity | Inferred | $<0.074\,(3\sigma)$
$\sqrt{e}\cos\omega$ | eccentricity parameter | $\mathcal{U}(-0.2,0.2)$ | $0.01_{-0.14}^{+0.13}$
$\sqrt{e}\sin\omega$ | eccentricity parameter | $\mathcal{U}(-0.2,0.2)$ | $0.03_{-0.13}^{+0.12}$
$\gamma$ | Radial velocity offset ( km s-1) | $\mathcal{U}(24,25)$ | $24.400_{-0.072}^{+0.077}$
$T_{14}$ | Transit duration (days) | Inferred | $0.2266^{+0.0029}_{-0.0030}$
$i$ | Orbital inclination (deg) | $\mathcal{U}(80,110)$ | 87.90${}^{+0.088}_{-0.084}$
$b$ | Impact parameter | Inferred | 0.780${}^{+0.014}_{-0.015}$
$\lambda$ | Projected spin-orbit angle (deg) | $\mathcal{U}(-185,185)$ | $1.41^{+0.76}_{-0.76}$
$\psi$ | 3D orbital obliquity (deg) | Derived | $2.4\pm 1.3$

Note: Gaussian priors are listed as $\mathcal{G}$(median,width) and uniform
priors are listed as $\mathcal{U}$(lower bound, upper bound). Inferred
parameters are calculated from the posterior distribution. TRES spectra were
used to derive a metallicity prior using the Stellar Parameter Classification
analysis (Buchhave et al., 2012) and the project rotational velocity and
macroturbulent velocity using a least squares deconvolution analysis (Zhou et
al., 2016). Parallax was obtained from the _Gaia_ DR3 release (Gaia
Collaboration et al., 2022a).

##  4 Stellar variability

TOI-4641 exhibits short timescale stellar variability at the 1 mmag level
during all five sectors of _TESS_ observations and is also noticed in the
ground-based observations. The variability is likely consistent with changing
rotational spot modulation on the stellar surface.

The top left panel of Figure 7 shows the sector-by-sector frequency power
spectrum of TOI-4641. Sectors 18 and 58 exhibit variability at the $0.96\pm
0.02$ cycles/day frequency, while the $3.93\pm 0.02$ cycles/day peak is
strongest for Sectors 42, 43, and 44. The variability does not exhibit equal
period spacing as is expected for $\gamma$ Dor variables, and the frequency is
too low for $\delta$ Scuti pulsations. We suggest the variability is most
consistent with rotational modulation. The rotational broadening velocity
$v\sin I_{\star}=86.3\pm 1.0\,\mathrm{km\,s}^{-1}$ propagates to an expected
rotation period of $1.06\pm 0.04$ days assuming $\sin I_{\star}=1$,
corresponding to $0.94\pm 0.04$ cycles/day, which is consistent with the peak
variability frequency in the light curve in Sectors 18 and 58. The 1 cycle/day
periodicity is the second most dominant peak in Sectors 42, 43, and 44, with
the 4 cycles/day peak likely due to the specific spot configuration during
this timeframe. Low-amplitude spot variability is seen in early-type stars in
the _Kepler_ sample. Sikora et al. (2020) notes that 10-30% of A and B stars
exhibit spot-induced variability, despite not showing spectroscopic signatures
of chemical peculiarity. Using all _TESS_ sectors of data, we derive a
$3\sigma$ lower limit on the stellar inclination $I_{\star}$ of 83 $\degr$.

Figure 7: TOI-4641 exhibits 1 day time scale variability over all observed
_TESS_ sectors. The per sector frequency spectra are shown in the top left
panel. The frequency spectrum of the entire _TESS_ dataset is shown on the
bottom. The per sector light curves, folded to the $0.96$ cycles/day
frequency, are shown on the top right panel. The 0.96 cycle per day peak is
consistent with the expected rotation period of the star as per the
spectroscopic rotational line broadening velocity. We interpret the
variability to be spot-induced modulations, with the spots likely changing in
configuration over multiple months.

To check if the variability is on target, we used the lightkurve code
(Lightkurve Collaboration et al., 2018) to extract light curves from multiple
target and background apertures, and found no changes in the amplitude and
shape of the variability. Figure 8 shows a set of example target apertures and
resulting light curves. The target is in a sparse field and there are no stars
within a 1’ field that are bright enough to cause the 1 mmag variability seen
on target after dilution. We also note that the rotational modulation was
similarly identified in the SPOC transit search of the combined TOI-4641 light
curves. These ‘Threshold Crossing Events’ (TCEs) attributed to the rotational
modulation were subjected to the same diagnostic tests as would any TCE
triggered by a transiting planet. The difference image centroid offsets
(Twicken et al., 2018) showed that the source of the modulation was consistent
with TOI-4641 at the 3$\sigma$ level and inconsistent at that level with all
other TIC v8.2 objects.

Figure 8: Photometry, using two different apertures, showing that the observed
variability is likely on target and not induced by a nearby background source.
The top panel shows a narrow 90-percentile (red) and a broad 50-percentile
(blue) aperture. The bottom panel shows a 2-day example section of their
respective light curves showing no difference in the frequency and amplitude
of the observed light curve as a function of aperture size.

##  5 Discussion

TOI-4641 is a warm Jupiter in a 22 day orbit around a bright (V=7.5), rapidly
rotating (86.3${}^{+1.00}_{-0.99}$ km s-1) F-star. The orbit is nearly
circular with eccentricity constrained at the 3$\sigma$ level to less than
0.074. This target was observed in five _TESS_ sectors and was photometrically
followed up by the Tierras ground-based Observatory to rule out the false
positive scenario of a nearby eclipsing binary contaminating the aperture.
Additionally, we obtained high-resolution images from the Speckle imager on
the WIYN 3.5m telescope and detected no secondary sources out to a radius of
1.2 arcseconds. TRES spectra allowed us to detect a 3-$\sigma$ upper limit
planetary mass of 3.87 $M_{\rm Jup}$. We also obtained two nights of TRES in-
transit spectroscopic data to measure the projected spin-orbit angle of
$1.41^{+0.76}_{-0.76}$$\degr$.

TOI-4641b is amongst the longest period planets to be thoroughly characterized
about a hot rapidly rotating star (Figure 9). Long-period planets provide
tests for mechanisms that induce primordial misalignment in planetary systems.
At such orbital distances, star-planet tidal interactions are too weak to
modify the orbital obliquity.

Chaotic accretion is one proposed method of primordial misalignments where
neighboring protostar material interacts or arrives at different times during
the accretion process potentially causing the protostar disk to tilt with
respect to the star (Bate et al., 2010; Thies et al., 2011; Fielding et al.,
2015; Bate, 2018; Kuffmeier et al., 2021). Another proposed mechanism for
misalignment is magnetic warping. In young stars, in particular, the twisting
of magnetic field lines between the ionized disk and the differential rotation
of the young star can cause misalignment (Foucart & Lai, 2011; Lai, 2012). In
this scenario, the misalignment torque must overcome the realignment torque
from accretion, magnetic braking, disk winds and viscosity. Stellar or
planetary companions can also cause misalignment during the primordial phase
of formation (Borderies et al., 1984; Lubow & Ogilvie, 2000; Batygin, 2012;
Matsakos & Königl, 2017). Rogers et al. (2012) proposed that internal gravity
waves excited at the radiative-convective boundary of early-type stars can
induce their surface layers to change in spin direction. Changes to the spin
axes of early-type stars will also lead to an apparent spin-orbit
misalignment, potentially contributing to the temperature-obliquity gradient
seen in the hot Jupiter population.

Critically, most of these proposed mechanisms do not have a strong dependence
on the host star properties beyond planet-star tidal interactions. We should
not observe strong differences in the obliquity distributions of longer-period
Jovian planets as a function of stellar mass. Should internal gravity waves
(Rogers et al., 2012) play a major role in shaping the spin axes of early-type
stars, no orbital distance trends should be observed in that population.

Testing these predictions motivate full characterizations of planets in long-
period orbits about early-type stars. TOI-4641b is the second such Jovian-
sized planet around a rapidly rotating early-type star, preceded only by
Kepler-448b (Bourrier et al., 2015; Johnson et al., 2017). Both planets have
been found in well-aligned geometries, whereas early-type stars with closer
orbiting giant planets tend to show a broad range of obliquities including
nearly-polar and retrograde orbits.

Figure 9: TOI-4641b amongst giant planets with obliquities measured. Top
Obliquity as a function of $a/R_{\star}$. Values adopted from Albrecht et al.
(2022) including only planets with $R_{p}>8\,R_{e}$ to represent the Jovian
population. The colors represent the effective temperature of the host star.
TOI-4641b is represented with a star. Middle Selecting only for planets about
stars $T_{\mathrm{eff}}>6500\,K$. TOI-4641 is one of the longest period
planets characterized about an early-type star. Bottom The distribution for
planets about cool stars $T_{\mathrm{eff}}<6500\,K$.

## Acknowledgments

We respectfully acknowledge the traditional custodians of the lands on which
we conducted this research and throughout Australia. We recognize their
continued cultural and spiritual connection to the land, waterways, cosmos and
community. We pay our deepest respects to all Elders, present and emerging
people of the Giabal, Jarowair and Kambuwal nations, upon whose lands this
research was conducted. CH thanks the support of the ARC DECRA program
DE200101840. GZ thanks the support of the ARC DECRA program DE210101893. JGM
acknowledges support by the Heising Simons Foundation through a 51 Pegasi B
Fellowship, and by the Pappalardo family through the MIT Pappalardo Fellowship
in Physics. JGM and DC thank the staff at the F. L. Whipple Observatory for
their assistance in the refurbishment and maintenance of the 1.3-m telescope.
Tierras is supported by grants from the John Templeton Foundation and the
Harvard Origins of Life Initiative. The opinions expressed in this publication
are those of the authors and do not necessarily reflect the views of the John
Templeton Foundation. SJM was supported by the Australian Research Council
(ARC) through Future Fellowship FT210100485. Funding for the TESS mission is
provided by NASA’s Science Mission Directorate. We acknowledge the use of
public TESS data from pipelines at the TESS Science Office and at the TESS
Science Processing Operations Center. This research has made use of the
Exoplanet Follow-up Observation Program website, which is operated by the
California Institute of Technology, under contract with the National
Aeronautics and Space Administration under the Exoplanet Exploration Program.
Resources supporting this work were provided by the NASA High-End Computing
(HEC) Program through the NASA Advanced Supercomputing (NAS) Division at Ames
Research Center for the production of the SPOC data products. This paper
includes data collected by the TESS mission that are publicly available from
the Mikulski Archive for Space Telescopes (MAST). This work has made use of
data from the European Space Agency (ESA) mission Gaia
(https://www.cosmos.esa.int/gaia), processed by the Gaia Data Processing and
Analysis Consortium (DPAC,
https://www.cosmos.esa.int/web/gaia/dpac/consortium). Funding for the DPAC has
been provided by national institutions, in particular the institutions
participating in the Gaia Multilateral Agreement.

## Data Availability

The data underlying this article will be shared on reasonable request to the
corresponding author.

## References

  * Albrecht et al. (2012) Albrecht S., et al., 2012, ApJ, 757, 18
  * Albrecht et al. (2022) Albrecht S. H., Dawson R. I., Winn J. N., 2022, PASP, 134, 082001
  * Bate (2018) Bate M. R., 2018, MNRAS, 475, 5618
  * Bate et al. (2010) Bate M. R., Lodato G., Pringle J. E., 2010, MNRAS, 401, 1505
  * Batygin (2012) Batygin K., 2012, Nature, 491, 418
  * Borderies et al. (1984) Borderies N., Goldreich P., Tremaine S., 1984, ApJ, 284, 429
  * Bourrier et al. (2015) Bourrier V., et al., 2015, A&A, 579, A55
  * Bourrier et al. (2017) Bourrier V., Cegla H. M., Lovis C., Wyttenbach A., 2017, A&A, 599, A33
  * Brown et al. (2013) Brown T. M., et al., 2013, PASP, 125, 1031
  * Buchhave et al. (2010) Buchhave L. A., et al., 2010, ApJ, 720, 1118
  * Buchhave et al. (2012) Buchhave L. A., et al., 2012, Nature, 486, 375
  * Castelli & Kurucz (2004) Castelli F., Kurucz R. L., 2004, ArXiv Astrophysics e-prints,
  * Claret (2017) Claret A., 2017, A&A, 600, A30
  * Claret & Bloemen (2011) Claret A., Bloemen S., 2011, A&A, 529, A75
  * Collier Cameron et al. (2010) Collier Cameron A., et al., 2010, MNRAS, 407, 507
  * Collins et al. (2017) Collins K. A., Kielkopf J. F., Stassun K. G., Hessman F. V., 2017, AJ, 153, 77
  * Cutri et al. (2003) Cutri R. M., et al., 2003, VizieR Online Data Catalog, p. II/246
  * Cutri et al. (2021a) Cutri R. M., et al., 2021a, VizieR Online Data Catalog, p. II/328
  * Cutri et al. (2021b) Cutri R. M., et al., 2021b, VizieR Online Data Catalog, p. II/328
  * Dawson & Johnson (2018) Dawson R. I., Johnson J. A., 2018, ARA&A, 56, 175
  * Donati et al. (1997) Donati J. F., Semel M., Carter B. D., Rees D. E., Collier Cameron A., 1997, MNRAS, 291, 658
  * Dotter (2016) Dotter A., 2016, ApJS, 222, 8
  * Eastman et al. (2013) Eastman J., Gaudi B. S., Agol E., 2013, PASP, 125, 83
  * Fűrész (2008) Fűrész G., 2008, PhD thesis, University of Szeged, Hungary
  * Fielding et al. (2015) Fielding D. B., McKee C. F., Socrates A., Cunningham A. J., Klein R. I., 2015, MNRAS, 450, 3306
  * Foucart & Lai (2011) Foucart F., Lai D., 2011, MNRAS, 412, 2799
  * Gaia Collaboration et al. (2022a) Gaia Collaboration et al., 2022a, arXiv e-prints, p. arXiv:2208.00211
  * Gaia Collaboration et al. (2022b) Gaia Collaboration et al., 2022b, arXiv e-prints, p. arXiv:2208.00211
  * Garcia-Mejia et al. (2020) Garcia-Mejia J., Charbonneau D., Fabricant D., Irwin J., Fata R., Zajac J. M., Doherty P. E., 2020, in Marshall H. K., Spyromilio J., Usuda T., eds, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series Vol. 11445, Ground-based and Airborne Telescopes VIII. p. 114457R (arXiv:2012.09744), doi:10.1117/12.2561467
  * Gray & Corbally (1994) Gray R. O., Corbally C. J., 1994, AJ, 107, 742
  * Guerrero et al. (2021) Guerrero N. M., et al., 2021, ApJS, 254, 39
  * Heller (1993) Heller C. H., 1993, ApJ, 408, 337
  * Høg et al. (2000) Høg E., et al., 2000, A&A, 355, L27
  * Howell et al. (2011) Howell S. B., Everett M. E., Sherry W., Horch E., Ciardi D. R., 2011, AJ, 142, 19
  * Huang et al. (2020) Huang C. X., et al., 2020, Research Notes of the American Astronomical Society, 4, 204
  * Irwin et al. (2015) Irwin J. M., Berta-Thompson Z. K., Charbonneau D., Dittmann J., Falco E. E., Newton E. R., Nutzman P., 2015, in 18th Cambridge Workshop on Cool Stars, Stellar Systems, and the Sun. pp 767–772 (arXiv:1409.0891), doi:10.48550/arXiv.1409.0891
  * Jenkins (2002) Jenkins J. M., 2002, ApJ, 575, 493
  * Jenkins et al. (2010) Jenkins J. M., et al., 2010, in Radziwill N. M., Bridger A., eds, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series Vol. 7740, Software and Cyberinfrastructure for Astronomy. p. 77400D, doi:10.1117/12.856764
  * Jenkins et al. (2016) Jenkins J. M., et al., 2016, in Chiozzi G., Guzman J. C., eds, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series Vol. 9913, Software and Cyberinfrastructure for Astronomy IV. p. 99133E, doi:10.1117/12.2233418
  * Jenkins et al. (2020) Jenkins J. M., Tenenbaum P., Seader S., Burke C. J., McCauliff S. D., Smith J. C., Twicken J. D., Chandrasekaran H., 2020, Kepler Data Processing Handbook: Transiting Planet Search, Kepler Science Document KSCI-19081-003, id. 9. Edited by Jon M. Jenkins.
  * Jensen (2013) Jensen E., 2013, Tapir: A web interface for transit/eclipse observability, Astrophysics Source Code Library, record ascl:1306.007 (ascl:1306.007)
  * Johnson et al. (2017) Johnson M. C., Cochran W. D., Addison B. C., Tinney C. G., Wright D. J., 2017, AJ, 154, 137
  * Koposov (2020) Koposov S., 2020, segasai/minimint: First official release, Zenodo, doi:10.5281/zenodo.4002972
  * Kovács et al. (2002) Kovács G., Zucker S., Mazeh T., 2002, A&A, 391, 369
  * Kraft (1967) Kraft R. P., 1967, ApJ, 150, 551
  * Kreidberg (2015) Kreidberg L., 2015, PASP, 127, 1161
  * Kuffmeier et al. (2021) Kuffmeier M., Dullemond C. P., Reissl S., Goicovic F. G., 2021, A&A, 656, A161
  * Kunimoto et al. (2022) Kunimoto M., Tey E., Fong W., Hesse K., Shporer A., Fausnaugh M., Vanderspek R., Ricker G., 2022, Research Notes of the American Astronomical Society, 6, 236
  * Kurucz (1992) Kurucz R. L., 1992, in Barbuy B., Renzini A., eds, Vol. 149, The Stellar Populations of Galaxies. p. 225
  * Lai (2012) Lai D., 2012, MNRAS, 423, 486
  * Li et al. (2019) Li J., Tenenbaum P., Twicken J. D., Burke C. J., Jenkins J. M., Quintana E. V., Rowe J. F., Seader S. E., 2019, PASP, 131, 024506
  * Lightkurve Collaboration et al. (2018) Lightkurve Collaboration et al., 2018, Lightkurve: Kepler and TESS time series analysis in Python, Astrophysics Source Code Library (ascl:1812.013)
  * Lin et al. (1996) Lin D. N. C., Bodenheimer P., Richardson D. C., 1996, Nature, 380, 606
  * Lubow & Ogilvie (2000) Lubow S. H., Ogilvie G. I., 2000, ApJ, 538, 326
  * Mandel & Agol (2002) Mandel K., Agol E., 2002, ApJ, 580, L171
  * Masuda & Winn (2020) Masuda K., Winn J. N., 2020, AJ, 159, 81
  * Matsakos & Königl (2017) Matsakos T., Königl A., 2017, AJ, 153, 60
  * McCully et al. (2022) McCully C., Daily M., Brandt G. M., Johnson M. C., Bowman M., Harbeck D.-R., 2022, in Software and Cyberinfrastructure for Astronomy VII. p. 1218914 (arXiv:2212.11381), doi:10.1117/12.2630667
  * McLaughlin (1924) McLaughlin D. B., 1924, ApJ, 60, 22
  * Perryman et al. (1997) Perryman M. A. C., et al., 1997, A&A, 500, 501
  * Pont et al. (2009) Pont F., et al., 2009, A&A, 502, 695
  * Queloz et al. (2010) Queloz D., et al., 2010, A&A, 517, L1
  * Rasio & Ford (1996) Rasio F. A., Ford E. B., 1996, Science, 274, 954
  * Ricker et al. (2015) Ricker G. R., et al., 2015, Journal of Astronomical Telescopes, Instruments, and Systems, 1, 014003
  * Rogers et al. (2012) Rogers T. M., Lin D. N. C., Lau H. H. B., 2012, ApJ, 758, L6
  * Rossiter (1924) Rossiter R. A., 1924, ApJ, 60, 15
  * Santerne et al. (2014) Santerne A., et al., 2014, A&A, 571, A37
  * Scott et al. (2018) Scott N. J., Howell S. B., Horch E. P., Everett M. E., 2018, PASP, 130, 054502
  * Sikora et al. (2020) Sikora J., Wade G. A., Rowe J., 2020, MNRAS, 498, 2456
  * Siverd et al. (2018) Siverd R. J., et al., 2018, in Evans C. J., Simard L., Takami H., eds, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series Vol. 10702, Ground-based and Airborne Instrumentation for Astronomy VII. p. 107026C, doi:10.1117/12.2312800
  * Skrutskie et al. (2006) Skrutskie M. F., et al., 2006, AJ, 131, 1163
  * Smith et al. (2012) Smith J. C., et al., 2012, PASP, 124, 1000
  * Stassun et al. (2019) Stassun K. G., et al., 2019, AJ, 158, 138
  * Stumpe et al. (2012) Stumpe M. C., et al., 2012, PASP, 124, 985
  * Stumpe et al. (2014) Stumpe M. C., Smith J. C., Catanzarite J. H., Van Cleve J. E., Jenkins J. M., Twicken J. D., Girouard F. R., 2014, PASP, 126, 100
  * Thies et al. (2011) Thies I., Kroupa P., Goodwin S. P., Stamatellos D., Whitworth A. P., 2011, MNRAS, 417, 1817
  * Twicken et al. (2018) Twicken J. D., et al., 2018, PASP, 130, 064502
  * Wijnen et al. (2017) Wijnen T. P. G., Pelupessy F. I., Pols O. R., Portegies Zwart S., 2017, A&A, 604, A88
  * Winn et al. (2009) Winn J. N., et al., 2009, ApJ, 703, 2091
  * Zhou et al. (2016) Zhou G., Latham D. W., Bieryla A., Beatty T. G., Buchhave L. A., Esquerdo G. A., Berlind P., Calkins M. L., 2016, MNRAS, 460, 3376
  * Zhou et al. (2019) Zhou G., et al., 2019, AJ, 157, 31


# Can Language Models Evaluate Human Written Text? Case Study on Korean
Student Writing for Education

Seungyoon Kim
Sangmoon High School
<EMAIL_ADDRESS>
Seungone Kim
Carnegie Mellon University
<EMAIL_ADDRESS>

###### Abstract

Large language model (LLM)-based evaluation pipelines have demonstrated their
capability to robustly evaluate machine-generated text. Extending this
methodology to assess human-written text could significantly benefit
educational settings by providing direct feedback to enhance writing skills,
although this application is not straightforward. In this paper, we
investigate whether LLMs can effectively assess human-written text for
educational purposes. We collected 100 texts from 32 Korean students across 15
types of writing and employed GPT-4-Turbo to evaluate them using
grammaticality, fluency, coherence, consistency, and relevance as criteria.
Our analyses indicate that LLM evaluators can reliably assess grammaticality
and fluency, as well as more objective types of writing, though they struggle
with other criteria and types of writing. We publicly release our dataset and
feedback 111https://github.com/seungyoon1/llm-as-a-judge-human-eval/tree/main.

##  1 Introduction

Recent works have demonstrated that when adeptly prompted or trained, large
language models (LLMs) can closely mimic human evaluation in assessing
machine-generated text [24, 14, 16]. This has led to the development of
numerous generative evaluation pipelines that use language models as judges
(denoted as ‘LLM-as-a-Judge’) [9, 20, 3]. Applying this methodology to human-
written text could greatly benefit educational settings by providing direct
feedback to enhance writing skills [5, 21]. However, human-written text often
possesses different characteristics from machine-generated text, particularly
in the unexpected use of words [6]. This discrepancy complicates the direct
application of the LLM-as-a-Judge pipeline for evaluating human-written text.
In this work, we explore whether the LLM-as-a-Judge pipeline can also be used
to evaluate human-written text. Specifically, we are interested in determining
if LLMs can effectively assess texts written by Korean students to improve
their writing skills across various formats, such as essays, reports, and
scripts.

For this study, we collected a corpus of 100 texts written by 32 Korean
students, ranging in age from 11 to 19, covering 15 types of writing. We
evaluated these texts using GPT-4-Turbo based on five criteria:
‘grammaticality,’ ‘fluency,’ ‘coherence,’ ‘consistency,’ and ‘relevance,’
resulting in a total of 500 judgments across the 100 texts. Subsequently, we
redistributed the verbal feedback generated by GPT-4-Turbo to the students,
asking them to assess whether the judgments were reasonable, overly critical,
or overly optimistic. The results indicated that while judgments on
‘grammaticality’ and ‘fluency’ were mostly deemed reasonable (87% and 93%,
respectively), those concerning the other three criteria were relatively
lower, suggesting that LLM-as-a-Judge pipeline has limitations. Notably, upon
examining individual types of writing, participants found the judgments to be
particularly unreasonable when assessing more subjective texts, such as
diaries and self-introduction essays.

Although it was not a rigorous comparison of writings on the same topic in a
controlled environment, our analysis of the evaluation results allowed us to
make several indirect observations that LLM-as-a-Judge could be utilized by
students to produce higher quality writing. First, we observed that while it
tends to give high scores for Consistency and Relevance, it gives lower scores
for Fluency. From this, we were able to indirectly observe through the
participants’ assessments that found the results reasonable, that LLM-as-a-
Judge could be used in educational settings to help students write more
fluently. Secondly, students often wrote descriptive essays and book reports,
and the evaluation results for these writing types were found to be reasonably
high. Most scores across the five criteria ranged from the high 3s to the low
4s, suggesting that LLM-as-a-Judge could be useful in elevating writings from
a 3.5 - 4 out of 5 to a full 5. Lastly, when comparing the average scores
between texts written by 11-13-year-olds and those by 17-19-year-olds, we
found that the latter tended to receive higher scores. This ability to
discriminate between the two age groups not only confirmed the robustness of
LLM-as-a-Judge but also suggested that it could be actively used to help
younger students elevate the quality of their writing to that of older
students.

##  2 Related Works

Due to the ambiguity in grading free-form responses generated by LLMs, word-
match or semantic-similarity based evaluation metrics have long been the de-
facto standards [15, 18, 23]. Recently, directly employing language models as
evaluators (LLM-as-a-Judge) has proven more effective, achieving a higher
correlation with human grading of machine-generated text [24, 14, 16, 2, 9, 3,
20]. Moreover, follow-up studies have demonstrated that employing detailed
evaluation criteria not only yields more accurate judgments but also enhances
interpretability [22, 12, 9, 11, 13, 10]. Inspired by this, our work employs
five specific evaluation axes: grammaticality, fluency, coherence,
consistency, and relevance. On the application side, prior works have used
LLM-as-a-Judge to evaluate various types of LLM-generated text, including the
soundness of research proposals [19], toxicity in life advice [8], and the
factuality of medical documents [7]. Our work distinguishes itself from these
examples as we evaluate human-written text across 15 distinct types of writing
from students.

In the field of adopting LLMs to enhance education, prior works have primarily
focused on distinguishing between LLM-generated and human-generated text [17,
1, 4]. While preventing the overuse of LLMs is crucial, it is equally
important to explore better educational models for using LLMs wisely. Our work
suggests that LLMs can be effectively utilized as tools to enhance students’
writing by providing verbal feedback on their writing.

##  3 Experimental Setting

In this section, we describe (1) the process by which we gathered and
preprocessed 100 human-written texts, (2) our assessment pipeline for
acquiring judgments from LLM evaluators, and (3) how we asked each student to
determine whether the judgments from LLM evaluators were reasonable.

###  3.1 Dataset Construction

Table 1: Statistics of age distribution of the 100 student-written texts. Age | Number of Instances
---|---
11 | 16
13 | 18
14 | 4
17 | 33
18 | 26
19 | 3

We gathered 32 participants, all of whom are Korean students. Each participant
was asked to provide a free-form text that they had manually written without
using LLMs, along with instructions for writing that text. This includes
writings from the past, for which we labeled the age at which the text was
written. We excluded texts that were part of classification tasks or were
short answers. After this process, the main authors categorized the text into
15 types of writing, including Book Report, Descriptive Essay, Process Essay,
Reflective Essay, Story Writing, Play Script, Linguistic Report, Scientific
Report, Presentation Script, Problem Creation, Argumentative Essay,
Presentation Report, Diary, Self Introduction Essay, and Letter.

The main authors manually typed and revised the texts based on the
instructions, as these documents are typically long and in PDF format. To
maintain the quality of the text, we did not revise the content or grammar.
Instead, we manually corrected line break issues using an online JSON
formatting tool 222https://jsoneditoronline.org/ to ensure compatibility of
the dataset. Each instance consists of six components: student id (ranging
between 0 and 31 to distinguish the writer of each text), age (ranging between
11 and 19, based on the age when the writer composed the text, not their
current age), language (either Korean or English), type of writing (among the
15 categories listed above), input (the instruction to write the corresponding
text), and output (the text that each student wrote). The statistics of the
age, type of writing, and language are shown in Table 1, Table 3, and Table 2.

Table 2: Statistics of age distribution of the 100 student-written texts. Type of Writing | Number of Instances
---|---
Book Report | 17
Descriptive Essay | 22
Process Essay | 4
Reflective Essay | 7
Story Writing | 1
Play Script | 3
Linguistic Report | 1
Scientific Report | 13
Presentation Script | 9
Problem Creation | 2
Argumentative Essay | 11
Presentation Report | 2
Diary | 5
Self Introduction Essay | 2
Letter | 1

###  3.2 Evaluation Pipeline

We mainly follow the evaluation pipeline of Prometheus [9, 11]. Specifically,
we utilize the prometheus-eval library 333https://github.com/prometheus-
eval/prometheus-eval, following the default hyper-parameters, and add five
custom score rubrics to evaluate texts written by students. The code and
prompts are provided in supplementary material. We use GPT-4-Turbo-April as
the evaluator for conducting 500 judgments. The judgments consist of verbal
feedback, which highlights the strengths and weaknesses of the student’s text
with respect to each evaluation criterion, and a scoring decision, which is an
integer between 1 and 5. The criteria are as follows: the fluency criterion
assesses if the text fluent and easy to read, considering it is written by a
Korean student. The coherence criterion evaluates whether the text is coherent
and logically organized, considering it is written by a Korean student. The
consistency criterion examines if the text is consistent in terms of style,
tone, and tense. The relevance criterion checks if the text is relevant to the
given instruction or topic. Lastly, the grammaticality criterion assesses
whether the text demonstrates proper grammatical usage.

Table 3: Statistics of age distribution of the 100 student-written texts. Written Language | Number of Instances
---|---
Korean | 63
English | 37

###  3.3 Verification of Evaluation Results

To verify if the evaluation results from Subsection 3.2 are valid, we
distribute the verbal feedback and scoring decisions back to the 32 student
participants who provided their writing. We ask if the scoring decisions and
the verbal feedback were valid or not (i.e., the scoring as well as the verbal
feedback is overly critical or overly optimistic). Each participant were paid
$ 10 for providing each writing piece and verifying the judgments (15 minutes
per instance). Although we were unable to confirm whether students’ writing
improved through direct revisions due to cost limitations, the goal was to
indirectly verify if they could improve their writing by reviewing whether the
feedback was valid. Future work could involve experimenting with students
directly revising their writing.

##  4 Experimental Results & Discussions

Table 4: GPT-4 evaluation results of 100 student-written texts across 5
evaluation criteria, and posthoc human meta-evaluation results on GPT-4’s
judgment. Across all criteria, humans verified that on 77% - 93% of the time,
the judgments were reasonable, which supports the claim that LLM-as-a-Judge
could be utilized to pinpoint the strength and weakness of each student’s
writing.

Criteria | LLM-as-a-Judge Evaluation Results | Human Meta-Evaluation Results
---|---|---
Score 1 | Score 2 | Score 3 | Score 4 | Score 5 | Avg Score | Valid | Overly Critical | Overly Optimistic
Grammaticality | 0 | 6 | 28 | 42 | 11 | 3.63 | 87% | 8% | 5%
Fluency | 0 | 3 | 72 | 23 | 2 | 3.24 | 93% | 6% | 1%
Coherence | 1 | 4 | 34 | 39 | 22 | 3.77 | 81% | 13% | 6%
Consistency | 1 | 1 | 20 | 53 | 25 | 4.00 | 82% | 4% | 14%
Relevance | 7 | 9 | 13 | 30 | 41 | 3.89 | 77% | 4% | 19%

#### Can LLM-as-a-Judge provide reasonable judgments?

The GPT-4 evaluation results and post-hoc human meta-evaluation results are
shown in Table 4 (on the next page). First, when examining GPT-4’s scoring
decisions for the same fixed 100 texts, consistency and relevance received
higher scores, while fluency received lower scores. Second, in the meta-
evaluation results conducted by the students, fluency had the highest
agreement on the validity of the judgment. Given that fluency achieved the
lowest score and humans determined it to be the most valid (93%), this
supports the claim that the LLM-as-a-Judge pipeline could be utilized to
enhance students’ writing. This finding is straightforward, considering that
the participants we tested are all Korean students who do not use English as
their first language. Also, for more objective evaluation criteria such as
grammaticality, the validity was high (87%).

Table 5: Human meta-evaluation results on GPT-4’s judgment for each writing category. Validity ratio is higher on more objective texts while lower on subjective texts. Type of Writing | Ratio of Validity
---|---
Process Essay | 100.00%
Descriptive Essay | 95.45%
Scientific Report | 92.31%
Self Introduction Essay | 50.00%
Argumentative Essay | 36.36%
Diary | 20.00%

#### Is there a type of writing where LLM-as-a-Judge does not function
properly?

Considering that our text collection consists of a wide variety of writings,
we further analyzed the patterns of which writings were deemed to consist of
more accurate judgments. Table 5 shows the top 1, 2, and 3 and lowest 1, 2,
and 3 validity ratios for the writing categories (excluding categories with
only one text). The validity ratio tends to be higher for texts with more
objective characteristics, including process essays, descriptive essays, and
scientific reports, while it is lower for self-introduction essays,
argumentative essays, and diaries. Notably, we observed that for subjective
tasks, texts are often written in a more informal tone. However, coherence and
consistency criteria, which should not consider informality, pointed this out
as a limitation, indicating that LLM-as-a-Judge does not function as intended
in these cases. We leave the development of better evaluation frameworks to
assess human-written text to assist their writing for future work.

Table 6: GPT-4 evaluation results for each age group across 5 criteria. Higher scores are given to senior students over junior students. Criteria | Age 11-14 | Age 17-19
---|---|---
Grammaticality | 3.34 | 3.95
Fluency | 3.08 | 3.25
Coherence | 3.24 | 3.95
Consistency | 3.53 | 4.23
Relevance | 3.54 | 3.95

#### Is GPT-4 capable of distinguishing between writings from senior and
junior students?

The average scores provided by GPT-4-Turbo evaluator is shown in Table 6.
Although it is not strictly accurate since the participants’ writings were not
compared based on the same instructions, we generally observed a difference in
scores between the two groups across all criteria. This indicates that the
LLM-as-a-Judge pipeline tends to give higher scores to senior students. This
finding indirectly supports the reliability of LLM-as-a-Judge, based on the
premise that older students are capable of writing more flexibly.

##  5 Conclusion

Recently, LLM-as-a-Judge has emerged as an effective paradigm for evaluating
LLM responses to subjective questions. In this work, we extend this paradigm
to investigate whether it can also effectively evaluate human-written texts
through preliminary experiments. We collected and evaluated 100 essays written
by a total of 32 Korean students of various ages, and then had each student
verify whether the evaluation results were reasonable. The experimental
results confirmed that LLM-as-a-Judge could serve as a good evaluator for more
objective types of writing, as well as for fluency and grammaticality
criteria. Additionally, we identified areas that need further improvement. We
hope that our work will serve as a foundation for future research exploring
the use of LLM-as-a-Judge to evaluate human-written texts more broadly.

## Acknowledgement

The first author, currently a highschool student, have proposed the idea for
this paper, conducted all the experiments, and wrote the paper. The
corresponding author have provided feedback throughout the project by having a
weekly 2 hour meeting.

## References

  * Bernabei et al. [2023] Margherita Bernabei, Silvia Colabianchi, Andrea Falegnami, and Francesco Costantino.  Students’ use of large language models in engineering education: A case study on technology acceptance, perceptions, efficacy, and detection chances.  _Computers and Education: Artificial Intelligence_ , 5:100172, 2023.
  * Chan et al. [2023] Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu.  Chateval: Towards better llm-based evaluators through multi-agent debate.  _arXiv preprint arXiv:2308.07201_ , 2023.
  * Dubois et al. [2024] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto.  Length-controlled alpacaeval: A simple way to debias automatic evaluators.  _arXiv preprint arXiv:2404.04475_ , 2024.
  * Gao et al. [2024] Chujie Gao, Dongping Chen, Qihui Zhang, Yue Huang, Yao Wan, and Lichao Sun.  Llm-as-a-coauthor: The challenges of detecting llm-human mixcase.  _arXiv preprint arXiv:2401.05952_ , 2024.
  * Henderson et al. [2019] Michael Henderson, Tracii Ryan, and Michael Phillips.  The challenges of feedback in higher education.  _Assessment & Evaluation in Higher Education_, 2019.
  * Holtzman et al. [2019] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi.  The curious case of neural text degeneration.  _arXiv preprint arXiv:1904.09751_ , 2019.
  * Joseph et al. [2024] Sebastian Antony Joseph, Lily Chen, Jan Trienes, Hannah Louisa Göke, Monika Coers, Wei Xu, Byron C Wallace, and Junyi Jessy Li.  Factpico: Factuality evaluation for plain language summarization of medical evidence.  _arXiv preprint arXiv:2402.11456_ , 2024.
  * Kim et al. [2023a] Minbeom Kim, Jahyun Koo, Hwanhee Lee, Joonsuk Park, Hwaran Lee, and Kyomin Jung.  Lifetox: Unveiling implicit toxicity in life advice.  _arXiv preprint arXiv:2311.09585_ , 2023a.
  * Kim et al. [2023b] Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al.  Prometheus: Inducing fine-grained evaluation capability in language models.  _arXiv preprint arXiv:2310.08491_ , 2023b.
  * Kim et al. [2024a] Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, et al.  The biggen bench: A principled benchmark for fine-grained evaluation of language models with language models.  _arXiv preprint arXiv:2406.05761_ , 2024a.
  * Kim et al. [2024b] Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo.  Prometheus 2: An open source language model specialized in evaluating other language models.  _arXiv preprint arXiv:2405.01535_ , 2024b.
  * Kim et al. [2023c] Tae Soo Kim, Yoonjoo Lee, Jamin Shin, Young-Ho Kim, and Juho Kim.  Evallm: Interactive evaluation of large language model prompts on user-defined criteria.  _arXiv preprint arXiv:2309.13633_ , 2023c.
  * Lee et al. [2024] Seongyun Lee, Seungone Kim, Sue Hyun Park, Geewook Kim, and Minjoon Seo.  Prometheus-vision: Vision-language model as a judge for fine-grained evaluation.  _arXiv preprint arXiv:2401.06591_ , 2024.
  * Li et al. [2023] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Alpacaeval: An automatic evaluator of instruction-following models.  https://github.com/tatsu-lab/alpaca_eval, 2023.
  * Lin [2004] Chin-Yew Lin.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81, 2004.
  * Liu et al. [2023] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu.  Gpteval: Nlg evaluation using gpt-4 with better human alignment.  _arXiv preprint arXiv:2303.16634_ , 2023.
  * Orenstrakh et al. [2023] Michael Sheinman Orenstrakh, Oscar Karnalim, Carlos Anibal Suarez, and Michael Liut.  Detecting llm-generated text in computing education: A comparative study for chatgpt cases.  _arXiv preprint arXiv:2307.07411_ , 2023.
  * Papineni et al. [2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318, 2002.
  * Shao et al. [2024] Yijia Shao, Yucheng Jiang, Theodore A. Kanell, Peter Xu, Omar Khattab, and Monica S. Lam.  Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models.  In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 2024.
  * Verga et al. [2024] Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis.  Replacing judges with juries: Evaluating llm generations with a panel of diverse models.  _arXiv preprint arXiv:2404.18796_ , 2024.
  * Wisniewski et al. [2020] Benedikt Wisniewski, Klaus Zierer, and John Hattie.  The power of feedback revisited: A meta-analysis of educational feedback research.  _Frontiers in psychology_ , 10:487662, 2020.
  * Ye et al. [2023] Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo.  Flask: Fine-grained language model evaluation based on alignment skill sets.  _arXiv preprint arXiv:2307.10928_ , 2023.
  * Zhang et al. [2019] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi.  Bertscore: Evaluating text generation with bert.  _arXiv preprint arXiv:1904.09675_ , 2019.
  * Zheng et al. [2023] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _arXiv preprint arXiv:2306.05685_ , 2023.