# Laminar vortex dynamics around forward-swept wings

Kai Zhang1 Email<EMAIL_ADDRESS>Kunihiko Taira2  1 Department of
Mechanical and Aerospace Engineering, Rutgers University, Piscataway, 08854,
NJ 2 Department of Mechanical and Aerospace Engineering, University of
California, Los Angeles, 90095, CA

###### Abstract

Forward-swept wings offer unique advantages in the aerodynamic performance of
air vehicles. However, the low-Reynolds-number characteristics of such wings
have not been explored in the past. In this work, we numerically study laminar
separated flows over forward-swept wings with semi aspect ratios $sAR=0.5$ to
2 at a chord-based Reynolds number of 400. Forward-swept wings generate wakes
that are significantly different from those of backward-swept wings. For low-
aspect-ratio forward wings, the wakes remain steady due to the strong downwash
effects induced by the tip vortices. For larger aspect ratio, the downwash
effects weaken over the inboard regions of the wing, allowing unsteady vortex
shedding to occur. Further larger aspect ratio allows for the formation of
streamwise vortices for highly-swept wings, stabilizing the flow. Forward-
swept wings can generate enhanced lift at high angles of attack than the
unswept and backward-swept wings, with the cost of high drag. We show through
force element analysis that the increased lift of forward-swept wings is
attributed to the vortical structure that is maintained by the tip-vortex-
induced downwash over the outboard region of wing span. The current findings
offer a detailed understanding of the sweep effects on laminar separated flows
over forward-swept wings, and invite innovative designs of high-lift devices.

##  1 Introduction

Low-Reynolds-number aerodynamics of finite-aspect-ratio lifting surfaces have
been widely studied for developing micro air vehicles and understanding
biological flights. The wakes of these small-scale wings embodies rich flow
physics comprised of unsteady separation, vortex formation, and wake
interactions, which are influenced by the Reynolds number, platform shape, and
unsteady maneuvers (Mueller & DeLaurier, 2003; Buchholz & Smits, 2006; Visbal
et al., 2013; Winslow et al., 2018; Eldredge & Jones, 2019). The sweep angle
has also been shown to play a key role in shaping the wake dynamics
(Küchemann, 1953; Harper & Maki, 1964; Visbal & Garmann, 2019; Zhang et al.,
2020a). Most of these studies have focused on the backward-swept wings.

Forward-swept wings, although less common in air vehicle designs, offer unique
advantages that make them suitable for agile flights at high angles of attack.
In general, swept wings have the property that their aft sections stall first.
In conventional backward-swept wings, stall occurs first near the wing tip
(Harper & Maki, 1964; Black, 1956; Zhang et al., 2020a), resulting in a loss
of aileron control. For forward-swept wings, since stall commences at the
inboard region, ailerons can still be effective at high angles of attack to
offer high maneuverability. This feature has been demonstrated by the Grumman
X-29 experimental aircraft, which gave pilots excellent control response up to
$45^{\circ}$ angle of attack (Putnam, 1984). In addition, many aerial animals
have been observed to sweep their wings forward during slow flights because
the wings are required to operate at high angles of attack providing high lift
to support body weight (Thomas & Taylor, 2001). Many species also exhibit
forward wings during high angle-of-attack perching and snatching maneuvers
(Manchester et al., 2017).

The aerodynamic benefits of forward-swept wings have attracted a number of
studies on their wake characteristics. Breitsamter & Laschka (2001) conducted
extensive experimental investigations on the flows over forward-swept wings at
$Re=4.6\times 10^{5}$. They observed that the wing tip vortex and the leading
edge vortex with an opposite sense of rotation dominate the flowfield. The
leading-edge vortex can burst at moderate angles of attack in the outer wing
region. Traub & Lawrence (2009) studied these flows over a thin $65^{\circ}$
swept wing in forward and backward sweep configurations, both revealing vortex
dominated flows. Surface flow visualization showed large extents of flow
separation near the wing root of the forward-swept wings, which feature
significantly lower lift compared to the aft swept wings. Lee & Ko (2009)
measured the near-field tip-vortex flow behind a forward-swept wing at
$Re=1.74\times 10^{5}$. Recently, Setoguchi & Kanazaki (2020) investigated the
separated vortical flows from the forward-swept wings at low speed and high
angle of attack conditions through Reynolds-averaged Navier-Stokes
simulations. They asserted that forward-swept wings exhibit benefits in terms
of stall characteristics, since separated vortices remain on the outboard wing
up to high angles of attack.

The above studies have revealed the aerodynamic characteristics of forward-
swept wings at relatively high Reynolds numbers. However, a detailed
characterization of the wake dynamics of forward-swept wings at low Reynolds
number remains unexplored. A thorough understanding of the low-$Re$ wakes of
forward-swept wings is not only of fundamental importance, but also lays the
foundation for design of small-scale high-lift devices. In this work, we
perform a large number of direct numerical simulations to study the three-
dimensional separated flows over forward-swept finite-aspect-ratio wings at a
Reynolds number of 400. The present study, together with our previous work on
backward-swept wings (Zhang et al., 2020a), offers a comprehensive coverage of
sweep effects on separated flows over finite-aspect-ratio wings. In what
follows, we present the computational setup and its validation in §2. The
results are discussed in §3, where we provide descriptions of the wake
vortical structures and the aerodynamic forces. We also present a force
element analysis to identify key wake structures that are responsible for lift
generation. We conclude this study by summarizing our findings in §4.

##  2 Computational setup

Figure 1: Case setup for flows over swept wings

We study the incompressible flows over finite-aspect-ratio forward-swept wings
with a NACA 0015 cross section. A schematic of the wing is shown in figure 1.
The wings are subjected to uniform flow with velocity $U_{\infty}$ in the $x$
direction. The $z$ axis aligns with the spanwise direction of the unswept
wing, and the $y$ axis points in the direction of lift. The sweep angle
$\Lambda$ is defined as the angle between the $z$ axis and the leading edge.
The symmetry boundary condition is prescribed along the midspan. Denoting half
wing span as $b$, the semi aspect ratio is defined as $sAR=b/c$, where $c$ is
the chord length. The Reynolds number, defined as $Re\equiv U_{\infty}c/\nu$
($\nu$ is the kinematic viscosity of the fluid), is fixed at 400. In what
follows, all the spatial variables are normalized by the chord length $c$,
velocity by $U_{\infty}$, and time by $c/U_{\infty}$. The lift and drag
coefficients are defined as $C_{L}=F_{L}/(\rho U_{\infty}^{2}bc/2)$ and
$C_{D}=F_{D}/(\rho U_{\infty}^{2}bc/2)$, where $F_{L}$ and $F_{D}$ are the
aerodynamic forces exerted on the semi-span wing in the $y$ and $x$
directions, respectively, and $\rho$ is the fluid density.

Figure 2: Comparisons of force coefficients ($a,b$) and wake vortical
structures ($c$) computed from _Cliff_ and _pimpleFoam_ for
$(sAR,\alpha,\Lambda)=(2,20^{\circ},0^{\circ})$ and
$(sAR,\alpha,\Lambda)=(2,20^{\circ},45^{\circ})$. The inset tables in ($a,b$)
summarize the time-averaged force coefficients $\overline{C_{L}}$ and
$\overline{C_{D}}$. The vortical structures in ($c$) are visualized with
isosurfaces of $Qc^{2}/U_{\infty}^{2}=1$, where $Q$ is the second invariant of
the velocity gradient tensor.

The finite-volume-based flow solver pimpleFoam of the OpenFOAM toolbox (Weller
et al., 1998) is used to simulate the flows with second-order spatial and
temporal accuracy. We employed a similar grid setup from our previous work on
backward-swept wings (Zhang et al., 2020a, b), for which a commercial CFD code
Cliff (CharLES package) was used (Ham & Iaccarino, 2004; Ham et al., 2006). To
validate the present numerical setup, we compare the force coefficients of the
unswept wing $(sAR,\alpha,\Lambda)=(2,20^{\circ},0^{\circ})$ and backward-
swept wing $(sAR,\alpha,\Lambda)=(2,20^{\circ},45^{\circ})$ computed from
Cliff and pimpleFoam in figures 2($a$) and ($b$). The time traces of the
forces obtained from the two solvers agree well with each other, and their
time-averaged values differ less than 1%. In addition, the vortical structures
of the backward-swept wing computed from both solvers match almost perfectly
as shown in figure 2($c$), validating the present computational results.

##  3 Results

###  3.1 Wake vortical structures

Figure 3: Representative wake vortical structures for forward-swept wings,
visualized by isosurfaces of $Qc^{2}/U_{\infty}^{2}=1$. Figure 4:
Classification of wake vortical structures behind swept wings for ($a$)
$sAR=0.5$, ($b$) $sAR=1$ and ($c$) $sAR=2$.  : steady flow due to tip effects;
$\mathop{\raisebox{-1.18399pt}{$\leavevmode\hbox to6.02pt{\vbox
to8.82pt{\pgfpicture\makeatletter\hbox{\hskip 0.42679pt\lower-1.65387pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{}{}{{}}\pgfsys@setlinewidth{0.43056pt}\pgfsys@invoke{
}\pgfsys@roundjoin\pgfsys@invoke{ }{}{{}}{} {}{} {}{} {}{} {}{}
{}\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0.850980392156863,0.325490196078431,0.0980392156862745}\pgfsys@color@rgb@stroke{0.850980392156863}{0.325490196078431}{0.0980392156862745}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.85358pt}\pgfsys@invoke{
}\definecolor[named]{pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1}\pgfsys@invoke{
}{}\pgfsys@moveto{0.0pt}{2.75554pt}\pgfsys@lineto{2.58333pt}{6.73817pt}\pgfsys@lineto{5.16666pt}{2.75554pt}\pgfsys@lineto{2.58333pt}{-1.22708pt}\pgfsys@lineto{0.0pt}{2.75554pt}\pgfsys@closepath\pgfsys@fillstroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}$}}$: unsteady
shedding near midspan;  : steady flow due to midspan effects;  : unsteady
shedding near wing tip;  : steady flow with streamwise vortices.  : steady
flow due to low $Re$. The dashed lines denote the approximate boundaries
between steady (filled symbols) and unsteady (empty symbols) flows. The
vortical structures are visualized by isosurfaces of $Qc^{2}/U_{\infty}^{2}=1$
for representative cases.

The wake vortical structures behind high-incidence forward-swept wings can
take three major forms, as shown in figure 3. These wakes are classified as
($a$) steady flow due to tip vortices ( ), ($b$) unsteady vortex shedding near
the midspan ($\mathop{\raisebox{-1.18399pt}{$\leavevmode\hbox to6.02pt{\vbox
to8.82pt{\pgfpicture\makeatletter\hbox{\hskip 0.42679pt\lower-1.65387pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{}{}{{}}\pgfsys@setlinewidth{0.43056pt}\pgfsys@invoke{
}\pgfsys@roundjoin\pgfsys@invoke{ }{}{{}}{} {}{} {}{} {}{} {}{}
{}\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0.850980392156863,0.325490196078431,0.0980392156862745}\pgfsys@color@rgb@stroke{0.850980392156863}{0.325490196078431}{0.0980392156862745}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.85358pt}\pgfsys@invoke{
}\definecolor[named]{pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1}\pgfsys@invoke{
}{}\pgfsys@moveto{0.0pt}{2.75554pt}\pgfsys@lineto{2.58333pt}{6.73817pt}\pgfsys@lineto{5.16666pt}{2.75554pt}\pgfsys@lineto{2.58333pt}{-1.22708pt}\pgfsys@lineto{0.0pt}{2.75554pt}\pgfsys@closepath\pgfsys@fillstroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}$}}$), and $(c)$
steady flow with streamwise vortices ( ). We map out these flows over the
$\Lambda$-$\alpha$ space for different aspect ratios in figure 4, in which the
backward-swept wing wakes (Zhang et al., 2020a) are also included. The wakes
for $\alpha\leq 12^{\circ}$ are clustered into one group as they are steady at
$Re=400$ regardless of aspect ratio and sweep angle.

The most prominent feature of the forward-swept wing wakes is the presence of
the tip vortices, as shown in figure 3. In comparison, the tip vortices can be
suppressed for backward-swept wings. This is due to the outboard spanwise flow
induced by the backward-swept wing that counteracts the roll-up of flow around
the tip (Zhang et al., 2020a). With increasing forward sweep angle, the tip
vortices grow stronger with increasing forward sweep angle, as observed in
figure 4. The wake dynamics of low-aspect-ratio forward-swept wings are
significantly influenced by the tip vortices.

At $sAR=0.5$, the tip vortices induce strong downwash effects over the entire
span of the wings, acting to suppress the shedding of the leading-edge vortex
sheet (Taira & Colonius, 2009; DeVoria & Mohseni, 2017; Zhang et al., 2020a,
b). As a result, the wakes of forward-swept wings with $sAR=0.5$ are mostly
steady, as shown in figure 4($a$). The leading-edge vortex sheet rolls up near
the midspan into a dome-like shape. Together with the pair of tip vortices,
the vortical structures of these forward-swept wings resemble the shape of a
joker’s hat. It is not until $\alpha=30^{\circ}$ that the unsteady vortex
shedding emerge. The forward-swept wing can achieve steady flow at smaller
sweep angle than the backward-swept case, as the former is subjected stronger
downwash effects from the tip vortices.

For higher-aspect-ratio forward-swept wings, the downwash induced by the tip
vortex becomes weaker near the midspan, allowing the detachment of the
leading-edge vortex sheet. The flow thus features unsteady vortex shedding as
shown in figure 3$(b)$. This type of flow prevails for high-incidence forward-
swept wings with $sAR=1$, as shown in figure 4($b$). The stability boundary in
the $\Lambda$-$\alpha$ space at $sAR=1$ is asymmetric with respect to
$\Lambda=0$, with the backward-swept wings exhibiting steady flows over a
larger region of the parameter space. These steady flows of backward-swept
wings are achieved due to the formation of a pair of vortical structures near
the midspan on the suction side, which impose downward velocity to each other
and stabilize the wake (Zhang et al., 2020a). Such midspan effects are
localized to the inboard region of backward-swept wings. For cases with large
angle of attack and higher aspect ratio, the backward-swept wings generate
unsteady vortex shedding near the tip region, which is not observed for
forward-swept wings.

For $sAR=2$, the increased span allows for the formation of streamwise
vortices (in addition to the tip vortices) over the inboard region, acting as
a wake stabilizing mechanism for both forward- and backward-swept wings with
high sweep angles. For the forward-swept wings, two pairs of such streamwise
vortices are observed for the $sAR=2$ wings. The first pair resides in the
near wake of the midspan region, and appear detached from the rest of the
vortical structures as shown in figure 3($c$). The other pair on trailing edge
grows from the tip region and contracts towards the midspan. This set of
streamwise vortices is similar to those found in the backward-swept cases, in
which the streamwise vortices grow from midspan and trails toward the tip
(Zhang et al., 2020a). This difference reflects the reverse of the spanwise
flow as the wings are swept from backward to forward. Since the formation of
such streamwise vortices is universal for inclined slender bodies with
inhomogeneous end boundary conditions (Sarpkaya, 1966; Thomson & Morrison,
1971; Zhang et al., 2020a), the stability boundary at $sAR=2$ in the
$\Lambda$-$\alpha$ space appear almost symmetric for both the forward- and
backward-swept wings.

###  3.2 Aerodynamic forces and moments

Figure 5: Time-averaged aerodynamic force coefficients of forward-swept wings.
$(a)$ lift coefficient $\overline{C_{L}}$; $(b)$ drag coefficient
$\overline{C_{D}}$; $(c)$ lift-to-drag ratio $\overline{C_{L}/C_{D}}$ and
$(d)$ pitch coefficient $\overline{C_{M}}$ about the quarter chord on the mid-
section between tip and root planes. The black dashed lines in the
$\overline{C_{L}}$ plots represent the inviscid limit for unswept wings. Left,
middle, and right columns show data for $sAR=0.5$, 1, and 2, respectively.

The time-averaged aerodynamic force coefficients $\overline{C_{L}}$,
$\overline{C_{D}}$, the lift-to-drag ratio $\overline{C_{L}/C_{D}}$, and the
pitch coefficient $\overline{C_{M}}$ of the swept wings are shown in figure 5.
Here, the pitch moment coefficient is defined as $C_{M}=M/(\rho
U_{\infty}^{2}bc^{2}/2)$, where $M$ is the half-span pitch moment about the
quarter-chord line on the mid-section $(z=sAR/2)$ between the tip and the root
planes, and positive $M$ acts to pitch the wing in the nose-up direction. For
the lift coefficients shown in figure 5($a$), the inviscid lift limit of
finite-aspect-ratio unswept wings calculated with
$C_{L}=2\pi\alpha/(\sqrt{1+(1/sAR)^{2}}+1/sAR)$ is shown (Helmbold, 1942).

The lift coefficients increase almost linearly for low angles of attack
($\alpha\lesssim 12^{\circ}$). The introduction of forward sweep has a
negative impact on $\overline{C_{L}}$ across different aspect ratios for these
low-incidence wings, while backward sweep can enhance lift for the $sAR=0.5$
and 1 wings. On the other hand, at high angles of attack ($\alpha\gtrsim
20^{\circ}$), the forward-swept wings generally feature higher
$\overline{C_{L}}$ than the backward-swept and unswept wings. This phenomenon
is associated with strong downwash effect provided by the tip vortices in
forward-swept wings, which will be discussed in §3.3.

The drag coefficients shown in figure 5$(b)$ generally exhibit an quadratic
growth with angle of attack over the studied range. At low angles of attack,
the drag coefficients remain close to each other among wings with different
sweep angles. At high $\alpha$, accompanying the high lift of forward-swept
wings, $\overline{C_{D}}$ also increases with forward sweep angle. As a
result, the lift-to-drag ratios of the forward-swept wings are generally
smaller than the backward-swept cases, as shown in figure 5($c$).

The pitch moments for both the forward- and backward-swept wings are positive,
and increase with the angle of attack monotonically, as shown in figure
5($d$). This suggests an inherent pitch instability with nose-up motion of the
swept wings around their mid-sections. As the aspect ratio increases, the
difference in pitch moments among cases with different sweep angle enlarges.
The pitch moments of forward-swept wings are generally smaller than the
backward-swept cases for wings with larger aspect ratios. In particular, for
$sAR=2$, both the forward- and backward-swept wings feature higher pitch
moments than the unswept cases, indicating highly uneven force distributions
along the wing span.

Figure 6: Sectional force distributions for $(sAR,\alpha)=(2,20^{\circ})$
wings with different sweep angles. $(a)$ Sectional lift coefficients. The
inset shows the lift distributions of forward- and backward-swept wings
operating at $Re=2\times 10^{7}$ (Gudmundsson, 2013; Vos & Farokhi, 2015).
$(b)$ Sectional drag distribution.

The sectional lift and drag coefficients for both forward- and backward-swept
wings with $(sAR,\alpha)=(2,20^{\circ})$ are shown in figure 6. As discussed
in Zhang et al. (2020a), the backward-swept wings feature higher sectional
lift at the inboard region, where a pair of vortical structures are formed on
the suction side due to the three-dimensional midspan effects. For the
forward-swept wings, the sectional lift increases from inboard to around half
chord away from the tip, and then decrease drastically towards the tip. The
high lift near the wing tip of forward-swept wings complements the low
sectional lift in the inboard region, and could lead to an overall increase in
lift for low-aspect-ratio wings as discussed with regards to figure 5($a$).
The fact that the sectional lift are higher at the upwind end of the wing span
(tip for forward-swept wing, and root for backward-swept wing) also explains
the positive pitch moments for both wings. We note that such lift distribution
of low-$Re$ swept wings is opposite to that of high-$Re$ attached flows, in
which the forward-swept wings feature higher lift inboard, and backward-swept
wings features higher lift at outboard (Gudmundsson, 2013; Vos & Farokhi,
2015), as shown in the insets of figure 6$(a)$. Such disparity highlights the
significant difference between the current vortex-dominated flows compared to
the high-$Re$ attached flows.

The drag coefficients of the forward- and backward-swept wings exhibit inverse
distribution along the wing span. In the former case, the sectional drag
increases with the spanwise location and finishes with a surge towards the
tip, while in the latter case, the highest sectional lift is found at the most
inboard section. The higher sectional lift and drag in the outboard region of
forward-swept wings lead to increased bending moments at the root plane,
calling for reinforced structural design.

###  3.3 Force element analysis

Next, let us use the force element theory (Chang, 1992) to identify the flow
structures that are responsible for exerting lift on the wing. To apply this
theory, we compute an auxiliary potential function $\phi_{L}$ (satisfying
$\bm{\nabla}^{2}\phi_{L}=0$) with the boundary condition
$-\bm{n}\cdot\bm{\nabla}\phi_{L}=\bm{n}\cdot\bm{e_{y}}$ on the wing surface
($\bm{e_{y}}$ is the unit vector in the lift direction). By taking the inner
product of the momentum equation for incompressible flow with
$\bm{\nabla}\phi_{L}$ and integrating over the entire fluid domain $V$, the
lift force can be recovered as

$F_{L}=\int_{V}\bm{\omega}\times\bm{u}\cdot\bm{\nabla}\phi_{L}\mathrm{d}V+\frac{1}{\Rey}\int_{\partial
V}\bm{\omega}\times\bm{n}\cdot(\bm{\nabla}\phi_{L}+\bm{e_{y}})\mathrm{d}S.$
(1)

The integrands in the first and second terms on the right hand side of this
equation are the volume and the surface lift elements, respectively. At
$Re=400$, the volume force elements contribute more significantly to the total
force than the surface force elements.

Figure 7: Lift elements for ($a$) forward-swept wing
$(sAR,\alpha,\Lambda)=(2,20^{\circ},-45^{\circ})$ and ($b$) backward-swept
wing $(sAR,\alpha,\Lambda)=(2,20^{\circ},45^{\circ})$. The lift elements from
-1 (blue) to 1 (red) are plotted on the isosurfaces of
$Qc^{2}/U_{\infty}^{2}=1$. The sectional slices shows colormaps of
$L_{e}\in[-1,1]$ at different spanwise locations.

We present the lift elements for forward-swept
$(sAR,\alpha,\Lambda)=(2,20^{\circ},-45^{\circ})$ and backward-swept wings
$(sAR,\alpha,\Lambda)=(2,20^{\circ},45^{\circ})$ on the isosurfaces of
$Qc^{2}/U_{\infty}^{2}=1$ in figure 7. In general, the vortex sheets emanating
from both the leading and trailing edges are associated with positive lift,
while the flow near the pressure side of the wing contribute to negative lift
(Menon & Mittal, 2021). For the forward-swept wing shown in figure 7($a$), the
positive lift elements on the suction side of the wing appear thicker and
closer to the wing surface towards the wing tip than those in the inboard
regions. This is due to the strong tip-vortex-induced downwash effects that
confines the vorticity in the vicinity of the suction side of the wing, as
discussed in §3.1. On the other hand, the backward-swept wing enhances lift by
the formation of a pair of vortical structures near the midspan, where the
concentrated lift elements are attached to the wing surface (Zhang et al.,
2020a). Despite these different mechanisms in maintaining the steady vortical
structures on the suction sides of the wings, both forward- and backward-swept
wings are able to harness separated flows to generate additional vortical lift
at high incidence, thus enhancing their aerodynamic performance.

##  4 Conclusions

We have performed three-dimensional direct numerical simulations to study the
effects of forward sweep on the wake dynamics and aerodynamic characteristics
of finite-aspect-ratio wings at a chord-based Reynolds number of 400, covering
aspect ratios of $sAR=0.5$ to $2$, angles of attack $\alpha=0^{\circ}$ to
$30^{\circ}$, and sweep angle $\Lambda=0^{\circ}$ to $-45^{\circ}$. The flows
over forward-swept wings generally feature a pair of counter-rotating tip
vortices, which shapes the wake dynamics significantly. For low-aspect-ratio
forward-swept wings, the wakes are strongly influenced by the tip-vortex-
induced downwash effects and remains steady. With increasing aspect ratio, the
downwash effects weaken over the inboard regions of the wing, resulting in
unsteady vortex shedding. For wings with higher forward sweep angles,
streamwise vortices develop in the wake, again stabilizeing the flow. While
the introduction of forward sweep can have a negative impact on the lift at
low $\alpha$, the forward-swept wings can experience enhanced lift from the
separated flows, resulting in higher lift than the backward swept wings. This
comes at the cost of high drag, which results in lower lift-to-drag ratios for
forward swept wings. Through a force element analysis, we show that the high
lift of forward swept wings is mainly contributed by the steady vortical
structure near the tip region, which is formed under the downwash induced by
the tip vortices. Together with our previous studies on backward-swept wings,
the present results provide a comprehensive understanding of the sweep effects
on the laminar separated flows over finite-aspect-ratio wings and offer
coverage of less-explored area of low-$Re$ aerodynamic database.

## Acknowledgement

KZ is grateful for the Office of Advanced Research Computing (OARC) at
Rutgers, The State University of New Jersey for providing access to the Amarel
cluster. KT acknowledges support from the US Air Force Office of Scientific
Research (FA9550-17-1-0222 and FA9550-22-1-0174).

## References

  * Black (1956) Black, J. 1956 Flow studies of the leading edge stall on a swept-back wing at high incidence. Aeronaut. J. 60 (541), 51–60.
  * Breitsamter & Laschka (2001) Breitsamter, C. & Laschka, B. 2001 Vortical flowfield structure at forward swept-wing configurations. J. Aircraft 38 (2), 193–207.
  * Buchholz & Smits (2006) Buchholz, J. H. J. & Smits, A. J. 2006 On the evolution of the wake structure produced by a low-aspect-ratio pitching panel. J. Fluid Mech. 546, 433–443.
  * Chang (1992) Chang, C. C. 1992 Potential flow and forces for incompressible viscous flow. Proc. R. Soc. Lond. A 437 (1901), 517–525.
  * DeVoria & Mohseni (2017) DeVoria, A. C. & Mohseni, K. 2017 On the mechanism of high-incidence lift generation for steadily translating low-aspect-ratio wings. J. Fluid Mech. 813, 110–126.
  * Eldredge & Jones (2019) Eldredge, J. D. & Jones, A. R. 2019 Leading-edge vortices: mechanics and modeling. Annu. Rev. Fluid Mech. 51, 75–104.
  * Gudmundsson (2013) Gudmundsson, S. 2013 General aviation aircraft design: Applied Methods and Procedures. Butterworth-Heinemann.
  * Ham & Iaccarino (2004) Ham, F. & Iaccarino, G. 2004 Energy conservation in collocated discretization schemes on unstructured meshes. Annual Research Briefs, Center for Turbulence Research, Stanford University pp. 3–14.
  * Ham et al. (2006) Ham, F., Mattsson, K. & Iaccarino, G. 2006 Accurate and stable finite volume operators for unstructured flow solvers. Annual Research Briefs, Center for Turbulence Research, Stanford University pp. 243–261.
  * Harper & Maki (1964) Harper, C. W. & Maki, R. L. 1964 A review of the stall characteristics of swept wings. Techical Note D-2373. NASA.
  * Helmbold (1942) Helmbold, H. B. 1942 Der unverwundene ellipsenflugel als tragende flanche. Jahrbuch 1942 der Deutch Luftfahrtforsch pp. I111–I113.
  * Küchemann (1953) Küchemann, D. 1953 Types of flow on swept wings - with special reference to free boundaries and vortex sheets. Techical Note Aero 2234. Royal Aircraft Establishment.
  * Lee & Ko (2009) Lee, T. & Ko, L. S. 2009 Vortex wake generated behind a forward swept wing. J. Aircraft 46 (2), 717–721.
  * Manchester et al. (2017) Manchester, Z. R., Lipton, J. I., Wood, R. J. & Kuindersma, S. 2017 A variable forward-sweep wing design for enhanced perching in micro aerial vehicles. AIAA Paper 2017-0011.
  * Menon & Mittal (2021) Menon, K. & Mittal, R. 2021 Significance of the strain-dominated region around a vortex on induced aerodynamic loads. J. Fluid Mech. 918.
  * Mueller & DeLaurier (2003) Mueller, T. J. & DeLaurier, J. D. 2003 Aerodynamics of small vehicles. Annu. Rev. Fluid Mech. 35 (1), 89–111.
  * Putnam (1984) Putnam, T. W. 1984 X-29 flight research program. Technical Memorandum 86025\. NASA.
  * Sarpkaya (1966) Sarpkaya, T. 1966 Separated flow about lifting bodies and impulsive flow about cylinders. AIAA J. 4 (3), 414–420.
  * Setoguchi & Kanazaki (2020) Setoguchi, N. & Kanazaki, M. 2020 Low-speed and high angle of attack aerodynamic characteristics of supersonic business jet with forward swept wing. AIAA Paper 2020-0534.
  * Taira & Colonius (2009) Taira, K. & Colonius, T. 2009 Three-dimensional flows around low-aspect-ratio flat-plate wings at low Reynolds numbers. J. Fluid Mech. 623, 187–207.
  * Thomas & Taylor (2001) Thomas, A. L. R. & Taylor, G. K. 2001 Animal flight dynamics I: Stability in gliding flight. J. Theor. Biol. 212 (3), 399–424.
  * Thomson & Morrison (1971) Thomson, K. D. & Morrison, D. F. 1971 The spacing, position and strength of vortices in the wake of slender cylindrical bodies at large incidence. J. Fluid Mech. 50 (4), 751–783.
  * Traub & Lawrence (2009) Traub, L. W. & Lawrence, J. 2009 Aerodynamic characteristics of forward and aft swept arrow wings. J. Aircraft 46 (4), 1454–1458.
  * Visbal et al. (2013) Visbal, M., Yilmaz, T. O. & Rockwell, D. 2013 Three-dimensional vortex formation on a heaving low-aspect-ratio wing: Computations and experiments. J. Fluids Struct. 38, 58–76.
  * Visbal & Garmann (2019) Visbal, M. R. & Garmann, D. J. 2019 Effect of sweep on dynamic stall of a pitching finite-aspect-ratio wing. AIAA J. 57 (8), 3274–3289.
  * Vos & Farokhi (2015) Vos, R. & Farokhi, S. 2015 Introduction to transonic aerodynamics. Springer.
  * Weller et al. (1998) Weller, H. G., Tabor, G., Jasak, H. & Fureby, C. 1998 A tensorial approach to computational continuum mechanics using object-oriented techniques. Comput. Phys. 12 (6), 620–631.
  * Winslow et al. (2018) Winslow, J., Otsuka, H., Govindarajan, B. & Chopra, I. 2018 Basic understanding of airfoil characteristics at low Reynolds numbers ($10^{4}-10^{5}$). J. Aircraft 55 (3), 1050–1061.
  * Zhang et al. (2020a) Zhang, K., Hayostek, S., Amitay, M., Burtsev, A., Theofilis, V. & Taira, K. 2020a Laminar separated flows over finite-aspect-ratio swept wings. J. Fluid Mech. 905, R1.
  * Zhang et al. (2020b) Zhang, K., Hayostek, S., Amitay, M., He, W., Theofilis, V. & Taira, K. 2020b On the formation of three-dimensional separated flows over wings under tip effects. J. Fluid Mech. 895, A9.


Further author information: (Send correspondence to Edoardo Cucchetti)
Edoardo Cucchetti: E-mail<EMAIL_ADDRESS>
# Testing the X-IFU calibration requirements: an example for quantum
efficiency and energy resolution

Edoardo Cucchetti  IRAP, Université de Toulouse, CNRS, CNES, 9 Av. du Colonel
Roche, 31400 Toulouse, France  François Pajot  IRAP, Université de Toulouse,
CNRS, CNES, 9 Av. du Colonel Roche, 31400 Toulouse, France  Etienne
Pointecouteau  IRAP, Université de Toulouse, CNRS, CNES, 9 Av. du Colonel
Roche, 31400 Toulouse, France  Philippe Peille  CNES, 18 Avenue Edouard Belin
31400 Toulouse, France  Gabriele Betancourt-Martinez  IRAP, Université de
Toulouse, CNRS, CNES, 9 Av. du Colonel Roche, 31400 Toulouse, France  Stephen
J. Smith  NASA/Goddard Space Flight Center, 8800 Greenbelt Rd., Greenbelt, MD
20771, United States  CRESST and University of Maryland, Baltimore County, MD
21250, USA; University of Maryland, College Park, MD 20742,USA  Marco Barbera
Università degli Studi di Palermo, Dipartimento di Fisica e Chimica, Via
Archirafi 36, 90123, Palermo, Italy  Istituto Nazionale di Astrofisica,
Osservatorio Astronomico di Palermo,
Piazza del Parlamento 1, 90134 Palermo, Italy  Megan E. Eckart  NASA/Goddard
Space Flight Center, 8800 Greenbelt Rd., Greenbelt, MD 20771, United States
Simon R. Bandler  NASA/Goddard Space Flight Center, 8800 Greenbelt Rd.,
Greenbelt, MD 20771, United States  Caroline A. Kilbourne  NASA/Goddard Space
Flight Center, 8800 Greenbelt Rd., Greenbelt, MD 20771, United States  Massimo
Cappi  INAF-IASF Bologna, Via Gobetti 101, 40129 Bologna, Italy  Didier Barret
IRAP, Université de Toulouse, CNRS, CNES, 9 Av. du Colonel Roche, 31400
Toulouse, France

###### Abstract

With its array of 3840 Transition Edge Sensors (TESs) operated at 90 mK, the
X-Ray Integral Field Unit (X-IFU) on board the ESA L2 mission Athena will
provide spatially resolved high-resolution spectroscopy (2.5 eV FWHM up to 7
keV) over the 0.2 to 12 keV bandpass. The in-flight performance of the X-IFU
will be strongly affected by the calibration of the instrument. Uncertainties
in the knowledge of the overall system, from the filter transmission to the
energy scale, may introduce systematic errors in the data, which could
potentially compromise science objectives – notably those involving line
characterisation e.g. turbulence velocity measurements – if not properly
accounted for. Defining and validating calibration requirements is therefore
of paramount importance. In this paper, we put forward a simulation tool based
on the most up-to-date configurations of the various subsystems (e.g. filters,
detector absorbers) which allows us to estimate systematic errors related to
uncertainties in the instrumental response. Notably, the effect of
uncertainties in the energy resolution and of the instrumental quantum
efficiency on X-IFU observations is assessed, by taking as a test case the
measurements of the iron K complex in the hot gas surrounding clusters of
galaxies. In-flight and ground calibration of the energy resolution and the
quantum efficiency is also addressed. We demonstrate that provided an accurate
calibration of the instrument, such effects should be low in both cases with
respect to statistics during observations.

###### keywords:

Athena, X-IFU, X-rays, Calibration, Performance verification

##  1 Introduction

The X-ray Integral Field Unit (X-IFU) [1, 2] on board the future European
X-ray observatory Athena [3] will provide a new look at the X-ray sky and
study the hot and energetic Universe with unprecedented capabilities. With a
spectral resolution of 2.5 eV required up to 7 keV and a 5” spatial resolution
over a 5’ equivalent diameter field-of-view, the X-IFU will provide spatially-
resolved high-resolution spectroscopy in the soft X-ray band (0.2 – 12 keV).
This will be achieved using an array of $\sim$ 3840 superconducting Transition
Edge Sensors (TESs) [4] micro-calorimeters operated at 90 mK, which will allow
individual energy measurements for each pixel.

The calibration of the instrument will be critical to achieve its challenging
science objectives. Notably, the instrumental response needs to be known with
a high level of accuracy to understand the future observations and to maximise
the performances of the instrument. This response will be calibrated on the
ground and modelled by specific response files for each pixel, which will
contain information on the effective area and energy redistribution of the
detector. However, an accurate per pixel calibration may not always be
possible for every physical parameter (e.g., per-pixel measurements of the
stopping power) and this may cause slight systematic effects in the data when
pixels are considered individually. Numerical simulations are therefore
required to verify the calibration accuracy required on the instrumental
parameters and to quantify the corresponding impact on the driving science
objectives of the X-IFU.

In this paper, we present a numerical approach to verify calibration
requirements for the X-IFU. Particular attention is given to the calibration
of the instrumental response. After explaining the motivation for this
approach and the calibration strategy for the detector response (Sect. 2), we
present the related simulation pipeline used to conduct theses studies (Sect.
3). This tool is applied to two examples to assess the systematic errors
related to uncertainties in the quantum efficiency (Sect. 4) and of the energy
resolution (Sect. 5). In both cases, the realistic variations expected in
instrumental parameters are evaluated in regard to the driving science case of
the iron K complex measurement in the core of galaxy clusters.

##  2 X-IFU instrumental response calibration

Whenever an astrophysical source is observed with the X-IFU, its spectrum will
be convolved with the response of the instrument. This transfer function will
depend on a large number of parameters. For X-ray instruments, the response is
modelled using specific response files, divided into two parts for each
detector:

Figure 1: (Left) Total effective area for the 15-row mirrow configuration of
the X-IFU (cm2) as a function of energy (keV). (Right) Total transmission of
the thermal filters placed on the aperture cylinder of the instrument
(courtesy M. Barbera) [5], which is included in the overall effective area of
the instrument.

  * •

The first part will model the total effective area of the detector as a
function of energy (Figure 1 – Left). In the case of the X-IFU, this includes
the effective area of the mirror, the transmission of the thermal filters [5]
(Figure 1 – Right), the absorber stopping power of the pixel (also known as
the quantum efficiency or QE) and the geometrical filling factor of the
detector. This information is usually contained in the Ancillary Response
Function (ARF) file.

  * •

The second part will model the energy redistribution of a given detector,
which gives the probability of a photon of energy $E$ to fall into the
instrumental channel of energy $E^{\prime}$. The core part of the
redistribution function is typically a Gaussian function centred on $E$, whose
full-width half maximum (FWHM) is by definition the energy resolution of the
detector at the given energy. The energy redistribution of the pixel is
contained in the Redistribution Matrix Function (RMF) file.

In this study, we consider the calibration of the instrumental responses for
the X-IFU. The X-IFU is required to know the energy resolution to better than
0.15 eV up to 7 keV, the normalisation of the effective area with an absolute
error lower than 4% at 1 keV (TBC) and with relative error (i..e, knowledge of
the shape of the effective area) below 3% over the 0.2 – 10 keV bandpass.

Strategies for calibration of the instrumental response include ground
calibration of the telescope and the filters, full-detector measurement of the
quantum efficiency (i.e., each pixel is given the average quantum efficiency
measured over the entire detector array TBC) and pixel-by-pixel measurements
of the energy resolution. Yet, these approaches are not immune to inducing
small systematic effects in the data. For instance, by assuming an average
value of the QE over the entire detector, small pixel-to-pixel variations in
the stopping power will in practice be overlooked. Likewise, although the
energy resolution will be measured for each detector, this will only be done
at several specific energies over the bandpass and interpolated, thus leaving
residual uncertainties in the energy resolution of the pixels. Due to limited
calibration time, the systematic errors introduced by these uncertainties may
not be fully investigated through ground calibration. Simulations are thus
required to quantify these effects on the science performed by the X-IFU.

##  3 Numerical validation of calibration requirements

To investigate the effects of uncertainties in the instrumental response on
the performances of the instrument, we have developed a dedicated pipeline
using modified responses and based on an iterative approach. The first step of
the simulation is to identify a representative science case which is most
affected by an error in the physical parameter under investigation (e.g., the
QE or the energy resolution). Starting from a realistic range of uncertainties
on the subsystems, this can be estimated by computing a set of changes in the
response functions and deducing where the highest effect on the science will
be (e.g., on specific lines, on an energy band, on the continuum). One
drawback of this approach is that more than one science objective may be
driving the calibration, in which case, multiple studies may be needed. Once a
relevant target is chosen, the simulation pipeline can be summarized as
follows:

  * •

The science case under consideration is represented by a typical input
spectrum generated using XSPEC [6] and seen by a perfect instrument (unitary
response). To study only the systematic effects related to the response
functions, this spectrum is generated using unrealistically high exposure
times.

  * •

Using the realistic variations of the parameter under study, we generate a
random value of the modified instrumental response function. The input
spectrum is convolved with this response to obtain a realistic observation by
the instrument.

  * •

The spectrum is fitted using C-statistics [7] with the same input model but
using instead the original value of the response function (i.e., not modified)
to estimate the systematic error on the recovered parameters.

By repeating this process a large number of times, the overall effect of the
instrumental response can be assessed. Physical parameters such as the QE may
however depend on multiple other variables (e.g., thickness or composition of
the absorbers, geometry…). To simplify the interpretation of the results, only
changes in one of these variables are considered in this study. Once the
effects are quantified, they can be compared to the current science
requirements for the X-IFU. If the errors are not compatible, mitigation
techniques or improvements on subsystems may be required. This process can
easily be generalised to other physical quantities and therefore provides an
interesting tool to verify the coherence of the calibration requirements
formulated for the X-IFU.

##  4 Quantum efficiency

The Quantum Efficiency (QE) of a TES detector is defined as the overall
stopping power of the pixel as a function of energy, and is de facto related
to the thickness of the absorbers placed above the TES. We investigate in this
section the effect of changes in the absorber thickness on the instrumental
response, at a single-pixel level.

###  4.1 Realistic variations in the detector’s quantum efficiency

We consider here that typical X-IFU detectors are composed of a 1.7 µm Au and
4.2 µm Bi absorber bi-layer with a geometrical filling factor of 96.8% [4]. In
the case of the gold layer, deposition is expected to be homogeneous within a
pixel, with only small gradients in gold thickness across the wafer (estimated
$\leq 5$% TBC). Bismuth will on the contrary have inhomogeneous distribution
within pixels [8], with differences up to 10% (presence of ‘craters’ and
‘valleys’). As long as the variation of bismuth is within this limit, the
quantum efficiency at a given energy will remain within a linear range of
variations over the bandpass of better than 1%. As such, we assume here that
the average pixel quantum efficiency should be (to the first order) comparable
to the quantum efficiency of a pixel with an average value of bismuth
thickness. In the rest of this study, changes in Bi are therefore represented
by an ‘average’ change in Bi thickness over the full pixel. This result
remains to be confirmed by more accurate measurements on representative TESs.

For both materials, changes in absorber thickness affect mainly the high-
energy QE of the pixel, notably above 6 keV (see Figure 2 – Left). To remain
below the required 3% change over the 0.2 – 10 keV bandpass at a pixel level,
variations in the absorber thickness should not be higher than $\pm 0.20$ µm
in gold and $\pm 0.33$ µm for bismuth i.e., respectively $\sim$ 12% and 8% of
their original value, as shown in Figure 2 (Right). In fact, changes in either
of these two materials have similar effects on the overall stopping power, as
no atomic edge is present below 10 keV. Variations in the QE remain below the
requirement as long as total relative variations of both absorbers are below
$\sim$ 4% (Figure 2 – Right). Although bismuth is more likely to have higher
absolute thickness variations, we assume that (in average over a pixel) these
variations should be comparable to that of the gold gradients. In the rest of
the study, we therefore use the same absolute change in thickness for both
absorbers (i.e., $\Delta$Au=$\Delta$Bi in µm). The same exercise has been
conducted for comparison using the same relative variation for both absorbers
(i.e., $\Delta$Au/Au=$\Delta$Bi/Bi in %), with consistent results.

Figure 2: (Left) Quantum efficiency of the absorber with 1.7 µm Au and 4.2 µm
Bi with a geometrical filling factor of 96.8%. The blue envelope represents
the change in QE for a $\pm 0.05$ µm change in Au and in Bi, used as $\pm
1\sigma$ envelope to generate realistic variations in the response. (Right)
Maximal absolute change in the absorber quantum efficiency over the 0.2 – 10
keV bandpass as a function of the change (in %) in the absorber thickness for
Au (red solid line), Bi (blue dash-dotted line) and both in Au and in Bi
(green dotted line). The horizontal dashed line recalls the 3% requirement.

To investigate this effect on the performance of the X-IFU, we select as
driving case the recovery of the physical properties of the hot plasma
embedded in clusters of galaxies using the iron K complex (5 – 7 keV
bandpass), which is likely to be affected by QE uncertainties above 6 keV.
These measurements will be particularly critical to assess the dynamics of the
cluster and its central object. In the current requirements of the instrument,
iron abundance and temperature shall be recovered within 4% of their original
value, while the velocity broadening of the lines shall be known to better
than 20 km/s.

A typical emission spectrum of the thermal plasma was generated using XSPEC
(bapec model [9]) for a local cluster ($z\sim 0.1$) with an iron abundance of
$Z=0.3~{}Z_{\odot}$ (solar abundances as per [10]) and a turbulent velocity of
100 km/s. This spectrum is used as input of the numerical pipeline presented
in Sect. 3. For each iteration of the pipeline, the modified response function
is generated by randomly interpolating a value of the detector QE (Figure 2 –
Left), assuming a normal distribution centred on the original function and
with a standard deviation corresponding to the stopping power curves created
for the same change of $\pm$ X µm in gold and bismuth thicknesses.

Figure 3: Effect of a change in detector thickness (in µm in gold and in
bismuth) on the recovery of different physical parameters in terms of relative
error (%) with respect to their original value for temperature (Top left) and
iron abundance (Top right) and in km/s for the turbulent velocity (Bottom).
Simulations are done for a cluster temperature of 4 (red diamonds), 6 (green
circles) and 8 keV (blue squares) using 10 000 iterations of the pipeline. The
vertical dashed line indicates the change in Au and in Bi (µm) such that the
maximal change in the absolute QE reaches 3% over 0.2 – 10 keV.

###  4.2 Impact on the observations

The previous pipeline was applied to a set of ‘toy’ model galaxy clusters,
with hot gas temperatures of 4, 6 and 8 keV. Figure 3 shows the cluster’s
temperature, iron metallicity and turbulent speed over the 5 – 7 keV bandpass
as function of the change in thickness used to modify the QE (both Au and Bi).
In this case, the 3% requirement is achieved for a change in 0.12 µm in both
layers, which represents a total $\sim$ 4% in the total thickness of the
absorber. We find that abundances and temperature are most affected, with
errors up to 1% for variations of the order of $\pm 0.1$ µm in both Au and Bi,
while systematic errors on the velocity broadening are minimal ($\leq$1 km/s,
see Figure 3 – Bottom). This can be explained by the shape of the changes in
QE, which mostly affect the continuum of the spectrum rather than the
broadening of the lines. In both cases, these effects are well below the
current requirements.

Given the current estimates expected on the thickness of the absorbers, the
systematic effect on the observation of the iron K complex in galaxy clusters
should be low per pixel. Most of the changes between pixels will besides be
either slow gradients for gold, or averaged over the pixel for bismuth. Since
data within neighbouring pixels will likely be grouped during post-processing
to increase the signal-to-noise ratio, the effect shown Figure 3 will be
further reduced within binned regions (for Bi notably). Even if single distant
pixels are used (e.g., to compute structure functions to determine the power
spectrum of the turbulence [11]), the expected variations across the wafer
will be low: systematic effects should therefore be negligible with respect to
statistical errors, which are of the order of a few percent for typical
cluster observations. Experimental verification of the residual gradients
across the wafer (for gold) and within pixels (for bismuth) should
nevertheless be performed. As shown by the small sample of ‘toy’ model
clusters, results obtained here are strongly related to the choice of the
input model, and ultimately to the choice of the science case. Further studies
on other science objectives and other contributors to the effective area
(e.g., filters) are therefore needed to validate the calibration requirements.

##  5 Energy resolution

We investigate here the effect of uncertainties in the energy resolution in
the instrumental response.

###  5.1 Impact on the energy resolution

The energy resolution for the X-IFU will be measured at the pixel level. It
will be calibrated by measuring the Line Spread Function (or LSF) of the
detectors over the bandpass of the instrument (using e.g., crystal-cut mono-
chromators or electron beam ion traps) and interpolated using polynomials
[12]. The Gaussian core LSF of the X-IFU micro-calorimeters should dominate
the total LSF. Current response files (RMF) therefore use a Gaussian spectral
redistribution function, and define its energy resolution as its FWHM $\Delta
E_{0}$. The knowledge of this energy resolution is required to be below 0.15
eV (up to 7 keV) for each pixel. Small deviations of the energy resolution are
however possible either due to the interpolation over the energy band or to
statistical errors on the line fit used for ground calibration.

Uncertainties in the energy resolution will affect measurements of the line
broadening in astrophysical sources. We select as driving science objective
the measurement of the iron K complex velocity broadening (crucial to
understand turbulence mechanisms in galaxy clusters), which requires an
accuracy lower than 20 km/s. The science case is modelled using the same bapec
model presented in Sect. 4, with a gas temperature of 4 keV. In the same way
as for the QE, the input spectrum generated using XSPEC is used as input of
the numerical pipeline Sect. 3. Changes in the response affect in this case
only the RMF file, which is tweaked to have a modified spectral resolution
$\Delta E$ between 5 and 7 keV.

###  5.2 Impact on turbulent speed measurements and analytical model

The results provided by the simulations are shown in Figure 4. For small
variations of $\Delta E$, the impact of the energy resolution on the
measurement is moderate. Even for uncertainties of the energy resolution of
the order of 0.15 eV, the corresponding systematic error will be below 2 km/s.
In an effort to generalise this study to other atomic lines, we tried to
derive an analytical expression of the systematic degradation on the line
broadening $\Delta\sigma_{\text{turb}}$ as a function of the error on $\Delta
E$. To do so, assuming emission lines measured by the X-IFU are well
represented by Gaussian, their total broadening $\sigma$ is given by:

$\sigma^{2}=\sigma^{2}_{\text{turb}}+\sigma^{2}_{\text{th}}+\sigma^{2}_{\text{res}}$
(1)

where $\sigma^{2}_{\text{th}}$ is the thermal broadening of the line and
$\sigma^{2}_{\text{res}}=\Delta E/\sqrt{8\times ln(2)}$. The thermal
broadening component is assumed to be fixed in this study, since temperature
is frozen in the fit. Assuming an uncertainty $\Delta\sigma_{\text{res}}$ on
the energy resolution, for the same line, the error made on the turbulent
velocity should be

$\displaystyle|\Delta\sigma_{\text{turb}}|$
$\displaystyle=\sigma_{\text{turb}}-\sqrt{\sigma_{\text{turb}}^{2}+\sigma_{\text{res,0}}^{2}-\sigma_{\text{res}}^{2}}~{}~{}~{}\text{if}~{}\Delta\sigma_{\text{res}}\geq
0$ (2) $\displaystyle|\Delta\sigma_{\text{turb}}|$
$\displaystyle=-\sigma_{\text{turb}}+\sqrt{\sigma_{\text{turb}}^{2}+\sigma_{\text{res,0}}^{2}-\sigma_{\text{res}}^{2}}~{}~{}~{}\text{if}~{}\Delta\sigma_{\text{res}}\leq
0$ (3)

where $\sigma_{\text{res,0}}=2.5$ eV and
$\sigma_{\text{res}}=\sigma_{\text{res,0}}+\Delta\sigma_{\text{res}}$ As
demonstrated in Figure 4, the corresponding analytical model provides
consistent results with the simulation. The systematic error on the broadening
is converted in terms of km/s by using

$\Delta V_{\text{turb}}=\Delta\sigma_{\text{turb}}\frac{c}{E_{w}}$ (4)

where $c$ is the speed of light and $E_{w}$ the profile-weighted energy of the
complex. Once again, this systematic effect will be higher when single pixels
are considered, but should be averaged whenever multiple pixels are binned
together. For purely Gaussian lines, Equation 3 should provide accurate first-
order estimates of the systematic error on the turbulent velocity over the
bandpass. Overall, these results suggest that the contribution of systematic
errors in the calibration requirement for the energy resolution should be low.
Statistics of the line will likely play a much larger part in the total error
contribution for science cases involving line broadening measurements, notably
in the case of weak lines.

Figure 4: Effect of an error in the knowledge of the energy resolution of a
pixel (eV) on the measurement of the iron K line broadening (red dots),
assuming a level of turbulence of $V_{\text{turb}}=100$ km/s. The analytical
model of Equation 3 is over-plotted in purple. Degradation on the broadening
is given both in eV and km/s (see Equation 4). The black dashed line gives the
20 km/s requirement, the red dotted line indicates the systematic error made
for a 0.15 eV error in the knowledge of $\Delta E$. The bottom panel indicates
the residuals (km/s) between the analytical model and the simulation.

##  6 Conclusion

In this paper we have presented a numerical approach to test and verify some
of the calibration requirements for the X-ray Integral Field Unit. A specific
look was given to the calibration of the instrumental response. Starting from
realistic variations of the physical parameters (e.g., thickness of TES
absorbers, LSF) and using modified response files, the pipeline was used to
investigate the calibration requirement for the quantum efficiency and the
energy resolution. Notably the effect of uncertainties in these parameters was
quantified on the driving science case of iron K measurements in galaxy
clusters. Overall, we have demonstrated that for the current design of the
instrument, these systematics can be considered as second-order effects and
will likely be averaged in binned pixel regions. However, most of the results
obtained here are model-dependent. Further investigations by using multiple
science cases along with actual measurements in the subsystems are now
required to continue the validation of the X-IFU calibration requirements.

## References

  * [1] Barret, D., Lam Trong, T., den Herder, J.-W., Piro, L., Barcons, X., Huovelin, J., Kelley, R., Mas-Hesse, J. M., Mitsuda, K., Paltani, S., Rauw, G., RoŻanska, A., Wilms, J., Barbera, M., Bozzo, E., Ceballos, M. T., Charles, I., Decourchelle, A., den Hartog, R., Duval, J.-M., Fiore, F., Gatti, F., Goldwurm, A., Jackson, B., Jonker, P., Kilbourne, C., Macculi, C., Mendez, M., Molendi, S., Orleanski, P., Pajot, F., Pointecouteau, E., Porter, F., Pratt, G. W., Prêle, D., Ravera, L., Renotte, E., Schaye, J., Shinozaki, K., Valenziano, L., Vink, J., Webb, N., Yamasaki, N., Delcelier-Douchin, F., Le Du, M., Mesnager, J.-M., Pradines, A., Branduardi-Raymont, G., Dadina, M., Finoguenov, A., Fukazawa, Y., Janiuk, A., Miller, J., Nazé, Y., Nicastro, F., Sciortino, S., Torrejon, J. M., Geoffray, H., Hernandez, I., Luno, L., Peille, P., André, J., Daniel, C., Etcheverry, C., Gloaguen, E., Hassin, J., Hervet, G., Maussang, I., Moueza, J., Paillet, A., Vella, B., Campos Garrido, G., Damery, J.-C., Panem, C., Panh, J., Bandler, S., Biffi, J.-M., Boyce, K., Clénet, A., DiPirro, M., Jamotton, P., Lotti, S., Schwander, D., Smith, S., van Leeuwen, B.-J., van Weers, H., Brand, T., Cobo, B., Dauser, T., de Plaa, J., and Cucchetti, E., “The Athena X-ray Integral Field Unit (X-IFU),” in [Space Telescopes and Instrumentation 2016: Ultraviolet to Gamma Ray ], Proc. SPIE 9905, 99052F (2016).
  * [2] Pajot, F., Lam Trong, T., den Herder, J.-W., Piro, L., and Cappi, M., “The Athena X-ray Integral Field Unit (X-IFU),” Journal Of Low Temperature Physics (Jan. 2018).
  * [3] Nandra, K., Barret, D., Barcons, X., Fabian, A., den Herder, J.-W., Piro, L., Watson, M., Adami, C., Aird, J., Afonso, J. M., and et al., “The Hot and Energetic Universe: A White Paper presenting the science theme motivating the Athena+ mission,” ArXiv e-prints (June 2013).
  * [4] Smith, S. J., Adams, J. S., Bandler, S. R., Betancourt-Martinez, G. L., Chervenak, J. A., Chiao, M. P., Eckart, M. E., Finkbeiner, F. M., Kelley, R. L., Kilbourne, C. A., Miniussi, A. R., Porter, F. S., Sadleir, J. E., Sakai, K., Wakeham, N. A., Wassell, E. J., Yoon, W., Bennett, D. A., Doriese, W. B., Fowler, J. W., Hilton, G. C., Morgan, K. M., Pappas, C. G., Reintsema, C. N., Swetz, D. S., Ullom, J. N., Irwin, K. D., Akamatsu, H., Gottardi, L., den Hartog, R., Jackson, B. D., van der Kuur, J., Barret, D., and Peille, P., “Transition-edge sensor pixel parameter design of the microcalorimeter array for the x-ray integral field unit on Athena,” in [Space Telescopes and Instrumentation 2016: Ultraviolet to Gamma Ray ], Proc. SPIE 9905, 99052H (July 2016).
  * [5] Barbera, M., Argan, A., Bozzo, E., Branduardi-Raymont, G., Ciaravella, A., Collura, A., Cuttaia, F., Gatti, F., Jimenez Escobar, A., Lo Cicero, U., Lotti, S., Macculi, C., Mineo, T., Nuzzo, F., Paltani, S., Parodi, G., Piro, L., Rauw, G., Sciortino, L., Sciortino, S., and Villa, F., “Thermal Filters for the ATHENA X-IFU: Ongoing Activities Toward the Conceptual Design,” Journal of Low Temperature Physics 184, 706–711 (Aug. 2016).
  * [6] Arnaud, K. A., “XSPEC: The First Ten Years,” in [Astronomical Data Analysis Software and Systems V ], Jacoby, G. H. and Barnes, J., eds., Astronomical Society of the Pacific Conference Series 101, 17 (1996).
  * [7] Cash, W., “Parameter estimation in astronomy through application of the likelihood ratio,” Astrophysical Journal 228, 939–947 (Mar. 1979).
  * [8] Sadleir, J. E., Bandler, S. R., Brekosky, R. P., Chervenak, J., Figueroa-Feliciano, E., Finkbeiner, F., Iyomoto, N., Kelley, R. L., Kilbourne, C. A., King, J. M., Porter, F. S., Robinson, I. K., Saab, T., and Talley, D. J., “Bismuth X-ray absorber studies for TES microcalorimeters,” Nuclear Instruments and Methods in Physics Research A 559, 447–449 (Apr. 2006).
  * [9] Smith, R. K., Brickhouse, N. S., Liedahl, D. A., and Raymond, J. C., “Collisional Plasma Models with APEC/APED: Emission-Line Diagnostics of Hydrogen-like and Helium-like Ions,” Astrophysical Journal 556, L91–L95 (Aug. 2001).
  * [10] Anders, E. and Grevesse, N., “Abundances of the elements - Meteoritic and solar,” Geochimica et Cosmochimica Acta 53, 197–214 (Jan. 1989).
  * [11] ZuHone, J. A., Markevitch, M., and Zhuravleva, I., “Mapping the Gas Turbulence in the Coma Cluster: Predictions for Astro-H,” Astrophysical Journal 817, 110 (Feb. 2016).
  * [12] Eckart, M. E., Adams, J. S., Boyce, K. R., Brown, G. V., Chiao, M. P., Fujimoto, R., Haas, D., den Herder, J. W., Ishisaki, Y., Kelley, R. L., Kilbourne, C. A., Leutenegger, M. A., McCammon, D., Mitsuda, K., Porter, F. S., Sato, K., Sawada, M., Seta, H., Sneiderman, G. A., Szymkowiak, A. E., Takei, Y., Tashiro, M., Tsujimoto, M., de Vries, C. P., Watanabe, T., Yamada, S., and Yamasaki, N. Y., “Ground calibration of the astro-h (hitomi) soft x-ray spectrometer,” Journal of Astronomical Telescopes, Instruments, and Systems 4, 4 – 4 – 22 (2018).


# An Empirical Investigation into the Effect of
Parameter Choices in Knowledge Distillation

Md Arafat Sultan, Aashka Trivedi, Parul Awasthy, Avirup Sil
IBM Research AI
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

We present a large-scale empirical study of how choices of configuration
parameters affect performance in knowledge distillation (kd). An example of
such a kd parameter is the measure of distance between the predictions of the
teacher and the student, common choices for which include the mean squared
error (mse) and the kl-divergence. Although scattered efforts have been made
to understand the differences between such options, the kd literature still
lacks a systematic study on their general effect on student performance. We
take an empirical approach to this question in this paper, seeking to find out
the extent to which such choices influence student performance across $13$
datasets from $4$ nlp tasks and $3$ student sizes. We quantify the cost of
making sub-optimal choices and identify a single configuration that performs
well across the board.

mO Avi[#1]

An Empirical Investigation into the Effect of
Parameter Choices in Knowledge Distillation

Md Arafat Sultan, Aashka Trivedi, Parul Awasthy, Avirup Sil IBM Research AI
<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Knowledge distillation (kd) from a larger teacher model into a smaller student
model Hinton et al. (2014) is a powerful approach in machine learning, often
yielding better models than ordinary 1-hot-label training. Yet setting up kd
and optimizing its performance for a new application can be challenging due to
its highly parameterized nature. For example, one can measure the distance
between the predictions of the teacher and the student as the mean squared
error (mse) between their output logits, or alternatively with cross-entropy
or kl-divergence between their probability estimates Kim et al. (2021). The
teacher, as another example, can be selected based on its performance
according to some evaluation metric or the quality of its probability
estimates Menon et al. (2021).

Crucially, our understanding of the extent to which the various available
options in kd represent fundamentally different processes, and not minor
variations of the same underlying mechanism with similar performance profiles,
is still very limited. As a result, users of kd often resort to personal
experience (and bias) to choose from such alternatives or even do so randomly
(Mukherjee and Hassan Awadallah, 2020; Sanh et al., 2019; Jiao et al., 2020;
Wang et al., 2021; Zhou et al., 2021), not knowing how their choices might
affect downstream results. In this paper, we take a first step towards
understanding the effects of different parameter choices on the performance of
kd by conducting a large-scale empirical investigation on $13$ carefully
chosen datasets (§3) from $4$ nlp tasks with models of different sizes and
capacities.

Besides the aforementioned distance measure and teacher selection criterion,
we include two more key kd parameters in our study (§2): whether or not to (a)
use human labels for supervision, and (b) temperature scale the student’s
output. We perform an approximate greedy search over the large space of
combinations of these choices (§3), and present empirical results answering
the following research questions (§4): (i) How much do parameter choices
matter in kd? We quantify and compare performances of strong, weak and random
kd configurations; (ii) What are the effects of individual parameters? We
examine if certain parameters play a bigger role in determining student
performance; and (iii) Does a single configuration exist that can perform well
across the board? We propose a simple method to find such a combination of
parameter choices using validation set results and analyze its test set
performance.

Our results show a relative performance gain of up to $9.4\%$ with a strong
student model over a weak one ($4.3\%$ in the 90th percentile), and that of up
to $3\%$ over a randomly chosen configuration. Since our search over kd
configurations is not exhaustive (§3), these are in fact lower bounds of the
corresponding quantities, indicating that a sub-optimal choice may lead to
undesired, even critical, performance loss for many applications. To that end,
the one configuration identified by our proposed method not only reduces the
performance gap across the board, but also matches or outperforms the original
best configurations – which are test-specific and variable – in $40\%$ of all
test cases.

##  2 Preliminaries

Given input $x$, let $q(x)$ be the 1-hot target distribution to learn in a
task. Let $l^{s}(x)$ and $p^{s}(x)$ be the output logits and the corresponding
softmax distribution predicted by the student $s$ being trained, and
$l^{t}(x)$ and $p^{t}(x)$ the same for the pre-trained teacher $t$. The kd
objective for $x$ takes the following general form (omitting arguments for
simplicity):

$\mathcal{L}_{KD}=(1-\alpha)\mathcal{L}_{1}(q,p^{s})+{\alpha}\mathcal{L}_{2}(o^{t}_{\tau_{t}},o^{s}_{\tau_{s}})$
(1)

where $\mathcal{L}_{1}$ is the distance of $p^{s}$ from $q$ and
$\mathcal{L}_{2}$ is the distance between the output $o^{t}$ of the teacher
and $o^{s}$ of the student, which can be either the logits $l^{t}$ and $l^{s}$
or the probabilities $p^{t}$ and $p^{s}$ depending on the choice of
$\mathcal{L}_{2}$, with $\alpha\in(0,1]$ modulating the weights of
$\mathcal{L}_{1}$ and $\mathcal{L}_{2}$ in the overall loss. $\tau_{t}$ and
$\tau_{s}$ are temperature hyperparameters that can be used to divide the
respective sets of logits to smoothen the corresponding distributions.

We investigate the following decision choices involving four key kd
parameters:

$\bullet$

Use of human labels. Setting $\alpha=1$ in Eq. 1 uses only the teacher’s
predictions as targets for distillation Jiao et al. (2020); Beyer et al.
(2022); alternatively, one can tune $\alpha\in(0,1]$ to also learn from 1-hot
expert labels Sun et al. (2019); Dasgupta et al. (2023). There is an important
trade-off associated with this choice: tuning $\alpha$ can increase the number
of training runs by a factor of $n_{\alpha}$, where $n_{\alpha}$ is the number
of $\alpha$ values to search over.

$\bullet$

Teacher-student distance measure. We compare performances of two
$\mathcal{L}_{2}$ options in Eq. 1: the mean-squared error (mse) between
$l^{s}$ and $l^{t}$ Kim et al. (2021); Rao et al. (2023) and the cross-entropy
(ce) between $p^{s}$ and $p^{t}$ Hinton et al. (2014); Yang et al. (2022). A
related study by Kim et al. (2021) discusses empirical observations in image
classification, but our more extensive investigation includes a larger set of
nlp tasks.

$\bullet$

Teacher selection. Menon et al. (2021) find that models that generate high-
quality probability estimates – as indicated by their low log-loss on a held-
out set – constitute better kd teachers than models with a high top-1
accuracy. While they compare teachers $t$ of different architectures and
sizes, our goal here is to identify the best checkpoint for $t$ stored during
training. For each teacher-student-dataset triple, we compare students
distilled from two teachers: (a) the highest-score teacher ($\bm{t_{hs}}$):
the teacher checkpoint with the highest validation set score for the chosen
evaluation metric, e.g., F1-score, and (b) the lowest loss teacher
($\bm{t_{ll}}$): the checkpoint with the lowest validation set loss.

$\bullet$

Temperature scaling of student logits. While Hinton et al. (2014) originally
applied an equal amount of temperature scaling to teacher and student outputs
($\tau_{s}=\tau_{t}$), some later work scaled only the teacher’s
($\tau_{s}=1$) Zhang and Sabuncu (2020); Sultan et al. (2022). The latter has
been found to generate better calibrated predictions Zhang and Sabuncu (2020)
and is arguably easier to interpret; here we investigate if it also leads to
significant performance differences across a range of tasks.

##  3 Methodology

Setting up a diverse set of test conditions. A study such as this must involve
a variety of test cases with a multitude of functions to learn as well as
models of different sizes and capacities. To that end, we evaluate kd
parameter choices on four nlp tasks: text classification, reading
comprehension, named entity recognition and machine translation. The first
three are extractive or labeling tasks with different label sets and the
fourth a generative task. To further enforce diversity, within each task, we
select datasets that vary in properties such as the size of their training
sets and the mix of mono and multilingual data in their splits. Table 1 lists
the datasets; see Appendix B for details.

Task | Datasets
---|---
Text Classification |  mrpc, mnli, paws-x, xnli
Named Entity Recognition |  ontonotes, conll, klue3
Reading Comprehension |  squad, natural questions,
mlqa, tydiqa
Machine Translation |  wmt16 de-en,
wmt16 ro-en
Table 1: The $4$ tasks and $13$ datasets used in our experiments. See Appendix
B for details.

For each non-generative task, we fine-tune a roberta-large Liu et al. (2019)
teacher for each dataset and distill it into a roberta-base as well as two
$6$\- and $4$-layer truncated roberta architectures. The two latter students
are initialized using a skip-layer strategy as in Sun et al. (2019). For
machine translation, we similarly fine-tune an mbart-large-cc25 Liu et al.
(2020) teacher ($12$ encoder and $12$ decoer layers) and distill it into
$12$-, $6$\- and $4$-layer mbart-cc25 skipped-layer students, each with an
equal number of encoder and decoder layers.

Parameter |  Options (§2)
---|---
$t$ |  $t_{hs}$, $t_{ll}$
$\mathcal{L}_{2}$ |  ce, mse
$\tau_{s}$ | $1,\tau_{t}$
$\alpha$ | {1}, $\\{.1,.2,...,.9,1\\}$
Table 2: kd parameter grid in our experiments.

Searching over a large space of parameter choices. Table 2 shows a typical
parameter search space for a single kd experiment. Two of the four parameters
in the table ($t$ and $\mathcal{L}_{2}$) are categorical and one ($\alpha$) is
bounded; for these, a standard grid search over a fixed set of values is
sufficient. $\tau_{t}$, however, is unbounded, and thus a two-stage search,
first over a sparse set of values, e.g., a geometric progression, and then
linearly within a local proximity of the optimal value from stage $1$ arguably
makes more sense for it (and for $\tau_{s}$ when $\tau_{s}=\tau_{t}$).

Assuming a grid size of $7$ for the first broad search for $\tau_{t}$ and of
$4$ for the second local search, the number of models to be distilled and
evaluated per dataset-student pair implied by Table 2 alone is roughly $440$;
this number must be multiplied by the size of the space of hyperparameters
associated with general model training, e.g., learning rate, to compute the
actual number of students to be trained. This shows why an exhaustive grid
search is practically infeasible in kd, not only for practitioners in general
but also for us in this study, as we intend to test the effects of such
choices across different tasks, datasets and model sizes.

We therefore adopt the following greedy search strategy to navigate important
points of the grid: First, we set $\alpha$ to $1$ – training the student with
only the teacher’s soft predictions – and optimize the remaining parameters
$\mathcal{L}_{2}$, $t$ and $\tau_{s}$; we then evaluate this optimal
$\alpha=1$ combination of the other three parameters for additional values of
$\alpha$ (Table 2) to determine if a mix of hard and soft labels can train
better students than only the latter. To find the best combination of
$\mathcal{L}_{2}$, $t$ and $\tau_{s}$ for $\alpha=1$, we start with the
configuration of Hinton et al. (2014) ($\mathcal{L}_{2}=$ ce, $t=$ $t_{hs}$,
$\tau_{s}=\tau_{t}$) and tune the learning rate to find the best model using
the validation set, and evaluate it on the test set. Then at each step of an
iterative process, we change the value of one kd parameter and evaluate it
similarly. The value of the parameter is set to the choice that yields the
higher score and the process continues. This method traverses four out of
eight possible configurations for $\alpha=1$ with an aim to evaluate the more
promising candidates.

##  4 Experimental Results

We compare different kd configurations using the following uniform method in
all our evaluations: (i) Compute the relative gain in accuracy with a target
configuration over a baseline for all $39$ teacher-student-dataset triples,
e.g., (squad, roberta-large, roberta-4layer), (ii) Split the $39$ gains up
into deciles, and (iii) Put different such comparisons side by side to assess
their magnitude in context. The Transformers library of Wolf et al. (2020) is
used for all implementations.

Figure 1:  Relative performance gain with the best kd configuration in our
evaluated sample over two baselines. The empirical upper bound of the cost of
making a bad parameter choice is $1.6\%$ with a $50\%$ probability and $4.3\%$
with a $90\%$ probability.

Figure 1 illustrates how the best kd configuration for each individual
dataset-teacher-student triple, i.e. test case, improves performance over two
baselines: the worst-performing configuration for the triple and a
configuration chosen randomly from among all candidates. At the median ($D5$),
we observe a gain of $0.4$–$1.6\%$ over the baselines, which goes up to
$1.4$–$4.3$% in the $9$th decile ($D9$). In other words, the empirical upper
bound of the cost of making a bad parameter choice is $1.6\%$ with a $50\%$
probability and $4.3\%$ with a $90\%$ probability. The interpretation of how
big these differences are can vary from one use case to another: the $3\%$
difference at the peak between a tuned and a random configuration in Figure 1
likely does not warrant any expensive tuning in, for example, an academic
research project where kd is being used simply as a tool among many and is not
the central research topic. In contrast, an ill-fated worse-case scenario
causing a near $10\%$ drop, as we observe for $D10$ of the worst-performing
configuration, can be critical in many real-life applications. Given that we
only evaluate a subset of all configurations (§3) – and thus the differences
we observe in Figure 1 are in fact lower bounds – kd configuration choices can
have far-reaching consequences in practice.

Figure 2:  Performance differences due to individual kd parameter choices.
Choices of all parameters except $\alpha$ can have a non-negligible impact on
performance.

To gain further insight, next we look at the impact of individual parameter
choices. As described in §3, at each step of our greedy grid search for
$\alpha=1$, we change the value of one of the other parameters $t$,
$\mathcal{L}_{2}$ and $\tau_{s}$ and evaluate the resulting configuration; we
take the performances of the two configurations before and after this change
as an estimate of the performance of the two corresponding parameter values.
In Figure 2, the use of hard labels ($\alpha\leq 1$) has the least overall
impact over its baseline ($\alpha=1$), which is an important finding as tuning
$\alpha$ can be expensive. For the remaining parameters, the difference in
student performance has a much wider distribution over the two choices, with
ce as a distance measure perhaps showing the clearest advantage over mse.
Individual choices make less of a difference than their combination, as
expected, when compared to Figure 1.

Parameter | Default | Alternative | $\bm{R}$
---|---|---|---
$t$ | $t_{hs}$ | $\bm{t_{ll}}$ | $+11$
$\mathcal{L}_{2}$ | ce | mse | $-5$
$\tau_{s}$ | $\tau_{t}$ | $\bm{1}$ | $+1$
Table 3: Rewards $R$ for overriding default parameter choices with their
alternatives. The best configuration implied by this table is thus: $t=$
$t_{ll}$, $\mathcal{L}_{2}=$ ce, $\tau_{s}=1$.

Our final research question concerns finding a single kd configuration that is
stable, i.e., performs well across different applications. Given the small
observed improvement from tuning $\alpha$, we explore this question only for
$\alpha=1$ and look for a stable combination of $t$, $\mathcal{L}_{2}$ and
$\tau_{s}$ values. Assuming mutual independence of these parameters, we
separately optimize for each on validation data, and evaluate the combination
of optimal values as our target configuration on all test sets. Table 3 shows
the validation set reward $R$ for overriding the Hinton et al. (2014) choice
(the default) for each parameter with the alternative choice, which we compute
as the difference between the number of times the alternative trains a better
and a worse student than the default across all $39$ of our test cases. A
positive $R$ implies that the alternative choice should be favored while $R<0$
indicates that the default is the better option; the magnitude of $R$
represents the strength of the finding. Our results confirm the superiority of
the low-loss teacher of Menon et al. (2021) in a larger context, and shows
$\tau_{s}=1$ to be on par with the traditional $\tau_{s}=\tau_{t}$, which is
desirable given its better known calibration properties Zhang and Sabuncu
(2020).

In Figure 3 we contrast test set performances of the best configuration of
Table 3 with those of a random and a lazy configuration; the latter simulates
a common user persona who chooses a random kd configuration along with
$\tau_{t}=1$, as in Ren et al. (2023). Our best single configuration shows the
least lag from the best-performing configuration specific to each individual
test case; in fact, it is at least as good as the latter in $40\%$ of all test
cases and better than both baselines across the board. This is yet another
significant discovery of our study, pointing to a relatively stable kd
configuration if no tuning were to be performed.

Figure 3: Our proposed configuration (green) optimized over validation data
performs at least as well as the original test-specific best configurations in
$40\%$ of all test cases, outperforming two baselines across the board.

##  5 Conclusion

We demonstrate substantial differences in student accuracy due to varying
choices of configuration parameters in knowledge distillation (kd), and
identify a single configuration that stabilizes performance across a range of
different target scenarios. While we only take a small, yet crucial, step
towards understanding the effects of parameter choices in kd, through this
work, we hope to inspire future explorations of this important topic.

## Limitations

First, due to a prohibitively large search space, we were able to perform only
an approximate grid search over kd parameters to find our best and worst
empirical configurations (§3). Second, even with a large-scale investigation
involving a variety of test conditions, it is generally hard to claim with
certainty that the findings of such a study will generalize to even larger
studies with more tasks. However, scientific knowledge often advances one
small step at a time, which is how we believe the contributions of this work
should be viewed.

## References

  * Beyer et al. (2022) Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. 2022.  Knowledge distillation: A good teacher is patient and consistent.  In _CVPR_.
  * Bojar et al. (2016) Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016.  Findings of the 2016 Conference on Machine Translation.  In _Proceedings of the First Conference on Machine Translation_.
  * Clark et al. (2020) Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020.  TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages.  _Transactions of the ACL_.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  XNLI: Evaluating Cross-lingual Sentence Representations.  In _EMNLP_.
  * Dasgupta et al. (2023) Sayantan Dasgupta, Trevor Cohn, and Timothy Baldwin. 2023.  Cost-effective Distillation of Large Language Models.  In _Findings of the ACL_.
  * Dolan and Brockett (2005) William B. Dolan and Chris Brockett. 2005.  Automatically Constructing a Corpus of Sentential Paraphrases.  In _International Workshop on Paraphrasing (IWP)_.
  * Florian et al. (2010) Radu Florian, John Pitrelli, Salim Roukos, and Imed Zitouni. 2010.  Improving Mention Detection Robustness to Noisy Input.  In _EMNLP_.
  * Hinton et al. (2014) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2014.  Distilling the Knowledge in a Neural Network.  In _NeurIPS Deep Learning Worksop_.
  * Jiao et al. (2020) Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020.  TinyBERT: Distilling BERT for Natural Language Understanding.  In _Findings of EMNLP_.
  * Kim et al. (2021) Taehyeon Kim, Jaehoon Oh, , Nak Yil Kim, Sangwook Cho, and Se-Young Yun. 2021.  Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation.  In _IJCAI_.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural Questions: A Benchmark for Question Answering Research.  _Transactions of the ACL_.
  * Lewis et al. (2020) Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020.  MLQA: Evaluating Cross-lingual Extractive Question Answering.  In _ACL_.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual Denoising Pre-training for Neural Machine Translation.  _Transactions of the ACL_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _arXiv preprint arXiv:1907.11692_.
  * Menon et al. (2021) Aditya Krishna Menon, Ankit Singh Rawat, Sashank J. Reddi, Seungyeon Kim, and Sanjiv Kumar. 2021.  A Statistical Perspective on Distillation.  In _ICML_.
  * Mukherjee and Hassan Awadallah (2020) Subhabrata Mukherjee and Ahmed Hassan Awadallah. 2020.  XtremeDistil: Multi-stage Distillation for Massive Multilingual Models.  In _ACL_.
  * Pradhan et al. (2012) Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012.  CoNLL-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotes.  In _EMNLP-CoNLL Shared Task_.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ Questions for Machine Comprehension of Text.  In _EMNLP_.
  * Rao et al. (2023) Jun Rao, Liang Ding, Shuhan Qi, Meng Fang, Yang Liu, Li Shen, and Dacheng Tao. 2023.  Dynamic contrastive distillation for image-text retrieval.  _IEEE Transactions on Multimedia_ , pages 8383–8395.
  * Ren et al. (2023) Yuxin Ren, Zihan Zhong, Xingjian Shi, Yi Zhu, Chun Yuan, and Mu Li. 2023.  Tailoring Instructions to Student’s Learning Levels Boosts Knowledge Distillation.  In _ACL_.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  In _$EMC^{2}$ at NeurIPS_.
  * Sultan et al. (2022) Md Arafat Sultan, Avirup Sil, and Radu Florian. 2022.  Not to Overfit or Underfit the Source Domains? An Empirical Study of Domain Generalization in Question Answering.  In _EMNLP_.
  * Sun et al. (2019) Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019.  Patient Knowledge Distillation for BERT Model Compression.  In _EMNLP-IJCNLP_.
  * Tjong Kim Sang (2002) Erik F. Tjong Kim Sang. 2002.  Introduction to the CoNLL-2002 Shared Task: Language-Independent Named Entity Recognition.  In _CoNLL_.
  * Wang et al. (2021) Fusheng Wang, Jianhao Yan, Fandong Meng, and Jie Zhou. 2021.  Selective Knowledge Distillation for Neural Machine Translation.  In _ACL-IJCNLP_.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference.  In _NAACL_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R´emi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-Art Natural Language Processing.  In _EMNLP (System Demonstrations)_.
  * Yang et al. (2022) Chenxiao Yang, Junwei Pan, Xiaofeng Gao, Tingyu Jiang, Dapeng Liu, and Guihai Chen. 2022.  Cross-task knowledge distillation in multi-task recommendation.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 36(4):4318–4326.
  * Yang et al. (2019) Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019.  PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification.  In _EMNLP-IJCNLP_.
  * Zhang and Sabuncu (2020) Zhilu Zhang and Mert R. Sabuncu. 2020.  Self-Distillation as Instance-Specific Label Smoothing.  In _NeurIPS_.
  * Zhou et al. (2021) Xuan Zhou, Xiao Zhang, Chenyang Tao, Junya Chen, Bing Xu, Wei Wang, and Jing Xiao. 2021.  Multi-Grained Knowledge Distillation for Named Entity Recognition.  In _NAACL_.

##  Appendix A Evaluation Metrics

We use standard evaluation metrics for all tasks: bleu for machine translation
and F1-score for the rest.

##  Appendix B Details of Datasets

Our datasets represent a diverse set of tasks, varying in properties such as
size, language and label space. In this section, we describe key attributes of
all $13$ of our datasets.

###  B.1 Text Classification

  * •

mrpc Dolan and Brockett (2005):

    * –

$\\#$ of training examples: $3,668$

    * –

$\\#$ of validation examples: $408$

    * –

$\\#$ of test examples: $1,725$

    * –

Training set language: English

    * –

Validation set language : English

    * –

Test set language: English

    * –

$\\#$ of labels: 2

  * •

mnli Williams et al. (2018):

    * –

$\\#$ of training examples: $382,887$

    * –

$\\#$ of validation examples: $9,815$

    * –

$\\#$ of test examples: $9,815$

    * –

Training set language: English

    * –

Validation set language : English

    * –

Test set language: English

    * –

$\\#$ of labels: 3

  * •

paws-x Yang et al. (2019):

    * –

$\\#$ of training examples: $345,807$

    * –

$\\#$ of validation examples: $14,000$

    * –

$\\#$ of test examples: $14,000$

    * –

Training set language: Multilingual

    * –

Validation set language : Multilingual

    * –

Test set language: Multilingual

    * –

$\\#$ of labels: 2

  * •

xnli Conneau et al. (2018):

    * –

$\\#$ of training examples: $392,702$

    * –

$\\#$ of validation examples: $37,350$

    * –

$\\#$ of test examples: $75,150$

    * –

Training set language: English

    * –

Validation set language: Multilingual

    * –

Test set language: Multilingual

    * –

$\\#$ of labels: 3

###  B.2 Reading Comprehension

  * •

squad Rajpurkar et al. (2016):

    * –

$\\#$ of training examples: $76,081$

    * –

$\\#$ of validation examples: $10,507$

    * –

$\\#$ of test examples: $10,507$

    * –

Training set language: English

    * –

Validation set language : English

    * –

Test set language: English

    * –

$\\#$ of labels: $384$ (context window size)

  * •

natural questions Kwiatkowski et al. (2019):

    * –

$\\#$ of training examples: $91,235$

    * –

$\\#$ of validation examples: $12,836$

    * –

$\\#$ of test examples: $12,836$

    * –

Training set language: English

    * –

Validation set language : English

    * –

Test set language: English

    * –

$\\#$ of labels: $384$ (context window size)

  * •

mlqa Lewis et al. (2020):

    * –

$\\#$ of training examples: $76,081$

    * –

$\\#$ of validation examples: $7,715$

    * –

$\\#$ of test examples: $75,712$

    * –

Training set language: English

    * –

Validation set language : Multilingual

    * –

Test set language: Multilingual

    * –

$\\#$ of labels: $384$ (context window size)

  * •

tydiqa Clark et al. (2020) (Gold Passages):

    * –

$\\#$ of training examples: $44,896$

    * –

$\\#$ of validation examples: $4,985$

    * –

$\\#$ of test examples: $5,077$

    * –

Training set language: Multilingual

    * –

Validation set language : Multilingual

    * –

Test set language: Multilingual

    * –

$\\#$ of labels: $384$ (context window size)

###  B.3 Named Entity Recognition

  * •

conll Tjong Kim Sang (2002):

    * –

$\\#$ of training examples: $39,115$

    * –

$\\#$ of validation examples: $8,275$

    * –

$\\#$ of test examples: $10,395$

    * –

Training set language: Multilingual

    * –

Validation set language: Multilingual

    * –

Test set language: Multilingual

    * –

$\\#$ of labels: $9$

  * •

ontonotes Pradhan et al. (2012):

    * –

$\\#$ of training examples: $82,728$

    * –

$\\#$ of validation examples: $10,510$

    * –

$\\#$ of test examples: $10,396$

    * –

Training set language: English

    * –

Validation set language : English

    * –

Test set language: English

    * –

$\\#$ of labels: $37$

  * •

klue3 Florian et al. (2010):

    * –

$\\#$ of training examples: $12,467$

    * –

$\\#$ of validation examples: $8,785$

    * –

$\\#$ of test examples: $10,816$

    * –

Training set language: English

    * –

Validation set language: Multilingual

    * –

Test set language: Multilingual

    * –

$\\#$ of labels: $103$

###  B.4 Machine Translation

  * •

wmt16 de-en Bojar et al. (2016) (maximum sequence length of 128 tokens):

    * –

$\\#$ of training examples: $500,000$ (sampled from $4,490,021$)

    * –

$\\#$ of validation examples: $2,168$

    * –

$\\#$ of test examples: $2,998$

    * –

Training set language: German (source), English (target)

    * –

Validation set language: German (source), English (target)

    * –

Test set language: German (source), English (target)

    * –

$\\#$ of labels: $50,265$ (vocabulary size)

  * •

wmt16 ro-en Bojar et al. (2016) (maximum sequence length of 128 tokens):

    * –

$\\#$ of training examples: $100,000$ (sampled from $596,163$)

    * –

$\\#$ of validation examples: $1,998$

    * –

$\\#$ of test examples: $1,996$

    * –

Training set language: Romanian (source), English (target)

    * –

Validation set language: Romanian (source), English (target)

    * –

Test set language: Romanian (source), English (target)

    * –

$\\#$ of labels: $50,265$ (vocabulary size)