#### Title

Predicting solution scattering patterns with explicit-solvent molecular
simulations

#### Running title

Explicit-solvent SAS predictions

#### Authors

Leonie Chatzimagas1 and Jochen S. Hub1

#### Affiliations

1Theoretical Physics and Center for Biophysics, Saarland University,
Saarbrücken 66123, Germany

#### Keywords

Small-angle scattering, SAXS, SANS, all-atom molecular dynamics simulations,
hydration layer, excluded solvent

#### Abstract

Small-angle X-ray or neutron scattering (SAXS/SANS/SAS) is widely used to
obtain structural information on biomolecules or soft-matter complexes in
solution. Deriving a molecular interpretation of the scattering signals
requires methods for predicting SAS patterns from a given atomistic structural
model. Such SAS predictions are non-trivial because the patterns are
influenced by the hydration layer of the solute, the excluded solvent, and by
thermal fluctuations. Many computationally efficient methods use simplified,
implicit models for the hydration layer and excluded solvent, leading to some
uncertainties and to free parameters that require fitting against experimental
data. SAS predictions based on explicit-solvent molecular dynamics (MD)
simulations overcome such limitations at the price of an increased
computational cost. To rationalize the need for explicit-solvent methods, we
first review the approximations underlying implicit-solvent methods. Next, we
describe the theory behind explicit-solvent SAS predictions that are easily
accessible via the WAXSiS web server. We present the workflow for computing
SAS pattern from a given molecular dynamics trajectory. The calculations are
available via a modified version of the GROMACS simulations software, coined
GROMACS-SWAXS, which implements the WAXSiS method. Practical considerations
for running routine explicit-solvent SAS predictions are discussed.

##  1 Introduction

Small- and wide-angle X-ray scattering (SWAXS) has originally been used to
probe the overall shape of biomolecules or soft-matter complexes in solution.
In recent decades, SWAXS has developed into an increasingly quantitative
probe, primarily due to improved sample preparation, more brilliant light
sources, and single-photon counting detectors (Koch et al., 2003; Putnam et
al., 2007; Graewert and Svergun, 2013). Thanks to the advent of free-electron
lasers, time-resolved SWAXS (TR-SWAXS) measurements are capable of tracking
ultrafast dynamics of biomolecules or small molecules in solution (Arnlund et
al., 2014; Levantino et al., 2015; Brinkmann and Hub, 2016). Complementary,
small-angle neutron scattering (SANS) has remained popular because it allows
for contrast-variation experiments by changing the D2O concentration of the
buffer (Gabel, 2015). Henceforth, we use the term SAS (small-angle scattering)
when referring to both SWAXS and SANS.

Analysis of the SAS curve gives information on the overall shape and
macroscopic properties of the probe, such as the radius of gyration, maximum
particle size, the structural order (globular vs. unfolded), molecular mass,
and particle volume. In contrast, an atomistic interpretation cannot be drawn
from the data alone owing to the low information content of the SAS curves,
but instead requires methods for theoretically predicting SAS intensities from
a given atomistic structure. Such SAS curve predictions, so-called forward
models, enable the validation or the refinement of structural models or
ensembles against SAS data.

The physical properties that give rise to a solution scattering curve $I(q)$
are well understood. Namely, the intensities are given by

$I(q)=\langle|F(\mathbf{q})|^{2}\rangle_{\Omega}\,,$ (1)

where $\langle\cdot\rangle_{\Omega}$ denotes the orientational average in
reciprocal space, and $F(\mathbf{q})$ is the Fourier transform of the electron
density contrast $\Delta\rho(\mathbf{r})$ between the solution and the pure-
buffer system:

$F(\mathbf{q})=\int\Delta\rho(\mathbf{r})\,e^{-i\mathbf{q}\cdot\mathbf{r}}\,\mathrm{d}^{3}r$
(2)

However, computing $I(q)$ from a given structural model such as a crystal
structure or a molecular dynamics (MD) simulation of a biomolecule is
complicated by several aspects:

  * (i)

Since SAS detects the electron density contrast in solution, computing $I(q)$
requires knowledge of the volume of solvent that is displaced by the solute.
For a biomolecule with internal disorder and a rough surface, the displaced
volume is far from obvious. In addition, as discussed below, the volume taken
by atoms of a certain chemical element depends on the chemical environment,
suggesting that tabulated atomic volumes are subject to marked uncertainties.

  * (ii)

The density of the hydration layer of biomolecules differs from the density of
bulk solvent, thus contributing to the density contrast $\Delta\rho$. The
hydration layer is influenced by properties such as the biomolecule’s charge
and geometry, the amino acid composition of the surface (anionic, cationic,
hydrophobic, hydrophilic neutral), or the salt type and concentration of the
buffer. For instance, highly charged biomolecules such as DNA/RNA were shown
to exhibit a tight hydration layer (Pollack, 2011). For biomolecules with
large surface-to-volume ratio, such as intrinsically disordered proteins, the
hydration layer is expected to strongly contribute to the SAS signal. However,
how such properties determine the hydration layer is not yet understood on a
quantitative level (Kim et al., 2016).

  * (iii)

SAS curves are influenced by thermal fluctuations of the biomolecule. Using
analytic models, Moore (2014) showed that including Debye-Waller factors or
correlated atomic fluctuations influences the calculated SAS curves at
relatively small scattering angles of $q\gtrsim 0.2$ Å-1. These findings are
in line with earlier experimental observations (Tiede et al., 2002) and with
MD simulations, which revealed alterations of SAXS curves at $q\gtrsim 0.25$
Å-1 upon modulating atomic fluctuations during the simulations (Chen and Hub,
2014).

In the last three decades, a wide range of methods have been developed for
predicting SAS intensities from atomistic structures, reflecting the
increasing importance of SAS experiments for biomolecular research and the
need for a physically founded interpretation of the data (Svergun et al.,
1995; Merzel and Smith, 2002a, b; Tjioe and Heller, 2007; Bardhan et al.,
2009; Ravikumar et al., 2013; Yang et al., 2009; Schneidman-Duhovny et al.,
2010, 2013; Stovgaard et al., 2010; Poitevin et al., 2011; Liu et al., 2012;
Nguyen et al., 2014; Putnam et al., 2015; Grudinin et al., 2017; Oroguchi et
al., 2009; Oroguchi and Ikeguchi, 2012; Grishaev et al., 2010; Virtanen et
al., 2011; Park et al., 2009; Köfinger and Hummer, 2013; Chen and Hub, 2014;
Knight and Hub, 2015). The methods mainly differ by

  * •

the method for modeling the hydration layer, involving explicit all-atom
models, increased atomic form factors for solvent-exposed solute atoms, layers
of uniform electron density, implicit models with internal structure, or
several others;

  * •

modeling of the excluded solvent based on tabulated atomic volumes up to
explicit all-atom models;

  * •

mathematical methods for computing the orientational average involving the
Debye equation, a spherical harmonics expansion, or numerical quadrature;

  * •

the spatial resolution, i.e., whether the SAS curves are computed from
individual atoms or from coarse-grained beads.

In this chapter, we focus on the calculation of SAS curves from atomistic
explicit-solvent MD simulations as implemented in GROMACS-SWAXS, an extension
of the GROMACS simulation software (https://gitlab.com/cbjh/gromacs-swaxs)
(Chen and Hub, 2014; Chen et al., 2019). The method is easily accessible via
the WAXSiS web server at https://waxsis.uni-saarland.de (Knight and Hub,
2015). As a rationale behind the explicit-solvent methods, we first review the
approximations and limitations underlying widely used implicit-solvent
methods, with a focus on the role of the hydration layer and excluded solvent
for obtaining the density contrast. Next, we briefly describe the theory of
explicit-solvent SAS curve predictions from explicit-solvent MD simulations as
used by GROMACS-SWAXS and WAXSiS. Finally, the workflow of SAS calculations
from an MD trajectory and practical considerations are discussed.

Figure 1: (A/C) Hydration layer (B/D) displaced solvent, as represented by
(A/B) explicit-solvent and (C/D) several implicit-solvent methods. (A) A
spatial envelope is defined (blue mesh) that includes the biomolecule and the
hydration layer, here shown for the protein PCNA (pdb code 4D2G) (De Biasio et
al., 2015). (B) The same spatial envelope is used to define the displaced
solvent. (C) Implicit-solvent methods use a simplified description of the
hydration layer, often modeled as a layer of uniform density, while (D) the
displaced solvent is typically described by water dummy beads with estimated
volumes taken from a publication by Traube (1895) and fitted against the data.

##  2 Implicit-solvent methods

Computationally efficient methods for computing SAS curves, as implemented in
CRYSOL, FoXS, PepsiSAXS, or several other tools, use an implicit
representation of the hydration layer and excluded solvent. To rationalize the
benefits of explicit-solvent SAS calculations, we discuss the limitation of
implicit-solvent methods in the following.

##### Displaced solvent.

Many methods represent the displaced (or excluded) solvent with the help of
water dummy beads, which are typically placed at the positions of the
biomolecule’s heavy atoms (see Fig. 1D) (Svergun et al., 1995; Grudinin et
al., 2017). The electron density of the bead is essentially taken as
$\rho_{d}(r)=\rho_{s}\exp(-\pi r^{2}/v^{2/3})$, where $\rho_{s}$ is the
electron density of the solvent and $v$ the displaced volume, such that the
bead carries $\rho_{s}v$ electrons, as desired. The atomic form factor of the
bead is given by the Fourier transform of the electron density,
$f_{d}(q)=\rho_{s}v\exp(-v^{2/3}q^{2}/4\pi)$. In the implicit-solvent
description, the buffer subtraction is carried out by reducing the atomic form
factor of the biomolecule’s heavy atom with the form factor of the dummy bead
$f_{d}(q)$, leading to so-called “reduced form factors”.

This method requires knowledge of the displaced volumes $v$ for all atoms.
However, estimates for the displaced volume differ substantially within the
literature, leading to some uncertainty in the predicted SAS curve (Hub,
2018). Most SAXS prediction methods use and cite the displaced volumes
reported by Fraser et al. (1978). These data originally date back to a
publication in 1895 by Isidor Traube, which was based on the volume change of
water in response to the solvation of various organic compounds (Traube,
1895). Table 1 presents the atomic volumes reported by Fraser et al. in
$\mathrm{\SIUnitSymbolAngstrom}$3 as well as the original values by Traube in
ccm/mol for several atoms or chemical groups. Traube and Fraser values are
identical since $\mathrm{ccm/mol}=1.66$ Å3.

More recently, atomic volumes have been obtained by Pontius et al. using
Voronoi tessellation applied to the core region of high-resolution protein
structures (Pontius et al., 1996). The volumes by Traube and Fraser greatly
differ from the volumes obtained from crystal structures (Table 1, columns 2
and 3). Compared to the Pontius values, the Fraser values greatly overestimate
the volumes of CH and CH2 groups, and they underestimate the volumes of N and
O atoms and of NH or OH groups. Notably, Traube reported volumes of N or O
atoms for alternative chemical environments such as nitro or (today outdated)
“pentavalent” environments as well as for hydroxyl groups; however, these were
not cited by Fraser, which may explain why they are not used in SAS
calculations today (Table 1, column 4).

Table 1: Displaced volumes from Voronoi tessellation of crystal structure cores by Pontius et al. (1996), as well as from densitometric data by Traube (1895), as cited by Fraser et al. (1978). Pontius values represent averages and standard deviations from all amino acids. Traube and Fraser values are identical, using $\mathrm{ccm/mol}=1.66\,$Å3. Fraser volumes of chemical groups represent sums of single-atom volumes. Atomic group | Pontius et al. [$\mathrm{\SIUnitSymbolAngstrom}$3] | Fraser et al., 1978 [$\mathrm{\SIUnitSymbolAngstrom}$3] | Traube, 1895 [ccm/mol]
---|---|---|---
H |  | 5.15 | 3.1
C |  | 16.44 | 9.9
N | 8.8 (0.8) | 2.49 | 1.5 (trivalent)
|  |  | 10.7 (pentavalent)
|  |  | 8.5–10.7 (in nitro compound)
O | 22.3 (0.4) | 9.13 | 5.5 (carbonyl oxygen)
|  |  | 2.3 or 0.4 (hydroxy oxygen)
OH | 23.9 (0.9) | 14.28 |
NH | 14.1 (0.3) | 7.64 |
CH | 11.8 (0.6) | 21.59 |
CH2 | 20.9 (1.8) | 26.74 |
CH3 | 33.9 (1.2) | 31.89 |

Besides the uncertainty owing to the choice of tabulated atomic volumes,
additional uncertainty may arise because the atomic volumes depend on the
chemical environment. Whereas the volumes by Pontius et al. represent the
atomic packing in the core of stable proteins, it seems unlikely that the
volumes also hold for more flexible protein surfaces, for detergent micelles,
or for nucleotides. For n-hexadecane, for instance, the Pointius and Fraser
volumes would imply a molecular volume of 360.4 Å3 and 438.1 Å3, respectively,
whereas the experimental value is 486.6 Å3.

##### Hydration layer (HL).

Several tools model the hydration layer (HL) as a layer with a predefined
thickness and a uniform excess density (Fig. 1C). For instance, CRYSOL
constructs a homogeneous $3\text{\,}\mathrm{\SIUnitSymbolAngstrom}$-wide layer
described by a two-dimensional angular function leading to a simplistic
representation of the HL, which may not be suitable for molecules with
cavities or non-globular shapes (Svergun et al., 1995). CRYSOL3 aims to
overcome this limitation by incrusting the surface (Franke et al., 2017))
similar to the grid-based density employed by PepsiSAXS (Grudinin et al.,
2017). In contrast, FoXS models the HL by increasing atomic form factors of
solvent-exposed atoms (Schneidman-Duhovny et al., 2010).

The density of HLs of biomolecules likely depends on the type of the
water–surface interactions and, therefore, on the chemical composition of the
surface. For instance, Kim et al. (2016) found that acidic amino acids
(Glu/Asp) exhibit a tighter HL as compared to basic amino acids (Arg/Lys), an
effect that our group recently observed in MD simulations (unpublished data).
Likewise, it seems plausible that the HL of maltoside or sodium dodecyl
sulfate (SDS) head groups of detergent micelles exhibit different densities as
compared to the HL of charged or hydrophobic amino acids. Such chemical
specificity of the HL is not captured by implicit solvent methods, explaining
why they require fitting of the HL against experimental data.

##### Risk of overfitting solvent-related parameters

Owing to the simplified description of the HL and excluded solvent, implicit-
solvent methods require two or three parameters that are adjusted upon fitting
the calculated curve to the experimental curve. One parameter adjusts the
excess density of the HL, while another parameter adjusts the volumes of the
dummy beads that represent the excluded solvent. Some methods adjust in
addition the overall displaced volume. The parameters are typically fitted by
minimizing the residuals between the calculated and experimental SAS curves.
Whether the fitted parameters reflect the physical situation such as the true
HL density, or whether they merely absorb errors in the structural model or in
the experimental data, is typically not clear. Hence, fitting solvent-related
parameter may hide structurally relevant information.

Figure 2: SAXS curve prediction and snapshots of a protein-detergent complex.
(A) MD simulation systems of aquaporin-0 solubilized in 250 to 330 $\beta$DDM
detergent molecules simulated with the Charmm36 force field. (B) Explicit-
solvent SAXS curve predictions for different number of $\beta$DDM using an
aggregated ensemble of $90\text{\,}\mathrm{ns}100\text{\,}\mathrm{ns}$ MD
trajectories (colored lines). Experimental data (black dashed line) from
Berthaud et al. (2012). (C) $\chi$-agreement between experimental SAXS curves
and explicit-solvent SAXS calculations from free MD simulations (colored bars)
or from the WAXSiS webserver (black bars). WAXSiS restrains the DDM molecules
in a short simulation, rationalizing the increased $\chi$-values.
$N_{\beta\mathrm{DDM}}=290$ reveals the best agreement with experiment. (D)
$\chi$-agreement by CRYSOL, FoXS, and AquaSAXS, revealing nearly constant
$\chi$ due to the adjustment of free parameters. Reprinted with permission
from Chen and Hub (2015b), Copyright 2020 American Chemical Society.

A possible consequence of fitting parameters is illustrated in Fig. 2, which
presents the validation of MD simulations of a protein-detergent complex
against SAXS data (Chen and Hub, 2015b; Berthaud et al., 2012). The complexes
were composed of an aquaporin-0 tetramer solvated in 250 to 330 $\beta$DDM
detergent molecules. Computing SAXS curves from the simulations with implicit-
solvent methods and fitting the curves to experimental data leads to
$\chi$-values that hardly depend on the number of $\beta$DDM molecules,
suggesting that alterations in the structural model have been absorbed into
the fitting parameters (Fig. 2D). Consequently, the experimental number of
$\beta$DDM molecules cannot be obtained by comparison with the data. This
example illustrates that structurally relevant information may be hidden upon
fitting free parameters. In contrast, when using explicit-solvent SAXS
calculations described below, the $\chi$-agreement of models with different
$\beta$DDM numbers with the data can be distinguished because no solvent-
related parameters are fitted, thereby revealing that the model with 290
$\beta$DDM exhibits the best agreement with the data (Fig. 2C).

##  3 Explicit-solvent SAS predictions with the WAXSiS method

Several methods for predicting SAS curves have been presented that model both
the excluded solvent and the HL with explicit-solvent MD simulations (Oroguchi
et al., 2009; Park et al., 2009; Köfinger and Hummer, 2013; Chen and Hub,
2014). Here, we focus on the WAXSiS method, which is based on the formalism by
Park et al. (2009). However, whereas the implementation by Park et al.
required a constrained biomolecule, the WAXSiS method allows SAS predictions
from MD simulations of flexible biomolecules by defining the SAS-contributing
solvent with a spatial envelope. In addition, the WAXSiS method corrects (i)
imprecise densities of water models and (ii) small density mismatches between
the biomolecule and the pure-solvent simulation systems, both of which may
influence the SAS prediction significantly. A careful comparison of different
implicit and explicit SAXS prediction methods has been presented by Bernetti
and Bussi (2021).

Figure 3: (A) Solvent electron density around Rnase A, taken from an explicit-
solvent MD simulations with position restraints on all heavy atoms, revealing
the marked structure of the hydration layer. Only solvent density inside an
envelope (blue mesh) is shown, where the envelope was constructed at a 7Å
distance from the protein surface. (B) Solvent density as function of the
distance from the Van der Waals surface for three different proteins (see
legend), averaged over the protein surface. Simulations were carried out with
the Amber99sb-ildn/Tip3p force fields. Here, the solvent density was scaled
with a small constant factor to match the experimental bulk density of 334 e
nm-3, as implemented in GROMACS-SWAXS to correct imprecise densities of some
water models.

In the WAXSiS method, a spatial envelope is defined that encloses the
biomolecule and the heterogeneous density of the HL (Fig. 1A, Fig. 3A). Hence,
the HL structure is fully defined by the force field and does not require a
free fitting parameter. The buffer subtraction is carried out by computing the
scattering intensity difference between (a) the biomolecule with the envelope-
enclosed solvent and (b) an explicit-solvent pure-buffer simulation, from
which an identical volume is taken with the help of the same envelope (Fig
1B). Thereby, the displaced solvent is modeled with atomic detail and without
need of knowing atomic volumes of solute atoms.

Compared to implicit-solvent methods, the explicit-solvent calculations are
computationally far more expensive as they require MD simulations, but they
have several crucial advantages:

  * (i)

The calculations do not require any free solvent-related parameters that must
be fitted to experimental data. Thus, the amount of structural information,
which can be extracted from the low-information SAS data, is not further
reduced. By circumventing the fitting of the HL density, the radius of
gyration is not adjusted against the data.

  * (ii)

The use of “reduced form factors” is avoided and, with this, the uncertainties
subject to atomic volumes. This may explain the better accuracy compared to
implicit-solvent methods for predicting the SAS intensities of heterogeneous
systems (such as protein-detergent complexes) and for molecules with non-
globular geometries (Ivanović et al., 2018a, 2020).

  * (iii)

The explicit-solvent description remains valid at wide scattering angles
($q>$$1.5\text{\,}\mathrm{\SIUnitSymbolAngstrom}$), where the internal
structure of water becomes relevant (Park et al., 2009).

  * (iv)

By computing the SAS patterns from an MD trajectory, the method naturally
accounts for thermal fluctuations, which influence the SAS patterns at
remarkably small scattering angles of $q\gtrsim 0.25$ Å-1 (Tiede et al., 2002;
Moore, 2014; Chen and Hub, 2014, 2015b).

Figure 4: (A) Experimental SAXS data of the ring-shaped protein PCNA (grey)
and SAXS curve computed with the WAXSiS model from a PCNA model with modeled
loops $\left(\chi^{2}=1.35\right)$ (Cordeiro et al., 2017). (B) Comparison of
SAXS curves of the PCNA crystal structure (pdb code 4D2G) computed with
different methods with default settings (see legend). Panel A adapted and
reused with permission from Cordeiro et al. (2017).

As an example, Fig. 4A compares the experimental SAXS curve of the ring-shaped
protein PCNA with a curve computed with the WAXSiS method. The WAXSiS curve
was computed from the 4D2G crystal structure of PCNA, to which missing termini
were added (Cordeiro et al., 2017), and the computed curve was henceforth
scaled with a constant factor
($I_{\mathrm{scaled}}(q)=f\,I_{\mathrm{WAXSiS}}(q)$). No other parameters were
adjusted. Figure 4B compares SAXS curves computed from the 4D2G structure
(without termini) computed with WAXSiS and with several implicit-solvent
methods. All methods used default settings, and the calculated curves were not
fitted against the experiment. Evidently, the predicted SAXS curves differ
significantly, presumably owing to different modeling of the hydration layer
and excluded solvent.

##  4 Theory

The experimental SAS intensity is given by the difference between the
scattering intensity of the solution $I_{\mathrm{sample}}(q)$ and the solvent
$I_{\mathrm{solvent}}(q)$:

$I(q)=I_{\mathrm{sample}}(q)-I_{\mathrm{solvent}}(q)$ (3)

To calculate this excess intensity $I(q)$ from MD trajectories, the WAXSiS
method follows the formalism by Park et al. (2009). The low-dilution limit is
considered, such that correlations between different solute molecules can be
neglected and the scattering experiment can be modeled by a single solute
molecule in a solvent bath. A spatial envelope is constructed around the
solute including the hydration layer (Fig. 1A). The instantaneous electron
density of the solute system $A(\textbf{r})$ and of the solvent system
$B(\textbf{r})$ is divided into the electron density inside and outside of the
envelope, indicated by subscripts $\mathrm{i}$ and $\mathrm{o}$, respectively,

$\displaystyle A(\textbf{r})$
$\displaystyle=A_{\mathrm{i}}(\textbf{r})+A_{\mathrm{o}}(\textbf{r})\,,$ (4)
$\displaystyle B(\textbf{r})$
$\displaystyle=B_{\mathrm{i}}(\textbf{r})+B_{\mathrm{o}}(\textbf{r})\,.$ (5)

Assuming that the envelope is large enough such that density correlations
between the inside and the outside of the envelope are due to bulk water, the
excess scattering intensity only requires knowledge of the Fourier transforms
of the electron densities inside the envelope
$\tilde{A}_{\mathrm{i}}(\textbf{q})$ and $\tilde{B}_{\mathrm{i}}(\textbf{q})$:

$I(q)=\big{\langle}D(\textbf{q})\big{\rangle}_{\Omega}\,,$ (6)

where

$D(\textbf{q})=\bigl{\langle}\bigl{|}\tilde{A}_{\mathrm{i}}(\textbf{q})\bigr{|}^{2}\bigr{\rangle}^{(\omega)}-\bigl{\langle}\bigl{|}\tilde{B}_{\mathrm{i}}(\textbf{q})\bigr{|}^{2}\bigr{\rangle}^{(\omega)}+2\mathrm{Re}\bigl{[}-\bigl{\langle}\tilde{B}_{\mathrm{i}}^{*}(\textbf{q})\bigr{\rangle}^{(\omega)}\bigl{\langle}\tilde{A}_{\mathrm{i}}(\textbf{q})-\tilde{B}_{\mathrm{i}}(\textbf{q})\bigr{\rangle}^{(\omega)}].$
(7)

Here, $\langle...\rangle_{\Omega}$ denotes the orientational average in
$\mathbf{q}$-space, $\langle...\rangle^{(\omega)}$ the average over solute and
solvent fluctuations at fixed solute orientation $\omega$, and $\mathrm{Re}$
is the real part. In practice, $\langle...\rangle^{(\omega)}$ represents the
average over MD frames after superimposing the solute onto a reference
structure of orientation $\omega$. $\tilde{A}_{\mathrm{i}}(\textbf{q})$ and
$\tilde{B}_{\mathrm{i}}(\textbf{q})$ are calculated using the atomic form
factors $f_{j}(q)$ and coordinates $\textbf{r}_{j}$ of atom $j$,

$\tilde{A}_{\mathrm{i}}(\textbf{q})=\sum_{j=1}^{N_{A}}f_{j}(q)e^{-i\textbf{q}\cdot\textbf{r}_{j}}\,,$
(8)

where $N_{A}$ is the number of atoms within the envelope in the respective MD
frame (Fig. 1A). $\tilde{B}_{i}(\textbf{q})$ is calculated analogously over
the solvent atoms within the envelope (Fig. 1B). The form factors are
approximated using the Cromer-Mann parameters $a_{k},b_{k}$ and $c$ of atom
$j$ (Cromer and Mann, 1968),

$f_{j}(q)=\sum_{k=1}^{4}a_{k}e^{-b_{k}(q/4\pi)^{2}}+c\,.$ (9)

The formalism is identical for SANS calculations, except that the atomic form
factors $f_{j}(q)$ are replaced by the coherent neutron scattering lengths
$b_{j}$ of the scattering atoms (Chen et al., 2019; Dias-Mirandela et al.,
2018).

#### Comparing to experimental data

To compare a predicted curve with experimental data, the experimental curve
$I_{\mathrm{exp}}$ can be fitted to the calculated curve $I_{\mathrm{c}}$ by
minimizing

$\chi^{2}=N_{q}^{-1}\sum_{i=1}^{N_{q}}\frac{\bigl{[}I_{\mathrm{c}}(q_{i})-\bigl{(}fI_{\mathrm{exp}}(q_{i})+c\bigr{)}\bigr{]}^{2}}{\sigma_{i}^{2}}\,,$
(10)

where $N_{q}$ denotes the number of $q$-points, and $\sigma_{i}$ are the
experimental errors. Besides the overall scale $f$, an offset $c$ may be
fitted to account for small experimental uncertainties from the buffer
subtraction. Here, the experimental curves are fitted instead of the
calculated curves, because the calculated curves are predicted without any
free parameters. In practice, it is advisable to fit also without the offset
$c$ and, thereby, to test whether $c$ absorbs only a small constant offset, as
desired, or whether fitting the offset would hide a structurally relevant
discrepancy between $I_{\mathrm{exp}}$ and $I_{\mathrm{c}}$.

##  5 Workflow

We present a brief workflow for computing a SAS curve from an explicit-solvent
MD simulation (Fig. 5). Because the SAS calculations are implemented into an
extension of GROMACS, it is convenient (but not required) to run the MD
simulation with GROMACS as well. This workflow is automated also on the web
server WAXSiS (https://waxsis.uni-saarland.de). However, the MD simulations on
WAXSiS are carried out with the Yasara software which allows fully automated
MD simulations even in presence of ligands or modified amino acids (Krieger
and Vriend, 2015). Several additional tutorials and more detailed
documentation for GROMACS-SWAXS are available at
https://cbjh.gitlab.io/gromacs-swaxs-docs.

Figure 5: Workflow for SAS curve predictions with GROMACS-SWAXS. *Suggested
mdp parameters are shown in listing 1.

###  5.1 A: With any GROMACS

1\. Run an MD simulation of your biomolecule in solution with either GROMACS-
SWAXS or any other GROMACS version. Choose the simulation box large enough,
such that the envelope (see below) will fit into the box. To generate the box
use, e.g.,

⬇

gmx editconf -d 1.5 -f protein.gro -bt dodecahedron -o box.gro

where for the -d option a distance between 1.5 and 2.0 is usually sufficient.

2\. Run an MD simulation of pure solvent for the buffer subtraction. Use the
same water model and MD parameter (mdp) file as used for the solute
simulations to ensure that solute and solvent simulations exhibit identical
bulk solvent densities, as required for an accurate buffer subtraction. The
solvent simulation box should be at least as large as your solute simulation
box. Produce $1\text{\,}\mathrm{ns}5\text{\,}\mathrm{ns}$ of simulation,
writing the simulation frames every
$1\text{\,}\mathrm{ps}5\text{\,}\mathrm{ps}$, i.e. $\sim$1000 frames.

###  5.2 B: With GROMACS-SWAXS

When the simulations of solute and pure solvent have finished, we continue
with preparing and running the SAS curve calculation. Once the steps below are
understood, they can be scripted to automate the SAS curve calculation.

1\. Download and compile GROMACS-SWAXS (https://gitlab.com/cbjh/gromacs-
swaxs), and load it into your path:

⬇

source /path/to/gromacs-swaxs/bin/GMXRC

####  5.2.1 Specify the atomic form factors/neutron scattering lengths

2\. Generate a run-input (tpr) file using gmx grompp of GROMACS-SWAXS to
detect the chemical elements, which are used below for assigning the Cromer-
Mann parameters or neutron scattering lengths. Chemical elements are
recognized using atom names and masses to ensure that a Cα carbon atom (C) is
not confused with calcium (Ca), Fluor (F) not with Iron (Fe), etc. Atomic
masses are reliably provided by a tpr file. An empty mdp file may be used for
this step:

⬇

rm -f anymdp.mdp; touch anymdp.mdp

gmx grompp -f anymdp.mdp -p topol.top -o tmp.tpr

3\. Now, the previously created tpr file is used to generate a scatter.itp
file for each molecule, which contains the Cromer-Mann parameters or the
neutron scattering lengths. If the solute contains non-default groups, such as
chromophore, a heme group, or similar, first prepare an index file with the
entire solute. Then run:

⬇

gmx genscatt -s tmp.tpr [-n index.ndx] [-vsites]

Select your complete solute, such as Protein, Protein_HEME or
Protein_Chromophore. If you use virtual sites, use -vistes and select Prot-
Masses (or a group with heme, the chromophore, etc.). The -visites option
instructs gmx genscatt that atomic masses deviate from the physical masses, as
common when using virtual sites. gmx genscatt writes one itp file for each
molecule type. These must be added to each moleculetype definition in the
topology, similar to the #include "posres.itp" line for defining position
restraints.

####  5.2.2 Generating the envelope

4\. Generate the envelope from the trajectory of the solute simulations

⬇

gmx genscatt -s tmp.tpr -f solutetraj.xtc -d 0.7

The output envelope.dat lists inner and outer radii of the envelope surface,
whereas envelope-ref.gro is a reference structure with orientation $\omega$
used to superimpose the solute frames onto the envelope (see Theory). The
envelope may be visualized with PyMol with the file envelope.py.

####  5.2.3 Generating the solute and solvent run-input files

5\. Generate tpr files with gmx grompp and with an mdp file that contains all
SAS-specific parameters, as presented in Listing 1.

⬇

gmx grompp -f rerun.mdp -c solutesystem.gro [-n index.ndx] -o solute.tpr

In addition, a tpr file of the pure-solvent system is required, prepared with
a mdp file with an empty solute entry (waxs-solute =)

⬇

gmx grompp -f solvent.mdp -c solventsystem.gro -o solvent.tpr

Listing 1: A typical set of SAS-specific mdp parameters for calculating a
SWAXS/SANS curve with GROMACS-SWAXS. The calculation of SANS curves is
optional, as indicated by square brackets.

⬇

; read scattering info from topology

define = -DSCATTER

; turn on SAXS calc. and, optionally, multiple SANS calc.

scatt-coupl = xray [neutron neutron ...]

; solute group

waxs-solute = Protein ; or Protein-Masses

; solvent group

waxs-solvent = Water_and_ions

; rotational fit group as used with gmx genenv

waxs-rotfit = C-alpha

; or use a PBC atom near the geometric center, as suggested by gmx genenv

waxs-pbcatom = -1

; for uniform average over all simulation frames

waxs-tau = 0

; number of q-points

waxs-nq = 101

; qmin and qmax in nm^(-1)

waxs-startq = 0

waxs-endq = 10

; number of q-vectors for spherical average, use ~0.2*(D*qmax)^2

waxs-nsphere = 1500

; experimental solvent density in e/nm3, for solvent density correction

waxs-solvdens = 334

; Use I(sample)-I(buffer) as buffer subtraction scheme

waxs-correct-buffer = no

; D2O concentrations of SANS calculations

waxs-deuter-conc = [0.42 1]

####  5.2.4 Compute the SAS curve

6\. Finally, compute the SAS curves from the solute simulations with the rerun
functionality of gmx mdrun. The envelope files are specified with environment
variables, as follows:

⬇

export GMX_ENVELOPE_FILE=envelope.dat

export GMX_WAXS_FIT_REFFILE=envelope-ref.gro

gmx mdrun -s solute.tpr -rerun solutetraj.xtc \

-sw solvent.tpr -fw solventtraj.xtc

The SAS calculations strongly benefit from the use of a GPU or from using
multiple CPU cores, for instance using gmx mdrun -ntomp 16 -gpu_id 0 .... The
SAS curves are written to the files waxs_final.xvg.

##  6 Practical considerations

###  6.1 Convergence

The number of simulation frames required to receive a converged SAS curve
depends on the contrast between solute and solvent, where a lower contrast
leads to slower convergence. The contrast is lower if the hydration layer is
large compared to the biomolecule, i.e., if the envelope contains mostly
water. Specifically, small biomolecules or intrinsically disordered proteins
(IDPs) require a larger number of simulation frames as compared to larger or
globular biomolecules. As a reference, for a smaller protein such as the GB3
domain, approximately 300 frames are required to obtain a well-converged SAS
curve. A larger protein such as glucose isomerase requires approx. 70 frames.
The statistical errors listed in waxs_final.xvg are computed with error
propagation, assuming that each simulation frame provides an independent
solvent structure, which is fulfilled when using frames with time spacing
$\gtrsim 1$ ps.

For IDPs, the convergence should be carefully assessed, for instance by
dividing the trajectory of both the biomolecule and the buffer into ten
independent blocks, and then comparing the ten SAS curves computed from the
blocks. The standard error over the blocks provides a rigorous measure for
statistical uncertainty.

###  6.2 Orientational average

The parameter $J$ (mdp option waxs-nsphere) determines the number of q-vectors
used to take the orientational average. It should be taken as
$J\approx\alpha(Dq_{\mathrm{max}})^{2}$. Where $D$ is the maximum diameter of
the solute, $q_{\mathrm{max}}$ the maximum momentum transfer (mdp option waxs-
endq), and the constant $\alpha$ determines the accuracy of the orientational
average. For most biomolecules, $\alpha=0.2$ is sufficient, while for
excessively elongated rod-like structures a value of $\alpha=0.5$ may be
required.

###  6.3 Envelope size

The distance of the envelope from the biomolecular surface (specified with gmx
genenv -d) should be at least
$0.6\text{\,}\mathrm{nm}0.7\text{\,}\mathrm{nm}$, thereby containing the
density modulations along the hydration layers. For highly charged
biomolecules, a larger envelope may be used to account for the entire counter
ion cloud. We found previously that a distance of $\sim$3 times the Debye
length is required to include most effects from the counter ion cloud on the
radius of gyration of glucose isomerase (Ivanović et al., 2018b). At a salt
concentration of 100 mM, this may require an envelope distance of $\sim$3 nm
and, hence, large simulation systems.

###  6.4 Solvent density correction

A solvent density correction is necessary for two reasons: (i) Due to finite-
size effects, the bulk solvent densities and correlations may slightly differ
between the solute simulation and the pure-solvent simulation (Köfinger and
Hummer, 2013). (ii) The density of certain water models, such as the popular
Tip3p model, deviate from the experimental water densities. The parameter for
solvent density correction should be chosen to match the experimental electron
density of the solvent (mdp option waxs-solvdens).

###  6.5 Atomic fluctuations

Structural fluctuations influence the SAS profile. Depending on the scientific
question, position restraints may or may not be applied to the biomolecule,
for instance with the aim to obtain the SAS curve for structure with given
backbone coordinates or for a flexible biomolecular ensemble, respectively. To
resolve the structure of the HL, it may be useful to restrain all heavy atoms
of the biomolecule to avoid that side chain fluctuations smear out the HL
structure, as used to obtain the solvent densities shown in Fig. 3.

###  6.6 Water models

We validated previously that most popular water force fields lead to nearly
identical SAS curves at moderate scattering angles, suggesting that most water
models exhibit similar packing at the biomolecular surface (Chen and Hub,
2014). Only at very wide angle of $q\gtrsim 2$ Å-1, where the water scattering
is dominant, large effects from the water models are visible. However, we
recently observed that water models with increased dispersion interactions
such as the Tip4p2005s or Tip4p-D may impose tighter packing on the protein
surface, leading to slightly increased radii of gyration (unpublished data)
(Best et al., 2014; Piana et al., 2015). These observations open a novel route
for validating protein–water interactions by modern force fields against
experimental SAS data.

###  6.7 Computational costs

Run times of SAS predictions depend on the number of atoms, on the applied
convergence criteria, and on the hardware. Typical costs may be illustrated by
the execution time for different proteins on the WAXSiS webserver at
https://waxsis.uni-saarland.de, which is currently (by August 2022) equipped
with a 16-core AMD Ryzen 9 processor and an Nvidia RTX 3070Ti graphics card.
Using the convergence setting “normal”, which is suitable for tests, SAS
predictions of lysozyme (PDB id 1LYS), glucose isomerase (PDB id 1MNZ), and
RNA polymerase II (PDB id 1TWF) required
$2.7\text{\,}\mathrm{m}\mathrm{i}\mathrm{n}$,
$4.8\text{\,}\mathrm{m}\mathrm{i}\mathrm{n}$, and
$15.2\text{\,}\mathrm{m}\mathrm{i}\mathrm{n}$, respectively. Using the
convergence setting “thorough”, which is recommended for publications, the
execution times for the same biomolecules increased to
$10.8\text{\,}\mathrm{m}\mathrm{i}\mathrm{n}$,
$14.2\text{\,}\mathrm{m}\mathrm{i}\mathrm{n}$, and
$36.3\text{\,}\mathrm{m}\mathrm{i}\mathrm{n}$, respectively. These numbers may
change as the hardware improves or as the convergence criteria are adapted in
future releases of the WAXSiS webserver.

##  7 Summary

Both implicit- and explicit-solvent SAS prediction methods are highly valuable
for the structural interpretation of experimental SAS methods. Implicit-
solvent methods are computationally efficient, thus allowing high-throughput
calculations on a laptop. They are unequivocally capable of validating the
overall shape of structural models against SAS data, and to reveal effects
from larger conformational transitions. To harvest structural details, as
encoded in high-precision SAS data collected from modern SEC-SAXS or SEC-SANS
experiments, explicit-solvent methods are needed. They are (i) based on an
accurate representation of the hydration layer and the excluded solvent,
thereby avoiding solvent-related fitting parameters, (ii) are accurate for
inhomogeneous systems since they do not require tabulated atomic volumes, and
(iii) include effects from structural fluctuations on SAS curves. The WAXSiS
method presented here may be further used for structure refinement simulations
by restraining an MD simulation to experimental SAS data (Chen and Hub, 2015a;
Chen et al., 2019). More details on structure refinement are provided in a
chapter of Part B of this monograph.

Running a SAS prediction on the WAXSiS webserver requires between 3 and 40 min
including the MD simulation. On a fast workstation or a computer cluster, tens
to hundreds of explicit-solvent calculations are feasible within days,
suggesting that the cost for explicit-solvent SAS calculations are negligible
as compared to the costs for experimental sample preparation or for data
collection at the beamline. Therefore, we expect that the methods presented
here will become a routinely used tool for SAS-based structural biology.

##  8 Acknowledgements

We thank Tobias Fischbach for support with preparing Figure 2A. This study was
supported by the Deutsche Forschungsgemeinschaft (HU 1971-3/1).

## References

  * Arnlund et al. (2014) Arnlund, D., Johansson, L. C., Wickstrand, C., Barty, A., Williams, G. J., Malmerberg, E., Davidsson, J., Milathianaki, D., DePonte, D. P., Shoeman, R. L., et al. (2014).  Visualizing a protein quake with time-resolved X-ray scattering at a free-electron laser.  Nat. Methods, 11(9):923–926.
  * Bardhan et al. (2009) Bardhan, J., Park, S., and Makowski, L. (2009).  It SoftWAXS: A computational tool for modeling wide-angle X-ray solution scattering from biomolecules.  J. Appl. Cryst., 42(5):932–943.
  * Bernetti and Bussi (2021) Bernetti, M. and Bussi, G. (2021).  Comparing state-of-the-art approaches to back-calculate SAXS spectra from atomistic molecular dynamics simulations.  Eur. Phys. J. B, 94(9):180.
  * Berthaud et al. (2012) Berthaud, A., Manzi, J., Pérez, J., and Mangenot, S. (2012).  Modeling detergent organization around aquaporin-0 using small-angle X-ray scattering.  J. Am. Chem. Soc., 134(24):10080–10088.
  * Best et al. (2014) Best, R. B., Zheng, W., and Mittal, J. (2014).  Balanced Protein–Water Interactions Improve Properties of Disordered Proteins and Non-Specific Protein Association.  Journal of Chemical Theory and Computation, 10(11):5113–5124.
  * Brinkmann and Hub (2016) Brinkmann, L. U. L. and Hub, J. S. (2016).  Ultrafast anisotropic protein quake propagation after CO-photodissociation in myoglobin.  Proc. Natl. Acad. Sci. USA, 113:10565–10570.
  * Chen and Hub (2014) Chen, P.-c. and Hub, J. S. (2014).  Validating solution ensembles from molecular dynamics simulation by wide-angle X-ray scattering data.  Biophys. J., 107:435–447.
  * Chen and Hub (2015a) Chen, P.-c. and Hub, J. S. (2015a).  Interpretation of solution X-ray scattering by explicit-solvent molecular dynamics.  Biophys. J., 108:2573–2584.
  * Chen and Hub (2015b) Chen, P.-c. and Hub, J. S. (2015b).  Structural Properties of Protein-Detergent Complexes from SAXS and MD Simulations.  J. Phys. Chem. Lett., 6:5116–5121.
  * Chen et al. (2019) Chen, P.-c., Shevchuk, R., Strnad, F. M., Lorenz, C., Karge, L., Gilles, R., Stadler, A. M., Hennig, J., and Hub, J. S. (2019).  Combined Small-Angle X-ray and Neutron Scattering Restraints in Molecular Dynamics Simulations.  Journal of Chemical Theory and Computation, 15(8):4687–4698.
  * Cordeiro et al. (2017) Cordeiro, T. N., Chen, P.-c., De Biasio, A., Sibille, N., Blanco, F. J., Hub, J. S., Crehuet, R., and Bernadó, P. (2017).  Disentangling polydispersity in the PCNA-p15PAF complex, a disordered, transient and multivalent macromolecular assembly.  Nucleic Acids Res., 45:1501–2015.
  * Cromer and Mann (1968) Cromer, D. T. and Mann, J. B. (1968).  X-ray scattering factors computed from numerical Hartree-Fock wave functions.  Acta Cryst. A, 24(2):321–324.
  * De Biasio et al. (2015) De Biasio, A., de Opakua, A. I., Mortuza, G. B., Molina, R., Cordeiro, T. N., Castillo, F., Villate, M., Merino, N., Delgado, S., Gil-Cartón, D., Luque, I., Diercks, T., Bernadó, P., Montoya, G., and Blanco, F. J. (2015).  Structure of p15PAF–PCNA complex and implications for clamp sliding during DNA replication and repair.  Nat Commun, 6(1):6439.
  * Dias-Mirandela et al. (2018) Dias-Mirandela, G., Tamburrino, G., Ivanović, M. T., Strnad, F. M., Byron, O., Rasmussen, T., Hoskisson, P. A., Hub, J. S., Zachariae, U., Gabel, F., Hoskisson, P., Hub, J. S., Zachariae, U., Gabel, F., and Javelle, A. (2018).  Merging in-solution X-ray and neutron scattering data allows fine structural analysis of membrane-protein detergent complexes.  J. Phys. Chem. Lett., 9:3910–3914.
  * Franke et al. (2017) Franke, D., Petoukhov, M. V., Konarev, P. V., Panjkovich, A., Tuukkanen, A., Mertens, H. D. T., Kikhney, A. G., Hajizadeh, N. R., Franklin, J. M., Jeffries, C. M., and Svergun, D. I. (2017).  _ATSAS_ _2.8_ : A comprehensive data analysis suite for small-angle scattering from macromolecular solutions.  J Appl Crystallogr, 50(4):1212–1225.
  * Fraser et al. (1978) Fraser, R., MacRae, T., and Suzuki, E. (1978).  An improved method for calculating the contribution of solvent to the X-ray diffraction pattern of biological molecules.  J. Appl. Cryst., 11(6):693–694.
  * Gabel (2015) Gabel, F. (2015).  Small-angle neutron scattering for structural biology of protein–rna complexes.  In Methods in Enzymology, volume 558, pages 391–415. Elsevier.
  * Graewert and Svergun (2013) Graewert, M. A. and Svergun, D. I. (2013).  Impact and progress in small and wide angle X-ray scattering (SAXS and WAXS).  Curr. Opin. Struct. Biol., 23(5):748–754.
  * Grishaev et al. (2010) Grishaev, A., Guo, L., Irving, T., and Bax, A. (2010).  Improved fitting of solution X-ray scattering data to macromolecular structures and structural ensembles by explicit water modeling.  J. Am. Chem. Soc., 132:15484–15486.
  * Grudinin et al. (2017) Grudinin, S., Garkavenko, M., and Kazennov, A. (2017).  Pepsi-SAXS: An adaptive method for rapid and accurate computation of small-angle X-ray scattering profiles.  Acta. Cryst. D, 73(5).
  * Hub (2018) Hub, J. S. (2018).  Interpreting solution X-ray scattering data using molecular simulations.  Curr. Opin. Struct. Biol., 49:18–26.
  * Ivanović et al. (2018a) Ivanović, M. T., Bruetzel, L. K., Lipfert, J., and Hub, J. S. (2018a).  Temperature-dependent atomic models of detergent micelles refined against small-angle X-ray scattering data.  Angew. Chem. Int. Ed., 57:5635–5639.
  * Ivanović et al. (2018b) Ivanović, M. T., Bruetzel, L. K., Shevchuk, R., Lipfert, J., and Hub, J. S. (2018b).  Quantifying the influence of the ion cloud on SAXS profiles of charged proteins.  Physical Chemistry Chemical Physics, 20(41):26351–26361.
  * Ivanović et al. (2020) Ivanović, M. T., Hermann, M. R., Wójcik, M., Pérez, J., and Hub, J. S. (2020).  Small-Angle X-ray Scattering Curves of Detergent Micelles: Effects of Asymmetry, Shape Fluctuations, Disorder, and Atomic Details.  The Journal of Physical Chemistry Letters, 11(3):945–951.
  * Kim et al. (2016) Kim, H. S., Martel, A., Girard, E., Moulin, M., Härtlein, M., Madern, D., Blackledge, M., Franzetti, B., and Gabel, F. (2016).  SAXS/SANS on supercharged proteins reveals residue-specific modifications of the hydration shell on supercharged proteins reveals residue-specific modifications of the hydration shell.  Biophys. J., 110(10):2185–2194.
  * Knight and Hub (2015) Knight, C. J. and Hub, J. S. (2015).  WAXSiS: A web server for the calculation of SAXS/WAXS curves based on explicit-solvent molecular dynamics.  Nucleic Acids Res., 43:W225–W230.
  * Koch et al. (2003) Koch, M. H., Vachette, P., and Svergun, D. I. (2003).  Small-angle scattering: A view on the properties, structures and structural changes of biological macromolecules in solution.  Quart. Rev. Biophys., 36(02):147–227.
  * Köfinger and Hummer (2013) Köfinger, J. and Hummer, G. (2013).  Atomic-resolution structural information from scattering experiments on macromolecules in solution.  Phys. Rev. E., 87:052712.
  * Krieger and Vriend (2015) Krieger, E. and Vriend, G. (2015).  New ways to boost molecular dynamics simulations.  J. Comput. Chem., 36(13):996–1007.
  * Levantino et al. (2015) Levantino, M., Schiro, G., Lemke, H. T., Cottone, G., Glownia, J. M., Zhu, D., Chollet, M., Ihee, H., Cupane, A., and Cammarata, M. (2015).  Ultrafast myoglobin structural dynamics observed with an X-ray free-electron laser.  Nature Commun., 6:6772.
  * Liu et al. (2012) Liu, H., Morris, R. J., Hexemer, A., Grandison, S., and Zwart, P. H. (2012).  Computation of small-angle scattering profiles with three-dimensional Zernike polynomials.  Acta Cryst. A, 68(2):278–285.
  * Merzel and Smith (2002a) Merzel, F. and Smith, J. C. (2002a).  Is the first hydration shell of lysozyme of higher density than bulk water?  Proc. Natl. Acad. Sci. USA, 99:5378–5383.
  * Merzel and Smith (2002b) Merzel, F. and Smith, J. C. (2002b).  SASSIM: A method for calculating small-angle X-ray and neutron scattering and the associated molecular envelope from explicit-atom models of solvated proteins.  Acta Cryst. D, 58:242–249.
  * Moore (2014) Moore, P. B. (2014).  The Effects of Thermal Disorder on the Solution-Scattering Profiles of Macromolecules.  Biophys. J., 106:1489–1496.
  * Nguyen et al. (2014) Nguyen, H. T., Pabit, S. A., Meisburger, S. P., Pollack, L., and Case, D. A. (2014).  Accurate small and wide angle x-ray scattering profiles from atomic models of proteins and nucleic acids.  J. Chem. Phys., 141(22):22D508.
  * Oroguchi et al. (2009) Oroguchi, T., Hashimoto, H., Shimizu, T., Sato, M., and Ikeguchi, M. (2009).  Intrinsic Dynamics of Restriction Endonuclease it EcoO109I Studied by Molecular Dynamics Simulations and X-Ray Scattering Data Analysis.  Biophys. J., 96:2808–2822.
  * Oroguchi and Ikeguchi (2012) Oroguchi, T. and Ikeguchi, M. (2012).  MD–SAXS method with nonspherical boundaries.  Chem. Phys. Lett., 541:117–121.
  * Park et al. (2009) Park, S., Bardhan, J. P., Roux, B., and Makowski, L. (2009).  Simulated x-ray scattering of protein solutions using explicit-solvent models.  J Chem Phys, 130:134114.
  * Piana et al. (2015) Piana, S., Donchev, A. G., Robustelli, P., and Shaw, D. E. (2015).  Water Dispersion Interactions Strongly Influence Simulated Structural Properties of Disordered Protein States.  The Journal of Physical Chemistry B, 119(16):5113–5123.
  * Poitevin et al. (2011) Poitevin, F., Orland, H., Doniach, S., Koehl, P., and Delarue, M. (2011).  AquaSAXS: A web server for computation and fitting of SAXS profiles with non-uniformally hydrated atomic models.  Nucl. Acids Res., 39:W184–W189.
  * Pollack (2011) Pollack, L. (2011).  Saxs studies of ion–nucleic acid interactions.  Annual review of biophysics, 40:225–242.
  * Pontius et al. (1996) Pontius, J., Richelle, J., and Wodak, S. J. (1996).  Deviations from standard atomic volumes as a quality measure for protein crystal structures.  J. Mol. Biol., 264(1):121–136.
  * Putnam et al. (2007) Putnam, C. D., Hammel, M., Hura, G. L., and Tainer, J. A. (2007).  X-ray solution scattering (SAXS) combined with crystallography and computation: Defining accurate macromolecular structures, conformations and assemblies in solution.  Quart. Rev. Biophys., 40(3):191–285.
  * Putnam et al. (2015) Putnam, D. K., Weiner, B. E., Woetzel, N., Lowe, E. W., and Meiler, J. (2015).  BCL::SAXS: GPU accelerated Debye method for computation of small angle X-ray scattering profiles.  Proteins Struct. Funct. Bioinf., 83(8):1500–1512.
  * Ravikumar et al. (2013) Ravikumar, K. M., Huang, W., and Yang, S. (2013).  Fast-SAXS-pro: A unified approach to computing SAXS profiles of DNA, RNA, protein, and their complexes.  J. Chem. Phys., 138(2):01B609.
  * Schneidman-Duhovny et al. (2010) Schneidman-Duhovny, D., Hammel, M., and Sali, A. (2010).  FoXS: A web server for rapid computation and fitting of SAXS profiles.  Nucl. Acids Res., 38:W540–W544.
  * Schneidman-Duhovny et al. (2013) Schneidman-Duhovny, D., Hammel, M., Tainer, J. A., and Sali, A. (2013).  Accurate SAXS profile computation and its assessment by contrast variation experiments.  Biophys. J., 105(4):962–974.
  * Stovgaard et al. (2010) Stovgaard, K., Andreetta, C., Ferkinghoff-Borg, J., and Hamelryck, T. (2010).  Calculation of accurate small angle X-ray scattering curves from coarse-grained protein models.  BMC Bioinformatics, 11(1):429.
  * Svergun et al. (1995) Svergun, D., Barberato, C., and Koch, M. H. J. (1995).  CRYSOL – a Program to Evaluate X-ray Solution Scattering of Biological Macromolecules from Atomic Coordinates.  J. Appl. Cryst., 28:768–773.
  * Tiede et al. (2002) Tiede, D. M., Zhang, R., and Seifert, S. (2002).  Protein conformations explored by difference high-angle solution x-ray scattering: Oxidation state and temperature dependent changes in cytochrome c.  Biochemistry, 41(21):6605–6614.
  * Tjioe and Heller (2007) Tjioe, E. and Heller, W. T. (2007).  Ornl_sas: software for calculation of small-angle scattering intensities of proteins and protein complexes.  Journal of Applied Crystallography, 40(4):782–785.
  * Traube (1895) Traube, J. (1895).  Ueber das Molekularvolumen.  Ber. Dtsch. Chem. Ges., 28(3):2722–2728.
  * Virtanen et al. (2011) Virtanen, J. J., Makowski, L., Sosnick, T. R., and Freed, K. F. (2011).  Modeling the hydration layer around proteins: Applications to small-and wide-angle x-ray scattering.  Biophys. J., 101(8):2061–2069.
  * Yang et al. (2009) Yang, S., Park, S., Makowski, L., and Roux, B. (2009).  A rapid coarse residue-based computational method for X-ray solution scattering characterization of protein folds and multiple conformational states of large protein complexes.  Biophys. J., 96(11):4449–4463.


# Seven Failure Points When Engineering a Retrieval Augmented Generation
System

Scott Barnett, Stefanus Kurniawan, Srikanth Thudumu, Zach Brannelly, Mohamed
Abdelrazek  scott.barnett, stefanus.kurniawan, srikanth.thudumu,
zach.brannelly<EMAIL_ADDRESS>Applied Artificial
Intelligence InstituteDeakin UniversityGeelongAustralia

###### Abstract.

Software engineers are increasingly adding semantic search capabilities to
applications using a strategy known as Retrieval Augmented Generation (RAG). A
RAG system involves finding documents that semantically match a query and then
passing the documents to a large language model (LLM) such as ChatGPT to
extract the right answer using an LLM. RAG systems aim to: a) reduce the
problem of hallucinated responses from LLMs, b) link sources/references to
generated responses, and c) remove the need for annotating documents with
meta-data. However, RAG systems suffer from limitations inherent to
information retrieval systems and from reliance on LLMs. In this paper, we
present an experience report on the failure points of RAG systems from three
case studies from separate domains: research, education, and biomedical. We
share the lessons learned and present 7 failure points to consider when
designing a RAG system. The two key takeaways arising from our work are: 1)
validation of a RAG system is only feasible during operation, and 2) the
robustness of a RAG system evolves rather than designed in at the start. We
conclude with a list of potential research directions on RAG systems for the
software engineering community.

Retrieval Augmented Generation, RAG, SE4AI, Case Study

††conference: 3rd International Conference on AI Engineering — Software
Engineering for AI; April 2024; Lisbon, Portugal††ccs: Software and its
engineering Empirical software validation

##  1\. Introduction

The new advancements of Large Language Models (LLMs), including ChatGPT, have
given software engineers new capabilities to build new HCI solutions, complete
complex tasks, summarise documents, answer questions in a given artefact(s),
and generate new content. However, LLMs suffer from limitations when it comes
to up-to-date knowledge or domain-specific knowledge currently captured in
company’s repositories. Two options to address this problem are: a) Finetuning
LLMs (continue training an LLM using domain specific artifacts) which requires
managing or serving a fine-tuned LLM; or b) use Retrieval-Augmented Generation
(RAG) Systems that rely on LLMs for generation of answers using existing
(extensible) knowledge artifacts. Both options have pros and cons related to
privacy/security of data, scalability, cost, skills required, etc. In this
paper, we focus on the RAG option.

Retrieval-Augmented Generation (RAG) systems offer a compelling solution to
this challenge. By integrating retrieval mechanisms with the generative
capabilities of LLMs, RAG systems can synthesise contextually relevant,
accurate, and up-to-date information. A Retrieval-Augmented Generation (RAG)
system combines information retrieval capabilities, and generative prowess of
LLMs. The retrieval component focuses on retrieving relevant information for a
user query from a data store. The generation component focuses on using the
retrieved information as a context to generate an answer for the user query.
RAG systems are an important use case as all unstructured information can now
be indexed and available to query reducing development time no knowledge graph
creation and limited data curation and cleaning.

Software engineers building RAG systems are expected to preprocess domain
knowledge captured as artifacts in different formats, store processed
information in appropriate data store (vector database), implement or
integrate the right query-artifact matching strategy, rank matched artifacts,
and call the LLMs API passing in user queries and context documents. New
advances for building RAG systems are constantly emerging (Lewis et al., 2020;
Hofstätter et al., 2023) but how they relate and perform for a specific
application context has to be discovered.

In this work we present the lessons learned and 7 failure points arising from
3 case studies. The purpose of this paper is to provide 1) a reference to
practitioners and 2) to present a research road map for RAG systems. To the
best of our knowledge, we present the first empirical insight into the
challenges with creating robust RAG systems. As advances in LLMs continue to
take place, the software engineering community has a responsibility to provide
knowledge on how to realise robust systems with LLMs. This work is an
important step for robustness in building RAG systems.

Research questions for this work include:

  * •

What are the failure points that occur when engineering a RAG system? (section
5) We present an empirical experiment using the BioASQ data set to report on
potential failure points. The experiment involved 15,000 documents and 1000
question and answer pairs. We indexed all documents then ran the queries and
stored the generated responses using GPT-4. All question and answer pairs were
then validated with OpenAI evals 111https://github.com/openai/evals. Manual
inspection (all discrepancies, all flagged as incorrect, and a sample of
correct labels) was analysed to identify the patterns.

  * •

What are the key considerations when engineering a RAG system? (section 6) We
present the lessons learned from three case studies involving the
implementation of a RAG system. This presents the challenges faced and
insights gained.

Contributions arising from this work include:

  * •

A catalogue of failure points (FP) that occur in RAG systems.

  * •

An experience report from 3 case studies of implementing a RAG system. Two
currently running at Deakin University.

  * •

A research direction for RAG systems based on the lessons learned from the 3
case studies.

##  2\. Related Work

Retrieval augmented generation encompasses using documents to augment large
language models through pre-training and at inference time (Izacard and Grave,
2020; Guu et al., 2020; Lewis et al., 2020). Due to the compute cost, data
preparation time and required resources using RAG without training or fine-
tuning is an attractive proposition. However, challenges arise when using
large language models for information extraction such as performance with long
text (Hofstätter et al., 2023).

A recent survey (Zhu et al., 2023) showed that large language models are used
across the RAG pipeline including retriever, data generation, rewriter, and
reader. Our work complements this survey by taking a software engineering
perspective to shine a light on what issues engineers will face and what
software engineering research is necessary to realise solutions with the
current state-of-the-art RAG systems.

Emerging work has looked at benchmarking RAG systems (Chen et al., 2023b) but
not at the failures occurring during implementation. Software engineering
research has investigated the use of RAG systems for code-related tasks
(Nashid et al., 2023). However, the application of RAG systems is broader than
software engineering tasks. This paper complements existing work by presenting
challenges faced during the implementation of a RAG system with a focus on
practitioners.

Errors and failures that arise from RAG systems overlap with other information
retrieval systems including 1) no metrics for query rewriting, 2) document re-
ranking, and 3) effective content summarisation (Zhu et al., 2023). Our
results confirm this The unique aspects are related to the semantic and
generative nature of the use of large language models including evaluating
factual accuracy (OpenAI, 2023).

##  3\. Retrieval Augmented Generation

Figure 1. Indexing and Query processes required for creating a Retrieval
Augmented Generation (RAG) system. The indexing process is typically done at
development time and queries at runtime. Failure points identified in this
study are shown in red boxes. All required stages are underlined. Figure
expanded from (Zhu et al., 2023).

With the explosion in popularity of large language model services such as
ChatGPT222https://chat.openai.com/, Claude333https://claude.ai/, and Bard
444https://bard.google.com/, people have explored their use as a question and
answering systems. While the performance is impressive (OpenAI, 2023) there
are two fundamental challenges: 1) hallucinations - where the LLM produces a
response that looks right but is incorrect, and 2) unbounded - no way to
direct or update the content of the output (other than through prompt
engineering). A RAG system is an information retrieval approach designed to
overcome the limitations of using a LLM directly.

RAG works by taking a natural language query is converted into an embedding
which is used to semantically search a set of documents. Retrieved documents
are then passed to a large language model to generate an answer. An overview
of a RAG system is shown in Figure 1 as two separate processes, Index and
Query. See this survey for more details (Zhu et al., 2023)

###  3.1. Index Process

In a RAG system, the retrieval system works using embeddings that provide a
compressed semantic representation of the document. An embedding is expressed
as a vector of numbers. During the Index process each document is split into
smaller chunks that are converted into an embedding using an embedding model.
The original chunk and the embedding are then indexed in a database. Software
engineers face design decisions around how best to chunk the document and how
large a chunk should be. If chunks are too small certain questions cannot be
answered, if the chunks are too long then the answers include generated noise.

Different types of documents require different chunking and processing stages.
For example, video content requires a transcription pipeline to extract the
audio and convert to text prior to encoding (see subsection 4.2. The choice of
which embedding to use also matters as changing the embedding strategy
requires re-indexing all chunks. An embedding should be chosen based on the
ability to semantically retrieve correct responses. This process depends on
the size of the chunks, the types of questions expected, the structure of the
content and the application domain.

###  3.2. Query Process

The Query process takes place at run time. A question expressed as natural
language is first converted into a general query. To generalise the query a
large language model is used which enables additional context such as previous
chat history to be included in the new query. An embedding is then calculated
from the new query to use for locating relevant documents from the database.
Top-k similar documents are retrieved using a similarity method such as cosine
similarity (vector databases have techniques such as inverted indexes to speed
up retrieval time). The intuition is that chunks that are semantically close
to the query are likely to contain the answer.

Retrieved documents are then re-ranked to maximise the likelihood that the
chunk with the answer is located near the top. The next stage is the
Consolidator which is responsible for processing the chunks. This stage is
needed to overcome the limitations of large language models 1) token limit and
2) rate limit. Services such as OpenAI have hard limits on the amount of text
to include in a prompt. This restricts the number of chunks to include in a
prompt to extract out an answer and a reduction strategy is needed to chain
prompts to obtain an answer. These online services also restrict the number of
tokens to use within a time frame restricting the latency of a system.
Software engineers need to consider these tradeoffs when designing a RAG
system.

The final stage of a RAG pipeline is when the answer is extracted from the
generated text. Readers are responsible for filtering the noise from the
prompt, adhering to formatting instructions (i.e. answer the question as a
list of options), and producing the output to return for the query.
Implementation of a RAG system requires customising multiple prompts to
process questions and answers. This process ensures that questions relevant
for the domain are returned. The use of large language models to answer real
time questions from documents opens up new application domains where question
and answering is new capability. Thus, RAG systems are difficult to test as no
data exists and needs to be experimentally discovered through either a)
synthetic data generation, or b) piloting the system with minimal testing.

##  4\. Case Studies

This study conducted three case studies to discover the challenges that arise
when implementing RAG systems. A summary of each of the case studies is shown
in Table 1. All scripts, data, and examples of each of the failure points for
the BioASQ case study are available online
555https://figshare.com/s/fbf7805b5f20d7f7e356. The other two case studies
have been excluded due to confidentiality concerns.

Case Study | Domain |  Doc Types | Dataset Size |  RAG Stages |  Sample Questions
---|---|---|---|---|---
Cognitive Reviewer* | Research |  PDFs | (Any size) |  Chunker, Rewriter, Retriever, Reader |  What are the key points covered in this paper?
AI Tutor* | Education |  Videos, HTML, PDF | 38 |  Chunker, Rewriter, Retriever, Reader |  What were the topics covered in week 6?
BioASQ | Biomedical |  Scientific PDFs | 4017 |  Chunker, Retriever, Reader |  Define pseudotumor cerebri. How is it treated?
Table 1. A summary of the RAG case studies presented in this paper. Case
studies marked with a * are running systems currently in use.

###  4.1. Cognitive Reviewer

Cognitive Reviewer is a RAG system designed to support researchers in
analysing scientific documents. Researchers specify a research question or
objective and then upload a collection of related research papers. All of the
documents are then ranked in accordance with the stated objective for the
researcher to manually review. The researcher can also ask questions directly
against all of the documents. Cognitive Reviewer is currently used by PhD
students from Deakin University to support their literature reviews. The
Cognitive Reviewer does the Index process at run time and relies on a robust
data processing pipeline to handle uploaded documents i.e. no quality control
possible at development time. This system also uses a ranking algorithm to
sort the uploaded documents.

###  4.2. AI Tutor

The AI Tutor is a RAG system where students ask questions about the unit and
answers are sourced from the learning content. Students are able to verify the
answers by accessing a sources list from where the answer came from. The AI
Tutor works by integrating into Deakin’s learning management system, indexing
all of the content including PDF documents, videos, and text documents. As
part of the Index process, videos are transcribed using the deep learning
model Whisper (Radford et al., 2023) before being chunked. The AI Tutor was
developed between August 2023 to November 2023 for a pilot in a unit with 200
students that commenced the 30th of October 2023. Our intention is to present
the lessons learned during implementation and present a followup findings at
the conclusion of the pilot. This RAG pipeline includes a rewriter to
generalise queries. We implemented a chat interface where previous dialogue
between the user and the AI Tutor was used as part of the context for each
question. The rewriter considers this context and rewrites the query to
resolve ambiguous requests such as ‘Explain this concept further.’

###  4.3. Biomedical Question and Answer

The previous case studies focused on documents with smaller content sizes. To
explore the issues at a larger scale we created a RAG system using the BioASQ
(Krithara et al., 2023) dataset comprised of questions, links to document, and
answers. The answers to questions were one of yes/no, text summarisation,
factoid, or list. This dataset was prepared by biomedical experts and contains
domain specific question and answer pairs. We downloaded 4017 open access
documents from the BioASQ dataset and had a total of 1000 questions. All
documents were indexed and the questions asked against the RAG system. The
generated questions were then evaluated using the OpenEvals technique
implemented by OpenAI666https://github.com/openai/evals. From the generated
questions we manually inspected 40 issues and all issues that the OpenEvals
flagged as inaccurate. We found that the automated evaluation was more
pessimistic than a human rater for this domain. However, one threat to
validity with this finding is that BioASQ is a domain specific dataset and the
reviewers were not experts i.e. the large language model may know more than a
non-expert.

##  5\. Failure Points of RAG Systems

From the case studies we identified a set of failure points presented below.
The following section addresses the research question What are the failure
points that occur when engineering a RAG system?

  1. FP1

Missing Content The first fail case is when asking a question that cannot be
answered from the available documents. In the happy case the RAG system will
respond with something like “Sorry, I don’t know”. However, for questions that
are related to the content but don’t have answers the system could be fooled
into giving a response.

  2. FP2

Missed the Top Ranked Documents The answer to the question is in the document
but did not rank highly enough to be returned to the user. In theory, all
documents are ranked and used in the next steps. However, in practice the top
K documents are returned where K is a value selected based on performance.

  3. FP3

Not in Context - Consolidation strategy Limitations Documents with the answer
were retrieved from the database but did not make it into the context for
generating an answer. This occurs when many documents are returned from the
database and a consolidation process takes place to retrieve the answer.

  4. FP4

Not Extracted Here the answer is present in the context, but the large
language model failed to extract out the correct answer. Typically, this
occurs when there is too much noise or contradicting information in the
context.

  5. FP5

Wrong Format The question involved extracting information in a certain format
such as a table or list and the large language model ignored the instruction.

  6. FP6

Incorrect Specificity The answer is returned in the response but is not
specific enough or is too specific to address the user’s need. This occurs
when the RAG system designers have a desired outcome for a given question such
as teachers for students. In this case, specific educational content should be
provided with answers not just the answer. Incorrect specificity also occurs
when users are not sure how to ask a question and are too general.

  7. FP7

Incomplete Incomplete answers are not incorrect but miss some of the
information even though that information was in the context and available for
extraction. An example question such as “What are the key points covered in
documents A, B and C?” A better approach is to ask these questions separately.

##  6\. Lessons and Future Research Directions

FP |  Lesson |  Description |  Case Studies
---|---|---|---
FP4 |  Larger context get better results (Context refers to a particular setting or situation in which the content occurs) |  A larger context enabled more accurate responses (8K vs 4K). Contrary to prior work with GPT-3.5 (Liu et al., 2023b) |  AI Tutor
FP1 |  Semantic caching drives cost and latency down |  RAG systems struggle with concurrent users due to rate limits and the cost of LLMs. Prepopulate the semantic cache with frequently asked questions (Bang, 2023). |  AI Tutor
FP5-7 |  Jailbreaks bypass the RAG system and hit the safety training. |  Research suggests fine-tuning LLMs reverses safety training (Lermen et al., 2023), test all fine-tuned LLMs for RAG system. |  AI Tutor
FP2, FP4 |  Adding meta-data improves retrieval. |  Adding the file name and chunk number into the retrieved context helped the reader extract the required information. Useful for chat dialogue. |  AI Tutor
FP2, FP4-7 |  Open source embedding models perform better for small text. |  Opensource sentence embedding models performed as well as closed source alternatives on small text. |  BioASQ, AI Tutor
FP2-7 |  RAG systems require continuous calibration. |  RAG systems receive unknown input at runtime requiring constant monitoring. |  AI Tutor, BioASQ
FP1, FP2 |  Implement a RAG pipeline for configuration. |  A RAG system requires calibrating chunk size, embedding strategy, chunking strategy, retrieval strategy, consolidation strategy, context size, and prompts. |  Cognitive Reviewer, AI Tutor, BioASQ
FP2, FP4 |  RAG pipelines created by assembling bespoke solutions are suboptima. |  End-to-end training enhances domain adaptation in RAG systems (Siriwardhana et al., 2023). |  BioASQ, AI Tutor
FP2-7 |  Testing performance characteristics are only possible at runtime. |  Offline evaluation techniques such as G-Evals (Liu et al., 2023a) look promising but are premised on having access to labelled question and answer pairs. |  Cognitive Reviewer, AI Tutor
Table 2. The lessons learned from the three case studies with key takeaways
for future RAG implementations

The lessons learned from the three case studies are shown in Table 2. We
present our findings for the research question: What are the key
considerations when engineering a RAG system? Based on our takeaways we
identified multiple potential research areas linked to RAG as follows:

###  6.1. Chunking and Embeddings

Chunking documents sounds trivial. However, the quality of chunking affects
the retrieval process in many ways and in particular on the embeddings of the
chunk then affects the similarity and matching of chunks to user queries.
There are two ways of chunking: heuristics based (using punctuation, end of
paragraph, etc.), and semantic chunking (using the semantics in the text to
inform start-end of a chunk). Further research should explore the tradeoffs
between these methods and their effects on critical downstream processes like
embedding and similarity matching. A systematic evaluation framework comparing
chunking techniques on metrics like query relevance and retrieval accuracy
would benefit the field.

Embeddings represent another active research area, including generating
embeddings for multimedia and multimodal chunks such as tables, figures,
formulas, etc. Chunk embeddings are typically created once during system
development or when a new document is indexed. Query preprocessing
significantly impacts a RAG system’s performance, particularly handling
negative or ambiguous queries. Further research is needed on architectural
patterns and approaches (Cummaudo et al., 2020a) to address the inherent
limitations with embeddings (quality of a match is domain specific).

###  6.2. RAG vs Finetuning

LLMs are great world models due to the amount of training data, and finetuning
tasks applied on the model before it’s released. However, these models are
general-purpose models (may not know the very specifics of your domain) and
also not up to date (there is a cutoff date on their knowledge). Fine-tuning
and RAG offer two potential customisation pathways, each with distinct
tradeoffs. Finetuning requires curating internal datasets to adapt and train
the LLM on. However, all your data are baked into the model and you need to
sort out the security/privacy (who can access what). Furthermore, as the
foundation model itself evolves or you get new data to add to the model, you
will need to run finetuning again. On the other side, RAG systems seem to
offer a pragmatic solution allowing you to chunk your data as needed and only
use relevant chunks into the context to ask the LLM to generate an answer from
the included context. This facilitates continuously updating the knowledge
with new documents and also gives the control over what chunks the user is
able to access. However, optimal strategies for chunk embedding, retrieval,
and contextual fusion remain active research. Further work should
systematically compare finetuning and RAG paradigms across factors including
accuracy, latency, operating costs, and robustness.

###  6.3. Testing and Monitoring RAG systems

Software engineering best practices are still emerging for RAG systems.
Software testing and test case generation are one of the areas for refinement.
RAG systems require questions and answers that are application specific often
unavailable when indexing unstructured documents. Emerging work has considered
using LLMs for generating questions from multiple documents (Chen et al.,
2023a). How to generate realistic domain relevant questions and answers
remains an open problem.

Once suitable test data is available quality metrics are also required to
assist engineers in making quality tradeoffs. Using large language models is
expensive, introduces latency concerns, and has performance characteristics
that all change with each new release. This characteristic has previously been
studied for machine learning systems (Cummaudo et al., 2020b, a) but the
required adaptations (if any) have yet to be applied to LLM based systems such
as RAGs. Another idea is to incorporate ideas from self-adaptive systems to
support monitoring and adapting RAG systems, preliminary work has started for
other machine learning applications (Casimiro et al., 2022).

##  7\. Conclusion

RAG systems are a new information retrieval that leverages LLMs. Software
engineers increasingly interact with RAG systems a) through implementing
semantic search, or b) through new code-dependent tasks. This paper presented
the lessons learned from 3 case studies including an empirical investigation
involving 15,000 documents and 1000 questions. Our findings provide a guide to
practitioners by presenting the challenges faced when implementing RAG
systems. We also included future research directions for RAG systems related
to 1) chunking and embeddings, 2) RAG vs Finetuning, and 3) Testing and
Monitoring. Large language models are going to continue to obtain new
capabilities of interest to engineers and researchers. This paper presents the
first investigation into RAG systems from a software engineering perspective.

###### Acknowledgements.

To Amanda Edgar, Rajesh Vasa, Kon Mouzakis, Matteo Vergani, Trish McCluskey,
Kathryn Perus, Tara Draper, Joan Sutherland and Ruary Ross for their support
and involvement in making the AI Tutor project possible.

## References

  * (1)
  * Bang (2023) Fu Bang. 2023.  GPTCache: An Open-Source Semantic Cache for LLM Applications Enabling Faster Answers and Cost Savings. In _3rd Workshop for Natural Language Processing Open Source Software_.
  * Casimiro et al. (2022) Maria Casimiro, Paolo Romano, David Garlan, Gabriel Moreno, Eunsuk Kang, and Mark Klein. 2022.  _Self-adaptive Machine Learning Systems: Research Challenges and Opportunities_.  133–155.  https://doi.org/10.1007/978-3-031-15116-3_7
  * Chen et al. (2023b) Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2023b.  Benchmarking Large Language Models in Retrieval-Augmented Generation.  _arXiv preprint arXiv:2309.01431_ (2023).
  * Chen et al. (2023a) Mingda Chen, Xilun Chen, and Wen-tau Yih. 2023a.  Efficient Open Domain Multi-Hop Question Answering with Few-Shot Data Synthesis.  _arXiv preprint arXiv:2305.13691_ (2023).
  * Cummaudo et al. (2020a) Alex Cummaudo, Scott Barnett, Rajesh Vasa, and John Grundy. 2020a.  Threshy: Supporting safe usage of intelligent web services. In _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 1645–1649.
  * Cummaudo et al. (2020b) Alex Cummaudo, Scott Barnett, Rajesh Vasa, John Grundy, and Mohamed Abdelrazek. 2020b.  Beware the evolving ‘intelligent’web service! An integration architecture tactic to guard AI-first components. In _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 269–280.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020.  Retrieval augmented language model pre-training. In _International conference on machine learning_. PMLR, 3929–3938.
  * Hofstätter et al. (2023) Sebastian Hofstätter, Jiecao Chen, Karthik Raman, and Hamed Zamani. 2023.  Fid-light: Efficient and effective retrieval-augmented text generation. In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1437–1447.
  * Izacard and Grave (2020) Gautier Izacard and Edouard Grave. 2020.  Leveraging passage retrieval with generative models for open domain question answering.  _arXiv preprint arXiv:2007.01282_ (2020).
  * Krithara et al. (2023) Anastasia Krithara, Anastasios Nentidis, Konstantinos Bougiatiotis, and Georgios Paliouras. 2023.  BioASQ-QA: A manually curated corpus for biomedical question answering.  _Scientific Data_ 10 (2023), 170.  Citation Key: 422.
  * Lermen et al. (2023) Simon Lermen, Charlie Rogers-Smith, and Jeffrey Ladish. 2023.  LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B.  arXiv:2310.20624 [cs.LG]
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020\.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ 33 (2020), 9459–9474.
  * Liu et al. (2023b) Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023b.  Lost in the middle: How language models use long contexts.  _arXiv preprint arXiv:2307.03172_ (2023).
  * Liu et al. (2023a) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023a.  G-eval: Nlg evaluation using gpt-4 with better human alignment, may 2023.  _arXiv preprint arXiv:2303.16634_ (2023).
  * Nashid et al. (2023) Noor Nashid, Mifta Sintaha, and Ali Mesbah. 2023.  Retrieval-based prompt selection for code-related few-shot learning. In _Proceedings of the 45th International Conference on Software Engineering (ICSE’23)_.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  https://doi.org/10.48550/ARXIV.2303.08774
  * Radford et al. (2023) Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023.  Robust speech recognition via large-scale weak supervision. In _International Conference on Machine Learning_. PMLR, 28492–28518.
  * Siriwardhana et al. (2023) Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen, Tharindu Kaluarachchi, Rajib Rana, and Suranga Nanayakkara. 2023.  Improving the domain adaptation of retrieval augmented generation (RAG) models for open domain question answering.  _Transactions of the Association for Computational Linguistics_ 11 (2023), 1–17.
  * Zhu et al. (2023) Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji-Rong Wen. 2023.  Large language models for information retrieval: A survey.  _arXiv preprint arXiv:2308.07107_ (2023).