# Unifying the Validation of Ambient Solar Wind Models

Martin A. Reiss Karin Muglach Richard Mullinix Maria M. Kuznetsova Chiu
Wiegand Manuela Temmer Charles N. Arge Sergio Dasso Shing F. Fung Jose Juan
Gonzalez Aviles Siegfried Gonzi Lan Jian Peter MacNeice Christian Moestl
Mathew Owens Barbara Perri Rui Pinto Lutz Rastaetter Pete Riley Evangelia
Samara and ISWAT H1-01 Team Members Space Research Institute, Austrian Academy
of Sciences, Graz, Austria  Heliophysics Science Division, NASA Goddard Space
Flight Center, Greenbelt, MD 20771, USA  Catholic University of America,
Washington, DC 20064, USA  Community Coordinated Modeling Center, Code 674,
NASA GSFC, Greenbelt, MD, USA  University of Graz, Institute of Physics, Graz,
Austria  Instituto de Astronomiia y Fiisica del Espacio, UBA-CONICET, Buenos
Aires, Argentina  Catedras Conacyt, Servicio de Clima Espacial Mexico,
Laboratorio Nacional de Clima Espacial, Instituto de Geofisca, Unidad
Michoacan, Universidad Nacional Autnoma de Mexico, 58190 Morelia, Michoacan,
Mexico  Met Office, Exeter, UK  Department of Meteorology, University of
Reading, Reading, UK RG6 6BB  Centre for Mathematical Plasma Astrophysics, KU
Leuven, Celestijnenlaan 200b-box 2400, B-3001 Leuven, Belgium  Universit ie de
Toulouse; UPS-OMP; IRAP; Toulouse, France  Predictive Science Inc., San Diego,
CA, USA

###### Abstract

Progress in space weather research and awareness needs community-wide
strategies and procedures to evaluate our modeling assets. Here we present the
activities of the Ambient Solar Wind Validation Team embedded in the COSPAR
ISWAT initiative. We aim to bridge the gap between model developers and end-
users to provide the community with an assessment of the state-of-the-art in
solar wind forecasting. To this end, we develop an open online platform for
validating solar wind models by comparing their solutions with in situ
spacecraft measurements. The online platform will allow the space weather
community to test the quality of state-of-the-art solar wind models with
unified metrics providing an unbiased assessment of progress over time. In
this study, we propose a metadata architecture and recommend community-wide
forecasting goals and validation metrics. We conclude with a status update of
the online platform and outline future perspectives.

††journal: Advances in Space Research

##  1 Introduction

The rate at which we develop and update space weather models has outpaced the
rate at which we build our data and validation infrastructure. Consequently,
questions such as ”How well does a model perform over a given time interval?”
or ”How much has this model improved over the past five years?” are, if at
all, difficult to answer. Progress in space weather research and awareness
therefore benefits from community-coordinated strategies and procedures for
validation.

Validation is the craft of assessing the quality of models by comparing their
solutions with observations. But the validation of space weather models faces
challenges. First, keeping up with the ever-growing number of models,
different versions thereof, and increasingly versatile user needs without
community-wide procedures is practically impossible. Next, the slow, iterative
process between model developers and end-users causes a bottleneck. Since new
space weather models are usually reported in the literature, models are often
outdated when a new paper gets published (MacNeice et al., 2018). Finally,
adapting to specific user needs is not possible in retrospect in validation
studies. End-users have to rely on metrics selected by the authors, which do
not always satisfy the increasingly versatile user needs. The Ambient Solar
Wind Validation Team embedded in the Committee on Space Research (COSPAR) -
International Space Weather Action Teams (ISWAT) initiative supports the
community in these challenges. While the team focuses on validating solar wind
models, our strategies can be adapted to adjacent disciplines.

Assessment of the state-of-the-art in solar wind forecasting is of pivotal
importance in space weather research and awareness. The reasons are three-fold
(see Cranmer et al., 2017). First, most of the time the solar wind determines
the prevailing conditions in our solar system, including bulk speed, magnetic
field strength and orientation (Luhmann et al., 2002). Second, fast and slow
solar wind flows interacting with each other are a recurrent driver of
moderate geomagnetic activity (Kilpua et al., 2017), particularly during solar
minimum (Verbanac et al., 2011). Third, the solar wind sets the conditions
through which the most extreme forms of space weather, interplanetary coronal
mass ejections, propagate (Gosling & Pizzo, 1999). Ample evidence shows that
solar wind flows can distort and deflect interplanetary coronal mass ejections
and thereby affect their severity (Riley et al., 1997; Odstrčil & Pizzo, 1999;
Case et al., 2008; Zhou & Feng, 2017).

The validation of solar wind models has received much attention; for instance,
Owens et al. (2005); MacNeice (2009b) and Reiss et al. (2016) developed
validation procedures that detect abrupt transitions from slow to fast wind.
Owens et al. (2017) and Henley & Pope (2017) discussed ensemble solar wind
forecasts using a ”cost-loss” analysis, and Owens (2018) proposed time window
approaches to complement traditional point-to-point comparison metrics.
Numerous studies have assessed the quality of ambient solar wind models by
comparing their solutions to in situ measurements (see, for example, Owens et
al., 2008; MacNeice, 2009b; Norquist & Meeks, 2010; Jian et al., 2011; Gressl
et al., 2014; Jian et al., 2015, 2016; Devos et al., 2014; Reiss et al., 2016,
2019; Hinterreiter et al., 2019; Li et al., 2020; Riley & Ben-Nun, 2021, among
others). Other observational tests focus on the Earth to Sun magnetic
connectivity (MacNeice et al., 2011), interplanetary scintillation (Kim et
al., 2014; Jackson et al., 2015; Gonzi et al., 2021), coronagraph images
(Jones et al., 2017; Lamy et al., 2019), and more. As an unbiased benchmark,
Owens et al. (2013) and Kohutova et al. (2016) proposed persistence models
that assume the solar wind condition repeat after each Carrington rotation.

Despite these and other important validation efforts, model development has
greatly outpaced the validation infrastructure development. To rectify this
issue, MacNeice (2018) recommended the development of automated protocols to
maintain up-to-date validation results. In line with this recommendation, we
formed the Ambient Solar Wind Validation Team with the following aims:

  1. 1.

Develop a comprehensive metadata architecture including metrics to enable
sustainable validation of the state-of-the-art and progress assessment over
time.

  2. 2.

Implement an open online platform in collaboration with NASA’s Community
Coordinated Modelling Center (CCMC) to validate solar wind models with
streamlined metrics.

  3. 3.

Quantitatively assesses the state-of-the-art in forecasting the solar wind
conditions at Earth and other planetary environments.

  4. 4.

Use our developed infrastructure to establish an ”Ambient Solar Wind
Scoreboard” to maintain up-to-date validation results.

This study presents our progress concerning Aim 1 and 2, while a follow-up
study will focus on Aim 3 and 4. Section 2 reviews the state-of-the-art in the
methodology of solar wind forecast models. Section 3 proposes the metadata
architecture used to register community models in the online platform. Section
4 recommends community-wide validation metrics, while Section 5 presents the
current status of the online platform. The discussion in Section 6 concludes
this study and outlines our future perspectives.

##  2 Ambient Solar Wind Forecasting in 2022

In what follows, we focus on forecast models with possible application in an
operational setting. Our discussion, therefore, excludes solar wind models
designed to study physical processes in experimental conditions. We
furthermore limit our discussion to ambient solar wind models that exclude the
dynamics of interplanetary coronal mass ejections.

State-of-the-art solar wind frameworks couple models of the global corona and
heliosphere. As a standard configuration, the coronal domain spans the
distance of 1 solar radius ($\textup{R}_{\odot}$) to a distance between 20 and
30 $\textup{R}_{\odot}$ (depending on the model), and the heliospheric domain
extends between the outer boundary of the coronal domain to 1 AU (or further
when needed). Figure 1 shows the flow of models starting from the inner
boundary of the coronal domain into the heliospheric domain. Synoptic
photospheric magnetograms determine the inner boundary in the coronal domain.
Magnetic flux transport models such as the Evolving Surface Flux Transport
Model (ESFTM; Schrijver & De Rosa, 2003), Air Force Data Assimilative
Photospheric Flux Transport (ADAPT; Arge et al., 2010; Hickmann et al., 2015),
and the SURface Flux Transport (SURF; Upton & Hathaway, 2014) are expected to
improve these photospheric magnetic field measurements. ADAPT, for example,
creates an ensemble of possible magnetograms to account for differential
rotation, meridional flows, and random flux emergence.

In the coronal domain, numerical models rely on extrapolations computed from
photospheric magnetograms. One of the workhorse models in the community to
reconstruct the global coronal magnetic field is the Potential Field Source
Surface model (PFSS; Altschuler & Newkirk, 1969). The PFSS model assumes that
regions of the photosphere are current-free, which means that coronal field
solutions can be expressed as the gradient of a scalar potential. Since
potential fields give closed field lines, an outer boundary condition is
added. This outer boundary is defined by a spherically-symmetric ”source
surface” where the field is assumed to be only radial (inward or outward-
directed). The source surface radius is set to a reference height of 2.5 solar
radii to best match observations (Hoeksema et al., 1982). Recent studies argue
that the source surface radius can vary between 1.5 and 3 solar radii
depending on the solar activity cycle (Réville et al., 2015; Asvestari et al.,
2019; Boe et al., 2020). The Schatten Current Sheet model (SCS; Schatten,
1971) combined with the PFSS model forms a more uniform radial field strength
solution. In this way, the SCS model accounts for Ulysses observations that
revealed invariance of the radial magnetic field component with increasing
latitude out of the ecliptic (Wang & Sheeley, 1995).

Fig. 1: Examples of state-of-the-art models for forecasting the solar wind in
Earth’s space weather environment according to domains.

Plasma drags and distorts magnetic field lines and thereby distorts the
coronal magnetic field from the assumed current-free configuration. Coronal
models should thus account for the complex dynamics by including
magnetohydrodynamical (MHD) effects and solving a set of nonlinear partial
differential equations. For example, the Magnetohydrodynamics Algorithm
outside a Sphere model (MAS; Linker et al., 1999; Mikić et al., 1999), the
Space Weather Modeling Framework (SWMF; Tóth et al., 2005), and WindPredict-AW
(Réville et al., 2020) are such three-dimensional MHD codes. PFSS model
solutions often determine their initial conditions. The MHD equations are then
integrated until the plasma and magnetic fields settle into equilibrium. In
the final solutions, the solar wind plasma is constrained in closed magnetic
field lines and accelerated to supersonic speeds along open field lines. An
alternative approach situated between global MHD models and more specialized
one-dimensional models is the MULTI-VP model (Pinto & Rouillard, 2017). The
underlying idea is to compute numerous one-dimensional model solutions that
can sample sub-domains of interest and the whole solar atmosphere.

The model interface between the coronal and heliospheric domain is often
determined by the topology of the coronal field extrapolation. Because the
dynamic pressure term in the momentum equation is governed by the bulk speed
($\propto\rho v^{2}$), heliospheric solutions are sensitive to the
empirically-derived solar wind speed at the model interface (see Riley et al.,
2015). Models for specifying the speed at the interface are the Wang-Sheeley
(WS; Wang & Sheeley, 1990), Distance from the Coronal Hole Boundary (DCHB;
Riley et al., 2001), and the Wang-Sheeley-Arge (WSA; Arge et al., 2003) model,
updated recently with an adaptive approach (Adaptive-WSA; Reiss et al., 2019,
2020). While the traditional WS model relies on an inverse relationship
between the flux tube expansion rate of open field lines and solar wind speed
measured at Earth, the DCHB model relies on the relation between the great
circle angular distance from the nearest coronal hole boundary at the solar
surface and the solar wind speed. The WSA model combines both aspects of the
WS and DCHB model, and the Adaptive-WSA model continually adapts the WSA model
coefficients to best match observations.

In the heliospheric domain, different approaches are used to evolve the solar
wind solutions from the model interface into space. These approaches include
kinematic mapping, one-dimensional upwind propagation, and global heliospheric
MHD modeling (Riley & Lionello, 2011). In kinematic mapping, the solar wind
flow is treated as a chain of plasma parcels. Each parcel is accelerated or
decelerated depending on the adjacent wind speed (Arge & Pizzo, 2000). In
contrast, heliospheric MHD models provide a complete picture of the spatial
and temporal evolution of the solar wind in the inner heliosphere. Examples
are HelioMAS (Linker et al., 1999; Mikić et al., 1999), Enlil (Odstrcil,
2003), SWMF (Tóth et al., 2005), SUSANOO-SW (Shiota et al., 2014), LFM-helio
(Merkin et al., 2016), and the European Heliospheric Forecasting Information
Asset (EUHFORIA; Pomoell & Poedts, 2018).

Upwind propagation tools bridge the gap between the kinematic mapping and
global MHD modeling. By neglecting the pressure gradient and the gravitation
term in the fluid momentum equation, upwind tools can evolve the solar wind
flows in a self-consistent way. Examples are the Heliospheric Upwind
Extrapolation (HUX; Riley & Lionello, 2011; Riley & Issan, 2021; Issan &
Riley, 2022) and the Tunable HUX (THUX; Reiss et al., 2020) for mapping the
solar wind from the Sun to any position in the inner heliosphere, and the
time-dependent HUXt (Owens et al., 2020) model for evolving interplanetary
coronal mass ejections in the ambient wind. Studying more than 40 years of
data in retrospect shows that HUXt bulk speed solutions are in agreement (to
within 6 percent) with MHD codes (see Owens et al., 2020).

In addition, researchers have developed useful predictive tools beyond
numerical frameworks coupling the global corona and inner heliosphere.
Predictive tools are often designed to forecast the solar wind at Earth while
not computing the large-scale solar wind pattern. Examples are based on data
assimilation (Lang et al., 2017; Lang & Owens, 2019), empirical relationships
(Robbins et al., 2006; Vršnak et al., 2007; Reiss et al., 2016), machine
learning algorithms (Yang et al., 2018; Chandorkar et al., 2020; Bailey et
al., 2021), pattern matching (Bussy-Virat & Ridley, 2014; Riley et al., 2017;
Owens et al., 2017), and persistence (Owens et al., 2013; Kohutova et al.,
2016; Temmer et al., 2018).

Present operational forecast services at the Met Office (Exeter, UK) and the
National Oceanic and Atmospheric Administration (NOAA, Boulder, USA) rely on
the coupled WSA-Enlil model framework to forecast the solar wind conditions in
the inner heliosphere.

Fig. 2: Overview of metadata components for registering an ambient solar wind
model in the database. This is an example of a standard model framework
consisting of a coronal and a heliospheric model. We note that the model chain
which consists of both the coronal and heliospheric domains, is surrounded by
the orange boundaries. A more detailed description of the individual
components is provided in the boxes below.

##  3 Metadata Architecture

A comprehensive metadata architecture is needed to support a sustainable
validation of solar wind models. Here we propose seven types of metadata, all
of which are required to register new community models in the online platform.
These metadata components include information on the observational input data,
data preprocessing, model description, model setting, model output, model
chain, and model solution. We will store the metadata components in an
associated metadata template file according to the Space Physics Archive
Search and Extract (SPASE) standards. More details on SPASE metadata can be
found in Fung et al. (2022). As the backbone of the online platform, the
Comprehensive Assessment of Models and Events Using Library Tools (CAMEL;
Rastätter et al., 2019) framework hosted by NASA’s CCMC will make the metadata
accessible to the community online.

Figure 2 shows the metadata architecture. The input observational data and the
related data preprocessing steps are needed to document the input to the model
run. Each run can consist of a chain of models used to compute the individual
model solutions. The model chain template includes a description of the
coronal and heliospheric model domain. As a general guideline, we distinguish
between models of the corona and those of the inner heliosphere. Each model in
these domains needs a detailed description of the model, settings, and
outputs. It is also possible to register more than one model per domain.

Executing a model chain with different model parameters results in a new model
run. The model developers will specify a model version label. The information
on how the models are linked to each other is explained in the model chain
metadata file. Finally, the model solution metadata describes the solution of
the model chain. While also part of the metadata, the metrics provided via
CCMC’s CAMEL will be discussed in Section 4. In the following, we discuss all
the components in more detail.

###  3.1 Observational Metadata

The observational input metadata supports the reproducibility of the model
solutions. Documentation of the observational input can also reduce the
storage requirements for some models. This type of metadata describes the
observations and explains how these observations were made. Information such
as the date and time of the observation, the name of the observatory, and
details on the instrumentation are required. Figure 2 shows a list of
essential information conveyed as part of the observational input metadata.

###  3.2 Data Preprocessing Metadata

The preprocessing of input data affects the output of solar wind models.
Preprocessing concerns not only synoptic photospheric magnetograms in
numerical frameworks but also solar imagery and other inputs to predictive
tools. Ample evidence shows that deficiencies in magnetograms that serve as an
inner boundary of coronal magnetic modes significantly affects the model
solutions. Flux transport models such as ADAPT, therefore, assimilate
photospheric magnetic fields on the Earth-facing side of the Sun and apply
surface flux transport to approximate the global photospheric magnetic field
better. To document information on the preprocessing of the model input data,
we propose the metadata information template shown in Figure 2.

###  3.3 Model Description Metadata

The individual models are the building blocks of the numerical framework. This
type of metadata describes the individual models in the model chain used to
model the solar wind in different domains. This metadata component describes
each model and stores the model domain, simulation type, and version
information.

###  3.4 Model settings metadata

The model settings metadata specifies the model input and settings used for
computing the model solutions. In addition, it documents the model parameter
settings and the data structure settings, such as grid parameters. This
metadata component also includes information on expanding the model solutions
from a single deterministic run to an ensemble mode.

###  3.5 Model Output Metadata

This type of metadata focuses on the description of the model output. It
describes the solutions produced and explains the underlying data format and
the computed physical quantities. Additionally, this metadata template file
collects information about the post-processing steps after the model run that
are needed to interpret the solutions.

###  3.6 Model Chain Metadata

This type of metadata describes how the individual models used to simulate the
solar wind conditions from the Sun to the near-Earth space environment are
connected. It focuses on how the models are linked to forecasting the solar
wind at a specified location in the inner heliosphere. Furthermore, it
provides details on the boundary conditions used to couple the models.

###  3.7 Model Solution Metadata

The model solution metadata describes the final output of the model chain. It
contains a list of the physical properties computed by the models and points
to space missions that can be used to validate the models.

##  4 Forecasting Goals and Metrics

After collecting feedback from the community during workshops and conferences,
we recommend testing the quality of community models in the online platform
based on two forecasting goals:

  1. 1.

The ability of the solar wind model to forecast the temporal evolution of the
solar wind speed, as well as abrupt changes from slow to fast solar wind.

  2. 2.

The ability of the solar wind model to forecast the magnetic polarity and
magnetic sector boundary crossings.

To assess the agreement between model solutions and measurements, we propose
using continuous and binary variables. The first can take on any real values,
while the second is restricted to categorical values such as event/non-event.
Time series of solar wind properties such as bulk speed and magnetic polarity
can be interpreted in both ways as shown in Figure 3. First, we discuss
commonly applied point-to-point comparison metrics (blue arrows). Second, we
investigate the quality of model solutions in terms of binary metrics, where
each time step in the predicted and observed time series is labeled as an
event/non-event based on the selected threshold value (bottom panel). Third,
we study procedures to detect abrupt transitions from slow to fast solar wind
(orange arrows) and quantify the ability in terms of event-based metrics.

All the validation metrics discussed here will be integrated into CAMEL’s web
application and made available online. In this way, end-users and model
developers can use streamlined metrics and representations to compare
community models.

Fig. 3: a) Observed (black) and modelled (red) solar wind speed as a function
of time with an example of proposed validation metrics explaining the concept
of point-to-point measures (blue). b) binary-event measures in the pink box,
and an event-based validation (orange). The solid blue line represents the
climatological mean. Courtesy: Modified from Owens (2018)

###  4.1 Point-to-point Comparison Metrics

We validate the quality of solar wind models by comparing their forecasts to
measurements. We first focus on the underlying statistical distributions using
the mean, median, and standard deviation. These measures contain valuable
information on the model’s tendency to over or under-estimate the observed
physical properties. We moreover study the predictive skill in terms of
established point-to-point error measures such as the mean error, mean
absolute error, and the root mean square error. Table 1 shows the definitions
of commonly used error functions, where $(f_{k},o_{k})$ is the $k$-th element
of $n$ total forecast and observation pairs in the time series. Although
strictly speaking not an error function, we also compute the Pearson
correlation coefficient (PCC).

To complement these error measures, we use the skill score (SS) that compares
the predictive skill to a simple baseline. Table 1 shows the definition of the
skill score, where $\textup{MSE}_{\textup{pred}}$ is the mean square error of
the forecasted time series, and $\textup{MSE}_{\textup{ref}}$ is the MSE of a
reference baseline model. A common example for a baseline model in the
literature is the climatological mean defined as the mean value of the
observation (Owens, 2018). For an ideal forecast, the MSE results in zero, and
the SS consequently results in a value of 1. A forecast that equals the skill
of the climatological mean results in an SS value of 0, and a forecast that is
less skillful than the baseline model results in a negative SS value. Other
reference models such as a model of 27-day persistence (assuming that the
solar wind conditions will repeat after each Carrington rotation) can replace
the climatological mean.

Table 1: Overview of point-to-point comparison metrics. Metric | Short Name | Definition
---|---|---
Mean error | ME | $\frac{1}{n}\sum_{k=1}^{n}(f_{k}-o_{k})$
Mean square error | MSE | $\frac{1}{n}\sum_{k=1}^{n}(f_{k}-o_{k})^{2}$
Mean absolute error | MAE | $\frac{1}{n}\sum_{k=1}^{n}\left|f_{k}-o_{k}\right|$
Root mean square error | RMSE | $\sqrt{\frac{1}{n}\sum_{k=1}^{n}(f_{k}-o_{k})^{2}}$
Skill score | SS | $1-\frac{\rm{MSE}}{\rm{MSE}_{\rm{ref}}}$

###  4.2 Binary Metrics

The previous section quantified the magnitude of the error at every time step.
This section focuses on an alternative procedure that classifies each time
step as an event/non-event. Owens (2018) summarized the advantages of this
procedure as follows. First, traditional error functions do not differentiate
between times of slow and fast solar wind conditions. Forecasters, however,
are often only interested in times when the solar wind conditions exceed a
certain threshold level, while the temporal evolution of the slow solar wind
is secondary. Second, outliers in the forecasted time series can significantly
affect the computed error functions and correlation coefficients. For these
reasons, a reasonable approach is to classify each time step in the solar wind
model solutions and measurements as an event/non-event.

To define events and non-events in solar wind time series, we first need to
define an event threshold. By cross-checking events and non-events in the
predicted and observed time series, we can compute the number of hits (true
positives; TPs), false alarms (false positives; FPs), misses (false negatives;
FNs), and correct rejections (true negatives; TNs). TPs are correctly
forecasted events, while FNs are observed events that were not forecasted.
Similarly, FPs are forecasted events that were not observed, and TNs are
correctly forecasted non-events. We can compute a set of skill measures from
the total counts of the different scenarios, summarized in the so-called
contingency table. Table 2 shows some examples.

A noteworthy skill measure is the so-called true skill statistics (TSS). The
TSS is defined in the range $[-1,1]$. While a TSS of 0 indicates no skill, a
perfect model solution would score 1 (or -1 for an inverse forecast). The TSS
has the advantage that it is unbiases by the ratio between forecasted and
observed events and uses all entries in the contingency table (Bloomfield et
al., 2012).

###  4.3 Event-based Metrics

Relying on point-to-point comparison metrics can be misleading (see, for
example, Owens et al., 2005; MacNeice, 2009b, a). This is the case when the
temporal evolution of solar wind properties is generally well-predicted, but
the arrival times differ in forecast and observation. Owens et al. (2005);
MacNeice (2009a); Reiss et al. (2016) proposed a three-stage strategy to
quantify these arrival time errors in bulk speed forecasts. First, is the
definition and detection of abrupt transitions from slow to fast solar wind
speed in forecast and measurements. Second is the association of the events
detected in the forecasts and observations. Third, is the computation of skill
measures to compare the predictive abilities of the models investigated. Table
2 shows several validation measures.

Table 2: Overview of binary metrics defined by the entries of a contingency table. Metric | Short Name | Definition
---|---|---
True Positive Rate | TPR | $\frac{\rm{TP}}{\rm{TP}+\rm{FN}}$
False Positive Rate | FPR | $\frac{\rm{FP}}{\rm{FP}+\rm{TN}}$
Threat Score | TS | $\frac{\rm{TP}}{\rm{TP}+\rm{FP}+\rm{FN}}$
True Skill Statistics | TSS | $\rm{TPR}-\rm{FPR}$
Bias | BS | $\frac{\rm{TP}+\rm{FP}}{\rm{TP}+\rm{FN}}$

##  5 An Open Platform for Model Validation

We use the existing Comprehensive Assessment of Models and Events Using
Library Tools (CAMEL; Rastätter et al., 2019) framework to make our
developments accessible to the community. The implementation of CAMEL consists
of a front-end and a back-end reaching deep into CCMC’s modeling
infrastructure.

The front-end of CAMEL will make our results visible to the community. It will
enable end-users to visualize state-of-the-art solar wind solutions and
compare them with spacecraft measurements and each other. By selecting time
intervals of interest, end-users will be able to compute the validation
metrics discussed in Section 4. The repertoire of metrics will be expanded and
updated in later versions of the platform to stay up-to-date with operational
and research user needs. CAMEL provides data interpolation options such as
nearest neighbor and linear interpolation for the computation of these
metrics. CAMEL’s front-end will furthermore make all the metadata information
discussed in Section 3 publically available. By doing so, end-users will be
able to access all the background information needed to interpret the model
solutions displayed in the online platform.

The back-end of CAMEL links to existing services at the CCMC. This includes,
for example, an Application Programming Interface (API) allowing end-users to
download the model solutions and computed metrics displayed in the CAMEL
front-end. In a future step, we will offer solar wind model developers the
opportunity to feed the solutions of models currently installed at the CCMC
directly into CAMEL. This way, model developers can make their models and
updated versions, including a comprehensive validation analysis, readily
accessible to the community. For more information on CAMEL, we refer the
interested reader to Rastätter et al. (2019).

##  6 Discussion and Outlook

Our ability to model the global corona and solar wind has grown significantly
over the past decades. The validation of state-of-the-art model solutions has,
however, stagnated. Validation of the quality of models for selected events
and time intervals, usage of individually developed metrics, and a slow
iterative process between developers and end-users cause a bottleneck. These
validation practices make a complete assessment of the state-of-the-art
difficult or even impossible. This paper has discussed the first steps of the
Ambient Solar Wind Validation Team to address these challenges by streamlining
the validation of solar wind models. By collecting feedback during team
meetings and international conferences, our team has identified forecasting
goals of importance to the research and forecasting communities, developed a
metadata architecture to register solar wind models, and recommended metrics
that reflect a broad spectrum of scientific and forecasting needs. We have
also taken the first steps towards developing an open online platform by
feeding all these developments into CAMEL (Rastätter et al., 2019) hosted by
NASA’s CCMC.

The online platform proposed here does not solve the open problems in large-
scale solar wind modeling, but it does help to bridge the gap between
developers and end-users and thereby accelerate the feedback loop to drive
innovation and progress in solar wind model development. Our approach for
streamlining the validation in space weather research is not new; the need for
automated validation protocols has been identified before in MacNeice (2018),
and similar community strategies have been successfully implemented in
adjacent space weather disciplines. The International Forum for Space Weather
Capabilities Assessment is devoted to creating a community-wide consensus on
continuous assessment of space weather predictive capabilities in line with
the COSPAR Space Weather Roadmap (Schrijver et al., 2015). Participating teams
in this forum such as the CME Arrival Time and Impact Working Team and the
Solar Energetic Particle Validation Team have developed a similar approach for
assessing the predictive skill of models (see Verbeke et al., 2019). Other
examples are Pulkkinen et al. (2013), Glocer et al. (2016), and the GEM CEDAR
Challenge.

While our effort can provide a step forward in validating the state-of-the-
art, additional factors need to be considered. First, we expect the proposed
metadata architecture to capture most of the details required to make the
model solutions reproducible. However, we cannot claim that the metadata will
completely capture all of these details. Frameworks of coupled corona and
inner heliosphere models are sophisticated numerical algorithms with many
parameters and settings. Second, the online platform is designed to capture
the current state-of-the-art. Although considerably faster than reporting new
model results in the literature, our approach in the present form still needs
overhead to upload new solutions manually. Third, and related to the last
note, in an ideal case, the validation should be carried out by an
independent, unbiased agent, without model developers uploading individual
solutions to the online platform. In the future, we expect to use model runs
directly from the CCMC, with the model developer’s permission, to allow
solutions to be fed directly into the online platform. Many models are already
hosted at the CCMC. We argue that trust in the scientific integrity of
participating models is necessary, and the opportunity to upload model
solutions from external sources will continue to allow a comprehensive
comparison of community models.

In the near future, work is ongoing to implement and improve upon these first
steps. Most metrics have already been implemented in a beta version of the
online platform. In the next step, we will release the first version of the
platform online to the public, including the complete list of metrics and
access to the metadata information of participating models. Specifically, we
will explore the physical properties of the evolving solar wind flow such as
solar wind speed, density, temperature, magnetic polarity, and magnetic sector
boundary crossings with either point-to-point comparison metrics and more
advanced event-based validation metrics. We will also continue to advertise
the open platform in the space weather community to add more solar wind models
and provide a more comprehensive state-of-the-art validation. Almost all model
developers listed in Figure 1 have agreed to provide their model solutions to
the online platform. Furthermore, we will focus on new data available from
recently launched space missions such as Parker Solar Probe (Fox et al., 2016)
and Solar Orbiter (Müller et al., 2013), and expand our repertoire of metrics
to validate probabilistic model solutions.

In our long-term planning, one future avenue is to use the infrastructure
developed here to implement an Ambient Solar Wind Scoreboard to assess
predictive model capabilities in real-time. A real-time assessment and display
of large-scale solar wind models can be a valuable tool for improved space
weather awareness at Earth and other planetary environments.

##  7 Summary

The Ambient Solar Wind Validation Team embedded in the Committee on Space
Research (COSPAR) - International Space Weather Action Teams (ISWAT)
initiative was formed to support the space weather community to maintain up-
to-date solar wind model validation and track progress over time. This study
presents our first steps towards developing an open platform for validating
solar wind forecasting models. We propose a metadata architecture consisting
of seven components to support a continuous, transparent, and reproducible
validation of solar wind models. Next, we recommend community-wide forecasting
goals, including the forecast of the temporal evolution of the solar wind and
abrupt changes from slow to fast solar wind, as well as the forecast of the
magnetic polarity and magnetic boundary crossings. We furthermore discuss a
comprehensive validation procedure based on the point-to-point comparison,
binary, and event-based metrics. We conclude with a status update of the
online platform and outline future perspectives of this community effort.

##  8 Acknowledgments

The authors thank all contributors to the COSPAR ISWAT initiative, which
supported this research effort. The authors acknowledge the following
organizations and programs: M.A.R., and C.M., acknowledge the Austrian Science
Fund (FWF): P34437, J4160-N27, P31659, P31521; K.M. acknowledges support by
the NASA HGI program (# 80HQTR19T0028), the NASA cooperative agreement
NNG11PL10A and 80NSSC21M0180; J.J.G.A. acknowledges to Investigadores por
México-CONACYT (CONACYT Fellow), CONACYT LN 315829 (2021) and CONACYT-AEM
2017-01-292684 grants. S.D. acknowledges support from the argentianean grants
PICT-2019-02754 (FONCyT-ANPCyT) and UBACyT-20020190100247BA (UBA).

## References

  * Schrijver et al. (2015) Schrijver et al., C. J. (2015).  Understanding space weather to shield society: A global road map for 2015-2025 commissioned by COSPAR and ILWS.  Advances in Space Research, 55, 2745–2807. doi:10.1016/j.asr.2015.03.023. arXiv:1503.06135.
  * Altschuler & Newkirk (1969) Altschuler, M. D., & Newkirk, G. (1969).  Magnetic Fields and the Structure of the Solar Corona. I: Methods of Calculating Coronal Fields.  Solar Physics, 9, 131--149. doi:10.1007/BF00145734.
  * Arge et al. (2010) Arge, C. N., Henney, C. J., Koller, J., Compeau, C. R., Young, S., MacKenzie, D., Fay, A., & Harvey, J. W. (2010).  Air Force Data Assimilative Photospheric Flux Transport (ADAPT) Model.  In M. Maksimovic, K. Issautier, N. Meyer-Vernet, M. Moncuquet, & F. Pantellini (Eds.), Twelfth International Solar Wind Conference (pp. 343--346).  volume 1216 of American Institute of Physics Conference Series.  doi:10.1063/1.3395870.
  * Arge et al. (2003) Arge, C. N., Odstrcil, D., Pizzo, V. J., & Mayer, L. R. (2003).  Improved Method for Specifying Solar Wind Speed Near the Sun.  In M. Velli, R. Bruno, F. Malara, & B. Bucci (Eds.), Solar Wind Ten (pp. 190--193).  volume 679 of American Institute of Physics Conference Series.  doi:10.1063/1.1618574.
  * Arge & Pizzo (2000) Arge, C. N., & Pizzo, V. J. (2000).  Improvement in the prediction of solar wind conditions using near-real time solar magnetic field updates.  Journal of Geophysical Research, 105, 10465--10480. doi:10.1029/1999JA000262.
  * Asvestari et al. (2019) Asvestari, E., Heinemann, S. G., Temmer, M., Pomoell, J., Kilpua, E., Magdalenic, J., & Poedts, S. (2019).  Reconstructing Coronal Hole Areas With EUHFORIA and Adapted WSA Model: Optimizing the Model Parameters.  Journal of Geophysical Research (Space Physics), 124(11), 8280--8297. doi:10.1029/2019JA027173. arXiv:1907.03337.
  * Bailey et al. (2021) Bailey, R. L., Reiss, M. A., Arge, C. N., Möstl, C., Henney, C. J., Owens, M. J., Amerstorfer, U. V., Amerstorfer, T., Weiss, A. J., & Hinterreiter, J. (2021).  Using Gradient Boosting Regression to Improve Ambient Solar Wind Model Predictions.  Space Weather, 19(5), e02673. doi:10.1029/2020SW002673. arXiv:2006.12835.
  * Bloomfield et al. (2012) Bloomfield, D. S., Higgins, P. A., McAteer, R. T. J., & Gallagher, P. T. (2012).  Toward Reliable Benchmarking of Solar Flare Forecasting Methods.  The Astrophysical Journal Letters, 747(2), L41. doi:10.1088/2041-8205/747/2/L41. arXiv:1202.5995.
  * Boe et al. (2020) Boe, B., Habbal, S., & Druckmüller, M. (2020).  Coronal Magnetic Field Topology from Total Solar Eclipse Observations.  The Astrophysical Journal, 895(2), 123. doi:10.3847/1538-4357/ab8ae6. arXiv:2004.08970.
  * Bussy-Virat & Ridley (2014) Bussy-Virat, C. D., & Ridley, A. J. (2014).  Predictions of the solar wind speed by the probability distribution function model.  Space Weather, 12, 337--353. doi:10.1002/2014SW001051.
  * Case et al. (2008) Case, A. W., Spence, H. E., Owens, M. J., Riley, P., & Odstrcil, D. (2008).  Ambient solar wind’s effect on ICME transit times.  Geophysical Research Letters, 35(15), L15105. doi:10.1029/2008GL034493.
  * Chandorkar et al. (2020) Chandorkar, M., Furtlehner, C., Poduval, B., Camporeale, E., & Sebag, M. (2020).  Dynamic time lag regression: Predicting what & when.  In International Conference on Learning Representations.
  * Cranmer et al. (2017) Cranmer, S. R., Gibson, S. E., & Riley, P. (2017).  Origins of the Ambient Solar Wind: Implications for Space Weather.  Space Science Reviews, 212(3-4), 1345--1384. doi:10.1007/s11214-017-0416-y. arXiv:1708.07169.
  * Devos et al. (2014) Devos, A., Verbeeck, C., & Robbrecht, E. (2014).  Verification of space weather forecasting at the Regional Warning Center in Belgium.  Journal of Space Weather and Space Climate, 4(27), A29. doi:10.1051/swsc/2014025.
  * Fox et al. (2016) Fox, N. J., Velli, M. C., Bale, S. D., Decker, R., Driesman, A., Howard, R. A., Kasper, J. C., Kinnison, J., Kusterer, M., Lario, D., Lockwood, M. K., McComas, D. J., Raouafi, N. E., & Szabo, A. (2016).  The Solar Probe Plus Mission: Humanity’s First Visit to Our Star.  Space Science Reviews, 204(1-4), 7--48. doi:10.1007/s11214-015-0211-6.
  * Fung et al. (2022) Fung, F. S., Lee, N. A., Andre, N., Bargatze, L. F., Bouchemit, M., Candey, R. M., Cecconi, B., De Zeeuw, D., Gnot, V., Jian, L., King, T., Masson, A., McGranaghan, R., Muglach, K., Reiss, M. A., Ringuette, R., Roberts, D. A., Vandegriff, J., & Wiegand, C. (2022).  SPASE Metadata as a Heliophysics Science-Enabling Tool.  Advances in Space Research, .
  * Glocer et al. (2016) Glocer, A., Rastätter, L., Kuznetsova, M., Pulkkinen, A., Singer, H. J., Balch, C., Weimer, D., Welling, D., Wiltberger, M., Raeder, J., Weigel, R. S., McCollough, J., & Wing, S. (2016).  Community-wide validation of geospace model local K-index predictions to support model transition to operations.  Space Weather, 14(7), 469--480. doi:10.1002/2016SW001387.
  * Gonzi et al. (2021) Gonzi, S., Weinzierl, M., Bocquet, F. X., Bisi, M. M., Odstrcil, D., Jackson, B. V., Yeates, A. R., Jackson, D. R., Henney, C. J., & Nick Arge, C. (2021).  Impact of Inner Heliospheric Boundary Conditions on Solar Wind Predictions at Earth.  Space Weather, 19(1), e02499. doi:10.1029/2020SW002499.
  * Gosling & Pizzo (1999) Gosling, J. T., & Pizzo, V. J. (1999).  Formation and Evolution of Corotating Interaction Regions and their Three Dimensional Structure.  Space Science Reviews, 89, 21--52. doi:10.1023/A:1005291711900.
  * Gressl et al. (2014) Gressl, C., Veronig, A. M., Temmer, M., Odstrčil, D., Linker, J. A., Mikić, Z., & Riley, P. (2014).  Comparative Study of MHD Modeling of the Background Solar Wind.  Solar Physics, 289, 1783--1801. doi:10.1007/s11207-013-0421-6. arXiv:1312.1220.
  * Henley & Pope (2017) Henley, E. M., & Pope, E. C. D. (2017).  Cost-Loss Analysis of Ensemble Solar Wind Forecasting: Space Weather Use of Terrestrial Weather Tools.  Space Weather, 15(12), 1562--1566. doi:10.1002/2017SW001758.
  * Hickmann et al. (2015) Hickmann, K. S., Godinez, H. C., Henney, C. J., & Arge, C. N. (2015).  Data Assimilation in the ADAPT Photospheric Flux Transport Model.  Solar Physics, 290(4), 1105--1118. doi:10.1007/s11207-015-0666-3. arXiv:1410.6185.
  * Hinterreiter et al. (2019) Hinterreiter, J., Magdalenic, J., Temmer, M., Verbeke, C., Jebaraj, I. C., Samara, E., Asvestari, E., Poedts, S., Pomoell, J., Kilpua, E., Rodriguez, L., Scolini, C., & Isavnin, A. (2019).  Assessing the Performance of EUHFORIA Modeling the Background Solar Wind.  Solar Physics, 294(12), 170. doi:10.1007/s11207-019-1558-8. arXiv:1907.07461.
  * Hoeksema et al. (1982) Hoeksema, J. T., Wilcox, J. M., & Scherrer, P. H. (1982).  Structure of the heliospheric current sheet in the early portion of sunspot cycle 21.  Journal of Geophysical Research, 87, 10331--10338. doi:10.1029/JA087iA12p10331.
  * Issan & Riley (2022) Issan, O., & Riley, P. (2022).  Theoretical refinements to the heliospheric upwind extrapolation technique and application to in-situ measurements.  Frontiers in Astronomy and Space Sciences, 8. URL: https://www.frontiersin.org/article/10.3389/fspas.2021.795323. doi:10.3389/fspas.2021.795323.
  * Jackson et al. (2015) Jackson, B. V., Odstrcil, D., Yu, H. S., Hick, P. P., Buffington, A., Mejia-Ambriz, J. C., Kim, J., Hong, S., Kim, Y., Han, J., & Tokumaru, M. (2015).  The UCSD kinematic IPS solar wind boundary and its use in the ENLIL 3-D MHD prediction model.  Space Weather, 13(2), 104--115. doi:10.1002/2014SW001130.
  * Jian et al. (2016) Jian, L. K., MacNeice, P. J., Mays, M. L., Taktakishvili, A., Odstrcil, D., Jackson, B., Yu, H. S., Riley, P., & Sokolov, I. V. (2016).  Validation for global solar wind prediction using Ulysses comparison: Multiple coronal and heliospheric models installed at the Community Coordinated Modeling Center.  Space Weather, 14(8), 592--611. doi:10.1002/2016SW001435.
  * Jian et al. (2015) Jian, L. K., MacNeice, P. J., Taktakishvili, A., Odstrcil, D., Jackson, B., Yu, H.-S., Riley, P., Sokolov, I. V., & Evans, R. M. (2015).  Validation for solar wind prediction at Earth: Comparison of coronal and heliospheric models installed at the CCMC.  Space Weather, 13, 316--338. doi:10.1002/2015SW001174.
  * Jian et al. (2011) Jian, L. K., Russell, C. T., Luhmann, J. G., MacNeice, P. J., Odstrcil, D., Riley, P., Linker, J. A., Skoug, R. M., & Steinberg, J. T. (2011).  Comparison of Observations at ACE and Ulysses with Enlil Model Results: Stream Interaction Regions During Carrington Rotations 2016 - 2018.  Solar Physics, 273(1), 179--203. doi:10.1007/s11207-011-9858-7.
  * Jones et al. (2017) Jones, S. I., Uritsky, V., & Davila, J. M. (2017).  Image-optimized Coronal Magnetic Field Models.  The Astrophysical Journal, 844(2), 93. doi:10.3847/1538-4357/aa7b7a. arXiv:1706.07316.
  * Kilpua et al. (2017) Kilpua, E. K. J., Balogh, A., von Steiger, R., & Liu, Y. D. (2017).  Geoeffective Properties of Solar Transients and Stream Interaction Regions.  Space Science Reviews, 212(3-4), 1271--1314. doi:10.1007/s11214-017-0411-3.
  * Kim et al. (2014) Kim, T. K., Pogorelov, N. V., Borovikov, S. N., Jackson, B. V., Yu, H. S., & Tokumaru, M. (2014).  MHD heliosphere with boundary conditions from a tomographic reconstruction using interplanetary scintillation data.  Journal of Geophysical Research (Space Physics), 119(10), 7981--7997. doi:10.1002/2013JA019755.
  * Kohutova et al. (2016) Kohutova, P., Bocquet, F.-X., Henley, E. M., & Owens, M. J. (2016).  Improving solar wind persistence forecasts: Removing transient space weather events, and using observations away from the Sun-Earth line.  Space Weather, 14(10), 802--818. doi:10.1002/2016SW001447.
  * Lamy et al. (2019) Lamy, P., Floyd, O., Mikić, Z., & Riley, P. (2019).  Validation of MHD Model Predictions of the Corona with LASCO-C2 Polarized Brightness Images.  Solar Physics, 294(11), 162. doi:10.1007/s11207-019-1549-9.
  * Lang et al. (2017) Lang, M., Browne, P., van Leeuwen, P. J., & Owens, M. (2017).  Data Assimilation in the Solar Wind: Challenges and First Results.  Space Weather, 15(11), 1490--1510. doi:10.1002/2017SW001681.
  * Lang & Owens (2019) Lang, M., & Owens, M. J. (2019).  A Variational Approach to Data Assimilation in the Solar Wind.  Space Weather, 17(1), 59--83. doi:10.1029/2018SW001857. arXiv:1806.07306.
  * Li et al. (2020) Li, H., Feng, X., & Wei, F. (2020).  Assessment of CESE-HLLD ambient solar wind model results using multipoint observation.  Journal of Space Weather and Space Climate, 10, 44. doi:10.1051/swsc/2020048.
  * Linker et al. (1999) Linker, J. A., Mikić, Z., Biesecker, D. A., Forsyth, R. J., Gibson, S. E., Lazarus, A. J., Lecinski, A., Riley, P., Szabo, A., & Thompson, B. J. (1999).  Magnetohydrodynamic modeling of the solar corona during Whole Sun Month.  Journal of Geophysical Research, 104, 9809--9830. doi:10.1029/1998JA900159.
  * Luhmann et al. (2002) Luhmann, J. G., Li, Y., Arge, C. N., Gazis, P. R., & Ulrich, R. (2002).  Solar cycle changes in coronal holes and space weather cycles.  Journal of Geophysical Research (Space Physics), 107, 1154. doi:10.1029/2001JA007550.
  * MacNeice (2009a) MacNeice, P. (2009a).  Validation of community models: 2. Development of a baseline using the Wang-Sheeley-Arge model.  Space Weather, 7, S12002. doi:10.1002/2009SW000489.
  * MacNeice (2009b) MacNeice, P. (2009b).  Validation of community models: Identifying events in space weather model timelines.  Space Weather, 7, S06004. doi:10.1029/2009SW000463.
  * MacNeice (2018) MacNeice, P. (2018).  On the Need to Automate Support for Quality Assessment Studies of Space Weather Models.  Space Weather, 16(11), 1627--1634. doi:10.1029/2018SW002039.
  * MacNeice et al. (2011) MacNeice, P., Elliott, B., & Acebal, A. (2011).  Validation of community models: 3. Tracing field lines in heliospheric models.  Space Weather, 9, S10003. doi:10.1029/2011SW000665.
  * MacNeice et al. (2018) MacNeice, P., Jian, L. K., Antiochos, S. K., Arge, C. N., Bussy-Virat, C. D., DeRosa, M. L., Jackson, B. V., Linker, J. A., Mikic, Z., Owens, M. J., Ridley, A. J., Riley, P., Savani, N., & Sokolov, I. (2018).  Assessing the Quality of Models of the Ambient Solar Wind.  Space Weather, 16(11), 1644--1667. doi:10.1029/2018SW002040.
  * Merkin et al. (2016) Merkin, V. G., Lyon, J. G., Lario, D., Arge, C. N., & Henney, C. J. (2016).  Time-dependent magnetohydrodynamic simulations of the inner heliosphere.  Journal of Geophysical Research (Space Physics), 121(4), 2866--2890. doi:10.1002/2015JA022200.
  * Mikić et al. (1999) Mikić, Z., Linker, J. A., Schnack, D. D., Lionello, R., & Tarditi, A. (1999).  Magnetohydrodynamic modeling of the global solar corona.  Physics of Plasmas, 6, 2217--2224. doi:10.1063/1.873474.
  * Müller et al. (2013) Müller, D., Marsden, R. G., St. Cyr, O. C., & Gilbert, H. R. (2013).  Solar Orbiter . Exploring the Sun-Heliosphere Connection.  Solar Physics, 285(1-2), 25--70. doi:10.1007/s11207-012-0085-7. arXiv:1207.4579.
  * Norquist & Meeks (2010) Norquist, D. C., & Meeks, W. C. (2010).  A comparative verification of forecasts from two operational solar wind models.  Space Weather, 8(12), S12005. doi:10.1029/2010SW000598.
  * Odstrcil (2003) Odstrcil, D. (2003).  Modeling 3-D solar wind structure.  Advances in Space Research, 32, 497--506. doi:10.1016/S0273-1177(03)00332-6.
  * Odstrčil & Pizzo (1999) Odstrčil, D., & Pizzo, V. J. (1999).  Three-dimensional propagation of CMEs in a structured solar wind flow: 1. CME launched within the streamer belt.  Journal of Geophysical Research, 104, 483--492. doi:10.1029/1998JA900019.
  * Owens et al. (2020) Owens, M., Lang, M., Barnard, L., Riley, P., Ben-Nun, M., Scott, C. J., Lockwood, M., Reiss, M. A., Arge, C. N., & Gonzi, S. (2020).  A Computationally Efficient, Time-Dependent Model of the Solar Wind for Use as a Surrogate to Three-Dimensional Numerical Magnetohydrodynamic Simulations.  Solar Physics, 295(3), 43. doi:10.1007/s11207-020-01605-3.
  * Owens (2018) Owens, M. J. (2018).  Time-Window Approaches to Space-Weather Forecast Metrics: A Solar Wind Case Study.  Space Weather, 16(11), 1847--1861. doi:10.1029/2018SW002059.
  * Owens et al. (2005) Owens, M. J., Arge, C. N., Spence, H. E., & Pembroke, A. (2005).  An event-based approach to validating solar wind speed predictions: High-speed enhancements in the wang-sheeley-arge model.  Journal of Geophysical Research: Space Physics, 110(A12), n/a--n/a. doi:10.1029/2005JA011343.  A12105.
  * Owens et al. (2013) Owens, M. J., Challen, R., Methven, J., Henley, E., & Jackson, D. R. (2013).  A 27 day persistence model of near-earth solar wind conditions: A long lead-time forecast and a benchmark for dynamical models.  Space Weather, 11(5), 225--236. doi:10.1002/swe.20040.
  * Owens et al. (2017) Owens, M. J., Riley, P., & Horbury, T. S. (2017).  Probabilistic Solar Wind and Geomagnetic Forecasting Using an Analogue Ensemble or ‘‘Similar Day’’ Approach.  Solar Physics, 292, #69. doi:10.1007/s11207-017-1090-7.
  * Owens et al. (2008) Owens, M. J., Spence, H. E., McGregor, S., Hughes, W. J., Quinn, J. M., Arge, C. N., Riley, P., Linker, J., & Odstrcil, D. (2008).  Metrics for solar wind prediction models: Comparison of empirical, hybrid, and physics-based schemes with 8 years of L1 observations.  Space Weather, 6(8), S08001. doi:10.1029/2007SW000380.
  * Pinto & Rouillard (2017) Pinto, R. F., & Rouillard, A. P. (2017).  A Multiple Flux-tube Solar Wind Model.  The Astrophysical Journal, 838(2), 89. doi:10.3847/1538-4357/aa6398. arXiv:1611.08744.
  * Pomoell & Poedts (2018) Pomoell, J., & Poedts, S. (2018).  EUHFORIA: European heliospheric forecasting information asset.  Journal of Space Weather and Space Climate, 8, A35. doi:10.1051/swsc/2018020.
  * Pulkkinen et al. (2013) Pulkkinen, A., RastäTter, L., Kuznetsova, M., Singer, H., Balch, C., Weimer, D., Toth, G., Ridley, A., Gombosi, T., Wiltberger, M., Raeder, J., & Weigel, R. (2013).  Community-wide validation of geospace model ground magnetic field perturbation predictions to support model transition to operations.  Space Weather, 11(6), 369--385. doi:10.1002/swe.20056.
  * Rastätter et al. (2019) Rastätter, L., Wiegand, C. P., Mullinix, R. E., & MacNeice, P. J. (2019).  Comprehensive Assessment of Models and Events Using Library Tools (CAMEL) Framework: Time Series Comparisons.  Space Weather, 17(6), 845--860. doi:10.1029/2018SW002043.
  * Reiss et al. (2019) Reiss, M. A., MacNeice, P. J., Mays, L. M., Arge, C. N., Möstl, C., Nikolic, L., & Amerstorfer, T. (2019).  Forecasting the Ambient Solar Wind with Numerical Models. I. On the Implementation of an Operational Framework.  The Astrophysical Journals, 240(2), 35. doi:10.3847/1538-4365/aaf8b3. arXiv:1905.04353.
  * Reiss et al. (2020) Reiss, M. A., MacNeice, P. J., Muglach, K., Arge, C. N., Möstl, C., Riley, P., Hinterreiter, J., Bailey, R. L., Weiss, A. J., Owens, M. J., Amerstorfer, T., & Amerstorfer, U. (2020).  Forecasting the Ambient Solar Wind with Numerical Models. II. An Adaptive Prediction System for Specifying Solar Wind Speed near the Sun.  The Astrophysical Journal, 891(2), 165. doi:10.3847/1538-4357/ab78a0. arXiv:2003.09336.
  * Reiss et al. (2016) Reiss, M. A., Temmer, M., Veronig, A. M., Nikolic, L., Vennerstrom, S., Schoengassner, F., & Hofmeister, S. J. (2016).  Verification of high-speed solar wind stream forecasts using operational solar wind models.  Space Weather, 14(7), 2016SW001390. doi:10.1002/2016SW001390.
  * Réville et al. (2015) Réville, V., Brun, A. S., Strugarek, A., Matt, S. P., Bouvier, J., Folsom, C. P., & Petit, P. (2015).  From Solar to Stellar Corona: The Role of Wind, Rotation, and Magnetism.  The Astrophysical Journal, 814(2), 99. doi:10.1088/0004-637X/814/2/99. arXiv:1509.06982.
  * Réville et al. (2020) Réville, V., Velli, M., Panasenco, O., Tenerani, A., Shi, C., Badman, S. T., Bale, S. D., Kasper, J. C., Stevens, M. L., Korreck, K. E., Bonnell, J. W., Case, A. W., de Wit, T. D., Goetz, K., Harvey, P. R., Larson, D. E., Livi, R., Malaspina, D. M., MacDowall, R. J., Pulupa, M., & Whittlesey, P. L. (2020).  The Role of Alfvén Wave Dynamics on the Large-scale Properties of the Solar Wind: Comparing an MHD Simulation with Parker Solar Probe E1 Data.  The Astrophysical Journal Supplement Series, 246(2), 24\. doi:10.3847/1538-4365/ab4fef. arXiv:1912.03777.
  * Riley & Ben-Nun (2021) Riley, P., & Ben-Nun, M. (2021).  On the Sources and Sizes of Uncertainty in Predicting the Arrival Time of Interplanetary Coronal Mass Ejections Using Global MHD Models.  Space Weather, 19(6), e02775. doi:10.1029/2021SW002775.
  * Riley et al. (2017) Riley, P., Ben-Nun, M., Linker, J. A., Owens, M. J., & Horbury, T. S. (2017).  Forecasting the properties of the solar wind using simple pattern recognition.  Space Weather, 15(3), 526--540. doi:10.1002/2016SW001589.
  * Riley et al. (1997) Riley, P., Gosling, J. T., & Pizzo, V. J. (1997).  A two-dimensional simulation of the radial and latitudinal evolution of a solar wind disturbance driven by a fast, high-pressure coronal mass ejection.  Journal of Geophysical Research, 102(A7), 14677--14686. doi:10.1029/97JA01131.
  * Riley & Issan (2021) Riley, P., & Issan, O. (2021).  Using a Heliospheric Upwinding eXtrapolation (HUX) Technique to Magnetically Connect Different Regions of the Heliosphere.  Frontiers in Physics, 9, 268. doi:10.3389/fphy.2021.679497.
  * Riley et al. (2015) Riley, P., Linker, J. A., & Arge, C. N. (2015).  On the role played by magnetic expansion factor in the prediction of solar wind speed.  Space Weather, 13, 154--169. doi:10.1002/2014SW001144.
  * Riley et al. (2001) Riley, P., Linker, J. A., & Mikić, Z. (2001).  An empirically-driven global MHD model of the solar corona and inner heliosphere.  Journal of Geophysical Research, 106, 15889--15902. doi:10.1029/2000JA000121.
  * Riley & Lionello (2011) Riley, P., & Lionello, R. (2011).  Mapping Solar Wind Streams from the Sun to 1 AU: A Comparison of Techniques.  Solar Physics, 270, 575--592. doi:10.1007/s11207-011-9766-x.
  * Robbins et al. (2006) Robbins, S., Henney, C. J., & Harvey, J. W. (2006).  Solar Wind Forecasting with Coronal Holes.  Solar Physics, 233, 265--276. doi:10.1007/s11207-006-0064-y. arXiv:astro-ph/0701275.
  * Schatten (1971) Schatten, K. H. (1971).  Current sheet magnetic model for the solar corona.  Cosmic Electrodynamics, 2, 232--245.
  * Schrijver & De Rosa (2003) Schrijver, C. J., & De Rosa, M. L. (2003).  Photospheric and heliospheric magnetic fields.  Solar Physics, 212(1), 165--200. doi:10.1023/A:1022908504100.
  * Shiota et al. (2014) Shiota, D., Kataoka, R., Miyoshi, Y., Hara, T., Tao, C., Masunaga, K., Futaana, Y., & Terada, N. (2014).  Inner heliosphere MHD modeling system applicable to space weather forecasting for the other planets.  Space Weather, 12(4), 187--204. doi:10.1002/2013SW000989.
  * Temmer et al. (2018) Temmer, M., Hinterreiter, J., & Reiss, M. A. (2018).  Coronal hole evolution from multi-viewpoint data as input for a STEREO solar wind speed persistence model.  Journal of Space Weather and Space Climate, 8, A18. doi:10.1051/swsc/2018007. arXiv:1801.10213.
  * Tóth et al. (2005) Tóth et al., G. (2005).  Space Weather Modeling Framework: A new tool for the space science community.  Journal of Geophysical Research (Space Physics), 110, A12226. doi:10.1029/2005JA011126.
  * Upton & Hathaway (2014) Upton, L., & Hathaway, D. H. (2014).  Effects of Meridional Flow Variations on Solar Cycles 23 and 24.  The Astrophysical Journal, 792(2), 142. doi:10.1088/0004-637X/792/2/142. arXiv:1408.0035.
  * Verbanac et al. (2011) Verbanac, G., Vršnak, B., Veronig, A., & Temmer, M. (2011).  Equatorial coronal holes, solar wind high-speed streams, and their geoeffectiveness.  Advances in Astronomy and Space Physics, 526, A20. doi:10.1051/0004-6361/201014617.
  * Verbeke et al. (2019) Verbeke, C., Mays, M. L., Temmer, M., Bingham, S., Steenburgh, R., Dumbović, M., Núñez, M., Jian, L. K., Hess, P., Wiegand, C., Taktakishvili, A., & Andries, J. (2019).  Benchmarking CME Arrival Time and Impact: Progress on Metadata, Metrics, and Events.  Space Weather, 17, 6--26. doi:10.1029/2018SW002046. arXiv:1811.10695.
  * Vršnak et al. (2007) Vršnak, B., Temmer, M., & Veronig, A. M. (2007).  Coronal Holes and Solar Wind High-Speed Streams: I. Forecasting the Solar Wind Parameters.  Solar Physics, 240, 315--330. doi:10.1007/s11207-007-0285-8.
  * Wang & Sheeley (1990) Wang, Y.-M., & Sheeley, N. R., Jr. (1990).  Solar wind speed and coronal flux-tube expansion.  The Astrophysical Journal, 355, 726--732. doi:10.1086/168805.
  * Wang & Sheeley (1995) Wang, Y.-M., & Sheeley, N. R., Jr. (1995).  Solar Implications of ULYSSES Interplanetary Field Measurements.  The Astrophysical Journal Letters, 447, L143. doi:10.1086/309578.
  * Yang et al. (2018) Yang, Y., Shen, F., Yang, Z., & Feng, X. (2018).  Prediction of Solar Wind Speed at 1 AU Using an Artificial Neural Network.  Space Weather, 16(9), 1227--1244. doi:10.1029/2018SW001955.
  * Zhou & Feng (2017) Zhou, Y., & Feng, X. (2017).  Numerical study of the propagation characteristics of coronal mass ejections in a structured ambient solar wind.  Journal of Geophysical Research (Space Physics), 122(2), 1451--1462. doi:10.1002/2016JA023053.