# Verification against in-situ observations for
Data-Driven Weather Prediction

Vivek Ramavajjala
Excarta Inc.
<EMAIL_ADDRESS>
&Peetak P. Mitra
Excarta Inc.
<EMAIL_ADDRESS>

###### Abstract

Data-driven weather prediction models (DDWPs) have made rapid strides in
recent years, demonstrating an ability to perform medium-range weather
forecasting a high degree of accuracy. The fast, accurate, and low-cost
forecasts promised by DDWPs make their use in operational forecasting an
attractive proposition, however, there remains work to be done in rigorously
evaluating DDWPs in a true operational setting. DDWPs have been trained and
evaluated only using the ERA5 reanalysis dataset, which is produced using a
combination of real-world weather observations and a Numerical Weather
Prediction (NWP) model. While the ERA5 dataset represents a very high quality
simulation of global weather, it is still influenced by the underlying NWP and
is not a replacement for real-world observations. Additional benchmarks are
needed to examine how well DDWPs perform when deployed in operational settings
and tested against real-world observations. Such benchmarks can provide
unbiased measures of model quality, and build confidence in DDWPs as they are
deployed in operational settings. We present a robust dataset of in-situ
observations derived from the NOAA Meteorological Assimilation Data Ingest
System (MADIS) program to serve as a benchmark to validate DDWPs in an
operational setting. We use this dataset to compare a well-known DDWP
(FourCastNet) against an NWP baseline (IFS), and observe that even though
FourCastNet outperforms IFS when tested against ERA5, it does not have a
performance benefit when tested against real-world observations. By providing
a large corpus of quality-controlled, in-situ observations, this dataset
provides a meaningful real-world task that all NWPs and DDWPs can be validated
against. We hope that this data can be used not only to rigorously and fairly
compare operational weather models but also to spur future research in new
directions.

##  1 Introduction

Data-driven weather prediction (DDWP) models have made rapid strides in recent
years, steadily improving forecast skill over longer time horizons at a
fraction of the cost of Numerical Weather Prediction (NWP) models. Given the
demonstrated improvements in forecast skill, low inference cost, and the
ability to target specific variables of interest, using DDWPs in operational
weather forecasting is an attractive proposition. Weather forecasts play a
foundational role in day-to-day operations of key sectors like agriculture,
energy, transportation, and in reducing loss of life due to extreme weather
events, thus allowing DDWPs to have a tangible social benefit. This potential
impact also makes it imperative that ML practitioners should take a holistic
view of DDWPs in the operational forecasting setting, testing DDWPs under the
same constraints faced by NWPs that currently underpin operational forecasts.
Similar examination of ML models is common in the development of large
language models (LLMs), where a "red-teaming" approach is used to identify how
a robustly trained LLM may yet produce undesirable outcomes in a real-world
scenario (e.g., Perez et al., (2022), Tamkin et al., (2021)). For DDWPs, a
good starting point is defining evaluation tasks that more accurately reflect
their use in operational forecasts.

The most promising DDWPs developed recently have been trained using the ERA5
reanalysis dataset (Hersbach et al., (2020)), typically using data from
1979-2017 for training and development, and using data from 2018 as a held-out
test set. Produced by combining historical observations and a NWP, the ERA5
data provides "maps without gaps" of essential climate variables (ECVs). The
ML models use a fixed subset of ECVs from ERA5 to represent the atmosphere at
time t, and predict the same subset of ECVs at t+$\delta$, where $\delta$ is
most commonly 6 hours, but varies from 1 hour to 24 hours. All ECVs are
typically normalized to zero mean and unit variance, and a mean-squared error
(MSE) loss is used to train the models. For evaluation, DDWPs are typically
initialized with data from ERA5 at 00UTC on a particular day, and iteratively
rolled out to generate a forecast for the next 14 days. The forecasts are
evaluated by comparing the predictions against ERA5 data using two metrics:
the root mean squared error (RMSE) and the anomaly correlation coefficient
(ACC), computed over the entire latitude-longitude grid, and weighted in
proportion to the latitude. To establish a NWP baseline, the RMSE and ACC
metrics are similarly calculated for ECMWF’s Integrated Forecasting System
(IFS). For details, we refer to FourCastNet (Pathak et al., (2022)), Pangu-
Weather (Bi et al., (2022)), or GraphCast (Lam et al., (2022)), which share
essentially the same procedure for training and evaluating different DDWPs.
DDWPs thus learn a high-quality approximation of the underlying NWP used to
construct the ERA5 dataset, and evaluations using ERA5 data test the quality
of the learned approximation.

###  1.1 Limitations of verification against reanalysis

Verifying DDWP forecasts against ERA5 reanalysis is similar to the
"verification against analysis" procedure used to evaluate operational NWPs,
where the forecast from an NWP is compared to a sequence of analyses produced
from the same NWP111For instance, see ”grid-to-grid” verification against
analysis published by NOAA:
https://www.emc.ncep.noaa.gov/users/verification/global/gfs/ops/grid2grid_all_models/rmse/..
While convenient, since the sequence of analyses depends on the NWP output
from the previous cycle, such evaluations can underestimate the forecast
error, especially for short lead times (Bowler et al., (2015), Bowler,
(2008)). Besides fundamental limitations of verification against analysis,
comparing against ERA5 is not sufficient to estimate how well the DDWP
performs in an operational setting, both objectively and relative to the IFS
baseline.

First, the analysis available for initializing operational DDWPs assimilate
observations from +3 hours ahead, and may not be as robust as the ERA5
reanalysis, which assimilates observations from +9 hours at 00UTC. Second,
evaluating IFS on the ERA5 dataset is not necessarily meaningful since it has
not been optimized to predict ERA5 data, whereas DDWPs have — giving DDWPs an
unfair advantage. Further, since the ERA5 dataset uses an underlying NWP to
create a picture of the global weather, it is inherently biased towards the
underlying NWP and its assumptions, and hence may differ from real weather
conditions, creating a noticeable difference between simulation and reality.
Indeed, simply because a DDWP outperforms the IFS model when tested against
ERA5 does not guarantee the same performance improvement holds in an
operational setting, where both start states and ground truth are different.
Such concerns have already been raised by Bi et al., (2023), who note the
limitations of using only reanalysis data for validation despite real-world
systems working with observational data. We lastly note that operational NWPs
are verified not just against analysis, but also verified against
observations, e.g. ECMWF scorecards report performance against both analysis
and observations222See
https://sites.ecmwf.int/ifs/scorecards/scorecards-48r1ENS.html. This
underscores the need for similar evaluation benchmarks for DDWPs.

In domains such as robotics or RL-based control, the "sim2real" challenge of
transferring and evaluating a model trained via simulation in a real-world
setting is well known (c.f., Kadian et al., (2020)). Viewing the ERA5 dataset
as a high-quality simulation of weather, there is a similar need for a
"sim2real" step that evaluates DDWPs against observations. We also note that
both the industry and common user primarily consume weather forecasts for
specific locations (vs. gridded forecasts), making such "verification against
observations" a meaningful test of operational weather models.

###  1.2 Our contributions

We introduce a dataset of quality-controlled in-situ observations derived from
the NOAA Meteorological Assimilation Data Ingest System (MADIS) program that
can be used to more thoroughly evaluate the performance of DDWPs in an
operational setting. Additionally, since in-situ observations are not modeled
in any way but instead reflect the actual weather on the ground, they can be
used for unbiased comparisons of DDWPs and NWPs on a task meaningful to
consumers of forecasts. In Section 3, we compare FourCastNet DDWP model
(Pathak et al., (2022)) and the IFS model against real-world observations.
FourCastNet has previously shown improvements over the IFS model when
evaluated against ERA5, here we examine its performance against real-world
observations.

Prior work has examined ERA5 data against observations for specific variables
and regions, e.g., Jiao et al., (2021) evaluate ERA5 for precipitation in
China, or to assess its suitability for specific tasks, e.g., choosing
suitable location for wind farms (Olauson, (2018). To our knowledge, this is
the first publicly available comparison of ERA5 against in-situ observations
at such a scale, as well as the first publicly available dataset for verifying
NWPs and DDWPs against observations. The dataset is made available at
https://huggingface.co/datasets/excarta/madis2020.

##  2 Dataset

###  2.1 Background of the MADIS database

MADIS333https://madis.ncep.noaa.gov/index.shtml (Meteorological Assimilation
Data Ingest System) is a database provided by NOAA (National Oceanic and
Atmospheric Administration) that contains meteorological observations covering
the entire globe. MADIS ingests data from NOAA and non-NOAA sources, including
observation networks ("mesonets") from US federal, state, and transportation
agencies, universities, volunteer networks, and data from private sectors like
airlines as well as public-private partnerships like CWOP (Citizen Weather
Observer Program). Whereas reanalysis data is influenced by the underlying
physics of the NWP used to reconstruct the reanlaysis, these observations
directly capture the weather on the ground and can be used to objectively
evaluate any data-driven or numerical weather model, without being biased
towards either of the approaches.

###  2.2 Curating the MADIS2020 dataset

We created a dataset "MADIS2020" consisting of hourly observations from the
Mesonet and METAR networks ingested by MADIS444Not all sources ingested by
MADIS are free for public use, we choose Mesonet and METAR sources as they are
publicly usable, and have large data volumes. for the entirety of 2020,
totaling over 700 million quality-checked observations. Mesonet observations
represent data from different weather observation networks, while METAR data
primarily indicates data collected at airports. We have additionally taken
care to use the subset of MADIS data fully open to
public555https://madis.ncep.noaa.gov/mesonet_providers.shtml, and to follow
the FAIR principles Wilkinson et al., (2016) in aggregating and disseminating
the data. Neither the raw nor the curated data contains any personally
identifiable information.

###  2.3 Dataset features and the quality thresholds

We limit ourselves to a few key surface meteorological variables that have an
outsized impact on society, discussed in Table 1 :

Table 1: Variables in dataset, provided at an hourly frequency for the entire year of 2020 Variable name | Description | Short name | Units
---|---|---|---
Temperature | Air temperature at 2m height | t2m | $\mathtt{K}$
Dewpoint | Dewpoint temperature at 2m height | d2m | $\mathtt{K}$
Wind speed | Windspeeds calculated from 10m u and v components | wind | $\mathtt{ms^{-1}}$

Temperature, dewpoint, and wind speeds are instantaneous observations and not
averaged over a time window. The MADIS database also includes quality control
flags that specify the level to which an observation has been
checked666https://madis.ncep.noaa.gov/madis_sfc_qc_notes.shtml:

  * •

Level 1: Observation is checked to lie within a feasible range:

Temperature between $\mathtt{-60F}$ to $\mathtt{130F}$

Dewpoint between $\mathtt{-90F}$ to $\mathtt{90F}$

Wind speed must lie between $\mathtt{0kts}$ \- $\mathtt{250kts}$

  * •

Level 2: Internal consistency, temporal consistency, and statistical spatial
consistency are checked. Internal consistency checks enforce meteorological
relationships between different variables at the same station, e.g., dewpoint
must not exceed temperature. The temporal consistency check flag observations
that change too rapidly, and the statistical spatial consistency check flag
observations that have failed quality checks 75% of the time in the last 7
days.

  * •

Level 3: Spatial consistency checks that verify one observation against nearby
observations.

Based on the QC levels above, different quality flags are defined and applied
for each observation:

  * •

C: Coarse pass, passed level 1

  * •

S: Screened, passed levels 1 and 2

  * •

V: Verified, passed levels 1, 2, and 3

  * •

X: Rejected, failed level 1

  * •

Q: Erroneous, passed level 1 but failed level 2 or 3

We include only observations that have QC flags S or V, i.e., passing at least
level 2 checks. Some observations pass level 2 checks, but are in regions too
data-sparse to support level 3 checks, and may still introduce noisy
observations. To remove such noisy observations, we ignore temperature and
dewpoint observations that were more than 20 C away from the ERA5 prediction.

###  2.4 Dataset geographical coverage

We partition the observations into different geographical regions, extending
from the list of regions used in ECMWF
scorecards777https://sites.ecmwf.int/ifs/scorecards/scorecards-48r1ENS.html,
graphically shown in the Appendix figure 5 and the latitude/longitude extents
are provided in Table 2.

It is to be noted that the above regions are not mutually exclusive, and a
small number of observations in the "global" region do not fall under any of
the defined regions. Figure 1 shows the distribution of observations globally
and for each region. All regions have at least 1,000,000 observations for
temperature, wind speeds, and dewpoint. Unsurprisingly, these data are not
uniformly distributed across the world, with North America and Europe having
the greatest density of observations, and the global South having a much lower
density of observation data.

Figure 1: Total number of observations for different variables in 2020, in log
(base 10) scale.

###  2.5 Comparing observations against ERA5

We first test how well the ERA5 dataset corresponds to real-world
observations, we compare each hourly observation for all weather stations in
MADIS2020 with the corresponding ERA5 estimate, using RMSE to quantify the
error for each region. To identify diurnal variation in quality, we report the
RMSE of ERA5 vs. observations for separately for each hour of the day. Figure
2 shows the RMSE for wind speeds, temperature, and dewpoint. We note that ERA5
shows a consistent performance across all regions for wind speeds, but has a
notable regional variation in quality for temperature and dewpoint. In
particular, regions in Asia and the global South have higher RMSE for
temperature and dewpoint.

Since this regional variation exists in the data itself, it is expected that
any models trained on ERA5 data would have similar regional variations as
well, and should be kept in mind when training models on ERA5 data.

Figure 2: RMSE of ERA5 data vs. real-world observations for different regions.

##  3 Validating a Data-Driven Weather Prediction model

Having set up the MADIS2020 dataset, we use it to examine whether a DDWP
performs as well against real-world data as it does in simulations. We choose
FourCastNet as the DDWP to evaluate, since it has a public implementation and
is relativel easy to replicate. As with prior work in DDWPs, we use the IFS
forecast from ECMWF as an NWP baseline.

###  3.1 Training the DDWP

We train a variant of the FourCastNet (FCN) model, using the same model
architecture and hyperparameters described by Pathak et al., (2022). The
original FCN implementation888A public implementation is available at
https://github.com/NVlabs/FourCastNet. produces 6-hourly forecasts, and we
train an additional FCN model to make 1-hourly forecasts, using 1-hourly ERA5
data but keeping other training and modeling choices the same as the 6-hourly
model. We use a hierarchical temporal aggregation process (similar to the
process followed by Bi et al., (2022)) to interleave the two models and obtain
hourly forecasts at 0.25° resolution. Additionally, we incorporate dewpoint at
2m height as an additional input and output variable.

###  3.2 Verification against reanalysis

Since MADIS2020 contains observations only from 2020, we use ERA5 data from
2020 to validate FCN and IFS. Following the prior AI-based methods, we
initialize the FCN model using ERA5 data at 00UTC and produce 48-hour
forecasts. We compute the latitude-weighted RMSE and Anomaly Correlation
Coefficient (ACC) metrics as defined below:

$\displaystyle\text{RMSE}(v,t)$
$\displaystyle=\sqrt{\frac{\sum_{i=1}^{N_{lat}}\sum_{j=1}^{N_{lon}}L(i)(\hat{X}^{v}_{i,j,t}-X^{v}_{i,j,t})^{2}}{N_{lat}N_{lon}}}$
$\displaystyle\text{ACC}(v,t)$
$\displaystyle=\frac{\sum_{i,j}L(i)\hat{X}^{\prime v}_{i,j,t}X^{\prime
v}_{i,j,t}}{\sqrt{\sum_{i,j}L(i)(\hat{X}^{\prime
v}_{i,j,t})^{2}\sum_{i,j}L(i)(X^{\prime v}_{i,j,t})^{2}}}$

where
$L(i)=N_{\text{lat}}\frac{\cos{\phi}}{\sum_{i^{\prime}=1}^{N_{\text{lat}}}\cos{\phi_{i^{\prime}}}}$
stands for the weight at latitude $\phi_{i}$ and X’ denotes the difference
between $X$ and the climatology. We calcuate the RMSE and ACC metrics for
temperature, dewpoint, and u10 (the eastward component of windspeed).
Following prior work in DDWPs, We exclude v10, the northward component of
windspeed, as it is highly correlated with u10. A lower RMSE and higher ACC
indicate a better performing model, we refer to Pathak et al., (2022), Bi et
al., (2022), Lam et al., (2022) for more detailed comparisons on other DDWPs
againste ERA5.

Figure 3 shows the RMSE and ACC for FCN and IFS for the first 48 hours of
forecast lead time. Compared to IFS, FourCastNet clearly shows a reduction in
RMSE and improvement in ACC, validating previously published results for
FourCastNet.

Figure 3: RMSE and ACC comparison for FourCastNet (FCN) and IFS for a 48 hour
forecast. The top row shows RMSE, while the bottom row shows the ACC for each
model and variable. In all cases, the FCN model outperforms IFS.

###  3.3 Verification against observations

####  3.3.1 Initializing the DDWP

To validate the FCN model in a real-world setting, it needs to be initialized
using operational analysis data and not reanalysis data from ERA5. The ideal
test would be to initialize FCN with the same analysis used to initialize IFS,
and while such data is available from the TIGGE archive, very long download
times made this impracticable. To approximate the lower quality of an
operational start state vs. a reanalysis start state, we construct start
states using the ensemble mean data from ERA5 instead of the HRES data from
ERA5. The ensemble mean data in ERA5 is produced at a lower resolution (63km)
than the HRES data
(31km)999https://confluence.ecmwf.int/display/CKB/ERA5%3A+data+documentation#heading-
Spatialgrid, thus making it a coarser reanalysis compared to HRES. In our
experiments, we saw a noticeable drop in the quality of predictions made using
ensemble mean start states compared to HRES start states, mimicking the effect
of having lower quality start states in operational settings.

####  3.3.2 Evaluating gridded forecasts on point observations

FCN and IFS forecasts are produced at a 0.25° grid resolution, whereas real-
world observations in MADIS2020 are at specific locations that may not
coincide with grid points. We use bilinear interpolation to obtain forecasts
at specific points from gridded forecasts, using the functionality provided by
the Xarray software package by Hoyer and Hamman, (2017). While bilinear
interpolation is not a perfect way of interpolating a gridded forecast to a
specific location, we note that the same limitations apply to FCN and IFS as
they are both produced on identical grids. Hence, bilinear interpolation
provides an unbiased way to evaluate both DDWPs and NWPs on point observation
data. We note that any improved interpolation methods, e.g., ones that account
for latitude distortion in the grid, would equally benefit NWPs and DDWPs.
Once gridded forecasts are converted to point forecasts, we can compute the
RMSE for each variable (wind speed, temperature, dewpoint) for each forecast
and region.

Figure 4 shows the performance of FCN and IFS for all variables different
regions, for a 48 hour forecast. It is immediately evident that both FCN and
IFS perform nearly identically in all regions and for all lead times, which
contradicts our expectation that FCN should outperform IFS, given how well it
does when tested against ERA5. This result underscores the need for testing
DDWPs in realistic settings as they are deployed in operational forecasts.

Figure 4: RMSE for different variables, comparing FCN and IFS against
observations for different regions of the world. Each row is one region, and
the columns represent the RMSE for wind speed, temperature, and dewpoint
respectively.

##  4 Discussion

DDWPs have made significant improvements in a short time, and show great
promise for use in operational weather forecasting. So far, DDWPs have been
evaluated using reanalysis data as ground truth, which effectively tests DDWPs
in a high-quality simulation. While valuable, such validation is incomplete
and more thorough evaluation is needed under settings that replicate their
operational use. While by no means exhaustive, the use of in-situ observations
from MADIS2020 provides one such way to evaluate DDWPs on a meaningful task,
with the benefit that they can be used to fairly compare all DDWPs and NWPs.
Compared to ERA5 data, real-world observations better represent the conditions
experienced by end-users of weather forecasts, making such an evaluation
benchmark particularly relevant when deploying DDWPs in operational forecasts.
Indeed, when comparing one well-known DDWP model (FourCastNet) against a
baseline NWP (the IFS model), we see that any performance advantage shown when
testing against ERA5 disappears when testing against real-world observations.

While further work is needed to identify why DDWPs may not perform well
against real-world data, one reason for the difference in performance comes
from the use of less accurate start states in operational settings. This may
be addressed by fine-tuning DDWPs on operational start states once they are
sufficiently trained on ERA5 data. The ERA5 dataset itself can be further
examined and uesd more effectively in training DDWPs. The uncertainty of ERA5
reanalysis varies by time and location and tends to be higher in the global
South, which can in turn influence the quality of any models trained on ERA5
data. One effect of such variation in data quality is seen in the regional
variation of model performance, and ML practitioners must take care to ensure
their evaluations do not focus purely on the data-rich parts of the world. The
ensemble "spread" available in ERA5 conveys a specific type of uncertainty
estimate, and may be used to improve the training of DDWPs by adjusting how
much to weight data from different locations and times. The uncertainty is
expected to also be higher the farther back in time we go (due to sparser
observations), and accounting for uncertainty may bias DDWPs towards recent
data and help reduce any training-inference skew caused by weather patterns
changing due to climate change.

As weather forecasts are often consumed at specific locations of high interest
like airports, interpolation from a grid to a specific latitude/longitude is a
necessary step. Bilinear interpolation is a common technique, but ML could be
applied to perform data-driven interpolation, taking into account sub-grid
features like topography, terrain, land use, etc., for which in-situ
observations are particularly useful as training and evaluation data. Improved
interpolation techniques could once again exploit observations from data-rich
regions for the benefit of data-poor regions. Such techniques can be compared
against techniques like Model Output Statistics (MOS), used to provide site-
specific guidance derived from the gridded GFS forecast issued by NOAA (Glahn
and Lowry, (1972)). Where a long history of observations is available, data-
driven debiasing can be used to remove systemic biases present in weather
models. The definition and use of appropriate metrics is necessary to build
trust amongst stakeholders and properly evaluate the quality of DDWPs. While
deterministic metrics like RMSE are convenient to report, they are not
appropriate for all variables (e.g. precipitation) and for longer lead times.
For longer lead times, it may be more appropriate to use probabilistic metrics
as well as NWP ensembles as a more relevant baseline.

The recent activity and progress in researching ML-based forecasting leaves
little doubt that data-driven weather prediction models will have a meaningful
role to play in operational forecasting. While much of the recent work has
been focused on learning an NWP approximation in an ERA5 simulation, imagining
a DDWP in an operational setting raises many more interesting questions (and
avenues for research) around their actual use. Our work should be seen as the
start of a conversation around answering such questions to realize the full
potential of ML-based weather forecasting.

## Acknowledgments and Disclosure of Funding

The authors would like to thank the European Center for Medium Weather
Forecasting (ECMWF) for making available the ERA5 dataset and the TIGGE
archived NWP forecasts, used as IFS in the study. Likewise, we would like to
thank NOAA for making publicly available such a large corpus of in-situ
observations from the MADIS program.

## References

  * Bi et al., (2022) Bi, K., Xie, L., Zhang, H., Chen, X., Gu, X., and Tian, Q. (2022).  Pangu-weather: A 3D high-resolution model for fast and accurate global weather forecast.  arXiv preprint arXiv:2211.02556.
  * Bi et al., (2023) Bi, K., Xie, L., Zhang, H., Chen, X., Gu, X., and Tian, Q. (2023).  Accurate medium-range global weather forecasting with 3d neural networks.  Nature, pages 1–6.
  * Bowler et al., (2015) Bowler, N., Cullen, M., and Piccolo, C. (2015).  Verification against perturbed analyses and observations.  Nonlinear Processes in Geophysics, 22(4):403–411.
  * Bowler, (2008) Bowler, N. E. (2008).  Accounting for the effect of observation errors on verification of MOGREPS.  Meteorological Applications: A journal of forecasting, practical applications, training techniques and modelling, 15(1):199–205.
  * Glahn and Lowry, (1972) Glahn, H. R. and Lowry, D. A. (1972).  The use of model output statistics (MOS) in objective weather forecasting.  Journal of Applied Meteorology and Climatology, 11(8):1203–1211.
  * Hersbach et al., (2020) Hersbach, H., Bell, B., Berrisford, P., Hirahara, S., Horányi, A., Muñoz-Sabater, J., Nicolas, J., Peubey, C., Radu, R., Schepers, D., et al. (2020).  The ERA5 global reanalysis.  Quarterly Journal of the Royal Meteorological Society, 146(730):1999–2049.
  * Hoyer and Hamman, (2017) Hoyer, S. and Hamman, J. (2017).  xarray: ND labeled arrays and datasets in python.  Journal of Open Research Software, 5(1).
  * Jiao et al., (2021) Jiao, D., Xu, N., Yang, F., and Xu, K. (2021).  Evaluation of spatial-temporal variation performance of ERA5 precipitation data in china.  Scientific Reports, 11(1):1–13.
  * Kadian et al., (2020) Kadian, A., Truong, J., Gokaslan, A., Clegg, A., Wijmans, E., Lee, S., Savva, M., Chernova, S., and Batra, D. (2020).  Sim2Real Predictivity: Does evaluation in simulation predict real-world performance?  IEEE Robotics and Automation Letters, 5(4):6670–6677.
  * Lam et al., (2022) Lam, R., Sanchez-Gonzalez, A., Willson, M., Wirnsberger, P., Fortunato, M., Pritzel, A., Ravuri, S., Ewalds, T., Alet, F., Eaton-Rosen, Z., Hu, W., Merose, A., Hoyer, S., Holland, G., Stott, J., Vinyals, O., Mohamed, S., and Battaglia, P. (2022).  Graphcast: Learning skillful medium-range global weather forecasting.
  * Olauson, (2018) Olauson, J. (2018).  ERA5: The new champion of wind power modelling?  Renewable energy, 126:322–331.
  * Pathak et al., (2022) Pathak, J., Subramanian, S., Harrington, P., Raja, S., Chattopadhyay, A., Mardani, M., Kurth, T., Hall, D., Li, Z., Azizzadenesheli, K., Hassanzadeh, P., Kashinath, K., and Anandkumar, A. (2022).  FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators.
  * Perez et al., (2022) Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., and Irving, G. (2022).  Red teaming language models with language models.
  * Tamkin et al., (2021) Tamkin, A., Brundage, M., Clark, J., and Ganguli, D. (2021).  Understanding the capabilities, limitations, and societal impact of large language models.  arXiv preprint arXiv:2102.02503.
  * Wilkinson et al., (2016) Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2016).  The FAIR guiding principles for scientific data management and stewardship.  Scientific data, 3(1):1–9.

##  5 Appendix for MADIS2020 Dataset

In this section we will provide a brief overview of the dataset directory,
code snippets to access and manipulate the data and provide links for open-
source implementations for the model results with the goal to improve
reproducibility.

## Dataset directory structure

The MADIS2020 dataset contains observations from two separate networks, namely

  1. 1.

METAR: Primarily a format for reporting weather information that is
predominantly used in aviation and provided by airports.

  2. 2.

MESONET: These are data obtained from private weather stations that measure
and collect data that are important to the purposes of providing a
comprehensive meteorological understanding.

The directory follows the following structure

… .1 madis2020. .2 month. .3 metar. .4 zarr datafile. .3 mesonet. .4 zarr
datafile.

The MADIS observation data is stored in zarr files, which can be opened using
the open-source Xarray library Hoyer and Hamman, (2017). The month range is
between 1 (for January) to 12 (for December). Each zarr data file comprises
observation for a single hour, therefore totaling 24 files for the same date.
The observations for a specific hour YYYY-MM-DD HH:00 are stored in the file
path MADIS2020/month/(mesonet|metar)/YYYYMMDD_HH00.zarr.

Each zarr file has the variables (as shown in Table 1) with quality control
flags (more on NOAA website
101010https://madis.ncep.noaa.gov/madis_sfc_qc_notes.shtml), however it is to
be noted that the accumulated precipitation variables are only available for
METAR.

Lastly, the dataset contains two variables latitude and longitude that specify
where the observation was taken.

## Example codes

In this section we will share some basic code snippets to access and
manipulate the data for evaluation, assuming the data is in a standard format
such as NetCDF or Zarr.

### Code to access data

⬇

1# Import python library

2import xarray as xr

3

4# Load file path

5FILE_PATH =
’<DIRECTORY_PATH>/madis2020/<month>/<metar|mesonet>/YYYYMMDD_HH00.zarr’

6

7# Access data from file

8hourly_data = xr.open_zarr(FILE_PATH)

9

10# Shows all variables including quality flags, latitude, and longitude

11weather_vars = hourly_data.data_vars

### Code to apply quality flags

The details regarding the QC flags have been discussed extensively in the main
paper. Here we provide a snippet of code examples to apply some flags. Further
details on the QC levels are available on the NOAA website
111111https://madis.ncep.noaa.gov/madis_sfc_qc_notes.shtml.

⬇

1# apply relevant quality flag

2valid_observation = ((hourly_data.windSpeedDD == b’S’) +
(hourly_data.windSpeedDD == b’V’))

3

4# code to ignore outliers

5validobs = valid_observation & (hourly_data.windSpeed < 50)

6

7# extract data

8speed_obs = hourly_data.windSpeed[validobs].load()

9lat_obs = hourly_data.latitude[validobs].load()

10

11# add longitude conversion if required in the following step

12lon_obs = (hourly_data.longitude[validobs].load()

This returns the arrays for wind observations (speed_obs) at relevant
locations (lat_obs and lon_obs). The model prediction at relevant locations
can then be used to compare against obervations.

### Comparing observations to model outputs

Given that ground observations are location-specific, the model outputs which
are often times in a gridded format, are required to interpolate from the
neighboring nodes to the relevant latitude/longitude in question. The reader
can use any standard package, for example, Xarray’s data array interpolation
121212https://docs.xarray.dev/en/stable/generated/xarray.DataArray.interp.html
for this purpose. We present a code snippet on how to evaluate one hour of
ERA5 data against MADIS observations, noting that similar procedures can be
used for any modeled data.

⬇

1# Import xarray library

2import xarray as xr

3import numpy as np

4

5# Step 1: Load the hourly data, and filter for valid temperature
observations.

6FILE_PATH =
’<DIRECTORY_PATH>/madis2020/<month>/<metar|mesonet>/YYYYMMDD_HH00.zarr’

7madis_data = xr.open_zarr(FILE_PATH)

8valid_observation = ((madis_data.temperatureDD == b’S’) +

9 (madis_data.temperatureDD == b’V’))

10valid_lats = madis_data.latitude[valid_observation]

11valid_lons = madis_data.longitude[valid_observation]

12valid_t2m = madis_data.temperature[valid_observation]

13

14# Step 2: Load the ERA5 data for temperature, for the same date and time.

15# The ERA5 data can be downloaded directly from ECMWF’s CDS API,

16# and a reference script to download data is available from

17# WeatherBench:

18# https://github.com/pangeo-data/WeatherBench/blob/master/src/download.py

19# In this example, the NetCDF file is expected to contain data for 24

20# hours of January 1, 2020. Such data can be downloaded by specifying

21# the following flags to the download script from WeatherBench:

22# \--variable=t2m \--level_type=single \--years 2020 \--month 1 \--day 1

23# Note that this would download data at a 0.703125 degree resolution.

24ERA5_PATH = ’<ERA5_DIRECTORY>/t2m_20200101.nc’

25era5_data = xr.open_dataset(ERA5_PATH)

26

27# Step 3: Interpolate the ERA5 data for hour 0 to the required
latitude/longitudes.

28# Note that the actual resolution of the data is encoded in the NetCDF file,

29# and we can exploit xarray’s functions to perform bilinear interpolation

30# for all desired points.

31era5_pred = era5_data.t2m[0].interp(latitude=xr.DataArray(valid_lats),

32 longitude=xr.DataArray(valid_lons))

33

34# Step 4: Compute the error between the observed temperature, and

35# the ERA5 modeled temperature.

36error = (valid_t2m.to_numpy().flatten()

37 - era5_pred.to_numpy().flatten())

38mse = np.mean(error**2)

## Bias in the Dataset

We conducted a study to verify any location-specific biase in the dataset and
divided the globe into specific regions, graphically shown in Figure 5 and the
latitude/longitude range is shown in Table 2.

Figure 5: Geographical extent of various regions used in evaluations. Table 2: The latitude and longitude extents of the various geographies used in the study Geographic Region | Longitude range | Latitude range
---|---|---
global |  $\mathtt{-180}$° to $\mathtt{180}$° |  $\mathtt{-90}$° to $\mathtt{90}$°
namer (North America) |  $\mathtt{-120}$° to $\mathtt{-75}$° |  $\mathtt{25}$° to $\mathtt{60}$°
europe (Europe) |  $\mathtt{-12.5}$° to $\mathtt{42.5}$° |  $\mathtt{35}$° to $\mathtt{75}$°
wasia (West Asia) |  $\mathtt{42.5}$° to $\mathtt{102.5}$° |  $\mathtt{0}$° to $\mathtt{60}$°
easia (East Asia) |  $\mathtt{102.5}$° to $\mathtt{150}$° |  $\mathtt{25}$° to $\mathtt{60}$°
seasia (South-East Asia) |  $\mathtt{95}$° to $\mathtt{130}$° |  $\mathtt{-12.5}$° to $\mathtt{25}$°
ausnz (Aus. & New Zealand) |  $\mathtt{110}$° to $\mathtt{180}$° |  $\mathtt{-48}$° to $\mathtt{-12.5}$°
samer (South America) |  $\mathtt{-80}$° to $\mathtt{-31}$° |  $\mathtt{-54}$° to $\mathtt{10}$°
nafr (Northern Africa) |  $\mathtt{-15}$° to $\mathtt{34}$° |  $\mathtt{4}$° to $\mathtt{36}$°
safr (Southern Africa) |  $\mathtt{-15}$° to $\mathtt{50}$° |  $\mathtt{-36}$° to $\mathtt{4}$°

In Figure 1 we plot the data distribution across various geographies and it
appears that the global south and the APAC region suffer disproportionately
with comparably fewer high-quality ground observations. This trend exacerbates
for variables such as precipitation which while only available for METAR, has
a huge discrepancy for non-North American geographies.

## Benchmarking

We evaluate three sources of weather data against real-world observations, -
the ERA5 reanalyses product from ECMWF, FourCastNet data driven model and IFS
Numerical Weather Prediction model made available by ECMWF. Full details of
the FourCastNet model implementation are discussed in Pathak et al., (2022)
and the original model implementation is publicly available
131313https://github.com/NVlabs/FourCastNet. For the IFS forecast, the TIGGE
archived NWP forecasts were used
141414https://www.ecmwf.int/en/research/projects/tigge. The ERA5 dataset
151515https://www.ecmwf.int/en/forecasts/dataset/ecmwf-reanalysis-v5 is also
made publicly available by ECMWF.

The all-weather verification in Figure 6 shows a close comparison between the
three sources of data with signs of uncorrelated errors between the data-
driven (FCN) and numerical weather prediction (IFS) model.

Figure 6: RMSE of ERA5, FCN, and IFS against observations for wind speeds,
temperature, dewpoint, and 6-hourly precipitation accumulation. Each column
represents a data source, and each row represents a variable. The x-axis
represents the time of day in UTC, and the y-axis represents the RMSE in
appropriate units. Note the different forecast depths for ERA5 vs. FCN and
IFS.