# Temporal Treasure Hunt: Content-based Time Series Retrieval System for
Discovering Insights

Chin-Chia Michael Yeh, Huiyuan Chen, Xin Dai, Yan Zheng, Yujie Fan, Vivian
Lai,
Junpeng Wang, Audrey Der∗, Zhongfang Zhuang, Liang Wang, Wei Zhang  Visa
Research, UC Riverside∗
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Time series data is ubiquitous across various domains such as finance,
healthcare, and manufacturing, but their properties can vary significantly
depending on the domain they originate from. The ability to perform Content-
based Time Series Retrieval (CTSR) is crucial for identifying unknown time
series examples. However, existing CTSR works typically focus on retrieving
time series from a single domain database, which can be inadequate if the user
does not know the source of the query time series. This limitation motivates
us to investigate the CTSR problem in a scenario where the database contains
time series from multiple domains. To facilitate this investigation, we
introduce a CTSR benchmark dataset that comprises time series data from a
variety of domains, such as motion, power demand, and traffic. This dataset is
sourced from a publicly available time series classification dataset archive,
making it easily accessible to researchers in the field. We compare several
popular methods for modeling and retrieving time series data using this
benchmark dataset. Additionally, we propose a novel distance learning model
that outperforms the existing methods. Overall, our study highlights the
importance of addressing the CTSR problem across multiple domains and provides
a useful benchmark dataset for future research.

###### Index Terms:

time series, information retrieval, neural net

##  I Introduction

Time series data is ubiquitous across various domains as almost all systems
change over time [1]. Financial indices, electrocardiograms, and pressure
sensors on manufacturing machines provide time-varying measurements of
distinct systems, among many other examples. This leads institutions to
commonly maintain collections of time series data from multiple sources,
requiring an effective Content-based Time Series Retrieval (CTSR) system for
users to efficiently solve their time series-related problems.

To understand how a CTSR system can assist users, let us consider a use case
as depicted in Fig. 1. Assume that the user acquires a time series from an
unknown source and wishes to determine what type of time series it is. To
solve this problem, the user queries a CTSR system with the time series, and
the system provides a ranked list of similar time series with relevant
metadata. For example, in our use case, the system returns a list where five
out of six similar time series are power consumption signatures for microwave
ovens. The user can then deduce that the unknown time series is likely a power
consumption signature for a microwave oven. Therefore, the CTSR system helps
the user to identify potential sources for the query time series.

Figure 1:  The use case for a Content-based Time Series Retrieval (CTSR)
system is illustrated in the example using the actual outputs generated by the
proposed Residual Network 2D (RN2D) model.

The primary challenge in addressing the CTSR problem is the distinct
characteristics exhibited by time series data from different sources. An
effective CTSR approach must be capable of distinguishing between diverse
concepts in various domains. Nevertheless, existing works in the field have
mainly focused on retrieval time series from single domain databases, with the
query and database originating from the same domain [2, 3, 1, 4, 5], thus
ignoring the challenge of retrieving time series from multiple domains.

This work contributes to the study of the CTSR problem in the following ways:

  * •

We present a detailed procedure for testing and comparing CTSR systems.

  * •

We benchmark six popular baseline methods that are based on commonly used
models for time series classification or general sequence modeling.

  * •

We propose a novel method called the Residual Network $2D$ (RN2D), which
demonstrates superior performance compared to the baseline methods with
statistical significance.

##  II Method

In this section, we will first present the baseline methods. Next, we will
introduce our proposed method and discuss its benefits by contrasting it with
the baselines. The CTSR problem is formulated as follows:

######  Problem Statement.

Given a set of time series
$\mathcal{X}=[\mathbf{x}_{1},\cdots,\mathbf{x}_{n}]$ and a query time series
$\mathbf{q}$, we aim to obtain a relevance score function $f(\cdot,\cdot)$
such that $f(\mathbf{x}_{i},\mathbf{q})>f(\mathbf{x}_{j},\mathbf{q})$ if
$\mathbf{x}_{i}$ is more relevant to $\mathbf{q}$ compared to
$\mathbf{x}_{j}$.

Note that the scoring function $f(\cdot,\cdot)$ can be either a predefined
similarity/distance function or a trainable function that is optimized using
the time series in $\mathcal{X}$ and their associated metadata.

The baseline methods we considered are:

  * •

Euclidean Distance (ED): We compute the Euclidean distance between the query
time series and each time series in the collection, and then order the
collection based on the computed distances. This is the simplest method for
solving the CTSR problem.

  * •

Dynamic Time Warping (DTW): This method is similar to the ED baseline, but it
uses the Dynamic Time Warping (DTW) distance instead. The DTW distance is
considered a simple but effective baseline for time series classification
problems [1, 6, 7], and therefore it is important to include this method in
our CTSR benchmark.

  * •

Long Short-Term Memory Network (LSTM): LSTM is one of the most popular types
of Recurrent Neural Networks (RNNs) used for modeling sequential data [8, 9,
10]. In this study, we optimize the LSTM models based on the Siamese network
architecture [11]. The input to the Siamese network is composed of two time
series. We begin by processing each time series using a $1D$ convolutional
layer to extract local features. We then feed the output of the convolutional
layer to a bi-directional LSTM model to obtain a hidden representation. The
hidden representation of the last time step is fed to a linear layer to get
the final representation of the input time series. We compute the relevance
score between the two input time series by applying the Euclidean distance
function to the two final representations (i.e., one from each input time
series).

  * •

Gated Recurrent Unit Network (GRU): The GRU is another popular RNN
architecture for modeling sequential data [12, 9, 10]. In our experiments, we
optimize the GRU models using the same Siamese network architecture as the
LSTM model. The only difference is that we replace the bi-directional LSTM
model with a bi-directional GRU model.

  * •

Transformer (TF): The transformer has emerged as a popular alternative to RNNs
for sequence modeling [13, 14, 10, 9, 15, 16]. To learn the hidden
representation of the input time series, we use the transformer encoder
introduced in [13]. In contrast to the previous methods that employed RNN-
based encoders/sub-networks within the Siamese network, we use transformer-
based encoders/sub-networks, where the transformer encoders are used to encode
the input time series and the relevance score between the two time series is
defined as the Euclidean distance between their final transformer outputs.

  * •

Residual Network 1D (RN1D): This network design is inspired by the success of
the residual network in computer vision [17, 18]. Instead of $2D$
convolutional layers, it uses $1D$ convolutional layers [17, 18]. The design
was first proposed in [18] and was shown to be one of the strongest
classifiers for time series classification in extensive evaluations conducted
by [19]. We optimize this model in a Siamese network architecture [11] in the
same way as the LSTM method.

Both the ED and DTW methods are parameter-free and do not require a training
phase. However, the DTW method is considered more effective for time series
data, as it considers all alignments between the input time series [1]. The
computation of the DTW distance can be described by a two-stage process. In
the first stage, a pairwise distance matrix $D\in\mathbb{R}^{w\times h}$ is
computed from the input time series $\mathbf{a}=[a_{1},\cdots,a_{w}]$ (where
$w$ is the length of $\mathbf{a}$) and $\mathbf{b}=[b_{1},\cdots,b_{h}]$
(where $h$ is the length of $\mathbf{b}$) as $D[i,j]=|a_{i}-b_{j}|$. In the
second stage, a fixed recursion function is applied to $D$, where each element
in $D$ is updated as $D[i,j]\leftarrow
D[i,j]+\textsc{min}(D[i-1,j],D[i,j-1],D[i-1,j-1])$. As a result, the DTW
method can be understood as applying a predefined function to the pairwise
distance matrix between the input time series.

The other four baseline methods utilize the distance learning framework of the
Siamese network. They utilize high-capacity111We use the term capacity to
describe the expressiveness [20] of a model. neural network models (LSTM, GRU,
TF, and RN1D) to learn the hidden representation of the input time series,
which is then used to compute the distance between two time series. However,
since the neural network models process the two input time series
independently, it is unlikely for the model to learn the distance based on
different alignments between a pair of time series.

###  II-A Residual Network 2D

To leverage the benefits of both alignment information and neural networks, we
propose the Residual Network 2D (RN2D) model, as illustrated in Fig. 2. The
RN2D model capitalizes on the rich alignment information of the pair-wise
distance matrix, similar to the DTW method. However, instead of using a fixed
function, the proposed method employs a high-capacity neural network to model
the distance computation. In other words, the RN2D model utilizes an
expressive model like the four neural network baselines. Fig. 2.a and Fig. 2.b
depict the building block and overall model design, respectively.

Figure 2:  The building block and network designs for the proposed Residual
Network 2D (RN2D) model are shown in Fig. 2.a and Fig. 2.b, respectively.

The RN2D model is designed with inspiration from deep residual networks used
in computer vision [17]. We employ the bottleneck building block design from
[17] as shown in Fig. 2.a. Given the input dimension $n_{in}$, bottleneck
dimension $n_{neck}$, and output dimension $n_{out}$, we first project the
input tensor $X_{\text{in}}\in\mathbb{R}^{w\times h\times n_{in}}$ to
$\mathbb{R}^{w\times h\times n_{neck}}$ space using a $1\times 1$
convolutional layer. We then apply a ReLU layer to the intermediate
representation, followed by a $3\times 3$ convolutional layer with a stride of
two, which reduces the size further to $\mathbb{R}^{w/2\times h/2\times
n_{neck}}$. Next, we use another ReLU layer to process the intermediate
representation, followed by a $1\times 1$ convolutional layer to project it to
$\mathbb{R}^{w/2\times h/2\times n_{out}}$ space. We refer to the output of
this layer as $X_{\text{out}}$. Since the size of $X_{\text{in}}$ does not
match that of $X_{\text{out}}$, we first process $X_{\text{in}}$ with a
$1\times 1$ convolutional layer before adding it to $X_{\text{out}}$ for the
skip connection. Finally, we process the merged representation with a ReLU and
exit the building block. If the input is in $\mathbb{R}^{w\times h\times
n_{in}}$ space, the output will be in $\mathbb{R}^{w/2\times h/2\times
n_{out}}$ space.

Our overall network design is also inspired by the design proposed in [17], as
depicted in Fig. 2.b. Given two input time series
$\mathbf{a}=[a_{1},\cdots,a_{w}]$ and $\mathbf{b}=[b_{1},\cdots,b_{h}]$, we
first compute the pairwise distance matrix $D\in\mathbb{R}^{w\times h}$
similar to the DTW method. The $(i,j)$-th position of $D$ is computed as
$D[i,j]=|a_{i}-b_{j}|$. Before applying the convolutional layer, we reshape
$D$ to $w\times h\times 1$ by adding an extra dimension. Next, a $7\times 7$
convolutional layer with a step size of two projects $D$ to
$\mathbb{R}^{w/2\times h/2\times 64}$ space. After a ReLU layer, the
intermediate representation passes through eight building blocks with a
$64{\to}16{\to}64$ setting. Subsequently, we apply a global average pooling
layer to reduce the spatial dimension, and the output of this layer is a
64-dimensional vector. Lastly, a linear layer projects the vector to a scalar
number, which is the relevance score between the two input time series.

##  III Dataset and Experiment Setup

We create the CTSR benchmark dataset from the UCR Archive [7], a collection of
128 time series classification datasets from various domains like motion,
power demand, and traffic. It is one of the most popular dataset archives for
benchmarking time series classification algorithms [6, 18, 19]. We convert the
UCR Archive [7] to a CTSR benchmark dataset following the steps outlined in
[21]. After the conversion steps, we end up with 136,377 time series in the
training set, 17,005 time series in the test set, and 17,005 time series in
the validation set. To evaluate the performance, we calculate standard
information retrieval metrics such as precision at $k$ (Prec@$k$), average
precision at $k$ (AP@$k$), and normalized discounted cumulative gain at $k$
(NDCG@$k$) for each query.

We utilized the Bayesian Personalized Ranking (BPR) loss [22] as the loss
function since the CTSR problem is a Learning to Rank problem. The loss
function is defined as follows for a batch of training data
$\mathcal{B}=[B_{0},\cdots,B_{m}]$:
$-\sum_{(\mathbf{t},\mathbf{t}_{+},\mathbf{t}_{-})\in\mathcal{B}}\log\sigma(f_{\bm{\theta}}(\mathbf{t},\mathbf{t}_{+})-f_{\bm{\theta}}(\mathbf{t},\mathbf{t}_{-}))$
where $m$ is the batch size, $\sigma(\cdot)$ is the sigmoid function, and
$f_{\bm{\theta}}(\cdot,\cdot)$ is the model. Each sample in the training data
is a tuple that consists of the query (or anchor), the positive, and the
negative time series, i.e.,
$B_{i}=(\mathbf{t},\mathbf{t}_{+},\mathbf{t}_{-})$. We used the average
NDCG@10 on validation data to select the model for testing. Further details
regarding the source code, hyper-parameter settings, and other information can
be found in [21].

##  IV Experiment Result

The performance of the different methods is computed by averaging the
performance measurements at $k=10$ across the 17,005 test queries, and the
results are presented in Table I. The table provides an easy way to compare
the performance of different methods using various metrics. To evaluate the
significance of differences between methods, we perform two-sample t-tests
with $\alpha=0.05$.

TABLE I: The proposed method outperforms the others in all performance measurements.  Method | PREC@10 | AP@10 | NDCG@10
---|---|---|---
ED | 0.7316 | 0.7655 | 0.7499
DTW | 0.8562 | 0.8792 | 0.8693
LSTM | 0.9205 | 0.9277 | 0.9260
GRU | 0.9108 | 0.9185 | 0.9166
TF | 0.9146 | 0.9212 | 0.9199
RN1D | 0.9086 | 0.9164 | 0.9146
RN2D | 0.9266 | 0.9342 | 0.9325

We first discuss the performance of the two non-neural network baselines,
namely ED and DTW. The DTW method significantly outperforms the ED method in
all three performance measurements, indicating that the use of alignment
information is beneficial for the CTSR problem. A similar observation has also
been made for the time series classification problem [6]. When considering the
four neural network baselines (LSTM, GRU, TF, and RN1D), they all
significantly outperform the DTW method, highlighting the importance of using
high-capacity models for the CTSR problem. One possible reason for this is
that the CTSR dataset comprises time series from diverse domains [7]. Higher
capacity models are required to capture the diverse patterns within the data.
Among the four methods, LSTM performs significantly better than the second-
best method in all three performance measurements. The proposed method, RN2D,
which is a high-capacity model that utilises alignment information,
significantly outperforms all other methods based on the results of t-tests.

Fig. 3 presents the performance measurements of different methods at various
values of $k$ (ranging from 5 to 15). This analysis allows us to ensure that
the superior performance of the proposed method is not limited to a specific
value of $k$. The performance of ED and DTW is notably lower than the other
methods, so we have excluded these two methods in the figures for improved
readability. The RN2D method consistently achieves the highest performance
across different values of $k$ for all three performance metrics. The order of
methods from the best to the worst is: RN2D, LSTM, TF, GRU, and RN1D, which
aligns with the findings in Table I.

Figure 3:  The proposed method outperforms the others with different settings
of $k$ for each performance measure.

We present the results of the top eight retrieved time series using different
methods in Fig. 4, where we sample two queries with different complexities
from the test dataset. The simpler query contains a single cycle of a pattern,
while the complex query contains periodic signals, which typically require
shift invariant distance measures [23] to correctly find the relevant items.
This figure shows the CTSR problem is challenging, as even irrelevant time
series are visually similar to the query.

Figure 4:  The top eight items of a given query time series were retrieved by
different methods. The retrieved time series is colored red if it is relevant
and black if it is irrelevant.

We make the following observations from the retrieved time series of different
methods. The ED method struggles with the more complex query because it is
unable to align the query to relevant time series. The DTW method performs
better than the ED method on the complex query, but the freedom of alignment
hurts its performance on the simple query. The four neural network baselines
(i.e., LSTM, GRU, TF, and RN1D) work better than both ED and DTW methods when
considering both queries. However, none of these baselines are able to
outperform the proposed RN2D method, which reliably retrieves relevant items.
Therefore, this figure demonstrates that the proposed RN2D method outperforms
other methods by effectively leveraging both the high-complexity model and
rich alignment information, enabling it to successfully handle both simple and
complex queries.

##  V Conclusion

In this paper, we investigated the Content-based Time Series Retrieval (CTSR)
problem and tested six baseline methods (ED, DTW, LSTM, GRU, TF, and RN1D),
and proposed a novel method, called RN2D. Our proposed method outperformed the
baselines with statistical significance. For future work, we plan to focus on
improving the computational efficiency of the proposed model by exploring
techniques discussed in [24, 25, 26, 27, 28, 29]. We could also improve the
effectiveness of the CTSR system by considering pretraining techniques [30,
31].

## References

  * [1] T. Rakthanmanon, B. Campana, A. Mueen, G. Batista, B. Westover, Q. Zhu, J. Zakaria, and E. Keogh, “Searching and mining trillions of time series subsequences under dynamic time warping,” in _Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining_ , 2012, pp. 262–270.
  * [2] M. M. Muhammad Fuad and P.-F. Marteau, “Multi-resolution approach to time series retrieval,” in _Proceedings of the Fourteenth International Database Engineering & Applications Symposium_, 2010, pp. 136–142.
  * [3] P. Esling and C. Agon, “Time-series data mining,” _ACM Computing Surveys (CSUR)_ , vol. 45, no. 1, pp. 1–34, 2012.
  * [4] D. Song, N. Xia, W. Cheng, H. Chen, and D. Tao, “Deep r-th root of rank supervised joint binary embedding for multivariate time series retrieval,” in _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2018, pp. 2229–2238.
  * [5] D. Zhu, D. Song, Y. Chen, C. Lumezanu, W. Cheng, B. Zong, J. Ni, T. Mizoguchi, T. Yang, and H. Chen, “Deep unsupervised binary coding networks for multivariate time series retrieval,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 02, 2020, pp. 1403–1411.
  * [6] A. Bagnall, J. Lines, A. Bostrom, J. Large, and E. Keogh, “The great time series classification bake off: a review and experimental evaluation of recent algorithmic advances,” _Data mining and knowledge discovery_ , vol. 31, no. 3, pp. 606–660, 2017.
  * [7] H. A. Dau, A. Bagnall, K. Kamgar, C.-C. M. Yeh, Y. Zhu, S. Gharghabi, C. A. Ratanamahatana, and E. Keogh, “The ucr time series archive,” _IEEE/CAA Journal of Automatica Sinica_ , vol. 6, no. 6, pp. 1293–1305, 2019.
  * [8] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [9] B. Lim and S. Zohren, “Time-series forecasting with deep learning: a survey,” _Philosophical Transactions of the Royal Society A_ , vol. 379, no. 2194, p. 20200209, 2021.
  * [10] H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang, “Informer: Beyond efficient transformer for long sequence time-series forecasting,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 12, 2021, pp. 11 106–11 115.
  * [11] D. Chicco, “Siamese neural networks: An overview,” _Artificial Neural Networks_ , pp. 73–94, 2021.
  * [12] K. Cho, B. Van Merriënboer, D. Bahdanau, and Y. Bengio, “On the properties of neural machine translation: Encoder-decoder approaches,” _arXiv preprint arXiv:1409.1259_ , 2014.
  * [13] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [14] S. Li, X. Jin, Y. Xuan, X. Zhou, W. Chen, Y.-X. Wang, and X. Yan, “Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [15] H. Chen, Y. Lin, M. Pan, L. Wang, C.-C. M. Yeh, X. Li, Y. Zheng, F. Wang, and H. Yang, “Denoising self-attentive sequential recommendation,” in _Proceedings of the 16th ACM Conference on Recommender Systems_ , 2022, pp. 92–101.
  * [16] C.-C. M. Yeh, H. Chen, Y. Fan, X. Dai, Y. Zheng, V. Lai, J. Wang, Z. Zhuang, L. Wang, W. Zhang, and E. K. Keogh, “Ego-network transformer for subsequence classification in time series data,” in _2023 IEEE International Conference on Big Data (Big Data)_. IEEE, 2023.
  * [17] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [18] Z. Wang, W. Yan, and T. Oates, “Time series classification from scratch with deep neural networks: A strong baseline,” in _2017 International joint conference on neural networks (IJCNN)_. IEEE, 2017, pp. 1578–1585.
  * [19] H. Ismail Fawaz, G. Forestier, J. Weber, L. Idoumghar, and P.-A. Muller, “Deep learning for time series classification: a review,” _Data mining and knowledge discovery_ , vol. 33, no. 4, pp. 917–963, 2019.
  * [20] Z. Lu, H. Pu, F. Wang, Z. Hu, and L. Wang, “The expressive power of neural networks: A view from the width,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [21] The Author(s), “Supplementary materials,” 2022, https://sites.google.com/view/ctsr.
  * [22] S. Rendle, C. Freudenthaler, Z. Gantner, and L. Schmidt-Thieme, “BPR: Bayesian personalized ranking from implicit feedback,” in _Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence_ , 2009, pp. 452–461.
  * [23] J. Paparrizos and L. Gravano, “k-shape: Efficient and accurate clustering of time series,” in _Proceedings of the 2015 ACM SIGMOD international conference on management of data_ , 2015, pp. 1855–1870.
  * [24] J. Wang, W. Liu, S. Kumar, and S.-F. Chang, “Learning to hash for indexing big data—a survey,” _Proceedings of the IEEE_ , vol. 104, no. 1, pp. 34–57, 2015.
  * [25] J. Wang, T. Zhang, N. Sebe, H. T. Shen _et al._ , “A survey on learning to hash,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 40, no. 4, pp. 769–790, 2017.
  * [26] C.-C. M. Yeh, M. Gu, Y. Zheng, H. Chen, J. Ebrahimi, Z. Zhuang, J. Wang, L. Wang, and W. Zhang, “Embedding compression with hashing for efficient representation learning in large-scale graph,” in _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ , 2022, pp. 4391–4401.
  * [27] H. Chen, X. Li, K. Zhou, X. Hu, C.-C. M. Yeh, Y. Zheng, and H. Yang, “Tinykg: Memory-efficient training framework for knowledge graph neural recommender systems,” in _Proceedings of the 16th ACM Conference on Recommender Systems_ , 2022, pp. 257–267.
  * [28] A. Andoni and D. Beaglehole, “Learning to hash robustly, guaranteed,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 599–618.
  * [29] C.-C. M. Yeh, H. Chen, X. Dai, Y. Zheng, J. Wang, V. Lai, Y. Fan, A. Der, Z. Zhuang, L. Wang _et al._ , “An efficient content-based time series retrieval system,” in _Proceedings of the 32nd ACM International Conference on Information and Knowledge Management_ , 2023, pp. 4909–4915.
  * [30] Q. Ma, Z. Liu, Z. Zheng, Z. Huang, S. Zhu, Z. Yu, and J. T. Kwok, “A survey on time-series pre-trained models,” _arXiv preprint arXiv:2305.10716_ , 2023.
  * [31] C.-C. M. Yeh, X. Dai, H. Chen, Y. Zheng, Y. Fan, A. Der, V. Lai, Z. Zhuang, J. Wang, L. Wang _et al._ , “Toward a foundation model for time series data,” in _Proceedings of the 32nd ACM International Conference on Information and Knowledge Management_ , 2023, pp. 4400–4404.


# Training neural mapping schemes for satellite altimetry with simulation data

###### Abstract

Satellite altimetry combined with data assimilation and optimal interpolation
schemes have deeply renewed our ability to monitor sea surface dynamics.
Recently, deep learning (DL) schemes have emerged as appealing solutions to
address space-time interpolation problems. The scarcity of real altimetry
dataset, in terms of space-time coverage of the sea surface, however impedes
the training of state-of-the-art neural schemes on real-world case-studies.
Here, we leverage both simulations of ocean dynamics and satellite altimeters
to train simulation-based neural mapping schemes for the sea surface height
and demonstrate their performance for real altimetry datasets. We analyze
further how the ocean simulation dataset used during the training phase
impacts this performance. This experimental analysis covers both the
resolution from eddy-present configurations to eddy-rich ones, forced
simulations vs. reanalyses using data assimilation and tide-free vs. tide-
resolving simulations. Our benchmarking framework focuses on a Gulf Stream
region for a realistic 5-altimeter constellation using NEMO ocean simulations
and 4DVarNet mapping schemes. All simulation-based 4DVarNets outperform the
operational observation-driven and reanalysis products, namely DUACS and
GLORYS. The more realistic the ocean simulation dataset used during the
training phase, the better the mapping. The best 4DVarNet mapping was trained
from an eddy-rich and tide-free simulation datasets. It improves the resolved
longitudinal scale from 151 kilometers for DUACS and 241 kilometers for GLORYS
to 98 kilometers and reduces the root mean squared error (RMSE) by 23% and
61%. These results open research avenues for new synergies between ocean
modelling and ocean observation using learning-based approaches.

Journal of Advances in Modeling Earth Systems (JAMES)

IMT Atlantique, Lab-STICC, Brest, France Université Grenoble Alpes, CNRS, IRD,
Grenoble, France Datlas, Grenoble, France INRIA Team ODYSSEY Quentin
<EMAIL_ADDRESS>
We propose to train neural mapping schemes for real altimeter data from ocean
simulation data.

The trained neural schemes significantly outperform the operational mapping of
real altimetry data for a Gulf Stream case-study.

More realistic simulation datasets improve the performance of the trained
neural mapping with a 20% improvement in the spatial scales.

## Plain Language Summary

For an artificial intelligence (AI) to learn, one need to describe a task
using data and an evaluation procedure. Here we aim at constructing images
related to the ocean surface currents. The satellite data we use provide
images of the ocean surface with a lot of missing data (around 95% of missing
pixels for a given day), and we aim at finding the values of the missing
pixels. Because we don’t know the full image, it is challenging to train an AI
on this task using only the satellite data. However, today’s physical
knowledge makes it possible to numerically simulate oceans on big computers.
For these simulated oceans, we have access to the gap-free image, so we can
train AI models by first hiding some pixels and checking if the model fill the
gaps with the correct values. Here, we explore under which conditions AIs
trained on simulated oceans are useful for the real ocean. We show that
today’s simulated oceans work well for training an AI on this task and that
training on more realistic simulated oceans improve the performance of the AI!

##  1 Introduction

Satellite altimeters have brought a great leap forward in the observation of
sea surface height on a global scale since the 80’s.

Altimetry data have greatly contributed to the monitoring and understanding of
key processes such as the sea-level rise and the role of mesoscale dynamics.
The scarce and irregular sampling of the measurements presents a challenge for
training deep neural networks. The retrieval of mesoscale-to-submesoscale sea
surface dynamics for horizontal scales smaller than 150 km however remains a
challenge for operational systems based on optimal interpolation [Taburet .
(2019)] and data assimilation [Lellouche . (2021)] schemes. This has motivated
a wealth of research to develop novel mapping schemes [Ballarotta . (2020),
Ubelmann . (2021), Guillou . (2021)].

In this context, data-driven and learning-based approaches [Alvera Azcarate .
(2005), Barth . (2022), Lguensat . (2017), Fablet, Amar . (2021), Martin .
(2023)] appear as appealing alternatives to make the most of the available
observation and simulation datasets. Especially, Observing System Simulation
Experiments (OSSE) have stressed the potential of neural schemes trained
through supervised learning for the mapping of satellite-derived altimetry
data [Fablet, Amar . (2021), Beauchamp . (2023)]. Their applicability to real
datasets has yet to be assessed and recent studies have rather explored
learning strategies from real gappy multi-year altimetry datasets [Martin .
(2023)]. Despite promising results, schemes trained with unsupervised
strategies do not reach the relative improvement of the operational processing
suggested by OSSE-based studies.

Here, we go beyond using OSSEs as benchmarking-only testbeds. We explore their
use for the training of neural mapping schemes and address the space-time
interpolation of real satellite altimetry observations. Through numerical
experiments on a Gulf Stream case-study with a 5-nadir altimeter
constellation, our main contributions are three-fold. We demonstrate the
relevance of the simulation-based learning of neural mapping schemes and their
generalization performance for real nadir altimetry data. We benchmark the
proposed approach with state-of-the-art operational products as well as neural
schemes trained from real altimetry datasets. We also assess how the
characteristics of the training datasets, especially in terms of resolved
ocean processes, drives the mapping performance. To ensure the reproducibility
of our results, our code is made available through an open source license
along with the considered datasets and the trained models [Febvre (2023)].

The content of this paper is organized as follows. Section 2 offers background
information on related work, Section 3 presents our method, Section 4 reports
our numerical experiments, and Section 5 elaborates on our main contributions.

##  2 Background

###  2.1 Gridded satellite altimetry products

The ability to produce gridded maps from scattered along-track nadir altimeter
measurements of sea surface height is key to the exploitation of altimeter
data in operational services and science studies [Abdalla . (2021)]. As
detailed below, we can distinguish three categories of approaches to produce
such maps: reanalysis products [Lellouche . (2021)] using data assimilation
schemes, observation-based products [Taburet . (2019)] and learning-based
approaches [Fablet, Amar . (2021)].

Reanalysis products such as the GLORYS12 reanalysis [Lellouche . (2021)]
leverage the full expressiveness of state-of-the-art ocean models. They aim at
retrieving ocean state trajectories close to observed quantities through data
assimilation methods including among others Kalman filters and variational
schemes [Carrassi . (2018)]. Such reanalyses usually exploit satellite-derived
and in situ data sources. For instance, GLORYS12 reanalysis assimilates
satellite altimetry data, but also satellite-derived observations of the sea
surface temperature, sea-ice concentration as well as in situ ARGO data [Wong
. (2020)].

The second category involves observation-based products. In contrast to
reanalyses, they only rely on altimetry data and address a space-time
interpolation problem. They usually rely on simplifying assumptions on sea
surface dynamics. In this category, optimal-interpolation-based product DUACS
(Data Unification and Altimeter Combination System) [Taburet . (2019)]
exploits a covariance-based prior, while recent studies involve quasi-
geostrophic dynamics to guide the interpolation scheme [Guillou . (2021),
Ballarotta . (2020)].

Data-driven and learning-based approaches form a third category of SSH mapping
schemes. Similarly to observation-based methods, they are framed as
interpolation schemes. Especially deep learning schemes have gained some
attention. Recent studies have explored different neural architectures both
for real and OSSE altimetry datasets [Archambault . (2023), Beauchamp .
(2021), Martin . (2023)]. These studies investigate both different training
strategies as well as different neural architectures from off-the-shelf
computer vision ones such as convolutional LSTMs and UNets [Ronneberger .
(2015)] to data-assimilation-inspired ones [Beauchamp . (2021), Fablet,
Chapron . (2021)].

###  2.2 Ocean Modeling and OSSE

Advances in modeling and simulating ocean physics have largely contributed to
a better understanding of the processes involved in the earth system and to
the development of operational oceanography [Barnier . (2006), Ajayi .
(2020)]. High-resolution simulations used in Observing System Simulation
Experiments (OSSE) also provide a great test-bed for the design and evaluation
of new of ocean observation systems [Benkiran . (2021)]. The availability of
numerical model outputs enables the computation of interpretable metrics
directly on the quantities of interest. This avoids challenges met when
working solely with observation data that may be incomplete, noisy or
indirectly related to the desired quantity. For example, in the case of the
recently launched SWOT mission, OSSEs combined ocean and instrument
simulations to address calibration issues and interpolation performance for
SWOT altimetry data [Dibarboure . (2022)]. Such OSSEs have also promoted novel
developments for the interpolation of satellite altimetry such as the BFN-QG
and 4DVarNet schemes [Guillou . (2021), Beauchamp . (2023)].

In OSSE settings, we can train learning-based mapping schemes in a supervised
manner using model outputs as the ”ground truth” during the training phase.
Nonetheless, these training methods cannot be straightforwardly applied to
Observing System Experiments (OSEs) due to a lack of comprehensive
groundtruthed observation datasets. Applied machine learning practitioners
often grapple with insufficient amount of labelled data during the training of
supervised learning schemes, as the collection of large annotated datasets for
a specific task can be costly or unattainable. Proposed solutions includes the
exploitation of large existing datasets (such as ImageNet
dengImageNetLargescaleHierarchical2009) to train general purpose models (like
heDeepResidualLearning2016). Another approach involves the generation of
synthetic datasets to facilitate the creation of groundtruthed samples [Gomez
Gonzalez . (2017), Dosovitskiy . (2015)]. OSSEs, which combine ocean model
outputs and observing system simulators [Boukabara . (2018)], can deliver such
large synthetic groundtruthed datasets. We propose to investigate how OSSE-
based training strategies apply to the analysis of real satellite altimetry
datasets. Recent results of SSH super-resolution model trained on simulation
datasets and evaluated on real ones [Buongiorno Nardelli . (2022)] support the
relevance of such strategies.

###  2.3 Physics-aware deep-learning

In the last decades, DL advances combined with the rise in computational
resources and amount of data have shown the power of extracting knowledge from
data in domains ranging from computer vision to language processing [LeCun .
(2015)]. Yet, despite to the universality of DL architectures [Hornik .
(1989)], a central challenge persists in learning from data: the
generalization performance beyond the distribution of the training data. To
tackle this problem, the literature includes a variety of strategies such as
data augmentation [Shorten Khoshgoftaar (2019)] and regularization techniques,
including dropout layers [Srivastava . (2014)] and weight decay schemes
[Krizhevsky . (2012)]. This is of critical importance for physical systems,
where models trained on past data will be challenged when the system evolves
and reaches dynamics absent from the training data. We can see evidence of
this shortcoming in the instability challenges faced by neural closures for
climate models [Brenowitz . (2020)].

There have been a variety of approaches to harness physical priors within
learning schemes to address this issue. Some injects trainable components in
classical integration schemes of physical models such as
yinAugmentingPhysicalModels2021b. Others leverage physical priors within their
learning setups which can been used in the training objective [Raissi .
(2019), Greydanus . (2019)], as well as in the architecture [Li . (2020), Wang
. (2020)]. However most of these works have focused on relatively simple
physical models and it remains challenging to combine current state-of-the-art
ocean models with such methods. Obstacles include the complexity and cost of
running the physical models, the differences in programming tools and the
computing infrastructures used in each domain, as well as the availability of
automatic differentiation tools for state-of-the-art ocean models.

The proposed simulation-based training strategy offers another way to benefit
from the advances in high-resolution ocean modeling in the design of deep
neural models for ocean reanalysis problems.

##  3 Method

###  3.1 Overview

We designate our approach as ”simulation-based”, it consists in leveraging
ocean models and simulations of observing systems to design supervised
training environments. In this section, we describe the proposed method for
assessing the potential of simulation-based neural schemes for the mapping
real altimetry tracks. We describe the architecture considered in our study,
as well as the different datasets used for training purposes. We also detail
our simulation-based training setup and the proposed evaluation framework on
real altimetry.

Figure 1: Overview of the experimental setup. On the left side we display the
simulation-based training strategy based on an ocean simulation which will be
used for 1) generating synthetic observation and 2) computing the training
objective of the neural mapping scheme. On the right side we show the
evaluation principle of splitting the available satellite observations to
evaluate the method on data that were not used for the inference.

###  3.2 Neural mapping scheme

The neural mapping scheme considered for this study is the 4DVarNet
framework[Fablet, Amar . (2021)]. We choose this scheme due to the performance
shown in the OSSE setup. As reported in
beauchamp4DVarNetSSHEndtoendLearning2023, it significantly outperforms the
DUACS product [Taburet . (2019)] in the targeted Gulf stream region. 4DVarNet
relies on a variational data assimilation formulation. The reconstruction
results from the minimization of a variational cost. This cost encapsulates a
data fidelity term and a regularization term. It exploits a prior on the
space-time dynamics through a convolutional neural network inspired from
[Fablet . (2018)], and an iterative gradient-based minimization based on a
recurrent neural network as introduced by andrychowiczLearningLearnGradient.
The overall architecture and components are similar to those presented in
beauchamp4DVarNetSSHEndtoendLearning2023. We adapt some implementation details
based on cross-validation experiments to improve the performance and reduce
the training time. We refer the reader to the code for more details [Febvre
(2023)].

###  3.3 SSH Data

|  | Resolution | Reanalysis | Tide | DAC
---|---|---|---|---|---
NATL60 | [Ajayi . (2020)] | 1/60∘ | No | No | No
eNATL60-t | [Brodeau . (2020)] | 1/60∘ | No | Yes | Yes
eNATL60-0 | [Brodeau . (2020)] | 1/60∘ | No | No | Yes
GLORYS12-r | [Lellouche . (2021)] | 1/12∘ | Yes | No | No
GLORYS12-f | [Lellouche . (2021)] | 1/12∘ | No | No | No
ORCA025 | [Barnier . (2006)] | 1/4∘ | No | No | No
Table 1: Summary table of the different synthetic SSH fields used for
training. The last column indicate whether the Dynamic Atmospheric Correction
was applied on the synthetic SSH. It justify the presence of both eNATL60-0
and NATL60 to isolate the impacts of resolution and tide.

We use numerical simulations of ocean general circulation models (OGCM) to
build our reference SSH datasets. Such simulations involve a multitude of
decisions that affect the resulting simulated SSH. Here we consider NEMO
(Nucleus for European Modelling of the Ocean) [Gurvan . (2022)] which is among
the state-of-the art OGCM in operational oceanography [Ajayi . (2020)] as well
as in climate studies [Voldoire . (2013)]. The selected SSH datasets reported
in Table 1 focus on three main aspects: the added-value of high-resolution
eddy-rich simulations, the impact of reanalysis datasets and the relevance of
tide-resolving simulations.

In order to evaluate the impact of eddy-rich simulations, we consider NATL60,
GLORYS12-f and ORCA025 free runs, respectively with a horizontal grid
resolution of 1/60∘, 1/12∘, and 1/4∘. Finer grids allow for more processes to
be simulated. We therefore expect higher-resolution simulations to exhibit
structures closer to the real ocean and the associated trained deep learning
model to perform better. Regarding the impact of reanalysis data, we compare
numerical experiments with the GLORYS12-r reanalysis and the associated free
run GLORYS12-f. This reanalysis dataset relies on the assimilation of
temperature, sea level and sea ice concentration observations. Besides, the
recent eNATL60 twin simulations eNATL60-t and eNATL60-0 allow us to evaluate
the impact of tide-resolving simulations. We summarize in Table 1 the
characteristics of the different datasets.

###  3.4 OSSE-based training setup

We sketch the proposed OSSE-based training setup on the left side of the
Figure 1. In order to fairly evaluate the datasets’ quality as a training
resource, we standardize the training procedure. We regrid all simulations to
the same resolution (1/20°) and we use daily-averaged SSH fields as training
targets. We generate noise-free pseudo-observations by sampling values of the
daily-averaged fields corresponding to realistic orbits of a 5 altimeter-
constellation. We train all models from a one-year dataset in a Gulfstream
domain from (66°W, 32°N) to (54°W, 44°N) in which we keep the same two months
for validation. The hyper-parameters of the model and training procedure such
as the number of epoch, learning rate scheduler are the same for all the
experiments. The detailed configuration can be found by the reader in the
available implementation. As training objective, we combine the mean square
errors for the SSH fields and the amplitude of the gradients as well as a
regularization loss for the prior model.

###  3.5 OSE-based evaluation setup

As sketched on the right side of the Figure 1, the evaluation setup relies on
real altimetry data from the constellation of 6 satellites from 2017
(SARAL/Altika, Jason 2, Jason 3, Sentinel 3A, Haiyang-2A and Cryosat-2 ). We
apply the standardized setup presented in a data-challenge
https://github.com/ocean-data-challenges/2021a_SSH_mapping_OSE. We use the
data from the first five satellites as inputs for the mapping and the last one
(Cryosat-2) for computing the performance metrics. We compute these metrics in
the along-track geometry. The evaluation domain spans from (65°W, 33°N) to
(55°W, 43°N) and the evaluation period from January 1st to December 31st
2017\. Given $\eta_{c2}$ and $\hat{\eta}$ the measured SSH and the
reconstructed SSH respectively, we compute the following two metrics:

  * •

$\mu_{ssh}$ is a score based on the normalized root mean squared (nRMSE) error
computed as $1-\displaystyle\frac{RMS(\hat{\eta}-\eta_{c2})}{RMS(\eta_{c2})}$

  * •

$\lambda_{x}$ is the wavelength at which the power spectrum density (PSD)
score $1-\displaystyle\frac{PSD(\hat{\eta}-\eta_{c2})}{PSD(\eta_{c2})}$
crosses the $0.5$ threshold, which characterize the scales resolved by the
reconstruction (the error below that wavelength makes up for more than half of
the total signal)

In Table 3, we also consider the root mean square error (RMSE) as well as the
nRMSE score of the sea level anomaly $\mu_{sla}$ obtained by subtracting the
mean dynamic topography to the SSH. Lastly, we assess the performance
degradation resulting from the transition from simulated to real data by
quantifying the improvement relative to DUACS in the resolved scale
$\lambda_{x}$ on our OSE setup as well as on the OSSE benchmarking setup
proposed in guillouMappingAltimetryForthcoming2021. This benchmarking setup
relies on NATL60-CJM165 OSSE dataset. We refer the reader to
https://github.com/ocean-data-challenges/2020a_SSH_mapping\\_NATL60 for a
detailed description of this experimental setup.

Figure 2: Samples Kinetic energy and relative vorticity of the training and
reconstruction data of January 6${}^{t}h$ . The reconstructed year is 2017
while the training year vary depending on the simulation. The first two
columns (a) and (b) show the training data while columns (c) and (d) show the
associated 4DVarNet reconstruction. The kinetic energy is displayed in columns
((a) and (c)) and the relative vorticity normalized by the local Coriolis
parameter in columns ((b) and (d)). Each row shows the experiment using
respectively: ORCA025 (I), GLORYS12-f (II), GLORYS12-r (III), NATL60 (IV),
eNATL60-t (V) and eNATL60-0 (VI)

##  4 Results

This section details our numerical experiments for the considered real
altimetry case-study for a Gulf Stream region as described in Section 3.5. We
first report the benchmarking experiments to assess the performance of the
proposed learning-based strategy with respect to (w.r.t.) state-of-the-art
mapping schemes. We then analyse how the characteristics of the training
datasets drive the mapping performance.

###  4.1 Benchmarking against the state of the art

We report in Table 2 the performance metrics of state-of-the-art approaches
including both operational observation products [Taburet . (2019), Ubelmann .
(2021)], deep-learning-based schemes trained on observation data [Archambault
. (2023), Martin . (2023)] as well as methods using explicitly a model-based
prior on sea surface dynamics [Guillou . (2021), Ballarotta . (2020),
Lellouche . (2021)]. We compare those methods with a 4DVarNet trained on
eNATL60-0 OSSE dataset. The latter outperforms all other methods on the two
metrics considered (22% improvement in RMSE w.r.t. the DUACS product and 33%
improvement in the resolved scale). We report a significantly worse
performance for GLORYS12 reanalysis. This illustrates the challenge of
combining large ocean general circulation models and observation data for the
mapping of the SSH.

The last column indicates that the 4DVarNet scheme leads to the best mapping
scores for both the OSE and OSSE setups. For the latter, the reported
improvement of 47% is twice greater than the second best at 22%. The
performance of the 4DVarNet drops by 11% when considering the former. By
contrast, other methods do not show such differences between the OSE and OSSE
case-studies. This suggests that the finer-scale structures that are well
reconstructed in the OSSE setup are not as beneficial in the OSE setup. While
one could question the representativeness of the OSSE datasets for the fine-
scale patterns in the true ocean, real nadir altimetry data may also involve
multiple processes which could impede the reconstruction and evaluation of
horizontal scales below 100km.

| SSH | Deep | Calibrated on | Physical | rmse | $\mu_{ssh}$ | $\lambda_{x}$ | $1-\frac{\lambda_{x}}{\lambda_{ref}}$
---|---|---|---|---|---|---|---|---
| Only | Learning | data from | Model | (cm) | () | (km) | (% ose, osse)
(a) 4DVarNet | Yes | Yes | Simulation | – | 5.9 | 0.91 | 100 |  33, 47
(b) MUSTI | No | Yes | Satellite | – | 6.3 | 0.90 | 112 | 26, 22
(c) ConvLstm-SST | No | Yes | Satellite | – | 6.7 | 0.90 | 108 | 28, –
(d) ConvLstm | Yes | Yes | Satellite | – | 7.2 | 0.89 | 113 | 25, –
(e) DYMOST | Yes | No | Satellite | QG | 6.7 | 0.90 | 131 | 13, 11
(f) MIOST | Yes | No | Satellite | – | 6.8 | 0.90 | 135 | 11, 10
(g) BFN-QG | Yes | No | Satellite | QG | 7.6 | 0.89 | 122 | 19, 21
(h) DUACS | Yes | No | Satellite | – | 7.7 | 0.88 | 151 |  0, 0
(i) GLORYS12 | No | No | Satellite | NEMO | 15.1 | 0.77 | 241 | -60, –
Table 2:  SSH reconstruction performance of the benchmarked methods (a) 4DVarNet from this study trained on eNATL60-0 (b) Archambault et al. (2023), (c and d) ConvLstm-SST and ConvLstm from Martin et al. (2023), (e) DYMOST from Ballarotta et al. (2020), (f) MIOST from Ubelmann et al. (2021), (g) BFN-QG from Guillou et al. (2021), (h) DUACS from Taburet et al. (2019), (i) GLORYS12 from Lellouche et al. (2021. The columns indicate from left to right: whether athe mapping schemes rely only on SSH data or also exploit additional data such as gap free SST products; if the method uses deep learning architectures; the data used to calibrate (or train) the mapping scheme; the numerical model of the ocean used for the mapping if any (QG stands for quasi-geostrophic); $\mu$ and $\lambda_{x}$ are the metrics as described in Section 3.5 Figure 3:  Space-time spectral densities of the training datasets (first row) and of their associated reconstruction (second row). Darker blue in the lower left corner indicates higher energy at larger wavelength and periods. The different SSH fields exhibit different energy cascades when moving to finer temporal (upward) or spatial (rightward) scales. Training Data | RMSE | $\mu_{ssh}$ | $\mu_{sla}$ | $\lambda_{x}$ | $1-\frac{\lambda_{x}}{\lambda_{ref}}$
---|---|---|---|---|---
| (cm) |  |  | (km) | (% ose, osse)
NATL60 | 5.9 | 0.91 | 0.80 | 98 | (35, –)
eNATL60-t | 5.9 | 0.91 | 0.80 | 100 | (33, 48)
eNATL60-0 | 5.9 | 0.91 | 0.80 | 100 | (33, 47)
GLORYS12-r | 6.3 | 0.90 | 0.78 | 106 | (30, 28)
GLORYS12-f | 6.7 | 0.90 | 0.77 | 119 | (21, 23)
ORCA025 | 7.1 | 0.89 | 0.76 | 126 | (17, 17)
Table 3: Performance of 4DVarNet mapping schemes trained on different
simulated datasets. The first column shows the source of the training dataset
as described in Table 1; the subsequent columns indicate the reconstruction
metrics described in Section 3.5. Note that the NATL60 could not be evaluated
on the OSSE setup since the evaluation data were used for validation during
the training stage.

###  4.2 Eddy-present datasets versus eddy-rich ones

Figure 4:  Spectral analysis of the training and reconstructed SSH datasets.
We display the PSD of the training dataset (left plot), reconstructed SSH
field (center plot) as well as the associated PSD score (right plot)

We analyse here in more detail the impact of the spatial resolution of the
training dataset onto the reconstruction performance. In Table 3, as expected,
the higher resolution grid in the ocean run simulation leads to a better
mapping with a 22% improvement in $\lambda_{x}$ and a 17% improvement in the
RMSE score between the experiments with the coarsest (ORCA025) and finest
(NATL60) resolutions. We also observe qualitative differences in the relative
vorticity fields in Figure 2. Residual artifacts due to the altimetry tracks
appear (60°W, 39°N) for the two lower-resolution training datasets. They are
greatly diminished when considering the NATL60 dataset. Despite these
differences, the reconstructed vorticity and kinetic energy fields in Figure 2
look very similar for the different 4DVarNet schemes, whatever the training
datasets. By contrast, the vorticity and kinetic energy fields in the training
datasets clearly depict fewer fine-scale structures and weaker gradients for
the lower-resolution simulation datasets, namely ORCA025 and GLORYS12-f. These
results support the generalization skills of 4DVarNet schemes to map real
altimetry tracks despite being trained on SSH sensibly different from the
reconstruction.

We draw similar conclusions from the analysis of the spectral densities shown
in Figure 4. The differences in the energy distribution of the training data
significantly reduce in the reconstructions. 4DVarNet schemes trained from
higher-resolution datasets however result in more faithful reconstruction at
all scales. The patterns observed for the temporal PSD are slightly different
in Figure 3. We do not observe the same homogenization as for the spatial PSD.
Lower-resolution training datasets involve a significant drop of an order of
magnitude for periods greater than 10 days and wavelength greater than 200km.

###  4.3 Forced simulation datasets versus reanalysis ones

Figure 5: Spectral impact of model reanalysis. We display the PSD of the
training dataset (left plot), reconstructed SSH field (center plot) as well as
the associated PSD score (right plot)

Looking in more specifically at the effect of ocean reanalysis between the two
experiments GLORYS12-f and GLORYS12-r. We can first note the impact of
observation data assimilation in Figure 3 where we see how the power spectrum
of the reanalysis is significantly raised compared to the free run. The
spectrum is closer to ones of the higher resolution simulations. Visually we
also clearly see stronger gradients in the kinetic energy in Figure 2.

We can observe a similar behavior as in Section 4.2 in Figure 5 with the gap
of in spectral density being diminished between the training and
reconstruction data, and the PSD score indicating a lower energy of the error
at all scales for the reanalysis-based experiment.

Quantitatively in Table 1 we see an improvement of 11% in both the RMSE and
the scale resolved, besides training on a reanalysis increase the relative
gain w.r.t. DUACS significantly more on real data (+9%) than on simulated data
(+5%) as we can see in the right most column. This suggests that the
reanalysis dataset conveys information on real world observations which
improves the generalization performance.

###  4.4 Tide-free datasets versus tide-resolving ones

We assess here the impact of tide-resolving simulation used as training data.
We use the twin eNATL60 runs eNATL60-t and eNATL60-0. Contrary to other runs,
those simulations contain barometric and wind forcing, we therefore remove the
Dynamic Atmospheric Correction [Carrere . (2016)] from the SSH fields.
Additionally since the barotropic tide signals are removed from real altimetry
tracks prior to interpolation, we also remove the signal from the training
data by subtracting the spatial mean over the training domain for each hourly
snapshot before calculating the daily averages.

Given those processing steps, the two training datasets exhibit very similar
wavenumber spectra as shown in Figures 3. We also find that training on those
two datasets produce little differences in the reconstructions both
quantitatively (see Table 3) and qualitatively (Fig. 2). The resulting
performance is comparable to that of the NATL60 experiment.

We identify two hypotheses for explaining why tide-resolving simulation do not
lead to better mapping schemes:

  * •

The preprocessing applied on the training field remove the main tide signals.
We therefore effectively measure the impact of tide modeling on other ocean
processes that may be less significant;

  * •

The evaluation procedure applied on altimetry tracks on which the barotropic
tide has been filtered may not be interpretable enough to measure the
reconstruction of residual tide signals. New instruments like the KaRIN
deployed in the SWOT mission may provide new ways to better quantify those
effects.

These findings provide motivation for carefully considering the purpose of the
learning-based model when making decisions about the training data. In our
case, explicitly modeling tide processes that are removed from the
observations in the evaluation setup added overheads in the computational cost
of running the simulation as well as in the preprocessing of the training
data. Additionally given the considered evaluation data and metrics, we were
not able to quantify any significant differences between the two trained
mapping schemes.

##  5 Discussion

This study has been greatly facilitated by the standardized tasks and
evaluation setups proposed in data-challenges https://ocean-data-
challenges.github.io/. Data-challenges are used to specify a targeted problem
of interest to domain experts through datasets and relevant evaluation
metrics. This preliminary work have been instrumental in constituting the
comprehensive benchmark and combining methods from different teams and
institution around the world. Additionally, it also constitutes a strong basis
for a trans-disciplinary collaboration between the ocean and machine learning
research communities.

Moreover, the results presented in this study introduce a use of ocean
simulations for developing altimetry products. This opens new ways for ocean
physicist, modelers and operational oceanographers to collaborate. In order to
assess the range of these new synergies, it would be interesting to explore if
the approach proposed here of training neural schemes using simulation data
would generalize to other tasks such as forecast or sensor calibration and to
other quantities like surface temperature, currents, salinity or biochemical
tracers.

If the simulation-based training approach introduced here is successfully
extended to other ocean problems, one could envision training large foundation
deep learning models [Brown . ()] capturing the inner structure of high
resolution ocean simulations which could then be used in many downstream
applications. This could be the way to capitalize on all the advancement in
ocean modeling without having to run OGCM numerical simulation for each
downstream products.

Furthermore, we would like to highlight the cost consideration when running
numerical simulation intended for training learning based schemes. Indeed
given that the eNATL60 run took 2700x CPU hours and 350x memory compared to
the ORCA025 run for a smaller domain, a trade-off arises between generating
multiple ”cheap” trajectories versus generating a single more realistic
trajectory.

To conclude, we have shown in this study that training machine learning models
on simulations datasets leads good performance on real altimetry data mapping
and outperforms current state of the art approaches. The model trained on
NATL60 reduces the RMSE by 18% compared neural schemes trained on observation
data and improves the scales resolved by 33% compared to the DUACS operational
product. Even the coarsest simulation considered ORCA025 provides competitive
results with current operational methods. We have shown that using a more
realistic SSH fields using reanalysis or higher resolution simulations
increases the performances of the trained model. This is an exciting result
that shows the potential for training operational products from ocean
simulations and how advances in ocean modeling in operational oceanography can
be beneficial. The results shown here are limited to the interpolation problem
on a regional domain but the robustness of the performance shown are
encouraging for further developing these results using a larger domain.

## Open Research Section

The authors provide the training data, source code, reconstructed maps and
trained model for each experiments of the manuscript at
https://doi.org/10.5281/zenodo.8064114.

###### Acknowledgements.

This work was supported by ANR Projects Melody and OceaniX and CNES. It
benefited from HPC and GPU resources from GENCI-IDRIS (Grant 2020-101030) and
Ifremer.

## References

  * Abdalla . (2021) abdallaAltimetryFutureBuilding2021Abdalla, S., Abdeh Kolahchi, A., Ablain, M., Adusumilli, S., Aich Bhowmick, S., Alou-Font, E.Zlotnicki, V.  202107\.  Altimetry for the Future: Building on 25 Years of Progress Altimetry for the future: Building on 25 years of progress.  Advances in Space Research682319–363.  10.1016/j.asr.2021.01.022
  * Ajayi . (2020) ajayiSpatialTemporalVariability2020Ajayi, A., Le Sommer, J., Chassignet, E., Molines, JM., Xu, X., Albert, A. Cosme, E.  2020\.  Spatial and Temporal Variability of the North Atlantic Eddy Field From Two Kilometric-Resolution Ocean Models Spatial and Temporal Variability of the North Atlantic Eddy Field From Two Kilometric-Resolution Ocean Models.  Journal of Geophysical Research: Oceans1255e2019JC015827.  10.1029/2019JC015827
  * Alvera Azcarate . (2005) alveraazcarateReconstructionIncompleteOceanographic2005Alvera Azcarate, A., Barth, A., Rixen, M. Beckers, JM.  2005\.  Reconstruction of Incomplete Oceanographic Data Sets Using Empirical Orthogonal Functions: Application to the Adriatic Sea Surface Temperature Reconstruction of incomplete oceanographic data sets using empirical orthogonal functions: Application to the Adriatic Sea surface temperature.  Ocean Modelling94.  10.1016/j.ocemod.2004.08.001
  * Andrychowicz . (2016) andrychowiczLearningLearnGradientAndrychowicz, M., Denil, M., Gómez, S., Hoffman, MW., Pfau, D. Schaul, T.  201611\.  Learning to Learn by Gradient Descent by Gradient Descent Learning to learn by gradient descent by gradient descent.
  * Archambault . (2023) archambaultMultimodalUnsupervisedSpatioTemporal2023Archambault, T., Filoche, A., Charantonnis, A. Béréziat, D.  202302\.  Multimodal Unsupervised Spatio-Temporal Interpolation of Satellite Ocean Altimetry Maps Multimodal Unsupervised Spatio-Temporal Interpolation of satellite ocean altimetry maps.  VISAPP. VISAPP.
  * Ballarotta . (2020) ballarottaDynamicMappingAlongTrack2020Ballarotta, M., Ubelmann, C., Rogé, M., Fournier, F., Faugère, Y., Dibarboure, G.Picot, N.  202009\.  Dynamic Mapping of Along-Track Ocean Altimetry: Performance from Real Observations Dynamic Mapping of Along-Track Ocean Altimetry: Performance from Real Observations.  Journal of Atmospheric and Oceanic Technology3791593–1601.  10.1175/JTECH-D-20-0030.1
  * Barnier . (2006) bernardImpactPartialSteps2006Barnier, B., Gurvan, M., Thierry, P., Jean-Marc, M., Anne-Marie, T., Julien, LS.Beverly, DC.  200612\.  Impact of Partial Steps and Momentum Advection Schemes in a Global Ocean Circulation Model at Eddy-Permitting Resolution Impact of partial steps and momentum advection schemes in a global ocean circulation model at eddy-permitting resolution.  Ocean Dynamics565543–567.  10.1007/s10236-006-0082-1
  * Barth . (2022) barthDINCAEMultivariateConvolutional2022Barth, A., Alvera-Azcárate, A., Troupin, C. Beckers, JM.  202203\.  DINCAE 2.0: Multivariate Convolutional Neural Network with Error Estimates to Reconstruct Sea Surface Temperature Satellite and Altimetry Observations DINCAE 2.0: Multivariate convolutional neural network with error estimates to reconstruct sea surface temperature satellite and altimetry observations.  Geoscientific Model Development1552183–2196.  10.5194/gmd-15-2183-2022
  * Beauchamp . (2021) beauchampDatadrivenLearningbasedInterpolations2021Beauchamp, M., fablet, r., Ubelmann, C., Ballarotta, M. Chapron, B.  202101\.  Data-Driven and Learning-Based Interpolations of along-Track Nadir and Wide-Swath SWOT Altimetry Observations Data-driven and learning-based interpolations of along-track Nadir and wide-swath SWOT altimetry observations.  Proceedings of the 10th International Conference on Climate Informatics Proceedings of the 10th International Conference on Climate Informatics ( 22–29).  New York, NY, USAAssociation for Computing Machinery.  10.1145/3429309.3429313
  * Beauchamp . (2023) beauchamp4DVarNetSSHEndtoendLearning2023Beauchamp, M., Febvre, Q., Georgenthum, H. Fablet, R.  202304\.  4DVarNet-SSH: End-to-End Learning of Variational Interpolation Schemes for Nadir and Wide-Swath Satellite Altimetry 4DVarNet-SSH: End-to-end learning of variational interpolation schemes for nadir and wide-swath satellite altimetry.  Geoscientific Model Development1682119–2147.  10.5194/gmd-16-2119-2023
  * Benkiran . (2021) benkiranAssessingImpactAssimilation2021Benkiran, M., Ruggiero, G., Greiner, E., Le Traon, PY., Rémy, E., Lellouche, JM.Tchonang, B.  2021\.  Assessing the Impact of the Assimilation of SWOT Observations in a Global High-Resolution Analysis and Forecasting System Part 1: Methods Assessing the Impact of the Assimilation of SWOT Observations in a Global High-Resolution Analysis and Forecasting System Part 1: Methods.  Frontiers in Marine Science8.
  * Boukabara . (2018) boukabaraCommunityGlobalObserving2018Boukabara, SA., Ide, K., Zhou, Y., Shahroudi, N., Hoffman, RN., Garrett, K.Atlas, R.  201810\.  Community Global Observing System Simulation Experiment (OSSE) Package (CGOP): Assessment and Validation of the OSSE System Using an OSSE–OSE Intercomparison of Summary Assessment Metrics Community Global Observing System Simulation Experiment (OSSE) Package (CGOP): Assessment and Validation of the OSSE System Using an OSSE–OSE Intercomparison of Summary Assessment Metrics.  Journal of Atmospheric and Oceanic Technology35102061–2078.  10.1175/JTECH-D-18-0061.1
  * Brenowitz . (2020) brenowitzInterpretingStabilizingMachineLearning2020Brenowitz, ND., Beucler, T., Pritchard, M. Bretherton, CS.  202012\.  Interpreting and Stabilizing Machine-Learning Parametrizations of Convection Interpreting and Stabilizing Machine-Learning Parametrizations of Convection.  Journal of the Atmospheric Sciences77124357–4375.  10.1175/JAS-D-20-0082.1
  * Brodeau . (2020) brodeauOceannextENATL60Material2020Brodeau, L., Sommer, JL. Albert, A.  202009\.  Ocean-next/eNATL60: Material Describing the Set-up and the Assessment of NEMO-eNATL60 Simulations. Ocean-next/eNATL60: Material describing the set-up and the assessment of NEMO-eNATL60 simulations.  Zenodo.  10.5281/zenodo.4032732
  * Brown . () brownLanguageModelsAreBrown, TB., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P.Henighan, T.  .  Language Models Are Few-Shot Learners Language Models are Few-Shot Learners.
  * Buongiorno Nardelli . (2022) buongiornonardelliSuperResolvingOceanDynamics2022Buongiorno Nardelli, B., Cavaliere, D., Charles, E. Ciani, D.  202201\.  Super-Resolving Ocean Dynamics from Space with Computer Vision Algorithms Super-Resolving Ocean Dynamics from Space with Computer Vision Algorithms.  Remote Sensing1451159.  10.3390/rs14051159
  * Carrassi . (2018) carrassiDataAssimilationGeosciences2018Carrassi, A., Bocquet, M., Bertino, L. Evensen, G.  2018\.  Data Assimilation in the Geosciences: An Overview of Methods, Issues, and Perspectives Data assimilation in the geosciences: An overview of methods, issues, and perspectives.  WIREs Climate Change95e535.  10.1002/wcc.535
  * Carrere . (2016) carrereMajorImprovementAltimetry2016Carrere, L., Faugère, Y. Ablain, M.  201606\.  Major Improvement of Altimetry Sea Level Estimations Using Pressure-Derived Corrections Based on ERA-Interim Atmospheric Reanalysis Major improvement of altimetry sea level estimations using pressure-derived corrections based on ERA-Interim atmospheric reanalysis.  Ocean Science123825–842.  10.5194/os-12-825-2016
  * Deng . (2009) dengImageNetLargescaleHierarchical2009Deng, J., Dong, W., Socher, R., Li, LJ., Li, K. Fei-Fei, L.  200906\.  ImageNet: A Large-Scale Hierarchical Image Database ImageNet: A large-scale hierarchical image database.  2009 IEEE Conference on Computer Vision and Pattern Recognition 2009 IEEE Conference on Computer Vision and Pattern Recognition ( 248–255).  10.1109/CVPR.2009.5206848
  * Dibarboure . (2022) dibarboureDataDrivenCalibrationAlgorithm2022Dibarboure, G., Ubelmann, C., Flamant, B., Briol, F., Peral, E., Bracher, G.Picot, N.  202201\.  Data-Driven Calibration Algorithm and Pre-Launch Performance Simulations for the SWOT Mission Data-Driven Calibration Algorithm and Pre-Launch Performance Simulations for the SWOT Mission.  Remote Sensing14236070.  10.3390/rs14236070
  * Dosovitskiy . (2015) dosovitskiyFlowNetLearningOptical2015Dosovitskiy, A., Fischer, P., Ilg, E., Häusser, P., Hazirbas, C., Golkov, V.Brox, T.  201512\.  FlowNet: Learning Optical Flow with Convolutional Networks FlowNet: Learning Optical Flow with Convolutional Networks.  2015 IEEE International Conference on Computer Vision (ICCV) 2015 IEEE International Conference on Computer Vision (ICCV) ( 2758–2766).  10.1109/ICCV.2015.316
  * Fablet, Amar . (2021) fabletENDTOENDPHYSICSINFORMEDREPRESENTATION2021Fablet, R., Amar, MM., Febvre, Q., Beauchamp, M. Chapron, B.  202106\.  END-TO-END PHYSICS-INFORMED REPRESENTATION LEARNING FOR SATELLITE OCEAN REMOTE SENSING DATA: APPLICATIONS TO SATELLITE ALTIMETRY AND SEA SURFACE CURRENTS END-TO-END PHYSICS-INFORMED REPRESENTATION LEARNING FOR SATELLITE OCEAN REMOTE SENSING DATA: APPLICATIONS TO SATELLITE ALTIMETRY AND SEA SURFACE CURRENTS.  ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information SciencesV-3-2021295–302.  10.5194/isprs-annals-V-3-2021-295-2021
  * Fablet, Chapron . (2021) fabletLearningVariationalData2021Fablet, R., Chapron, B., Drumetz, L., Mémin, E., Pannekoucke, O. Rousseau, F.  2021\.  Learning Variational Data Assimilation Models and Solvers Learning Variational Data Assimilation Models and Solvers.  Journal of Advances in Modeling Earth Systems1310e2021MS002572.  10.1029/2021MS002572
  * Fablet . (2018) fabletBilinearResidualNeural2018Fablet, R., Ouala, S. Herzet, C.  201809\.  Bilinear Residual Neural Network for the Identification and Forecasting of Geophysical Dynamics Bilinear Residual Neural Network for the Identification and Forecasting of Geophysical Dynamics.  2018 26th European Signal Processing Conference (EUSIPCO) 2018 26th European Signal Processing Conference (EUSIPCO) ( 1477–1481).  10.23919/EUSIPCO.2018.8553492
  * Febvre (2023) febvreCodeDataRelease2023aFebvre.  202306\.  Code and Data Release Simulation-based 4dVarNets for Satellite Altimetry Code and Data release Simulation-based 4dVarNets for satellite altimetry.  10.5281/zenodo.8119300
  * Gomez Gonzalez . (2017) gomezgonzalezVIPVortexImage2017Gomez Gonzalez, CA., Wertz, O., Absil, O., Christiaens, V., Defrère, D., Mawet, D.Surdej, J.  201707\.  VIP: Vortex Image Processing Package for High-contrast Direct Imaging VIP: Vortex Image Processing Package for High-contrast Direct Imaging.  The Astronomical Journal1547.  10.3847/1538-3881/aa73d7
  * Greydanus . (2019) greydanusHamiltonianNeuralNetworks2019Greydanus, S., Dzamba, M. Yosinski, J.  2019\.  Hamiltonian Neural Networks Hamiltonian Neural Networks.  Advances in Neural Information Processing Systems Advances in Neural Information Processing Systems ( 32).  Curran Associates, Inc.
  * Guillou . (2021) guillouMappingAltimetryForthcoming2021Guillou, FL., Metref, S., Cosme, E., Ubelmann, C., Ballarotta, M., Sommer, JL. Verron, J.  202104\.  Mapping Altimetry in the Forthcoming SWOT Era by Back-and-Forth Nudging a One-Layer Quasigeostrophic Model Mapping Altimetry in the Forthcoming SWOT Era by Back-and-Forth Nudging a One-Layer Quasigeostrophic Model.  Journal of Atmospheric and Oceanic Technology384697–710.  10.1175/JTECH-D-20-0104.1
  * Gurvan . (2022) gurvanNEMOOceanEngine2022Gurvan, M., Bourdallé-Badie, R., Chanut, J., Clementi, E., Coward, A., Ethé, C.Moulin, A.  202203\.  NEMO Ocean Engine NEMO ocean engine.  10.5281/zenodo.6334656
  * He . (2016) heDeepResidualLearning2016He, K., Zhang, X., Ren, S. Sun, J.  2016\.  Deep Residual Learning for Image Recognition Deep Residual Learning for Image Recognition.  Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition ( 770–778).
  * Hornik . (1989) hornikMultilayerFeedforwardNetworks1989Hornik, K., Stinchcombe, M. White, H.  198901\.  Multilayer Feedforward Networks Are Universal Approximators Multilayer feedforward networks are universal approximators.  Neural Networks25359–366.  10.1016/0893-6080(89)90020-8
  * Krizhevsky . (2012) krizhevskyImageNetClassificationDeep2012Krizhevsky, A., Sutskever, I. Hinton, GE.  2012\.  ImageNet Classification with Deep Convolutional Neural Networks ImageNet Classification with Deep Convolutional Neural Networks.  Advances in Neural Information Processing Systems Advances in Neural Information Processing Systems ( 25).  Curran Associates, Inc.
  * LeCun . (2015) lecunDeepLearning2015LeCun, Y., Bengio, Y. Hinton, G.  201505\.  Deep Learning Deep learning.  Nature5217553436–444.  10.1038/nature14539
  * Lellouche . (2021) jean-michelCopernicusGlobal122021Lellouche, J., Greiner, E., Bourdallé-Badie, R., Garric, G., Melet, A., Drévillon, M.Le Traon, P.  2021\.  The Copernicus Global 1/12° Oceanic and Sea Ice GLORYS12 Reanalysis The Copernicus Global 1/12° Oceanic and Sea Ice GLORYS12 Reanalysis.  Frontiers in Earth Science9.
  * Lguensat . (2017) lguensatAnalogDataAssimilation2017Lguensat, R., Tandeo, P., Ailliot, P., Pulido, M. Fablet, R.  201710\.  The Analog Data Assimilation The Analog Data Assimilation.  Monthly Weather Review145104093–4107.  10.1175/MWR-D-16-0441.1
  * Li . (2020) li2020fourierLi, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A. Anandkumar, A.  2020\.  Fourier Neural Operator for Parametric Partial Differential Equations. Fourier neural operator for parametric partial differential equations.
  * Martin . (2023) martinSynthesizingSeaSurface2023Martin, SA., Manucharyan, GE. Klein, P.  2023\.  Synthesizing Sea Surface Temperature and Satellite Altimetry Observations Using Deep Learning Improves the Accuracy and Resolution of Gridded Sea Surface Height Anomalies Synthesizing Sea Surface Temperature and Satellite Altimetry Observations Using Deep Learning Improves the Accuracy and Resolution of Gridded Sea Surface Height Anomalies.  Journal of Advances in Modeling Earth Systems155e2022MS003589.  10.1029/2022MS003589
  * Raissi . (2019) raissiPhysicsinformedNeuralNetworks2019Raissi, M., Perdikaris, P. Karniadakis, GE.  201902\.  Physics-Informed Neural Networks: A Deep Learning Framework for Solving Forward and Inverse Problems Involving Nonlinear Partial Differential Equations Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.  Journal of Computational Physics378686–707.  10.1016/j.jcp.2018.10.045
  * Ronneberger . (2015) ronnebergerUNetConvolutionalNetworks2015Ronneberger, O., Fischer, P. Brox, T.  2015\.  U-Net: Convolutional Networks for Biomedical Image Segmentation U-Net: Convolutional Networks for Biomedical Image Segmentation.  N. Navab, J. Hornegger, WM. Wells AF. Frangi (), Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015 Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015 ( 234–241).  ChamSpringer International Publishing.  10.1007/978-3-319-24574-4_28
  * Shorten Khoshgoftaar (2019) shortenSurveyImageData2019Shorten, C. Khoshgoftaar, TM.  201907\.  A Survey on Image Data Augmentation for Deep Learning A survey on Image Data Augmentation for Deep Learning.  Journal of Big Data6160.  10.1186/s40537-019-0197-0
  * Srivastava . (2014) srivastavaDropoutSimpleWay2014Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. Salakhutdinov, R.  2014\.  Dropout: A Simple Way to Prevent Neural Networks from Overfitting Dropout: A Simple Way to Prevent Neural Networks from Overfitting.  Journal of Machine Learning Research15561929–1958.
  * Taburet . (2019) taburetDUACSDT2018252019Taburet, G., Sanchez-Roman, A., Ballarotta, M., Pujol, MI., Legeais, JF., Fournier, F.Dibarboure, G.  201909\.  DUACS DT2018: 25 Years of Reprocessed Sea Level Altimetry Products DUACS DT2018: 25 years of reprocessed sea level altimetry products.  Ocean Science1551207–1224.  10.5194/os-15-1207-2019
  * Ubelmann . (2021) ubelmannReconstructingOceanSurface2021Ubelmann, C., Dibarboure, G., Gaultier, L., Ponte, A., Ardhuin, F., Ballarotta, M. Faugère, Y.  2021\.  Reconstructing Ocean Surface Current Combining Altimetry and Future Spaceborne Doppler Data Reconstructing Ocean Surface Current Combining Altimetry and Future Spaceborne Doppler Data.  Journal of Geophysical Research: Oceans1263e2020JC016560.  10.1029/2020JC016560
  * Voldoire . (2013) voldoireCNRMCM5GlobalClimate2013Voldoire, A., Sanchez-Gomez, E., Salas y Mélia, D., Decharme, B., Cassou, C., Sénési, S.Chauvin, F.  201305\.  The CNRM-CM5.1 Global Climate Model: Description and Basic Evaluation The CNRM-CM5.1 global climate model: Description and basic evaluation.  Climate Dynamics4092091–2121.  10.1007/s00382-011-1259-y
  * Wang . (2020) Wang2020TFWang, R., Kashinath, K., Mustafa, M., Albert, A. Yu, R.  2020\.  Towards Physics-informed Deep Learning for Turbulent Flow Prediction Towards physics-informed deep learning for turbulent flow prediction.  Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.
  * Wong . (2020) wongArgoData19992020Wong, APS., Wijffels, SE., Riser, SC., Pouliquen, S., Hosoda, S., Roemmich, D.Park, HM.  2020\.  Argo Data 1999–2019: Two Million Temperature-Salinity Profiles and Subsurface Velocity Observations From a Global Array of Profiling Floats Argo Data 1999–2019: Two Million Temperature-Salinity Profiles and Subsurface Velocity Observations From a Global Array of Profiling Floats.  Frontiers in Marine Science7.
  * Yin . (2021) yinAugmentingPhysicalModels2021bYin, Y., Guen, VL., Dona, J., de Bézenac, E., Ayed, I., Thome, N. Gallinari, P.  202112\.  Augmenting Physical Models with Deep Networks for Complex Dynamics Forecasting* Augmenting physical models with deep networks for complex dynamics forecasting*.  Journal of Statistical Mechanics: Theory and Experiment202112124012.  10.1088/1742-5468/ac3ae5