# On the Scaling Laws of Geographical Representation
in Language Models

###### Abstract

Language models have long been shown to embed geographical information in
their hidden representations. This line of work has recently been revisited by
extending this result to Large Language Models (LLMs). In this paper, we
propose to fill the gap between well-established and recent literature by
observing how geographical knowledge evolves when scaling language models. We
show that geographical knowledge is observable even for tiny models, and that
it scales consistently as we increase the model size. Notably, we observe that
larger language models cannot mitigate the geographical bias that is inherent
to the training data.

Keywords: language models, geographic, bias

On the Scaling Laws of Geographical Representation
in Language Models

Nathan Godey$\mkern 6.0mu{}^{1,2}$ Éric de la Clergerie1 Benoît Sagot1
---
1Inria, 2Sorbonne Université
Paris, France
<EMAIL_ADDRESS>

Abstract content

## 1\. Introduction & Related work

In recent years, numerous studies analyzing the hidden representations of
self-supervised language models have provided insights into how these models
incorporate linguistic knowledge from their training data (Gupta et al., 2015;
Köhn, 2015; Shi et al., 2016; Hupkes and Zuidema, 2018; Conneau et al., 2018;
Jawahar et al., 2019).

This line of work has been called probing, as most approaches are generally
based on the training of classifiers—or probes—upon frozen hidden
representations.

Analyzing the representations of language models can point out sociocultural
biases that were inherently learned by the models during training (Zhao et
al., 2018), and training probes can help with mitigating these biases
(Ravfogel et al., 2020; Iskander et al., 2023).

Among probing tasks, several works have focused on geographical
representations that are implicitly embedded in language models. Louwerse and
Benesh (2012) show that coordinates of places in the Middle-Earth can be
predicted by just using the co-occurence matrix extracted from the Lord of the
Rings novels. Faisal and Anastasopoulos (2022) build networks from
geographical representations based on monolingual and multilingual models of
different sizes. They show that all models embed more accurate geographical
representations for countries of the Global North.

This geographical discrepancy can be explained by biases that are inherent to
the datasets used for pretraining Faisal et al. (2022). Imbalanced frequency
distributions of geographical references in pretraining data causes
distortions in the representational space (Zhou et al., 2021). These
distortions lead to a loss in the models’ ability to differentiate between
under-represented locations.

Recently, Gurnee and Tegmark (2023) have probed large language models from the
Llama-2 suite (Touvron et al., 2023) to extract coordinates of prompted
locations from hidden representations across layers. They show that models
ranging from 7B to 70B parameters are able to convincingly embed geographical
coordinates on a world map when representing basic prompts.

In this work, we propose to extend the analysis by Gurnee and Tegmark (2023)
to smaller language models, in order to observe how scale affects the ability
of models to implicitly embed geographical information from raw training data.
We show that such ability consistently improves with model size, and that even
tiny models are able to produce visually meaningful world maps.

We make several contributions:

  * •

We show that geographical information can be extracted to a certain extent
from representations at every model scale;

  * •

We observe that larger models are more geographically biased than their
smaller counterparts;

  * •

We find that the performance of models in terms of geographical probing is
correlated with the frequency of corresponding country names in the training
data.

## 2\. Scaling Laws of Geographical Probing

(a) Pythia 14M ($R^{2}=34.34$)

(b) Pythia 160M ($R^{2}=55.28$)

(c) Pythia 1B ($R^{2}=67.94$)

(d) Pythia 2.8B ($R^{2}=74.97$)

Figure 1: Predicted coordinates of test set instances for different model
sizes. Each color represents a different continent.

In this section, we train geographical probes for a wide variety of models at
different scales.

### 2.1. Methodology

We use the World dataset from Gurnee and Tegmark (2023) as a geographical data
source. It contains 39,504 location names from the whole world along with
corresponding longitude and latitude. We use the same train-test split
strategy as in the original article, thus keeping 20% of samples for testing
purposes.

For each location name $X$, we prompt models with the text: “Where is $X$ in
the world?”. We then infer with a given model on the whole dataset, and use
the last token belonging to the entity $X$ as the model’s representation. To
follow the linear probing paradigm used in Gurnee and Tegmark (2023), we train
a Ridge linear regressor (Hoerl and Kennard, 1970) to predict latitude and
longitude based on the model’s representations. We then measure the probe’s
performance on the test set using the $R^{2}$ correlation coefficient.

### 2.2. Results

In Figure 1, we display the predictions of the probe for the most performant
layer, which is generally the last one. We observe that geographical
information can be extracted from models even for a very small parameter
count. The performance of the probes seem to increase with the model size.

(a) Decoder models

(b) Encoder models

Figure 2: Evolution of the $R^{2}$ coefficient on the test set for various
model suites.

We show in Figure 2 that the performance of language models evolves
consistently with model size, regardless of the architecture. We validate this
property on several decoder model families: GPT-2 (Radford et al., 2019), OPT
(Zhang et al., 2022), Pythia (Biderman et al., 2023), GPT-Neo (Black et al.,
2021), the multilingual mGPT (Shliazhko et al., 2023), and Llama-2 (Touvron et
al., 2023). We also display results for several encoder models: BERT (Devlin
et al., 2019; Turc et al., 2019), RoBERTa (Liu et al., 2019), ELECTRA (Clark
et al., 2020), and DeBERTa-v3 (He et al., 2020). This property also applies
for encoder models, for which we notice that the BERT suite unexpectedly
outperforms its counterparts. The performance of encoder models is comparable
with the one of equivalent decoder models. We can underline the fact that
BERT-Large (336M parameters) is as accurate as the three times larger
Pythia-1B.

Interestingly, the multilingual XLM-R (Conneau et al., 2020) underperforms its
counterparts, even though multilingual data must have increased the training
data’s geographical diversity to some extent (Faisal and Anastasopoulos,
2021). The mGPT suite also slightly underperforms Pythia models at equivalent
model sizes.

We verified that the better performance of larger models was not solely
related with the ability of the probes to extract better patterns from their
higher-dimensionality hidden representations. We achieved this by
concatenating representations with themselves to increase dimensionality
without introducing novel knowledge. It led to slightly worse performance for
all tested models, thus showing that performance was not a consequence of
dimensionality alone.

## 3\. Geographical Bias and Scale

In Figure 1, it seems at first glance that as the model size increases, the
predictions tend to be more accurate for locations of the Southern Hemisphere.
In this section, we propose to quantify this hypothesized behavior by
measuring the bias across countries and continents for various scales. We also
correlate the models’ accuracy with both lexical and geographical factors.

### 3.1. Measuring bias

We group probe performance as measured by mean-squared error (MSE) on
predicted coordinates, and average measures by continent in Figure 3. While we
notice that the performance increases consistently for every continent, we do
not observe a significant reduction in the performance gap across continents
as model size increases.

Figure 3: Average MSE by continent for different sizes in the Pythia suite.

To measure the heterogeneity of the probing performance of language models
across countries, we use the Gini coefficient (Gini, 1912) that is widely used
in economics. Given a series of observed variables $(x_{i})_{i\in[1,N]}$, the
Gini coefficient is defined as:

$Gini(x)=\frac{\sum_{i,j\in[1,N]}|x_{i}-x_{j}|}{N\cdot\sum_{i=1}^{N}x_{i}}$

A Gini coefficient of 1 reflects perfect heterogeneity, while a Gini of 0
implies perfect homogeneity.

Figure 4: Gini coefficients of MSE on the test set averaged by country or by
continent, as model size increases.

Figure 4 shows that the larger the model is, the more heterogeneous the probe
performance is across countries and continents. This contradicts the
impression given by Figure 1, and shows that scale does not solve the
geographical discrepancy caused by bias inherent to the training data.

Figure 5: Test log-MSE for Pythia-1B as plotted on a World map.

In Figure 5, we locally average log-MSE on a World map, and report results
agglomerated accorded to latitude and longitude. We clearly observe that the
model performs poorly in Oceania, South Asia and South America. We also see
that the error is minimal around the latitude of North America and Europe,
while it increases in the Southern Hemisphere.

### 3.2. Identifying sources of bias

We attempt to correlate the performance of our geographical probes with
several factors. First, the dataset from (Gurnee and Tegmark, 2023) provides
each location with an estimate of the corresponding population count when
relevant. We also consider training data distribution as a potential factor of
heterogeneity. Finally, we consider latitude and longitude as potential
factors of bias.

To account for training data distribution, we look for exact string matches of
country names from the Gurnee and Tegmark (2023) dataset in an extract of The
Pile (Gao et al., 2020) containing 3.5 million samples
111https://huggingface.co/datasets/ola13/small-the_pile. We select this
dataset as it was used to pretrain the models from the Pythia suite (Biderman
et al., 2023) we evaluate in this section. We find 15 million matches,
covering 98% of the countries of the dataset.

We do not count occurrences of location names directly, as matching locations
on the basis of their names does not account for named entity ambiguity. An
example of ambiguous location name is Fully, which is a town in Switzerland.
An exact match strategy overestimates by large margins the occurrence count of
this location, because of the corresponding English word fully. Disambiguation
techniques have been designed (Hoffart et al., 2011; Orr et al., 2020), but we
prefer to avoid the risk of bias propagation and the cost of using such
methods on a large corpus.

Figure 6: Pearson correlation coefficients of various factors with location-
wise MSE, for several Pythia model sizes. *: Tests that yielded p-values above
0.05.

We display Pearson correlations between each of the aforementioned factors and
the entity-level MSE for each model size in Figure 6. As in 1(a), we observe
that the error on coordinates prediction is negatively correlated with the
latitude, i.e. southern locations are less accurately identified. This
correlation slowly decays as the model size increases. Meanwhile, longitude
seems to be mildly correlated with the probe performance.

Interestingly, the population count is not correlated with the error level.
The occurrence count of the location country is negatively correlated with the
error level, thus showing that the more country names appear in the training
dataset, the more the probes are able to recover coordinates from locations in
these countries. However, this correlation is mild and even below the
significance threshold for the smallest model.

We also measure the correlation between country occurrences and other metrics
to account for the bias inherent to the data. We observe that country name
occurrences are positively correlated with latitude with a p-value of 0.06,
and not correlated with the longitude. More importantly, the population count
of a country and the count of this country name in the data are heavily
correlated (factor of +0.52 and p-value of 3e-23). Thus, even though the data
seems guided by demographic factors, this is not the case of the model’s
representations.

## 4\. Discussion

We believe that quantifying sociocultural bias in representations of language
models and pretraining datasets allows to better understand the roots of the
biases that can be observed during generation.

Bender et al. (2021) discuss the relevance of scaling models to ever larger
magnitudes, with regard to environmental and financial costs. Our study shows
that scale can also increase language modeling bias when it comes to
geographical representation, given a pretraining dataset. We advocate in favor
of measuring and mitigating bias in pretraining datasets to avoid scaling bias
along with performance.

## Conclusion

In this study, we show that a wide variety of language models, varying in
architecture and sizes, implicitly embed geographical data to some extent. As
we consider larger models, the performance of geographical probes consistently
increases towards levels shown in Gurnee and Tegmark (2023).

We show numerically that the geographical probe performance is correlated with
latitude across model sizes, but also with the number of occurrence of
corresponding country names in the pretraining data. Conversely, the
population count of the location seems uncorrelated with the probe
performance. This indicates that a minority of people benefit from better
geographical understanding when using language models, which does not maximize
the social utility of these systems.

While it may initially seem that this performance increase mitigates
heterogeneity between Southern and Northern countries, we actually show that
larger models tend to be more biased according to the Gini coefficient taken
on prediction error. This tends to show that scaling language models can
amplify discrepancies in their geographical knowledge.

## Acknowledgements

This work was funded by the last authors’ chair in the PRAIRIE institute,
funded by the French national agency ANR as part of the “Investissements
d’avenir” program under the reference ANR-19-P3IA-0001. We thank Stella
Biderman for her insightful advice.

## Bibliographical References

##

  * Bender et al. (2021) Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the dangers of stochastic parrots: Can language models be too big? In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’21, page 610–623, New York, NY, USA. Association for Computing Machinery.
  * Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, Usvsn Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. 2023.  Pythia: A suite for analyzing large language models across training and scaling.  In _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 2397–2430. PMLR.
  * Black et al. (2021) Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021.  GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow.  If you use this software, please cite it using these metadata.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  ELECTRA: pre-training text encoders as discriminators rather than generators.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.
  * Conneau et al. (2018) Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. 2018.  What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2126–2136, Melbourne, Australia. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Faisal and Anastasopoulos (2021) Fahim Faisal and Antonios Anastasopoulos. 2021.  Investigating post-pretraining representation alignment for cross-lingual question answering.  In _Proceedings of the 3rd Workshop on Machine Reading for Question Answering_ , pages 133–148, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Faisal and Anastasopoulos (2022) Fahim Faisal and Antonios Anastasopoulos. 2022.  Geographic and geopolitical biases of language models.  In _arXiv:2212.10408_ , Online.
  * Faisal et al. (2022) Fahim Faisal, Yinkai Wang, and Antonios Anastasopoulos. 2022.  Dataset geography: Mapping language data to language users.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3381–3411, Dublin, Ireland. Association for Computational Linguistics.
  * Gini (1912) Corrado Gini. 1912.  _Variabilità e mutabilità: contributo allo studio delle distribuzioni e delle relazioni statistiche.[Fasc. I.]_.  Tipogr. di P. Cuppini.
  * Gupta et al. (2015) Abhijeet Gupta, Gemma Boleda, Marco Baroni, and Sebastian Padó. 2015.  Distributional vectors encode referential attributes.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 12–21, Lisbon, Portugal. Association for Computational Linguistics.
  * Gurnee and Tegmark (2023) Wes Gurnee and Max Tegmark. 2023.  Language models represent space and time.
  * He et al. (2020) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020.  Deberta: Decoding-enhanced BERT with disentangled attention.  _CoRR_ , abs/2006.03654.
  * Hoerl and Kennard (1970) Arthur E. Hoerl and Robert W. Kennard. 1970.  Ridge regression: Biased estimation for nonorthogonal problems.  _Technometrics_ , 12(1):55–67.
  * Hoffart et al. (2011) Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Hagen Fürstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011.  Robust disambiguation of named entities in text.  In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing_ , pages 782–792, Edinburgh, Scotland, UK. Association for Computational Linguistics.
  * Hupkes and Zuidema (2018) Dieuwke Hupkes and Willem Zuidema. 2018.  Visualisation and ’diagnostic classifiers’ reveal how recurrent and recursive neural networks process hierarchical structure (extended abstract).  In _Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18_ , pages 5617–5621. International Joint Conferences on Artificial Intelligence Organization.
  * Iskander et al. (2023) Shadi Iskander, Kira Radinsky, and Yonatan Belinkov. 2023.  Shielded representations: Protecting sensitive attributes through iterative gradient-based projection.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 5961–5977, Toronto, Canada. Association for Computational Linguistics.
  * Jawahar et al. (2019) Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019.  What does BERT learn about the structure of language? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3651–3657, Florence, Italy. Association for Computational Linguistics.
  * Köhn (2015) Arne Köhn. 2015.  What’s in an embedding? analyzing word embeddings through multilingual evaluation.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 2067–2073, Lisbon, Portugal. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Louwerse and Benesh (2012) Max M. Louwerse and Nick Benesh. 2012.  Representing spatial structure through maps and language: Lord of the rings encodes the spatial structure of middle earth.  _Cognitive Science_ , 36(8):1556–1569.
  * Orr et al. (2020) Laurel Orr, Megan Leszczynski, Simran Arora, Sen Wu, Neel Guha, Xiao Ling, and Christopher Re. 2020.  Bootleg: Chasing the tail with self-supervised named entity disambiguation.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Ravfogel et al. (2020) Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020\.  Null it out: Guarding protected attributes by iterative nullspace projection.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7237–7256, Online. Association for Computational Linguistics.
  * Shi et al. (2016) Xing Shi, Inkit Padhi, and Kevin Knight. 2016.  Does string-based neural MT learn source syntax? In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1526–1534, Austin, Texas. Association for Computational Linguistics.
  * Shliazhko et al. (2023) Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina. 2023.  mgpt: Few-shot learners go multilingual.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  Llama: Open and efficient foundation language models.
  * Turc et al. (2019) Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Well-read students learn better: On the importance of pre-training compact models.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022.  Opt: Open pre-trained transformer language models.
  * Zhao et al. (2018) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018\.  Gender bias in coreference resolution: Evaluation and debiasing methods.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 15–20, New Orleans, Louisiana. Association for Computational Linguistics.
  * Zhou et al. (2021) Kaitlyn Zhou, Kawin Ethayarajh, and Dan Jurafsky. 2021.  Frequency-based distortions in contextualized word embeddings.

## Language Resource References

##

  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020.  The pile: An 800gb dataset of diverse text for language modeling.


$\displaystyle\left|\frac{(2h)^{d}}{\\#\mathcal{I}(x_{0})}\sum_{i\in\mathcal{I}(x_{0})}g(x_{i})-\int_{\mathcal{S}(x_{0})}g(x)\mathrm{d}x\right|$
$\displaystyle\leq\sum_{i\in\mathcal{I}(x_{0})}\int_{\mathcal{X}_{i}(x_{0})}\left|g(x_{i})-g(x)\right|\mathrm{d}x$
(258) $\displaystyle\leq c_{d}L\frac{T}{n_{0}}h^{d}\,,$ (259)

where $\mathcal{X}_{i}(x_{0})$, $i\in\mathcal{I}(x_{0})$ form a suitable
partition of $\mathcal{S}(x_{0})$ with $x_{i}\in\mathcal{X}_{i}(x_{0})$ and
$\operatorname{\mathsf{diam}}(\mathcal{X}_{i}(x_{0}))\leq c_{d}T/n_{0}$.
Applying this to
$g_{h}(x):=\left(v^{\\!\top}\\!\psi(z(x))\right)^{2}K(\left\|z(x)\right\|_{2})$,
we obtain

$\left|\frac{nh^{d}(2h)^{d}}{T^{d}\\#\mathcal{I}(x_{0})}v^{\\!\top}\\!B(x_{0})v-\int_{\mathcal{S}(x_{0})}g_{h}(x)\mathrm{d}x\right|\leq
c_{d}\frac{T}{n_{0}}h^{d}\sup_{x\in[0,T]^{d}}\left\|Dg_{h}(x)\right\|_{\mathsf{op}}\,.$
(260)

As $g_{h}(x)=g_{1}(x/h)$, with Lipschitz-continuity of $K$ by StrictKernel,

$\sup_{x\in\mathcal{S}_{h}(x_{0})}\left\|Dg_{h}(x)\right\|_{\mathsf{op}}=h^{-1}\sup_{x\in\mathcal{S}_{1}(x_{0})}\left\|Dg_{1}(x)\right\|_{\mathsf{op}}\leq
c_{\ell,K}h^{-1}\,.$ (261)

Note that
$\lfloor(2hn_{0}/T)\rfloor^{d}\leq\\#\mathcal{I}(x_{0})(\lfloor(2hn_{0}/T)\rfloor+1)^{d}$.
Thus,

$\left|v^{\\!\top}\\!B(x_{0})v-h^{-d}\int_{\mathcal{S}(x_{0})}g_{h}(x)\mathrm{d}x\right|\leq
c_{d,\ell,K}\frac{T}{n_{0}h}\,.$ (262)

This means, if $T/(n_{0}h)$ is small enough, it only remains to show a
positive lower bound on

$h^{-d}\int_{\mathcal{S}(x_{0})}g_{h}(x)\mathrm{d}x=\int_{\mathbb{R}^{d}}\left(v^{\\!\top}\\!\psi\\!\left(x\right)\right)^{2}K(\left\|x\right\|_{2})\mathrm{d}x$
(263)

With the same arguments as in the proof of [Tsy09, Lemma 1.4] and using
StrictKernel, one can show

$\int_{\mathbb{R}^{d}}\left(v^{\\!\top}\\!\psi\\!\left(x\right)\right)^{2}K(\left\|x\right\|_{2})\mathrm{d}x\geq
c_{d,\ell,K}>0\,.$ (264)

∎

##  Appendix C Polynomial Interpolation

###  C.1 Univariate Polynomial Interpolation

######  Proposition C.1.

Let $\ell\in\mathbb{N}_{0}$. Let
$u\in\mathcal{D}^{\ell+1}(\mathbb{R},\mathbb{R})$. Let
$(t_{i})_{i\in{\left\llbracket 0,\ell\right\rrbracket}}$ with
$t_{0}<\dots<t_{\ell}$. Let $p\in\mathcal{P}_{1,\ell}$ be the (Lagrange)
interpolation polynomial of $(t_{i},u(t_{i}))_{i\in{\left\llbracket
0,\ell\right\rrbracket}}$. Let $k\in{\left\llbracket 0,\ell\right\rrbracket}$.
Then,

$\sup_{t\in[t_{0},t_{\ell}]}\left|D^{k}u(t)-D^{k}p(t)\right|\leq\frac{1}{k!(\ell+1-k)!}\sup_{t\in[t_{0},t_{\ell}]}\left|D^{k}\omega(t)\right|\sup_{t\in[t_{0},t_{\ell}]}\left|D^{\ell+1}u(t)\right|\,,$
(265)

where $\omega(t):=\prod_{i=0}^{\ell}(t_{i}-t)$.

###### Proof.

[How91, Theorem 2 and 3]. ∎

In following lemma, we apply this result to the uniform grid
$t_{i}=i\Delta\\!t$.

######  Lemma C.2.

Let $\ell\in\mathbb{N}_{0}$. Let $\Delta\\!t\in\mathbb{R}_{>0}$. Let
$u\in\mathcal{D}^{\ell+1}(\mathbb{R},\mathbb{R})$. Let
$p\in\mathcal{P}_{1,\ell}$ be the interpolation polynomial of
$(i\Delta\\!t,u(i\Delta\\!t))_{i\in{\left\llbracket 0,\ell\right\rrbracket}}$.
Let $k\in{\left\llbracket 0,\ell\right\rrbracket}$. Then

$\sup_{t\in[0,\ell\Delta\\!t]}\left|D^{k}u(t)-D^{k}p(t)\right|\leq
c_{k,\ell}\Delta\\!t^{\ell+1-k}\left|D^{\ell+1}u\right|_{\infty}\,.$ (266)

###  C.2 Univariate Polynomial Comparison

######  Lemma C.3.

Let $p_{n},\tilde{p}_{n}\in\mathcal{P}_{1,\ell-1}$. Let
$h_{n},\delta_{n}\in\mathbb{R}_{>0}$. Let $c_{1},\dots,c_{\ell}\in\mathbb{R}$
be distinct. Let $c_{\mathsf{min}},c_{\mathsf{max}}\in\mathbb{R}_{>0}$.
Assume, for all $i=1,\dots,\ell$,

$\left|p_{n}(c_{i}h_{n})-\tilde{p}_{n}(c_{i}h_{n})\right|\preccurlyeq\delta_{n}\,.$
(267)

Then

$\sup_{t\in[-c_{\mathsf{min}}h_{n},c_{\mathsf{max}}h_{n}]}\left|D^{k}p_{n}(t)-D^{k}\tilde{p}_{n}(t)\right|\preccurlyeq
h_{n}^{-k}\delta_{n}$ (268)

for $k=0,\dots,\ell-1$.

###### Proof.

Let $p_{n}(t)-\tilde{p}_{n}(t)=:q(t)=\sum_{k=1}^{\ell}a_{k}t^{k-1}$. Let
$y_{i}:=q(c_{i}h_{n})$. Then $a=V^{-1}y$, where
$V\in\mathbb{R}^{\ell\times\ell}$ is the Vandermonde matrix
$V_{i,j}=(c_{i}h_{n})^{j-1}$. By the inverse formula of the Vandermonde
matrix, $\left|(V^{-1})_{i,j}\right|\preccurlyeq h_{n}^{1-i}$. Thus,

$\left|a_{i}\right|=\left|\sum_{j=1}^{\ell}(V^{-1})_{i,j}y_{j}\right|\preccurlyeq
h_{n}^{1-i}\delta_{n}\,.$ (269)

Furthermore, for $c\in[-c_{\mathsf{min}},c_{\mathsf{max}}]$,

$\displaystyle\left|D^{k}q(ch_{n})\right|$
$\displaystyle=\left|\sum_{i=1}^{\ell-k}a_{i+k}\left(ch_{n}\right)^{i-1}\prod_{s=1}^{k}(i+s)\right|$
(270)
$\displaystyle\preccurlyeq\sum_{i=1}^{\ell-k}\left|c\right|^{i-1}h_{n}^{-k}\delta_{n}$
(271) $\displaystyle\preccurlyeq h_{n}^{-k}\delta_{n}\,.$ (272)

∎

###  C.3 Multivariate Polynomial Interpolation

Recall Definition 4.11 for the objects $\psi(x)$, $\Psi(\mathbf{x})$, and
$I(\mathbf{x},\mathbf{y},x)$.

######  Lemma C.4 (Linear Transformation).

Let $\ell,d\in\mathbb{N}$. Set $N:=\dim(\mathcal{P}_{d,\ell})$. Let
$b\in\mathbb{R}^{d}$. Let $B\in\mathbb{R}^{d\times d}$. Then there is
$A\in\mathbb{R}^{N\times N}$ such that $\psi(Bx+b)=A\psi(x)$ for all
$x\in\mathbb{R}^{d}$. Furthermore, if $B$ is invertible, then so is $A$.

###### Proof.

First we need to define
$\tilde{\psi}_{\ell}\colon\mathbb{R}^{d}\to\mathbb{R}^{M}$ as a variant of the
monomial transformation $\psi_{\ell}\colon\mathbb{R}^{d}\to\mathbb{R}^{N}$,
where commutativity is not applied, i.e., $\tilde{\psi}_{\ell}(x)$ contains a
dimension for $x_{1}x_{2}$ and a different one for $x_{2}x_{1}$. The monomial
transform without commutativity $\tilde{\psi}_{\ell}$ can be expressed using
the Kronecker product, see, e.g., [HJ94, Chapter 4].

Denote by $\otimes$ the operator for the Kronecker product. For
$x\in\mathbb{R}^{d}$, $k\in\mathbb{N}_{0}$, define $x^{\otimes
k}\in\mathbb{R}^{d^{k}}$ as follows: Let $x^{\otimes 0}:=1$. For
$k\in\mathbb{N}$, define $x^{\otimes k}=x^{\otimes(k-1)}\otimes x$. Denote

$M:=\sum_{k=0}^{\ell}d^{k}=\frac{d^{\ell+1}-1}{d-1}\,.$ (273)

Define the monomial transform without commutativity
$\tilde{\psi}_{\ell}\colon\mathbb{R}^{d}\to\mathbb{R}^{M}$ as

$\tilde{\psi}_{\ell}(x):=\begin{pmatrix}x^{\otimes 0}\\\ x^{\otimes 1}\\\
\vdots\\\ x^{\otimes\ell}\end{pmatrix}\,.$ (274)

Our intermediate goal is to find a matrix $\tilde{A}\in\mathbb{R}^{M\times M}$
with the property

$\tilde{\psi}(Bx+b)=\tilde{A}\tilde{\psi}(x)\,.$ (275)

For this, we first recall some properties of the Kronecker product: The
Kronecker product is bilinear. In particular,

$M_{1}\otimes(M_{2}+M_{3})=M_{1}\otimes M_{2}+M_{1}\otimes M_{3}\ \text{and}\
(M_{1}+M_{2})\otimes M_{3}=M_{1}\otimes M_{3}+M_{2}\otimes M_{3}$ (276)

for compatible matrices $M_{1},\dots,M_{3}$. Moreover, The mixed-product
property of the Kronecker product states

$\left(M_{1}\otimes M_{2}\right)\left(M_{3}\otimes
M_{4}\right)=\left(M_{1}M_{3}\right)\otimes\left(M_{2}M_{4}\right)$ (277)

for compatible matrices $M_{1},\dots,M_{4}$. The Kronecker product is not
commutative, but $M_{1}\otimes M_{2}$ and $M_{2}\otimes M_{1}$ are permutation
equivalent: There are permutation matrices $P,Q$ such that

$M_{1}\otimes M_{2}=P(M_{2}\otimes M_{1})Q$ (278)

for compatible matrices $M_{1},M_{2}$.

Now, using the mixed-product property, we obtain by induction

$\left(Bx\right)^{\otimes k}=B^{\otimes k}x^{\otimes k}\,.$ (279)

Next, the bilinearity and the permutation equivalence imply, also using
induction, there are matrices $\bar{A}_{j}\in\mathbb{R}^{d^{k}\times d^{j}}$

$\left(b+x\right)^{\otimes k}=\sum_{j=0}^{k}\bar{A}_{j}x^{\otimes k}\,.$ (280)

Thus there is are matrices $\tilde{A}_{0},\tilde{A}_{1}\in\mathbb{R}^{M\times
M}$ such that

$\tilde{\psi}(x+b)=\tilde{A}_{0}\tilde{\psi}(x)\qquad\text{and}\qquad\tilde{\psi}(Bx)=\tilde{A}_{1}\tilde{\psi}(x)$
(281)

and, hence,

$\tilde{\psi}(Bx+b)=\tilde{A}\tilde{\psi}(x)$ (282)

with $\tilde{A}=\tilde{A}_{0}\tilde{A}_{1}$. From (279), we can see that
$\tilde{A}_{1}$ is block diagonal with blocks $B^{\otimes k}$. From (280), we
can see that $\tilde{A}_{0}$ is block triangular with identity blocks
$I_{d^{k}}\in\mathbb{R}^{d^{k}\times d^{k}}$ on the diagonal. Hence,
$\tilde{A}$ is block triangular with blocks $B^{\otimes k}$ on the diagonal.
For the determinant of the Kronecker product, we have

$\det(M_{1}\otimes M_{2})=\det(M_{1})^{d_{2}}\det(M_{2})^{d_{1}}$ (283)

for $M_{1}\in\mathbb{R}^{d_{1}\times d_{1}}$ and
$M_{2}\in\mathbb{R}^{d_{2}\times d_{2}}$. Thus,

$\det\\!\left(\tilde{A}\right)=\prod_{k=0}^{\ell}\det\\!\left(B^{\otimes
k}\right)=\det(B)^{\kappa}\,,$ (284)

where

$\kappa:=\sum_{k=0}^{\ell}kd^{k-1}=\begin{cases}\frac{\ell(1+\ell)}{2}&\text{
if }d=1\\\ \frac{1+\ell d^{\ell+1}-(\ell+1)d^{\ell}}{(d-1)^{2}}&\text{
otherwise.}\end{cases}$ (285)

To see this, show inductively using (283) that

$\det(B^{\otimes k}):=\det(B)^{kd^{k-1}}\,.$ (286)

Now, we have the existence of $\tilde{A}\in\mathbb{R}^{M\times M}$ with

$\tilde{\psi}(Bx+b)=\tilde{A}\tilde{\psi}(x)$ (287)

and the property that if $B$ is invertible so is $\tilde{A}$. In the final
step, we covert from $\tilde{\psi}$ to $\psi$ via a linear transformation of
full rank: Let $Z\in\mathbb{R}^{N\times M},\tilde{Z}\in\mathbb{R}^{M\times N}$
be linear mappings that fulfill

$\psi_{\ell}(x)=Z\tilde{\psi}_{\ell}(x)\qquad\text{and}\qquad\tilde{\psi}_{\ell}(x)=\tilde{Z}\psi_{\ell}(x)$
(288)

for all $x\in\mathbb{R}^{d}$. Note that the rank of $Z$ and of $\tilde{Z}$ is
$N$, i.e., both matrices have full rank. Set

$A:=Z\tilde{A}\tilde{Z}\,.$ (289)

Then

$\psi(Bx+b)=Z\tilde{\psi}(Bx+b)=Z\tilde{A}\tilde{\psi}(x)=Z\tilde{A}\tilde{Z}\psi(x)=A\psi(x)\,.$
(290)

Moreover, if $B$ is invertible, so is $\tilde{A}$. Then, as $\tilde{A}$,
$\tilde{Z}$, and $Z$ have full rank, $A$ also has full rank and is invertible.
∎

######  Lemma C.5 (Linear Transformation).

Let $\ell,d_{\mathsf{x}},d_{\mathsf{y}}\in\mathbb{N}$. Set
$N:=\dim(\mathcal{P}_{d_{\mathsf{x}},\ell})$. Let
$b\in\mathbb{R}^{d_{\mathsf{x}}}$. Let $B\in\mathbb{R}^{d_{\mathsf{x}}\times
d_{\mathsf{x}}}$. Let $\mathbf{x}\in(\mathbb{R}^{d_{\mathsf{x}}})^{N}$ and
$\mathbf{y}\in(\mathbb{R}^{d_{\mathsf{y}}})^{N}$. Let
$x\in\mathbb{R}^{d_{\mathsf{x}}}$. Assume $B$ and $\Psi(\mathbf{x})$ are
invertible. Then

  1. (i)

$\Psi(B\mathbf{x}+b)$ is invertible,

  2. (ii)

$I(\mathbf{x},\mathbf{y},x)=I(B\mathbf{x}+b,\mathbf{y},Bx+b)\,,$ (291)

  3. (iii)

$\psi(x)^{\\!\top}\\!\Psi(\mathbf{x})^{-1}=\psi(Bx+b)^{\\!\top}\\!\Psi(B\mathbf{x}+b)^{-1}\,.$
(292)

###### Proof.

Part (i) follows directly from Lemma C.4.

As the interpolation is done componentwise, it is enough to show the results
for the case $d_{\mathsf{y}}=1$. Let us assume $d_{\mathsf{y}}=1$. The
function $g\colon\mathbb{R}^{d_{\mathsf{x}}}\to\mathbb{R},\,x\mapsto
I(\mathbf{x},\mathbf{y},x)$ is the unique interpolation polynomial of
$(\mathbf{x},\mathbf{y})$ with degree at most $\ell$. The mapping
$\tilde{g}\colon\mathbb{R}^{d}\to\mathbb{R},x\mapsto
I(B\mathbf{x}+b,\mathbf{y},Bx+b)$ is a polynomial of degree at most $\ell$ and
fulfills $\tilde{g}(x_{k})=y_{k}$ for $k\in{\left\llbracket
N\right\rrbracket}$. Thus $\tilde{g}=g$. This shows (ii).

As (ii) is true for all $\mathbf{y}$ and
$I(\mathbf{x},\mathbf{y},x)=\psi(x)^{\\!\top}\\!\Psi(\mathbf{x})^{-1}\mathbf{y}$,
(iii) follows. ∎

######  Lemma C.6 (Approximation).

Let $\ell,d\in\mathbb{N}$. Set $N:=\dim(\mathcal{P}_{d,\ell})$. Let
$\mathbf{x}\in(\mathbb{R}^{d})^{N}$. Let $g\colon\mathbb{R}^{d}\to\mathbb{R}$
be $(\ell+1)$-times continuously differentiable and set
$L:=\sup_{x\in\mathbb{R}^{d}}\|D^{\ell+1}g(x)\|_{\mathsf{op}}$. Let
$\mathbf{y}=(g(x_{1}),\dots,g(x_{N}))$. Then

$\sup_{x\in\mathring{\mathsf{ch}}(\mathbf{x})}\left\|g(x)-I_{\ell}(\mathbf{x},\mathbf{y},x)\right\|_{2}\leq
LC_{\ell}(\mathbf{x})\operatorname{\mathsf{diam}}(\mathbf{x})^{\ell+1}\,,$
(293)

where $C_{\ell}(\mathbf{x})$ is defined as follows: For $k\in{\left\llbracket
N\right\rrbracket}$, let $e_{k}\in\mathbb{R}^{N}$ be the $k$-th standard base
vector. Define

$C_{\ell}(\mathbf{x}):=\begin{cases}\frac{1}{(\ell+1)!}\sum_{k=1}^{N}\sup_{x\in\mathring{\mathsf{ch}}(\mathbf{x})}\left|I(\mathbf{x},e_{k},x)\right|&\text{
if }\Psi(\mathbf{x})\text{ is invertible,}\\\ \infty&\text{
otherwise.}\end{cases}$ (294)

###### Proof.

[CR72, Theorem 2]. ∎

######  Lemma C.7.

Let $\ell,d\in\mathbb{N}$. Set $N:=\dim(\mathcal{P}_{d,\ell})$. If
$\Psi(\mathbf{x})$ is invertible,

$C_{\ell}(\mathbf{x})\leq\frac{N^{\frac{3}{2}}}{(\ell+1)!}\left\|\Psi(\eta_{\mathbf{x}}(\mathbf{x}))^{-1}\right\|_{\mathsf{op}}\,.$
(295)

###### Proof.

By Lemma C.5, we have

$\sup_{x\in\mathring{\mathsf{ch}}(\mathbf{x})}\left|I({\mathbf{x}},e_{j},x)\right|=\sup_{x\in\mathring{\mathsf{ch}}(\mathbf{x})}\left|I\\!\left(\eta_{\mathbf{x}}(\mathbf{x}),e_{j},\eta_{\mathbf{x}}(x)\right)\right|\leq\sup_{x\in\mathrm{B}^{d}(0,1)}\left|I\\!\left(\eta_{\mathbf{x}}(\mathbf{x}),e_{j},x\right)\right|\,.$
(296)

For $x\in\mathrm{B}^{d}(0,1)$,

$\left|I(\eta_{\mathbf{x}}(\mathbf{x}),e_{j},x)\right|\leq\|\psi(x)\|_{2}\left\|\Psi(\eta_{\mathbf{x}}(\mathbf{x}))^{-1}\right\|_{\mathsf{op}}\left\|e_{j}\right\|_{2}\leq\sqrt{N}\left\|\Psi(\eta_{\mathbf{x}}(\mathbf{x}))^{-1}\right\|_{\mathsf{op}}\,.$
(297)

Thus,

$\displaystyle C_{\ell}(\mathbf{x})$
$\displaystyle\leq\frac{1}{(\ell+1)!}\sum_{j=1}^{N}\sqrt{N}\left\|\Psi(\eta_{\mathbf{x}}(\mathbf{x}))^{-1}\right\|_{\mathsf{op}}$
(298)
$\displaystyle\leq\frac{N^{\frac{3}{2}}}{(\ell+1)!}\left\|\Psi(\eta_{\mathbf{x}}(\mathbf{x}))^{-1}\right\|_{\mathsf{op}}\,.$
(299)

∎

##  Appendix D Ordinary Differential Equations and Smoothness

###  D.1 Difference Bound

######  Lemma D.1.

Let $f\colon\mathbb{R}\to\mathbb{R}$ be Lipschitz continuous. Let
$x\in\mathbb{R}$. Set $u:=U(f,x,\cdot)$. Let $v\colon\mathbb{R}\to\mathbb{R}$
be differentiable with $v(0)\leq x$ and $\dot{v}(t)\leq f(v(t))$ for all
$t\in\mathbb{R}_{\geq 0}$. Then

$v(t)\leq u(t)$ (300)

for all $t\in\mathbb{R}_{\geq 0}$.

###### Proof.

Assume there is $t_{1}\in\mathbb{R}_{>0}$ with $v(t_{1})>u(t_{1})$. Then the
first intersection time $t_{0}:=\inf\\{t\in\mathbb{R}_{\geq 0}\mid
v(t)>u(t)\\}$ exists. As $v$ and $u$ are continuous, we have
$v(t_{0})=u(t_{0})$. Let $t_{2}=\sup\\{t\in\mathbb{R}_{\geq t_{0}}\mid\forall
s\in[t_{0},t]\colon v(s)\geq u(s)\\}$. Because of the existence of $t_{1}$ and
continuity of $u$ and $v$, we have $t_{2}>t_{0}$. Let $L\in\mathbb{R}_{>0}$ be
the Lipschitz constant of $f$. Set $\delta(t):=v(t)-u(t)$. Let
$t\in[t_{0},t_{2}]$. Then

$\displaystyle\delta(t)$
$\displaystyle=\int_{t_{0}}^{t}\dot{\delta}(s)\mathrm{d}s$ (301)
$\displaystyle\leq\int_{t_{0}}^{t}f(v(s))-f(u(s))\mathrm{d}s$ (302)
$\displaystyle\leq L\int_{t_{0}}^{t}\delta(s)\mathrm{d}s\,.$ (303)

As $\delta$ is continuous, there is
$t_{\mathsf{max}}\in\operatorname*{arg\,max}_{t\in[t_{0},\min(t_{0}+1/L,t_{2})]}\delta(t)$.
By definition of $t_{0}$ and $t_{\mathsf{max}}$, we have
$\delta(t_{\mathsf{max}})>0$. Thus, using $\delta(t_{0})=0$, the continuity of
$\delta$, and $t_{\mathsf{max}}-t_{0}\leq 1/L$, we obtain

$\delta(t_{\mathsf{max}})\leq
L\int_{t_{0}}^{t_{\mathsf{max}}}\delta(s)\mathrm{d}s<L\left(t_{\mathsf{max}}-t_{0}\right)\delta(t_{\mathsf{max}})\leq\delta(t_{\mathsf{max}})\,,$
(304)

a contradiction. ∎

######  Lemma D.2.

Let $d\in\mathbb{N}$, $L\in\mathbb{R}_{>0}$. Let $f,g\in\Sigma^{d\to d}(1,L)$.
Let $x_{0}\in\mathbb{R}^{d}$. Denote $u:=U(f,x_{0},\cdot)$,
$v:=U(g,x_{0},\cdot)$. Then, for $t\in\mathbb{R}_{\geq 0}$,

$\left\|u(t)-v(t)\right\|_{2}\leq\frac{\exp(Lt)-1}{L}\left|f-g\right|_{\infty}\,.$
(305)

In particular, if $t\in[0,\frac{1}{L}]$, then

$\left\|u(t)-v(t)\right\|_{2}\leq 2t\left|f-g\right|_{\infty}\,.$ (306)

###### Proof.

Set $C:=\left|f-g\right|_{\infty}$. We represent the right-hand side of (305)
with the solution $z$ of an ODE: Define $z\colon\mathbb{R}_{\geq
0}\to\mathbb{R}$ as the solution of the ODE $\dot{z}(t)=Lz(t)+C$ with
$z(0)=0$. Then,

$z(t)=\frac{C}{L}\left(\exp(Lt)-1\right)\,.$ (307)

Next consider the left-hand side of (305). Define
$w(t):=\left\|u(t)-v(t)\right\|_{2}$. Then $w(0)=0$ and

$\displaystyle w(t)$
$\displaystyle=\left\|\int_{0}^{t}f(u(s))-g(v(s))\mathrm{d}s\right\|_{2}$
(308)
$\displaystyle\leq\left\|\int_{0}^{t}f(u(s))-f(v(s))\mathrm{d}s\right\|_{2}+\left\|\int_{0}^{t}f(v(s))-g(v(s))\mathrm{d}s\right\|_{2}$
(309) $\displaystyle\leq L\int_{0}^{t}w(s)\mathrm{d}s+Ct\,.$ (310)

Thus, Lemma D.1 yields, $w(t)\leq z(t)$. ∎

###  D.2 Smoothness of Solutions in Time

Let $d,\beta\in\mathbb{N}$. Let $L_{{\left\llbracket
0,\beta\right\rrbracket}}\subseteq\mathbb{R}_{>0}$. Assuming
$f\in\bar{\Sigma}^{d\to d}(\beta,L_{{\left\llbracket
0,\beta\right\rrbracket}})$ we want to find $\tilde{L}_{{\left\llbracket
1,\beta+1\right\rrbracket}}\subseteq\mathbb{R}_{>0}$ such that
$U(f,x,\cdot)\in\Sigma^{1\to d}(\beta+1,\tilde{L}_{{\left\llbracket
0,\beta+1\right\rrbracket}})$ with $\tilde{L}_{0}=\infty$.

As detailed in [But16, Chapter 3], higher order derivatives of $u$ can be
expressed using tree structures. To state this result in Theorem D.6 below, we
first need to introduce increasing trees (Definition D.3) and the derivative
operator associated to such a tree (Definition D.5).

In the following the term finite ordered set refers to a finite set with
strict total order that is denoted as $<$.

######  Definition D.3 (Increasing tree).

Let $S$ be a finite ordered set.

  1. (i)

An increasing tree $\mathcal{T}$ is a tuple $(S,E)$, where
$E\subseteq\\{(n_{1},n_{2})\mid n_{1},n_{2}\in S,\,n_{1}<n_{2}\\}$ with
$\\#E=\\#S-1$. We call $S$ the nodes, $E$ the edges, and $\min S$ the root.
Furthermore, denote $\\#\mathcal{T}:=\\#S$.

  2. (ii)

Let $\mathfrak{T}_{S}$ be the set of increasing trees with nodes $S$.

  3. (iii)

We define $[\mathcal{T}_{1},\dots,\mathcal{T}_{k}]$ as the increasing tree
that connects all trees $\mathcal{T}_{j}$ at their roots to a new root $s$:

Let $k\in\mathbb{N}_{0}$. Let $S_{0}=\\{s\\}$ and $S_{1},\dots,S_{k}$ be
disjoint finite ordered sets so that $S:=\bigcup_{j\in{\left\llbracket
0,k\right\rrbracket}}S_{k}$ is also ordered with $s=\min S$. Let
$\mathcal{T}_{1}=(S_{1},E_{1}),\dots,\mathcal{T}_{k}=(S_{k},E_{k})$ be
increasing trees. Define $[\mathcal{T}_{1},\dots,\mathcal{T}_{k}]:=(S,E)$,
where $E:=\bigcup_{j\in{\left\llbracket 0,k\right\rrbracket}}E_{j}$ and
$E_{0}:=\\{(s,\min S_{j})\mid j\in{\left\llbracket k\right\rrbracket}\\}$.

######  Lemma D.4.

Let $S$ be a finite ordered set. Then

$\\#\mathfrak{T}_{S}=(\\#S-1)!\,.$ (311)

###### Proof.

For $S=\\{s\\}$, we have $\mathfrak{T}_{S}=\\{(\\{s\\},\emptyset)\\}$. Hence,
$\\#\mathfrak{T}_{S}=(\\#S-1)!$ is true. Now assume $\\#S=k+1$, $\max S=s$,
and $S^{\prime}=S\setminus\\{s\\}$. Then $\mathfrak{T}_{S}$ can be constructed
from $\mathfrak{T}_{S^{\prime}}$, by adding the node $s$ to the node $n$ of
$\mathcal{T}$ for all trees $\mathcal{T}\in\mathfrak{T}_{S^{\prime}}$ and all
nodes $n\in S^{\prime}$. Hence,

$\\#\mathfrak{T}_{S}=\\#S^{\prime}\\#\mathfrak{T}_{S^{\prime}}=k\,\\#\mathfrak{T}_{S^{\prime}}\,.$
(312)

The desired result now follows by induction. ∎

######  Definition D.5 (Tree derivative operator).

Let $S$ be a finite ordered set. Let $\mathcal{T}\in\mathfrak{T}_{S}$. Let
$k\in\mathbb{N}_{0}$. Assume
$\mathcal{T}=[\mathcal{T}_{1},\dots,\mathcal{T}_{k}]$. Let $d\in\mathbb{N}$.
Let $f\in\mathcal{D}^{k}(\mathbb{R}^{d},\mathbb{R}^{d})$. Let
$x\in\mathbb{R}^{d}$. Define the tree derivative of $f$ at $x$, denoted as
$D^{\mathcal{T}}f(x)$, as

$D^{\mathcal{T}}f(x):=D^{k}f(x)[D^{\mathcal{T}_{1}}f(x),\dots,D^{\mathcal{T}_{k}}f(x)]\,.$
(313)

######  Theorem D.6.

Let $d,k\in\mathbb{N}$. Let
$f\in\mathcal{D}^{k-1}(\mathbb{R}^{d},\mathbb{R}^{d})$. Let
$x\in\mathbb{R}^{d}$. Assume $f$ is Lipschitz continuous. Set
$u:=U(f,x,\cdot)$. Then

$D^{k}u(t)=\sum_{\mathcal{T}\in\mathfrak{T}_{{\left\llbracket
k\right\rrbracket}}}D^{\mathcal{T}}f(u(t))\,.$ (314)

###### Proof.

[But16, Theorem 311B]. ∎

######  Corollary D.7.

Let $d,\beta\in\mathbb{N}$. Let $L_{{\left\llbracket
0,\beta\right\rrbracket}}\subseteq\mathbb{R}_{>0}$. Let
$f\in\bar{\Sigma}^{d\to d}(\beta,L_{{\left\llbracket
0,\beta\right\rrbracket}})$. Let $\beta\in\mathbb{N}$. Let
$x_{0}\in\mathbb{R}^{d}$. Set $u:=U(f,x_{0},\cdot)$. Set
$L:=\sup_{k=0,\dots,\beta}L_{k}$. Then, for all
$k\in{\left\llbracket\beta+1\right\rrbracket}$,

$\left|D^{k}u\right|_{\infty}\leq(k-1)!L^{k}\,.$ (315)

###### Proof.

Let $S$ be a finite ordered set. Let $\mathcal{T}\in\mathfrak{T}_{S}$. We want
to show by induction that

$\left|D^{\mathcal{T}}f\right|_{\infty}\leq L^{\\#\mathcal{T}}\,.$ (316)

If $\\#\mathcal{T}=1$,
$\left|D^{\mathcal{T}}f\right|_{\infty}=\left|f\right|_{\infty}\leq L$. Now,
assume $\mathcal{T}=[\mathcal{T}_{1},\dots,\mathcal{T}_{\kappa}]$ and
$\left|D^{\mathcal{T}_{j}}f\right|_{\infty}\leq L^{\\#{\mathcal{T}_{j}}}$.
Then,

$\displaystyle\left|D^{\mathcal{T}}f\right|_{\infty}$
$\displaystyle=\left|D^{\kappa}f(\cdot)[D^{\mathcal{T}_{1}}f(\cdot),\dots,D^{\mathcal{T}_{\kappa}}f(\cdot)]\right|_{\infty}$
(317) $\displaystyle\leq
L\prod_{j\in{\left\llbracket\kappa\right\rrbracket}}\left|D^{\mathcal{T}_{j}}f\right|_{\infty}$
(318) $\displaystyle\leq L^{1+\sum_{j=1}^{\kappa}\\#\mathcal{T}_{j}}$ (319)
$\displaystyle=L^{\\#\mathcal{T}}\,.$ (320)

Hence, we have shown (316). Using Theorem D.6, (316), and Lemma D.4, we obtain

$\displaystyle\left|D^{k}u\right|_{\infty}$
$\displaystyle\leq\sum_{\mathcal{T}\in\mathfrak{T}_{{\left\llbracket
k\right\rrbracket}}}\left|D^{\mathcal{T}}f(u(\cdot))\right|_{\infty}$ (321)
$\displaystyle\leq\\#\mathfrak{T}_{{\left\llbracket k\right\rrbracket}}L^{k}$
(322) $\displaystyle\leq(k-1)!L^{k}\,.$ (323)

∎

###  D.3 Smoothness of Solutions in the Initial Conditions

We state a simple consequence of the basic form of Grönwall’s inequality,
which we will make use of below.

######  Lemma D.8 (Grönwall’s inequality).

Let $d\in\mathbb{N}$. Let $u\colon\mathbb{R}\to\mathbb{R}^{d}$ be
differentiable. Let $L\in\mathbb{R}_{\geq 0}$. Assume

$\left\|\dot{u}(t)\right\|_{2}\leq L\left\|u(t)\right\|_{2}\qquad\text{for
}t\in\mathbb{R}\,.$ (324)

Then, for all $t\in\mathbb{R}$, we have

$\displaystyle\left\|u(t)\right\|_{2}$
$\displaystyle\leq\left\|u(0)\right\|_{2}\exp(L\left|t\right|)\,,$ (325)
$\displaystyle\left\|u(t)-u(0)\right\|_{2}$
$\displaystyle\leq\left\|u(0)\right\|_{2}\left(\exp(L\left|t\right|)-1\right)\,.$
(326)

###### Proof.

We assume below that $t>0$ and $u(t)\neq 0$ for all $s\in[0,t]$. If $t<0$,
consider $\tilde{u}(t)=u(-t)$ instead. If there are $s\in[0,t]$ with $u(s)=0$,
the proof applies to all intervals with $[a,b]$ with $u(\tilde{s})\neq 0$ for
$\tilde{s}\in[a,b]$, which yields an upper bound that is at least as strong as
claimed.

Assume $t>0$ and $u(t)\neq 0$ for all $s\in[0,t]$. Set
$v(t):=\left\|u(t)\right\|_{2}$. Then

$\dot{v}(t)=\frac{u(t)^{\\!\top}\\!\dot{u}(t)}{\left\|u(t)\right\|_{2}}\leq\left\|\dot{u}(t)\right\|_{2}\leq
L\left\|u(t)\right\|_{2}=Lv(t)\,.$ (327)

Thus, by Grönwall’s inequality in its standard form, we have

$v(t)\leq v(0)\exp(Lt)\,,$ (328)

which is the first inequality that we want to prove. Now, we can calculate

$\displaystyle\left\|u(t)-u(0)\right\|_{2}$
$\displaystyle\leq\int_{0}^{t}\left\|\dot{u}(s)\right\|_{2}\mathrm{d}s$ (329)
$\displaystyle\leq L\int_{0}^{t}v(s)\mathrm{d}s$ (330) $\displaystyle\leq
L\left\|u(0)\right\|_{2}\int_{0}^{t}\exp(Ls)\mathrm{d}s$ (331)
$\displaystyle=\left\|u(0)\right\|_{2}\left(\exp(Lt)-1\right)\,.$ (332)

∎

######  Lemma D.9 (Derivative with respect to initial conditions).

Let $d\in\mathbb{N}$. Let $L_{1}\in\mathbb{R}_{>0}$. Let
$f\in\bar{\Sigma}(1,\infty,L_{1})$. Then, for all $t\geq 0$,

$\left|DU(f,\cdot,t)\right|_{\infty}\leq\exp(L_{1}t)\quad\text{and}\quad\left|D\Upsilon(f,t,\cdot)\right|_{\infty}\leq\exp(L_{1}t)-1\,.$
(333)

###### Proof.

Let $v\in\mathbb{D}_{d}$ be a direction. Set $u(t):=u_{x}(t):=D_{x,v}U(f,x,t)$
denote the derivative of the solution $U(f,x,t)$ with respect to the initial
conditions $x$. Then, using the chain rule Lemma A.5 (ii),

$\displaystyle\dot{u}(t)$ $\displaystyle=D_{x,v}\dot{U}(f,x,t)$ (334)
$\displaystyle=D_{x,v}\left(f(U(f,x,t))\right)$ (335)
$\displaystyle=Df(U(f,x,t))[D_{x,v}U(f,x,t)]$ (336)
$\displaystyle=Df(U(f,x,t))[u(t)]\,.$ (337)

Furthermore,

$u(0)=D_{x,v}U(f,x,0)=v\,.$ (338)

We have established the differential inequality

$\left\|\dot{u}(t)\right\|_{2}\leq\left\|(Df)(U(f,x,t))[u(t)]\right\|_{2}\leq
L_{1}\left\|u(t)\right\|_{2}$ (339)

with $\|u(0)\|_{2}=1$. Thus, by Lemma D.8,

$\left\|u(t)\right\|_{2}\leq\exp(L_{1}\left|t\right|)\,.$ (340)

Hence,

$\displaystyle\left|DU(f,\cdot,t)\right|_{\infty}$
$\displaystyle=\sup_{v\in\mathbb{D}_{d}}\sup_{x\in\mathbb{R}^{d}}\left\|D_{x,v}U(f,x,t)\right\|_{2}$
(341) $\displaystyle\leq\exp(L_{1}t)\,.$ (342)

Set $u^{\circ}(t):=u(t)-u(0)$. Then, $\dot{u}^{\circ}=\dot{u}$ and
$u^{\circ}(0)=0$. Thus, using Lemma D.8 again,

$\left\|u^{\circ}(t)\right\|_{2}\leq\exp(L_{1}t)-1\,.$ (343)

We obtain,

$\left|D\Upsilon(f,t,\cdot)\right|_{\infty}=\left|DU(f,\cdot,t)-DU(f,\cdot,0)\right|_{\infty}\leq\exp(L_{1}t)-1\,.$
(344)

∎

For higher order derivatives of $U(f,x,t)$ with respect to $t$, we used the
tree derivatives in Theorem D.6. For higher order derivatives of $U(f,x,t)$
with respect to $x$, we introduce partition derivatives.

######  Definition D.10.

  1. (i)

Let $k\in\mathbb{N}$. Let $\mathbb{A}_{k}$ be the set of partitions of
${\left\llbracket k\right\rrbracket}$.

  2. (ii)

Let $k\in\mathbb{N}$. Let
$\mathbf{v}=(v_{1},\dots,v_{k})\in\mathbb{D}_{d}^{k}$. Define the set of
partitions of $\mathbf{v}$, as

$\mathbb{A}(\mathbf{v}):=\left\\{(v_{A_{1}},\dots,v_{A_{\ell}})\,\kern-1.2pt\mathopen{}\left|\vphantom{(v_{A_{1}},\dots,v_{A_{\ell}})}\,(A_{1},\dots,A_{\ell})\in\mathbb{A}_{k}\right.\mathclose{}\kern-1.2pt\right\\}\,.$
(345)

  3. (iii)

Let $k\in\mathbb{N}$. Let $\mathbf{v}\in\mathbb{D}_{d}^{k}$. Let
$f,g\in\mathcal{D}^{k}(\mathbb{R}^{d},\mathbb{R}^{d})$. Let
$\pi=(\mathbf{v}_{1},\dots,\mathbf{v}_{\ell})\in\mathbb{A}(\mathbf{v})$.
Denote the partition derivative of $f\circ g$ as

$D^{\pi}(f\circ
g)(x):=(D^{\ell}f)(g(x))[D_{\mathbf{v}_{1}}g(x),\dots,D_{\mathbf{v}_{\ell}}g(x)]\,.$
(346)

The following theorem is a version of Faà di Bruno’s formula [Cra05].

######  Theorem D.11.

Let $k\in\mathbb{N}$. Let $\mathbf{v}\in\mathbb{D}_{d}^{k}$. Let
$f\in\mathcal{D}^{k-1}(\mathbb{R}^{d},\mathbb{R}^{d})$. Assume that $f$ is
Lipschitz continuous. Then

$D_{x,\mathbf{v}}\dot{U}(f,x,t)=\sum_{\pi\in\mathbb{A}(\mathbf{v})}D^{\pi}\left(f\circ
U(f,\cdot,t)\right)(x)\,.$ (347)

###### Proof.

For $\mathbf{v}\in\mathbb{D}_{d}^{k}$, denote
$u_{\mathbf{v}}(x,t):=D_{x,\mathbf{v}}U(f,x,t)$. We prove the statement by
induction over $k$. See the proof of Lemma D.9 for $k=1$. Assume the statement
is true for all $\mathbf{v}\in\mathbb{D}_{d}^{k}$. Let $v\in\mathbb{D}_{d}$
and $\mathbf{v^{\prime}}:=(\mathbf{v},v)\in\mathbb{D}_{d}^{k+1}$. Then

$\displaystyle\dot{u}_{\mathbf{v^{\prime}}}(x,t)$
$\displaystyle=D_{x,v}\dot{u}_{\mathbf{v}}(x,t)$ (348)
$\displaystyle=\sum_{(\mathbf{v_{1}},\dots,\mathbf{v_{\ell}})\in\mathbb{A}(\mathbf{v})}D_{x,v}\left(D^{\ell}f(U(f,x,t))[u_{\mathbf{v_{1}}}(x,t),\dots,u_{\mathbf{v_{\ell}}}(x,t)]\right)$
(349)

By the product rule, Lemma A.5 (i),

$\displaystyle
D_{x,v}\left((D^{\ell}f)(U(f,x,t))[u_{\mathbf{v_{1}}}(x,t),\dots,u_{\mathbf{v_{\ell}}}(x,t)]\right)$
(350)
$\displaystyle=D_{x,v}\left(D^{\ell}f(U(f,x,t))\right)[u_{\mathbf{v_{1}}}(x,t),\dots,u_{\mathbf{v_{\ell}}}(x,t)]$
(351) $\displaystyle\phantom{=}\
+\sum_{j=1}^{\ell}\left((D^{\ell}f)(U(f,x,t))[u_{\mathbf{v_{1}}}(x,t),\dots,D_{x,v}u_{\mathbf{v_{j}}}(x,t),\dots,u_{\mathbf{v_{\ell}}}(x,t)]\right)\,.$
(352)

By the chain rule, Lemma A.5 (ii),

$\displaystyle
D_{x,v}\left(D^{\ell}f(U(f,x,t))\right)[u_{\mathbf{v_{1}}}(x,t),\dots,u_{\mathbf{v_{\ell}}}(x,t)]=D^{\ell+1}f(U(f,x,t))[u_{\mathbf{v_{1}}}(x,t),\dots,u_{\mathbf{v_{\ell}}}(x,t),u_{v}(x,t)]\,.$
(353)

The set of all partitions of $\mathbf{v}^{\prime}$ can be created from the set
of all partitions of $\mathbf{v}$ as follows: Let
$\mathbf{v_{1}},\dots,\mathbf{v_{\ell}}$ be a partition of $\mathbf{v}$. Then
$(v,\mathbf{v_{1}},\dots,\mathbf{v_{\ell}})$ and
$(\mathbf{v_{1}},\dots,(\mathbf{v}_{j},v),\dots,\mathbf{v_{\ell}})$,
$j\in{\left\llbracket\ell\right\rrbracket}$ are partitions of
$\mathbf{v}^{\prime}$ (the order of the partition components does not matter).
Iterating this construction over all partitions of $\mathbf{v}$ creates all
partitions of $\mathbf{v}^{\prime}$.

Now putting all previous equations in this proof together, we obtain

$\displaystyle D_{x,\mathbf{v}^{\prime}}\dot{U}(f,x,t)$
$\displaystyle=\dot{u}_{\mathbf{v^{\prime}}}(x,t)$ (354)
$\displaystyle=\sum_{(\mathbf{v_{1}},\dots,\mathbf{v_{\ell}})\in\mathbb{A}(\mathbf{v}^{\prime})}D^{\ell}f(U(f,x,t))[u_{\mathbf{v_{1}}}(x,t),\dots,u_{\mathbf{v_{\ell}}}(x,t)]$
(355)
$\displaystyle=\sum_{\pi\in\mathbb{A}(\mathbf{v}^{\prime})}D^{\pi}\left(f\circ
U(f,\cdot,t)\right)(x)\,.$ (356)

∎

######  Corollary D.12.

Let $\beta\in\mathbb{N}$, $L_{{\left\llbracket
1,\beta\right\rrbracket}}\in\mathbb{R}_{>0}$,
$k\in{\left\llbracket\beta\right\rrbracket}$, $d\in\mathbb{N}$. Let
$f\in\bar{\Sigma}^{d\to d}(\beta,L_{{\left\llbracket
0,\beta\right\rrbracket}})$ with $L_{0}=\infty$. Let
$\Delta\\!t_{0}\in\mathbb{R}_{>0}$. Assume $\Delta\\!t\in[0,\Delta\\!t_{0}]$.
Then

$\left|D^{k}\Upsilon(f,\Delta\\!t,\cdot)\right|_{\infty}\leq
C_{k}\left(\exp(L_{1}\Delta\\!t)-1\right)\,,$ (357)

where $C_{k}\in\mathbb{R}_{>0}$ depends only on $k$, $\Delta\\!t_{0}$, and
$L_{{\left\llbracket 1,k\right\rrbracket}}$.

###### Proof.

Define $C_{1}:=1$, $\tilde{C}_{1}:=\exp(L_{1}\Delta\\!t_{0})$, and, for
$k\in{\left\llbracket 2,\beta\right\rrbracket}$ with an arbitrary
$\mathbf{v}\in\mathbb{D}_{d}^{k}$,

$\displaystyle C_{k}$
$\displaystyle:=\frac{1}{L_{1}}\sum_{(\mathbf{v}_{1},\dots,\mathbf{v}_{\ell})\in\mathbb{A}(\mathbf{v})\setminus\\{\mathbf{v}\\}}L_{\ell}\prod_{j=1}^{\ell}\tilde{C}_{\\#\mathbf{v}_{j}}\,,$
(358) $\displaystyle\tilde{C}_{k}$
$\displaystyle:=C_{k}\left(\exp(L_{1}\Delta\\!t_{0})-1\right)\,.$ (359)

Let $k\in{\left\llbracket\beta\right\rrbracket}$. Let
$\mathbf{v}\in\mathbb{D}_{d}^{k}$. Set
$u_{\mathbf{v}}(x,t)=D_{x,\mathbf{v}}U(f,t,x)$. We will show, by induction
over $k$, that

$\sup_{x\in\mathbb{R}^{d}}\left\|u_{\mathbf{v}}(x,t)\right\|_{2}\leq\begin{cases}\exp(L_{1}t)&\text{
if }k=1\,,\\\ C_{k}\left(\exp(L_{1}t)-1\right)&\text{
otherwise}\,.\end{cases}$ (360)

Lemma D.9 shows the case $k=1$. Now let $k\geq 2$ and assume (360) for all
$\mathbf{v}\in\mathbb{D}_{d}^{k-1}$. Let $\mathbf{v}\in\mathbb{D}_{d}^{k}$. By
Theorem D.11,

$\dot{u}_{\mathbf{v}}(x,t)=\sum_{(\mathbf{v}_{1},\dots,\mathbf{v}_{\ell})\in\mathbb{A}(\mathbf{v})}(D^{\ell}f)(U(f,x,t))[u_{\mathbf{v}_{1}}(x,t),\dots,u_{\mathbf{v}_{\ell}}(x,t)]\,.$
(361)

Thus,

$\displaystyle\left\|\dot{u}_{\mathbf{v}}(x,t)\right\|_{2}$
$\displaystyle\leq\sum_{(\mathbf{v}_{1},\dots,\mathbf{v}_{\ell})\in\mathbb{A}(\mathbf{v})}\left|D^{\ell}f\right|_{\infty}\prod_{j=1}^{\ell}\left\|u_{\mathbf{v}_{j}}(x,t)\right\|_{2}$
(362) $\displaystyle\leq
L_{1}\left\|u_{\mathbf{v}}(x,t)\right\|_{2}+\sum_{(\mathbf{v}_{1},\dots,\mathbf{v}_{\ell})\in\mathbb{A}(\mathbf{v})\setminus\\{\mathbf{v}\\}}L_{\ell}\prod_{j=1}^{\ell}\left\|u_{\mathbf{v}_{j}}(x,t)\right\|_{2}\,.$
(363)

Set $z(t):=\left\|u_{\mathbf{v}}(x,t)\right\|_{2}$. Then

$\dot{z}(t)=\frac{\dot{u}_{\mathbf{v}}(x,t)^{\\!\top}\\!u_{\mathbf{v}}(x,t)}{\left\|u_{\mathbf{v}}(x,t)\right\|_{2}}\leq\left\|\dot{u}_{\mathbf{v}}(x,t)\right\|_{2}\leq
L_{1}\left(z(t)+C_{k}\right)\,.$ (364)

Consider the ODE

$\dot{v}(t)=L_{1}\left(v(t)+C_{k}\right)$ (365)

with $v(0)=\left\|u_{\mathbf{v}}(x,0)\right\|_{2}=0$ for $k\geq 2$. Then

$v(t)=C_{k}\left(\exp(L_{1}t)-1\right)\,.$ (366)

By the Lemma D.1,

$z(t)\leq v(t)\,.$ (367)

Hence

$\left\|u_{\mathbf{v}}(x,t)\right\|_{2}\leq
C_{k}\left(\exp(L_{1}t)-1\right)\,.$ (368)

For $k\geq 2$, $D^{k}\Upsilon(f,t,x)=D^{k}U(f,x,t)$. Thus,

$\sup_{x\in\mathbb{R}^{d}}\|D^{k}\Upsilon(f,t,x)\|_{\mathsf{op}}\preccurlyeq\sup_{x\in\mathbb{R}^{d}}\sup_{\mathbf{v}\in\mathbb{D}_{d}^{k}}\|u_{\mathbf{v}}(x,t)\|_{2}\leq
C_{k}\left(\exp(L_{1}t)-1\right)\,.$ (369)

For $k=1$, the result follows from Lemma D.9 with $C_{1}=1$. ∎

## References

  * [Ant07] Anestis Antoniadis  “Wavelet methods in statistics: some recent developments and their applications”  In _Stat. Surv._ 1, 2007, pp. 16–55  DOI: 10.1214/07-SS014
  * [Arn06] Vladimir I. Arnold  “Ordinary differential equations” Translated from the Russian by Roger Cooke, Second printing of the 1992 edition, Universitext  Springer-Verlag, Berlin, 2006, pp. ii+334
  * [Bis08] Jaya P.. Bishwal  “Parameter estimation in stochastic differential equations” 1923, Lecture Notes in Mathematics  Springer, Berlin, 2008, pp. xiv+264  DOI: 10.1007/978-3-540-74448-1
  * [BRR98] Peter J. Bickel, Ya’acov Ritov and Tobias Rydén  “Asymptotic normality of the maximum-likelihood estimator for general hidden Markov models”  In _Ann. Statist._ 26.4, 1998, pp. 1614–1635  DOI: 10.1214/aos/1024691255
  * [Bru08] Nicolas J-B. Brunel  “Parameter estimation of ODE’s via nonparametric estimators”  In _Electron. J. Stat._ 2, 2008, pp. 1242–1267  DOI: 10.1214/07-EJS132
  * [But16] J.. Butcher  “Numerical methods for ordinary differential equations” With a foreword by J. M. Sanz-Serna  John Wiley & Sons, Ltd., Chichester, 2016, pp. xxiii+513  DOI: 10.1002/9781119121534
  * [CG20] Fabienne Comte and Valentine Genon-Catalot  “Nonparametric drift estimation for i.i.d. paths of stochastic differential equations”  In _Ann. Statist._ 48.6, 2020, pp. 3336–3365  DOI: 10.1214/19-AOS1933
  * [Che+18] Ricky T.. Chen, Yulia Rubanova, Jesse Bettencourt and David Duvenaud  “Neural Ordinary Differential Equations”  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , NIPS’18  Montréal, Canada: Curran Associates Inc., 2018, pp. 6572–6583
  * [CR72] P.. Ciarlet and P.. Raviart  “General lagrange and hermite interpolation in Rn with applications to finite element methods”  In _Archive for Rational Mechanics and Analysis_ 46.3, 1972, pp. 177–199  DOI: 10.1007/BF00252458
  * [Cra05] Alex D.. Craik  “Prehistory of Faà di Bruno’s formula”  In _Amer. Math. Monthly_ 112.2, 2005, pp. 119–130  DOI: 10.2307/30037410
  * [Dat21] Itai Dattner  “Differential equations in data analysis”  In _Wiley Interdiscip. Rev. Comput. Stat._ 13.6, 2021, pp. Paper No. e153420  DOI: 10.1002/wics.1534
  * [DK15] Itai Dattner and Chris A.. Klaassen  “Optimal rate of direct estimators in systems of ordinary differential equations linear in functions of the parameters”  In _Electron. J. Stat._ 9.2, 2015, pp. 1939–1973  DOI: 10.1214/15-EJS1053
  * [Don10] T. Donovan  “Replay: The History of Video Games”  Yellow Ant, 2010
  * [FY03] Jianqing Fan and Qiwei Yao  “Nonlinear time series” Nonparametric and parametric methods, Springer Series in Statistics  Springer-Verlag, New York, 2003, pp. xx+551  DOI: 10.1007/b97702
  * [GK12] Shota Gugushvili and Chris A.. Klaassen  “$\sqrt{n}$-consistent parameter estimation for systems of ordinary differential equations: bypassing numerical integration via smoothing”  In _Bernoulli_ 18.3, 2012, pp. 1061–1098  DOI: 10.3150/11-BEJ362
  * [GR21] Georg A. Gottwald and Sebastian Reich  “Supervised learning from noisy observations: combining machine-learning techniques with data assimilation”  In _Phys. D_ 423, 2021, pp. Paper No. 13291115  DOI: 10.1016/j.physd.2021.132911
  * [Har02] Philip Hartman  “Ordinary differential equations” Corrected reprint of the second (1982) edition [Birkhäuser, Boston, MA; MR0658490 (83e:34002)], With a foreword by Peter Bates 38, Classics in Applied Mathematics  Society for IndustrialApplied Mathematics (SIAM), Philadelphia, PA, 2002, pp. xx+612  DOI: 10.1137/1.9780898719222
  * [Hei+18] Markus Heinonen, Cagatay Yildiz, Henrik Mannerström, Jukka Intosalmi and Harri Lähdesmäki  “Learning unknown ODE models with Gaussian processes”  In _International Conference on Machine Learning_ , 2018, pp. 1959–1968  PMLR
  * [HJ94] Roger A. Horn and Charles R. Johnson  “Topics in matrix analysis” Corrected reprint of the 1991 original  Cambridge University Press, Cambridge, 1994, pp. viii+607
  * [How91] Gary W Howell  “Derivative error bounds for Lagrange interpolation: An extension of Cauchy’s bound for the error of Lagrange interpolation”  In _Journal of Approximation Theory_ 67.2, 1991, pp. 164–173  DOI: 10.1016/0021-9045(91)90015-3
  * [Lah+24] Kamel Lahouel, Michael Wells, Victor Rielly, Ethan Lew, David Lovitz and Bruno M. Jedynak  “Learning nonparametric ordinary differential equations from noisy data”  In _J. Comput. Phys._ 507, 2024, pp. Paper No. 11297122  DOI: 10.1016/j.jcp.2024.112971
  * [Mar+23] Youssef Marzouk, Zhi Ren, Sven Wang and Jakob Zech  “Distribution learning via neural differential equations: a nonparametric statistical perspective”, 2023  DOI: 10.48550/arXiv.2309.01043
  * [MMP15] Kevin McGoff, Sayan Mukherjee and Natesh Pillai  “Statistical inference for dynamical systems: a review”  In _Stat. Surv._ 9, 2015, pp. 209–252  DOI: 10.1214/15-SS111
  * [MRS12] Enno Mammen, Christoph Rothe and Melanie Schienle  “Nonparametric regression with nonparametrically generated covariates” Publisher: Institute of Mathematical Statistics  In _The Annals of Statistics_ 40.2, 2012, pp. 1132–1170  DOI: 10.1214/12-AOS995
  * [Olv06] Peter J. Olver  “On multivariate interpolation”  In _Stud. Appl. Math._ 116.2, 2006, pp. 201–240  DOI: 10.1111/j.1467-9590.2006.00335.x
  * [QZ10] Xin Qi and Hongyu Zhao  “Asymptotic efficiency and finite-sample properties of the generalized profiling estimation of parameters in ordinary differential equations”  In _Ann. Statist._ 38.1, 2010, pp. 435–481  DOI: 10.1214/09-AOS724
  * [RH17] James Ramsay and Giles Hooker  “Dynamic data analysis” Modeling data with differential equations, Springer Series in Statistics  Springer, New York, 2017, pp. xvii+230  DOI: 10.1007/978-1-4939-7190-9
  * [RW06] Carl Edward Rasmussen and Christopher K.. Williams  “Gaussian Processes for Machine Learning”  The MIT Press, 2006
  * [RW94] D. Ruppert and M.. Wand  “Multivariate locally weighted least squares regression”  In _Ann. Statist._ 22.3, 1994, pp. 1346–1370  DOI: 10.1214/aos/1176325632
  * [Sch20] Johannes Schmidt-Hieber  “Nonparametric regression using deep neural networks with ReLU activation function”  In _Ann. Statist._ 48.4, 2020, pp. 1875–1897  DOI: 10.1214/19-AOS1875
  * [SS24] Christof Schötz and Maximilian Siebel  “Lower Bounds for Nonparametric Estimation of Ordinary Differential Equations”, 2024
  * [Str24] Steven Strogatz  “Nonlinear Dynamics and Chaos”  New York: ChapmanHall/CRC, 2024  DOI: 10.1201/9780429398490
  * [Tal21] Michel Talagrand  “Upper and lower bounds for stochastic processes—decomposition theorems” Second edition 60, Ergebnisse der Mathematik und ihrer Grenzgebiete. 3. Folge. A Series of Modern Surveys in Mathematics  Springer, Cham, 2021, pp. xviii+726  DOI: 10.1007/978-3-030-82595-9
  * [Tsy09] Alexandre B. Tsybakov  “Introduction to nonparametric estimation” Revised and extended from the 2004 French original, Translated by Vladimir Zaiats, Springer Series in Statistics  Springer, New York, 2009, pp. xii+214  DOI: 10.1007/b13794
  * [Xia19] Luo Xiao  “Asymptotic theory of penalized splines”  In _Electron. J. Stat._ 13.1, 2019, pp. 747–794  DOI: 10.1214/19-ejs1541