=-1 Causal representation learning promises to extend causal models to hidden causal variables from raw entangled measurements. However, most progress has focused on proving identifiability results in different settings, and we are not aware of any successful real-world application. At the same time, the field of dynamical systems benefited from deep learning and scaled to countless applications but does not allow parameter identification. In this paper, we draw a clear connection between the two and their key assumptions, allowing us to apply identifiable methods developed in causal representation learning to dynamical systems. At the same time, we can leverage scalable differentiable solvers developed for differential equations to build models that are both identifiable and practical. Overall, we learn explicitly controllable models that isolate the trajectory-specific parameters for further downstream tasks such as out-of-distribution classification or treatment effect estimation. We experiment with a wind simulator with partially known factors of variation. We also apply the resulting model to real-world climate data and successfully answer downstream causal questions in line with existing literature on climate change.

§ INTRODUCTION

Causal representation learning (CRL) [Schölkopf et al., 2021] focuses on provably retrieving high-level latent variables from low-level data. Recently, there have been many casual representation learning works compiling, in various settings, different theoretical identifiability results for these latent variables [Kivva et al., 2022, Lachapelle et al., 2024, Sturma et al., 2024, Brehmer et al., 2022, Lippe et al., 2022, Lippe et al., 2022, Xu et al., 2024, Von Kügelgen et al., 2021, von Kügelgen et al., 2024, Zhang et al., 2024, Varici et al., 2023, Squires et al., 2023]. The main open challenge that remains for this line of work is the broad applicability to real-world data.
Following earlier works in disentangled representations (see [Locatello et al., 2019] for a summary of data sets), existing approaches have largely focused on visual data

This is challenging for various reasons. Most notably, it is unclear what the causal variables should be in computer vision problems and what would be interesting or relevant causal questions. The current standard is to test algorithms on synthetic data sets with “made-up” latent causal graphs, e.g., with the object class of a rendered 3d shape causing its position, hue, and rotation [Von Kügelgen et al., 2021].

=-1In parallel, the field of machine learning for science [Mjolsness and DeCoste, 2001, Raghu and Schmidt, 2020] shows promising results on various real-world time series data collected from some underlying dynamical systems. Some of these works primarily focus on time-series forecasting, i.e., building a neural emulator that mimics the behavior of the given times series data [Chen et al., 2018, Chen et al., 2021, Kidger et al., 2021]; while others try to additionally learn an explicit ordinary differential equation simultaneously [Brunton et al., 2016, Brunton et al., 2016, Kaheman et al., 2020, d’Ascoli et al., 2022, d'Ascoli et al., 2024, Schröder and Macke, 2023]. However, to the best of our knowledge, none of these methods provide explicit identifiability analysis indicating whether the discovered equation recovers the ground truth underlying governing process given time series observations; or even whether the learned representation relates to the underlying steering parameters. At the same time, many scientific questions are inherently causal, in the sense that physical laws govern the measurements of all the natural data we can record, e.g., across different environments and experimental settings. Identifying such an underlying physical process can boost scientific understanding and reasoning in numerous fields; for example, in climate science, one could conduct sensitivity analysis of layer thickness parameter on atmosphere motion more efficiently, given a neural emulator that identifies the layer thickness in its latent space. However, whether mechanistic models can be practically identified from data is so far unclear <cit.>.

This paper aims to identify the underlying time-invariant physical parameters from real-world time series, such as the previously mentioned layer thickness parameter, while still preserving the ability to forecast efficiently. Thus, we connect the two seemingly faraway communities, causal representation learning and machine learning for dynamical systems, by phrasing parameter estimation problems in dynamical systems as a latent variable identification problem in CRL.
The benefits are two folds:
(1) we can import all identifiability theories for free from causal representation learning works, extending discovery methods with additional identifiability analysis and, e.g., multiview training constructs;
(2) we showcase that the scalable mechanistic neural networks <cit.> recently developed for dynamical systems can be directly employed with causal representation learning, thus providing a scalable implementation for both identifying and forecasting real-world dynamical systems.

=-1Starting by comparing the common assumptions in the field of parameter estimation in dynamical systems and causal representation learning, we carefully justify our proposal to translate any parameter estimation problem into a latent variable identification problem;
we differentiate three types of identifiability: full identifiability, partial identifiability and non-identifiability.
We describe concrete scenarios in dynamical systems where each kind of identifiability can be theoretically guaranteed and restate exemplary identifiability theorems from the causal representation learning literature with slight adaptation towards the dynamical system setup. We provide a step-by-step recipe for reformulating a parameter estimation problem into a causal representation learning problem and discuss the challenges and pitfalls in practice. Lastly, we successfully evaluate our parameter identification framework on various simulated and real-world climate data. We highlight the following contributions:

* We establish the connection between causal representation learning and parameter estimation for differential equations by pinpointing the alignment of common assumptions between two communities and providing hands-on guidance on how to rephrase the parameter estimation problem as a latent variable identification problem in causal representation learning.

* We equip discovery methods with provably identifiable parameter estimation approaches from the causal representation learning literature and their specific training constructs. This enables us to maintain both the theoretical results from the latter and the scalability of the former.

* We successfully apply causal representation learning approaches to simulated and real-world climate data, demonstrating identifiability via domain-specific downstream causal tasks (OOD classification and treatment-effect estimation), pushing one step further on the applicability of causal representation for real-world problems.

Remark on the novelty of the paper:  Our main contribution is establishing a connection between the dynamical systems and causal representation learning fields. As such, we do not introduce a new method per se. Meanwhile, this connection allows us to introduce CRL training constructs in methods that otherwise would not have any identification guarantees. Further, it provides the first avenue for causal representation learning applications on real-world data. These are both major challenges in the respective communities, and we hope this paper will serve as a building block for cross-pollination.

§ PARAMETER ESTIMATION IN DYNAMICAL SYSTEMS

=-1We consider dynamical systems in the form of

\begin{equation}
\dot \xb (t) = \fb_{\thetab}(\xb(t)) \quad \quad \xb(0) = \xb_0, \, \, \thetab \sim p_{\thetab}, \, \, t \in [0, t_{\max}]
\label{eq:ODE}
\end{equation}

where $\xb(t) \in \Xcal \subseteq \RR^d$ denotes the state of a system at time $t$, $f_{\thetab} \in \Ccal^1(\Xcal, \Xcal)$ is some smooth differentiable vector field representing the constraints that define the system's evolution, characterized by a set of physical parameters $\thetab \in \Thetab = \Thetab_1 \times \dots \times \Thetab_N$, where $\Thetab \subseteq \RR^N$ is an open, simply connected real space associated with the probability density $p_{\thetab}$. Formally, $f_{\thetab}$ can be considered as a functional mapped from $\thetab$ through $M: \Thetab \to \Ccal^1(\Xcal, \Xcal)$.
In our setup, we consider time-invariant, trajectory-specific parameters $\thetab$ that remain constant for the whole time span $[0, t_{\max}]$, but variable for different trajectories.
For instance, consider a robot arm interacting with multiple objects of different mass; a parameter $\thetab$ could be the object's masses $m \in \RR_{+}$ in Newton's second law $\ddot x(t) = \nicefrac{\Fcal(t)}{m}$, with $\Fcal(t)$ denote the force applied at time $t$. Depending on the object the robot arm interacts with, $m$ can take different values, following the prior distribution $p_{\thetab}$.
$\xb(0) = \xb_0 \in \Xcal$ denotes the initial value of the system. Note that higher-order ordinary differential equations can always be rephrased as a first-order ODE. For example, a $\nu$-th order ODE in the following form:

\begin{equation*}
x^{(\nu)}(t) = f =(x(t), x^{(1)}(t), \dots, x^{(\nu - 1)}(t), {\thetab}),
\end{equation*}

can be written as $\dot \xb (t) = f_{\thetab}(\xb(t))$,
where $\xb(t) = (x(t), x^{(1)}(t), \dots, x^{(\nu - 1)}(t)) \in \RR^{\nu \cdot d}$ denotes state vector constructed by concatenating the derivatives. Formally, the solution of such a dynamical system can be obtained by integrating the vector field over time: $\xb(t) = \int_{0}^t f(\xb(\tau), {\thetab}) d\tau $.

=-1What do we mean by “parameters”?
The parameters $\thetab$ that we consider can be both explicit and implicit. When the functional form of the ODE is given, like Newton's second law, the set of parameters is defined explicitly and uniquely. For real-world physical processes where the functional form of the state evolution is unknown, such as the sea-surface temperature change, we can consider latitude-related features as parameters. Overall, we use parameters to generally refer to any time-invariant, trajectory-specific components of the underlying dynamical system.

[Existence and uniqueness]

For every $\xb_0 \in \Xcal$, $\thetab \in \Thetab$, there exists a unique continuous solution $\xb_{\thetab}: [0, t_{\max}] \to \Xcal$ satisfying the ODE eq:ODE for all $t \in [0, t_{\max}]$ [Lindelöf, 1894, Ince, 1956].

[Structural identifiability]

An ODE eq:ODE is structurally identifiable in the sense that for any $\thetab_1, \thetab_2 \in \Thetab$,
$\xb_{\thetab_1}(t) = \xb_{\thetab_2}(t) \, \forall t \in [0, t_{\max}]$ holds if and only if $\thetab_1 = \thetab_2$ [Bellman and Åström, 1970, Walter et al., 1997, Wieland et al., 2021].

=-1 <Ref>

implies that it is in principle possible to identify the parameter $\thetab$ from a trajectory $\xb_{\thetab}$ [Miao et al., 2011].
Since this work focuses on providing concrete algorithms that guarantee parameter identifiability given infinite number of samples,
the structural identifiability assumption is essential as a theoretical ground for further algorithmic analysis.
It is noteworthy that a non-structurally identifiable system can become identifiable by reparamatization. For example, linear ODE $\dot \xb(t) = ab \xb(t)$ with parameters $a, b \in \RR^2$ is structurally non-identifiable as $a,b$ are commutative. But if we define $c:= ab$ as the overall growth rate of the linear system, then $c$ is structurally identifiable.

=-1Problem setting. Given an observed trajectory $\xb := (\xb_{\thetab}(t_0), \dots, \xb_{\thetab}(t_T)) \in \Xcal^T$ over the discretized time grid $\Tcal:= (t_0, \dots, t_T)$, our goal is to investigate the identifiability of structurally identifiable parameters by formulating concrete conditions under which the parameter $\thetab$ is (i) fully identifiable, (ii) partially identifiable, or (iii) non-identifiable from the observational data. We establish the identifiability theory for dynamical systems by converting classical parameter estimation problems [Bellman and Åström, 1970] into a latent variable identification problem in causal representation learning [Schölkopf et al., 2021]. For both (i) and (ii), we empirically showcase that existing CRL algorithms with slight adaptation can successfully (partially) identify the underlying physical parameters.

§ IDENTIFIABILITY OF DYNAMICAL SYSTEMS

=-1 This section provides different types of theoretical statements on the identifiability of the underlying time-invariant, trajectory-specific physical parameters $\thetab$, depending on whether the functional form of $f_{\thetab}$ is known or not. We show that the parameters from an ODE with a known functional form can be fully identified while parameters from unknown ODEs are in general non-identifiable. However, by incorporating some weak form of supervision, such as multiple similar trajectories generated from certain overlapping parameters [Locatello et al., 2020, Von Kügelgen et al., 2021, Daunhawer et al., 2023, Yao et al., 2024], parameters from an unknown ODE can also be partially identified. Detailed proofs of the theoretical statements are provided in <ref>.

§.§ Identifiability of dynamical systems with known functional form

=-1 We begin with the identifiability analysis of the  physical parameters of an ODE with known functional form. Many real-world data we record are governed by known physical laws. For example, the bacteria growth in microbiology could be modeled with a simple logistic equation under certain conditions, where the parameter of interest in this case would be the growth rate $r \in \RR_{+}$ and maximum capacity $K \in \RR_{+}$. Identifying such parameters would be helpful for downstream analysis. To this end, we introduce the definition of full identifiability of a physical parameter vector $\thetab$.

=-1A parameter vector $\thetab \in \Thetab$ is fully identified if the estimator $\hat{\thetab}$ converges to the ground truth parameter $\thetab$ almost surely.

=-1An ODE solver $F : \Thetab \to \Xcal^T$ computes the solution $\xb$ of the ODE $f_{\thetab} = M(\thetab)$ eq:ODE over a discrete time grid $T = (t_1, \dots, t_{T})$.

[Full identifiability with known functional form]corollaryfullIdent

Consider a trajectory $\xb \in \Xcal^T$ generated from a ODE $f_{\thetab}(\xb(t))$ satisfying <ref>, let $\hat \thetab$ be an estimator minimizing the following objective:

\begin{equation}
\label{eq:loss_full_ident}
\textstyle \Lcal (\hat \thetab) =
\norm{F({\hat \thetab}) -
\xb}_2^2
\end{equation}

then the parameter $\thetab$ is fully-identified def:full_ident by the estimator $\hat \thetab$.

=-1 The estimator $\hat \theta$ of <ref> is considered as some learnable parameters that can be directly optimized. If we have multiple trajectories $\xb$ generated from different realizations of $\thetab \sim p_{\thetab}$, we can also amortize the prediction $\hat{\thetab}$ using a smooth encoder $g: \Xcal^T \to \Thetab$. In this case, the loss above can be rewritten as:
$ \Lcal (g) = \EE_{\xb, t}
\norm{F({g(\xb)}) -
\xb(t)}_2^2
then the optimal encoder $g^* \in \argmin \Lcal(g)$ can generalize to unseen trajectories $\xb$ that follow the same class of physical law $f$ and fully identify their trajectory-specific parameters $\thetab$.

=-1 In <ref>, we consider an ideal setup glossing over several practical challenges: (i) Although closed-form solution of $\thetab^*$ is provided by linear least squares when $f$ is linear in $\thetab$ (see <ref> for details), finding the global optimum $\thetab^*$ in the nonlinear case using gradient descent is challenging in practice, both computationally and, despite the guarantee of theoretical full identifiability, it ignores non-convexity.

(ii) Since the functional form $f_{\thetab}$ is known, we assume that the ODE solver is exact in the sense that the generated solution of the ground truth parameter $F(\thetab)$ perfectly aligns with the observation $\xb$, i.e., $\Lcal(\thetab) = 0$. However, in practice, numerical solvers preserve certain approximation errors [Lötstedt and Petzold, 1986]. Although recent advances propose neural network-based ODE solvers [Chen et al., 2018] to alleviate this issue, end-to-end training that involves solving an ODE in the forward pass is not trivial. Most of the differentiable ODE solvers [Chen et al., 2018, Chen et al., 2021, Chen, 2018] solve the ODE autoregressively; thus, the time dimension cannot be parallelized in the GPU. To tackle this problem,   provided a highly efficient ODE solver that can be utilized in our framework. A more extensive discussion about different types of neural network-based solvers is provided in <ref>.

=-1Discussion. Many works on machine learning for dynamical system identification follow the principle presented in <ref>, and most of them solely differ concerning the architecture they choose for the ODE solver.
For example, SINDy-like ODE discovery methods [Brunton et al., 2016, Brunton et al., 2016, Kaheman et al., 2020, Kaptanoglu et al., 2021, Pervez et al., 2024] approximate the ground truth vector field $f$ using a linear weighted sum over a set of library functions and learn the linear coefficients by sparse regression.
For any ODE $f$ that is linear in $\thetab$, i.e., the ground truth vector field is in the form of $\textstyle f_{\thetab}(\xb, t) = \sum_{i = 1}^m \theta_i \phi_i(\xb)$ for a set of known base functions $\{\phi_i\}_{i \in [m]}$, SINDy-like approaches can fully identify the parameters by imposing some sparsity constraint.
Another line of work, gradient matching [Wenk et al., 2019], estimates the parameters probabilistically by modeling the vector field $f_{\thetab}$ using a Gaussian Process (GP).
The modeled solution $\xb(t)$ is thus also a GP since GP is closed under integrals (a linear operator).
Given the functional form of $f_{\thetab}$, the model aims to match the estimated gradient $\dot \xb$ and the evaluated vector field $f_{\thetab}(\xb(t))$ by maximizing the likelihood, which is equivalent to minimizing the least-squares loss eq:loss_full_ident under Gaussianity assumptions.
Hence, the gradient matching approaches can theoretically identify the underlying parameters under <ref>.
Formal statements and proofs for both SINDy-like and gradient matching approaches are provided in <ref>. Note that most ODE discovery approaches [Wenk et al., 2019, Brunton et al., 2016, Brunton et al., 2016, Kaheman et al., 2020, Kaptanoglu et al., 2021, Pervez et al., 2024] refrain from making identifiability statements and explicitly states it is unknown which settings yield identifiability.

§.§ Identifiability of dynamical systems without known functional form

Comparing typical assumptions of parameter estimation for dynamical systems and latent variable identification in causal representation learning. We justify that the common assumptions in both fields are aligned, providing theoretical ground for applying identifiable CRL methods to learning-based parameter estimation approaches in dynamical systems.

2cparam. estimation     2cCRL     1p6.0cmExplanation

ref     assumption      assumption     ref     1p6.0cm

<ref>     euexistence & uniqueness        codeterm. gen.    <ref>     Both <ref> and <ref> implies deterministic generative process.

ov$supp({\thetab})=\Thetab$     <ref>                  <ref> implies <ref> as $\xb_{\thetab}$ uniquely exists for all $\thetab \in \Thetab$.

<ref>    trstructural identifiability          iginjectivity                                                  <ref>      <ref> implies <ref> of the solution $\xb_{\thetab}$.

[overlay, remember picture,
round/.style=circle, inner sep=0, minimum size=1mm, fill=white, draw=black, thick,
[round] (ov_) at ($(ov.west) + (-.2, 0)$) ;

[thick] ($(eu.east) + (0.2, 0)$) node[round]  to[out=0, in=180] ($(co.west) + (-0.2, 0)$) node[round] ;
[thick, dashed] ($(tr.east) + (0.2, 0)$) node[round, solid]  to[out=0, in=180] (ov_);
[thick, dashed] ($(eu.east) + (0.2, 0)$) node[round, solid]  to[out=0, in=180] (ov_);
[thick] ($(tr.east) + (0.2, 0)$) node[round]  to[out=0, in=180] ($(ig.west) + (-0.2, 0)$) node[round] ;

=-1 In traditional dynamical systems, identifiability analysis usually assumes the functional form of the ODE is known [Miao et al., 2011]; however, for most real-world time series data, the functional form of underlying physical laws remains uncovered. Machine learning-based approaches for dynamical systems work in a black-box manner and can clone the behavior of an unknown system [Chen et al., 2018, Chen et al., 2021, Norcliffe et al., 2020], but understanding and identifiability guarantees of the learned parameters are so far missing. Since most of the physical processes are inherently steered by a few underlying time-invaraint parameters, identifying these parameters can be helpful in answering downstream scientific questions. For example, identifying climate zone-related parameters from sea surface temperature data could improve understanding of climate change because the impact of climate change significantly differs in polar and tropical regions.
Hence, we aim to provide identifiability analysis for the underlying parameters of an unknown dynamical system by converting the classical parameter estimation problem of dynamical systems into a latent variable identification problem in causal representation learning.
We start by listing the common assumptions in CRL and comparing the ground assumptions between these two fields.

The data generation process is deterministic in the sense that observation $\xb$ is generated from some latent vector $\thetab$ using a deterministic solver $F$ def:solve.

For each observation $\xb$, there is only one corresponding latent vector $\thetab$, i.e., the ODE solve function $F$ def:solve is injective in $\thetab$.

[Continuity and full support]

=-1$p_{\thetab}$ is smooth and continuous on $\Thetab$ with $p_{\thetab} > 0$ a.e.

=-1Assumption justification. <Ref> summarizes common assumptions in traditional parameter estimation in dynamical systems and causal representation learning literature.
We observe strong alignment between the ground assumptions in these two fields that justifies our idea of employing causal representation learning methods in parameter estimation problems for dynamical systems: (1) <Ref> implies that given a fixed initial value $\xb_0 \in \Xcal$, there exists a unique solution $\xb(t), \, t \in [0, t_{\max}]$ for any $f_{\thetab}$ with $\thetab \in \Thetab$. In other words, parameter domain $\Thetab$ is fully supported (<ref>), and these ODE solving processes from $F(\thetab)$ (<ref>) are deterministic, which aligns with the standard <ref> in CRL. Since the ODE solution $F(\thetab)$ sec:probem_setup is continuous by definition, the continuity assumption from CRL (<ref>) is also fulfilled. (2) <ref> emphasizes that each trajectory $\xb$ can only be uniquely generated from one parameter vector $\thetab \in \Thetab$, which means the generating process $F$ (<ref>) is injective in $\thetab$ (<ref>).

=-1 Next, we reformulate the parameter estimation problem in the language of causal representation learning.
We first cast the generative process of the dynamical system $f_{\thetab}(\xb(t))$ as a latent variable model by considering the underlying physical parameters $\thetab \sim p_{\thetab}$ as a set of latent variables.
Given a trajectory $\xb$ generated by a set of underlying factors $\thetab$ based on the vector field $f_{\thetab}(\xb(t))$, we consider the observed trajectory as some unknown nonlinear mixing of the underlying $\thetab$, with the mixing process specified by individual vector field $f_{\thetab}(\xb(t))$. This interpretation of observations aligns with the standard setup of causal representation learning; for instance, high-dimensional images are usually generated from some lower-dimensional latent generating factors through an unknown nonlinear process.
Thus, estimating the parameters of unknown dynamical systems becomes equivalent to inferring the underlying generating factors in causal representation learning.

=-1 After transforming the parameter estimation into a latent variable identification problem in CRL, we can directly invoke the identifiability theory from the literature. Based on <cit.>, we conclude that the underlying parameters from an unknown system are in general non-identifiable.
Nevertheless, several works proposed different weakly supervised learning strategies that can partially identify the latent variables [Locatello et al., 2020, Ahuja et al., 2022, Brehmer et al., 2022, Von Kügelgen et al., 2021, Daunhawer et al., 2023, Yao et al., 2024].
To this end, we define partial identifiability in the context of dynamical systems by slightly adapting the definition of block-identifiability proposed by Von Kügelgen et al., 2021:

A partition $\thetab_S := (\thetab_i)_{i \in S}$ with $S \subseteq [N]$ of parameter $\thetab \in \Thetab$ is partially identified by an encoder $g: \Xcal^T \to \Thetab$ if the estimator $\hat{\thetab}_S := g(\xb)_S$ contains all and only information about the ground truth partition $\thetab_S$, i.e. $\hat{\thetab}_S = h(\thetab_S)$ for some invertible mapping $h: \Thetab_S \to \Thetab_S$ where $\Thetab_S := \times_{i \in S} \Thetab_i$.

=-1 Note that the inferred partition $\hat{\thetab}_S$ can be a set of entangled latent variables rather than a single one. In the multivariate case, one can consider the $\hat{\thetab}_S$ as a bijective mixture of the ground truth parameter $\thetab_S$.

[Identifiability without known functional form]corollaryunknownf

Assume a dynamical system $f$ satisfying <ref>, a pair of trajectories $\xb, \tilde{\xb}$ generated from the same system $f$ but specified by different parameters $\thetab, \tilde{\thetab}$, respectively. Assume a partition of parameters $\thetab_S$ with $S \subseteq [N]$ is shared across the pair of parameters $\thetab, \tilde{\thetab}$.
Let $g: \Xcal^T \to \Theta$ be some smooth encoder and $\hat{F}: \Thetab \to \Xcal^T$ be some left-invertible smooth solver that minimizes the following objective:

\begin{equation}
\label{eq:loss_part_ident}
\textstyle \Lcal(g, \hat{F}) = \EE_{
\xb, \tilde\xb
\underbrace{\norm{g(\xb)_S - g(\tilde{\xb})_S}_2^2}_{\text{\textcolor{Green}{Alignment}}}
\norm{\solve{g(\xb)} - \xb}_2^2
+\norm{\solve{g(\tilde{\xb})} - \tilde{\xb}}_2^2}_{\text{\textcolor{Peach}{Sufficiency}}},
\end{equation}

then the shared partition $\thetab_S$ is partially identified def:par_ident by $g$ in the statistical setting.

We remark that an implicit ODE solver $\hat{F}$ is introduced in <ref> because the functional form $f_{\thetab}$ is unknown. Intuitively,  <Ref> provides partial identifiability results for the shared partition of parameters between two trajectories. We can consider the trajectories to be different simulation experiments but with certain sharing conditions, such as two wind simulations that share the same layer thickness parameter. This partial identifiability statement is mainly concluded from the theory in the multiview CRL literature [Ahuja et al., 2022, Locatello et al., 2020, Brehmer et al., 2022, Schölkopf et al., 2021, Von Kügelgen et al., 2021, Daunhawer et al., 2023, Yao et al., 2024].
Note that this corollary is one exemplary demonstration of achieving partial identifiability in dynamical systems. Many identifiability results from the causal representation works can be reformulated similarly by replacing their decoder with a differentiable ODE solver $\hat F$.
The high-level idea of multiview CRL is to identify the shared part between different views by enforcing alignment on the shared coordinates while preserving a sufficient information representation.
Alignment can be obtained by either minimizing the $L_2$ loss between the encoding from different views on the shared coordinates [Von Kügelgen et al., 2021, Daunhawer et al., 2023, Yao et al., 2024] or maximizing the correlation on the shared dimensions correspondingly [Lyu et al., 2021, Lyu and Fu, 2022];
Sufficiency of the learned representation is often prompted by maximizing the entropy [Zimmermann et al., 2021, Von Kügelgen et al., 2021, Daunhawer et al., 2023, Yao et al., 2024] or minimizing the reconstruction error [Locatello et al., 2020, Ahuja et al., 2022, Schölkopf et al., 2021, Brehmer et al., 2022].
Other types of causal representation learning works will be further discussed in <ref>.

§ CRL-CONSTRUCT OF IDENTIFIABLE NEURAL EMULATORS FOR DYNAMICAL SYSTEMS

=-1 This section provides a step-by-step construct of a neural emulator that can (1) identify the time-invariant, trajectory-specific physical parameters from some unknown dynamical systems if the identifiability conditions are met and (2) efficiently forecast future time steps. Identifiability can be guaranteed by employing causal representation learning approaches sec:identifiability_dynamical_systems while forecasting ability can be obtained by using an efficient mechanistic solver [Pervez et al., 2024] as a decoder.
For the sake of simplicity, we term these identifiable neural emulators as identifiers.
We remark that the general architecture remains consistent for most CRL approaches, while the learning object differs slightly in latent regularization, which is specified by individual identifiability algorithms. Intuitively, the latent regularization can be interpreted as an additional constraint put on the learned encodings imposed by the setting-specific assumptions, such as the alignment term in multiview CRL cor:partial_id.
In the following,
we demonstrate building an identifier in the multiview setting from scratch and showcase how it can be easily generalized to other CRL approaches with slight adaptation.

=-1Architecture. Since the parameters of interest are time-invariant and trajectory-specific sec:probem_setup, we input the whole trajectory $\xb = (\xb(t_1), \dots, \xb(t_T))$ to a smooth encoder $g: \Xcal^T \to \Thetab$, as shown in <ref>. Then, we decode the trajectory $\hat \xb$ from estimated parameter vector $\hat{\thetab} := g(\xb)$ using a mechanistic solver [Pervez et al., 2024].
The high-level idea of mechanistic neural networks is to approximate the underlying dynamical system using a set of explicit ODEs $\Ucal_{\hat \thetab}: C(\alphab, \hat \thetab) = 0$ with learnable coefficients $\alphab \in \RR^{d_{\alphab}}$. The explicit ODE family $\Ucal_{\hat \thetab}$ can then be interpreted as a constrained optimization problem and can thus be solved using a neural relaxed linear programming solver <cit.>.

In more detail, the original design of MNN predicts the coefficients from the input trajectory $\xb$ using an MNN encoder $g_{\mnn}$; however, as we enforce the estimated parameter $\thetab$ to preserve sufficient information of the entire trajectory $\xb$, we instead predict the coefficients $\alphab$ from the estimated parameter $\hat{\thetab}$ with the encoder $g_{\mnn}: \Thetab \to \RR^{d_{\alphab}}$. Formally, the coefficients $\alphab$ are computed as
\alphab = g_{\mnn}(\hat{\thetab})$ where  $\hat{\thetab} = g(\xb).
The resulting ODE family $\Ucal_{\hat \thetab}$ provides a broad variability of ODE parametrizations. A detailed formulation of $\Ucal_{\hat \thetab}$ at $t$ <cit.> is given by

\begin{equation}
\label{eq:mnn_ode}
\underbrace{ \textstyle \sum_{i=0}^{l} c_i(t; \hat \thetab) u^{(i)} }_{\text{linear terms}}+ \underbrace{ \textstyle \sum_{j=0}^r \phi_{k}(t; \hat \thetab) g_k(t, \{u^{(j)}\})}_{\text{nonlinear terms}} = b(t; \hat \thetab),
\end{equation}

where $u^{({i})}$ is $i-$th order approximations of the ground truth state $\xb$. Like in any ODE solving in practice, solving <ref> requires discretization of the continuous coefficients in time (e.g., $c_i(t; \hat \thetab)$). Discretizing the ODE representation $\Ucal_{\hat \thetab}$ gives rise to:

\begin{equation}
\textstyle
\sum_{i=0}^{l} c_{i, t} u_t^{(i)} + \sum_{j=0}^r \phi_{k, t} g_k(\{u_t^{(j)}\}) = b_t \quad s.t. \quad
(u_{t_1}, u^\prime_{t_1}, \dots) = \omega,
\end{equation}

where $\omega$ denotes the initial state vector of the ODE representation $\Ucal_{\hat \thetab}$. To this end, we present the explicit definition of the learnable coefficients $\alphab:= (c_{i, t}, \phi_{k, t}, b_t, s_t, \omega)$ with $t \in \Tcal, i \in [l], k \in [r]$, which is a concatenation of linear coefficients $c_{i, t}$, nonlinear coefficients $\phi_{i, k}$, adaptive step sizes $s_t$ and initial values $\omega$. Note that we dropped the $\hat \thetab$ in the notation for simplicity, but all of these coefficients $\alphab$ are predicted from $\hat \thetab$, as described previously.
At last, MNN converts ODE solving into a constrained optimization problem by representing the $\Ucal_{\hat \thetab}$ using a set of constraints, including ODE equation constraints, initial value constraints, and smoothness constraints <cit.>. This optimization problem is then solved by neural relaxed linear programming solver <cit.> in a time-parallel fashion, thus making the overall mechanistic solver scalable and GPU-friendly.

=-1Learning objective and latent regularizers.
Depending on whether the functional form of the underlying dynamical system is known or not, the proposed neural emulator can be trained using the losses given in <ref> or <ref>, respectively.
When the functional form is unknown, we employ CRL approaches to partially identify the physical parameters.
We remark that the causal representation learning schemes mainly differ in the latent regularizers, specified by the assumptions and settings. Therefore, we provide a more extensive summary of different causal representation learning approaches and their corresponding latent regularizer in <ref>.

§ RELATED WORK

=-1Multi-environment CRL. Another important line of work in causal representation learning focuses on the multi-environment setup, where the data are collected from multiple different environments and thus non-identically distributed. One common way to collect multi-environmental data is to perform single node interventions [Ahuja et al., 2023, Buchholz et al., 2024, Varici et al., 2023, Squires et al., 2023, Von Kügelgen et al., 2021, Zhang et al., 2024]. Identifiability proofs were provided for different settings, varying from types of mixing functions, causal models and interventions. For example,  considers linear Gaussian model and linear mixing functions, showing identifiability under both hard and soft interventions; Ahuja et al., 2023 considers a more general causal model with bounded support, together with finite degree polynomial mixing function, and provides identifiability proof for do and hard interventions. Buchholz et al., 2024 extends Squires et al., 2023 to general nonlinear mixing functions and linear Gaussian latent model. Zhang et al., 2024 show identifiability guarantee for a nonlinear causal model with polynomial mixing functions under soft interventions. Jin et al., 2023 considers linear mixing function with nonlinear model or linear non-Gaussian model under soft interventions. Overall, given the fruitful literature in multi-environment causal representation learning, we believe applying multi-environments methods to build identifiable neural emulators sec:crl-guide would be an exciting future avenue.

=-1ODE discovery. The ultimate goal of ODE discovery is to learn a human-interpretable equation for an unknown system, given discretized observations generated from this system.
Recently, many machine learning frameworks have been used for ODE discovery, such as sparse linear regression [Brunton et al., 2016, Brunton et al., 2016, Rudy et al., 2017, Kaheman et al., 2020], symbolic regression [d’Ascoli et al., 2022, Becker et al., 2023, d'Ascoli et al., 2024], simulation-based inference [Schröder and Macke, 2023, Cranmer et al., 2020].
d’Ascoli et al., 2022, Becker et al., 2023 exploit transformer-based approaches to dynamical symbolic regression for univariate ODEs, which is extended by d'Ascoli et al., 2024 to multivariate case. Schröder and Macke, 2023 employs simulation-based variational inference to jointly learn the operators (like addition or multiplication) and the coefficients. However, this approach typically runs simulations inside the training loop, which could introduce a tremendous computational bottleneck when the simulator is inefficient. On the contrary, our approach works offline with pre-collected data, avoiding simulating on the fly. Although ODE discovery methods can provide symbolic equations for data from an unknown trajectory, the inferred equation does not have to align with the ground truth. In other words, theoretical identifiability guarantees for these methods are still missing.

=-1Identifiability of dynamical systems. Identifiability of dynamical systems has been studied on a case-by-case basis in traditional system identification literature [Åström and Eykhoff, 1971, Villaverde et al., 2016, Miao et al., 2011]. Liang and Wu, 2008 studied ODE identifiability under measurement error. Scholl et al., 2023 investigated the identifiability of ODE discovery with non-parametric assumption, but only for univariate cases. More recently, several works have advanced in  identifiability analysis of linear ODEs from a single trajectory [Qiu et al., 2022, Stanhope et al., 2014, Duan et al., 2020]. Overall, current theoretical results cannot conclude whether an unknown nonlinear ODE can be identified from observational data. Hence, in our work, we do not aim to identify the whole equation of the dynamical systems but instead focus on identifying the time-invariant parameters.

§ EXPERIMENTS

Wind simulation: $u, v$ components [m/s] of simulated air motion over the globe.

=-1 This section provides experiments and results on both simulated and real-world climate data. In both cases, the true functional form of the underlying physical process is unknown, so we employ the multiview CRL approach together with mechanistic neural networks to build our identifiable neural emulator (termed as mechanistic identifier), following the steps in <ref>. We compare mechanistic identifier with three baselines: (1) Ada-GVAE [Locatello et al., 2020], a traditional multiview model that uses a vanilla decoder instead of a mechanistic solver. (2) Time-invariant MNN, proposed by [Pervez et al., 2024]. We choose this variant of MNN as our baseline for a fair comparison. (3) Contrastive identifier, a contrastive loss-based CRL approach without a decoder [Von Kügelgen et al., 2021, Daunhawer et al., 2023, Yao et al., 2024].
We train mechanistic identifier using <ref> and other baselines following the steps given in the original papers.
After training, we evaluate these methods on their identifiability and long-term forecasting capability.

§.§ Wind simulation

=-1Experimental setup.
Our experiment considers longitudinal and latitudinal wind velocities (also termed $u, v$ wind components) from the global wind simulation data generated by various layer-thickness parameters.
<Ref> depicts the wind simulation output at a certain time point.
To train the multiview approaches,
we generate a tuple of three views: After sampling the first view $\xb^1$ randomly throughout the whole training set, we sample another trajectory $\xb^2$ from a different location which shares the same simulation condition as the first one, compared to the first view, the third view $\xb^3$ is then sampled from another simulation but at the same location.
Overall, $\xb^1, \xb^2$ share the global simulation conditions like the layer thickness parameter while $\xb^1, \xb^3$ only share the local features. All three views share global atmosphere-related features that are not specified as simulation conditions.

More details about the data generation process and training pipeline are provided in <ref>.

=-1Parameter identification.
In this experiment, we use the learned representation to classify the ground-truth labels generated by discretizing the generating factor layer thickness, and report the accuracy in <ref>. In more detail, we use =12 for all models and split the learned encodings into three partitions $S_1, S_2, S_3$, with four dimensions each. Then, we individually predict the ground truth layer thickness labels from each partition. According to the previously mentioned view-generating process, the layer thickness parameter should be encoded in $S_1$ for both contrastive and mechanistic identifiers. This hypothesis is verified by <ref> since both contrastive and mechanistic identifiers show a high accuracy of $\approx$1 in the first partition $S_1$ and low accuracy in other partitions. On the contrary, Ada-GVAE and TI-MNN performed significantly worse with an average acc. of 60% everywhere. Overall, <ref> shows both the necessity of explicit time modeling using MNN solver (compared to Ada-GVAE) and identifiability power of multiview CRL (compared to TI-MNN).

figure=-1Prediction accuracy on layer thickness parameter on wind simulation data, evaluated on individual encoding partitions $S_1, S_2, S_3$. Results averaged from three random runs.

=-1Performance evaluation on the SST-V2 data on various types of tasks. Results averaged over three random seeds with standard deviation, provided as (m $\pm$ std).

Gray!20      3cSST V2

Acc.(ID)$(\uparrow)$     Acc.(OOD)$(\uparrow)$     Forecast. error$(\downarrow)$

Ada-GVAE     $0.468 \pm 0.001$     $0.467 \pm 0.000$      $0.043 \pm 0.044$

TI-MNN      $0.697 \pm 0.049$     $0.668 \pm 0.074$     $0.024 \pm 0.016$

Contr. Identifier      $\mathbf{0.904} \pm 0.011$     $\mathbf{0.861} \pm 0.022$

Mech. Identifier     $\mathbf{0.902} \pm 0.005$     $0.824 \pm 0.016$     $\mathbf{0.007} \pm 0.003$

§.§ Real-world sea surface temperature

=-1Experimental setup.
We evaluate the models on sea surface temperature dataset SST-V2 [Huang et al., 2021].
For the multiview training, we generate a pair trajectories from a small neighbor region ($\pm 5^{\circ}$) along the same latitude. We believe these pairs share certain climate properties as the locations from the same latitude share roughly the amount of direct sunlight which will directly affect the sea surface temperature. Further infromation about the dataset and training procedure is provided in <ref>.

=-1Time series forecasting.
We chunk the time series into slices of 4 years in training while keeping last four years as out-of-distribution forecasting task. To predict the last chunk, we input data from 2015 to 2018 to get the learned representation $\hat{\thetab}$. Since we assume $\hat{\thetab}$ to be time-inavriant, we decode $\hat{\thetab}$ together with 10 initial steps of 2019 to predict the last chunk. Note that contrastive identifier is excluded from this task as it does not have a decoder.
As shown in <ref>, the forecasting performance of mechanistic Identifier surpasses Ada-GVAE by a great margin, showcasing the superiority of integrating scalable mechanistic solvers in real-world time series datasets. At the same time, TI-MNN performed worse and unstably despite the MNN component, verifying the need of the additional information bottleneck (parameter encoder $g$) and the multiview learning scheme.

=-1Climate-zone classification.
Since there is no ground truth latitude-related parameters available, we design a downstream classification task that verifies our learned representation encodes the latitude-related information.
The goal of the task is to predict the climate zone (tropical, temperate, polar) from the learned shared representation because the latitude uniquely defines climate zones.
We evaluated the methods in both in-distribution (ID) and out-of-distribution (OOD) setup for all baselines. In the OOD setting, we input data from longitude $10^{\circ}$ to longitude $360^{\circ}$ when training the classifier while keeping the first $10$ degree as our out-of-distribution test data.
<Ref> show that both contrastive and mechanistic identifiers perform decently, supporting the applicability of identifiable multiview CRL algorithms in dynamical systems. Overall, the performance of multiview CRL-based approaches (contrastive and mechanistic identifiers) far exceeds Ada-GVAE and TI-MNN, again showcasing the superiority of the combination of causal representation learning and mechanistic solvers.

Causal effect estimation on SST-V2 by considering tropical or polar climate zones as treatments and zonal average temperature as outcome. Results computed over three independent runs.

=-1Average treatment effect estimation.
We further investigate the effect of climate zone on average temperature along one specific latitude through average treatment effect (ATE) estimation. Formally, we consider the latitudinal average temperature as outcome $Y$, two climate zones (tropical $(T=0)$, polar$(T=1)$) as binary treatments, and the predicted latitude-specific features as unobserved mediators.
Formally, ATE is defined as: $\text{ATE}:=\mathbb{E}[Y|do(T=1)]-\mathbb{E}[Y|do(T=0)]$.
Since ATE cannot be computed directly [Holland, 1986], we estimate it using the popular AIPW estimator [Robins et al., 1994]. <Ref> illustrates the estimated ATE change ratio from 1990 to 2020, computed by $\nicefrac{ATE(year) - ATE(1990)}{ATE(1990)}$. We observe that the recent ATE ratio has risen to  compared to 1990, which surprisingly aligns with the fact that the Arctic Ocean recently became at least twice as warm as before [Rantanen et al., 2022].

§ LIMITATIONS AND CONCLUSION

In this paper, we build a bridge between causal representation learning and dynamical system identification. By virtue of this connection, we successfully equipped existing mechanistic models (focusing on [Pervez et al., 2024] in practice for scalability reasons) with identification guarantees. Our analysis covers a large number of papers, including [Wenk et al., 2019, Brunton et al., 2016, Brunton et al., 2016, Kaheman et al., 2020, Kaptanoglu et al., 2021, Pervez et al., 2024] explicitly refraining from making identifiability statements. At the same time, our work demonstrated that causal representation learning training constructs are ready to be applied in the real world, and the connection with dynamical systems offers untapped potential due to its relevance in the sciences. This was an overwhelmingly acknowledged limitation of the causal representation learning field [Locatello et al., 2020, Von Kügelgen et al., 2021, Daunhawer et al., 2023, Yao et al., 2024, Ahuja et al., 2023, Buchholz et al., 2024, Varici et al., 2023, Squires et al., 2023]. Having clearly demonstrated the mutual benefit of this connection, we hope that future work will scale up identifiable mechanistic models and apply them to even more complex dynamical systems and real scientific questions.
Nevertheless, this paper has several technical limitations that could be addressed in future work. First of all, the proposed theory explicitly requires determinism as one of the key assumptions assmp:determinism, which directly excludes another important type of differential equation: Stochastic Differential Equations. Second, we assume we directly observe the state $\xb$ without considering measurement noise. Although the empirical results were promising on real-world noisy data subsec:sst, we believe explicitly modeling measurement noise would elevate the theory. Finally, our identifiability analysis focuses on the infinite data regime, which is unrealistic in real-world scenarios.

[Ahmed et al., 1974]
Nasir Ahmed, T_ Natarajan, and Kamisetty R Rao.
Discrete cosine transform.
IEEE transactions on Computers, 1000 (1):0 90–93, 1974.

[Ahuja et al., 2022]
Kartik Ahuja, Jason S Hartford, and Yoshua Bengio.
Weakly supervised representation learning with sparse perturbations.
Advances in Neural Information Processing Systems, 35:0 15516–15528, 2022.

[Ahuja et al., 2023]
Kartik Ahuja, Divyat Mahajan, Yixin Wang, and Yoshua Bengio.
Interventional causal representation learning.
In International Conference on Machine Learning, pages 372–407. PMLR, 2023.

[Åström and Eykhoff, 1971]
Karl Johan Åström and Peter Eykhoff.
System identification—a survey.
Automatica, 70 (2):0 123–162, 1971.

[Becker et al., 2023]
Sören Becker, Michal Klein, Alexander Neitz, Giambattista Parascandolo, and Niki Kilbertus.
Predicting ordinary differential equations with transformers.
In International Conference on Machine Learning, pages 1978–2002. PMLR, 2023.

[Bellman and Åström, 1970]
Ror Bellman and Karl Johan Åström.
On structural identifiability.
Mathematical biosciences, 70 (3-4):0 329–339, 1970.

[Brehmer et al., 2022]
Johann Brehmer, Pim De Haan, Phillip Lippe, and Taco S Cohen.
Weakly supervised causal representation learning.
Advances in Neural Information Processing Systems, 35:0 38319–38331, 2022.

[Brunton et al., 2016]
Steven L Brunton, Joshua L Proctor, and J Nathan Kutz.
Discovering governing equations from data by sparse identification of nonlinear dynamical systems.
Proceedings of the national academy of sciences, 1130 (15):0 3932–3937, 2016a.

[Brunton et al., 2016]
Steven L Brunton, Joshua L Proctor, and J Nathan Kutz.
Sparse identification of nonlinear dynamics with control (sindyc).
IFAC-PapersOnLine, 490 (18):0 710–715, 2016b.

[Buchholz et al., 2024]
Simon Buchholz, Goutham Rajendran, Elan Rosenfeld, Bryon Aragam, Bernhard Schölkopf, and Pradeep Ravikumar.
Learning linear causal representations from interventions under general nonlinear mixing.
Advances in Neural Information Processing Systems, 36, 2024.

[Chen, 2018]
Ricky T. Q. Chen.
torchdiffeq, 2018.
URL <https://github.com/rtqichen/torchdiffeq>.

[Chen et al., 2018]
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud.
Neural ordinary differential equations.
Advances in Neural Information Processing Systems, 2018.

[Chen et al., 2021]
Ricky T. Q. Chen, Brandon Amos, and Maximilian Nickel.
Learning neural event functions for ordinary differential equations.
International Conference on Learning Representations, 2021.

[Cranmer et al., 2020]
Kyle Cranmer, Johann Brehmer, and Gilles Louppe.
The frontier of simulation-based inference.
Proceedings of the National Academy of Sciences, 1170 (48):0 30055–30062, 2020.

[d'Ascoli et al., 2024]
Stéphane d'Ascoli, Sören Becker, Philippe Schwaller, Alexander Mathis, and Niki Kilbertus.
ODEFormer: Symbolic regression of dynamical systems with transformers.
In The Twelfth International Conference on Learning Representations, 2024.

[Daunhawer et al., 2023]
Imant Daunhawer, Alice Bizeul, Emanuele Palumbo, Alexander Marx, and Julia E Vogt.
Identifiability results for multimodal contrastive learning.
arXiv preprint arXiv:2303.09166, 2023.

[Duan et al., 2020]
Xiaoyu Duan, JE Rubin, and David Swigon.
Identification of affine dynamical systems from a single trajectory.
Inverse Problems, 360 (8):0 085004, 2020.

[d’Ascoli et al., 2022]
Stéphane d’Ascoli, Pierre-Alexandre Kamienny, Guillaume Lample, and Francois Charton.
Deep symbolic regression for recurrence prediction.
In International Conference on Machine Learning, pages 4520–4536. PMLR, 2022.

[Holland, 1986]
Paul W Holland.
Statistics and causal inference.
Journal of the American statistical Association, 810 (396):0 945–960, 1986.

[Huang et al., 2021]
Boyin Huang, Chunying Liu, Viva Banzon, Eric Freeman, Garrett Graham, Bill Hankins, Tom Smith, and Huai-Min Zhang.
Improvements of the daily optimum interpolation sea surface temperature (doisst) version 2.1.
Journal of Climate, 340 (8):0 2923–2939, 2021.

[Ince, 1956]
Edward L Ince.
Ordinary differential equations.
Courier Corporation, 1956.

[Jin et al., 2023]
Songyao Jin, Feng Xie, Guangyi Chen, Biwei Huang, Zhengming Chen, Xinshuai Dong, and Kun Zhang.
Structural estimation of partially observed linear non-gaussian acyclic model: A practical approach with identifiability.
In The Twelfth International Conference on Learning Representations, 2023.

[Kaheman et al., 2020]
Kadierdan Kaheman, J Nathan Kutz, and Steven L Brunton.
Sindy-pi: a robust algorithm for parallel implicit sparse identification of nonlinear dynamics.
Proceedings of the Royal Society A, 4760 (2242):0 20200279, 2020.

[Kaptanoglu et al., 2021]
Alan A Kaptanoglu, Jared L Callaham, Aleksandr Aravkin, Christopher J Hansen, and Steven L Brunton.
Promoting global stability in data-driven models of quadratic nonlinear dynamics.
Physical Review Fluids, 60 (9):0 094401, 2021.

[Kidger et al., 2021]
Patrick Kidger, Ricky T. Q. Chen, and Terry J. Lyons.
"hey, that's not an ode": Faster ode adjoints via seminorms.
International Conference on Machine Learning, 2021.

[Kivva et al., 2022]
Bohdan Kivva, Goutham Rajendran, Pradeep Ravikumar, and Bryon Aragam.
Identifiability of deep generative models without auxiliary information.
In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 15687–15701. Curran Associates, Inc., 2022.

[Klöwer and the SpeedyWeather.jl Contributors, 2023]
Milan Klöwer and the SpeedyWeather.jl Contributors.
Speedyweather.jl, 2023.
URL <https://github.com/SpeedyWeather/SpeedyWeather.jl>.

[Lachapelle and Lacoste-Julien, 2022]
Sébastien Lachapelle and Simon Lacoste-Julien.
Partial disentanglement via mechanism sparsity.
arXiv preprint arXiv:2207.07732, 2022.

[Lachapelle et al., 2023]
Sébastien Lachapelle, Tristan Deleu, Divyat Mahajan, Ioannis Mitliagkas, Yoshua Bengio, Simon Lacoste-Julien, and Quentin Bertrand.
Synergies between disentanglement and sparsity: Generalization and identifiability in multi-task learning.
In International Conference on Machine Learning, pages 18171–18206. PMLR, 2023.

[Lachapelle et al., 2024]
Sébastien Lachapelle, Divyat Mahajan, Ioannis Mitliagkas, and Simon Lacoste-Julien.
Additive decoders for latent variables identification and cartesian-product extrapolation.
Advances in Neural Information Processing Systems, 36, 2024.

[Liang and Wu, 2008]
Hua Liang and Hulin Wu.
Parameter estimation for differential equation models using a framework of measurement error in regression models.
Journal of the American Statistical Association, 1030 (484):0 1570–1583, 2008.

[Lindelöf, 1894]
Ernest Lindelöf.
Sur l’application de la méthode des approximations successives aux équations différentielles ordinaires du premier ordre.
Comptes rendus hebdomadaires des séances de l’Académie des sciences, 1160 (3):0 454–457, 1894.

[Lippe et al., 2022]
Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Efstratios Gavves.
Causal representation learning for instantaneous and temporal effects in interactive systems.
In The Eleventh International Conference on Learning Representations, 2022a.

[Lippe et al., 2022]
Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Stratis Gavves.
Citris: Causal identifiability from temporal intervened sequences.
In International Conference on Machine Learning, pages 13557–13603. PMLR, 2022b.

[Locatello et al., 2019]
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem.
Challenging common assumptions in the unsupervised learning of disentangled representations.
In international conference on machine learning, pages 4114–4124. PMLR, 2019.

[Locatello et al., 2020]
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen.
Weakly-supervised disentanglement without compromises.
In International Conference on Machine Learning, pages 6348–6359. PMLR, 2020.

[Lötstedt and Petzold, 1986]
Per Lötstedt and Linda Petzold.
Numerical solution of nonlinear differential equations with algebraic constraints. i. convergence results for backward differentiation formulas.
Mathematics of computation, 460 (174):0 491–516, 1986.

[Lu et al., 2022]
Peter Y Lu, Joan Ariño Bernad, and Marin Soljačić.
Discovering sparse interpretable dynamics from partial observations.
Communications Physics, 50 (1):0 206, 2022.

[Lyu and Fu, 2022]
Qi Lyu and Xiao Fu.
On finite-sample identifiability of contrastive learning-based nonlinear independent component analysis.
In International Conference on Machine Learning, pages 14582–14600. PMLR, 2022.

[Lyu et al., 2021]
Qi Lyu, Xiao Fu, Weiran Wang, and Songtao Lu.
Understanding latent correlation-based multiview learning and self-supervision: An identifiability perspective.
arXiv preprint arXiv:2106.07115, 2021.

[Miao et al., 2011]
Hongyu Miao, Xiaohua Xia, Alan S Perelson, and Hulin Wu.
On identifiability of nonlinear ode models and applications in viral dynamics.
SIAM review, 530 (1):0 3–39, 2011.

[Mjolsness and DeCoste, 2001]
Eric Mjolsness and Dennis DeCoste.
Machine learning for science: state of the art and future prospects.
science, 2930 (5537):0 2051–2055, 2001.

[Moran et al., 2022]
Gemma Elyse Moran, Dhanya Sridhar, Yixin Wang, and David Blei.
Identifiable deep generative models via sparse decoding.
Transactions on Machine Learning Research, 2022.
ISSN 2835-8856.
URL <https://openreview.net/forum?id=vd0onGWZbE>.

[Norcliffe et al., 2020]
Alexander Norcliffe, Cristian Bodnar, Ben Day, Nikola Simidjievski, and Pietro Liò.
On second order behaviour in augmented neural odes.
Advances in neural information processing systems, 33:0 5911–5921, 2020.

[Pervez et al., 2024]
Adeel Pervez, Francesco Locatello, and Efstratios Gavves.
Mechanistic neural networks for scientific machine learning.
International Conference on Machine Learning, 2024.

[Qiu et al., 2022]
Xing Qiu, Tao Xu, Babak Soltanalizadeh, and Hulin Wu.
Identifiability analysis of linear ordinary differential equation systems with a single trajectory.
Applied Mathematics and Computation, 430:0 127260, 2022.

[Raghu and Schmidt, 2020]
Maithra Raghu and Eric Schmidt.
A survey of deep learning for scientific discovery.
arXiv preprint arXiv:2003.11755, 2020.

[Rantanen et al., 2022]
Mika Rantanen, Alexey Yu Karpechko, Antti Lipponen, Kalle Nordling, Otto Hyvärinen, Kimmo Ruosteenoja, Timo Vihma, and Ari Laaksonen.
The arctic has warmed nearly four times faster than the globe since 1979.
Communications earth & environment, 30 (1):0 168, 2022.

[Robins et al., 1994]
James M Robins, Andrea Rotnitzky, and Lue Ping Zhao.
Estimation of regression coefficients when some regressors are not always observed.
Journal of the American statistical Association, 890 (427):0 846–866, 1994.

[Rudy et al., 2017]
Samuel H Rudy, Steven L Brunton, Joshua L Proctor, and J Nathan Kutz.
Data-driven discovery of partial differential equations.
Science advances, 30 (4):0 e1602614, 2017.

[Schölkopf et al., 2021]
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio.
Toward causal representation learning.
Proceedings of the IEEE, 1090 (5):0 612–634, 2021.

[Scholl et al., 2023]
Philipp Scholl, Aras Bacho, Holger Boche, and Gitta Kutyniok.
The uniqueness problem of physical law learning.
In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023.

[Schröder and Macke, 2023]
Cornelius Schröder and Jakob H Macke.
Simultaneous identification of models and parameters of scientific simulators.
arXiv preprint arXiv:2305.15174, 2023.

[Squires et al., 2023]
Chandler Squires, Anna Seigal, Salil S. Bhate, and Caroline Uhler.
Linear causal disentanglement via interventions.
In International Conference on Machine Learning, volume 202, pages 32540–32560. PMLR, 2023.

[Stanhope et al., 2014]
Shelby Stanhope, Jonathan E Rubin, and David Swigon.
Identifiability of linear and linear-in-parameters dynamical systems from a single trajectory.
SIAM Journal on Applied Dynamical Systems, 130 (4):0 1792–1815, 2014.

[Sturma et al., 2024]
Nils Sturma, Chandler Squires, Mathias Drton, and Caroline Uhler.
Unpaired multi-domain causal representation learning.
Advances in Neural Information Processing Systems, 36, 2024.

[Tonolini et al., 2020]
Francesco Tonolini, Bjørn Sand Jensen, and Roderick Murray-Smith.
Variational sparse coding.
In Uncertainty in Artificial Intelligence, pages 690–700. PMLR, 2020.

[Varici et al., 2023]
Burak Varici, Emre Acartürk, Karthikeyan Shanmugam, and Ali Tajer.
Score-based causal representation learning from interventions: Nonparametric identifiability.
In Causal Representation Learning Workshop at NeurIPS 2023, 2023.
URL <https://openreview.net/forum?id=MytNJ6lXAV>.

[Villaverde et al., 2016]
Alejandro F Villaverde, Antonio Barreiro, and Antonis Papachristodoulou.
Structural identifiability of dynamic systems biology models.
PLoS computational biology, 120 (10):0 e1005153, 2016.

[Von Kügelgen et al., 2021]
Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello.
Self-supervised learning with data augmentations provably isolates content from style.
Advances in neural information processing systems, 34:0 16451–16467, 2021.

[von Kügelgen et al., 2024]
Julius von Kügelgen, Michel Besserve, Liang Wendong, Luigi Gresele, Armin Kekić, Elias Bareinboim, David Blei, and Bernhard Schölkopf.
Nonparametric identifiability of causal representations from unknown interventions.
Advances in Neural Information Processing Systems, 36, 2024.

[Walter et al., 1997]
Eric Walter, Luc Pronzato, and John Norton.
Identification of parametric models from experimental data, volume 1.
Springer, 1997.

[Wenk et al., 2019]
Philippe Wenk, Alkis Gotovos, Stefan Bauer, Nico S Gorbach, Andreas Krause, and Joachim M Buhmann.
Fast gaussian process based gradient matching for parameter identification in systems of nonlinear odes.
In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1351–1360. PMLR, 2019.

[Wieland et al., 2021]
Franz-Georg Wieland, Adrian L Hauber, Marcus Rosenblatt, Christian Tönsing, and Jens Timmer.
On structural and practical identifiability.
Current Opinion in Systems Biology, 25:0 60–69, 2021.

[Xu et al., 2024]
Danru Xu, Dingling Yao, Sébastien Lachapelle, Perouz Taslakian, Julius von Kügelgen, Francesco Locatello, and Sara Magliacane.
A sparsity principle for partially observable causal representation learning.
International Conference on Machine Learning, 2024.

[Yao et al., 2024]
Dingling Yao, Danru Xu, Sebastien Lachapelle, Sara Magliacane, Perouz Taslakian, Georg Martius, Julius von Kügelgen, and Francesco Locatello.
Multi-view causal representation learning with partial observability.
In The Twelfth International Conference on Learning Representations, 2024.
URL <https://openreview.net/forum?id=OGtnhKQJms>.

[Zhang et al., 2024]
Jiaqi Zhang, Kristjan Greenewald, Chandler Squires, Akash Srivastava, Karthikeyan Shanmugam, and Caroline Uhler.
Identifiability guarantees for causal disentanglement from soft interventions.
Advances in Neural Information Processing Systems, 36, 2024.

[Zimmermann et al., 2021]
Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel.
Contrastive learning inverts the data generating process.
In International Conference on Machine Learning, pages 12979–12990. PMLR, 2021.

ODE former: https://arxiv.org/pdf/2310.05573.pdf