[Neural networks initialise close to the 0 constant function.]Initialisation of MLPs. Outputs given by $f(\vC_1 + \alpha(\vC_2 - \vC_1); \vtheta)$ where $\vC_1 \in \mathbb{R}^I$ with every entry equal to 1 and $\vC_2 = - \vC_1$ where $f$ is an MLP with 500 neurons per layer; the input dimension $I=150528 = 224 \times 224 \times 3$. The mean model outputs fig:init_mlp_mean and max model outputs fig:init_mlp_max are closer to 0 at initialisation as depth increases. Uncertainty is obtained by using multiple seeds to initialise the network.

To empirically assess the effect of the number of layers on GC at initialisation, we use MLPs initialised either with the default initialisation in many deep learning packages, which we will call `standard initialisation' (with weights sampled from a truncated normal distribution bounded between -2 and 2 and scaled by the square root of number of incoming units), and Glorot initialisation [Glorot and Bengio, 2010]. We show in Figure <ref> that the value of GC at initialisation decreases as the number of layers increases.
These results show deep MLPs are smoother at initialisation, in that they are less sensitive to the value of the input and are close to the constant function $0$; this is a useful initialisation to start with as otherwise the training procedure might have to undo the effects of a bad initialisation (where for example inputs from the same class might be initialised to have very different output logits), and furthermore increases reliability as training schemes become less dependent on initialisations. We also observe that GC captures this effect at initialisation, and helps explain why deeper models can better model the training data.

[Geometric Complexity (GC) at initialisation decreases as the number of layers in the network increases.]Geometric Complexity (GC) at initialisation decreases as the number of layers in an MLP increases; this reflects the initialisation phenomenon showcased in Figure <ref>. This effect shows that the deeper the MLP, the smoother it is at initialisation.

We have seen how deep MLPs initialised with commonly used initialisation schemes start training at low (close to 0) GC. We also observe that as training progresses and the model captures the training data, its GC increases; we see this in Figure <ref> and we will consistently observe this when training larger models as well (see Figures <ref> and <ref>). During training however, deep networks do not learn overly complex functions, even as GC increases; as shown in Figure <ref>, even when using a large MLP to train 10 data points, the model fits the data without overfitting. We now turn to explore the potential mechanisms for this observation.

[0 epochs.]

[10 epochs.]

[1000 epochs.]

[10000 epochs.]

[Geometric Complexity (GC) increases as training progresses.]For a large MLP fitting 10 points in 1 dimension, the GC of the function being learned gradually grows in training. Despite having capacity to overfit, the model learns a simple function while fitting the data.

§ GC AND REGULARISATION

We now highlight how many implicit and explicit regularisation methods regularise GC, despite not targetting it directly. To disentangle the effects studied
here from optimisation choices, we use stochastic gradient descent without momentum to train all
models unless otherwise specified. We also study the impact of a given training heuristic on GC in isolation
of other techniques to avoid masking effects. For this reason, we do not use data augmentation or
learning rate schedules.

§.§ Explicit regularisation

$L_2$ norm regularisation. Section <ref> has shown the connection between GC and weight norms in the case of rectifier neural networks. $L_2$ norm regularisation adds the regulariser $\zeta \sum_{l=1}^N \norm{\vW_l}_2^2$ with $\zeta > 0$ to the loss function, minimising the magnitude of each weight parameter. This results in a decreased GC in the case of rectifier neural networks where the GC can be written using the form in Eq (<ref>), where the value of each piecewise linear function will be determined by the magnitude of the network's weights. While this is not a formal argument, we confirm in Figure <ref> that as the $L_2$ regularisation strength increases, GC decreases.

[Gradient norm.]

[Common explicit regularisation methods lead to decreased GC when using vanilla stochastic gradient descent.]Common explicit regularisation methods lead to decreased GC when using vanilla stochastic gradient descent to train a Resnet-18 model on CIFAR-10. GC decreases with increased regularisation strength. Similar trends are observed when momentum is used, as shown in Section <ref> of the Appendix, where we also show loss curves.

Gradient norm regularisation. Another popular form of regularisation is to add an explicit regulariser with the norm of the gradient $\norm{\nabla_{\vtheta} E(\vtheta)}_2^2$ to the loss function [Geiping et al., 2022, Smith et al., 2021, Balduzzi et al., 2018, Mescheder et al., 2017, Nagarajan and Kolter, 2017, Qin et al., 2020]. While GC is concerned with gradients with respect to inputs, and the gradient norm regulariser penalises the gradient norm with respect to parameters, the two are connected via backpropagation; we previously used neural network structure and backpropagation to connect the effect of an input smoothness regularisation technique—Spectral Normalisation—to scaling of gradients with respect to parameters in Chapter <ref>. As we see in backpropagation Eqs (<ref>)—(<ref>), decreasing the gradient norm $\norm{\nabla_{\vtheta} E(\vtheta)}_2^2$ can be achieved by decreasing the magnitude of the weight matrix entries or that of the activation derivative $a'(\vz)$, which for ReLu networks can be achieved with increased sparsity; both of these will lead to a decrease in GC. We show in Figure <ref> that indeed, increased gradient norm regularisation consistently leads to a decrease in GC.

Spectral Normalisation/regularisation. GC is deeply connected to Lipschitz smoothness; as we have shown in Section <ref>, the Lipschitz constant of the model is an upper bound on its GC. Hard Lipschitz normalisation methods, such as Spectral Normalisation, will thus impose a fixed upper bound on the GC of the model, while regularisation methods such as Spectral Regularisation induce a regularisation pressure to decrease that bound; for an overview of these methods as well as a contrast between soft and hard constraints, see Chapter <ref>.
We show in Figure <ref> that minimising the spectral norm of each layer, as done by Spectral Regularisation, leads to a decrease in GC. Furthermore, as the strength of Spectral Regularisation increases—by increasing the regularisation coefficient—GC decreases.

Jacobian regularisation. Jacobian regularisation is an existing form of explicit regularisation which directly minimises GC, i.e. it adds $\zeta GC(f;\vtheta, \mathcal{D}) = \frac{\zeta}{|\mathcal{D}|}\sum_{\vx\in \mathcal{D}} \|\jacparam{\vx}{f(\vx; \vtheta)}\|_2^2$ to the loss function. Jacobian regularisation has been correlated with increased generalisation but also robustness to input distributional shift [Hoffman et al., 2019, Sokolić et al., 2017, Varga et al., 2017, Yoshida and Miyato, 2017]. Sokolić et al., 2017  show that adding Jacobian regularisation to the loss function can lead to an increase in test set accuracy (their Tables III, IV, and V). We confirm their findings, namely an increase in test accuracy and a decrease in GC with Jacobian/GC regularisation in Figure <ref>. As we have noted previously, too much smoothness can hurt model capacity and decrease performance, which is something that we observe here too for large regularisation coefficients.

[GC decreases with increased regularisation strength.]

[Accuracy increases with GC regularisation strength, unless too much regularisation is used.]

[Explicit minimising GC leads to increased generalisation.]Jacobian (GC) regularisation on CIFAR-10. Explicit regularisation minimising GC (already existing in literature as Jacobian regularisation) leads to increased generalisation, unless the regularisation coefficient is too large. This effect reflects what we have observed in other settings, where too much smoothness can hurt performance, from an illustrative example in Figure <ref>, to the Spectral Normalisation example in Figure <ref> and the reinforcement learning examples in Figure <ref>. Results obtained using 1 seed.

§.§ Implicit regularisation via discretisation drift

We now show how the implicit regularisation induced by discretisation drift in gradient descent optimisation, previously discussed in Chapters <ref>, <ref> and <ref>, can lead to a decrease in GC due the connection between gradient norm minimisation and GC in neural networks. Since the strength of implicit regularisation is affected by optimisation hyperparameters such as learning rate and batch size, we empirically observe how changing these hyperparameters leads to direct effect on GC.

Learning rate. Large learning rates have been consistently shown to have an implicit regularisation effect and lead to increased generalisation [Li et al., 2019, Barrett and Dherin, 2021, Smith et al., 2021, Hoffer et al., 2017]. Attempts at formalising this include the Implicit Gradient Regularisation (IGR) flow discussed in Chapter <ref>, which suggests that gradient descent implicitly minimises the loss function $\tilde{E}(\vtheta) = E(\vtheta) + \frac h 4 \norm{\nabla_{\vtheta} E}^2$, where $h$ is the learning rate. In the previous section, we described how backpropagation reveals shared components between gradients w.r.t. parameters—$\nabla_{\vtheta} E$ present in IGR—and  gradients w.r.t. inputs—$\nabla_{\vx}f$ present in GC. We then showed that explicitly regularising the gradient norm $\norm{\nabla_{\vtheta} E}^2$ leads to a decrease in GC. Based on these results, we expect that increasing the learning rate $h$ leads to a decrease in GC due to increased implicit gradient regularisation pressure $\frac h 4 \norm{\nabla_{\vtheta} E}^2$, which we  empirically observe in Figure <ref>.

[Train loss.]

[Test accuracy.]

[The implicit regularisation effect of large learning rates on GC.]The implicit regularisation effect of large learning rates on GC: increasing the learning rate results in decreased GC and increased performance. The results here use vanilla gradient descent; a similar trend is observed when momentum is used,  as shown in Section <ref> of the Appendix.

Batch size.
The increased generalisation performance of lower batch sizes has been long studied in deep learning, often under the name `the generalisation gap' [LeCun et al., 2012, Hoffer et al., 2017, Keskar et al., 2017, Smith et al., 2020, Masters and Luschi, 2018]. While traditionally this increased generalisation effect has been attributed to gradient noise due to the sampling process, recent work has revisited this assumption:
the generalisation gap was recently bridged with Geiping et al., 2022 showing that full-batch training can achieve the same test set performance as stochastic gradient descent. Geiping et al., 2022 use an explicit regulariser inspired by IGR, with the intuition that the strength of the implicit gradient norm regulariser induced by discretisation drift $\norm{\nabla_{\vtheta} E}^2$ is stronger for stochastic gradient descent compared to full-batch gradient descent, as described by Smith et al., 2021; we previously discussed the implicit regularisation induced by discretisation drift in stochastic gradient descent and the results of Smith et al., 2021 in Chapter <ref>. Thus, Geiping et al., 2022 use $\frac{h}{4 |\mathcal{B}|}\sum_{B \in \mathcal{B}} \norm{\frac{1}{B}\sum_{\vX \in B} \nabla_{\vtheta} E(\vX;\vtheta)}^2$ instead, where $\mathcal{B}$ is the set of batches used for training, in contrast to $\frac{h}{4} \norm{\frac{1}{|\mathcal{D}|}\sum_{\vX \in \mathcal{D}} \nabla_{\vtheta} E(\vX;\vtheta)}^2$, the IGR regulariser in the full-batch case. Since the gradient norm is computed over a batch  rather than the entire dataset, outliers, such as examples with a large gradient, are less likely to be averaged out and this leads to a stronger regularisation pressure.
Their result further strengthens the connection between small batch sizes and increased gradient norm regularisation pressure, which we have consistently seen to lead to smaller GC. Coupling these results, suggesting that for smaller batch sizes there is a stronger effect due to IGR, with what we have previously observed, namely that minimising $\norm{\nabla_{\vtheta} E}^2$ leads to a decrease in GC, we should expect that decreasing batch sizes leads to a decrease in GC. We empirically assess this question and show results in Figure <ref>, observing a consistent decrease in GC with smaller batch sizes. These results show the implicit regularisation effect of batch sizes on model smoothness.

[Train loss.]

[Test accuracy.]

[The implicit regularisation effect of small batch sizes on GC.]The implicit regularisation effect of small batch sizes on GC: decreasing the batch size results in decreased GC and increased performance. The results here use vanilla gradient descent; a similar trend is observed when momentum is used,  as shown in Section <ref> of the Appendix.

Results and justifications in this section use vanilla gradient descent, but we note that the gradient norm minimisation effect induced by the discretisation drift of momentum has recently been mathematically formalised by Ghosh et al., 2023, and thus we expect our arguments to hold for gradient descent with momentum too. We confirm this with empirical results in Appendix <ref>.

§ DOUBLE DESCENT AND GC

When neural network complexity is measured using a simple parameter count, a double descent phenomenon has been consistently observed: as the number of parameters increases, the test loss decreases at first, before increasing again, followed by a second descent toward a low test error value [Belkin, 2021, Belkin et al., 2019, Nakkiran et al., 2019].  An excellent overview of the double descent  phenomena in deep learning can be found in Belkin, 2021, together with connections to smoothness as an inductive bias of deep networks and the role of optimisation.
We previously discussed the double descent phenomenon in Section <ref>, where we contrasted it with the U-shaped curve associated with standard machine learning methods and complexity measures in Figure <ref>. We discussed the double descent phenomenon from the smoothness perspective and intuited that smoothness likely plays an important role in the second descent: we postulated that in the first descent, the generalisation error is decreasing as the model is given extra capacity to
capture the decision surface; the increase happens when the model has enough capacity to fit the training data, but it cannot do so and retain smoothness, and thus overfits; the second descent occurs as the capacity increases and smoothness can be retained.

To explore whether GC as a measure of model complexity based on input smoothness can capture the double descent phenomena we follow the set up introduced in Nakkiran et al., 2019: we train multiple ResNet-18 networks on CIFAR-10 with increasing layer width, and show results in Figure <ref>. We make two observations: first, like the test loss, GC follows a double descent curve as width increases (Figure <ref>); second, when plotting GC against the test loss, we observe a U-shape curve, recovering the traditional expected behaviour of complexity measures (Figure <ref>). Importantly, we observe the connection between the generalisation properties of over-parametrised models and GC: after the critical region, increase in model size leads to a decrease in GC. These results provide further evidence that GC is able to capture model capacity in a meaningful way, suggestive of a reconciliation of traditional complexity theory and deep learning, as well as provide an empirical basis for our previously highlighted connection between smoothness and double descent.

[GC and train/test losses as the network width increases.]

[Train and test losses as GC increases.]

[Double descent and GC.]Double descent and GC. fig:gc_train_test_dd: GC captures the double descent phenomenon. fig:gc_vs_train: GC captures the traditional U-shape curve, albeit with some noise, showing (top) GC compared to train and test losses for all models and (bottom)  GC compared to train and test losses averaged across different seeds. We fit a 6 degree polynomial to the curves to showcase the trend.

§ RELATED WORK

Complexity measures for deep neural networks. There has been a recent effort to create complexity measures that capture both the generalisation phenomenon of deep learning and are computationally tractable. Lee et al., 2020 introduce `Neural Complexity Measures' in which they use a neural network to approximate the excess risk—the difference between empirical and true risk, which the author call `the generalisation gap'—from validation data and use it as a regulariser for the model;
Nagarajan and Kolter, 2019 argue for an initialisation dependent measure of complexity; Jiang et al., 2018 approximate classification decision margins for neural networks and show their predictive generalisation gap correlates to the true generalisation gap. Maddox et al., 2020 use the effective dimensionality [MacKay, 1991] of the Hessian matrix $\nabla_{\vtheta}^2 E$ and show that it can capture the double descent phenomenon; since effective dimensionality depends on the eigenspectrum of the Hessian which is computationally expensive to compute for large networks, the authors select the leading eigenvalues to estimate the metric.

Most recently, Grant and Wu, 2022 modernised the Generalised Degrees of Freedom
(GDof) [Efron, 1983, Ye, 1998] complexity measure by providing a tractable approach to estimate it online during training. GDof is a similar measure to GC in that it computes the model sensitivity with respect to the dataset labels (GC instead computes the sensitivity to inputs). Grant and Wu, 2022 showed that GDof can resolve the double descent phenomenon; we hope that future work will assess the connection between GC and GDof both empirically and theoretically.

Huh et al., 2023 argue that deep neural networks exhibit a low-rank bias, measured as the rank of the Gram matrix of feature embeddings.
By measuring a relationship between data points, a low-rank measure is less local than GC, which assess how sensitive the model is to individual input changes.
Huh et al., 2023 connect low-rank bias with depth and initialisation, but contend that depth plays a more significant role than initialisation and optimisation. Instead, we have associated the implicit bias induced by GC with optimisation and optimisation hyperparameters.

Jacobian regularisation. Using GC as an explicit regulariser has existed previously as Jacobian regularisation, used to induce model robustness with respect to inputs and increase generalisation [Hoffman et al., 2019, Sokolić et al., 2017, Varga et al., 2017, Yoshida and Miyato, 2017]. Our goal here was not to introduce a new regulariser, but to show that many existing implicit or explicit regularisers minimise GC, and that GC can be used as a neural network complexity measure.

Gradient penalties. Gradient penalties are used to regularise $f: \mathbb{R}^I \rightarrow \mathbb{R}$ using a penalty of the form  $\sum_{\vx} (||\nabla_{\vx}{f(\vx; \vtheta)}||- K)^2$ with $K$ a hyperparameter; instead of minimising the Jacobian norm, these methods ensure it is close $K$. For additional discussion on gradient penalties, see Section <ref>. Gradient penalties have been primarily used for GAN training [Fedus et al., 2018, Gulrajani et al., 2017, Kodali et al., 2017];
we leave the investigation of the importance of GC outside supervised learning for future work.

Smoothness and generalisation.
Novak et al., 2018 is perhaps the closest work to ours: they show the correlation across multiple datasets between generalisation and
$\frac{1}{\mathcal{D}_{test}} \sum_{\vx_i \in \mathcal{D}_{test}} \norm{\evaljac{\vx}{(\tau \circ f(\cdot; \vtheta))}{\vx_i}}_2$  where $\tau$ is the last layer activation function (such as the softmax). They also
briefly show that their Jacobian norm metric captures the regularisation effect of stochastic gradient descent compared to full-batch training using L-BFGS [Liu and Nocedal, 1989]. Since our goal is to find a complexity measure we do not use the test set but the training set instead; our main focus is to show the connections between GC implicit and explicit regularisation, as well as initialisation in deep learning.

Bartlett et al., 2017 provide a generalisation bound depending on classification margins and the product of spectral norms of the model's weights and show empirically that the product of spectral norms correlates with excess risk. Bubeck and Sellke, 2021 connect Lipschitz smoothness with over-parametrisation, by providing a probabilistic upper bound on the model's Lipschitz constant based on its training performance and the inverse of its number of parameters.

Smoothness and double descent.
In concurrent work, Gamba et al., 2022 discuss the connection between the Jacobian and the Hessian of the loss with respect to inputs, namely the empirical average of $||\nabla_{\vx} E||_2$ and $||\nabla_{\vx}^2 E||_2$ over the training set and show that they follow a double descent curve. Like our work with GC, Gamba et al., 2022 mention how their work suggests our postulate in Chapter 3: `Finally, Rosca et al. (2020) investigate how to encourage smoothness, either on the entire input space or around training
data points. Interestingly, they postulate a connection between model-wise double descent and smoothness: during
the first ascent the model size is large enough to fit the training data at the expense of smoothness, while the second
descent happens as the model size becomes large enough for smoothness to increase. [...] Our work provides empirical
evidence supporting the postulate of Rosca et al. (2020).'

§ LIMITATIONS AND FUTURE WORK

Our experiments use the Resnet-18 and MLP architectures, and the ReLu activation function. We hope that future work will further validate these results in a variety of settings, including for other architectures, activation functions, and input domains.

Our results suggest that GC is able to capture a model's complexity and its generalisation abilities on a test set obtained using i.i.d. sampling from the same underlying distribution as the training set. We hope future work will investigate the properties of GC when the test time distribution differs from the training distribution, as is the case in transfer learning [Zhuang et al., 2020, Iman et al., 2022], or when the training set gradually changes in time, as in continual learning [Parisi et al., 2019]. We intuit that the results will depend on the distance between the training and test distribution in the case of transfer learning, or the rate of change of the input in the case of continual learning.

We believe incorporating GC for model selection has the potential to construct criteria analogous to the Bayesian information criterion [Bishop and Nasrabadi, 2006], adapted to the deep learning domain.
For architecture search GC could prove to be useful by excluding architectures that have high GC at initialisation on the training data. While we did not investigate this here, we hope this is a fruitful avenue for future work.

§ CONCLUSION

We introduced Geometric Complexity (GC), a measure of model complexity defined as the average Jacobian norm of a network's logits with respect to inputs from the training dataset. We showed that GC is closely connected to Lipschitz smoothness, but unlike the Lipschitz constant of a deep network, GC is tractable and can be measured during training. We have empirically shown how common explicit regularisers that target other quantities, such as weight norm regularisation and gradient norm regularisation, implicitly regularise GC.  We then connected GC with the implicit regularisation induced by the discretisation drift of gradient descent. We saw how mechanisms that induce higher implicit regularisation, such as decreased batch size and increased learning rate, lead to a decrease in GC. Last but not least, we showed how GC can resolve the double descent challenge associated with traditional complexity measures and recovers the expected U-shaped curve.

CHAPTER: CONCLUSION

Throughout the thesis we have seen that despite being fundamental ingredients in the melting pot behind deep learning's recipe for success, the effects of gradient-based optimisation and smoothness regularisation are far from being understood. Our aim was to take a few steps towards a practical theory of deep learning, inspired by great recent works making strides in that direction.
To this end, we started with investigating gradient descent, a perhaps deceptively simple algorithm, which forms the basis of all modern deep learning optimisation.
We took a continuous-time perspective to understanding gradient descent, either by quantifying discretisation drift between existing continuous-time flows and gradient descent or by finding new flows that better describe the gradient descent trajectory.
We then constructed novel continuous-time analyses to determine sources of instability in  supervised learning and two-player games such as Generative Adversarial Networks, which led to mitigation strategies either in the form of adaptive learning rates or explicit regularisers.
We continued our investigation by studying interactions between optimisation and models, in particular model smoothness with respect to inputs and related regularisers.
We expanded the use of smoothness regularisation to a new domain, reinforcement learning, where we observed that Spectral Normalisation leads to a consistent gain in agent performance that can be recovered through changes to optimisation.  When revisiting the use of Spectral Normalisation in Generative Adversarial Networks, however, we encountered a mixture of model regularisation and optimisation effects. We concluded by introducing a measure of model complexity based on smoothness and explored its interactions with other ingredients of the deep learning melting pot, such as implicit regularisation effects induced by discretisation drift,  initialisation, and explicit regularisation.

Motivated by our desire to dig deeper into why deep learning works so well, we employed theoretical and empirical tools to  perform an investigation into individual deep learning components and their many interactions. Using our novel intuitions and understanding of deep learning, we constructed methods that led to increased training stability or improved evaluation performance across multiple problem domains.
We hope both the approach and the results presented here can form the fruitful basis of further lines of inquiry, for the author as well as the wider community.

[Jumper et al., 2021]
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov,
Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna
Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J.
Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, Rishub
Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy,
Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer,
Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray
Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis.
Highly accurate protein structure prediction with alphafold.
In Nature, 2021.

[Ravuri et al., 2021]
Suman Ravuri, Karel Lenc, Matthew Willson, Dmitry Kangin, Remi Lam, Piotr
Mirowski, Megan Fitzsimons, Maria Athanassiadou, Sheleem Kashem, Sam Madge,
Rachel Prudden, Amol Mandhane, Aidan Clark, Andrew Brock, Karen Simonyan,
Raia Hadsell, Niall Robinson, Ellen Clancy, Alberto Arribas, and Shakir
Skilful precipitation nowcasting using deep generative models of
In Nature, 2021.

[Fawzi et al., 2022]
Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino
Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Francisco J. R.
Ruiz, Julian Schrittwieser, Grzegorz Swirszcz, David Silver, Demis Hassabis,
and Pushmeet Kohli.
Discovering faster matrix multiplication algorithms with
reinforcement learning.
In Nature, 2022.

[Ramesh et al., 2022]
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.
Hierarchical text-conditional image generation with clip latents.
arXiv preprint arXiv:2204.06125, 2022.

[Brown et al., 2020]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla
Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell,
Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon
Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris
Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess,
Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever,
and Dario Amodei.
Language models are few-shot learners.
In Advances in Neural Information Processing Systems, 2020.

[Hoffmann et al., 2022]
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor
Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes
Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den
Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich
Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre.
Training compute-optimal large language models.
In Advances in Neural Information Processing Systems, 2022.

[Sauer et al., 2023]
Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila.
Stylegan-t: Unlocking the power of gans for fast large-scale
text-to-image synthesis.
arXiv preprint arXiv:2301.09515, 2023.

[Silver et al., 2017]
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja
Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton,
Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den
Driessche, Thore Graepel, and Demis Hassabis.
Mastering the game of go without human knowledge.
In Nature, 2017a.

[Silver et al., 2017]
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew
Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore
Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis.
Mastering chess and shogi by self-play with a general reinforcement
learning algorithm.
arXiv preprint arXiv:1712.01815, 2017b.

[Berner et al., 2019]
OpenAI: Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung,
Przemysław Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq
Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub
Pachocki, Michael Petrov, Henrique P. d.O. Pinto, Jonathan Raiman, Tim
Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever,
Jie Tang, Filip Wolski, and Susan Zhang.
Dota 2 with large scale deep reinforcement learning.
arXiv preprint arXiv:1912.06680, 2019.

[Arora et al., 2018]
Sanjeev Arora, Nadav Cohen, and Elad Hazan.
On the optimization of deep networks: Implicit acceleration by
In International Conference on Machine Learning,

[Barrett and Dherin, 2021]
David GT Barrett and Benoit Dherin.
Implicit gradient regularization.
In International Conference on Learning Representations, 2021.

[Smith et al., 2021]
Samuel L Smith, Benoit Dherin, David GT Barrett, and Soham De.
On the origin of implicit regularization in stochastic gradient
In International Conference on Learning Representations, 2021.

[Arora et al., 2018]
Raman Arora, Amitabh Basu, Poorya Mianjy, and Anirbit Mukherjee.
Understanding deep neural networks with rectified linear units.
In International Conference on Learning Representations,

[Mescheder et al., 2017]
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger.
The numerics of gans.
In Advances in Neural Information Processing Systems, 2017.

[Nagarajan and Kolter, 2019]
Vaishnavh Nagarajan and J Zico Kolter.
Generalization in deep networks: The role of distance from

[Nagarajan and Kolter, 2017]
Vaishnavh Nagarajan and J Zico Kolter.
Gradient descent gan optimization is locally stable.
In Advances in Neural Information Processing Systems, 2017.

[Keskar et al., 2017]
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy,
and Ping Tak Peter Tang.
On large-batch training for deep learning: Generalization gap and
sharp minima.
International Conference for Learing Representations, 2017.

[Dinh et al., 2017]
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio.
Sharp minima can generalize for deep nets.
In International Conference on Machine Learning, 2017.

[Jiang et al., 2019]
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy
Fantastic generalization measures and where to find them.
In International Conference on Learning Representations, 2019.

[Smith et al., 2018]
Samuel L. Smith, Pieter-Jan Kindermans, and Quoc V. Le.
Don't decay the learning rate, increase the batch size.
In International Conference on Learning Representations, 2018.

[Li et al., 2018]
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein.
Visualizing the loss landscape of neural nets.
In Advances in Neural Information Processing Systems, 2018.

[Bjorck et al., 2018]
Nils Bjorck, Carla P Gomes, Bart Selman, and Kilian Q Weinberger.
Understanding batch normalization.
In Advances in Neural Information Processing Systems, 2018.

[Brock et al., 2021]
Andrew Brock, Soham De, and Samuel L Smith.
Characterizing signal propagation to close the performance gap in
unnormalized resnets.
In International Conference on Learning Representations, 2021.

[Ding et al., 2022]
Zhiyan Ding, Shi Chen, Qin Li, and Stephen J Wright.
Overparameterization of deep resnet: zero loss and mean-field
In Journal of machine learning research, 2022.

[Sriperumbudur et al., 2009]
Bharath K Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard
Schölkopf, and Gert RG Lanckriet.
On integral probability metrics,$\backslash$phi-divergences and
binary classification.
arXiv preprint arXiv:0901.2698, 2009.

[Arjovsky et al., 2017]
Martin Arjovsky, Soumith Chintala, and Léon Bottou.
Wasserstein generative adversarial networks.
In International Conference on Machine Learning, 2017.

[Fedus et al., 2018]
William Fedus, Mihaela Rosca, Balaji Lakshminarayanan, Andrew M Dai, Shakir
Mohamed, and Ian Goodfellow.
Many paths to equilibrium: Gans do not need to decrease a divergence
at every step.
In International Conference on Learning Representations, 2018.

[Hoffman et al., 2019]
Judy Hoffman, Daniel A Roberts, and Sho Yaida.
Robust learning with jacobian regularization.
arXiv preprint arXiv:1908.02729, 2019.

[Miyato et al., 2018]
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.
Spectral normalization for generative adversarial networks.
In International Conference on Learning Representations, 2018.

[Radford et al., 2015]
Alec Radford, Luke Metz, and Soumith Chintala.
Unsupervised representation learning with deep convolutional
generative adversarial networks.
In International Conference on Learning Representations, 2015.

[Goodfellow et al., 2014]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,
Sherjil Ozair, Aaron Courville, and Yoshua Bengio.
Generative adversarial nets.
In Advances in Neural Information Processing Systems, 2014.

[Brock et al., 2018]
Andrew Brock, Jeff Donahue, and Karen Simonyan.
Large scale gan training for high fidelity natural image synthesis.
In International Conference on Learning Representations, 2018.

[Karras et al., 2019]
Tero Karras, Samuli Laine, and Timo Aila.
A style-based generator architecture for generative adversarial
In Conference on Computer Vision and Pattern Recognition,

[Nowozin et al., 2016]
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka.
f-gan: Training generative neural samplers using variational
divergence minimization.
In Advances in Neural Information Processing Systems,

[Gulrajani et al., 2017]
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C
Improved training of wasserstein gans.
In Advances in Neural Information Processing Systems, 2017.

[Bińkowski et al., 2018]
Mikołaj Bińkowski, Dougal J Sutherland, Michael Arbel, and Arthur
Demystifying mmd gans.
In International Conference on Learning Representations, 2018.

[Mao et al., 2017]
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen
Paul Smolley.
Least squares generative adversarial networks.
In International Conference on Computer Vision, 2017.

[Zhang et al., 2019]
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena.
Self-attention generative adversarial networks.
In International Conference on Machine Learning, 2019.

[Karras et al., 2021]
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne
Hellsten, Jaakko Lehtinen, and Timo Aila.
Alias-free generative adversarial networks.
In Advances in Neural Information Processing Systems, 2021.

[de Masson d'Autume et al., 2019]
Cyprien de Masson d'Autume, Shakir Mohamed, Mihaela Rosca, and Jack Rae.
Training language gans from scratch.
In Advances in Neural Information Processing Systems, 2019.

[Werbos, 1982]
Paul J Werbos.
Applications of advances in nonlinear sensitivity analysis.
In System modeling and optimization, 1982.

[Rumelhart et al., 1986]
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams.
Learning representations by back-propagating errors.
In Nature, 1986.

[LeCun et al., 1989]
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard,
Wayne Hubbard, and Lawrence D Jackel.
Backpropagation applied to handwritten zip code recognition.
In Neural Computation, 1989.

[45]
Augustin Cauchy et al.
Méthode générale pour la résolution des systemes
d’équations simultanées.
In Comp. Rend. Sci. Paris, number 1847.

[Robbins and Monro, 1951]
Herbert Robbins and Sutton Monro.
A stochastic approximation method.
In The annals of mathematical statistics, 1951.

[Kingma and Ba, 2015]
Diederik P Kingma and Jimmy Ba.
Adam: A method for stochastic optimization.
In International Conference on Learning Representations, 2015.

[Tieleman and Hinton, 2012]
Tijmen Tieleman and Geoffrey Hinton.
Rmsprop: Divide the gradient by a running average of its recent
magnitude. coursera: Neural networks for machine learning.
Coursera Neural Networks Machine Learning course, 2012.

[Du et al., 2019]
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai.
Gradient descent finds global minima of deep neural networks.
In International Conference on Machine Learning, 2019.

[Zou et al., 2020]
Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu.
Gradient descent optimizes over-parameterized deep relu networks.
In Machine Learning. Springer, 2020.

[Du et al., 2017]
Simon S Du, Chi Jin, Jason D Lee, Michael I Jordan, Aarti Singh, and Barnabas
Gradient descent can take exponential time to escape saddle points.
In Advances in Neural Information Processing Systems, 2017.

[Cohen et al., 2021]
Jeremy M Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar.
Gradient descent on neural networks typically occurs at the edge of
In International Conference on Learning Representations, 2021.

[Lewkowycz et al., 2020]
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy
The large learning rate phase of deep learning: the catapult
arXiv preprint arXiv:2003.02218, 2020.

[Elkabetz and Cohen, 2021]
Omer Elkabetz and Nadav Cohen.
Continuous vs. discrete optimization of deep neural networks.
In Advances in Neural Information Processing Systems, 2021.

[Glendinning, 1994]
Paul Glendinning.
Stability, instability and chaos: an introduction to the theory
of nonlinear differential equations.
Cambridge university press, 1994.

[Saxe et al., 2014]
Andrew M Saxe, James L McClelland, and Surya Ganguli.
Exact solutions to the nonlinear dynamics of learning in deep linear
neural networks.
In International Conference on Learning Represenatations,

[Lampinen and Ganguli, 2019]
Andrew K Lampinen and Surya Ganguli.
An analytic theory of generalization dynamics and transfer learning
in deep linear networks.
In International Conference on Learning Represenatations,

[Advani et al., 2020]
Madhu S Advani, Andrew M Saxe, and Haim Sompolinsky.
High-dimensional dynamics of generalization error in neural networks.
In Neural Networks, 2020.

[Vardi and Shamir, 2021]
Gal Vardi and Ohad Shamir.
Implicit regularization in relu networks with the square loss.
In Conference on Learning Theory, 2021.

[França et al., 2020]
Guilherme França, Jeremias Sulam, Daniel Robinson, and René Vidal.
Conformal symplectic and relativistic optimization.
In Advances in Neural Information Processing Systems, 2020.

[Balduzzi et al., 2018]
David Balduzzi, Sebastien Racaniere, James Martens, Jakob Foerster, Karl Tuyls,
and Thore Graepel.
The mechanics of n-player differentiable games.
In International Conference on Machine Learning, 2018.

[May, 1976]
Robert M May.
Simple mathematical models with very complicated dynamics.
In Nature, 1976.

[Yaida, 2018]
Sho Yaida.
Fluctuation-dissipation relations for stochastic gradient descent.
In International Conference on Learning Representations, 2018.

[Liu et al., 2021]
Kangqiao Liu, Liu Ziyin, and Masahito Ueda.
Noise and fluctuation of finite learning rate stochastic gradient
In International Conference on Machine Learning, 2021.

[Kunin et al., 2021]
Daniel Kunin, Javier Sagastuy-Brena, and Hidenori Tanaka Ganguli, Surya Daniel
L.K. Yamins.
Symmetry, conservation laws, and learning dynamics in neural
In International Conference on Learning Representations, 2021.

[Schäfer and Anandkumar, 2019]
Florian Schäfer and Anima Anandkumar.
Competitive gradient descent.
In Advances in Neural Information Processing Systems, 2019.

[Qin et al., 2020]
Chongli Qin, Yan Wu, Jost Tobias Springenberg, Andy Brock, Jeff Donahue,
Timothy Lillicrap, and Pushmeet Kohli.
Training generative adversarial networks by solving ordinary
differential equations.
In Advances in Neural Information Processing Systems, 2020.

[Mescheder et al., 2018]
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin.
Which training methods for gans do actually converge?
In International Conference on Machine Learning, 2018.

[Wang et al., 2019]
Yuanhao Wang, Guodong Zhang, and Jimmy Ba.
On solving minimax optimization locally: A follow-the-ridge approach.
In International Conference on Learning Representations, 2019.

[Mazumdar et al., 2019]
Eric V Mazumdar, Michael I Jordan, and S Shankar Sastry.
On finding local nash equilibria (and only local nash equilibria) in
zero-sum games.
arXiv preprint arXiv:1901.00838, 2019.

[Parisi et al., 2019]
German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan
Continual lifelong learning with neural networks: A review.
In Neural Networks, 2019.

[Zhuang et al., 2020]
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui
Xiong, and Qing He.
A comprehensive survey on transfer learning.
In Proceedings of the Institute of Electrical and Electronics
Engineers, 2020.

[Shalev-Shwartz and Ben-David, 2014]
Shai Shalev-Shwartz and Shai Ben-David.
Understanding machine learning: From theory to algorithms.
Cambridge university press, 2014.

[Mitchell, 1980]
Tom M Mitchell.
The need for biases in learning generalizations.
Citeseer, 1980.

[MacKay, 1991]
David MacKay.
Bayesian model comparison and backprop nets.
In Advances in Neural Information Processing Systems, 1991.

[Srivastava et al., 2014]
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan
Dropout: a simple way to prevent neural networks from overfitting.
In The Journal of Machine Learning Research, 2014.

[Belkin, 2021]
Mikhail Belkin.
Fit without fear: remarkable mathematical phenomena of deep learning
through the prism of interpolation.
In Acta Numerica. Cambridge University Press, 2021.

[Belkin et al., 2019]
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal.
Reconciling modern machine-learning practice and the classical
bias–variance trade-off.
In National Academy of Sciences, 2019a.

[Nakkiran et al., 2019]
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya
Deep double descent: Where bigger models and more data hurt.
In International Conference on Learning Representations, 2019.

[Yoshida and Miyato, 2017]
Yuichi Yoshida and Takeru Miyato.
Spectral norm regularization for improving the generalizability of
deep learning.
arXiv preprint arXiv:1705.10941, 2017.

[Bartlett et al., 2017]
Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky.
Spectrally-normalized margin bounds for neural networks.
In Advances in Neural Information Processing Systems, 2017.

[Sokolić et al., 2017]
Jure Sokolić, Raja Giryes, Guillermo Sapiro, and Miguel RD Rodrigues.
Robust large margin deep neural networks.
In Transactions on Signal Processing, 2017.

[Novak et al., 2018]
Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha
Sensitivity and generalization in neural networks: an empirical
In International Conference on Learning Representations, 2018.

[Cisse et al., 2017]
Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas
Parseval networks: improving robustness to adversarial examples.
In International Conference on Machine Learning, 2017.

[Vahdat and Kautz, 2020]
Arash Vahdat and Jan Kautz.
Nvae: A deep hierarchical variational autoencoder.
In Advances in Neural Information Processing Systems, 2020.

[He et al., 2016]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition.
In Conference on Computer Vision and Pattern Recognition,

[Ioffe and Szegedy, 2015]
Sergey Ioffe and Christian Szegedy.
Batch normalization: Accelerating deep network training by reducing
internal covariate shift.
In International Conference on Machine Learning, 2015.

[Rosca et al., 2022]
Mihaela Rosca, Yan Wu, Chongli Qin, and Benoit Dherin.
On a continuous-time model of gradient descent dynamics and
instability in deep learning.
In Transactions on Machine Learning Research, 2022.

[Rosca et al., 2021]
Mihaela Rosca, Yan Wu, Benoit Dherin, and David GT Barrett.
Discretization drift in two-player games.
In International Conference on Machine Learning,

[Rosca et al., 2020]
Mihaela Rosca, Theophane Weber, Arthur Gretton, and Shakir Mohamed.
A case for new neural network smoothness constraints.

[Gogianu et al., 2021]
Florin Gogianu, Tudor Berariu, Mihaela Rosca, Claudia Clopath, Lucian Busoniu,
and Razvan Pascanu.
Spectral normalisation for deep reinforcement learning: an
optimisation perspective.
In International Conference on Machine Learning, 2021.

[Dherin et al., 2022]
Benoit Dherin, Michael Munn, Mihaela Rosca, and David GT Barrett.
Why neural networks find simple solutions: The many regularizers of
geometric complexity.
In Advances in Neural Information Processing Systems, 2022.

[Chen et al., 2018]
Changyou Chen, Chunyuan Li, Liqun Chen, Wenlin Wang, Yunchen Pu, and
Lawrence Carin Duke.
Continuous-time flows for efficient inference and density estimation.
In International Conference on Machine Learning,

[Chen et al., 2018]
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud.
Neural ordinary differential equations.
In Advances in Neural Information Processing Systems,

[Kidger, 2022]
Patrick Kidger.
On neural differential equations.
arXiv preprint arXiv:2202.02435, 2022.

[Grathwohl et al., 2019]
Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, Ilya Sutskever, and David
Ffjord: Free-form continuous dynamics for scalable reversible
generative models.
In International Conference on Learning Representations, 2019.

[Song et al., 2021]
Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon.
Maximum likelihood training of score-based diffusion models.
In Advances in Neural Information Processing Systems, 2021.

[Mohamed et al., 2020]
Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih.
Monte carlo gradient estimation in machine learning.
In Journal of Machine Learning Research, 2020.

[Hairer et al., 2006]
Ernst Hairer, Marlis Hochbruck, Arieh Iserles, and Christian Lubich.
Geometric numerical integration.
Oberwolfach Reports, 2006a.

[Jacot et al., 2018]
Arthur Jacot, Franck Gabriel, and Clément Hongler.
Neural tangent kernel: Convergence and generalization in neural
In Advances in Neural Information Processing Systems, 2018.

[Du et al., 2018]
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh.
Gradient descent provably optimizes over-parameterized neural
In International Conference on Learning Representations,

[Bartlett et al., 2018]
Peter Bartlett, Dave Helmbold, and Philip Long.
Gradient descent with identity initialization efficiently learns
positive definite linear transformations by deep residual networks.
In International Conference on Machine Learning,

[Dauphin et al., 2014]
Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli,
and Yoshua Bengio.
Identifying and attacking the saddle point problem in
high-dimensional non-convex optimization.
In Advances in Neural Information Processing Systems, 2014.

[Li et al., 2021]
Zhiyuan Li, Yi Zhang, and Sanjeev Arora.
Why are convolutional nets more sample-efficient than fully-connected
In International Conference on Learning Representations, 2021.

[Li et al., 2019]
Yuanzhi Li, Colin Wei, and Tengyu Ma.
Towards explaining the regularization effect of initial large
learning rate in training neural networks.
In Advances in Neural Information Processing Systems, 2019.

[Riedmiller and Braun, 1993]
Martin Riedmiller and Heinrich Braun.
A direct adaptive method for faster backpropagation learning: The
rprop algorithm.
In International Conference on Neural Networks, 1993.

[Mnih et al., 2015]
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness,
Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland,
Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis
Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and
Demis Hassabis.
Human-level control through deep reinforcement learning.
In Nature, 2015.

[Kearney et al., 2018]
Alex Kearney, Vivek Veeriah, Jaden B Travnik, Richard S Sutton, and Patrick M
Tidbd: Adapting temporal-difference step-sizes through stochastic
arXiv preprint arXiv:1804.03334, 2018.

[Hessel et al., 2018]
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski,
Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver.
Rainbow: Combining improvements in deep reinforcement learning.
In Association for the Advancement of Artificial Intelligence,

[Sutskever et al., 2013]
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton.
On the importance of initialization and momentum in deep learning.
In International Conference on Machine Learning, 2013.

[Polyak, 1964]
Boris T Polyak.
Some methods of speeding up the convergence of iteration methods.
Ussr computational mathematics and mathematical physics, 1964.

[Gupta et al., 2021]
Aman Gupta, Rohan Ramanath, Jun Shi, and S Sathiya Keerthi.
Adam vs. sgd: Closing the generalization gap on image classification.
In Workshop on Optimization for Machine Learning, Advances in
Neural Information Processing Systems, 2021.

[Zhou et al., 2020]
Pan Zhou, Jiashi Feng, Chao Ma, Caiming Xiong, Steven Hoi, and Weinan E.
Towards theoretically understanding why sgd generalizes better than
adam in deep learning.
In Advances in Neural Information Processing Systems, 2020.

[Reddi et al., 2019]
Sashank J Reddi, Satyen Kale, and Sanjiv Kumar.
On the convergence of adam and beyond.
In International Conference on Learning Representations, 2019.

[Liu et al., 2020]
Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han.
Understanding the difficulty of training transformers.
arXiv preprint arXiv:2004.08249, 2020a.

[Choi et al., 2019]
Dami Choi, Christopher J Shallue, Zachary Nado, Jaehoon Lee, Chris J Maddison,
and George E Dahl.
On empirical comparisons of optimizers for deep learning.
arXiv preprint arXiv:1910.05446, 2019.

[Bellemare et al., 2017]
Marc G Bellemare, Will Dabney, and Rémi Munos.
A distributional perspective on reinforcement learning.
In International Conference on Machine Learning, 2017.

[Hartman, 1960]
Philip Hartman.
A lemma in the theory of structural stability of differential
In Proceedings of the American Mathematical Society, 1960.

[Metz et al., 2017]
Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein.
Unrolled generative adversarial networks.
In International Conference on Learning Representations, 2017.

[Jin et al., 2020]
Chi Jin, Praneeth Netrapalli, and Michael Jordan.
What is local optimality in nonconvex-nonconcave minimax
In International Conference on Machine Learning, 2020.

[Farnia and Ozdaglar, 2020]
Farzan Farnia and Asuman Ozdaglar.
Do gans always have nash equilibria?
In International Conference on Machine Learning, 2020.

[Fiez et al., 2020]
Tanner Fiez, Benjamin Chasnov, and Lillian J Ratliff.
Convergence of learning dynamics in stackelberg games.
In International Conference on Machine Learning, 2020.

[Berard et al., 2020]
Hugo Berard, Gauthier Gidel, Amjad Almahairi, Pascal Vincent, and Simon
A closer look at the optimization landscapes of generative
adversarial networks.
In International Conference on Learning Representations, 2020.

[Gur-Ari et al., 2018]
Guy Gur-Ari, Daniel A Roberts, and Ethan Dyer.
Gradient descent happens in a tiny subspace.
arXiv preprint arXiv:1812.04754, 2018.

[Gilmer et al., 2022]
Justin Gilmer, Behrooz Ghorbani, Ankush Garg, Sneha Kudugunta, Behnam
Neyshabur, David Cardoze, George Dahl, Zachary Nado, and Orhan Firat.
A loss curvature perspective on training instability in deep
In International Conference on Learning Representations, 2022.

[Bartlett et al., 2018]
Peter L Bartlett, Steven N Evans, and Philip M Long.
Representing smooth functions as compositions of near-identity
functions with implications for deep network optimization.
arXiv preprint arXiv:1804.05012, 2018b.

[Gunasekar et al., 2018]
Suriya Gunasekar, Jason D Lee, Daniel Soudry, and Nati Srebro.
Implicit bias of gradient descent on linear convolutional networks.
In Advances in Neural Information Processing Systems,

[Allen-Zhu et al., 2019]
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song.
A convergence theory for deep learning via over-parameterization.
In International Conference on Machine Learning, 2019.

[Du and Hu, 2019]
Simon Du and Wei Hu.
Width provably matters in optimization for deep linear neural
In International Conference on Machine Learning, 2019.

[Ziyin et al., 2021]
Liu Ziyin, Botao Li, James B Simon, and Masahito Ueda.
Sgd can converge to local maxima.
In International Conference on Learning Representations, 2021.

[Du et al., 2018]
Simon S Du, Wei Hu, and Jason D Lee.
Algorithmic regularization in learning deep homogeneous models:
Layers are automatically balanced.
In Advances in Neural Information Processing Systems,

[Amari, 1998]
Shun-Ichi Amari.
Natural gradient works efficiently in learning.
In Neural computation, 1998.

[Ollivier, 2015]
Yann Ollivier.
Riemannian metrics for neural networks: feedforward networks.
In Information and Inference: A Journal of the Institute of
Mathematics, 2015a.

[Ollivier, 2015]
Yann Ollivier.
Riemannian metrics for neural networks ii: recurrent networks and
learning symbolic data sequences.
Information and Inference: A Journal of the IMA,

[Song et al., 2018]
Yang Song, Jiaming Song, and Stefano Ermon.
Accelerating natural gradient with higher-order invariance.
In International Conference on Machine Learning, 2018.

[Rosca et al., 2021]
Mihaela C Rosca, Yan Wu, Benoit Dherin, and David GT Barrett.
Discretization drift in two-player games.
In International Conference on Machine Learning,

[Lee et al., 2019]
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha
Sohl-Dickstein, and Jeffrey Pennington.
Wide neural networks of any depth evolve as linear models under
gradient descent.
In Advances in Neural Information Processing Systems, 2019.

[Wanner and Hairer, 1996]
Gerhard Wanner and Ernst Hairer.
Solving ordinary differential equations II.
Springer Berlin Heidelberg New York, 1996.

[Rosenbrock, 1960]
HoHo Rosenbrock.
An automatic method for finding the greatest or least value of a
The computer journal, 1960.

[Asuncion and Newman, 2007]
Arthur Asuncion and David Newman.
Uci machine learning repository, 2007.

[Jastrzebski et al., 2019]
Stanisław Jastrzebski, Zachary Kenton, Nicolas Ballas, Asja Fischer, Yoshua
Bengio, and Amos Storkey.
On the relation between the sharpest directions of dnn loss and the
sgd step length.
In International Conference on Learning Representations,

[Lee et al., 2016]
Jason D Lee, Max Simchowitz, Michael I Jordan, and Benjamin Recht.
Gradient descent only converges to minimizers.
In Conference on learning theory, 2016.

[Kawaguchi, 2016]
Kenji Kawaguchi.
Deep learning without poor local minima.
In Advances in Neural Information Processing Systems, 2016.

[Sagun et al., 2017]
Levent Sagun, Utku Evci, V Ugur Guney, Yann Dauphin, and Leon Bottou.
Empirical analysis of the hessian of over-parametrized neural
In Workshop track - International Conference on Learning
Representations 2018, 2017.

[Ghorbani et al., 2019]
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao.
An investigation into neural net optimization via hessian eigenvalue
In International Conference on Machine Learning, 2019.

[Papyan, 2018]
Vardan Papyan.
The full spectrum of deepnet hessians at scale: Dynamics with sgd
training and sample size.
arXiv preprint arXiv:1811.07062, 2018.

[Petersen and Pedersen, 2008]
Kaare Brandt Petersen and Michael Syskind Pedersen.
The matrix cookbook.

[Damian et al., 2022]
Alex Damian, Eshaan Nichani, and Jason D Lee.
Self-stabilization: The implicit bias of gradient descent at the edge
of stability.
In Workshop on Optimization for Machine Learning, Advances in
Neural Information Processing Systems, 2022.

[Loshchilov and Hutter, 2017]
Ilya Loshchilov and Frank Hutter.
Sgdr: Stochastic gradient descent with warm restarts.
In International Conference on Learning Representations, 2017.

[He et al., 2019]
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li.
Bag of tricks for image classification with convolutional neural
In Conference on Computer Vision and Pattern Recognition,

[Geiping et al., 2022]
Jonas Geiping, Micah Goldblum, Phillip E Pope, Michael Moeller, and Tom
Stochastic training is not necessary for generalization.
In International Conference on Learning Representations, 2022.

[Jastrzebski et al., 2019]
Stanisław Jastrzebski, Maciej Szymczak, Stanislav Fort, Devansh Arpit, Jacek
Tabor, Kyunghyun Cho, and Krzysztof Geras.
The break-even point on optimization trajectories of deep neural
In International Conference on Learning Representations,

[Foret et al., 2020]
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur.
Sharpness-aware minimization for efficiently improving
In International Conference on Learning Representations, 2020.

[Duchi et al., 2011]
John Duchi, Elad Hazan, and Yoram Singer.
Adaptive subgradient methods for online learning and stochastic
In Journal of machine learning research, 2011.

[Lee et al., 2020]
Jaehoon Lee, Samuel Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao,
Roman Novak, and Jascha Sohl-Dickstein.
Finite versus infinite neural networks: an empirical study.
In Advances in Neural Information Processing Systems,

[Shi et al., 2021]
Bin Shi, Simon S Du, Michael I Jordan, and Weijie J Su.
Understanding the acceleration phenomenon via high-resolution
differential equations.
In Mathematical Programming, 2021.

[Kovachki and Stuart, 2021]
Nikola B Kovachki and Andrew M Stuart.
Continuous time analysis of momentum methods.
In Journal of Machine Learning Research, 2021.

[França et al., 2021]
Guilherme França, Michael I Jordan, and René Vidal.
On dissipative symplectic integration with applications to
gradient-based optimization.
Journal of Statistical Mechanics: Theory and Experiment, 2021.

[Li et al., 2017]
Qianxiao Li, Cheng Tai, and Weinan E.
Stochastic modified equations and adaptive stochastic gradient
In International Conference on Machine Learning, 2017.

[160]
Taiki Miyagawa.
Toward equation of motion for deep neural networks: Continuous-time
gradient descent and discretization error analysis.
In Advances in Neural Information Processing Systems.

[Ahn et al., 2022]
Kwangjun Ahn, Jingzhao Zhang, and Suvrit Sra.
Understanding the unstable convergence of gradient descent.
In International Conference on Machine Learning, 2022.

[Arora et al., 2022]
Sanjeev Arora, Zhiyuan Li, and Abhishek Panigrahi.
Understanding gradient descent on edge of stability in deep learning.
In International Conference on Machine Learning,, 2022.

[Ma et al., 2022]
Chao Ma, Lei Wu, and Lexing Ying.
The multiscale structure of neural network loss functions: The effect
on optimization and origin.
arXiv preprint arXiv:2204.11326, 2022.

[Chen and Bruna, 2022]
Lei Chen and Joan Bruna.
On gradient descent convergence beyond the edge of stability.
arXiv preprint arXiv:2206.04172, 2022.

[Ziyin et al., 2022]
Liu Ziyin, Kangqiao Liu, Takashi Mori, and Masahito Ueda.
Strength of minibatch noise in sgd.
In International Conference on Learning Representations, 2022.

[Fletcher, 2013]
Roger Fletcher.
Practical methods of optimization.
John Wiley & Sons, 2013.

[Martens and Grosse, 2015]
James Martens and Roger Grosse.
Optimizing neural networks with kronecker-factored approximate
In International Conference on Machine Learning, 2015.

[Hochreiter and Schmidhuber, 1997]
Sepp Hochreiter and Jürgen Schmidhuber.
Flat minima.
In Neural computation, 1997.

[Kaur et al., 2022]
Simran Kaur, Jeremy Cohen, and Zachary C Lipton.
On the maximum hessian eigenvalue and generalization.

[Sutton and Barto, 2018]
Richard S Sutton and Andrew G Barto.
Reinforcement learning: An introduction.
MIT press, 2018.

[Rajeswaran et al., 2020]
Aravind Rajeswaran, Igor Mordatch, and Vikash Kumar.
A game theoretic framework for model based reinforcement learning.
In International Conference on Machine Learning, 2020.

[Singh et al., 2000]
Satinder P Singh, Michael J Kearns, and Yishay Mansour.
Nash convergence of gradient dynamics in general-sum games.
In Uncertainty in Artificial Intelligence, 2000.

[Heusel et al., 2017]
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp
Gans trained by a two time-scale update rule converge to a local nash
In Advances in Neural Information Processing Systems, 2017.

[Neal, 2001]
Radford M Neal.
Annealed importance sampling.
Statistics and computing, 2001.

[Jolicoeur-Martineau, 2019]
Alexia Jolicoeur-Martineau.
The relativistic discriminator: a key element missing from standard
In International Conference on Learning Representations, 2019.

[Grover et al., 2018]
Aditya Grover, Manik Dhar, and Stefano Ermon.
Flow-gan: Combining maximum likelihood and adversarial learning in
generative models.
In Proceedings of the AAAI conference on artificial
intelligence, 2018.

[Salimans et al., 2016]
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and
Xi Chen.
Improved techniques for training gans.
In Advances in Neural Information Processing Systems, 2016.

[Lu, 2021]
Haihao Lu.
An $o(sr)$-resolution ode framework for understanding discrete-time
algorithms and applications to the linear convergence of minimax problems.
In Mathematical Programming, 2021.

[Hairer and Lubich, 1997]
Ernst Hairer and Christian Lubich.
The life-span of backward error analysis for numerical integrators.
In Numerische Mathematik, 1997.

[Daskalakis and Panageas, 2018]
Constantinos Daskalakis and Ioannis Panageas.
The limit points of (optimistic) gradient descent in min-max
In Advances in Neural Information Processing Systems, 2018.

[You et al., 2018]
Yang You, Zhao Zhang, Cho-Jui Hsieh, James Demmel, and Kurt Keutzer.
Imagenet training in minutes.
In International Conference on Parallel Processing, 2018.

[Meurer et al., 2017]
Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, Ondřej
Čertík, Sergey B. Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu
Ivanov, Jason K. Moore, Sartaj Singh, Thilina Rathnayake, Sean Vig, Brian E.
Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats,
Fredrik Johansson, Fabian Pedregosa, Matthew J. Curry, Andy R. Terrel,
Štěpán Roučka, Ashutosh Saboo, Isuru Fernando, Sumith Kulal,
Robert Cimrman, and Anthony Scopatz.
Sympy: symbolic computing in python.
PeerJ Computer Science, 2017.

[Unlu and Aitchison, 2020]
Ali Unlu and Laurence Aitchison.
Gradient regularisation as approximate variational inference.
In Symposium on Advances in Approximate Bayesian Inference,

[Jastrzebski et al., 2021]
Stanisław Kamil Jastrzebski, Devansh Arpit, Oliver Åstrand, Giancarlo Kerg,
Huan Wang, Caiming Xiong, Richard Socher, Kyunghyun Cho, and Krzysztof J.
Catastrophic fisher explosion: Early phase fisher matrix impacts
In International Conference on Machine Learning, 2021.

[Schäfer et al., 2020]
Florian Schäfer, Hongkai Zheng, and Animashree Anandkumar.
Implicit competitive regularization in gans.
In International Conference on Machine Learning, 2020.

[Su et al., 2016]
Weijie Su, Stephen Boyd, and Emmanuel J Candes.
A differential equation for modeling nesterov's accelerated gradient
method: theory and insights.
The Journal of Machine Learning Research, 2016.

[Wibisono et al., 2016]
Andre Wibisono, Ashia C Wilson, and Michael I Jordan.
A variational perspective on accelerated methods in optimization.
proceedings of the National Academy of Sciences, 2016.

[Shoham and Leyton-Brown, 2008]
Yoav Shoham and Kevin Leyton-Brown.
Multiagent systems: Algorithmic, game-theoretic, and logical
Cambridge University Press, 2008.

[Iman et al., 2022]
Mohammadreza Iman, Khaled Rasheed, and Hamid R Arabnia.
A review of deep transfer learning and recent advancements.
arXiv preprint arXiv:2201.09679, 2022.

[Goodfellow et al., 2015]
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy.
Explaining and harnessing adversarial examples.
In International Conference on Learning Representations, 2015.

[He et al., 2016]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition.
In Conference on Computer Vision and Pattern Recognition,

[Lassance et al., 2018]
Carlos Eduardo Rosar Kos Lassance, Vincent Gripon, and Antonio Ortega.
Laplacian networks: Bounding indicator function smoothness for neural
network robustness.
In Transactions on Signal and Information Processing, 2018.

[D'Oro and Jaśkowski, 2020]
Pierluca D'Oro and Wojciech Jaśkowski.
How to learn a useful critic? model-based action-gradient-estimator
policy optimization.
In Advances in Neural Information Processing Systems, 2020.

[Tschannen et al., 2020]
Michael Tobias Tschannen, Josip Djolonga, Paul Kishan Rubenstein, Sylvain
Gelly, and Mario Lučić.
On mutual information maximization for representation learning.
In International Conference on Learning Representations, 2020.

[Federer, 1996]
Herbert Federer.
Geometric measure theory.
Springer, 1996.

[Loshchilov and Hutter, 2018]
Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization.
In International Conference on Learning Representations, 2018.

[Mises and Pollaczek-Geiringer, 1929]
RV Mises and Hilda Pollaczek-Geiringer.
Praktische verfahren der gleichungsauflösung.
ZAMM-Journal of Applied Mathematics and Mechanics/Zeitschrift
für Angewandte Mathematik und Mechanik, 90 (1):0 58–77,

[Gouk et al., 2020]
Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael J Cree.
Regularisation of neural networks by enforcing lipschitz continuity.
In Machine Learning, 2020a.

[Kim et al., 2021]
Hyunjik Kim, George Papamakarios, and Andriy Mnih.
The lipschitz constant of self-attention.
In International Conference on Machine Learning, 2021.

[Tsuzuku et al., 2018]
Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama.
Lipschitz-margin training: Scalable certification of perturbation
invariance for deep neural networks.
In Advances in Neural Information Processing Systems, 2018.

[Arbel et al., 2018]
Michael Arbel, Dougal Sutherland, Mikołaj Bińkowski, and Arthur Gretton.
On gradient regularizers for mmd gans.
In Advances in Neural Information Processing Systems, 2018.

[Kodali et al., 2017]
Naveen Kodali, Jacob Abernethy, James Hays, and Zsolt Kira.
On convergence and stability of gans.
arXiv preprint arXiv:1705.07215, 2017.

[Vapnik, 2013]
Vladimir Vapnik.
The nature of statistical learning theory.
Springer science & business media, 2013.

[Rasmussen and Ghahramani, 2001]
Carl Edward Rasmussen and Zoubin Ghahramani.
Occam's razor.
In Advances in Neural Information Processing Systems, 2001.

[Bartlett, 1997]
Peter L Bartlett.
For valid generalization the size of the weights is more important
than the size of the network.
In Advances in Neural Information Processing Systems, 1997.

[Golowich et al., 2018]
Noah Golowich, Alexander Rakhlin, and Ohad Shamir.
Size-independent sample complexity of neural networks.
In Conference On Learning Theory, 2018.

[Hardt et al., 2016]
Moritz Hardt, Ben Recht, and Yoram Singer.
Train faster, generalize better: Stability of stochastic gradient
In International Conference on Machine Learning, 2016.

[Krizhevsky et al., 2012]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.
Imagenet classification with deep convolutional neural networks.
In Advances in Neural Information Processing Systems, 2012.

[Simonyan and Zisserman, 2015]
Karen Simonyan and Andrew Zisserman.
Very deep convolutional networks for large-scale image recognition.
In International Conference on Learning Representations, 2015.

[Jun et al., 2020]
Heewoo Jun, Rewon Child, Mark Chen, John Schulman, Aditya Ramesh, Alec Radford,
and Ilya Sutskever.
Distribution augmentation for generative modeling.
In International Conference on Machine Learning, 2020.

[Belkin et al., 2019]
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal.
Reconciling modern machine-learning practice and the classical
bias–variance trade-off.
In Proceedings of the National Academy of Sciences. National
Acad Sciences, 2019b.

[van Amersfoort et al., 2020]
Joost van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal.
Simple and scalable epistemic uncertainty estimation using a single
deep deterministic neural network.
In International Conference on Machine Learning, 2020.

[Liu et al., 2020]
Jeremiah Zhe Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax-Weiss, and
Balaji Lakshminarayanan.
Simple and principled uncertainty estimation with deterministic deep
learning via distance awareness.
In Advances in Neural Information Processing Systems,

[Lakshminarayanan et al., 2017]
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.
Simple and scalable predictive uncertainty estimation using deep
In Advances in Neural Information Processing Systems, 2017.

[Szegedy et al., 2013]
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan,
Ian Goodfellow, and Rob Fergus.
Intriguing properties of neural networks.
arXiv preprint arXiv:1312.6199, 2013.

[Papernot et al., 2016]
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay
Celik, and Ananthram Swami.
The limitations of deep learning in adversarial settings.
In European symposium on Security and Privacy, 2016.

[Chakraborty et al., 2018]
Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep
Adversarial attacks and defences: A survey.
In Transactions on Intelligence Technology, 2018.

[Kurakin et al., 2016]
Alexey Kurakin, Ian Goodfellow, and Samy Bengio.
Adversarial machine learning at scale.
In International Conference on Learning Representations, 2016.

[Moosavi-Dezfooli et al., 2017]
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal
Universal adversarial perturbations.
In Conference on Computer Vision and Pattern Recognition,

[Madry et al., 2018]
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and
Adrian Vladu.
Towards deep learning models resistant to adversarial attacks.
In International Conference on Learning Representations, 2018.

[Gilmer et al., 2018]
Justin Gilmer, Luke Metz, Fartash Faghri, Samuel S Schoenholz, Maithra Raghu,
Martin Wattenberg, and Ian Goodfellow.
Adversarial spheres.
In Workshop Track, International Conference on Learning
Representations, 2018.

[Kingma and Welling, 2014]
Diederik P Kingma and Max Welling.
Auto-encoding variational bayes.
In International Conference on Learning Representations, 2014.

[Rezende et al., 2014]
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra.
Stochastic backpropagation and approximate inference in deep
generative models.
In International Conference on Machine Learning, 2014.

[Rezende and Mohamed, 2015]
Danilo Rezende and Shakir Mohamed.
Variational inference with normalizing flows.
In International Conference on Machine Learning, 2015.

[Behrmann et al., 2019]
Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and
Jörn-Henrik Jacobsen.
Invertible residual networks.
In International Conference on Machine Learning, 2019.

[Nguyen et al., 2010]
XuanLong Nguyen, Martin J Wainwright, and Michael I Jordan.
Estimating divergence functionals and the likelihood ratio by convex
risk minimization.
In Transactions on Information Theory, 2010.

[Nowozin et al., 2016]
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka.
f-gan: Training generative neural samplers using variational
divergence minimization.
In Advances in Neural Information Processing Systems,

[Arbel et al., 2021]
Michael Arbel, Liang Zhou, and Arthur Gretton.
Generalized energy based models.
In International Conference on Learning Representations, 2021.

[Zhou et al., 2019]
Zhiming Zhou, Jiadong Liang, Yuxuan Song, Lantao Yu, Hongwei Wang, Weinan
Zhang, Yong Yu, and Zhihua Zhang.
Lipschitz generative adversarial nets.
In International Conference on Machine Learning, 2019.

[Belghazi et al., 2018]
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua
Bengio, Aaron Courville, and R Devon Hjelm.
Mine: mutual information neural estimation.
In International Conference on Machine Learning, 2018.

[Oord et al., 2018]
Aaron van den Oord, Yazhe Li, and Oriol Vinyals.
Representation learning with contrastive predictive coding.
arXiv preprint arXiv:1807.03748, 2018.

[Tolstikhin et al., 2018]
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf.
Wasserstein auto-encoders.
In International Conference on Learning Representations, 2018.

[Ostrovski et al., 2018]
Georg Ostrovski, Will Dabney, and Remi Munos.
Autoregressive quantile networks for generative modeling.
In International Conference on Machine Learning, 2018.

[Dabney et al., 2018]
Will Dabney, Georg Ostrovski, David Silver, and Remi Munos.
Implicit quantile networks for distributional reinforcement learning.
In International Conference on Machine Learning,

[Finlay et al., 2020]
Chris Finlay, Jörn-Henrik Jacobsen, Levon Nurbekyan, and Adam M Oberman.
How to train your neural ode.
In International Conference on Machine Learning, 2020.

[Silvia et al., 2020]
Chiappa Silvia, Jiang Ray, Stepleton Tom, Pacchiano Aldo, Jiang Heinrich, and
Aslanides John.
A general approach to fairness with optimal transport.
In Association for the Advancement of Artificial Intelligence,

[Jiang et al., 2020]
Ray Jiang, Aldo Pacchiano, Tom Stepleton, Heinrich Jiang, and Silvia Chiappa.
Wasserstein fair classification.
In Uncertainty in Artificial Intelligence, 2020.

[Villani, 2008]
C. Villani.
Optimal Transport: Old and New.
Grundlehren der mathematischen Wissenschaften. Springer Berlin
Heidelberg, 2008.

[Jacobsen et al., 2018]
Joern-Henrik Jacobsen, Jens Behrmann, Richard Zemel, and Matthias Bethge.
Excessive invariance causes adversarial vulnerability.
In International Conference on Learning Representations, 2018.

[Fazlyab et al., 2019]
Mahyar Fazlyab, Alexander Robey, Hamed Hassani, Manfred Morari, and George
Efficient and accurate estimation of lipschitz constants for deep
neural networks.
In Advances in Neural Information Processing Systems, 2019.

[Kurach et al., 2019]
Karol Kurach, Mario Lučić, Xiaohua Zhai, Marcin Michalski, and
Sylvain Gelly.
A large-scale study on regularization and normalization in gans.
In International Conference on Machine Learning, 2019.

[LeCun et al., 1999]
Yann LeCun, Patrick Haffner, Léon Bottou, and Yoshua Bengio.
Object recognition with gradient-based learning.
In Shape, contour and grouping in computer vision. Springer,

[Czarnecki et al., 2019]
Wojciech M Czarnecki, Razvan Pascanu, Simon Osindero, Siddhant Jayakumar,
Grzegorz Swirszcz, and Max Jaderberg.
Distilling policy distillation.
In International Conference on Artificial Intelligence and
Statistics, 2019.

[Bengio et al., 2020]
Emmanuel Bengio, Joelle Pineau, and Doina Precup.
Interference and generalization in temporal difference learning.
In International Conference on Machine Learning, 2020.

[Vinyals et al., 2019]
Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew
Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko
Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang,
Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander S.
Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden,
Yury Sulsky, James Molloy, Tom L. Paine, Caglar Gulcehre, Ziyu Wang, Tobias
Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney,
Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis
Hassabis, Chris Apps, and David Silver.
Grandmaster level in starcraft ii using multi-agent reinforcement
In Nature, 2019.

[Schaul et al., 2016]
T. Schaul, John Quan, Ioannis Antonoglou, and D. Silver.
Prioritized experience replay.
In International Conference for Learning Representations,

[247]
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian
Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin,
Charles Blundell, and Shane Legg.
Noisy networks for exploration.
In International Conference on Learning Representations.

[Osband et al., 2016]
Ian Osband, Charles Blundell, A. Pritzel, and Benjamin Van Roy.
Deep exploration via bootstrapped dqn.
In Advances in Neural Information Processing Systems, 2016.

[Ostrovski et al., 2017]
Georg Ostrovski, Marc G Bellemare, Aäron van den Oord, and Rémi Munos.
Count-based exploration with neural density models.
In International Conference on Machine Learning, 2017.

[Dabney et al., 2018]
Will Dabney, Mark Rowland, Marc Bellemare, and Rémi Munos.
Distributional reinforcement learning with quantile regression.
In Association for the Advancement of Artificial Intelligence,

[Hessel et al., 2018]
Matteo Hessel, Joseph Modayil, H. V. Hasselt, T. Schaul, Georg Ostrovski,
W. Dabney, Dan Horgan, B. Piot, Mohammad Gheshlaghi Azar, and D. Silver.
Rainbow: Combining improvements in deep reinforcement learning.
In Association for the Advancement of Artificial Intelligence,

[Watkins and Dayan, 1992]
Christopher JCH Watkins and Peter Dayan.
Machine learning, 1992.

[Dabney et al., 2021]
Will Dabney, André Barreto, Mark Rowland, Robert Dadashi, John Quan, Marc G
Bellemare, and David Silver.
The value-improvement path: Towards better representations for
reinforcement learning.
In Association for the Advancement of Artificial Intelligence,

[Van Hasselt et al., 2016]
Hado Van Hasselt, Arthur Guez, and David Silver.
Deep reinforcement learning with double q-learning.
In Association for the Advancement of Artificial Intelligence,

[Wang et al., 2016]
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando
Dueling network architectures for deep reinforcement learning.
In International Conference on Machine Learning, 2016.

[Henderson et al., 2018]
Peter Henderson, Joshua Romoff, and Joelle Pineau.
Where did my optimum go?: An empirical analysis of gradient descent
optimization in policy gradient methods.

[Hessel et al., 2019]
Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojciech Czarnecki, Simon Schmitt,
and Hado van Hasselt.
Multi-task deep reinforcement learning with popart.
In Association for the Advancement of Artificial Intelligence,

[Bellemare et al., 2013]
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling.
The arcade learning environment: An evaluation platform for general
In Journal of Artificial Intelligence Research, 2013.

[Young and Tian, 2019]
Kenny Young and Tian Tian.
Minatar: An atari-inspired testbed for thorough and reproducible
reinforcement learning experiments.
arXiv preprint arXiv:1903.03176, 2019.

[Ceron and Castro, 2021]
Johan Samir Obando Ceron and Pablo Samuel Castro.
Revisiting rainbow: Promoting more insightful and inclusive deep
reinforcement learning research.
In International Conference on Machine Learning, 2021.

[Gouk et al., 2020]
Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael J Cree.
Regularisation of neural networks by enforcing lipschitz continuity.
In Machine Learning, 2020b.

[Nair and Hinton, 2010]
Vinod Nair and Geoffrey E Hinton.
Rectified linear units improve restricted boltzmann machines.
In International Conference on Machine Learning, 2010.

[Xu et al., 2015]
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li.
Empirical evaluation of rectified activations in convolutional
arXiv preprint arXiv:1505.00853, 2015.

[Salimans and Kingma, 2016]
Tim Salimans and Diederik P Kingma.
Weight normalization: A simple reparameterization to accelerate
training of deep neural networks.
In Advances in Neural Information Processing Systems, 2016.

[Van Laarhoven, 2017]
Twan Van Laarhoven.
L2 regularization versus batch and weight normalization.
arXiv preprint arXiv:1706.05350, 2017.

[Hoffer et al., 2018]
Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry.
Norm matters: efficient and accurate normalization schemes in deep
In Advances in Neural Information Processing Systems, 2018.

[Arora et al., 2019]
Sanjeev Arora, Zhiyuan Li, and Kaifeng Lyu.
Theoretical analysis of auto rate-tuning by batch normalization.
In International Conference on Learning Representations, 2019.

[Givchi and Palhang, 2014]
A. Givchi and M. Palhang.
Quasi newton temporal difference learning.
In Asian Conference on Machine Learning, 2014.

[Sun et al., 2020]
Tao Sun, Han Shen, Tianyi Chen, and Dongsheng Li.
Adaptive temporal difference learning with linear function
In Transactions on Pattern Analysis and Machine Intelligence,

[Romoff et al., 2020]
Joshua Romoff, Peter Henderson, David Kanaa, Emmanuel Bengio, Ahmed Touati,
Pierre-Luc Bacon, and Joelle Pineau.
Tdprop: Does jacobi preconditioning help temporal difference
arXiv preprint arXiv:2007.02786, 2020.

[Lin et al., 2021]
Zinan Lin, Vyas Sekar, and Giulia Fanti.
Why spectral normalization stabilizes gans: Analysis and
In Advances in Neural Information Processing Systems, 2021.

[Fortmann-Roe, 2012]
Scott Fortmann-Roe.
Understanding the bias-variance tradeoff.
URL: http://scott. fortmann-roe. com/docs/BiasVariance. html
(hämtad 2019-03-27), 2012.

[Neyshabur et al., 2014]
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro.
In search of the real inductive bias: On the role of implicit
regularization in deep learning.
In Workshop Track, International Conference on Learning
Representations, 2014.

[Zhang et al., 2021]
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.
Understanding deep learning (still) requires rethinking
In Communications of the Association for Computing Machinery,

[Koltchinskii and Panchenko, 1999]
V. Koltchinskii and D. Panchenko.
Rademacher processes and bounding the risk of function learning.
In High Dimensional Probability II. Birkhäuser, 1999.

[Bartlett and Mendelson, 2002]
Peter L Bartlett and Shahar Mendelson.
Rademacher and gaussian complexities: Risk bounds and structural
In Journal of Machine Learning Research, 2002.

[Vapnik and Chervonenkis, 1971]
V. N. Vapnik and A. Y. Chervonenkis.
On the uniform convergence of relative frequencies of events to their
In Theory of Probability and its Applications, 1971.

[Neyshabur, 2017]
Behnam Neyshabur.
Implicit regularization in deep learning.
arXiv preprint arXiv:1709.01953, 2017.

[Ma and Ying, 2021]
Chao Ma and Lexing Ying.
The sobolev regularization effect of stochastic gradient descent.
arXiv preprint arXiv:2105.13462, 2021.

[Bartlett et al., 2002]
Peter L Bartlett, Stéphane Boucheron, and Gábor Lugosi.
Model selection and error estimation.
In Machine Learning, 2002.

[Neyshabur et al., 2018]
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan
The role of over-parametrization in generalization of neural
In International Conference on Learning Representations, 2018.

[Anthony and Bartlett, 1999]
Martin Anthony and Peter L Bartlett.
Neural network learning: Theoretical foundations.
Cambridge university press, 1999.

[Jiang et al., 2018]
Yiding Jiang, Dilip Krishnan, Hossein Mobahi, and Samy Bengio.
Predicting the generalization gap in deep networks with margin
In International Conference on Learning Representations, 2018.

[Schmidhuber, 1997]
J. Schmidhuber.
Discovering neural nets with low kolmogorov complexity and high
generalization capability.
In Neural networks : the official journal of the International
Neural Network Society, 1997.

[Hinton and Zemel, 1994]
Geoffrey E Hinton and Richard Zemel.
Autoencoders, minimum description length and helmholtz free energy.
In J. Cowan, G. Tesauro, and J. Alspector, editors, Advances in
Neural Information Processing Systems. Morgan-Kaufmann, 1994.

[Blier and Ollivier, 2018]
Léonard Blier and Yann Ollivier.
The description length of deep learning models.
Advances in Neural Information Processing Systems, 2018.

[Serra et al., 2018]
Thiago Serra, Christian Tjandraatmadja, and Srikumar Ramalingam.
Bounding and counting linear regions of deep neural networks.
In International Conference on Machine Learning, 2018.

[Alquier, 2021]
Pierre Alquier.
User-friendly introduction to pac-bayes bounds.
arXiv preprint arXiv:2110.11216, 2021.

[Dziugaite and Roy, 2017]
Gintare Karolina Dziugaite and Daniel M Roy.
Computing nonvacuous generalization bounds for deep (stochastic)
neural networks with many more parameters than training data.
Uncertainty in Artificial Intelligence, 2017.

[Goodfellow et al., 2016]
Ian Goodfellow, Yoshua Bengio, and Aaron Courville.
Deep learning.
MIT press, 2016.

[Seong et al., 2018]
Sihyeon Seong, Yegang Lee, Youngwook Kee, Dongyoon Han, and Junmo Kim.
Towards flatter loss surface via nonmonotonic learning rate
In Uncertainty in Artificial Intelligence, 2018.

[Smith et al., 2020]
Samuel Smith, Erich Elsen, and Soham De.
On the generalization benefit of noise in stochastic gradient
In International Conference on Machine Learning, 2020.

[Roberts, 2018]
Daniel A. Roberts.
Sgd implicitly regularizes generalization error.
In Advances in Neural Information Processing Systems 2018
Workshop. 2018.

[Glorot and Bengio, 2010]
Xavier Glorot and Yoshua Bengio.
Understanding the difficulty of training deep feedforward neural
In International Conference on Artificial Intelligence and
Statistics, 2010a.

[Gunasekar et al., 2018]
Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro.
Characterizing implicit bias in terms of optimization geometry.
In International Conference on Machine Learning,

[He et al., 2015]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Delving deep into rectifiers: Surpassing human-level performance on
imagenet classification.
In International Conference on Computer Vision, 2015.

[Li and Liang, 2018]
Yuanzhi Li and Yingyu Liang.
Learning overparameterized neural networks via stochastic gradient
descent on structured data.
In Advances in Neural Information Processing Systems, 2018.

[Zhang et al., 2020]
Yaoyu Zhang, Zhi-Qin John Xu, Tao Luo, and Zheng Ma.
A type of generalization error induced by initialization in deep
neural networks.
In Mathematical and Scientific Machine Learning, 2020.

[Virmaux and Scaman, 2018]
Aladin Virmaux and Kevin Scaman.
Lipschitz regularity of deep neural networks: analysis and efficient
In Advances in Neural Information Processing Systems, 2018.

[Combettes and Pesquet, 2019]
Patrick L Combettes and Jean-Christophe Pesquet.
Lipschitz certificates for neural network structures driven by
averaged activation operators.
arXiv preprint arXiv:1903.01014, 2019.

[Agarap, 2018]
Abien Fred Agarap.
Deep learning using rectified linear units (relu).
arXiv preprint arXiv:1803.08375, 2018.

[Maas et al., 2013]
Andrew L Maas, Awni Y Hannun, and Andrew Y Ng.
Rectifier nonlinearities improve neural network acoustic models.
In International Conference on Machine Learning, 2013.

[Glorot and Bengio, 2010]
Xavier Glorot and Yoshua Bengio.
Understanding the difficulty of training deep feedforward neural
In International Conference on Artificial Intelligence and
Statistics, 2010b.

[Varga et al., 2017]
Dániel Varga, Adrián Csiszárik, and Zsolt Zombori.
Gradient regularization improves accuracy of discriminative models.
arXiv preprint arXiv:1712.09936, 2017.

[Hoffer et al., 2017]
Elad Hoffer, Itay Hubara, and Daniel Soudry.
Train longer, generalize better: closing the generalization gap in
large batch training of neural networks.
In Advances in Neural Information Processing Systems, 2017.

[LeCun et al., 2012]
Yann A LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller.
Efficient backprop.
In Neural networks: Tricks of the trade. Springer, 2012.

[Masters and Luschi, 2018]
Dominic Masters and Carlo Luschi.
Revisiting small batch training for deep neural networks.
arXiv preprint arXiv:1804.07612, 2018.

[Ghosh et al., 2023]
Avrajit Ghosh, He Lyu, Xitong Zhang, and Rongrong Wang.
Implicit regularization in heavy-ball momentum accelerated stochastic
gradient descent.
In International Conference on Learning Representations, 2023.

[Lee et al., 2020]
Yoonho Lee, Juho Lee, Sung Ju Hwang, Eunho Yang, and Seungjin Choi.
Neural complexity measures.
In Advances in Neural Information Processing Systems,

[Maddox et al., 2020]
Wesley J Maddox, Gregory Benton, and Andrew Gordon Wilson.
Rethinking parameter counting in deep models: Effective
dimensionality revisited.
arXiv preprint arXiv:2003.02139, 2020.

[Grant and Wu, 2022]
Erin Grant and Yan Wu.
Predicting generalization with degrees of freedom in neural networks.
In International Conference on Machine Learning 2022 2nd AI for
Science Workshop, 2022.

[Efron, 1983]
Bradley Efron.
Estimating the error rate of a prediction rule: improvement on
In Journal of the American Statistical Association. Taylor &
Francis, 1983.

[Ye, 1998]
Jianming Ye.
On measuring and correcting the effects of data mining and model
Journal of the American Statistical Association, 1998.

[Huh et al., 2023]
Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and
Phillip Isola.
The low-rank simplicity bias in deep networks.
Transactions on Machine Learning Research, 2023.

[Liu and Nocedal, 1989]
Dong C Liu and Jorge Nocedal.
On the limited memory bfgs method for large scale optimization.
In Mathematical programming, 1989.

[Bubeck and Sellke, 2021]
Sébastien Bubeck and Mark Sellke.
A universal law of robustness via isoperimetry.
In Advances in Neural Information Processing Systems, 2021.

[Gamba et al., 2022]
Matteo Gamba, Erik Englesson, Mårten Björkman, and Hossein Azizpour.
Deep double descent via smooth interpolation.
arXiv preprint arXiv:2209.10080, 2022.

[Bishop and Nasrabadi, 2006]
Christopher M Bishop and Nasser M Nasrabadi.
Pattern recognition and machine learning.
Springer, 2006.

[Grobman, 1959]
David M Grobman.
Homeomorphism of systems of differential equations.
Doklady Akademii Nauk SSSR, 1959.

[LeCun et al., 1998]
Yan LeCun, Corinna Cortes, and Christopher J.C. Burges.
The mnist database of handwritten digits.
http://yann. lecun. com/exdb/mnist/, 1998.

[Krizhevsky, 2009]
Alex Krizhevsky.
Learning multiple layers of features from tiny images.

[Deng et al., 2009]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.
Imagenet: A large-scale hierarchical image database.
In Conference on Computer Vision and Pattern Recognition,

[Clevert et al., 2015]
Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter.
Fast and accurate deep network learning by exponential linear units
In International Conference on Learning Representations, 2015.

[Hairer et al., 2006]
Ernst Hairer, Christian Lubich, and Gerhard Wanner.
Geometric numerical integration.
Springer-Verlag, Berlin, second edition, 2006b.

[Bradbury et al., 2018]
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary,
Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye
Wanderman-Milne, and Qiao Zhang.
JAX: composable transformations of Python+NumPy programs.
URL <http://github.com/google/jax>.

[Hennigan et al., 2020]
Tom Hennigan, Trevor Cai, Tamara Norman, and Igor Babuschkin.
Haiku: Sonnet for JAX.
URL <http://github.com/deepmind/dm-haiku>.

[Hessel et al., 2020]
Matteo Hessel, David Budden, Fabio Viola, Mihaela Rosca, Eren Sezener, and Tom
Optax: composable gradient transformation and optimisation, in jax!
URL <http://github.com/deepmind/optax>.

[Cuturi and Avis, 2014]
Marco Cuturi and David Avis.
Ground metric learning.
In Journal of Machine Learning Research, 2014.

[Machado et al., 2018]
Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew
Hausknecht, and Michael Bowling.
Revisiting the arcade learning environment: Evaluation protocols and
open problems for general agents.
In Journal of Artificial Intelligence Research, 2018.

[Toromanoff et al., 2019]
Marin Toromanoff, Emilie Wirbel, and Fabien Moutarde.
Is deep reinforcement learning really superhuman on atari? leveling
the playing field.
arXiv preprint arXiv:1908.04683, 2019.