# Energy-Based Models for Code Generation
under Compilability Constraints

Tomasz Korbak,1, Hady Elsahar,2 Marc Dymetman,2 Germán Kruszewski2
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
1University of Sussex, United Kingdom
2Naver Labs Europe, France
Work done during a research internship at Naver Labs Europe.

###### Abstract

Neural language models can be successfully trained on source code, leading to
applications such as code completion. However, their versatile autoregressive
self-supervision objective overlooks important global sequence-level features
that are present in the data such as syntactic correctness or compilability.
In this work, we pose the problem of learning to generate compilable code as
constraint satisfaction. We define an Energy-Based Model (EBM) representing a
pre-trained generative model with an imposed constraint of generating only
compilable sequences. We then use the KL-Adaptive Distributional Policy
Gradient algorithm Khalifa et al. (2021) to train a generative model
approximating the EBM. We conduct experiments showing that our proposed
approach is able to improve compilability rates without sacrificing diversity
and complexity of the generated samples.

##  1 Introduction

Code completion is an essential feature of any modern Integrated Development
Environment (IDEs). It supports developers with recommendations about the next
token to write given a context, speeding up software development and reducing
the number of mistakes. A large body of work has relied on statistical
language modeling, treating programming languages as natural languages using
probabilistic grammars Raychev et al. (2014); Bielik et al. (2016), and more
recently relying on neural language models Liu et al. (2016a); Svyatkovskiy et
al. (2020a, b); Arkesteijn et al. (2020); Ciniselli et al. (2021).111See
Allamanis et al. (2018) for a survey. In particular, neural autoregressive
language models have been favoured due to their scalability and generic
training procedure that can exploit large codebases (e.g. open source code
repositories available on GitHub) through self-supervised training.

Despite these desirable traits, neural language models, trained in the
standard way, are known to suffer from myopia and to overlook global sequence-
level features that are present in the data and which might be crucial for the
quality of generated sequences Parshakova et al. (2019b). This leads to
repetitions, hallucinations and failing to capture long-distance consistency
requirements. In a code generation context, this is demonstrated in
compilation errors that are a common failure mode in such tasks as translation
between programming languages (Roziere et al., 2020). This problem has
inspired a large body of work on different fronts on injecting sequence-level
priors by either directly optimizing sequence-level features Ranzato et al.
(2016) or through fusion with grammars and automata Xiao et al. (2016). These
techniques aim to balance between the desirable traits and fast inference of
neural autoregressive models trained in the standard way and the satisfaction
of global sequence-level features.

In this work, we formulate compilable code generation as a constraint
satisfaction problem. We show that this formulation leads to a unique
distribution represented by an Energy-Based Model (EBM). This unique
distribution by definition fully satisfies the compilability constraints while
having a minimal KL divergence from the original autoregressive generative
model trained through cross entropy. We then train an auto-regressive
generative model to approximate the underlying distribution of this EBM using
the KL-Adaptive Distributional Policy Gradient algorithm Khalifa et al.
(2021).

In our experiments, we show that our approach significantly improves
compilability rates without sacrificing diversity or complexity of the
generated examples. This alleviates the drawbacks of reinforcement learning
fine-tuning techniques that maximize compilability but deviate significantly
from the original generative model, which leads to severe loss in diversity
and complexity of the generated samples. Finally, we complement our
experiments with a qualitative analysis of the effect of several fine-tuning
approaches on the distribution of compilation errors.

##  2 Related Work

#### Imposing compilability constraints on generative models

There is a body of work focusing on unconditional code generation or code
completion: generating a piece of source code given a preceding piece of
source code Nguyen et al. (2013); Raychev et al. (2014); Karpathy et al.
(2015); Bielik et al. (2016). That work, however, focuses on perplexity and
similarity with respect to ground truth completions (in terms of exact-match
accuracy, Levensthein distance and ROUGE scores) Svyatkovskiy et al. (2020a);
Lu et al. (2021), usually failing to measure and control for compilability of
generated sequences or semantic and syntactic constraints in general.222One
exception is the work of Maddison and Tarlow (2014), who augment neural
probabilistic context free grammars with semantic constraints and use them for
unconditional generation. On the other hand, semantic and syntactic
constraints are frequently considered in language-to-code translation or
program synthesis. For instance, Zhong et al. (2017), who used policy
gradients to train a model for translating natural language questions to
corresponding SQL queries and – in addition for rewarding for query execution
results – added a penalty for syntactically invalid queries. Taking that one
step further, Kulal et al. (2019) use compilation errors (with their precise
location) to guide search over the space of possible programs.

#### Optimizing sequence-level rewards for text generation

Most previous attempts at steering autoregressive model to conform to global
constraints defined over entire sequence have employed reinforcement learning
(RL). This includes using Reinforce (Williams, 1992a) for machine translation
Ranzato et al. (2016) or actor critic Konda and Tsitsiklis (2000) for
abstractive summarization (Paulus et al., 2018), caption generation Liu et al.
(2016b), dialogue Li et al. (2016b), and video captioning (Pasunuru and
Bansal, 2017). Some approaches (for instance, in machine translation and
summarization (Ranzato et al., 2016; Bahdanau et al., 2017)) directly optimize
performance metrics such as BLEU and ROUGE at training time. Others use
heuristic rewards (for instance Li et al. (2016b) for dialogue generation and
Tambwekar et al. (2019) for story generation) in order to obtain certain a
priori desirable features of generated sequences that then incentivize good
performance on target metrics. A weakness of using RL in fine-tuning
generative models is the problem of catastrophic forgetting: maximizing
global, sequence-level rewards leads to very large deviations from the
original autoregressive model trained through cross-entropy. This often
results in significant reductions in fluency and diversity of generated
samples. The catastrophic forgetting problem is sometimes addressed by
imposing a penalty term to the rewards, such as the KL divergence between the
trained policy and the auto-regressive model. This approach, termed
“conservative fine-tuning”, was applied to generating melodies with music
theory rewards and organic molecules with synthesizability rewards by Jaques
et al. (2017) as well fine-tuning language models for controllable language
generation by Ziegler et al. (2019). This solution doesn’t have an explicit
notion of the optimal policy and often has hard time balancing between the
reward term and the KL penalty term, leading to instability in training
Khalifa et al. (2021). Unlike this approach, our formulation defines the
optimal distribution that satisfies both requirements.

#### Energy-based models for text

Energy-based models (EBMs) (Hinton, 2002; LeCun et al., 2006; Ranzato et al.,
2007) are a family of probabilistic graphical models in which learning and
inference are done by associating an unnormalized probability with each
configuration of observed and latent variables. Early examples of EBMs applied
to natural language processing include sequence labeling problems (e.g.
tagging) exploiting global properties of a sequence (Andor et al., 2016;
Belanger and McCallum, 2016). A recent surge of interest in EBMs (Du and
Mordatch, 2019) has not left text generation unaffected (see Bakhtin et al.
(2020) for a survey). Tu et al. (2020) proposed an energy-based inference
networks for non-autoregressive machine translation. Parshakova et al. (2019b)
and Deng et al. (2020) augment a autoregressive language models with an
additional global factor to obtain a lower perplexity on the training data.
Khalifa et al. (2021) develop a novel approach to distributional controllable
text generation by constructing an EBM satisfying desired statistical
constraints imposed on the set of generated sequences (such as topic or gender
statistics over the sequences) and then train an autoregressive policy to
approximate it, which can be sampled from efficiently. We build on Khalifa et
al.’s approach by applying it to a novel domain outside natural language and
defining a new kind of constraint: compilability.

##  3 Method

Following Khalifa et al. (2021), we formulate compilable code generation as a
constraint satisfaction problem over a space of generative models. There are
two constraints that a target generative model $p$ must satisfy. First, $p$
must have minimal divergence -in the distribution space- from an original
generative model $a$ pre-trained using a standard autoregressive language
modeling objective. Second, it must generate only sequences that satisfy a
certain sequence level constraint $b$. In our case, $b(x)=1$ iff $x$ is a
syntactically correct Python program and $b(x)=0$ otherwise. There two
constraints can be represented as a product-of-experts (Hinton, 2002) energy-
based model

$P(x)=a(x)b(x).$ (1)

$p(x)$ can be obtained from $P(x)$ by dividing it by a normalization constant
$Z$:

$p(x)=\frac{1}{Z}P(x),$ (2)

where

$Z\doteq\sum_{x}P(x).$ (3)

This EBM $P$ is unique, it represents a distribution $p$ that optimally
reconciles the two constraints. It is a special case of the generalized
maximum entropy formulation presented in (Csiszár and Shields, 2004) for
applying constraints over distributions.

However, one problem still remains: it is not straightforward how to draw
samples $x\sim p(x)$ or even evaluating probability $p(x)$ from this optimal
unique distribution. A simple method for drawing samples from the $p$
distribution could be sampling sequences from $a$ and filtering on $b(x)$.
While this method sounds simple, there’s no direct way of using it for
interactive code completion as sampling full sequences till the end is
necessary to filter through the sequence-level filter $b(x)$. Therefore our
objective here is to obtain another autoregressive policy ${\pi_{\theta}}$ to
directly approximate $p$.

To attain this, Khalifa et al. (2021) (following Parshakova et al. (2019a))
developed a training procedure called KL-Adaptive Distributional Policy
Gradients (KL-DPG) to train ${\pi_{\theta}}$ to minimize the KL divergence
between $p$ and ${\pi_{\theta}}$. The gradient of this KL turns out to be
tractable:

$\displaystyle\nabla_{\theta}D_{\mathrm{KL}}(p,{\pi_{\theta}})$
$\displaystyle=\nabla_{\theta}\mathbb{E}_{x\sim
p}\log\frac{p(x)}{{\pi_{\theta}}(x)}$ (4)
$\displaystyle=-\nabla_{\theta}\mathbb{E}_{x\sim p}\log{\pi_{\theta}}(x)$ (5)
$\displaystyle=-\mathbb{E}_{x\sim p}\nabla_{\theta}\log{\pi_{\theta}}(x)$ (6)
$\displaystyle=-\frac{1}{Z}\sum_{x}P(x)\nabla_{\theta}\log{\pi_{\theta}}(x)$
(7)

Let us now absorb the constant $-1/Z$ into a learning rate $\alpha^{(\theta)}$
and estimate the expectation over $p(x)$ using importance sampling Owen (2013)
from yet another generative model $q$:

$\nabla_{\theta}D_{\mathrm{KL}}(p,{\pi_{\theta}})\propto\mathbb{E}_{x\sim
q}\frac{P(x)}{q(x)}\nabla_{\theta}\log{\pi_{\theta}}(x).$ (8)

During training, both ${\pi_{\theta}}$ and $q$ are initialized as $a$. Then,
$q$ is periodically updated to ${\pi_{\theta}}$ if ${\pi_{\theta}}$ surpasses
$q$ in being closer to $p$ (in terms of KL). For a pseudo-code of the whole
KL-DPG training procedure, see Algorithm 1.

The gradient in (8) is similar to an estimate obtained using policy gradients
methods in standard reinforcement learning Sutton et al. (1999) with
$P(x)/q(x)$ playing the role of a pseudoreward. This similarity, however, is
superficial. Our objective is approximating a target generative model $p$ by
minimizing $D_{\mathrm{KL}}(p,{\pi_{\theta}})$ rather than maximizing expected
reward $b(x)$ or $P(x)$ or $P(x)/q(x)$. As we show in Section 5, these
objectives produce vastly different policies which diverge from $p$ and
catastrophically forget what the pretrained model $a$ knew about its training
domain. Furthermore, since $q$ will always be close to ${\pi_{\theta}}$, our
pseudoreward $P(x)/q(x)$ effectively depends on policy parameters $\theta$.

Algorithm 1 KL-DPG

1:EBM $P$, initial generative model $a$

2:$\pi_{\theta}\leftarrow a$

3:$q\leftarrow a$

4:for each iteration do

5: for each episode do

6: sample $x$ from $q(x)$

7: $\theta\leftarrow\theta+\alpha^{(\theta)}\frac{P(x)}{q(x)}\
\nabla_{\theta}\log\pi_{\theta}(x)$

8: if $D_{\mathrm{KL}}(p||\pi_{\theta})<D_{\mathrm{KL}}(p||q)$ then

9: $q\leftarrow\pi_{\theta}$

10:$\pi_{\theta}$

##  4 Experiments

###  4.1 Setup

#### Dataset:

To prepare the training dataset, we started from the Python150 dataset, which
consists of 150k Python source code files obtained from GitHub Raychev et al.
(2016). Then, using the code from Roziere et al. (2020), we extracted 713k
Python functions (both methods and standalone functions) from it (250 MB of
raw text data). The additional filtering criteria were compilability
(according to $b(x)$) and being less than 128 BPE tokens long. The dataset was
then split into a training subset $\mathcal{D}_{\text{train}}$ and test subset
$\mathcal{D}_{\text{test}}$.

#### Initial generative model $a$:

We implemented $a$ using the GPT-2 Radford et al. (2019) architecture with
117m parameters (gpt2-small) and kept all the original hyperparameters (see
Table 1 in the Appendix). We trained a byte-level BPE tokenizer Sennrich et
al. (2016) with special BOS and EOS tokens to obtain a vocabulary of 50k
tokens. The model was trained for one epoch.

#### Compilability Scorer $b$:

To check for compilability, we call the compile_command function from codeop
module of Python Standard
Library333https://docs.python.org/3/library/codeop.html with a sequence $x$ as
argument and check if it returns a code object. We apply no postprocessing
other than removing BOS and EOS tokens. codeop.compile_command is the
implementation that Python interactive interpreters use in read-eval-print
loop (REPL) to determine whether a string is a valid Python code. The method
tries to compile a string of Python code and raise and exception if there is a
problem with the Python code, in particular a SyntaxError for invalid Python
syntax and ValueError or OverflowError if there is an invalid literal.

This notion of compilability is concerned only with syntactic correctness and
does not execute the body of a function. However, we found the initial
compilability rate $\mathbb{E}_{x\sim a}b(x)$ of functions $x$ sampled from
$a(x)$ to be only 0.56, which leaves a large margin for improvement.444Note
that initial compilability rate will be equal to our $Z$ because
$\mathbb{E}_{x\sim a}b(x)=\sum_{x}a(x)b(x)=\sum_{x}P(x)=Z$.

#### KL-DPG training

${\pi_{\theta}}$ and $q$ share their architecture with $a$ but have separate
weights which are only initially identical to $a$’s. Throughout the training,
${\pi_{\theta}}$ will be updated to approximate $p$. See Table 2 in the
Appendix for a complete list of hyperparameters used for training
${\pi_{\theta}}$ and $q$ using KL-DPG.

###  4.2 Baselines

We compare our method to a common approach of using standard reinforcement
learning to fine-tune a generative model to conform to desired constraints. We
use the Reinforce algorithm (Williams, 1992b) which instead of minimizing
divergence from the target distribution $p$ tries to maximize expected reward
$\mathbb{E}_{{\pi_{\theta}}}R(x)$. We consider two kinds of reward $R(x)$:

  * •

$R(x)=b(x)$, where the generative model is simply rewarded for generating
sequences that compile;

  * •

$R(x)=P(x)$, where the generative model is simply rewarded proportionally to
the score our EBM assigns to $x$. Intuitively, this objective gives reward for
both compilability and respecting the original generative model $a$.

###  4.3 Evaluation Metrics

We evaluate KL-DPG and two baselines in terms of the following metrics:

  1. 1.

$\mathbb{E}_{x\sim{\pi_{\theta}}}b(x)$, compilability rate of sequences
sampled from ${\pi_{\theta}}(x)$,

  2. 2.

$D_{\mathrm{KL}}(p,{\pi_{\theta}})$, the forward KL divergence from the
optimal distribution $p$,

  3. 3.

$D_{\mathrm{KL}}({\pi_{\theta}},a)$, the reverse KL divergence from the
original pretrained generative model,

  4. 4.

Distinct-1 score, a measure of text diversity in terms of the frequency of
token repetitions in a sample $x$, proposed in the context of NLP by (Li et
al., 2016a),

  5. 5.

Self-BLEU-5, a measure of text diversity _across_ samples, proposed in the
context of NLP by (Zhu et al., 2018),

  6. 6.

Perplexity measured on $\mathcal{D}_{\text{test}}$, a held-out subset of the
data used for training $a$, calculated as

$\exp\Big{[}-\frac{1}{N}\sum_{x\in\mathcal{D}_{\text{test}}}\log{\pi_{\theta}}(x)\Big{]},$

where $N$ is the overall number of tokens in $\mathcal{D}_{\text{test}}$.

  7. 7.

Sequence length, the average number of characters in generated sequence $x$
after detokenization,

  8. 8.

AST node count, the average number of nodes in an abstract syntax tree (AST)
of sequences that compile. Samples are parsed to their corresponding ASTs
using the ast module from Python Standard
Library.555https://docs.python.org/3/library/ast.html Intuitively, this metric
should indicate the logical (as opposed to surface) complexity of generated
programs,

  9. 9.

PEP8 error frequency, the average number of violations of PEP8, the style
guide for Python,666https://www.python.org/dev/peps/pep-0008/ measured using
pycodestyle,777https://github.com/PyCQA/pycodestyle an off-the-shelf linter
(static code analysis tool). We report the average number of errors per
character to avoid confounding by sequence length.

While high compilability rate is the target, the remaining metrics control for
various aspects of fluency, quality and diversity of generated samples. Most
but not all of these aspects reduce to the constraint of staying close to $a$;
for instance, it is possible for ${\pi_{\theta}}$ to actually outperform $a$
in matching the statistics of $a$’s own training distribution $p^{*}(x)$.

Figure 1: Compilability rate $\mathbb{E}_{{x\sim{\pi_{\theta}}}b(x)}$
($\uparrow$ better) of samples from policies obtained from KL-DPG, and two
baselines: Reinforce with reward $R(x)=b(x)$ and with reward $R(x)=P(x)$.
Figure 2: Evaluation metrics $\text{KL}(p|\pi_{\theta})$ ($\downarrow$
better), $\text{KL}(\pi_{\theta}|a)$ ($\downarrow$ better), Self-BLEU-5
($\downarrow$ better), Distinct-1 ($\uparrow$ better), AST node count
($\uparrow$ better), PEP8 error count ($\downarrow$ better), sequence length
($\uparrow$ better), and perplexity ($\downarrow$ better) for policies
obtained from KL-DPG, and two baselines: Reinforce with reward $R(x)=b(x)$ and
with reward $R(x)=P(x)$. Figure 3: The frequency (measured as the percentage
of samples from ${\pi_{\theta}}(x)$ causing a given error) of each kind
compilation error for the original generative model $a$ and policies fine-
tuned using KL-DPG and Reinforce with $R(x)=b(x)$. The policy fine-tuned using
Reinforce with $R(x)=P(x)$ was excluded because the single sequence it
produces causes no compilation errors. Percentages were computed using 500
samples while confidence intervals were based on 3 repeats of the sampling
procedure.

##  5 Results

We present the evolution of nine evaluation metrics as a function of gradient
updates on Figures 1 and 2.

Reinforce with $R(x)=b(x)$ quickly improves compilability by a large margin
but this improvement is mirrored by an equally large divergence from $p$ and
$a$. This divergence translates into generating sequences much shorter (in
terms of the number of characters) and logically simpler (in terms of the
number of nodes in its AST) than an average sequence sampled from $a$. This
heavily decreased sequence length (most of the generated functions are one-
liners) seems to artificially increase diversity metrics (Self-BLEU-5 and
Distinct-1).

Reinforce with $R(x)=P(x)$ doesn’t improve compilability rate until an
inflection point after which it quickly reaches perfect compilability at a
price of heavily diverging from both $a$ and (perhaps counterintuitively) $p$.
The reason behind that, however, is that the policy heavily peaks around a
single sequence that is compilable. To understand what causes this behavior,
first note that the objective for Reinforce with $R(x)=P(x)$ is to maximize
$\mathbb{E}_{x\sim{\pi_{\theta}}}[a(x)b(x)]$. Because $R(x)=0$ for
uncompilable sequences, compilation rate will improve. But for compilable
sequences, the effective reward is $R(x)=a(x)$ meaning that ${\pi_{\theta}}$
is rewarded most for generating the most probable sequences (according to
$a(x)$), making them even more probable. Eventually,
$\mathbb{E}_{x\sim{\pi_{\theta}}}a(x)$ is maximized by a policy peaking on a
single sample $x$ that was the most probable one according to $a(x)$. This
failure mode is reflected in diversity metrics and perplexity. The sequence
the policy peaks on is also shorter and less complex than an average sequence
sampled from $a$.

KL-DPG is the only method that consistently improves compilability rate while
decreasing divergence from $p$, maintaining the diversity of $a$ and only
slightly decreasing sequence length and the number of nodes in ASTs. Moreover,
as a by-product of improving compilability, KL-DPG is also able to slightly
decrease the perplexity and the frequency of PEP8 violations per character. We
conjecture the decrease in perplexity is because compilability provides a
training signal enabling ${\pi_{\theta}}$ to fit the $a$’s training
distribution $p^{*}(x)$ better than $a$ was able to.888This mirrors the
results obtained by Parshakova et al. (2019b), who also defined an EBM
augmenting an autoregressive model with prior knowledge about features of the
training set and observed a decrease in perplexity compared to pure
autoregressive training. The decrease in the frequency of PEP8 violations
might be due to the fact that compilability is correlated with PEP8
compliance.

###  5.1 Qualitative evaluation

To further analyze effects of different fine-tuning approaches on sample
diversity, we measured the frequency of BPE tokens in generated samples. For
each of four analyzed generative models, we sampled 1000 sequences using pure
ancestral sampling. We then computed the frequency for each BPE token (the
number of times it occurs) and its rank (its index in a sorted list of
tokens). We plotted these results on Figure 4. This qualitative evaluation
paints a similar picture: fine-tuning using Reinforce incurs a large (with
$R(x)=b(x)$) or extreme (with $R(x)=P(x)$) decrease in token diversity. In
contrast, KL-DPG is able to maintain a relatively long tail of token
frequencies, not departing too far from $a$.

Figure 4: Token frequency against token rank computed for tokens found in
samples from from KL-DPG, and two baselines. Longer tails imply more diverse
samples.

Moreover, in order to gain better understanding of how different fine-tuning
methods affect generative models we measured the frequency of different
categories of compilation errors for samples from $a$ and from fine-tuned
policies. This analysis is presented on Figure 3. We categorized errors using
error messages produced by Python interpreter trying to compile an
uncompilable sequence. invalid syntax is the most common failure mode (30% of
all sequences sampled from $a$), with a long tail of other error categories.
We can see that both KL-DPG and Reinforce with $R(x)=b(x)$ consistently
decrease error frequency across almost all the categories.

Finally, in the Appendix we present randomly generated samples from each
discussed policy. Tables 3-6 contain samples obtained through unconditional
generation. In addition to that, to illustrate the applicability of obtained
policies for code completion, in Tables 7-9 we present samples obtained
through conditional generation, i.e. $x\sim{\pi_{\theta}}(x|c)$, where the
context $c$ is a function name. In either case, samples were obtained using
pure ancestral sampling.

##  6 Discussion

In the paper, we presented a new energy-based model formulation for the
problem of imposing the constraint of compilability on an autoregressive
generative model for source code. In contrast with standard reinforcement
learning approaches, the solution we propose – KL-DPG – is able to improve
compilability rate without sacrificing diversity and complexity of generated
samples.

One obvious application of the presented approach is improving the accuracy of
code completion, i.e. tools assisting in programming by predicting the next
tokens based on context (Svyatkovskiy et al., 2020a). The fact that fine-
tuning using KL-DPG has a beneficial effect on perplexity and PEP8 error
frequency suggests that it can provide a training signal complementary to that
in a language modeling objective. The benefits of this auxilary training
signal would arguably diminish with increased training time and datatset size,
but that still leaves room for significant improvement in low-resource
domains.

A limitation of the current KL-DPG approach is that it is restricted to
unconditional generation. This is because for a conditional EBM $P(x,c)$ the
proportionality constant $-1/Z$ from (4) would depend on a context $c$.
Nevertheless, one can imagine using a policy ${\pi_{\theta}}$ fine-tuned using
KL-DPG as initialization of a decoder for conditional generation, e.g.
transpilation (translation between programming languages) or program synthesis
(translation from a natural language to a programming language).

## References

  * Allamanis et al. (2018) Miltiadis Allamanis, Earl T. Barr, Premkumar T. Devanbu, and Charles Sutton. 2018\.  A survey of machine learning for big code and naturalness.  _ACM Comput. Surv._ , 51(4):81:1–81:37.
  * Andor et al. (2016) Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016.  Globally Normalized Transition-Based Neural Networks.
  * Arkesteijn et al. (2020) Youri Arkesteijn, Nikhil Saldanha, and Bastijn Kostense. 2020.  Code completion using neural attention and byte pair encoding.  _CoRR_ , abs/2004.06343.
  * Bahdanau et al. (2017) Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio. 2017.  An actor-critic algorithm for sequence prediction.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
  * Bakhtin et al. (2020) A. Bakhtin, Y. Deng, S. Gross, Myle Ott, Marc’Aurelio Ranzato, and Arthur Szlam. 2020.  Energy-based models for text.  _ArXiv_ , abs/2004.10188.
  * Belanger and McCallum (2016) David Belanger and Andrew McCallum. 2016.  Structured prediction energy networks.  In _Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48_ , ICML’16, pages 983–992. JMLR.org.
  * Bielik et al. (2016) Pavol Bielik, Veselin Raychev, and Martin Vechev. 2016.  Phog: Probabilistic model for code.  In _Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48_ , ICML’16, page 2933–2942. JMLR.org.
  * Ciniselli et al. (2021) Matteo Ciniselli, Nathan Cooper, Luca Pascarella, Denys Poshyvanyk, Massimiliano Di Penta, and Gabriele Bavota. 2021.  An empirical study on the usage of BERT models for code completion.  _CoRR_ , abs/2103.07115.
  * Csiszár and Shields (2004) Imre Csiszár and Paul C. Shields. 2004.  Information theory and statistics: A tutorial.  _Commun. Inf. Theory_ , 1(4):417–528.
  * Deng et al. (2020) Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Marc’Aurelio Ranzato. 2020\.  Residual energy-based models for text generation.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Du and Mordatch (2019) Yilun Du and Igor Mordatch. 2019.  Implicit generation and modeling with energy based models.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Hinton (2002) Geoffrey E. Hinton. 2002.  Training products of experts by minimizing contrastive divergence.  _Neural Comput._ , 14(8):1771–1800.
  * Jaques et al. (2017) Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, Jose Miguel Hernandez Lobato, Richard E. Turner, and Doug Eck. 2017.  Tuning recurrent neural networks with reinforcement learning.
  * Karpathy et al. (2015) A. Karpathy, J. Johnson, and Li Fei-Fei. 2015.  Visualizing and understanding recurrent networks.  _ArXiv_ , abs/1506.02078.
  * Khalifa et al. (2021) Muhammad Khalifa, Hady Elsahar, and Marc Dymetman. 2021.  A distributional approach to controlled text generation.  In _International Conference on Learning Representations_.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Konda and Tsitsiklis (2000) Vijay Konda and John Tsitsiklis. 2000.  Actor-critic algorithms.  In _Advances in Neural Information Processing Systems_ , volume 12. MIT Press.
  * Kulal et al. (2019) Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. 2019.  Spoc: Search-based pseudocode to code.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * LeCun et al. (2006) Yann LeCun, Sumit Chopra, Raia Hadsell, Marc’Aurelio Ranzato, and Fu Jie Huang. 2006\.  A Tutorial on Energy-Based Learning.  In _Predicting Structured Data_. MIT Press.
  * Li et al. (2016a) Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a.  A diversity-promoting objective function for neural conversation models.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 110–119, San Diego, California. Association for Computational Linguistics.
  * Li et al. (2016b) Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016b.  Deep reinforcement learning for dialogue generation.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016_ , pages 1192–1202. The Association for Computational Linguistics.
  * Liu et al. (2016a) Chang Liu, Xin Wang, Richard Shin, Joseph E Gonzalez, and Dawn Song. 2016a.  Neural code completion.
  * Liu et al. (2016b) Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. 2016b.  Optimization of image description metrics using policy gradient methods.  _CoRR_ , abs/1612.00370.
  * Lu et al. (2021) Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2021.  Codexglue: A machine learning benchmark dataset for code understanding and generation.  _CoRR_ , abs/2102.04664.
  * Maddison and Tarlow (2014) Chris J. Maddison and Daniel Tarlow. 2014.  Structured generative models of natural source code.  In _Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32_ , ICML’14, page II–649–II–657. JMLR.org.
  * Nguyen et al. (2013) Tung Thanh Nguyen, Anh Tuan Nguyen, Hoan Anh Nguyen, and Tien N. Nguyen. 2013.  A statistical semantic language model for source code.  In _Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering_ , ESEC/FSE 2013, page 532–542, New York, NY, USA. Association for Computing Machinery.
  * Owen (2013) Art B. Owen. 2013.  Importance Sampling.  In _Monte Carlo theory, methods and examples_ , chapter 9.
  * Parshakova et al. (2019a) Tetiana Parshakova, Jean-Marc Andreoli, and Marc Dymetman. 2019a.  Distributional Reinforcement Learning For Energy-Based Sequential Models.  _CoRR_.
  * Parshakova et al. (2019b) Tetiana Parshakova, Jean-Marc Andreoli, and Marc Dymetman. 2019b.  Global Autoregressive Models for Data-Efficient Sequence Learning.  In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_ , pages 900–909, Hong Kong, China. Association for Computational Linguistics.
  * Pasunuru and Bansal (2017) Ramakanth Pasunuru and Mohit Bansal. 2017.  Reinforced video captioning with entailment rewards.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017_ , pages 979–985. Association for Computational Linguistics.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems 32_ , pages 8024–8035. Curran Associates, Inc.
  * Paulus et al. (2018) Romain Paulus, Caiming Xiong, and Richard Socher. 2018.  A deep reinforced model for abstractive summarization.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI Blog_ , 1(8):9.
  * Ranzato et al. (2007) Marc’Aurelio Ranzato, Y-Lan Boureau, Sumit Chopra, and Yann LeCun. 2007.  A unified energy-based framework for unsupervised learning.  In _Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics, AISTATS 2007, San Juan, Puerto Rico, March 21-24, 2007_ , volume 2 of _JMLR Proceedings_ , pages 371–379. JMLR.org.
  * Ranzato et al. (2016) Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016.  Sequence level training with recurrent neural networks.  In _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_.
  * Raychev et al. (2016) Veselin Raychev, Pavol Bielik, and Martin Vechev. 2016.  Probabilistic model for code with decision trees.  _SIGPLAN Not._ , 51(10):731–747.
  * Raychev et al. (2014) Veselin Raychev, Martin Vechev, and Eran Yahav. 2014.  Code completion with statistical language models.  _SIGPLAN Not._ , 49(6):419–428.
  * Roziere et al. (2020) Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. 2020\.  Unsupervised translation of programming languages.  _Advances in Neural Information Processing Systems_ , 33.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Sutton et al. (1999) Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. 1999.  Policy gradient methods for reinforcement learning with function approximation.  In _Proceedings of the 12th International Conference on Neural Information Processing Systems_ , NIPS’99, page 1057–1063, Cambridge, MA, USA. MIT Press.
  * Svyatkovskiy et al. (2020a) Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. 2020a.  Intellicode compose: Code generation using transformer.  In _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , ESEC/FSE 2020, page 1433–1443, New York, NY, USA. Association for Computing Machinery.
  * Svyatkovskiy et al. (2020b) Alexey Svyatkovskiy, Sebastian Lee, Anna Hadjitofi, Maik Riechert, Juliana Franco, and Miltiadis Allamanis. 2020b.  Fast and memory-efficient neural code completion.  _CoRR_ , abs/2004.13651.
  * Tambwekar et al. (2019) Pradyumna Tambwekar, Murtaza Dhuliawala, Lara J. Martin, Animesh Mehta, Brent Harrison, and Mark O. Riedl. 2019.  Controllable neural story plot generation via reward shaping.  In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019_ , pages 5982–5988. ijcai.org.
  * Tu et al. (2020) Lifu Tu, Richard Yuanzhe Pang, Sam Wiseman, and Kevin Gimpel. 2020.  Engine: Energy-based inference networks for non-autoregressive machine translation.  _ArXiv_ , abs/2005.00850.
  * Williams (1992a) Ronald J. Williams. 1992a.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  _Mach. Learn._ , 8:229–256.
  * Williams (1992b) Ronald J. Williams. 1992b.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  In _Machine Learning_ , pages 229–256.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _CoRR_ , abs/1910.03771.
  * Xiao et al. (2016) Chunyang Xiao, Marc Dymetman, and Claire Gardent. 2016.  Sequence-based structured prediction for semantic parsing.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1341–1350, Berlin, Germany. Association for Computational Linguistics.
  * Zhong et al. (2017) Victor Zhong, Caiming Xiong, and Richard Socher. 2017.  Seq2sql: Generating structured queries from natural language using reinforcement learning.  _arXiv preprint arXiv:1709.00103_.
  * Zhu et al. (2018) Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018.  Texygen: A benchmarking platform for text generation models.  In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018_, pages 1097–1100. ACM.
  * Ziegler et al. (2019) Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019.  Fine-tuning language models from human preferences.  _CoRR_ , abs/1909.08593.

##  Appendix A Hyperparameters and implementation details

We implemented all models using PyTorch (Paszke et al., 2019) and HuggingFace
(Wolf et al., 2019). Training the initial generative model took 10 days on 3
Nvidia Tesla T4 GPUs. For a detailed list of hyperparameter values, see Table
1.

Hyperparameter | Value
---|---
base LM | gpt2-small
number of params | 117m
number of layers | 12
number of heads | 12
vocabulary size | 50257
sequence length | 128
hidden state size | 768
activation function | gelu
optimizer | Adam (Kingma and Ba, 2014)
initial learning rate | $5\times 10^{-5}$
learning rate scheduler | linear
batch size | 24
total gradient updates | 20069
dropout rate | 0.1
Table 1: Hyperparameters used for training the initial generative model $a$

The implementation of KL-DPG was based on code published by Khalifa et al.
(2021).999https://github.com/naver/gdc Each fine-tuning run took approximately
5 days on 2 Nvidia V100 GPUs. For a detailed list of hyperparameter values,
see Table 2.

Hyperparameter | Value
---|---
optimizer | Adam (Kingma and Ba, 2014)
learning rate $\alpha^{(\theta)}$ | $1.41\times 10^{-6}$
learning rate scheduler | linear
batch size | 2048
warmup gradient updates | 100
total gradient updates | 250
sequence length | 128
dropout rate | 0.1
Table 2: Hyperparameters used for training ${\pi_{\theta}}$ using KL-DPG and Reinforce $b(x)$ | Program
---|---
0 |  ⬇ def test_3_invalid(self):  serializer = serializer.validated_manager[’quarterly_ cred’]  serializer.user = ’token’  self.verify_token(epsg = serializer.DBModes,[serializer.user])  \par
0 |  ⬇ def delete(self,username,password = None):  if username:  if username.startswith(”oil”,None)or username.startswith(’””,True):  raise HttpRequest()  db.model.delete.assert_called_with(username,’password’)  \par
1 |  ⬇ def mode(self):  self._mode = ’modeM_GB’  return self \par
0 |  ⬇ def _update_update_tbl(self,new_worksheet):  self._merge_tbl(new_worksheet,old_worksheet)  self._create_where(’x1’)  self._update_tbl(’x1’,{ }).extend([str(new_fh.getvalue()))  self._clear_sql()  self.clear_lstrip()  self.pop.set(’x1’)[int(col)for param in[’x1’,’y1’]]  self.flush.update()  \par
1 |  ⬇ def _callResourceCost(self,server):  response = urllib.Request(’GET’,’//api//log//body = urllib. urllib2.urlencode(body)  response.headers[’X-Basic-Control-Authorization’]= self.oauth_client.Client.CertResponse(response.body)  return response \par
0 |  ⬇ def _pre_save(self,data):  self.calculate_updates([item.resolve(data[’output’)]= yield data[’output’].find(’top’,[’mybounce’,’geodeIB’])))  \par
1 |  ⬇ def read(self):  self.offset -= 1  start = O8(self)  while time.time()- start:  return self.get_index(start)  \par
0 |  ⬇ def Pub(self):  r = PCHAP()  r[’where’]= struct.unpack(’!T’,self.digest))  response = MKchronosOPS(’R’)  self.sendMessage(response)  return self.Response(response)  \par
1 |  ⬇ def __init__(self,current_node):  self.current_node = current_loadbalancer self.assign_current_node = None self.parenting = None if self.menu:  self.getNodeSelector(Index(RemovelineToRow,self.parent.position),0,2.0,5.0)  self.show_parent()  \par
1 |  ⬇ def get_response_data(self):  return { ’from_blob_client’:self.to_blob_key,’as_blob_secret’:self.to_project_secret.to_secret(),’json’:self.to_storage }  \par
0 |  ⬇ def put(self,key,expire = True):  if not invert:  dict = { }  dict.update(key,self.__TestStepities[key])  self.cs.put(self._uZED_ATTRIBUTES_ =[(”sequential_command”,”duration”,key,expire)]= ”//?modified:\par
1 |  ⬇ def testPath(self):  t = Gaffer.Reader(self.callback)  dupe = ””  f.mkdir(t)  f = sys.stdout.tell()  f.write(’_’)  self.assertEqual(f,dataponCollision)  \par
1 |  ⬇ def get_count(self):  return self.get_implicit_count()  \par
1 |  ⬇ def is_alive(self):  return(self.pid,)and(self.pid == 400)  \par
Table 3: Sequences sampled from the original generative model $a$ $b(x)$ | Program
---|---
0 |  ⬇ def fetch_size(self,page):  response = self.fetch(page,max((2))  constant(response.json(),response.pop(’utf-8’))  payload = ”return payload \par
0 |  ⬇ def setUp(self):  self.project_loader = testutil.FileSentenceDependencyGraph(extensions =[’file’,’path’])  self.schema =RelatedPackage preserveLoader(root_loader)  self.extension_context = XMLLoader()  \par
1 |  ⬇ def __getattr__(self,perm):  return self._memo.get(perm)  \par
1 |  ⬇ def expand(self,text):  value.strip()  return extract_cseq(text)  \par
1 |  ⬇ def test_Obze(self):  w = Command()  self.assertEqual(w.callHeader.callHeader,self.result)  \par
0 |  ⬇ def start_stream(self,addressFamily,opcode):  logger.info(”OpenlibwriteStructBegin chunkon.csv’,OperationalError())  error_message = self.get_stream([None,None])  message,message = self.block_messages[0]  message = message[0]  self._process_message(message,message,message,message)  \par
0 |  ⬇ def set_dense(self,srs,fit_to):  if dup in self.scalar:  return if not isinstance(modality,(pyobj):  self.sq =SUBNET self.basic = asim.bin.sample(srs,rng = self.ctypes,trials = self.rng,dtype = self.dtype)  \par
1 |  ⬇ def _act(self,value):  self._result.set_argument(’value’,value)  \par
1 |  ⬇ def _verify_ssling_access_admin(self,ip_name):  self._check_proxy(ip_name)  \par
0 |  ⬇ def __str__(self):  r =[]  for s in self.__dict__.items():  if s[0]in BoundCacheContents():  break if s[:- 1]:Elements([(”Unsupported Ctreturn ”Data attribute ’\par
0 |  ⬇ def test_FaceIP_3D_14(self):  self.assertTrue(self.doTestFace(self.doTestFace([self.doTestFace([False,False)])  \par
0 |  ⬇ def __init__(self,** options):  super(_ChoiceTest,self).__init__(** options)  self.action_classes = options[”cells_store”]  self.choices =(1.2,** options[”mysql”]= FakeMissingTuple())  self.parser = Message(list.__init__(option_forms))  \par
1 |  ⬇ def main(self,client):  remove_home_config(client,”client_snapshot_url”)  self.client.client_snapshot.update(client)  \par
1 |  ⬇ def _stop_signal(self,emitter,datafile,for_attachment):  vim.gui.target_cancel()  \par
Table 4: Sequences sampled from a policy fine-tuned using KL-DPG $b(x)$ | Program
---|---
1 |  ⬇ def invalidateKey(self):  self.action.rooms = { }  \par
1 |  ⬇ def get(self):  return self.handler.identifier \par
1 |  ⬇ def flush(self):  self.write(”ready”)  \par
1 |  ⬇ def get_flavor(self,resource,path,** metadata):  return self.context.get(resource,path,** metadata)  \par
1 |  ⬇ def test_api_set_to_result(self):  X = T.ListHead()  self.assertEquals(quantiles(X),self._cache.annotations)  \par
1 |  ⬇ def is_cmp(self,other):  return not self._safe_eq(other,self.link)  \par
1 |  ⬇ def __iter__(self):  return iter(self._reverse())  \par
1 |  ⬇ def cancel(self):  return self.enhanced_window.set_timeout()  \par
1 |  ⬇ def __str__(self):  return str(self.repository)  \par
1 |  ⬇ def summary(self):  return self._series \par
1 |  ⬇ def Lazypeer(self):  return self._peer \par
1 |  ⬇ def ByteSize(self):  n = 0  n += self.lengthString(len(self.parameters_))  return n + self.lengthString(number(self.value_))  \par
1 |  ⬇ def setUp(self):  super(TestMaUserRoleTestCase,self).setUp()  self.core =BER()  self.topsetup_existing = False \par
1 |  ⬇ def __init__(self,** kwargs):  self.sourcemersListComp = kwargs.get(’stretch {}’.format(self.__class__.twsourceCentOS_text))  \par
Table 5: Sequences sampled from a policy fine-tuned using Reinforce with $R(x)=b(x)$ $b(x)$ | Program
---|---
1 |  ⬇ def set_OwnerId(self,OwnerId):  self.add_query_param(’OwnerId’,OwnerId)  \par
1 |  ⬇ def set_OwnerId(self,OwnerId):  self.add_query_param(’OwnerId’,OwnerId)  \par
1 |  ⬇ def set_OwnerId(self,OwnerId):  self.add_query_param(’OwnerId’,OwnerId)  \par
1 |  ⬇ def set_OwnerId(self,OwnerId):  self.add_query_param(’OwnerId’,OwnerId)  \par
1 |  ⬇ def set_OwnerId(self,OwnerId):  self.add_query_param(’OwnerId’,OwnerId)  \par
1 |  ⬇ def set_OwnerId(self,OwnerId):  self.add_query_param(’OwnerId’,OwnerId)  \par
1 |  ⬇ def set_OwnerId(self,OwnerId):  self.add_query_param(’OwnerId’,OwnerId)  \par
1 |  ⬇ def set_OwnerId(self,OwnerId):  self.add_query_param(’OwnerId’,OwnerId)  \par
1 |  ⬇ def set_OwnerId(self,OwnerId):  self.add_query_param(’OwnerId’,OwnerId)  \par
1 |  ⬇ def set_OwnerId(self,OwnerId):  self.add_query_param(’OwnerId’,OwnerId)  \par
1 |  ⬇ def set_OwnerId(self,OwnerId):  self.add_query_param(’OwnerId’,OwnerId)  \par
1 |  ⬇ def set_OwnerId(self,OwnerId):  self.add_query_param(’OwnerId’,OwnerId)  \par
1 |  ⬇ def set_OwnerId(self,OwnerId):  self.add_query_param(’OwnerId’,OwnerId)  \par
1 |  ⬇ def set_OwnerId(self,OwnerId):  self.add_query_param(’OwnerId’,OwnerId)  \par
1 |  ⬇ def set_OwnerId(self,OwnerId):  self.add_query_param(’OwnerId’,OwnerId)  \par
1 |  ⬇ def set_OwnerId(self,OwnerId):  self.add_query_param(’OwnerId’,OwnerId)  \par
Table 6: Sequences sampled from a policy finetuned using Reinforce with $R(x)=P(x)$ $b(x)$ | Program
---|---
Sequences sampled from the original generative model $a$
1 |  ⬇ def closeEvent(self):  self._isalive = False self._original_resume = True \par
1 |  ⬇ def close_file(self):  pass \par
1 |  ⬇ def closeWorking(self):  pass \par
Sequences sampled from a policy fine-tuned using KL-DPG
1 |  ⬇ def close(self):  if not self.closed:  self.closed = True self.translation.close()  \par
1 |  ⬇ def close(self):  self.queue.Importer.close(self.info)  self.open_input.close()  self.graph.close(self.gamma)  \par
1 |  ⬇ def close(self):  try:  self.srv.get_browser.mac(self.bus_process.name,vm_output = True)  except suspended as ex:  self.socket.stop(ex)  \par
Sequences sampled from a policy fine-tuned using Reinforce with $R(x)=b(x)$
1 |  ⬇ def close(self):  self._stdout.close()  \par
1 |  ⬇ def close(self):  self.idb.close()  \par
1 |  ⬇ def close(self):  self.reuse = subprocess.Popen(’CONNECTION’,”).unregisterProducer()  p = subprocess.Popen()  p.communicate().close()  return u.close()  \par
Sequences sampled from a policy fine-tuned using Reinforce with $R(x)=P(x)$
1 |  ⬇ def close(self,object):  self.api.close(self.uid.length)  \par
1 |  ⬇ def close(self):  self.job_closed.remove(self)  \par
1 |  ⬇ def close(self):  self.buffer.flush()  \par
Table 7: Samples obtained from policies conditioned on prompt def close $b(x)$ | Program
---|---
Sequences sampled from the original generative model $a$
0 |  ⬇ def fit_pdf(self,hop,theta,theta):  asserttriangular is self._fit_rewrite(hop, kernel,theta,theta)- gtheta,70)  assertworkspace isTType.ACCEPTED_ignore assert subset in(coeff,Y)  assert self._Xfd != xOpenStackBackendError assert isinstance(750,Win,T,Vector)  \par
0 |  ⬇ def fit(self,X,y):  self._ y = y self._children -= 1  assert isinstance(self._labels,_MOD_’”)  x[:]= 0  y[:]=Bio_OFFSET y *= self._labels y * y * y y //= y return y \par
1 |  ⬇ def fit(self,X = None,y = None,result = None):  sts = self.get_appId(self.mesh_filename,X,y = y,d = result)  self.mirror_logpdf([0x9]* indented)  \par
Sequences sampled from a policy fine-tuned using KL-DPG
1 |  ⬇ def fit(self,X,y,* args,** kwargs):  X = self.transform(X,y,* args,** kwargs)  data = np.DataFrame(data)  for i in self.fallback_array.iteration_two(* data):  data[i].labels[i].tolist()  return data \par
0 |  ⬇ def fit(self, initial_output = None):  if initial_output:  self.force_input = False else:  self.cells_done = tuple(initial_output)  if initial_input == self.WK_MASK:  self.output_output += self.osfstorage_NORMAL self.outputs = list([self.inputState.NORMAL_READ valid])  return 1  \par
1 |  ⬇ def fit(self,reshape,a,b):  return frappe. filediff(islice(a,b),b)  \par
Sequences sampled from a policy fine-tuned using Reinforce with $R(x)=b(x)$
1 |  ⬇ def fit(self,X,y):  self.x = y \par
1 |  ⬇ def fit(self,fit,d):  self.fit =followers return super(PositionUntilLockedSequence,self).fit(marks)  \par
0 |  ⬇ def fit(self,X_acc):  X_exog = self.xc1.exog y = self.instance.exog y,= self.model.w2 preserve_uniform(os.environ.XMANllf,y_y))  y += self.model.t2le continX y = self.transition.fit(y)  y.y = self.model.y * y y.red = self.model.gw.urmpopow(y)  return y \par
Sequences sampled from a policy fine-tuned using Reinforce with $R(x)=P(x)$
0 |  ⬇ def fit(self,fit,X,y,z):  self.learning = indices[np.zeros(axis = 1Dot,y = y,motion = self. np.loss,y = res.scale)]  self.index = y \par
1 |  ⬇ def fit(self,params):  self.params_param = params \par
1 |  ⬇ def fit(self,X,y = None):  self.x = x self.y = x \par
Table 8: Samples obtained from policies conditioned on prompt def fit $b(x)$ | Program
---|---
Sequences sampled from the original generative model $a$
0 |  ⬇ def generate_samples_with_prompt(self,input_value,decimal = False):  use_full = False full_input_string = escape_input[decimal]  newprefix = local_input_format.split(”<return newprefix \par
1 |  ⬇ def generate_samples_with_prompt_publish(self):  self.overflow = self.percent \par
0 |  ⬇ def generate_samples_with_prompt_line(self):  lines =[]  for line in rc:  if line.startswith(’_’,’-’):  lines.append(”{}0‘lines.append(”: ”.join(lines))  lines.appenddsets()  lines.append_):  if len(lines)> 0:  lines.append(lines[0])  return lines \par
Sequences sampled from a policy fine-tuned using KL-DPG
1 |  ⬇ def generate_samples_with_prompt(self):  result = self._generate_blobs().generate(self._name,self._amount_in,lambda x:x.lower())  return result \par
0 |  ⬇ def generate_samples_with_prompt_token(self,impdly,red,name,restdeclarations,restid_with_mucmapreduce_access_reference,tpversion):  if prefix_to_acked_ level_per_pbfrom_account_version(MACRO256):  return 71212000x00 * c201402E64D + 204  self.generate_cant_rgb_signature(FLAG,name,comtop header, ”0|02”,[”-20001500e6fsha”]  \par
0 |  ⬇ def generate_samples_with_prompt(self):  tsMAIN_SIZE =(0,1)  tsSBream_bin = self.1000  if if tsody_size is not None:  tsbleations = y size = ts86.data.get_input(vid_ docs).get_language()  for address in data.SerializeToString()if not region:  cpu_ratio = np.zeros(freq.encode(”Now”))  tsLOCATION_examples =[self.read_format(addr)for dir in tsningAssignmentInt()])  \par
Sequences sampled from a policy fine-tuned using Reinforce with $R(x)=b(x)$
1 |  ⬇ def generate_samples_with_prompt(self):  pass \par
1 |  ⬇ def generate_samples_with_prompt_indices(self):  return self.raw_results_with.raw_options.random_encoding \par
0 |  ⬇ def generate_samples_with_prompt(self,* args,** kwargs):  return self.fit_sum(kwargs -(n))):  \par
Sequences sampled from a policy fine-tuned using Reinforce with $R(x)=P(x)$
0 |  ⬇ def generate_samples_with_prompt(self,cached_done,keep = False):  if not hasattr(upstream_show,’normalize’):  return sm =wb. cppProcessor(cached_TLS = False)  self.maxOccurs = self.concurrency. anno_DealList()  tool.is(csrf_restore,lazy = True)  self.salt_made(csrf)  \par
1 |  ⬇ def generate_samples_with_prompt(self):  start = back_start - self.start + self.test_samples().set_ofmid result =[]  for step in range(start):  result.append(step)  result.append(step)  return result \par
0 |  ⬇ def generate_samples_with_prompt(self,type::phone_shard = None):  return int(int(self.last_offsets_best_timescale,type_op = ”0”))  \par
Table 9: Samples obtained from policies conditioned on prompt def
generate_samples_with_prompt


# Hölder estimates for solutions of parabolic SPDEs.

S.B. Kuksin, N.S. Nadirashvili and A.L. Piatnitski  Institut de Mathémathiques
de Jussieu–Paris Rive Gauche, CNRS, Université Paris Diderot, UMR 7586,
Sorbonne Paris Cité, F-75013, Paris, France & School of Mathematics, Shandong
University, Jinan, Shandong, China<EMAIL_ADDRESS>Aix-Marseille
Université, 39, rue F. Joliot-Curie, 13453 Marseille, France,
<EMAIL_ADDRESS>Arctic University of Norway, campus
Narvik, P.O.Box 385, 8505 Narvik, Norway & Institute for Information
Transmission Problems of RAS, Bolshoi Karetny per. 19, 127051, Moscow, Russia,
<EMAIL_ADDRESS>
###### Abstract

This paper considers second-order stochastic partial differential equations
with additive noise given in a bounded domain of $\mathbb{R}^{n}$. We suppose
that the coefficients of the noise are $L^{p}$-functions with sufficiently
large $p$. We prove that the solutions are Hölder-continuous functions almost
surely (a.s.) and that the respective Hölder norms have finite momenta of any
order.

Keywords: stochastic equation, Hölder-continuous function.

##  1 Introduction

This paper is devoted to $L^{\infty}$\- and Hölder estimates for solutions of
second-order parabolic stochastic partial differential equations (SPDEs) in
bounded domains, under the minimal regularity assumptions on the coefficients.
In recent years several works focused on the $L^{p}$\- and $W^{l,p}$-theory
for parabolic SPDEs; however, the $W^{l,p}$-estimates obtained in these works
rely on rather strong regularity assumptions on the coefficients. On the other
hand, in various qualitative and applied problems the estimates in terms of
$L^{\infty}$\- or even $L^{p}$-norms of the coefficients are very important.
Major progress in this area was achieved by Krylov in [1], where the
regularity assumptions were reduced essentially for equations posed in the
whole $R^{d}$.

In our work we propose another technique which allows us to further reduce the
regularity assumptions and to treat initial-boundary problems for solutions of
SPDEs with additive noise. Although this class of equations is smaller than
that in [1], still it is sufficiently large and contains many important
operators. Our approach differs essentially from that in [1] and gives rise to
a shorter proof of the main results.

In this paper we study SPDEs of the form

$\begin{array}[]{c}\displaystyle
du=Audt+\sum\limits_{j=1}^{N}f^{j}(x,t)dw_{t}^{j},\qquad(x,t)\in
Q\times(0,+\infty),\\\ \displaystyle u|_{t=0}=0,\qquad u|_{\partial
Q}=0,\end{array}$ (1)

where $Q$ is a smooth bounded domain in $\mathbb{R}^{n}$, $w^{j}_{t}$ are
standard independent one-dimensional Brownian motions, and $f^{j}(x,t)$ are
progressively measurable $L^{\infty}$\- or $L^{p}$-functions; in the latter
case $p$ is sufficiently large. Regarding the operator $A$ we assume that $A$
is either a generic second-order uniformly elliptic operator with sufficiently
smooth coefficients or is a divergence form uniformly elliptic operator with
measurable bounded coefficients. In this paper we prove that a solution of (1)
is a Hölder-continuous function a.s. and that the corresponding $L^{\infty}$\-
and Hölder norms have finite momenta of any order.

For a stochastically perturbed heat equation (i.e., for (1) with $A=\Delta$)
with periodic boundary conditions these estimates were proved by one of the
authors in the appendix of [2]. It should be noted that the results of this
paper can be generalized to the equations involving the infinite-dimensional
stochastic term $\sum_{j=1}^{\infty}f^{j}(x,t)dw_{t}^{j}$ instead of the
finite-dimensional one; $L^{\infty}$-convergence (or $L^{p}$-convergence with
sufficiently large $p$) of the sum $\sum_{j=1}^{\infty}|f^{j}(x,t)|$ is then
required.

On the basis of the $L^{\infty}$-estimates obtained in this work, one can
examine the regularity of solutions of some nonlinear SPDEs and prove the
solvability of these equations in certain functional spaces (see [2]). When
analyzing SPDEs, the $L^{\infty}$-estimates of this type can be used in the
same way as the classical maximum principle in the case of deterministic
equations.

In the first part of the paper we obtain several estimates for Hölder and
$L^{\infty}$-norms of solutions of deterministic parabolic equations. In what
follows we make use of these estimates to derive point-wise bounds for
increments of the solution of (1) and to estimate the corresponding function
norms.

##  2  Setting of the problem. Technical estimates.

Let $w^{j}_{t}$ be standard independent Brownian motions defined on a
probability space $(\Omega,\mathcal{F},\mathbf{P})$, and let
$(\mathcal{F}_{t})$ be the corresponding filtration of $\sigma$-algebras. We
assume that $f^{j}(x,t)$ are $\mathcal{F}\times\mathcal{B}$-measurable,
$(\mathcal{F}_{t})$-adapted functions, where $\mathcal{B}$ stands for the
Borel $\sigma$-algebra, and that at least one of the following two assumptions
holds:

$\begin{array}[]{lrrlll}\mathbf{B}^{\infty}&\\!\\!\\!:&\ \
|f(x,t)|&\\!\\!\\!\leqslant 1&\hbox{ almost surely (a.s.) for all $x$ and
$t$},\\\\[5.69054pt] \mathbf{B}^{p}&\\!\\!\\!:&\ \
\|f(\textperiodcentered,t)\|_{L^{p}(Q)}&\\!\\!\\!\leqslant 1&\hbox{ a.s. for
every }t.\end{array}$

In the following two statements, $A$ is a uniformly elliptic second-order
differential operator of the form

$A=a_{ij}(x)\frac{\partial}{\partial x_{i}}\frac{\partial}{\partial
x_{j}}+b_{i}(x))\frac{\partial}{\partial x_{i}}+c(x)$

with smooth coefficients.

######  Theorem 2.1.

Let $\mathbf{B}^{\infty}$ hold. Then under the above conditions
(a) for any $k\geqslant 0$ and any $T\geqslant 0$, a solution of problem (1)
satisfies the estimate

$\mathbf{E}\|u\|_{L^{\infty}(Q_{T})}^{k}\leqslant C_{k}\exp(\overline{c}T),$
(2)

where $Q_{T}=Q\times[T,T+1]$ and $\overline{c}=\max(0,\max_{Q}c(x))$;
(b) the solution $u(x,t)$ is Hölder-continuous a.s., and for any $T\geqslant
0$, $k\geqslant 0$, and $\theta<\frac{1}{2}$ the upper bound

$\mathbf{E}\|u\|_{C^{\theta}(Q_{T})}^{k}\leqslant
C(k,\theta)\exp(\overline{c}T)$ (3)

holds.

In estimates (2), (3), and everywhere below, for brevity we write
$|u|_{L^{\infty}(Q_{T})}$ instead of $|(u|_{Q_{T}})|_{L^{\infty}(Q_{T})}$,
etc.

######  Theorem 2.2.

If $\mathbf{B}^{p}$ holds for some $p>d$, then
(a) for any $k\geqslant 0$ and any $T\geqslant 0$, a solution of problem (1)
satisfies the estimate (2);
(b) the solution $u(x,t)$ is Hölder-continuous a.s., and for any $T\geqslant
0$, $k\geqslant 0$, and $\theta<1/2-d/(2p)$ the upper bound (3) holds.

######  Remark 2.1.

If the zero-order term $c(x)$ of the operator $A$ is non-positive, then the
exponential factor disappears in all the estimates (2)–(3) above. The case of
a generic operator that satisfies our assumptions can be reduced to the case
from Remark 1 by means of the following change of the unknown function
$U=\exp(-\alpha t)u$, with a proper choice of $\alpha$.

From now on we assume without loss of generality that $c(x)\leqslant 0$.

For a uniformly elliptic divergence form operator $A$ with bounded measurable
coefficients one has the following result.

######  Theorem 2.3.

Let $A$ be a uniformly elliptic divergence form operator with bounded
coefficients. Then under the assumption $\mathbf{B}^{\infty}$,
(a) for any $k\geqslant 0$ and any $T\geqslant 0$, the solution of problem (1)
satisfies the estimate

$\mathbf{E}\,\|u\|^{k}_{L^{\infty}(Q_{T})}\leqslant C_{k};$

(b) the solution $u(x,t)$ is a.s. Hölder-continuous, and there is $\alpha>0$
depending only on the lower and upper ellipticity constants of $A$ and on the
geometry of the domain $Q$ such that for any $T\geqslant 0$, $k\geqslant 0$,
and $\theta<\alpha$ the upper bound

$\mathbf{E}\|u\|^{k}_{C^{\theta}(Q_{T})}\leqslant C(k,\theta)$ (4)

holds. Under the assumption $\mathbf{B}^{p}$ with $p>d$,
(c) there exists $\alpha_{0}>0$ such that estimate (4) holds true for every
$T\geqslant 0$, $k\geqslant 0$, and $\theta\leqslant\alpha_{0}$.

The proof of these theorems relies on several auxiliary statements. Let us
consider the following initial-boundary problem:

$\frac{\partial v}{\partial t}-Av=0,\qquad v\big{|}_{t=0}=F(x),\qquad
v\big{|}_{\partial Q}=0,$ (5)

and denote by $S_{t}$ the corresponding semigroup:

$v(\cdot,t)=(S_{t}F)(\cdot),\qquad F=F(\cdot).$

The following statement deals with a “smooth” operator $A$.

######  Lemma 2.1.

Let $|F(x)|<M$. Then, for any $\theta<1$, the following estimates hold with
$c>0$:

$\displaystyle\|v(\cdot,t)\|_{C^{\theta}(Q)}$ $\displaystyle\leqslant$
$\displaystyle c(\theta)Mt^{-\frac{\theta}{2}}\exp(-ct),$ (6)
$\displaystyle|v(x,t+\delta)-v(x,t)|$ $\displaystyle\leqslant$ $\displaystyle
c(\theta)Mt^{-\theta}\delta^{\theta}\exp(-ct).$ (7)

Moreover, if $\|F\|_{L^{q}(Q)}\leqslant M$ and $q>1$, then

$\displaystyle\|v(\cdot,t)\|_{C^{\theta}(Q)}$ $\displaystyle\leqslant$
$\displaystyle c(\theta)Mt^{-\frac{\theta}{2}-\frac{d}{2q}}\exp(-ct),$ (8)
$\displaystyle|v(x,t+\delta)-v(x,t)|$ $\displaystyle\leqslant$ $\displaystyle
c(\theta)Mt^{-\theta-\frac{d}{2q}}\delta^{\theta}\exp(-ct).$ (9)

###### Proof.

Denoting by $G(x,y,t)$ the Green function associated with problem (5), we have

$\|G(x,\cdot,t)\|_{L^{p}(Q)}\leqslant
c(p)t^{-d/(2q)},\qquad\frac{1}{p}+\frac{1}{q}=1$ (10)

(see, for example, [3]). This implies the inequality

$\|v(\cdot,t)\|_{L^{\infty}(\mathbb{R}^{d})}\leqslant
c(p)t^{-d/(2q)}\|F\|_{L^{q}(\mathbb{R}^{d})}$ (11)

for any $F\in L^{q}(\mathbb{R}^{d})$.

In addition to problem (5), we consider in the domains $Q_{\gamma}=\gamma
Q=\\{\gamma x:x\in Q\\}$, $\gamma\geqslant 1$, the initial-boundary problems

$\begin{array}[]{c}\displaystyle\frac{\partial V}{\partial
t}-A_{\gamma}V=0,\qquad x\in Q_{\gamma},\\\\[5.69054pt]
V\big{|}_{t=0}=F(\gamma x),\qquad V\big{|}_{\partial
Q_{\gamma}}=0,\end{array}$ (12)

where

$A_{\gamma}=a_{ij}(\gamma x)\frac{\partial}{\partial
x_{i}}\frac{\partial}{\partial x_{j}}+\gamma^{-1}b_{i}(\gamma
x)\frac{\partial}{\partial x_{i}}+\gamma^{-2}c(\gamma x).$

Due to the smoothness of the coefficients of $A$ and of the domain $Q$, by the
standard elliptic estimates one has

$\left|\frac{\partial}{\partial
t}V(x,1)\right|+\left|\nabla_{x}V(x,1)\right|\leqslant
c\left\|V(\cdot,\frac{1}{2})\right\|_{L^{\infty}(Q_{\gamma})}$ (13)

uniformly in $x\in Q_{\gamma}$, with the constant $c$ that does not depend on
$\gamma$ (see [6], [5]). After an appropriate rescaling, the last estimate
reads

$t\left|\frac{\partial}{\partial
t}v(x,t)\right|+t^{\frac{1}{2}}\left|\nabla_{x}v(x,t)\right|\leqslant
c\Big{\|}v\big{(}\cdot,\frac{t}{2}\big{)}\Big{\|}_{L^{\infty}(Q)}$ (14)

for all $t$, $0<t\leqslant 1$. From (11) and (14) it follows that

$\left|\frac{\partial}{\partial t}v(x,t)\right|\leqslant
ct^{-1-\frac{d}{2q}}\|F\|_{L^{q}(Q)},$ (15) $\left|\frac{\partial}{\partial
x}v(x,t)\right|\leqslant ct^{-\frac{1}{2}-\frac{d}{2q}}\|F\|_{L^{q}(Q)},$ (16)

for all $t\leqslant 1$. Finally, we use the interpolation
inequality333Denoting $\|v\|_{L^{\infty}}$ by $\gamma_{0}$ and $\|v\|_{C^{1}}$
by $\gamma_{1}$, we have $|v(x+y)-v(x)|\leqslant 2\gamma_{0}$ and
$|v(x+y)-v(x)|\leqslant 2\gamma_{1}|y|$. Raising these inequalities to the
degrees $(1-\theta)$ and $\theta$, respectively, and multiplying the new ones,
we get that $|v(x+y)-v(x)|\leqslant
2^{1-\theta}\gamma_{0}^{1-\theta}\gamma_{1}^{\theta}|y|^{\theta}$. This
relation implies the required inequality.

$\|v\|_{C^{\theta}}\leqslant
2\|v\|^{1-\theta}_{L^{\infty}}\|v\|^{\theta}_{C^{1}}$

to derive (6), (7), and (8), (9) from (11), (15), and (16). For large $t$ the
required estimates follow from the exponential decay of the Green function.
This completes the proof of the lemma. ∎

For operators $A$ with measurable coefficients we have the following result.

######  Lemma 2.2.

Let $A$ be a divergence form uniformly elliptic operator with bounded
coefficients. Then there exists $\alpha>0$ depending only on the ellipticity
constants of $A$ and on the domain $Q$, such that for any $\theta<\alpha$ a
solution $v$ of problem (5) admits all the estimates of Lemma 2.1.

###### Proof.

The proof is rather similar to that of the preceding lemma and we outline it
briefly. According to [4] the inequality (10) (and thus (11)) is still valid.
The bound (13) is not valid any more but instead we can use the Nash-type
estimates for the Hölder norms of the solution $V(x,t)$ (see [5], [6]) to get
that

$\|V\|_{C^{\alpha}(Q_{\gamma}\times[1,2])}\leqslant
c\left\|V\big{(}\cdot,\frac{1}{2}\big{)}\right\|_{L^{\infty}(Q)}$

for some $\alpha>0$ and $c$ independent of $\gamma$. The rest of the proof is
the same as in the previous lemma. ∎

Now we proceed with problem (1) in a “smooth” case. Its solution does exist,
is unique, and can be written in the form (see [7, Chapter 5] and [1])

$u(t)=\int_{0}^{t}\sum\limits_{j=1}^{N}S_{t-\tau}f^{j}(x,\tau)dw_{\tau}.$

The last formula is, in fact, the definition of a mild solution. Clearly it is
sufficient to consider one element of the sum above. In what follows the
summation sign and the index $j$ will be omitted. We begin by obtaining point-
wise estimates of increments of the solution in $x$ variables. To this end we
denote $[S_{t-\tau}f(x_{1},\tau)-S_{t-\tau}f(x_{2},\tau)]$ by $g(t,\tau)$ and
introduce a random variable

$U=u(t,x_{1})-u(t,x_{2})=\int_{0}^{t}g(t,\tau)dw_{\tau}.$

The quadratic characteristics of the stochastic integral is given by
$X(t)=\int_{0}^{t}g^{2}(t,\tau)d\tau$. From Lemma 2.1 we derive the estimate

$X(t)\leqslant
M^{2}c(\theta)|x1-x2|^{2\theta}\int_{0}^{t}t^{-\theta}\exp(-2ct)\,dt\leqslant
M^{2}c^{\prime}(\theta)|x_{1}-x_{2}|^{2\theta},$

valid for any $\theta<1$. Thus, by the Burkholder–Davis–Gundy inequality we
have

$|U|^{p}\leqslant c_{1}(p)\mathbf{E}(X^{p/2})\leqslant
c(p,\theta)M^{p}|x_{1}-x_{2}|^{p\theta},\qquad\theta<1.$ (17)

Similarly, for a time increment we define

$\begin{array}[]{rl}\displaystyle
R=\\!\\!&\\!\\!u(x,t+\delta)-u(x,t)=\displaystyle\int_{t}^{t+\delta}S_{t+\delta-\tau}f(x,\tau)dw_{\tau}\\\\[5.69054pt]
&\displaystyle+\int_{0}^{t}\big{(}S_{t+\delta-\tau}f(x,\tau)-S_{t-\tau}f(x,\tau)\big{)}dw_{\tau}=I_{1}+I_{2}.\end{array}$

The first integral $I_{1}$ can be estimated as follows:

$\mathbf{E}|I_{1}|^{p}\leqslant c\delta^{p/2}M^{p}.$ (18)

Indeed, if we denote $S_{t+\delta-\tau}f(x,\tau)$ by $h_{1}(t,\tau)$, then

$\int_{t}^{t+\delta}h^{2}_{1}(t,\tau)\tau\leqslant M^{2}\delta,$

and required estimate (18) is a direct consequence of the
Burkholder–Davis–Gundy inequality.

In order to estimate $I_{2}$, we define

$h_{2}(t,\tau)=S_{t+\delta-\tau}f(x,\tau)-S_{t-\tau}f(x,\tau).$

By Lemma 2.1 we have

$|h_{2}(t,\tau)|\leqslant
c(\theta)M(t-\tau)^{-\theta}\delta^{\theta}\exp\big{(}-c(t-\tau)\big{)}$

for any $\theta<\frac{1}{2}$. Thus,

$\int_{0}^{t}h_{2}^{2}(t,\tau)d\tau\leqslant
M^{2}\delta^{2\theta}\int_{0}^{t}(t-\tau)^{-2\theta}\exp\big{(}-2c(t-\tau)\big{)}d\tau\leqslant
c(\theta)M^{2}\delta^{2\theta}$

and by the Burkholder–Davis–Gundy inequality we have

$\mathbf{E}|I_{2}|^{p}\leqslant c(\theta)\delta^{\theta
p}M^{p},\qquad\theta<\frac{1}{2}.$ (19)

Combining (17), (18), and (19), we obtain

$\mathbf{E}\left|u(x_{1},t_{1})-u(x_{2},t_{2})\right|^{p}\leqslant
c(\theta)\big{(}|t_{1}-t_{2}|+|x_{1}-x_{2}|\big{)}^{\theta p}$ (20)

for any $p>1$ and any $\theta<\frac{1}{2}$.

##  3  Boundedness and Hölder-continuity of the solution; higher momenta.

This section deals with the estimation of $L^{\infty}$ and $C^{\theta}$ norms
of the solution of problem (1). The first statement here is given by the
following lemma.

######  Lemma 3.1.

Let the functions $f^{j}(x,t)$ satisfy condition $\mathbf{B}^{\infty}$ of the
preceding section. Then

  * (a)

there is a.s. a continuous version of the solution $u(x,t)$ of problem (1);

  * (b)

under this choice of $u(x,t)$, for any $T\geqslant 0$ and any $p\geqslant 1$,
we have

$\mathbf{E}\|u\|^{p}_{L^{\infty}(Q\times[T,T+1])}\leqslant c(p)<\infty.$

###### Proof.

The first statement follows immediately from (20) by the Kolmogorov theorem;
it just suffices to choose a sufficiently large $p$.

To prove the second one we consider a cylinder $Q^{T}=Q\times[T,T+1]$ and
define a sequence of sets

$\mathcal{K}_{n}=\\{k\in\mathbb{Z}^{d+1}\big{|}k2^{-n}\in Q^{T}\\},\qquad
n=1,2,\ldots.$

For an arbitrary vector $e\in\mathbb{Z}^{d+1}$ such that

$|e|_{1}\sim\max\limits_{1\leqslant j\leqslant d+1}|e_{j}|=1,$

we then set $\xi_{k}^{n,e}=|u((k+e)2^{-n})-u(k2^{-n})|$. By Lemma 2.1 we have

$\mathbf{E}|\xi_{k}^{n,e}|^{p}\leqslant c(\theta)2^{-np\theta},\qquad
0<\theta<\frac{1}{2},$ (21)

for any $p\geqslant 1$. Let us introduce the events

$\mathcal{A}^{n,e}_{k,q}=\left\\{\omega\in\Omega\,|\,\xi_{k}^{n,e}\geqslant
Kq^{n}\right\\}.$

From (21) by the Chebyshev inequality we get

$\mathbf{P}(\mathcal{A}^{n,e}_{k,q})\leqslant\frac{\mathbf{E}|\xi_{k}^{n,e}|^{p}}{K^{p}q^{np}}\leqslant
c(\theta)\frac{2^{-np\theta}}{K^{p}q^{np}}.$

For each $n$ the total number of the events $\mathcal{A}^{n,e}_{k,q}$ is not
greater than $2^{\bar{c}(d,Q)n}$. Thus the probability of the union
$\mathcal{A}^{n}_{q}=\bigcup_{k\in\mathcal{K}}\big{(}\bigcup_{|e|_{1}=1}\mathcal{A}^{n,e}_{k,q}\big{)}$
satisfies the estimate

$\mathbf{P}(\mathcal{A}^{n}_{q})\leqslant
c(\theta)K^{-p}q^{-pn}2^{\bar{c}(d,Q)n-\theta pn}=c(\theta)K^{-p}\alpha^{n},$

where $\alpha=2^{\bar{c}(d,Q)}/(2^{\theta}q)^{p}$. If we set
$\theta=\frac{1}{3}$, $q=2^{-1/6}$, then $\alpha=2\bar{c}(d,Q)/2^{p/6}$.
Finally, taking $p>6\bar{c}(d,Q)+1$, we have $\alpha<\frac{1}{2}$. Hence, for
the probability of the event $\mathcal{A}=\bigcup_{n>1}\mathcal{A}^{n}_{q}$
the following estimate holds true:

$\mathbf{P}(\mathcal{A})\leqslant c(\theta)K^{-p}.$ (22)

From now on we assume without loss of generality that $Q$ is a $d$-dimensional
cube $(0,1)^{d}$. Indeed, to reduce the case of a general domain to that of a
cube it suffices to extend the function $u(x,t)$ in the exterior of $Q$ as
zero and to rescale the arguments if necessary.

Next, any point of $Q$ can be represented in the form

$x=\sum\limits_{j=1}^{\infty}e(j)2^{-j},\qquad|e(j)|_{1}\leqslant 1.$

Let us denote $x(m)=\sum_{j=1}^{m}e(j)2^{-j}$ (as usual, $x(0)=0$). Clearly,
$u(x(0),t)=0$. Then, by the definition of the set $\mathcal{A}^{n,e}_{k,q}$,
for any $\omega\not\in\mathcal{A}$ we have

$\big{|}u(x(m),t)-u(x(m+1),t)\big{|}\leqslant Kq^{m}=K2^{-m/6}.$

Therefore,

$|u(x,t)|\leqslant K\sum\limits_{m=1}^{\infty}2^{-m/6}=K2^{1/6}(2^{1/6}-1).$
(23)

It now follows from (22) and (23) that

$\mathbf{P}\big{\\{}\|u\|_{L^{\infty}(Q^{T})}\geqslant K\big{\\}}\leqslant
c_{p}(K+1)^{-p}$

for any $p>\bar{c}(d,Q)$. Finally, denoting $U_{T}=\|u\|_{L^{\infty}(Q^{T})}$,
we find that

$\mathbf{E}U^{s}_{T}=\int_{0}^{\infty}x^{s}d\mathbf{P}\\{U_{t}\leqslant
x\\}=s\int_{0}^{\infty}x^{s-1}\mathbf{P}\\{U_{t}\leqslant x\\}dx$ $\leqslant
sc_{p}\int_{0}^{\infty}x^{s-1}(x+1)^{-p}dx<\infty$

if $p>s+1$. The lemma is proved. ∎

In order to estimate the Hölder norms of $u(x,t)$, let us first formulate the
following simple assertion.

######  Lemma 3.2.

Let a function $g(y)$ satisfy the estimate

$\mathop{\mathrm{osc}}\limits_{J}g(\cdot)\leqslant\gamma_{n}$ (24)

in any small cube $J$ which is a mesh of the grid $2^{-n}\mathbb{Z}^{d+1}$,
i.e., in any $J=2^{-n}j+[0,2^{-n}]^{d+1}$, where $j\in\mathbb{Z}^{d+1}$. Then,
for any ${\scriptstyle\Delta}\in\mathbb{R}^{d+1}$, one has

$\big{|}g(y+{\scriptstyle\Delta})-g(y)\big{|}\leqslant
2\gamma\big{.}_{[\log_{2}(1/|{\scriptstyle\Delta}|)]},$

where $[\cdot]$ stands for the integer part.

###### Proof.

For $n=[\log_{2}(1/|{\scriptstyle\Delta}|)]$ let us consider the family of
closed cubes defined above. Clearly, both $y$ and $y+{\scriptstyle\Delta}$
either belong to the same cube or they are situated in adjacent cubes having
at least one point in common. So the statement of the lemma follows. ∎

Next, for a solution of (1) and for any $\omega\not\in\mathcal{A}$, inequality
(24) holds with $\gamma_{n}=Kq^{n}$. Thus,

$\displaystyle\big{|}u((x,t)+{\scriptstyle\Delta})-u(x,t)\big{|}$
$\displaystyle\leqslant$ $\displaystyle 2Kq^{[\log_{2}(1/|\Delta|)]}\leqslant
2Kq^{\log_{2}(1/|\Delta|)-1}$ $\displaystyle=$ $\displaystyle
2Kq^{-1}2^{\log_{2}(q)\log_{2}(1/|\Delta|)}=2Kq^{-1}|{\scriptstyle\Delta}|^{\log_{2}(1/q)}.$

If we put $q=2^{-\theta_{1}}$ with $\theta_{1}<\theta<\frac{1}{2}$, then
$\log_{2}(1/q)=\theta_{1}$, and the above inequality implies

$\big{|}u((x,t)+{\scriptstyle\Delta})-u(x,t)\big{|}\leqslant
2K2^{\theta_{1}}|{\scriptstyle\Delta}|^{\theta_{1}}\leqslant
4K|{\scriptstyle\Delta}|^{\theta_{1}}.$

Therefore, for all $\omega\not\in\mathcal{A}$,

$\|u\|_{C^{\theta_{1}}(Q^{T})}\leqslant 4K.$

Taking into account (22) we derive from the last estimate that

$\mathbf{P}\left\\{\|u\|_{C^{\theta_{1}}(Q^{T})}>K\right\\}\leqslant
c(p,\theta_{1})K^{-p}$

for all $p>1$. Finally, for any $\theta_{1}<\frac{1}{2}$ and any $m\geqslant
1$ the bound

$\mathbf{E}\|u\|^{m}_{C^{\theta_{1}}(Q^{T})}\leqslant c(m,\theta_{1})$

can be proved in the usual way, and we arrive at the following assertion.

######  Lemma 3.3.

For any $\theta<\frac{1}{2}$ a solution $u(x,t)$ of problem (1) is a.s. a
$C^{\theta}$-function. Moreover, for any cylinder $Q^{T}$ and any $m\geqslant
1$, the inequality $\mathbf{E}\|u\|^{m}_{C^{\theta}(Q^{T})}\leqslant
c(m,\theta)$ holds.

Now, to complete the proof of Theorem 2.1 it is merely sufficient to refer to
Lemmas 3.1 and 3.3.

Other statements of this work can be justified in the same way.

## References

  * [1] N. V. Krylov, On $L^{p}$-theory of stochastic partial differential equations in the whole space, SIAM J. Math. Anal., 27 (1996), 313–340.
  * [2] S. B. Kuksin, A stochastic nonlinear Schrödinger equation 1. A priori estimates, Proc. Steklov Inst. Math., 225 (1999), 219–242.
  * [3] A. Friedman, Partial Differential Equations of Parabolic Type, Prentice-Hall, Englewood Cliffs, NJ, 1964.
  * [4] D. G. Aronson, Bounds for the fundamental solution of a parabolic equation, Bull. Amer. Math. Soc., 73 (1967), 890–896.
  * [5] D. Gilbarg and N. S. Trudinger, Elliptic Partial Differential Equations of Second-Order, Springer-Verlag, Berlin, Heidelberg, 1977.
  * [6] O. A. Ladyzenskaja, V. A. Solonnikov, and N. N. Ural’tseva, Linear and Quasilinear Equations of Parabolic Type, Transl. Math. Monographs 23, AMS, Providence, RI, 1967.
  * [7] G. Da Prato and J. Zabczyk, Stochastic Equations in Infinite Dimensions, Cambridge University Press, Cambridge, UK, 1992.