# Designing Biological Sequences via Meta-Reinforcement Learning and Bayesian
Optimization

Leo Feng &Padideh Nouri &Aneri Muni &Yoshua Bengio 1 &Pierre-Luc Bacon 2&
Mila – Université de Montréal Correspondence to: Leo Feng
<EMAIL_ADDRESS>
1CIFAR Senior Fellow, 2CIFAR AI Chair

###### Abstract

The ability to accelerate the design of biological sequences can have a
substantial impact on the progress of the medical field. The problem can be
framed as a global optimization problem where the objective is an expensive
black-box function such that we can query large batches restricted with a
limitation of a low number of rounds. Bayesian Optimization is a principled
method for tackling this problem. However, the astronomically large state
space of biological sequences renders brute-force iterating over all possible
sequences infeasible. In this paper, we propose MetaRLBO where we train an
autoregressive generative model via Meta-Reinforcement Learning to propose
promising sequences for selection via Bayesian Optimization. We pose this
problem as that of finding an optimal policy over a distribution of MDPs
induced by sampling subsets of the data acquired in the previous rounds. Our
in-silico experiments show that meta-learning over such ensembles provides
robustness against reward misspecification and achieves competitive results
compared to existing strong baselines.

##  1 Introduction

Over the past few years, there has been a rapid interest in expediting the
drug discovery process using Artificial Intelligence(AI). Leading
pharmaceutical companies such as Pfizer, Sanofi and Roche are benefiting from
AI in their pipelines to be able to keep up with the growing industry [21]. In
particular, designing biological sequences has been a long-standing challenge
due to the large chemical space with very sparse functionally interesting
sequences [8]. The design of such sequences can be framed as a black-box
optimization problem. In this case, the time and cost-intensive wet-lab
evaluation correspond to the objective black-box function.

Bayesian Optimisation is a principled way to tackle black-box optimization
problems that approximates the expensive objective function (true oracle) with
a cheap surrogate model. BO evaluates samples using an acquisition function to
explore (i.e. select uncertain but informative sequences) or exploit (i.e.
select sequences with high predicted value). Although acquisition functions
are relatively cheap to evaluate, naively searching over the space of possible
sample locations can quickly become intractable. In previous work, Swersky et
al. [31] proposed to train a policy such that it modifies a population of
sequences to directly optimise the acquisition function. Belanger et al. [4]
propose to brute-force enumerate over the search space for shorter sequences
and to use regularized evolution [25] for designing longer sequences. Romero
et al. [26] propose to construct sequences with enhanced thermostability using
crossover on an existing dataset of sequences and selects sequences using GP-
UCB [30].

In this work, we treat this discovery of biological sequence problem as
training a generator to propose promising candidate sequences and use Bayesian
Optimization as a selection procedure. Generative models give us the ability
to jump to the regions of the landscape that was never searched before. In
this paper, we propose a method called MetaRLBO where we frame the problem of
training a generator as a Meta-Reinforcement Learning problem and apply
Bayesian Optimization for batch black-box function optimization. In each
round, we design our tasks by constructing a distribution of proxy oracles and
train our generator via Meta-Reinforcement Learning. Afterwards, we generate
sequences with this meta-learned generator and finally select the sequences to
query the true oracle via Bayesian Optimization. Our experiments show that
MetaRLBO achieves competitive performance compared to existing baselines. In
addition, we analyse the uncertainty estimates given by various surrogate
models for batch sequence design.

##  2 Background

###  2.1 Problem Setting

The optimisation problem consist of generating a sequence that maximises the
expected reward $s^{*}_{t}=\mathrm{argmax}_{s_{t}\in S}f(s_{t})$. The sequence
design can be viewed as a Markov decision process (MDP) where $s_{T}$ is a
sequence of length $T$ and of alphabet $\Sigma_{t=1}^{T}a_{t}$, that maximises
an experimentally measured value (oracle) $f:S\rightarrow\mathbb{R}$. For
example in the AMP task, $f(s)$ is defined as the antimicrobial efficacy of
the candidate sequence s which can be computed using wet-lab experiments. In
particular, we are interested in a specific case of black-box optimisation:
one which requires few rounds of evaluations over large batches, typical of
wetlab experiments over biological assays (DNA, proteins, RNA, etc). This
settings is one where in each round $r$ we want to minimize an objective
function using the information of previously generated sequences. In each
round, our algorithm generates a set of candidate sequences
$\mathcal{B}_{r}=\bigcup_{i=1}^{B}s_{i}$ where $\mathrm{B}$ is the size of the
batch. The sequences are then evaluated according to our oracle $f$ and added
to the queried data
$\mathcal{D}\leftarrow\mathcal{D}\bigcup\\{(s,f(s))|s\in\mathcal{B}_{r}\\}$
and a new round is started. The objective is to maximise the score of
aggregated sequences $\mathrm{max}_{s\in\bigcup_{i}\mathcal{B}_{i}}f(s)$.

###  2.2 Meta-Reinforcement Learning

The specific flavor of Meta-Reinforcement Learning (Meta-RL) considered in
this paper is one where we have a distribution of Markov Decision Problems
(MDPs) $\Omega(\mathcal{M})$ which we aim to control optimally. Each such MDP
$\mathcal{M}_{i}\in\Omega(\mathcal{M})$ is defined over discrete state and
action space $\mathcal{S}$ and $\mathcal{A}$ respectively. In the general
case, each MDP can also have its own transition probability function
$P_{i}:S\times\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R}$ and reward
function $R_{i}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$. In our
context, the problem structure is such that the transition function is known
and shared across MDPs but the reward function varies.

We consider a performance criterion based on the expected discounted return
over a finite horizon of length $H$ in searching for an optimal randomized
policy $\pi:\mathcal{S}\to\text{Dist}(\mathcal{A})$. We write the expected
return $\mathcal{J}$ under $\pi$:

$\mathcal{J}(\pi)=\mathbb{E}_{P_{i},\pi}\left[\sum_{t=0}^{H-1}\gamma^{t}r(s_{t},a_{t},s_{t+1})\right]\enspace,$

$\gamma$ is a discount factor. While dynamic programming [Puterman1994] offer
solution methods to solve finite-horizon problems, they are restricted to the
set of deterministic policies and are only applicable to small known MDPs.
Here, stochastic policies are specifically sought for due to the exploration
problem.

In adopting a learning perspective, our meta-RL problem becomes one where we
want to learn a policy from the data gathered across a collection of MDPs
sampled during training such that it generalises well to new MDPs
$\mathcal{M}^{\prime}\in\Omega(\mathcal{M})$ given little data. By
generalization, we mean that we expect the learned policy to attain good
performance in the new MDP with few adaptation samples. The process of
training a policy $\pi$ over sampled MDPs from the distribution of MDPs is
meta-training and that of evaluating the generalisation of $\pi$ is meta-
testing.

Model Agnostic Meta-Learning (MAML) MAML [11] tackles a specific instance of
the meta-learning problem [6, 27] with a gradient-based algorithm inspired by
bi-level optimization [3]. More specifically, the MAML formulation considers
the problem of learning initialisation parameters $\theta_{0}$ such that the
policy adapts (improves) quickly within a few gradient updates to a new task.
More precisely, the meta-RL problem considered in MAML can be concisely
formulated as:

$\displaystyle\min_{\theta_{0}}\,\,$
$\displaystyle\mathbb{E}_{\mathcal{M}^{\prime}\sim\Omega(\mathcal{M})}[\mathcal{L}(\theta_{K}^{\mathcal{M}};\mathcal{M}^{\prime})]$
s.t.
$\displaystyle\theta_{k+1}=\theta_{k}-\alpha\nabla_{\theta}\mathcal{L}(\theta_{k}^{\mathcal{M}};\mathcal{M})$
$\displaystyle\theta_{0}^{\mathcal{M}}=\theta\quad\quad\\{\forall\mathcal{M}\in\Omega(\mathcal{M})\\}$
$\displaystyle\mathcal{L}(\theta_{k};\mathcal{M})=\frac{1}{T}\sum_{t}G_{t}\log\pi_{\theta_{k}}(a_{t}|s_{t})$

where $K$ refers to the number of inner loop updates, $T$ is the number of
timesteps (i.e. the horizon), $\theta$ refers to the meta-parameters,
$\\{(s_{t},a_{t},r_{t})\\}_{t=0}^{T-1}$ refers to a trajectory generated by
the policy, and $G_{t}$ is the return from timestep $t$.

###  2.3 Bayesian Optimization

Bayesian Optimization (BO) aims to maximise a black-box (potentially non-
differentiable) objective function in a few evaluations. It does so by
building a surrogate model (e.g., a GP [24] or ensemble) of the true (oracle)
objective function such that it can be queried at a lower cost. Given the
surrogate model, BO then proceeds to efficiently compute the posterior
distribution over the function scores and suggest promising candidates
according to an acquisition function to be evaluated by the true oracle.
Examples of acquisition functions are the following: Upper Confidence Bound
(UCB) [30], Posterior Mean, Thompson Sampling [32], Entropy Search [14, 15],
or Expected Improvement [16, 22] Acquisition functions not only take into
account the score of a candidate as predicted by the surrogate model but also
its uncertainty. As a result, BO methods aim to find a suitable trade-off
between exploration (gathering informative data) and exploitation
(maximization of the black-box score function).

While our algorithm is compatible with any acquisition function, we found that
Upper Confidence Bound (UCB) and Posterior Mean heuristics perform well in
practice while being simple to implement. The first strategy, UCB, is defined
as $\text{AF}_{\text{UCB}}=\mu(s)+\beta\sigma(s)$ where $s$ is a sample,
$\mu(s)$ and $\sigma(s)$ are the mean and standard deviation predicted by the
surrogate model and $\beta$ is a hyperparameter controlling the exploration-
exploitation trade-off and with larger values favouring sequences of higher
uncertainty. In contrast, the Posterior Mean approach selects only sequences
with high predicted scores. As a result, the method does not explicitly try to
explore uncertain areas. Posterior Mean can therefore be found as a subcase of
UCB for the value of $\beta=0$.

While Gaussian Processes have been the de facto choice in BO application,
their poor scalability in high-dimensions [37] and over large datasets
rendered them incompatible with modern deep learning tools. In this work, we
use instead an ensemble of convolutional neural networks as the surrogate
model. In this case, $\mu(s)$ is given by the mean of the predictions from the
ensemble and $\sigma(s)$ is similarly estimated using the standard deviation.

##  3 Methodology

###  3.1 Problem Setting

Sequence Generation as a Markov Decision Problem In order for the BO procedure
to scale over large batches of sequences with the fewest query to the oracle,
we need to learn a probabilistic model allowing us to sample promising
sequences according to their estimated score and degree of diversity. The
requirement to be able to sample better candidates according to some specified
objective function renders off-the-shelf probabilistic modelling inapplicable.
Furthermore, it does not suffice to merely capture the statistical properties
of a given dataset. A generator needs to be able to extrapolate beyond it and
steer the BO towards promising regions of the solution space.

We approach this problem from a reinforcement learning perspective and show
that problem of learning a good generator can be seen as one of finding an
optimal policy in an MDP with a specific structure. In this framework, our
generator iteratively constructs new sequences by appending one element at a
time to a string (initially empty) until it chooses to stop. We view this
procedure as a sequential decision making problem where the action space is
discrete (for the problems presented in this paper) and consists of all the
possible symbols of a given alphabet (eg. amino acids) and the state space
$\mathcal{S}=\bigcup_{t=1,\ldots L}\mathcal{A}^{t}$ is the set of all possible
sequence prefixes where $L$ is the length of the sequence. In other words, at
time step $t$, the state is $s_{t}=a_{0},\ldots a_{t-1}\text{ s.t.
}a_{i}\in\mathcal{A}$. Because the state changes according to the action of
the concatenation operation, the transition probability function in this MDP
is fully known and deterministic, ie:
$P(s_{t+1}|s_{t},a_{t})\triangleq\mathds{1}\left(s_{t+1}=\text{concat}(s_{t},a_{t})\right)$
In the current instantiation of our algorithm, we provide our system with
feedback on the generated string only at the end of the sequence construction
and not during. Note that if additional evaluative feedback is available for
partial sequences, it could also be encoded within the reward function without
any change to our algorithm. In all applications considered in this paper, we
construct sparse reward functions which incorporate domain-specific scoring
metrics as well as a general diversity bonus to encourage the generation of
different enough sequences across rounds.

Figure 1: Schematic of the MetaRLBO Framework.

###  3.2 Inner-Outer Loop Optimisation

The nature of our problem prevents us from querying the oracle (wetlab)
directly to evaluate the reward function. Hence, we devise a hierarchical
approach in which a Bayesian optimisation procedure orchestrates the
interaction with the oracle while using samples from the RL generator in the
inner loop. Figure 1 provides an overview of the interaction protocol between
all components of our system. In this inner loop, the generator is trained
over an ensemble of estimated reward functions acting as a proxy to the
expensive true oracle. We are therefore under a model-based RL setting, but
one where the transition dynamics are fully known and deterministic.

Learning an ensemble of reward functions offers many advantages: it provides
uncertainty estimates readily usable by the BO procedure and provides more
robustness in the presence of error in the estimated reward model. However, in
order for this approach to be practical, we need to be able to learn good
policies sufficiently fast for each such proxy oracle (reward function). We
address this issue using a MAML-like meta-learning approach where we meta-
learn initial parameters for the generator.

####  3.2.1 Inducing a Distribution over Proxy MDPs

We learn a distribution over proxy oracles $\Omega(f)$ by training on the
queried dataset $D=\bigcup_{i}B_{i}$. The initial dataset $B_{0}$ can either
be derived from previous experiments (e.g., related wet-lab experiments) or
sampled randomly according to some specified prior distribution. In order to
induce a distribution over possible MDPs, we sample different subsets of the
current dataset $D$, fit a proxy-oracle model to predict the score, and train
an RL agent (generator) for each sub-sampled dataset of size $p|D|$ where
$0<p\leq 1$.

####  3.2.2 Training the generator via Meta-RL

The particular flavor of meta-learning considered in this paper is based on
the adaptation procedure of [11]. In each meta-training step, we sample a set
of $V$ proxy oracles $\hat{f}_{1},\hat{f}_{2},\ldots\hat{f}_{V}\in\Omega(f)$.
Each proxy oracle specifies a different MDP (i.e. task) $M_{i}$ for which the
reward function is given by $\hat{f}_{i}$. During the adaptation phase of our
generator, we fine-tune $\pi_{\theta}$ for $K$ steps using a REINFORCE
gradient estimator from a learned $\theta^{0}$.

$\displaystyle\theta^{i}_{t+1}\leftarrow\theta^{i}_{t}-\alpha\nabla_{\theta}\mathcal{L}(\theta^{i}_{t};M_{i})\enspace,$

In order to learn this initialisation, we backpropagate through the $K$
gradient steps as follows:

$\displaystyle\theta^{0}_{t+1}\leftarrow\theta^{0}_{t}-\gamma\frac{1}{V}\sum_{i=1}^{V}\nabla_{\theta}\mathcal{L}(\theta^{i}_{K};M_{i})$

While higher-order differentiation of the REINFORCE surrogate loss results in
a biased meta-gradient [12], we choose to ignore this technicality in favour
of increased computation efficiency as commonly done in practice with other
algorithms.

####  3.2.3 Generating sequences

When generating sequences to evaluate, we sample a number of proxy oracles
$\bar{f}_{1},\bar{f}_{2},\ldots,\bar{f}_{Q}\in\Omega(f)$, each reflecting a
belief of the true oracle $f$. We then finetune each generator for some pre-
defined number of steps for each tasks, giving us a set of policies
$\pi_{\theta^{(1)}},\pi_{\theta^{(2)}},\ldots\pi_{\theta^{(Q)}}$. At this end
of this procedure, we then query $|P|$ sequences from each policy that we
gather into a batch $\mathcal{P}=\bigcup_{i=1,\ldots,Q}P_{i}$. Since the
reward function of each MDP is different, generated sequences from different
policies $\pi_{\theta^{(1)}},\pi_{\theta^{(2)}},\ldots\pi_{\theta^{(Q)}}$
provide uncertainty about the generation process as a whole.

Promoting Diversity within a round In each round of interaction at the outer
level (from BO to wetlab), a batch of candidate sequences to be presented to
the Bayesian Optimization must be formed using the ensemble of generators. By
virtue of using a different reward function in each MDP, the corresponding
learned generators tend to be different from each other, i.e. the learned
optimal policies need not be the same. Hence, the combination of subset
sampling and meta-learning readily provides a mechanism for generating more
diverse candidates in each round. That is: it provides a form of meta-
exploration on top of the built-in exploration mechanism of softmax
(randomized) policies.

Promoting Diversity Across Rounds Similar to the existing literature on count-
based methods and density models for promoting exploration [5, 2], we augment
the reward with an exploration bonus. This method encourages the generation of
diverse sequences relative to the ones queried in the previous rounds
(diversity across rounds). This augmented reward is defined as
$r=f(s)-\lambda\mathrm{dens}_{\epsilon}(s)$ where
$\mathrm{dens}_{\epsilon}(s)$ is the weighted number of sequences with a
distance less than a threshold $\epsilon$ from $s$. $\lambda$ is a
hyperparameter that controls the the strength of exploration.

####  3.2.4 Selecting via Bayesian Optimization

When selecting sequences, we evaluate the generated sequences $P$ according to
an acquisition function. For the surrogate model, we can re-use the proxy
models ($\bar{f}_{1},\ldots\bar{f}_{Q}$) sampled from the proxy oracle
distribution $\Omega(f)$ that were used for training our policies. Aggregating
the proxy models together, we get a surrogate model comprising of an ensemble
of neural networks without the need for additional computation. Alternatively,
we can train a new surrogate model given the queried dataset
$\bigcup_{i}B_{i}$. However, in our experiments, we found that re-using the
proxy models works well in practice and provides good uncertainty estimates.
For evaluation, we greedily select the sequences $B_{i}\subseteq P$ that
maximise the acquisition function and query them using the objective function
(i.e. true oracle) $f$.

##  4 Experiments

###  4.1 Training details

Policy Network We implement the policy as a feedforward neural network that
takes as input the flattened one-hot encoding of the generated string, i.e. a
vector of dimension $\mathbf{R}^{L\times A}$ where $L$ is the length of the
sequence and $A$ is the size of the alphabet. The network output is the logits
for the distribution over the next character to be generated. We also use
entropy bonus to avoid premature convergence [1]. We apply positional encoding
[34] to the one-hot representation of the input string.

Surrogate Model In our experiments, we use an ensemble of CNNs for our proxy
models. For each ensemble member, the model is trained for 10 epochs using the
ADAM optimizer [17] with a batch size of 50 and mean squared error loss. In
practice, we found setting $p=1.0$ sufficient for training neural networks
since diversity can be induced by the different neural network initialisation.
However, setting $p<1.0$ is necessary for evaluating different proxy models
(see Section 4.5)

Training During meta-training, we sample $V=4$ proxy models per meta-updates.
When generating sequences, we sample $Q=32$ proxy oracles and generate
$|P|=64$ sequences from each of $\pi_{\theta}^{(i)}$. In our experiments, we
assume no prior knowledge is given, generating $B_{0}$ via a random policy.

###  4.2 Datasets

We test our approach on three kinds of sequence optimization problems:
designing antimicrobial peptide sequences (AMPs), ribonucleic acid (RNA)
sequences, and sequences that maximise a synthetic Alternating Ising Model. In
the AMP problem, the sequences are of variable lengths. In the Alternating
Ising and the RNA14 Task, the sequence length is fixed.

Antimicrobial Peptides (AMP) AMPs are small peptides with amino acids as their
building blocks and a length generally ranging from 8 to 75 (amino acids).
This task consists of an alphabet of size 20 and a maximum length of 50, which
gives rise to a search space of size $O(20^{50})$. Following [2], we train a
random forest classifier to predict whether a sequence is antimicrobial
towards a certain pathogen and we use that as our ground truth (wet-lab)
simulator. In our experiments, we perform 12 rounds with a batch size of 250.

Alternating Ising Model We consider the synthetic problem of generating a
string of alternating characters [31]. The string with the highest score is
one that only alternates between two characters. We consider the problem
setting with lengths of 20 with an alphabet of size 20. As such, the search
space of this problem is $O(20^{20})$. In our experiments, we perform 16
rounds with a batch size of 500.

Ribonucleic Acid (RNA) We design RNA sequences with length 14 from the
alphabet of size 4 nucleotides, this would give us a search space of
$O(4^{14})$. Here the optimisation problem can be defined as finding a
sequence that maximises the negative binding energy towards a hidden RNA
target of length 50. To simulate the ground truth oracle we use FLEXS package.
[29, 20] In our experiments, we perform 12 rounds with a batch size of 100.

###  4.3 Baselines

In our experiments, we compare against several baselines including exploration
algorithms proposed in FLEXS [29] and Bayesian Optimization methods proposed
in [31]. Primarily, we consider: (1) Random: A baseline that mutates a random
previously measured sequence. (2) Genetic: A naive genetic algorithm that uses
a Wright-Fisher model and single point mutations and recombinations. [29]. (3)
CMA-ES: Covariance Matrix Adaptation Evolution Strategies [13, 29]. (4)
DynaPPO: A model-based RL algorithm that learns a reward function given by the
mean predicted value of an ensemble of various different models such as random
forests, gaussian processes, and bayesian ridge regression. [2, 29]. (5)
Adalead: A model-guided evolutionary greedy algorithm [29].

###  4.4 Comparison with baselines

For a fair comparison, mutative methods are initialised with a random string,
ensuring no prior knowledge is given to the method. Figure 2(a) shows results
using MetaRLBO on the AMP, RNA, and Alternating Ising Model problems. In Table
2, we compare our method with Bayesian Optimization methods and evolutonary
methods. We see that MetaRLBO outperforms several existing baselines.

(a) AMP

(b) RNA14

(c) Ising20

Figure 2: Cumulative maximum scores given by the objective function $f$. (a)
Antimicrobial Peptide Design (b) Ribonucleic Acid (c) Alternating Ising Model

Alternating Chain | Length 20
---|---
Acq. Function | UCB | POST
Single Mutant | 14.67 | 14.67
Regularized Evol | 14.67 | 14.67
BO + Single Mutant | 15.33 | 13.67
BO + Regularized Evol | 16.67 | 15.00
BO + DES | 16.67 | 16.33
MetaRLBO | 18.00 | 17.00
Table 1: Alternating Chain Ising20 task. Previous results obtained from [31]

Proxy Model | Cumul. Max
---|---
CNN | 18
MLP | 15
Table 2: Ising20 Task using UCB. We compare the performance of MetaRLBO when
trained on different proxy oracles.

###  4.5 Analysis

(a) Uncertainty Calibration Plot

(b) Box Plot of NLL

(c) Ensemble of Policies

Figure 3: (a) Uncertainty Calibration Plot comparing the expected vs.
empirical rates of observing an outcome $y_{t}$. A perfect model would achieve
the line $y=x$. (b) Box Plot of the Negative Log Likelihood comparing an
ensemble of MLP and an ensemble of CNNs. The surrogate models are trained on
data selected in the first $n$ rounds and evaluated on sequences selected in
the $n+1$-th round. (c) An ablation comparing an ensemble of fine-tuned
policies from a meta-learned initialization and an ensemble of policies
trained from scratch.

Uncertainty Model Prior work considered using an ensemble of MLPs as the
surrogate model for Bayesian Optmization [4, 31] In Figure 3(a), we show an
uncertainty calibration plot comparing various surrogate models trained on
sequences selected in the first $n$ rounds and evaluated on the sequences
selected in the $n+1$-th round. This is a measure of how well a surrogate
model is at estimating uncertainty in a multi-round setting. Calibration in
the regression setting [19] means that $y_{t}$ should fall in a $c\%$
confidence interval $c\%$ of the time. This means that in Figure 3(a), we
would want surrogate models to achieve as close to $y=x$ as possible.

To generate the data for the analysis, we run MetaRLBO on the AMP task using
UCB as our acquisition function and an ensemble of CNNs as our surrogate
model. In Figure 3(a), we considered surrogate models such as an ensemble of
feedforward neural networks with three layers (32, 8, 4 units and $p=1.0$)
used in Belanger et al. [4] and Swersky et al. [31], ensemble of convolutional
neural networks ($p=1.0$), ensemble of Bayesian Ridge Regressors ($p=0.8$),
ensemble of K-Neighbors Regressors ($p=0.8$), and Gaussian Processes.
Additional plots are included in the Appendix. The results show an an ensemble
of CNNs is better calibrated than the other surrogate models. For a more
comprehensive analysis, we also measure the Negative Log-likelihood, another
popular metric for evaluating predictive uncertainty [7]. We see that an
ensemble of MLP and an ensemble of CNN are within intervals of each other
under this metric. However, the ensemble of MLP has higher variance.

We also evaluate the uncertainty models empirically in practice on the
Alternating Ising Model task. In this experiment, we run MetaRLBO from scratch
using either MLP or CNN as the proxy model $\Omega(f)$. In practice, we found
CNNs to perform better (see Table 2). The result of our analysis suggests that
convolutional neural networks are better suited for uncertainty estimation in
biological sequence design than that of feedforward neural networks.

Ensemble of Policies. As an ablation, we compare MetaRLBO with training an
ensemble of policies. Instead of fine-tuning different $Q$ policies from a set
of meta-trained parameters, we train $Q$ different policies from scratch per
round such that each policy optimises for a different proxy oracle. In these
experiments, we set $Q=8$, generating $256$ sequences from each policy. In our
MetaRLBO experiments, we additionally set $V=4$ and $K=2$. Empirically, we
found that training a policy via meta-reinforcement learning improved the
performance (see Figure 3(c)).

##  5 Related Work

Machine learning has been effective in optimizing DNA and protein sequences
[36, 9, 10]. We can divide existing methods for biological sequence design
into three categories [2] : directed evolution [38], optimization using
discriminative models (e.g. Bayesian Optimization), and optimization using
generative model [33, 23].

Optimization using discriminative models alternates between 1) fitting a proxy
model $f^{\prime}(x)$ to approximate $f(x)$ and 2) selecting a batch of
sequences using an acquisition function based on $f^{\prime}(x)$. Swersky et
al. [31] introduces Deep Evol. Solver (DES) to train a policy to evolve a
population of strings with character-level edits to optimize an acquisition
function. Evolutionary methods perform a local search in the space of
sequences, in a hill-climbing fashion. As a result, they have poor sample
efficiency. Directed evolution which mimics the natural selection involves the
generation of a large set of diverse candidates to find functionally
interesting candidates [38]. Optimization methods based on generative models
learn a generator $p_{\theta}(x)$ that aims to maximise the expected value of
$f(x)$: $\mathbf{E}_{x\sim P_{\theta}(x)}[f(x)]$. One way to optimize this is
via RL. Wang et al. [35] propose to train a long short-term memory (LSTM)
generative model and a bidirectional LSTM classification model for designing
AMP sequences. Schuchardt et al. [28] propose to train an RL agent how to
mutate, perform selection, and cross-over on sequences. DynaPPO [2] is a
model-based RL algorithm that trains an agent to generate sequences that
maximise the predicted mean given by an ensemble of various models such as
Random Forest, Bayesian Ridge Regression, MLP, etc. Our proposed method
(MetaRLBO) lies in the intersection between optimization using discriminative
models and optimization using generative models. As far as we are aware, we
are the first to propose combining Meta-RL and Bayesian Optimization for
biological sequence design.

##  6 Conclusions and Future Work

We have introduced MetaRLBO, a method that combines a generator based on Meta-
Reinforcement Learning and a selection procedure via Bayesian Optimization to
propose promising sequences that optimise a black-box function. We also
analyse the uncertainty estimates of surrogate models and find that an
ensemble of CNNs is better calibrated than an ensemble of MLPs as used in
prior work.

When selecting sequences, MetaRLBO currently selects sequences greedily
according to the acquisition function. However, instead of querying two
promising but similar sequences, it would be advantageous to query
sufficiently different sequences to explore the search space. This additional
level of exploration can be tackled with batch-aware bayesian optimization
methods such as BatchBald [18] and would be an interesting future direction to
explore.

## References

  * Ahmed et al. [2019] Z. Ahmed, N. Le Roux, M. Norouzi, and D. Schuurmans.  Understanding the impact of entropy on policy optimization.  In _International Conference on Machine Learning_ , pages 151–160. PMLR, 2019.
  * Angermueller et al. [2020] C. Angermueller, D. Dohan, D. Belanger, R. Deshpande, K. Murphy, and L. Colwell.  Model-based reinforcement learning for biological sequence design.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=HklxbgBKvr.
  * Bard [1998] J. F. Bard.  _Practical Bilevel Optimization_.  Springer US, 1998.
  * Belanger et al. [2019] D. Belanger, S. Vora, Z. Mariet, R. Deshpande, D. Dohan, C. Angermueller, K. Murphy, O. Chapelle, and L. Colwell.  Biological sequences design using batched bayesian optimization.  2019\.
  * Bellemare et al. [2016] M. G. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos.  Unifying count-based exploration and intrinsic motivation.  _Advances in Neural Information Processing Systems_ , (Nips):1479–1487, 2016.  ISSN 10495258.
  * Bengio et al. [2002] Y. Bengio, S. Bengio, and J. Cloutier.  Learning a synaptic learning rule.  01 2002.  doi: 10.1109/IJCNN.1991.155621.
  * Candela et al. [2006] J. Q. Candela, C. E. Rasmussen, F. Sinz, O. Bousquet, B. Schölkopf, and J. Quiñonero Candela.  _Evaluating Predictive Uncertainty Challenge_ , volume 3944 of _Lecture Notes in Computer Science_ , page 1–27.  Springer, machine learning challenges - evaluating predictive uncertainty, textual entailment and object recognition systems edition, January 2006.  URL https://www.microsoft.com/en-us/research/publication/evaluating-predictive-uncertainty-challenge/.
  * Capecchi and Reymond [2021] A. Capecchi and J. L. Reymond.  Peptides in chemical space.  _Medicine in Drug Discovery_ , 9:100081, 2021.  ISSN 25900986.  doi: 10.1016/j.medidd.2021.100081.  URL https://doi.org/10.1016/j.medidd.2021.100081.
  * Chhibbar and Joshi [2019] P. Chhibbar and A. Joshi.  Generating protein sequences from antibiotic resistance genes data using generative adversarial networks.  _arXiv preprint arXiv:1904.13240_ , 2019.
  * de Jongh et al. [2020] R. de Jongh, A. van Dijk, M. Julsing, P. Schaap, and D. de Ridder.  Designing eukaryotic gene expression regulation using machine learning.  _Trends in Biotechnology_ , 38(2):191–201, Feb. 2020.  ISSN 0167-7799.  doi: 10.1016/j.tibtech.2019.07.007.
  * Finn et al. [2017] C. Finn, P. Abbeel, and S. Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _International Conference on Machine Learning_ , pages 1126–1135. PMLR, 2017.
  * Foerster et al. [2018] J. N. Foerster, G. Farquhar, M. Al-Shedivat, T. Rocktäschel, E. P. Xing, and S. Whiteson.  Dice: The infinitely differentiable monte carlo estimator.  In J. G. Dy and A. Krause, editors, _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018_ , volume 80 of _Proceedings of Machine Learning Research_ , pages 1524–1533. PMLR, 2018\.
  * Hansen [2016] N. Hansen.  The cma evolution strategy: A tutorial.  _arXiv preprint arXiv:1604.00772_ , 2016.
  * Hennig and Schuler [2011] P. Hennig and C. Schuler.  Entropy search for information-efficient global optimization.  _Journal of Machine Learning Research_ , 13, 12 2011.
  * Hernández-Lobato et al. [2014] J. M. Hernández-Lobato, M. W. Hoffman, and Z. Ghahramani.  Predictive entropy search for efficient global optimization of black-box functions.  In _NIPS_ , 2014.
  * Jones et al. [1998] D. Jones, M. Schonlau, and W. Welch.  Efficient global optimization of expensive black-box functions.  _Journal of Global Optimization_ , 13:455–492, 12 1998\.  doi: 10.1023/A:1008306431147.
  * Kingma and Ba [2015] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations_ , 2015.
  * Kirsch et al. [2019] A. Kirsch, J. Van Amersfoort, and Y. Gal.  Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning.  _Advances in neural information processing systems_ , 32:7026–7037, 2019.
  * Kuleshov et al. [2018] V. Kuleshov, N. Fenner, and S. Ermon.  Accurate uncertainties for deep learning using calibrated regression.  In _International Conference on Machine Learning_ , pages 2796–2804. PMLR, 2018.
  * Lorenz et al. [2011] R. Lorenz, S. H. Bernhart, C. H. Zu Siederdissen, H. Tafer, C. Flamm, P. F. Stadler, and I. L. Hofacker.  ViennaRNA package 2.0.  _Algorithms for molecular biology_ , 6(1):26, 2011\.
  * Mak and Pichika [2019] K. K. Mak and M. R. Pichika.  Artificial intelligence in drug development: present status and future prospects.  _Drug Discovery Today_ , 24(3):773–780, 2019\.  ISSN 18785832.  doi: 10.1016/j.drudis.2018.11.014.  URL https://doi.org/10.1016/j.drudis.2018.11.014.
  * Močkus [1975] J. Močkus.  On bayesian methods for seeking the extremum.  In G. I. Marchuk, editor, _Optimization Techniques IFIP Technical Conference Novosibirsk, July 1–7, 1974_ , pages 400–404, Berlin, Heidelberg, 1975. Springer Berlin Heidelberg.  ISBN 978-3-540-37497-8.
  * Müller et al. [2018] A. T. Müller, J. A. Hiss, and G. Schneider.  Recurrent Neural Network Model for Constructive Peptide Design.  _Journal of Chemical Information and Modeling_ , 58(2):472–479, 2018.  ISSN 15205142.  doi: 10.1021/acs.jcim.7b00414.
  * Rasmussen and Williams [2006] C. E. Rasmussen and C. K. I. Williams.  _Gaussian processes for machine learning_.  Adaptive computation and machine learning. MIT Press, 2006.
  * Real et al. [2019] E. Real, A. Aggarwal, Y. Huang, and Q. V. Le.  Regularized evolution for image classifier architecture search.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 4780–4789, 2019.
  * Romero et al. [2013] P. A. Romero, A. Krause, and F. H. Arnold.  Navigating the protein fitness landscape with gaussian processes.  _Proceedings of the National Academy of Sciences_ , 110(3):E193–E201, 2013.  doi: 10.1073/pnas.1215251110.  URL https://www.pnas.org/content/110/3/E193.
  * Schmidhuber [1987] J. Schmidhuber.  Evolutionary principles in self-referential learning. on learning now to learn: The meta-meta-meta…-hook.  Diploma thesis, Technische Universitat Munchen, Germany, 14 1987.
  * Schuchardt et al. [2019] J. Schuchardt, V. Golkov, and D. Cremers.  Learning to evolve.  _arXiv preprint arXiv:1905.03389_ , 2019.
  * Sinai et al. [2020] S. Sinai, R. Wang, A. Whatley, S. Slocum, E. Locane, and E. D. Kelsic.  Adalead: A simple and robust adaptive greedy search algorithm for sequence design.  _arXiv preprint arXiv:2010.02141_ , 2020.
  * Srinivas et al. [2009] N. Srinivas, A. Krause, S. M. Kakade, and M. Seeger.  Gaussian process optimization in the bandit setting: No regret and experimental design.  _arXiv preprint arXiv:0912.3995_ , 2009.
  * Swersky et al. [2020] K. Swersky, Y. Rubanova, D. Dohan, and K. Murphy.  Amortized bayesian optimization over discrete spaces.  In J. Peters and D. Sontag, editors, _Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI)_ , volume 124 of _Proceedings of Machine Learning Research_ , pages 769–778. PMLR, 03–06 Aug 2020.  URL https://proceedings.mlr.press/v124/swersky20a.html.
  * Thompson [1933] W. R. Thompson.  On the likelihood that one unknown probability exceeds another in view of the evidence of two samples.  _Biometrika_ , 25(3/4):285–294, 1933.  ISSN 00063444.  URL http://www.jstor.org/stable/2332286.
  * Van Oort et al. [2021] C. M. Van Oort, J. B. Ferrell, J. M. Remington, S. Wshah, and J. Li.  AMPGAN v2: Machine Learning-Guided Design of Antimicrobial Peptides.  _Journal of Chemical Information and Modeling_ , 61(5):2198–2207, 2021.  ISSN 15205142.  doi: 10.1021/acs.jcim.0c01441.
  * Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , pages 6000–6010, 2017.
  * Wang et al. [2021] C. Wang, S. Garlick, and M. Zloh.  Deep learning for novel antimicrobial peptide design.  _Biomolecules_ , 11:471, 03 2021.  doi: 10.3390/biom11030471.
  * Wang et al. [2019] Y. Wang, H. Wang, L. Wei, S. Li, L. Liu, and X. Wang.  Synthetic promoter design in escherichia coli based on generative adversarial network.  _bioRxiv_ , 2019.  doi: 10.1101/563775.  URL https://www.biorxiv.org/content/early/2019/04/25/563775.
  * Wang et al. [2016] Z. Wang, F. Hutter, M. Zoghi, D. Matheson, and N. de Feitas.  Bayesian optimization in a billion dimensions via random embeddings.  _Journal of Artificial Intelligence Research_ , 55:361–387, 2016.
  * Yang et al. [2019] K. K. Yang, Z. Wu, and F. H. Arnold.  Machine-learning-guided directed evolution for protein engineering.  _Nature Methods_ , 16(8):687–694, 2019.  ISSN 15487105.  doi: 10.1038/s41592-019-0496-6.  URL http://dx.doi.org/10.1038/s41592-019-0496-6.

## Appendix

Figure 4: Comparison of the Uncertainty Calibration across different rounds.
We can see that the ensemble of CNNs are better calibrated than the other
surrogate models across the vast majority of the rounds.