# Latent CLAP Loss for Better Foley Sound Synthesis

Tornike Karchkhadze† University of California San Diego
San Diego, USA
<EMAIL_ADDRESS>Hassan Salami Kavaki† The Graduate Center, CUNY
New York, USA
<EMAIL_ADDRESS>Mohammad Rasool Izadi  Bose Corp.
Framingham, USA
<EMAIL_ADDRESS>Bryce Irvin  Mikolaj Kegler  Bose Corp.
Framingham, USA
<EMAIL_ADDRESS>Bose Corp.
Framingham, USA
<EMAIL_ADDRESS>Ari Hertz  Bose Corp.
Framingham, USA
<EMAIL_ADDRESS>Shuo Zhang  Bose Corp.
Framingham, USA
<EMAIL_ADDRESS>Marko Stamenovic  Bose Corp.
Framingham, USA
<EMAIL_ADDRESS>
###### Abstract

Foley sound generation, the art of creating audio for multimedia, has recently
seen notable advancements through text-conditioned latent diffusion models.
These systems use multimodal text-audio representation models, such as
Contrastive Language-Audio Pretraining (CLAP), whose objective is to map
corresponding audio and text prompts into a joint embedding space. AudioLDM, a
text-to-audio model, was the winner of the DCASE2023 task 7 Foley sound
synthesis challenge. The winning system fine-tuned the model for specific
audio classes and applied a post-filtering method using CLAP similarity scores
between output audio and input text at inference time, requiring the
generation of extra samples, thus reducing data generation efficiency. We
introduce a new loss term to enhance Foley sound generation in AudioLDM
without post-filtering. This loss term uses a new module based on the CLAP
model—Latent CLAP encoder—to align the latent diffusion output with real audio
in a shared CLAP embedding space. Our experiments demonstrate that our method
effectively reduces the Fréchet Audio Distance (FAD) score of the generated
audio and eliminates the need for post-filtering, thus enhancing generation
efficiency.

###### Index Terms:

Foley sound synthesis, latent diffusion, CLAP

††footnotetext: Authors with equal contribution. This work was performed
during an internship at Bose Corporation.

##  I Introduction

In recent years, deep learning models have made tremendous advances in the
domain of sound generation [1, 2, 3, 4, 5]. In light of these advancements,
user-controlled neural audio synthesis has the potential to revolutionize
numerous domains, including Foley sound synthesis [6, 7], the art of creating
or reproducing everyday sound effects that are added to film, video, and other
media in post-production. This innovation could impact a variety of fields
that utilize Foley sound by automating the labor-intensive manual sound
design.

Diffusion models [8] have gained significant attention for their ability to
learn complex distributions, which makes them well-suited for data types such
as audio. By integrating the latent diffusion model (LDM) [9] for generation
and the contrastive language-audio pretraining (CLAP) [10] model as a text
encoder, AudioLDM [1] stands as one of the current state-of-the-art text-to-
audio generation systems, demonstrating strong performance in Foley sound
generation tasks. Models built upon AudioLDM [11, 12] emerged as the winner of
the DCASE2023 Task 7 Foley Sound Synthesis Challenge [13]. The winning model
used AudioLDM, fine-tuned on a set of sound classes, alongside a post-
filtering technique that generated superfluous samples and sub-selected the
best among them using a heuristic. Although post-filtering improved the output
quality, it significantly compromised the efficiency of generation.

We propose augmenting the AudioLDM framework by introducing a Latent CLAP
loss, which is incorporated through a Latent CLAP encoder module. This module
is integrated into the fine-tuning process with an objective to increase the
similarity in the CLAP embedding space between the generated and real audio at
training time, thus enhancing the quality of the generated samples.

Our experiments show that the proposed method not only improves Fréchet Audio
Distance (FAD) [14], an objective generative audio metric, but also alleviates
the need for post-filtering, thereby enhancing the quality of generated sounds
while greatly improving the efficiency of the audio generation process.
Additionally, a perceptual study involving subjective evaluations from a
cohort of human listeners confirm our objective metrics, indicating strong
preferences towards the data generated using the proposed model, as compared
to AudioLDM fine-tuned on the DCASE2023 Task 7 data.

##  II Background

###  II-A AudioLDM System Overview

AudioLDM comprises a text-audio encoder, a generator, an autoencoder, and a
vocoder. CLAP serves as the text-audio encoder, mapping audio and text into a
shared embedding space. This is succeeded by an LDM that serves as the main
generator of the system. To train on limited computational resources while
retaining generation quality, the LDM operates within the latent space of a
Variational Autoencoder (VAE) [15]. This VAE is pretrained to compress and
reconstruct Mel-spectrograms. Finally, a HiFi-GAN [16] vocoder is used to
synthesize the audio output from the generated Mel-spectrograms.

Each component—the CLAP encoder, VAE, and HiFi-GAN vocoder—is pretrained
independently and frozen within the overall system, for the subsequent
training of the LDM. AudioLDM overcomes a primary challenge in the field of
audio machine learning, the need for vast, high-quality audio-to-text
datasets, by directly extracting embeddings from audio using the CLAP model.
Additionally, AudioLDM utilizes classifier-free guidance (CFG) [17], which
allows for the adjustment of adherence to conditioning during inference
through CFG weighting.

###  II-B Foley Sound Synthesis Task and Baseline Models

This study follows the challenge set out by DCASE2023, specifically Task 7 on
Foley Sound Synthesis [13]. The challenge was set up as a category-to-sound
generation with seven distinct categories: dog bark, footstep, gunshot,
keyboard, moving motor vehicle, rain, and sneeze/cough. The challenge
organizers supplied a hand-curated development dataset containing the above
mentioned 7 categories.

For our baseline comparisons, we select the winning model from the DCASE2023
Challenge Task 7 on Foley Sound Synthesis [11], and its subsequent
modification [12]. Both baselines employ a pretrained AudioLDM, fine-tuned on
the text-to-audio pairs from the challenge dataset. In this process, class
labels are transformed into descriptive sentences, which are then fed into the
text branch of the CLAP model. To extract better semantic embeddings for each
class, [12] used an additional tuning layer between the CLAP and LDM modules.
Both models applied a post-filtering technique to the generated audio samples.
The post-filtering process involved calculating the cosine similarity between
CLAP embeddings of the generated audio and the target text. Audio samples
achieving similarity scores beyond a predefined threshold were then selected
for the final output pool. In both studies, the post-filtering thresholds, as
well as the CFG weights (see II-A) were determined empirically for each sound
class.

##  III Method

Our system, depicted in Fig. 1, is a modification of AudioLDM [2] with the
addition of a novel Latent CLAP encoder for enhanced loss computation.

Figure 1: Overview of the overall system for Foley audio generation.

###  III-A CLAP Encoder

The CLAP model comprises two parallel, jointly-trained transformer-based
encoders: one for text and one for audio. The text encoder in the CLAP model
converts the text input $x_{\text{text}}$ into an embedding vector denoted by
$E_{\text{text}}$. Similarly, the audio encoder converts the audio input
$x_{\text{audio}}$ into $E_{\text{audio}}$ that matches the dimensions of
$E_{\text{text}}$. These two encoders are trained with a contrastive loss,
resulting in an aligned latent space of the same dimensionality. A comparative
analysis [10] shows that the best architecture for the CLAP audio encoder is
HTSAT [18], and the best architecture for the text encoder is the RoBERTa [19]
language model. In our work, we utilize the publicly released model
†††https://github.com/LAION-AI/CLAP from [10] pretrained on AudioCaps [20],
Clotho [21], LAION-Audio-630K [10], and Audioset [22] datasets.

Similarly to the baseline model [12], we add a fully connected tuning layer
after the text CLAP encoder (see Fig. 1). We evaluate the system with and
without this layer.

###  III-B LDM Generator

The LDM generator, operating in latent space, when conditioned on the text
embedding $E_{\text{text}}$, produces a compressed representation of Mel-
spectrogram $s_{n}\in\mathbb{R}^{C\times\frac{T}{r}\times\frac{F}{r}}$, where
$r$ shows the compression ratio, $C$ denotes the number of channels, $T$ and
$F$ represent the time and frequency dimensions of Mel-spectrogram
respectively. The variable $n\in[1,\ldots,N]$ in $s_{n}$ represents the step
number in the diffusion model’s forward or reverse steps. During the forward
phase of the diffusion process, the original clean compressed representation
of the Mel-spectrogram $s_{0}$ is gradually turned into an isotropic Gaussian
noise $s_{N}$ with distribution $\mathcal{N}(0,I)$ over $N$ steps, by
incrementally adding Gaussian noise $\epsilon$ at each step. Conversely, in
the reverse phase of the diffusion process, the clean representation $s_{0}$
is incrementally reconstructed through a denoising process, where at each time
step $n$, the model predicts the injected noise $\epsilon$ to reconstruct
$s_{n-1}$ from $s_{n}$.

The training objective of the model is to minimize the mean square error (MSE)
between the predicted noise $\epsilon_{\theta}$ and the actual Gaussian noise
$\epsilon$, following the classic Denoising Diffusion Probabilistic Model’s
(DDPM) [8] loss approach:

$\displaystyle{L_{DDPM}}({\theta})=\mathbb{E}_{s_{0},\epsilon,n}\|\epsilon-\epsilon_{\theta}(s_{n},n,E_{\text{text}})\|^{2},$
(1)

where $\theta$ corresponds to the parameters of the LDM model.

###  III-C Variational Autoencoder and HiFi-GAN Vocoder

We employ a VAE model [7] to bridge the Mel-spectrogram domain with the latent
space used by the LDM. The VAE encoder projects Mel-spectrograms into the
latent space and the VAE decoder reconstructs the Mel-spectrograms from
compressed representations. During training, the input audio
$x_{\text{audio}}$ is first converted into a Mel-spectrogram $x_{\text{Mel}}$
as shown in Fig. 1. Then, the VAE encoder maps $x_{\text{Mel}}$ to a latent
compressed representation $x_{\text{latent}}$
$\in\mathbb{R}^{C\times\frac{T}{r}\times\frac{F}{r}}$, which is then compared
with the LDM’s output $s_{n}$. During inference, the VAE decoder transforms
the denoised compressed representation output of LDM, $s_{0}$, into the Mel-
spectrogram $\hat{x}_{\text{Mel}}$. The HiFi-GAN vocoder subsequently maps
$\hat{x}_{\text{Mel}}$ into an audio waveform $\hat{x}_{\text{audio}}$.

###  III-D Latent CLAP Encoder

To improve the model performance without the need for post-filtering, we
attempted to include CLAP similarity score into the loss function, aiming to
maximize similarity during training. Unfortunately, this end-to-end training
setup required extensive GPU resources and proved to be ineffective. To
overcome this limitation, we developed a new module that maps the VAE latent
space, in which LDM operates, to the CLAP embedding space. This module, which
we refer to as the Latent CLAP encoder, utilizes the Pretrained Audio Neural
Network (PANN) architecture [23], a popular approach for tasks such as audio
classification. Concretely, we use the ‘PANN-10’ design, with modified input
and output layers which process compressed audio representations –
$x_{\text{latent}}$ during its own training and $s_{n}$ during fine-tuning of
the AudioLDM system – and predicts the corresponding CLAP embedding
$E_{\text{latent},n}$.

As illustrated in Fig. 2, we trained the Latent CLAP encoder $\phi$ in
conjunction with the frozen audio branch of the pretrained CLAP model using
MSE objective to match the embedding spaces of the two encoders:

$L(\phi)=\mathbb{E}_{x_{\text{latent}}}\|E_{\text{audio}}-E_{\text{latent}}\|^{2}.\vspace{-1pt}$
(2)

The audio inputs $x_{\text{audio}}$ are first transformed into Mel-
spectrograms $x_{\text{Mel}}$ and then compressed into a latent representation
$x_{\text{latent}}$ by a pretrained, frozen VAE encoder before being fed into
Latent CLAP. The dataset employed for training was WaveCaps [24], a collection
of around 400K Chat-GPT labeled audio clips. Apart from the labels that were
not used in our training process, the WaveCaps compilation includes popular
audio datasets such as Audioset, BBC, Freesound, and Soundbible, sharing
substantial commonalities with the data that was used to train the original
CLAP model. Our rationale for using a similar dataset was to align the
performance of the new Latent CLAP encoder as closely as possible with the
existing branches of CLAP.

TABLE I: Evaluation: FAD Scores (lower is better, $\downarrow$).

System | Dog Bark | Footstep | Gun Shot | Keyboard | Motor Vehicle | Rain | Sneeze/Cough | Average | Std.
---|---|---|---|---|---|---|---|---|---
Real Audio | 3.19 | 5.51 | 5.02 | 3.42 | 6.24 | 4.46 | 4.43 | 4.61 | 1.09
LDM [11] | 5.69 | 10.24 | 7.25 | 5.11 | 19.77 | 9.75 | 3.38 | 8.74 | 5.45
LDM+Tuning [12] | 6.58 | 10.31 | 5.07 | 4.7 | 11.57 | 9.59 | 4.5 | 7.47 | 2.95
LDM+Latent | 4.54 | 7.08 | 5.6 | 5.31 | 16.14 | 7.84 | 3.16 | 7.1 | 4.28
LDM+Tuning+Latent | 5.91 | 9.9 | 4.51 | 4.46 | 7.3 | 8.99 | 4.55 | 6.52 | 2.26
LDM+filter [11] | 4.11 | 9.47 | 5.2 | 4.48 | 23.45 | 9.56 | 3.4 | 8.52 | 7.05
LDM+Tuning+filter [12] | 5.64 | 9.91 | 4.56 | 4.93 | 12.54 | 8.86 | 4.33 | 7.25 | 3.2
LDM+Latent+filter | 4.46 | 8.24 | 5.27 | 4.71 | 10.91 | 8.83 | 2.87 | 6.47 | 2.88
LDM+Tuning+Latent+filter | 6.24 | 9.01 | 4.4 | 4.86 | 10.7 | 9.94 | 4.24 | 7.05 | 2.76

As shown in Fig. 1, we introduce the Latent CLAP loss to our Foley generation
system at the fine-tuning stage, aiming to enhance the similarity between
LDM’s latent audio output and the actual ground truth audio, facilitating
improvements in sample quality. The objective of the loss function in the
training process is minimizing the MSE between the CLAP embedding vector
outputs of real audio and generated latent representations, as follows:

$\displaystyle{L_{LCLAP}}({\theta})$
$\displaystyle=\mathbb{E}_{n}\Bigl{[}w(n)\cdot\|E_{\text{audio}}-E_{\text{latent},n}\|^{2}\Bigl{]},$
(3)

where the embedding vectors $E_{\text{audio}}$ and $E_{\text{latent},n}$
represent the outputs of the CLAP audio branch and latent CLAP encoder,
respectively. The audio branch processes the incoming audio
$x_{\text{audio}}$, while the Latent CLAP encoder operates on the noisy,
compressed representation $s_{n}$, which is the output of the LDM model at
step $n$. The $w(n)$ in the equation is a $n$-dependent weighting function
that defines the contribution of higher $n$ (noisier) and lower $n$ (cleaner)
samples in the loss calculation, addressing the observed ineffectiveness of
the method when treating noisy and less noisy samples uniformly.

The Foley synthesis model’s total loss combines the conventional DDPM loss
with the newly introduced Latent CLAP loss with weight $\lambda$ as follows:

$\displaystyle L({\theta})={L_{DDPM}}({\theta})+\lambda{L_{LCLAP}}({\theta})$
(4)

Figure 2:  Latent CLAP encoder pretraining.

##  IV Experimental Setup

###  IV-A Data and Metrics

Our experiments utilized the DCASE2023 Task 7 challenge dataset, comprising
6.1 hours of audio, distributed across seven given categories, with each
containing 600 to 800 clips of 4 seconds long. We pre-processed the data by
resampling it from 22 kHz to 16 kHz and employing a repeating method to extend
the clips from 4 to 10.24 seconds in length. Using a window length of 1024 and
a hop size of 160 samples, the audio clips were transformed into Mel-
spectrograms with dimensions $F\times T=64\times 1024$, representing Mel-
frequency bins and time frames respectively. Text prompts were transformed
from class labels to sentences using the phrase “This is a sound of <label>.”
We adopted the FAD metric for evaluation, utilizing the means and standard
deviations of VGGish embeddings provided by the challenge organizers as the
reference distribution for the evaluation set.

###  IV-B Parameter Settings

As a starting point for our fine-tuning process, we took a pretrained
audioldm-m-full variant of the AudioLDM model from publicly released
checkpoints†††github.com/haoheliu/AudioLDM. When fine-tuning with the
DCASE2023 development dataset, all parts of the model were frozen except for
the parameters of the LDM and the tuning layer as demonstrated in Fig. 1. Our
setup uses a VAE with a compression ratio $r=4$, encoding Mel-spectrograms of
size $F\times T=64\times 1024$ into latent vectors of size
$C\times\frac{F}{r}\times\frac{T}{r}=8\times 16\times 256$ in channel,
frequency and time dimensions, respectively. The embedding sizes for audio,
text, and latent representations from the CLAP encoder are 512. During fine-
tuning, we use the Adam optimizer with a learning rate of $3\times 10^{-6}$
for up to 500 epochs. The number of LDM denoising steps is set to $N=1000$
during training and $N=200$ during inference. To evaluate model performance,
we generate 100 clips per class and compute their FAD with respect to the
evaluation set.

In our experiments, we standardized the CFG weight (see II-A) to 2.0 for all
categories, unlike the baselines [11, 12] in which it varied between 1.5 and
2.5. Also in contrast to the baselines, which used varying AudioLDM models for
different classes, we only used audioldm-m-full, with a goal of analyzing
Latent CLAP’s impact rather than minimizing FAD scores. For post-filtering, we
used the same CLAP similarity threshold as in the baselines (see II-B). Namely
0.2 for all classes, except for keyboard (0.15) and motor vehicle (0.75).

For Latent CLAP loss in (3), we set the denoising-step-dependent weighting
function as $w(n)=10^{-\frac{n}{200}}$, where $n$ is the denoising step in the
latent diffusion process. The function is an exponent decay attenuating
contributions of overly noisy samples in the training above $N=200$ step,
which is also the number of steps used at the inference stage. For the
weighting coefficient $\lambda$ balancing the DDPM and Latent CLAP loss
contributions in (4), we found that a setting of $\lambda=2000$ provided the
best results for the model using latent CLAP loss, whereas $\lambda=1000$
yielded the best results for the model utilizing both latent CLAP loss and the
tuning layer.

##  V Results

###  V-A Objective Metrics

We evaluated our models, which incorporate Latent CLAP loss, both with and
without the inclusion of the tuning layer, against baselines [12, 11]. The
performance of our proposed method on the DCASE2023 Task 7 validation set is
reported in Table I. As shown in rows 2-5, models using Latent CLAP loss in
both settings outperform the baselines in terms of FAD score (1.64 and 0.95
avg. FAD improvement, respectively), indicating substantially better
correspondence of the generated set with the evaluation set in the matter of
objective metrics.

We observed that integration of Latent CLAP loss showed minimal effect on the
similarity scores between generated audio and text embeddings
($.65\rightarrow.66$ with the tuning layer and $.30\rightarrow.31$ without the
tuning layer). Our results demonstrate that the addition of latent CLAP
enhances the output audio quality without altering the alignment between the
text prompts and generated audio. This outcome is consistent with our emphasis
on improving audio fidelity, as opposed to increasing the similarity between
text and audio, which is the underlying assumption of the baseline’s use of
post-filtering.

Furthermore, we observed that audio and text embedding similarity score-
dependent post-filtering is ineffective for our model (Table I, lower
section). In contrast to the baseline models, where post-filtering has a
noticeable effect, our method, particularly combining the tuning layer and
Latent CLAP loss, shows that post-filtering not only becomes superfluous and
can also slightly increase the FAD score.

The elimination of the post-filtering stage provided significant data
generation efficiency improvements. Here, using NVIDIA GeForce RTX 3090
generating 100 10-seconds-long clips, the elimination of post-filtering
increases the generation speed by an order of magnitude.

Figure 3: Impact of Latent CLAP Loss Weight on FAD Score. Note: these FAD
scores were obtained for the models trained for only 100 epochs, while Table I
presents the evaluation of the models trained for 500 epochs. This study aims
to explore the relative performance changes as a functions of $\lambda$.

###  V-B Latent CLAP Loss Weight Analysis

To test the robustness of Latent CLAP loss to hyperparameter selections, we
varied the weighting parameter $\lambda$ from (4) and calculated the average
FAD scores across all classes, comparing models with and without the tuning
layer trained over 100 epochs (note that the models in Table I were trained
for 500 epochs). Results presented in Fig. 3 illustrate that the increase in
the $\lambda$ gradually improves the avg. FAD score until the optimal point is
reached, beyond which the performance starts to decrease. This indicates that
the latent CLAP loss is consistently contributing to the overall performance
without unexpected $\lambda$-dependent effects.

###  V-C Subjective Evaluation

Human perception of audio quality and class relevance can be highly subjective
and nonlinear, often correlating poorly with objective metrics [25]. Human
ratings are still considered the gold standard in audio quality evaluation.
Thus to better ground our work, the audio generated by the proposed method was
compared to those from the baseline model involving the tuning layer (LDM +
Tuning) [12] and ground truth audio samples in a series of 7 online surveys.
Each survey focused on a specific sound class, included 15 audio samples from
each of the three models and was completed by 40 listeners. Respondents rated
audio clip for their quality and class relevance on a scale from 0 to 10. Our
survey’s audio samples and analyses are publicly available†††
https://github.com/karchkha/Latent-CLAP-subjective-evaluation.

As reported in Table II, the average subjective ratings reveal that the Latent
CLAP loss model outscored the baseline model and ground truth samples in both
audio quality and category fitness. A mixed-design ANOVA showed that the main
effect of model was significant for both audio quality ($F$(1.9,509.6) =
115.3, $p$ $<$ .001, $\eta_{p}^{2}$ = .3) and category fit ($F$(1.9,522) =
156.6, $p$ $<$ .001, $\eta_{p}^{2}$ = .36), while post-hoc paired t-tests with
Bonferroni corrections further support the finding that the proposed model was
rated significantly higher than both the baseline model and ground truth
samples across both items ($p$ $<$ .001 for all comparisons).

The lower ratings for the ground truth samples is unexpected and could be
attributed to the fact that real-world recordings often contain extraneous
noises or recording artifacts. Our generated audio presents cleaner sounds
with features that are more distinctly aligned with the target class, offering
a potentially clearer representation of the intended sound event, which may
contribute to higher scores in human evaluations.

TABLE II: Subjective evaluation of Audio Quality and class relevance Model | Audio quality | Class relevance
---|---|---
Ground Truth | 5.55 | 6.03
LDM + Tuning [12] | 6.00 | 6.55
LDM + Tuning + Latent | 6.40 | 7.03

##  VI Conclusion

This work introduces an improved AudioLDM model for Foley sound generation
with the addition of the Latent CLAP loss. The concept of the proposed method
is to integrate the post-filtering process within the model to enhance audio
quality while maintaining inference efficiency. As indicated by both objective
(FAD score) and subjective evaluations (listening study), the proposed model
generates higher quality audio in comparison to baselines. This approach is
not limited to the closed-set Foley generation problem and can be applied to
general-purpose text-to-audio models without any further modifications. Future
research will explore these applications further, potentially enhancing the
fidelity of generative audio models.

## References

  * [1] H. Liu, Z. Chen, Y. Yuan, X. Mei, X. Liu, D. Mandic, W. Wang, and M. D. Plumbley, “AudioLDM: Text-to-audio generation with latent diffusion models,” in _ICML_ , vol. 202, 2023, pp. 21 450–21 474.
  * [2] H. Liu, Q. Tian, Y. Yuan, X. Liu, X. Mei, Q. Kong, Y. Wang, W. Wang, Y. Wang, and M. D. Plumbley, “Audioldm 2: Learning holistic audio generation with self-supervised pretraining,” arXiv:2308.05734, 2023.
  * [3] D. Yang, J. Yu, H. Wang, W. Wang, C. Weng, Y. Zou, and D. Yu, “Diffsound: Discrete diffusion model for text-to-sound generation,” _IEEE TASLP_ , vol. 31, pp. 1720–1733, 2023.
  * [4] F. Kreuk, G. Synnaeve, A. Polyak, U. Singer, A. Défossez, J. Copet, D. Parikh, Y. Taigman, and Y. Adi, “Audiogen: Textually guided audio generation,” in _ICLR_ , 2023.
  * [5] R. Huang, J. Huang, D. Yang, Y. Ren, L. Liu, M. Li, Z. Ye, J. Liu, X. Yin, and Z. Zhao, “Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models,” arXiv:2301.12661, 2023.
  * [6] L. Chen, S. Srivastava, Z. Duan, and C. Xu, “Deep cross-modal audio-visual generation,” in _Proc. of the on Thematic Workshops of ACM Multimedia_ , 2017, pp. 349–357.
  * [7] V. Iashin and E. Rahtu, “Taming visually guided sound generation,” in _British Machine Vision Conference (BMVC)_ , 2021.
  * [8] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” in _NeurIPS_ , vol. 33, 2020, pp. 6840–6851.
  * [9] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 10 684–10 695.
  * [10] Y. Wu, K. Chen, T. Zhang, Y. Hui, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” in _Proc. ICASSP_ , 2023, pp. 1–5.
  * [11] Y. Yuan, H. Liu, X. Liu, X. Kang, P. Wu, M. D. Plumbley, and W. Wang, “Text-driven foley sound generation with latent diffusion model,” in _DCASE-workshop_ , 2023.
  * [12] Y. Yuan, H. Liu, X. Liu, X. Kang, M. D. Plumbley, and W. Wang, “Latent diffusion model based foley sound generation system for dcase challenge 2023 task 7,” arXiv:2305.15905, 2023.
  * [13] K. Choi, J. Im, L. Heller, B. McFee, K. Imoto, Y. Okamoto, M. Lagrange, and S. Takamichi, “Foley sound synthesis at the dcase 2023 challenge,” _arXiv: 2304.12521_ , 2023.
  * [14] K. Kilgour, M. Zuluaga, D. Roblek, and M. Sharifi, “Fréchet audio distance: A metric for evaluating music enhancement algorithms,” arXiv:1812.08466, 2019.
  * [15] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” arXiv:1312.6114, 2013.
  * [16] J. Kong, J. Kim, and J. Bae, “Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,” in _NeurIPS_ , vol. 33, 2020, pp. 17 022–17 033.
  * [17] J. Ho and T. Salimans, “Classifier-free diffusion guidance,” in _NeurIPS Workshop on Deep Generative Models and Downstream Applications_ , 2021.
  * [18] K. Chen, X. Du, B. Zhu, Z. Ma, T. Berg-Kirkpatrick, and S. Dubnov, “HTS-AT: A hierarchical token-semantic audio transformer for sound classification and detection,” in _Proc. ICASSP_ , 2022, pp. 646–650.
  * [19] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized BERT pretraining approach,” in _ICLR_ , 2020.
  * [20] C. D. Kim, B. Kim, H. Lee, and G. Kim, “AudioCaps: Generating captions for audios in the wild,” in _NAACL-HLT_ , 2019, pp. 119–132.
  * [21] K. Drossos, S. Lipping, and T. Virtanen, “Clotho: an audio captioning dataset,” in _Proc. ICASSP_ , 2020, pp. 736–740.
  * [22] J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in _Proc. ICASSP_ , 2017.
  * [23] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, “PANNs: Large-scale pretrained audio neural networks for audio pattern recognition,” _IEEE TASLP_ , vol. 28, pp. 2880–2894, 2019.
  * [24] X. Mei, C. Meng, H. Liu, Q. Kong, T. Ko, C. Zhao, M. D. Plumbley, Y. Zou, and W. Wang, “Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research,” arXiv:2303.17395, 2023.
  * [25] A. Vinay and A. Lerch, “Evaluating generative audio systems and their metrics,” in _Proc. ISMIR_ , 2022, pp. 858–865.


#  GP-ConvCNP: Better Generalization for Convolutional Conditional Neural
Processes on Time Series Data

Jens Petersen Division of Medical Image Computing
German Cancer Research Center
Heidelberg, Germany  Gregor Köhler  Division of Medical Image Computing
German Cancer Research Center
Heidelberg, Germany  David Zimmerer  Division of Medical Image Computing
German Cancer Research Center
Heidelberg, Germany  Fabian Isensee  HIP Applied Computer Vision Lab
Division of Medical Image Computing
German Cancer Research Center  Paul F. Jäger  Interactive Machine Learning
Group
German Cancer Research Center  Klaus H. Maier-Hein  Division of Medical Image
Computing
German Cancer Research Center
Heidelberg, Germany

###### Abstract

Neural Processes (NPs) are a family of conditional generative models that are
able to model a distribution over functions, in a way that allows them to
perform predictions at test time conditioned on a number of context points. A
recent addition to this family, _Convolutional Conditional Neural Processes_
(ConvCNP), have shown remarkable improvement in performance over prior art,
but we find that they sometimes struggle to generalize when applied to time
series data. In particular, they are not robust to distribution shifts and
fail to extrapolate observed patterns into the future. By incorporating a
Gaussian Process into the model, we are able to remedy this and at the same
time improve performance within distribution. As an added benefit, the
Gaussian Process reintroduces the possibility to sample from the model, a key
feature of other members in the NP family.

##  1 Introduction

Neural Processes [Garnelo et al., 2018a, b] have been proposed as a way to
leverage the expressiveness of neural networks to learn a distribution over
functions (often referred to as a _stochastic process_), so that they can
condition their predictions on observations given at test time, a so-called
_context_. But what does it mean to successfully learn such a distribution? We
believe that it should be characterized by the following: 1) accurate
predictions, meaning predictions should be as close as possible to the true
underlying function, 2) good reconstruction of the given observations, 3)
generalization, because we assume that there will be some underlying
generative process from which the distribution originates and which is valid
beyond the finite data we observe. The latter is especially important when
only few context observations are given that could be explained by several
different functions. Follow-up work to Neural Processes has mostly emphasized
the first two aspects, the most prominent of which are _Attentive Neural
Processes_ (ANP) [Kim et al., 2019] and _Convolutional Conditional Neural
Process_ (ConvCNP) [Gordon et al., 2020], each improving upon its predecessor
in terms of both prediction accuracy and reconstruction ability.

We propose a model that addresses all of the above, with a particular focus on
the ability to generalize. By combining ConvCNP with a Gaussian Process, we
achieve a significant improvement in generalization: the model, which we call
GP-ConvCNP, can better extrapolate far from the provided context
observations—meaning into future given past and present observations—and is
more robust to a distribution shift at test time. It further reintroduces the
ability to sample from the model, something that ConvCNP is incapable of,
showing a better sample distribution than both NP and ANP. Finally, we find
that our proposed model often yields a significant improvement in predictive
performance on in-distribution data as well. We focus our evaluation on time
series data, where we see the greatest potential for applications of our
model. In this context, we consider several synthetic datasets as well as real
time series, specifically weather data and predator-prey population dynamics.
We provide a complete implementation111https://github.com/MIC-DKFZ/gpconvcnp,
including data for convenience, to reproduce all experiments in this work.

Figure 1: Our work proposes GP-ConvCNP, an extension of ConvCNP that
reintroduces sampling and improves generalization on time series data. Shown
here are examples for the different synthetic time series and methods
evaluated in this work (mean prediction in blue, samples in red). While the
mean predictions from ConvCNP and GP-ConvCNP look similar—and significantly
better than those from Neural Processes (NP) and Attentive Neural Processes
(ANP)—only GP-ConvCNP combines high quality predictions (a feature of ConvCNP)
with the ability to sample (a feature of NP and ANP). While synthetic data
measures in-distribution performance, we evaluate generalization capabilities
on real data.

##  2 Problem Statement & Methods

In the framework of Neural Processes [Garnelo et al., 2018a, b] we assume that
we are given a set of $N$ observations
$C=\\{(x_{c},y_{c})\\}_{c=1}^{N}\eqqcolon(\mathbf{x_{c}},\mathbf{y_{c}})$,
often called the _context_ , where $x_{c}\in X$ are samples from the input
space $X$ and $y_{c}\in Y$ are samples from the output space $Y$ (commonly
$X=\mathbb{R}^{d_{X}}$ and $Y=\mathbb{R}^{d_{Y}}$, in this work we restrict
ourselves to $X=\mathbb{R}$, because time is scalar). It is assumed that these
observations were generated by some function $f:X\rightarrow Y$, i.e.
$y_{c}=f(x_{c})$, and our goal is to infer $f$ from $C$ so that we may
evaluate it at arbitrary new input locations $x_{t}$. In reality, this will
most likely mean we have collected a number of measurements over time and are
interested in an $f$ that lets us interpolate and extrapolate those
measurements. Note that when we speak of predictive performance, we refer to
both of those cases and not in a temporal sense. The problem is ill-posed
without placing further assumptions on $f$, which is why we typically restrict
it to some family $\mathcal{F}$: polynomials of some order, a combination of
oscillating functions with different frequencies, etc.. However, in many cases
it is undesired or even impossible to manually specify $\mathcal{F}$, so
Neural Processes propose to use neural networks to learn an approximate
representation of $\mathcal{F}$ by observing many examples $f\in\mathcal{F}$.
The latter are typically represented as a context set $C$ (the measurements we
have) and a _target_ set
$T=\\{(x_{t},y_{t})\\}_{t=1}^{M}\eqqcolon(\mathbf{x_{t}},\mathbf{y_{t}})$ (the
measurements we’re interested in). By learning to reconstruct the examples $f$
from a limited number of context points a model should implicitly form a
representation of $\mathcal{F}$, which leads to the following learning
objective:

$\displaystyle\max_{\theta}\sum_{f\in\mathcal{F}}\log
p_{\theta}(\mathbf{y_{t}}|\mathbf{x_{t}},\mathbf{x_{c}},\mathbf{y_{c}})$ (1)
$\displaystyle=$
$\displaystyle\max_{\theta}\sum_{f\in\mathcal{F}}\sum_{t}\log\mathcal{N}(y_{t};g^{\mu}_{\theta}(Z,x_{t}),g^{\sigma}_{\theta}(Z,x_{t}))$
(2)

This objective is common to all approaches we evaluate in our work, and the
second line formalizes the fact that we choose to always model the output as a
diagonal Gaussian, parametrized by mean and variance functions
$g^{\mu}_{\theta},g^{\sigma}_{\theta}$ that seek to maximize the log-
likelihood of the targets $\mathbf{y_{t}}$. The output variance can also be
fixed, but Le et al. [2018] show that a learned output variance is preferable.
$Z$ is a representation of the context $(\mathbf{x_{c}},\mathbf{y_{c}})$, i.e.
there is a mapping $E:X,Y\rightarrow Z$. The implementation of $E$ is where
the members of the Neural Process family differ most, and we visualize them in
Fig. A.1.

###  2.1 (Attentive) Neural Processes

The original Neural Processes [Garnelo et al., 2018b] implement $E$ as a
neural network that _encodes_ individual context observations
$(\mathbf{x_{c}},\mathbf{y_{c}})$ into a finite-dimensional space. These
representations are then averaged to form the global representation $Z$.
Similar to Eq. 2, $Z$ parametrizes a Gaussian distribution, which enables NP
to sample from this latent space and produce diverse predictions; we do not
consider the deterministic NP variant [Garnelo et al., 2018a] in this work. NP
are trained by maximizing a lower bound on Eq. 2, similar to variational
autoencoders. In our NP implementation $E$ and
$(g^{\mu}_{\theta},g^{\sigma}_{\theta})$ are symmetric 6-layer MLP, with a
representation size of 128. Attentive Neural Processes [Kim et al., 2019] are
motivated by the observation that NP poorly reconstruct the provided context,
i.e. the predictions seem to miss the context points, as seen for example in
Fig. 1. To mitigate this effect, ANP augment NP with an additional
deterministic encoder-decoder path. Instead of averaging the individual
representations, a learned attention mechanism combines them, conditioned on a
target point $x_{t}$. So while NP need to compress representations to a single
point in $Z$, ANP don’t have this bottleneck, which likely contributes to
their improved performance. In our ANP implementation, the deterministic path
mirrors the variational path, with both the representation dimension and the
embedding dimension of the attention mechanism being 128. Le et al. [2018]
evaluated several hyperparameter configurations for NP and ANP and our
implementation matches their best performing one.

Table 1: Results for synthetically created data. Test data was generated with the same parameters as the training data, so we’re looking at _in-distribution_ performance. $\uparrow$/$\downarrow$ indicate that higher/lower is better. Errors represent 1 standard deviation over 5 runs with different seeds (standard error of the mean for GPs, because seed influence is negligible), where each run was evaluated with $102\,400\text{\,}\mathrm{(}$$30\,720\text{\,}\mathrm{f}$or $W_{2}$) samples. Bold indicates that the method(s) are significantly better than all non-bold methods, i.e. when the difference is larger than the root sum of squares of the standard deviations. Overall, GP-ConvCNP outperforms the competing approaches, especially in terms of predictive log-likelihood and sample diversity (compared to an oracle) where applicable. In terms of reconstruction error, our method outperforms prior art on three datasets, but is on par with ConvCNP on two of those. Interestingly, the EQ-GP, which is what our model uses as an initial estimate, performs rather poorly in all but the first example. In the first example, where the EQ-GP is already a decent estimate, our approach leverages that information and matches the oracle GP in predictive performance! The reconstruction error and $W_{2}$ of the oracle are zero, so we don’t show them here. The dependence of model performance on the number of context points is visualized in Fig. A.2 for the two GP examples.  |  | Matern-5/2 GP | Weakly Per. GP | Fourier Series | Step Functions
---|---|---|---|---|---
Predictive LL$\uparrow$ | GP (EQ) | $1.031\pm 0.075$ | $-8.034\pm 2.260$ | $-0.241\pm 0.752$ | $-2\text{\times}{10}^{17}$
GP (Oracle) | $1.933\pm 0.095$ | $1.876\pm 0.026$ |  |
| NP | $-0.496\pm 0.027$ | $-1.161\pm 0.007$ | $-1.743\pm 0.020$ | $-3.287\pm 0.491$
| ANP | $0.723\pm 0.046$ | $-1.047\pm 0.008$ | $-0.976\pm 0.028$ | $-65.141\pm 60.979$
| ConvCNP | $1.710\pm 0.038$ | $-0.153\pm 0.033$ | $0.372\pm 0.065$ | $-0.522\pm 0.163$
| GP-ConvCNP | $1.930\pm 0.031$ | $-0.090\pm 0.021$ | $1.632\pm 0.079$ | $-0.532\pm 0.044$
Recon. Error$\downarrow$ | GP (EQ) | $0.001\pm 0.001$ | $0.028\pm 0.001$ | $0.004\pm 0.001$ | $0.097\pm 0.001$
| NP | $0.027\pm 0.001$ | $0.500\pm 0.003$ | $0.845\pm 0.074$ | $0.292\pm 0.010$
| ANP | $0.008\pm 0.002$ | $0.491\pm 0.004$ | $0.181\pm 0.018$ | $0.284\pm 0.013$
| ConvCNP | $0.025\pm 0.020$ | $0.109\pm 0.077$ | $0.042\pm 0.027$ | $0.121\pm 0.017$
| GP-ConvCNP | $0.013\pm 0.002$ | $0.061\pm 0.007$ | $0.040\pm 0.023$ | $0.116\pm 0.017$
$W_{2}\downarrow$ | GP (EQ) | $4.294\pm 0.007$ | $4.521\pm 0.003$ |  |
| NP | $1.836\pm 0.021$ | $2.745\pm 0.004$ |  |
| ANP | $1.369\pm 0.048$ | $2.708\pm 0.002$ |  |
| ConvCNP |  |  |  |
| GP-ConvCNP | $0.987\pm 0.086$ | $1.800\pm 0.045$ |  |

###  2.2 From ConvCNP to GP-ConvCNP

With the goal of enabling translation equivariance (i.e. independence of the
value range of $\mathbf{x_{c}}$ and $\mathbf{x_{t}}$) in Neural Processes, the
authors of Convolutional Conditional Neural Processes (ConvCNP) [Gordon et
al., 2020] approach their work from the perspective of _learning on sets_
[Zaheer et al., 2017]. While NP and ANP map the context set into a finite-
dimensional representation, ConvCNP map it into an infinite-dimensional
function space. The authors show that in this scenario translation
equivariance (as well as permutation invariance) can only be achieved if the
mapping $E$ can be represented in the form

$\displaystyle E(\mathbf{x_{c}},\mathbf{y_{c}})$
$\displaystyle=\rho(E^{\prime}(\mathbf{x_{c}},\mathbf{y_{c}}))$ (3)
$\displaystyle E^{\prime}(\mathbf{x_{c}},\mathbf{y_{c}})$
$\displaystyle=\sum_{c}\phi(y_{c})\psi(\cdot-x_{c})$ (4)

where $\phi:Y\rightarrow\mathbb{R}^{2}$ and $\psi:X\rightarrow\mathbb{R}$, so
that $E^{\prime}$ defines a function and $\rho$ operates in function space and
must be translation equivariant. The similar naming of $E,E^{\prime}$ is
deliberate, because herein lies a key difference to NP (and also ANP): NP
learn a powerful mapping (i.e. neural network) from the context to a
representation and then another one from this representation to the output
space, whereas ConvCNP employs a very simple mapping to another representation
(to function space, because $\phi$ and $\psi$ are defined with kernels, see
below). A powerful approximator is then learned that operates _within_ this
representation space, as $\rho$ is a CNN operating on a discretization of
$E^{\prime}$. The mapping back to output space is again a simple one, usually
also $\psi$ combined with a linear map. In this sense, both $E$ and
$E^{\prime}$ can be thought of as representations when we make the connection
to NP. See also Fig. A.1 for a visualization of these differences. In Gordon
et al. [2020], $\psi$ is chosen to be a simple Gaussian kernel, and $\phi$
such that the resulting $E^{\prime}$ has two components:

$E^{\prime}(\mathbf{x_{c}},\mathbf{y_{c}})=\left(\sum_{c}k(\cdot,x_{c})\;,\;\sum_{c}\frac{y_{c}k(\cdot,x_{c})}{\sum_{c^{\prime}}k(\cdot,x_{c^{\prime}})}\right)$
(5)

which is the combination of a kernel density estimator and a Nadaraya-Watson
estimator. This estimate is discretized on a suitable grid and a CNN $\rho$ is
applied, the result of which is again turned into a continuous function by
convolving with the (Gaussian) kernel $\psi$. We use the official
implementation222https://github.com/cambridge-mlg/convcnp in our experiments.
Note that $k$ in Eq. 5 is the same as $\psi$ in the implementation.

In this work, we propose GP-ConvCNP, a model that replaces the deterministic
kernel density estimate $E^{\prime}$ in ConvCNP with a Gaussian Process
posterior [Rasmussen and Williams, 2006]. Gaussian Processes (GP) are a
popular choice for time series analysis [Roberts et al., 2013], but typically
require a lot of prior knowledge about a problem to choose an appropriate
kernel. We will find that this is not the case for GP-ConvCNP, which is even
able to learn periodicity when the chosen kernel is not periodic.

The posterior in a GP is a normal distribution with a mean function
$m(\mathbf{x_{t}})$ conditioned on the context and a covariance function
$K(\mathbf{x_{t}})$ specified by some kernel $k$:

$\displaystyle m(\mathbf{x_{t}})$
$\displaystyle=k_{tc}^{T}\left(k_{cc}+\sigma^{2}\mathbb{I}\right)^{-1}\mathbf{y_{c}}$
(6) $\displaystyle K(\mathbf{x_{t}})$
$\displaystyle=k_{tt}+\sigma^{2}-k_{tc}^{T}\left(k_{cc}+\sigma^{2}\mathbb{I}\right)^{-1}k_{tc}$
(7)

where $k_{tc}=k(\mathbf{x_{t}},\mathbf{x_{c}})$ etc. and $\sigma^{2}$ is a
noise parameter that essentially determines how close the prediction will be
to the context points. We make this parameter learnable. Note that Eq. 6 is
very similar to Eq. 5: it corresponds to the second component of the Nadaraya-
Watson estimator with only a changed denominator.

The first obvious benefit of this model is that we can sample from the GP
posterior distribution and thus also from our model, recovering one very
compelling property of NP that ConvCNP lacks. Another advantage we see is that
by working with a distribution instead of a deterministic estimate as input to
the CNN, the data distribution is implicitly smoothed. It has been established
that such smoothing reduces overfitting and improves generalization, e.g. by
adding noise to inputs [Bishop, 1995, p.347] or more generally doing data
augmentation [Volpi et al., 2018]. Working with a distribution instead of a
deterministic estimate, we need to perform Monte-Carlo integration to get a
prediction from our model. During training, however, we only use a single
sample, as is commonly done e.g. in variational autoencoders when training
with mini-batch stochastic gradient descent. To facilitate comparison, the
kernel we use in our GP is the same as in ConvCNP, i.e. a Gaussian kernel with
a learnable length scale.

Note that our model retains all desirable characteristics of the competing
approaches, in particular permutation invariance with respect to the inputs
(present in all prior art) and translation equivariance (present in
ConvCNP)333As in ConvCNP, this obviously requires a stationary kernel.. For
details on the various optimization parameters etc. we refer to the provided
implementation.

Figure 2: Examples from the temperature time series test set. For the interpolation task (top) we provide context points from the full sequence, for the extrapolation task (bottom) we provide context points in the first half of the sequence and evaluate the second. Both ConvCNP and GP-ConvCNP capture the periodicity of day/night changes in temperature well and are able to extrapolate it. We find that GP-ConvCNP often better matches the amplitude of the true signal, especially for the extrapolation task, which could explain its superior performance in Table 2. Because the temperature signal is periodic, we also show a periodic GP with a commonly used Exponential Sine-Squared kernel. NP and ANP are only shown in the appendix in Fig. A.3, as they were unable to fit the data (similar to the weakly periodic GP data in Fig. 1). Table 2: Results on the real world datasets. Again, $\uparrow$/$\downarrow$ indicate that higher/lower is better and errors represent 1 standard deviation over 5 runs with different seeds (standard error of the mean for GP, because seed influence is negligible). (left) For the temperature interpolation task, context points are randomly sampled from the test interval, for the temperature extrapolation task we provide context points in the first half of the interval and measure performance on the second half (as seen in Fig. 2). For comparison, we also show a periodic GP with an Exponential Sine-Squared kernel. (right) For the population dynamics, models are trained on simulated data, so the real world data (also shown in Fig. 3) is likely out-of-distribution, as evidenced by the stark drop in performance. There is no obvious choice of kernel if one wanted to apply a GP to this problem.  |  | Temperature Time Series | Population Dynamics
---|---|---|---
|  | interpolation | extrapolation | simulated | real
Predictive LL$\uparrow$ | GP (per.) | $-2.075\pm 0.237$ | $-46.611\pm 2.557$ |  |
| NP | $-0.855\pm 0.003$ | $-1.267\pm 0.011$ | $0.527\pm 0.051$ | $-33.070\pm 7.636$
| ANP | $-0.733\pm 0.008$ | $-1.938\pm 0.381$ | $1.027\pm 0.033$ | $-29.714\pm 9.210$
| ConvCNP | $-0.522\pm 0.008$ | $-1.261\pm 0.062$ | $1.374\pm 0.017$ | $-23.540\pm 12.441$
| GP-ConvCNP | $-0.515\pm 0.019$ | $-1.190\pm 0.016$ | $1.337\pm 0.029$ | $-5.382\pm 2.625$
Recon. Error$\downarrow$ | GP (per.) | $0.274\pm 0.001$ |  |  |
| NP | $0.238\pm 0.002$ |  | $0.018\pm 0.001$ | $1.053\pm 0.015$
| ANP | $0.198\pm 0.007$ |  | $0.008\pm 0.004$ | $0.772\pm 0.020$
| ConvCNP | $0.106\pm 0.002$ |  | $0.002\pm 0.001$ | $0.374\pm 0.019$
| GP-ConvCNP | $0.123\pm 0.018$ |  | $0.004\pm 0.001$ | $0.411\pm 0.026$

##  3 Experiments

We design our experiments with the purpose of evaluating how well members of
the Neural Process family, including the one we propose, are suited for the
task of learning distributions over functions, i.e. stochastic processes,
specifically for time series data. Like the works we compare ourselves with,
we evaluate both predictive performance (_How good is our prediction between
context points?_) via the predictive log-likelihood and the reconstruction
performance (_How good is our prediction at the context points?_) via the
root-mean-square error (RSME), because predictions directly at the context
points are usually extremely narrow Gaussians, leading to unstable
likelihoods.

As outlined in the introduction, one defining aspect of successfully learning
a distribution over functions is a model’s ability to generalize. This can
mean several things, for example independence with respect to the input value
range, called translation equivariance. This is a key feature of ConvCNP (as
long as a stationary kernel is used for interpolation), and we retain this
property in GP-ConvCNP. We evaluate two further attributes of generalization,
both on real world data: one is the ability to extrapolate the context
information, i.e. to produce good predictions well into the future by
inferring an underlying pattern; the other is the ability to deal with a
distribution shift at test time, in our case a shift from simulated to real
world data.

On top of the above, we are also interested in how well the distribution of
samples from a model matches the ideal distribution. In general, the latter is
not accessible, but for some synthetic examples we describe below,
specifically those from a Gaussian Process, we do have access, simply by using
the generating GP as an oracle. We can then compare this reference—a Gaussian
distribution—with the distribution of samples from our model. Note that one
sample is a prediction at all target points at once, as seen for example in
Fig. 1. The majority of approaches that estimate differences between
distributions fall into the categories of either _$f$ -divergences_ or
_Integral Probability Measures_ (for an overview see for example Sriperumbudur
et al. [2009]). The former require evaluations of likelihoods for both
distributions, while we only have individual samples from our model. We opt
for a parameter-free representative of the IPM category, the Wasserstein
distance $W_{2}$. We elaborate further on the definition and motivation in
Section C.4.

We initially test our method on diverse synthetic time series. The first two
have also been used in Gordon et al. [2020], and they allow us to evaluate the
sample diversity, as outlined above: (1) Samples from a Gaussian Process with
a Matern-5/2 kernel. (2) Samples from a Gaussian Process with a weakly
periodic kernel. (3) Fourier series with a variable number of components, each
of which has random bias, amplitude and phase. (4) Step functions, which were
specifically chosen to challenge our model, as the kernel we employ introduces
smoothness assumptions that are ill-suited for this problem. All of these are
described in greater detail in Appendix C as well as the provided
implementation. The size $N$ of the context set is drawn uniformly from
$[3,100)$ and the size $M$ of the target set from $[N,100)$ following Le et
al. [2018]. We further join the context set into the target set as done in
Garnelo et al. [2018a, b]. Examples can be seen in Fig. 1.

The first real world dataset we look at are weather recordings for several
different US, Canadian and Israeli cities. In particular we focus on
temperature measurements in hourly intervals that have been collected over the
course of 5 years (see Section C.2). Temperatures in each city are normalized
by their respective means and standard deviations. We randomly sample
sequences of $\sim$1 month as instances and evaluate two tasks, taking US and
Canadian cities as the training set and Israeli cities as the test set:

  1. 1.

Interpolation, where we draw context points and target points randomly from
the entire sequence (i.e. the same as in the synthetic examples).

  2. 2.

Extrapolation, where context points are drawn from the first half of the
sequence and performance is evaluated on the second half (as shown in Fig. 2).
We can reasonably be sure that temperature changes between day and night occur
in the future with the same frequency, so extrapolating this pattern is a good
test of a model’s ability to generalize.

The second real world dataset are measurements of a predator-prey population
of lynx and hare. Such population dynamics are often approximated by Lotka-
Volterra equations [Leigh, 1968], so we train models on simulated population
dynamics and test on both the simulated and real world data. Gordon et al.
[2020] used this dataset as well, but only to qualitatively show that ConvCNP
can be applied to it. The analysis will allow us to quantify how robust the
models are to a shift in distribution at test time, as the simulation
parameters are almost certainly not an ideal fit for the real world data. For
details on the simulation process we refer to Section C.3.

Finally, even though the focus of our work is on time series data, we include
some image experiments, mainly for the purpose of a more nuanced direct
comparison with ConvCNP. In particular, we compare the models on MNIST [Lecun
et al., ], CIFAR10 [Krizhevsky, 2009] and CelebA [Liu et al., 2015]. For the
latter two, we work on resampled versions at $32^{2}$ resolution. More details
are given in Appendix E.

Figure 3: Example of ConvCNP and GP-ConvCNP applied to the simulated Lotka-
Volterra population dynamics (top) and to the real Hudson Bay Company lynx-
hare dataset (bottom). Both perform well on the simulated (i.e. in-
distribution) data and seem to struggle fitting the test interval on the real
world data. Not however how the predicted uncertainty is larger for GP-
ConvCNP. We display the best out of 5 models in each case, and for ConvCNP the
performance is much more volatile, as seen in Table 2. NP and ANP perform
poorly on the real world data, the corresponding figure is Fig. A.4.

##  4 Results

Table 1 shows results for the various synthetic time series. In this
experiment the models are trained and tested on random samples generated in
the same way, so these results measure _in-distribution_ performance. We find
that GP-ConvCNP is the overall best performing method, significantly so in
terms of predictive performance for 3 out of the 4 time series and performing
on par with ConvCNP on the other. Reconstruction performance is on par with
ConvCNP in 3 out of 4 instances and significantly better in one. For
reference, we also show results for a Gaussian Process with EQ kernel (what
our model uses) and the oracle where available. Evidently, the initial GP
estimate in our model doesn’t have to be very good, but when it is, like in
the Matern-5/2 case, our approach leverages this and even matches the oracle
in performance. For examples originating from a Gaussian Process, we can
evaluate the sample diversity with respect to the oracle GP, finding that GP-
ConvCNP significantly outperforms the other methods in this regard. It is
important to note, however, that this measure does not fully isolate the
sample diversity. A low reconstruction error, for example, will also improve
the $W_{2}$, which is likely the reason that ANP still performs better than
NP, even though the former hardly displays any variation in its samples, as
seen in Fig. 1. The figure also shows how NP and ANP struggle to fit high
frequency signals, while ConvCNP and GP-ConvCNP are able to. The sample
diversity in GP-ConvCNP is larger than in ANP, but samples are only
significantly different from the mean prediction when further away from the
context points in areas of high predictive uncertainty (shaded areas
correspond to $1\sigma$). In contrast, samples from the NP are more diverse
throughout, at the expense of accurately matching the context points.

Table 3: Results for the image experiments, in terms of predictive log-likelihood (i.e. higher is better) on the respective test sets. Errors represent 1 standard deviation over 10 runs with different seeds. Bold indicates a significant difference, i.e. when the difference is larger than the root sum of squares of the standard deviations. GP-ConvCNP outperforms ConvCNP overall, with a slight (non-significant) advantage for ConvCNP on MNIST. Visual examples and more details on the image experiments are given in Appendix E. | ConvCNP | GP-ConvCNP
---|---|---
MNIST | $4.133\pm 0.057$ | $4.077\pm 0.026$
CIFAR10 | $2.462\pm 0.006$ | $2.744\pm 0.008$
CelebA | $2.212\pm 0.006$ | $2.468\pm 0.008$

Examples from the temperature time series dataset can be seen in Fig. 2. The
key characteristic of the signal is the temperature change between day and
night, making it a high frequency signal not unlike the weakly periodic GP
samples in the synthetic dataset. NP and ANP were not able to fit these
signals, as can be seen in Fig. A.3. The top row of Fig. 2 shows an example of
the regular interpolation task, the bottom row an example of the extrapolation
task, which we deem an important aspect of generalization. ConvCNP and GP-
ConvCNP are both able to interpolate as well as extrapolate the correct
temperature pattern, but occasionally ConvCNP underestimates the amplitude
when extrapolating. We also show an example of a periodic GP using an
Exponential Sine-Squared kernel, which is a common choice for periodic
signals. It fails to capture finer variations in the signal and often
struggles to infer the right frequency, which results in its poor
extrapolation performance in Table 2. We find that while ConvCNP and GP-
ConvCNP perform on par for the interpolation task, GP-ConvCNP performs
significantly better than the other methods on the extrapolation task.

To measure how robust the different members of the Neural Process family are
to a distribution shift at test time, we train models on population dynamics
simulated as Lotka-Volterra processes, and evaluate performance both on
simulated (_in-distribution_) and real world (_out-of-distribution_) data. The
real world dataset, along with a simulated example, can be seen in Fig. 3.
While both ConvCNP and GP-ConvCNP fit the simulated data well, they struggle
with the test interval on the real data. This is reflected in Table 2 as well,
where we find that ConvCNP performs better than GP-ConvCNP (even significantly
so, albeit not with a huge difference) on the simulated data. Applied to the
real world dataset, all methods experience a large drop in performance,
indicating that this is indeed a significant distribution shift. GP-ConvCNP is
by far the best performing method here, which is likely because of a better
estimate of the preditive uncertainty. Note how the uncertainty predicted by
ConvCNP is smaller than that of GP-ConvCNP in Fig. 3 (the figure shows
$1\sigma$). The predictions we show here are from the best performing seed in
each case, other ConvCNP models predicted an even narrower distribution. We
selected this particular interval for testing because it’s the same interval
Gordon et al. [2020] show in the ConvCNP paper. We also evaluated with context
points drawn randomly from the entire interval (i.e. the same way we evaluate
on the simulated data), and GP-ConvCNP still performs significantly better
than the competing approaches (see Table A.1).

ConvCNP also showed performance improvements compared to NP and ANP when
applied to image data. While the focus of our work is on time series, we were
also interested to see if our model yields any benefits in this domain. It
does indeed, as seen in Table 3, where GP-ConvCNP outperforms ConvCNP on both
CIFAR10 and CelebA (ConvCNP has a non-significant advantage on MNIST).
Examples are given in Appendix E, where we don’t see any meaningful difference
in visual quality. The latter only “measures” the quality of the mean
prediction, so we suspect that the performance improvement is due to a more
accurate predictive uncertainty.

##  5 Related Work

Neural Processes have inspired a number of works outside of the ones we
discuss. Louizos et al. [2019] propose to not merge observations into a global
latent space, but instead learn conditional relationships between them. This
is especially suitable for semantically meaningful clustering and
classification. Singh et al. [2019] and Willi et al. [2019] address the
problem of overlapping and changing dynamics in the generating process of the
data, a special case we do not include here. With a simple Gaussian kernel, we
wouldn’t expect our model to perform well in that scenario, but one could of
course introduce inductive bias in the form of e.g. non-stationary kernels,
when translation equivariance is no longer desired. NPs have also been scaled
to extremely complex output spaces like in _Generative Query Networks_ [Eslami
et al., 2018, Rosenbaum et al., 2018], where a single observation is a full
image. GQN directly relates to the problem of (3D) scene understanding
[Sitzmann et al., 2019, Engelcke et al., 2020].

Gordon et al. [2020] build their work (ConvCNP) upon recent contributions in
the area of _learning on sets_ , i.e. neural networks with set-valued inputs
[Zaheer et al., 2017, Wagstaff et al., 2019], which has mostly been explored
in the context of point clouds [Qi et al., 2017b, a, Wu et al., 2019].
Especially the work of Wu et al. [2019] is closely related to Gordon et al.
[2020], also employing a CNN on a kernel density estimate, but their
application is not concerned with time series. _Bayesian Neural Networks_
[Neal, 1996, Graves, 2011, Hernández-Lobato and Adams, 2015] also address the
problem of learning distributions over functions, but often implicitly, in the
sense that the distributions over the weights are used to estimate uncertainty
[Blundell et al., 2015, Gal and Ghahramani, 2016]. We are interested in this
too, but in our scenario we want to be able to condition on observations at
test time.

The main limitation of _Gaussian Processes_ is their computational complexity
and many works are dedicated to improving this aspect, often via
approximations based on inducing points [Snelson and Ghahramani, 2006,
Titsias, 2009, Gardner et al., 2018, Wilson and Nickisch, 2015] but also other
approaches [Deisenroth and Ng, 2015, Rahimi and Recht, 2007, Le et al., 2013,
Cheng and Boots, 2017, Hensman et al., 2013, 2015, Salimbeni et al., 2018],
even for exact GPs [Wang et al., 2019]. Rather than competing with these
approaches, our model will be able to leverage developments in this area. Some
of the above try to find more efficient kernel representations and are thus
closely related to the idea of _kernel learning_ , i.e. the idea to combine
the expressiveness of (deep) learning approaches with the flexibility of
kernel methods, for example Yang et al. [2015], Wilson et al. [2016b, a],
Tossou et al. [2019], Calandra et al. [2016]. The key difference to our work
is that these approaches attempt to learn kernels as an input to a kernel
method, while we learn to make the output of a kernel method more expressive.

##  6 Discussion

We have presented a new model in the Neural Process family that extends
ConvCNP by incorporating a Gaussian Process into it. We show on both synthetic
and real time series that this improves performance overall, but most markedly
when generalization is required: our model, GP-ConvCNP, can better extrapolate
to regions far from the provided context points and is more robust when moving
to real world data after training on simulated data. We further retain
translation equivariance, a key feature of ConvCNP, as long as a stationary
kernel is used for the GP. The introduction of the latter also allows us to
draw multiple samples from the model, where the distribution of samples from
our model better matches the samples from an oracle than those from a regular
Neural Process or an Attentive Neural Process do. Our model uses the
prediction from a GP with an EQ-kernel as an initial estimate. Interestingly,
this estimate needn’t be very good—our model can learn periodicity even with a
non-periodic input kernel—but when it is, our model can fully leverage it and
even match the performance of an oracle, as seen in Table 1. An advantage all
Neural Process flavors enjoy compared to many conventional time series
prediction methods such as ARIMA models (see e.g. Hyndman and Athanasopoulos
[2018]) is that they naturally work on non-uniform time series, with
observations acquired at arbitrary times.

Of course, with the benefits of GPs we also inherit their limitations. GPs are
typically slow, naively requiring $O(N^{3})$ operations in the number of
context observations, and our model inherits this complexity. While this was a
non-issue on the time series data used in our work, GP-ConCNP was noticably
slower than ConvCNP (roughly 1.5x) in the image experiments, which we included
for a more complete comparison with ConvCNP. Our model still outperformed
ConvCNP, but for larger images the improved performance will likely not be
worth the additional cost. Making GPs faster is a very active research area,
as outlined above. For our model specifically it seems reasonable to leverage
work on deep kernels [Wilson et al., 2016b] or to learn mappings before the GP
prediction like in Calandra et al. [2016] in order to learn more meaningful GP
posteriors that capture information about the training distribution. We do
expect that our model is well suited to also work with these approximate
methods, as we modify the prediction from the GP with a powerful neural
network that should be able to correct minor approximation errors. For
example, KISS-GP Wilson and Nickisch [2015] only has linear complexity, so
incorporating it or one of the many other efficient approximate methods into
our model should allow it to scale to much larger datasets. We leave a
verification of this for future work.

## References

  * Bishop [1995] Christopher M. Bishop.  _Neural Networks for Pattern Recognition_.  Oxford University Press, Inc., 1995.
  * Blundell et al. [2015] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra.  Weight uncertainty in neural networks.  In _International Conference on Machine Learning_ , pages 1613–1622, 2015.
  * Calandra et al. [2016] Roberto Calandra, Jan Peters, Carl Edward Rasmussen, and Marc Peter Deisenroth.  Manifold gaussian processes for regression.  _arXiv:1402.5876 [cs, stat]_ , 2016.
  * Cheng and Boots [2017] Ching-An Cheng and Byron Boots.  Variational inference for gaussian process models with linear complexity.  In _Advances in Neural Information Processing Systems 30_ , pages 5184–5194. 2017.
  * Deisenroth and Ng [2015] Marc Deisenroth and Jun Wei Ng.  Distributed gaussian processes.  In _International Conference on Machine Learning_ , pages 1481–1490, 2015.
  * Engelcke et al. [2020] Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner.  GENESIS: Generative scene inference and sampling with object-centric latent representations.  In _International Conference on Learning Representations_ , 2020.
  * Eslami et al. [2018] S. M. Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S. Morcos, Marta Garnelo, Avraham Ruderman, Andrei A. Rusu, Ivo Danihelka, Karol Gregor, David P. Reichert, Lars Buesing, Theophane Weber, Oriol Vinyals, Dan Rosenbaum, Neil Rabinowitz, Helen King, Chloe Hillier, Matt Botvinick, Daan Wierstra, Koray Kavukcuoglu, and Demis Hassabis.  Neural scene representation and rendering.  _Science_ , 360(6394):1204–1210, 2018.
  * Gal and Ghahramani [2016] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _International Conference on Machine Learning_ , pages 1050–1059, 2016.
  * Gardner et al. [2018] Jacob Gardner, Geoff Pleiss, Ruihan Wu, Kilian Weinberger, and Andrew Wilson.  Product kernel interpolation for scalable gaussian processes.  In _International Conference on Artificial Intelligence and Statistics_ , pages 1407–1416, 2018.
  * Garnelo et al. [2018a] Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and S. M. Ali Eslami.  Conditional neural processes.  In _International Conference on Machine Learning_ , pages 1704–1713, 2018a.
  * Garnelo et al. [2018b] Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, S. M. Ali Eslami, and Yee Whye Teh.  Neural processes.  In _ICML Workshop on Theoretical Foundations and Applications of Deep Generative Models_ , 2018b.
  * Gordon et al. [2020] Jonathan Gordon, Wessel P. Bruinsma, Andrew Y. K. Foong, James Requeima, Yann Dubois, and Richard E. Turner.  Convolutional conditional neural processes.  In _International Conference on Learning Representations_ , 2020.
  * Graves [2011] Alex Graves.  Practical variational inference for neural networks.  In _Advances in Neural Information Processing Systems 24_ , pages 2348–2356. 2011.
  * Hensman et al. [2013] James Hensman, Nicolò Fusi, and Neil D. Lawrence.  Gaussian processes for big data.  In _Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence_ , pages 282–290, 2013.
  * Hensman et al. [2015] James Hensman, Alexander Matthews, and Zoubin Ghahramani.  Scalable variational gaussian process classification.  In _Internation Conference on Artificial Intelligence and Statistics_ , pages 351–360, 2015.
  * Hernández-Lobato and Adams [2015] José Miguel Hernández-Lobato and Ryan P. Adams.  Probabilistic backpropagation for scalable learning of bayesian neural networks.  In _International Conference on Machine Learning_ , volume 37, pages 1861–1869, 2015.
  * Hyndman and Athanasopoulos [2018] Rob J. Hyndman and George Athanasopoulos.  _Forecasting: principles and practice_.  OTexts, 2018.
  * Kim et al. [2019] Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh.  Attentive neural processes.  In _International Conference on Learning Representations_ , 2019.
  * Krizhevsky [2009] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  Technical report, 2009.
  * Le et al. [2013] Quoc Le, Tamas Sarlos, and Alexander Smola.  Fastfood - computing hilbert space expansions in loglinear time.  In _International Conference on Machine Learning_ , pages 244–252, 2013.
  * Le et al. [2018] Tuan Anh Le, Hyunjik Kim, Marta Garnelo, Dan Rosenbaum, Jonathan Schwarz, and Yee Whye Teh.  Empirical evaluation of neural process objectives.  In _NeurIPS Bayesian Deep Learning Workshop_ , 2018.
  * [22] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner.  Gradient-based learning applied to document recognition.  86(11):2278–2324.
  * Leigh [1968] Egbert G Leigh.  _Ecological role of Volterra’s equations_.  Lectures on mathematics in the life sciences. Princeton University, 1968\.
  * Liu et al. [2015] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In _Proceedings of International Conference on Computer Vision (ICCV)_ , December 2015.
  * Louizos et al. [2019] Christos Louizos, Xiahan Shi, Klamer Schutte, and Max Welling.  The functional neural process.  In _Advances in Neural Information Processing Systems 32_ , pages 8743–8754. 2019.
  * Neal [1996] Radford M. Neal.  _Bayesian Learning for Neural Networks_.  Lecture Notes in Statistics. Springer, 1996.
  * Qi et al. [2017a] Charles R. Qi, Hao Su, Mo Kaichun, and Leonidas J. Guibas.  PointNet: Deep learning on point sets for 3d classification and segmentation.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , pages 77–85, 2017a.
  * Qi et al. [2017b] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.  PointNet++: Deep hierarchical feature learning on point sets in a metric space.  In _Advances in Neural Information Processing Systems 30_ , pages 5099–5108. 2017b.
  * Rahimi and Recht [2007] Ali Rahimi and Benjamin Recht.  Random features for large-scale kernel machines.  In _Advances in Neural Information Processing Systems 20_ , pages 1177–1184. 2007.
  * Rasmussen and Williams [2006] Carl Edward Rasmussen and C. K. I. Williams.  _Gaussian Processes for Machine Learning_.  MIT Press, 2006.
  * Roberts et al. [2013] S. Roberts, M. Osborne, M. Ebden, S. Reece, N. Gibson, and S. Aigrain.  Gaussian processes for time-series modelling.  371(1984):20110550, 2013.
  * Rosenbaum et al. [2018] Dan Rosenbaum, Frederic Besse, Fabio Viola, Danilo J. Rezende, and S. M. Ali Eslami.  Learning models for visual 3d localization with implicit mapping.  In _NeurIPS Bayesian Deep Learning Workshop_ , 2018.
  * Salimbeni et al. [2018] Hugh Salimbeni, Ching-An Cheng, Byron Boots, and Marc Deisenroth.  Orthogonally decoupled variational gaussian processes.  In _Advances in Neural Information Processing Systems 31_ , pages 8711–8720. 2018.
  * Singh et al. [2019] Gautam Singh, Jaesik Yoon, Youngsung Son, and Sungjin Ahn.  Sequential neural processes.  In _Advances in Neural Information Processing Systems 32_ , pages 10254–10264. 2019.
  * Sitzmann et al. [2019] Vincent Sitzmann, Michael Zollhoefer, and Gordon Wetzstein.  Scene representation networks: Continuous 3d-structure-aware neural scene representations.  In _Advances in Neural Information Processing Systems 32_ , pages 1119–1130. 2019.
  * Snelson and Ghahramani [2006] Edward Snelson and Zoubin Ghahramani.  Sparse gaussian processes using pseudo-inputs.  In _Advances in Neural Information Processing Systems 18_ , pages 1257–1264. 2006.
  * Sriperumbudur et al. [2009] Bharath K. Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard Schölkopf, and Gert R. G. Lanckriet.  On integral probability metrics, phi-divergences and binary classification.  _arXiv:0901.2698 [cs, math]_ , 2009.
  * Titsias [2009] Michalis Titsias.  Variational learning of inducing variables in sparse gaussian processes.  In _International Conference on Artificial Intelligence and Statistics_ , pages 567–574, 2009.
  * Tossou et al. [2019] Prudencio Tossou, Basile Dura, Francois Laviolette, Mario Marchand, and Alexandre Lacoste.  Adaptive deep kernel learning.  _arXiv:1905.12131 [cs, stat]_ , 2019.
  * Volpi et al. [2018] Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John Duchi, Vittorio Murino, and Silvio Savarese.  Generalizing to unseen domains via adversarial data augmentation.  In _Advances in Neural Information Processing Systems 31_ , page 5339–5349, 2018.
  * Wagstaff et al. [2019] Edward Wagstaff, Fabian B. Fuchs, Martin Engelcke, Ingmar Posner, and Michael Osborne.  On the limitations of representing functions on sets.  In _International Conference on Machine Learning_ , 2019.
  * Wang et al. [2019] Ke Wang, Geoff Pleiss, Jacob Gardner, Stephen Tyree, Kilian Q Weinberger, and Andrew Gordon Wilson.  Exact gaussian processes on a million data points.  In _Advances in Neural Information Processing Systems 32_ , pages 14622–14632. 2019.
  * Willi et al. [2019] Timon Willi, Jonathan Masci, Jürgen Schmidhuber, and Christian Osendorfer.  Recurrent neural processes.  _arXiv:1906.05915 [cs, stat]_ , 2019.
  * Wilson et al. [2016a] Andrew G Wilson, Zhiting Hu, Russ R Salakhutdinov, and Eric P Xing.  Stochastic variational deep kernel learning.  In _Advances in Neural Information Processing Systems 29_ , pages 2586–2594. 2016a.
  * Wilson and Nickisch [2015] Andrew Gordon Wilson and Hannes Nickisch.  Kernel interpolation for scalable structured gaussian processes (KISS-GP).  In _International Conference on Machine Learning_ , 2015.
  * Wilson et al. [2016b] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing.  Deep kernel learning.  In _International Conference on Artificial Intelligence and Statistics_ , 2016b.
  * Wu et al. [2019] Wenxuan Wu, Zhongang Qi, and Li Fuxin.  PointConv: Deep convolutional networks on 3d point clouds.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , pages 9621–9630, 2019.
  * Yang et al. [2015] Zichao Yang, Andrew Wilson, Alex Smola, and Le Song.  A la carte – learning fast kernels.  In _International Conference on Artificial Intelligence and Statistics_ , pages 1098–1106, 2015.
  * Zaheer et al. [2017] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola.  Deep sets.  In _Advances in Neural Information Processing Systems 30_ , pages 3391–3401. 2017.

Figure A.1: Schematic overview of the different methods used in this work.
Dotted lines indicate sampling and we use the following acronyms: multilayer
perceptron (MLP), kernel density estimate (KDE), Gaussian Process (GP), linear
layer (LIN), convolutional neural network (CNN). (_First row_) Neural
Processes (NP) encode each context point $(x_{c},y_{c})$ into a representation
$z_{c}$. These are then averaged to form a global representation $z$. A sample
from the global representation is concatenated with the target input $x_{t}$
to predict the target output $y_{t}$. (_Second row_) Attentive Neural
Processes (ANP) contain a NP, but have a second deterministic path. In this
path, the context pairs are also encoded separately into representations
$\mathbf{a_{c}}$. These are then combined via an atttention mechanism that
uses $x_{t}$ as the query, $\mathbf{x_{c}}$ as the keys and $\mathbf{a_{c}}$
as the values. The resulting representation $a$ is concatenated with the
representation from the NP path and the target input to predict the target
output. (_Third row_) ConvCNP performs a kernel density estimate on the
context observations $(\mathbf{x_{c}},\mathbf{y_{c}})$, thus mapping to a
continuous representation. This representation is evaluated on a grid, i.e.
discretized, and a projection and CNN operate on the discretized
representation. The result is evaluated at a target input $x_{t}$ by
performing a convolution with the discretized representation and finally
projected to predict the target output. (_Fourth row_) GP-ConvCNP works
similar to ConvCNP, but instead of a deterministic kernel density estimate a
Gaussian Process is applied to the context. We sample from the GP posterior
and discretize the result, continuing with the same operations as in ConvCNP.
Note that for visual purposes, the KDE and GP outputs are one-dimensional, but
in reality the output space can have any number of dimensions.

##  Appendix A Method Descriptions

Fig. A.1 shows schematic representations of the different methods used in this
work, and a description is given in the figure caption. The MLPs in both NP
and ANP have 6 hidden layers with 128 channels each, and the input and output
sizes are adjusted to match the dimensions of data and latent representations.
The latent representation in both models has 128 dimensions, so that the
encoders for the NP and the NP path in ANP have 256 output channels to
represent both the mean and the standard deviation of a Gaussian distribution
(in practice, we predict the log-variance, not the standard deviation). The
attention mechanism in ANP also uses 128 as the embedding dimension. These
configurations follow Le et al. [2018], who evaluated several different
configurations for NP and ANP.

ConvCNP and GP-ConvCNP both use a Gaussian kernel with a learnable length
scale $l$ to map the input to a continuous representation, given by

$k(x,x^{\prime})=\exp\left(-\frac{|x-x^{\prime}|^{2}}{2l^{2}}\right)$ (8)

The result is discretized onto a grid, which we obtain by taking the minimum
and maximum of the target inputs as the value range, padded by 0.1 units. The
grid is constructed over this range with a resolution of 20 points per unit.
The discretized representations are projected to 8 channels before a CNN is
applied. The CNN is a 12-layer residual network with ReLU activations. The
number of channels in the convolutional layers doubles every second layer for
the first 6 layers and is then decreased symmetrically, leading to 8 output
channels. Residual connections are implemented via concatenation. Predictions
are obtained by convolving the CNN output with a target input, followed by a
final projection.

##  Appendix B Optimization

Recall that our optimization objective is

$\max_{\theta}\sum_{f\in\mathcal{F}}\log
p_{\theta}(\mathbf{y_{t}}|\mathbf{x_{t}},\mathbf{x_{c}},\mathbf{y_{c}})$ (9)

which we can rewrite as

$\max_{\theta}\sum_{f\in\mathcal{F}}\log
p_{\theta}(\mathbf{y_{t}}|\mathbf{x_{t}},Z)$ (10)

where $Z$ is given by the different $E$ that _encode_ the context defined in
Section 2. For ConvCNP this is deterministic, so we can maximize Eq. 9
directly. For the other methods we can again rewrite the summands as

$\log
p(\mathbf{y_{t}}|\mathbf{x_{t}},\mathbf{x_{c}},\mathbf{y_{c}})=\log\mathop{\mathbb{E}}_{z\sim
p(z|\mathbf{x_{c}},\mathbf{y_{c}})}p(\mathbf{y_{t}}|\mathbf{x_{t}},z)$ (11)

where we now distinguish $z$ as an expression of $Z$. In GP-ConvCNP,
$p(z|\mathbf{x_{c}},\mathbf{y_{c}})$ is given by the GP posterior, so for
training we would need to integrate over the posterior. In practice, we just
draw a single sample, which is common practice in stochastic mini-batch
training. Approximating the expectation with this sample, we can also directly
maximize the log-likelihood.

In contrast to the above, $p(z|\mathbf{x_{c}},\mathbf{y_{c}})$ is an unknown
or intractable mapping in NP and ANP, so we employ variational inference, i.e.
we approximate $p(z|\mathbf{x_{c}},\mathbf{y_{c}})$ with a member of some
family $Q$ that we can find by optimization. The log-likelihood then becomes

$\displaystyle\log
p(\mathbf{y_{t}}|\mathbf{x_{t}},\mathbf{x_{c}},\mathbf{y_{c}})\geq$
$\displaystyle\mathop{\mathbb{E}}_{z\sim
q(z|\mathbf{x_{t}},\mathbf{y_{t}})}\log p(\mathbf{y_{t}}|\mathbf{x_{t}},z)$
$\displaystyle-
D_{KL}\left(q(z|\mathbf{x_{t}},\mathbf{y_{t}})||p(z|\mathbf{x_{c}},\mathbf{y_{c}})\right)$
(12) $\displaystyle\approx$ $\displaystyle\mathop{\mathbb{E}}_{z\sim
q(z|\mathbf{x_{t}},\mathbf{y_{t}})}\log p(\mathbf{y_{t}}|\mathbf{x_{t}},z)$
$\displaystyle-
D_{KL}\left(q(z|\mathbf{x_{t}},\mathbf{y_{t}})||q(z|\mathbf{x_{c}},\mathbf{y_{c}})\right)$
(13)

where the inequality follows from Jensen’s inequality. To maximize the LHS it
is sufficient to maximize the RHS, and Eq. 13 is what is being optimized in NP
and ANP. $q$ corresponds to what we designated as $E$ in Section 2. Like for
GP-ConvCNP, we approximate the expectation with a single sample during
training.

In our implementation, we use Adam with an initial learning rate of $0.001$.
We train each model for $600\,000\text{\,}\mathrm{b}$atches with a batch size
of 256. We repeatedly multiply the learning rate by $\gamma=0.995$ after
training for 1000 batches.

##  Appendix C Data & Evaluation Details

###  C.1 Synthetic Data

For all synthetic time series draws we define the x-axis to cover the interval
$[-3,3]$. As outlined in Section 3.2, we draw $N$ context points randomly from
this interval, with $N$ a random integer from the range $[3,100)$. We then
draw $M$ target points in the same manner, with $M$ a random integer from
$[N,100)$. During training, we add the context points to the target set so
that the methods learn to reconstruct the context. These are the different
types we evaluate:

  1. 1.

Samples from a Gaussian Process with a Matern-5/2 kernel with lengthscale
parameter $l=0.5$. The kernel is given by

$\displaystyle k(x,x^{\prime})=$
$\displaystyle\left(1+\frac{\sqrt{5}|x-x^{\prime}|}{l}+\frac{5|x-x^{\prime}|^{2}}{3l^{2}}\right)$
$\displaystyle\cdot\;$
$\displaystyle\exp\left(-\frac{5|x-x^{\prime}|}{l}\right)$ (14)

  2. 2.

Samples from a Gaussian Process with a weakly periodic kernel that is given by

$\displaystyle k(x,x^{\prime})=$
$\displaystyle\exp\left(-\frac{|x-x^{\prime}|^{2}}{8}\right)$
$\displaystyle\cdot\;$ $\displaystyle\exp\Big{(}(\cos(8\pi x)-\cos(8\pi
x^{\prime}))^{2}\Big{)}$ $\displaystyle\cdot\;$
$\displaystyle\exp\Big{(}(\sin(8\pi x)-\sin(8\pi x^{\prime}))^{2}\Big{)}$ (15)

  3. 3.

Fourier series that are given by

$f(x)=a_{0}+\sum_{k=1}^{K}a_{k}\cos(kx-\phi_{k})$ (16)

where $K$ is a random integer from $[10,20)$ and $a_{k}$ (including $a_{0}$)
as well as $\phi_{k}$ are random real numbers drawn from $[-1,1]$.

  4. 4.

Step functions, where we draw $S$ stepping points along the x-axis, with $S$ a
random integer from $[3,10)$. The interval between two stepping points is
assigned a constant value that is drawn from $[-3,3]$. We ensure that each
interval is at least $0.1$ units wide and that the step difference is also at
least $0.1$ units in magnitude.

###  C.2 Temperature Time Series

The temperature dataset we work with is taken from
https://www.kaggle.com/selfishgene/historical-hourly-weather-data. It consists
of hourly temperature measurements in 30 US and Canadian cities as well as 6
Israeli cities, taken continuously over the course of $\sim$5 years.
Occasionally there are NaN values reported in the dataset, we either crop
those when at the begging/end of a sequence or fill them via linear
interpolation. We use the US/Canadian cities as our training and validation
set and the Israeli cities as our test set. For both training and testing we
draw random sequences of length 720 (i.e. 30 days) from the corresponding set,
and then draw $N$ context points and $M$ target points from the sequence, with
$N$ from the interval $[20,100)$ and $M$ from $[N,100)$. The temperatures for
each city are normalized by their respective means and standard deviations,
and we define the time range for a given sequence to be $[0,3]$, so that one
time unit is equivalent to 10 days. We evaluate each seed for a model with 100
random samples and report the mean and standard deviation over 5 seeds for
each model. For convenience, we include the data with our implementation.

###  C.3 Population Dynamics

We simulate population dynamics of a predator-prey population with a Lotka-
Volterra model. Let $X$ be the number of predators at a given time and $Y$ the
number of prey. We draw initial numbers $X$ from $[50,100)$ and $Y$ from
$[100,150)$. We then draw time increments from an exponential distribution and
after each time increment one of the following events occurs:

  1. 1.

A single predator is born with probability proportional to the rate
$\theta_{0}\cdot X\cdot Y$

  2. 2.

A single predator dies with probability proportional to the rate
$\theta_{1}\cdot X$

  3. 3.

A single prey is born with probability proportional to the rate
$\theta_{2}\cdot Y$

  4. 4.

A single prey dies with probability proportional to the rate $\theta_{3}\cdot
X\cdot Y$

The rate of the exponential distribution we draw time increments from is the
sum of the above rates. Each population is simulated for 10000 events, and we
reject populations that have died out, populations that exceed a total number
of 500 individuals at any given point, as well as those where the accumulated
time is larger than 100 units. To get value ranges that are better suitable
for training, we rescale the time axis by a factor $0.1$ and the population
axis by a factor $0.01$. For each population we draw $\theta_{0}$ from
$[0.005,0.01]$, $\theta_{1}$ from $[0.5,0.8]$, $\theta_{2}$ from $[0.5,0.8]$
and $\theta_{3}$ from $[0.005,0.01]$. These parameters result in roughly 2/3
of the simulated populations matching our criteria. We also tried the
parameters reported in Gordon et al. [2020], but found that we had to reject
more than $90\%$ of populations, which meant an unreasonably long training
time, as the simulation process for the populations is difficult to
parallelize and thus rather slow. The $N$ context points and $M$ target points
are again drawn randomly from a population, with $N$ from $[20,100)$ and $M$
from $[\max(70,N),150)$.

We evaluate models trained on simulated data on real world measurements of a
lynx-hare population. The data were recorded at the end of the 19th and the
start of the 20th century by the Hudson’s Bay Company. To the best of our
knowledge, the data represent recorded trades of pelts from the two animals
and not direct measurements of the populations. There is no unique source for
the data in a tabular format, but we used https://github.com/stan-dev/example-
models/blob/master/knitr/lotka-volterra/hudson-bay-lynx-hare and include the
data with our code for convenience. For evaluation, we normalize the data so
that the mean population matches the mean of populations in the simulated data
and the time interval matches the mean duration of a simulated population.

###  C.4 Wasserstein Distance

As outlined in Section 3, we seek to compare the distribution of samples from
a model with a reference distribution, which we have access to for the
synthetic examples sampled from a GP in the form of the prediction from the
same GP. Comparing distributions is usually done with either some form of _$f$
-divergence_ (e.g. the Kullback-Leibler divergence) or with an _Integral
Probability Measure_ (IPM) f-divergences require evaluations of likelihoods in
both distributions, while we can only evaluate those under the GP posterior
but not in our models. IPM only compare samples from the distributions and are
thus suited for our scenario. One of the more well-known measures from this
group is the Wasserstein distance given by:

$W_{p}(P,Q)=\min_{\pi}\left(\sum_{i=1}^{|P|}||x_{i}-y_{\pi(i)}||^{p}\right)^{1/p}$
(17)

where $P=\\{x_{i}\\}_{i}$ and $Q=\\{y_{i}\\}_{i}$ are collections of samples
from the two distributions. In colloquial terms, the Wasserstein distance is
the minimum overall distance between sample pairs, taken over all possible
pairings between samples from the two distributions. For this reason the
Wasserstein-1 distance is also called the _Earth Mover Distance_. $p$ is the
only hyperparameter we need to select, making this measure a very convenient
choice. We set $p=2$ so that the underlying distance metric becomes the
Euclidean distance.

##  Appendix D Additional Results

Table A.1: This table corresponds to the rightmost column in Table 2, i.e. it shows results on the real world population dynamics data. In Table 2, the evaluation was performed as seen in Fig. 3, meaning one contiguous interval on the data was selected as the target region and the rest of the data is provided as context, following Gordon et al. [2020]. Here we instead sample the context and target points randomly from the entire interval, like we do in the other experiments as well. For each seed, we average over 100 random draws and report the standard deviation over 5 seeds as errors. While ConvCNP maintains leading performance in terms of reconstruction error, GP-ConvCNP significantly outperforms the other methods in predictive performance, similar to what we found in Table 2. All methods perform worse compared to the evaluation method used in Table 2. | Predictive LL$\uparrow$ | Recon. Error$\downarrow$
---|---|---
NP | $-36.735\pm 4.137$ | $0.952\pm 0.024$
ANP | $-38.717\pm 3.572$ | $0.718\pm 0.018$
ConvCNP | $-28.762\pm 1.958$ | $0.272\pm 0.008$
GP-ConvCNP | $-19.252\pm 1.846$ | $0.343\pm 0.020$

In this section we show some additional results, specifically we show the
performance as a function of the number of context points (Fig. A.2) as well
as examples for NP and ANP on the temperature time series dataset in Fig. A.3
and on the population dynamics dataset in Fig. A.4. We also show results on
the population dynamics dataset in Table A.1, using a different evaluation
method compared to the main manuscript.

Figure A.2: Performance as a function of the number of context points for the
two synthetic GP examples. For sparse context points, our model and ConvCNP
are on par, while an increasing number of context points leads to an advantage
for our model. Figure A.3: Examples from the temperature time series test set
for NP and ANP. For the interpolation task (left) we provide context points
from the full sequence, for the extrapolation task (right) we provide context
points in the first half of the sequence and evaluate the second. Both methods
are unable to fit the context points, likely because the frequency is too high
to be represented in the models. Figure A.4: Example of NP and ANP applied to
the simulated Lotka-Volterra population dynamics (top) and to the real Hudson
Bay Company lynx-hare dataset (bottom). Similar to ConvCNP and GP-ConvCNP,
seen in Fig. 3, both work well on simulated data. On the real data, however,
both struggle to fit the context points and produce a poor prediction for the
test interval.

##  Appendix E Image Experiments

For a more complete comparison of our model with ConvCNP, we include image
experiments, specifically MNIST, CIFAR10 and CelebA. For the latter two, we
work with resampled images at $32^{2}$ resolution. The context set has a size
drawn from $[20,400)$ ($[20,300)$ for MNIST), the target set a size drawn from
$[50,400)$, and we reconstruct both target and context points during training.
We evaluate the average log-likehood of the model predictions on the
respective test sets, as seen in Table 3. The implementation of ConvCNP is
again taken directly from the official repository, and we leave the
architecture unchanged with the exception of swapping the kernel interpolation
for a GP to make the comparison fair. All other hyper parameters are the same
as in the time series experiments.

Examples for both ConvCNP and GP-ConvCNP can be seen in Fig. A.5, Fig. A.6 and
Fig. A.7, with each example taken from the test sets. There is not noticeable
visual difference between the two model, so we assume that the improved
performance is due to better estimates of the predictive uncertainty (i.e. the
standard deviation of the predicted Gaussian). In terms of performance, we
found that inference takes roughly 1.5x as long for GP-ConvCNP as it does for
ConvCNP, which we believe is still an acceptable tradeoff.

Figure A.5: Examples for ConvCNP and GP-ConvCNP applied on MNIST test data.
Models were trained on the training set. Numbers indicate the number of
context points and the top left panel shows the reference image for each case.
Figure A.6: Examples for ConvCNP and GP-ConvCNP applied on CIFAR10 test data.
Models were trained on the training set. Numbers indicate the number of
context points and the top left panel shows the reference image for each case.
Figure A.7: Examples for ConvCNP and GP-ConvCNP applied on CelebA test data,
resized to 32x32 resolution. Models were trained on the training set. Numbers
indicate the number of context points and the top left panel shows the
reference image for each case.