# DistPred: A Distribution-Free Probabilistic Inference Method for Regression
and Forecasting

David S. Hippocampus
Department of Computer Science
Cranberry-Lemon University
Pittsburgh, PA 15213
<EMAIL_ADDRESS>
Use footnote for providing further information about author (webpage,
alternative address)— _not_ for acknowledging funding agencies.

###### Abstract

Traditional regression and prediction tasks often only provide deterministic
point estimates. To estimate the uncertainty or distribution information of
the response variable, methods such as Bayesian inference, model ensembling,
or MC Dropout are typically used. These methods either assume that the
posterior distribution of samples follows a Gaussian process or require
thousands of forward passes for sample generation. We propose a novel approach
called DistPred for regression and forecasting tasks, which overcomes the
limitations of existing methods while remaining simple and powerful.
Specifically, we transform proper scoring rules that measure the discrepancy
between the predicted distribution and the target distribution into a
differentiable discrete form and use it as a loss function to train the model
end-to-end. This allows the model to sample numerous samples in a single
forward pass to estimate the potential distribution of the response variable.
We have compared our method with several existing approaches on multiple
datasets and achieved state-of-the-art performance. Additionally, our method
significantly improves computational efficiency. For example, compared to
state-of-the-art models, DistPred has a 90x faster inference speed.
Experimental results can be reproduced through this anonymous link.

##  1 Introduction

Traditional deterministic point estimates are no longer sufficient to meet the
needs of AI safety and uncertainty quantification. For example, we may want to
obtain confidence intervals for predicted points to make important decisions,
such as deciding whether to travel based on weather forecasts or how to invest
based on stock predictions. Moreover, this is particularly important in high-
security AI application areas such as autonomous driving, risk estimation, and
decision-making.

In this paper, we consider the underlying distribution behind predicting the
response variable because it reflects the confidence intervals of all levels.
For example, based on this distribution, we can calculate confidence intervals
at any level, coverage rate, and uncertainty quantification. Currently,
predicting the distribution of the response variable poses a challenge because
at a specific moment, the response variable can only take on a single
deterministic value. This point can be viewed as a maximum likelihood sample
from its underlying distribution, but it fails to reflect the overall state of
the underlying distribution.

Currently, the primary method used to address distribution prediction and
uncertainty quantification in regression and prediction tasks is frequentist
sampling. These methods involve sampling numerous samples by perturbing the
explanatory variable or model to approximate the underlying distribution of
the response variable. For instance, Bayesian neural networks (BNNs) simulate
this uncertainty by assuming that their parameters follow a Gaussian
distribution, thereby capturing the model’s uncertainty given the data
(wierstra2015weight). Similarly, ensemble-based methods have been proposed to
combine multiple deep models with random outputs to capture prediction
uncertainty. MC Dropout (gal2016dropout) demonstrates that enabling dropout
during each testing process yields results akin to model ensembling.
Additionally, models based on GANs and diffusion are introduced for
conditional density estimation and prediction uncertainty quantification.
These models utilize noise during the generation or diffusion process to
obtain different predicted values for estimating the uncertainty of the
response variable.

The common characteristic of these methods mentioned above is the requirement
of $K$ forward passes to sample $K$ representative samples. For example,
Bayesian framework-based methods require $K$ learnable parameter samples to be
inferred to obtain $K$ representative samples; ensemble methods require $K$
models to jointly infer; MC Dropout requires $K$ forward passes with random
dropout activation; generative models require $K$ forward or diffusion
processes. However, the excessive forward passes result in significant
computational overhead and slow speed, a drawback that becomes increasingly
apparent for AI applications with high real-time requirements.

Furthermore, several methods (Xie1994; salinas2020deepar; NEURIPS2020_NTK)
transform distribution prediction and uncertainty quantification into
predicting statistical variables such as mean and variance by assuming that
the response variable follows a known continuous distribution. For instance,
DeepAR (salinas2020deepar) assumes the response variable follows a Gaussian
distribution, thereby directly utilizing GaussianNLLLoss (Xie1994) to optimize
its mean and variance. These methods only predict statistical variables,
reducing the inference cost, but strong distribution assumptions often fail to
capture the true data distribution, resulting in inferior performance.

Additionally, there are methods that train models using quantiles as loss
functions to obtain specific quantile intervals (QIs) of the response
variable. For example, to predict the 5th and 95th percentile confidence
intervals, one can directly predict the corresponding quantiles. MRQNN
predicts specified quantiles using quantile loss. Similarly, TFT enables the
model to directly output three quantiles: 0.1, 0.5, and 0.9. These methods
cannot provide distribution information about the response variable but only
offer fixed QIs, thus lacking flexibility in their predictions.

Figure 1: DistPred can provide $K$ predicted values $\hat{y}$ of the response
variable $y$ given the predictor variable $x$ in a single forward process,
denoted as $\mathbb{E}(\hat{Y}|x)$, where $\hat{Y}$ represents a maximum
likelihood sample of $y$. Based on this sampling, the probability mass
distribution (PMD) $P_{\theta}(\hat{y}|x)$, cumulative distribution function
(PDF) $F_{\theta}(\hat{y}|x)$, and confidence curve $CC_{\theta}(\hat{y}|x)$
for the response variable $y$ can be computed, thereby yielding comprehensive
statistical insights into $y$. For instance, this includes confidence
intervals (CI) at any desired level, as well as p-values.

To address this issue, we propose a novel method called DistPred, which is a
distribution-free probabilistic inference method for regression and
forecasting tasks. DistPred is a simple and powerful method that can estimate
the distribution of the response variable in a single forward pass.
Specifically, we contemplate employing all predictive quantiles to specify the
potential cumulative density function (CDF) of the predictor variable, and we
show that the full quantiles’ prediction can be translated into calculating
the minimum expected score of the response variable and the predictive
ensemble variables. Based on this, we transform proper scoring rules that
measure the discrepancy between the predicted distribution and the target
distribution into a differentiable discrete form and use it as a loss function
to train the model end-to-end. This allows the model to sample numerous
samples in a single forward pass to estimate the potential distribution of the
response variable. DistPred is orthogonal to other methods, enabling its
combination with alternative approaches to enhance estimation performance.
Further, we show that DistPred can provide comprehensive statistical insights
into the response variable, including confidence intervals at any desired
level, p-values, and other statistical information, as shown in Figure 1.
Experimental results show that DistPred outperforms existing methods in terms
of both accuracy and computational efficiency. Specifically, DistPred has a
90x faster inference speed than state-of-the-art models.

##  2 Method

Assume that the dataset $D=\\{x_{i},y_{i}\\}_{i=1}^{N}$ consists of $N$
sample-label pairs. The subscript $i$ will be omitted if it does not cause
ambiguity in the context. Our objective is to utilize a machine learning model
$M$ with parameters $\theta$ to predict the underlying distribution $P(y)$ of
the response variable $y$ from $D$, aiming to acquire comprehensive
statistical insights such as obtaining confidence intervals (CI) and
quantifying uncertainty at any desired level.

Direct prediction of distribution $P$ is not feasible, because:

  1. 1.

Without distributional assumptions, we cannot give a valid representation of
the probability density function (PDF) or cumulative density function (CDF) of
the predicted distribution $P_{\theta}(\hat{y})$.

  2. 2.

We can only obtain a singular deterministic value for the response variable
$y$, lacking access to its distributional information to guide model learning.

Figure 2: Relationship between CDF and all predictive quantiles.

To address the aforementioned issues, we contemplate employing all predictive
quantiles $\hat{q}_{1},\hat{q}_{2},\cdots$, at levels
$\alpha_{1},\alpha_{2},\cdots$ to specify the potential CDF
$F_{\theta}(\hat{y})$ of the predictor variable $\hat{y}$. This is because if
we know the cumulative distribution function of a random variable, we can find
any quantile by setting $F(y)=q$. Conversely, if we have a complete set of
quantiles, we can approximate or reconstruct the cumulative distribution
function of the random variable. As shwon in Figure 2, quantiles provide
discrete ‘snapshots’ of the distribution, while the CDF is a continuous,
smooth version of these snapshots, offering a complete description of the
cumulative probability from the minimum to the maximum value.

Next, we will present that the full predictive quantiles outlined earlier
serves as a proper approximation of the response variable’s CDF. And the full
quantiles’ prediction can be translated into calculating the minimum expected
score of the response variable and the predictive ensemble variables
$\hat{Y}$, where $\hat{Y}=\\{\hat{y}_{1},\cdots,\hat{y}_{K}\\}$. Prior to
delving into this analysis, we shall initially introduce the scoring rules
utilized for assessing the propriety of predicted distributions.

###  2.1 Utilize proper scoring rules as the loss function

Scoring rules offer a concise measure for assessing probabilistic forecasts by
assigning numerical scores according to the forecast distribution and
predicted outcomes (gneiting2007strictly; jordan2017evaluating). Specifically,
Let $\Omega$ denote the set of possible values of the quantity of interest,
and let $\mathcal{P}$ denote a convex class of probability distributions on
$\Omega$. A scoring rule is a function

$S:\Omega\times\mathcal{P}\rightarrow\mathbb{R}\cup\\{\infty\\}$ (1)

that assigns numerical values to pairs of forecasts $P\in\mathcal{P}$ and
observations $y\in\Omega$. We identify probabilistic forecasts $P$ with the
associated CDF $F$ or PDF $f$, and consider scoring rules to be negatively
oriented, where a lower score signifies a more accurate forecast. A proper
scoring rule is optimized when the forecast aligns with the true distribution
of the observation, i.e., if

$E_{Y\sim Q}[S(Q,Y)]\leq E_{Y\sim Q}[S(P,Y)]$ (2)

for all $P,Q\in\mathcal{P}$. A scoring rule is termed strictly proper when
equality is achieved only when $P=Q$. Proper scoring rules (PSR) are essential
for comparative evaluation, particularly in ranking forecasts. In practice,
the forecaster with the lowest average score across multiple forecast cases
typically demonstrates the best predictive performance. Proper scoring rules
incentivize forecasters to accurately report their perception of the true
distribution in this scenario. Therefore, PSR provide attractive loss and
utility functions that can be tailored to a regression or forecast problem. To
estimate $\theta$, we might measure the goodness-of-fit by the mean score

$S_{n}(\theta)=\frac{1}{N}\sum_{i=1}^{N}S(P_{\theta}(\hat{y}_{i}),y_{i}).$ (3)

Let $\theta^{*}$ denotes the true parameter value, then asymptotic arguments
indicate that $\text{argmin}_{\theta}S_{n}(\theta)\rightarrow\theta^{*}$ as
$n\rightarrow\infty$. This suggests a general approach to transform PSR into
loss functions for training models, which implicitly minimizes the divergence
between predictive and true distributions.

###  2.2 Property of full predictive quantiles

We contemplate probabilistic predictions pertaining to a continuous quantity,
manifested as full predictive quantiles $\hat{q}_{1},\cdots,\hat{q}_{K}$. For
$P\in\mathcal{P}$, let $q_{1},\cdots,q_{K}$ denote the true $P$-quantiles at
levels $\alpha_{1},\cdots,\alpha_{K}\in(0,1)$. Then, the expected score
$S(q_{1},\cdots,q_{K};P)$ can be defined as

$S(\hat{q}_{1},\cdots,\hat{q}_{K};P)=\int
S(\hat{q}_{1},\cdots,\hat{q}_{K};y)\,\text{d}P(y).$ (4)

Further, a scoring rule S is proper if $S(q_{1},\cdots,q_{K};P)\geq
S(\hat{q}_{1},\cdots,\hat{q}_{K};P)$. Based on this, the following theorem is
holds (gneiting2007strictly; schervish2012characterization).

######  Theorem 1.

Assuming that $s_{k},i\in[1,\cdots K]$ is nondecreasing and $h$ is arbitrary,
then the scoring rule

$S(\hat{q}_{1},\cdots,\hat{q}_{K};P)=\sum_{k=1}^{K}\left(\alpha_{i}s_{k}(\hat{q}_{k})+(s_{k}(y)-s_{k}(\hat{q_{k}})\mathbbm{1}\\{y\leq\hat{q}_{k}\\})\right)$
(5)

is proper for predicting the quantiles at levels
$\alpha_{1},\cdots,\alpha_{K}$ when $K\rightarrow\infty$.
$\mathbbm{1}\\{y\leq\hat{q}_{k}\\}$ denotes the indicator function which is
one if $y\leq\hat{q}_{k}$ and zero otherwise.

Theorem 1 shows that full predictive quantiles is proper. In essence,
delineating a predictive CDF is equivalent to specifying all predictive
quantiles. Consequently, we can formulate scoring rules for the predictive
distribution based on the scoring rules for the quantiles. Specifically, let
$S_{\alpha}$ denotes a proper scoring rule for the quantile at level $\alpha$,
then the scoring rule

$S(F,y)=\int_{0}^{1}S_{\alpha}(F^{-1}(\alpha);y)\,\text{d}\alpha=\int_{-\infty}^{\infty}S(F(\hat{y}),\mathbbm{1}\\{y\leq\hat{y}\\})\,\text{d}\hat{y}$
(6)

is proper. Here, we can find that the right of Equation 6 corresponds to the
CRPS in which $S$ is the quadratic or Brier score, which is defined as

$C(F,y)=\int_{-\infty}^{\infty}(F(\hat{y})-\mathbbm{1}\\{y\leq\hat{y}\\})^{2}\,\text{d}\hat{y}.$
(7)

If the first moment of $F$ is finite, the CRPS can be written as

$C(F,y)=\mathbb{E}_{F}[|\hat{Y}-y|]-\frac{1}{2}\mathbb{E}_{FF}[|\hat{Y}-\hat{Y}^{\prime}|],$
(8)

where $\hat{Y}$ and $\hat{Y}^{\prime}$ denote independent predictive variables
with distribution $F$ (gneiting2007strictly).

Figure 3: The workflow of DistPred. An ensemble of predictive variables
$\hat{Y}$ is inferred in a forward pass and PSR $S(\mathbb{E}(\hat{Y}|x),y)$
is utilized to train the learner end-to-end.

###  2.3 End-to-end ensemble inference

Figure 4: DistPred’s architecture.

Based on the proof provided above, it is evident that predicting the full
quantiles is equivalent to minimizing Equation 8 w.r.t
$\mathbb{E}(\hat{Y}|y)$. Hence, as the workflow shown in Figure 3, we can
develop a model $M$ with parameters $\theta$ that infers an ensemble of
predictive variables $\hat{Y}$ in a forward pass and utilize Equation 8 to
train it end-to-end. This allows the model to sample numerous samples in a
single forward pass to estimate the empirical CDF $\hat{F}$ by the predictive
ensemble variables

$C(\hat{F},y)=\frac{1}{K}\sum_{k=1}^{K}|\hat{y}_{k}-y|-\frac{1}{2K^{2}}\sum_{k=1}^{K}\sum_{j=1}^{K}|\hat{y}_{k}-\hat{y}^{\prime}_{j}|.$
(9)

It’s worth noting that Equation 9 is a differentiable discrete form w.r.t
$\hat{Y}$ and $\hat{Y^{\prime}}$ that strictly satisfies PSR. However,
implementations of Equation 9 exhibit inefficiency due to their computational
complexity of $O(K^{2})$. This can be enhanced by employing representations
based on the generalized quantile function (laio2007verification) and the
sorted predictive ensemble variables

$C(\hat{F},y)=\frac{1}{2K^{2}}\sum_{k=1}^{K}(\overrightarrow{y_{k}}-y)(k\mathbbm{1}\\{y\leq\overrightarrow{y_{k}}\\}-i+\frac{1}{2}).$
(10)

Since a sorting operation is involved, the computational complexity of
Equation 10 is $O(KlogK)$. To conserve memory, we also suggest utilizing
Equation 10 as the loss function, as predicting ensemble variables in long-
term forecasting tasks may lead to out-of-memory issues on GPUs.

###  2.4 Incorporate alternative methodologies

DistPred is orthogonal to other methods, enabling its combination with
alternative approaches to enhance estimation performance. Here, with a focus
on computational efficiency and memory conservation, we opt to integrate MC
Dropout with DistPred, thereby denoting the amalgamation as DistPred-MCD. In
our experiments, we observed that DistPred-MCD can further enhance uncertainty
quantification performance, albeit with a marginal increase in computational
effort.

##  3 Experiments

In this paper, our focus centers on regression (proposed by
hernandez2015probabilistic) and prediction (proposed by Zhou2021Informer)
tasks, where we validate the application of the proposed DistPred method to
these specific endeavors.

###  3.1 PICP and QICE metrics

In recent years, BNNs (hernandez2015probabilistic; gal2016dropout) have
emerged as a class of models that aims at estimating the uncertainty providing
a more complete picture of $p(y|x)$. In (han2022card), it is contended that
the metric utilized in BNNs to assess uncertainty estimates, namely the
negative log-likelihood (NLL), is computed based on Gaussian densities. This
assumption implies that they consider both the conditional distribution
$p(y|x=x^{\prime})$ for all $x^{\prime}$ are Gaussian. However, this
assumption is very difficult to verify for real-world datasets. We follow CARD
(han2022card) and use the following two metrics, both of which are designed to
empirically evaluate the degree of similarity between learned and true
conditional distributions:

  * •

PICP (Prediction Interval Coverage Probability) (yao2019quality) is a metric
that measures the proportion of true labels that fall within the prediction
interval.

  * •

QICE (Quantile Interval Calibration Error) (han2022card) is a metric that
measures the average difference between the predicted and true quantiles at a
given level $\alpha$.

The PICP is calculated as

$PICP:=\frac{1}{N}\sum_{n=1}^{N}\mathbbm{1}\\{\hat{y}_{n}\geq
q_{\alpha/2}\\}\cdot\mathbbm{1}\\{\hat{y}_{n}\leq q_{1-\alpha/2}\\},$ (11)

where $q_{\alpha/2}$ and $q_{1-\alpha/2}$ represent the low and high
percentiles, respectively, that we have selected for the predicted $\hat{y}$
outputs given the same $x$ input. This metric evaluates the proportion of
accurate observations that lie within the percentile range of the generated
$\hat{y}$ samples corresponding to each $x$ input. In essence, with a strong
representation of the true distribution by the learned distribution, this
metric is expected to closely align with the discrepancy between the
designated low and high percentiles. Within this study, we opt for the $2.5$th
and $97.5$th percentiles, signifying that an optimal PICP value for the
acquired model would ideally reach 95%.

However, a caveat of the PICP metric becomes apparent in the measurement of
distribution differences. For instance, consider a situation where the 2.5th
to 97.5th percentile range of the learned distribution encompasses the data
falling between the 1st and 96th percentiles of the true distribution. Even
with a sufficient number of samples, the resulting PICP value would still
approach 95%. However, it becomes evident that a discrepancy exists between
the learned distribution and the true distribution.

Drawing from this reasoning, han2022card introduces a novel empirical metric
called QICE. This metric can be perceived as an enhanced version of PICP,
offering finer granularity and addressing the issue of uncovered quantile
ranges. To calculate QICE, the initial step involves generating an adequate
number of samples for each $x$ value. These samples are then divided into $M$
bins of approximately equal sizes. Subsequently, the quantile values are
determined at each boundary within these bins. The definition of QICE entails
computing the mean absolute error (MAE) between the proportion of true data
encompassed by each quantile interval and the optimal proportion, which is 1/M
for all intervals:

$QICE:=\frac{1}{M}\sum_{m=1}^{M}|r_{m}-\frac{1}{M}|,\quad\text{where}\ \
r_{m}=\frac{1}{N}\sum_{n=1}^{N}\mathbbm{1}\\{\hat{y}_{n}\geq
q_{\alpha/2}\\}\cdot\mathbbm{1}\\{\hat{y}_{n}\leq q_{1-\alpha/2}\\}.$ (12)

In this paper, we followed han2022card and set $M=10$ for all experiments.

###  3.2 Toy Examples

To demonstrate the effectiveness of DistPred, we initially conduct experiments
on $8$ toy examples as done in CARD (han2022card). These examples are
specifically crafted with distinct statistical characteristics in their data
generating functions: some have a uni-modal symmetric distribution for their
error term (linear regression, quadratic regression, sinusoidal regression),
others have heteroscedasticity (log-log linear regression, log-log cubic
regression) or multi-modality (inverse sinusoidal regression, 8 Gaussians,
full circle).

The research demonstrates that a trained DistPred models has the capability to
produce samples that closely resemble the true response variable for novel
covariates. Additionally, it can quantitatively match the true distribution
based on certain summary statistics. The study visualizes scatter plots
comparing real and generated data for all eight tasks in Figure 5. In cases
where the tasks involve unimodal conditional distributions, the interest
region fills the region between the $2.5$th and $97.5$th percentiles of the
generated $\hat{y}$ values.

We note that within every task, the generated samples seamlessly integrate
with the authentic test instances, indicating the potential of DistPred to
reconstruct the inherent data generation process. This experiment visually
demonstrates that DistPred effectively reconstructs the sample potential
distribution of the target response variable. This indicates that the
advantages of DistPred mentioned earlier can be fully harnessed in
distribution prediction.

Figure 5: Scatter plot of DistPred’s regression results on $8$ toy examples.

###  3.3 UCI Regression Tasks

For experiments conducted on real-world datasets, we utilize the same $10$ UCI
regression benchmark datasets (asuncion2007uci) and follow the experimental
protocol introduced by hernandez2015probabilistic, which has also been
followed by gal2016dropout and lakshminarayanan2017simple, as well as by
han2022card. The dataset information can be found in Table
LABEL:tb_reg_uci_dim located in Appendix LABEL:app_dataset.

We compare DistPred with other state-of-the-art methods, including PBP
(hernandez2015probabilistic), MC Dropout (gal2016dropout), DeepEnsemble
(lakshminarayanan2017simple), and another deep generative model that estimates
a conditional distribution sampler, GCDS (jointmatching), as well as diffusion
model, CARD (han2022card). The multiple train-test splits are applied with a
90%/10% ratio, following the same methodology as hernandez2015probabilistic
and han2022card (20 folds for all datasets except 5 for Protein and 1 for
Year). The reported metrics are presented as the mean and standard deviation
across all splits. As pointed out by han2022card, we compare the QICE of
different methods on various UCI datasets. Additional information regarding
the experimental setup for these models is available in Appendix
LABEL:app_implementation. The experimental results, along with corresponding
metrics, are presented in Table LABEL:tb_uci_qice. The frequency with which
each model achieves the best corresponding metric is reported in the
penultimate row, while the frequency with which it achieves the top two
positions is reported in the last row.

The results demonstrate that the DistPred method outperforms existing methods,
often by a considerable margin. It is worth noting that these impressive
results are achieved in a single forward pass of the DistPred method.
Crucially, the performance of uncertain quantization can be enhanced even
further by leveraging DistPred-MCD, a hybrid approach combining DistPred and
MC Dropout.

The implementation of DistPred on UCI regression tasks follows a
straightforward approach: We employ a basic multilayer perceptron (MLP) as the
foundational framework, complemented by Equation 9 serving as the loss
function for end-to-end training. Due to the fact that DistPred necessitates
solely a single forward inference, its inference speed is notably rapid. Table
1 presents a comparison of the training and inference speeds of mainstream
models. It should be noted that, for a fair comparison, the implementations of
various models are constructed on the same backbone and utilized the same
equipment. It is evident that DistPred is approximately $230$ times faster in
training and about $90$ times faster in inference compared to the state-of-
the-art model CARD. The inference speed of DistPred is slower than its
training speed because it involves calculating distribution statistical
metrics like QICE and PICP.

Table 1: Comparison of model training and inference times (minutes) on UCI
boston datasets.

Models | DistPred | DistPred-MCD | PBP | MC Dropout | CARD
---|---|---|---|---|---
Training | 0.035 $\pm$ 6E-3 | 0.035 $\pm$ 6E-3 | 0.04 $\pm$ 8E-3 | 0.031 $\pm$ 0.01 | 8.14$\pm$ 0.05
Inference | 0.095 $\pm$ 4E-3 | 0.027 $\pm$ 5E-3 | 5.23 $\pm$ 0.10 | 4.62 $\pm$ 0.06 | 8.31 $\pm$ 0.17

###  3.4 Ablation study of the number of samples and ensembles

We investigate the influence of the number of samples generated by DistPred,
as well as the number of ensembles of DistPred-MCD, on their respective
performances. As shown in Figure 6, with an increase in the number of output
samples and ensembles, the model’s performance shows a gradual improvement,
eventually reaching a point of saturation.

Figure 6: Ablation study of the number of samples in DistPred (a) and the
number of ensembles in DistPred-MCD (b).

###  3.5 Time Series Distribution Forecasting

We extend time series forecasting (Zhou2021Informer; wu2021autoformer;
zhou2022fedformer; liu2023itransformer) from point estimation to the task of
distribution prediction to infer about more statistical information about a
certain moment.

Baselines: We employ recent 10 SOTA methods for comparisons, including
iTransformer liu2023itransformer, PatchTST nie2022time, SCINet liu2022scinet,
TimesNet wu2022timesnet, DLinear zeng2023transformers, FEDformer
zhou2022fedformer, Autoformer wu2021autoformer, Informer Zhou2021Informer,
LogTrans li2019enhancing and Reformer Kitaev2020Reformer. We use the same
experimental setup as (Zhou2021Informer) and (liu2022scinet) and follow the
same experimental protocol as (Zhou2021Informer). Univariate results can be
found in Appendix LABEL:app_univariate.

Datasets and Setting: The detailed information pertaining to the datasets can
be located in Appendix LABEL:app_dataset. The models used in the experiments
are evaluated over a wide range of prediction lengths to compare performance
on different future horizons: 96, 192, 336 and 720. The experimental settings
are the same for both multivariate and univariate tasks. We use the average of
the MSE and MAE ($\frac{MSE+MAE}{2}$) to evaluate the overall performance of
the model. It is noteworthy that DistPred provides an ensemble $\hat{Y}$ of
response variables. Consequently, we employ the mean value of $\hat{Y}$ as the
point estimate at that moment.

Main Results: The results for multivariate TS forecasting are outlined in
Table LABEL:tb2, with the optimal results highlighted in bold and the second-
best results emphasized with underlined. It can be found that, despite not
utilizing MSE and MAE, DistPred achieves state-of-the-art performance across
all datasets and prediction length configurations. iTransformer and PatchTST
stand out as the latest models acknowledged for their exceptional average
performance. Compared with them, the proposed DistPred demonstrates an average
performance increase of 3.5% and 16.5%, respectively, achieving a substantial
performance improvement. We provide metrics, e.g., CRPS, QICE, PICP, for
comparison by the future research community.

###  3.6 Visualization of the predictive distribution

The predictive distribution of DistPred is visualized in Figure 7. It can be
observed that DistPred gives a family of predictions. Given all predictive
ensemble values, the model can estimate the distribution of the response
variable. Therefore, we can calculate confidence intervals at different
levels, as shwon in the right subplot of Figure 7.

Figure 7: Visualization of the prediction results and the confidence
intervals.

##  4 Conclusion

In this paper, we propose a novel method called DistPred, which is a
distribution-free probabilistic inference method for regression and
forecasting tasks. DistPred is a simple and powerful method that can estimate
the distribution of the response variable in a single forward pass.
Specifically, we transform proper scoring rules that measure the discrepancy
between the predicted distribution and the target distribution into a
differentiable discrete form and use it as a loss function to train the model
end-to-end. This allows the model to sample numerous samples in a single
forward pass to estimate the potential distribution of the response variable.
We also propose a hybrid method called DistPred-MCD, which combines DistPred
with MC Dropout to further improve the performance of uncertain quantization.
We conduct experiments on both toy examples and real-world datasets, and the
results demonstrate that DistPred outperforms existing methods, often by a
considerable margin. We also extend time series forecasting from point
estimation to distribution prediction and achieve state-of-the-art performance
on multivariate time series forecasting tasks. In the future, we plan to
extend DistPred to other tasks, such as classification and reinforcement
learning.

## NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine
learning research, addressing issues of reproducibility, transparency,
research ethics, and societal impact. Do not remove the checklist: The papers
not including the checklist will be desk rejected. The checklist should follow
the references and follow the (optional) supplemental material. The checklist
does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to
answer these questions. For each question in the checklist:

  * •

You should answer [Yes] , [No] , or [N/A] .

  * •

[N/A]  means either that the question is Not Applicable for that particular
paper or the relevant information is Not Available.

  * •

Please provide a short (1–2 sentence) justification right after your answer
(even for NA).

The checklist answers are an integral part of your paper submission. They are
visible to the reviewers, area chairs, senior area chairs, and ethics
reviewers. You will be asked to also include it (after eventual revisions)
with the final version of your paper, and its final version will be published
with the paper.

The reviewers of your paper will be asked to use the checklist as one of the
factors in their evaluation. While "[Yes] " is generally preferable to "[No]
", it is perfectly acceptable to answer "[No] " provided a proper
justification is given (e.g., "error bars are not reported because it would be
too computationally expensive" or "we were unable to find the license for the
dataset we used"). In general, answering "[No] " or "[N/A] " is not grounds
for rejection. While the questions are phrased in a binary way, we acknowledge
that the true answer is often more nuanced, so please just use your best
judgment and write a justification to elaborate. All supporting evidence can
appear either in the main paper or the supplemental material, provided in
appendix. If you answer [Yes]  to a question, in the justification please
point to the section(s) where related material for the question can be found.

IMPORTANT, please:

  * •

Delete this instruction block, but keep the section heading “NeurIPS paper
checklist",

  * •

Keep the checklist subsection headings, questions/answers and guidelines
below.

  * •

Do not modify the questions and only use the provided macros for your answers.

  1. 1.

Claims

  2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

  3. Answer: [TODO]

  4. Justification:

  5. Guidelines:

     * •

The answer NA means that the abstract and introduction do not include the
claims made in the paper.

     * •

The abstract and/or introduction should clearly state the claims made,
including the contributions made in the paper and important assumptions and
limitations. A No or NA answer to this question will not be perceived well by
the reviewers.

     * •

The claims made should match theoretical and experimental results, and reflect
how much the results can be expected to generalize to other settings.

     * •

It is fine to include aspirational goals as motivation as long as it is clear
that these goals are not attained by the paper.

  6. 2.

Limitations

  7. Question: Does the paper discuss the limitations of the work performed by the authors?

  8. Answer: [TODO]

  9. Justification:

  10. Guidelines:

     * •

The answer NA means that the paper has no limitation while the answer No means
that the paper has limitations, but those are not discussed in the paper.

     * •

The authors are encouraged to create a separate "Limitations" section in their
paper.

     * •

The paper should point out any strong assumptions and how robust the results
are to violations of these assumptions (e.g., independence assumptions,
noiseless settings, model well-specification, asymptotic approximations only
holding locally). The authors should reflect on how these assumptions might be
violated in practice and what the implications would be.

     * •

The authors should reflect on the scope of the claims made, e.g., if the
approach was only tested on a few datasets or with a few runs. In general,
empirical results often depend on implicit assumptions, which should be
articulated.

     * •

The authors should reflect on the factors that influence the performance of
the approach. For example, a facial recognition algorithm may perform poorly
when image resolution is low or images are taken in low lighting. Or a speech-
to-text system might not be used reliably to provide closed captions for
online lectures because it fails to handle technical jargon.

     * •

The authors should discuss the computational efficiency of the proposed
algorithms and how they scale with dataset size.

     * •

If applicable, the authors should discuss possible limitations of their
approach to address problems of privacy and fairness.

     * •

While the authors might fear that complete honesty about limitations might be
used by reviewers as grounds for rejection, a worse outcome might be that
reviewers discover limitations that aren’t acknowledged in the paper. The
authors should use their best judgment and recognize that individual actions
in favor of transparency play an important role in developing norms that
preserve the integrity of the community. Reviewers will be specifically
instructed to not penalize honesty concerning limitations.

  11. 3.

Theory Assumptions and Proofs

  12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

  13. Answer: [TODO]

  14. Justification:

  15. Guidelines:

     * •

The answer NA means that the paper does not include theoretical results.

     * •

All the theorems, formulas, and proofs in the paper should be numbered and
cross-referenced.

     * •

All assumptions should be clearly stated or referenced in the statement of any
theorems.

     * •

The proofs can either appear in the main paper or the supplemental material,
but if they appear in the supplemental material, the authors are encouraged to
provide a short proof sketch to provide intuition.

     * •

Inversely, any informal proof provided in the core of the paper should be
complemented by formal proofs provided in appendix or supplemental material.

     * •

Theorems and Lemmas that the proof relies upon should be properly referenced.

  16. 4.

Experimental Result Reproducibility

  17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

  18. Answer: [TODO]

  19. Justification:

  20. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

If the paper includes experiments, a No answer to this question will not be
perceived well by the reviewers: Making the paper reproducible is important,
regardless of whether the code and data are provided or not.

     * •

If the contribution is a dataset and/or model, the authors should describe the
steps taken to make their results reproducible or verifiable.

     * •

Depending on the contribution, reproducibility can be accomplished in various
ways. For example, if the contribution is a novel architecture, describing the
architecture fully might suffice, or if the contribution is a specific model
and empirical evaluation, it may be necessary to either make it possible for
others to replicate the model with the same dataset, or provide access to the
model. In general. releasing code and data is often one good way to accomplish
this, but reproducibility can also be provided via detailed instructions for
how to replicate the results, access to a hosted model (e.g., in the case of a
large language model), releasing of a model checkpoint, or other means that
are appropriate to the research performed.

     * •

While NeurIPS does not require releasing code, the conference does require all
submissions to provide some reasonable avenue for reproducibility, which may
depend on the nature of the contribution. For example

       1. (a)

If the contribution is primarily a new algorithm, the paper should make it
clear how to reproduce that algorithm.

       2. (b)

If the contribution is primarily a new model architecture, the paper should
describe the architecture clearly and fully.

       3. (c)

If the contribution is a new model (e.g., a large language model), then there
should either be a way to access this model for reproducing the results or a
way to reproduce the model (e.g., with an open-source dataset or instructions
for how to construct the dataset).

       4. (d)

We recognize that reproducibility may be tricky in some cases, in which case
authors are welcome to describe the particular way they provide for
reproducibility. In the case of closed-source models, it may be that access to
the model is limited in some way (e.g., to registered users), but it should be
possible for other researchers to have some path to reproducing or verifying
the results.

  21. 5.

Open access to data and code

  22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

  23. Answer: [TODO]

  24. Justification:

  25. Guidelines:

     * •

The answer NA means that paper does not include experiments requiring code.

     * •

Please see the NeurIPS code and data submission guidelines
(https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.

     * •

While we encourage the release of code and data, we understand that this might
not be possible, so “No” is an acceptable answer. Papers cannot be rejected
simply for not including code, unless this is central to the contribution
(e.g., for a new open-source benchmark).

     * •

The instructions should contain the exact command and environment needed to
run to reproduce the results. See the NeurIPS code and data submission
guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more
details.

     * •

The authors should provide instructions on data access and preparation,
including how to access the raw data, preprocessed data, intermediate data,
and generated data, etc.

     * •

The authors should provide scripts to reproduce all experimental results for
the new proposed method and baselines. If only a subset of experiments are
reproducible, they should state which ones are omitted from the script and
why.

     * •

At submission time, to preserve anonymity, the authors should release
anonymized versions (if applicable).

     * •

Providing as much information as possible in supplemental material (appended
to the paper) is recommended, but including URLs to data and code is
permitted.

  26. 6.

Experimental Setting/Details

  27. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

  28. Answer: [TODO]

  29. Justification:

  30. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The experimental setting should be presented in the core of the paper to a
level of detail that is necessary to appreciate the results and make sense of
them.

     * •

The full details can be provided either with the code, in appendix, or as
supplemental material.

  31. 7.

Experiment Statistical Significance

  32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

  33. Answer: [TODO]

  34. Justification:

  35. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The authors should answer "Yes" if the results are accompanied by error bars,
confidence intervals, or statistical significance tests, at least for the
experiments that support the main claims of the paper.

     * •

The factors of variability that the error bars are capturing should be clearly
stated (for example, train/test split, initialization, random drawing of some
parameter, or overall run with given experimental conditions).

     * •

The method for calculating the error bars should be explained (closed form
formula, call to a library function, bootstrap, etc.)

     * •

The assumptions made should be given (e.g., Normally distributed errors).

     * •

It should be clear whether the error bar is the standard deviation or the
standard error of the mean.

     * •

It is OK to report 1-sigma error bars, but one should state it. The authors
should preferably report a 2-sigma error bar than state that they have a 96%
CI, if the hypothesis of Normality of errors is not verified.

     * •

For asymmetric distributions, the authors should be careful not to show in
tables or figures symmetric error bars that would yield results that are out
of range (e.g. negative error rates).

     * •

If error bars are reported in tables or plots, The authors should explain in
the text how they were calculated and reference the corresponding figures or
tables in the text.

  36. 8.

Experiments Compute Resources

  37. Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

  38. Answer: [TODO]

  39. Justification:

  40. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The paper should indicate the type of compute workers CPU or GPU, internal
cluster, or cloud provider, including relevant memory and storage.

     * •

The paper should provide the amount of compute required for each of the
individual experimental runs as well as estimate the total compute.

     * •

The paper should disclose whether the full research project required more
compute than the experiments reported in the paper (e.g., preliminary or
failed experiments that didn’t make it into the paper).

  41. 9.

Code Of Ethics

  42. Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

  43. Answer: [TODO]

  44. Justification:

  45. Guidelines:

     * •

The answer NA means that the authors have not reviewed the NeurIPS Code of
Ethics.

     * •

If the authors answer No, they should explain the special circumstances that
require a deviation from the Code of Ethics.

     * •

The authors should make sure to preserve anonymity (e.g., if there is a
special consideration due to laws or regulations in their jurisdiction).

  46. 10.

Broader Impacts

  47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

  48. Answer: [TODO]

  49. Justification:

  50. Guidelines:

     * •

The answer NA means that there is no societal impact of the work performed.

     * •

If the authors answer NA or No, they should explain why their work has no
societal impact or why the paper does not address societal impact.

     * •

Examples of negative societal impacts include potential malicious or
unintended uses (e.g., disinformation, generating fake profiles,
surveillance), fairness considerations (e.g., deployment of technologies that
could make decisions that unfairly impact specific groups), privacy
considerations, and security considerations.

     * •

The conference expects that many papers will be foundational research and not
tied to particular applications, let alone deployments. However, if there is a
direct path to any negative applications, the authors should point it out. For
example, it is legitimate to point out that an improvement in the quality of
generative models could be used to generate deepfakes for disinformation. On
the other hand, it is not needed to point out that a generic algorithm for
optimizing neural networks could enable people to train models that generate
Deepfakes faster.

     * •

The authors should consider possible harms that could arise when the
technology is being used as intended and functioning correctly, harms that
could arise when the technology is being used as intended but gives incorrect
results, and harms following from (intentional or unintentional) misuse of the
technology.

     * •

If there are negative societal impacts, the authors could also discuss
possible mitigation strategies (e.g., gated release of models, providing
defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms
to monitor how a system learns from feedback over time, improving the
efficiency and accessibility of ML).

  51. 11.

Safeguards

  52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

  53. Answer: [TODO]

  54. Justification:

  55. Guidelines:

     * •

The answer NA means that the paper poses no such risks.

     * •

Released models that have a high risk for misuse or dual-use should be
released with necessary safeguards to allow for controlled use of the model,
for example by requiring that users adhere to usage guidelines or restrictions
to access the model or implementing safety filters.

     * •

Datasets that have been scraped from the Internet could pose safety risks. The
authors should describe how they avoided releasing unsafe images.

     * •

We recognize that providing effective safeguards is challenging, and many
papers do not require this, but we encourage authors to take this into account
and make a best faith effort.

  56. 12.

Licenses for existing assets

  57. Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

  58. Answer: [TODO]

  59. Justification:

  60. Guidelines:

     * •

The answer NA means that the paper does not use existing assets.

     * •

The authors should cite the original paper that produced the code package or
dataset.

     * •

The authors should state which version of the asset is used and, if possible,
include a URL.

     * •

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

     * •

For scraped data from a particular source (e.g., website), the copyright and
terms of service of that source should be provided.

     * •

If assets are released, the license, copyright information, and terms of use
in the package should be provided. For popular datasets,
paperswithcode.com/datasets has curated licenses for some datasets. Their
licensing guide can help determine the license of a dataset.

     * •

For existing datasets that are re-packaged, both the original license and the
license of the derived asset (if it has changed) should be provided.

     * •

If this information is not available online, the authors are encouraged to
reach out to the asset’s creators.

  61. 13.

New Assets

  62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

  63. Answer: [TODO]

  64. Justification:

  65. Guidelines:

     * •

The answer NA means that the paper does not release new assets.

     * •

Researchers should communicate the details of the dataset/code/model as part
of their submissions via structured templates. This includes details about
training, license, limitations, etc.

     * •

The paper should discuss whether and how consent was obtained from people
whose asset is used.

     * •

At submission time, remember to anonymize your assets (if applicable). You can
either create an anonymized URL or include an anonymized zip file.

  66. 14.

Crowdsourcing and Research with Human Subjects

  67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

  68. Answer: [TODO]

  69. Justification:

  70. Guidelines:

     * •

The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.

     * •

Including this information in the supplemental material is fine, but if the
main contribution of the paper involves human subjects, then as much detail as
possible should be included in the main paper.

     * •

According to the NeurIPS Code of Ethics, workers involved in data collection,
curation, or other labor should be paid at least the minimum wage in the
country of the data collector.

  71. 15.

Institutional Review Board (IRB) Approvals or Equivalent for Research with
Human Subjects

  72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

  73. Answer: [TODO]

  74. Justification:

  75. Guidelines:

     * •

The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.

     * •

Depending on the country in which research is conducted, IRB approval (or
equivalent) may be required for any human subjects research. If you obtained
IRB approval, you should clearly state this in the paper.

     * •

We recognize that the procedures for this may vary significantly between
institutions and locations, and we expect authors to adhere to the NeurIPS
Code of Ethics and the guidelines for their institution.

     * •

For initial submissions, do not include any information that would break
anonymity (if applicable), such as the institution conducting the review.