Table 21: Average MSE and NLL on weather tasks, with $m=10$ and $\gamma=0.5$.

task | TempMin | TempMax | Humidity
---|---|---|---
metric | MSE | NLL | MSE | NLL | MSE | NLL
MTNP-A | 0.0067 $\pm$ 0.0007 | -1.0773 $\pm$ 0.0231 | 0.0119 $\pm$ 0.0012 | -0.8658 $\pm$ 0.0405 | 0.0745 $\pm$ 0.0033 | -0.0471 $\pm$ 0.0278
MTNP-P | 0.0046 $\pm$ 0.0003 | -1.1614 $\pm$ 0.0129 | 0.0071 $\pm$ 0.0002 | -1.0166 $\pm$ 0.0145 | 0.0581 $\pm$ 0.0029 | -0.1463 $\pm$ 0.0232
MTNP-SA | 0.0039 $\pm$ 0.0001 |  -1.1881 $\pm$ 0.0074 | 0.0058 $\pm$ 0.0005 | -1.0990 $\pm$ 0.0203 |  0.0542 $\pm$ 0.0024 |  -0.2083 $\pm$ 0.0283
MTNP-SP |  0.0037 $\pm$ 0.0001 | -1.1832 $\pm$ 0.0165 |  0.0054 $\pm$ 0.0001 |  -1.1049 $\pm$ 0.0154 | 0.0546 $\pm$ 0.0021 | -0.1006 $\pm$ 0.0696

task | Precip | Cloud | Dew
---|---|---|---
metric | MSE | NLL | MSE | NLL | MSE | NLL
MTNP-A | 0.2514 $\pm$ 0.0049 | 0.6655 $\pm$ 0.0212 | 0.2578 $\pm$ 0.0070 | 0.7186 $\pm$ 0.0369 | 0.0114 $\pm$ 0.0012 | -0.9103 $\pm$ 0.0316
MTNP-P | 0.2400 $\pm$ 0.0072 |  0.6205 $\pm$ 0.0210 | 0.2360 $\pm$ 0.0070 |  0.6532 $\pm$ 0.0166 | 0.0083 $\pm$ 0.0001 | -1.0183 $\pm$ 0.0039
MTNP-SA |  0.2262 $\pm$ 0.0041 | 0.6270 $\pm$ 0.0288 | 0.2242 $\pm$ 0.0042 | 0.6825 $\pm$ 0.0280 | 0.0074 $\pm$ 0.0004 |  -1.0409 $\pm$ 0.0200
MTNP-SP | 0.2276 $\pm$ 0.0028 | 0.6557 $\pm$ 0.0433 |  0.2215 $\pm$ 0.0043 | 0.6660 $\pm$ 0.0141 |  0.0073 $\pm$ 0.0003 | -1.0331 $\pm$ 0.0147

To investigate whether the self-attention modules operate as desired in per-
task attention stage, we visualize the attention weights assigned to each
context point. For each task, we compute an averaged attention weight placed
on a set of context points by averaging dimensions of the attention map of
shape $(nL,nH,nQ,nK)$ to $(1,1,1,nK)$ where $nL$ and $nH$ denote the number of
self-attention layers and heads in each layer, and $nQ=nK$ refer to the number
of query ($Q$) and key ($K$) vectors and $Q=K$. Note that this averaged
attention weight can be interpreted as the averaged importance put on each of
the context points by the self-attention module. Results are visualized in
Figure 24. As shown in the figure, points located at representative positions
or at sparse regions are attended more than others. This shows that self-
attention in per-task paths operates as desired.

Figure 24: Visualization of per-task attention weights of MTNP assigned on
different context points in synthetic tasks. Size and color of markers
represent the amount of attention weight assigned on each point.

###  H.2 Effect of Parameter-Sharing in Per-Task Branches

In this study, we explore the effect of parameter-sharing the per-task encoder
networks and decoder networks in STNP and MTNP. The variants of STNP and MTNP
are as follows. (1) STNP-S: STNP with shared encoder and decoder, (2) STNP-TS:
STNP with task-specific encoders and decoders, (3) MTNP-S: MTNP with shared
encoder and decoder in per-task branches, (4) MTNP-TS: MTNP with task-specific
encoders and decoders in per-task branches. Note that we consistently use
STNP-TS and MTNP-S architectures for the 1D experiments (synthetic and
weather) and STNP-TS and MTNP-TS architectures for the 2D experiments (CelebA)
in Section 5

As can be seen in the tables, in the weather dataset, we observe that the
models with parameter sharing (STNP-S & MTNP-S) show comparable performances
to their respective non-sharing baselines (STNP-TS & MTNP-TS). On the other
hand, the parameter sharing technique slightly improves the performances of
the models in the 1-D synthetic case. We conjecture that the utilization of
the same architecture and its parameter for all tasks acts as a good inductive
bias to the models considering that all tasks share the same global parameter
a,b,c,w. Nonetheless, we still find that MTNPs consistently outperform STNPs
regardless of whether the parameters are shared or not, validating the
effectiveness of MTNP to capture and exploit functional correlation for multi-
task learning problems.

Table 22: Average normalized MSE on synthetic tasks, with varying context size
($m$) and $\gamma=0.5$.

task | Sine | Tanh
---|---|---
$m$ | 5 | 10 | 20 | 5 | 10 | 20
STNP-S | 0.5387 $\pm$ 0.0270 | 0.2596 $\pm$ 0.0351 | 0.0845 $\pm$ 0.0216 | 0.1255 $\pm$ 0.0143 | 0.0459 $\pm$ 0.0095 | 0.0123 $\pm$ 0.0019
STNP-TS | 0.5212 $\pm$ 0.0157 | 0.2609 $\pm$ 0.0382 | 0.0993 $\pm$ 0.0182 | 0.1307 $\pm$ 0.0134 | 0.0468 $\pm$ 0.0074 | 0.0159 $\pm$ 0.0028
MTNP-S |  0.2636 $\pm$ 0.0105 |  0.1137 $\pm$ 0.0078 |  0.0485 $\pm$ 0.0034 |  0.0435 $\pm$ 0.0047 |  0.0115 $\pm$ 0.0021 |  0.0040 $\pm$ 0.0002
MTNP-TS | 0.2901 $\pm$ 0.0164 | 0.1297 $\pm$ 0.0079 | 0.0532 $\pm$ 0.0050 | 0.0618 $\pm$ 0.0069 | 0.0172 $\pm$ 0.0041 | 0.0063 $\pm$ 0.0006

task | Sigmoid | Gaussian
---|---|---
$m$ | 5 | 10 | 20 | 5 | 10 | 20
STNP-S | 0.0177 $\pm$ 0.0034 | 0.0050 $\pm$ 0.0009 | 0.0012 $\pm$ 0.0003 | 0.0801 $\pm$ 0.0124 | 0.0371 $\pm$ 0.0048 | 0.0181 $\pm$ 0.0028
STNP-TS | 0.0203 $\pm$ 0.0034 | 0.0067 $\pm$ 0.0013 | 0.0025 $\pm$ 0.0005 | 0.0799 $\pm$ 0.0098 | 0.0409 $\pm$ 0.0041 | 0.0222 $\pm$ 0.0042
MTNP-S |  0.0066 $\pm$ 0.0019 |  0.0014 $\pm$ 0.0001 |  0.0006 $\pm$ 0.0001 |  0.0360 $\pm$ 0.0018 |  0.0132 $\pm$ 0.0008 |  0.0069 $\pm$ 0.0012
MTNP-TS | 0.0093 $\pm$ 0.0015 | 0.0031 $\pm$ 0.0010 | 0.0014 $\pm$ 0.0001 | 0.0426 $\pm$ 0.0025 | 0.0173 $\pm$ 0.0019 | 0.0100 $\pm$ 0.0006

Table 23: Average MSE and NLL on weather tasks, with $m=10$ and $\gamma=0.5$.

task | TempMin | TempMax | Humidity
---|---|---|---
metric | MSE | NLL | MSE | NLL | MSE | NLL
STNP-S | 0.0049 $\pm$ 0.0003 | -1.1381 $\pm$ 0.0223 | 0.0066 $\pm$ 0.0006 | -1.0527 $\pm$ 0.0241 | 0.0621 $\pm$ 0.0069 | 0.0240 $\pm$ 0.1210
STNP-TS | 0.0046 $\pm$ 0.0004 | -1.1514 $\pm$ 0.0181 | 0.0069 $\pm$ 0.0004 | -1.0390 $\pm$ 0.0106 | 0.0632 $\pm$ 0.0072 | 0.1273 $\pm$ 0.1898
MTNP-S | 0.0037 $\pm$ 0.0001 |  -1.1832 $\pm$ 0.0165 | 0.0054 $\pm$ 0.0001 |  -1.1049 $\pm$ 0.0154 | 0.0546 $\pm$ 0.0021 |  -0.1006 $\pm$ 0.0696
MTNP-TS |  0.0036 $\pm$ 0.0002 | -1.1818 $\pm$ 0.0076 |  0.0053 $\pm$ 0.0003 | -1.0885 $\pm$ 0.0246 |  0.0519 $\pm$ 0.0013 | -0.0662 $\pm$ 0.0828

task | Precip | Cloud | Dew
---|---|---|---
metric | MSE | NLL | MSE | NLL | MSE | NLL
STNP-S | 0.2675 $\pm$ 0.0104 | 0.9537 $\pm$ 0.1347 | 0.2629 $\pm$ 0.0043 | 0.7847 $\pm$ 0.0503 | 0.0084 $\pm$ 0.0007 | -0.9877 $\pm$ 0.0312
STNP-TS | 0.2607 $\pm$ 0.0082 | 1.1242 $\pm$ 0.2362 | 0.2631 $\pm$ 0.0044 | 0.8563 $\pm$ 0.0637 | 0.0086 $\pm$ 0.0008 | -0.9815 $\pm$ 0.0283
MTNP-S | 0.2276 $\pm$ 0.0028 |  0.6557 $\pm$ 0.0433 |  0.2215 $\pm$ 0.0043 |  0.6660 $\pm$ 0.0141 | 0.0073 $\pm$ 0.0003 |  -1.0331 $\pm$ 0.0147
MTNP-TS |  0.2187 $\pm$ 0.0043 | 0.7213 $\pm$ 0.0953 | 0.2253 $\pm$ 0.0100 | 0.6663 $\pm$ 0.0283 |  0.0071 $\pm$ 0.0003 | -1.0232 $\pm$ 0.0144

###  H.3 Effect of Latent and Deterministic Encoders

In this study, we compare STNP, JTNP, and MTNP without using deterministic
encoder, each corresponds to STNP-L, JTNP-L, and MTNP-L in the table. Note
that these variants correspond to the direct NP implementations of
STNP/JTNP/MTNP rather than ANP. The results are provided in the tables below.
The overall trends are the same as the models with deterministic encoder,
which demonstrates that the effectiveness of MTNP does not depend on a
specific choice of architecture (vanilla NP or ANP).

Table 24: Average normalized MSE on synthetic tasks, with varying context size
($m$) and $\gamma=0.5$.

task | Sine | Tanh
---|---|---
$m$ | 5 | 10 | 20 | 5 | 10 | 20
STNP-L | 0.5615 $\pm$ 0.0199 | 0.3006 $\pm$ 0.0407 | 0.1254 $\pm$ 0.0159 | 0.1362 $\pm$ 0.0119 | 0.0585 $\pm$ 0.0109 | 0.0234 $\pm$ 0.0020
S+JTNP-L | 0.4151 $\pm$ 0.0273 | 0.2560 $\pm$ 0.0184 | 0.1395 $\pm$ 0.0106 | 0.1055 $\pm$ 0.0171 | 0.0506 $\pm$ 0.0054 | 0.0224 $\pm$ 0.0023
MTNP-L |  0.3003 $\pm$ 0.0147 |  0.1543 $\pm$ 0.0130 |  0.0755 $\pm$ 0.0080 |  0.0563 $\pm$ 0.0069 |  0.0162 $\pm$ 0.0020 |  0.0067 $\pm$ 0.0003

task | Sigmoid | Gaussian
---|---|---
$m$ | 5 | 10 | 20 | 5 | 10 | 20
STNP-L | 0.0213 $\pm$ 0.0025 | 0.0086 $\pm$ 0.0008 | 0.0045 $\pm$ 0.0006 | 0.0809 $\pm$ 0.0101 | 0.0405 $\pm$ 0.0063 | 0.0234 $\pm$ 0.0026
S+JTNP-L | 0.0201 $\pm$ 0.0030 | 0.0106 $\pm$ 0.0008 | 0.0061 $\pm$ 0.0003 | 0.0687 $\pm$ 0.0056 | 0.0376 $\pm$ 0.0014 | 0.0227 $\pm$ 0.0019
MTNP-L |  0.0096 $\pm$ 0.0022 |  0.0027 $\pm$ 0.0005 |  0.0012 $\pm$ 0.0002 |  0.0417 $\pm$ 0.0037 |  0.0169 $\pm$ 0.0010 |  0.0091 $\pm$ 0.0007

Table 25: Average MSE and NLL on weather tasks, with $m=10$ and $\gamma=0.5$.

task | TempMin | TempMax | Humidity
---|---|---|---
metric | MSE | NLL | MSE | NLL | MSE | NLL
STNP-L | 0.0065 $\pm$ 0.0010 | -1.0783 $\pm$ 0.0207 | 0.0102 $\pm$ 0.0008 | -0.9202 $\pm$ 0.0205 | 0.0828 $\pm$ 0.0067 | 0.0341 $\pm$ 0.0401
S+JTNP-L | 0.0085 $\pm$ 0.0016 | -1.0632 $\pm$ 0.0252 | 0.0136 $\pm$ 0.0021 | -0.8601 $\pm$ 0.0236 | 0.0966 $\pm$ 0.0108 | 0.0737 $\pm$ 0.0699
MTNP-L |  0.0046 $\pm$ 0.0005 |  -1.1580 $\pm$ 0.0167 |  0.0072 $\pm$ 0.0005 |  -1.0203 $\pm$ 0.0171 |  0.0596 $\pm$ 0.0027 |  -0.1554 $\pm$ 0.0274

task | Precip | Cloud | Dew
---|---|---|---
metric | MSE | NLL | MSE | NLL | MSE | NLL
STNP-L | 0.3381 $\pm$ 0.0026 | 0.8792 $\pm$ 0.0008 | 0.3411 $\pm$ 0.0049 | 0.8798 $\pm$ 0.0066 | 0.0106 $\pm$ 0.0005 | -0.8804 $\pm$ 0.0118
S+JTNP-L | 0.2832 $\pm$ 0.0095 | 0.7347 $\pm$ 0.0179 | 0.2851 $\pm$ 0.0123 | 0.7692 $\pm$ 0.0111 | 0.0127 $\pm$ 0.0007 | -0.8827 $\pm$ 0.0220
MTNP-L |  0.2432 $\pm$ 0.0046 |  0.6267 $\pm$ 0.0220 |  0.2372 $\pm$ 0.0056 |  0.6659 $\pm$ 0.0163 |  0.0085 $\pm$ 0.0003 |  -1.0006 $\pm$ 0.0065

We also compare MTNP with deterministic encoder only, which corresponds to
MTNP-D in the table below. To emphasize the benefits of generative modeling of
MTNPs, we include MTNP evaluated on the best sample among 25 predictive
samples, which corresponds to MTNP-best in the table. We observe that MTNP and
MTNP-D are comparable in synthetic and weather datasets, which seems
reasonable as we designed the deterministic encoder to mimic the latent
encoder of MTNP (_e.g._ , they employ both per-task and across-task
inferences). However, we can see that MTNP-best clearly outperforms MTNP-D,
which implies that MTNP can generate more accurate samples while MTNP-D
cannot.

Table 26: Average normalized MSE on synthetic tasks, with varying context size
($m$) and $\gamma=0.5$.

task | Sine | Tanh
---|---|---
$m$ | 5 | 10 | 20 | 5 | 10 | 20
MTNP-D | 0.1963 $\pm$ 0.0039 | 0.0985 $\pm$ 0.0059 | 0.0447 $\pm$ 0.0034 | 0.0423 $\pm$ 0.0079 | 0.0110 $\pm$ 0.0015 | 0.0042 $\pm$ 0.0007
MTNP | 0.2636 $\pm$ 0.0105 | 0.1137 $\pm$ 0.0078 | 0.0485 $\pm$ 0.0034 | 0.0435 $\pm$ 0.0047 | 0.0115 $\pm$ 0.0021 | 0.0040 $\pm$ 0.0002
MTNP-best |  0.1239 $\pm$ 0.0087 |  0.0491 $\pm$ 0.0068 |  0.0176 $\pm$ 0.0032 |  0.0169 $\pm$ 0.0028 |  0.0032 $\pm$ 0.0009 |  0.0009 $\pm$ 0.0001

task | Sigmoid | Gaussian
---|---|---
$m$ | 5 | 10 | 20 | 5 | 10 | 20
MTNP-D | 0.0073 $\pm$ 0.0011 | 0.0015 $\pm$ 0.0004 | 0.0006 $\pm$ 0.0002 | 0.0281 $\pm$ 0.0018 | 0.0127 $\pm$ 0.0015 | 0.0073 $\pm$ 0.0012
MTNP | 0.0066 $\pm$ 0.0019 | 0.0014 $\pm$ 0.0001 | 0.0006 $\pm$ 0.0001 | 0.0360 $\pm$ 0.0018 | 0.0132 $\pm$ 0.0008 | 0.0069 $\pm$ 0.0012
MTNP-best |  0.0025 $\pm$ 0.0007 |  0.0004 $\pm$ 0.0001 |  0.0002 $\pm$ 0.0000 |  0.0146 $\pm$ 0.0021 |  0.0046 $\pm$ 0.0006 |  0.0021 $\pm$ 0.0005

Table 27: Average MSE and NLL on weather tasks, with $m=10$ and $\gamma=0.5$.

task | TempMin | TempMax | Humidity
---|---|---|---
metric | MSE | NLL | MSE | NLL | MSE | NLL
MTNP-D | 0.0039 $\pm$ 0.0001 | -1.1892 $\pm$ 0.0079 | 0.0059 $\pm$ 0.0002 | -1.0944 $\pm$ 0.0168 | 0.0535 $\pm$ 0.0023 | -0.2142 $\pm$ 0.0359
MTNP | 0.0037 $\pm$ 0.0001 | -1.1832 $\pm$ 0.0165 | 0.0054 $\pm$ 0.0001 | -1.1049 $\pm$ 0.0154 | 0.0546 $\pm$ 0.0021 | -0.1006 $\pm$ 0.0696
MTNP-best |  0.0032 $\pm$ 0.0001 |  -1.2143 $\pm$ 0.0118 |  0.0047 $\pm$ 0.0001 |  -1.1441 $\pm$ 0.0111 |  0.0489 $\pm$ 0.0019 |  -0.2320 $\pm$ 0.0460

task | Precip | Cloud | Dew
---|---|---|---
metric | MSE | NLL | MSE | NLL | MSE | NLL
MTNP-D | 0.2257 $\pm$ 0.0035 | 0.6034 $\pm$ 0.0315 | 0.2260 $\pm$ 0.0038 | 0.6536 $\pm$ 0.0130 | 0.0077 $\pm$ 0.0003 | -1.0434 $\pm$ 0.0150
MTNP | 0.2276 $\pm$ 0.0028 | 0.6557 $\pm$ 0.0433 | 0.2215 $\pm$ 0.0043 | 0.6660 $\pm$ 0.0141 | 0.0073 $\pm$ 0.0003 | -1.0331 $\pm$ 0.0147
MTNP-best |  0.2160 $\pm$ 0.0026 |  0.5491 $\pm$ 0.0269 |  0.2091 $\pm$ 0.0041 |  0.6044 $\pm$ 0.0090 |  0.0065 $\pm$ 0.0003 |  -1.0763 $\pm$ 0.0102

###  H.4 Effect of Task Embeddings

In this study, we compare to different types of task embeddings for MTNP.
MTNP-Onehot uses one-hot encoded vector for task embedding $e^{t}$ while MTNP-
learnable uses learnable vector for the task embedding. Note that we
consistently use MTNP-learnable for the 1D experiments in Section 5

Table 28: Average normalized MSE on synthetic tasks, with varying context size
($m$) and $\gamma=0.5$.

task | Sine | Tanh
---|---|---
$m$ | 5 | 10 | 20 | 5 | 10 | 20
MTNP-Onehot |  0.2507 $\pm$ 0.0174 |  0.1047 $\pm$ 0.0123 |  0.0426 $\pm$ 0.0075 | 0.0440 $\pm$ 0.0050 |  0.0110 $\pm$ 0.0014 |  0.0036 $\pm$ 0.0005
MTNP-Learnable | 0.2636 $\pm$ 0.0105 | 0.1137 $\pm$ 0.0078 | 0.0485 $\pm$ 0.0034 |  0.0435 $\pm$ 0.0047 | 0.0115 $\pm$ 0.0021 | 0.0040 $\pm$ 0.0002

task | Sigmoid | Gaussian
---|---|---
$m$ | 5 | 10 | 20 | 5 | 10 | 20
MTNP-Onehot | 0.0067 $\pm$ 0.0013 |  0.0014 $\pm$ 0.0003 |  0.0005 $\pm$ 0.0001 |  0.0338 $\pm$ 0.0017 |  0.0130 $\pm$ 0.0009 |  0.0062 $\pm$ 0.0008
MTNP-Learnable |  0.0066 $\pm$ 0.0019 |  0.0014 $\pm$ 0.0001 | 0.0006 $\pm$ 0.0001 | 0.0360 $\pm$ 0.0018 | 0.0132 $\pm$ 0.0008 | 0.0069 $\pm$ 0.0012

Table 29: Average MSE and NLL on weather tasks, with $m=10$ and $\gamma=0.5$.

task | TempMin | TempMax | Humidity
---|---|---|---
metric | MSE | NLL | MSE | NLL | MSE | NLL
MTNP-Onehot |  0.0036 $\pm$ 0.0002 |  -1.1975 $\pm$ 0.0088 | 0.0055 $\pm$ 0.0001 | -1.1046 $\pm$ 0.0090 |  0.0526 $\pm$ 0.0014 | -0.0835 $\pm$ 0.1149
MTNP-Learnable | 0.0037 $\pm$ 0.0001 | -1.1832 $\pm$ 0.0165 |  0.0054 $\pm$ 0.0001 |  -1.1049 $\pm$ 0.0154 | 0.0546 $\pm$ 0.0021 |  -0.1006 $\pm$ 0.0696

task | Precip | Cloud | Dew
---|---|---|---
metric | MSE | NLL | MSE | NLL | MSE | NLL
MTNP-Onehot |  0.2265 $\pm$ 0.0034 | 0.7003 $\pm$ 0.0482 | 0.2244 $\pm$ 0.0044 | 0.6909 $\pm$ 0.0264 |  0.0070 $\pm$ 0.0004 |  -1.0479 $\pm$ 0.0155
MTNP-Learnable | 0.2276 $\pm$ 0.0028 |  0.6557 $\pm$ 0.0433 |  0.2215 $\pm$ 0.0043 |  0.6660 $\pm$ 0.0141 | 0.0073 $\pm$ 0.0003 | -1.0331 $\pm$ 0.0147

We observe that MTNP with one-hot embedding is comparable to MTNP with
learnable embedding. To further investigate the effect of learnable embedding,
we visualize the learned task embedding by MTNP using the t-SNE algorithm. We
include the visualization results in Figure 25 and 26. As shown in the figure,
we find that the learned task embeddings are well-separated from each other
and uniformly distributed on the embedding space. From the observations, we
conjecture that well-separated task embeddings are sufficient task information
for MTNP.

Figure 25: t-SNE plot (with 2 components) of the learned task embeddings of
MTNP in synthetic tasks. Figure 26: t-SNE plot (with 2 components) of the
learned task embeddings of MTNP in weather tasks.