# EMOE: Expansive Matching of Experts for Robust Uncertainty Based Rejection

Yunni Qu
Department of Computer Science
University of North Carolina at Chapel Hill
201 S Columbia St, Chapel Hill, NC 27599
<EMAIL_ADDRESS>&James Wellnitz
Eshelman School of Pharmacy
University of North Carolina at Chapel Hill
301 Pharmacy Lane, Chapel Hill, NC 27599
<EMAIL_ADDRESS>
Alexander Tropsha
Eshelman School of Pharmacy
University of North Carolina at Chapel Hill
301 Pharmacy Lane, Chapel Hill, NC 27599
<EMAIL_ADDRESS>
&Junier Oliva
Department of Computer Science
University of North Carolina at Chapel Hill
201 S Columbia St, Chapel Hill, NC 27599
<EMAIL_ADDRESS>
###### Abstract

Expansive Matching of Experts (EMOE) is a novel method that utilizes support-
expanding, extrapolatory pseudo-labeling to improve prediction and uncertainty
based rejection on out-of-distribution (OOD) points. We propose an expansive
data augmentation technique that generates OOD instances in a latent space,
and an empirical trial based approach to filter out augmented expansive points
for pseudo-labeling. EMOE utilizes a diverse set of multiple base experts as
pseudo-labelers on the augmented data to improve OOD performance through a
shared MLP with multiple heads (one per expert). We demonstrate that EMOE
achieves superior performance compared to state-of-the-art methods on tabular
data.

##  1 Introduction

Figure 1: _Mockup_. Our approach trains a set of diverse base experts on
training data. After, we consider a novel augmentation to expand the
distributional support in a latent space. Then, using a shared neural network
(MLP) with multiple heads (one per expert) we train the network to match the
expert labels on the expanded data.

It is well-known that the generalization capabilities of models can be
severely limited when tested on out-of-distribution (OOD) data that deviates
from the distribution seen at training time (Torralba and Efros, 2011; Liu et
al., 2021; Freiesleben and Grote, 2023). This in turn affects many real-world
applications where models may be evaluated on distribution-shifted data during
deployment. For instance, these issues commonly arise in medical applications
where patient distributions at inference time may deviate from the training
data (Lee et al., 2023). A potential strategy for the safe deployment of
models in real-world applications is to employ novelty-based rejection
(Dubuisson and Masson, 1993; Hendrickx et al., 2024), where predictions are
rejected whenever the model is evaluated on an instance that deviates from the
data distribution seen during training. While such an approach is appropriate
in certain scenarios (for example, whenever it is expected that a human will
be in the loop and thus can easily intervene upon rejection), this prevalent
strategy is overly-conservative as it foregoes any potential
extrapolation111We use the term extrapolation to loosely encompass prediction
outside of the training data distribution support (without consideration of
the data’s convex-hull). by design. That is, novelty-rejection forbids any
form of extrapolation (predictions outside of the training data support), even
when the model may be capable. Instead, in this work we attempt to _better
assess_ (and improve) the limits of models’ ability to extrapolate based on
dataset augmentation and self-training on a set of experts (see Fig. 1).

(a) Poor Extrapolative Rejection Wastes Resources

(b) Filtration on Extrapolative Data.

Figure 2: _Virtual screening_. (a) Current application of ML models to virtual
screening frequently yields unreliable predictions that will waste resources
on unsuccessful experiments (Kimber et al., 2021). (b) We seek reliable
extrapolatory predictions (✓) to enable more economical use of experimental
resources.

Broader Impacts As a motivating application, consider the use of ML models for
scientific _discovery_. In such discovery applications, ML models should, by
definition, characterize data that is novel and distinct from what has been
previously characterized. These, however, are exactly the sort of inferences
that are disallowed by novelty-rejection methods, which essentially forbid any
potential characterization of instances that expand the support of
characterized data (_regardless of a model’s capability to do so_). Take, for
instance, drug discovery (a driving application for this paper) where one
hopes to predict desirable properties of molecules that are quite distinct
from molecules that have been previously characterized. (I.e., “scaffold
hoping” (Hu et al., 2017), leveraging existing data to discover a desirable
molecule with significantly different chemical structure.) Novelty-based
rejection would reject any prediction on molecules that do not fall firmly in
the support of what has been previously characterized and used for model
training. Therefore, these typical rejection techniques only allow predictions
on molecules that are similar to those already characterized in the training
set (akin to the notion of ‘interpolation’) preventing their utility in
discovering structurally novel molecules. Instead, we wish gain a better
understanding of _what_ OOD (‘extrapolatory’) instances may be well predicted
by our model. In drug discovery, the use of models with poor confidence
filtration will result in _false positives_ that waste resources on
unsuccessful experiments where screened molecules do not present desirable
properties (see Fig. 2); hence, it is key that high-confidence predictions on
OOD instances correspond with accuracy (i.e., that confidence filtration on
OOD instances leads to higher-quality predictions).

Contributions Unfortunately, in-distribution confidence based measurements
fail to properly characterize model capabilities on OOD data (Arjovsky et al.,
2019; Creager et al., 2020). In this work we show that we can improve
confidence-based filtration of predictions (as measured by area under
precision-recall and receiver operating characteristic curves, AURPC/AUROC) on
OOD instances with a novel training scheme of a multi-headed network based on
matching with augmented (expanded) data (see Fig. 1, Alg. 1). In particular,
our contributions are as follows: 1) we propose a novel expansive data
augmentation technique that generates OOD instances in a latent space; 2) we
propose a novel empirical trial based approach to filter out augmented
expansive points for psuedo-labeling; 3) we develop a straight-forward but
effective strategy that yields a strong, diverse set of base-experts for self-
training; 4) we develop our novel EMOE approach for training a multi-headed
network; 5) we show state-of-the-art (SOTA) performance in rejecting
predictions via estimated confidences via AUPRC-based metrics.

Algorithm 1 Expansive Matching of Experts (EMOE) Approach

1:Learn a latent space.

2:Train a set of base experts.

3:Expand training set with data that falls beyond the support in the latent
space.

4:Train a network with multiple heads (one per base expert) matching the
experts on expanded and original training dataset.

5:Infer with a combination of the base experts and network heads.

##  2 Related Work

#### Domain Generalization

Domain generalization (DG) aims to learn a model that is able to generalize to
multiple domains. A typical approach is to learn a domain invariant
representation across multiple source domains. Domain invariant representation
learning can be done by minimizing variations in feature distributions (Li et
al., 2018; Ding et al., 2022) and imposing a regularizer to balance between
predictive power and invariance (Arjovsky et al., 2019; Koyama and Yamaguchi,
2020). Another line of research incorporates data augmentation to improve
generalizability. Basic transformations like rotation and translation, varying
in magnitude, are commonly used on images to diversify the training data
(Cubuk et al., 2019; Berthelot et al., 2020). More sophisticated augmentation
techniques have recently surfaced: (Zhang et al., 2018) introduced mixup,
which linearly combines two training samples; (Yun et al., 2019) proposed
CutMix, blending two images by replacing a cutout patch with a patch from
another image; (Zhong et al., 2022) adversarially augment images to prevent
over fitting to source domains. We focus on augmentations that are general and
applicable across modalities.

#### Self-Training

Self-training aims to utilize an earlier model as a pseudo-labeler to populate
the training data with more labelled instances by labelling unlabelled data at
each iteration. Then, the new labelled set is combined with the previous
training set to train a new model. The concept of pseudo-labeling was
initially proposed by (Lee, 2013), suggesting a straightforward approach of
retaining instances where the teacher model has high prediction probabilities.
Following (Lee, 2013; Zou et al., 2018) proposed to use a proportion of the
most confident unlabelled data points instead of a fixed threshold.
Researchers then combined curriculum learning with pseudo labeling, where
thresholds for acquiring unlabeled data for each class is dynamically adjusted
at different time steps, allowing the most informative unlabeled data to be
incorporated (Cascante-Bonilla et al., 2020; Zhang et al., 2021). Another line
of research improves robustness of pseudo-labeling by encouraging diversity in
the pseudo-labeler. In their work, Ghosh et al. (2021) employed an ensemble of
models as teacher to provide pseudo-labels for the student model. On the other
hand, (Xie et al., 2019) injected noise into the pseudo-labeler model by
incorporating Dropout (Srivastava et al., 2014) and data augmentation
techniques to provide more robust pseudo-labels. In this work we show how to
utilize multiple pseudo-labelers on extrapolatory augmented data to improve
OOD performance.

#### Selective Classification

Reject option methods (also known as selective classification) aim to identify
instances where the model should not predict. Many selective classification
approaches rely on a post-training processing strategy. Following this
strategy, once the model has finished training, a rejection metric is
computed. Then, predictions are rejected or accepted based on a predefined
threshold. A simple choice for a rejection metric is to utilize the
conditional output probability from ML models (Stefano et al., 2000; Fumera et
al., 2000). Building upon these works, (Devries and Taylor, 2018) proposed to
train a confidence branch alongside of the prediction branch by incentivizing
a neural network to produce a confidence measure during training; Geifman and
El-Yaniv (2017) proposed a method for constructing a probability-calibrated
selective classifier with guaranteed control over the true risk. Recently,
methods adopting end-to-end training approaches have been proposed
(Thulasidasan et al., 2019) (Ziyin et al., 2019) (Geifman and El-Yaniv, 2019).
In these works, an extra class is added when predictions are made. If the
extra class has the highest class probability for a sample, the sample is
rejected. Most reject-option approaches are geared towards in-distribution
rejection and utilize novelty-rejection when encountering any OOD points
(Torralba and Efros, 2011; Liu et al., 2021; Freiesleben and Grote, 2023);
instead, we propose to learn better conditional output probabilities on OOD
data for more effective, capability-aware rejection.

#### Ensemble Modeling

Ensemble techniques aim to utilize a diverse set of models jointly for better
performance. Early methodologies for ensembles aggregate (bag) predictions
from all models (Dietterich, 2007)(Kussul et al., 2010) or a subset of the
models in the ensemble (Jordan and Jacobs, 1993), (Eigen et al., 2013). In the
OOD setting, prior works addressed this problem by enforcing prediction
diversity on OOD data (Pagliardini et al., 2022), ensembling moving average
models (Arpit et al., 2022a), and training an ensemble of domain specific
classifiers (Yao et al., 2023). EMOE adopts a multi-headed architecture that
produces an ensemble to improve predictions on OOD data.

##  3 Method

Training Data Throughout, we assume the ‘ _single-source_ ’ generalization
setting (Qiao et al., 2020), where we observe a single in-distribution (ID)
training dataset $\mathcal{D}=\\{(x_{i},y_{i})\\}_{i=1}^{N}$, and instances
are drawn _iid_ $(x_{i},y_{i})\sim\mathcal{P}_{\mathrm{in}}$ _without_ any
accompanying environmental/domain/source information. For simplicity, we write
to the binary classification case, $y_{i}\in\\{0,1\\}$, but our methodology is
easily extendable to other supervised tasks. We design our method to work in
general, non-modality specific222In particular, we avoid any modality or
domain-specific augmentation of data. (e.g., image, text, audio) settings such
as the real-valued case $x_{i}\in\mathbb{R}^{d}$.

Base Collection of Experts EMOE leverages a set of diverse initial experts
$\\{g_{k}\\}_{k=1}^{K}$ to guide the training of a secondary model. There are
many mixture of experts (Jordan and Jacobs, 1993), (Eigen et al., 2013) (Du et
al., 2021) and ensembling (Arpit et al., 2022a) (Dietterich, 2007)
(Pagliardini et al., 2022) (Yao et al., 2023) methods available; we observe
good empirical performance (see Sec. 4) using a collection of strong base-
learners trained on uniform sub-selections of instances _and_ features (akin
to the construction of a random forest).

###  3.1 Expansive Augmentation of Training Data

Figure 3: Expansion in latent space: points (black) are augmented (gray) and
expand the distributional support.

We begin with the simple intuition that if we want to improve extrapolatory
performance of models, then we should consider training signals on instances
that lie outside of the original data support. While there has been much
recent attention in strong augmentations to improve OOD performance in
modality-specific settings (Zhong et al., 2022) (Xie et al., 2019), performing
such augmentations on general data remains a challenge. To reason about the
support of the training data, and how to _expand_ past it, we propose to
leverage a latent factor space, $\varphi:\mathbb{R}^{d}\mapsto\mathbb{R}^{s}$.
While learning semantically meaningful latent factor spaces remains an active
area of research, we observed strong performance utilizing autoencoding
techniques (see Sec. 4), which carry a corresponding decoder
$\gamma:\mathbb{R}^{s}\mapsto\mathbb{R}^{d}$. Without loss of generality, we
consider centered latent spaces such that $\mathbb{E}[\varphi(X)]=0$.

We propose a novel, yet straightforward strategy to expand data outside of
training distributional support: perturb instances to lie further away from
the origin in latent space. In particular, if we have latent vector
$z=\varphi(x)$, we propose to consider perturbations of the form
$z^{\prime}=(1+|\epsilon|)z$ where $\epsilon\sim\mathcal{N}(0,1)$, and one can
utilize the decoder $x^{\prime}=\gamma(z^{\prime})$. That is, we define our
expansion operation on a set of points as:

$\textbf{Ex}(\\{x_{i}\\}_{i=1}^{N})\equiv\left\\{\gamma\left((1+|\epsilon_{i}|)\varphi(x_{i})\right)\mid\epsilon_{i}\sim\mathcal{N}(0,1)\right\\}_{i=1}^{N}.$
(1)

Ex will be a _stochastic_ mapping. Clearly, the perturbed latent codes will
tend away from areas of support (see Fig. 3). However, unlike with small
jitter-based perturbations, where one can retain an original instance label,
it is less clear how to derive an accompanying training signal for expansive
augmentations. Here, we propose to leverage a pseudo-labeling scheme where we
derive $K$ labels with the base experts
$(f_{1}(x^{\prime}),\ldots,f_{K}(x^{\prime}))$333One may also train base
experts directly on the latent space,
$(f_{1}(z^{\prime}),\ldots,f_{K}(z^{\prime}))$, and avoid the decoder.. We
expound on utilizing the base expert labels below.

###  3.2 Trustworthy Expansive Signals with Extrapolatory Directional Mining

Recent works have noted that confidence based filtration of pseudo-labels
improves self-training techniques (Lee, 2013; Sohn et al., 2020). Here we
present a novel, complementary approach to filter out pseudo-labels on
expansive augmented data (as above). We wish to filter out expansive
augmentations where predicted labels may not be accurate or helpful. Of
course, predicting the quality of extrapolatory performance is in itself a
difficult task, since we do no have access to any OOD data in our setting. Our
approach is based on empirical _non-iid_ held-out trials to judge the
difficulty of labeling extrapolatory instances in some direction (in latent
space), and keep track of directions that are easiest to extrapolate to for
later augmentation. That is, for a given direction on the hyper-sphere
$v\in\mathbb{S}^{s-1}$, we withhold the training instances that have the
highest projections w.r.t. $v$ (are in the highest $q$-th quantile),
$\mathcal{T}^{\mathrm{held}}_{v}\equiv\\{(x,y)\in\mathcal{D}\mid\varphi(x)^{T}v\geq
t_{v,q}\\},$ where $t_{v,q}$ is the threshold of the $q$-th quantile for
projections of training latent vectors onto $v$. The remaining points in the
training set are then used to train a model for our trial on direction $v$:
$\mathcal{T}^{\mathrm{train}}_{v}\equiv\mathcal{D}\setminus\mathcal{T}^{\mathrm{held}}_{v}.$
We train a simple parametric model (e.g., a linear logistic-regression model)
on $\mathcal{T}^{\mathrm{train}}_{v}$, $f_{v}$ and evaluate a performance
metric, $\rho$ (e.g., accuracy, F1-score, etc.), on the held-out trial data
$\rho(\mathcal{T}^{\mathrm{held}}_{v},f_{v})$. We take well-performing trials
(based on $\rho(\mathcal{T}^{\mathrm{held}}_{v},f_{v})$) as evidence that
extrapolation to extreme values in $v$ is possible with the data, and hence
accrue instances in $\mathcal{T}^{\mathrm{held}}_{v}$ into a large collection
$\mathcal{E}$ for pseudo-labeling with the base experts (see Alg. 2 for
details).

Algorithm 2 Extrapolatory Directional Mining

1:procedure
get_directional_expansion_points($\varphi,M,T,q,\mathcal{D}$)$\triangleright$
get quality points to expand, $\mathcal{E}$, based on top $M$ (of $T$)
performing (based on metric $\rho$) directional empirical held-out trials
withholding the $q$-th percentile of projections in latent space $\varphi$.

2: heap.init()

3: for $j\in\ \\{1,\ldots,T\\}$ do

4: $v\leftarrow\mathrm{Unif}(\\{\frac{x}{\|x\|}\mid(x,y)\in\mathcal{D}\\})$
$\triangleright$ random direction

5:
$t_{v,q}\leftarrow\texttt{quantile}(\\{\varphi(x)^{T}v\mid(x,y)\in\mathcal{D}\\},q)$

6:
$\mathcal{T}^{\mathrm{held}}_{v}\leftarrow\\{(x,y)\in\mathcal{D}\mid\varphi(x)^{T}v\geq
t_{v,q}\\}$ $\triangleright$ withhold extreme points on direction

7:
$\mathcal{T}^{\mathrm{train}}_{v}\leftarrow\mathcal{D}\setminus\mathcal{T}^{\mathrm{held}}_{v}.$
$\triangleright$ train on rest

8:
$f_{v}\leftarrow\texttt{model.fit(}\mathcal{T}^{\mathrm{train}}_{v}\texttt{)}$

9:
$\texttt{heap.push(}\rho(\mathcal{T}^{\mathrm{held}}_{v},f_{v}),\mathcal{T}^{\mathrm{held}}_{v}\texttt{)}$
$\triangleright$ order based on performance

10: end for

11: $\mathcal{E}=[\,]$

12: for $j\in\ \\{1,\ldots,M\\}$ do $\triangleright$ get points in top $M$
trials

13: $\mathcal{T}\leftarrow\texttt{heap.pop()}$

14: $\mathcal{E}\texttt{.append(}\\{x\mid(x,y)\in\mathcal{T}\\}\texttt{)}$

15: end for

16: return $\mathcal{E}$

17:end procedure

###  3.3 Matching Network with Experts on Expansive Data

We propose learning a multi-headed network based on self-training with base
experts’ predictions on the (filtered) expanded set of training data,
$\mathcal{E}$. That is, we propose learning a network composed of a shared
multilayer perceptron (MLP), $\phi:\mathbb{R}^{d}\mapsto\mathbb{R}^{m}$, and
$K$ expert matching heads, $h_{1},\ldots,h_{K}$; e.g., mapping to logit space
for binary classification, $h_{j}:\mathbb{R}^{m}\mapsto\mathbb{R}$. Our loss
incorporates a per head loss that matches experts on a set $\mathcal{S}$:

$\mathcal{L}_{\mathrm{match}}(\phi,\\{h_{j}\\}_{j=1}^{K},\\{g_{j}\\}_{j=1}^{K};\mathcal{S})\equiv\frac{1}{|\mathcal{S}|K}\sum_{x\in
S}\sum_{j=1}^{K}\ell(h_{j}(\phi(x)),g_{j}(x)),$ (2)

where $\ell(\hat{y},y)$ is a supervised loss (e.g., the cross-entropy loss).
Moreover, we will utilize a mean-matching L1 loss

$\mathcal{L}_{\mathrm{mean}}(\phi,\\{h_{j}\\}_{j=1}^{K},\\{g_{j}\\}_{j=1}^{K};\mathcal{S})\equiv\frac{1}{|\mathcal{S}|}\sum_{x\in
S}\left\|\frac{1}{K}\sum_{j=1}^{K}\sigma(h_{j}(\phi(x)))-\frac{1}{K}\sum_{j=1}^{K}g_{j}(x))\right\|_{1},$
(3)

where $\sigma(\cdot)$ is the sigmoid. Our full expansive matching of experts
loss is then:

$\displaystyle\mathcal{L}_{\mathrm{EMOE}}(\phi,\\{h_{j}\\}_{j=1}^{K},\\{g_{j}\\}_{j=1}^{K};\mathcal{D},\textbf{Ex}(\mathcal{E}))\equiv$
$\displaystyle\
\mathcal{L}_{\mathrm{mean}}(\phi,\\{h_{j}\\}_{j=1}^{K},\\{g_{j}\\}_{j=1}^{K};\mathcal{D})$
(4)
$\displaystyle+\mathcal{L}_{\mathrm{match}}(\phi,\\{h_{j}\\}_{j=1}^{K},\\{g_{j}\\}_{j=1}^{K};\mathcal{D})$
(5)
$\displaystyle+\lambda\,\mathcal{L}_{\mathrm{match}}(\phi,\\{h_{j}\\}_{j=1}^{K},\\{g_{j}\\}_{j=1}^{K};\textbf{Ex}(\mathcal{E})),$
(6)

where $\mathcal{E}$ is our set of points to expand (e.g., using Alg. 2). Note
that we provide additional supervisory losses on non-augmented $\mathcal{D}$
via $\mathcal{L}_{\mathrm{mean}}$. In practice, we considered simple linear
heads. At an intuitive level, this forces the MLP to learn a robust feature
embedding that can ‘mimic’ the diverse views that the base experts provide.
Empirical results show (see Sec. 4) that the network heads learn an effective
(often better) estimator than the base expects. However, we see more
consistent improvements by not forgetting the base experts and bagging as:

$f_{\mathrm{EMOE}}(x)\equiv\frac{1}{2K}\sum_{j=1}^{K}g_{j}(x)+h_{j}(\phi(x)).$
(7)

Motivation Below we include high-level hypotheses on how the EMOE approach may
learn better estimates on OOD data through _variance reduction and
regularization_. Previous work has decomposed OOD generalization into
bias/variance terms (Yang et al., 2020; Arpit et al., 2022b):

$\mathbb{E}_{(x,y)\sim\mathcal{P}_{\mathrm{out}}}\mathbb{E}_{\mathcal{D}\sim\mathcal{P}_{\mathrm{in}}}[\mathrm{CE}(y,f(x;\mathcal{D}))]=\mathbb{E}_{(x,y)}[\mathrm{CE}(y,\bar{f}(x))]+\mathbb{E}_{x,\mathcal{D}}[\mathrm{KL}(\bar{f}(x),f(x;\mathcal{D}))]$
(8)

where $\mathrm{CE}$ is the cross-entropy loss, $f(x;\mathcal{T})$ is the model
fit on dataset $\mathcal{T}$,
$\bar{f}(x)=\mathbb{E}_{\mathcal{D}}[f(x;\mathcal{D})]$ is the expected
prediction when averaging out draws on the (in-distribution) training dataset
$\mathcal{D}$, and $\mathcal{P}_{\mathrm{out}}$ is the OOD data distribution
at inference time. Letting
$\bar{g}(x)\equiv\frac{1}{K}\sum_{j=1}^{K}g_{j}(x)$, we may view $\bar{g}(x)$
as a bootstrap-like estimate for $\bar{f}(x)$. One may then take
$\mathbb{E}_{x,\mathcal{D}}[\mathrm{KL}(\bar{g}(x),f(x;\mathcal{D}))]$ as a
proxy for
$\mathbb{E}_{x,\mathcal{D}}[\mathrm{KL}(\bar{f}(x),f(x;\mathcal{D}))]$ and
roughly consider

$\mathbb{E}_{(x,y)\sim\mathcal{P}_{\mathrm{out}}}\mathbb{E}_{\mathcal{D}\sim\mathcal{P}_{\mathrm{in}}}[\mathrm{CE}(y,f(x;\mathcal{D}))]\approx\mathbb{E}_{(x,y)}[\mathrm{CE}(y,\bar{f}(x))]+\mathbb{E}_{x,\mathcal{D}}[\mathrm{KL}(\bar{g}(x),f(x;\mathcal{D}))],$
(9)

which connects to (5) when interpreting our expanded points as a proxy for the
OOD distribution $\mathcal{P}_{\mathrm{out}}$ and
$\mathcal{L}_{\mathrm{match}}(\phi,\\{h_{j}\\}_{j=1}^{K},\\{g_{j}\\}_{j=1}^{K};\textbf{Ex}(\mathcal{E}))$
as a proxy for
$\mathbb{E}_{x,\mathcal{D}}[\mathrm{KL}(\bar{g}(x),f(x;\mathcal{D}))]$.

##  4 Experiments

We conduct experiments on _tabular_ data to test the OOD generalizability of
EMOE. We considered the _single source_ domain generalization setting (e.g.,
(Qiao et al., 2020)), where our model is trained solely on ID data without any
(labeled or unlabelled) OOD data during training/validation, and without any
accompanying environmental/domain/source information from ID training
instances. Moreover, we note that we avoided utilizing any modality-specific
information in EMOE (e.g., we do not utilize any domain specific
augmentations) for generality. We utilized XGB Classifiers (Chen and Guestrin,
2016) fitted to random subsets of data instances and features as the base
collection of experts. For a fair/realistic evaluation, we avoided any hyper-
parameter tuning on EMOE and utilized a fixed architecture of a $2$ layer
$512$ ELU (Clevert et al., 2015) hidden-unit MLP with $1024$ linear-output
heads (please see other hyperparameters in Supp. Mat.A.1). For our latent
space, we utilize PCA with $128$ components. While OOD generalization is an
active field of research (Freiesleben and Grote, 2023; Liu et al., 2021),
methodology for general (non-modality specific) single source domain
generalization is more limited. Given that EMOE integrates numerous
techniques, we provide context for our results and compared EMOE with existing
strong domain generalization methods that approach the problem from various
perspectives and strategies (and are applicable in the single-source setting).
In our experiments, we include two baselines that utilize data augmentation:
AdvStyle (Zhong et al., 2022) and Mixup (Zhang et al., 2018), as well as two
baselines employing ensemble methods: D-BAT(Pagliardini et al., 2022) and
EoA(Arpit et al., 2022a). Mixup linearly combines two ID samples, AdvStyle
adversarially augments ID data, D-BAT enforces prediction diversity on OOD
data, and EoA ensembles moving average models. For prediction thresholding
(rejection), we directly utilize the conditional probability $P(Y=1\mid X=x)$
generated by the models. Many real-world applications (e.g. in drug discovery
and virtual screen, see Fig. 2) shall utilize only high-confidence
predictions. Thus, alongside AUPRC and AUROC, we paid close attention to high-
confidence filtration and reported percent of AUPRC at conservative recall
thresholds (e.g., ‘$\mathrm{AURPC@R}\\!\\!\leq\\!\\!.2$’). This metric is
computed as the area under the PR-curve up to the specified recall threshold
and dividing by the maximum possible area for that threshold (i.e., the
threshold). We report both base expert (‘EMOE Base’) and EMOE (7) ensemble
performance. Our implementation shall be open-sourced upon publication.

Table 1: Experiment results on ChEMBL (Gaulton et al., 2011) and Therapeutics Data Commons (Huang et al., 2021) datasets. We bold best scores based on the mean minus 1 standard deviation. |  | hERG | A549_cells | cyp_2D6 | Ames
---|---|---|---|---|---
AUPRC@ | D-BAT | 84.48±3.90 | 98.26±0.32 | 91.40±2.20 | 99.04±0.53
R<0.2 | AdvStyle | 88.21±1.73 | 97.77±0.63 | 84.83±2.08 | 99.05±0.38
| EoA | 63.80±0.94 | 61.31±0.70 | 61.77±0.92 | 78.74±0.96
| Mixup | 82.25±3.37 | 95.04±0.56 | 87.09±5.18 | 91.02±2.36
| EMOE base | 94.49±0.54 | 98.29±0.24 | 94.96±0.41 | 98.14±0.22
| EMOE | 95.18±0.79 | 98.95±0.22 | 96.38±0.65 | 98.66±0.37
AUPRC | D-BAT | 54.60±3.55 | 67.04±1.18 | 47.42±1.91 | 70.44±1.70
| AdvStyle | 51.54±2.08 | 65.02±1.60 | 44.41±2.15 | 74.98±1.09
| EoA | 43.30±1.15 | 44.95±0.54 | 37.37±2.12 | 59.43±0.41
| Mixup | 42.42±1.91 | 50.52±1.11 | 27.79±3.34 | 60.94±1.78
| EMOE base | 72.51±0.23 | 84.09±0.10 | 72.72±0.20 | 87.50±0.07
| EMOE | 73.73±0.42 | 84.67±0.09 | 73.77±0.32 | 88.53±0.19
AUROC | D-BAT | 76.58±1.01 | 78.16±0.51 | 67.54±1.06 | 83.82±0.34
| AdvStyle | 75.84±1.02 | 76.13±0.62 | 65.51±1.39 | 85.56±1.59
| EoA | 68.02±0.76 | 68.33±0.53 | 60.50±1.07 | 74.77±0.55
| Mixup | 73.96±0.57 | 76.57±0.94 | 67.53±2.02 | 78.43±1.09
| EMOE base | 74.87±0.10 | 79.17±0.07 | 70.30±0.20 | 81.86±0.11
| EMOE | 76.16±0.28 | 79.54±0.07 | 70.54±0.42 | 83.59±0.24

###  4.1 Tabular Datasets

To test how well EMOE generalizes to data with domain shifts, we considered a
total of seven datasets from ChEMBL (Gaulton et al., 2011), Therapeutics Data
Commons (Huang et al., 2021), and DrugOOD (Ji et al., 2022). For all datasets,
we represented molecules using extended-connectivity fingerprints (Rogers and
Hahn, 2010) with radius 2 (ECFP4) and with dimensionality 1024. ECFP4 is a
standard method for molecular representation and was chosen for its simplicity
in calculation as well as its ability to perform comparably to learned
representations, such as those generated by graph neural networks on relevant
classification tasks (Zagidullin et al., 2021). Datasets for inhibition of
human Ether-à-go-go-Related Gene (hERG), cytotoxicity of human A549 cells
(A549_cells), and agonists for Cytochrome P450 2D6 (cyp_2D6) were collected
from ChEMBL (Gaulton et al., 2011). For these datasets, binary classification
labels were generated using a pChEMBL threshold of 5.0. We also considered an
additional binary classification dataset for Ames mutagenicity (Ames) that was
taken from Therapeutics Data Commons (TDC) (Huang et al., 2021). For the
ChEMBL and TDC datasets (hERG, A549_cells, cyp_2D6, and Ames), ID and OOD
splits were determined based on the Murko scaffold of a molecule, such that
OOD data have molecular scaffolds not present in the ID data, mimicking the
“scaffold domain” approach utilized in DrugOOD (Ji et al., 2022).

We also considered the “core ec50,” “refined ec50,” and “core ic50” ligand-
based affinity prediction datasets (lbap) from DrugOOD (Ji et al., 2022) (the
three hardest OOD performance gap datasets). For these datasets ID and OOD
splits were determined based on size, or the number of atoms in a molecule,
such that larger molecules are considered the OOD set and smaller molecules,
the ID set. Datasets are organized by domain and subsequently divided into
training, OOD validation, and OOD testing sets in sequential order. Hence, the
OOD validation set from DrugOOD differs in distribution from the OOD testing
set. While larger in samples, the DrugOOD datasets ignore the impact of
biological targets on label, resulting in a modeling task that has limited
relevance to drug discovery. The additional ChEMBL and TDC datasets, though
smaller, have direct relevance for drug discovery tasks. In our experiments,
we assessed performance on both of these sets. Our results are shown in Table
1 and Table 2.

Table 2: Experiment results on DrugOOD datasets. We bold best scores based on the mean minus 1 standard deviation. |  | core ec50 val | core ec50 test | refined ec50 val | refined ec50 test | core ic50 test | core ic50 test
---|---|---|---|---|---|---|---
AUPRC@ | D-BAT | 93.81±0.49 | 84.35±3.01 | 96.97±0.36 | 88.78±0.90 | 98.13±0.19 | 91.79±0.84
R<0.2 | AdvStyle | 94.84±0.69 | 84.51±5.27 | 95.13±0.29 | 88.21±0.83 | 97.04±0.38 | 89.05±0.50
| EoA | 81.85±0.53 | 71.84±1.01 | 85.03±0.14 | 78.79±0.32 | 88.56±0.12 | 77.03±0.29
| Mixup | 83.97±1.37 | 73.04±0.94 | 85.39±0.52 | 79.78±0.75 | 88.99±0.96 | 78.07±1.36
| EMOE base | 97.88±0.30 | 68.91±0.57 | 98.18±0.22 | 89.38±0.68 | 99.13±0.02 | 94.10±0.26
| EMOE | 98.56±0.19 | 70.68±1.04 | 98.22±0.15 | 89.99±0.63 | 99.11±0.07 | 94.45±0.17
AUPRC | D-BAT | 76.64±1.10 | 54.87±2.21 | 84.70±1.15 | 70.08±1.55 | 90.84±0.62 | 73.45±2.02
| AdvStyle | 81.17±3.85 | 58.40±4.88 | 83.01±2.01 | 69.48±4.83 | 88.54±3.22 | 72.11±3.33
| EoA | 64.16±1.14 | 36.50±3.30 | 69.66±1.06 | 57.71±1.77 | 79.12±0.21 | 56.52±0.95
| Mixup | 73.03±3.73 | 60.84±9.64 | 80.36±1.96 | 72.88±3.73 | 86.88±0.32 | 74.99±0.33
| EMOE base | 88.48±0.10 | 71.94±0.13 | 91.26±0.08 | 82.55±0.18 | 94.87±0.04 | 84.14±0.11
| EMOE | 89.58±0.13 | 71.55±0.50 | 91.59±0.07 | 83.27±0.32 | 95.31±0.04 | 84.77±0.08
AUROC | D-BAT | 75.26±0.62 | 58.21±0.58 | 72.09±0.43 | 60.32±0.56 | 80.31±0.18 | 64.82±0.41
| AdvStyle | 75.97±0.88 | 58.86±0.55 | 70.78±0.78 | 59.62±0.68 | 78.36±0.52 | 64.14±0.60
| EoA | 64.91±0.75 | 52.71±0.98 | 59.27±0.44 | 54.63±0.54 | 62.99±0.35 | 55.83±0.41
| Mixup | 68.20±1.48 | 56.33±1.00 | 60.39±0.90 | 56.50±0.82 | 64.24±2.75 | 57.75±1.79
| EMOE base | 73.69±0.09 | 56.58±0.09 | 70.26±0.10 | 59.72±0.19 | 77.66±0.11 | 64.93±0.12
| EMOE | 75.69±0.20 | 54.62±0.69 | 71.47±0.22 | 60.90±0.63 | 79.77±0.13 | 66.04±0.15

###  4.2 Ablation Studies

We empirically validate our per expert matching, and trail-based filtration
(Sec. 3.2) with ablations.

Table 3: ChEMBL datasets’ mean $\Delta$AUPRC over base experts ablating
augmentation.

|<EMAIL_ADDRESS>|  $\Delta$@R<1
---|---|---
SH MLP + MM | 0.59 | 0.45
MH MLP + MM | 0.25 | 0.55
EMOE | 1.11 | 1.03

#### Matching Ablations

We begin by ablating the matching scheme on base experts and explore a mean-
only matching approach on expanded points as an alternative. First, we
consider a similar training scheme to EMOE (5), but utilizing a single-headed
(SH) MLP, $f(x)$ ($512\rightarrow 512\rightarrow 1$), which is trained via a
mean matching loss
$\mathcal{L}_{\mathrm{MM}}(f,\\{g_{j}\\}_{j=1}^{K};\mathcal{S})\equiv\frac{1}{|\mathcal{S}|}\sum_{x\in
S}\ell(f(x),\frac{1}{K}\sum_{j=1}^{K}g_{j}(x))$, rather than the per-expert
matching loss, $\mathcal{L}_{\mathrm{match}}$ (2) We also explored the effect
of training our multi-headed (MH) architecture ($512\rightarrow 512\rightarrow
1024$) using mean-matching,
$\mathcal{L}^{\prime}_{\mathrm{MM}}(\\{h_{j}\\}_{j=1}^{K},\\{g_{j}\\}_{j=1}^{K};\mathcal{S})\equiv\frac{1}{|\mathcal{S}|}\sum_{x\in
S}\ell(\frac{1}{K}\sum_{j=1}^{K}\sigma(h_{j}(x)),\frac{1}{K}\sum_{j=1}^{K}g_{j}(x)).$
Lastly, we compare these ablations to our EMOE approach which trains the
multi-headed architecture with per-expert matching (5). We report results as
the average change ($\Delta$) in performance for AUPRC using the EMOE
predictions (7) vs the base experts at various recall thresholds. (Greater
values indicate greater improvement over the base experts.) As seen in Tab. 3,
individual expert matching yielded the best results.

#### Trial Based Filtration

Table 4: ChEMBL datasets’ mean $\Delta$AUPRC over base experts ablating
augmentation.

|<EMAIL_ADDRESS>|  $\Delta$@R<1
---|---|---
EMOE Conf. | 0.93 | 0.96
EMOE | 1.11 | 1.03

Next we ablate our augmentation strategy. We considered an alternative generic
confidence based strategy (‘Conf.’), that expands the dataset on randomly
drawn points and also performs random convex combinations of pairs of points.
These points are then labeled with the

EMOE base experts and those with high confidence are kept for matching. In
contrast, in our EMOE approach we only expand those points that were contained
in successful trials (see Sec. 3.2), and filter them further based on
confidence. Although our approach is stable w.r.t. different augmentation
strategies we see the best results by incorporating our extrapolatory trials
for filtration.

Figure 4: _Experts correlations between EMOE base experts and EMOE network
heads on ‘Ames‘ dataset._ (a) EMOE experts has a high correlation with EMOE
base experts on sample the base experts makes correct predictions. (b) EMOE
experts shown low correlations with EMOE base experts on sample the base
experts makes incorrect predictions.

###  4.3 Discussion

Figure 5: _Mean predicted probabilities from EMOE and EMOE base model_.
Starred points denote instances where the base model initially makes incorrect
predictions but are corrected when we average the predicted probabilities from
EMOE and EMOE base.

Below we expound on major takeaways from our results. First, it is worth
noting the base-experts are providing relatively competitive performance
compared to the more complicated baselines. This motivates our philosophy of
building on, and strengthening, the predictions of the base experts. Moreover,
we see consistent improvements over the base experts by EMOE. EMOE especially
stands out for its precision at lower recall values. We can visualize reasons
for the improvement in performance as follows. As can be seen in the scatter
plot (Fig. 5) the multi-headed network rectifies several of the base expert’s
mistakes on OOD data. Moreover, it can be seen that the multi-headed network
is also increasing the certainty of predictions on OOD data as predictions are
moving away from $0.5$. It is well-known that diversity in ensembles improve
performance (Fort et al., 2019). However, we want diversity of ensemble on
potentially erroneous predictions. This is exactly the behavior that we
observe in Fig. 4; while the heads of the EMOE network show significant
agreement on OOD samples where the base experts make correct predictions, they
display significantly less agreement (more diversity) on OOD samples where
base experts make incorrect predictions. This diversity contributes to
improved performance.

##  5 Conclusion

In summary, this work presents Expansive Matching on Experts (EMOE), a novel
method that utilizes support-expanding, extrapolatory pseudo-labeling to
improve prediction and uncertainty based rejection on OOD points. Our
techniques are general and not specific to any data-modality. Moreover, they
are largely complementary to other existing approaches. Thus, we envision the
potential for future work that incorporates our methodology jointly with other
OOD generalization techniques. Moreover, we are encouraged by results in
chemical-property prediction and shall further explore future work in these
directions.

## References

  * Torralba and Efros (2011) Antonio Torralba and Alexei A. Efros.  Unbiased look at dataset bias.  _CVPR 2011_ , pages 1521–1528, 2011.  URL https://api.semanticscholar.org/CorpusID:2777306.
  * Liu et al. (2021) Jiashuo Liu, Zheyan Shen, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui.  Towards out-of-distribution generalization: A survey.  _arXiv preprint arXiv:2108.13624_ , 2021.
  * Freiesleben and Grote (2023) Timo Freiesleben and Thomas Grote.  Beyond generalization: a theory of robustness in machine learning.  _Synthese_ , 202(4):109, 2023.
  * Lee et al. (2023) Seungyeon Lee, Changchang Yin, and Ping Zhang.  Stable clinical risk prediction against distribution shift in electronic health records.  _Patterns_ , 4, 2023.  URL https://api.semanticscholar.org/CorpusID:261165136.
  * Dubuisson and Masson (1993) Bernard Dubuisson and Mylène Masson.  A statistical decision rule with incomplete knowledge about classes.  _Pattern Recognit._ , 26:155–165, 1993.  URL https://api.semanticscholar.org/CorpusID:5710992.
  * Hendrickx et al. (2024) Kilian Hendrickx, Lorenzo Perini, Dries Van der Plas, Wannes Meert, and Jesse Davis.  Machine learning with a reject option: A survey.  _Machine Learning_ , pages 1–38, 2024.
  * Kimber et al. (2021) Talia B. Kimber, Yonghui Chen, and Andrea Volkamer.  Deep learning in virtual screening: Recent applications and developments.  _International Journal of Molecular Sciences_ , 22, 2021.  URL https://api.semanticscholar.org/CorpusID:233463467.
  * Hu et al. (2017) Ye Hu, Dagmar Stumpfe, and Jurgen Bajorath.  Recent advances in scaffold hopping: miniperspective.  _Journal of medicinal chemistry_ , 60(4):1238–1246, 2017.
  * Arjovsky et al. (2019) Martín Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz.  Invariant risk minimization.  _ArXiv_ , abs/1907.02893, 2019.  URL https://api.semanticscholar.org/CorpusID:195820364.
  * Creager et al. (2020) Elliot Creager, Jörn-Henrik Jacobsen, and Richard S. Zemel.  Environment inference for invariant learning.  In _International Conference on Machine Learning_ , 2020.  URL https://api.semanticscholar.org/CorpusID:232097557.
  * Li et al. (2018) Ya Li, Mingming Gong, Xinmei Tian, Tongliang Liu, and Dacheng Tao.  Domain generalization via conditional invariant representations.  In _AAAI Conference on Artificial Intelligence_ , 2018.  URL https://api.semanticscholar.org/CorpusID:19158057.
  * Ding et al. (2022) Yuzhu Ding, Lei Wang, Binxin Liang, Shuming Liang, Yang Wang, and Fangxiao Chen.  Domain generalization by learning and removing domain-specific features.  _ArXiv_ , abs/2212.07101, 2022.  URL https://api.semanticscholar.org/CorpusID:254636222.
  * Koyama and Yamaguchi (2020) Masanori Koyama and Shoichiro Yamaguchi.  Out-of-distribution generalization with maximal invariant predictor.  _ArXiv_ , abs/2008.01883, 2020.  URL https://api.semanticscholar.org/CorpusID:220968862.
  * Cubuk et al. (2019) Ekin Dogus Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le.  Randaugment: Practical automated data augmentation with a reduced search space.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pages 3008–3017, 2019.  URL https://api.semanticscholar.org/CorpusID:208006202.
  * Berthelot et al. (2020) David Berthelot, Nicholas Carlini, Ekin Dogus Cubuk, Alexey Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel.  Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring.  In _International Conference on Learning Representations_ , 2020.  URL https://api.semanticscholar.org/CorpusID:213757781.
  * Zhang et al. (2018) Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  In _International Conference on Learning Representations_ , 2018.
  * Yun et al. (2019) Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Young Joon Yoo.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 6022–6031, 2019.  URL https://api.semanticscholar.org/CorpusID:152282661.
  * Zhong et al. (2022) Zhun Zhong, Yuyang Zhao, Gim Hee Lee, and Nicu Sebe.  Adversarial style augmentation for domain generalized urban-scene segmentation.  _Advances in Neural Information Processing Systems_ , 35:338–350, 2022.
  * Lee (2013) Dong-Hyun Lee.  Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks.  2013\.  URL https://api.semanticscholar.org/CorpusID:18507866.
  * Zou et al. (2018) Yang Zou, Zhiding Yu, B. V. K. Vijaya Kumar, and Jinsong Wang.  Unsupervised domain adaptation for semantic segmentation via class-balanced self-training.  In _European Conference on Computer Vision_ , 2018.  URL https://api.semanticscholar.org/CorpusID:52954862.
  * Cascante-Bonilla et al. (2020) Paola Cascante-Bonilla, Fuwen Tan, Yanjun Qi, and Vicente Ordonez.  Curriculum labeling: Revisiting pseudo-labeling for semi-supervised learning.  In _AAAI Conference on Artificial Intelligence_ , 2020.  URL https://api.semanticscholar.org/CorpusID:228096598.
  * Zhang et al. (2021) Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki.  Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling.  In _Neural Information Processing Systems_ , 2021.  URL https://api.semanticscholar.org/CorpusID:239016453.
  * Ghosh et al. (2021) Soumyadeep Ghosh, Sanjay Kumar, Janu Verma, and Awanish Kumar.  Self training with ensemble of teacher models.  _ArXiv_ , abs/2107.08211, 2021.  URL https://api.semanticscholar.org/CorpusID:236087412.
  * Xie et al. (2019) Qizhe Xie, Eduard H. Hovy, Minh-Thang Luong, and Quoc V. Le.  Self-training with noisy student improves imagenet classification.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10684–10695, 2019.  URL https://api.semanticscholar.org/CorpusID:207853355.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  _J. Mach. Learn. Res._ , 15:1929–1958, 2014.  URL https://api.semanticscholar.org/CorpusID:6844431.
  * Stefano et al. (2000) Claudio De Stefano, Carlo Sansone, and Mario Vento.  To reject or not to reject: that is the question-an answer in case of neural classifiers.  _IEEE Trans. Syst. Man Cybern. Part C_ , 30:84–94, 2000.  URL https://api.semanticscholar.org/CorpusID:7594035.
  * Fumera et al. (2000) Giorgio Fumera, Fabio Roli, and Giorgio Giacinto.  Reject option with multiple thresholds.  _Pattern Recognit._ , 33:2099–2101, 2000.  URL https://api.semanticscholar.org/CorpusID:9209281.
  * Devries and Taylor (2018) Terrance Devries and Graham W. Taylor.  Learning confidence for out-of-distribution detection in neural networks.  _ArXiv_ , abs/1802.04865, 2018.  URL https://api.semanticscholar.org/CorpusID:3271220.
  * Geifman and El-Yaniv (2017) Yonatan Geifman and Ran El-Yaniv.  Selective classification for deep neural networks.  _ArXiv_ , abs/1705.08500, 2017.  URL https://api.semanticscholar.org/CorpusID:491127.
  * Thulasidasan et al. (2019) Sunil Thulasidasan, Tanmoy Bhattacharya, Jeff A. Bilmes, Gopinath Chennupati, and Jamaludin Mohd-Yusof.  Combating label noise in deep learning using abstention.  In _International Conference on Machine Learning_ , 2019.  URL https://api.semanticscholar.org/CorpusID:166227922.
  * Ziyin et al. (2019) Liu Ziyin, Zhikang T. Wang, Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippe Morency, and Masahito Ueda.  Deep gamblers: Learning to abstain with portfolio theory.  In _Neural Information Processing Systems_ , 2019.  URL https://api.semanticscholar.org/CorpusID:195767452.
  * Geifman and El-Yaniv (2019) Yonatan Geifman and Ran El-Yaniv.  Selectivenet: A deep neural network with an integrated reject option.  In _International Conference on Machine Learning_ , 2019.  URL https://api.semanticscholar.org/CorpusID:59316904.
  * Dietterich (2007) Thomas G. Dietterich.  Ensemble methods in machine learning.  2007\.  URL https://api.semanticscholar.org/CorpusID:10765854.
  * Kussul et al. (2010) Ernst M. Kussul, Oleksandr Makeyev, Tatiana Baidyk, and Daniel Calderon Reyes.  Neural network with ensembles.  _The 2010 International Joint Conference on Neural Networks (IJCNN)_ , pages 1–7, 2010.  URL https://api.semanticscholar.org/CorpusID:993561.
  * Jordan and Jacobs (1993) Michael I. Jordan and Robert A. Jacobs.  Hierarchical mixtures of experts and the em algorithm.  _Neural Computation_ , 6:181–214, 1993.  URL https://api.semanticscholar.org/CorpusID:67000854.
  * Eigen et al. (2013) David Eigen, Marc’Aurelio Ranzato, and Ilya Sutskever.  Learning factored representations in a deep mixture of experts.  _CoRR_ , abs/1312.4314, 2013.  URL https://api.semanticscholar.org/CorpusID:11492613.
  * Pagliardini et al. (2022) Matteo Pagliardini, Martin Jaggi, François Fleuret, and Sai Praneeth Karimireddy.  Agree to disagree: Diversity through disagreement for better transferability.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Arpit et al. (2022a) Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming Xiong.  Ensemble of averages: Improving model selection and boosting performance in domain generalization.  In _Advances in Neural Information Processing Systems_ , 2022a.
  * Yao et al. (2023) Huaxiu Yao, Xinyu Yang, Xinyi Pan, Shengchao Liu, Pang Wei Koh, and Chelsea Finn.  Improving domain generalization with domain relations.  2023\.  URL https://api.semanticscholar.org/CorpusID:256615216.
  * Qiao et al. (2020) Fengchun Qiao, Long Zhao, and Xi Peng.  Learning to learn single domain generalization.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 12553–12562, 2020.  URL https://api.semanticscholar.org/CorpusID:214713888.
  * Du et al. (2021) Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen S. Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V. Le, Yonghui Wu, Z. Chen, and Claire Cui.  Glam: Efficient scaling of language models with mixture-of-experts.  _ArXiv_ , abs/2112.06905, 2021.  URL https://api.semanticscholar.org/CorpusID:245124124.
  * Sohn et al. (2020) Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin Dogus Cubuk, Alexey Kurakin, Han Zhang, and Colin Raffel.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  _ArXiv_ , abs/2001.07685, 2020.  URL https://api.semanticscholar.org/CorpusID:210839228.
  * Yang et al. (2020) Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma.  Rethinking bias-variance trade-off for generalization of neural networks.  In _International Conference on Machine Learning_ , pages 10767–10777. PMLR, 2020.
  * Arpit et al. (2022b) Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming Xiong.  Ensemble of averages: Improving model selection and boosting performance in domain generalization.  _Advances in Neural Information Processing Systems_ , 35:8265–8277, 2022b.
  * Chen and Guestrin (2016) Tianqi Chen and Carlos Guestrin.  Xgboost: A scalable tree boosting system.  _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , 2016.  URL https://api.semanticscholar.org/CorpusID:4650265.
  * Clevert et al. (2015) Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter.  Fast and accurate deep network learning by exponential linear units (elus).  _arXiv: Learning_ , 2015.  URL https://api.semanticscholar.org/CorpusID:5273326.
  * Gaulton et al. (2011) Anna Gaulton, Louisa J. Bellis, A. Patrícia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, and John P. Overington.  Chembl: a large-scale bioactivity database for drug discovery.  _Nucleic Acids Research_ , 40:D1100 – D1107, 2011.  URL https://api.semanticscholar.org/CorpusID:16681789.
  * Huang et al. (2021) Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor W. Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik.  Therapeutics data commons: Machine learning datasets and tasks for drug discovery and development, 2021.
  * Ji et al. (2022) Yuanfeng Ji, Lu Zhang, Jiaxiang Wu, Bingzhe Wu, Long-Kai Huang, Tingyang Xu, Yu Rong, Lanqing Li, Jie Ren, Ding Xue, Houtim Lai, Shaoyong Xu, Jing Feng, Wei Liu, Ping Luo, Shuigeng Zhou, Junzhou Huang, Peilin Zhao, and Yatao Bian.  Drugood: Out-of-distribution (ood) dataset curator and benchmark for ai-aided drug discovery – a focus on affinity prediction problems with noise annotations, 2022.
  * Rogers and Hahn (2010) David Rogers and Mathew Hahn.  Extended-connectivity fingerprints.  _Journal of Chemical Information and Modeling_ , 50(5):742–754, 2010.  doi: 10.1021/ci100050t.  URL https://doi.org/10.1021/ci100050t.  PMID: 20426451.
  * Zagidullin et al. (2021) B Zagidullin, Z Wang, Y Guan, E Pitkänen, and J Tang.  Comparative analysis of molecular fingerprints in prediction of drug combination effects.  _Briefings in Bioinformatics_ , 22(6):bbab291, 08 2021.  ISSN 1477-4054.  doi: 10.1093/bib/bbab291.  URL https://doi.org/10.1093/bib/bbab291.
  * Fort et al. (2019) Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan.  Deep ensembles: A loss landscape perspective.  _ArXiv_ , abs/1912.02757, 2019.  URL https://api.semanticscholar.org/CorpusID:208637294.
  * Kingma and Ba (2014) Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _CoRR_ , abs/1412.6980, 2014.  URL https://api.semanticscholar.org/CorpusID:6628106.
  * Maas (2013) Andrew L. Maas.  Rectifier nonlinearities improve neural network acoustic models.  2013\.  URL https://api.semanticscholar.org/CorpusID:16489696.

##  Appendix A Addtional Experiment Details

###  A.1 EMOE training details

In all of our experiments we used the Adam [Kingma and Ba, 2014] optimizer and
mini-batches of size 256. One Nvidia A100 GPU with 40GB GPU memory was used to
run our experiments, and duration for model training is approximately 0.5
hours. During the extrapolatory directional mining trials, we held out 0.1 of
the data for each of the 1000 trials. Subsequently, from the top-performing
trials, determined by accuracy evaluation, we selected the top held-out points
from 0.15 of the best-performing trials for data expansion. $\lambda$ = 0.5
was used for the $\mathcal{L}_{\mathrm{match}}$ for the expanded points. As
noted in Sec. 3.1 we trained the EMOE models directly in the latent space to
avoid the need for the decoder (and also allowed baselines to do this if it
aided their performance). In the experiments on hERF, A549_cells, CYP_2D6,
Ames, core ec50, refined ec50, EMOE was trained for 20000 iterations.
Arithmetic mean between EMOE and EMOE base was reported. On the M/C-D and
M/F-D datasets, we took the raw pixel values as input and trained EMOE
networks for 4000 iterations due to lower observed losses. Harmonic mean
between EMOE and EMOE base was reported on these datasets. We performed 8
trails on each of the datasets for EMOE.

###  A.2 Baseline Setup

We re-implemented all baselines we are comparing against EMOE following the
implementation details in their paper and/or using Github implementations (if
available). Since the fingerprints representation of chemicals are quite
sparse, we preformed dimension reduction using PCA with 128 components on all
chemical datasets. For D-BAT[Pagliardini et al., 2022] with existing
implementations designed for tabular data, we utilized their original model
architectures. For the other three baseline methods without implementation
specifically for tabular data, we adopted a structure comprising two 512
ELU[Clevert et al., 2015] layers to closely mimic the EMOE network
architecture. The Adam [Kingma and Ba, 2014] optimizer was used for training
baseline models.

D-BAT In our experiments, the D-Bat[Pagliardini et al., 2022] models used MLP
architecture with one 128 LeakyRelu[Maas, 2013] layer following the
architecture in their Github. Their paper [Pagliardini et al., 2022] discussed
two settings, and we focused on the scenario where perturbation data differs
from the distribution of test data, adhering to the single-source domain
generalization setting. We trained an ensemble of five models sequentially for
the D-bat baseline models and the predictions from the 5 models were averaged
to obtain the final prediction.

EoA We trained an ensemble of 5 simple moving average model following the
method described in [Arpit et al., 2022a]. We start calculating the moving
average at iteration 50 and trained the models for 200 iterations. The
predictions from the 5 models were averaged to obtain the final prediction for
EoA.

For AdvStyle [Zhong et al., 2022] and Mixup [Zhang et al., 2018], the
methodologies were straightforward. We experimented with training using
various numbers of iterations and reported the most promising results. Note
that we used alpha=0.7 when combining the 2 samples for Mixup. We executed all
baseline experiments five times on each dataset to ensure a precise estimation
of performance.

##  Appendix B Additional Experiment and Ablation Results

###  B.1 Full Experiment Results on CheMBL and Therapeutics Data Commons

In Table 5, we report the full results on hERG, A549_cells, cyp_2D6. and Ames.

Table 5: Full experiment results on ChEMBL [Gaulton et al., 2011] and Therapeutics Data Commons [Huang et al., 2021] datasets. We bold best scores based on the mean minus 1 standard deviation. |  | hERG | A549_cells | cyp_2D6 | Ames
---|---|---|---|---|---
AUPRC | D-BAT | 88.55±3.75 | 98.57±0.36 | 95.71±1.99 | 99.07±0.52
$\mathrm{@R}\\!\\!\leq\\!\\!.1$ | AdvStyle | 93.27±1.25 | 96.89±0.66 | 84.21±4.62 | 99.52±0.26
| EoA | 63.80±0.94 | 61.31±0.70 | 61.77±0.92 | 78.74±0.96
| Mixup | 82.80±3.49 | 95.04±0.56 | 87.39±6.91 | 91.02±2.36
| EMOE base | 96.69±0.62 | 99.79±0.11 | 99.55±0.58 | 99.30±0.25
| EMOE | 98.98±0.56 | 99.85±0.14 | 99.26±0.59 | 99.76±0.37
AUPRC@ | D-BAT | 84.48±3.90 | 98.26±0.32 | 91.40±2.20 | 99.04±0.53
$\mathrm{@R}\\!\\!\leq\\!\\!.2$ | AdvStyle | 88.21±1.73 | 97.77±0.63 | 84.83±2.08 | 99.05±0.38
| EoA | 63.80±0.94 | 61.31±0.70 | 61.77±0.92 | 78.74±0.96
| Mixup | 82.25±3.37 | 95.04±0.56 | 87.09±5.18 | 91.02±2.36
| EMOE base | 94.49±0.54 | 98.29±0.24 | 94.96±0.41 | 98.14±0.22
| EMOE | 95.18±0.79 | 98.95±0.22 | 96.38±0.65 | 98.66±0.37
AUPRC@ | D-BAT | 82.44±3.56 | 97.37±0.53 | 87.65±1.29 | 98.61±0.53
$\mathrm{@R}\\!\\!\leq\\!\\!.3$ | AdvStyle | 85.05±1.93 | 96.47±0.64 | 82.76±2.13 | 98.71±0.44
| EoA | 63.80±0.94 | 61.31±0.70 | 61.77±0.92 | 78.74±0.96
| Mixup | 81.51±3.01 | 94.95±0.55 | 84.53±6.46 | 90.88±2.24
| EMOE base | 91.05±0.50 | 97.34±0.21 | 91.60±0.32 | 98.05±0.15
| EMOE | 91.57±0.61 | 98.11±0.25 | 93.10±0.81 | 98.41±0.29
AUPRC | D-BAT | 54.60±3.55 | 67.04±1.18 | 47.42±1.91 | 70.44±1.70
| AdvStyle | 51.54±2.08 | 65.02±1.60 | 44.41±2.15 | 74.98±1.09
| EoA | 43.30±1.15 | 44.95±0.54 | 37.37±2.12 | 59.43±0.41
| Mixup | 42.42±1.91 | 50.52±1.11 | 27.79±3.34 | 60.94±1.78
| EMOE base | 72.51±0.23 | 84.09±0.10 | 72.72±0.20 | 87.50±0.07
| EMOE | 73.73±0.42 | 84.67±0.09 | 73.77±0.32 | 88.53±0.19
AUROC | D-BAT | 76.58±1.01 | 78.16±0.51 | 67.54±1.06 | 83.82±0.34
| AdvStyle | 75.84±1.02 | 76.13±0.62 | 65.51±1.39 | 85.56±1.59
| EoA | 68.02±0.76 | 68.33±0.53 | 60.50±1.07 | 74.77±0.55
| Mixup | 73.96±0.57 | 76.57±0.94 | 67.53±2.02 | 78.43±1.09
| EMOE base | 74.87±0.10 | 79.17±0.07 | 70.30±0.20 | 81.86±0.11
| EMOE | 76.16±0.28 | 79.54±0.07 | 70.54±0.42 | 83.59±0.24

###  B.2 Full Experiment Results on DrugOOD

In Table 6, we report the full results on core ec50, refined ec 50, and core
ic50 from DrugOOD [Ji et al., 2022].

Table 6: Full experiment results on DrugOOD datasets. We bold best scores based on the mean minus 1 standard deviation. |  | core ec50 val | core ec50 test | refined ec50 val | refined ec50 test | core ic50 test | core ic 50 test
---|---|---|---|---|---|---|---
AUPRC@ | D-BAT | 94.04±0.55 | 86.59±3.17 | 97.19±0.43 | 88.93±1.07 | 98.25±0.21 | 91.89±0.86
$\mathrm{@R}\\!\\!\leq\\!\\!.1$ | AdvStyle | 95.79±0.45 | 84.56±5.28 | 96.37±0.72 | 88.69±0.95 | 98.10±0.37 | 89.39±0.73
| EoA | 81.85±0.53 | 71.84±1.01 | 85.03±0.14 | 78.79±0.32 | 88.56±0.12 | 77.03±0.29
| Mixup | 83.97±1.37 | 73.03±0.93 | 85.39±0.52 | 79.78±0.75 | 88.99±0.96 | 78.07±1.36
| EMOE base | 98.85±0.33 | 66.19±1.14 | 99.09±0.11 | 92.72±0.66 | 99.56±0.01 | 96.85±0.17
| EMOE | 99.05±0.26 | 68.12±1.16 | 98.85±0.20 | 91.26±0.80 | 99.38±0.08 | 96.37±0.31
AUPRC@ | D-BAT | 93.81±0.49 | 84.35±3.01 | 96.97±0.36 | 88.78±0.90 | 98.13±0.19 | 91.79±0.84
$\mathrm{@R}\\!\\!\leq\\!\\!.2$ | AdvStyle | 94.84±0.69 | 84.51±5.27 | 95.13±0.29 | 88.21±0.83 | 97.04±0.38 | 89.05±0.50
| EoA | 81.85±0.53 | 71.84±1.01 | 85.03±0.14 | 78.79±0.32 | 88.56±0.12 | 77.03±0.29
| Mixup | 83.97±1.37 | 73.04±0.94 | 85.39±0.52 | 79.78±0.75 | 88.99±0.96 | 78.07±1.36
| EMOE base | 97.88±0.30 | 68.91±0.57 | 98.18±0.22 | 89.38±0.68 | 99.13±0.02 | 94.10±0.26
| EMOE | 98.56±0.19 | 70.68±1.04 | 98.22±0.15 | 89.99±0.63 | 99.11±0.07 | 94.45±0.17
AUPRC@ | D-BAT | 93.73±0.48 | 81.40±2.00 | 96.89±0.33 | 87.77±0.71 | 98.08±0.19 | 90.71±1.17
$\mathrm{@R}\\!\\!\leq\\!\\!.3$ | AdvStyle | 94.52±0.82 | 83.57±4.65 | 94.71±0.29 | 88.05±0.81 | 96.69±0.48 | 88.93±0.42
| EoA | 81.85±0.53 | 71.84±1.01 | 85.03±0.14 | 78.79±0.32 | 88.56±0.12 | 77.03±0.29
| Mixup | 83.97±1.37 | 73.06±0.96 | 85.39±0.52 | 79.78±0.75 | 88.99±0.96 | 78.07±1.36
| EMOE base | 96.85±0.28 | 70.30±0.45 | 97.30±0.25 | 87.82±0.53 | 98.69±0.03 | 92.32±0.23
| EMOE | 97.86±0.20 | 71.35±0.87 | 97.49±0.09 | 89.01±0.54 | 98.84±0.07 | 93.11±0.13
AUPRC | D-BAT | 76.64±1.10 | 54.87±2.21 | 84.70±1.15 | 70.08±1.55 | 90.84±0.62 | 73.45±2.02
| AdvStyle | 81.17±3.85 | 58.40±4.88 | 83.01±2.01 | 69.48±4.83 | 88.54±3.22 | 72.11±3.33
| EoA | 64.16±1.14 | 36.50±3.30 | 69.66±1.06 | 57.71±1.77 | 79.12±0.21 | 56.52±0.95
| Mixup | 73.03±3.73 | 60.84±9.64 | 80.36±1.96 | 72.88±3.73 | 86.88±0.32 | 74.99±0.33
| EMOE base | 88.48±0.10 | 71.94±0.13 | 91.26±0.08 | 82.55±0.18 | 94.87±0.04 | 84.14±0.11
| EMOE | 89.58±0.13 | 71.55±0.50 | 91.59±0.07 | 83.27±0.32 | 95.31±0.04 | 84.77±0.08
AUROC | D-BAT | 75.26±0.62 | 58.21±0.58 | 72.09±0.43 | 60.32±0.56 | 80.31±0.18 | 64.82±0.41
| AdvStyle | 75.97±0.88 | 58.86±0.55 | 70.78±0.78 | 59.62±0.68 | 78.36±0.52 | 64.14±0.60
| EoA | 64.91±0.75 | 52.71±0.98 | 59.27±0.44 | 54.63±0.54 | 62.99±0.35 | 55.83±0.41
| Mixup | 68.20±1.48 | 56.33±1.00 | 60.39±0.90 | 56.50±0.82 | 64.24±2.75 | 57.75±1.79
| EMOE base | 73.69±0.09 | 56.58±0.09 | 70.26±0.10 | 59.72±0.19 | 77.66±0.11 | 64.93±0.12
| EMOE | 75.69±0.20 | 54.62±0.69 | 71.47±0.22 | 60.90±0.63 | 79.77±0.13 | 66.04±0.15

###  B.3 Ablations Results on Vanilla and MoE Models without Ensemble Base
Model

In this ablation study, we focused on examining the impact of the vanilla vs.
Mixture of Experts (MoE) architecture on OOD performance. Both the MoE MLP and
Vanilla MLP were configured with two layers of 512 ELU units. The MoE MLP has
1024 output heads, whereas the vanilla MLP had only one. For noisy
augmentation, we applied small perturbations drawn from a standard normal
distribution to the training set while retaining the original labels. We run 3
trials on each of the CheMBL dataset and the results are shown in Table 7 .
Although we observed that the MoE model achieved a AUPRC compared to the
vanilla MLP, we found that simple noisy augmentation did not lead to any
significant difference in performance.

Table 7: Ablation results on MLP architectures with CheMBL datasets. | AUPRC$\mathrm{@R}\\!\\!\leq\\!\\!.1$ | AUPRC$\mathrm{@R}\\!\\!\leq\\!\\!.2$ | AUPRC$\mathrm{@R}\\!\\!\leq\\!\\!.3$ | AUPRC | AUROC
---|---|---|---|---|---
Vanilla MLP | 68.96 | 68.96 | 68.96 | 49.73 | 68.26
Vanilla MLP+noisy aug | 69.12 | 69.12 | 69.12 | 48.43 | 67.44
MH_MLP | 70.96 | 70.96 | 70.96 | 53.18 | 69.08
MH_MLP+noisy aug | 70.49 | 70.49 | 70.49 | 54.54 | 69.15
EMOE | 99.04 | 96.21 | 92.90 | 73.73 | 70.59

##  Appendix C Limitations

It is important to acknowledge a reliance on the EMOE base model and the
latent space; we observed good performance with simple implementations,
indicating the potential for better performance with other choices. Moreover,
to preserve generality, this work limited itself to augmentations in a real-
value vector setting; however, when it may be possible to exploit modality-
specific augmentations in applications.


#  Hybrid NOMA for Future Radio Access: Design, Potentials and Limitations

Kuntal Deka1 and Sanjeev Sharma2 (Member, IEEE)
1School of Electrical Sciences, IIT Goa, India Email<EMAIL_ADDRESS>
2Department of Electronics Engineering, IIT (BHU) Varanasi, India Email:
<EMAIL_ADDRESS>
###### Abstract

Next-generation internet of things (IoT) applications need trillions of low-
powered wireless mobile devices to connect with each other having ultra-
reliability and low-latency. Non-orthogonal multiple access (NOMA) is a
promising technology to address massive connectivity for 5G and beyond by
accommodating several users within the same orthogonal resource block.
Therefore, this article explores hybrid NOMA (HNOMA) for massive multiple
access in the uplink scenarios due to its higher spectral efficiency. The
HNOMA includes both power domain and code domain NOMA method due to diverse
channel conditions in practice. We highlight that polar coded based data
transmission can achieve higher reliability and lower latency in HNOMA-based
wireless networks. Further, at the base station (BS), channel state
information (CSI) of each link is not perfectly available or very complex to
estimate due to non-orthogonal links. Therefore, we analyze and review the
performance of uplink based system involving HNOMA transmission in the
presence of imperfect CSI. Furthermore, we summarize some key technical
challenges as well as their potential solutions in futuristic IoT applications
using HNOMA transmission. Finally, we offer some design guidelines for HNOMA-
based systems using deep learning approach to implement adaptive and efficient
wireless networks.

##  I Introduction

Forthcoming society will get highly benefited using the data-driven solutions
and methods. In the data-driven environment, trillions of low-power mobile
devices will connect to communicate and share their decisions. However,
connecting trillions of devices together through a central base station (BS)
will be challenging due to limited available orthogonal resources with very
low-latency wireless links. Therefore, non-orthogonal multiple access (NOMA)
and hybrid NOMA (HNOMA) transmission techniques can be suitable methods as
compared to orthogonal multiple access (OMA) techniques to connect large
devices in the internet of things (IoT) scenarios [1, 2, 3]. Since the non-
orthogonal transmission techniques enhance spectral efficiency and lower the
latency of a wireless system [1] The HNOMA uses both power domain (PD) and
code domain (CD) NOMA for multiple access. Further, HNOMA provides higher
spectral efficiency as compared to NOMA and OMA techniques [1], and we analyze
its potential applications and challenges for 6G networks.

IoT applications can be divided into two main categories as massive IoT in
which low power devices continuously send their observation to the cloud, and
critical IoT which includes control signaling, healthcare, and remote
manufacturing. Some upcoming applications, wireless sensing, and signaling is
essential, like in traffic management, self driving vehicles. Therefore, in
this article, we discuss the massive connectivity of low power devices with
ultra-reliability and low-latency based IoT.

###  I-A Smart IoT Network

A scenario of smart IoT applications is shown in Fig. 1. In upcoming 5G and
beyond IoT networks, various devices will connect to each other using the
virtual cloud networks. For example, smart farming can benefit from the data-
driven design by analyzing the soil and atmospheric conditions of the
environment for a particular type of crop. A similar technique can be adopted
for IoT enabled intra-vehicular network (IVN), where large number of sensors
are connected with each other for sharing the vehicle’s status information in
order to develop a smart vehicular system. Further, traffic management can be
significantly improved by using sensing and wireless communication methods to
develop smart cities. Therefore, the integration of various areas like
industries, offices, power grids, farming, healthcare, etc. using IoT platform
will make society more conformable, flexible and sustainable.

A smart IoT network, heterogeneous infrastructures are connected to a central
cloud network, as shown in Fig. 1. Therefore, each type of infrastructure like
smart grid can be optimized by accessing available data at the central
cloud/node. Hence, smart infrastructure will give rise to dense/ultra-dense
deployment of access points (AP) in 5G and beyond network.

Figure 1: A view of smart IoT Network. RAP: remote access point, AP: access
point.

###  I-B Wireless multiple access techniques

In order to connect seamlessly trillions low-powered IoT devices, we need
higher spectral efficiency and availability of a wide frequency spectrum.
Currently, millimeter-wave (mmWave) range of the spectrum is explored for
wireless communications [4]. High bandwidth at mmWave will help to connect
large devices together; however, it alone can not be sufficient for IoT
applications. Currently, NOMA techniques are preferred over OMA to enhance the
wireless systems’ spectral efficiency [5, 6, 7]. In general, either PD or CD
NOMA methods are considered in the literature [1]. In CD-based NOMA technique
includes interleave division multiple access (IDMA), multi-user shared access
(MUSA) technique, pattern division multiple access (PDMA), and sparse code
multiple access (SCMA) [1]. In PD NOMA, superposition coding techniques at the
transmitter and successive interference cancellation (SIC) at the receiver are
used unlike the OMA. Further, HNOMA, which is an integrated method of PD and
CD NOMA, is more suitable and has higher spectral efficiency over conventional
PD or CD NOMA. Therefore, in this article, we discuss HNOMA for 5G and beyond
wireless networks.

####  I-B1 A simple view of wireless connectivity using HNOMA

In Fig. 2, a pictorial representation of the connectivity of low powered
devices to a remote access point (RAP) is shown. For easy processing and low
latency, the large number of devices are divided into total $J$ clusters in a
RAP area. Further, each cluster has $K$ multiple groups based on their
distance from the RAP, as shown in Fig. 2. Therefore, a device ${D}$ in RAP is
accessed using cluster and group indices, as ${D}_{j,k},\ j=1,\cdots
J,k=1,\cdots,K$. Each cluster has $Z$ orthogonal (time/frequency/code)
resources for communications. Further, these $Z$ orthogonal resources are
reused in other clusters in the system. Therefore, all the groups in a cluster
use the same $Z$ orthogonal resources for communications, as shown in Fig. 2.
However, $K$ groups are distinguished using their channel gain at the RAP.
Further, devices in each group may have similar or correlated channel impulse
response. Therefore, total $JK$ devices are connected to the RAP using $Z$
orthogonal resources, as depicted in Fig. 2. In the absence of strict power
control and sharp frequency cut-off, we assume each cluster may get some
interference from other adjacent clusters. However, we have not considered
interference from other adjacent clusters in the system.

Figure 2: A sketch for devices connectivity to remote access point using
HNOMA.

####  I-B2 Example: HNOMA transmission in V2X communications

Automotive industries are moving towards connected and autonomous vehicles to
improve traffic congestion, fuel efficiency, safety, etc. [8]. Therefore, a
large number of sensor nodes are required to exchange information. In vehicle-
to-everything (V2X) communication, cloud network gather information through
several wireless access points such as local area central node, base station,
etc. Therefore, to connect trillions low-powered devices, high spectral
efficiency of network is essential with low latency and efficient signal
processing methods. IEEE 802.11p standard is proposed for V2X Communications
to support intelligent transportation system (ITS) applications. However,
still technologies for ITS will have to improve for bounded access channel
delay and V2X wireless connections. In V2X, grant free111User node
transmission that does not require transmit scheduling request i.e. randomly
enter or leave the system. HNOMA with polar coding can be used to achieve low
latency and high reliability with high spectral efficiency.

The rest of the article is organized as follows. Given the potential
advantages of HNOMA system, in Section II, we illustrate HNOMA based uplink
wireless system for massive connectivity in IoT. Section III briefly focuses
for polar coded HNOMA-based system. The impact of channel estimation error in
HNOMA in Section IV. An example of HNOMA system in the uplink scenario is
presented in Section V. After that, we highlight some research challenges in
upcoming IoT networks for heterogeneous applications and HNOMA techniques in
Section VI. Then, we discuss and propose a deep learning-based detection and
estimation for HNOMA in Section VII. Section VIII concludes this article.

##  II HNOMA

Recently, HNOMA scheme is proposed for multiple access [1, 9]. The HNOMA
scheme using both PD and CD NOMA is more spectral efficient than the PD-NOMA
plus OMA-based system [1]. Therefore, in this paper, HNOMA uses jointly PD and
SCMA-based NOMA. The HNOMA uses both power difference and sparsity among the
users and orthogonal resources, respectively for transmission. Consider the
uplink scenario of HNOMA system with two groups only, where Group-$1$ and
Group-$2$ have $J_{1}$ and $J_{2}$ communicating nodes, respectively. By
considering Group-$1$ and Group-$2$ as the far and the near groups,
respectively, we can get the desired variation in strengths of the received
signals at the BS. Group-$2$ (near group) receives higher signal power than
the Group-$1$ in the HNOMA system. Therefore, the data for Group-$2$ is
decoded first by considering the Group-$1$ signal as an interference using
massage passing algorithm (MPA). After decoding Group-$2$, the signal
corresponding to Group-$2$ is subtracted from the received signal at the BS.
Then, from the interference-canceled signal, Group-$1$ symbols are decoded.
Therefore, Group-$1$ has full diversity, while Group-$2$ has interference-
limited scenarios. Hence, at the BS, both MPA and successive interference
cancellation (SIC) based detection is used in HNOMA [1]. In HNOMA, multiple
groups can communicate by optimizing transmit power among the different groups
and codebook design within a group. Therefore, $K$ groups can communicate to
BS in HNOMA, where each group has $J_{k},k=1,2,...,K$ user nodes. The
overloading factor $\lambda$ can be defined as the ratio of the total number
$\sum_{k=1}^{K}J_{k}$ of users to the number $Z$ of the orthogonal resources
in the system.

Therefore, HNOMA-based system can achieve higher spectral efficiency in the
IoT network by marginally increasing the detection complexity. However, the BS
or the central node can support a higher complexity in the system. Further,
deep learning based information detection can also be used in HNOMA. Deep
neural network (DNN) can be trained off-line and is being used for online data
detection in wireless communication systems [10]. Furthermore, hybrid non-
orthogonal transmission can also include orthogonal and non-orthognal multiple
access schemes for next-generation adaptive networks depending on nodes’ data
rate constraint.

The impact of overloading in HNOMA is shown in Fig. 3. In each group, the
number of user nodes are $J_{1}=J_{2}=6,8$ and orthogonal resources are four,
i.e., $Z=4$. The transmitted signals of the user nodes experience independent
flat Rayleigh fading. Codebooks in [1] are used by user nodes in Fig. 3. The
performance degrades in high overloading scenario ($400\%$) as compared to
$300\%$ overloading. However, it is good in most of IoT applications. Further,
the detection performance of HNOMA-based system can be improved using polar
coding techniques.

Figure 3: Impact of overloading factor on the hybrid system’s SER performance
over Rayleigh fading.

The HNOMA-based system includes both the SIC and MPA based detection, whose
complexity increases as the number of groups in a cluster and the overloading
factor increases. To limit the complexity, the number devices in the IoT
networks can be partitioned into small groups and clusters based on their
channel gains, as mentioned in Fig. 2. Further, efficient SIC ordering is also
essential in HNOMA to reduce the error propagation effect [11]. Further, we
introduce SCMA-based NOMA scheme with polar codes to improve the reliability
of a wireless network below.

##  III Polar Codes

Erdal Arikan invented a new class of channel codes known as polar codes [12].
These are the first class of codes which can achieve the Shannon capacity in a
mathematically provable fashion. These codes are based on the principle of
channel polarization. Here, with the help of polar encoding, $n$ synthetic bit
channels are manufactured from a given set of $n$ identical channels. The
newly created bit-channels are polarized in the sense that a fraction of them
become noise-free and others become completely noisy channel. The key aspect
here is that the fraction of the noiseless channels is equal to the capacity
of the original underlying channel. Therefore, a capacity-achieving coding
scheme can be devised where the information bits are sent over the noiseless
channels and a set of frozen bits (known to the decoder) are sent over the
completely noisy channels. The decoding is done with the help of successive
cancellation (SC) algorithm. Unlike other modern codes like, Turbo and low-
density parity-check (LDPC) codes, the decoding of polar codes is non-
iterative. The performance of the polar codes with SC decoding was not
impressive at practical block-lengths. Then, Tal and Vardy proposed successive
cancellation list decoding for polar codes [13]. It has been found that aided
by a small cyclic-redundancy check (CRC) code, the list decoding of polar
codes can provide significantly better performance than LDPC and turbo codes
when block-length is in the small to medium range.

Figure 4: Polar coded SCMA system for uplink.

The performance of an SCMA system can be significantly improved by employing
an error correcting scheme. Here we consider the polar codes as they can
provide the best performance amongst all the modern codes. The block diagram
is shown in Fig. 4. The coding scheme is a concatenated one with a small CRC
code of length $n_{c}$ and polar code of length $n$. The $j$th user’s data are
divided into message frames ${\bf{u}}_{j}$ of size $k$. Here we have
$k=n\times r-n_{c}$, where $r$ is the rate of the coding scheme. The frame
${\bf{c}}_{j}$ at the output of the CRC encoder has length $n_{c}$. The frame
${\bf{c}}_{j}$ is sent to polar encoder as the input. The output of the polar
encoder is ${\bf{x}}_{j}$. Suppose the SCMA mapper involves $M-$ary
constellation. Then the bits in the frame ${\bf{x}}_{j}$ are converted into
$n_{s}=\frac{n}{\log_{2}M}$ symbols with each symbol containing $m=\log_{2}M$
bits. We have
${\bf{x}}_{j}=\left({{x}}_{j}^{1},\cdots{{x}}_{j}^{n_{s}}\right)$. For every
symbol slot ${{s}}\in\left\\{1,2,\ldots,n_{s}\right\\}$, the SCMA encoding
(mapping) and detection is performed for all the $J$ users. The SCMA detection
is carried out by MPA. From the MPA detector, the soft values i.e. the
probability of an user’s data being a particular symbol are considered. These
symbol-wise probability values are converted to bitwise probability values
which are fed to the individual polar decoder. Every polar decoder employs
CRC-aided list decoding algorithm. The estimated bits across all symbol
duration are accumulated and finally the estimated frame $\hat{{\bf{u}}}_{j}$
is obtained.

Figure 5: SER performance of various polar coded $6\times 4$ SCMA systems.

Fig. 5 shows the performance of polar coded SCMA systems with various
parameters. In recent communication standards, the short polar codes are
considered for error correction [14]. Therefore, we have shown the results for
polar codes of block-lengths $n=64$ and $n=256$. For decoding, CRC-aided list
decoding is used. The CRC code employed is of length 16 with generator
polynomial $g(x)=x^{16}+x^{12}+x^{5}+1$. This code is known as CRC-16-CCITT.
The list size of $L=4$ is considered. The performance can be further improved
by considering a higher value of list size. Observe from Fig. 5 that the with
the help of short polar code of length $n=64$, the performance can be
drastically improved from that of the uncoded SCMA system. With a slightly
higher value of block-length $n=256$, the performance can be further enhanced
as shown in Fig. 5.

The selection of the frozen bits known as the construction process is an
important step for polar coding. Historically, polar codes have been
constructed by Bhattacharyya-parameter-based algorithm designed for binary
erasure channels (BP-BEC). The other popular methods are Monte-Carlo-based
method, density-evolution-based method, Gaussian-approximation (GA) method,
Tal and Vardy method, etc. The features considered for BP-BEC, Monte-Carlo-
based method, density-evolution-based method GA and Tal and Vardy method are
Bhattacharyya parameter, BER values, probability density function (PDF) of the
LLRs, mean of the LLRs and approximate PDF of the LLRs respectively. Here, we
consider Monte-Carlo-based method for the construction of polar codes in the
multiple-access channel or uplink system.

##  IV Imperfect Channel Estimation

Symbol detection at the BS requires accurate information of a channel state
response of each communicating user node. The received signal is the
superposition of each device’s data symbol and their channel information.
Therefore, accurate channel estimation at the BS is complex or impossible due
to large number of IoT devices with HNOMA schemes. Since the BS needs more
parameters estimation in the uplink scenario than the downlink. The channel
impulse response (CIR) at the BS is expressed as
$\textbf{h}=\textbf{h}_{\mathrm{est}}+\textbf{e}$, where h,
$\textbf{h}_{\mathrm{est}}$, and e is the actual CIR, estimated CIR and error
in channel estimation, respectively. e can be modeled as complex Gaussian
random vector with zero mean vector and the covariance matrix
$\sigma_{\mathrm{e}}^{2}\mathbf{I}$. Further, the variance of error depends on
the SNR level and is expressed as
$\sigma_{\mathrm{e}}^{2}=\sigma_{\mathrm{h}}^{2}/(1+\rho\text{SNR})$, where
$\sigma_{\mathrm{h}}^{2}\mathbf{I}$ is the covariance matrix of h and $\rho$
indicates quality of channel estimation. Further, $\rho=\infty$ represents
perfect channel estimation in the detection and estimation process. Therefore,
the impact of channel estimation should be considered for massive IoT
applications in practice. Some better channel destination algorithms can also
be explored in near future for NOMA and HNOMA-based systems.

Further, large channel gain differences among the groups in a cluster leads to
a better channel estimation in HNOMA. However, imperfect channel information
also results in residual error in the combined SIC and MPA based detection
while subtracted the stronger user’s signal from the received signal.
Therefore, spectrally efficient HNOMA-based systems, channel estimation
algorithms and their effect can be explored in near future.

##  V Example of HNOMA-based System in Uplink Scenario

We show an example to achieve massive connectivity in IoT using the HNOMA. To
reduce the processing complexity and latency, low power IoT devices are
divided into multiple clusters and each cluster has several groups of devices
depending on their channel gains. For a single cluster, the received signal at
BS is the superposition of each group’s signal and complex additive white
Gaussian noise. The detection performance in HNOMA depends on the channel
estimation error and noise as shown in Fig. 6. In imperfect channel estimation
$\sigma_{\mathrm{h}}^{2}=1$ is used. Overloading factor $\lambda=300\%$ and
$Z=4$ are used in Fig. 6. Imperfect channel estimation limits the performance
of HNOMA-based System, as observed in Fig. 6. However, the impact of channel
estimation can be minimized using some coding techniques and complex channel
estimation algorithms in the system to achieve high reliability in IoT
networks.

Figure 6: Impact of imperfect channel estimation on SER performance over
Rayleigh fading in HNOMA-based systems.

##  VI Research challenges in upcoming IoT networks

###  VI-A Efficient energy enhancement

In next-generation wireless networks, several kinds of BS like micro, femto
and pico will be deployed to get ultra-high density of users. Energy
efficiency optimization will provide green future cellular networks
infrastructure. In general, using smaller cells, BS power consumption can be
reduced. Further, letting BS go to sleeping mode during light or no traffic
load scenarios result in higher energy efficiency in the network. Therefore,
dynamic management of users and BS in the wireless network reduce the power
consumption, however, it is a difficult task at large scale level. Therefore,
the virtual distribution of resources and their optimization can be a
strategic approach to optimize the energy efficiency of 5G and beyond
networks. Further, simultaneous wireless information and power transfer
(SWIPT) and wireless power transmission can be used in IoT networks.

###  VI-B Asynchronous communications

Most of the data in upcoming applications will be short, bursty and
asynchronous in nature. Therefore, we need some signal processing techniques
which will shift the paradigm from current systems to asynchronous and non-
orthogonal systems. Further, different mobility level of devices in the
massive IoT will pose severe challenges to achieve lower latency. Therefore,
detection and estimation must be robust to asynchronous signaling and must
possess the capability to deal with uncoordinated interference in next-
generation wireless networks. Performance of HNOMA and NOMA-based systems can
be analyzed in asynchronous and high mobility scenarios for 5G and beyond
networks.

###  VI-C Optimum devices’ grouping and activity detection

The performance of the HNOMA mainly depends on the grouping of devices in a
cluster. Grouping of devices can be done using their SNR values at the BS.
Similar SNR devices are partitioned into the same group in a cluster. However,
high mobility of devices and fading can make the user paring difficult in the
massive networks to achieve higher spectral efficiency. In literature, some
sub-optimal iterative methods are proposed for efficient user paring. Further,
using a large data set based on users’ history, efficient user paring can be
achieved by employing machine learning approach in complex and large wireless
networks.

###  VI-D Robust and efficient signal processing

In some applications like the smart power grid and underwater communications,
ambient noise may not be Gaussian in nature i.e., impulsive noise. Therefore,
the performance of optimal detection methods (designed using Gaussian noise)
can deteriorate significantly. To remove the outliers in large data size using
conventional methods like time series before the detection/estimation is a
challenging task in massive IoT network due to a higher dimension of the data.
Hence, some non-linear signal processing techniques can be used to efficiently
remove/reduce the impact of the impulse noise before the decision in the IoT
network. Further, recently deep learning based wireless communication
detection and estimation method can also use to overcome the effect of impulse
noise in wireless network. Furthermore, some robust detection/estimation
methods are required to overcome the impact of channel estimation error and
hardware impairments, in HNOMA-based IoT applications to achieve massive
connectivity.

##  VII Deep learning based detection

Humans’ daily living activities will fully integrate to various devices to get
a comfortable, sustainable and efficient lifestyle [15]. Upcoming 5G and
beyond networks will be based on data-driven and machine learning (ML)
approaches due to extremely high data rate with heterogenous devices and
applications. ML-based smart networks will be adaptive, efficient, and
predictive in nature. Therefore, human intervention will be significantly
reduced in smart IoT applications. Further, large IoT network of trillion
connected devices will be enhanced using ML techniques.

ML or deep learning will also play a vital role in the health care system to
get an easy medical facility in remote areas and early detection of diseases.
Further, seamlessly continuous wireless monitoring and assistance will also
benefit old age people in society. Therefore, ML-based solutions give
adaptability, scalability, and flexibility in IoT applications.

Higher spectral and power-efficient communication systems can be accomplished
using deep learning-based detection and estimation methods. The availability
of large data size will benefit the deep neural network (DNN) for efficient
training and parameters optimization. DNN can provide simple and power
efficient optimization of large and complex 5G and beyond networks. Since
conventional approaches may be complex and power inefficient in 6G network.
For example, recently deep learning based techniques are used in massive MIMO
(multiple-input and multiple-output), channel estimation, D2D (device-to-
device) for symbol detection, power optimization, and localization in wireless
systems [4]. Another application of deep learning can be used in cognitive
radio network for primary user detection and adaptive spectrum management in
6G networks [10]. Further, personal data security and privacy will be great
concerns in next-generation wireless networks. Therefore, ML methods can also
enhance security by detecting abnormal behaviors and intrusions in network
before they happen.

Next, we highlight the importance of DNN for symbol detection in HNOMA. In
Fig. 7, the average SER performance of HNOMA using DNN and MPA based detection
is shown. DNN outperforms MPA at high SNR as compared to MPA due to effective
learning of DNN. MPA performance is also limited by interference in HNOMA
system. DNN with input and output, and three hidden layers (with $480$, $10$,
$10$ neurons) trained off-line at $\text{SNR}=30$ dB. DNN also does not need
channel information explicitly to decode the users’ data symbols, unlike MPA.
The training SNR and number of hidden layers are optimized using multiple
trials. Therefore, DNN can help to decode the information in complex and large
systems, especially in interference-limited and imperfect channel scenarios.

Figure 7: Comparison between DNN and MPA based detection of hybrid non-
orthogonal transmission-based system over Rayleigh fading.

Further, HNOMA can be analyzed with massive MIMO technology using conventional
and ML-based detection methods to achieve the massive connectivity in low
powered IoT network. However, HNOMA with massive MIMO needs some simple and
efficient detection methods to deploy this technology in practice. Therefore,
the processing power in complex wireless systems needs to be minimized along
with transmitted power to fulfill the demands of next-generation green
networks. Since at a time, the active devices are very less as compared to the
total connected devices in the network. Therefore, ML-based active device
detection and their data symbol estimation can also be employed in IoT network
for grant-free communication to achieve low latency. .

##  VIII Conclusion

We have highlighted the importance of HNOMA to connect low power trillion
devices in upcoming next-generation IoT applications. The HNOMA can achieve
higher spectral efficiency by serving multiple users at a time in the network.
Further, the impact of channel estimation error on the detection performance
is highlighted. The inclusion of polar codes can benefit the massive IoT
networks to achieve the reliability in adverse and channel mismatch scenarios
as highlighted. Further, some research challenges and their solutions are
suggested to fulfill the demand of next-generation IoT networks in this paper.
At last, we have suggested and proposed deep learning-based smart wireless
system design for HNOMA. Since, interference and channel impact can be
minimized in HNOMA/NOMA-based systems using deep learning. Therefore, deep
learning and data-driven solutions will play an important role in next-
generation wireless networks.

## References

  * [1] S. Sharma, K. Deka, V. Bhatia, and A. Gupta, “Joint power-domain and SCMA-based NOMA system for downlink in 5G and beyond,” _IEEE Communications Letters_ , vol. 23, no. 6, pp. 971–974, 2019.
  * [2] Y. Y. Nam, “Multiuser activity and data detection via sparsity-blind greedyrecovery for uplink grant-free NOMA,” _IEEE Communications Letters_ , vol. PP, no. PP, pp. 1–4, 2019.
  * [3] Y. Liu, H. Zhang, K. Long, A. Nallanathan, and V. C. M. Leung, “Energy-efficient subchannel matching and power allocation in NOMA autonomous driving vehicular networks,” _IEEE Wireless Communications_ , vol. 26, no. 4, pp. 88–93, 2019.
  * [4] P. Yang, Y. Xiao, M. Xiao, and S. Li, “6G wireless communications: Vision and potential techniques,” _IEEE Network_ , vol. 33, no. 4, pp. 70–75, 2019\.
  * [5] Q. Wang, R. Zhang, L.-L. Yang, and L. Hanzo, “Non-orthogonal multiple access: A unified perspective,” _IEEE Wireless Communications_ , vol. 25, no. 2, pp. 10–16, 2018.
  * [6] D. Wan, M. Wen, F. Ji, H. Yu, and F. Chen, “Non-orthogonal multiple access for cooperative communications: Challenges, opportunities, and trends,” _IEEE Wireless Communications_ , vol. 25, no. 2, pp. 109–117, 2018.
  * [7] K. Yang, N. Yang, N. Ye, M. Jia, Z. Gao, and R. Fan, “Non-orthogonal multiple access: achieving sustainable future radio access,” _IEEE Communications Magazine_ , vol. 57, no. 2, pp. 116–121, 2018.
  * [8] L. Liang, H. Ye, and G. Y. Li, “Toward intelligent vehicular networks: A machine learning framework,” _IEEE Internet of Things Journal_ , vol. 6, no. 1, pp. 124–135, 2018.
  * [9] Y. Sun, Z. Ding, X. Dai, and G. K. Karagiannidis, “A feasibility study on network NOMA,” _IEEE Transactions on Communications_ , vol. 66, no. 9, pp. 4303–4317, 2018.
  * [10] C. Jiang, H. Zhang, Y. Ren, Z. Han, K.-C. Chen, and L. Hanzo, “Machine learning paradigms for next-generation wireless networks,” _IEEE Wireless Communications_ , vol. 24, no. 2, pp. 98–105, 2016.
  * [11] D. Tse and P. Viswanath, _Fundamentals of wireless communication_. Cambridge University Press, 2005.
  * [12] E. Arikan, “Channel polarization: A method for constructing capacity-achieving codes for symmetric binary-input memoryless channels,” _IEEE Transactions on Information Theory_ , vol. 55, no. 7, pp. 3051–3073, July 2009.
  * [13] I. Tal and A. Vardy, “List decoding of polar codes,” _IEEE Transactions on Information Theory_ , vol. 61, no. 5, pp. 2213–2226, May 2015\.
  * [14] Peiying Zhu, “Polar code for 5G NR,” ITW 2018 Keynote, available online: http://itw2018.org/static/resource/Keynote-Peiying.pdf.
  * [15] B. Qolomany, A. Al-Fuqaha, A. Gupta, D. Benhaddou, S. Alwajidi, J. Qadir, and A. C. Fong, “Leveraging machine learning and big data for smart buildings: A comprehensive survey,” _IEEE Access_ , vol. 7, pp. 90 316–90 356, 2019\.