# MO-DEHB: Evolutionary-based Hyperband for Multi-Objective Optimization

Noor Awad1 Equal Contribution Ayushi Sharma111footnotemark: 1 Philipp Muller2
Janek Thomas2 Frank Hutter1 1Department of Computer Science, University of
Freiburg, Germany
2Department of Statistics, University of Munich, Germany
<EMAIL_ADDRESS>
###### Abstract

Hyperparameter optimization (HPO) is a powerful technique for automating the
tuning of machine learning (ML) models. However, in many real-world
applications, accuracy is only one of multiple performance criteria that must
be considered. Optimizing these objectives simultaneously on a complex and
diverse search space remains a challenging task. In this paper, we propose MO-
DEHB, an effective and flexible multi-objective (MO) optimizer that extends
the recent evolutionary Hyperband method DEHB. We validate the performance of
MO-DEHB using a comprehensive suite of 15 benchmarks consisting of diverse and
challenging MO problems, including HPO, neural architecture search (NAS), and
joint NAS & HPO, with objectives including accuracy, latency and algorithmic
fairness. A comparative study against state-of-the-art MO optimizers
demonstrates that MO-DEHB clearly achieves the best performance across our 15
benchmarks.

##  1 Introduction

Designing the architecture and tuning hyperparameters for complex machine
learning (ML) methods can be costly, time-consuming, and error-prone. In
recent years, there has been a significant amount of research on methods such
as hyperparameter optimization (HPO) Feurer and Hutter (2019) and Neural
Architecture Search (NAS) Elsken et al. (2019) to address these challenges.
Various methods have been developed to efficiently solve these tasks, however,
many of them only focus on optimizing a single objective, such as predictive
accuracy. In real-world applications, practitioners often need to optimize
multiple objectives that may conflict with each other, such as finding a
smaller model with minimal drop in predictive performance. This can be a
difficult task, particularly when deploying models on resource-constrained
devices. Another omnipresent example is ensuring predictive performance while
guaranteeing a certain level of fairness. This poses a challenge and
significant trade-offs between performance and different fairness notions need
to be explored Corbett-Davies et al. (2017). Despite efforts to adapt single-
objective (SO) optimizers for multi-objective (MO) settings, there have been
relatively few studies that have examined the use of these methods for HPO and
NAS in complex search spaces.

Because HPO and NAS require a lot of computational resources when executed
naively, multi-fidelity (MF) techniques have gained popularity. These methods
exploit cheap approximations, such as evaluations at fewer epochs or on
smaller data subsamples, of the objective function to speed up the
optimization. Successive Halving (SH) Jamieson and Talwalkar (2016) is a first
simple, yet effective MF method that allocates resources to configurations
that perform sufficiently well, promoting them to run for larger computational
budgets, rather than evaluating poor ones for the maximum resource budget. The
Hyperband (HB) Li et al. (2017) algorithm instantiates SH with different
levels of its lowest fidelity, to guard from sub-optimal choices of fidelity
levels. While SH and HB are effective in choosing the fidelity level of the
configurations they evaluate, they rely on simple random search for choosing
those configurations and therefore do not learn over the course of the
algorithm. To address this issue, BOHB Falkner et al. (2018) introduces the
use of Bayesian Optimization (BO) via Tree Parzen Estimates (TPE) Bergstra et
al. (2011), and DEHB Awad et al. (2021) uses an evolutionary algorithm (EA)
based on Differential Evolution (DE) Storn and Price (1997) to further improve
the search at each budget level.

A SO optimizer can be extended to handle MO by using (1) scalarization
technique (i.e., optimizing a linear combination of multiple competing
objectives to a single objective point) or (2) non-dominated sorting (NDS)
(i.e., approaches that take the geometry of the whole Pareto front into
account). Although scalarization is a simple convenient technique in MO, it
has multiple shortcomings: (1) it captures only restricted pareto shapes, (2)
linear scalarization can cause failures for non-convex pareto fronts, (3) it
encourages a single direction exploration, (4) it fails to recover the full
pareto front if the weights are not chosen carefully and (5) it is greatly
affected by re-scaling the values of competing objectives.

To mitigate the aforementioned issues of scalarization methods, in this work
we extend DEHB to optimize multiple objectives by the use of NDS. To
summarize, our main contributions are:

  1. 1.

We extend the HB component in DEHB to MO considering two variants: (1) NSGA-II
and (2) EpsNet.

  2. 2.

We adapt the evolutionary search component in DEHB to handle MO by
incorporating NDS and dominated hypervolume contribution in DE selection
strategy to approximate the desired trade-offs between the objectives.

  3. 3.

We comprehensively evaluate our two MO-DEHB variants on a wide range of 15 MO
problems, and demonstrate that they establish the new state of the art in MO
compared to many state-of-the-art baselines.

The paper proceeds as follows: Section 2 presents related work. Section 3
provides background on MO, a summary of MO evolutionary algorithms (EAs), and
an overview of DEHB. Section 4 presents our proposed method, MO-DEHB. Section
5 empirically evaluates MO-DEHB on a broad range of 15 benchmarks from three
benchmark families, showing that it defines a new state-of-the-art
performance. Finally, Section 6 concludes the paper.

##  2 Related Work

BO and EAs are popular choices for solving MO HPO problems. MO EAs are usually
based on Pareto dominance (e.g., NSGA-II Deb et al. (2002a)), decomposition
(e.g., MOEA/D Zhang and Li (2007a)) or indicators (e.g., SMS-EMOA Emmerich et
al. (2005)) Emmerich and Deutz (2018). MO BO algorithms are usually based on
scalarization (e.g., ParEGO Knowles (2006)), aggregating acquisition functions
(e.g., SMS-EGO Ponweiser et al. (2008)), multiple acquisition functions (e.g.,
MultiEGO Jeong and Obayashi (2005)) or information theoretic (e.g., PESMO
Hernández-Lobato et al. (2016)). A recent review of these methods and their
use in MO HPO can be found in Karl et al. (2022). However, these methods do
not generally support multi-fidelity evaluations. Recent work introduces the
first MO HB method by using a simple scalarization method Schmucker et al.
(2020) or NDS Schmucker et al. (2021b) to overcome the shortcomings of
scalarization methods. The NDS approach has been used to jointly optimize
hyperparameters and hardware configuration Salinas et al. (2021a).

##  3 Background

###  3.1 Multi-objective Optimization

Let $f:\Lambda\to\mathds{R}^{n}$ be a function over the search space
$\Lambda$. A multi-objective optimization problem aims to minimize a function
$f:\Lambda\to\mathds{R}^{n}$ over the search space $\Lambda$ where $n$ is the
number of the objectives.

$\displaystyle\min_{\lambda\in\Lambda}f(\lambda)=\min_{\lambda\in\Lambda}\left(f_{1}(\lambda),f_{2}(\lambda),\ldots,f_{n}(\lambda)\right)\
.$ (1)

In our application this would represent $n$ different performance measures of
a machine learning model, configured by the hyperparameter vector $\lambda$.
One objective is the estimated generalization error and others can, among
other things, be related to model size, inference time and fairness. Given two
configurations $\lambda^{(1)},\lambda^{(2)}\in\Lambda$, we say
$\lambda^{(1)}\succ\lambda^{(2)}$ if $\lambda^{(2)}$ is dominated by
$\lambda^{(1)}$, that is, $f_{i}(\lambda^{(2)})\geq
f_{i}(\lambda^{(1)})\forall i\in[n]$ and
$f_{j}(\lambda^{(2)})>f_{j}(\lambda^{(1)})$ for at least one $j\in[n]$.
Configurations $\lambda^{*}$ that are not dominated by any other candidates
are called Pareto optimal, and the set of Pareto optimal configurations forms
the Pareto front. Any MO algorithm aims to find the true Pareto set or an
approximation set of non-dominated solutions. In practice, it is rarely
possible to compute the true Pareto set due to the complexity of the search
space $\Lambda$, potential noise in the objective functions and the difficulty
in optimizing multiple competing objectives.

Hypervolume (HV) is one of the most commonly used quality indicators in MO
Zitzler and Thiele (1998); Knowles et al. (2003). The hypervolume indicator
$\mathcal{H}(A)$ of A is the measure of the region weakly dominated by A and
bounded above by a reference point $r$:

$\displaystyle\mathcal{H}(A)=\Lambda({\lambda\in\mathcal{R}^{d}|\exists p\in
A:p\succeq\lambda\wedge\lambda\succeq r}),$ (2)

where $\Lambda$ is the Lebesgue measure. HV has been shown to promote the
convergence towards the Pareto front, as well as to ensure a representative
distribution of points along the front. Maximzing HV is equivalent to
identifying the Pareto set, and with a limited number of configurations,
maximizing HV metric results in subsets of the Pareto front that are evenly
distributed Beume et al. (2007); Guerreiro et al. (2021).

###  3.2 Multi-objective Evolutionary Algorithms

Non-dominated sorting (NDS) is a widely used method in multi-objective EAs
Srinivas and Deb (1994). It is used to sort candidates based on their non-
domination level, which is determined by partitioning the objective space into
fronts. NDS partitions an objective space set into fronts $F_{1}\prec
F_{2}\prec\cdots\prec F_{n}$ such that a configuration $\lambda^{1}\in F_{i}$
outperforms another candidate $\lambda^{2}\in F_{j}$ with respect to all
objectives if $i<j$. We can easily compute the partitioning by computing all
non-dominated points $F_{1}$, removing them, then computing the next layer of
non-dominated points $F_{2}$, and so on.

The candidates within the same front are further sorted by their crowding
distance Deb et al. (2002b), which is calculated by comparing the distance
between a candidate and its two nearest neighbors for each objective function.
To calculate Crowding distance, we follow the following steps for every
objective $f_{j}$: (i) Sort points by $f_{j}$. (ii) Normalize scores to [0,1].
(iii) The boundary solutions (solutions with smallest and largest function
values) are assigned an infinite distance value. Therefore, they are always
selected. (iv) For each point a distance score, which is the distance between
its 2 next-neighbors w.r.t. the sorting of $f_{j}$ is assigned. The overall
crowding-distance value is calculated as the sum of individual distance values
corresponding to each objective. Another approach to sort candidates within
the same front is the EpsNet exploration strategy Salinas et al. (2021b). This
strategy starts by removing the first point from the ”front” set (which
represents all points on the front that we aim to sort) and placing it in an
empty set, referred to as the ”ranked” set. Subsequently, the remaining points
in the ”front” set are iteratively selected based on their Euclidean distance
from the points already in the ”ranked” set. Specifically, the strategy
selects the point that has the highest Euclidean distance with respect to its
closest point in the ”ranked” set. The selected point is then removed from the
”front” set and placed into the ”ranked” set. This process continues
iteratively until all points have been ranked. By selecting the point that has
the highest distance to its closest point in the ”ranked” set, the strategy
ensures that the selected candidates are as dissimilar as possible. This can
be useful for exploring different regions of the solution space and for
identifying trade-offs between different objectives.

###  3.3 Overview of DEHB

DEHB is a recently proposed multi-fidelity evolutionary optimizer that uses a
new amalgam of Differential Evolution and Hyperband for HPO Awad et al.
(2021). DEHB fulfills the desiderata of a good HPO optimizer such as: (1)
conceptually simple (2) computationally cheap (3) strong final and anytime
performance and (4) effectively takes advantage of parallel resources. Adding
to that, DEHB is very effective in optimizing high dimensional and discrete
search spaces. A key design principle of DEHB is to share information across
the runs it executes at various fidelities. Unlike HB that only prompt the top
performing configurations from lower fidelity to higher one, DEHB maintains a
sub-population for each fidelity and uses an evolutionary search based on DE.
Each sub-population is evolved and updated using mutation, crossover and
selection operations. Figure 1 illustrates the framework of a DEHB iteration
where $\eta=3$ minimum fidelity $b_{min}=3$ and maximum fidelity $b_{max}=27$.
The first SH bracket of DEHB iteration initializes all the sub-populations
randomly and follows the SH subroutine as in vanilla HB. Starting from the
second iteration, DEHB performs a modified SH using DE evolution. For each SH
iteration (top to bottom), DEHB collects the top performing configurations for
each fidelity level in a parent pool which is used to select the parents for
the mutation instead of using the current sub-population itself. By doing so,
the DE evolution can still incorporate information from the good performing
region w.r.t. the lower fidelity. When the size of a sub-population is small
111if the population size is less than 3 (i.e., minimum number of parents)
needed to perform the used rand1 mutation strategy in DE which usually occurs
on full fidelity level, a global population that consists of all the
configurations from all the sub-population is used to select additional
parents to perform mutation.

The crossover then combines each individual and its corresponding mutant
vector to generate the final offspring. Next, the selection operation updates
the sub-population on the current fidelity if the new offspring is better that
its parent. The sub-populations (right to left) are forwarded and evolved from
a SH iteration to a next one. The final evolved sub-populations in each SH are
then forwarded to start a new DEHB iteration.

Figure 1: DEHB iteration

##  4 MO-DEHB: Algorithm Description

Multi-objective EAs usually navigate the objective space and optimize the
Pareto front using NDS as discussed in Section 3.2. To leverage suitability
and flexibility of DEHB for the MO domain, we follow a similar approach and
apply two changes. First, we follow a similar approach as proposed in
Schmucker et al. (2021a) to adapt the HB component for selecting the top-
performing configurations in MO domain using NDS with the use of NSGA-II or
Epsnet to rank the Pareto fronts resulting in two variants. Second, we
extended the evolutionary search using DE to MO based on NDS and dominated HV.
The following subsections explain these changes in more detail.

###  4.1 Multi-objective Hyperband

The single-objective HB starts by instantiating a SH subroutine by making use
of the input parameters (i.e., minimum fidelity, maximum fidelity and $\eta$)
that used to trade-off the number of configurations and fidelity. The first
iteration of SH samples $N$ random configurations with a small initial budget,
evaluates them and forwards the top $1/\eta$ configurations to the next higher
budget. This process is repeated until the highest budget is reached.
Subsequent SH iterations promote the top $1/\eta$ configurations from the
lower fidelity to the parent pool which is further used in DE evolution. To
select the top-performing configurations, we adapt HB for the MO domain by
employing a methodology similar to that presented in Schmucker et al. (2021b).
This approach is based on NDS to rank the configurations in the lowest budget
by taking into account the global geometry of the Pareto front approximations
in two stages. Referring to Figure 1, for each SH bracket we first rank the
population into multiple fronts using NDS. Then, to rank the candidates in the
same front (i.e., to choose a subset of the population from a specific front)
we employ one of two strategies: The first strategy is based on NSGA-II (we
dub it MO-DEHBNSGA-II) which uses the crowding-distance to select the
candidates with the lower density in the objective space. The second strategy,
EpsNet (giving rise to MO-DEHBEPSNET), aims to iteratively select candidates
by maximizing the distance between them and the previously chosen
configurations in the objective space. Both of these strategies aim to select
a sparse set of candidates that effectively represents the front, with the
goal of promoting diversity of the parent pool when we later use them to
perform the mutation in DE search to find good solutions.

For example, in the $SH_{1}$ bracket in Figure 1, we rank the population,
$pop_{c_{1}}^{b_{1}}$, into multiple fronts using NDS. Let us assume
$pop_{c_{1}}^{b_{1}}$ is divided into three fronts $F_{1}$, $F_{2}$ and
$F_{3}$. Next, assuming the number of candidates to be promoted to higher
fidelity (which is determined by HB) is greater than the number of candidates
in $F_{1}$ but smaller than $F_{2}$, we select all the candidates from $F_{1}$
and select the remaining ones from $F_{2}$ by either crowding-distance or
EpsNet.

###  4.2 Multi-objective DE

In DEHB, the configurations at each fidelity level form a sub-population and
are evolved using an evolutionary search based on the following DE operations:
mutation, crossover and selection. For each SH iteration (top to bottom in
Figure 1), the parent pool is used to select three random parents to perform
the mutation operation (i.e., rand1 mutation strategy Awad et al. (2021)) and
generate a new mutant configuration. We then apply the binomial crossover
between the generated mutant and the target configuration to generate the
final offspring. Next, the offspring is evaluated and compared to its target
configuration to select which one performs better to replace the other. To
adapt DE to MO, we extend the selection operation to handle multiple
objectives as we present in Algorithm 1. We utilize the global population in
DEHB that contains the most recent sub-populations that have evolved across
all fidelities. We incorporate new offspring into it and sort it according to
NDS, resulting in multiple fronts. The front rankings of both the target and
offspring are then determined, which helps in positioning them in the
objective space with their respective trade-offs. If the offspring is on a
lower ranked front than the target, it replaces the target in the sub-
population. If the target is on a lower ranked front, the sub-population is
not updated. If they are on the same front, then the one with the least
dominated hypervolume in the last front is chosen to be replaced by the
offspring, as it has the least contribution in approximating the desired
trade-offs.

Input:

$\rho$ $\leftarrow$ parent_pool; - Parent pool from lower fidelity

$pop$ $\leftarrow$ population; - Population for current fidelity

$global\\_pop$ $\leftarrow$ \- Population across all fidelity

Output: Return evolved popluation

1 for  _$parent$ in $pop$_ do

2 $mutant$ = generate_mutant($\rho$);

3 $offspring$ $\leftarrow$ crossover($mutant$, $parent$);

4 $f$ = evaluate($offspring$);

5 $C$ $\leftarrow$ MO-selection($pop$,$parent$,$offspring$);

6 update population($C$, $offspring$, $global_{p}op$)

7 end for

8

9 Function  _MO-selection(_$global\\_pop,parent,offspring$_)_:

10 $[F_{1},...,F_{m}]\xleftarrow{}NDS(global\\_pop)$

11 $R\\_parent,R\\_offspring$ =
get_front_ranking($parent,offspring,[F_{1},...,F_{m}]$)

12 if  _$R\\_parent >R\\_offspring$_ then

13 return $parent$

14 else if  _$R\\_parent <R\\_offspring$_ then

15 return $offspring$

16 else

17 $least\\_hv\\_contributor$ = minimum_HV($F\\_m$)

18 return $least\\_hv\\_contributor$

19

20

21

return $pop$

Algorithm 1 MO-DE

##  5 Experiments

We evaluated MO-DEHB on a broad collection of MO problems with a total of $15$
benchmarks from diverse domains: (i) NAS (9 benchmarks), (ii) joint NAS & HPO
(5 benchmarks) and (iii) algorithmic fairness (1 benchmark). In the following
subsection we explain our benchmarks, experimental setup and results in
detail.

###  5.1 Benchmarks

####  5.1.1 Neural Architecture Search

The complexity of ML models has led to a widespread use of NAS to find
efficient architectures Elsken et al. (2019). However, NAS methods often focus
solely on achieving the optimal performance and do not take into account other
important objectives such as latency. To address this, we conduct experiments
on NAS-Bench-101 Ying et al. (2019), NAS-Bench-1shot1 Zela et al. (2020) and
NAS-Bench-201 Dong and Yang (2020), which are 9 tabular benchmarks. Our
experiments aim to optimize for both validation accuracy and model size. For
more information about these benchmarks, refer to HPOBench suite Eggensperger
et al. (2021).

####  5.1.2 Joint NAS & HPO

While there has been extensive research on optimizing architecture and model
hyperparameters separately, few studies have investigated optimizing them
together. In this experiment, we use two raw benchmarks. The first one,
proposed by Izquierdo et al. (2021), involves tuning Convolutional Neural
Networks (CNNs) composed of a variable number of fully-connected layers and a
15D mixed and hierarchical search space. These networks are trained on the
Oxford-Flowers dataset Nilsback and Zisserman (2008) and Fashion-MNIST Xiao et
al. (2017). For these two datasets, we optimize validation accuracy and model
size. For more information on the datasets and search spaces, refer to
Izquierdo et al. (2021). The second benchmark uses three surrogate benchmarks
Zela et al. (2022) from the recently introduced JAHS-Bench-201 suite Bansal et
al. (2022) (CIFAR-10, Colorectal-Histology and Fashion-MNIST), and we optimize
validation accuracy and latency.

####  5.1.3 Algorithmic Fairness

We adopt the experimental setup from Schmucker et al. (2021a) and train a
multi-layer perceptron (MLP) to optimize both predictive accuracy and
fairness. Specifically, we use the fairness metric of difference in
statistical parity (DSP) which measures the absolute difference in predicted
outcome between two subgroups distinguished by a protected attribute (i.e.,
sex). The MLP is trained on the Adult dataset Kohavi and others (1996). For
more information about this raw benchmark, refer to Schmucker et al. (2021a).

We provide more details about the benchmarks in Appendix B.

###  5.2 Experimental Setup

Evaluation protocol We ran our experiments for $10$ repetitions for each
benchmark using different seeds. We set the wallclock runtime limit to $24$h
for raw benchmarks (Flower, Fashion and Adult) and $4$h for surrogate and
tabular benchmarks. For all benchmarks, we further introduce a limit on the
number of target algorithm executions, determined by
$\big{\lceil}20+80\times\sqrt{\text{\\#HPs}}\big{\rceil}$, similar to
Pfisterer et al. (2022) where HPs denotes the number of hyperparameters. We
report the difference in log HV from the best possible Pareto set to the one
currently observed by an optimizer:
$\text{LogHVDiff}(\text{HV})~{}=~{}log_{10}(\text{HV}-\text{HV}_{emp})$ where
$\text{HV}_{emp}$ is the normalized HV of all observed configurations over all
the runs.

Hardware For all surrogate and tabular benchmarks, we run experiments on a
compute cluster equipped with Intel(R) Xeon(R) Gold 6242 CPUs @ 2.80GHz. For
the raw benchmarks, we ran all jobs on a compute cluster equipped with NVIDIA
GeForce RTX 2080Ti.

Baselines In our comparison, we compare our two proposed variants (MO-
DEHBNSGA-II and MO-DEHBEPSNET) against state-of-the-art baselines. We use two
multi-objective BO optimizers: QNPAREGO Daulton et al. (2020) and SMAC
Lindauer et al. (2022) and three multi-objective EAs: NSGA-III Blank et al.
(2019), AGE-MOEA Panichella (2019) and MOEA/D Zhang and Li (2007b). We further
add random search (RS) as a baseline. We use the default settings to run these
baselines and report that as well as more information about them in Appendix
A.

###  5.3 Results

|  |
---|---|---
|  |

Figure 2: Difference in dominated hypervolume of the Pareto front
approximations for the different methods with respect to the combined front
approximation over time on Imagenet and Cifar100 from NAS-Bench-201, Flowers,
JAHS-Fashion and JAHS-Colorectal-Histology from Joint NAS & HPO and Adult from
algorithmic fairness

|
---|---
|

Figure 3: We report summary-attainment-surfaces for on Imagenet and Cifar100
from NAS-Bench-201, Flowers, JAHS-Fashion and JAHS-Colorectal-Histology from
Joint NAS & HPO and Adult from algorithmic fairness. Upper and lower bound
correspond to the first and ninth summary-attainment-surface.

Figure 4: Average rank over time: We aggregate all 10 runs and compute the
average rank over all benchmarks.

In this section, for space limitations, we present the results for 6
representative benchmarks: 2 from NAS (Imagenet and Cifar100 from NAS-
Bench-201), 3 from joint NAS & HPO (Flowers, JAHS-Fashion and JAHS-Colorectal-
Histology) and algorithmic fairness on Adult dataset. The qualitatively
similar plots for all the benchmarks can be found in Appendix C. Figure 2
presents the difference in dominated HV of the Pareto front approximations
with respect to the combined front approximation over time on the 6 selected
benchmarks for the different baseline methods. Overall, the MO-DEHB variants
clearly perform best. As seen in Figure 2, both MO-DEHB variants outperform
all other methods in benchmarks such as JAHS-Fashion and fairness-Adult,
showing exceptional performance throughout the optimization process. However,
in benchmarks such as NAS-201-Cifar100 and JAHS-Colorectal-Histology, the MO-
DEHB variants performed poorly initially but later became competitive with
other methods. Interestingly, RS performed well on the NAS-201-Imagenet
benchmark, similar to QNPAREGO, and remained competitive till the end of
optimization. Additionally, SMAC performed well during the first phase of
exploration on this benchmark. The performance of the MO-DEHB variants varied
in some benchmarks, such as on the Flower dataset where MO-DEHBEPSNET
performed well initially, but later MO-DEHBNSGA-II performed better, although
it was outperformed by AGE-MOEA.

Figure 3 visualizes a summary of attainment surfaces to assess the method’s
ability to cover the entire Pareto front Knowles (2005). We show the first,
median and ninth attainment surfaces to visualize the distribution of Pareto
fronts achieved by the methods to simplify visually inspecting the differences
than plotting all 10 attainment surfaces. In Figure 3 for the Adult dataset,
it can be seen that MO-DEHB has a consistent performance as indicated by the
small difference between the upper and lower bound. While SMAC performs better
than the MO-DEHB variants on the Fashion and Colorectal-Histology benchmarks
from the JAHS-Bench, the MO-DEHB variants still exhibit consistent and good
performance. On the Imagenet benchmark, all methods perform similarly. This
may be due to the high evaluation limits and the tabular nature of the
benchmark.

###  5.4 Results Summary

We now compare MO-DEHB variants and the other baseline methods across all the
used benchmarks. We compute the average rank by hypervolume over time for each
method in Figure 4. Both MO-DEHB variants show exceptional performance and
perform very well early on. Later during the optimization process and for a
short period of time, RS, SMAC and QNPAREGO perform competitively. At the end,
both MO-DEHB variants show superior performance. We also report the mean and
standard deviation for log HV difference for 10 repetitions across all the
benchmarks for all the compared algorithms in Table 1.

| RS | MOEA/D | AGE-MOEA | NSGAIII | SMAC | QNPAREGO |  MO-DEHBEPSNET |  MO-DEHBNSGA-II
---|---|---|---|---|---|---|---|---
NAS-101-CifarA |  -2.897$\pm$0.042 |  -2.684$\pm$0.161 |  -2.901$\pm$0.033 |  -2.885$\pm$0.045 |  -2.858$\pm$0.028 |  -2.889$\pm$0.038 |  -2.895$\pm$0.038 | -2.916$\pm$0.0217
NAS101-CifarB |  -2.887$\pm$0.0349 |  -2.832$\pm$0.0515 |  -2.888$\pm$0.0394 |  -2.866$\pm$0.0297 |  -2.890$\pm$0.0338 |  -2.843$\pm$0.038 |  -2.893$\pm$0.025 | -2.905$\pm$0.037
NAS101-CifarC |  -2.896$\pm$0.040 |  -2.752$\pm$0.178 |  -2.876$\pm$0.041 |  -2.870$\pm$0.038 |  -2.856$\pm$0.038 | -2.924$\pm$0.027 |  -2.889$\pm$0.037 |  -2.894$\pm$0.031
NAS1shot1-SS1 |  -2.865$\pm$0.031 |  -2.795$\pm$0.074 |  -2.862$\pm$0.026 |  -2.853$\pm$0.023 |  -2.857$\pm$0.022 |  -2.865$\pm$0.028 | -2.867$\pm$0.0182 |  -2.851$\pm$0.029
NAS1shot1-SS2 |  -2.903$\pm$0.041 |  -2.826$\pm$0.057 |  -2.899$\pm$0.043 |  -2.884$\pm$0.039 |  -2.909$\pm$0.033 | -2.913$\pm$0.026 |  -2.905$\pm$0.035 |  -2.891$\pm$0.026
NAS1shot1-imagenet |  -0.487$\pm$0.024 |  -0.487$\pm$0.024 |  -0.487$\pm$0.024 |  -0.487$\pm$0.024 |  -0.484$\pm$0.021 |  -0.475$\pm$0.016 | -0.489$\pm$0.016 |  -0.474$\pm$0.031
NAS1shot1-cifar10 |  -1.982$\pm$0.018 |  -1.873$\pm$0.077 |  -1.983$\pm$0.018 |  -1.938$\pm$0.0402 |  -1.986$\pm$0.027 |  -1.986$\pm$0.0426 | -1.998$\pm$0.043 |  -1.99$\pm$0.025
NAS1shot1-cifar100 |  -1.066$\pm$0.013 |  -1.051$\pm$0.030 |  -1.066$\pm$0.013 |  -1.070$\pm$0.024 |  -1.073$\pm$0.017 |  -1.069$\pm$0.013 |  -1.069$\pm$0.027 | -1.0914$\pm$0.0191
Fashion |  -0.695$\pm$0.0945 |  -0.469$\pm$0.392 |  -0.286$\pm$0.120 |  -0.538$\pm$0.325 |  -0.740$\pm$0.056 | -0.970$\pm$0.034 | -0.919$\pm$0.043 | -0.780$\pm$0.103
Flower |  -1.066$\pm$0.013 |  -1.051$\pm$0.030 |  -1.066$\pm$0.013 |  -1.070$\pm$0.024 |  -1.073$\pm$0.017 |  -1.069$\pm$0.013 |  -1.069$\pm$0.027 | -1.091$\pm$0.019
JAHS-cifar10 |  -7.201$\pm$0.636 |  -6.059$\pm$0.628 |  -5.665$\pm$0.633 |  -6.096$\pm$0.661 |  -7.807$\pm$0.601 | -7.650$\pm$0.850 | -7.891$\pm$0.649 | -8.103$\pm$0.286
JAHS-color-histology |  -8.532$\pm$0.329 |  -8.324$\pm$0.759 | -7.910$\pm$0.533 |  -8.071$\pm$0.618 |  -9.191$\pm$0.468 |  -8.697$\pm$0.334 |  -9.128$\pm$0.596 | -9.387$\pm$0.583
JAHS-fashion |  -7.515$\pm$1.044 |  -5.813$\pm$0.949 |  -5.393$\pm$1.149 |  -5.813$\pm$0.949 |  -7.620$\pm$0.423 |  -7.582$\pm$0.441 |  -8.385$\pm$0.512 | -8.430$\pm$0.342
Adult |  -1.919$\pm$0.004 |  -1.889$\pm$0.024 |  -1.898$\pm$0.012 |  -1.890$\pm$0.006 |  -1.882$\pm$0.012 | -1.864$\pm$0.014 | -1.932$\pm$0.004 | -1.932$\pm$0.002
Table 1: A summary of mean $\pm$ standard deviation for the log HV difference
for $10$ receptions for all compared algorithms across all benchmarks

##  6 Conclusion

In this paper, we extended DEHB, a very effective evolutionary-based
hyperband, to multi-objective optimization in the context of HPO. We use
techniques that optimize the Pareto geometry based on non-dominating sorting
and propose two variants: (1) MO-DEHBNSGA-II and (2) MO-DEHBEPSNET. We
evaluate the performance of MO-DEHB using comprehensive benchmark suites
consisting of diverse and challenging MO problems from three benchmark
families: NAS, NAS+HPO and algorithmic fairness. We demonstrated that MO-DEHB
outperforms state-of-the-art MO optimizers on most of the benchmark problems.
These results suggest that MO-DEHB is a promising new MO optimizer that is
well-suited to a wide range of real-world optimization problems. For future
work, we aim to propose an asynchronous version of MO-DEHB that takes
advantage of parallel resources as an improved version of the original
parallel settings in DEHB. Our reference implementation of MO-DEHB is
available at https://anonymous.4open.science/r/MODEHB-F38E.

## Ethical Statement

There are no ethical issues.

## References

  * Awad et al. [2021] N. Awad, N. Mallik, and F. Hutter.  DEHB: Evolutionary hyberband for scalable, robust and efficient Hyperparameter Optimization.  In Proc. of IJCAI’21, pages 2147–2153, 2021.
  * Bansal et al. [2022] Archit Bansal, Danny Stoll, Maciej Janowski, Arber Zela, and Frank Hutter.  Jahs-bench-201: A foundation for research on joint architecture and hyperparameter search.  In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
  * Bergstra et al. [2011] J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl.  Algorithms for hyper-parameter optimization.  In Proc. of NeurIPS’11, pages 2546–2554, 2011.
  * Beume et al. [2007] Nicola Beume, Boris Naujoks, and Michael Emmerich.  Sms-emoa: Multiobjective selection based on dominated hypervolume.  European Journal of Operational Research, 181(3):1653–1669, 2007\.
  * Blank et al. [2019] Julian Blank, Kalyanmoy Deb, and Proteek Chandan Roy.  Investigating the normalization procedure of nsga-iii.  In International Conference on Evolutionary Multi-Criterion Optimization, pages 229–240. Springer, 2019.
  * Corbett-Davies et al. [2017] S. Corbett-Davies, E. Pierson, A. Feller, S. Goel, and A. Huq.  Algorithmic decision making and the cost of fairness.  In Proceedings of the 23rd acm sigkdd international conference on knowledge discovery and data mining, pages 797–806, 2017.
  * Daulton et al. [2020] S. Daulton, M. Balandat, and E. Bakshy.  Differentiable expected hypervolume improvement for parallel Multi-Objective Bayesian optimization.  In Proc. of NeurIPS’20, pages 9851–9864, 2020.
  * Deb et al. [2002a] K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan.  A fast and elitist multiobjective genetic algorithm: Nsga-ii.  IEEE transactions on evolutionary computation, 6(2):182–197, 2002\.
  * Deb et al. [2002b] Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan.  A fast and elitist multiobjective genetic algorithm: Nsga-ii.  IEEE transactions on evolutionary computation, 6(2):182–197, 2002\.
  * Dong and Yang [2020] Xuanyi Dong and Yi Yang.  Nas-bench-201: Extending the scope of reproducible neural architecture search.  arXiv preprint arXiv:2001.00326, 2020.
  * Eggensperger et al. [2021] K. Eggensperger, P. Müller, N. Mallik, M. Feurer, R. Sass, A. Klein, N. Awad, M. Lindauer, and F. Hutter.  HPOBench: A collection of reproducible multi-fidelity benchmark problems for HPO.  In Proc. of NeurIPS’21 Datasets and Benchmarks Track, 2021.
  * Elsken et al. [2019] T. Elsken, J. Metzen, and F. Hutter.  Neural Architecture Search: A survey.  JMLR, 20(55):1–21, 2019.
  * Emmerich and Deutz [2018] M. Emmerich and A. Deutz.  A tutorial on multiobjective optimization: fundamentals and evolutionary methods.  Natural computing, 17(3):585–609, 2018.
  * Emmerich et al. [2005] M. Emmerich, N. Beume, and B. Naujoks.  An emo algorithm using the hypervolume measure as selection criterion.  In International Conference on Evolutionary Multi-Criterion Optimization, pages 62–76. Springer, 2005.
  * Falkner et al. [2018] S. Falkner, A. Klein, and F. Hutter.  BOHB: Robust and efficient Hyperparameter Optimization at scale.  In Proc. of ICML’18, pages 1437–1446, 2018.
  * Feurer and Hutter [2019] M. Feurer and F. Hutter.  Hyperparameter Optimization.  In F. Hutter, L. Kotthoff, and J. Vanschoren, editors, Automated Machine Learning: Methods, Systems, Challenges, chapter 1, pages 3 – 38. Springer, 2019.  Available for free at http://automl.org/book.
  * Guerreiro et al. [2021] Andreia P Guerreiro, Carlos M Fonseca, and Luís Paquete.  The hypervolume indicator: Computational problems and algorithms.  ACM Computing Surveys (CSUR), 54(6):1–42, 2021.
  * Hernández-Lobato et al. [2016] D. Hernández-Lobato, J. Hernández-Lobato, A. Shah, and R. Adams.  Predictive entropy search for multi-objective bayesian optimization.  In Proc. of ICML’16, page 1492–1501, 2016.
  * Izquierdo et al. [2021] Sergio Izquierdo, Julia Guerrero-Viu, Sven Hauns, Guilherme Miotto, Simon Schrodi, André Biedenkapp, Thomas Elsken, Difan Deng, Marius Lindauer, and Frank Hutter.  Bag of baselines for multi-objective joint neural architecture search and hyperparameter optimization.  In 8th ICML Workshop on Automated Machine Learning (AutoML), 2021\.
  * Jamieson and Talwalkar [2016] K. Jamieson and A. Talwalkar.  Non-stochastic best arm identification and Hyperparameter Optimization.  In Proc. of AISTATS’16, 2016.
  * Jeong and Obayashi [2005] S. Jeong and S. Obayashi.  Efficient global optimization (ego) for multi-objective problem and data mining.  In Proc. of CEC’05, volume 3, pages 2138–2145. IEEE, 2005.
  * Karl et al. [2022] F. Karl, T. Pielok, J. Moosbauer, F. Pfisterer, S. Coors, M. Binder, L. Schneider, J. Thomas, J. Richter, M. Lang, G. Eduardo, B. Juergen, and B. Bischl.  Multi-objective hyperparameter optimization–an overview.  arXiv preprint arXiv:2206.07438, 2022.
  * Knowles et al. [2003] Joshua D Knowles, David W Corne, and Mark Fleischer.  Bounded archiving using the lebesgue measure.  In The 2003 Congress on Evolutionary Computation, 2003. CEC’03., volume 4, pages 2490–2497. IEEE, 2003.
  * Knowles [2005] Joshua Knowles.  A summary-attainment-surface plotting method for visualizing the performance of stochastic multiobjective optimizers.  In 5th International Conference on Intelligent Systems Design and Applications (ISDA 05), pages 552–557. IEEE, 2005.
  * Knowles [2006] J. D. Knowles.  ParEGO: a hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems.  IEEE Transactions on Evolutionary Computation, 10(1):50–66, 2006\.
  * Kohavi and others [1996] Ron Kohavi et al.  Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid.  In Kdd, volume 96, pages 202–207, 1996.
  * Li et al. [2017] L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar.  Hyperband: Bandit-based configuration evaluation for Hyperparameter Optimization.  In Proc. of ICLR’17, 2017.
  * Lindauer et al. [2022] Marius Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp, Difan Deng, Carolin Benjamins, Tim Ruhkopf, René Sass, and Frank Hutter.  Smac3: A versatile bayesian optimization package for hyperparameter optimization.  J. Mach. Learn. Res., 23:54–1, 2022.
  * Nilsback and Zisserman [2008] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008.
  * Panichella [2019] Annibale Panichella.  An adaptive evolutionary algorithm based on non-euclidean geometry for many-objective optimization.  In Proceedings of the Genetic and Evolutionary Computation Conference, pages 595–603, 2019.
  * Pfisterer et al. [2022] F. Pfisterer, L. Schneider, J. Moosbauer, M. Binder, and B. Bischl.  Yahpo gym-an efficient multi-objective multi-fidelity benchmark for hyperparameter optimization.  In First Conference on Automated Machine Learning (Main Track), 2022\.
  * Ponweiser et al. [2008] W. Ponweiser, T. Wagner, D. Biermann, and M. Vincze.  Multiobjective optimization on a limited budget of evaluations using model-assisted $\mathcal{S}$-Metric selection.  In Parallel Problem Solving from Nature - PPSN X, pages 784–794, 2008.
  * Salinas et al. [2021a] D. Salinas, V. Perrone, O. Cruchant, and C. Archambeau.  A multi-objective perspective on jointly tuning hardware and hyperparameters, 2021.
  * Salinas et al. [2021b] David Salinas, Valerio Perrone, Olivier Cruchant, and Cedric Archambeau.  A multi-objective perspective on jointly tuning hardware and hyperparameters.  arXiv preprint arXiv:2106.05680, 2021.
  * Schmucker et al. [2020] R. Schmucker, M. Donini, V. Perrone, and C. Archambeau.  Multi-objective multi-fidelity hyperparameter optimization with application to fairness.  In MetaLearn’20, 2020.
  * Schmucker et al. [2021a] R. Schmucker, M. Donini, M. Zafar, D. Salinas, and C. Archambeau.  Multi-objective asynchronous successive halving, 2021.
  * Schmucker et al. [2021b] Robin Schmucker, Michele Donini, Muhammad Bilal Zafar, David Salinas, and Cédric Archambeau.  Multi-objective asynchronous successive halving.  arXiv preprint arXiv:2106.12639, 2021.
  * Srinivas and Deb [1994] Nidamarthi Srinivas and Kalyanmoy Deb.  Muiltiobjective optimization using nondominated sorting in genetic algorithms.  Evolutionary computation, 2(3):221–248, 1994.
  * Storn and Price [1997] R. Storn and K. Price.  Differential evolution – a simple and efficient heuristic for global optimization over continuous spaces.  JGO, 11:341–359, 1997.
  * Xiao et al. [2017] Han Xiao, Kashif Rasul, and Roland Vollgraf.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.  arXiv preprint arXiv:1708.07747, 2017.
  * Ying et al. [2019] Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter.  Nas-bench-101: Towards reproducible neural architecture search.  In International Conference on Machine Learning, pages 7105–7114. PMLR, 2019.
  * Zela et al. [2020] Arber Zela, Julien Siems, and Frank Hutter.  Nas-bench-1shot1: Benchmarking and dissecting one-shot neural architecture search.  arXiv preprint arXiv:2001.10422, 2020.
  * Zela et al. [2022] A. Zela, J. Siems, L. Zimmer, J. Lukasik, M. Keuper, and F. Hutter.  Surrogate NAS benchmarks: Going beyond the limited search spaces of tabular NAS benchmarks.  In Proc. of ICLR’22, 2022.
  * Zhang and Li [2007a] Q. Zhang and H. Li.  Moea/d: A multiobjective evolutionary algorithm based on decomposition.  IEEE Transactions on evolutionary computation, 11(6):712–731, 2007\.
  * Zhang and Li [2007b] Qingfu Zhang and Hui Li.  Moea/d: A multiobjective evolutionary algorithm based on decomposition.  IEEE Transactions on evolutionary computation, 11(6):712–731, 2007\.
  * Zitzler and Thiele [1998] E. Zitzler and L. Thiele.  Multiobjective optimization using evolutionary algorithms — a comparative case study.  In Parallel Problem Solving from Nature, PPSN, pages 292–301, 1998\.


431–447, 2020.

[23]
T. Takacs and B. Jüttler, “${H^2}$ regularity properties of singular
parameterizations in isogeometric analysis,” Graphical Models,
vol. 74, no. 6, pp. 361–372, 2012.


# Predictive Analysis for Detection of Human Neck Postures using a robust
integration of kinetics and kinematics

Korupalli V Rajesh Kumar
Research Scholar
School of Electronics Engineering
Vellore Institute of Technology Chennai Campus
Chennai, India 600127
<EMAIL_ADDRESS>
Susan Elias
Associate Professor
School of Electronics Engineering
Vellore Institute of Technology Chennai Campus
Chennai, India 600127
<EMAIL_ADDRESS>

###### Abstract

Human neck postures and movements need to be monitored, measured, quantified
and analyzed, as a preventive measure in healthcare applications. Improper
neck postures are an increasing source of neck musculoskeletal disorders,
requiring therapy and rehabilitation. The motivation for the research
presented in this paper was the need to develop a notification mechanism for
improper neck usage. Kinematic data captured by sensors have limitations in
accurately classifying the neck postures. Hence, we propose an integrated use
of kinematic and kinetic data to efficiently classify neck postures. Using
machine learning algorithms we obtained 100% accuracy in the predictive
analysis of this data. The research analysis and discussions show that the
kinetic data of the Hyoid muscles can accurately detect the neck posture given
the corresponding kinematic data captured by the neck-band. The proposed
robust platform for the integration of kinematic and kinetic data has enabled
the design of a smart neck-band for the prevention of neck musculoskeletal
disorders.

_K_ eywords

Human Neck Postures, Inertial Measurement Unit, OpenSim, Kinetics, Kinematics
and Predictive Analysis.

##  1 Introduction and Related Work

Enabling technology to monitor, measure and manage human movements has been an
active area of research with a wide spectrum of applications ranging from
medical diagnostics, rehabilitation, sports, fitness, behaviour analysis and
gait based bio-metrics. The historic landscape of research publications in the
field, has presented the use of technology that are vision based (video
cameras), sensor based (Inertial Measurement Units IMUs), Infra-red and RADAR
based innovations to study human movement [1, 2, 3, 4]. Quantitative analysis
have traditionally been carried out using either kinetic or kinematic data for
various applications. Kinematic data is obtained using IMUs and has been used
in the study of Musculoskeletal Disorders (MSD) [5, 6] and in gait
recognition. On the other hand, kinetic data provides details of the force of
the component in motion and helps to analyse the activation of muscles
associated with the joint in motion. Kinetic data are computed from the
signals obtained from Kinesiological Electro Myography (KEMG) device and these
are quantitatively analysed for understanding muscle force and fatigue [7, 8,
9, 10, 11]. The motivation for the research presented in this paper, was to
identify a robust methodology to integrate the kinetic and kinematic features
for predictive analysis of human postures and movements. IMU-based technology
is the most popular for kinematic data capture as the IMU is a versatile
Micro-Electro-Mechanical System (MEMs) based device [12]. IMU data have been
integrated earlier with several proprietary and open source based simulation
platforms for analysis and visualization of movement data [13, 14, 15, 16, 17,
18, 19]. Here the integration of IMU with OpenSim is presented and it is
currently a focus area in top research laboratories as well. OpenSim is an
free open source simulation and modelling tool developed at Stanford
University (https://opensim.stanford.edu/). The in-built features, modeling
capabilities and the contributions by the OpenSim community makes it a
scalable and reliable tool for analysing human movement. Besides presenting
the step-wise procedure to integrate IMU data with OpenSim, this paper
presents a novel methodology of combining kinetic and kinematic data for
generating insightful analysis of human movements. OpenSim provides details of
muscle activation and supports joint modelling of kinetic and kinematic
parameters. In the following subsections the detailed methodology is
presented. The results and discussions that follow will highlight the
significance and applications of the proposed robust integration platform for
movement analysis.

##  2 Motivation and Proposed work

The goal of the proposed research is to present a methodology to measure and
accurately identify the postures of the human neck, for prevention and
rehabilitation of musculoskeletal disorders of the cervical region. Improper
neck posture due to sedentary lifestyle is a major cause of Cervical spine
dysfunction in all age groups ranging from children to the elderly. The neck
region can also be affected due to improper neck postures that are inherent in
certain kinds of occupations. Other lifestyle related activities, including
sleeping in the sitting position during travel can trigger musculoskeletal
problems around the neck. The motivation for the research was to propose a
novel methodology to prevent disorders of the neck through timely detection
and notifications. A sensor based smart neck-band that can precisely detect
neck postures was designed to monitor improper use of the neck region and to
generate alert messages as a preventive measure [20]. This neck-band can also
be used to take measurements of the range of motion of the neck regions during
therapy and rehabilitation [21]. Research works related to the use of sensors
to track movements by obtaining kinematic data have been presented extensively
in literature under the field called Actigraphy. These approaches have also
been widely used for tracking movement through commercially available wrist
worn fitness monitors. But in order to identify the postures of the neck,
there are several challenges and limitation in using only kinematic data
obtained using sensors. Hence we explored the possibility of integrating
kinetic and kinematic data for better accuracy in detection of neck postures.
In this paper we present a robust integrated platform for predictive analysis
of human neck postures using kinetic and kinematic data.

##  3 Materials and Methods

In this research, an IMU based device is used for acquiring the kinematic data
of the neck, and the OpenSim simulation modeling tool is used to generate the
kinetic data of the corresponding neck movements. Predictive Analysis to
detect neck postures from the kinetic and kinematic data, is done using
Machine Learning methods. In the following sections, the methods and materials
used in this research are presented.

###  3.1 Kinematic data acquisition using Smart Neck-Band

####  3.1.1 IMU Based Neck-Band:

Inertial Measurement Unit (IMU) embedded in an elastic neck-band captures the
kinematic data required for the analysis. IMU devices are available in
miniature-sizes, and can be used to design wearable products. For this
research, Metawear CPRO, IMU device developed at MBIENTLAB has been used. It
has on-chip memory, processing unit, accelerometer, gyroscope sensor,
magnetometer sensor, pressure sensor and temperature sensor. In addition to
these sensors, this device has inbuilt Bluetooth support, to establish
communication with the associated mobile application or any Bluetooth device
to transmit the device data. This IMU device can stream data for 6 hours
continuously using a 3.3 V coin sized battery and is attached to a wearable
band with an adjustable strap to fit it firmly around the neck. This neck-band
when integrated with the proposed predictive analysis can be referred to as a
smart neck-band, for its context-aware functionalities.

Figure 1: Kinematic Data Acquisition Process

####  3.1.2 Mobile Application- MetaBase:

The commercially available IMU device used in this research comes with an
associated mobile application called MetaBase, developed by MBIENTLAB. It is
used to retrieve the recorded sensor data from the IMU device. MetaBase
establishes a Bluetooth communication channel with the IMU hardware and
receives the sensor data from the device. Once data transmission from the
sensor to the mobile application is completed, the data can be shared to a
cloud storage or to any local machine. Figure 1 summarises the data
acquisition process of the neck kinematics.

####  3.1.3 Sensor Data Format:

The IMU device, has in-built Accelerometer, Gyroscope and Magnetometer sensors
to record various kinematic aspects of the movements.The format of the data
captured by these sensors are given below:

  * •

Accelerometer sensor : epoch(ms), time, elapsed(s), x(g), y(g), z(g)

  * •

Gyroscope sensor : epoch(ms), time, elapsed(s), x(deg/s), y(deg/s), z(deg/s)

  * •

Magnetometer : epoch(ms), time, elapsed(s), x(T), y(T), z(T) etc.

For the kinematic analysis of neck postures, dealt with in this research, the
accelerometer sensor data was sufficient. The accelerometer sensor was
operated at 100Hz frequency with 8+/-g.

###  3.2 Kinetic data generation using Neck Musculoskeletal Model

Electro Myography (EMG) and Surface EMG (SEMG) are the standard methods for
muscle related data acquisition for movement analysis. The kinetic data
obtained from EMG based methods are accurate, but are limited to laboratory
based study. In order to carry out movement analysis using both kinematic and
kinetic data, a novel methodology has been proposed in this paper. Here we
focus on the detection of postures of the neck using an innovative approach.
Instead of capturing kinetic data related to muscle activation using any of
the EMG based methods, in the proposed approach kinetic data of the
corresponding neck postures is generated using a Neck Musculoskeletal
Simulation Model. The smart neck-band captures the real-world kinematic data
of the neck postures and this data is used to generate the corresponding
kinetic data relating to the muscles around the neck region using the OpenSim
simulation tool.

####  3.2.1 OpenSim - Simulation Modeling Tool:

OpenSim is the most popular open-source tool, used to create and study human
musculoskeletal models and provides extensive data on kinematics and kinetics
of human movement. In-built functionalities such as Scale, Inverse
Kinematics(IK), Inverse Dynamics (ID), Residual Reduction (RR), Static
Optimization (STO), Computed Muscle Control (CMC) and Analyze, provide support
to extract all information related to the muscles, tendons, joints kinetics
and kinematics. An overview of the functionalities of OpenSim and the inbuilt
tools is shown in Figure 2 and detailed information is made available by the
OpenSim contributors [22, 23, 24, 25].

Figure 2: Overview of OpenSim Functionalities

####  3.2.2 Neck Musculoskeletal Model:

In this research, the neck musculoskeletal model presented in [26] (Figure 3)
was used. This is a fully flexible model for head and neck movements, and
versatile compared to other models [27]. The neck region consists of Cervical
joints (C1-C7), sixty four muscles and various associated tendons and
ligaments. The Hyoid muscles plays a vital role in supporting the neck
movements and hence has a key role in the proposed predictive analysis. The
Neck Musculoskeletal Model used in this research has integrated the Hyoid
muscles and this was a big advantage for our experimental analysis. The
kinetic data provided by OpenSim includes forces, length, power and activation
levels of joints, muscles and tendons. With the neck musculoskeletal model and
research insights provided in [26, 28] used in simulation modeling. We used
the kinetic data of the Hyoid muscles, extracted from the CMC tool during
simulation for our research analysis. The Hyoid muscles are highlighted in
green colour and shown in Figure 3.

Figure 3: Hyoid Muscles (in green colour)

##  4 Experiments and Research Analysis

In general the human neck has three-degrees of freedom, horizontal plane,
vertical plane, and rolling of the head. All other asymmetric movements of the
neck are variations and combinations of these three fundamental movements. At
any point of time the neck will be in one of the following nine static
positions called neck postures or it can move in any random order between
these nine postures. The nine static positions or neck postures are:

  * •

Neck at Extreme Up (NU)

  * •

Neck at Extreme Down (ND)

  * •

Neck at Extreme Right (NR)

  * •

Neck at Extreme Left (NL)

  * •

Neck at Right Up (NRU)

  * •

Neck at Right Down (NRD)

  * •

Neck at Left Up (NLU)

  * •

Neck at Left Down( NLD)

  * •

Neck at Middle (NM).

The goal of the research presented in this paper is to design a methodology to
detect neck postures by training the Machine Learning algorithms using the
kinematic and kinetic data. In this research we used Random Forest, an
ensemble learning method for the prediction and classification. The smart
neck-band captures the real world data and integrates it with the OpenSim
simulation platform. In order to effectively capture the neck kinematics, an
experimental study was first carried out to determine the ideal location for
the IMU device. The IMU device was fixed onto the elastic band, and the neck-
band could be worn in a manner that located the IMU device either in Front or
at the Back of the neck region.

###  4.1 Experimental study for Location of IMU

In this research, eight volunteers participated and details of their physical
attributes are given in Table 1.

Gender | No.of Persons | Height | Weight
---|---|---|---
Male | 5 |  $\sim$5.3 cms | +/- 75.6 kgs
Female | 3 |  $\sim$5.1 cms | +/- 69.3 kgs
Table 1: Participants - Physical Attributes
Figure 4: IMU located FRONT & BACK and corresponding Musculoskeletal Model

The subjects wore the neck-band and participated in the research study. The
participants were asked to keep their neck in the nine static positions for a
duration ranging from 1 minute to 2 minutes based on their comfort level. IMU
data was recorded for all participants for each of the nine neck postures with
the sensor located at FRONT side of the neck region and similarly for BACK
side of neck. Figure 4, shows the kinematic and kinetic data extraction
methods using IMU device and OpenSim simulation model for both Front and Back
neck locations. Details of the dataset are presented in Table 2.

IMU - Device Placement | Total Time Duration |  | Dataset Size
---
(After Pre-processing)
Sensor Placed at Front Side |  | 1080 sec
---
(Nine Static positions)
| 1080 * 6
---
Rows * Columns
Sensor Placed at Back Side |  | 1080 sec
---
(Nine Static positions)
| 1080 * 6
---
Rows * Columns
Table 2: Dataset Quantitative Parameters

Datasets were pre-processed and modularised based on the time stamp provided
by the sensor and labelled manually. The observations in the dataset were also
validated using the video footage of the corresponding experimental study. As
part of the pro-processing task, NaN’s(Not a Number) and NA’s (Not Available)
were interpreted, outliers were removed and data was normalised.

###  4.2 Predictive Analysis using Random Forest Algorithm

The data collected by keeping IMU device at the FRONT side of the neck was
divided into training and testing data in the ratio of 75 : 25, and given as
input to the Random Forest algorithm, to classify the nine static neck
positions and to find the accuracy of the classification. Similarly the data
collected by keeping the IMU device at the BACK side of the neck was similarly
processed.

Figure 5: Confusion Matrix - Classification of Neck Positions

The confusion matrix and performance metrics for the Front and Back location
and are shown in Figure 5 and 6. 100% accuracy was achieved for Front location
based classification and 99% for the Back location based classification. From
this research study we can infer that the ideal location for the IMU device
during data capture is the Front side of the neck. This inference correlates
with our decision to use the Hyoid muscles located in Front of the neck for
accurate classification of neck postures. In this research paper, we proposed
the idea of generating kinetic data related to the Hyoid muscles and to use
this data along with the associated kinematic data to detect neck posture
accurately using classification techniques. This is presented in the following
section.

Figure 6: Performance Metrics

##  5 Robust Integration of Kinematic and Kinetic Data

In this section we first present the procedure to integrate kinematic and
kinetic data for the experiment research analysis. Integrating the IMU based
kinematic data to the OpenSim simulation modeling platform is a challenging
and interesting task.

###  5.1 IMU data integration with OpenSim

  * •

The IMU based accelerometer sensor data format provides three dimensional
kinematic data (X,Y,Z).

  * •

To export IMU kinematic data into OpenSim simulation tool, mathematical and
functional analysis is required. In OpenSim, a file with the extension .trc
(Track Row Column) is used as an input file for the Inverse Kinematics (IK)
tool, and using this information this tool provides joint movement data as a
motion file with the extension .mot.

  * •

The neck skeletal model has seven sets of markers around the skull and
cervical region (four on the skull, one at Sternum Jugular Notch and two at
acromioclavicular right and left joints ). In the front side of the neck,
there is a marker called Sternum Jugular Notch (SJN). The IMU based kinematics
data is mapped onto the data of SJN - X,Y,Z coordinates. The other markers are
calibrated according to the functional movements. The .trc file contains the
details of these markers and it is the input file for the Inverse Kinematics
(IK) and the motion file (.mot) is obtained as the output.

  * •

The information in this .mot file is fed as input to Computed Muscle Control
(CMC) tool, which produces the data related to Neck kinematics, kinetics,
joints, muscles, forces etc.

  * •

The functional integration of IMU data and OpenSim is shown in Figure 7.

Figure 7: IMU to OpenSim - Data Flow

###  5.2 Simulation using Neck musculoskeletal model

The Spine and Cervical joints C1, C2, C3, C4, C5, C6 and C7 play a pivotal
role for any kind of neck movements. The Spine and Cervical variations in the
OpenSim for the various neck postures presented in Figure 8.

Figure 8: Head and Neck Musculoskeletal Model

In this research we recorded and analyzed kinematic data for all the subjects.
In this section we present the simulation results and analysis for Subject ID:
04. The data collected from all the other subjects were used in training and
testing of the classification model. Figure 9, shows the neck postures for
research Subject ID: 4 and the corresponding musculoskeletal model. The images
were captured during the simulation of the Musculoskeletal neck model and this
helped to correlate the dataset with the image of the neck posture.

Figure 9:  Subject Specific Neck Postures with OpenSim Musculoskeletal Model

###  5.3 Neck Musculoskeletal Model based Kinematic and Kinetic Data Analysis

OpenSim generates the Neck kinematics information based on the input data:
Acceleration, Position and Velocity. We used the Acceleration and Position
data to classify and predict the human neck posture. From the Figures 10 and
11, we can observe the response of the cervical joints and other associated
joints during the experimentation task. The subjects changed the neck posture
from one position to another after a time gap of about 120 sec, and with a
total of nine positions in the study, 1080 secs of data was captured. Figure
10 shows the variations in the Neck Acceleration with respective to joint
kinematics and similarly Figure 11 shows the variations in the Position of the
neck.

Figure 10: Kinematics-Neck Acceleration  Figure 11: Kinematics-Neck Position

There are eight important sets of Hyoid muscles (shown in Figure 3) and many
other associated hyoid muscles are attached with hyoid bone in neck region.
These muscles help in providing free movement generation and flexibility to
the neck [26]. The OpenSim based CMC tool provides kinetic information like
forces, activation, lengths etc. Using the CMC tool, kinetic data is extracted
for the corresponding kinematic data that was captured and integrated in
OpenSim. Here, the response of the tendon forces of the Neck Hyoid muscles
were analysed as shown in Figure 12.

Figure 12: Kinetics - Tendon Force

###  5.4 Predictive Analysis - Kinematics and Kinetics

Figure 13: Kinematics-Neck Acceleration

The Random Forest Algorithm was used to classify and to predict the neck
posture based on the kinematics and kinetics data generated by OpenSim. The
Random Forest method achieved 100% accuracy in the classification and
prediction of neck postures using Neck acceleration and position data. These
results are presented in Figures 13 and 14.

Figure 14: Kinematics-Neck Position

Similarly, Random Forest classifier predicted nine neck postures using the
response of the tendon force data of Hyoid muscles and achieved 100% accuracy
in the prediction. The results of the kinetic tendon force classification is
shown in Figure 15.

Figure 15: Kinetics-Tendon Force

##  6 Conclusion

This paper presents a novel methodology to identify neck postures using
kinetic and kinematic data. Improper neck postures can lead to neck cervical
pain and musculoskeletal disorders. This research includes the design of a
smart neck-band that captures kinematic data of the neck postures and
movements. The OpenSim simulation tool and a neck musculoskeletal model, were
used to simulate the related kinetic data for classification of the neck
postures. The Machine Learning algorithms achieved 100% accuracy in prediction
of the neck postures and this innovative methodology can support notification
based applications for prevention of neck musculoskeletal disorders.

## References

  * [1] P. C. Ribeiro, J. Santos-Victor, and P. Lisboa, “Human activity recognition from video: modeling, feature selection and classification architecture,” in The Proceedings of International Workshop on Human Activity Recognition and Modelling,(HAREM) Oxford, United Kingdom, 2005.
  * [2] V. Kellokumpu, G. Zhao, and M. Pietikäinen, “Human activity recognition using a dynamic texture based method.,” in The Proceedings of British Machine Vision Conference 2008, Leeds, United Kingdom, 2008.
  * [3] K. K. Biswas and S. K. Basu, “Gesture recognition using microsoft kinect.,” in The Proceedings of The 5th IEEE international conference on automation, robotics and applications, Wellington, New Zealand, 2011.
  * [4] A. Nandy, R. Chakraborty, and P. Chakraborty, “Cloth invariant gait recognition using pooled segmented statistical features,” Neurocomputing, vol. 191, pp. 117–140, 2016.
  * [5] R. Balakrishnan, E. Chinnavan, and T. Feii, “An extensive usage of hand held devices will lead to musculoskeletal disorder of upper extremity among student in amu: a survey method,” International Journal of Physical Education, Sports and Health, vol. 3, no. 2, pp. 368–372, 2016.
  * [6] B. Thanathornwong, S. Suebnukarn, Y. Songpaisan, and K. Ouivirach, “A system for predicting and preventing work-related musculoskeletal disorders among dentists,” Computer methods in biomechanics and biomedical engineering, vol. 17, no. 2, pp. 177–185, 2014.
  * [7] M. Kadaba, H. Ramakrishnan, M. Wootten, J. Gainey, G. Gorton, and G. Cochran, “Repeatability of kinematic, kinetic, and electromyographic data in normal adult gait,” Journal of Orthopaedic Research, vol. 7, no. 6, pp. 849–860, 1989.
  * [8] S. Pizzolato, L. Tagliapietra, M. Cognolato, M. Reggiani, H. Müller, and M. Atzori, “Comparison of six electromyography acquisition setups on hand movement classification tasks,” PloS one, vol. 12, no. 10, 2017.
  * [9] N. U. Ahamed, Z. Taha, M. Alqahtani, O. Altwijri, M. Rahman, and A. Deboucha, “Age related differences in the surface emg signals on adolescent’s muscle during contraction,” in The Proceedings of IOP Conference Series: Materials Science and Engineering. Malaysia & Indonesia, 2016.
  * [10] S. Deslandes, J. Mariot, B. Beaune, S. Durand, and S. Serveto, “Isokinetics for neck muscular fatigue assessment,” Computer Methods in Biomechanics and Biomedical Engineering, vol. 11, no. S1, pp. 87–88, 2008.
  * [11] H. Saraç, S. Laporte, and B. Sandoz, “Evaluation of a passive model to mimic dynamic head/neck movements,” Computer Methods in Biomechanics and Biomedical Engineering, vol. 20, no. sup1, pp. S191–S192, 2017.
  * [12] T. Seel, J. Raisch, and T. Schauer, “Imu-based joint angle measurement for gait analysis,” Sensors, vol. 14, no. 4, pp. 6891–6909, 2014.
  * [13] E. J. Shiroma, M. Schepps, J. Harezlak, K. Chen, C. Matthews, A. Koster, P. Caserotti, N. Glynn, and T. Harris, “Daily physical activity patterns from hip-and wrist-worn accelerometers,” Physiological measurement, vol. 37, no. 10, pp. 1852–1861, 2016.
  * [14] A. Moncada-Torres, K. Leuenberger, R. Gonzenbach, A. Luft, and R. Gassert, “Activity classification based on inertial and barometric pressure sensors at different anatomical locations,” Physiological measurement, vol. 35, no. 7, pp. 1245–1263, 2014.
  * [15] N. D. Nguyen, P. H. Truong, and G.-M. Jeong, “Daily wrist activity classification using a smart band,” Physiological measurement, vol. 38, no. 9, pp. L10–L16, 2017.
  * [16] N. L. Mifsud, N. H. Kristensen, M. Villumsen, J. Hansen, and U. G. Kersting, “Portable inertial motion unit for continuous assessment of in-shoe foot movement,” Procedia Engineering, vol. 72, pp. 208–213, 2014.
  * [17] C. Crema, A. Depari, A. Flammini, E. Sisinni, T. Haslwanter, and S. Salzmann, “Imu-based solution for automatic detection and classification of exercises in the fitness scenario,” in The Proceedings of IEEE Sensors Applications Symposium (SAS), Glassboro, New Jersey, 2017.
  * [18] M. Georgi, C. Amma, and T. Schultz, “Recognizing hand and finger gestures with imu based motion and emg based muscle activity sensing.,” in The Proceedings of the International Conference on Bio-inspired Systems and Signal Processing (BIOSIGNALS), Lisbon, Portugal, 2015.
  * [19] R. Chavarriaga, H. Sagha, A. Calatroni, S. T. Digumarti, G. Tröster, J. d. R. Millán, and D. Roggen, “The opportunity challenge: A benchmark database for on-body sensor-based activity recognition,” Pattern Recognition Letters, vol. 34, no. 15, pp. 2033–2042, 2013.
  * [20] K. V. RajeshKumar and E. Susan, “Smart neckband for rehabilitation of musculoskeletal disorders,” 2019.  INDIAN PATENTS Reference: 201941042978.
  * [21] K. V. R. Kumar and S. Elias, “Smart neck-band for rehabilitation of musculoskeletal disorders,” in The proceedings of IEEE 2020 International Conference on COMmunication Systems & NETworkS (COMSNETS), Bengaluru, India, 2020.
  * [22] S. L. Delp, F. C. Anderson, A. S. Arnold, P. Loan, A. Habib, C. T. John, E. Guendelman, and D. G. Thelen, “Opensim: open-source software to create and analyze dynamic simulations of movement,” IEEE transactions on biomedical engineering, vol. 54, no. 11, pp. 1940–1950, 2007.
  * [23] A. Seth, M. Sherman, J. A. Reinbolt, and S. L. Delp, “Opensim: a musculoskeletal modeling and simulation framework for in silico investigations and exchange,” Procedia Iutam, vol. 2, pp. 212–232, 2011\.
  * [24] D. G. Thelen and F. C. Anderson, “Using computed muscle control to generate forward dynamic simulations of human walking from experimental data,” Journal of biomechanics, vol. 39, no. 6, pp. 1107–1115, 2006.
  * [25] A. Seth, J. L. Hicks, T. K. Uchida, A. Habib, C. L. Dembia, J. J. Dunne, C. F. Ong, M. S. DeMers, A. Rajagopal, M. Millard, et al., “Opensim: Simulating musculoskeletal dynamics and neuromuscular control to study human and animal movement,” PLoS computational biology, vol. 14, no. 7, 2018\.
  * [26] J. D. Mortensen, A. N. Vasavada, and A. S. Merryweather, “The inclusion of hyoid muscles improve moment generating capacity and dynamic simulations in musculoskeletal models of the head and neck,” PloS one, vol. 13, no. 6, 2018.
  * [27] D. Cazzola, T. P. Holsgrove, E. Preatoni, H. S. Gill, and G. Trewartha, “Cervical spine injuries: A whole-body musculoskeletal model for the analysis of spinal loading,” PloS one, vol. 12, no. 1, 2017.
  * [28] J. Mortensen, M. Trkov, and A. Merryweather, “Improved ergonomic risk factor assessment using opensim and inertial measurement units,” in The Proceedings of IEEE/ACM International Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE) Washington, D.C, 2018\.