# Keywords for Bias

Abdurrezak Efe  Huawei Turkey R&D CenterIstanbulTurkey<EMAIL_ADDRESS>,
Gizem Gezici  Huawei Turkey R&D CenterIstanbulTurkey<EMAIL_ADDRESS>,
Aysenur Uzun  Huawei Turkey R&D CenterIstanbulTurkey<EMAIL_ADDRESS>and  Uygar Kurt  Huawei Turkey R&D CenterIstanbulTurkey<EMAIL_ADDRESS>
###### Abstract.

This work proposes to analyse some keywords for bias analysis. For this, we
are using several NLP approaches and compare them based on their capability of
analysing keywords to analyse bias. The overall findings show that our
proposed approach gives comparable results with state-of-the-art.

††copyright: rightsretained

##  1\. Introduction

Key-phrase extraction aims to choose a set of important words or phrases
automatically that summarize the main points of a given document. Key-phrases
can help readers to search information quickly and accurately, especially in
large text collections. Researchers have already proposed various approaches
to extract key-phrases. For an in-depth study, please refer to (Hasan and Ng,
2014).

The problem of extracting key-phrases can be mainly solved using two different
kinds of approaches: supervised and unsupervised. Although supervised
approaches generally perform better on domain-specific tasks, data annotation
is a costly process, and the trained models might not be leveraged for other
datasets. Unsupervised models can easily be applied to distinct datasets with
no customization or little modification needed, i.e., one can only use the
suitable models for different languages, yet they can provide comparable
results.

Successful unsupervised key-phrase extraction methods are composed of graph-
based and embedding-based approaches (Bennani-Smires et al., 2018; Boudin,
2018; Bougouin et al., 2013; Sun et al., 2020; Wan and Xiao, 2008). SingleRank
(Wan and Xiao, 2008) initially constructs a set of neighbour documents for
each document $d$. Then, this set is used to generate candidate key-phrases
with a graph ranking algorithm that has both local (of $d$ itself) and global
(of the whole neighbor document set) components. Unlike SingleRank which also
uses corpus information that a given document might be linked to, sBERT-Rank
only requires the given document at a time. From the given document, sBERT-
Rank first detects the noun-phrases with the Stanza dependency parser Compared
to SingleRank (Wan and Xiao, 2008) this is a more efficient and realistic
approach. In this sense, sBERT-Rank is a corpus-independent approach that
leverages the language information embedded in the pre-trained model of BERT
(Devlin et al., 2018). On the other hand, TopicRank (Bougouin et al., 2013)
constructs a graph of clusters of candidate phrases for various topics
regarding the given document. Similar to SingleRank (Wan and Xiao, 2008), a
graph ranking algorithm is utilized to choose the top clusters (nodes).
Finally, the key-phrases are extracted from these top topics. In a follow-up
work, namely MultipartiteRank (Boudin, 2018), the authors additionally propose
to add candidate phrases as nodes to the graph in a multipartite graph
structure. Nonetheless, the major difference from TopicRank (Bougouin et al.,
2013) is that MultipartiteRank (Boudin, 2018) adds position information to the
edge weights to capture the semantics of longer documents.

In addition to those graph-based methods, embedding-based approaches have also
been used for extracting key-phrases, since external information from pre-
trained language models can help to improve the model capability (Bennani-
Smires et al., 2018; Sun et al., 2020). Our proposed pipeline is similar to
the embedding-based ranking methods of EmbedRank (Bennani-Smires et al., 2018)
and SIFRank (Sun et al., 2020). EmbedRank generates candidate phrases by
initially extracting all possible phrases between the lengths of 2 to 8 tokens
from a given document by only using POSTag information Then, these phrases,
along with the original document, are embedded into a vector space using the
embedding methods of Sent2Vec (Pagliardini et al., 2017) and Doc2Vec (Lau and
Baldwin, 2016). Further, these phrases are ranked with respect to their cosine
similarities with the corresponding document from which they were extracted.
sBERT-Rank is mainly different from EmbedRank in two ways. sBERT-Rank uses a
dependency tree instead of only POSTag information to generate candidate
phrases (noun-phrases) in a more efficient and effective manner. In this way,
our approach firstly selects suitable key-phrases using word relations rather
than applying simple word permutations, which brings extra computation for
less informative phrases. The second difference is that our key-phrase
extraction pipeline uses sBERT (Reimers and Gurevych, 2019) which also
leverages the hidden positional embedding mechanism of BERT instead of the
embedding methods of Sent2Vec and Doc2Vec.

In addition, in SIFRank (Sun et al., 2020), the authors suggest to firstly
extract the topic of a given document by using sentence embeddings. Then, the
candidate phrases are compared with the topic (not the sentences or the
document itself) for ranking. Subsequently, a position-biased weight is
assigned to each candidate phrase to improve the model’s performance on long
documents. Unlike the SIFRank, the performance of the sBERT-Rank is not
affected by the document length at all since the ranking score is computed in
two steps. These subsequent steps help us to select the candidate phrases that
are informative of the corresponding sentence from which the candidates were
extracted, and the sentence is informative of the given document. Apart from
this, SIFRank uses (Arora et al., 2017), while we use sBERT for sentence
embeddings.

In this work, we introduce a simple, corpus-independent, i.e., only requires
the current document itself, and an easily-extendible to other languages,
unsupervised embedding-based pipeline to extract key-phrases from a given
document. To the best of our knowledge, this is the only work that uses all
four diverse benchmark datasets in the literature to properly evaluate the
capability of our proposed pipeline of sBERT-Rank as well as provide a
comprehensive evaluation of other state-of-the-art approaches. This is
important in the sense that unsupervised approaches are generally expected to
give comparable and consistent results irrespective of the dataset properties,
i.e. longer or shorter documents, news articles of scientific papers etc.,
thus we believe that it is also necessary to assess this property of our
pipeline as well as the-state-of-the-art approaches.

The remainder of the paper is structured as follows. In Section 2 we present
the details of our pipeline, sBERT-Rank. In Section 3 we provide the
experimental results and then discuss them. About the experimental setup and
then display the results. Lastly, we conclude the paper in Section 4.

##  2\. sBERT-Rank

In this section, we present the core components of our unsupervised key-phrase
extraction pipeline. Our pipeline is composed of three main parts. First, we
extract informative noun-phrases as candidates from a given document. Then,
these candidates and the current document are represented in a high
dimensional vector space using sBERT. Lastly, we rank the candidate phrases by
using the cosine similarity scores. A simple flowchart explaining this process
can be seen in 1.

sentNoun-Phrase Chunkerlist of ${NP}$sBERTlist of candidate embeddingsRanking
Candidatesinformativeness of candidates Figure 1. sBert-Rank Pipeline

###  2.1. Noun-Phrase Chunker

Key-phrases are the most informative keywords or multiple words for the
document at hand. They are mostly nouns as well as noun or adjective phrases.
Hence, in this work, we aim to catch noun-phrases including adjective phrases
within a given document.

In order to detect noun-phrases, a widely-used stable dependency parser of
Stanza (Qi et al., 2020) proposed by the Stanford NLP Group is used. The
current document is first split into , and these sentences are given to the
Stanza as inputs to generate its dependency tree for the sentences in the
document. This allows us to generate dependency trees both for long and short
documents. Since dependency trees contain valuable word- information, after
creating the dependency tree for a given document, its noun-phrases are
extracted with a rule-based approach which leverages the dependency tree
information.

To exploit dependency tree information, we propose a one general rule where we
consider a noun-phrase (_NP_) as a recurrence relation. The general rule is
defined as follows:

(1) $NP=(NOUN,ADJ)(NP)$

In the given rule, $NP$ initially starts with a noun or adjective, then it is
followed by a noun-phrase itself. Furthermore, if we consider a given
${NP}_{i}$, which is comprised of $n$ tokens in total we define each of these
tokens as $t_{j}$, where $j=1,2,3,...n$. Note that ${NP}_{i}$ is defined as a
list of $n$ tokens with the order of ${t_{1},t_{2},t_{3},...,t_{n}}$. Then,
our rule requires that the head of each $t_{i}$, $1\leq i\leq n$, extracted
from the dependency tree, which is denoted as $h(t_{i})$, should be $t_{n}$.
That is, all of the tokens within the noun-phrase should be modified by the
last token. This condition helps us to filter the noun-phrases in such a way
that each candidate is a complete phrase by itself. Thus, as mentioned in
Section 1 differently from (Bennani-Smires et al., 2018) which uses only
POSTag information in the candidate generation step, we obtain a more
distilled set of candidate phrases using dependency tree information. In this
way, the distilled set generally provides more informative candidates which
are smaller in size than the ones generated by (Bennani-Smires et al., 2018)
which could provide a more effective as well an efficient processing of the
documents in a given dataset. Note that for each document, only distinct noun-
phrases are taken. Apart from these, since stanza is available for more than
70 languages 111https://stanfordnlp.github.io/stanza/, the first step of our
pipeline can easily be extended to extract key-phrases in other languages.

Table 1. Performance of sBERT-Rank on INSPEC, DUC2001, NUS and SemEval2017 Datasets |  |  | INSPEC |  |  | DUC2001 |  |  | NUS |  |  | SemEval2017 |
---|---|---|---|---|---|---|---|---|---|---|---|---|---
N | Model | R | P | F1 | R | P | F1 | R | P | F1 | R | P | F1
5 | MultipartiteRank | 0.21 | 0.28 | 0.24 | 0.18 | 0.15 | 0.16 | 0.10 | 0.19 | 0.13 | 0.11 | 0.34 | 0.16
SingleRank | 0.2 | 0.26 | 0.23 | 0.11 | 0.10 | 0.10 | 0.02 | 0.04 | 0.03 | 0.11 | 0.34 | 0.17
TopicRank | 0.21 | 0.28 | 0.24 | 0.16 | 0.14 | 0.15 | 0.09 | 0.17 | 0.12 | 0.10 | 0.34 | 0.16
EmbedRank | 0.25 | 0.41 | 0.31 | 0.22 | 0.34 | 0.27 | 0.02 | 0.05 | 0.03 | 0.13 | 0.45 | 0.20
SIFRank | 0.21 | 0.43 | 0.29 | 0.25 | 0.40 | 0.31 | 0.03 | 0.04 | 0.03 | 0.15 | 0.50 | 0.23
sBERT-Rank | 0.28 | 0.29 | 0.29 | 0.24 | 0.22 | 0.23 | 0.14 | 0.18 | 0.15 | 0.13 | 0.32 | 0.19
10 | MultipartiteRank | 0.33 | 0.24 | 0.28 | 0.26 | 0.12 | 0.17 | 0.17 | 0.16 | 0.17 | 0.18 | 0.28 | 0.22
SingleRank | 0.37 | 0.26 | 0.30 | 0.22 | 0.11 | 0.14 | 0.05 | 0.05 | 0.05 | 0.21 | 0.33 | 0.26
TopicRank | 0.31 | 0.23 | 0.27 | 0.25 | 0.11 | 0.15 | 0.14 | 0.14 | 0.14 | 0.17 | 0.27 | 0.21
EmbedRank | 0.40 | 0.35 | 0.37 | 0.35 | 0.28 | 0.28 | 0.05 | 0.06 | 0.05 | 0.23 | 0.40 | 0.30
SIFRank | 0.39 | 0.39 | 0.39 | 0.37 | 0.30 | 0.33 | 0.04 | 0.04 | 0.04 | 0.26 | 0.45 | 0.33
sBERT-Rank | 0.43 | 0.24 | 0.31 | 0.39 | 0.19 | 0.25 | 0.22 | 0.14 | 0.17 | 0.24 | 0.29 | 0.26
15 | MultipartiteRank | 0.43 | 0.22 | 0.29 | 0.33 | 0.11 | 0.16 | 0.22 | 0.14 | 0.17 | 0.24 | 0.26 | 0.25
SingleRank | 0.47 | 0.24 | 0.32 | 0.29 | 0.09 | 0.15 | 0.08 | 0.05 | 0.06 | 0.29 | 0.30 | 0.29
TopicRank | 0.38 | 0.21 | 0.27 | 0.31 | 0.09 | 0.14 | 0.17 | 0.11 | 0.13 | 0.22 | 0.23 | 0.22
EmbedRank | 0.49 | 0.31 | 0.38 | 0.44 | 0.24 | 0.31 | 0.07 | 0.05 | 0.06 | 0.32 | 0.37 | 0.34
SIFRank | 0.48 | 0.33 | 0.39 | 0.45 | 0.25 | 0.32 | 0.06 | 0.04 | 0.05 | 0.35 | 0.41 | 0.38
sBERT-Rank | 0.52 | 0.21 | 0.30 | 0.50 | 0.16 | 0.24 | 0.28 | 0.12 | 0.17 | 0.32 | 0.26 | 0.29

###  2.2. sBERT

After obtaining the unique noun-phrases which are the candidates for each
document, it is necessary to assess the informativeness level of each unique
noun-phrase for the document to which it belongs. For this, candidate phrases
and the document itself are represented in a high-dimensional vector space by
using sBERT embeddings. sBERT modifies the pre-trained BERT network by using
siamese networks to derive semantically meaningful sentences. Yet, in the
scope of this work, we use sBERT not only for sentences, but for computing
semantic textual similarity 222https://www.sbert.net/. sBERT specifically
provides a pre-trained model for English as well as a multilingual model for
other languages. Thus, in addition to the first step of sBERT-Rank, the second
step can easily be extended to other languages as well by using the
multilingual model of sBERT.

###  2.3. Ranking Candidates

In the last step of our pipeline, for ranking the candidate phrases that are
represented in a high-dimensional space, two cosine similarity scores are
computed. First, the cosine similarity scores between each sentence in the
current document and the document itself are computed. Then, the cosine
similarity scores between each candidate phrase and the sentence from which
the phrase is extracted are also computed. In this way, the informativeness
level of each candidate phrase is calculated by multiplying the computed
cosine similarity scores. After that, the candidate phrases are ranked with
respect to their informativeness level scores in ascending order.

Let’s assume that a given document ${doc}_{i}$ contains $n$ sentences. As the
first step, ${doc}_{i}$ is splitted into its sentences of ${sent}_{j}$ for
$j=1,2,3,...n$. Then, each sentence ${sent}_{j}$ is given to the Stanza and
based on the dependency tree information and the rule-based approach let’s
assume that in total $m$ candidate phrases are extracted from ${sent}_{j}$. In
this case, for each candidate phrase ${cand}_{k}$ where $k=1,2,3,...m$, the
informativeness level of ${inf}_{{cand}_{k}}$ is computed as follows:

(2)
${inf}_{{cand}_{k}}=cos({cand}_{k},{sent}_{j})*cos({sent}_{j},{doc}_{i})\vspace{1em}$

Note that $cos(a,b)$ computes the cosine similarity score between the textual
contents of $a$ and $b$. In Eq. (2) the first component is the cosine
similarity score computed between the candidate phrase and the sentence;
while, the second component denotes the similarity score between the sentence
and the whole document. After computing the score of ${inf}_{{cand}_{k}}$ for
all $m$ extracted candidate phrases in ${sent}_{j}$, we fulfill the same
computation for all the candidate phrases of $n$ sentences. Then, for each
document ${doc}_{i}$, all of these scores are aggregated into a list which is
sorted with respect to the computed informativeness level scores – the
candidate phrases with higher scores are expected to be more informative about
${doc}_{i}$.

##  3\. Experimental Setup

In this section, we give the details of our experiments. For this, first we
provide information about the evaluation datasets, and then we report the
results as displayed in Table LABEL:tab:keyphrase and discuss them.

###  3.1. Dataset

For evaluating our key-phrase pipeline capability, we used the same four
benchmark datasets, namely  _INSPEC_ ,  _DUC_ ,  _NUS_ , and  _SemEval2017_
that have been used in unsupervised key-phrase extraction literature.
_INSPEC_ is a scientific paper abstract dataset that contains around $2000$
abstracts with the assigned key-phrases. On the other hand,  _DUC_ is composed
of $308$ newspaper articles and the articles are annotated with key-phrases
from TREC-9 by the articles’ authors.  _NUS_ is a dataset of $211$ full
scientific papers, and the papers are assigned with key-phrases by both
authors and annotators. Lastly,  _SemEval2017_ is a dataset of $493$
paragraphs selected from ScienceDirect 333https://www.sciencedirect.com/
articles, in the research fields of Computer Science, Material Sciences, and
Physics. The articles are assigned with key-phrases by undergraduate students
and expert annotators.

Based on the information, datasets show distinct properties since they contain
various kinds of textual content, i.e. news articles, scientific papers, etc.,
as well as their annotation procedure for assigning the key-phrases are
different from each other. Apart from these, based on the dataset statistics
in Table 2, the datasets are also dissimilar in the sense that they contain
longer or shorter documents. For instance,  _INSPEC_ is composed of shorter
documents while  _NUS_ includes x40 longer documents. Additionally, datasets
include a different number of key-phrases on average for each document. All of
these diverse dataset properties can affect the key-phrase extraction
capability of a given model; therefore we believe that our evaluation results
are sufficiently comprehensive. Nonetheless, the-state-of-the-art approaches
displayed in Table 1 evaluated their models only on three benchmark datasets,
MultipartiteRank and EmbedRank used  _INSPEC_ ,  _DUC_ ,  _NUS_ whereas
SIFRank used  _INSPEC_ ,  _DUC_ , and  _SemEval2017_. Finally, SingleRank used
only  _DUC_ and TopicRank utilized  _INSPEC_ ,  _SemEval2010_ ,  _WikiNews_
and  _DEFT_.

| INSPEC | DUC2001 | NUS | SemEval2017
---|---|---|---|---
# key-phrases | 14.11 | 8.07 | 11.71 | 17.30
# tokens | 124.25 | 732.78 | 7056.17 | 168.86
# documents | 2000 | 308 | 211 | 493
Table 2. Average number of tokens and key-phrases for each document in the
four benchmark datasets used.

###  3.2. Results & Discussion

Our results in Table 1 show that sBERT-Rank outperforms the state-of-the-art
unsupervised key-phrase extraction approaches in the  _NUS_ dataset in terms
of recall values, and it also gives comparable F1-scores with the other
methods. Moreover, sBERT-Rank achieves the highest recall values on the
_INSPEC_ and  _DUC_ datasets and comparable results in terms of F1-scores,
especially on the  _INSPEC_ dataset. Apart from this, sBERT-Rank achieves the
second-highest recall values on the  _SemEval2017_ dataset after the SIFRank,
and gives the third-highest F1-scores among the other approaches. Even if the
SIFRank surpasses sBERT-Rank on the  _SemEval2017_ , it gives very low scores
of all precision, recall and F1-scores on the  _NUS_ dataset – in terms of
F1-scores for instance sBERT-Rank outweighs the SIFRank by at least 3.5 times.
This shows that the SIFRank cannot provide consistent results, probably for
datasets containing very long documents like the  _NUS_ dataset. It might not
be successful for extracting key-phrases. Unlike our proposed pipeline, sBERT-
Rank gives consistent results irrespective of the dataset properties. This is
the only pipeline in Table 1 that either outperforms other approaches, or
achieves comparable scores (among the top-3 highest scores). We believe that
achieving comparable results on different datasets regardless of their
instrinsic properties is an important property for an unsupervised pipeline
that is expected to be leveraged on diverse datasets from various domains in
practice.

This important property of our introduced pipeline sBERT-Rank can probably be
attributed to the fact that the informativeness level of a given candidate
phrase is computed by multiplying the two similarity scores. The candidate can
obtain a high informativeness score, only if it is informative of the
corresponding sentence as well as if the sentence is informative of the
document at hand. This step not only helps us to select the most informative
candidate phrases from the first set of extracted ones, but it also alleviates
the scalability-related issues that most of the aforementioned state-of-the-
art approaches possibly suffer in practice. Note that regarding the
scalability, not only the second step of selecting informative candidates
based on their informativeness scores, but also the initial step of extracting
the first set of candidate phrases helps since we obtain a smaller first set
since we have an additional constraint that was mentioned in Section 2.1.

In addition to these, our proposed pipeline sBERT-Rank has its own
limitations. Except for the  _NUS_ dataset, sBERT-Rank gives comparable yet
lower precision scores. Improving the precision scores by examining the
experimental results with the corresponding dataset properties is left as
future work. Nonetheless, the results indicate that the precision scores of
sBERT-Rank gradually decrease while their recall scores gradually increase
when the candidate number increases from $5$ to $15$ on all the four benchmark
datasets. For reproducibility purposes, you can find our code at
https://github.com/abdurrezzak/sBERT-Rank.

The hypothetical function generating the key-phrases might be way to complex
to be approximated by the aforementioned scores. Thus, as the number of
candidates increase, the approximation that supposedly works with a small
number of candidates might diverge from being a good predictor. Which is
counter-intuitive to the fact that recall values increase in all of the
methods as the number of candidates increases but is also reasonable when we
consider that the number of key-phrases to detect is the same and generated by
a constant hypothetical base distribution.

##  4\. Conclusion & Future Work

In this work, we proposed a new unsupervised pipeline, sBERT-Rank which is a
simple, easily-extendible and corpus-independent pipeline for key-phrase
extraction. sBERT-Rank leverages noun-phrases extracted from dependency
parsing in order to detect candidate phrases with the rule-based approach.
This step helps us to create a good set of informative candidate phrases.
Then, using sBERT, candidate phrases, sentences, and documents are all
represented in a high-dimensional vector space, which serves to capture the
semantic similarity of the given textual contents. The scoring method of
sBERT-Rank provides the user with informativeness scores for each candidate
regardless of its position in the given text. Furthermore, the scoring
mechanism with two components does not favor a candidate if its sentence not
relevant or not informative for the current document at hand.

As a future work, we aim to evaluate the model capability of sBERT-Rank on
other languages since it is easily-extendible. Nonetheless, to improve the
overall precision scores and to alleviate the problem of gradually decreasing
precision scores, different scoring functions can be investigated. Moreover,
key-phrase results with the corresponding dataset properies can be elaborately
examined for finding better scoring functions or the pipeline steps can be
slightly modified to prevent the gradual increase/decrease of recall/precision
scores when the candidate number increases.

## References

  * (1)
  * Arora et al. (2017) Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017.  A simple but tough-to-beat baseline for sentence embeddings. In _International conference on learning representations_.
  * Bennani-Smires et al. (2018) Kamil Bennani-Smires, Claudiu Musat, Andreea Hossmann, Michael Baeriswyl, and Martin Jaggi. 2018\.  Simple unsupervised keyphrase extraction using sentence embeddings.  _arXiv preprint arXiv:1801.04470_ (2018).
  * Boudin (2018) Florian Boudin. 2018\.  Unsupervised keyphrase extraction with multipartite graphs.  _arXiv preprint arXiv:1803.08721_ (2018).
  * Bougouin et al. (2013) Adrien Bougouin, Florian Boudin, and Béatrice Daille. 2013\.  Topicrank: Graph-based topic ranking for keyphrase extraction. In _International joint conference on natural language processing (IJCNLP)_. 543–551.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Hasan and Ng (2014) Kazi Saidul Hasan and Vincent Ng. 2014.  Automatic keyphrase extraction: A survey of the state of the art. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 1262–1273.
  * Lau and Baldwin (2016) Jey Han Lau and Timothy Baldwin. 2016.  An empirical evaluation of doc2vec with practical insights into document embedding generation.  _arXiv preprint arXiv:1607.05368_ (2016).
  * Pagliardini et al. (2017) Matteo Pagliardini, Prakhar Gupta, and Martin Jaggi. 2017\.  Unsupervised learning of sentence embeddings using compositional n-gram features.  _arXiv preprint arXiv:1703.02507_ (2017).
  * Qi et al. (2020) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D Manning. 2020.  Stanza: A python natural language processing toolkit for many human languages.  _arXiv preprint arXiv:2003.07082_ (2020).
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_ (2019).
  * Sun et al. (2020) Yi Sun, Hangping Qiu, Yu Zheng, Zhongwei Wang, and Chaoran Zhang. 2020.  SIFRank: a new baseline for unsupervised keyphrase extraction based on pre-trained language model.  _IEEE Access_ 8 (2020), 10896–10906.
  * Wan and Xiao (2008) Xiaojun Wan and Jianguo Xiao. 2008.  CollabRank: towards a collaborative approach to single-document keyphrase extraction. In _Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008)_. 969–976.


# TMoE-P: Towards the Pareto Optimum
for Multivariate Soft Sensors

Licheng Pan, , Hao Wang, , Zhichao Chen, Yuxing Huang, Xinggao Liu The authors
are with the State Key Laboratory of Industrial Control Technology, College of
Control Science and Engineering, Zhejiang University, Hangzhou 310027, China
(e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>lxg@zju.edu.cn).

###### Abstract

Multi-variate soft sensor seeks accurate estimation of multiple quality
variables using measurable process variables, which have emerged as a key
factor in improving the quality of industrial manufacturing. The current
progress stays in some direct applications of multitask network architectures;
however, there are two fundamental issues remain yet to be investigated with
these approaches: (1) negative transfer, where sharing representations despite
the difference of discriminate representations for different objectives
degrades performance; (2) seesaw phenomenon, where the optimizer focuses on
one dominant yet simple objective at the expense of others. In this study, we
reformulate the multi-variate soft sensor to a multi-objective problem, to
address both issues and advance state-of-the-art performance. To handle the
negative transfer issue, we first propose an Objective-aware Mixture-of-
Experts (OMoE) module, utilizing objective-specific and objective-shared
experts for parameter sharing while maintaining the distinction between
objectives. To address the seesaw phenomenon, we then propose a Pareto
Objective Routing (POR) module, adjusting the weights of learning objectives
dynamically to achieve the Pareto optimum, with solid theoretical supports. We
further present a Task-aware Mixture-of-Experts framework for achieving the
Pareto optimum (TMoE-P) in multi-variate soft sensor, which consists of a
stacked OMoE module and a POR module. We illustrate the efficacy of TMoE-P
with an open soft sensor benchmark, where TMoE-P effectively alleviates the
negative transfer and seesaw issues and outperforms the baseline models.

###### Index Terms:

Soft Sensor, Multi-objective Optimization, Negative Transfer, Seesaw.

##  I Introduction

Process monitoring plays a significant role in contemporary industry and is
directly associated to the manufacture of critical industrial products, e.g.,
oil, gas, rare metals, iron, and steel, which are integral to modern human
life and national economies. Monitoring the dynamics of critical quality
variables in manufacturing has become one of the main concerns in order to
meet urgent and demanding requirements, including increasing yields, reducing
material consumption, protecting the environment and ensuring the safety of
manufacturing processes. The main challenge with monitoring is that some
quality variables are extremely difficult to measure. For instance, in deep
water gas-lift oil well process, down-hole pressure is an extremely useful
indicator for assessing the manufacture quality [1], but it is difficult to
measure with hardware sensors (e.g., permanent down-hole pressure gauges) due
to the inability of hardware sensors in high pressure and salinity
environments [2].

Soft sensors aims to estimate immeasurable quality variables with measurable
process variables, which can be categorized as model-driven and data-driven
approaches. With the success of machine learning and database technology,
data-driven approaches have been predominant for building effective soft
sensors, which involves building statistical estimates of quality variables
with process variables. At the very beginning, the data-driven soft sensors
were implemented with linear statistical approaches, such as principal
component regression [5, 6], partial least squares [7] and gaussian process
[8]. To depict nonlinear relationships between variables, nonlinear models in
the machine learning community were further involved in soft sensors,
represented by the support vector regression [9, 10], decision tree [11] and
kernel regression [12]. More recently, with the great advancement of deep
learning techniques [3, 4], soft sensors have been dominated by deep neural
methods. Representative methods can be roughly categorized into several types:
autoencoder networks [13, 14, 15], recurrent neural networks [16, 17, 18],
convolution neural networks [19, 20], graph neural networks [21, 22], and
self-attentive networks [23, 24], with each type of methods its own strengths
and weaknesses. For example, Autoencoder-based sensors can be refined with
semi-supervised setting, but they struggle to capture long-term sequence
patterns; RNN-based sensors can be incrementally updated for real-time
monitoring, but they suffer from sub-optimal accuracy and single-threaded
computational paradigm; self-attention based sensors can process the whole
sequence in a parallel manner, but they suffer from huge computational cost
and overfitting risk.

Overall, the aforementioned line of research focuses on developing more
efficient and effective architectures with the aim of improving the estimation
accuracy of _single_ quality variable. Despite their success, in real-world
practice there is more than one quality variable to be estimated, which
requires the construction of an effective multi-variate soft sensor (MVSS).
For example, the product concentration and reactant concentration in the
reactive distillation process must be measured concurrently to track the
separation energy consumption and product purity.

Different from Multi-task Learning (MTL), Multi-objective Optimization (MOO)
is a technology that can coordinate multiple objectives with internal
conflicts and connections to obtain Pareto optimal solutions, mainly through
the sharing mechanism. Generally, parameter sharing mechanism can be separated
into two categories: hard parameter sharing and soft parameter sharing. Hard
parameter sharing refers to the uniform sharing of the network’s underlying
parameters while maintaining the network’s top-level parameters’ independence
for objectives, such as UberNet [25], multilinear relationship networks [26],
and stochastic filter groups [27]. However, in circumstances when the
connection between objectives is poor, the direct sharing of underlying
parameters will result in performance decrease, which is known as NT
phenomenon. Soft parameter sharing introduces distinctive underlying shared
expert parameters to each objective, such as cross-stitch networks [28],
sluice networks [29], and MMoE [30, 33] The involvement of experts helps
mitigate the NT issue, but because the variations and interactions among
experts are neglected, it is challenging for multiple objectives to reach the
same performance of a single objective at once, leading to the seesaw
phenomenon.

This paper proposes a Task-aware Mixture-of-Experts framework for achieving
the Pareto optimum (TMoE-P) to solve issues of NT and seesaw, which were
brought on by disregarding the commonality and variability across experts in
parameter sharing. The model is made up of two modules: Objective-aware
Mixture-of-Experts (OMoE) and Pareto Objective Routing (POR). To start, the
OMoE module explicitly distinguishes between objective-specific and objective-
shared experts, reducing potentially damaging interactions among
representations. Secondly, the POR module solves an optimization problem
concerning the gradient of shared parameters using the gradient of each
objective, and then obtains the Pareto optimal model parameters, which
alleviates some objectives’ performance sacrifice throughout the model
training process.

The main contributions of this paper are summarized as follows:

  1. 1.

The soft sensor problem is stated from the standpoint of MOO, and it is
highlighted that the regression problem of multiple quality variables is
transformed into the joint optimization of multiple regression objectives. A
MOO model will be used to model the relationships between and within sequences
of quality variables.

  2. 2.

A TMoE-P model based on the OMoE module and POR module is proposed to solve
the NT and seesaw issues of multi-objective soft sensor problem. The solutions
for the two modules are explicit objective-specific with objective-shared
representations learning, and Pareto optimal MOO.

  3. 3.

Numerous off-line experiments were conducted in the Sulfur Recovery Unit
process to assess the efficiency of TMoE-P in resolving the issues of NT and
seesaw. The results and mathematical proofs reveal that TMoE-P outperforms the
baseline models in soft sensor applications.

The rest of this paper is organized as follows: Section II covers the
preliminaries of multi-objective soft sensor problem and Pareto theory.
Section III introduced the novel TMoE-P approach. Section IV is an
experimental study on the well-known Sulfur Recovery Unit. The final section
is the conclusion.

##  II Preliminaries

###  II-A Multi-Objective Soft Sensor Problem

Given a dataset with i.i.d. data points
$\left\\{\boldsymbol{x}_{i},\boldsymbol{y}_{i}\right\\}_{i\in\\{1,2,\ldots,N\\}}$
where $\boldsymbol{x}_{i}=\left[x_{i,1},x_{i,2},\ldots,x_{i,D}\right]^{\top}$
is the $D$-dimensional process variables,
$\boldsymbol{y}_{i}=\left[y_{i,1},y_{i,2},\ldots,y_{i,K}\right]^{\top}$ is the
$K$-dimensional quality variables, $y_{i,k}$ is the label of the
$k^{\text{th}}$ objective, $K$ is the number of objectives, and $N$ is the
total number of data points. Multi-objective soft sensor (MOSS) seeks to
establish an inference mathematical model
$f\left(\boldsymbol{x}|\boldsymbol{\theta}\right)$ from process variables to
quality variables through sharing mechanism. When a new process variables
sequence $\boldsymbol{x}_{N+m}|_{m\geq 1}$ arrives, the quality variables
sequence $\hat{\boldsymbol{y}}_{N+m}$ can be predicted as follows:

$\hat{\boldsymbol{y}}_{N+m}=f(\boldsymbol{x}_{N+m}|\boldsymbol{\theta})\cong\left[y_{N+m,1},\ldots,y_{N+m,K}\right]^{\top}.$
(1)

Once we have set the associated loss function for each objective as
$\mathcal{L}_{k}$, the mathematical form of MOSS is given as:

$\displaystyle\min_{\boldsymbol{\theta}_{\text{sh}},\boldsymbol{\theta}_{1},\ldots,\boldsymbol{\theta}_{K}}\mathcal{L}\left(\boldsymbol{\theta}_{\text{sh}},\boldsymbol{\theta}_{1},\ldots,\boldsymbol{\theta}_{K}\right)$
(2) $\displaystyle=$
$\displaystyle\min_{\boldsymbol{\theta}_{\text{sh}},\boldsymbol{\theta}_{1},\ldots,\boldsymbol{\theta}_{K}}\left[\hat{\mathcal{L}}_{1}\left(\boldsymbol{\theta}_{\text{sh}},\boldsymbol{\theta}_{1}\right),\ldots,\hat{\mathcal{L}}_{L}\left(\boldsymbol{\theta}_{\text{sh}},\boldsymbol{\theta}_{K}\right)\right]^{\top}.$

where $\boldsymbol{\theta}_{\text{sh}}$ are shared parameters between
objectives, $\boldsymbol{\theta}_{k}$ are objective-specific,
$\hat{\mathcal{L}}_{k}\left(\boldsymbol{\theta}_{\text{sh}},\boldsymbol{\theta}_{k}\right)$
is the empirical loss of $k^{\text{th}}$ objective on the dataset, defined as
$\hat{\mathcal{L}}_{k}\left(\boldsymbol{\theta}_{\text{sh}},\boldsymbol{\theta}_{k}\right)=\frac{1}{N}\sum_{i=1}^{N}\mathcal{L}_{k}\left(f_{k}\left(\boldsymbol{x}_{i}|\boldsymbol{\theta}_{\text{sh}},\boldsymbol{\theta}_{k}\right),y_{i,k}\right)$
and $f_{k}$ is the inference submodel in $f$ corresponding to $k^{\text{th}}$
objective.

(a) Hard Sharing

(b) Cross-Talk

(c) Soft Sharing

Figure 1: Network routing for hard and soft parameter sharing.

By leveraging both common features shared between objectives and objective-
specific features, MOSS is capable of improving efficiency and generalization
performance. There are two types of parameter sharing mechanisms that are
frequently utilized nowadays. One is hard parameter sharing shown in Fig.
1(a), which involves a shared underlying model structure with common hidden
layers across objectives. Although this structure minimizes the likelihood of
overfitting, optimization conflicts resulting from objective heterogeneity
still exists. The other type is soft parameter sharing, which shares experts
capable of cross-talk at the bottom layer or fuses experts information via
gating networks, as illustrated in Fig. 1(b) and 1(c). Compared with hard
parameter sharing, soft parameter sharing has more objective-specific
parameters and can nevertheless achieve higher performance when objective
dependencies are complicated and parameter conflicts arise.

###  II-B Negative Transfer & Seesaw Phenomenon In MOSS

NT and seesaw phenomenons are two major issues that must be addressed
throughout the MOO process of multiple quality variables in MOSS. Specially,
NT phenomenon means that the multi-objective performance attained through
training via (2) has deteriorated in comparison to the performance of
individual modeling training for each objective. The typical explanation is
that there is a weak or loose connection between the multiple objectives of
joint modeling or that the MOO model is difficult to learn the similarities
and differences between the various objectives.

The seesaw phenomenon usually occurs when the association between multiple
objectives is complex. It implies that multiple objectives cannot be performed
at a high level at the same time, and that secondary objectives are frequently
sacrificed to achieve excellent performance on the main objective.

###  II-C Pareto Optimality For MOSS

Figure 2: The structure of Objective-aware Mixture-of-Experts module.

The MOSS problem specified in (1) tries to obtain estimates
$\left\\{\hat{y}_{k}\right\\}_{k\in\\{1,2,\ldots,K\\}}$ for each objective by
minimizing the prediction loss of the inference model $f$ on each objective by
solving the MOP defined in (2). However, MOP is tough to solve and acquire the
optimal solution. The main reason is that MOP has multiple objective
functions, and the superiority and inferiority of MOP solutions cannot be
compared and ranked by traditional size relation comparison. To evaluate the
correlation between solutions, the Pareto optimality theory is proposed and
defined as follows [31] in the parameter sharing mode:

  1. 1.

A parameter solution $\boldsymbol{\theta}$ is said to dominate another
parameter solution $\widetilde{\boldsymbol{\theta}}$ if
$\hat{\mathcal{L}}_{k}\left(\boldsymbol{\theta}_{\text{sh}},\boldsymbol{\theta}_{k}\right)\leq\hat{\mathcal{L}}_{k}\left(\widetilde{\boldsymbol{\theta}}_{\text{sh}},\widetilde{\boldsymbol{\theta}}_{k}\right)$
for all objectives $k$, and there is a objective $k^{\prime}$ that satisfies
$\hat{\mathcal{L}}_{k^{\prime}}\left(\boldsymbol{\theta}_{\text{sh}},\boldsymbol{\theta}_{k^{\prime}}\right)<\hat{\mathcal{L}}_{k^{\prime}}\left(\widetilde{\boldsymbol{\theta}}_{\text{sh}},\widetilde{\boldsymbol{\theta}}_{k^{\prime}}\right)$.

  2. 2.

A parameter solution $\boldsymbol{\theta}^{*}$ is called Pareto optimal if and
only if there is no parameter solution $\boldsymbol{\theta}$ dominating
$\boldsymbol{\theta}^{*}$.

In fact, the solution of MOP is not unique, but rather a set of Pareto optimal
solutions, often known as Pareto set. Besides, the KKT conditions listed below
need to be met while solving MOP:

  1. 1.

There exists $c_{1},c_{2},\ldots,c_{K}$ that satisfies $\sum_{k=1}^{K}c_{k}=1$
and
$\sum_{k=1}^{K}c_{k}\nabla_{\boldsymbol{\theta}_{\text{sh}}}\hat{\mathcal{L}}_{k}\left(\boldsymbol{\theta}_{\text{sh}},\boldsymbol{\theta}_{k}\right)=0$.

  2. 2.

For all objectives $k$,
$\nabla_{\boldsymbol{\theta}_{k}}\hat{\mathcal{L}}_{k}\left(\boldsymbol{\theta}_{\text{sh}},\boldsymbol{\theta}_{k}\right)=0$
is satisfied.

A solution that meets the above conditions is called a Pareto stationary
point, and all Pareto optimal points are Pareto stationary, although the
opposite is not always true. Therefore, while training the MOO model, we focus
on leveraging Pareto stationary point to approximate the optimal MOO model
parameters.

##  III Proposed Method

###  III-A Objective-aware Mixture-of-Experts Module

The OMoE module is a multi-gate hybrid expert module inspired by [32] that was
designed under the proposed framework. In contrast to the multi-gate hybrid
expert models used in [30] and [33], the OMoE module explicitly distinguishes
between objective-shared experts and objective-specific experts. Based on the
MMoE structure of using objective-specific gate to fuse shared expert
information to mitigate the NT phenomenon, the OMoE module increases
objective-specific experts to achieve single-objective modeling performance,
thereby solving the seesaw phenomenon caused by MOO imbalance.

Specifically, the OMoE module is composed of several objective-specific and
objective-shared feature extraction blocks, each with its own set of
objective-specific expert module, objective-shared expert module, and
corresponding gating networks. Multi-layer networks associated with each
objective make up the OMoE module’s top layer. The number of objective-shared
and objective-specific experts in the feature extraction blocks, as well as
the width and depth of the experts networks and tower networks, are all
hyperparameters. Fig. 2 shows an example of a single feature extraction block
to illustrate data flow in the OMoE module. The inputs of the current feature
extraction block are respectively derived from the selected outputs of
objective-specific and objective-shared expert module in the previous feature
extraction block. For the $j^{\text{th}}$ feature extraction block, the
outputs of $k^{\text{th}}$ objective-specific expert module and the objective-
shared expert module will be selected and concatenated as follows to obtain
features $O_{k}$ and $O_{s}$:

$\displaystyle O_{k}\left(\boldsymbol{x}^{(k)}\right)$
$\displaystyle=\left[\text{cat}\left\\{E_{(k,p)}^{\top}\right\\}_{p=1}^{n_{k}},\text{cat}\left\\{E_{(s,q)}^{\top}\right\\}_{q=1}^{n_{s}}\right]^{\top},$
(3) $\displaystyle O_{s}\left(\boldsymbol{x}^{(s)}\right)$
$\displaystyle=\left[\text{cat}\left\\{\left\\{E_{(k,p)}^{\top}\right\\}_{p=1}^{n_{k}}\right\\}_{k=1}^{K},\text{cat}\left\\{E_{(s,q)}^{\top}\right\\}_{q=1}^{n_{s}}\right]^{\top}.$

where
$\text{cat}\left\\{\boldsymbol{\alpha}^{\top},\boldsymbol{\beta}^{\top}\right\\}=\left[\boldsymbol{\alpha}^{\top},\boldsymbol{\beta}^{\top}\right]$
is the concatenation function, $\boldsymbol{x}^{(k)}$ is the input of the
$k^{\text{th}}$ objective-specific expert module, $E_{(k,p)}$ is the output
features of the $p^{\text{th}}$ expert in the $k^{\text{th}}$ objective-
specific expert module, $\boldsymbol{x}^{(s)}$, $E_{(s,q)}$ correspond to the
objective-shared expert module, and $n_{k}$, $n_{s}$ are the number of experts
in the $k^{\text{th}}$ objective-specific expert module and objective-shared
expert module respectively. It should be noted that the corresponding inputs
for all modules in the first feature extraction block is the same, that is
$\boldsymbol{x}^{(k)}=\boldsymbol{x}^{(s)}$.

Figure 3: The workflow of gating network.

Following selective concatenation, the output features are selectively fused
via gating networks. The gating network is a single-layer fully connected
network with softmax as the activation function, and the input is used as a
filter to determine the weighted sum of the chosen splicing vectors, as shown
in Fig. 3. Taking the $k^{\text{th}}$ objective-specific gating network as an
example, its output after feature extraction block (FEB) is also used as the
input of the $k^{\text{th}}$ objective-specific expert module in block $j+1$
as follows:

$\text{FEB}\left(\boldsymbol{x}^{(k)}\right)=g_{k}\left(\boldsymbol{x}^{(k)}\right)O_{k}\left(\boldsymbol{x}^{(k)}\right).$
(4)

where $g_{k}$ is the weight calculation network and also the input linear
transformation with a sofmax layer:

$g_{k}\left(\boldsymbol{x}^{(k)}\right)=\text{softmax}\left(\boldsymbol{W}_{k}\boldsymbol{x}^{(k)}\right).$
(5)

where $\boldsymbol{W}_{k}\in\mathbb{R}^{n\times d}$ is a trainable matrix, $n$
is the number of experts in $O_{k}$ and $d$ is the dimension of input
$\boldsymbol{x}^{(k)}$.

Finally, the output of the gating network is fed into the matching tower
network to obtain the predicted value:

$\hat{y}_{k}=h_{k}\left(g_{k}^{(n_{b})}\left({\boldsymbol{x}}^{(k,n_{b})}\right)\right).$
(6)

where $h_{k}$ is the $k^{\text{th}}$ tower network and $n_{b}$ is the number
of feature extraction block.

The OMoE module is a MOO model of soft parameter sharing framework, but unlike
MMoE and cross-stitch networks, OMoE divides feature extraction into
objective-specific and objective-shared parts. This design adopts separate
gating networks to integrate expert information for multiple objectives to
mitigate the NT phenomenon, while using objective-specific experts to achieve
performance comparable to single-objective modeling to mitigate the seesaw
phenomenon.

###  III-B Pareto Objective Routing Module

The genesis of the seesaw phenomenon is more than just a problem with the
architecture of MOO models. Another cause of the uneven distribution of
training resources among objectives is the imbalance of loss weights. The
optimization direction of the MOO model is determined by the weighted sum of
the loss weight and the objective loss, and the incorrect setting of the loss
weight may sacrifice the performance of the secondary objectives to improve
the performance of the primary objectives. Therefore, the POR module is
designed in such a way that the loss weight is dynamically modified as the
model parameters are optimized.

####  III-B1 Problem Statement

As previously stated, the MOP defined by (2) is difficult to have a unique
solution to minimize the loss of all objectives. The typical solution is to
convert the loss vector optimization in (2) into scalar optimization, which is
realized by the weighted sum of the following formula:

$\min_{\boldsymbol{\theta}}\mathcal{L}_{\boldsymbol{w}}\left(\boldsymbol{\theta}\right)=\min_{\boldsymbol{\theta}}\boldsymbol{w}^{\top}\mathcal{L}\left(\boldsymbol{\theta}\right)=\min_{\boldsymbol{\theta}}\sum_{k=1}^{K}w_{k}\hat{\mathcal{L}}_{k}\left(\boldsymbol{\theta}_{\text{sh}},\boldsymbol{\theta}_{k}\right).$
(7)

where $\boldsymbol{w}=\left[w_{1},\ldots,w_{K}\right]^{\top}$ is the loss
weight, satisfying $\sum_{k=1}^{K}w_{k}=1$ and $w_{k}\geq 0$ for all
objectives $k$.

The Pareto stationary point of the multi-objective loss minimization problem
in scalar form can be approximated using the following sub-optimization
problem:

$\min_{\boldsymbol{w}}L\left(\boldsymbol{w}\right)=\min_{\boldsymbol{w}}\left\|\nabla_{\boldsymbol{\theta}_{\text{sh}}}\mathcal{L}_{\boldsymbol{w}}\right\|_{2}^{2}.$
(8)

where $\|\cdot\|_{2}$ is the $L_{2}$-norm.

The sub-optimization problem (8) is a linearly constrained optimization
problem about the loss weight $\boldsymbol{w}$, and its optimal solution
$\boldsymbol{w}^{*}$ can be obtained through linear search algorithm Frank-
Wolfe [34].

Algorithm 1 POR for Parameter Optimization
---
Input: $\boldsymbol{\theta}$: parameters of the MOO model;
$\nabla_{\boldsymbol{\theta}_{\text{sh}}}\hat{\mathcal{L}}_{k}$: gradient of
each objective loss to the objective-shared parameters;
$\nabla_{\boldsymbol{\theta}_{k}}\hat{\mathcal{L}}_{k}$: gradient of each
objective loss to the objective-specific parameters.
Parameter: $R$: the maximum number of iterations; $\boldsymbol{e}_{\hat{k}}$:
a $K$-dimensional unit vector whose $\hat{k}^{\text{th}}$ element is 1.
Output: $\boldsymbol{w}^{*}$: loss weight which minimizes
$L\left(\boldsymbol{w}\right)$; $\boldsymbol{\theta}$: optimized parameters.
1: |  Initialize $\boldsymbol{w}^{(0)}=\left[w_{1},\ldots,w_{K}\right]^{\top}=\left[\frac{1}{T},\ldots,\frac{1}{T}\right]^{\top}$
2: |  Compute $\boldsymbol{M}$ with element $\boldsymbol{M}_{i,j}=\left(\nabla_{\boldsymbol{\theta}_{\text{sh}}}\hat{\mathcal{L}}_{i}\right)^{\top}\left(\nabla_{\boldsymbol{\theta}_{\text{sh}}}\hat{\mathcal{L}}_{j}\right)$
3: |  Set $r=0$
4: |  repeat
5: |  $\hat{k}\leftarrow\text{argmin}_{i}\sum_{j=1}^{K}w_{j}\boldsymbol{M}_{i,j}$
6: |  $v^{*}\leftarrow\text{argmin}_{v}\left[v\boldsymbol{e}_{\hat{k}}+(1-v)\boldsymbol{w}\right]^{\top}\boldsymbol{M}\left[v\boldsymbol{e}_{\hat{k}}+(1-v)\boldsymbol{w}\right]$
7: |  $\boldsymbol{w}\leftarrow v^{*}\boldsymbol{e}_{\hat{k}}+(1-v^{*})\boldsymbol{w}$
8: |  $r=r+1$
9: |  until $v^{*}\rightarrow 0$ or $r=R$
10: |  Update $\boldsymbol{\theta}_{k}\leftarrow\boldsymbol{\theta}_{k}-\eta\nabla_{\boldsymbol{\theta}_{k}}\hat{\mathcal{L}}_{k}\left(\boldsymbol{\theta}_{\text{sh}},\boldsymbol{\theta}_{k}\right)$ for all objectives
11: |  Update $\boldsymbol{\theta}_{\text{sh}}\leftarrow\boldsymbol{\theta}_{\text{sh}}-\eta\sum_{k=1}^{K}w_{k}\nabla_{\boldsymbol{\theta}_{\text{sh}}}\hat{\mathcal{L}}_{k}\left(\boldsymbol{\theta}_{\text{sh}},\boldsymbol{\theta}_{k}\right)$

####  III-B2 Frank-Wolfe Algorithm

The Frank-Wolfe algorithm is a method for approximately minimizing the
objective function in the feasible zone after first-order linearization. The
loss function $L\left(\boldsymbol{w}\right)$ in (8) can be linearized as
follows:

$\min_{\boldsymbol{w}}L\left(\boldsymbol{w}\right)\rightarrow\min_{\boldsymbol{w}}\nabla_{\boldsymbol{w}}L\left(\boldsymbol{w}^{(r)}\right)^{\top}\boldsymbol{w}.$
(9)

where $\boldsymbol{w}^{(r)}$ is the approximation point in current iteration.

In the case of two objectives, (8) can be generalized as
$\min_{w}\|w\boldsymbol{l}_{1}+(1-w)\boldsymbol{l}_{2}\|_{2}^{2}$, where
$\boldsymbol{l}_{1},\boldsymbol{l}_{2}$ are the loss gradient vectors of the
two objectives relative to the objective-shared parameters. The Frank-Wolfe
algorithm points out that the weight $w^{*}$ that makes the norm in the
preceding formula the smallest is:

$w^{*}=\left\\{\begin{array}[]{lc}1&\text{if}\
\boldsymbol{l}_{1}^{\top}\boldsymbol{l}_{2}\geq\boldsymbol{l}_{1}^{\top}\boldsymbol{l}_{1}\\\
0&\text{if}\
\boldsymbol{l}_{1}^{\top}\boldsymbol{l}_{2}\geq\boldsymbol{l}_{2}^{\top}\boldsymbol{l}_{2}\\\
\frac{(\boldsymbol{l}_{2}-\boldsymbol{l}_{1})^{\top}\boldsymbol{l}_{2}}{\|\boldsymbol{l}_{1}-\boldsymbol{l}_{2}\|_{2}^{2}}&\text{otherwise}\end{array}\right..$
(10)

For the common situation where the number of objectives is higher than two,
the objective gradient interconnection matrix $\boldsymbol{M}$ should be
firstly defined as follows:

$\boldsymbol{M}_{i,j}=\left(\nabla_{\boldsymbol{\theta}_{\text{sh}}}\hat{\mathcal{L}}_{i}\left(\boldsymbol{\theta}_{\text{sh}},\boldsymbol{\theta}_{i}\right)\right)^{\top}\left(\nabla_{\boldsymbol{\theta}_{\text{sh}}}\hat{\mathcal{L}}_{j}\left(\boldsymbol{\theta}_{\text{sh}},\boldsymbol{\theta}_{j}\right)\right).$
(11)

Further calculate the weighted row sum of $\boldsymbol{M}$ to find the
objective $\hat{t}$ that minimizes the weighted sum of its own gradient
multiplied by the gradients of other objectives. The weight $v^{*}$ that makes
$\left[v\boldsymbol{e}_{\hat{k}}+(1-v)\boldsymbol{w}^{(r)}\right]^{\top}\boldsymbol{M}\left[v\boldsymbol{e}_{\hat{k}}+(1-v)\boldsymbol{w}^{(r)}\right]$
the least can be derived by (10). Finally, the loss weight $\boldsymbol{w}$ is
updated as follows:

$\boldsymbol{w}^{(r+1)}\leftarrow
v^{*}\boldsymbol{e}_{\hat{k}}+(1-v^{*})\boldsymbol{w}^{(r)}.$ (12)

where $r$ is the number of iterations and
$\boldsymbol{e}_{\hat{k}}\in\mathbb{R}^{K}$ is a unit vector whose
$\hat{k}^{\text{th}}$ element is 1.

Iteratively update the loss weight $\boldsymbol{w}$ until $v^{*}$ approaches
zero, and then find the loss weight $\boldsymbol{w}^{*}$ that eventually meets
the normalization and non-negative conditions. Ultimately, the objective-
specific parameters are updated in the same way as the regular network
parameters, with the objective-shared parameters updated using the loss
weights:

$\displaystyle\boldsymbol{\theta}_{k}$
$\displaystyle\leftarrow\boldsymbol{\theta}_{k}-\eta\nabla_{\boldsymbol{\theta}_{k}}\hat{\mathcal{L}}_{k}\left(\boldsymbol{\theta}_{\text{sh}},\boldsymbol{\theta}_{k}\right)\quad
k\in\\{1,\ldots,K\\},$ (13) $\displaystyle\boldsymbol{\theta}_{\text{sh}}$
$\displaystyle\leftarrow\boldsymbol{\theta}_{\text{sh}}-\eta\sum_{k=1}^{K}w_{k}\nabla_{\boldsymbol{\theta}_{\text{sh}}}\hat{\mathcal{L}}_{k}\left(\boldsymbol{\theta}_{\text{sh}},\boldsymbol{\theta}_{k}\right).$

where $\eta$ is the learning rate.

Algorithm 1 displays the POR module for parameter optimization of the MOO
model.

####  III-B3 Analysis of Primal-Dual Convergence

We shall demonstrate the primal-dual convergence of the Frank-Wolfe algorithm,
which is stronger than the convergence of the primal problem, using a simple
dual gap proof framework. The convergence of the primal error is given first
in Theorem III.1.

######  Theorem III.1.

For each $r\geq 1$, the iterates $\boldsymbol{w}^{(r)}$ of the Frank-Wolfe
algorithm satisfy

$L(\boldsymbol{w}^{(r)})-L(\boldsymbol{w}^{*})\leq\frac{2C_{f}}{r+2}(1+\delta).$
(14)

where $\delta\geq 0$ is the accuracy to which the linear subproblems (9) are
solved, and $C_{f}$ is the curvature constant of loss function $L$ which is
defined as

$C_{f}=\sup_{\gamma\in[0,1]}\frac{2}{\gamma^{2}}(L(\boldsymbol{w}^{\prime})-L(\boldsymbol{w})-\langle\boldsymbol{w}^{\prime}-\boldsymbol{w},\nabla
L(\boldsymbol{w})\rangle).$ (15)

where
$\boldsymbol{w}^{\prime}=\boldsymbol{w}+\gamma(\boldsymbol{s}-\boldsymbol{w})$
and $\boldsymbol{s}$ is a feasible solution.

The proof of the above convergence theorem is given in Appendix for
completeness. Theorem III.1 guarantees a small raw error, while the optimal
value $L(\boldsymbol{w}^{*})$ and the curvature constant $C_{f}$ are usually
unknown. The surrogate duality gap is defined below for more convenient
estimation of the approximation quality

$\phi(\boldsymbol{w})=\max\ \langle\boldsymbol{w}-\boldsymbol{s},\nabla
L(\boldsymbol{w})\rangle.$ (16)

Convexity of $L$ implies that the linearization
$L(\boldsymbol{w})+\langle\boldsymbol{w}-\boldsymbol{s},\nabla
L(\boldsymbol{w})\rangle$ always lies below the function $L$, which provides
the property of the duality gap, $\phi(\boldsymbol{w})\geq
L(\boldsymbol{w})-L(\boldsymbol{w}^{*})$.

The Frank-Wolfe algorithm obtain guaranteed small duality gap
$\phi(\boldsymbol{w})\leq\epsilon$ after $O(\frac{1}{\epsilon})$ iterations
even if the linear subproblems 9 are only solved approximately.

######  Theorem III.2.

If the Frank-Wolfe algorithm is run for $R\geq 2$ iterations, then the
algorithm has a bounded duality gap with iterate $\boldsymbol{w}^{(r)}$,
$1\leq r\leq R$

$\phi(\boldsymbol{w}^{(r)})\leq\frac{2\beta C_{f}}{R+2}(1+\delta).$ (17)

where $\beta=\frac{27}{8}$.

By proving that the duality gap cannot be kept large in multiple iterations,
Theorem III.2 is proved by contradiction in Appendix.

###  III-C Architecture & Procedure

Figure 4: The general architecture of the proposed Task-aware Mixture-of-
Experts for achieving the Pareto optimum.

Fig. 4 depicts the general architecture of the proposed TMoE-P model. The
model is divided into two components: the OMoE module and the POR module. The
input is first routed through the OMoE module, where objective-specific and
objective-shared features are progressively separated using stacked feature
extraction blocks. After that, the gating networks screen and fuse the
separated features through (3) and (4). Additionally, the output of the last
feature extraction block will disgard the part of the objective-shared gating
network and estimate the quality variables via the tower networks.

The predicted quality variables sequence is utilized in conjunction with the
real values to calculate the loss of each objective, while backpropagation is
employed to derive the gradient of each objective loss for the objective-
shared parameters. The gradient will be submitted to the POR module, which
will thereafter iteratively search for the loss weight $\boldsymbol{w}^{*}$
that minimizes (8). The multi-objective loss $\mathcal{L}_{\boldsymbol{w}}$ is
calculated after $\boldsymbol{w}^{*}$ and the loss of each objective have been
weighted. Finally, the network parameters are optimized using (13) to approach
the Pareto optimal network parameters.

##  IV Experiments

In this section, we carry out extensive experiments using real material
measurement data to assess the effectiveness of the proposed model.

###  IV-A Experiment Setup

####  IV-A1 Dataset

Figure 5: Process flow of the SRU.

The dataset comes from Sulfur Recovery unit (SRU), which is an essential part
of managing sulfur emissions [35]. Fig. 5 depicts an SRU made up of four
parallel sulfur production lines, the major inputs of which are
$\text{H}_{2}\text{S}$-enriched MEA gas and $\text{H}_{2}\text{S}$\- and
$\text{NH}_{3}$-enriched SWS gas.

The key reaction in SRU that can purify acid gas and generate sulfur as a
byproduct is as follows:

$\left\\{\begin{array}[]{ll}3\text{H}_{2}\text{S}+\frac{1}{2}\text{O}_{2}&\rightarrow\text{SO}_{2}+2\text{H}_{2}\text{S}+\text{H}_{2}\text{O}\\\
\text{SO}_{2}+2\text{H}_{2}\text{S}&\rightarrow\text{S}_{x}+2\text{H}_{2}\text{O}\end{array}\right..$
(18)

In practical industrial applications, plant operators must manually adjust the
air-to-feed ratio such that the concentration ratio of $\text{H}_{2}\text{S}$
and $\text{SO}_{2}$ is roughly 2:1, where online analyzers are inseparable.
However, it’s simple for acid gas to harm the instrument itself, hence soft
sensors should be considered.

TABLE I: Process variables and quality variables for SRU

Variable | Description | Unit
---|---|---
$x_{1}$ | Flow of MEA gas | $\text{m}^{3}\text{/s}$
$x_{2}$ | Flow of air | $\text{m}^{3}\text{/s}$
$x_{3}$ | Flow of secondary air | $\text{m}^{3}\text{/s}$
$x_{4}$ | Flow of SWS gas | $\text{m}^{3}\text{/s}$
$x_{5}$ | Flow of air in SWS zone | $\text{m}^{3}\text{/s}$
$y_{1}$ | Concentration of $\text{H}_{2}\text{S}$ in tail gas | $\text{moles/}\text{m}^{3}$
$y_{2}$ | Concentration of $\text{SO}_{2}$ in tail gas | $\text{moles/}\text{m}^{3}$

Table. I displays the process variables and quality variables collected from
the SRU industrial history data. Taking into account the process dynamics and
time-delay characteristics, the actual input of the soft sensor model at time
$t$ is designed as follows:

$\boldsymbol{x}_{t}=\left[x_{1}(t),\ldots,x_{1}(t-9),\dots,x_{D}(t),\ldots,x_{D}(t-9)\right]^{\top}.$
(19)

where $D=5$, $x_{d}(t),d=1,2,\ldots,D$ are the input variables at time $t$,
$x_{d}(t-z),z=1,2,\dots,9$ are corresponding lagged values. After removing
null values, the dataset comprises a total of 10,000 data samples.

####  IV-A2 Baseline Models

In experiments, we compare the proposed model to conventional regression
prediction models such as partial least squares regression (PLSR), AE, variant
of AE like stacked autoencoder (SAE) and gated stacked target-related
autoencoder (GSTAE) [36], long short-term memory network (LSTM), variable
attention LSTM (VA-LSTM) network [37] and supervised LSTM (SLSTM) network
[38]. In addition, we compared SOTA MTL models such as MMoE [30], PLE [32],
and BMoE [33]. The appendix provides descriptions of baseline models we used.
Since certain regression prediction models can only predict univariate, we
undertake a fair assessment of quality variables using univariate modeling.
And for regression prediction models capable of multivariate prediction, we
substitute their multivariate prediction heads with objective-specific tower
networks and transform them into MOO models for comparison.

####  IV-A3 Training & Evaluation

All quality variable prediction models in experiments are trained and
evaluated through MSE loss. The SRU dataset is split into training, validation
and test set at a ratio of 6:2:2. In all MOO models and single-objective
models, we use RELU-activated MLP networks for each expert or feature
extraction layer. The proposed model necessitates the adjustment of several
hyperparameters: the number of feature extraction blocks, objective-specific
experts, and objective-shared experts. The evaluation metrics employed RMSE,
MAE, and $\text{R}^{2}$, and all experiments were carried out on Ubuntu
18.04.2 LTS based on Python 3.7.

###  IV-B Performance Evaluation

TABLE II: Experimental results on the SRU dataset

Models | $\text{H}_{2}\text{S}$ | $\text{SO}_{2}$
---|---|---
RMSE | MAE | $\text{R}^{2}$ | RMSE | MAE | $\text{R}^{2}$
ST | PLSR | 0.0323$\pm$0.0002 | 0.0187$\pm$0.0002 | 0.6299$\pm$0.0100 | 0.0271$\pm$0.0003 | 0.0191$\pm$0.0001 | 0.7730$\pm$0.0010
AE | 0.0264$\pm$0.0037 | 0.0171$\pm$0.0017 | 0.6950$\pm$0.0936 | 0.0243$\pm$0.0023 | 0.0176$\pm$0.0009 | 0.7833$\pm$0.0456
SAE | 0.0198$\pm$0.0010 | 0.0133$\pm$0.0003 | 0.8008$\pm$0.0228 | 0.0240$\pm$0.0012 | 0.0171$\pm$0.0005 | 0.7916$\pm$0.0131
GSTAE | 0.0226$\pm$0.0019 | 0.0142$\pm$0.0008 | 0.7731$\pm$0.0129 | 0.0247$\pm$0.0012 | 0.0177$\pm$0.0006 | 0.7767$\pm$0.0179
LSTM | 0.0221$\pm$0.0024 | 0.0146$\pm$0.0009 | 0.7719$\pm$0.0127 | 0.0228$\pm$0.0005 | 0.0170$\pm$0.0003 | 0.8045$\pm$0.0178
VA-LSTM | 0.0202$\pm$0.0032 | 0.0131$\pm$0.0009 | 0.8183$\pm$0.0194 | 0.0225$\pm$0.0008 | 0.0170$\pm$0.0004 | 0.8180$\pm$0.0080
SLSTM | 0.0215$\pm$0.0037 | 0.0136$\pm$0.0061 | 0.8124$\pm$0.0674 | 0.0202$\pm$0.0038 | 0.0148$\pm$0.0047 | 0.8568$\pm$0.0488
MT | GSTAE | 0.0226$\pm$0.0020 | 0.0141$\pm$0.0008 | 0.8319$\pm$0.0230 | 0.0249$\pm$0.0012 | 0.0178$\pm$0.0005 | 0.8065$\pm$0.0172
VA-LSTM | 0.0205$\pm$0.0023 | 0.0133$\pm$0.0007 | 0.8351$\pm$0.0140 | 0.0238$\pm$0.0008 | 0.0175$\pm$0.0004 | 0.8091$\pm$0.0098
SLSTM | 0.0245$\pm$0.0073 | 0.0193$\pm$0.0074 | 0.8051$\pm$0.1166 | 0.0247$\pm$0.0051 | 0.0164$\pm$0.0066 | 0.8024$\pm$0.0852
MMoE | 0.0225$\pm$0.0018 | 0.0147$\pm$0.0008 | 0.8444$\pm$0.0163 | 0.0235$\pm$0.0018 | 0.0173$\pm$0.0004 | 0.8292$\pm$0.0267
BMoE | 0.0207$\pm$0.0013 | 0.0149$\pm$0.0007 | 0.8665$\pm$0.0163 | 0.0222$\pm$0.0006 | 0.0176$\pm$0.0006 | 0.8534$\pm$0.0079
PLE | 0.0192$\pm$0.0010 | 0.0131$\pm$0.0005 | 0.8764$\pm$0.0240 | 0.0225$\pm$0.0012 | 0.0167$\pm$0.0012 | 0.8371$\pm$0.0236
Ours |  0.0177$\pm$0.0004* |  0.0122$\pm$0.0008* | 0.8845$\pm$0.0206 | 0.0207$\pm$0.0023 | 0.0153$\pm$0.0017 | 0.8626$\pm$0.0243

The baseline approaches and the proposed TMoE-P model’s prediction performance
evaluation metrics for the SRU dataset are provided in Table. II, together
with the mean and variation. And Fig. 6 depicts the prediction curve, the
accompanying residual histogram, and the kernel density estimation curve.

Figure 6: Prediction curve, accompanying residual histogram, and the kernel
density estimation curve of TMoE-P. (a) and (c) for $\text{H}_{2}\text{S}$,
(b) and (d) for $\text{SO}_{2}$.

The outcomes demonstrate that the proposed TMoE-P model greatly outperforms
the baseline models in the concentration prediction of both
$\text{H}_{2}\text{S}$ and $\text{SO}_{2}$. As seen from SRU’s reaction
process (18), there is a inverse and complex relationship between between the
concentration of $\text{H}_{2}\text{S}$ and $\text{SO}_{2}$, leading to the
occurence of NT phenomenon. Because the dependencies between objectives are
challenging to describe when multi-objective joint modeling is required for a
single-objective regression prediction model, this is notably evident in the
SLSTM. In single-objective modeling of $\text{SO}_{2}$, the SLSTM model scored
the greatest metric among most of the baseline approaches; however, when
employed in multi-objective modeling, the $\text{R}^{2}$ of $\text{SO}_{2}$
concentration prediction declined by 0.054.

Furthermore, our experimental results reveal that in the multi-objective
scenario of $\text{H}_{2}\text{S}$ and $\text{SO}_{2}$ concentration
prediction, the majority of the multi-objective models suffer from seesaw
phenomenon. Specifically, several models decrease the accuracy of
$\text{SO}_{2}$ while improving the accuracy of $\text{H}_{2}\text{S}$. Take
the VA-LSTM model for example, the $\text{R}^{2}$ of $\text{H}_{2}\text{S}$
concentration prediction increased by 0.02 while the $\text{R}^{2}$ of
$\text{SO}_{2}$ concentration prediction decreased by about 0.01. On this
premise, the MMoE and BMoE models utilize the mixture-of-experts structure to
balance multiple objectives, and significantly improve $\text{H}_{2}\text{S}$
with a slight decrease in $\text{SO}_{2}$ performance. The PLE model
incorporates objective-specific experts, which improves the accuracy of
$\text{H}_{2}\text{S}$ concentration prediction while exacerbating the seesaw
phenomenon. The proposed model TMoE-P employs objective-aware mixture-of-
experts, uses the POR module to dynamically modify target weights, and can
effectively extract correlation information between objectives, outperforming
or even outperforming single modeling of each objective.

The kernel density estimation curve in Fig. 6 ensures that the residuals are
Gaussian white noise with zero mean and variance concentration, which verifies
that TMoE-P is an unbiased estimation model in statistical significance

Figure 7: Performance of dynamic weights and constant equal weights during
model training. (a),(c) and (e) are the metrics of $\text{H}_{2}\text{S}$
concentration; (b),(d) and (f) are the metrics of $\text{SO}_{2}$
concentration. Figure 8: Dynamic target weights and corresponding losses
during training process of TMoE-P. Figure 9: Ablation study of POR module
under different $n_{e}$. (a) for $\text{H}_{2}\text{S}$, and (b) for
$\text{SO}_{2}$.

###  IV-C Ablation Studies

####  IV-C1 On the POR Module

The introduction of the POR module to dynamically modify the target weights in
the updating of the OMoE network parameters is one of the proposed TMoE-P
model’s primary advances. This subsection will verify the effectiveness of
dynamic weights in the POR module by constrasting the OMoE model trained with
constant target weights on the same parameters.

The trend chart of metrics in the $\text{H}_{2}\text{S}$ and $\text{SO}_{2}$
concentration prediction objectives of the TMoE-P network trained with
constant target weights and dynamic weights is shown in Fig. 7. It is obvious
that the model trained with the POR module’s dynamic weights outperforms the
model trained with constant weights in terms of steady-state metrics values
after convergence. In particular, the $\text{R}^{2}$ of $\text{H}_{2}\text{S}$
has risen from 0.8764 to 0.8845 while the $\text{R}^{2}$ of $\text{SO}_{2}$
has risen from 0.8371 to 0.8626. Furthermore, dynamic weights outperform
single-objective modeling by 8.1% and 0.67% respectively, while eliminating
the seesaw phenomenon and NT phenomenon. To better understand the evolution
trend of dynamic weights, we extracted target weights during the TMoE-P
training process, as illustrated in Fig. 8. The figure shows that as the
objective-specific features and objective-shared features are progressively
separated, the target weight evolves from equilibrium to focus on
$\text{H}_{2}\text{S}$ concentration prediction, eventually settling at
roughly 0.7:0.3. The dynamic adjustment of target weights allows the TMoE-P to
overcome the imbalance of training resources produced by excessively
complicated objective correlations, bringing it closer to the Pareto optimal
model, demonstrating the effectiveness of the POR module in MOO.

We further verified the improvement of POR module to OMoE module under
different number of experts $n_{e}$, as shown in Fig. 9. It can be seen from
the step value in the histogram that, with the same number of experts, the
proposed TMoE-P has a higher $\text{R}^{2}$ than OMoE alone, and the variance
is generally lower, ensuring more stable and accurate predictions.

In addition, we applied the POR module to the traditional soft sensor models
VA-LSTM, SLSTM and the multi-task learning model MMoE, and the results are
shown in Fig. 10. It should be noted that the shared parameter of the first
two models is the subject feature extraction network, and the objective-
specific parameter is the prediction head. And it can be clearly seen that POR
module can produce effective accuracy improvement no matter it is in the
traditional soft sensor model or in the multi-task learning model.

Figure 10: Ablation study of POR module under different model. (a) for
$\text{H}_{2}\text{S}$, and (b) for $\text{SO}_{2}$.

####  IV-C2 On the Experts Module

In Table III, we remove objective-shared and objective-specific components
from the OMoE module to study the differences between target-alone and hard
parameter sharing modeling versus hybrid modeling. We find that hybrid
modeling, which combines objective-shared and objective-specific experts, is
the best MOO framework across all parameter groups. OMoE without objective-
shared experts fully ignores the correlation information between quality
variables, making it challenging to improve the multi-objective metrics by
increasing the number of experts and executing information fusion through the
gating network. And regrettably, the $\text{R}^{2}$ metric fell by 2% $\sim$
6% for predictions of two quality variables. The OMoE structure without
objective-specific experts, on the other hand, is analogous to the MMoE
structure, which cannot escape the seesaw phenomenon in MOO.

TABLE III: Ablation study of components in TMoE-P

$n_{k}$ | $n_{s}$ | $\text{H}_{2}\text{S}-\text{R}^{2}$ | $\text{SO}_{2}-\text{R}^{2}$
---|---|---|---
1 | 0 |  0.8764$\pm$0.0258 |  0.8379$\pm$0.0276
0 | 1 |  0.8646$\pm$0.0165 |  0.8462$\pm$0.0259
1 | 1 | 0.8845$\pm$0.0206 | 0.8626$\pm$0.0243
2 | 0 |  0.7580$\pm$0.0304 |  0.7833$\pm$0.0245
0 | 2 |  0.7526$\pm$0.0388 |  0.7841$\pm$0.0218
2 | 2 |  0.8009$\pm$0.0210 |  0.7914$\pm$0.0358
3 | 0 |  0.8256$\pm$0.0171 |  0.8143$\pm$0.0176
0 | 3 |  0.8119$\pm$0.0217 |  0.8123$\pm$0.0222
3 | 3 |  0.8306$\pm$0.0312 |  0.8222$\pm$0.0206
4 | 0 |  0.7622$\pm$0.0378 |  0.7854$\pm$0.0157
0 | 4 |  0.7954$\pm$0.0273 |  0.7896$\pm$0.0252
4 | 4 |  0.8045$\pm$0.0418 |  0.7884$\pm$0.0288

Figure 11: Parameter sensitivity analysis of the hyperparameters on metrics
RMSE and $\text{R}^{2}$. (a),(c) and (e) for RMSE metric; (b),(d) and (f) for
$\text{R}^{2}$ metric.

###  IV-D Parameter Sensitivity Studies

The main hyperparameters of the proposed TMoE-P model are the number of
objective-specific experts $n_{e}$, the number of feature extraction blocks
$n_{b}$, and the number of expert network layers $n_{l}$. The sensitivity
analysis of the TMoE-P prediction performance will be performed on the three
variable hyperparameters in this subsection, with the remaining parameters
fixed and one of the above hyperparameters changed through the control
variable approach.

Fig. 11 depicts the fluctuation diagram of the evaluation metrics RMSE and
$\text{R}^{2}$ when $n_{e}$, $n_{b}$ and $n_{l}$ vary between $[1,5]$, $[1,4]$
and $[1,5]$. According to the variations in the two metrics, it can be seen
that when $n_{e}=1$, the sum of TMoE-P’s metrics in the prediction of
$\text{H}_{2}\text{S}$ and $\text{SO}_{2}$ concentrations is the highest,
reaching 0.8845 and 0.8626, respectively. However, when $n_{e}$ grows, the
performance of TMoE-P falls dramatically. One plausible explanation is that
the gating network shown in Fig. 3 is difficult to integrate large-capacity
expert knowledge, and the TMoE-P with single expert is adequate to realize
joint prediction of $\text{H}_{2}\text{S}$ and $\text{SO}_{2}$ concentrations.

When it comes to the effect of $n_{b}$, TMoE-P performs best for $n_{b}=2$,
and as $n_{b}$ decreases or grows, TMoE-P performs worse and worse. Indeed,
when $n_{b}=1$, TMoE-P can only separate and fuse objective-specific and
objective-shared features via a objective-specific gating network, and when
$n_{b}$ is too large, the network structure becomes too complex for both to
avoid the seesaw phenomenon. Additionally, we discovered that when $n_{l}=3$,
the objective-specific expert structure of TMoE-P attained highest
performance, outperforming the $n_{l}=2$ and $n_{l}=4$ models in terms of
$\text{R}^{2}$, which improved by 0.7% and 4.0% for $\text{H}_{2}\text{S}$ and
1.0% and 2.8% for $\text{SO}_{2}$, respectively.

To summarize the sensitivity experiment results for the aforementioned
hyperparameters, we recommend that during the TMoE-P training process, first
set $n_{e}$ to 1 and then change the hyperparameters $n_{b}$ and $n_{l}$
within $[1,3]$ and $[2,4]$.

##  V Related Work

Solving industrial soft sensor problems based on the concept of MOO is a novel
and demanding task, as there are frequently contradicting but closely
connected linkages between quality variables. Broadly speaking, current work
in this field can be categorised in two groups. The first is to utilize MOO as
the model hyperparameter optimizer to find the best model structure. The
second instead generalize and solve the soft sensor problem using the
mathematical form of MOO.

He et al.[39] optimized the hyperparameters of the four data-driven models
including Random Forest, Gradient Boosting Regression, Ridge Regression, and
K-Nearest Neighbor using the intelligent evolutionary algorithm NSGA II, but
did not further integrate the models. Instead, Jin et al.[40] simply performed
an optimization search of similarity measures for Just-in-time learning models
and developed a stacked Just-in-time learning model with a two-layer structure
for online estimation of quality variables. While considering the
dimensionality of features and the weight of regularization, Ribeiro et
al.[41] proposed a two-stage MOO to improve the resilience of Partial Least
Squares Regression.

Yan et al.[42] suggested an end-to-end multi-quality variable correlation
model for joint optimization, with the prediction loss of quality variables
and the correlation loss of features serving as the optimization objectives in
MOO. However, the nonlinear feature transformation considered in this model is
merely Multilayer Perceptron, which is a standard hard parameter sharing
model. Huang et al.[33] characterized the soft sensor problem as a multi-task
regression problem, which is a special case of MOO, and used the MMoE
structure and GradNorm module to balance the task gradient to improve multi-
task performance. Futhermore, Lei et al.[43] extended the regression task with
generation and classification tasks, and employed the upgraded Capsule Network
and weighted objective loss for stochastic gradient multi-task optimization.

Our approach is most similar to Huang, in that we both use gradients to create
target weights to optimize network parameters. The distinction is that we
concentrate primarily on three aspects: we begin by solving the MOP. Compared
with multi-task learning, MOO focuses on approximating the Pareto optimal
solution, and the network update mechanism is more challenging. Second, the
OMoE and MMoE structures use different soft parameter sharing approaches,
adding objective-specific experts, which can mitigate the seesaw phenomenon
produced by full sharing of the underlying parameters when objective
correlation is weak. Finally, our POR optimization algorithm approaches the
Pareto optimal model parameters, whereas GradNorm lacks theoretical backing
and relies on intuition to discover target weights that balance multi-task
learning speed.

##  VI Conclusion

In this paper, we propose a Task-aware Mixture-of-Experts model combining the
OMoE and the POR module capable of approaching Pareto optimality (TMoE-P) for
the field of industrial soft sensor. The proposed TMoE-P uses the OMoE network
to explicitly separate objective-specific parameters from objective-shared
parameters in order to achieve soft sensor accuracy that exceeds individual
modeling under shared mode. Simultaneously, we employ the POR module, which
can approach the Pareto optimality, to dynamically modify the proportion of
soft sensor objectives during the training of the OMoE network, avoiding the
phenomenon of negative transfer and seesaw. The experimental results on the
SRU chemical industry data set reveal that TMoE-P outperforms the SOTA MTL
model and the convensional regression prediction model. Our future research
will mainly focus on investigating the fusion mechanism of the gating network
with a large number of experts, MTL modeling for time-series data in the soft
sensor field, and the Pareto stationary point solving algorithm under the MOP
optimization framework.

[Baseline Description] The baseline model we used in the verification
experiments of TMoE are as follows:

  * •

PLSR finds a linear regression model by projecting quality variables and
process variables into a new space;

  * •

AE performs representation learning on the input data to obtain dimensionality
reduction features, and predicts quality variables after combining with a
fully connected neural network;

  * •

SAE increases the depth of the hidden layer based on AE and improves feature
extraction ability through layer-wise unsupervised pre-training;

  * •

GSTAE [36] adds the prediction error of the quality variables to the loss
function during the layer-wise unsupervised pre-training, guiding the feature
learning process with target-related information, and uses gated neurons to
learn the information flow of the final output neurons;

  * •

LSTM is a recurrent neural network capable of processing quality variable data
arranged in sequences and capturing temporal dependencies between sequences;

  * •

VA-LSTM [37] consists of two LSTM networks, one considers the correlation of
process variables with quality variables and assigns attention weights to
process variables based on the correlations, and the other captures the long-
term dependencies of weighted input to predict quality variables;

  * •

SLSTM [38] consists of two LSTM networks, one considers the correlation of
process variables with quality variables and assigns attention weights to
process variables based on the correlations, and the other captures the long-
term dependencies of weighted input to predict quality variables;

  * •

MMoE [30] is multi-gate mixture-of-experts, which characterizes task
correlation and learns specific tasks based on shared representations;

  * •

BMoE [33] leverages GradNorm on the basis of MMoE to balance the learning
gradient between tasks;

  * •

PLE [32] explicitly distinguishes between task-specific and task-shared
components in the model, avoiding the seesaw phenomenon caused by loose task
correlation;

## Proof of Convergence Theorem

The proof of the convergence of the primal error depends on the following
Lemma .1 on the improvement in each iteration.

######  Lemma .1.

For a step
$\boldsymbol{w}^{(r+1)}=\boldsymbol{w}^{(r)}+\gamma(\boldsymbol{s}-\boldsymbol{w}^{(r)})$
with arbitrary step-size $\gamma\in[0,1]$, if $\boldsymbol{s}$ is a
appropriate descent direction on the linear approximation to $L$, it holds
that

$L\left(\boldsymbol{w}^{(r+1)}\right)\leq
L\left(\boldsymbol{w}^{(r)}\right)-\gamma\phi\left(\boldsymbol{w}^{(r)}\right)+\frac{\gamma^{2}}{2}C_{f}(1+\delta).$
(20)

###### Proof.

To simplify the notation, we write $\boldsymbol{w}=\boldsymbol{w}^{(r)}$,
$\boldsymbol{w}^{\prime}=\boldsymbol{w}^{(r+1)}$, and $d_{w}=\nabla
L\left(\boldsymbol{w}^{(r)}\right)$. From the definition of the curvature
constant $C_{f}$, we have

$\displaystyle L(\boldsymbol{w}^{\prime})$
$\displaystyle=L(\boldsymbol{w}+\gamma(\boldsymbol{s}-\boldsymbol{w}))$ (21)
$\displaystyle\leq
L(\boldsymbol{w})+\gamma\left\langle\boldsymbol{s}-\boldsymbol{w},d_{w}\right\rangle+\frac{\gamma^{2}}{2}C_{f}.$

Then we pick $\boldsymbol{s}$ as an linear minimizer for $L$, and it satisfies
$\left\langle\boldsymbol{s},\nabla
L\left(\boldsymbol{w}^{(r)}\right)\right\rangle\leq\min\left\langle\hat{\boldsymbol{s}},\nabla
L\left(\boldsymbol{w}^{(r)}\right)\right\rangle+\frac{1}{2}\delta\gamma
C_{f}$, which can be rewritten as

$\displaystyle\left\langle\boldsymbol{s}-\boldsymbol{w},d_{w}\right\rangle$
$\displaystyle\leq\min\left\langle\boldsymbol{w}^{\prime},d_{w}\right\rangle-\left\langle\boldsymbol{w},d_{w}\right\rangle+\frac{1}{2}\delta\gamma
C_{f}$ (22) $\displaystyle=-\phi(\boldsymbol{w})+\frac{1}{2}\delta\gamma
C_{f}.$

Therefore, we obtain $L(\boldsymbol{w}^{\prime})\leq
L(\boldsymbol{w})-\gamma\phi(\boldsymbol{w})+\frac{\gamma^{2}}{2}C_{f}(1+\delta)$,
which proves the lemma. ∎

######  Theorem .1.

For each $r\geq 1$, the iterates $\boldsymbol{w}^{(r)}$ of the Frank-Wolfe
algorithm satisfy

$L(\boldsymbol{w}^{(r)})-L(\boldsymbol{w}^{*})\leq\frac{2C_{f}}{r+2}(1+\delta).$
(23)

where $\delta\geq 0$ is the accuracy to which the linear subproblems (9) are
solved, and $C_{f}$ is the curvature constant of loss function $L$ which is
defined as

$C_{f}=\sup_{\gamma\in[0,1]}\frac{2}{\gamma^{2}}(L(\boldsymbol{w}^{\prime})-L(\boldsymbol{w})-\langle\boldsymbol{w}^{\prime}-\boldsymbol{w},\nabla
L(\boldsymbol{w})\rangle).$ (24)

where
$\boldsymbol{w}^{\prime}=\boldsymbol{w}+\gamma(\boldsymbol{s}-\boldsymbol{w})$
and $\boldsymbol{s}$ is a feasible solution.

###### Proof.

From Lemma .1 we know that $L\left(\boldsymbol{w}^{(r+1)}\right)\leq
L\left(\boldsymbol{w}^{(r)}\right)-\gamma\phi\left(\boldsymbol{w}^{(r)}\right)+\gamma^{2}C$
holds for every step of Frank-Wolfe algorithm where we define
$C=\frac{C_{f}}{2}(1+\delta)$ and take fixed step $\gamma=\frac{2}{r+2}$.

Writing the primal error as
$\zeta(\boldsymbol{w})=L(\boldsymbol{w})-L(\boldsymbol{w}^{*})$ at any point
$\boldsymbol{w}$, this implies that

$\displaystyle\zeta\left(\boldsymbol{w}^{(r+1)}\right)$
$\displaystyle\leq\zeta\left(\boldsymbol{w}^{(r)}\right)-\gamma\phi\left(\boldsymbol{w}^{(r)}\right)+\gamma^{2}C$
(25)
$\displaystyle\leq\zeta\left(\boldsymbol{w}^{(r)}\right)-\gamma\zeta\left(\boldsymbol{w}^{(r)}\right)+\gamma^{2}C$
$\displaystyle=(1-\gamma)\zeta\left(\boldsymbol{w}^{(r)}\right)+\gamma^{2}C.$

where we have used weak duality
$\zeta(\boldsymbol{w})\leq\phi(\boldsymbol{w})$. We will now use induction
over $r$ to prove bound claimed as follows

$\zeta\left(\boldsymbol{w}^{(r+1)}\right)\leq\frac{4C}{r+3}\quad r=0,1,\ldots$
(26)

The base-case $r=0$ follows from (25) applied for the first step through
$\gamma=\gamma^{(0)}=\frac{2}{0+2}=1$. Now considering $r\geq 1$,

$\displaystyle\zeta\left(\boldsymbol{w}^{(r+1)}\right)$
$\displaystyle\leq(1-\gamma^{(r)})\zeta\left(\boldsymbol{w}^{(r)}\right)+{\gamma^{(r)}}^{2}C$
(27)
$\displaystyle=(1-\frac{2}{r+2})\zeta\left(\boldsymbol{w}^{(r)}\right)+{\frac{2}{r+2}}^{2}C$
$\displaystyle\leq(1-\frac{2}{r+2})\frac{4C}{r+2}+{\frac{2}{r+2}}^{2}C.$

Simply rearranging the terms gives the bound claimed above

$\displaystyle\zeta\left(\boldsymbol{w}^{(r+1)}\right)$
$\displaystyle\leq\frac{4C}{r+2}(1-\frac{1}{r+2})=\frac{4C}{r+2}\frac{r+1}{r+2}$
(28) $\displaystyle\leq\frac{4C}{r+2}\frac{r+2}{r+3}=\frac{4C}{r+3}.$

∎

######  Theorem .2.

If the Frank-Wolfe algorithm is run for $R\geq 2$ iterations, then the
algorithm has a bounded duality gap with iterate $\boldsymbol{w}^{(r)}$,
$1\leq r\leq R$

$\phi(\boldsymbol{w}^{(r)})\leq\frac{2\beta C_{f}}{R+2}(1+\delta).$ (29)

where $\beta=\frac{27}{8}$.

###### Proof.

We do not prove that the above duality gap holds for the entire iteration, but
for the last third of the $R$ iteration. To simplify notation, we denote the
primal and dual errors as $\zeta^{(r)}=\zeta\left(\boldsymbol{w}^{(r)}\right)$
and $\phi^{(r)}=\phi\left(\boldsymbol{w}^{(r)}\right)$ for $r\geq 0$.

By primal convergence Theorem .1, we already know that the primal error
satisfies $\zeta^{(r)}\leq\frac{C}{r+2}$, where we reset $C=2C_{f}(1+\delta)$.

We firstly make following contradict assumption: $\phi^{(r)}$ always stays
larger than $\frac{\beta C}{U}$ in the last third of the $R$ iterations, which
can be formally defined as

$\phi^{(r)}>\frac{\beta C}{U}\quad\text{ for }\quad r\in\\{\lceil\mu
U\rceil-2,\ldots,R\\}$ (30)

where $U=R+2$ for simple notation, and $0<\mu<1$ is an arbitrary fixed
parameter. We will find later that $\mu=\frac{2}{3}$ is a good choice.

Lemma .1 can be further read as follows if we choose $\gamma=\frac{2}{r+2}$

$\displaystyle\zeta^{(r+1)}$
$\displaystyle\leq\zeta^{(r)}-\frac{2}{r+2}\phi^{(r)}+\frac{2}{(r+2)^{2}}C_{f}(1+\delta)$
(31) $\displaystyle=\zeta^{(r)}-\frac{2}{r+2}\phi^{(r)}+\frac{C}{(r+2)^{2}}.$

Considering the assumptions we made in (30), we obtain

$\zeta^{(r+1)}<\zeta^{(r)}-\frac{2}{r+2}\frac{\beta
C}{U}+\frac{C}{(r+2)^{2}}.$ (32)

If we define $r_{\text{min}}=\lceil\mu U\rceil-2$, then $r_{\text{min}}\geq 0$
for $R\geq\frac{2(1-\mu)}{\mu}$. While the steps $r$ satisfy
$r_{\text{min}}\leq r\leq R$, then $\mu U\leq r+2\leq U$, the inequality in 32
now reads as

$\displaystyle\zeta^{(r+1)}$ $\displaystyle<\zeta^{(r)}-\frac{2}{U}\frac{\beta
C}{U}+\frac{C}{(\mu U)^{2}}$ (33) $\displaystyle=\zeta^{(r)}-\frac{2\beta
C-C/\mu^{2}}{U^{2}}.$

We now sum up this inequality over the last third of the steps from
$r_{\text{min}}$ up to $R$, then we get

$\displaystyle\begin{split}\zeta^{(R+1)}&<\zeta^{(r_{\text{min}})}-(R-r_{\text{min}}+1)\frac{2\beta
C-C/\mu^{2}}{U^{2}}\\\ &\leq\frac{C}{\mu
U}-\tau\frac{2\mu\beta-1/\mu}{U}\frac{C}{\mu U}\\\ &=\frac{C}{\mu
U}\left(1-\tau\frac{2\mu\beta-1/\mu}{U}\right).\end{split}$ (34)

where $\tau=(1-\mu)U\leq R+2-(\lceil\mu U\rceil-1)=R-r_{\text{min}}+1$, and in
the last inequality we have used Theorem .1 giving
$\zeta^{(r_{\text{min}})}\leq\frac{C}{r_{\text{min}}+2}\leq\frac{C}{\mu U}$.
For $\mu=\frac{2}{3}$ and $\beta=\frac{27}{8}$, the following term become
zero: $1-\tau\frac{2\mu\beta-1/\mu}{U}=1-(1-\mu)(2\mu\beta-1/\mu)=0$. Then we
arrive at the contradiction that $\zeta^{(R+1)}<0$ and our assumption on the
gap is refuted, so the claimed bound has been proven. ∎

## References

  * [1] E. Camponogara, A. Plucenio, A. F. Teixeira, and S. R. Campos, “An automation system for gas-lifted oil wells: Model identification, control, and optimization,” _Journal of petroleum science and engineering_ , vol. 70, no. 3-4, pp. 157–167, 2010.
  * [2] H. J. Galicia, Q. P. He, and J. Wang, “A reduced order soft sensor approach and its application to a continuous digester,” _Journal of Process Control_ , vol. 21, no. 4, pp. 489–500, 2011.
  * [3] H. Wang, T. Chang, T. Liu, J. Huang, Z. Chen, C. Yu, R. Li, and W. Chu, “ESCM2: entire space counterfactual multi-task model for post-click conversion rate estimation,” in _SIGIR_. ACM, 2022, pp. 363–372.
  * [4] J. Fan, Y. Zhuang, Y. Liu, H. Jianye, B. Wang, J. Zhu, H. Wang, and S.-T. Xia, “Learnable behavior control: Breaking atari human world records via sample-efficient behavior selection,” in _ICLR_.
  * [5] L. McElroy, J. Bao, C. T. Jayasundara, R. Yang, and A. Yu, “A soft-sensor approach to impact intensity prediction in stirred mills guided by dem models,” _Powder technology_ , vol. 219, pp. 151–157, 2012.
  * [6] X. Yuan, Z. Ge, Z. Song, Y. Wang, C. Yang, and H. Zhang, “Soft sensor modeling of nonlinear industrial processes based on weighted probabilistic projection regression,” _IEEE Transactions on Instrumentation and Measurement_ , vol. 66, no. 4, pp. 837–845, 2017.
  * [7] J. Zheng and Z. Song, “Semisupervised learning for probabilistic partial least squares regression model and soft sensor application,” _Journal of process control_ , vol. 64, pp. 123–131, 2018.
  * [8] Y. Liu, T. Chen, and J. Chen, “Auto-switch gaussian process regression-based probabilistic soft sensors for industrial multigrade processes with transitions,” _Industrial & Engineering Chemistry Research_, vol. 54, no. 18, pp. 5037–5047, 2015.
  * [9] S. Herceg, Ž. U. Andrijić, and N. Bolf, “Development of soft sensors for isomerization process based on support vector machine regression and dynamic polynomial models,” _Chemical Engineering Research and Design_ , vol. 149, pp. 95–103, 2019.
  * [10] Q. Li, Q. Du, W. Ba, and C. Shao, “Multiple-input multiple-output soft sensors based on kpca and mkls-svm for quality prediction in atmospheric distillation column,” _International Journal of Innovative Computing, Information and Control_ , vol. 8, no. 12, pp. 8215–8230, 2012.
  * [11] X. Zeng and G. Wu, “Data-driven soft sensor modeling for dct clutch pressure based on decision tree and stepwise regression,” in _2020 7th International Conference on Information Science and Control Engineering (ICISCE)_. IEEE, 2020, pp. 436–442.
  * [12] J. Ji, H. Wang, K. Chen, Y. Liu, N. Zhang, and J. Yan, “Recursive weighted kernel regression for semi-supervised soft-sensing modeling of fed-batch processes,” _Journal of the Taiwan Institute of Chemical Engineers_ , vol. 43, no. 1, pp. 67–76, 2012.
  * [13] B. Shen, L. Yao, and Z. Ge, “Nonlinear probabilistic latent variable regression models for soft sensor application: From shallow to deep structure,” _Control Engineering Practice_ , vol. 94, p. 104198, 2020.
  * [14] X. Yuan, B. Huang, Y. Wang, C. Yang, and W. Gui, “Deep learning-based feature representation and its application for soft sensor modeling with variable-wise weighted sae,” _IEEE Transactions on Industrial Informatics_ , vol. 14, no. 7, pp. 3235–3243, 2018.
  * [15] X. Wang and H. Liu, “Data supplement for a soft sensor using a new generative model based on a variational autoencoder and wasserstein gan,” _Journal of Process Control_ , vol. 85, pp. 91–99, 2020.
  * [16] L. Z. Chen, S. K. Nguang, X. M. Li, and X. D. Chen, “Soft sensors for on-line biomass measurements,” _Bioprocess and Biosystems Engineering_ , vol. 26, pp. 191–195, 2004.
  * [17] G. Kataria and K. Singh, “Recurrent neural network based soft sensor for monitoring and controlling a reactive distillation column,” _Chemical Product and Process Modeling_ , vol. 13, no. 3, 2018.
  * [18] J. Loy-Benitez, S. Heo, and C. Yoo, “Soft sensor validation for monitoring and resilient control of sequential subway indoor air quality through memory-gated recurrent neural networks-based autoencoders,” _Control Engineering Practice_ , vol. 97, p. 104330, 2020.
  * [19] K. Wang, C. Shang, L. Liu, Y. Jiang, D. Huang, and F. Yang, “Dynamic soft sensor development based on convolutional neural networks,” _Industrial & Engineering Chemistry Research_, vol. 58, no. 26, pp. 11 521–11 531, 2019\.
  * [20] W. Zhu, Y. Ma, Y. Zhou, M. Benton, and J. Romagnoli, “Deep learning based soft sensor and its application on a pyrolysis reactor for compositions predictions of gas phase components,” in _Computer Aided Chemical Engineering_. Elsevier, 2018, vol. 44, pp. 2245–2250.
  * [21] Y. Huang, C. Zhang, J. Yella, S. Petrov, X. Qian, Y. Tang, X. Zhu, and S. Bom, “Grassnet: Graph soft sensing neural networks,” in _2021 IEEE International Conference on Big Data (Big Data)_. IEEE, 2021, pp. 746–756.
  * [22] M. Jia, Y. Dai, D. Xu, T. Yang, Y. Yao, and Y. Liu, “Deep graph network for process soft sensor development,” in _2021 8th International Conference on Information, Cybernetics, and Computational Social Systems (ICCSS)_. IEEE, 2021, pp. 1–6.
  * [23] R. Guo, H. Liu, G. Xie, Y. Zhang, and D. Liu, “A self-interpretable soft sensor based on deep learning and multiple attention mechanism: From data selection to sensor modeling,” _IEEE Transactions on Industrial Informatics_ , 2022.
  * [24] Z. Geng, Z. Chen, Q. Meng, and Y. Han, “Novel transformer based on gated convolutional neural network for dynamic soft sensor modeling of industrial processes,” _IEEE Transactions on Industrial Informatics_ , vol. 18, no. 3, pp. 1521–1529, 2021.
  * [25] I. Kokkinos, “Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 6129–6138.
  * [26] M. Long, Z. Cao, J. Wang, and P. S. Yu, “Learning multiple tasks with multilinear relationship networks,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [27] F. J. Bragman, R. Tanno, S. Ourselin, D. C. Alexander, and J. Cardoso, “Stochastic filter groups for multi-task cnns: Learning specialist and generalist convolution kernels,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 1385–1394.
  * [28] I. Misra, A. Shrivastava, A. Gupta, and M. Hebert, “Cross-stitch networks for multi-task learning,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 3994–4003.
  * [29] S. Ruder, J. Bingel, I. Augenstein, and A. Søgaard, “Sluice networks: Learning what to share between loosely related tasks,” _arXiv preprint arXiv:1705.08142_ , vol. 2, 2017.
  * [30] J. Ma, Z. Zhao, X. Yi, J. Chen, L. Hong, and E. H. Chi, “Modeling task relationships in multi-task learning with multi-gate mixture-of-experts,” in _Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining_, 2018, pp. 1930–1939.
  * [31] O. Sener and V. Koltun, “Multi-task learning as multi-objective optimization,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [32] H. Tang, J. Liu, M. Zhao, and X. Gong, “Progressive layered extraction (ple): A novel multi-task learning (mtl) model for personalized recommendations,” in _Proceedings of the 14th ACM Conference on Recommender Systems_ , 2020, pp. 269–278.
  * [33] Y. Huang, H. Wang, Z. Liu, L. Pan, H. Li, and X. Liu, “Modeling task relationships in multi-variate soft sensor with balanced mixture-of-experts,” _IEEE Transactions on Industrial Informatics_ , 2022\.
  * [34] M. Frank and P. Wolfe, “An algorithm for quadratic programming,” _Naval research logistics quarterly_ , vol. 3, no. 1-2, pp. 95–110, 1956.
  * [35] L. Fortuna, A. Rizzo, M. Sinatra, and M. Xibilia, “Soft analyzers for a sulfur recovery unit,” _Control Engineering Practice_ , vol. 11, no. 12, pp. 1491–1500, 2003.
  * [36] Q. Sun and Z. Ge, “Gated stacked target-related autoencoder: A novel deep feature extraction and layerwise ensemble method for industrial soft sensor application,” _IEEE Transactions on Cybernetics_ , vol. 52, no. 5, pp. 3457–3468, 2020.
  * [37] X. Yuan, L. Li, Y. Wang, C. Yang, and W. Gui, “Deep learning for quality prediction of nonlinear dynamic processes with variable attention-based long short-term memory network,” _The Canadian Journal of Chemical Engineering_ , vol. 98, no. 6, pp. 1377–1389, 2020.
  * [38] X. Yuan, L. Li, and Y. Wang, “Nonlinear dynamic soft sensor modeling with supervised long short-term memory network,” _IEEE transactions on industrial informatics_ , vol. 16, no. 5, pp. 3168–3176, 2019.
  * [39] Z. He, J. Qian, J. Li, M. Hong, and Y. Man, “Data-driven soft sensors of papermaking process and its application to cleaner production with multi-objective optimization,” _Journal of Cleaner Production_ , vol. 372, p. 133803, 2022.
  * [40] H. Jin, B. Pan, X. Chen, and B. Qian, “Ensemble just-in-time learning framework through evolutionary multi-objective optimization for soft sensor development of nonlinear industrial processes,” _Chemometrics and Intelligent Laboratory Systems_ , vol. 184, pp. 153–166, 2019.
  * [41] V. H. A. Ribeiro and G. Reynoso-Meza, “Feature selection and regularization of interpretable soft sensors using evolutionary multi-objective optimization design procedures,” _Chemometrics and Intelligent Laboratory Systems_ , vol. 212, p. 104278, 2021.
  * [42] S. Yan and X. Yan, “Joint monitoring of multiple quality-related indicators in nonlinear processes based on multi-task learning,” _Measurement_ , vol. 165, p. 108158, 2020.
  * [43] L. OuYang, N. Jin, L. Bai, and W. Ren, “Soft measurement of oil–water two-phase flow using a multi-task sequence-based capsnet,” _ISA transactions_ , 2022.