* Li et al. (2020) Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li.  A survey on deep learning for named entity recognition.  _IEEE Trans. Knowl. Data Eng._ , 34(1):50–70, 2020.
  * Li et al. (2022) Jingzheng Li, Hailong Sun, and Jiyi Li.  Beyond confusion matrix: learning from multiple annotators with awareness of instance features.  _Machine Learn._ , pp. 1–23, 2022.
  * Long et al. (2016) Chengjiang Long, Gang Hua, and Ashish Kapoor.  A Joint Gaussian Process Model for Active Visual Recognition with Expertise Estimation in Crowdsourcing.  _Int. J. Comput. Vis._ , 116(2):136–160, 2016\.
  * Loshchilov & Hutter (2017) Ilya Loshchilov and Frank Hutter.  SGDR: Stochastic Gradient Descent with Warm Restarts.  In _Int. Conf. Learn. Represent._ , Toulon, France, 2017.
  * Loshchilov & Hutter (2019) Ilya Loshchilov and Frank Hutter.  Decoupled Weight Decay Regularization.  In _Int. Conf. Learn. Represent._ , New Orleans, LA, 2019.
  * Minaee et al. (2021) Shervin Minaee, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos.  Image segmentation using deep learning: A survey.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 44(7):3523–3542, 2021.
  * Naeini et al. (2015) Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht.  Obtaining Well Calibrated Probabilities Using Bayesian Binning.  In _AAAI Conf. Artif. Intell._ , Austin, TX, 2015.
  * Netzer et al. (2011) Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng.  Reading Digits in Natural Images with Unsupervised Feature Learning,.  In _Deep Learn. Unsupervised Feature Learn. Workshop @ Adv. Neural Inf. Process. Syst._ , Granada, Spain, 2011.
  * Ovadia et al. (2019) Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek.  Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift.  In _Adv. Neural Inf. Process. Syst._ , Vancouver, BC, 2019.
  * Piech et al. (2015) Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J. Guibas, and Jascha Sohl-Dickstein.  Deep Knowledge Tracing.  In _Adv. Neural Inf. Process. Syst._ , Montreal, QC, 2015.
  * Platanios et al. (2020) Emmanouil A. Platanios, Maruan Al-Shedivat, Eric Xing, and Tom Mitchell.  Learning from Imperfect Annotations.  _arXiv:2004.03473_ , 2020.
  * Pouyanfar et al. (2018) Samira Pouyanfar, Saad Sadiq, Yilin Yan, Haiman Tian, Yudong Tao, Maria Presa Reyes, Mei-Ling Shyu, Shu-Ching Chen, and Sundaraja S Iyengar.  A Survey on Deep Learning: Algorithms, Techniques, and Applications.  _ACM Comput. Surv._ , 51(5):1–36, 2018.
  * Qu et al. (2016) Yanru Qu, Han Cai, Kan Ren, Weinan Zhang, Yong Yu, Ying Wen, and Jun Wang.  Product-based Neural Networks for User Response Prediction.  In _IEEE Int. Conf. Data Min._ , pp. 1149–1154, Barcelona, Spain, 2016.
  * Raykar et al. (2010) Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy.  Learning from Crowds.  _J. Mach. Learn. Res._ , 11(4):1297–1322, 2010\.
  * Reimers & Gurevych (2017) Nils Reimers and Iryna Gurevych.  Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks.  _arXiv:1707.06799_ , 2017.
  * Rodrigues & Pereira (2018) Filipe Rodrigues and Francisco Pereira.  Deep Learning from Crowds.  In _AAAI Conf. Artif. Intell._ , pp. 1611–1618, New Orleans, LA, 2018.
  * Rodrigues et al. (2013) Filipe Rodrigues, Francisco Pereira, and Bernardete Ribeiro.  Learning from multiple annotators: Distinguishing good from random labelers.  _Pattern Recognit. Lett._ , 34(12):1428–1436, 2013.
  * Rodrigues et al. (2014) Filipe Rodrigues, Francisco Pereira, and Bernardete Ribeiro.  Gaussian Process Classification and Active Learning with Multiple Annotators.  In _Int. Conf. Machine Learn._ , pp. 433–441, Beijing, China, 2014\.
  * Rodrigues et al. (2017) Filipe Rodrigues, Mariana Lourenco, Bernardete Ribeiro, and Francisco C. Pereira.  Learning Supervised Topic Models for Classification and Regression from Crowds.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 39(12):2409–2422, 2017.
  * Ronneberger et al. (2015) Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-Net: Convolutional Networks for Biomedical Image Segmentation.  In _Med. Image Comput. Comput.-Assist. Interv._ , pp. 234–241, Munich, Germany, 2015.
  * Rühling Cachay et al. (2021) Salva Rühling Cachay, Benedikt Boecking, and Artur Dubrawski.  End-to-End Weak Supervision.  In _Adv. Neural Inf. Process. Syst._ , Virtual Conf., 2021.
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  ImageNet Large Scale Visual Recognition Challenge.  _Int. J. of Comp. Vis._ , 115:211–252, 2015.
  * Schlagwein et al. (2019) Daniel Schlagwein, Dubravka Cecez-Kecmanovic, and Benjamin Hanckel.  Ethical norms and issues in crowdsourcing practices: A Habermasian analysis.  _Inf. Syst. J._ , 29(4):811–837, 2019.
  * Simonyan & Zisserman (2015) Karen Simonyan and Andrew Zisserman.  Very Deep Convolutional Networks for Large-Scale Image Recognition.  In _Int. Conf. Learn. Represent._ , San Diego, CA, 2015.
  * Tanno et al. (2019) Ryutaro Tanno, Ardavan Saeedi, Swami Sankaranarayanan, Daniel C. Alexander, and Nathan Silberman.  Learning from Noisy Labels by Regularized Estimation of Annotator Confusion.  In _Conf. Comput. Vis. Pattern Recognit._ , pp. 11244–11253, Long Beach, CA, 2019.
  * Uma et al. (2021) Alexandra Nnemamaka Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio.  Learning from Disagreement: A Survey.  _J. Artif. Intell. Res._ , 72:1385–1470, 2021.
  * Vapnik (1995) Vladimir Vapnik.  _The Nature of Statistical Learning Theory_.  Springer, 1995.
  * Vaughan (2018) Jennifer W. Vaughan.  Making Better Use of the Crowd: How Crowdsourcing Can Advance Machine Learning Research.  _J. Mach. Learn. Res._ , 18(193):1–46, 2018.
  * Vuurens et al. (2011) Jeroen Vuurens, Arjen P. de Vries, and Carsten Eickhoff.  How Much Spam Can You Take? An Analysis of Crowdsourcing Results to Increase Accuracy.  In _ACM SIGIR Workshop Crowdsourcing Inf. Retr._ , pp. 21–26, Beijing, China, 2011.
  * Wei et al. (2022) Hongxin Wei, Renchunzi Xie, Lei Feng, Bo Han, and Bo An.  Deep Learning From Multiple Noisy Annotators as A Union.  _IEEE Trans. Neural Netw. Learn. Syst._ , 2022.
  * Wojke & Bewley (2018) Nicolai Wojke and Alex Bewley.  Deep Cosine Metric Learning for Person Re-Identification.  In _IEEE Winter Conf. Appl. Comput. Vis._ , pp. 748–756, Lake Tahoe, NV, 2018.
  * Xia & McKernan (2020) Huichuan Xia and Brian McKernan.  Privacy in Crowdsourcing: a Review of the Threats and Challenges.  _Comput. Support. Coop. Work_ , 29:263–301, 2020.
  * Xiao et al. (2017) Han Xiao, Kashif Rasul, and Roland Vollgraf.  Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms.  _arXiv:1708.07747_ , 2017.
  * Yan et al. (2014) Yan Yan, Rómer Rosales, Glenn Fung, Ramanathan Subramanian, and Jennifer Dy.  Learning from multiple annotators with varying expertise.  _Machine Learn._ , 95(3):291–327, 2014.
  * Yang et al. (2018) Jie Yang, Thomas Drake, Andreas Damianou, and Yoelle Maarek.  Leveraging Crowdsourcing Data for Deep Active Learning an Application: Learning Intents in Alexa.  In _Int. World Wide Web Conf._ , pp. 23–32, Lyon, France, 2018\.
  * Yu et al. (2022) Peilin Yu, Tiffany Ding, and Stephen H. Bach.  Learning from Multiple Noisy Partial Labelers.  In _Int. Conf. Artif. Intell. Stat._ , pp. 11072–11095, Virtual Conf., 2022.
  * Zagoruyko & Komodakis (2016) Sergey Zagoruyko and Nikos Komodakis.  Wide Residual Networks.  _arXiv:1605.07146_ , 2016.
  * Zhang (2022) Jing Zhang.  Knowledge Learning With Crowdsourcing: A Brief Review and Systematic Perspective.  _IEEE/CAA J. Autom. Sin._ , 9(5):749–762, 2022\.
  * Zhang et al. (2016) Jing Zhang, Xindong Wu, and Victor S. Sheng.  Learning from Crowdsourced Labeled Data: A Survey.  _Artif. Intell. Rev._ , 46(4):543–576, 2016\.
  * Zhang et al. (2020) Le Zhang, Ryutaro Tanno, Mou-Cheng Xu, Chen Jin, Joseph Jacob, Olga Cicarrelli, Frederik Barkhof, and Daniel Alexander.  Disentangling Human Error from Ground Truth in Segmentation of Medical Images.  In _Adv. Neural Inf. Process. Syst._ , pp. 15750–15762, Virtual Conf., 2020.
  * Zhang et al. (2023) Le Zhang, Ryutaro Tanno, Moucheng Xu, Yawen Huang, Kevin Bronik, Chen Jin, Joseph Jacob, Yefeng Zheng, Ling Shao, Olga Ciccarelli, et al.  Learning from Multiple Annotators for Medical Image Segmentation.  _Pattern Recognit._ , pp. 109400, 2023.
  * Zhang et al. (2019) Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay.  Deep learning based recommender system: A survey and new perspectives.  _ACM Comput. Surv._ , 52(1):1–38, 2019.
  * Zhou et al. (2017) Lina Zhou, Shimei Pan, Jianwu Wang, and Athanasios V. Vasilakos.  Machine learning on big data: Opportunities and challenges.  _Neurocomputing_ , 237:350–361, 2017.

##  Appendix A Proofs

###### Proof of Proposition 1.

Minimizing the loss in Eq. 1 results in the following Bayes optimal
prediction:

$\displaystyle y_{\mathrm{GT}}(\mathbf{x})$
$\displaystyle=\operatorname*{arg\,min}_{y^{\prime}\in\Omega_{Y}}\left(\mathbb{E}_{y\mid\mathbf{x}}\left[\delta(y\neq
y^{\prime}))\right]\right)=\operatorname*{arg\,min}_{y^{\prime}\in\Omega_{Y}}\left(\sum_{y\in\Omega_{Y}}\Pr(y\mid\mathbf{x})\delta(y\neq
y^{\prime})\right)$
$\displaystyle=\operatorname*{arg\,min}_{y^{\prime}\in\Omega_{Y}}\left(\sum_{y\in\Omega_{Y}\setminus\\{y^{\prime}\\}}\Pr(y\mid\mathbf{x})\right)=\operatorname*{arg\,min}_{y^{\prime}\in\Omega_{Y}}\left(1-\Pr(y^{\prime}\mid\mathbf{x})\right)=\operatorname*{arg\,max}_{y^{\prime}\in\Omega_{Y}}\left(\Pr(y^{\prime}\mid\mathbf{x})\right).\qed$

###### Proof of Proposition 2.

Minimizing the loss in Eq. 4 results in the following Bayes optimal
prediction:

$\displaystyle y_{\mathrm{AP}}(\mathbf{x},\mathbf{a})$
$\displaystyle=\operatorname*{arg\,min}_{y^{\prime}\in\\{0,1\\}}\left(\mathbb{E}_{y\mid\mathbf{x}}\left[\mathbb{E}_{z\mid\mathbf{x},\mathbf{a},y}\left[\delta\left(y^{\prime}\neq\delta\left(y\neq
z\right)\right)\right]\right]\right)$
$\displaystyle=\operatorname*{arg\,min}_{y^{\prime}\in\\{0,1\\}}\left(\sum_{y\in\Omega_{Y}}\Pr(y\mid\mathbf{x})\left(\sum_{z\in\Omega_{Y}}\Pr(z\mid\mathbf{x},\mathbf{a},y)\delta(y^{\prime}\neq\delta(y\neq
z))\right)\right)$
$\displaystyle=\operatorname*{arg\,min}_{y^{\prime}\in\\{0,1\\}}\left(\sum_{y\in\Omega_{Y}}\Pr(y\mid\mathbf{x})\left(\sum_{z\in\Omega_{Y}\setminus\\{y\\}}\Pr(z\mid\mathbf{x},\mathbf{a},y)\delta(y^{\prime}\neq
1)+\Pr(y\mid\mathbf{x},\mathbf{a},y)\delta(y^{\prime}\neq 0)\right)\right)$
$\displaystyle=\operatorname*{arg\,min}_{y^{\prime}\in\\{0,1\\}}\left(\sum_{y\in\Omega_{Y}}\Pr(y\mid\mathbf{x})\Big{(}(1-\Pr(y\mid\mathbf{x},\mathbf{a},y))\delta(y^{\prime}\neq
1)+\Pr(y\mid\mathbf{x},\mathbf{a},y)\delta(y^{\prime}\neq 0)\Big{)}\right)$
$\displaystyle=\delta\left(\sum_{y\in\Omega_{Y}}\Pr(y\mid\mathbf{x})\Pr(y\mid\mathbf{x},\mathbf{a},y)<\sum_{y\in\Omega_{Y}}\Pr(y\mid\mathbf{x})\left(1-\Pr(y\mid\mathbf{x},\mathbf{a},y)\right)\right)$
$\displaystyle=\delta\left(\sum_{y\in\Omega_{Y}}\Pr(y\mid\mathbf{x})\Pr(y\mid\mathbf{x},\mathbf{a},y)<\sum_{y\in\Omega_{Y}}\Pr(y\mid\mathbf{x})-\sum_{y\in\Omega_{Y}}\Pr(y\mid\mathbf{x})\Pr(y\mid\mathbf{x},\mathbf{a},y)\right)$
$\displaystyle=\delta\left(\sum_{y\in\Omega_{Y}}\Pr(y\mid\mathbf{x})\Pr(y\mid\mathbf{x},\mathbf{a},y)<1-\sum_{y\in\Omega_{Y}}\Pr(y\mid\mathbf{x})\Pr(y\mid\mathbf{x},\mathbf{a},y)\right)$
$\displaystyle=\delta\left(\sum_{y\in\Omega_{Y}}\Pr(y\mid\mathbf{x})\Pr(y\mid\mathbf{x},\mathbf{a},y)<0.5\right).\qed$

###### Proof of Theorem 1.

Applying assumption $(\star)$ ‣ 1 of Theorem 1 to Eq. 21, the weight
$w(\mathbf{a}_{m})$ for an annotator $\mathbf{a}_{m}$ is given by: w(a_m)
=$(\star)$ ‣ 1 MG ∑_g=1^Gδ(m ∈A(g))—A(g)—. Accordingly, the sums of the
annotator weights are uniformly distributed across the $G$ groups:

$\sum_{m\in\mathcal{A}^{(1)}}w(\mathbf{a}_{m})=\dots=\sum_{m\in\mathcal{A}^{(G)}}w(\mathbf{a}_{m})=\frac{M}{G}.$
$(\diamond)$

Inserting these annotator weights into the weighted log-likelihood function
and making use of assumption $(\dagger)$ ‣ 1 in Theorem 1, we get

$\displaystyle\sum_{n=1}^{N}\sum_{m=1}^{M}w(\mathbf{a}_{m})\ln\left(\Pr(z_{nm}\mid\mathbf{x}_{n},\mathbf{a}_{m})\right)$
$\displaystyle=\sum_{n=1}^{N}\sum_{g=1}^{G}\sum_{m\in\mathcal{A}^{(g)}}w(\mathbf{a}_{m})\ln\left(\Pr(z_{nm}\mid\mathbf{x}_{n},\mathbf{a}_{m})\right)$
$\displaystyle\stackrel{{\scriptstyle\text{\ref{eq:assumption-1}}}}{{=}}\sum_{n=1}^{N}\sum_{g=1}^{G}\left(\sum_{m\in\mathcal{A}^{(g)}}w(\mathbf{a}_{m})\right)\ln\left(\Pr(z_{n{m_{g}}}\mid\mathbf{x}_{n},\mathbf{a}_{m_{g}})\right)$
$\displaystyle\stackrel{{\scriptstyle\text{\ref{eq:assumption-3}}}}{{=}}\sum_{n=1}^{N}\sum_{g=1}^{G}\frac{M}{G}\ln\left(\Pr(z_{n{m_{g}}}\mid\mathbf{x}_{n},\mathbf{a}_{m_{g}})\right)$
$\displaystyle\propto\sum_{n=1}^{N}\sum_{g=1}^{G}\ln\left(\Pr(z_{n{m_{g}}}\mid\mathbf{x}_{n},\mathbf{a}_{m_{g}})\right).\qed$

##  Appendix B End-to-end Training Algorithm Example

This appendix illustrates the steps of Algorithm 1 for a sampled mini-batch.
Let us assume a classification problem with $C=2$ classes, $M=2$ annotators,
and a mini-batch size of $B=1$. We further suppose there is no prior
information about annotators such that we represent the two annotators via the
one-hot encoded vectors

$\mathbf{a}_{1}=(1,0)^{\mathrm{T}}\text{ and
}\mathbf{a}_{2}=(0,1)^{\mathrm{T}}.$ (33)

In the first step, we compute the instances’ class-membership probabilities.
For a mini-batch size of $B=1$, we exemplarily assume we have a single
arbitrary instance $\mathbf{x}_{n}$, for which we obtain

$\mathbf{\hat{p}}_{\bm{\theta}}(\mathbf{x}_{n})=(0.8,0.2)^{\mathrm{T}}$ (34)

as class-membership probabilities outputted by the GT model with parameters
$\bm{\theta}$. In the second step, we compute the confusion matrix for each
instance-annotator pair. For a mini-batch size of $B=1$ and $M=2$ annotators,
we obtain

$\mathbf{\hat{P}}_{\bm{\omega}}(\mathbf{x}_{n},\mathbf{a}_{1})=\begin{pmatrix}1&0\\\
0&1\end{pmatrix}\text{ and
}\mathbf{\hat{P}}_{\bm{\omega}}(\mathbf{x}_{n},\mathbf{a}_{2})=\begin{pmatrix}0.5&0.5\\\
0.5&0.5\end{pmatrix}$ (35)

as two exemplary confusion matrices outputted by the AP model with parameters
$\bm{\omega}$. Thus, we currently expect annotator $\mathbf{a}_{1}$ to be
error-free and $\mathbf{a}_{2}$ to randomly guess. In the third step, we
determine the similarities between all pairs of annotator embeddings to
compute their weights in the fourth step. Since the Gaussian kernel
$k_{\gamma}$ is symmetric and there are only two annotators, we obtain

$\hat{w}_{\gamma}(\mathbf{a}_{1})=\hat{w}_{\gamma}(\mathbf{a}_{2})=1$ (36)

as annotator weights in our example. We refer to Fig. 7 for a more complex
example of computing annotator weights. Before evaluating the loss function,
we assume $\mathbf{z}_{n}=(2,2)^{\mathrm{T}}$ as the vector of class labels
assigned by the annotators $\mathbf{a}_{1}$ and $\mathbf{a}_{2}$ to instance
$\mathbf{x}_{n}$. Moreover, we take $\gamma=1$ as an example bandwidth and
$\alpha=2,\beta=1$ as parameters of the gamma distribution. Now, we have all
the ingredients to evaluate the loss function in Eq. 25:

$\displaystyle
L_{\mathbf{x}_{n},\mathbf{a}_{1},\mathbf{a}_{2},\mathbf{z}_{n},\alpha,\beta}(\bm{\theta},\bm{\omega},\gamma)=$
$\displaystyle\underbrace{-\frac{1}{2}\cdot
1\cdot\ln\left((0,1)\begin{pmatrix}1&0\\\
0&1\end{pmatrix}^{\mathrm{T}}\begin{pmatrix}0.8\\\
0.2\end{pmatrix}\right)}_{\text{prediction loss for annotator
}\mathbf{a}_{1}}$ (37) $\displaystyle\underbrace{-\frac{1}{2}\cdot
1\cdot\ln\left((0,1)\begin{pmatrix}0.5&0.5\\\
0.5&0.5\end{pmatrix}^{\mathrm{T}}\begin{pmatrix}0.8\\\
0.2\end{pmatrix}\right)}_{\text{prediction loss for annotator
}\mathbf{a}_{2}}$ (38)
$\displaystyle\underbrace{-\ln\left(\mathrm{Gam}\left(1\mid
2,1\right)\right)}_{\text{regularization term for the bandwidth }\gamma}$ (39)
$\displaystyle=-\frac{1}{2}\ln\left(0.2\right)-\frac{1}{2}\ln\left(0.5\right)+1.$
(40)

Eq. 37 and Eq. 38 compute the cross-entropy loss between the estimated
annotation probabilities

$\mathbf{\hat{p}}_{\bm{\theta},\bm{\omega}}(\mathbf{x}_{n},\mathbf{a}_{1})=\begin{pmatrix}1&0\\\
0&1\end{pmatrix}^{\mathrm{T}}\begin{pmatrix}0.8\\\
0.2\end{pmatrix}=\begin{pmatrix}0.8\\\
0.2\end{pmatrix},\qquad\mathbf{\hat{p}}_{\bm{\theta},\bm{\omega}}(\mathbf{x}_{n},\mathbf{a}_{1})=\begin{pmatrix}0.5&0.5\\\
0.5&0.5\end{pmatrix}^{\mathrm{T}}\begin{pmatrix}0.8\\\
0.2\end{pmatrix}=\begin{pmatrix}0.5\\\ 0.5\end{pmatrix}$ (41)

and the two provided annotations as one-hot encoded targets to learn
annotation patterns. In contrast, Eq. 39 computes the logarithmic probability
density of the current bandwidth value in relation to the gamma distribution
to regularize the possible bandwidth values. Finally, we can use a common
optimizer for DNNs to update the parameters $\bm{\theta},\bm{\omega},\gamma$.

##  Appendix C Ablation Study

This appendix presents the results of an ablation study regarding MaDL’s
hyperparameters. Table 9 provides the results regarding the two datasets music
and labelme with real-world annotators and Table 10 presents the results for
the dataset letter with the four annotator sets simulated according to Table
3. We design the ablation study following a one-factor-at-a-time approach in
favor of reducing the computational cost. This means we define a default MaDL
variant for each experiment and change the value of only one hyperparameter at
a time. For example, we study the effect of the AP prior $\eta\in(0,1)$ by
taking the default MaDL variant and changing only the value of this
hyperparameter. Our default MaDL variant corresponds to the hyperparameter
values described in Section 4. For the combination of the dataset letter with
the annotator set inductive, the default MaDL variant gets annotator features
containing prior information instead of one-hot encoded annotator features as
input. The general setup of an experiment, e.g., the number of repeated runs,
the splits into the training, test, and validation sets, etc., is identical to
the one described in Section 5. In the following, we analyze the effects of
the individual hyperparameters, whose default values are given in brackets:

fnum@@desciiiitemEmbedding size $(Q=R=16)$:

The embedding size controls the dimensionality of the instance and annotator
embeddings learned by the AP model. For the two datasets with real-world
annotators in Table 9, an embedding size of $Q=R=16$ clearly works best. In
contrast, the results of Table 10 indicate that an embedding size of $Q=R=8$
is superior for the dataset letter with simulated annotator sets. Although
$Q=R=16$ is a robust default embedding size for the tested learning tasks, it
is crucial to consider the characteristics of each learning task individually.
Thereby, the number of annotators and instance features are of particular
importance.

fnum@@desciiiitemAP prior $(\eta=0.8)$:

The AP prior controls the initialization of the AP output layer’s biases (cf.
Eq. 19) and thus the parametrization of initial annotator confusion matrices.
This hyperparameter is relevant for the identifiability of the class-
membership probabilities and annotator confusion matrices. The results of
Table 9 and 10 confirm this importance. A low value of $\eta=0.1$ leads to
poor performance across all tested datasets since it cannot identify the
annotation noise. Selecting high values, e.g., $\eta\in\\{0.7,0.8,0.9\\}$,
resolves such an issue and thus leads to much better performances.

fnum@@desciiiitemOuter product $(\mathrm{True})$:

The outer product layer is one option, adopted from literature (Qu et al.,
2016), to model the interactions between instance and annotator embeddings.
Training MaDL with such a product layer leads to performance gains for music
and labelme as datasets with real-world annotators (cf. Table 9). In contrast,
there are no clear performance differences for the dataset letter with the
four simulated annotator sets (cf. Table 10). Since this modeling of
interactions resembles recommender systems, testing alternatives, e.g.,
computing the outer product of instance and annotator embeddings as input to a
convolutional layer (He et al., 2018), may be worthwhile.

fnum@@desciiiitemResidual block $(\mathrm{True})$:

This hyperparameter determines whether we implement the residual connection
(True) or not (False) into the block shown in Fig. 4. The idea of this
connection is to prioritize the annotator embeddings when computing APs.
Inspecting the results, we see clear performance gains for music and labelme
as datasets with real-world annotators (cf. Table 9). In contrast, the
residual connection leads for two of the simulated annotator sets to better
and for the other two to worse results for the dataset letter. Similar to
investigating alternatives to the outer product layer, optimizing the entire
architecture in Fig. 4 may be valuable in future work.

fnum@@desciiiitemGamma prior $(\alpha=1.25,\beta=0.25)$:

This pair of hyperparameters specifies the prior gamma distribution of the
bandwidth $\gamma$ used for computing kernel values between annotator
embeddings. Table 9 and 10 list the evaluated pairs of $\alpha$ and $\beta$,
including the option of no annotator weights. The values for $\alpha$ and
$\beta$ are selected to test different combinations of modes, i.e.,
$\nicefrac{{(\alpha-1)}}{{\beta}}\in\\{0.5,1,2\\}$, and variances, i.e.,
$\nicefrac{{\alpha}}{{\beta^{2}}}\in\\{6,20\\}$, for the gamma distribution.
Table 10 indicates no large performance difference of varying value pairs of
$\alpha$ and $\beta$. As previously shown for RQ2 in Section 5, the weak
results of the option “No Weights“ for the dataset letter with random-
correlated as annotator set demonstrate the importance of modeling annotator
correlations. Table 9 indicates larger performance differences for the varying
parametrizations. Here, a mode of $\nicefrac{{(\alpha-1)}}{{\beta}}=1$,
corresponding to $(\alpha=1.25,\beta=0.25)$ and $(\alpha=1.5,\beta=0.5)$,
leads to competitive performances across both datasets. In general, one could
also replace the Gaussian kernel with other common kernels or similarity
functions to estimate annotators’ densities as a basis for quantifying their
correlations. A popular alternative would be the cosine similarity function.
For example, Wojke & Bewley (2018) demonstrate how to learn an embedding space
where this similarity function is optimized by re-parametrizing the
conventional softmax classification output.

Table 9:  Ablation study on MaDL’s hyperparameters for the datasets music and
labelme with real-world annotators:

Best

and

second best

performances are highlighted per annotator set and evaluation score.  Parameter | Value | Ground Truth Model | Ground Truth Model
---|---|---|---
ACC $\uparrow$ |  NLL $\downarrow$ |  BS $\downarrow$ |  ACC $\uparrow$ |  NLL $\downarrow$ |  BS $\downarrow$
| music | labelme
Default: MaDL |  0.743$\pm$0.020 |  0.877$\pm$0.034 |  0.381$\pm$0.013 |  0.867$\pm$0.004 |  0.623$\pm$0.138 |  0.214$\pm$0.009
Embedding Size | $Q=R=8$ |  0.742$\pm$0.011 |  0.907$\pm$0.048 |  0.378$\pm$0.010 |  0.852$\pm$0.010 |  0.728$\pm$0.146 |  0.235$\pm$0.018
$Q=R=32$ |  0.724$\pm$0.020 |  0.889$\pm$0.067 |  0.388$\pm$0.029 |  0.856$\pm$0.012 |  0.940$\pm$0.280 |  0.242$\pm$0.026
AP Prior | $\eta=0.1$ |  0.200$\pm$0.101 |  5.652$\pm$1.197 |  1.305$\pm$0.213 |  0.139$\pm$0.055 |  10.514$\pm$5.138 |  1.483$\pm$0.351
$\eta=0.7$ |  0.734$\pm$0.027 |  0.935$\pm$0.071 |  0.377$\pm$0.020 |  0.851$\pm$0.012 |  0.882$\pm$0.309 |  0.248$\pm$0.026
$\eta=0.9$ |  0.735$\pm$0.005 |  0.907$\pm$0.047 |  0.395$\pm$0.008 |  0.854$\pm$0.004 |  1.000$\pm$0.277 |  0.246$\pm$0.012
Outer Product | False |  0.725$\pm$0.030 |  0.896$\pm$0.076 |  0.394$\pm$0.034 |  0.860$\pm$0.003 |  1.014$\pm$0.221 |  0.238$\pm$0.006
Residual Block | False |  0.734$\pm$0.015 |  0.911$\pm$0.065 |  0.387$\pm$0.032 |  0.852$\pm$0.006 |  1.121$\pm$0.461 |  0.249$\pm$0.022
Gamma Prior | No Weights |  0.736$\pm$0.014 |  0.857$\pm$0.033 |  0.377$\pm$0.011 |  0.870$\pm$0.008 |  0.634$\pm$0.177 |  0.211$\pm$0.014
$\alpha=1.118,\beta=0.236$ |  0.731$\pm$0.026 |  0.922$\pm$0.071 |  0.387$\pm$0.021 |  0.853$\pm$0.011 |  1.077$\pm$0.218 |  0.253$\pm$0.018
$\alpha=1.226,\beta=0.452$ |  0.722$\pm$0.015 |  0.948$\pm$0.086 |  0.399$\pm$0.023 |  0.852$\pm$0.011 |  0.902$\pm$0.316 |  0.245$\pm$0.027
$\alpha=1.5,\beta=0.5$ |  0.748$\pm$0.009 |  0.869$\pm$0.078 |  0.372$\pm$0.025 |  0.864$\pm$0.001 |  0.685$\pm$0.153 |  0.222$\pm$0.010
$\alpha=1.56,\beta=0.28$ |  0.722$\pm$0.018 |  0.847$\pm$0.026 |  0.381$\pm$0.012 |  0.860$\pm$0.012 |  0.678$\pm$0.211 |  0.225$\pm$0.019
$\alpha=2.22,\beta=0.61$ |  0.716$\pm$0.006 |  0.947$\pm$0.068 |  0.405$\pm$0.023 |  0.857$\pm$0.008 |  0.785$\pm$0.244 |  0.236$\pm$0.013

The above analyses show that the chosen default hyperparameters do not always
give the best results but are competitive for most datasets studied. They also
indicate that having a (small) validation set with GT labels is required if
maximum performance is crucial. Obtaining such a validation set in a setting
with error-prone annotators can be expensive. Therefore, future research may
examine methods to design such validation sets cost-efficiently. In this
context, it is also essential to elaborate further on the theoretical
foundations of multi-annotator supervised learning techniques, e.g., by
deriving theoretical guarantees for specific GT and AP distribution types.

Table 10:  Ablation study on MaDL’s hyperparameters for the dataset letter and
four simulated annotated sets:

Best

and

second best

performances are highlighted per annotator set and evaluation score.  Parameter | Value | Ground Truth Model | Annotator Performance Model
---|---|---|---
ACC $\uparrow$ |  NLL $\downarrow$ |  BS $\downarrow$ |  ACC $\uparrow$ |  NLL $\downarrow$ |  BS $\downarrow$ |  BAL-ACC $\uparrow$
letter (independent)
Default: MaDL |  0.935$\pm$0.006 |  0.303$\pm$0.102 |  0.098$\pm$0.010 |  0.766$\pm$0.004 |  0.491$\pm$0.007 |  0.317$\pm$0.005 |  0.702$\pm$0.005
Embedding Size | $Q=R=8$ |  0.937$\pm$0.004 |  0.285$\pm$0.116 |  0.097$\pm$0.006 |  0.773$\pm$0.003 |  0.476$\pm$0.006 |  0.309$\pm$0.002 |  0.708$\pm$0.003
$Q=R=32$ |  0.935$\pm$0.007 |  0.327$\pm$0.125 |  0.099$\pm$0.011 |  0.763$\pm$0.004 |  0.501$\pm$0.005 |  0.322$\pm$0.003 |  0.698$\pm$0.005
AP Prior | $\eta=0.1$ |  0.587$\pm$0.048 |  7.686$\pm$1.229 |  0.766$\pm$0.095 |  0.656$\pm$0.018 |  1.194$\pm$0.139 |  0.554$\pm$0.037 |  0.614$\pm$0.015
$\eta=0.7$ |  0.938$\pm$0.005 |  0.273$\pm$0.047 |  0.095$\pm$0.007 |  0.767$\pm$0.002 |  0.487$\pm$0.003 |  0.314$\pm$0.002 |  0.702$\pm$0.004
$\eta=0.9$ |  0.935$\pm$0.007 |  0.250$\pm$0.027 |  0.095$\pm$0.009 |  0.768$\pm$0.002 |  0.486$\pm$0.003 |  0.315$\pm$0.002 |  0.703$\pm$0.003
Outer Product | False |  0.933$\pm$0.010 |  0.365$\pm$0.177 |  0.103$\pm$0.018 |  0.765$\pm$0.003 |  0.499$\pm$0.012 |  0.321$\pm$0.006 |  0.701$\pm$0.004
Residual Block | False |  0.935$\pm$0.003 |  0.326$\pm$0.131 |  0.099$\pm$0.007 |  0.762$\pm$0.004 |  0.513$\pm$0.014 |  0.326$\pm$0.005 |  0.699$\pm$0.004
Gamma Prior | No Weights |  0.939$\pm$0.007 |  0.302$\pm$0.097 |  0.094$\pm$0.011 |  0.767$\pm$0.002 |  0.491$\pm$0.007 |  0.316$\pm$0.003 |  0.702$\pm$0.002
$\alpha=1.118,\beta=0.236$ |  0.937$\pm$0.005 |  0.285$\pm$0.053 |  0.096$\pm$0.008 |  0.768$\pm$0.004 |  0.484$\pm$0.003 |  0.314$\pm$0.002 |  0.701$\pm$0.004
$\alpha=1.226,\beta=0.452$ |  0.936$\pm$0.003 |  0.307$\pm$0.081 |  0.097$\pm$0.008 |  0.767$\pm$0.004 |  0.486$\pm$0.006 |  0.315$\pm$0.003 |  0.702$\pm$0.006
$\alpha=1.5,\beta=0.5$ |  0.940$\pm$0.002 |  0.274$\pm$0.042 |  0.092$\pm$0.004 |  0.767$\pm$0.003 |  0.489$\pm$0.006 |  0.317$\pm$0.003 |  0.703$\pm$0.004
$\alpha=1.56,\beta=0.28$ |  0.936$\pm$0.003 |  0.266$\pm$0.046 |  0.095$\pm$0.006 |  0.767$\pm$0.001 |  0.491$\pm$0.006 |  0.316$\pm$0.002 |  0.702$\pm$0.002
$\alpha=2.22,\beta=0.61$ |  0.936$\pm$0.002 |  0.283$\pm$0.048 |  0.097$\pm$0.005 |  0.766$\pm$0.003 |  0.493$\pm$0.005 |  0.318$\pm$0.002 |  0.703$\pm$0.002
letter (correlated)
Default: MaDL |  0.947$\pm$0.003 |  0.282$\pm$0.077 |  0.080$\pm$0.004 |  0.887$\pm$0.001 |  0.308$\pm$0.004 |  0.175$\pm$0.002 |  0.756$\pm$0.001
Embedding Size | $Q=R=8$ |  0.953$\pm$0.003 |  0.219$\pm$0.017 |  0.072$\pm$0.004 |  0.888$\pm$0.002 |  0.303$\pm$0.003 |  0.172$\pm$0.002 |  0.757$\pm$0.002
$Q=R=32$ |  0.949$\pm$0.005 |  0.303$\pm$0.089 |  0.080$\pm$0.008 |  0.885$\pm$0.002 |  0.324$\pm$0.018 |  0.178$\pm$0.004 |  0.754$\pm$0.002
AP Prior | $\eta=0.1$ |  0.588$\pm$0.089 |  9.442$\pm$2.241 |  0.773$\pm$0.166 |  0.782$\pm$0.031 |  0.894$\pm$0.186 |  0.377$\pm$0.062 |  0.659$\pm$0.030
$\eta=0.7$ |  0.947$\pm$0.006 |  0.292$\pm$0.080 |  0.078$\pm$0.008 |  0.887$\pm$0.002 |  0.309$\pm$0.005 |  0.174$\pm$0.003 |  0.756$\pm$0.003
| $\eta=0.9$ |  0.948$\pm$0.003 |  0.255$\pm$0.079 |  0.079$\pm$0.006 |  0.887$\pm$0.001 |  0.311$\pm$0.001 |  0.175$\pm$0.002 |  0.757$\pm$0.002
Outer Product | False |  0.948$\pm$0.003 |  0.305$\pm$0.109 |  0.081$\pm$0.005 |  0.887$\pm$0.001 |  0.314$\pm$0.005 |  0.177$\pm$0.002 |  0.757$\pm$0.001
Residual Block | False |  0.936$\pm$0.016 |  0.541$\pm$0.486 |  0.101$\pm$0.031 |  0.881$\pm$0.005 |  0.339$\pm$0.032 |  0.185$\pm$0.010 |  0.751$\pm$0.005
Gamma Prior | No Weights |  0.946$\pm$0.007 |  0.293$\pm$0.092 |  0.083$\pm$0.010 |  0.883$\pm$0.003 |  0.314$\pm$0.002 |  0.178$\pm$0.002 |  0.751$\pm$0.003
$\alpha=1.118,\beta=0.236$ |  0.950$\pm$0.004 |  0.260$\pm$0.089 |  0.077$\pm$0.007 |  0.887$\pm$0.001 |  0.308$\pm$0.004 |  0.174$\pm$0.003 |  0.758$\pm$0.001
$\alpha=1.226,\beta=0.452$ |  0.951$\pm$0.004 |  0.250$\pm$0.070 |  0.075$\pm$0.006 |  0.888$\pm$0.002 |  0.309$\pm$0.005 |  0.174$\pm$0.003 |  0.758$\pm$0.002
$\alpha=1.5,\beta=0.5$ |  0.951$\pm$0.005 |  0.271$\pm$0.093 |  0.075$\pm$0.007 |  0.887$\pm$0.001 |  0.311$\pm$0.003 |  0.174$\pm$0.002 |  0.757$\pm$0.002
$\alpha=1.56,\beta=0.28$ |  0.947$\pm$0.004 |  0.286$\pm$0.072 |  0.082$\pm$0.008 |  0.885$\pm$0.002 |  0.318$\pm$0.013 |  0.177$\pm$0.004 |  0.754$\pm$0.002
$\alpha=2.22,\beta=0.61$ |  0.947$\pm$0.002 |  0.289$\pm$0.052 |  0.082$\pm$0.003 |  0.885$\pm$0.002 |  0.313$\pm$0.001 |  0.177$\pm$0.001 |  0.754$\pm$0.003
letter (random-correlated)
Default: MaDL |  0.932$\pm$0.004 |  0.277$\pm$0.043 |  0.101$\pm$0.006 |  0.940$\pm$0.000 |  0.204$\pm$0.004 |  0.101$\pm$0.001 |  0.519$\pm$0.001
Embedding Size | $Q=R=8$ |  0.935$\pm$0.003 |  0.232$\pm$0.012 |  0.097$\pm$0.004 |  0.940$\pm$0.000 |  0.202$\pm$0.001 |  0.101$\pm$0.000 |  0.519$\pm$0.000
$Q=R=32$ |  0.935$\pm$0.003 |  0.266$\pm$0.057 |  0.096$\pm$0.007 |  0.939$\pm$0.000 |  0.214$\pm$0.003 |  0.104$\pm$0.000 |  0.519$\pm$0.000
AP Prior | $\eta=0.1$ |  0.598$\pm$0.069 |  7.537$\pm$1.922 |  0.744$\pm$0.136 |  0.931$\pm$0.002 |  0.251$\pm$0.012 |  0.120$\pm$0.004 |  0.512$\pm$0.001
$\eta=0.7$ |  0.934$\pm$0.009 |  0.263$\pm$0.035 |  0.098$\pm$0.012 |  0.940$\pm$0.000 |  0.204$\pm$0.003 |  0.101$\pm$0.000 |  0.519$\pm$0.000
| $\eta=0.9$ |  0.932$\pm$0.003 |  0.247$\pm$0.015 |  0.101$\pm$0.004 |  0.939$\pm$0.000 |  0.205$\pm$0.002 |  0.102$\pm$0.001 |  0.519$\pm$0.000
Outer Product | False |  0.932$\pm$0.003 |  0.292$\pm$0.038 |  0.102$\pm$0.006 |  0.939$\pm$0.000 |  0.207$\pm$0.004 |  0.102$\pm$0.001 |  0.519$\pm$0.000
Residual Block | False |  0.936$\pm$0.005 |  0.269$\pm$0.034 |  0.098$\pm$0.006 |  0.939$\pm$0.000 |  0.207$\pm$0.001 |  0.103$\pm$0.000 |  0.518$\pm$0.000
Gamma Prior | No Weights |  0.548$\pm$0.037 |  1.902$\pm$0.241 |  0.673$\pm$0.071 |  0.801$\pm$0.049 |  0.423$\pm$0.037 |  0.265$\pm$0.031 |  0.506$\pm$0.007
$\alpha=1.118,\beta=0.236$ |  0.935$\pm$0.001 |  0.270$\pm$0.034 |  0.097$\pm$0.004 |  0.940$\pm$0.000 |  0.204$\pm$0.001 |  0.101$\pm$0.000 |  0.519$\pm$0.000
$\alpha=1.226,\beta=0.452$ |  0.934$\pm$0.002 |  0.255$\pm$0.039 |  0.097$\pm$0.005 |  0.940$\pm$0.000 |  0.204$\pm$0.002 |  0.101$\pm$0.000 |  0.519$\pm$0.000
$\alpha=1.5,\beta=0.5$ |  0.935$\pm$0.006 |  0.252$\pm$0.047 |  0.098$\pm$0.009 |  0.940$\pm$0.000 |  0.204$\pm$0.003 |  0.101$\pm$0.001 |  0.519$\pm$0.000
$\alpha=1.56,\beta=0.28$ |  0.932$\pm$0.004 |  0.306$\pm$0.087 |  0.103$\pm$0.010 |  0.939$\pm$0.000 |  0.210$\pm$0.003 |  0.103$\pm$0.000 |  0.519$\pm$0.000
$\alpha=2.22,\beta=0.61$ |  0.932$\pm$0.005 |  0.279$\pm$0.048 |  0.101$\pm$0.008 |  0.939$\pm$0.000 |  0.209$\pm$0.003 |  0.103$\pm$0.001 |  0.519$\pm$0.000
letter (inductive)
Default: MaDL(A) |  0.914$\pm$0.004 |  0.303$\pm$0.010 |  0.124$\pm$0.006 |  0.718$\pm$0.004 |  0.583$\pm$0.007 |  0.383$\pm$0.004 |  0.665$\pm$0.005
Embedding Size | $Q=R=8$ |  0.919$\pm$0.004 |  0.287$\pm$0.018 |  0.120$\pm$0.005 |  0.704$\pm$0.012 |  0.569$\pm$0.015 |  0.384$\pm$0.012 |  0.645$\pm$0.012
$Q=R=32$ |  0.916$\pm$0.005 |  0.330$\pm$0.041 |  0.122$\pm$0.006 |  0.712$\pm$0.007 |  0.632$\pm$0.028 |  0.398$\pm$0.012 |  0.657$\pm$0.008
AP Prior | $\eta=0.1$ |  0.642$\pm$0.065 |  4.982$\pm$1.017 |  0.630$\pm$0.127 |  0.652$\pm$0.017 |  0.999$\pm$0.148 |  0.525$\pm$0.042 |  0.605$\pm$0.015
$\eta=0.7$ |  0.912$\pm$0.006 |  0.332$\pm$0.030 |  0.128$\pm$0.009 |  0.704$\pm$0.004 |  0.588$\pm$0.005 |  0.390$\pm$0.003 |  0.650$\pm$0.005
| $\eta=0.9$ |  0.913$\pm$0.003 |  0.306$\pm$0.021 |  0.127$\pm$0.003 |  0.717$\pm$0.005 |  0.582$\pm$0.005 |  0.383$\pm$0.003 |  0.665$\pm$0.005
Outer Product | False |  0.914$\pm$0.008 |  0.403$\pm$0.151 |  0.127$\pm$0.011 |  0.725$\pm$0.003 |  0.596$\pm$0.013 |  0.381$\pm$0.004 |  0.676$\pm$0.003
Residual Block | False |  0.921$\pm$0.003 |  0.286$\pm$0.032 |  0.116$\pm$0.006 |  0.721$\pm$0.004 |  0.593$\pm$0.032 |  0.378$\pm$0.011 |  0.668$\pm$0.004
Gamma Prior | No Weights |  0.915$\pm$0.004 |  0.301$\pm$0.016 |  0.124$\pm$0.003 |  0.719$\pm$0.007 |  0.577$\pm$0.010 |  0.377$\pm$0.006 |  0.663$\pm$0.007
$\alpha=1.118,\beta=0.236$ |  0.917$\pm$0.006 |  0.308$\pm$0.033 |  0.123$\pm$0.008 |  0.718$\pm$0.004 |  0.593$\pm$0.015 |  0.384$\pm$0.006 |  0.667$\pm$0.006
$\alpha=1.226,\beta=0.452$ |  0.918$\pm$0.002 |  0.298$\pm$0.024 |  0.121$\pm$0.002 |  0.719$\pm$0.006 |  0.583$\pm$0.016 |  0.381$\pm$0.007 |  0.666$\pm$0.006
$\alpha=1.5,\beta=0.5$ |  0.914$\pm$0.000 |  0.304$\pm$0.006 |  0.126$\pm$0.003 |  0.716$\pm$0.007 |  0.581$\pm$0.018 |  0.383$\pm$0.010 |  0.663$\pm$0.005
$\alpha=1.56,\beta=0.28$ |  0.910$\pm$0.006 |  0.341$\pm$0.059 |  0.132$\pm$0.007 |  0.711$\pm$0.008 |  0.588$\pm$0.009 |  0.389$\pm$0.007 |  0.659$\pm$0.006
$\alpha=2.22,\beta=0.61$ |  0.917$\pm$0.007 |  0.334$\pm$0.072 |  0.124$\pm$0.009 |  0.714$\pm$0.007 |  0.586$\pm$0.015 |  0.385$\pm$0.008 |  0.660$\pm$0.005

##  Appendix D Extended Results regarding Research Question 3

Table 11 extends Table 8 by the AP models’ results regarding the 75 annotators
providing class labels for training. The GT models’ results of both tables are
identical and are given to avoid switching between the two tables. The
additional AP models’ results confirm the main takeaway for property P5.
Excluding the UB, MaDL(A), which uses annotator features including prior
information, makes the most accurate AP predictions. This means it also
outperforms its counterpart MaDL($\overline{\text{A}}$), which has no prior
information about the annotators. LIA(A) estimates the APs more accurately
than LIA($\overline{\text{A}}$) for three of the four datasets. Comparing the
AP models of CoNAL(A) and CoNAL($\overline{\text{A}}$), performance gains of
using annotator features are observable for two of the four datasets, while
there are no notable differences for the other two datasets.

Table 11:  Additional results regarding RQ3 for datasets with simulated
annotators:

Best

and

second best

performances are highlighted per dataset and evaluation score while excluding the performances of the UB. In contrast to Table 8, the AP models’ results refer to the 75 annotators providing class labels for training.  Technique | P5 | P6 | Ground Truth Model | Annotator Performance Model
---|---|---|---|---
ACC $\uparrow$ |  NLL $\downarrow$ |  BS $\downarrow$ |  ACC $\uparrow$ |  NLL $\downarrow$ |  BS $\downarrow$ |  BAL-ACC $\uparrow$
letter (inductive)
UB | ✓ | ✓ | 0.962$\pm$0.002 | 0.129$\pm$0.003 | 0.058$\pm$0.002 | 0.729$\pm$0.004 | 0.565$\pm$0.010 | 0.369$\pm$0.005 | 0.677$\pm$0.004
LB | ✓ | ✓ |  0.861$\pm$0.005 |  1.090$\pm$0.017 |  0.429$\pm$0.008 |  0.572$\pm$0.008 |  0.719$\pm$0.008 |  0.515$\pm$0.006 |  0.552$\pm$0.004
LIA($\overline{\text{A}}$) | ✗ | ✗ |  0.875$\pm$0.006 |  0.901$\pm$0.060 |  0.350$\pm$0.024 |  0.622$\pm$0.002 |  0.712$\pm$0.015 |  0.495$\pm$0.007 |  0.593$\pm$0.003
LIA(A) | ✓ | ✓ |  0.876$\pm$0.006 |  1.006$\pm$0.177 |  0.397$\pm$0.074 |  0.618$\pm$0.028 |  1.441$\pm$0.925 |  0.581$\pm$0.129 |  0.572$\pm$0.038
CoNAL($\overline{\text{A}}$) | ✗ | ✗ |  0.875$\pm$0.009 |  0.804$\pm$0.119 |  0.186$\pm$0.010 |  0.646$\pm$0.001 |  0.666$\pm$0.019 |  0.455$\pm$0.007 |  0.595$\pm$0.001
CoNAL(A) | ✓ | ✗ |  0.874$\pm$0.007 |  0.808$\pm$0.116 |  0.186$\pm$0.011 |  0.646$\pm$0.001 |  0.662$\pm$0.015 |  0.453$\pm$0.006 |  0.595$\pm$0.001
MaDL($\overline{\text{A}}$) | ✗ | ✗ |  0.911$\pm$0.006 |  0.334$\pm$0.026 |  0.129$\pm$0.008 |  0.685$\pm$0.011 |  0.601$\pm$0.008 |  0.403$\pm$0.009 |  0.626$\pm$0.012
MaDL(A) | ✓ | ✓ |  0.914$\pm$0.004 |  0.303$\pm$0.009 |  0.124$\pm$0.005 |  0.718$\pm$0.004 |  0.583$\pm$0.006 |  0.383$\pm$0.004 |  0.665$\pm$0.004
fmnist (inductive)
UB | ✓ | ✓ | 0.909$\pm$0.002 | 0.246$\pm$0.005 | 0.131$\pm$0.003 | 0.759$\pm$0.001 | 0.486$\pm$0.002 | 0.320$\pm$0.001 | 0.693$\pm$0.002
LB | ✓ | ✓ |  0.881$\pm$0.002 |  0.876$\pm$0.005 |  0.370$\pm$0.002 |  0.621$\pm$0.020 |  0.662$\pm$0.006 |  0.469$\pm$0.006 |  0.562$\pm$0.009
LIA($\overline{\text{A}}$) | ✗ | ✗ |  0.852$\pm$0.003 |  1.011$\pm$0.020 |  0.436$\pm$0.010 |  0.657$\pm$0.022 |  0.641$\pm$0.013 |  0.449$\pm$0.013 |  0.589$\pm$0.015
LIA(A) | ✓ | ✓ |  0.855$\pm$0.002 |  0.972$\pm$0.012 |  0.417$\pm$0.006 |  0.685$\pm$0.031 |  0.623$\pm$0.023 |  0.432$\pm$0.021 |  0.610$\pm$0.025
CoNAL($\overline{\text{A}}$) | ✗ | ✗ |  0.889$\pm$0.002 |  0.322$\pm$0.005 |  0.163$\pm$0.003 |  0.705$\pm$0.009 |  0.547$\pm$0.010 |  0.372$\pm$0.009 |  0.653$\pm$0.010
CoNAL(A) | ✓ | ✗ |  0.890$\pm$0.002 |  0.323$\pm$0.011 |  0.163$\pm$0.005 |  0.713$\pm$0.011 |  0.540$\pm$0.011 |  0.366$\pm$0.010 |  0.661$\pm$0.013
MaDL($\overline{\text{A}}$) | ✗ | ✗ |  0.895$\pm$0.002 |  0.297$\pm$0.004 |  0.152$\pm$0.002 |  0.753$\pm$0.003 |  0.496$\pm$0.004 |  0.328$\pm$0.003 |  0.683$\pm$0.005
MaDL(A) | ✓ | ✓ |  0.893$\pm$0.004 |  0.297$\pm$0.008 |  0.153$\pm$0.004 |  0.755$\pm$0.003 |  0.492$\pm$0.004 |  0.325$\pm$0.003 |  0.686$\pm$0.005
cifar10 (inductive)
UB | ✓ | ✓ | 0.931$\pm$0.002 | 0.527$\pm$0.022 | 0.122$\pm$0.003 | 0.712$\pm$0.001 | 0.555$\pm$0.002 | 0.372$\pm$0.002 | 0.655$\pm$0.001
LB | ✓ | ✓ |  0.781$\pm$0.003 |  1.054$\pm$0.035 |  0.447$\pm$0.016 |  0.584$\pm$0.006 |  0.682$\pm$0.004 |  0.489$\pm$0.003 |  0.534$\pm$0.004
LIA($\overline{\text{A}}$) | ✗ | ✗ |  0.798$\pm$0.008 |  1.072$\pm$0.014 |  0.455$\pm$0.006 |  0.598$\pm$0.013 |  0.676$\pm$0.014 |  0.482$\pm$0.013 |  0.547$\pm$0.008
LIA(A) | ✓ | ✓ |  0.804$\pm$0.004 |  1.056$\pm$0.022 |  0.447$\pm$0.011 |  0.602$\pm$0.022 |  0.673$\pm$0.017 |  0.479$\pm$0.016 |  0.549$\pm$0.014
CoNAL($\overline{\text{A}}$) | ✗ | ✗ |  0.835$\pm$0.002 |  0.576$\pm$0.016 |  0.245$\pm$0.005 |  0.670$\pm$0.002 |  0.599$\pm$0.002 |  0.415$\pm$0.002 |  0.618$\pm$0.001
CoNAL(A) | ✓ | ✗ |  0.834$\pm$0.006 |  0.574$\pm$0.017 |  0.248$\pm$0.007 |  0.670$\pm$0.001 |  0.599$\pm$0.001 |  0.415$\pm$0.001 |  0.618$\pm$0.002
MaDL($\overline{\text{A}}$) | ✗ | ✗ |  0.811$\pm$0.008 |  0.626$\pm$0.036 |  0.277$\pm$0.014 |  0.690$\pm$0.002 |  0.565$\pm$0.002 |  0.386$\pm$0.001 |  0.624$\pm$0.003
MaDL(A) | ✓ | ✓ |  0.837$\pm$0.003 |  0.557$\pm$0.028 |  0.242$\pm$0.006 |  0.703$\pm$0.004 |  0.561$\pm$0.015 |  0.378$\pm$0.005 |  0.640$\pm$0.005
svhn (inductive)
UB | ✓ | ✓ | 0.965$\pm$0.001 | 0.393$\pm$0.015 | 0.063$\pm$0.002 | 0.650$\pm$0.003 | 0.589$\pm$0.002 | 0.410$\pm$0.002 | 0.578$\pm$0.004
LB | ✓ | ✓ |  0.927$\pm$0.002 |  0.805$\pm$0.016 |  0.328$\pm$0.009 |  0.579$\pm$0.006 |  0.709$\pm$0.005 |  0.513$\pm$0.004 |  0.531$\pm$0.003
LIA($\overline{\text{A}}$) | ✗ | ✗ |  0.929$\pm$0.003 |  0.818$\pm$0.133 |  0.336$\pm$0.068 |  0.548$\pm$0.043 |  0.695$\pm$0.029 |  0.502$\pm$0.027 |  0.528$\pm$0.009
LIA(A) | ✓ | ✓ |  0.932$\pm$0.001 |  0.754$\pm$0.152 |  0.303$\pm$0.079 |  0.600$\pm$0.009 |  0.672$\pm$0.027 |  0.480$\pm$0.024 |  0.539$\pm$0.004
CoNAL($\overline{\text{A}}$) | ✗ | ✗ |  0.941$\pm$0.001 |  0.258$\pm$0.009 |  0.090$\pm$0.003 |  0.627$\pm$0.005 |  0.608$\pm$0.002 |  0.428$\pm$0.002 |  0.551$\pm$0.006
CoNAL(A) | ✓ | ✗ |  0.942$\pm$0.001 |  0.260$\pm$0.012 |  0.090$\pm$0.002 |  0.632$\pm$0.007 |  0.605$\pm$0.003 |  0.425$\pm$0.003 |  0.556$\pm$0.007
MaDL($\overline{\text{A}}$) | ✗ | ✗ |  0.928$\pm$0.002 |  0.299$\pm$0.019 |  0.109$\pm$0.005 |  0.631$\pm$0.004 |  0.599$\pm$0.006 |  0.418$\pm$0.002 |  0.548$\pm$0.002
MaDL(A) | ✓ | ✓ |  0.935$\pm$0.001 |  0.256$\pm$0.009 |  0.098$\pm$0.002 |  0.641$\pm$0.002 |  0.592$\pm$0.001 |  0.413$\pm$0.001 |  0.562$\pm$0.002

##  Appendix E A Case Study on Varying Annotation Ratios

This appendix presents a case study on the impact of varying annotation ratios
on multi-annotator supervised learning techniques’ performances. Fig. 9
displays evaluation results for four different annotation ratios, i.e., 0.2,
0.4, 0.6, and 0.8, as curves for the dataset cifar10 with the simulated
annotator set independent. For a better interpretation of the results, the
upper right plot displays the means and standard deviations over five runs for
two descriptive statistics of the training data. On the one hand, we calculate
the annotation accuracy (ANNOT-ACC), and on the other hand, we calculate the
accuracy of the observed annotations aggregated via the majority rule (MR-
ACC). Mathematically, both statistics can be expressed as follows:

$\displaystyle\text{ANNOT-ACC}(\mathbf{y},\mathbf{Z})$
$\displaystyle\coloneqq\frac{1}{|\mathbf{Z}|}\sum_{n=1}^{N}\sum_{m\in\mathcal{A}_{n}}\delta\left(y_{n}=z_{nm}\right),$
(42) $\displaystyle\text{MR-ACC}(\mathbf{y},\mathbf{\overline{z}})$
$\displaystyle\coloneqq\frac{1}{N}\sum_{n=1}^{N}\delta\left(y_{n}=\overline{z}_{n}\right),$
(43)

where
$\mathbf{\overline{z}}=(\overline{z}_{1},\dots,\overline{z}_{N})^{\mathrm{T}}\in\Omega_{Y}^{N}$
denotes the vector of observed annotations aggregated per instance via the
majority rule. The ANNOT-ACC is almost constant across the different
annotation ratios, as the selection of which annotations are observed is made
randomly. In comparison, the MR-ACC increases because, on average, the
annotators are independent and better than random guessing. Therefore, the
probability of correctly aggregated annotation increases with more observed
annotations per instance.

The curves in the other seven plots report multi-annotator supervised learning
techniques’ mean evaluation scores and standard deviations over five runs. The
evaluated MaDL variant corresponds to its default hyperparameter
configuration. The GT models’ evaluation curves in the first row of Fig. 9
confirm our intuition that more annotations lead to better results. Further,
we observe superior performances of the UB regarding ACC and BS, while the UB
repeatedly provides inferior NLL scores. Since the NLL per instance is
unbounded, the latter observation is likely caused by highly overconfident
predictions. As expected, the GT model of the LB performs worst in terms of
the ACC for each tested annotation ratio and thus confirms the general
usefulness of multi-annotator supervised learning. However, its results for
the NLL and BS are surprisingly better than LIA, possibly due to the
difficulties of training via an EM algorithm (cf. Section 3). The GT model of
MaDL outperforms other multi-annotator supervised learning techniques
regarding the ACC and BS results, while it is approximately on par with CoNAL
regarding the NLL results. As the annotation ratio increases, the ACC of MaDL
approaches that of the UB. Still, MaDL cannot reach the ACC results of the UB,
even for the highest tested annotation ratio of 0.8. This is because the
negative impact of annotation noise is problem-dependent (Gu et al., 2022),
e.g., the impact may depend on the data distribution. The annotator simulation
is an additional impactful aspect in our concrete case. For cifar10, the
negative impact of annotation noise is more significant than for the other
datasets, particularly evident in the large gap in the ACC between the LB and
the UB. The AP models’ evaluation curves in the second row of Fig. 9 show
similar trends to the ones of the GT models. In particular, there are two
noteworthy aspects. First, the AP estimates of MaDL are close to the UB.
Second, the AP estimates of the LB are better than the ones of the other
multi-annotator supervised learning techniques (excluding MaDL and UB) for
annotation ratios of 0.6 and 0.8. The well-performing AP model architecture
adopted from MaDL likely leads to these results.

GT-ACC $\uparrow$

Annotation Ratios

GT-NLL $\downarrow$

Annotation Ratios

GT-BS $\downarrow$

Annotation Ratios

ACC $\uparrow$

Annotation Ratios

AP-ACC $\uparrow$

Annotation Ratios

AP-NLL $\downarrow$

Annotation Ratios

AP-BS $\downarrow$

Annotation Ratios

AP-BAL-ACC $\uparrow$

Annotation Ratios Figure 9: Evaluation results for four different annotation
ratios on cifar10 (independent).

##  Appendix F A Case Study on CIFAR100

This appendix presents a case study analyzing multi-annotator supervised
learning techniques’ performances on classification tasks with more classes
than in the main experiments. For this purpose, we use the dataset cifar100,
which is similar to the cifar10 dataset (cf. Table 2), with the major
difference that $C=100$ classes occur (Krizhevsky, 2009). The general design
of the experiments follows that of RQ1 (cf. Section 5.1), except that we adopt
the wide residual network (WRN-28-10, Zagoruyko & Komodakis, 2016) as the
basic network architecture and stochastic GD as the optimizer with a mini-
batch size of $B=128$. For a fairer comparison, we perform a small grid search
with the learning rates $\\{0.1,0.01\\}$ and weight decays $\\{0.0005,0.0\\}$
for each technique. Thereby, we reduce the learning rate with cosine annealing
(Loshchilov & Hutter, 2017) over the 100 training epochs.

Table 12 presents the obtained GT and AP models’ test performances, where the
evaluated MaDL variant corresponds to its default hyperparameter
configuration. As expected, training with GT labels as the UB achieves the
best performance in terms of GT and AP estimates. The performance gap between
the UB and other techniques is substantial, highlighting the challenges of
learning the cifar100 dataset with partially erroneous annotations. However,
the benefits of employing multi-annotator supervised learning techniques are
also significant. This advantage becomes particularly apparent when comparing
MaDL with the LB because MaDL improves the GT-ACC by approximately $19\,\%$.
The comparison with the other multi-annotator supervised learning techniques
further confirms the state-of-the-art results of MaDL for cifar100. We note
that only LIA and REAC appear to roughly learn the classification task among
the related techniques. A conceivable explanation might be that the default
values of the hyperparameters of CL, UNION, and CoNAL are only well-suited for
datasets with fewer classes.

Table 12:  Results on the cifar100 dataset with simulated annotators:

Best

and

second best

performances are highlighted per evaluation score while excluding the performances of the UB.  Technique | Ground Truth Model | Annotator Performance Model
---|---|---
ACC $\uparrow$ |  NLL $\downarrow$ |  BS $\downarrow$ |  ACC $\uparrow$ |  NLL $\downarrow$ |  BS $\downarrow$ |  BAL-ACC $\uparrow$
cifar100 (independent)
UB | 0.795 $\pm$ 0.001 | 0.818 $\pm$ 0.008 | 0.297 $\pm$ 0.003 | 0.727 $\pm$ 0.002 | 0.530 $\pm$ 0.001 | 0.355 $\pm$ 0.001 | 0.665 $\pm$ 0.003
LB |  0.430 $\pm$ 0.010 |  2.593 $\pm$ 0.038 |  0.762 $\pm$ 0.012 |  0.599 $\pm$ 0.015 |  0.665 $\pm$ 0.006 |  0.472 $\pm$ 0.005 |  0.577 $\pm$ 0.011
CL |  0.077 $\pm$ 0.013 |  10.555 $\pm$ 0.802 |  1.616 $\pm$ 0.059 |  0.594 $\pm$ 0.000 |  1.466 $\pm$ 0.013 |  0.764 $\pm$ 0.001 |  0.500 $\pm$ 0.000
REAC |  0.553 $\pm$ 0.006 |  2.037 $\pm$ 0.175 |  0.630 $\pm$ 0.023 |  0.629 $\pm$ 0.029 |  0.657 $\pm$ 0.071 |  0.465 $\pm$ 0.060 |  0.543 $\pm$ 0.035
UNION |  0.033 $\pm$ 0.003 |  4.815 $\pm$ 0.067 |  1.031 $\pm$ 0.014 |  0.594 $\pm$ 0.000 |  1.838 $\pm$ 0.008 |  0.794 $\pm$ 0.001 |  0.500 $\pm$ 0.000
LIA |  0.560 $\pm$ 0.008 |  1.995 $\pm$ 0.051 |  0.619 $\pm$ 0.009 |  0.672 $\pm$ 0.004 |  0.608 $\pm$ 0.003 |  0.417 $\pm$ 0.003 |  0.611 $\pm$ 0.006
CoNAL |  0.168 $\pm$ 0.008 |  7.762 $\pm$ 0.344 |  1.283 $\pm$ 0.016 |  0.594 $\pm$ 0.000 |  1.209 $\pm$ 0.021 |  0.700 $\pm$ 0.003 |  0.500 $\pm$ 0.000
MaDL |  0.621 $\pm$ 0.003 |  1.628 $\pm$ 0.011 |  0.508 $\pm$ 0.005 |  0.665 $\pm$ 0.006 |  0.646 $\pm$ 0.015 |  0.442 $\pm$ 0.011 |  0.628 $\pm$ 0.003

##  Appendix G Practitioner’s Guide

This appendix provides an overview of various aspects to consider when
employing MaDL in practical classification tasks. Note that these aspects are
closely related to each other and that the associated recommendations do not
have general validity.

fnum@@descivitemData modalities:

We can train MaDL with different modalities of data. Depending on the
instances’ data modality, we need to modify the GT model’s architecture. For
example, we may use TabNet (Arik & Pfister, 2021) for tabular data, ResNet (He
et al., 2016) for image data, or BERT (Devlin et al., 2018) for text data.
These considerations apply analogously to the annotator features and the
architecture of the AP model. However, annotator features are commonly tabular
since they are collected via surveys, or only anonymized identifiers of the
annotators are known if there is no prior annotator information. In the latter
case, one-hot encoding converts these identifiers into tabular annotator
features. Recommendation: Use common architectures from the literature to fit
the data modality.

fnum@@descivitemNumber of instances:

The number of annotated instances available for training is critical to DNNs’
generalization performances (Hestness et al., 2017) and, consequently, to
MaDL’s generalization performance. Typically, the more annotated instances we
have, the better MaDL can learn the underlying annotation patterns as a basis
for inferring the GT class labels and APs. Learning curves allow us to study
this behavior (Hoiem et al., 2021). Fig. 10 shows such exemplary learning
curves of MaDL’s GT-ACC and AP-ACC compared to the LB and UB for the dataset
letter. Each curve represents the means and standard deviations over five
runs. The trend of improving performance with an increasing number of
annotated instances is confirmed for both evaluation scores. Further, we
observe that MaDL improves upon the LB, even for smaller training sets. Our
findings, coupled with the case study on varying annotation ratios in Appendix
E, suggest that an increase in both the number of annotated instances and the
number of annotations per instance generally contributes to improved results.
Recommendation: If the annotation budget allows, increase the number of
annotated instances for better results.

GT-ACC $\uparrow$

Number of Annotated InstancesNumber of Annotated Instances

AP-ACC $\uparrow$

Figure 10: Exemplary learning curves on the letter dataset.

fnum@@descivitemNumber of classes:

With an increasing number of classes, there is a growing chance that some
classes will be easy to separate and some classes will be highly confusable
(Gupta et al., 2014). Detecting such pairs of confusable classes requires the
accurate estimation of annotators’ confusion matrices, whose concept is
illustrated in Fig. 11. The three matrices represent three variants of MaDL
with varying assumptions about APs (cf. property P1 in Section 3). The class-
independent variant corresponds to estimating a single scalar as the only
degree of freedom, i.e., $\nu=1$, to build the entire confusion matrix. In our
example, this scalar corresponds to $0.80$ as an annotator’s correctness
probability. Accordingly, this variant cannot distinguish between easy- and
difficult-to-recognize classes. The degree of freedom of the partially class-
dependent variant equals the number of classes, i.e., $\nu=C$, because there
is an individual correctness probability for each class. These probabilities
build the diagonal of the confusion matrix, which consists of the values
$0.60,0.80$, and $1.00$ in our example. Only the fully class-dependent variant
can capture pairs of highly confusable classes. As a result, the degree of
freedom is considerably higher with $\nu=C\cdot(C-1)$. On the one hand, our
experiments show that this variant is robust across different datasets (cf.
Section 5), and even works for datasets with $C=100$ classes (cf. Appendix F).
On the other hand, the complexity increases quadratically with the number of
classes, so this variant is computationally costly for datasets with hundreds
or thousands of classes. Note that MaDL can estimate these confusion matrices
not only annotator-dependent but also instance-dependent. Recommendation: If
no assumptions about the annotators are known, and the computational effort is
feasible, use the fully class-dependent MaDL variant.

Class-independent (I)
---
$\nu=1$
| Annotated Class
GT Class | $z=1$ | $z=2$ | $z=3$
$y=1$ | $0.80$ | $0.10$ | $0.10$
$y=2$ | $0.10$ | $0.80$ | $0.10$
$y=3$ | $0.10$ | $0.10$ | $0.80$
Partially Class-dependent (P)
---
$\nu=C=3$
| Annotated Class
GT Class | $z=1$ | $z=2$ | $z=3$
$y=1$ | $0.60$ | $0.20$ | $0.20$
$y=2$ | $0.10$ | $0.80$ | $0.10$
$y=3$ | $0.00$ | $0.00$ | $1.00$
Fully Class-dependent (F)
---
$\nu=C\cdot(C-1)=3\cdot 2=6$
| Annotated Class
GT Class | $z=1$ | $z=2$ | $z=3$
$y=1$ | $0.60$ | $0.35$ | $0.05$
$y=2$ | $0.20$ | $0.80$ | $0.00$
$y=3$ | $0.00$ | $0.00$ | $1.00$
Figure 11: Illustrative confusion matrices for $C=3$ classes.

fnum@@descivitemNumber of annotators:

As the number of annotators increases, the AP model usually needs to learn
more varying annotation patterns. For this, the AP model’s size must be
sufficiently large, e.g., by increasing the annotator embeddings’
dimensionality. Moreover, in settings with many annotators, prior information
about the annotators can be particularly helpful for learning correlations
between them. Recommendation: Ensure that the AP model has a sufficient size
to learn annotation patterns.

fnum@@descivitemTraining, validation, and testing:

In principle, MaDL can be trained without a validation or test set,
demonstrating robust results with default hyperparameters across diverse
datasets. Nevertheless, like most DNNs, careful selection of the optimizer’s
hyperparameters, such as the learning rate, is essential. Without a validation
set, the training loss curve can serve as a rudimentary guide to assess if the
loss is being minimized effectively. Still, better results typically stem from
hyperparameter tuning using a validation set. Next to popular regularization
techniques, e.g., dropout and weight decay, such a validation set is also
essential to detect and avoid overfitting. Finally, a separate test set allows
us to assess MaDL’s final performance reliably. The setting of error-prone
annotators makes obtaining a validation or test set challenging. Ideally, we
obtain the GT labels from experts. Alternatively, we may assume the majority
votes of numerous annotators as GT labels. Recommendation: For hyperparameter
tuning, avoidance of overfitting, and a reliable performance assessment,
consider acquiring a validation and test set before deploying MaDL.

fnum@@descivitemDeployment:

After training, MaDL’s GT and AP models offer various deployment options: We
can

  * •

directly apply the GT model as a downstream classifier to the associated
learning task,

  * •

use the GT model’s predictions to improve the dataset’s label quality,

  * •

study the annotation patterns and correlations between annotators via the AP
model,

  * •

or leverage the AP model to assess which annotator is best for annotating a
particular instance.


# A deep learning method for solving high-order nonlinear soliton equation

Shikun Cui  Zhen Wang<EMAIL_ADDRESS>Jiaqi Han  Xinyu Cui  School of
Mathematical Sciences, Dalian University of Technology, Dalian, 116024, China
Key Laboratory for Computational Mathematics and Data Intelligence of Liaoning
Province, Dalian, 116024, China

###### Abstract

We propose effective scheme of deep learning method for high-order nonlinear
soliton equation and compare the activation function for high-order soliton
equation. The neural network approximates the solution of the equation under
the conditions of differential operator, initial condition and boundary
condition. We apply this method to high-order nonlinear soliton equation, and
verify its efficiency by solving the fourth-order Boussinesq equation and the
fifth-order Korteweg de Vries equation. The results show that deep learning
method can solve the high-order nonlinear soliton equation and reveal the
interaction between solitons.

###### keywords:

deep learning method, physics-informed neural networks, high-order nonlinear
soliton equation, interaction between solitons, numerical driven solution.

2.5cm2.5cm1cm2cm

##  1 Introduction

Nonlinear soliton equation is an important part of the field of Mathematical
Physics, which is used to describe the state or process changing with time in
physics, mechanics or other natural sciences[1][2]. As the carrier of soliton
theory, the development of nonlinear equation has always been the focus of
mathematical physics researchers. In recent years, a deep learning numerical
method has been developed to solve many problems related to nonlinear
evolution equation. Deep learning method approximate potential solutions by
using deep neural network , which is usually more effective than ordinary
numerical methods[3][4][5]. Raissi M et al[3] proposed physics-informed neural
networks to solve partial differential equation, such neural networks are
constrained to respect any symmetries, invariances, or conservation
principles. Han et al[6] reconstruct partial differential equations from
backward stochastic differential equations, and use neural networks to
approximate the gradient of unknown solutions to solve general high-
dimensional parabolic partial differential equations. Justin et al[7] used
DGM( depth Galerkin method) to study the numerical driven solution of high
dimensional partial differential equation. Li and Chen[8][9] used deep
learning method to solve many second-order nonlinear evolution equations and
many third-order nonlinear evolution equations, such as KdV equation, Burgers
equation. They also proposed a new residual neural network to solve the sine-
Gordon equation[10]. Wang and Yan[11] study the data-driven solutions of the
defocused nonlinear NLS equation by using PINNs. Marcucci G et al[12] studied
theoretically artificial neural networks with a nonlinear wave as a reservoir
layer and developed a new computing model driven by nonlinear partial
differential equations.

At present, deep learning method is only used to solve low-order nonlinear
problems or low-order linear problems,and its applicability to high-order
nonlinear problems is undiscovered. We apply the deep learning method and
physics-informed neural networks[3] to solve high-order nonlinear equation and
show the efficiency and effectiveness. Specifically, we will study the
numerical driven solutions of the fourth-order Boussinesq equation and the
fifth-order KdV equation.

Boussinesq equation, as a kind of nonlinear equation closely related to wave
phenomena, has been widely studied in many fields of
physics[13][14][15][16][17]. The classical Boussinesq equation describes the
evolution of waves in shallow water[13]. Some progress has been made in the
analytical solution of Boussinesq equation[18][19][20][20][21].

KdV equation has always been an important part of nonlinear mathematical and
physical model, and it is also a hotspot of numerical methods[25][26][27]. The
numerical solution of the fifth-order KdV equation has also made some
progress[28][29][30].The numerical driven solution of KdV equation has been
solved by Raissi M et al[3] and Li, Chen[9], Raissi M et al used discrete time
model to solve KdV equation, Li and Chen used continuous time model to solve
KdV equation. However, due to the complexity of high-order nonlinear equation,
the numerical driven solution of high-order nonlinear problem have not been
solved. We will illustrate the applicability of the deep learning method to
the fifth-order nonlinear soliton equation by solving the fifth-order KdV
equation.

The paper is organized as follows. In section 2, we will introduce the method
of deep learning to solve high-order equations. In section 3, we use the deep
learning method to reproduce the one-soliton and the two-soliton numerical
driven solution of the fourth-order Boussinesq equation. In numerical driven
solution, we find the dynamic behavior of solitons interaction. Specifically,
we solve the chasing-soliton and collising-soliton of Boussinesq equation, and
find the dynamic behavior between solitons from the numerical driven solution.
In section 4, we get the one-soliton numerical driven solution and the two-
soliton numerical driven solution of the fifth-order KdV equation. In the
numerical driven solution, we also find the dynamic behavior of solitons
interaction. Finally, some concluding discussions and remarks are contained in
section 5.

##  2 Method

We consider the following form of (1 + 1)-dimensional fourth-order and fifth-
order nonlinear soliton equation,

$\displaystyle\chi_{1}(u_{t},u_{tt})=\chi_{2}(u,u_{x},u_{xx},u_{xxx},u_{xxxx},u_{xxxxx}),$
(1)

and solve their soliton solution, where the subscripts $t$ and $x$ denote the
partial derivatives, and $\chi_{1}$ is a linear function of the time
derivative of $u(x,t)$, $\chi_{2}$ is a nonlinear function of $u(x,t)$ and its
partial derivative to space variable $x$. Specifically, we build a multi-layer
neural network to approximate the potential solution, and use the automatic
differentiation technique to obtain its derivatives in time and space[31].

The residual network is defined

$\displaystyle
f=\chi_{1}(u_{t},u_{tt})-\chi_{2}(u,u_{x},u_{xx},u_{xxx},u_{xxxx},u_{xxxxx}).$
(2)

The shared parameters of neural networks can be learned by minimizing the loss
of mean square error,

$\displaystyle LOSS=LOSS_{u}+LOSS_{f},$ (3) $\displaystyle
LOSS_{u}=\frac{1}{N_{u}}\sum_{n=1}^{N_{u}}|u(t^{i}_{u},x^{i}_{u})-u^{i}|\ ,$
(4) $\displaystyle
LOSS_{f}=\frac{1}{N_{f}}\sum_{n=1}^{N_{f}}|f(t^{i}_{f},x^{i}_{f})|\ .$

$LOSS_{u}$ is the mean square error of initial and boundary, $LOSS_{f}$ is the
internal calculation error, $t^{i}_{u}$ and $x^{i}_{u}$ represent the initial
and boundary training values, $t^{i}_{f}$ and $x^{i}_{f}$ are the point
collected in $f$. $N_{u}$ is the total number of selected boundary points and
initial points, $N_{f}$ is the total number of selected internal collection
points. A common deep feedforward neural fully connected network used to deal
with high-order nonlinear problems. Figure 1 shows the framework of physics-
informed neural networks. There are activation function, weights and bias
between each layer. PINNs updates weights and biases by reducing error $LOSS$,
and the neural network stops operation until the error is lower than the
specified standard.

Figure 1: Architecture of physics-informed neural networks

Where $k$ represent the number of hidden layers, $n_{k}$ represents the number
of neurons corresponding to the hidden layer. We will improve the neural
network from the following aspects to ensure that PINNs can deal with high-
order nonlinear problems. Firstly, we will determine the appropriate number of
hidden layers and the corresponding number of neurons. Secondly, we will
improve the efficiency of neural network by improving the activation function.

In high-order nonlinear problem, it is very important to select the number of
hidden layers and neurons in each layer. If the selected neural network
structure is not suitable, there will occur over fitting( the deep learning
model is superior in training set, but the final result is not good), under
fitting( the deep learning model does not capture the characteristics of the
data well and can not solve the equation well), loss of operational efficiency
and other problems easily. Compared with low-order problems, high-order
nonlinear problems are more difficult due to the complexity of their equation.
Through a large number of experiments, we decide to use a neural network with
four hidden layers to deal with high-order nonlinear soliton equation.

On the selection of activation function, Chen and Li[9] proved that
trigonometric function as activation function is effective on solitary wave
solution of third-order nonlinear soliton equation. In this paper, we will
study the deep learning algorithm of high-order nonlinear soliton equation and
explore the effectiveness of trigonometric function as activation function for
high-order nonlinear soliton equation and find the most suitable activation
function. In addition, we use L-BFGS optimization algorithm[31] to set all
parameters of the target to minimize the loss function Equation (3). All
numerical examples reported here are run on a Dell computer with Intel Xeon
Gold 6320R i5 processor and 32 GB memory.

##  3 Boussinesq equation

Boussinesq equation with Dirichlet boundary condition and initial condition
given by

$\left\\{\begin{array}[]{lr}u_{tt}-u_{xx}-u_{xxxx}-3(u^{2})_{xx}=0,x\in[-20,20],t\in[-5,5],\\\
u(x,-5)=u_{0}(x),\\\ u(-20,t)=u(20,t),\end{array}\right.$ (5)

where $u_{0}(x)$ is a given real valued smooth function.

We use the deep learning method to find the one-soliton and two-soliton
numerical driven solution of equation (5) with tanh as activation function,
and try to reproduce the dynamic behavior between solitons.

###  3.1 One-soliton solution

In this subsection, the numerical driven one-soliton solution of Boussinesq
equation will be solved. The one-soliton analytic solution of Boussinesq
equation can be obtained by using Hirota method[18][19],

$\displaystyle
u(x,t)=\frac{k_{1}^{2}}{2}sech^{2}\Bigg{(}\frac{k_{1}x+\sqrt{k_{1}^{2}+k_{1}^{4}}t}{2}+\xi_{0}\Bigg{)}.$
(6)

We could set $k_{1}$ = 1, $\xi_{0}=0$. The corresponding initial condition
becomes

$\displaystyle
u_{0}(x)=\frac{1}{2}sech^{2}\Bigg{(}\frac{x-5\sqrt{2}}{2}\Bigg{)}.$ (7)

We generate the data of 201 snapshots directly on regular space-time grid with
$\triangle t$ = 0.05s. A small training data subset is generated by randomly
latin hypercube sampling method[24], the number of collection points are
$N_{u}$ = 100, $N_{f}$ = 20000. The latent solution $u(t,x)$ can be learned by
minimizing the loss function Equation (3). Top panel of Figure 2 shows
comparison of the predicted spatiotemporal solution and exact solution. The
model achieves a relative $L^{2}$ error of size $2.0\times 10^{-2}$ in a
runtime of 152s. The model is iterated 170 times to complete the operation.
Bottom panel of Figure 2 shows the detailed comparison of exact solution and
predicted spatiotemporal solution at different times t = -2.5, t = 0, t = 2.5
respectively. One-soliton solution of Boussinesq equation is reconstructed by
using deep learning method accurately. From Figure 3, we can clearly observe
the reconstructed solitary wave motion.

(a) One-soliton Evolution of Exact Solution (b) One-soliton Evolution of
Learned Solution

(c) $t=-2.5$ (d) $t=0$ (e) $t=2.5$

Figure 2: Subgraph (a) and (b) are comparison of one-soliton exact solution
and learned solution of Boussinesq equation, subgraph (c)-(e) are the detailed
comparison of exact solution and learned spatiotemporal solution at the
specific time.

(a) One-soliton Numerical Driven Solution (b) One-soliton Exact Solution

Figure 3: Spatiotemporal evolution of one-soliton numerical driven solution
and exact solution of Boussinesq equation.

In order to verify the universality of our neural network architecture for the
one-soliton solution of Boussinesq equation, we try to change the value of
$k_{1}$ and give the numerical solution respectively. The results shows that
our neural network architecture is effective in solving one-soliton of
Boussinesq equation.

Table 1: The results of different one-soliton solution of Boussinesq equation calculated by deep learning method. $k_{1}$ | 0.8 | 0.9 | 1.0 | 1.1 | 1.2 | 1.3 | 1.4
---|---|---|---|---|---|---|---
$L^{2}$ error | $2.79\times 10^{-2}$ | $3.36\times 10^{-2}$ | $2.03\times 10^{-2}$ | $1.9\times 10^{-2}$ | $1.67\times 10^{-2}$ | $1.60\times 10^{-2}$ | $1.62\times 10^{-2}$
Time(s) | 224 | 191 | 152 | 160 | 294 | 284 | 480
Iterations | 141 | 155 | 170 | 195 | 576 | 416 | 753

###  3.2  Two-soliton solution

In this subsection, we will calculate the numerical driven two-soliton
solution of the Boussinesq equation and reproduce the solitons interaction
process. The two-soliton solution of Boussinesq equation is given[18][19].

$\displaystyle
u(x,t)=2\frac{k_{1}^{2}e^{k_{1}x+\omega_{1}t+\delta_{1}}+k_{2}^{2}e^{k_{2}x+\omega_{2}t+\delta_{2}}+(k_{1}+k_{2})^{2}e^{(k_{1}+k_{2})x+(\omega_{1}+\omega_{2})t+\delta_{1}+\delta_{2}+\delta_{0}}}{1+e^{k_{1}x+\omega_{1}t+\delta_{1}}+e^{k_{2}x+\omega_{2}t+\delta_{2}}+e^{(k_{1}+k_{2})x+(\omega_{1}+\omega_{2})t+\delta_{1}+\delta_{2}+\delta_{0}}}$
(8)
$\displaystyle-2\frac{(k_{1}e^{k_{1}x+\omega_{1}t+\delta_{1}}+k_{2}e^{k_{2}x+\omega_{2}t+\delta_{2}}+(k_{1}+k_{2})e^{(k_{1}+k_{2})x+(\omega_{1}+\omega_{2})t+\delta_{1}+\delta_{2}+\delta_{0}})^{2}}{(1+e^{k_{1}x+\omega_{1}t+\delta_{1}}+e^{k_{2}x+\omega_{2}t+\delta_{2}}+e^{(k_{1}+k_{2})x+(\omega_{1}+\omega_{2})t+\delta_{1}+\delta_{2}+\delta_{0}})^{2}},$
$\displaystyle
e^{\delta_{0}}=-\frac{(\omega_{1}-\omega_{2})^{2}-(k_{1}-k_{2})^{2}-(k_{1}-k_{2})^{4}}{(\omega_{1}+\omega_{2})^{2}-(k_{1}+k_{2})^{2}-(k_{1}+k_{2})^{4}},$

where $\delta_{1}$ and $\delta_{2}$ is constant, $\omega_{1}$and $\omega_{2}$
meet the conditions $\omega_{1}^{2}=k_{1}^{2}+k_{1}^{4}$ ,
$\omega_{2}^{2}=k_{2}^{2}+k_{2}^{4}$ respectively.

Two-soliton solution of Boussinesq equation have two states, colliding-
soliton( the two solitons have different directions) and chasing-soliton( the
two solitons have the same direction and different amplitude). We will find
the numerical driven solution of the two forms respectively, and study their
dynamic behavior and the interaction between solitons.

####  3.2.1 Colliding-soliton

We just set $k_{1}$ = $k_{2}$ = 1.1, $\delta_{1}$ = $\delta_{2}$ =0,
$\omega_{1}$and $\omega_{2}$ have opposite sign. We generate the data of 201
snapshots directly on regular space-time grid with $\triangle t$ = 0.05s. A
small training data subset is generated by randomly latin hypercube sampling
method[24], the number of collection points are $N_{u}$ = 100, $N_{f}$ =
25000. The latent solution $u(x,t)$ is learned by minimizing the loss function
Equation (3). Top panel of Figure 4 shows the comparison of predicted
spatiotemporal solution and exact solution. Bottom panel of Figure 4 shows the
detailed comparison of actual solution and predicted spatiotemporal solution
at different time t = -4.5, t = 0, t = 2.5 respectively. The specific
spatiotemporal evolution of colliding-soliton of Boussinesq equation is given
in Figure 5. The model achieves a relative $L^{2}$ error of size $7.63\times
10^{-2}$ in a runtime of 430s. The model is iterated 578 times to complete the
operation.

(a) Colliding-soliton Interaction of Exact Solution (b) Colliding-soliton
Interaction of Learned Solution

(c) $t=-4.5$ (d) $t=0$ (e) $t=2.5$

Figure 4: Subgraph (a) and (b) are comparison of colliding-soliton exact
solution and learned solution of Boussinesq equation, subgraph (c)-(e) are the
detailed comparison of exact solution and learned spatiotemporal solution at
the specific time.

(a) Colliding-soliton Numerical Driven Solution (b) Colliding-soliton Exact
Solution

Figure 5: Spatiotemporal evolution of colliding-soliton numerical driven
solution and exact solution.

From Figure 5, we learn the spatiotemporal evolution process of separation-
fusion-separation of colliding-soliton. Amplitude becomes high during the
colliding process and shape remains unchanged before and after the
interaction, which is consistent with the known fact. The ‘phase shift’
phenomenon also occur in numerical driven solution.

In order to verify the universality of our neural network architecture for the
colliding-soliton, we calculate the different colliding-soliton solutions of
Boussinesq equation by deep learning method. The result shows that the deep
learning method is effective in solving the colliding-soliton solution of
Boussinesq equation.

Table 2: The results of different colliding-soliton solution of Boussinesq equation calculated by deep learning method. $k_{1}$ | 0.8 | 0.9 | 1.0 | 1.1 | 1.2
---|---|---|---|---|---
$k_{2}$ | 0.8 | 0.9 | 1.0 | 1.1 | 1.2
$L^{2}$ error | $8.55\times 10^{-2}$ | $8.10\times 10^{-2}$ | $1.16\times 10^{-1}$ | $7.63\times 10^{-2}$ | $7.64\times 10^{-2}$
time(s) | 358 | 551 | 642 | 430 | 1065
Iterations | 320 | 635 | 801 | 578 | 755

####  3.2.2 Chasing-soliton

We could set $k_{1}$ =1.5, $k_{2}$ = 0.9, $\delta_{1}$ = $\delta_{2}$ =0,
$\omega_{1}$and $\omega_{2}$ are positive. In this condition, two-solitons
have same direction and different magnitude, so the soliton chasing phenomenon
occurs. We generate the data of 201 snapshots directly on the regular space-
time grid with $\triangle t$ = 0.05s. A small training data subset is
generated by randomly latin hypercube sampling method[24], the number of
collection points are $N_{u}$ = 100, $N_{f}$ = 25000. Top panel of 6 shows the
comparison of predicted spatiotemporal solution and exact solution. Bottom
panel of Figure 6 shows the detailed comparison of exact solution and learned
spatiotemporal solution at different time t = -4.5, t = 0, t = 2.5
respectively. The model achieves a relative $L^{2}$ error of size $6.73\times
10^{-2}$ in a runtime of 892s.

Figure 7 shows the spatiotemporal evolution process of separation-fusion-
separation. Amplitude becomes low during the fusion process and shape remains
unchanged before and after the interaction, which is consistent with the known
fact. We also observe the ‘phase shift’ phenomenon in the chasing-soliton
numerical driven solution.

(a) Chasing-soliton Interaction of Exact Solution (b) Chasing-soliton
Interaction of Learned Solution

(c) $t=-4.5$ (d) $t=0$ (e) $t=2.5$

Figure 6: Subgraph (a) and (b) are comparison of chasing-soliton exact
solution and learned solution of Boussnesq equation, subgraph (c)-(e) are the
detailed comparison of exact solution and learned spatiotemporal solution at
the specific time.

(a) Numerical Driven Solution (b) Exact Solution

Figure 7: Spatiotemporal evolution of chasing-soliton numerical driven
solution and exact solution.

In exploring the effectiveness of activation function, we find tanh function
is more effective than trigonometric function. We take one-soliton solution as
an example, the calculation results is shown in Tabel 3. We find that both
tanh and trigonometric function is useful in Boussinesq equation, and tanh
cost less computational source. Compared with one-soliton solution, the two-
soliton solution of Boussinesq equation cost more computational source.

Table 3: The results of different activation function of one-soliton calculated by deep learning method. activation function | tanh | cos | sin | sigmoid | relu
---|---|---|---|---|---
$L^{2}$ error | $1.97\times 10^{-2}$ | $1.86\times 10^{-1}$ | $1.27\times 10^{-1}$ | $9.22\times 10^{-1}$ | $8.37\times 10^{-1}$
time(s) | 160 | 1670 | 1531 | 105 | 16
Iterations | 195 | 4722 | 3080 | 0 | 4

##  4 Fifth-order KdV equation

In this section, we consider the fifth-order KdV equation with Dirichlet
periodicity boundary condition and initial condition,

$\left\\{\begin{array}[]{lr}u_{t}+(\alpha u_{xxxx}+\beta uu_{xx}+\gamma
u^{3})_{x}=0,x\in[-10,10],t\in[0,2\pi],\\\ u(x,0)=u_{0}(x),\\\
u(-10,t)=u(10,t),\end{array}\right.$ (9)

where $\alpha$, $\beta$, $\gamma$ are arbitrary constant, $u_{0}(x)$ is a
given real valued smooth function.

We choose cos as activation function, and explore the effectiveness of
trigonometric function as activation function of the fifth-order KdV equation.
A deep learning method is used to find the one-soliton and two-soliton
solution of the equation, and reproduce the dynamic behavior between the
solitons.

###  4.1 One-soliton solution

Using Hirota bilinear method, the one-soliton analytical solution of fifth-
order KdV equation (9) can be obtained[23],

$\displaystyle u(x,t)=\frac{15\alpha
k_{1}^{2}}{2\beta}sech^{2}\Bigg{(}\frac{k_{1}x-{\alpha
k_{1}^{5}}t+\xi_{0}}{2}\Bigg{)}.$ (10)

We set $\alpha$=1, $\beta$=15, $\gamma$=15, $k_{1}$ = 1, $\xi_{0}=3$, the
equation is also called C-D-J-K equation. Correspondingly,

$\displaystyle u_{0}(x)=\frac{1}{2}sech^{2}\Bigg{(}\frac{x}{2}-1\Bigg{)}.$
(11)

In order to obtain high-precision data set, we generate the data of 201
snapshots directly on the regular space-time grid with $\Delta t$ = 0.05s. A
small training data subset is generated by randomly latin hypercube sampling
method [24], the number of collection points are $N_{u}$ = 100, $N_{f}$ =
20000. Top panel of Figure 8 shows the comparison of predicted spatiotemporal
solution and exact solution, and bottom panel of Figure 8 shows the detailed
comparison of exact solution and predicted spatiotemporal solution at
different time t=1.57, t=3.14, 4.71 respectively. Figure 9 shows specific
spatiotemporal evolution of one-soliton solution of fifth-order KdV equation.
The model achieves a relative $L^{2}$ error of size $1.36\times 10^{-2}$ in a
runtime of 537s. The model is iterated 803 times to complete the operation.

(a) One-soliton Evolution of Exact Solution (b) One-soliton Evolution of
Learned Solution

(c) $t=1.57$ (d) $t=3.14$ (e) $t=4.71$

Figure 8: Subgraph (a) and (b) are comparison of one-soliton exact solution
and learned solution of fifth-order KdV equation, subgraph (c)-(e) are the
detailed comparison of exact solution and learned spatiotemporal solution at
the specific time.

(a) One-soliton Numerical Driven Solution (b) One-soliton Exact Solution

Figure 9: Spatiotemporal evolution of one-soliton numerical driven solution
and exact solution of fifth-order KdV equation

From Figure 8 and Figure 9, we learn that velocity and shape of the numerical
driven solution of the one-soliton remain unchanged during the motion, which
shows the dynamic behavior of the soliton well.

To verify the universality of our neural network architecture for the one-
soliton numerical driven solution of fifth-order KdV equation, we calculate
the different one-soliton solution of fifth-order KdV equation.The result
shows that the our neural network architecture is very effective in solving
one-soliton solution of fifth-order KdV equation.

Table 4: The results of different one-soliton solution of fifth-order KdV equation calculated by deep learning method. $k_{1}$ | 0.9 | 0.95 | 1.0 | 1.05 | 1.1
---|---|---|---|---|---
$L^{2}$ error | $1.47\times 10^{-2}$ | $2.57\times 10^{-2}$ | $1.36\times 10^{-2}$ | $4.64\times 10^{-2}$ | $2.36\times 10^{-2}$
time(s) | 687 | 680 | 537 | 600 | 1166
Iterations | 300 | 803 | 803 | 883 | 1604

###  4.2 Two-soliton solution

In order to observe the solitons interaction behavior well, we set
$x\in[-20,20]$ and $t\in[-5,5]$. Equation (9) becomes

$\left\\{\begin{array}[]{lr}u_{t}+(u_{xxxx}+15uu_{xx}+15u^{3})_{x}=0,x\in[-20,20],t\in[-5,5],\\\
u(x,0)=u_{0}(x),\\\ u(-20,t)=u(20,t).\end{array}\right.$ (12)

By using Hirota bilinear method, the two-soliton analytical solution of fifth-
order KdV Equation (12) can be obtained[23]

$\displaystyle
u(x,t)=2\frac{k_{1}^{2}e^{k_{1}x+\omega_{1}t+\delta_{1}}+k_{2}^{2}e^{k_{2}x+\omega_{2}t+\delta_{2}}+(k_{1}+k_{2})^{2}e^{(k_{1}+k_{2})x+(\omega_{1}+\omega_{2})t+\delta_{1}+\delta_{2}+\delta_{0}}}{1+e^{k_{1}x+\omega_{1}t+\delta_{1}}+e^{k_{2}x+\omega_{2}t+\delta_{2}}+e^{(k_{1}+k_{2})x+(\omega_{1}+\omega_{2})t+\delta_{1}+\delta_{2}+\delta_{0}}}$
(13)
$\displaystyle-2\frac{(k_{1}e^{k_{1}x+\omega_{1}t+\delta_{1}}+k_{2}e^{k_{2}x+\omega_{2}t+\delta_{2}}+(k_{1}+k_{2})e^{(k_{1}+k_{2})x+(\omega_{1}+\omega_{2})t+\delta_{1}+\delta_{2}+\delta_{0}})^{2}}{(1+e^{k_{1}x+\omega_{1}t+\delta_{1}}+e^{k_{2}x+\omega_{2}t+\delta_{2}}+e^{(k_{1}+k_{2})x+(\omega_{1}+\omega_{2})t+\delta_{1}+\delta_{2}+\delta_{0}})^{2}},$
$\displaystyle\omega_{1}=-k_{1}^{5},\omega_{2}=-k_{2}^{5},e^{\delta_{0}}=\frac{(k_{1}-k_{2})^{2}+(k_{1}^{2}-k_{1}k_{2}+k_{2}^{2})}{(k_{1}+k_{2})^{2}+(k_{1}^{2}+k_{1}k_{2}+k_{2}^{2})}.$

We could set $k_{1}$ = 1, $k_{2}$ = 0.8, $\xi_{1}=\xi_{2}=0$. In order to
obtain high-precision data set, we generate the data of 201 snapshots directly
on the regular space-time grid with $\Delta t$ = 0.05s. A small training data
subset is generated by randomly latin hypercube sampling [24], the number of
collection points are $N_{u}$= 100, $N_{f}$= 20000. Top panel of Figure 10
shows the comparison of predicted spatiotemporal solution and exact solution.
Bottom panel of Figure 10 shows the detailed comparison of exact solution and
predicted spatiotemporal solution at different time t = -4.5, t = 0, t = 2.5
respectively. Specific spatiotemporal evolution of two-soliton is given in
Figure 11. The model achieves a relative $L^{2}$ error of size $7.22\times
10^{-2}$ in a runtime of 1774s. The model is iterated 3606 times to complete
the operation.

(a) Two-soliton Interaction of Exact Solution (b) Two-soliton Interaction of
Learned Solution

(c) $t=-4.5$ (d) $t=0$ (e) $t=2.5$

Figure 10: Subgraph (a) and (b) are comparison of two-soliton exact solution
and learned solution of fifth-order KdV equation, subgraph (c)-(e) are the
detailed comparison of exact solution and learned spatiotemporal solution at
the specific time.

(a) Two-soliton Numerical Driven Solution (b) Two-soliton Exact Solution

Figure 11: Spatiotemporal evolution of two-soliton numerical driven solution
and exact solution of fifth-order KdV equation

On the selection of activation function, we try a lot of other activation
functions of two-soliton, the results is given in Figure 5. The result shows
that the trigonometric function is more effective in solving the fifth-order
KdV equation.

Table 5: The results of different activation function of two-soliton calculated by deep learning method. activation function | tanh | cos | sin | sigmoid | relu
---|---|---|---|---|---
$L^{2}$ error | $1.46\times 10^{-1}$ | $7.22\times 10^{-2}$ | $1.27\times 10^{-1}$ | $8.18\times 10^{-1}$ | $6.13\times 10^{-1}$
time(s) | 623 | 1740 | 1531 | 237 | 48
Iterations | 1390 | 3600 | 3080 | 0 | 5

In addition, a small amplitude noise is given to verify the rationality of the
structure of the fifth-order KdV equation neural network. The results show
that the deep learning method is effective in solving two-soliton solutions of
fifth-order KdV equation.

Table 6: The results of different two-soliton solution of fifth-order KdV equation calculated by deep learning method. $k_{1}$ | 1.0 | 0.99 | 1.01 | 1.0 | 1.0
---|---|---|---|---|---
$k_{2}$ | 0.8 | 0.8 | 0.8 | 0.79 | 0.81
$L^{2}$ error | $7.22\times 10^{-2}$ | $8.27\times 10^{-2}$ | $1.05\times 10^{-1}$ | $9.05\times 10^{-2}$ | $1.0\times 10^{-1}$
time(s) | 1774 | 913 | 1011 | 1000 | 657
Iterations | 3606 | 1654 | 1738 | 2224 | 1240

##  5 Summary and discussion

We find a neural network architecture of PINNs suitable for solving high-order
nonlinear soliton equation. Specifically, it has four hidden layers, the
number of corresponding neurons are 256, 128, 64, 32 and 40, 40, 40, 40. And
we study the numerical driven solution of high-order nonlinear problems (
fourth-order Boussinesq equation and fifth-order KdV equation ), and control
$L^{2}$ error to $10^{-2}$ magnitude, which shows that the deep learning
method is effective. We extend the deep learning method to the solution of
fourth-order and fifth-order equation, but the ability of deep learning method
to deal with higher-order equation still needs to be explored. We summarize
the conclusions as follows. Firstly, the deep learning method is suitable to
solve the soliton solution of fourth-order Boussinesq equation and fifth-order
KdV equation. The deep learning method can recover the dynamic behavior of
solitons in high-order nonlinear soliton equation. From numerical driven
solution, we can observe the ‘phase shift’ phenomenon, and the shape remains
unchanged after the interaction, which is consistent with known facts.
Secondly, trigonometric function is effective in high-order nonlinear
problems. Compared with the low-order problem, high-order problems have higher
sensitivity in selection of architecture of neural network.

## Acknowledgment

Project supported by LiaoNing Revitalization Talents Program (XLYC1907014) and
“the Fundamental Research Funds for the Central Universities” (DUT21ZD205).

## Reference

## References

  * [1] Mayers J 2008 Phys. Rev. A 78 3 033618
  * [2] Manojlovic N and Marugan GAM 1995 Int. J. Mod. Phys. D 4 6 749-766
  * [3] Raissi M, Perdikaris P and Karniadakis G E 2017 J. Comput.Phys. 348 686-707
  * [4] Lagaris I E, Likas A and Fotiadis D I 1998 IEEE Trans. Neural Networks 9 987-1000
  * [5] Raissi M and Karniadakis G E 2018 J. Comput. Phys. 357 125-141
  * [6] Han J, Jentzen A and Weinan E 2018 Proc. Natl. Acad. Sci. 115 8505-8510
  * [7] Sirignano J and Spiliopoulos K 2018 J.Comput.Phys. 375 1339-1364
  * [8] Li J and Chen Y 2020 Commun. Theor. Phys. 72 105005
  * [9] Li J and Chen Y 2020 Commun. Theor. Phys. 72 115003
  * [10] Li J and Chen Y 2021 Commun. Theor. Phys. 73 015001
  * [11] Wang L and Yan ZY 2021 Phys. Lett. A 404 127408
  * [12] Marcucci G, Pierangeli D and Conti C 2020 Phys. Rev. Lett. 125 9 093901
  * [13] Boussinesq J 1872 J. Math. Pures Appl. 17 55-108
  * [14] Ursell F 1953 Proc. Cambridge Philos. Soc. 49 4 685-694
  * [15] Lu CN, Fu C and Yang HW 2018 Appl. Math. Comput. 327 104-116
  * [16] Guo BX, Gao ZJ and Lin J 2016 Commun. Theor. Phys. 66 6 589-594
  * [17] Himonas AA and Mantzavinos D 2015 J. Differ. Equ. 134 1 62-100
  * [18] Zhang Y, Ye LY, Lv YN and Zhao HQ 2007 J. Phys. A-Math. Theor. 40 21 5539-5549
  * [19] Hietarinta J 1987 J. Math. Phys. 28 8 1732-1742
  * [20] Zhang L, Zhang LF and Li CY 2008 Chin. Phys. B 17 2 403-410
  * [21] Clarkson PA 2008 Anal. Appl. 6 4 349-369
  * [22] Moleleki LD 2014 Adv. Math. Phys. 2014 672679
  * [23] Hietarinta J 1987 J. Math. Phys. 28 9 2094-2101
  * [24] Stein M L 1987 Technometrics 29 143-151
  * [25] Kuo PY and Wu HM 1981 J. Math. Anal. Appl. 82 2 334-345
  * [26] Vliegenthart Ac 1971 J. Eng. Math. 5 2 137-155
  * [27] Zhang YN, Tam HW and Hu XB 2014 J. Phys. A-Math. Theor. 47 4 045202
  * [28] Hu WP and Deng ZC 2008 Chin. Phys. B 17 11 3923-3929
  * [29] Ahmad H, Khan TA and Yao SW 2020 Open Math. 18 738-748
  * [30] Kaya D 2003 Appl. Math. Comput. 144 2-3 353-363
  * [31] Liu D C and Nocedal J 1989 Math. Program. 45 503-528