# Mixture of Experts based Multi-task Supervise Learning from Crowds

David S. Hippocampus
Department of Computer Science
Cranberry-Lemon University
Pittsburgh, PA 15213
<EMAIL_ADDRESS>
Use footnote for providing further information about author (webpage,
alternative address)— _not_ for acknowledging funding agencies.

###### Abstract

Existing truth inference methods in crowdsourcing aim to map redundant labels
and items to the ground truth. They treat the ground truth as hidden variables
and use statistical or deep learning-based worker behavior models to infer the
ground truth. However, worker behavior models that rely on ground truth hidden
variables overlook workers’ behavior at the item feature level, leading to
imprecise characterizations and negatively impacting the quality of truth
inference. This paper proposes a new paradigm of multi-task supervised
learning from crowds, which eliminates the need for modeling of items’s ground
truth in worker behavior models. Within this paradigm, we propose a worker
behavior model at the item feature level called Mixture of Experts based
Multi-task Supervised Learning from Crowds (MMLC). Two truth inference
strategies are proposed within MMLC. The first strategy, named MMLC-owf,
utilizes clustering methods in the worker spectral space to identify the
projection vector of the oracle worker. Subsequently, the labels generated
based on this vector are considered as the inferred truth. The second
strategy, called MMLC-df, employs the MMLC model to fill the crowdsourced
data, which can enhance the effectiveness of existing truth inference methods.
Experimental results demonstrate that MMLC-owf outperforms state-of-the-art
methods and MMLC-df enhances the quality of existing truth inference methods.

##  1 Introduction

Truth inference in crowdsourcing aims to derive accurate results from noisy
data provided by online workers. Existing truth inference methods can be
broadly classified into two categories: weakly supervised and supervised
approaches. In the weakly supervised approach, unknown ground truth is treated
as hidden variables. This involves utilizing statistics from workers’ noisy
answers to calculate results directly. Alternatively, it entails creating
worker behavior models and employing unsupervised learning methods such as the
EM algorithm to estimate unknown parameters and infer the ground truth. The
weakly supervised approach can further be classified into statistical learning
and deep learning methods based on whether it consider the features of the
items. Statistical learning methods, such as Majority voting (MV) Imamura et
al., (2018), Dawid&Skene model (DS) Dawid and Skene, (1979), and homologous
Dawid-Skene model (HDS) Karger et al., (2011); Li and Yu, (2014), do not
incorporate item features. In contrast, deep learning methods like Training
Deep Neural Nets Gaunt et al., (2016) take item features into account. In the
supervised approach, a classifier model is first constructed with item
features as the input and ground truth as the output. Then, a worker behavior
model is created based on a confusion matrix that establishes the relationship
between the item’s ground truth and the worker labels. On this basis,
supervised learning is implemented using the classifier model and the worker
behavior model by treating the worker labels as supervisory information.
Finally, the output of the classifier model is used as the inferred ground
truth. In recent years, various truth inference methods based on supervised
learning have been proposed, such as CrowdlayerRodrigues and Pereira, (2018),
CoNALChu et al., (2021), and UnionNetWei et al., (2022). However, the worker
behavior model based on the confusion matrix faces challenges in effectively
capturing variations in feature characteristics across different items.
Neglecting these variations in worker behavior under different conditions can
result in inaccurate representations of worker behavior, consequently
impacting the quality of truth inference. For example, in handwritten digit
recognition, workers generally have high accuracy. Suppose there are two
items: one closely resembles the digit “1,” but its ground truth is actually
“7,” and the other is a normal “7.” The former receives many labels as “1,”
while the latter rarely gets labeled as “1.” Under the worker behavior model
based on the confusion matrix, it is difficult to accurately model the
labeling behavior of such high-difficulty items. Therefore, there is a high
probability that the model will interpret the former with “1” as the ground
truth, leading to incorrect judgments. The quality of truth inference is
influenced by uncertainty from hidden variables, the method’s data
adaptability, and the accuracy in characterizing worker behavior. The purpose
of this paper is to develop a supervised model that can achieve high-quality
truth inference with a worker behavior model at the item feature level.

In this paper, we propose Multi-task Supervised Learning from Crowds (MLC), a
novel paradigm for crowdsourcing learning. Unlike traditional paradigm, MLC
does not rely on the ground truth of the items but instead focuses on
understanding the unique behavior of individual workers across different
items. When multiple workers handle the same item, they share the item’s
features, leading to a multi-task learning paradigm. Within this paradigm, we
propose a method called Mixture of Experts-based Multi-task Supervised
Learning from Crowds (MMLC). MMLC does not utilize a single classifier but
instead creates multiple expert modules. The outputs from these expert modules
serve as the _bases_ of the worker spectral space. Each worker is represented
by his or her projection vector in the spectral space that characterizes their
behavior. The worker behavior model provides a more precise depiction of their
behavior across different items by accurately modeling the workers’ behavior
on item features. However, it is important to note that the model itself
cannot determine the ground truth. To address this limitation, we introduce
two truth inference strategies based on MMLC. The first strategy involves
analyzing the distribution of workers’ projections in the worker spectral
space. We identify the projection of the oracle worker by applying clustering
methods, and consider its labels as the ground truth. This approach is
referred to as Oracle Worker Finding of MMLC (MMLC-owf). The second strategy
leverages the sparsity of crowdsourced data to fill the original dataset with
MMLC outputs, generating a new crowdsourcing dataset. Existing truth inference
methods can then be applied within this framework, which is called Data
Filling of MMLC (MMLC-df). The main contributions are as follows:

  * •

We introduce a novel paradigm of multi-task supervised learning from crowds
and propose a new worker behavior model called MMLC. This model is well-suited
for crowdsourcing learning and offers a more accurate way to characterize
worker behavior in item labeling.

  * •

We leverage MMLC to identify the oracle worker for labeling items as the
ground truth, referred to as MMLC-owf. Experimental results demonstrate that
the labels obtained using this method exhibit higher quality compared to
state-of-the-art methodes.

  * •

We introduce a truth inference framework called MMLC-df, which leverages the
MMLC model to fill sparse crowdsourced data. This framework then applies truth
inference methods to determine the ground truth. Experimental results
demonstrate that MMLC-df significantly enhances truth inference methods,
leading to higher-quality results.

##  2 Related Work

Weakly supervised approach: This class of methods focuses on modeling the
relationship between workers’ noisy answers and the ground truth. In this
approach, the ground truth is treated as a hidden variable, and weakly
supervised learning techniques are employed to infer the ground truth. The
most direct and widely used method is Majority Voting (MV) Sheng et al.,
(2017). It involves counting the responses from workers and considering the
answer with the majority of votes as the ground truth. However, MV overlooks
the variations in worker behavior during the labeling process and treats all
workers’ labels equally. To address this limitation, Tao et al. Tao et al.,
(2020) proposed a method that considers the deterministic information of
majority and minority categories separately. They establish a voting model
that takes into account the labeling quality of workers in different
situations. The Dawid-Skene (DS) Dawid and Skene, (1979) method utilizes a
confusion matrix to describe each worker’s behavior when handling an item. The
EM algorithm is then used to estimate the worker’s confusion matrix parameters
and the ground truth. HDS Raykar et al., (2010) assumes equal probabilities of
wrong worker selections while evolving the confusion matrix. GLAD Whitehill et
al., (2009) not only considers workers’ abilities but also accounts for the
difficulty of item processing. It creates a worker behavior model using a
sigmoid function and employs the EM algorithm to determine the ground truth of
an item. Some weakly supervised inference methods utilize deep learning
techniques to infer the ground truth. These methods typically use aggregation
algorithms to construct an initial answer table, which combines the noisy
labels provided by workers. Subsequently, a neural network model is trained
and used for classification based on the selected label set. While these
methods Gaunt et al., (2016); Ghiassi et al., (2022); Zhu et al., (2023)
introduce reliability measurements of labels, the accuracy of these
measurements directly affects the results of truth inference. Existing weakly
supervised truth inference methods have achieved some degree of success.
However, the data provided by workers is often sparse. Additionally, they
treat the ground truth as a hidden variable. As a result, the accuracy of
truth inference remains limited.

Supervised approach: This approach primarily involves building a classifier
model that establishes a connection between item features and the
corresponding ground truth. Additionally, it constructs a worker behavior
model based on the confusion matrix, which captures the relationship between
the item’s ground truth and the labels provided by workers. These worker
labels are used as supervision information, and the two models are jointly
trained in a supervised learning manner. Ultimately, the output of the
classifier model is used as the ground truth for inference Chu and Wang,
(2021); Ibrahim et al., (2023). Various techniques have been introduced within
this paradigm. For instance, the Expectation-Maximization (EM) algorithm has
been employed to integrate label aggregation and classifier training for truth
inference. Crowdlayer Rodrigues and Pereira, (2018) introduces the concept of
a crowd layer, which replaces the traditional confusion matrix used in the DS
model. This approach seamlessly integrates label reasoning and classifier
training in an end-to-end manner, resulting in more precise and reliable
outcomes. Tan and Chen applied the worker’s confusion matrix Tanno et al.,
(2019) and the labeled transfer matrix Cao et al., (2023) along with
classifier predictions to optimize model parameters and improve the accuracy
of item processing. Gao et al., (2022) introduces worker weight vectors, while
Cao et al., (2023) focuses on worker label reliability modeling. These
approaches aim to estimate the characteristics of workers’ abilities. UnionNet
Wei et al., (2022) considers all workers collectively and combines their
annotations into a parameter transfer matrix. It then aggregates classifier
predictions to facilitate model training. The CoNAL method Chu et al., (2021)
effectively handles noise from different sources by categorizing labeling
noise into common noise and individual noise. Although supervised learning can
be used to infer worker parameters, it struggles to accurately characterize
worker behavior. Consequently, when the discrimination among candidate answers
is poor, it is difficult to achieve good results using this approach.

Diverging from these methods, our method does not treat the ground truth of
the item as a latent variable. Instead, we focus on modeling the relationship
between the behavior of workers and the difficulty of item processing. This
allows us to establish a supervised learning framework to obtain a worker
behavior model. Subsequently, we use this model to identify oracle worker who
can accurately label items, thereby determining the ground truth.
Alternatively, we use this model to fill in sparse crowdsourced data and then
employ existing truth inference methods to obtain the ground truth from the
completed data.

##  3 Problem Formulation

This section provides a formal problem description. Our main goal is to create
a worker behavior model and achieve joint learning of worker abilities by
utilizing multi-task learning to infer the ground truth. Let
$\mathcal{W}=\\{w_{j}\\}$ denote the worker set, where $w_{j}$ represents an
individual worker, and $\mathcal{X}=\\{x_{i}\\}$ denote the set of items,
where $x_{i}$ represents a single item to be labeled. The labels for each item
belong to the category set $\mathcal{K}=\\{k\\}$. We use $y_{ij}$ to denote
the category label assigned by worker $w_{j}$ to item $x_{i}$. We have an
indicator function $I_{ij}$, where $I_{ij}=1$ if $y_{ij}$ exists and
$I_{ij}=0$ otherwise. Consequently, we obtain the crowdsourced triples dataset
$\mathcal{D}=\\{<x_{i},w_{j},y_{ij}>|I_{ij}=1\\}$. With regards to truth
inference in crowdsourcing, we provide the following definition:

######  Definition 1 (Problem of Truth Inference (TI Problem))

By modeling and learning from the crowdsourced label dataset $\mathcal{D}$,
the problem of truth inference aims to find a function
$g^{*}:\mathcal{X}\rightarrow\mathcal{K}$ such that

$g^{*}=\arg\min_{g\in\mathcal{H}}\sum_{i=1}^{|\mathcal{X}|}\mathcal{L}\left(\hat{z}_{i},g(x_{i})\right)+\lambda\|g\|_{\mathcal{H}}.$
(1)

Here, $\mathcal{H}$ denotes the hypothesis space of functions,
$\|\cdot\|_{\mathcal{H}}$ denotes the norm of hypothesis space, $\lambda$ is
the regularization coefficient, $\mathcal{L}$ is the loss function, and
$\hat{z}_{i}=t_{i}(\mathcal{D})$ is the estimation of label $z_{i}$ for item
$x_{i}$ from learning on the dataset. Since the crowdsourced truth inference
problem is an unsupervised learning problem without supervised information,
the estimation of ground truth is utilized instead of the goal of learning.

######  Definition 2 (Problem of Multi-task supervise Learning from Crowds
(MLC Problem))

Let $\mathcal{S}_{j}=\\{(<x_{i},w_{j}>,y_{ij})\\}_{x_{i}\in\mathcal{X}_{j}}$
denote the crowdsourced training dataset for worker $w_{j}$, where
$\mathcal{X}_{j}=\\{x_{i}|I_{ij}=1\\}$. The labels provided by worker $j$ can
be regarded as the $j$-th task for the corresponding item. Consequently, we
obtain the dataset as $\mathcal{S}=\bigcup_{j}\mathcal{S}_{j}$. The problem of
multi-task supervised learning from crowds is to find a worker behavior
function $f^{*}:\mathcal{X}\times\mathcal{W}\rightarrow\mathcal{K}$ such that

$f^{*}=\arg\min_{f\in\mathcal{H}}\sum_{j=1}^{|\mathcal{W}|}\frac{1}{|\mathcal{X}_{j}|}\sum_{x_{i}\in\mathcal{X}_{j}}\mathcal{L}\left(y_{ij},f(<x_{i},w_{j}>)\right)+\lambda\|f\|_{\mathcal{H}}.$
(2)

We can observe that the solution to MLC problem does not directly address TI
problem. Therefore, we provide two approaches to tackle this issue. The first
approach is to identify an oracle worker $w_{oracle}$ based on the
distribution of workers. We then consider the labels provided by this oracle
worker as the ground truth, that is,

$g^{*}(x_{i})=f^{*}(<x_{i},w_{oracle}>).$ (3)

The second approach considers the sparsity characteristic of crowdsourced
data, where workers do not annotate every item. Consequently, we can utilize
the results of MLC to generate a new dataset for inference. The new
crowdsourced data can be defined as follows:

$\mathcal{D}^{\prime}=\mathcal{D}\cup\left\\{<x_{i},w_{j},\hat{y}_{ij}>\Big{|}\hat{y}_{ij}=f^{*}(<x_{i},w_{j}>),I_{ij}=0\right\\}.$
(4)

##  4 Proposed Methodology

Figure 1: Model Structure of MMLC.

To address the MLC problem, we propose a Mixture of Experts based Multi-task
Supervised Learning from Crowds (MMLC) model. This model utilizes mixture of
experts to characterize the varying attention of workers towards different
item features, , aiming to capture the feature-level behavior differences of
workers when dealing with various items. The framework of the model is shown
in Fig. 1. It consists of three main modules: expert module, gate module, and
output module.

In the expert module, each item is processed by a feature extractor to obtain
an item feature vector $x_{i}$. Then, the item feature vector is fed into $m$
expert modules, where each module captures the unique characteristics of
worker behavior associated with different feature information. Each expert
module performs transformations and compressions on the feature vector,
resulting in the output matrix of the expert module:
$\bm{U}(x_{i})=\left(\bm{u}_{1}(x_{i}),\bm{u}_{2}(x_{i}),...,\bm{u}_{E}(x_{i})\right)$.
Each expert sub-module follows the same structure, consisting of multiple
layers of fully connected neural networks with ReLU activation functions in
each layer. For each expert sub-module $\bm{u}_{e}$, the high-dimensional
feature vector $x_{i}$ is transformed into a low-dimensional vector
$\bm{u}_{e}(x_{i})$.

In the gate module, a gate network is constructed to control the selection of
expert modules. This gate network takes worker data as input and generates a
projection vector of the worker in the worker spectral space, with a length of
$E$. The _bases_ of the worker spectral space are the outputs of the expert
sub-modules. Specifically, the module takes the one-hot encoded vector
representing each worker $w_{j}$ as input. After passing through a fully
connected ReLU layer, the data proceeds through an attention layer and a
softmax layer. Finally, it produces a worker projection vector
$\bm{v}(w_{j})=(v_{1}(w_{j}),v_{2}(w_{j}),...,v_{E}(w_{j}))^{T}$ with a length
of $E$. The projection of worker $w_{j}$ in the worker spectral space with the
expert sub-modules as the _bases_ is:

$proj_{\bm{U}(x_{i})}(w_{j})=\sum_{e=1}^{E}v_{e}(w_{j})\bm{u}_{e}(x_{i}).$ (5)

In the output module, the worker’s labels for the item are generated. The
output module generates labels for each worker based on their chosen expert
modules. It involves mapping worker behavior through the gate network, which
includes weighting and summing the outputs of the expert modules.
Subsequently, through a fully connected ReLU layer and a softmax layer, the
model produces the label output of worker $w_{j}$ for item $x_{i}$ as follows:

$f_{\bm{\Theta}}(<x_{i},w_{j}>)=\bm{o}\left(proj_{\bm{U}(x_{i})}(w_{j})\right),$
(6)

where $\bm{o}(\cdot)$ denotes the output function, and $\bm{\Theta}$ is the
parameter set of the functions $\bm{U}$, $\bm{v}$, and $\bm{o}$ within the
MMLC model. The MMLC model deals with a classification problem with
$|\mathcal{K}|$ categories. The network’s output is a
$|\mathcal{K}|$-dimensional vector, where each element represents the
predicted probability of a category.

The model’s loss function combines a cross-entropy loss term with a
regularization term. The loss function is formulated as follows:

$\mathcal{L}_{\bm{\Theta}}=-\frac{1}{|\mathcal{D}|}\sum_{w_{j}\in\mathcal{W}}\sum_{x_{i}\in\mathcal{X}_{j}}\sum_{k\in\mathcal{K}}y_{ij}^{k}\log\left(f_{\bm{\Theta}}(<x_{i},w_{j}>)\right)+\lambda\|\bm{\Theta}\|_{F},$
(7)

The first term denotes the multi-class cross-entropy loss, while the second
term represents the regularization of the model’s parameter set $\bm{\Theta}$
to prevent overfitting. In the equation, $\lambda$ is the regularization
coefficient, and $\|\cdot\|_{F}$ denotes the Frobenius norm. By minimizing the
loss function, we can obtain the final model
$\mathcal{M}^{*}:f_{\bm{\Theta}^{*}}(\cdot)$. This model uses the function
$f_{\bm{\Theta}^{*}}$ to predict the labels of worker $w_{j}$ for item
$x_{i}$, where $\bm{\Theta}^{*}$ represents the optimized parameters.

Truth Inference by Oracle Worker Finding (MMLC-owf): The MMLC model does not
directly generate the ground truth for inference. To address this issue, this
section proposes a method for inferring the ground truth by identifying the
oracle worker’s projection vector in the worker spectral space. Specifically,
each worker is theoretically associated with a projection in the worker
spectral space, representing their unique characteristics. Workers are
distributed in the spectral space. We assume the existence of an omniscient
oracle worker who possesses a projection vector in the spectral space and is
capable of providing the ground truth in the MMLC model. Therefore, by
identifying the projection vector of the oracle worker in the worker spectral
space, we can consider its output as the inferred truth. If we treat any
worker as a random expression of the oracle worker’s error, then the center of
the worker’s distribution projected onto the spectral space can be regarded as
the projection vector of the oracle worker, that is,

$\bm{v}_{oracle}=\bm{\tau}\left(\bm{v}(\mathcal{W})\right),$ (8)

where the function $\bm{\tau}(\cdot)$ is used to determine the distribution
center, which can be found using methods such as kernel density estimation,
mean, median, etc. According to the MMLC model, the outcome of the Oracle
Worker Finding method (MMLC-owf) for inferring the ground truth of item
$x_{i}$ can be expressed as follows:

$f_{\bm{\Theta}^{*}}(<x_{i},w_{oracle}>)=\bm{o}\left(\bm{U}(x_{i})\bm{v}_{oracle}\right).$
(9)

Truth Inference Framework by Data Filling (MMLC-df): In addition to the MMLC-
owf method, we propose a truth inference framework using data filling under
the MMLC model called MMLC-df, which utilizes the sparsity of crowdsourced
data. A new crowdsourced dataset $\mathcal{D}^{\prime}$ is constructed through
data filling as follows:

$\mathcal{D}^{\prime}=\mathcal{D}\cup\left\\{<x_{i},w_{j},\hat{y}_{ij}>\Big{|}\hat{y}_{ij}=f_{\bm{\Theta}^{*}}(<x_{i},w_{j}>),I_{ij}=0\right\\}.$
(10)

Subsequently, any truth inference method applied to this new crowdsourced
dataset can infer higher-quality ground truth compared to that obtained from
the original data.

##  5 Experiments

In this section, we verify the effectiveness of our method through
experiments. Our MMLC-owf method is a truth inference method, and we compare
it with the following baselines: MVSheng et al., (2008) directly uses majority
voting to determine the ground truth; DSDawid and Skene, (1979) employs a
confusion matrix to characterize the labeling behavior of workers and uses the
EM algorithm to infer the ground truth; HDSKarger et al., (2011) simplifies
the DS method by assuming that each worker has the same probability of being
correct under different truth values and equal probabilities for incorrect
options; FDSSinha et al., (2018) is a simple and efficient algorithm based on
DS, designed to achieve faster convergence while maintaining the accuracy of
truth inference; Max-MIGCao et al., (2019) utilizes the EM algorithm to
integrate label aggregation and classifier training; CoNALChu et al., (2021)
distinguishes between common noise and individual noise by predicting a joint
worker confusion matrix using classifiers; CrowdARCao et al., (2023) estimates
worker capability features through classifier prediction and models the
reliability of joint worker labels.

Our truth inference framework, MMLC-df, incorporates a core component that
performs data filling. We compare it with the following baselines: G_MV Sheng,
(2011) utilizes truth inference results from the MV algorithm to evaluate
worker ability and assign new labels accordingly; G_IRT Baker et al., (2017)
utilizes joint maximum likelihood estimation to estimate parameters of the IRT
model, such as worker abilities and item difficulties, and generates new
labels based on these parameters; TDG4Crowd Fang et al., (2023) learns the
feature distributions of workers and items separately using worker models and
item models. An inference component is used to learn the label distribution
and generate new labels.

We conduct experiments using three crowdsourced datasets with item features:

  * •

_LableMe_ Rodrigues and Pereira, (2018); Russell et al., (2008): This dataset
consists of 1000 images categorized into 8 classes, with a total of 2547
labels provided by 59 workers. Each image is represented by 8192-dimensional
features extracted using a pre-trained VGG-16 model.

  * •

_Text_ Dumitrache et al., (2018): This dataset comprises 1594 sentences
extracted from the CrowdTruth corpus, categorized into 13 groups. The dataset
includes 14,228 labels provided by 154 workers. Each sentence is represented
by 768-dimensional features extracted using a pre-trained BERT model.

  * •

_Music_ Rodrigues et al., (2014): This dataset consists of 700 music
compositions, each with a duration of 30 seconds, and categorized into 10
groups. It includes 2,945 labels provided by 44 workers. Each music
composition is represented by 124-dimensional features extracted using the
Marsyas Rodrigues et al., (2013) music retrieval tool.

To accommodate the feature scales of the three experimental datasets, our
model’s architecture varies accordingly. For the _LableMe_ dataset, our model
employs 16 expert modules, each comprising 3 fully connected ReLU layers, with
a final layer output dimension of 32. For the _Text_ and _Music_ datasets, we
utilize 10 expert modules. Each module consists of 3 and 2 fully connected
ReLU layers, with output dimensions of 32 and 16, respectively. We adopt the
settings from the Max-MIG, CoNAL, and CrowdAR truth inference methods, we
adopt the settings from their source code for the _LableMe_ and _Music_
datasets. Since there is no source code available for the _Text_ dataset, we
adopt the settings used in the _LableMe_ dataset. Regarding the TDG4Crowd data
filling algorithm, we utilize the settings from its source code. The remaining
methods do not use deep network structures and rely on default settings.

###  5.1 Evaluation of Oracle Worker Finding (MMLC-owf)

Method | _LableMe_ | _Text_ | _Music_
---|---|---|---
MV | 76.76 | 71.33 | 71.42
DS | 79.73 | 70.70 | 77.14
FDS | 77.78 | 71.45 | 77.57
HDS | 76.66 | 71.20 | 78.01
Max-MIG | 80.02$\pm$0.68 | 70.31$\pm$0.23 | 74.22$\pm$0.57
CoNAL | 81.46$\pm$0.49 | 72.75$\pm$0.49 | 76.02$\pm$0.36
CrowdAR | 82.14$\pm$0.36 | 70.48$\pm$0.42 | 78.54$\pm$0.59
MMLC-owf | 81.74$\pm$0.47 | 74.31$\pm$0.39 | 79.14$\pm$0.21

Table 1: Accuracy of Truth Inference Methods on Three Crowdsourced Datasets.

Figure 2: Accuracy of MMLC-owf with Various Clustering Methods.

Main Result: Our method, MMLC-owf, was evaluated alongside seven other methods
through five rounds experiments. The average accuracy results are shown in
Tab. 1. In our method, we utilized kernel density estimation (KDE) to compute
the projection vector of the oracle worker in the worker spectral space. Our
method, MMLC-owf, achieved the highest accuracy in the _Text_ and _Music_
datasets. In the _LableMe_ dataset, it ranked second, with only a 0.4%
difference from the top-performing CrowdAR method. Deep learning-based methods
typically produce better results when analyzing datasets with high-dimensional
item features like _LableMe_. In datasets with fewer features, the advantage
of deep learning methods was not significant.

Impact of Redundancy: We examine how varying levels of redundancy affect the
accuracy of our method. Due to the varying redundancy of data items, a maximum
redundancy parameter $R$ is set. We randomly keep $R$ labels for items with
more than $R$ labels and discard the rest. This process generates a dataset
with a maximum redundancy $R$. By conducting five repeated experiments and
averaging the accuracy and standard deviation, the results are shown in Fig.
3. As the average number of worker responses increases, all methods show an
upward trend in their results. The analysis of various redundancy levels
across the datasets indicates that higher redundancy levels are more
advantageous for our method. Our method can effectively utilize worker
behavior descriptions on datasets with higher redundancy but may face
underfitting on datasets with lower redundancy.

Figure 3: Accuracy Under Various Redundancies. (Left:_LableMe_ , Center:_Text_
, Right: _Music_)

Clustering Methods in Oracle Worker Finding: Our method, MMLC-owf, utilizes a
clustering method to determine the center of the distribution of the
projection vector of workers in the worker spectral space as the projection
vector of the oracle worker. Here, we examine how different clustering methods
affect truth inference outcomes. We compare three clustering methods: kernel
density estimation (KDE), Mean, and Median, as well as their worker-weighted
variants: KDE-W, Mean-W, and Median-W. Worker weights are calculated based on
the proportion of items answered by each worker relative to the total number
of items, considering data imbalance. In addition, we optimize the projection
vector in the worker spectral space using the ground truth of the items as an
upper-bound method for clustering. The parameters of the expert modules and
output modules are fixed in the pre-trained MMLC model, referred to as
“Truth.” By conducting five repeated experiments and averaging the results, as
shown in Fig. 2. For example, in the _LableMe_ dataset, the oracle worker’s
projection vector is derived using KDE, KDE-W, Median, Median-W, Mean, and
Mean-W. The MMLC-owf uses the oracle worker to generate ground truth. The
quality of the ground truth obtained by the following five methods in each
dataset is very similar, but the oracle worker generated using KDE produces
the highest quality ground truth in each dataset. The “Truth" method, which
represents the theoretical upper limit with clustering methods, achieved
accuracy rates of 96.32%, 91.25%, and 92.97% on the three datasets
respectively. The quality of the ground truth generated by oracle workers
using six clustering methods still slightly deviates from theoretical upper
limits. This implies that the MMLC-owf method can provide high-quality ground
truth by optimally projecting the worker spectral space, approaching the
theoretical upper limit. The model has strong expressive ability, with a small
gap between the projected spectral space and the ground truth. There is
potential to enhance MMLC-owf by choosing a more effective clustering method.

(a) _LabelMe_

(b) _Text_

(c) _Music_

Figure 4: Worker Scatter Plot in Worker Spectral Space.

Worker Distribution in Worker Spectral Space: We assume that each worker is an
oracle worker with random errors in their expression. The center of the
distribution of workers projected onto the spectral space corresponds to the
projection vector of the oracle worker. To validate this assumption, we
employed the IOSMAP dimensionality reduction method to reduce the worker
projection vectors obtained from the MMLC model to 2D, resulting in the
scatter plot shown in Fig. 4. We calculated the accuracy of each worker on the
dataset, where the closer the point’s color on the graph is to green, the
worker’s accuracy is higher. The closer the point’s color is to red, the lower
the worker’s accuracy. The plot also shows the projection obtained by the KDE
method for the oracle worker, represented by blue asterisks. From the
distribution of worker projections, although the shapes of the distributions
differ across datasets, there is a noticeable trend where workers with higher
accuracy tend to cluster closer to the projection of the oracle worker. This
observation demonstrates a clear tendency towards aggregation and provides
some degree of confirmation for the validity of our assumption.

###  5.2 Evaluation of Data Filling (MMLC-df)

Data | Method | Original | G_MV | G_IRT | TDG4Crowd | MMLC-df
---|---|---|---|---|---|---
| MV | 76.76 | -0.03$\pm$0.41 | -2.87$\pm$0.58 | +1.88$\pm$0.41 | +3.89$\pm$0.17
| DS | 79.73 | -2.87$\pm$0.77 | +0.06$\pm$0.26 | -1.05$\pm$0.37 | +1.72$\pm$0.20
| FDS | 77.78 | +0.07$\pm$0.92 | +0.20$\pm$0.62 | +0.93$\pm$0.23 | +3.38$\pm$0.21
_LableMe_ | HDS | 76.66 | -0.23 $\pm$0.44 | +0.18$\pm$0.33 | +2.02$\pm$0.32 | +2.48$\pm$0.36
| Max-MIG | 80.02$\pm$0.68 | +1.90$\pm$0.79 | +1.76$\pm$0.14 | +2.23$\pm$0.21 | +4.81$\pm$0.21
| CoNAL | 81.46$\pm$0.49 | -1.92$\pm$0.41 | -0.78$\pm$0.68 | +0.02$\pm$0.72 | +1.73$\pm$0.57
| CrowdAR | 82.14$\pm$0.36 | +3.21$\pm$0.28 | +2.52$\pm$0.28 | -0.69$\pm$0.56 | +2.49$\pm$0.41
| MMLC-owf | 81.74$\pm$0.47 | -2.91$\pm$0.46 | +0.05$\pm$0.61 | -1.41$\pm$0.54 | +1.65$\pm$0.66
| MV | 71.33 | +0.53$\pm$0.31 | -0.26$\pm$0.44 | -0.01$\pm$0.08 | +3.35$\pm$0.31
| DS | 70.72 | +1.80$\pm$0.53 | +0.01$\pm$0.51 | +0.38$\pm$0.21 | +4.02$\pm$0.26
| FDS | 71.45 | +0.88$\pm$0.62 | -0.63$\pm$0.48 | -0.16$\pm$0.04 | +3.26$\pm$0.35
_Text_ | HDS | 71.21 | +0.24$\pm$0.17 | -0.39$\pm$0.34 | -1.30$\pm$0.48 | +2.18$\pm$0.19
| Max-MIG | 70.31$\pm$0.23 | -1.24$\pm$0.70 | -1.87$\pm$0.17 | +0.33$\pm$0.64 | +3.88$\pm$0.51
| CoNAL | 72.75$\pm$0.49 | -1.33$\pm$0.22 | -2.27$\pm$0.43 | +0.62$\pm$0.32 | +2.16$\pm$0.61
| CrowdAR | 70.48$\pm$0.42 | +0.37$\pm$0.62 | -0.33$\pm$0.21 | +1.96$\pm$0.12 | +3.63$\pm$0.39
| MMLC-owf | 74.31$\pm$0.39 | -2.04$\pm$0.41 | -0.15$\pm$0.37 | +0.39$\pm$0.26 | +1.46$\pm$0.52
| MV | 71.42 | -0.95$\pm$0.35 | +6.72$\pm$0.23 | +5.73$\pm$0.28 | +7.58$\pm$0.45
| DS | 77.14 | -5.95$\pm$0.31 | +0.27$\pm$0.43 | +1.57$\pm$0.13 | +2.41$\pm$0.41
| FDS | 77.57 | -6.47$\pm$0.64 | +0.41$\pm$0.57 | +0.81$\pm$0.08 | +1.57$\pm$0.22
_Music_ | HDS | 78.01 | -7.25$\pm$0.87 | +0.52$\pm$0.43 | +0.70$\pm$0.25 | +0.12$\pm$0.54
| Max-MIG | 74.22$\pm$0.57 | +1.42$\pm$0.50 | +0.32$\pm$0.98 | +5.54$\pm$0.72 | +4.77$\pm$0.41
| CoNAL | 76.02$\pm$0.36 | +7.97$\pm$0.54 | +4.65$\pm$0.54 | +5.55$\pm$0.51 | +6.37$\pm$0.66
| CrowdAR | 78.54$\pm$0.59 | +1.46$\pm$0.42 | +2.31$\pm$0.20 | +1.97$\pm$0.11 | +2.47$\pm$0.28
| MMLC-owf | 79.14$\pm$0.21 | +0.24$\pm$0.37 | +0.75$\pm$ 0.32 | +1.62$\pm$0.31 | +1.43$\pm$ 0.42

Table 2: The Change of Accuracy After Data Filling.

Main Result: We compared MMLC-df with three filling methods: G_MV, G_IRT, and
TDG4Crowd. We used the filled data with eight truth inference methods from the
previous section to infer the ground truth. We used three real datasets and
applied eight truth value inference methods to infer the ground truth,
resulting in a total of 24 scenarios. We conducted five rounds of experiments,
and the mean and variance of all ground truth accuracy are presented in Tab.
2. It can be observed that our MMLC-df framework achieves enhanced performance
compared to the original data in 100% of the scenarios, with 79.2% of the
scenarios achieving optimal enhancement. On the other hand, G_MV, G_IRT, and
TDG4Crowd achieve enhanced results in 50%, 62.5%, and 75% of the scenarios,
respectively. In terms of the enhancement magnitude, our method performs the
best on the _Text_ dataset. While other methods may achieve better results in
certain scenarios for other datasets, their performance is unstable, and there
are cases where the results deteriorate. This indicates that our MMLC-df
framework demonstrates good stability and consistent performance.

Impact of Data Filling’s Density: Our MMLC-df framework leverages the sparsity
of crowdsourced data for data filling. To clarify, we define the data density
of non-empty crowdsourced data as $d_{\mathcal{D}}\in(0,1]$ and
$d_{\mathcal{D}}=\frac{|\mathcal{D}|}{|\mathcal{W}|\times|\mathcal{X}|}$. The
data densities of the three original datasets _LabelMe_ , _Text_ , and _Music_
are 0.0431, 0.0579, and 0.0956, respectively. The original data seems sparse.
We gradually fill the data until reaching a data density to 1 for the analysis
of its impact of data density on the results. We set a threshold for the
number of items to be filled, denoted as $n_{interval}$. For workers with
items exceeding this threshold, we replace their labeled items with predicted
values. By adjusting the threshold from large to small, we gradually fill the
data until all workers have completed their items, achieving a data density of
1. Due to the large amount of filled data, deep learning methods can be time-
consuming. The accuracy obtained by various methods shows a similar trend to
the density transformation. Therefore, we conducted this experiment using only
statistical machine learning methods. The experiment is repeated five rounds,
and the average accuracy and standard deviation are shown in Fig. 5. The
trends are generally consistent across all methods, but the variations differ
significantly among different datasets. In _Text_ dataset, as density
increases, the algorithm’s accuracy stabilizes rapidly and then reaches a
plateau. In the _LableMe_ dataset, accuracy fluctuates significantly as
density increases. Higher density often improves accuracy. In _Music_ dataset,
as density increases, accuracy initially fluctuates rapidly before
stabilizing. The _Text_ data filling performs the best, likely due to the
larger scale of this dataset compared to the other two, resulting in a more
significant impact.

Figure 5: Accuracy with Various Data Filling’s Density. (Left:_LableMe_ ,
Center:_Text_ , Right: _Music_)

##  6 Conclusion

This paper introduces a novel crowdsourced learning paradigm called MLC.Within
this paradigm, we propose a feature-level worker behavior model called MMLC.
Based on this model, we develop two truth inference methods: MMLC-owf, which
uses oracle worker finding, and MMLC-df, a truth inference framework based on
crowdsourced data filling. Experimental results demonstrate the superior
performance of MMLC-owf compared to other methods. Furthermore, we assess the
theoretical upper performance limit of the MMLC-owf method, demonstrating its
potential to enhance clustering method selection and validate its strong
performance. The experiments also validate the effectiveness and stability of
the MMLC-df framework in enhancing truth inference methods through
crowdsourced data filling. Furthermore, we observed that our model exhibited
better performance on datasets with a higher number of annotations per worker.
On real crowdsourcing platforms, workers continuously engage in annotation
tasks, resulting in an increasing average number of annotations per worker.
Consequently, our model holds significant practical value for real-world
applications.

## References

  * Baker et al., (2017) Baker, F. B., Kim, S.-H., et al. (2017).  The basics of item response theory using R, volume 969.  Springer.
  * Cao et al., (2019) Cao, P., Xu, Y., Kong, Y., and Wang, Y. (2019).  Max-mig: an information theoretic approach for joint learning from crowds.  arXiv preprint arXiv:1905.13436.
  * Cao et al., (2023) Cao, Z., Chen, E., Huang, Y., Shen, S., and Huang, Z. (2023).  Learning from crowds with annotation reliability.  In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2103–2107.
  * Chu et al., (2021) Chu, Z., Ma, J., and Wang, H. (2021).  Learning from crowds by modeling common confusions.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 5832–5840.
  * Chu and Wang, (2021) Chu, Z. and Wang, H. (2021).  Improve learning from crowds via generative augmentation.  In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 167–175.
  * Dawid and Skene, (1979) Dawid, A. P. and Skene, A. M. (1979).  Maximum likelihood estimation of observer error-rates using the em algorithm.  Journal of the Royal Statistical Society: Series C (Applied Statistics), 28(1):20–28.
  * Dumitrache et al., (2018) Dumitrache, A., Aroyo, L., and Welty, C. (2018).  Crowdsourcing semantic label propagation in relation classification.  arXiv preprint arXiv:1809.00537.
  * Fang et al., (2023) Fang, Y., Shen, C., Gu, H., Han, T., and Ding, X. (2023).  Tdg4crowd: test data generation for evaluation of aggregation algorithms in crowdsourcing.  In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 2984–2992.
  * Gao et al., (2022) Gao, Z., Sun, F.-K., Yang, M., Ren, S., Xiong, Z., Engeler, M., Burazer, A., Wildling, L., Daniel, L., and Boning, D. S. (2022).  Learning from multiple annotator noisy labels via sample-wise label fusion.  In European Conference on Computer Vision, pages 407–422. Springer.
  * Gaunt et al., (2016) Gaunt, A., Borsa, D., and Bachrach, Y. (2016).  Training deep neural nets to aggregate crowdsourced responses.  In Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence. AUAI Press, volume 242251.
  * Ghiassi et al., (2022) Ghiassi, A., Birke, R., Chen, L. Y., et al. (2022).  Labnet: A collaborative method for dnn training and label aggregation.  In ICAART (2), pages 56–66.
  * Ibrahim et al., (2023) Ibrahim, S., Nguyen, T., and Fu, X. (2023).  Deep learning from crowdsourced labels: Coupled cross-entropy minimization, identifiability, and regularization.  arXiv preprint arXiv:2306.03288.
  * Imamura et al., (2018) Imamura, H., Sato, I., and Sugiyama, M. (2018).  Analysis of minimax error rate for crowdsourcing and its application to worker clustering model.  Cornell University - arXiv,Cornell University - arXiv.
  * Karger et al., (2011) Karger, D., Oh, S., and Shah, D. (2011).  Iterative learning for reliable crowdsourcing systems.  Neural Information Processing Systems,Neural Information Processing Systems.
  * Li and Yu, (2014) Li, H. and Yu, B. (2014).  Error rate bounds and iterative weighted majority voting for crowdsourcing.  arXiv preprint arXiv:1411.4086.
  * Raykar et al., (2010) Raykar, V., Yu, S., Zhao, L., Valadez, G., Florin, C., Bogoni, L., and Moy, L. (2010).  Learning from crowds.  Journal of Machine Learning Research,Journal of Machine Learning Research.
  * Rodrigues and Pereira, (2018) Rodrigues, F. and Pereira, F. (2018).  Deep learning from crowds.  In Proceedings of the AAAI conference on artificial intelligence, volume 32.
  * Rodrigues et al., (2013) Rodrigues, F., Pereira, F., and Ribeiro, B. (2013).  Learning from multiple annotators: distinguishing good from random labelers.  Pattern Recognition Letters, 34(12):1428–1436.
  * Rodrigues et al., (2014) Rodrigues, F., Pereira, F., and Ribeiro, B. (2014).  Gaussian process classification and active learning with multiple annotators.  In International conference on machine learning, pages 433–441. PMLR.
  * Russell et al., (2008) Russell, B. C., Torralba, A., Murphy, K. P., and Freeman, W. T. (2008).  Labelme: a database and web-based tool for image annotation.  International journal of computer vision, 77:157–173.
  * Sheng, (2011) Sheng, V. S. (2011).  Simple multiple noisy label utilization strategies.  In 2011 IEEE 11th International Conference on Data Mining, pages 635–644. IEEE.
  * Sheng et al., (2008) Sheng, V. S., Provost, F., and Ipeirotis, P. G. (2008).  Get another label? improving data quality and data mining using multiple, noisy labelers.  In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 614–622.
  * Sheng et al., (2017) Sheng, V. S., Zhang, J., Gu, B., and Wu, X. (2017).  Majority voting and pairing with multiple noisy labeling.  IEEE Transactions on Knowledge and Data Engineering, 31(7):1355–1368.
  * Sinha et al., (2018) Sinha, V. B., Rao, S., and Balasubramanian, V. N. (2018).  Fast dawid-skene: A fast vote aggregation scheme for sentiment classification.  arXiv preprint arXiv:1803.02781.
  * Tanno et al., (2019) Tanno, R., Saeedi, A., Sankaranarayanan, S., Alexander, D. C., and Silberman, N. (2019).  Learning from noisy labels by regularized estimation of annotator confusion.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11244–11253.
  * Tao et al., (2020) Tao, F., Jiang, L., and Li, C. (2020).  Label similarity-based weighted soft majority voting and pairing for crowdsourcing.  Knowledge and Information Systems, 62:2521–2538.
  * Wei et al., (2022) Wei, H., Xie, R., Feng, L., Han, B., and An, B. (2022).  Deep learning from multiple noisy annotators as a union.  IEEE transactions on neural networks and learning systems.
  * Whitehill et al., (2009) Whitehill, J., Wu, T.-f., Bergsma, J., Movellan, J., and Ruvolo, P. (2009).  Whose vote should count more: Optimal integration of labels from labelers of unknown expertise.  Advances in neural information processing systems, 22.
  * Zhu et al., (2023) Zhu, K., Xue, S., and Jiang, L. (2023).  Improving label quality in crowdsourcing using deep co-teaching-based noise correction.  International Journal of Machine Learning and Cybernetics, pages 1–14.


# SentEmojiBot: Empathising Conversations Generation with Emojis

Akhilesh Ravi, Amit Yadav, Jainish Chauhan, Jatin Dholakia, Naman Jain, and
Mayank Singh
Indian Institute of Technology Gandhinagar
Gujarat, India
<EMAIL_ADDRESS>

###### Abstract

The increasing use of dialogue agents makes it extremely desirable for them to
understand and acknowledge the implied emotions to respond like humans with
empathy. Chatbots using traditional techniques analyze emotions based on the
context and meaning of the text and lack the understanding of emotions
expressed through face. Emojis representing facial expressions presents a
promising way to express emotions. However, none of the AI systems utilises
emojis for empathetic conversation generation. We propose, SentEmojiBot, based
on SentEmoji dataset, to generate empathetic conversations with a combination
of emojis and text. Evaluation metrics show that BERT-based model outperforms
the vanilla transformer model. A user study indicates that the dialogues
generated by our model were understandable and adding emojis improved
empathetic traits in conversations by 9.8%.

##  1 Introduction

Humans acknowledge the feelings of their interlocutor while responding with
caring attitude to achieve an engaging and comforting conversation. This
behaviour is termed as empathetic responding Rashkin et al. (2018). With the
onset of technologies such as chatbots and voice assistants, humans have
started to expect empathetic responses from the machine-mediated automatic
communication systems Reeves and Nass (1996). Many studies have proved that
empathetic responses results in better outcomes from both goal-oriented and
informal conversations. Levinson et al. (2000); Wentzel (1997); Bickmore and
Cassell (2001); Kim et al. (2004); Fraser et al. (2018). In recent years,
researchers have been successful in generating meaningful responses Zhou and
Wang (2018); Wang and Wan (2018); Zhou et al. (2018); Hu et al. (2017) and
embedding empathetic behaviour in the semantics of a chatbot’s response Ritter
et al. (2010); Zhang et al. (2018); Mazaré et al. (2019); Rashkin et al.
(2018); Lin et al. (2019). However, these works have been able to generate
responses by focusing purely on textual responses.

Figure 1: Comparison of responses from various systems: 1) Siri, 2) Rashkin et
al. (2018), 3) Our model

Research shows that facial expressions plays a key role in clearly
communicating the message of the speaker Busso et al. (2004). They help the
listener to clearly resolve the ambiguity in emotions, intention and tonality
of the message. Modern application softwares have introduced Emojis, the
animated faces with expressions, as an alternative to facial expressions in
chat rooms to eliminate the ambiguity related to the response of the user.
Previous works have analysed and supported the significance of emojis in
social media conversations through improved performances in understanding NLP
tasks such as sentiment, emotion, and sarcasm detection Felbo et al. (2017);
Wood and Ruder (2016); Li et al. (2019). Even though we find rich literature
that use emojis to improvise semantic understanding of text, to the best of
our knowledge, we did not find any work that uses emojis to enhance the
generation of empathetic responses in automated communication systems.

In this paper, we formalise the task of generating empathising responses using
emojis by proposing SentEmojiBot, a model trained on textual conversations and
emojis data. We present the experiments with appropriate evaluation methods to
prove the significance of emojis in conveying empathising messages. Figure 1
shows an example of a chatbot interface where Speaker(human) initiates the
conversation. The figure compares various systems and clearly shows the
positive impact of empathising text and emojis through the gradual improvement
in empathetic behaviour from Siri to SentEmojiBot. SentEmojiBot is a BERT-
based model that generates responses based on the emotion and context of the
text. In our experiments, the BERT based model outperformed the vanilla
transformer model. Moreover, a user survey shows that SentEmojiBot added
relevant emojis to conversations which improved the empathising behaviour of
the responses by 9.8%, compared to purely text-based response. Hence, our work
showcases the possibility of building natural, engaging, and empathetic
dialogue agents over the traditional text-based language models.

Our main contributions are SentEmojiBot - a pipeline for generating empathetic
responses with emojis, and a user-study showing an increase in empathetic
behaviour when emoji is added to a textual traditional response.

Figure 2: Example of a conversation snippet with multiple utterances from SE
dataset

##  2 Dataset

We utilise SentEmoji (hereafter ‘SE’) dataset released by Ravi et al. (2020)
containing empathetic responses with emojis. The dataset contains 24,850
conversations and 79,190 utterances, with an average utterance length of 15.2
words. The dataset has 10 fundamental emotional categories. These categories
are mutually exclusive from each other, in terms of appraisal, antecedent
events, probable behavioural response and physiology Kowalska and Wróbel
(2017). Figure 2 presents an example of conversation snippet from the SE
dataset. “Emotion” tells about the implied emotion in the conversation.
“Context” sets a situation for conversation based on the emotion. In every
conversation, “Speaker” refers to human and “Listener” refers to automated
dialogue agent. Each dialogue is considered as one utterance and each
utterance contains an emoji to either highlight the speaker’s emotion or
generate empathetic response from the listener.

##  3 Methodology

This section discusses the experimental setup and the architecture of
SentEmojiBot (Figure 3).

###  3.1 Data Preparation

In a conversation, people only have the information about the utterances, with
their interlocutor, that have been discussed in the past in order to analyse
and convey their response in return. Hence, we concatenate utterances prior to
the listener’s response, from the SE’s conversations as the “context
utterance” and the listener’s response as the “response utterance”. The
context utterance is fed as an input to the model to obtain response utterance
as an output. In total, there are 53,372 context-response utterance pairs. We
do not use emotion and context in the training process and do not consider
speaker’s response as the “response utterance” because speaker drives the
conversation for the listener and expects a response in return. Also, in the
real world deployment of SentEmojiBot, listener is expected to be an automated
model output whereas speaker is expected to be a human. We tokenised the
context utterance using the BertTokenizer Wolf et al. (2019) and the sequence
length is set to 100. The result is fed to the language models described below
to get an empathetic response.

###  3.2 Generating “Response Utterance”

To generate an empathetic text response, we perform experiments on retrieval-
based systems consisting of Transformers. In retrieval-based systems, the
model selects the best possible response from a set of candidate responses.
The following methodology has been formalised by Rashkin et al. (2018).

Figure 3: Architecture of SentEmojiBot

  * •

BERT-based: We used BERT Devlin et al. (2018) as the base architecture to
encode candidates (hy) and contexts (hx). The model is fine-tuned over pre-
trained weights Wolf et al. (2019) on SE dataset, all layers are trained for
12 epochs with a batch size of 16, an embedding layer of size 300, the
learning rate of $5\times 10^{-5}$, and the Adamax optimizer.

  * •

Vanilla Transformers-based: We use two transformer encoders separately
embedding context (hx) and candidates (hy) Yang et al. (2018). The learning
rate is set to $8\times 10^{-4}$, with an Adamax optimizer. The model is fine-
tuned for 25 epochs with a batch size of 128.

We provide the “context utterance” as an input and predict the next most
probable “response utterance” from the model. The model chooses a response
according to a softmax on the dot product (hx·hy) out of all candidates. We
minimise the negative log-likelihood of selecting the correct response. The
utterances from the SE dataset were split into three parts: training data
(80%), validation data (10%) and test data (10%). The number of training
epochs was decided to avoid over-fitting on the data and due to resource
constraints.

  * •

BERT-based: We used BERT Devlin et al. (2018) as the base architecture to
encode candidates (hy) and contexts (hx). The model is fine-tuned over pre-
trained weights Wolf et al. (2019) on SE dataset, all layers are trained for
12 epochs with a batch size of 16, an embedding layer of size 300, the
learning rate of $5\times 10^{-5}$, and the Adamax optimizer.

  * •

Vanilla Transformers-based: We use two transformer encoders separately
embedding context (hx) and candidates (hy) Yang et al. (2018). The learning
rate is set to $8\times 10^{-4}$, with an Adamax optimizer. The model is fine-
tuned for 25 epochs with a batch size of 128.

We provide the “context utterance” as an input and predict the next most
probable “response utterance” from the model. The model chooses a response
according to a softmax on the dot product (hx·hy) out of all candidates. We
minimise the negative log-likelihood of selecting the correct response. The
utterances from the SE dataset were split into three parts: training data
(80%), validation data (10%) and test data (10%). The number of training
epochs was decided to avoid over-fitting on the data and due to resource
constraints.

###  3.3 Incorporating Emoji

Table 1: Distribution of conversations in each emotion and the group of emojis
relevant to an emotion

Once we have a text-based response, we append the relevant emoji at the end.
We achieve this task by identifying the emotion of the generated response from
language models using CNN-based classifier and then selecting the most
relevant emoji based on the emotion as shown in Table 1.

  * •

Identifying emotion: Figure 3 shows the architecture of the CNN-based emotion
classifier inspired from Kim (2014). We trained the emotion classifier on the
“Context” of each conversation as an input and their corresponding “Emotion”
labels in the SE dataset as an output. We chose “Context” attribute of each
conversation instead of the utterances because “Context” summarises the
content of the conversation without directly revealing the details of the
conversation. Figure 2 shows an example of context and emotion pair. We split
the dataset into 72-8-20 for train-validation-test split required for the
evaluation and tuning. We trained the model with an Adam optimizer at a
learning rate of 0.001, and a decay of $10^{-6}$ for two epochs with a batch
size of 128 using cross-entropy loss. After training, we used the emotion
classifier with the generated text from language models to obtain the
appropriate emotion related to the sentence.

  * •

Getting relevant emoji: After getting the generated sentence’s emotion, we
need a relevant emoji which can be embedded in the text. Using the emotion
from the classifier, we obtain a group of emojis which signify the output
emotion. We obtain this bucket of emojis using Table 1. Table 1 is obtained by
mapping the most commonly used emojis to their corresponding emotion Novak et
al. (2015). After obtaining the bucket, the next step is to get the most
relevant emoji from the bucket since the bucket may contain more than one
emojis per emotion. To select the most relevant emoji, we compare the cosine
similarity between each emoji’s embedding and sentence embedding of the
generated response.

We obtain the emoji’s embedding using Emoji2Vec Eisner et al. (2016) and the
word embeddings for the sentence embedding using pre-trained Word2Vec
Demeester et al. (2016). Sentence embedding is generated using the method
proposed by Arora et al. (2016). Since Emoji2Vec generates embeddings using a
pre-trained model of Word2Vec on the words associated with the emoji, we chose
to use Word2Vec embeddings for the generated textual response instead of BERT
embeddings. This technique helps in providing the same space to sentence and
emoji embedding. Finally, the emoji with maximum cosine similarity with
sentence embedding is taken as the most relevant emoji from the bucket. We add
the emoji at the end of the sentence to generate an empathetic response.

Although, the emotion classifier provide us the emotion imbibed in the
generated sentence, still the emotion may not be explicit enough to add an
emoji. Thus, only when the cosine similarity is above a threshold, the emoji
is added. This way, we avoided adding emojis to all sentences, and hence
avoided their unrealistic and excessive use.

##  4 Evaluation

Model | Average | P@1,100
---|---|---
BLUE Score
Transformer | 4.38 | 3.65%
BERT | 5.78 | 36%

Table 2: Automatic evaluation metrics on the test set

Automated Metrics: Following the practice of earlier works in dialogue
generation Li et al. (2015); Wen et al. (2015), we compared the model
generated response with the actual response using the BLEU scores. The BLEU
scores (average of BLEU-1, BLEU-2, BLEU-3, and BLEU-4) of all the samples in
the test set were averaged for Transformer and BERT based models. Then, we
computed the P@1,100 Rashkin et al. (2018) to evaluate the performance of the
response-retrieval systems. Table 2 summarises the results and shows that
BERT-model outperforms the Transformer-based approach in terms of both the
metrics.

On evaluating the emotion classifier, we achieved the micro accuracy of 55.4%,
macro accuracy of 54.6%, and macro F1-score of 55.9%. According to Liu (2018),
extracting emotions is the biggest challenge in identifying the emoji. Hence,
our results are consistent with the experiments by Liu (2018). Even though the
results can be improved with advanced models, our pipeline is an attempt to
formalise the problem statement and provide its significance.

Human Evaluation: We evaluate 80 dialogues generated from BERT-based
SentEmojiBot: 40 dialogues with emojis and the same 40 dialogues without
emojis. We split the dialogues into four sets of 20 randomly chosen dialogues.
All the sets are mutually exclusive from each other. Each set was shared with
five English-speaking human evaluators (different from the authors of paper),
that evaluated each dialogue on a Likert scale (1–5) Joshi et al. (2015). The
total number of evaluators were 20. The evaluators rated the dialogues on the
basis of two criteria i.e. the empathy of generated dialogue and the relevance
of the added emoji. For dialogues without emoji, the relevance of added emoji
is not rated. All the ratings are averaged across each of the tasks to obtain
the final evaluation score shown in Table 3. We observed that emojis improved
the empathy score by 0.49. Furthermore, the relevance score of 3.11 reflects
that the evaluators feel that the emojis were relevant to the context on the
Likert scale.

User-Study | Empathy | Relevance
---|---|---
of emoji
Responses | 2.88/5 | -
without emojis
Responses | 3.37/5 | 3.11/5
with emojis
Table 3: Human ratings: Empathy and Relevance

##  5 Discussions And Conclusion

We showed the efficacy of emojis to improve empathetic responses and developed
a system- SentEmojiBot to generate empathetic responses inculcating emojis. As
shown in Table 2, SentEmojiBot performed well in terms of the metrics. The
human ratings in Table 3 show that added emojis were satisfactory relevant and
increased empathy of responses. We hope our pipeline and results will promote
more research on using cross-modality data like emojis for improving
empathetic behaviour of dialogue agents. Our current work is limited to
including emojis (a) at the end of sentences, and (b) after generating text-
based dialogues. However, humans often use emojis in between dialogues, hence,
in the future, generating emojis as a part of the dialogue itself can be
another direction to make the response more natural and empathetic.

## References

  * Arora et al. (2016) Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2016.  A simple but tough-to-beat baseline for sentence embeddings.
  * Bickmore and Cassell (2001) Timothy Bickmore and Justine Cassell. 2001.  Relational agents: a model and implementation of building user trust.  In _Proceedings of the SIGCHI conference on Human factors in computing systems_ , pages 396–403. ACM.
  * Busso et al. (2004) Carlos Busso, Zhigang Deng, Serdar Yildirim, Murtaza Bulut, Chul Min Lee, Abe Kazemzadeh, Sungbok Lee, Ulrich Neumann, and Shrikanth Narayanan. 2004.  Analysis of emotion recognition using facial expressions, speech and multimodal information.  In _Proceedings of the 6th International Conference on Multimodal Interfaces_ , ICMI ’04, page 205–211, New York, NY, USA. Association for Computing Machinery.
  * Demeester et al. (2016) Thomas Demeester, Tim Rocktäschel, and Sebastian Riedel. 2016.  Lifted rule injection for relation embeddings.  _EMNLP 2016 - Conference on Empirical Methods in Natural Language Processing, Proceedings_ , pages 1389–1399.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  (Mlm).
  * Eisner et al. (2016) Ben Eisner, Tim Rocktäschel, Isabelle Augenstein, Matko Bosnjak, and Sebastian Riedel. 2016.  emoji2vec: Learning Emoji Representations from their Description.  pages 48–54.
  * Felbo et al. (2017) Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017\.  Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm.  _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_.
  * Fraser et al. (2018) Jamie Fraser, Ioannis Papaioannou, and Oliver Lemon. 2018.  Spoken conversational ai in video games: Emotional dialogue management increases user engagement.  In _IVA_ , pages 179–184.
  * Hu et al. (2017) Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. 2017\.  Toward controlled generation of text.  _34th International Conference on Machine Learning, ICML 2017_ , 4:2503–2513.
  * Joshi et al. (2015) Ankur Joshi, Saket Kale, Satish Chandel, and D Kumar Pal. 2015.  Likert scale: Explored and explained.  _Current Journal of Applied Science and Technology_ , pages 396–403.
  * Kim et al. (2004) Sung Soo Kim, Stan Kaplowitz, and Mark V Johnston. 2004.  The effects of physician empathy on patient satisfaction and compliance.  _Evaluation & the health professions_, 27(3):237–251.
  * Kim (2014) Yoon Kim. 2014.  Convolutional neural networks for sentence classification.  _EMNLP 2014 - 2014 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference_ , pages 1746–1751.
  * Kowalska and Wróbel (2017) Magda Kowalska and Monika Wróbel. 2017.  Basic Emotions.
  * Levinson et al. (2000) W. Levinson, R. Gorawara-Bhat, and J. Lamb. 2000.  A study of patient clues and physician responses in primary care and surgical settings.  _Journal of the American Medical Association_ , 284(8):1021–1027.
  * Li et al. (2015) Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015.  A diversity-promoting objective function for neural conversation models.  _CoRR_ , abs/1510.03055.
  * Li et al. (2019) Mingyang Li, Sharath Guntuku, Vinit Jakhetiya, and Lyle Ungar. 2019.  Exploring (dis-) similarities in emoji-emotion association on twitter and weibo.  In _Companion Proceedings of The 2019 World Wide Web Conference_ , pages 461–467.
  * Lin et al. (2019) Zhaojiang Lin, Peng Xu, Genta Indra Winata, Farhad Bin Siddique, Zihan Liu, Jamin Shin, and Pascale Fung. 2019.  CAiRE: An End-to-End Empathetic Chatbot.  pages 1–2.
  * Liu (2018) Man Liu. 2018.  EmoNLP at SemEval-2018 Task 2: English Emoji Prediction with Gradient Boosting Regression Tree Method and Bidirectional LSTM.  pages 390–394.
  * Mazaré et al. (2019) Pierre-Emmanuel Mazaré, Samuel Humeau, Martin Raison, and Antoine Bordes. 2019\.  Training Millions of Personalized Dialogue Agents.  pages 2775–2779.
  * Novak et al. (2015) Petra Kralj Novak, Jasmina Smailović, Borut Sluban, and Igor Mozetič. 2015.  Sentiment of emojis.  _PLoS ONE_ , 10(12):1–22.
  * Rashkin et al. (2018) Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2018.  Towards Empathetic Open-domain Conversation Models: a New Benchmark and Dataset.
  * Ravi et al. (2020) Akhilesh Ravi, Amit Kumar Singh Yadav, Jainish Chauhan, Jatin Dholakia, and Naman Jain. 2020.  Sentemoji: A dataset to generate empathising conversations.  In _Proceedings of the 7th ACM IKDD CoDS and 25th COMAD_ , CoDS COMAD 2020, page 345–346, New York, NY, USA. Association for Computing Machinery.
  * Reeves and Nass (1996) Byron Reeves and Clifford Ivar Nass. 1996.  _The media equation: How people treat computers, television, and new media like real people and places._ Cambridge University Press, New York, NY, US.
  * Ritter et al. (2010) Alan Ritter, Colin Cherry, and Bill Dolan. 2010.  Unsupervised modeling of twitter conversations.  _NAACL HLT 2010 - Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Proceedings of the Main Conference_ , (June):172–180.
  * Wang and Wan (2018) Ke Wang and Xiaojun Wan. 2018.  Sentigan: Generating sentimental texts via mixture adversarial networks.  _IJCAI International Joint Conference on Artificial Intelligence_ , 2018-July:4446–4452.
  * Wen et al. (2015) Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. 2015.  Semantically conditioned LSTM-based natural language generation for spoken dialogue systems.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 1711–1721, Lisbon, Portugal. Association for Computational Linguistics.
  * Wentzel (1997) Kathryn R Wentzel. 1997.  Student motivation in middle school: The role of perceived pedagogical caring.  _Journal of educational psychology_ , 89(3):411.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _ArXiv_ , abs/1910.03771.
  * Wood and Ruder (2016) Ian Wood and Sebastian Ruder. 2016.  Emoji as emotion tags for tweets.  In _Proceedings of the Emotion and Sentiment Analysis Workshop LREC2016, Portorož, Slovenia_ , pages 76–79.
  * Yang et al. (2018) Yinfei Yang, Steve Yuan, Daniel Cer, Sheng-Yi Kong, Noah Constant, Petr Pilar, Heming Ge, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018.  Learning semantic textual similarity from conversations.  _arXiv preprint arXiv:1804.07754_.
  * Zhang et al. (2018) Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018.  Personalizing dialogue agents: I have a dog, do you have pets too? _ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)_ , 1:2204–2213.
  * Zhou et al. (2018) Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan Zhu, and Bing Liu. 2018.  Emotional chatting machine: Emotional conversation generation with internal and external memory.  _32nd AAAI Conference on Artificial Intelligence, AAAI 2018_ , pages 730–738.
  * Zhou and Wang (2018) Xianda Zhou and William Yang Wang. 2018.  Mojitalk: Generating emotional responses at scale.  _ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)_ , 1:1128–1137.