11institutetext: School of Mechanical Science and Engineering, Huazhong
University of Science and Technology, Wuhan, China 22institutetext: School of
Artificial Intelligence and Automation, Huazhong University of Science and
Technology, Wuhan, China

# Co-Driven Recognition of Semantic Consistency via the Fusion of Transformer
and HowNet Sememes Knowledge

Fan Chen 11 Yan Huang Corresponding Author: Yan Huang (Email:
platanus@hust.edu.cn)22 Xinfang Zhang 11 Kang Luo 11 Jinxuan Zhu 11 Ruixian He
11

###### Abstract

Semantic consistency recognition aims to detect and judge whether the
semantics of two text sentences are consistent with each other. However, the
existing methods usually encounter the challenges of synonyms, polysemy and
difficulty to understand long text. To solve the above problems, this paper
proposes a co-driven semantic consistency recognition method based on the
fusion of Transformer and HowNet sememes knowledge. Multi-level encoding of
internal sentence structures via data-driven is carried out firstly by
Transformer, sememes knowledge base HowNet is introduced for knowledge-driven
to model the semantic knowledge association among sentence pairs. Then,
interactive attention calculation is carried out utilizing soft-attention and
fusion the knowledge with sememes matrix. Finally, bidirectional long short-
term memory network (BiLSTM) is exploited to encode the conceptual semantic
information and infer the semantic consistency. Experiments are conducted on
two financial text matching datasets (BQ, AFQMC) and a cross-lingual
adversarial dataset (PAWSX) for paraphrase identification. Compared with
lightweight models including DSSM, MwAN, DRCN, and pre-training models such as
ERNIE etc., the proposed model can not only improve the accuracy of semantic
consistency recognition effectively (by 2.19%, 5.57% and 6.51% compared with
the DSSM, MWAN and DRCN models on the BQ dataset), but also reduce the number
of model parameters (to about 16M). In addition, driven by the HowNet sememes
knowledge, the proposed method is promising to adapt to scenarios with long
text.

###### Keywords:

semantic consistency HowNet transformer sememes knowledge knowledge fusion.

##  1 Introduction

Semantic consistency recognition (or text semantic matching) task is one of
the important tasks of natural language processing (NLP), and can be applied
to a wide variety of downstream tasks, such as information retrieval, question
answering, dialogue system, machine translation, etc. The input of this task
are mainly sentence pairs, different from natural language inference (and text
entailment) which aims to recognize the semantic relationship (neutral,
entailment, contradiction) between the sentence pairs, the final objective of
semantic consistency recognition is to judge whether the semantic meanings of
the sentence pairs are consistency or similar.

The task of semantic consistency recognition is one of the challenging tasks
due to the characteristics of polysemy and synonymy, which are prominent and
may lead to ambiguity and misunderstanding especially in Chinese language.
However, most of the existing algorithms including bag-of-word (BOW), vector
space model(VSM), term frequency–inverse document frequency (TF-IDF) and DSSM
[2], MwAN [3], DRCN [4] cannot catch the semantic meanings accurately, which
mainly solve the matching or similarity problem at the lexical level and are
difficult to understand text semantics accurately from context. In detail, the
existing text matching algorithms based on lexical coincidence and data-driven
neural network has the following limitations:

(1) The semantic diversity of word vocabularies. The same word can express
different meanings in different context, such as "Apple(苹果)" may denotes a
kind of fruit in a menu but can also denotes "Apple Inc(苹果公司)" in the
electronic product market.

(2) The phrase meaning depends on the order of expression. For a lexical
phrase, the meaning expressed will be completely different if exchange the
order, such as "奶牛(cow)" and "牛奶(milk)" in Chinese, "one another(彼此,互相)" and
"another one(另一个)" in English.

In order to solve the above problems, this paper proposes a semantic
consistency recognition method based on Transformer [1] and HowNet [22], which
expands the research on sentence semantic information acquisition. First, we
use Transformer to conduct multi-level encoding via data-driven for
representation of the internal structure and intrinsic semantics of text
sentences. We then introduce an external knowledge base, i.e. HowNet, to
conduct knowledge-driven modeling of semantic knowledge association between
vocabularies. In addition, soft-attention is exploited to calculate mutual
attention and to achieve knowledge fusion with the semantic matrix. Finally,
BiLSTM is incorporated to further encode the semantic information of the
conceptual level of text and infer the semantic consistency. A number of
experiments show that, compared with the existing lightweight models and the
pre-training models, the fusion of HowNet sememes knowledge and Transformer’s
advantage for long text can improve the accuracy of the semantic consistency
recognition to a certain extent.

The model proposed in this paper has the following innovations:

  * •

In order to solve the problem of text synonyms, polysemy and difficulty to
understand long text for semantic consistency recognition, a co-driven method
based on the fusion of Transformer and HowNet sememes knowledge is proposed.
In addition to Transformer encoding and model pre-training via data-driven,
HowNet sememes are incorporated to enhance the understanding of synonyms and
polysemy via knowledge-driven embedding and inference.

  * •

A technical approach of semantic knowledge fusion is proposed. The multi-level
internal structure and semantic information of sentence pairs are encoded
through Transformer, and the external knowledge base HowNet is introduced to
model the semantic knowledge similarity between sememes sequences. At last,
soft-attention is utilized to calculate the interactive attention and conduct
knowledge fusion via the sememes matrix.

  * •

Experiments are conducted on text matching of ant finance, banking finance
scenarios and multiple paraphrase identification application. Compared with
lightweight models such as DSSM [2], MwAN [3], DRCN [4] and pre-training
models such as ERNIE, the proposed method can not only improve the accuracy of
text semantic consistency recognition effectively, but also reduce the model
parameters. It is worth highlighting that our model is capable of adapting to
long text. The code will be released at https://github.com/Platanus-
hy/sememes_codriven_text_matching.

##  2 Related works

In recent years, with the rapid development of machine learning, a large
number of methods have been proposed to solve the problem of text consistency
recognition. In terms of semantic information acquisition, the classical short
text matching model DSSM [2] solved the problem of dictionary explosion in LSA
(latent semantic analysis), LDA (latent dirichlet analysis) and other methods,
but also loses context information due to the use of the word bag model. The
ESIM [5] model, proposed in 2016, utilizes the BiLSTM and attention mechanism
comprehensively and conducts interaction between sentence pairs in local
reasoning for the first time. The DIIN [6] model proposed in 2018 uses CNN and
LSTM for feature extraction, but the author uses both word vectors and local
vectors in its input layer, inputs some additional syntactic features, and
uses DenseNet for feature extraction. The DRCN model proposed in 2018 learns
form DenseNet [7]’s intensive connection operation for image recognition, it
retains the most original information of the text through intensive connection
to RNN and continuously adds interactive information to the matrix vector
through multiple circulations, and finally outputs via a full connection
layer. The KIM [8] model proposed in 2018 uses the external knowledge base
WordNet [9] to infer the logical relationship between sentence pairs and embed
external prior knowledge into the similarity matrix. In the MwAN [3] model
proposed in 2018, the authors use a variety of attention mechanisms (splicing,
bilinear, dot multiplication, subtraction) to fully capture the relationship
between sentence pairs. Finally, multiple results are weighted and combined to
output the final probability through GRU and full connection layers.

In terms of sentence structure, CT-LSTM [10], which was proposed in 2015,
introduces a tree shaped LSTM to solve the problem that LSTM cannot extract
the structural information of sentences as well as discussed the long sequence
dependency problem. Different from the commonly used RNN sequence modeling, it
uses the dependency relationship of sentences as the input of LSTM, which also
has some inspiration for future research.

With the proposal of BERT [11] model in 2018, a trend of pre-training models
swept the whole NLP world and ranked among the top in the major NLP lists.
BERT has a complete Encoder Decoder framework. Its basic composition is
Transformer, which is mainly composed of multi-head attention. It is a model
built with pure attention, which can solve the problem of long-distance
dependency in text sequences, i.e. the attention mechanism is capable to model
and remember semantic context over longer distance. The advantage of BERT
model is that it can learn more grammatical and semantic information from
large corpus, making the output word vector more representative, the larger
number of parameters improve the expressive ability and make it perform well
in various downstream tasks. In order to accelerate the training speed of the
pre-training models and to tackle the high hardware requirements on consumer
GPUs, Tim Dettmers [12] proposed LLM.Int8() for Transformer, which enables
ordinary consumer GPUs to use very large models without sacrificing
performance. In addition to Transformers, Hanxiao [13] proposed another
simple, attention independent architecture, gMLP, which achieves the same
level as Transformers in some indicators of pre-training, and even outperforms
Transformers in some downstream tasks.

Due to the special polysemy of lexical words, there are certain difficulties
in text matching. Different words often express the same meaning, such as
"China" and "Huaxia", which are semantically consistent, but not related in
terms of grapheme. In order to solve this problem, many researchers choose to
use the speech, dependency syntax and other information to calculate the
similarity. For example, Yan [14] and others tried to label the text with part
of speech, and only reserved nouns, verbs and adjectives. They obtained word
pairs by combining dependency syntax analysis. With PageRank [15] and degree
centrality as indicators, they established a grammar network for a large
number of text, and proposed a text similarity calculation method combining
syntactic relations and lexical semantics. Yan [16] proposed a discourse-level
text automatic generation model based on topical constraints. The synonymy of
the keyword set is used to generate multiple article topic plans. Lin [17]
introduced the concept vector space and represented document as a set of
concept words to build a vector space, then calculated the semantic similarity
through cosine similarity, which performs better than the bag of word (BOW)
model with Word2Vec [18].

The LET model proposed by Boer [19] in 2021 utilizes HowNet for text
entailment recognition. They transformed the initial vectors of all the
sememes for each lexical word with graph attention, then fused the sememes
vectors through attention pooling for each word, and obtained the final word
vector through the integration of GRU and BERT word vectors. Lexical words are
often ambiguous, only a few relevant sememes are essential to identify the
correct semantics, which will lead to the fact that the vector of sememes
obtained mismatch with the actual sentence, resulting in redundant semantic
information. In this paper, the sememes words are filtered in advance before
incorporating into the interaction matrix to avoid adding redundant
information.

##  3 Research Method

This section mainly introduces the co-driven text semantic consistency
recognition model based on the fusion of Transformer and HowNet sememes
knowledge, and analyzes the model structures and their functions. The proposed
model structure is shown in Figure 1, which is divided into 6 layers, namely
the Transformer encoding layer, the attention layer, the BiLSTM layer, the
pooling layer, the fully connected layer and the prediction layer.

Figure 1: Model structure.

###  3.1 The Transformer Encoding Layer

The role of the encoding layer is to model the sequential text to obtain deep
semantic information through the neural network. Instead of utilizing the
commonly used neural networks such as CNN, LSTM, etc., we exploit Transformer
encoder architecture for sentence pair encoding, which is mainly composed of
multi-head attention modules and can alleviate the problem of gradient
vanishing. The multi-head attention mechanism is formalized as in Eq.(1) to
Eq.(4):

${Q}=\mathrm{W}^{Q}X$ (1) ${K}=\mathrm{W}^{K}X$ (2) ${V}=\mathrm{W}^{V}X$ (3)

$\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V$
(4)

where $X$ is the input sentence, $W^{Q},W^{K},W^{V}\in
R^{d_{\text{model}}\times d_{k}}$ are weight matrix, $Q$, $K$, $V$ are
embeddings of $X$ for query, key and value respectively, i.e. query for the
appropriate key using $Q$ and $K$, then select the corresponding value $V$ via
softmax classifier.

In addition, the absolute position encodings are calculated as in Eq.(5) and
Eq.(6).

$PE_{(p,2m)}=\sin\left(\frac{p}{10000^{2m/d}}\right)$ (5)

$PE_{(p,2m+1)}=\cos\left(\frac{p}{10000^{2m/d}}\right)$ (6)

where $d$ denotes the dimension of the word embedding, $p$ denotes the index
of the word in the sentence.

###  3.2 The Attention Layer

Attention layer is an important component of text consistency recognition
model, which has the advantages of fast, effective and lightweight. Various
types of attention mechanisms are proposed in the recent years, such as Soft-
attention, Hard attention, Self-attention, etc. Haili [20] also proposed an
attention mechanism which focus on fine-grained sentiments. This paper adopts
the commonly used soft-attention mechanism, but incorporates the semantic
matrix generated based on HowNet sememes knowledge, which are balanced by
trainable weights $\gamma$

HowNet is a lexical sememes knowledge base for Chinese and English, which
discloses the semantic relationships between the concepts and their sememes.
This paper mainly uses HowNet to obtain all the lexical sememes corresponding
to the words in the sentence pairs. Namely, if two words share the same
sememes, the value of their corresponding position in the HowNet sememes
matrix will be set to 1, otherwise set to 0. Figure 2 shows an example of
sememes interactive calculating for sentence "我们都是中华儿女(We are all childrens of
China)" and sentence "华夏起源于先秦(Huaxia originated in the pre-Qin Dynasty)".

Figure 2: Analysis of HowNet semantics.

The boxes in the top represent the segmented sentences of $P$ and $H$ as input
for word embedding, the middle box represents the multiple sememes information
corresponding to the each word, and the lower box represents the interaction
of the sememes information of two words. The HowNet interaction matrix is
calculated as in Eq.(7) and Eq.(8). It can be seen from the Figure 2 that the
word "China(中华)" has sememes including "China(中国)", “ country(国家)”,
“local(地方)” and "Asia(亚洲)","politics(政)", and the word "Huaxia(华夏)" has
sememes including "borrow(借入)", "finance(金融)", "money(货币)", "China(中国)",
"place(地方)" and "Asia(亚洲)". The intersection of the two words is “China(中国)”,
“place(地方)” and “Asia(亚洲)”.

$M_{i,j}=\left\\{\begin{array}[]{cc}1&\text{ HowNet
}\left(P_{i},H_{j}\right)\neq 0\\\
0&\operatorname{How}\operatorname{Net}\left(P_{i},H_{j}\right)=0\end{array}\right.$
(7) $M=\left[\begin{array}[]{ccc}0&\cdots&1\\\ \vdots&\ddots&\vdots\\\
1&\cdots&0\end{array}\right]$ (8)

where $P$ and $H$ represent the embedding results of word segmentation (via
tools including Jieba, PKUseg, HanLP, etc.) of the sentence pairs
respectively. The vocabularies are vectorized via random initialization when
without pre-training. For pre-training, the word embeddings of Bert are used.
The generation process of the attention matrix is formalized as Eq.(9):

$e=PH^{T}+\gamma\cdot M$ (9)

where $\gamma$ is a trainable parameter. The attention matrix $e$ not only
integrates the structure and semantic information between sentences, but also
obtains the synonymous and polysemous relationships of word pairs between
sentences. The heat map of matrix change is shown in Figure 4. After injecting
the sememes information, the weights of some intersection positions increased,
which indicates that the positions reflect the key features of semantic
consistency between two sentences. After incorporating the attention matrix,
the soft-attention is calculated as follows:

$\hat{P}=\sum_{j=1}^{l_{h}}\frac{\exp\left(e_{ij}\right)}{\sum_{k=1}^{l_{h}}\exp\left(e_{ik}\right)}P_{tf},\forall
i\in\left[1,2,\cdots,l_{p}\right]$ (10)

$\hat{H}=\sum_{j=1}^{l_{p}}\frac{\exp\left(e_{ij}\right)}{\sum_{k=1}^{l_{p}}\exp\left(e_{ik}\right)}H_{tf},\forall
i\in\left[1,2,\cdots,l_{h}\right]$ (11)

In Eq.(10) and Eq.(11), $P_{tf}$ and $H_{tf}$ are embedding matrix of sentence
pairs after encoding via Transformer. $l_{p}$ and $l_{h}$ indicate the length
of the sentences as well as the output of the soft-attention module.

Figure 3: Changes of attention matrix.

###  3.3 The BiLSTM Layer

This layer is used to process the output of Transformer layer after the soft-
attention mechanism, i.e. $\hat{P}$ and $\hat{H}$. Through the Bidirectional
Long Short-Term Memory (BiLSTM), the forward encoding and backward encoding
are concatenated to obtain the context information.

The output of the Long Short-Term Memory (LSTM) network is as follows:

$P_{bi-lstm}=\operatorname{BiLSTM}(\hat{P})$ (12)

where $\hat{P}$ refers to the output of the sentence $P$ via the soft-
attention mechanism, and $P_{bi-lstm}$ refers to the encoding result of
$\hat{P}$ via the BiLSTM module.

###  3.4 Average Pooling and Max-Pooling

In order to fuse the text information after Transformer and BiLSTM, this model
splices multiple inputs through max-pooling and average pooling. The purpose
of this layer is to transform the vector dimension of two sentences from
$R^{l\times d}$ to $R^{l\times d}$ and facilitate the subsequent input of full
connection layer:

$P_{o}=\left[P_{tf};P_{bi-lstm}\right]$ (13)
$P_{rep}=\left[\operatorname{MaxPool}\left(P_{o}\right);\operatorname{AvgPool}\left(P_{o}\right)\right]$
(14)

where $P_{tf}$ denotes the output of Transformer encoding module, $P_{bi-
lstm}$ denotes the output of BiLSTM module, $P_{O}$ is the concatenation of
$P_{tf}$ and $P_{bi-lstm}$, $P_{rep}$ denotes the result after concatenation
of max-pooling and average pooling output of $P_{O}$.

###  3.5 The Fully Connected Layer

After obtaining the complete sentence vector expression for the sentence
pairs, i.e. $P_{rep}$ and $H_{rep}$, the commonly used vector splicing method
is to direct splice and input them into the multi-layer feed-forward neural
network to obtain the results. When conducting splicing, the information in
the HowNet matrix is considered, the sum of two different dimensions in the
HowNet matrix is obtained as $HN_{row}$ (sum for rows) and $HN_{col}$ (sum for
cols). The final input $H$ of the feed forward neural network is obtained via
the concatenation with $P_{rep}$ and $H_{rep}$.

$\begin{gathered}HN_{\text{row}}=\operatorname{sum}(M,axis=0)\\\
HN_{\text{col}}=\operatorname{sum}(M,axis=1)\\\
H=\operatorname{concat}\left(P_{rep};H_{\text{rep}};P_{rep}-H_{rep};HN_{\text{col
}};HN_{\text{row }}\right)\end{gathered}$ (15)

where $sum(M,axis)$ denotes summation of $M$ along the axis dimension. For
example, $HN_{row}$ denotes the result of the sum of the HowNet matrix along
the first dimension, that is, the HowNet information corresponding to sentence
$P$. Through vector concatenation, the corresponding semantic information of
the two sentences is obtained, in which, $P_{rep}-H_{rep}$ also represents the
difference between the two sentence vectors.

###  3.6 The Prediction Layer

After obtaining the final sentence vector representation of the sentence
pairs, the model uses a two-layer fully connected neural network and a softmax
layer to classify the sentence matching results into 1(positive) or
0(negative) as in Eq.(16). The cross entropy loss function is calculated as in
Eq.(17).

$p=softmax(FFN(FFN(H)))$ (16)

$Loss=\frac{1}{N}\sum_{i}^{N}-\left[y_{i}\times\log\left(p_{i}\right)+\left(1-y_{i}\right)\times\log\left(1-p_{i}\right)\right]$
(17)

where $y_{i}$ represents the label of the $i$-th sample $(P_{i},H_{i})$, the
label of positive sample is 1, otherwise 0. $p_{i}$ indicates the probability
that the sample is predicted to be a positive sample.

In addition to the commonly used cross entropy loss function, we also tried
the CoSent loss function, which forces the similarity of positive sample pairs
greater than that of negative samples, and makes the distance between positive
and negative samples in the vector space as far as possible. The experiment
shows that using the CoSent loss function has certain effect on pre-training
methods such as BERT and SentenceBERT [21], reducing the converge time cost of
pre-training models. But for the proposed model in this paper, the effect of
CoSent loss reduces compared with cross entropy loss when without pre-
training.

In the training phase, we used MultiStepLR to dynamically adjust the learning
rate, which reduces the learning rate of each parameter group by a decay rate
of 0.5 once the number of epoch reaches one of the milestones (i.e. the 20th,
50th, 80th, 100th and 150th iterations of the experiment). By dynamically
adjusted the learning rate as the number of iterations increases, the
convergence speed of the model increases, and its variation trend is shown in
Figure 4.

Figure 4: The decay trend of learning rate.

##  4 Experiments And Analysis

###  4.1 Datasets

Table 1: Dataset Size Dataset Name | Training Set Size | Validation Set Size | Test Set Size
---|---|---|---
PAWSX | 49401 | 2000 | 2000
AFQMC | 34334 | 4316 | 3861
BQ Corpus | 100000 | 10000 | 10000

In order to verify the effectiveness of the text consistency recognition model
based on Transformer and HowNet dual drivers proposed in this paper, this
paper conducts experiments on three open datasets respectively. The data sets
are respectively PAWSX data set, AFQMC data set and BQ Corpus data set.

PAWSX dataset is a multilingual definition pair dataset released by Google. It
is characterized by highly overlapping vocabulary, which is helpful to further
improve the model’s judgment on difficult samples, and can test the model’s
judgment ability on similar samples. The AFQMC dataset is an ant financial
similarity dataset, which contains 34,334 training data, 4316 validation data
and 3861 test data. BQ Corpus is the problem matching data set in the banking
and financial field, including the problem pairs extracted from the online
banking system log of one year. It is the largest problem matching data in the
banking field at present, including 10000 training data, 10000 verification
data, and 10000 test data.

Table 2: Example of BQ Dataset Sentence1 | Sentence2 | label
---|---|---
微信消费算吗 (Does Wechat consumption count) |  还有多少钱没还 (How much money is still unpaid) | 0(negative)
下周有什么好产品 (What are the good products next week) |  元月份有哪些理财产品 (What financial products are available in January) | 1(positive)
能查账单吗 (May I check the bill) |  可以查询账单 (You can check the bill) | 0(negative)
现在无法借款 (It’s unable to borrow now) |  QQ有微粒贷平台 (QQ has Weilidai loan product) | 0(negative)

###  4.2 Experimental Setup

The experiments of this paper are conducted on a 4-card GPU server with
RTX2080ti. The parameters and softwares used for model training are shown in
Table 3.

Table 3: Initial training parameters of the model Parameter | Value
---|---
Word-embedding dim | 300
Number of hidden layers | 128
Maximum sequence len | 100
Batch_size | 64
Number of Transformer layers | 10
Optimizer | Adam
Initial learning rate | 0.01
Software | Version
python | 3.6.13
torch | 1.10.2
OpenHowNet | 2.0
transformers | 4.18.0

###  4.3 Comparison of experimental results

In order to verify the actual effect of the model proposed in this paper,
three classical text matching models including DSSM, MwAN and DRCN, are
selected for comparison without pre-training. For the pre-training model, we
chose BERT-wwm-ext, BERT and Baidu ERNIE.

The selected data sets are PAWSX, AFQMC and BQ Corpus. In order to ensure the
unity of the experiment, all models use the same Jieba vocabulary for the same
dataset, and the indicators of comparison are ACC and F1-score.

Table 4: Experimental results on the BQ Dataset Model name | Pre-trained | Acc | F1
---|---|---|---
DSSM | ✗ | 77.12 | 76.47
MwAN | ✗ | 73.99 | 73.29
DRCN | ✗ | 74.65 | 76.02
Ours | ✗ | 78.81 | 76.62
Improvement | ✗ | +2.19% | +1.96%
BERT-wwm-ext | ✔ | 84.71 | 83.94
BERT | ✔ | 84.50 | 84.00
ERNIE | ✔ | 84.67 | 84.20
Ours-BERT | ✔ | 84.82 | 84.33
Improvement | ✔ | +0.177% | +0.464%
Table 5: Experimental Results on the AFQMC Dataset Model name | Pre-trained | Acc | F1
---|---|---|---
DSSM | ✗ | 57.02 | 30.75
MwAN | ✗ | 65.43 | 28.63
DRCN | ✗ | 66.05 | 40.60
Ours | ✗ | 66.62 | 42.93
Improvement | ✗ | +0.86% | +5.7%
BERT-wwm-ext | ✔ | 81.76 | 80.62
BERT | ✔ | 81.43 | 79.77
ERNIE | ✔ | 81.54 | 80.81
Ours-BERT | ✔ | 81.84 | 81.93
Improvement | ✔ | +0.097% | +1.38%
Table 6: Experimental Results on the PAWSX Dataset Model name | Pre-trained | Acc | F1
---|---|---|---
DSSM | ✗ | 42.64 | 59.43
MwAN | ✗ | 52.70 | 52.65
DRCN | ✗ | 61.24 | 56.52
Ours | ✗ | 62.55 | 59.72
Improvement | ✗ | +2.13% | +0.48%
BERT-wwm-ext | ✔ | 77.23 | 76.52
BERT | ✔ | 77.06 | 77.16
ERNIE | ✔ | 78.02 | 77.59
Ours-BERT | ✔ | 78.33 | 77.96
Improvement | ✔ | +0.397% | +0.476%

It can be seen from Table 4 that the accuracy of the proposed model in BQ
dataset is higher than that of other models. As shown in Table 5, from the
perspective of data sets, the results of the three models on AFQMC are not
very good. Preliminary analysis shows that the language standardization of the
sample data is poor, such as incomplete sentences, such as "可以帮我冻结花呗吗(Can you
freeze my Huabei account)" and "冻结花呗额度(Freeze the Huabei credit)". The label
is consistent, which leads to poor results in training set and test set.
However, the model proposed in this paper integrates Transformer and the
HowNet sememes knowledge base, which has a better effect. The results show
that for non-standard text, the performance can be improved by obtaining the
semantic information of some words and matching them. As shown in Table 6, for
the PAWSX dataset where the data samples are difficult to sample, the
traditional DSSM model cannot obtain interactive information and context
information, so the effect is poor. For hard-negative samples, the highly
similar sentence pairs lead to too similar semantic information. The HowNet
matrix generated by each pair of sentences is almost identical, so the method
of obtaining semantic knowledge is not good for judging the positive and
negative samples of difficult samples.

From the perspective of error analysis, because we directly used Jieba word
segmentation to preprocess the text, the word segmentation error produced by
it has different degrees of influence on the experimental results. Although
there are segmentation errors, all models use the same vocabulary for the same
dataset, and the model proposed in this paper is more effective than others.

###  4.4 Ablation Study

In this section, in order to evaluate the necessity and effectiveness of each
module of the model, ablation studies were conducted on the BQ dataset for
different structures of the model proposed. Table 7 shows the evaluation
results of the impact utilizing different word segmentation tools with or
without HowNet. Table 8 shows the evaluation results of the impact with
different sequence length. In addition, the number of Transformer layers for
text encoding is evaluated in Table 9.

Table 7: Semantic consistency recognition results on the BQ dataset using different tokenizers with or without HowNet. Tokenizer | HowNet | Acc | F1
---|---|---|---
Jieba | ✔ | 0.7881 | 0.7662
✗ | 0.7783 | 0.7624
PKUseg | ✔ | 0.7869 | 0.7653
✗ | 0.7792 | 0.7611
HanLP | ✔ | 0.7853 | 0.7599
✗ | 0.7735 | 0.7512

It can be seen from the experimental results in Table 7 that using HowNet can
improve the performance of the model to some extent. Compared with not using
HowNet, the accuracy of various word segmentation tools is improved. If there
are some words with complex semantics in the data sample, the introduction of
external knowledge base can significantly improve the sensitivity of the model
to polysemy and synonym, and can significantly improve the performance of the
model.

Table 8: Semantic consistency recognition results on the BQ dataset for different sequence length with or without HowNet. Sequence length | HowNet | Acc | F1
---|---|---|---
1$\sim$15 | ✔ | 0.7869 | 0.7662
| ✗ | 0.7763 | 0.7521
15$\sim$50 | ✔ | 0.7884 | 0.7684
| ✗ | 0.7792 | 0.7545

According to the results in Table 8, HowNet can effectively improve the
performance of both text with a length of less than 15 and text with a length
of more than 15 and less than 50, and can obtain more valid semantic
information in longer text to achieve better results. After the longest text
segment experiment in the dataset, the longest text length that this model can
handle is 50 on the basis of ensuring the experimental effect.

Table 9: Semantic consistency recognition results on the BQ dataset with different number of Transformer encoding layers. Number of Transformer layers | Acc | F1
---|---|---
2 | 0.7433 | 0.7353
4 | 0.7648 | 0.7572
6 | 0.7752 | 0.7731
8 | 0.7853 | 0.7658
10 | 0.7881 | 0.7662

From the result data in Table 9, the higher the number of Transformer layers,
the better the effect of the model. By stacking the Transformer coding layer,
the performance of the model can be improved to a certain extent, but at the
same time, the number of parameters of the model and the training time of the
model will be significantly increased, and the convergence speed will also be
significantly slower. We take the model with 6 coding layers as the optimal
model, and its parameter quantity is 16M, while the parameter of the DRCN
model which performs the best in the models without pre-training is 19M.
During training, the changes of trainable parameters are as follows:/par

Figure 5: Variation diagram of trainable parameters

It can be seen from Figure 5 that during the experiment, by observing the
changes of the trainable parameters of the attention matrix, the weight of the
sememes information matrix obtained by HowNet in the attention matrix is
gradually increased with the increase of the number of iterations, which
indicates that the HowNet sememes information generated by the original text
has a positive role in improving the model effect.

##  5 Conclusion

This paper proposed a new text semantic consistency recognition model based on
Transformer and HowNet, which uses the HowNet sememes knowledge to tackle the
synonyms and polysemy problems for semantic matching of sentence pairs. The
experiments on datasets including BQ, AFQMC and PAWSX show that, compared with
models without pre-training including DSSM, MwAN and DRCN, as well as pre-
training models such as ERNIE, the proposed method has a certain improvement.
By stacking Transformer layers, the performance of semantic consistency
recognition can be effectively improved while introducing more parameters to
some extent. In contrast, the model proposed in this paper has fewer
parameters and better performance. Compared with LET, which also uses HowNet
as the external information base, this paper filters the irrelevant semantic
information while utilizing HowNet sememes information, avoiding the impact of
redundant sememes on the results, which is more accurate and more intuitive.
Experiments show that the model proposed in this paper can improve the
accuracy of consistency recognition effectively using HowNet sememes
knowledge, and can also adapt to long text scenarios within 50 characters.
There are obvious improvements either with the lightweight model or with the
pre-training model.

In the future, we will study the knowledge limitation of language expression
in depth, i.e. a sentence is wrong in combination with common sense, but is
correct in terms of morphology and syntax. For example, "the sun travels
around the earth" et al. Thus, more work is needed to supplement the external
knowledge, we will continue to utilize commonsense knowledge to further
improve the semantic understanding of raw text.

## References

  * [1] Vaswani A, Shazeer N, Parmar N , et al. Attention Is All You Need. arXiv, 2017.
  * [2] Huang P S, He X, Gao J, et al. Learning deep structured semantic models for web search using clickthrough data，Proceedings of the 22nd ACM international conference on Conference on information $\&$ knowledge management. ACM, 2013
  * [3] Tan C, Wei F, Wang W, et al. Multiway attention networks for modeling sentence pairs, Twenty-Seventh International Joint Conference on Artificial Intelligence. IJCAI-18. 2018.
  * [4] Kim S, Kang I, Kwak N. Semantic Sentence Matching with Densely-connected Recurrent and Co-attentive Information. 2018.
  * [5] Chen Q, hu X, Ling Z, et al. Enhanced LSTM for Natural Language Inference. 2016.
  * [6] Yichen G, Heng L, Jian Z, et al. Natural Language Inference over Interaction Space, ICLR 2018.
  * [7] Huang G, Liu Z, Laurens V, et al. Densely Connected Convolutional Networks, IEEE Computer Society. IEEE Computer Society, 2016.
  * [8] Qian C, Zhu X, Ling Z H, et al. Neural Natural Language Inference Models Enhanced with External Knowledge，Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018.
  * [9] Lin D. Review of: WordNet: An Electronic Lexical Database, computational linguistics, 2002.
  * [10] Tai K S, Socher R, CD Manning. Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks, Computer Science, 2015, 5(1):: 36.
  * [11] Raffel C, Shazeer N, Roberts A, et al. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, 2019.
  * [12] Dettmers T, Lewis M, Belkada Y, et al. LLM.int8: 8-bit Matrix Multiplication for Transformers at Scale. arXiv e-prints, 2022.
  * [13] Hanxiao Liu, Zihang Dai, David R. So, et al. Pay Attention to MLPs. NeurlPS 2021.
  * [14] Jiao Y, Jing M, Kang F. Text Semantic similarity calculation in syntactic network based on Fusion co-occurrence distance, Modern library and information technology, 2019, 003(012):93-100.
  * [15] Page, Lawrence, Brin, et al. The PageRank citation ranking: Bringing order to the web. stanford digital libraries working paper, 1998.
  * [16] Yan H, Haili S, Ke X, at al. Text generation method based on topic constraint. Natural Science Edition of Peking University, 2020, 56(1): 9-15.
  * [17] Lin L, Hui L. A text similarity calculation method based on concept vector space. Data analysis and knowledge discovery, 2018, 2(5): 11.
  * [18] Mikolov T, Chen K, Corrado G, et al. Efficient Estimation of Word Representations in Vector Space. Computer Science, 2013.
  * [19] Boer Lyu, Lu Chen, et al. Let:Linguistic Knowledge Enhanced Graph Transformer for Chinese Short Text Matching. Proceedings of the AAAI Conference on artificial Intelligence. 2021.
  * [20] Haili Sun, Yan Huang, Songfeng Lu. Improving Fine-grained Text Sentiment Transfer for Diverse Review Generation. In: Proceedings of the 2020 IEEE International Conference on Artificial Intelligence and Computer Applications (ICAICA 2020), Dalian, China, June 27-29, 2020. IEEE, 2020: 261-266
  * [21] Reimers N, Gurevych I. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
  * [22] Zhendong Dong, Qiang Dong. HowNet - a hybrid language and knowledge resource. International Conference on Natural Language Processing and Knowledge Engineering, 2003. Proceedings. 2003, Beijing, China, 2003, pp. 820-824, doi: 10.1109/NLPKE.2003.1276017.


# Network Structure shapes the Impact of Diversity in Collective Learning

Fabian Baumann, Agnieszka Czaplicka, and Iyad Rahwan∗

Center for Humans and Machines,
Max Planck Institute for Human Development, Lentzeallee 94, Berlin 14195,
Germany.
Corresponding author<EMAIL_ADDRESS>
> It is widely believed that diversity arising from different skills enhances
> the performance of teams, and in particular, their ability to learn and
> innovate. However, diversity has also been associated with negative effects
> on the communication and coordination within collectives. Yet, despite the
> importance of diversity as a concept, we still lack a mechanistic
> understanding of how its impact is shaped by the underlying social network.
> To fill this gap, we model skill diversity within a simple model of
> collective learning and show that its effect on collective performance
> differs depending on the complexity of the task and the network density. In
> particular, we find that diversity consistently impairs performance in
> simple tasks. In contrast, in complex tasks, link density modifies the
> effect of diversity: while homogeneous populations outperform diverse ones
> in sparse networks, the opposite is true in dense networks, where diversity
> boosts collective performance. Our findings also provide insight on how to
> forge teams in an increasingly interconnected world: the more we are
> connected, the more we can benefit from diversity to solve complex problems.

## Introduction

Living in an increasingly interconnected world humans communicate with higher
rates than ever before [1]. With the goal to optimize collective problem-
solving, previous studies have focused on the effects of network structure on
the overall performance of a population in different tasks. Counter
intuitively, it was found that groups do not necessarily benefit from densely
connected networks, where information can be exchanged freely and spreads
rapidly [2, 3, 4, 5, 6, 7, 8, 9, 10]. In addition to enhanced communication,
increasing global connectivity also means that individuals from different
backgrounds and cultures interact and work together to solve problems
collectively [11]. It is therefore also crucial to investigate how individual
differences within networked groups or societies – often referred to as
diversity – shape the success of collective social endeavors.

Empirical studies have found contradictory, i.e., positive and negative,
effects of diversity on the performance of groups and larger organisations
[12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25]. For instance, an
increased professional diversity predicts greater productivity [26, 13] of
cities. Furthermore, within organizations, the teams with the highest
diversity in educational backgrounds have been found to be the ones that are
most innovative [12]. It was also shown that diverse teams are less prone to
detrimental groupthink [23]. However, there is also evidence that diversity
can negatively impact collective performance through various adverse effects.
Specifically, diversity may impair the ability of groups to communicate and
coordinate [17, 18, 19, 20], be associated with reduced trust between
individuals [21, 13], and can be a source for social tension and conflict [14,
22]. From a theoretical perspective, there is limited understanding of how
network structure and diversity affect collective performance, as previous
modelling efforts have focused on homogeneous populations of identical
individuals [2, 27, 3].

Figure 1: Schematic overview of the model. Panel A depicts the features of the
model that are varied including (i) task complexity, (ii) link density of the
underlying network, and (iii) the composition of the population in terms of
skill diversity. In panels B and C, we schematically depict how agents’ skills
map onto payoff functions $P$. In a homogeneous population, all agents have
the same set of skills (panel B). This results in a single payoff function,
which maps a latent solution $S_{i}$ to payoff value $P(S_{i})$ an agent is
able to extract from it. In homogeneous populations, all agents can extract
the same payoff from a given solution $S_{i}$, and hence there is one optimal
solution $S_{O}$ (star symbol) for all agents. In panel C, we show the case of
a diverse population, where agents generally have different distributions of
skills. For brevity, we depict two agents with different skill sets and their
corresponding distinct payoff functions $P_{\alpha}$ and $P_{\beta}$. As
depicted two agents in diverse populations will generally not extract the same
amount of payoff from a particular solution $S_{i}$. Accordingly, also their
optimal solutions differ (shown as star symbols).

Under which circumstances do networked collectives benefit from diversity? To
answer this question, we build on established modeling efforts, however,
overcome their limitations by considering heterogeneous individuals. In
particular, we study skill diversity within the paradigm of social learning,
which has also been used in empirical studies on collective problem solving
including experiments on the evolution of culture [28, 5, 6], and more narrow
cases where groups solved specific real world tasks [29, 27].

From a theoretical perspective, the dynamics of social learning can be mapped
onto a collective search process in an underlying solution space [2, 3], where
the quality of a solution $S$ is assessed using a payoff function $P(S)$: the
higher the payoff $P(S)$, the better the solution $S$. Importantly, we assume
that the payoff function $P$ does not solely result from the collective task
itself, but is shaped by the skills of agents. Put differently, an agent’s
skill set determines how much value $P$ they can extract from $S$, i.e.,
$P(S)$ corresponds to their contribution to the collective performance of the
group. The mapping of task and skills onto a payoff function is schematically
depicted in Fig. 1.

To navigate the space of latent solutions efficiently, i.e., to rapidly
discover beneficial solutions, models of social learning typically assume that
agents balance exploitative and explorative behaviour [30, 2, 3, 4, 31, 32].
While exploitation corresponds to copying solutions from other agents,
exploration refers to individual learning, where agents discover new solutions
by themselves. The balance between these two types of learning, i.e.,
individual learning and copying, typically leads to an iterative improvement
of both individual and collective performance [30, 3].

Following recent approaches to the study of social learning [5, 4, 3, 33, 34,
2], we assume that individuals are embedded in a social network and can
therefore only copy solutions from a subset of other agents, i.e., their
neighbors in the network. To copy solutions, agents consider the neighbor
which holds the currently best solution and adopt it only if it improves their
payoff [3, 4, 2]. As in previous models, agents innovate locally, i.e., they
can only explore solutions that are close to their current ones. We quantify
the collective performance of a population in terms of the average payoff of
agents $\langle P\rangle=N^{-1}\sum_{i=1}^{N}P_{i}$, where $P_{i}$ corresponds
to the contribution of agent $i$. Hence, each agent contributes to $\langle
P\rangle$ with equal weight. In general, $\langle P\rangle$ can take values
between 0 and 1, and the higher the value of $\langle P\rangle$, the better is
the collective performance. For a more detailed description of the model, see
Methods.

In the following, we investigate the collective dynamics along three
dimensions by varying (i) task complexity, (ii) network structure, and (iii)
group composition with regard to skill diversity. The setup is schematically
depicted in Fig. 1A.

Real-world tasks come in different complexities and it has been shown
theoretically as well as empirically that task complexity strongly impacts
collective performance [2, 3, 29]. The complexity of a task is usually defined
by the statistical properties of its payoff function, in particular, the
correlations between different solutions [35, 2, 3, 29, 4]. In this picture, a
simple problem is defined by a smooth payoff function with a single peak that
corresponds to the optimal solution $S_{O}$, i.e., the solution with the
highest payoff [3, 35]. Social learning is generally well adapted to solve
simple problems and groups are usually much faster in finding $S_{O}$ as
compared to isolated problem solvers [3, 2].

Complex tasks are characterized by a rugged payoff landscape. While rugged
landscapes have an optimal solution $S_{O}$, complex tasks give rise to
additional locally optimal but globally suboptimal solutions to which
collectives tend to converge, and from where they cannot escape [2, 3]. This
feature complicates the navigation of rugged solution spaces and makes complex
tasks harder to solve, both individually and as a group [2, 29, 3]. One of our
goals is to investigate if, and how, the complexity of a task modifies the
effects of skill diversity on the collective performance. In panel A of Fig.
1, task complexity is depicted schematically in terms of its associated payoff
landscape: a smooth landscape with a single optimum is associated with a
simple task and a rugged landscape with multiple (suboptimal) peaks
corresponds to a complex task. Following previous works, we use the NK model
[35, 2, 3] to generate payoff functions for both simple and complex tasks. For
details on the NK model, see Method section.

The effects of network structure on the collective dynamics of homogeneous
problem solvers have gained a lot of attention. Empirical studies [5, 27] and
theoretical models [3, 2] have shown that the structure of the communication
network strongly affects collective problem solving. An important finding is
that it is often not the most densely connected populations that perform best,
but rather the partially connected ones, especially when the task to be solved
is complex [2, 5, 6]. Given the crucial role of networks, we study how skill
diversity and network structure interact to influence the performance of a
group. Specifically, we focus on the interplay between link density and
diversity and consider random networks with different average degrees that
obey a Poisson degree distribution. By that we cover the range from very
sparse to more dense networks, including the limiting case of fully connected
networks.

Overall, our focus is on the dynamics of collective problem solving induced by
populations with (increasing) skill diversity in contrast to homogeneous ones.
As in previous studies, we assume that homogeneous populations are composed of
identical agents [2, 3, 27]. By definition, in homogeneous populations all
agents possess the same distribution of skills, which results in a single
payoff function $P$. Each agent therefore extracts the same amount of value,
$P(S)$, from a particular solution $S$, and there is consequently a unique
solution $S_{O}$, which defines the optimum for all agents with $P(S_{O})=1$.
The mapping from the task and agents’ skills onto the payoff function $P$ is
schematically depicted in Fig. 1B. The optimal solution with $P=1$ is marked
by the star symbol.

In diverse populations the situation is different. We formalize diversity by
assuming that agents generally differ with respect to their skills. As
depicted for two agents with different skill distributions in panel C of Fig.
1, this translates into distinct payoff functions ($P_{\alpha}$ and
$P_{\beta}$) that the agents use to navigate the space of latent solutions. It
generally holds that $P_{\alpha}(S_{i})\neq P_{\beta}(S_{i})$ for a particular
solution $S_{i}$, i.e., the two agents extract different payoff values from a
particular solution. It is a direct consequence of diverse skill sets that
there is no well-defined optimal solution $S_{O}$. Instead, different skill
sets map to different solutions, from which agents extract the highest payoff
(star symbols). In multi-disciplinary teams agents should focus on solutions
that match their abilities. For instance, an individual with high analytical
abilities is expected to extract more payoff from a solution associated with a
mathematical issue, while a co-worker who is more strong in the field of
linguistics should focus on parts of the task that are connected to language
in order to maximize their contribution to the collective performance.

If not denoted otherwise, we consider populations of $N=1000$ agents. We
assume that diverse populations can be subdivided into equally sized classes
of agents with identical skill distributions. Thus by increasing the number of
skill classes, we increase the level of diversity in a population. Note that
the collective performance of homogeneous and diverse populations are indeed
comparable as each payoff function is normalized. Hence, we generally find
that the optimal collective performance is $\langle P\rangle=1$, independent
of the level of diversity.

Consistent with empirical studies [12, 13, 14, 15, 16], we find strong effects
of diversity on collective performance, and we demonstrate that those effects
are mixed, i.e., groups may benefit but also suffer from skill diversity. In
the model, positive and negative effects of diversity arise as a result of
varying the task complexity and the network structure. Diversity generally
reduces the performance of groups in simple tasks. In the case of complex
tasks, the link density of the network modifies the effect of diversity: the
more we are connected, the more we can benefit from diversity to solve complex
problems.

Figure 2: Diversity in simple tasks. Panels A-C show the average payoffs over
time (A), average times $\langle T_{O}\rangle$ to discover the globally
optimal solutions (B), and the normalized histograms of $T_{O}$ (C) for
homogeneous and (increasingly) diverse populations on sparse networks with an
average degree of $\langle k\rangle=4$. Panel D depicts $\langle T_{O}\rangle$
(color coded) as a function of the average degree of the underlying network
and the level of diversity. The reported results are averages over 2500
realizations.

## Results

### Diversity in simple tasks

Figure 2 shows results obtained for simple tasks. In panels A-C, we consider
populations coupled via sparse random networks with an average degree of
$\langle k\rangle=4$, and colors encode different levels of diversity. In
panel D, we vary both the level of diversity and the average degree of the
underlying network.

Figure 2A depicts the collective performance quantified as the average payoff
$\langle P\rangle$ over time. We find that all populations, both homogeneous
(black) and diverse (color coded) ones, eventually converge to the solutions
$S_{O}$ with maximum payoff, $P(S_{O})=1$. However, the time to reach $S_{O}$,
i.e., to extract the maximum payoff collectively depends on the level of
diversity.

To quantify the observed effect of diversity further, we consider the times,
$T_{O}$, that agents need to converge to optimal solutions. Figure 2B depicts
the mean time $\langle T_{O}\rangle$ after which individual agents encounter
$S_{O}$. Clearly, $\langle T_{O}\rangle$ is smallest in homogeneous
populations, increases with diversity, and saturates for high levels of
diversity.

In Fig. 2C, we show the distributions of $T_{O}$ for the different levels of
diversity. While in the case of homogeneous populations the distribution is
narrowly centered around a small value, the distributions are much broader and
characterized by a pronounced positive skew for diverse populations. Note,
that the qualitative change in shape of the distribution already emerges for
low levels of diversity, i.e., in the case of two different skill classes
(cyan line).

Finally, Fig. 2D shows $\langle T_{O}\rangle$ as a function of diversity and
the link density of the network, which is tuned by the value of the average
degree $\langle k\rangle$. While for increasing $\langle k\rangle$, the effect
of diversity becomes less pronounced the qualitative behavior is the same: as
populations become more diverse, the average time $\langle T_{O}\rangle$ to
reach optimal solutions increases. We also find that to solve simple problems
collectively dense networks are generally beneficial as agents converge faster
to optimal solutions in networks with larger values of $\langle k\rangle$.
Note that this trend can be observed for all levels of diversity, varying
along the vertical axis in panel D of Fig. 2.

Taken together, the results reported in Fig. 2 suggest that in simple tasks
diversity hampers collective performance by reducing the speed of collective
problem solving. We find that homogeneous populations are most efficient in
optimizing collective payoffs, and diverse populations need more time. This
(qualitative) conclusion holds independent of the average degree $\langle
k\rangle$. To complement the findings depicted in Fig. 2, in the SM we show
results for different types of networks, which show the same qualitative
behavior.

### Diversity in complex tasks

Figure 3: Diversity in complex tasks. In panels A and B, we depict average
payoffs over time of homogeneous and diverse populations on sparse ($\langle
k\rangle=4$) and more dense ($\langle k\rangle=40$) random networks,
respectively. Panel C depicts the average final payoffs for increasing average
degrees of the underlying network. Panel D shows the average final payoffs (in
gray scale) as a function of the average degree of the underlying network and
the level of diversity. The reported results are averages over 2500
realizations. The shaded areas in panel C depict standard deviations.

Figure 3 summarizes the results for complex tasks. In particular, in panels
A-C, we compare the collective dynamics and final performance of homogeneous
and diverse populations on sparse and dense random networks. In panel D, we
systematically vary both the diversity and the average degree of the
underlying networks over wider ranges.

In Fig. 3A, we depict the collective performance over time on sparse networks
with the same average degree as in panels A-C of Fig. 2, i.e., of $\langle
k\rangle=4$. First, we note that both the homogeneous and diverse populations
(20 payoff classes) are not able reach the optimal collective performance
$\langle P\rangle=1$. Instead, due to the ruggedness of the complex solution
space populations tend to converge to globally suboptimal peaks, which reduces
the performance to $\langle P\rangle<1$. As in the case of simple tasks,
homogeneous and diverse populations converge quickly and more slowly to their
performance level, respectively. More importantly, however, we find that the
final performance of diverse populations is worse than the one reached by
homogeneous populations. As we will see in the following, this is not always
the case and homogeneous populations do not generally outperform diverse ones
in complex tasks. Instead, the effect of diversity on collective performance
depends on the link density of the underlying network.

In Fig. 3B, we show results for a random network with the tenfold average
degree as compared to panel A, i.e., $\langle k\rangle=40$. By fixing all
other model parameters, we find that the increased link density significantly
impacts the collective dynamics and qualitatively reverses the previous
result. Although, again, diverse populations converge more slowly, the final
collective performance of diverse populations is superior to the one reached
by homogeneous populations. These findings suggest that collective problem
solving in networked populations exhibits a transition point with respect to
link density above which diversity becomes beneficial and boosts performance.

To visualize this phenomenon more clearly, Fig. 3C depicts the final
performance of homogeneous and diverse populations in networks of increasing
average degree. For low average degrees, homogeneous populations outperform
diverse ones, i.e., final solutions found by homogeneous populations have on
average higher payoffs. However, as $\langle k\rangle$ increases, diversity
becomes beneficial and the final collective performance reached by diverse
populations exceeds the one of homogeneous populations. As we show in the SM,
this result holds in the limiting case of a fully connected network, where the
final collective performance is lowest for homogeneous populations and
improves for higher levels of diversity.

In Fig. 3D, we shed further light on the combined effect of link density and
diversity by varying both simultaneously. In gray scale we show the final
collective performance as a function of the average degree and the number of
skill classes. As we have already learnt from Fig. 3C, the performance of
homogeneous populations is impaired as the average degree of the network
increases. Similarly, on sparse networks the introduction of diversity, i.e.,
the increase of the number of skill classes, leads to a strong decrease in
performance. This is not the case for networks with a higher link density
(increasing values of $\langle k\rangle$). Instead, we find that the more
dense the network becomes, the higher is the ideal level of diversity, which
optimizes the final collective performance of the population. To demonstrate
that these findings are not restricted to random networks with Poisson degree
distribution, we performed additional simulation on random networks with fixed
degree and small world networks, see SM.

## Discussion

While various experimental works have found strong effects of diversity on
collective performance, theoretical studies that provide a mechanistic
understanding of the underlying dynamics are scarce. Here, we fill this gap by
means of a simple model of social learning that we extend by agents that
differ with respect to their skills. In line with previous studies, we find
that diversity can be both beneficial and detrimental to collective
performance [12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 36]. Our modeling
approach allows to draw intuitive conclusions on the effects of diversity that
are related to the underlying network structure and the complexity of the
task.

For simple problems, all populations – regardless of their level of diversity
– finally reach the optimal collective performance. To gauge the effect of
diversity, we have quantified how much time it takes for populations to
converge to $\langle P\rangle=1$. Diversity generally slows down the speed of
convergence optimal performance and therefore hampers collective problem
solving in terms of efficiency. While this effect is more pronounced on sparse
networks, it holds qualitatively true on densely connected networks.

In the case of complex tasks, the situation is more differentiated. There is
no constant effect of diversity on collective problem solving. Instead, the
effects of skill diversity depend on the underlying network structure. On
sparse networks with a low average degree, diversity impairs collective
problem solving and diverse populations perform worse as compared to
homogeneous ones. However, as soon as networks become sufficiently dense
diversity becomes beneficial and can boost collective performance beyond that
of homogeneous populations.

There are two separate effects introduced by skill diversity that allow to
qualitatively understand these findings. First, in diverse populations a
particular solution $S$ is not evaluated equally by all agents. Thus, a
solution may be profitable for some agents (they can extract a high payoff
value from it), but not for others (low payoff): we generally find
$P_{\alpha}(S)\neq P_{\beta}(S)$ as depicted in panel B of Fig. 1. This
feature of diversity introduces noise to the dynamics of collective problem
solving. It negatively affects the collective filtering and efficient
dissemination of solutions throughout the population.

In the case of complex tasks, there is a second effect of diversity that
results from the ruggedness of the payoff landscapes. It is not only that two
agents with different skill distributions extract different payoff values from
a particular solution $S$. Additionally, the local neighborhood of a solution
in the payoff space may differ. Indeed, while $S$ may correspond to a local
peak, or maximum, in the payoff function of one agent where they can get
stuck, $S$ potentially corresponds to a local minimum for a differently
skilled agent. The second agent is then able to improve upon the solution by
local exploration. This feature gives rise to a mechanism that helps agents in
diverse populations to escape suboptimal solutions to complex task. In other
words, precisely because two agents have different skill sets, they can
mutually benefit from sharing information to improve their solutions.

In simple tasks, the payoff landscape is smooth and agents are able to reach
solutions that lead to optimal performance, even by individual and local
exploration only [37]. Copying from peers with identical skills accelerates
this process as time resources spent on individual learning are saved [30]. In
essence, solutions are evaluated equally by all agents and homogeneous
populations can efficiently filter and disseminate beneficial solutions. The
same process is disrupted in diverse populations, as agents with different
skills do not agree on the value of a particular solution ($P_{\alpha}(S)\neq
P_{\beta}(S)$), ultimately decreasing the convergence rate towards optimal
collective performance $\langle P\rangle=1$. The finding is in line with
empirical studies, suggesting that diversity often reduces the ability of
groups to communicate efficiently, thereby creating an obstacle for collective
problem solving in simple tasks [14].

In complex tasks, the situation is different and diversity can boost
collective performance: diverse populations outperform homogeneous ones on
densely connected networks. As argued previously, homogeneous populations
rapidly collapse to, and get trapped at, suboptimal solutions when they copy
solutions from their best performing peers [2, 3]. Noise from various sources,
originating either from different social learning strategies, less well-
connected networks or distrust between individuals, has been shown to lead to
better exploration of the solution space and improved performance [3, 4, 2,
38]. Here we identify diversity as another way to boost collective
performance. Different skill sets that map onto distinct payoff functions
allow agents to mutually assist each other in escaping local optima, a benefit
of diversity that is based on the information exchange among dissimilar peers.

While our work provides a novel perspective on the relationship between
diversity, network structure, and collective performance it comes with
noteworthy limitations. We considered the specific type of diversity, where
the population can be subdivided in equally sized skill classes. Generally,
however, it can be assumed that groups of any kind are not of the same size.
As previously shown for different types of social dynamics ranging from
collective opinion change to the emergence of conventions a broader
distribution of group sizes can affect system level outcomes [39, 40]. An
interesting extension to our presented results could consider different task
complexities simultaneously. More specifically, our framework allows to
investigate cases where a particular task is complex for most agents in the
population, while others can solve it easier. This may shed light on the role
of diversity in the context of generalists and specialists, which has
previously been studied for homogeneous populations [33].

Overall, our findings suggest that diversity offers a pathway to boost
collective performance in complex tasks. Specifically, we have shown that the
more we are connected, the more we can benefit from diversity to solve complex
problems – a finding which informs the compilation of problem solving teams in
an increasingly interconnected and diverse world.

## Methods

### Dynamics of social learning

We consider a populations of $N$ agents, which are embedded in undirected
networks. In each discrete time step $t$, agent $i$ aims to improve their
solution in a two-step procedure. First, they attempt to learn, or copy, a
solution from a neighbour in the social network. As in previous studies (e.g.
Refs. [2, 3, 4]), we implement a copying strategy, where agents follow their
best neighbor, defined as the agent that currently holds the best solution,
i.e., the solution that gives rise to the highest payoff for agent $i$. If the
solution improves the current payoff of agent $i$, they adopt it. Otherwise,
agent $i$ switches to innovation, i.e. they locally explore the solution
landscape in search of a new, unknown solution. To innovate a new solution by
local exploration agents modify a single digit of their current solution. As
in the case of copying, the innovated solution is only adopted if it increases
the agent’s current payoff [41, 2, 35, 3].

To evaluate solutions, and to compare them to each other, agents use a payoff
function $P$, which maps a solution $S$ to a numerical value $P(S)$ that lies
between zero and one: the higher $P$, the more value an agent can extract,
i.e., the better the solution. In homogeneous populations all agents utilize
the same payoff function $P$ and therefore evaluate each solution equally.
Instead, in a diverse population agents use different payoff functions. A
particular solution is evaluated differently by two agents $i$ and $j$, which
belong to payoff classes $\alpha$ and $\beta$, respectively, as we generally
find $P_{\alpha}(S)\neq P_{\beta}(S)$.

### Generating payoff functions

To generate payoff landscapes corresponding to varying task complexities we
use the NK model [35]. In its original version the model is fully defined by
two parameters, $N_{\mathrm{NK}}$ and $K_{\mathrm{NK}}$. Each solution $S$ in
the problem space can be regarded as a vector of $N_{\mathrm{NK}}$ bits, or
elements, and has a well defined payoff value $P$. The total number of
possible solutions is thus given by $2^{N_{\mathrm{NK}}}$. The payoff
contribution of a particular solution $S$ is defined as the mean payoff
contribution of each element, which is a random number drawn from a uniform
distribution on the interval between 0 and 1. For $K_{\mathrm{NK}}=0$, the
average is simply taken over the contribution of each element $b_{i}$, i.e. we
have $P=N^{-1}\sum_{i}b_{i}$. For $K_{\mathrm{NK}}>0$, the contribution of
each element $f(b_{i}|b_{i},b_{i+1},\dots,b_{K_{\mathrm{NK}}})$ is determined
by $K_{\mathrm{NK}}$ additional and independent elements. The total payoff is
then computed as
$P=N^{-1}\sum_{i}f(b_{i}|b_{i},b_{i+1},\dots,b_{K_{\mathrm{NK}}})$, where the
$K_{\mathrm{NK}}$ elements are determined randomly.

Following previous studies, we first normalize the payoff function $P$ by its
maximum value $P_{\mathrm{max}}$ [3, 2, 42]. For intermediate values of
$K_{\mathrm{NK}}$ the distribution of payoffs tends towards a normal
distribution, such that most solutions have similar payoffs, which makes it
hard to distinguish collective performance in different conditions [3]. We
therefore take the normalized payoff function to the power of 8, such that
there are only very few good solutions, as was done in [3, 2]. We distinguish
between two scenarios: simple and complex tasks for which we set the
parameters of the NK model to $(N_{\mathrm{NK}},K_{\mathrm{NK}})=(15,0)$ and
$(N_{\mathrm{NK}},K_{\mathrm{NK}})=(15,7)$, respectively.

## Code availability

Simulation code is available at https://github.com/naibaf2/diversity

## References

  * [1] Homero Gil de Zúñiga and Sebastián Valenzuela.  The mediating path to a stronger citizenship: Online and offline networks, weak ties, and civic engagement.  Communication Research, 38(3):397–421, 2011.
  * [2] David Lazer and Allan Friedman.  The network structure of exploration and exploitation.  Administrative Science Quarterly, 52(4):667–694, 2007.
  * [3] Daniel Barkoczi and Mirta Galesic.  Social learning strategies modify the effect of network structure on group performance.  Nature Communications, 7(1):1–8, 2016.
  * [4] Winter Mason and Duncan J Watts.  Collaborative learning in networks.  Proceedings of the National Academy of Sciences, 109(3):764–769, 2012.
  * [5] Maxime Derex and Robert Boyd.  Partial connectivity increases cultural accumulation within groups.  Proceedings of the National Academy of Sciences, 113(11):2982–2987, 2016.
  * [6] Maxime Derex, Charles Perreault, and Robert Boyd.  Divide and conquer: intermediate levels of population fragmentation maximize cultural accumulation.  Philosophical Transactions of the Royal Society B: Biological Sciences, 373(1743):20170062, 2018.
  * [7] Winter A Mason, Andy Jones, and Robert L Goldstone.  Propagation of innovations in networked groups.  Journal of Experimental Psychology: General, 137(3):422, 2008.
  * [8] Maxime Derex and Alex Mesoudi.  Cumulative cultural evolution within evolving population structures.  Trends in Cognitive Sciences, 24(8):654–667, 2020.
  * [9] Andrea B Migliano, Federico Battiston, Sylvain Viguier, Abigail E Page, Mark Dyble, Rodolph Schlaepfer, Daniel Smith, Leonora Astete, Marilyn Ngales, Jesus Gomez-Gardenes, et al.  Hunter-gatherer multilevel sociality accelerates cumulative cultural evolution.  Science Advances, 6(9):eaax5913, 2020.
  * [10] Mauricio Cantor, Michael Chimento, Simeon Q Smeele, Peng He, Danai Papageorgiou, Lucy M Aplin, and Damien R Farine.  Social network architecture and the tempo of cumulative cultural evolution.  Proceedings of the Royal Society B, 288(1946):20203107, 2021.
  * [11] Ankita Saxena.  Workforce diversity: A key to improve productivity.  Procedia Economics and Finance, 11:76–85, 2014.
  * [12] Suzanne T Bell, Anton J Villado, Marc A Lukasik, Larisa Belau, and Andrea L Briggs.  Getting specific about demographic diversity variable and team performance relationships: A meta-analysis.  Journal of Management, 37(3):709–743, 2011.
  * [13] Alberto Alesina and Eliana La Ferrara.  Who trusts others?  Journal of Public Economics, 85(2):207–234, 2002.
  * [14] Robin Schimmelpfennig, Layla Razek, Eric Schnell, and Michael Muthukrishna.  Paradox of diversity in the collective brain.  Philosophical Transactions of the Royal Society B, 377(1843):20200316, 2022.
  * [15] Niccolò Pescetelli, Alex Rutherford, and Iyad Rahwan.  Modularity and composite diversity affect the collective gathering of information online.  Nature Communications, 12(1):1–10, 2021.
  * [16] Sujin K Horwitz and Irwin B Horwitz.  The effects of team diversity on team outcomes: A meta-analytic review of team demography.  Journal of Management, 33(6):987–1015, 2007.
  * [17] Roberto A Weber and Colin F Camerer.  Cultural conflict and merger failure: An experimental approach.  Management Science, 49(4):400–415, 2003.
  * [18] Ivan Dale Steiner.  Group Process and Productivity.  Academic press, 1972.
  * [19] Sharolyn Converse, JA Cannon-Bowers, and E Salas.  Shared mental models in expert team decision making.  Individual and Group Decision Making: Current issues, 221:221–46, 1993.
  * [20] Matthew A Cronin and Laurie R Weingart.  Representational gaps, information processing, and conflict in functionally diverse teams.  Academy of Management Review, 32(3):761–773, 2007.
  * [21] Robert D Putnam.  E pluribus unum: Diversity and community in the twenty-first century the 2006 johan skytte prize lecture.  Scandinavian Political Studies, 30(2):137–174, 2007.
  * [22] Ulrich J Eberle, J Vernon Henderson, Dominic Rohner, and Kurt Schmidheiny.  Ethnolinguistic diversity and urban agglomeration.  Proceedings of the National Academy of Sciences, 117(28):16250–16257, 2020.
  * [23] Jeffrey L Coles, Naveen D Daniel, and Lalitha Naveen.  Director overlap: groupthink versus teamwork.  Available at SSRN 3650609, 2020.
  * [24] Ray Reagans and Ezra W Zuckerman.  Networks, diversity, and productivity: The social capital of corporate r&d teams.  Organization science, 12(4):502–517, 2001.
  * [25] Damon Centola.  The network science of collective intelligence.  Trends in Cognitive Sciences, 2022.
  * [26] Esteban Moro, Morgan R Frank, Alex Pentland, Alex Rutherford, Manuel Cebrian, and Iyad Rahwan.  Universal resilience patterns in labor markets.  Nature communications, 12(1):1–8, 2021.
  * [27] Duncan J Watts and Steven H Strogatz.  Collective dynamics of ‘small-world’networks.  Nature, 393(6684):440–442, 1998.
  * [28] Robert Boyd and Peter J Richerson.  Culture and the evolutionary process.  University of Chicago press, 1988.
  * [29] Abdullah Almaatouq, Mohammed Alsobay, Ming Yin, and Duncan J Watts.  Task complexity moderates group synergy.  Proceedings of the National Academy of Sciences, 118(36):e2101062118, 2021.
  * [30] Kevin N Laland.  Social learning strategies.  Animal Learning & Behavior, 32(1):4–14, 2004.
  * [31] Luke Rendell, Robert Boyd, Daniel Cownden, Marquist Enquist, Kimmo Eriksson, Marc W Feldman, Laurel Fogarty, Stefano Ghirlanda, Timothy Lillicrap, and Kevin N Laland.  Why copy others? insights from the social learning strategies tournament.  Science, 328(5975):208–213, 2010.
  * [32] Christina Fang, Jeho Lee, and Melissa A Schilling.  Balancing exploration and exploitation through structural design: The isolation of subgroups and organizational learning.  Organization Science, 21(3):625–642, 2010.
  * [33] Marco Smolla and Erol Akçay.  Cultural selection shapes network structure.  Science Advances, 5(8):eaaw0609, 2019.
  * [34] Abdullah Almaatouq, Alejandro Noriega-Campero, Abdulrahman Alotaibi, PM Krafft, Mehdi Moussaid, and Alex Pentland.  Adaptive social networks promote the wisdom of crowds.  Proceedings of the National Academy of Sciences, 117(21):11379–11386, 2020.
  * [35] Stuart Kauffman and Simon Levin.  Towards a general theory of adaptive walks on rugged landscapes.  Journal of Theoretical Biology, 128(1):11–45, 1987.
  * [36] Lu Hong and Scott E Page.  Groups of diverse problem solvers can outperform groups of high-ability problem solvers.  Proceedings of the National Academy of Sciences, 101(46):16385–16389, 2004.
  * [37] Daniel A Levinthal.  Adaptation on rugged landscapes.  Management Science, 43(7):934–950, 1997.
  * [38] Giovanni F Massari, Ilaria Giannoccaro, and Giuseppe Carbone.  Are distrust relationships beneficial for group performance? the influence of the scope of distrust on the emergence of collective intelligence.  International Journal of Production Economics, 208:343–355, 2019\.
  * [39] Damon Centola, Joshua Becker, Devon Brackbill, and Andrea Baronchelli.  Experimental evidence for tipping points in social convention.  Science, 360(6393):1116–1119, 2018.
  * [40] Andrea Civilini, Nejat Anbarci, and Vito Latora.  Evolutionary game model of group choice dilemmas on hypergraphs.  Physical Review Letters, 127(26):268301, 2021.
  * [41] Felipe A Csaszar and Nicolaj Siggelkow.  How much to copy? determinants of effective imitation breadth.  Organization Science, 21(3):661–676, 2010.
  * [42] Nicolaj Siggelkow and Jan W Rivkin.  Speed and search: Designing organizations for turbulence and complexity.  Organization Science, 16(2):101–122, 2005.

## Supplemental Information

In this Supplemental Information, we report results which demonstrate that our
main findings on the effects of skill diversity hold for varied system sizes
and different types of networks.

### Simple tasks

In Fig. 4, we show results for the collective performance in simple tasks for
different types of networks. All other model parameters are identical to those
considered in Fig. 2 of the main text.

Panel A shows results for random networks that are generated by the
configuration model with a constant degree of $k=4$, i.e., with a degree
distribution $p(k)=\delta(k-4)$. In panel B, we consider the case of small-
world networks, generated by the Watts-Strogatz model.

As for random networks with a Poisson degree distribution (considered in the
main text), we find that in both cases, i.e., for random networks with fixed
degree and small-world networks, diversity is detrimental in simple tasks: the
time to reach optimal performance increases with diversity.

### Complex tasks

#### Performance on a fully connected network

Figure 7 shows the collective performance over time obtained on a fully
connected network for a complex task. Different colors correspond to different
levels of diversity.

In the main text, we have reported that in complex tasks the benefits of
diversity increase for increasing link densities of the underlying random
networks. Here we show that the result holds in the limit of a fully connected
network: final collective performance increases with increasing levels of
(skill) diversity.

#### Varying the system size

In the main text, we considered populations of size $N=1000$. In Fig. 5, we
show additional results for different system sizes.

In particular, we reproduced Fig. 3 (panels A and B) of the main text with
half ($N=500$, Fig. 5A) and double the system size ($N=2000$, Fig. 5B). As
expected, we find that the final values of the collective performance differ
slightly. In general, higher values of $N$ lead to better performance, due to
the fact that large populations can sample the payoff landscape more
thoroughly than small populations.

More importantly, however, the qualitative behavior reported in the main text
is recovered. In both cases, for smaller (panel A) and larger (panel B) system
sizes, diverse populations outperform homogeneous ones on dense networks,
while the opposite is true for sparse networks.

#### Different network models

In Fig. 6, we reproduced Fig. 3 (panels A and B) of the main text for
different types of networks, while fixing all other model parameters.

In panel A, we show results for random networks generated by the configuration
model with fixed degree $k$, which are either sparse ($k=4$, left) or dense
($k=40$, right). In panel B, we consider the case of sparse ($\langle
k\rangle=4$) and dense ($\langle k\rangle=40$) small-world networks.

In both cases our main conclusion holds: link density modifies the effect of
diversity, and diverse populations outperform homogeneous ones in densely
connected networks.

Figure 4: Collective performance in simple tasks. Mean times $\langle
T_{0}\rangle$ to reach optimal performance (left) and the corresponding
distributions of $T_{0}$ for different levels of diversity (right). In panel
A, we show results for random networks generated by the configuration model
with a fixed degree of $k=4$, i.e. each node has four adjacent nodes. In panel
B, we show results for small world networks generated by the Watts-Strogatz
model, where each link of a regular grid network is rewired with probability
$p_{r}$. Here we chose a regular gird network where each node is connected to
its $k=4$ nearest neighbors and set the rewiring probability to $p_{r}=0.1$.
In all shown cases the system size is $N=1000$. The results are averaged over
2500 realizations. Figure 5: Collective performance in complex tasks for
$N=500$ (panel A) and $N=2000$ (panel B). The average degrees of the
underlying random networks are identical to those of Fig. 3 (panels A and B)
of the main text, i.e., we set $\langle k\rangle=4$ and $\langle k\rangle=40$
for sparse and dense networks, respectively. The results are averaged over
2500 realizations. Figure 6: Collective performance in complex tasks. Panel A
shows results for networks generated by the configuration model with a
constant degree for each node: $k=4$ (left) and $k=40$ (right). In panel B, we
show results for small-world networks generated by the Watts-Strogatz model,
where each link of a regular grid network is rewired with probability $p_{r}$.
Here we chose a regular gird network where each node is connected to its $k=4$
(left) and $k=40$ (right) nearest neighbors and set the rewiring probability
to $p_{r}=0.1$. In all shown cases the system size is $N=1000$. The results
are averaged over 2500 realizations. Figure 7: Collective performance over
time for populations with increasing levels of diversity (number of skill
classes) on a fully connected network. The colors of the trajectories encode
the number of skill classes in populations of $N=1000$ agents. The results are
averaged over 2500 realizations.