# A Perceptual Quality Assessment Exploration for AIGC Images

###### Abstract

AI Generated Content (AIGC) has gained widespread attention with the
increasing efficiency of deep learning in content creation. AIGC, created with
the assistance of artificial intelligence technology, includes various forms
of content, among which the AI-generated images (AGIs) have brought
significant impact to society and have been applied to various fields such as
entertainment, education, social media, etc. However, due to hardware
limitations and technical proficiency, the quality of AIGC images (AGIs)
varies, necessitating refinement and filtering before practical use.
Consequently, there is an urgent need for developing objective models to
assess the quality of AGIs. Unfortunately, no research has been carried out to
investigate the perceptual quality assessment for AGIs specifically.
Therefore, in this paper, we first discuss the major evaluation aspects such
as technical issues, AI artifacts, unnaturalness, discrepancy, and aesthetics
for AGI quality assessment. Then we present the first perceptual AGI quality
assessment database, AGIQA-1K, which consists of 1,080 AGIs generated from
diffusion models. A well-organized subjective experiment is followed to
collect the quality labels of the AGIs. Finally, we conduct a benchmark
experiment to evaluate the performance of current image quality assessment
(IQA) models.

###### Index Terms:

AI-generated content (AIGC), AGI, quality assessment, subjective experiment

##  I Introduction

AI Generated Content (AIGC) refers to any form of content, such as text,
images, audio, or video, that is created with the help of artificial
intelligence technology. With the flourishing development of deep learning,
the efficiency of AIGC generation has increased, and AIGC Images (AGIs) are
becoming more prevalent in areas such as culture, entertainment, education,
social media, etc. Unlike natural scene images (NSIs) that are captured from
the natural scenes, AGIs are directly generated from AI models as shown in Fig
1. Namely, diffusion models [1] and generative adversarial networks [2] are
capable of generating a great number of images according to our needs.
However, due to the hardware limitations and technical proficiency, the
quality of AGIs is inconsistent and various, which often requires refinement
and filtering before exhibition and being put into practical use. Thus,
objective models for evaluating the quality of AGIs are urgently needed.

Fig. 1: Illustration of the generation process of AGIs and NSIs, where NSIs
are captured from the natural scenes and AGIs are directly generated from AI
models.

During the last decade, large amounts of effort have been put into
constructing image quality assessment (IQA) databases and proposing IQA
methods for common image contents, such as NSIs [3], JPEG2000-compressed [4],
cartoon [5], computer-generated [6], contrast-changed[7], high dynamic range
(HDR) [8], in-the-wild [9], screen content[10], and omnidirectional [11]
images. All these kinds of images can share some common technical quality
assessment dimensions such as illumination, blur, contrast, texture, etc. AGIs
obtain some unique quality characteristics and viewers tend to evaluate the
quality of AGIs from some different aspects although AGIs are generated under
restrictions to be similar to the training images such as NSIs. We summarize
some major quality assessment aspects for AGIs here: a) Technical issues,
which refer to the common distortions that affect the visibility of the image
content; b) AI artifacts, which indicates the confusing and unexpected
components appeared in the images; c) Unnaturalness, which stands for the
unnaturalness that goes against common sense and the discomfort during the
viewing experience; d) Discrepancy, which denotes the mismatch extent between
the AGIs and our expectation; e) Aesthetics, which refers to the overall
visual appeal and beauty of the images.

Fig. 2: Sample images from the AGIQA-1k database, where the first to sixth rows show AGIs with (_bird, cat, batman, kid, man, woman_) as the main objects respectively.  Table I: Illustration of text keywords for generating the AGIs. All the keywords mentioned in the table are used for the stable-diffusion-v2 while the keywords marked with * are excluded for the stable-inpainting-v1. Keywords | Content
---|---
Main Objects | Bird, Cat, Dog, Batman,
Snoppy, Teddy bear, Kid*, Man,
Woman, Alien, Demon*, Witch*
Second Objects | Driving Aircraft*/Bike*/Car*,
Having Hamburger*/Ice-cream*/Pizza,
Playing Baseball/Football/Yoga,
Using CD/Laptop/Phone,
Wearing Coat/Hat/Shirt
Places | City, Wild
Styles | Anime Style, Realistic Style

However, there has been no scientific research specifically targeted on the
perceptual quality of AGIs currently. Therefore, in this paper, we embark on a
certain exploration to address the challenge of evaluating the quality of AIGC
by carrying out a first-of-a-kind perceptual quality assessment database for
AGIs, named AGIQA-1K. Specifically, we employ two latent text-to-image
diffusion models [1] stable-inpainting-v1 and stable-diffusion-v2 as the AGI
models. Then we choose several most popular text keywords from the Internet
for AGI generation and a total of 1,080 AGIs are obtained. Afterward, we carry
out a subjective experiment in a well-controlled laboratory environment, where
the subjects are asked to perceptually evaluate the quality of AGIs following
the major quality aspects discussed above. Finally, a benchmark experiment is
conducted to evaluate the performance of current IQA models and in-depth
discussions are given as well. Our contributions are proposed as follows:

  * •

We propose a thorough quality assessment guideline for AGIs, the major
evaluation aspects include technical issues, AI artifacts, unnaturalness,
discrepancy, and aesthetics.

  * •

We are the first to carry out a perceptual AGI quality assessment database
(AGIQA-1K), which provides 1,080 AGIs along with quality labels.

  * •

A benchmark experiment is conducted to evaluate the performance of current IQA
models.

(a) NSI distributions (b) AGI distributions

Fig. 3: The normalized probability distributions of the quality-related
attributes for NSIs and AGIs. The distributions are obtained from 10,073 NSIs
in the KonIQ-10k IQA database [12] and 1,080 AGIs in the proposed AGIQA-1k
database respectively. The ’color’ indicates the colorfulness of the images
and the ’SI’ (spatial information) stands for the content diversity of the
images.

(a) Blur

(b) Unexpected artifact

(c) Unnaturalness

(d) Simple and Text unmatch

Fig. 4: Exhibition for some common AGI distortions, where the generation
keywords are marked in the top right. The content of (a) is poorly visible due
to the blur and inexplicit texture. Unexpected artifacts are introduced to the
bottom left of (b). (c) contains unnatural content such as the hands of the
woman do not cope with common sense. (d) is too simple and does not fit the
text keyword of “Wearing Coat”.

##  II Database Construction

###  II-A AGIs Collection

Considering the success of stable diffusion models, we select two text-to-
image diffusion models stable-inpainting-v1 and stable-diffusion-v2 (sub-
models derived from [1]) as the AGI models. To ensure content diversity and
catch up with the popular trends, we use the hot keywords from the PNGIMG
website 111https://pngimg.com/ for AGIs generation, and the employed keywords
are exhibited in Table I, which contains the main objects, the second objects,
places, and styles. Some sample AGIs are further illustrated in Fig. 2.

In order to evaluate the statistical discrepancy between NSIs and AGIs, we
present the distributions of five quality-related attributes for comparison.
The NSIs are sourced from the in-the-wild KonIQ-10k IQA database [12], while
the AGIs are collected through the proposed AGIQA-1K database. The quality-
related attributes under consideration are light, contrast, colorfulness,
blur, and spatial information (SI). Detailed descriptions of these attributes
can be found in [13]. As shown in Fig. 3, the quality-related attribute
distributions of NSIs and AGIs are quite similar and tend to be Gaussian-like.
Specifically, AGIs are relatively blurrier and contain more spatial
information than NSIs.

###  II-B Subjective Experiment

To evaluate the quality of AGIs, a subjective experiment is conducted
following the guidelines of ITU-R BT.500-13 [14]. The subjects are asked to
rate the overall quality levels of exhibited AGIs from the technical issues,
AI artifacts, unnaturalness, discrepancy, and aesthetic aspects. Some typical
distortion examples are shown in Fig. 4. The AGIs are presented in random
order on an iMac monitor with a resolution of up to 4096 $\times$ 2304, using
an interface designed with Python Tkinter, as shown in Fig. 5. The interface
allows viewers to browse the previous and next AGIs and rate them using a
quality scale that ranges from 0 to 5, with a minimum interval of 0.1. A total
of 22 graduate students (10 males and 12 females) participate in the
experiment, and they are seated at a distance of around 1.5 times the screen
height (45cm) in a laboratory with normal indoor lighting.

To limit the experiment time for each session to less than half an hour, the
experiment is split into 5 sessions, each of which includes the subjective
quality evaluation for about 200 AGIs. This results in more than
22$\times$1,080=23,760 quality ratings.

Fig. 5: An example of the quality assessment interface, where the AGI and
corresponding keywords are shown at the same time. The subject can then
evaluate the quality of AGIs and record the quality scores with the scroll bar
on the right.

###  II-C Subjective Data Analysis

After the subjective experiment, all quality ratings from the subjects are
collected. The raw rating judged by the $i$-th subject on the $j$-th image is
denoted by $r_{ij}$. Z-scores are obtained from the raw ratings using the
following formula:

$z_{ij}=\frac{r_{ij}-\mu_{i}}{\sigma_{i}},$ (1)

where $\mu_{i}=\frac{1}{N_{i}}\sum_{j=1}^{N_{i}}r_{ij}$,
$\sigma_{i}=\sqrt{\frac{1}{N_{i}-1}\sum_{j=1}^{N_{i}}(r_{ij}-\mu_{i})^{2}}$,
and $N_{i}$ is the number of images judged by subject $i$. Next, ratings from
unreliable subjects are removed using the subject rejection procedure
recommended by ITU-R BT.500-13 [14]. The mean opinion score (MOS) of image $j$
is computed by averaging the rescaled z-scores:

$MOS_{j}=\frac{1}{M}\sum_{i=1}^{M}z_{ij}^{{}^{\prime}},$ (2)

where $MOS_{j}$ indicates the MOS for the $j$-th AGI, $M$ is the number of
valid subjects, and $z_{ij}^{{}^{\prime}}$ are the rescaled z-scores. The
corresponding MOS distribution in Fig. 6 is consistent with previous works
[15] [16] about subjective diversity.

##  III Experiment

###  III-A Benchmark Models

Due to the absence of pristine reference images in the proposed AGIQA-1k
database, only no-reference (NR) IQA models are selected for comparison. The
selected models can be classified into three groups:

  * •

Handcrafted-based models: This group includes BMPRI [17], CEIQ [18], DSIQA
[19], NIQE[20], and SISBLIM [21]. These models extract handcrafted features
based on prior knowledge about image quality.

  * •

Handcrafted &SVR-based models: This group includes friquee [22], GMLF [23],
HIGRADE [24], NFERM [25], and NFSDM [26]. These models combine handcrafted
features from a Support Vector Regression (SVR) to represent perceptual
quality.

  * •

Deep learning-based models: This group includes ResNet50 [27], StairIQA [28],
and MGQA [29]. These models characterize quality-aware information by training
deep neural networks from labeled data.

Notably, the models mentioned above have exhibited strong performance in
previous IQA tasks for natural scenes.

Fig. 6: Illustration of the MOS probability distribution.

###  III-B Evaluation Criteria

In this study, three primary metrics are utilized to evaluate the consistency
between the predicted scores and Mean Opinion Scores (MOSs): Spearman Rank
Correlation Coefficient (SRoCC), Pearson Linear Correlation Coefficient
(PLCC), Kendall’s Rank Correlation Coefficient (KRoCC). The SRoCC metric
measures the similarity between two sets of rankings, while the PLCC metric
computes the linear correlation between two groups of rankings. The KRoCC
metric, on the other hand, estimates the ordinal relationship between two
measured quantities.

To map the predicted scores to MOSs, a five-parameter logistic function is
applied, which is a standard practice suggested in [30]:

$\hat{X}=\alpha_{1}\left(0.5-\frac{1}{1+e^{\alpha_{2}\left(X-\alpha_{3}\right)}}\right)+\alpha_{4}X+\alpha_{5},$
(3)

where $\\{\alpha_{i}\mid i=1,2,\ldots,5\\}$ represent the parameters to be
fitted, $y$ and $\hat{y}$ stand for predicted and mapped scores, respectively.

###  III-C Experimental Setup

All the benchmark models in III-A are validated on the proposed AGIQA-1k
database. The database is split randomly in an 80/20 ratio for
training/testing while ensuring the image with the same object label falls
into the same set. The partitioning and evaluation process is repeated several
times for a fair comparison while considering the computational complexity,
and the average result is reported as the final performance. For &SVR-based
models, the repeating time is 1,000, implemented by LIBSVM [31] with radial
basis function (RBF) kernel. For deep learning-based models, the repeating
time is 10, using ResNet50 [27] as the network backbone. The Adam optimizer
[32] (with an initial learning rate of 0.00001 and batch size 40) is used for
100-epochs training on an NVIDIA GTX 4090Ti GPU.

Table II: Performance results on the AGIQA-1k database and two different generative model subsets. The best performance results are marked in RED and the second performance results are marked in BLUE. Metric | Database | All | stable-inpainting-v1 | stable-diffusion-v2
---|---|---|---|---
Corr | SRoCC | KRoCC | PLCC | SRoCC | KRoCC | PLCC | SRoCC | KRoCC | PLCC
Hand crafted- based |  BMPRI [17] | 0.0651 | 0.0400 | 0.1646 | 0.3746 | 0.2643 | 0.4094 | -0.0158 | -0.0112 | -0.0111
CEIQ [18] | 0.3069 | 0.2097 | 0.2836 | 0.2348 | 0.1607 | 0.2000 | 0.1314 | 0.0898 | 0.1392
DSIQA [19] | -0.3047 | -0.2148 | -0.0559 | 0.0428 | 0.0241 | 0.4106 | 0.0046 | 0.0041 | 0.0184
NIQE [20] | -0.5490 | -0.3824 | -0.5048 | 0.0414 | 0.0240 | 0.0712 | -0.2275 | -0.1564 | -0.2392
SISBLIM [21] | -0.1309 | -0.0889 | -0.3575 | -0.2410 | -0.1666 | -0.4802 | 0.0541 | 0.0371 | 0.0305
Hand crafted &SVR- based |  friquee [20] | 0.4938 | 0.3469 | 0.4192 | 0.4231 | 0.3024 | 0.3989 | 0.1783 | 0.1244 | 0.2069
GMLF [23] | 0.5575 | 0.4052 | 0.6356 | 0.5062 | 0.3649 | 0.6167 | 0.1501 | 0.1039 | 0.1713
HIGRADE [24] | 0.4056 | 0.2860 | 0.4425 | 0.2493 | 0.1732 | 0.2886 | 0.1358 | 0.0943 | 0.1308
NFERM [25] | 0.4540 | 0.3224 | 0.5396 | 0.3874 | 0.2743 | 0.4901 | 0.1193 | 0.0817 | 0.1474
NFSDM [26] | 0.4314 | 0.3055 | 0.4714 | 0.3840 | 0.2743 | 0.4576 | 0.1002 | 0.0690 | 0.0911
Deep learning- based |  ResNet50[27] | 0.6365 | 0.4777 | 0.7323 | 0.6000 | 0.4485 | 0.7728 | 0.3961 | 0.2785 | 0.4739
StairIQA [28] | 0.5504 | 0.4039 | 0.6088 | 0.4669 | 0.2519 | 0.5050 | 0.3486 | 0.2519 | 0.4186
MGQA [29] | 0.6011 | 0.4456 | 0.6760 | 0.5618 | 0.4250 | 0.7206 | 0.3715 | 0.2584 | 0.3593

###  III-D Performance Discussion

The performance results on the proposed AGIQA-1K database and corresponding
two different generative model subsets are exhibited in Table II, from which
we can make several conclusions. 1) The handcrafted-based methods achieve poor
performance on the whole database and two subsets, which indicates the
extracted handcrafted features are not effective for modeling the quality
representation of AGIs. This is because most employed handcrafted features of
these methods are based on the prior knowledge learned from NSIs, which
apparently do not hold for the AGIs. 2) The deep learning-based methods
achieve relatively more competitive performance results on the whole database
and two subsets. However, they are still far away from satisfactory. 3) Nearly
all the IQA models achieve the best performance on the whole database and
undergo significant performance drops on the stable-diffusion-v2 subsets. We
attempt to give the reasons for such a phenomenon. More keywords are utilized
for the stable-diffusion-v2 model, therefore making the AGIs generated by a
such model more diverse and complicated. This makes it more challenging for
the IQA models to extract quality-aware features from AGIs, which inevitably
leads to performance drops.

We further validate the performance of the IQA models on the AGIQA-1K database
with the anime and realistic styles. The experimental results are listed in
Table III. It seems that the IQA models gain similar performance across
different styles, which suggests that the styles have a limited impact on the
performance of current IQA models.

Table III: Performance results on the AGIQA-1K database with different styles. The best performance results are marked in RED and the second performance results are marked in BLUE. Metric | Database | Anime | Realistic
---|---|---|---
Corr | SRoCC | PLCC | SRoCC | PLCC
Hand crafted- based |  BMPRI [17] | 0.1029 | 0.2452 | 0.0257 | 0.1001
CEIQ [18] | 0.2948 | 0.2746 | 0.3143 | 0.2873
DSIQA [19] | -0.3282 | -0.0741 | -0.3372 | -0.0857
NIQE [20] | -0.5256 | -0.4901 | -0.5853 | -0.5423
SISBLIM [21] | -0.3133 | -0.4331 | -0.0135 | -0.3009
Hand crafted &SVR- based |  friquee[22] | 0.4654 | 0.3752 | 0.5165 | 0.4153
GMLF [23] | 0.5200 | 0.6338 | 0.5946 | 0.6356
HIGRADE [24] | 0.4244 | 0.4895 | 0.4561 | 0.4858
NFERM [25] | 0.4778 | 0.5618 | 0.4834 | 0.5480
NFSDM [26] | 0.4270 | 0.4768 | 0.3728 | 0.4134
Deep learning- based |  ResNet50 [27] | 0.5769 | 0.6769 | 0.6686 | 0.7577
StairIQA [28] | 0.5947 | 0.6385 | 0.5007 | 0.5879
MGQA [29] | 0.6138 | 0.6876 | 0.5734 | 0.6613

##  IV Conclusion

AIGC has become increasingly popular as deep learning techniques keep
improving. However, due to hardware constraints and technical limitations, the
quality of AGIs can vary, necessitating refinement and filtering prior to
practical usage. Therefore, there is a critical need for developing objective
models to assess the quality of AGIs. In this paper, we first discuss
significant evaluation aspects, such as technical issues, AI artifacts,
unnaturalness, discrepancy, and aesthetics for AGI quality assessment. Then,
we carry out the first perceptual AGI quality assessment database, AGIQA-1K,
containing 1,080 AGIs generated from diffusion models. A well-organized
subjective experiment is conducted to collect quality labels for the AGIs.
Subsequently, a benchmark experiment is carried out to evaluate the
performance of current IQA models. The experimental results reveal that the
current IQA models are not well qualified to deal with AGIQA task and there is
still a long way to go.

## References

  * [1] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer,  “High-resolution image synthesis with latent diffusion models,”  in IEEE/CVF CVPR, 2022.
  * [2] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio,  “Generative adversarial networks,”  Communications of the ACM, 2020.
  * [3] Zhenyu Peng, Qiuping Jiang, Feng Shao, Wei Gao, and Weisi Lin,  “Lggd+: Image retargeting quality assessment by measuring local and global geometric distortions,”  IEEE TCSVT, 2022.
  * [4] Luhong Liang, Shiqi Wang, Jianhua Chen, Siwei Ma, Debin Zhao, and Wen Gao,  “No-reference perceptual image quality metric using gradient profiles for jpeg2000,”  Signal Processing: Image Commun, 2010.
  * [5] Chunyi Li, Zicheng Zhang, Wei Sun, Xiongkuo Min, and Guangtao Zhai,  “A full-reference quality assessment metric for cartoon images,”  in IEEE MMSP, 2022.
  * [6] Zicheng Zhang, Wei Sun, Xiongkuo Min, Tao Wang, Wei Lu, and Guangtao Zhai,  “Distinguishing computer-generated images from photographic images: a texture-aware deep learning-based method,”  in IEEE VCIP, 2022.
  * [7] Shiqi Wang, Kede Ma, Hojatollah Yeganeh, Zhou Wang, and Weisi Lin,  “A patch-structure representation method for quality assessment of contrast changed images,”  IEEE SPL, 2015.
  * [8] Ke Gu, Shiqi Wang, Guangtao Zhai, Siwei Ma, Xiaokang Yang, Weisi Lin, Wenjun Zhang, and Wen Gao,  “Blind quality assessment of tone-mapped images via analysis of information, naturalness, and structure,”  IEEE TMM, 2016.
  * [9] Zhenqiang Ying, Haoran Niu, Praful Gupta, Dhruv Mahajan, Deepti Ghadiyaram, and Alan Bovik,  “From patches to pictures (paq-2-piq): Mapping the perceptual space of picture quality,”  in IEEE/CVF CVPR, 2020.
  * [10] Shiqi Wang, Ke Gu, Xinfeng Zhang, Weisi Lin, Siwei Ma, and Wen Gao,  “Reduced-reference quality assessment of screen content images,”  IEEE TCSVT, 2018.
  * [11] Mai Xu, Chen Li, Shanyi Zhang, and Patrick Le Callet,  “State-of-the-art in 360° video/image processing: Perception, assessment and compression,”  IEEE JSTSP, 2020.
  * [12] V. Hosu, H. Lin, T. Sziranyi, and D. Saupe,  “Koniq-10k: An ecologically valid database for deep learning of blind image quality assessment,”  IEEE TIP, 2020.
  * [13] Vlad Hosu, Franz Hahn, Mohsen Jenadeleh, Hanhe Lin, Hui Men, Tamás Szirányi, Shujun Li, and Dietmar Saupe,  “The konstanz natural video database (konvid-1k),”  in IEEE QoMEX, 2017.
  * [14] I. T. Union,  “Methodology for the subjective assessment of the quality of television pictures,”  ITU-R Recommendation BT. 500-11, 2002.
  * [15] Yixuan Gao, Xiongkuo Min, Yucheng Zhu, Jing Li, Xiao-Ping Zhang, and Guangtao Zhai,  “Image quality assessment: From mean opinion score to opinion score distribution,”  in ACM MM, 2022.
  * [16] Hossein Talebi and Peyman Milanfar,  “Nima: Neural image assessment,”  IEEE TIP, 2018.
  * [17] Xiongkuo Min, Guangtao Zhai, Ke Gu, Yutao Liu, and Xiaokang Yang,  “Blind image quality estimation via distortion aggravation,”  IEEE TBC, 2018.
  * [18] Jia Yan, Jie Li, and Xin Fu,  “No-reference quality assessment of contrast-distorted images using contrast enhancement,”  arXiv preprint arXiv:1904.08879, 2019.
  * [19] Niranjan D Narvekar and Lina J Karam,  “A no-reference perceptual image sharpness metric based on a cumulative probability of blur detection,”  in QoMEX, 2009.
  * [20] Anish Mittal, Rajiv Soundararajan, and Alan C Bovik,  “Making a “completely blind” image quality analyzer,”  IEEE SPL, 2012.
  * [21] Ke Gu, Guangtao Zhai, Xiaokang Yang, and Wenjun Zhang,  “Hybrid no-reference quality metric for singly and multiply distorted images,”  IEEE TBC, 2014.
  * [22] Deepti Ghadiyaram and Alan C Bovik,  “Perceptual quality prediction on authentically distorted images using a bag of features approach,”  Journal of Vision, 2017.
  * [23] Wufeng Xue, Xuanqin Mou, Lei Zhang, Alan C Bovik, and Xiangchu Feng,  “Blind image quality assessment using joint statistics of gradient magnitude and laplacian features,”  IEEE TIP, 2014.
  * [24] D Kundu, D Ghadiyaram, AC Bovik, and BL Evans,  “Large-scale crowdsourced study for high dynamic range images,”  IEEE TIP, 2017.
  * [25] Ke Gu, Guangtao Zhai, Xiaokang Yang, and Wenjun Zhang,  “Using free energy principle for blind image quality assessment,”  IEEE TMM, 2014.
  * [26] Ke Gu, Guangtao Zhai, Xiaokang Yang, Wenjun Zhang, and Longfei Liang,  “No-reference image quality assessment metric by combining free energy theory and structural degradation model,”  in IEEE ICME, 2013.
  * [27] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in IEEE/CVF CVPR, 2016.
  * [28] Wei Sun, Huiyu Duan, Xiongkuo Min, Li Chen, and Guangtao Zhai,  “Blind quality assessment for in-the-wild images via hierarchical feature fusion strategy,”  in IEEE BMSB, 2022.
  * [29] Tao Wang, Wei Sun, Xiongkuo Min, Wei Lu, Zicheng Zhang, and Guangtao Zhai,  “A multi-dimensional aesthetic quality assessment model for mobile game images,”  in IEEE VCIP, 2021.
  * [30] Hamid R Sheikh, Muhammad F Sabir, and Alan C Bovik,  “A statistical evaluation of recent full reference image quality assessment algorithms,”  IEEE TIP, 2006.
  * [31] Chih-Chung Chang and Chih-Jen Lin,  “Libsvm: a library for support vector machines,”  ACM TIST, 2011.
  * [32] Diederik P Kingma and Jimmy Ba,  “Adam: A method for stochastic optimization,”  in ICLR, 2015.


#  UAVM: Towards Unifying Audio and Visual Models

Yuan Gong, Alexander H. Liu, Andrew Rouditchenko, and James Glass Code at
github.com/yuangongnd/uavm. The authors are with the Computer Science and
Artificial Intelligence Laboratory, Massachusetts Institute of Technology, USA
(e-mail: {yuangong, glass}@mit.edu).

###### Abstract

Conventional audio-visual models have independent audio and video branches. In
this work, we _unify_ the audio and visual branches by designing a Unified
Audio-Visual Model (UAVM). The UAVM achieves a new state-of-the-art audio-
visual event classification accuracy of 65.8% on VGGSound. More interestingly,
we also find a few intriguing properties of UAVM that the modality-independent
counterparts do not have.

###### Index Terms:

audio-visual learning, unified model

##  I Introduction

Humans perceive and understand their world by combining different sensory
input modalities including sound and vision. To enable AI systems to have a
similar ability, _audio-visual multi-modal learning_ has been extensively
studied [1, 2]. Due to the inherent differences between audio and video,
conventional audio-visual learning methods typically either use handcrafted
modality-specific features or have two _independent_ audio and visual branches
with different _architectures_ , _training schemes_ , and _model weights_ [3,
4, 5, 6, 7].

This modality-specific paradigm began to change after the Transformer [8]
showed its effectiveness for various tasks and modalities including audio [9,
10, 11] and video [12, 13]. Specifically, the Perceiver [14] model shows that
it is possible to handle arbitrary configurations of different modalities
using a _unified model architecture_ , although the training method is still
modality-specific. In addition to unified model architecture, data2vec [15]
further demonstrates that a _unified training scheme_ can be applied to
different modalities to obtain state-of-the-art performance. Despite the
unified model architecture and training scheme, the models of different
modalities are trained independently and have independent weights. To go one
step even further, SkillNet [16], EAO [17], VATT [18], and PolyViT [19] share
part of the _model weights_ among different modalities. Specifically, SkillNet
[16] and EAO [17] are single models that handle multiple modalities, but
different parts of the model weights are specialized for processing different
modalities. VATT [18] uses a modality-agnostic, single-backbone Transformer
and shares weights among different modalities. However, the training scheme of
VATT is modality-asymmetric. PolyViT [19] shares model weights except the
input tokenizer and task head.

Figure 1: An illustration of the unified audio-visual model (UAVM).

The motivations for cross-modality model unification are multifold. First, it
minimizes the use of handcrafted priors and inductive biases for each
individual modality, which is more data-driven and saves manual effort.
Second, a unified model that handles multiple modalities can be more
parameter-efficient than a set of modality-specific models. Third, unified
models are ideal _foundation models_ [20] that can be adapted to a wide range
of downstream tasks of multiple modalities.

However, how unified models differ from modal-specific models remains unclear.
Are unified models just two modal-independent models “glued” together? Does a
unified model indeed encode audio and visual input into a unified
representation space? We answer these questions by building and analyzing a
Unified Audio-Visual Model (UAVM) that unifies 1) model architecture; 2)
weights for high layers including the decision layer; and 3) training with a
unified algorithm for audio and video. On VGGSound [21], UAVM achieves a new
state-of-the-art accuracy for audio-visual event classification.

##  II The Unified Audio-Visual Model

###  II-A UAVM Model Architecture

The unique aspect of the UAVM is the shared Transformer and classification
layer whose weights are shared between different modalities. This feature
enables the UAVM to process both audio _and_ video independently. Before the
shared Transformer, we still have modal-specific feature extractors and
optional modal-specific Transformers for each modality.

As shown in Figure 1, the audio or video is first input to the corresponding
modality-specific feature extractor. Both audio and video feature extractors
are ConvNeXt-Base [22]. For video, we use ImageNet pretrained ConvNeXt as the
feature extractor. We uniformly sample RGB frames from the video at 3 FPS,
input each frame to the ConvNeXt-Base, and get the mean-pooled penultimate
layer representation of 1024-dimensions, i.e., for each 10-second video, the
video features are 30 1024-dimensional vectors
$\\{V^{F}_{1},...,V^{F}_{30}\\}$. For audio, we train a ConvNeXt using in-
domain audio data (AudioSet or VGGSound) with ImageNet initialization [23] and
then use it as the audio feature extractor. Each 10-second waveform is first
converted to a $1000\times 128$ log Mel filterbank (fbank) feature vector
computed with a 25ms Hanning window every 10ms and input to the feature
extractor. The output of the penultimate layer of ConvNeXt is a 30
(time)$\times$4 (frequency)$\times$1024 tensor. We apply a frequency mean
pooling to produce 30 1024-dimension audio features
$\\{A^{F}_{1},...,A^{F}_{30}\\}$. Note that we intend to make the audio and
video features synchronized. Both feature extractors are frozen during
training.

We then input the L2-normalized $\\{A^{F}\\}$ or $\\{V^{F}\\}$ to
corresponding $N$-layer modality-specific Transformers to produce
$\\{A_{1},...,A_{30}\\}$ or $\\{V_{1},...,V_{30}\\}$. After that, either
$\\{A_{1},...,A_{30}\\}$ or $\\{V_{1},...,V_{30}\\}$ is used as an input
$\\{S^{I}_{1},...,S^{I}_{30}\\}$ to the shared, modality-agnostic Transformer
of $N_{s}$ layers. This approach is fundamentally different from concatenating
audio and visual tokens as input to the shared Transformer as in MBT [24] and
Merlot Reserve [25]. We mean-pool the output of the shared Transformer
$\\{S_{1},...,S_{30}\\}$ and input it to a shared linear classification layer.
When only one modality is input, we use the output of the shared linear
classification layer as the single-modality prediction; when both modalities
are input, we run one forward pass for each modality and average the outputs
of the two passes as a fused prediction (Algorithm 1). In other words, UAVM is
robust to missing modalities.

Throughout the experiments, all Transformer layers have 4 attention heads and
the modality-specific Transformer always has an embedding dimension of 1024.
We tune the embedding dimension of the shared Transformer $S_{dim}$ from 16 to
1024 to control its capacity. We fix the total number of Transformer layers to
6 ($N+N_{s}=6$), and tune the number of modal-specific Transformer layers,
$N$, and shared Transformer layers, $N_{s}$, from 0 to 6 to control the model
unification level. When $N=0$ and $N_{s}=6$, the audio and visual models are
maximally unified; when $N=6$ and $N_{s}=0$, the audio and visual models are
completely independent including the classification layer. We set the model
with $N=N_{s}=3$ and $S_{dim}=1024$ as the base UAVM model.

Algorithm 1 UAVM Model Training and Inference

1:Dataset $\mathcal{D}=\\{A,V,Label\\}$, UAVM Model
$\mathcal{M}=\\{\theta_{a},\theta_{v},\theta_{s}\\}$

2:

3:Training ($\mathcal{D},\mathcal{M},\lambda_{\text{MT}}$)

4:

5:while $i<max\;training\;iteration$ do
$\triangleright$ One modality is used to train the model at an iteration

6: if $unif(0,1)<$ modality training weight $\lambda_{\text{MT}}$ then

7: sample a batch of audio $\\{A_{i},Label_{i}\\}$

8: $Pred_{a}=\mathcal{M}_{\\{\theta_{a},\theta_{s}\\}}(A_{i})$

9: $\mathcal{L}=Loss(Label,Pred_{a})$

10: backpropagate and update $\\{\theta_{a},\theta_{s}\\}$

11: else

12: sample a batch of video $\\{V_{i},Label_{i}\\}$

13: $Pred_{v}=\mathcal{M}_{\\{\theta_{v},\theta_{s}\\}}(V_{i})$

14: $\mathcal{L}=Loss(Label,Pred_{v})$

15: backpropagate and update $\\{\theta_{v},\theta_{s}\\}$ return
$\mathcal{M}$

16:

17:Inference ($\\{A,V\\},\mathcal{M}$)

18:

19:if $A\;!=None$ and $V\;!=None$ then

20:
$Pred=(\mathcal{M}_{\\{\theta_{a},\theta_{s}\\}}(A)+\mathcal{M}_{\\{\theta_{v},\theta_{s}\\}}(V))/2$

21:else if one modality is missing then

22: $Pred=\mathcal{M}_{\\{\theta_{v},\theta_{s}\\}}(V)$ when $A==None$

23: $Pred=\mathcal{M}_{\\{\theta_{a},\theta_{s}\\}}(A)$ when $V==None$ return
$Pred$

###  II-B UAVM Model Training

We train UAVM with Algorithm 1. In each training iteration, only one modality
is used. This is implemented by using a modality training weight
$\lambda_{\text{MT}}$ and uniform sampling, i.e., audio and video have
$\lambda_{\text{MT}}$ and $1-\lambda_{\text{MT}}$ probability to be used,
respectively. In other words, UAVM does not explicitly leverage the audio-
visual correspondence information and can be trained with unpaired audio and
video data (though in this paper, we focus on parallel datasets for
simplicity). By default, we use modality training weight
$\lambda_{\text{MT}}=0.5$. As in prior work [9, 23, 24], we train the model
with mixup [26], balanced sampling, label smoothing, and random time shifts.
These training techniques are applied to both modalities with exactly the same
hyperparameters, i.e., we use a unified training pipeline for the two
modalities.

##  III Experiment and Discussion

###  III-A Experiment Settings

####  III-A1 Dataset

We use two widely-used datasets for audio and video event classification:
AudioSet [27] and VGGSound [21]. AudioSet is a collection of 2M 10-second
YouTube video clips labeled with the sounds that the clip contains from a set
of 527 labels. We downloaded 1.8M training and 17K evaluation audio and video
samples for our experiment. VGGSound [21] is a collection of 200K 10-second
YouTube video clips annotated with 309 classes. We download 184K training and
15K test samples for our experiments. One advantage of VGGSound is that the
sound source is always visually present in the video clip. Also, its moderate
size allows us to conduct extensive experiments with our computational
resources. Therefore, while we compare UAVM performance with existing methods
on both AudioSet and VGGSound, we conduct all ablation studies and analyses on
VGGSound only.

####  III-A2 Training Details

For all experiments, we train the model with a batch size of 144 and the Adam
optimizer [28]. For the main experiments, we use an initial learning rate of
1e-5 and 5e-5 for AudioSet and VGGSound, respectively, and decrease the
learning rate with a factor of 0.5 every epoch. We train the model for 10
epochs and report the last epoch result. For ablation studies, we tune hyper-
parameters to ensure a fair comparison. We repeat all experiments 3 times with
different random seeds and report the mean and standard deviation. The
standard deviation is shown as shaded areas in Figure 2-5.

###  III-B Model Performance Comparison

We compare the performance of three base models:

1\. UAVM. The base UAVM described in Section II-A with 3 modal-specific
Transformer layers and 3 shared Transformer layers ($N=N_{s}=3$) and
$S_{dim}=1024$.

2\. Modal-Independent Model. The base UAVM without shared Transformer layers
(i.e., $N=6,N_{s}=0$), so the audio and visual models are completely
independent including the classification layer.

3\. Cross-Modal Attention Model. The base UAVM model with 3 modal-specific
Transformer layers and 3 shared Transformer layers ($N=N_{s}=3$) but instead
of inputting one modality at a time to the shared Transformer, this model
_concatenates_ the outputs of modal-specific Transformers $\\{A\\}$ and
$\\{V\\}$ together and inputs $\\{A,V\\}$ to the shared Transformer. This
model only works when both modalities are input.

We show the results in Table I. Key conclusions are as follows: First,
compared with the modal-independent model, UAVM achieves almost the same
fusion performance when both modalities are input, and even slightly better
results when a single modality is input  with 76% of the parameters,
demonstrating the feasibility of using a single network for two different
modalities. Second, comparing UAVM with the “MBT-style” cross-modal attention
model, we find a performance discrepancy between the datasets, i.e., UAVM is
noticeably better on VGGSound while the cross-modal attention model is
noticeably better on AudioSet. This is potentially because the sound source is
always visually present in VGGSound videos but not in AudioSet videos. The
UAVM strategy of giving equal weight to both modalities performs better than
cross-modal attention models that could be dominated by one modality on tasks
like VGGSound (video is always informative), but worse on AudioSet (video is
not always informative). Finally, Transformer models with pretrained frozen
features are a strong baseline with low computational cost. As a consequence,
UAVM achieves a new SOTA performance on VGGSound, and outperforms all previous
methods except MBT [24] on AudioSet.  Note this is a fair comparison as both
UAVM and MBT use ImageNet pretraining.  Compared with MBT raw patches, the
frozen feature input sequence length is much shorter (30 vs 1,500+), making it
more computationally efficient since the Transformer has quadratic complexity.

TABLE I: Model Performance Comparison on VGGSound and AudioSet. (∗ single-
modal model trained independently.
† modality-missing results of a multi-modal model. ) VGGSound (Top-1 Accuracy, %) | Audio | Video | Fusion
---|---|---|---
Chen _et al._ [21] | 48.8 | - | -
AudioSlowFast [29] | 50.1 | - | -
MBT [24] |  52.3∗ |  51.2∗ | 64.1
Our Cross-Modal Attention Model | - | - |  62.9$\pm$0.2
Our Modal-Independent Model |  56.5$\pm$0.1∗ |  49.7$\pm$0.2∗ |  65.7$\pm$0.2
Our UAVM Model | 56.5$\pm$0.1† | 49.9$\pm$0.2† | 65.8$\pm$0.1
Full AudioSet (mAP) | Audio | Video | Fusion
GBlend [30] |  32.4∗ |  18.8∗ | 41.8
Attn Audio-Visual [31] |  38.4∗ |  25.7∗ | 46.2
Perceiver [14] |  38.4∗ |  25.8∗ | 44.2
MBT [24] (w/ 500k training samples) |  44.3∗ |  32.3∗ | 52.1
Our Cross-Modal Attention Model | - | - | 50.4$\pm$0.1
Our Modal-Independent Model |  45.5$\pm$0.0∗ |  26.8$\pm$0.1∗ |  48.1$\pm$0.1
Our UAVM Model | 45.6$\pm$0.0† | 27.4$\pm$0.1† |  48.0$\pm$0.0

(a)  (b)  (c)

Figure 2: The audio-based, video-based, and fused accuracy on VGGSound with
various shared embedding dimension $S_{dim}$ (upper), number of shared layers
$N_{s}$ (middle), and modality training weight $\lambda_{\text{MT}}$ (lower).

We then conduct a series of ablation studies. We set models with $N=N_{s}=3$
and $S_{dim}=1024$ as the base UAVM and change one factor at a time to observe
the performance change. First, we constrain the shared Transformer embedding
dimension $S_{dim}$ to smaller values to lower its capacity for UAVM and
compare it with other models. For a fair comparison, the modal-independent and
cross-modal attention models also have three 1024-dimensional Transformer
layers and three $S_{dim}$-dimensional Transformer layers. With a limited
capacity, the shared Transformer is forced to share neurons for two
modalities. As shown in Figure 2 (upper), we find model performance generally
improves with larger $S_{dim}$, but even when $S_{dim}$ is very small, UAVM
still performs similarly or even better than the modal-independent model.
Second, we tune the number of shared layers $N_{s}$ to change the level of
model unification. Note that we fix the total number of Transformer layers to
be 6, i.e., $N+N_{s}=6$. As shown in Figure 2 (middle), when $S_{dim}$ is
small, the single-modality accuracy improves with more shared layers while
when $S_{dim}$ is large, the number of shared layers does not impact the
performance much.  This is true even when all 6 layers are shared (65.6%
accuracy). Finally, as shown in Figure 2 (lower), even though audio and video
are not equally informative for the event classification task (reflected in
different accuracies), using a 0.5 $\lambda_{\text{MT}}$ leads to optimal
fusion results.

###  III-C Unified Audio-Visual Representation Space

One core question about the unified model is if it indeed encodes two
modalities in a unified latent space, or just processes each modality with a
part of its parameters. We explore this by checking if a logistic regression
model can successfully classify the input modality based on the mean-pooled
penultimate layer representation of the UAVM model. Specifically, we use half
of the VGGSound test set to train a logistic regression model and use the
other half for testing. As shown in Figure 3.A, when the shared Transformer
has a small $S_{dim}$ and limited capacity, a logistic regression model is
unable to predict the input modality based on the penultimate layer
representation, in other words, the two modalities are encoded in a unified
space. However, the modality classification accuracy gradually increases with
$S_{dim}$, indicating that the model, though with shared weights, tends to
encode two modalities in separate spaces when it has redundant capacity.
Nevertheless, we do not see one or a small number of dimensions of the
representation specifically encoding the input modality information because
the classifier does not have a dominant coefficient (Figure 3.C). Further, as
shown in Figure 3.B, the modality of the input and modal-specific layer (layer
1-3) representations can be perfectly classified but the modality
classification accuracy drops suddenly with the first shared layer (layer 4)
representation, indicating that it is the shared Transformer that maps the two
very different inputs to a unified space. The above findings can also be
confirmed with the t-SNE plot of the representations of each layer with audio
and video input shown in Figure 3.D.

(a)  (b)

Figure 3: (A) Modality classification accuracy based on the last layer
representation of UAVM with various $S_{dim}$. (B) Modality classification
accuracy based on intermediate representations of each layer of UAVM models.
(C) The sorted modality classifier coefficients. (D) t-SNE plot of the
intermediate representations of each layer with audio input (red) and video
input (blue).  Figure 4: (Left) A-V retrieval performance based on the last
shared Transformer layer representation of UAVM models ($N_{s}=3$) with
various shared embedding dimensions $S_{dim}$. (Right) A-V retrieval
performance based on the representation of various layers of models with
$N_{s}=0$ (fully independent model), $N_{s}=3$, and $N_{s}=6$ (fully unified
model) and $S_{dim}=128$.

###  III-D Audio-Visual Representation Correspondence

One further question is how UAVM _aligns_ paired audio and visual input in the
latent representation space. Interestingly, we find a clear discrepancy
between UAVM and modal-independent models. As a probing task, we calculate the
A-V retrieval recall based on cosine similarity on a 1.5k subset of the
VGGSound evaluation set (309 classes, 5 samples per class). Note that with
reasonable single-modal classification accuracies, the audio and video of the
same class can be naturally retrieved from each other. However, as shown in
Figure 4 (right), for the modal-independent model ($N_{s}=0$), the A-V
retrieval recall is close to 0 for representations of all layers except the
final linear classification head (i.e., the prediction logits) while for
UAVMs, the A-V retrieval recall is much higher for representations for front
layers (the more shared layers, the earlier a high A-V retrieval recall is
achieved), indicating that the shared Transformer layers  tend to propagate
the supervision signal to front layers.  Nonetheless, the unified models learn
A-V correspondences beyond just intra-class correspondence as the A-V
retrieval recall of UAVMs ($S_{dim}=128$) is about 20% higher than the modal-
independent model, while their classification accuracies are similar. This is
particularly interesting because UAVMs do not see simultaneous audio and video
pairs, nor has any explicit audio-visual correspondence loss been applied
during training. Further, as shown in Figure 4 (left), when $S_{dim}>128$, the
A-V retrieval recall starts to decrease with the increase of shared embedding
dimension $S_{dim}$, while the classification accuracy consistently improves
with larger $S_{dim}$ (Figure 2, upper), indicating UAVM with redundant
capacity is worse in audio-visual alignment and closer to modal-independent
models, which is consistent with the finding in Section III-C.

Figure 5: (Left) Difference between the last layer attention maps with
(paired) audio and video input for UAVM with various $N_{s}$. (Right) Temporal
attention heatmap of 4 attention heads for a sample input (label: firing
cannon).

###  III-E Attention Maps

Even when the audio and video features are temporally synchronized, the
informative part for event classification could be different. We show the
temporal attention heatmap of 4 attention heads of the last Transformer layer
in Figure 5 (right). The UAVM, even when the shared Transformer has very
limited capacity ($S_{dim}=16$), can pay attention to different parts of the
audio and video, demonstrating its ability to process two modalities
simultaneously. A cross-modal attention model, however, could be dominated by
one modality while the other modality still contains useful information.
Interestingly, by quantitatively calculating the mean absolute error (MAE)
between attention maps of paired audio and video inputs, we find that the more
modal-specific layers, the smaller the difference between the audio and video
attention maps (Figure 5, left), indicating the modal-specific Transformers
layers are forced to temporally align the informative part of the two
modalities, so that succeeding shared Transformer layers can have a relatively
more modal-agnostic attention map.

##  IV Conclusion

In this work, we unify the audio and visual branches of a multi-modality model
and build UAVM. Performance-wise, UAVM is similar to a modal-independent
model, and outperforms cross-modal attention models on VGGSound. We find the
capacity of the weight-sharing network greatly impacts the behavior of UAVM.
When its capacity is constrained, UAVM shows some intriguing unique
properties, e.g., it maps audio and video in a unified latent space, and
better aligns paired audio and video in the latent space. Such unique
properties gradually disappear with increasing model capacity, and UAVM
behaves closer to modal-independent models.

##  V Acknowledgments and Disclosure of Funding

This research is supported by the MIT-IBM Watson AI Lab.

## References

  * [1] D. Ramachandram and G. W. Taylor, “Deep multimodal learning: A survey on recent advances and trends,” _IEEE signal processing magazine_ , pp. 96–108, 2017.
  * [2] H. Zhu, M.-D. Luo, R. Wang, A.-H. Zheng, and R. He, “Deep audio-visual learning: A survey,” _International Journal of Automation and Computing_ , pp. 351–376, 2021.
  * [3] Y. Aytar, C. Vondrick, and A. Torralba, “Soundnet: Learning sound representations from unlabeled video,” _Advances in neural information processing systems_ , 2016.
  * [4] R. Arandjelovic and A. Zisserman, “Look, listen and learn,” in _IEEE International Conference on Computer Vision_ , 2017, pp. 609–617.
  * [5] A. Rouditchenko, A. Boggust, D. Harwath, B. Chen, D. Joshi, S. Thomas, K. Audhkhasi, H. Kuehne, R. Panda, R. Feris _et al._ , “Avlnet: Learning audio-visual language representations from instructional videos,” in _Interspeech_ , 2021.
  * [6] M. Monfort, S. Jin, A. Liu, D. Harwath, R. Feris, J. Glass, and A. Oliva, “Spoken moments: Learning joint audio-visual representations from video descriptions,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 14 871–14 881.
  * [7] O. Chang, O. Braga, H. Liao, D. Serdyuk, and O. Siohan, “On robustness to missing video for audiovisual speech recognition,” _Transactions on Machine Learning Research_ , 2022.
  * [8] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , 2017.
  * [9] Y. Gong, Y.-A. Chung, and J. Glass, “AST: Audio Spectrogram Transformer,” in _Interspeech_ , 2021, pp. 571–575.
  * [10] Y. Gong, C.-I. Lai, Y.-A. Chung, and J. Glass, “SSAST: Self-Supervised Audio Spectrogram Transformer,” in _AAAI Conference on Artificial Intelligence_ , 2022, pp. 10 699–10 709.
  * [11] J. Ao, R. Wang, L. Zhou, C. Wang, S. Ren, Y. Wu, S. Liu, T. Ko, Q. Li, Y. Zhang _et al._ , “Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing,” in _ACL_ , 2022.
  * [12] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” in _International Conference on Learning Representations_ , 2020\.
  * [13] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, “Training data-efficient image transformers & distillation through attention,” in _International Conference on Machine Learning_ , 2021, pp. 10 347–10 357.
  * [14] A. Jaegle, F. Gimeno, A. Brock, O. Vinyals, A. Zisserman, and J. Carreira, “Perceiver: General perception with iterative attention,” in _International conference on machine learning_ , 2021, pp. 4651–4664.
  * [15] A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli, “data2vec: A general framework for self-supervised learning in speech, vision and language,” in _International Conference on Machine Learning_ , 2022, pp. 1298–1312.
  * [16] Y. Dai, D. Tang, L. Liu, M. Tan, C. Zhou, J. Wang, Z. Feng, F. Zhang, X. Hu, and S. Shi, “One model, multiple modalities: A sparsely activated approach for text, sound, image, video and code,” _arXiv preprint arXiv:2205.06126_ , 2022.
  * [17] N. Shvetsova, B. Chen, A. Rouditchenko, S. Thomas, B. Kingsbury, R. S. Feris, D. Harwath, J. Glass, and H. Kuehne, “Everything at once-multi-modal fusion transformer for video retrieval,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 20 020–20 029.
  * [18] H. Akbari, L. Yuan, R. Qian, W.-H. Chuang, S.-F. Chang, Y. Cui, and B. Gong, “Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text,” _Advances in Neural Information Processing Systems_ , pp. 24 206–24 221, 2021.
  * [19] V. Likhosherstov, A. Arnab, K. Choromanski, M. Lucic, Y. Tay, A. Weller, and M. Dehghani, “Polyvit: Co-training vision transformers on images, videos and audio,” _arXiv preprint arXiv:2111.12993_ , 2021.
  * [20] R. Bommasani, D. A. Hudson, E. Adeli _et al._ , “On the opportunities and risks of foundation models,” _ArXiv_ , 2021. [Online]. Available: https://crfm.stanford.edu/assets/report.pdf
  * [21] H. Chen, W. Xie, A. Vedaldi, and A. Zisserman, “Vggsound: A large-scale audio-visual dataset,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020, pp. 721–725.
  * [22] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A convnet for the 2020s,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 11 976–11 986.
  * [23] Y. Gong, Y.-A. Chung, and J. Glass, “Psla: Improving audio tagging with pretraining, sampling, labeling, and aggregation,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 2021.
  * [24] A. Nagrani, S. Yang, A. Arnab, A. Jansen, C. Schmid, and C. Sun, “Attention bottlenecks for multimodal fusion,” _Advances in Neural Information Processing Systems_ , pp. 14 200–14 213, 2021.
  * [25] R. Zellers, J. Lu, X. Lu, Y. Yu, Y. Zhao, M. Salehi, A. Kusupati, J. Hessel, A. Farhadi, and Y. Choi, “Merlot reserve: Neural script knowledge through vision and language and sound,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 16 375–16 387.
  * [26] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” in _International Conference on Learning Representations_ , 2018.
  * [27] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2017, pp. 776–780.
  * [28] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _International Conference on Learning Representations_ , 2015.
  * [29] E. Kazakos, A. Nagrani, A. Zisserman, and D. Damen, “Slow-fast auditory streams for audio recognition,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021, pp. 855–859.
  * [30] W. Wang, D. Tran, and M. Feiszli, “What makes training multi-modal classification networks hard?” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 12 695–12 705.
  * [31] H. M. Fayek and A. Kumar, “Large scale audiovisual learning of sounds with weakly labeled data,” in _International Joint Conferences on Artificial Intelligence_ , 2021, pp. 558–565.


# ToxBuster: In-game Chat Toxicity Buster with BERT

Zachary Yang
Ubisoft La Forge
McGill University, Mila
<EMAIL_ADDRESS>
Yasmine Maricar
Ubisoft MURL
<EMAIL_ADDRESS>
Mohammadreza Davari
Ubisoft La Forge
Concordia University, Mila
<EMAIL_ADDRESS>
Nicolas Grenon-Godbout
Ubisoft MURL
<EMAIL_ADDRESS>
Reihaneh Rabbany
McGill University, Mila
CIFAR AI chair
<EMAIL_ADDRESS>

###### Abstract

Detecting toxicity in online spaces is challenging and an ever more pressing
problem given the increase in social media and gaming consumption. We
introduce ToxBuster, a simple and scalable model trained on a relatively large
dataset of 194k lines of game chat from Rainbow Six Siege and For Honor,
carefully annotated for different kinds of toxicity. Compared to the existing
state-of-the-art, ToxBuster achieves 82.95% (+7) in precision and 83.56% (+57)
in recall. This improvement is obtained by leveraging past chat history and
metadata. We also study the implication towards real-time and post-game
moderation as well as the model transferability from one game to another.

##  1 Introduction

Toxic speech plagues online spaces, from social media platforms (e.g.,
Facebook (Ciftci et al., 2017), Twitter (Watanabe et al., 2018), Reddit (Mohan
et al., 2017), YouTube (Döring and Mohseni, 2020)), in-game chats (Silva et
al., 2020) to the comment section on news websites (Zannettou et al., 2020).
Marginalized groups also continue to be targeted more heavily online, with the
2021 survey from the Anti-Defamation League (ADL) citing 64% of LGTBQ+
respondents, 36% of Jewish respondents, and 17% of Asian-Americans having been
subject to identify-based / race-based harassment (ADL, 2021). Exposure to
toxic language not only alienate users, but can also lead to a wide range of
psychological harms and even incite real-world violence. To ensure a healthy
online community, companies have attempted various methods to curb the spread
of toxic speech, ranging from censoring words, (shadow) banning users or
blocking them from communicating Maher (2016); Lewington (2021). The sheer
volume of user generated data and the rapid evolution of language have made it
nearly impossible to implement a consistent level of moderation.

Building on recent developments in large language models, we utilize them to
design accurate and transferable models for effective content moderation. In
particular, contextual language embeddings (such as BERT (Devlin et al.,
2018)) are at the core of many state-of-the-art toxic speech detection models
(Lees et al., 2022; Hanu and Unitary team, 2020; Pavlopoulos et al., 2020a).
However, most works ignore context and those that do not report only slight
improvements.

Here, we propose the first in-game chat toxicity detection model which can
effectively incorporate broader context for a significant boost in
performance. More specifically, we trained ToxBuster on our annotated dataset.
Our dataset captures a diverse perspective (e.g. labels around the use of
targeted toxic words) by including annotators that self-identify from
marginalized groups. By leveraging and adapting innovations from
conversational AI, our model properly encodes chat history and metadata. Its
architecture can also be easily adapted for social media platforms. From our
experiments, we show that language used in social media and in-game chat are
different enough to warrant their own dataset. A high precision and low recall
toxicity identifier is needed to help prioritize and maximize the impact of
the often limited resources of manual content moderation. We observe our model
at 90.0% precision level can identify 82% of chat reported players. We also
show that the model is transferable between games of the same nature.

In summary, the contributions of this work are threefold:

  1. 1.

ToxBuster which is trained on a diverse game-chat toxicity dataset, leveraging
previous chat history and metadata to boost model performance and an ablation
study for the importance of each feature.

  2. 2.

Study on the implication towards moderation

  3. 3.

Study on model transferablity between games.

##  2 Background & Related Works

Toxicity detection has garnered increasing attention in recent years. Van Aken
et al. (2018) attributes the challenges around detection to not only the text
itself, e.g., out of vocabulary words, but also the lack of consensus on the
exact definition of toxicity, resulting in a multitude of varying defined
tasks and solutions. Toxicity definitions can range from simply toxicity
(Georgakopoulos et al., 2018) to further differentiating to include categories
such as hate speech (Gambäck and Sikdar, 2017), abusive language (Park and
Fung, 2017), cyberbullying (Zhong et al., 2016) and other offensive language.
In our setup, we adapt categories as defined by FairPlayAlliance and ADL
(2020) for disruptive behavior for online games.

Detecting toxicity is a supervised classification task that can be tackled
with traditional NLP models paired with manual feature engineering (Watanabe
et al., 2018), deep neural networks (Gambäck and Sikdar, 2017; Zhong et al.,
2016; Gao and Huang, 2017; Fehn Unsvåg and Gambäck, 2018), and pretrained
language models (Almerekhi et al., 2022; Jhaveri et al., 2022; Pavlopoulos et
al., 2020a; Lees et al., 2022). Attempts at including additional context to
language models have not led to significant performance gain (<1% in model
performance). Gao and Huang (2017) and Mubarak et al. (2017) tried including
news article titles and username as additional context for news articles
comments. Fehn Unsvåg and Gambäck (2018) tried including Twitter user metadata
for tweets. More recently, Pavlopoulos et al. (2020a) used the parent comment
and discussion title of Wikipedia comments. From Jigsaw’s Toxic Comment
Classification, they created two labelled datasets where context was and
wasn’t provided to the annotators. The toxicity ratio between these two
datasets is 4.4% to 6.4% at the aggregate level, suggesting that more context
may be required to achieve a consistent improvement in performance. Therefore,
in our setup, we do not limit the context to only the previous chat line, but
the full chat history.

Lastly, toxicity detection on social media platforms and in-game chat can be
reframed as a problem around conversational AI. Hence, we also look at
innovations from (multi-turn) conversational models. We adapt both methods
Speaker Segmentation and a naive dialogue augmentation that Lu et al. (2020)
suggests for improving BERT for multi-turn conversations.

##  3 Methodology

This section describes our dataset, ToxBuster, and the baseline models.

###  3.1 Dataset

The dataset consists of a total of three separate sessions (S) of data
annotations, consisting of chat logs extracted from regions that communicate
prominently in English. The first two sessions are from Rainbow Six Siege
(R6S) which is a multi-plyaer first person shooter. The last session is from
For Honor (FH), a multi-player melee action game. We oversample matches with
high amounts of chat lines and / or with at least one player getting reported
by another player. Further details of each phase can be found in Table 1 and
Appendix A.

| S1 | S2 | S3
---|---|---|---
Game | R6S | R6S | FH
Annotators | 9 | 30 | 20
# of Lines | 60,116 | 35,496 | 99,371
Timeframe | 2021-07 | 2021-07 | 2022-05
Table 1: Basic statistics for our dataset.

We adapt categories gathered from the “disruptive behavior in online game” as
highlighted by FairPlayAlliance and ADL (2020) as the following: Hate and
Harassment, Threats, Minor Endangerment, Extermism, Scams and Ads, Insults and
Flaming, Spam and Other offensive text. Exact definitions of each category can
be found in Appendix B. After the annotations, we aggregated the labels by 1)
using only the minimum intersecting span of words from all three annotations
and 2) using the most popular label, using the most severe label if none. We
report the final number of chat lines per toxic category and calculate the
Fleiss $\kappa$ for each session in Table 2.

Category Name | S1 | S2 | S3
---|---|---|---
Hate and Harassment | 3,852 | 1,630 | 4,453
Threats | 436 | 182 | 421
Minor Endangerment | 386 | 239 | 109
Extremism | 209 | 183 | 173
Scams and Ads | 179 | 277 | 53
Insults and Flaming | 4,580 | 4,244 | 11,329
Spam | 6,707 | 4,420 | 2,210
Other Offensive | 1,859 | 1,258 | 2,077
Non-toxic | 41,898 | 23,039 | 78,292
Fleiss $\kappa$ | 0.50 | 0.56 | 0.47

Table 2: Number of labelled chat lines per category per session.

###  3.2 ToxBuster

ToxBuster is a BERT model trained on the token task classification task. All
experiments with ToxBuster used a 60-20-20 train-val-test split with 5
different random seeds. We report mean and standard deviations of each metric.
Our model gained performance through two methods: Chat History and Chat
Speaker Segmentation.

####  3.2.1 Chat History

The main goal was to include previous chat lines as context. Pavlopoulos et
al. (2020b) suggested that using the previous comment as context does not
significantly improve the model’s performance, but including further history
may, as it was not tested. With our particular data being lots of short text
and conversational in nature, we decided to include as much previous chat
lines as possible. We further drew inspiration from the question-answering
task. In particular, we took advantage of the sentence-pair structure with
sentence A (question) being concatenated previous chat lines and sentence B
(answer) as the current chat line. Our model does differ in the truncation,
where we created a custom truncator that prioritizes truncating the sentence A
on the left, and truncating on the right of sentence B if removing sentence A
was not enough. Similar to the question-answering task, the toxicity labels
for the previous chat lines are -100, i.e. to not be counted in the loss
function.

Lastly, drawing inspiration from Lu et al. (2020) who proposed dialogue
augmentation, we implemented different chat mode filters: personal, team,
global and moderator. Personal mode filters past chat to include on those
written by the current chat line’s player. Team mode further includes past
chat from players of same team the current player is on. Global mode further
includes past chat that is broadcasted to all players. Moderator includes the
rest of the chat lines (i.e. chat lines from the enemy teams that isn’t
broadcasted to all players). Table 3 is a fabricated sample chat at the
beginning of a match.

1 |  (Team) Player 0: Hf
---|---
2 |  (All) Player 1: Hf
3 |  (Team) Player 6: Which site?
4 |  (Team) Player 7: A
5 |  (All) Player 7: Glhf
Table 3: Chat history changes based on the different chat modes. Players 0-4
are on one team, players 5-9 are on another. The chat history for line 5 for
personal, team, global and moderator is respectively line 4, lines 3-4, lines
2-4, and lines 1-4.

####  3.2.2 Chat Speaker Segmentation

The main goal was to include conversational and additional game information to
the model. Lu et al. (2020) also proposed speaker segmentation. We adapted
speaker segmentation to include three metadata on the current chat line:
playerID, chat type, and teamID. PlayerID is the integer representing the
player writing the chat line. This integer represents the order of each unique
player from the chat history of the current chat line where the writer of the
current chat line is always player 0. Chat type can be either “all” or “team”
chat, where “all” chat is broadcasted to all players. Note, depending on the
game, “all” chat is not enabled by default, but rather must be enabled
manually by the player. “Team” chat can only be read by the team the current
player is on. TeamID is the integer representing the team the player writing
the chat line is on. Similar to playerID, the writer of the current chat line
is always team 0. Figure 1 illustrates the change in the input embedding
layer.

Figure 1:  ToxBuster with Chat Speaker Segmentation. The input embeddings are
the sum of the corresponding token, position, teamID, chat type and playerID.
Chat history consists of as many lines as possible.

###  3.3 Baselines

Here, we detail 3 major models that we used as baselines to compare against
ToxBuster.

####  3.3.1 Cleanspeak

Cleanspeak is a paid tool that has “premier profanity filter &
moderation”111https://cleanspeak.com/docs/3.x/tech/apis/ based on user-defined
keywords and regexes. Toxic chat lines are those whose API response contains
matched text towards any toxic category. Currently, the toxic categories are
“bigotry_racism”, “harm_abuse”, “threats”, “grooming”, “terrorism”, “pii”
(personal identifiable information), “spam”, “bullying”, “vulgarity”,
“sexual”, “alcohol_drug”, “asciiart” which can be easily mapped to our
categories.

####  3.3.2 Perspective API

Perspective API222https://perspectiveapi.com/ is the tool that the Jigsaw and
Google’s Counter Abuse Technology team is building for better discussions
online Lees et al. (2022). We used version v1alpha1 of the API. As noted by
the research team, we classified a chat line as toxic only if the returned
toxic score is >= 0.7. We also note that the API returns an error code for
unsupported language. This impacted around 13% of the chat lines for each
dataset. We removed these lines during the calculation of the metrics.

####  3.3.3 Detoxify

Detoxify Hanu and Unitary team (2020) is a repository with three BERT-based
models respectively called original, unbiased, multilingual that are trained
on their respective Jigsaw toxicity datasets. Further details of how the
models were trained can be found in their repository
333https://github.com/unitaryai/detoxify. We classify a chat line as toxic
only if the returned toxic score for any category is >= 0.5.

##  4 Results and Discussions

In the following sections, we discuss ToxBuster’s performance compared to
baselines and in terms of the individual toxic categories, implication towards
actual moderation and ablation study to understand the impact of each
component of the full model.

###  4.1 Baselines Comparison

We compare ToxBuster with baselines on 5 different test splits of our S1&S2
dataset in Table 4. Since the toxic categories are different across the models
and dataset, we compared them solely on the binary task of classifying whether
each chat line is toxic or non-toxic. Our model performs the best in terms of
precision, recall and F1 score at 82.95, 83.56 and 83.25 respectively. For the
baselines, Perspective API performs the best in terms of precision but all
models have a poor recall rate, resulting in a low F1 score. We note that all
of these models cannot make use of previous chat history as context. This also
shows that the language used in comments from online spaces is different from
those used in in-game chats.

| Precision | Recall | F1 Score
---|---|---|---
Cleanspeak | 66.62 ± 0.83 | 29.10 ± 1.58 | 40.48 ± 1.58
Perspective API | 75.11 ± 1.69 | 24.38 ± 1.02 | 36.81 ± 1.28
Detoxifyoriginal | 62.72 ± 0.93 | 35.82 ± 1.77 | 45.58 ± 1.61
Detoxifyunbiased | 63.47 ± 1.21 | 29.58 ± 1.51 | 40.33 ± 1.54
Detoxifymultilingual | 62.33 ± 1.28 | 33.15 ± 1.98 | 43.26 ± 1.92
ToxBusterbase | 77.21 ± 1.30 | 77.91 ± 1.04 | 77.36 ± 1.31
ToxBusterfull | 82.95 ± 0.31 | 83.56 ± 0.27 | 83.25 ± 0.30

Table 4: Comparing ToxBusterto baseline models on our S1&S2 dataset on
predicting whether a chat line is toxic or non-toxic. ToxBusterbase is trained
without chat history and chat speaker segmentation.

###  4.2 ToxBuster’s Performance

We also analyze our model’s performance for each toxic category with results
shown in Table 5. The model can easily differentiate amongst non-toxic, toxic
words and spam. We attribute the lower F1 score in threats and minor
endangerment to their minority. While extremism and scams and ads have even
fewer samples, the language for both these two categories are usually very
unique. We notice that the model often confuses amongst hate and harassment,
threats, other offensive as the words are often very similar and additional
context from chat history, in game knowledge and social constructs are needed.
Annotators also reported it was often hard to choose between these categories
as well.

Category Name | Precision | Recall | F1 Score
---|---|---|---
Hate and Harassment | 63.78 ± 2.32 | 56.40 ± 3.81 | 59.76 ± 2.01
Threats | 31.53 ± 3.73 | 22.85 ± 4.55 | 26.45 ± 4.33
Minor Endangerment | 38.28 ± 7.12 | 29.21 ± 3.69 | 32.72 ± 3.43
Extremism | 54.58 ± 8.03 | 40.86 ± 8.88 | 45.93 ± 5.02
Scams and Ads | 56.89 ± 4.95 | 45.62 ± 9.54 | 56.12 ± 6.93
Insults and Flaming | 58.97 ± 3.53 | 53.72 ± 2.33 | 50.18 ± 6.93
Spam | 84.15 ± 3.75 | 78.42 ± 3.79 | 81.11 ± 2.61
Other Offensive | 47.52 ± 4.17 | 44.20 ± 3.03 | 45.76 ± 3.30
Non-toxic | 88.32 ± 0.77 | 91.85 ± 0.91 | 90.05 ± 0.17

Table 5: ToxBuster precision, recall and F1 score per category.

###  4.3 Implication towards Actual Moderation

For this model to run as a production system, it would need to be very
reliable. Compared to Cleanspeak, an existing production ready system, our
model significantly performs better, having a 83% recall rate compared to 29%.
For further refinement in a real world setting, we plot the PR curve for non-
toxic versus the rest in Figure 2. Figure 3 showcases for each class versus
the rest . Our model achieves an average precision rate of 0.95 when comparing
non-toxic versus toxic tokens. In Table 6, we report the corresponding recall
at really high precision levels per category. At a 99.9% precision, the model
maintains between a 0.4-6.5% recall rate on different categories.

Figure 2: PR Curve. ToxBuster’s average precision for differentiating between
non-toxic and toxic words is at 95%

Category Name | 90.0% | 99.0% | 99.9%
---|---|---|---
Hate and Harassment | 14.22% | 2.35% | 2.35%
Threats | 0.26% | 0.26% | 0.26%
Minor Endangerment | 7.32% | 6.69% | 6.69%
Extremism | 0.50% | 0.50% | 0.50%
Scams and Ads | 8.84% | 4.76% | 4.76%
Insults and Flaming | 1.18% | 0.39% | 0.39%
Spam | 66.63% | 42.01% | 6.14%
Other Offensive | 2.09% | 0.72% | 0.72%

Table 6: Recall rate per toxic category at 90.0%, 99.0% and 99.9% precision
level.

We continue to analyze ToxBuster when operated at high precision levels. The
model can intercept 5.38%, 2.07% and 0.81% of chat lines at 90.0%, 99.0% and
99.9% precision respectively. This alludes towards a solution of real-time
chat toxicity and moderation that dynamically uses the past chat history which
Perspective API currently cannot do. In Table 7, we compare the intersection
amongst the distinct players flagged by ToxBuster and players reported for
their chat or disruptive behavior by other players in a week. We use these
player reports as a proxy to understand the model’s precision from the player
base. At the time of collection, R6S only allows players to report others for
one reason. Hence, a player that is both toxic in chat and disruptive in game
(e.g. cheating) will most likely be reported for their behavior rather than
their chat. The viability of an automated moderation system after the game is
finished and is reported is shown. A very simple and naive one operating
ToxBuster at 90.0% precision can perform automatic moderation effort for 82.1%
cases of chat reported players. Since ToxBuster also returns different toxic
categories that representing different severity, we can have varying
moderation efforts that take these signals into consideration.

% of Players | 90.0% | 99.0% | 99.9%
---|---|---|---
${F}$ | 29.48% | 11.64% | 7.89%
${F\cap CR}$ | 82.1% | 51.1% | 41.3%
${F\cap R}$ | 55.49% | 26.92% | 19.57%

Table 7: % of distinct players flagged ($F$) by ToxBuster. $CR$ and $R$ are
chat reported and reported players respectively. ${CR}$ and ${R}$ players
represent 5.47% and 25.64% of all distinct players respectively.

###  4.4 Model Transferability

Finally, we also compare our model on S3 of our dataset to see how
transferable our model is from R6S to FH . S3 is from FH and thus will have
slightly different data, i.e. mentions of game-specific characters, map,
skills, or events. Toxicity arising from R6S and FH may also be different. For
this, we use 20,339 chat lines (20% of the dataset) as the test set. We
perform transfer learning by training our best ToxBuster trained on R6S in
terms of F1 score on increasing number of chat lines from FH . The results can
be seen in Table 8. ToxBusterFH is trained solely on S3 dataset. We can
conclude that the model can transfer pretty well between R6S and FH . The
performance difference between all settings are minute. We do see that on low
data setting, transfer learning the model is not worth the effort and could
lead to slightly lower performance (+691 shows overall lower performance. We
also see that transfer learning can boost the model, evident from the 1.5% in
precision from ToxBusterFH and ToxBusterR6S+62,528.

| Precision | Recall | F1 Score
---|---|---|---
ToxBusterR6S | 84.14 ± 0.32 | 85.81 ± 0.27 | 84.46 ± 0.30
\+ 691 | 84.01 ± 0.39 | 85.18 ± 0.41 | 84.37 ± 0.33
\+ 15,787 | 84.57 ± 0.12 | 84.27 ± 0.07 | 84.23 ± 0.04
\+ 25,448 | 85.04 ± 0.33 | 85.43 ± 0.34 | 85.09 ± 0.28
\+ 35,410 | 85.11 ± 0.35 | 85.65 ± 0.61 | 85.18 ± 0.43
\+ 53,582 | 85.43 ± 0.52 | 85.89 ± 0.31 | 85.46 ± 0.40
\+ 62,528 | 85.23 ± 0.43 | 85.93 ± 0.63 | 85.45 ± 0.49
ToxBusterFH | 84.88 ± 0.64 | 85.62 ± 0.57 | 85.09 ± 0.54

Table 8: ToxBuster’s transferability. ToxBusterR6S and ToxBusterFH are fully
trained on their respective game. $+$ shows the number of samples from FH used
to finetune ToxBusterR6S.

###  4.5 Ablation Study

In this section, we describe the series of ablation study on the two main
components: chat history and chat speaker segmentation

####  4.5.1 Chat History

For this experiment, we do not include any chat speaker segmentation to narrow
down the impact of the different chat modes, with results shown in Table 9.
Intuitively, the more context you have, the more reliable you can predict
whether the current chat line is toxic or not. This intuition is confirmed by
the close to 4% difference in the model’s performance with the inclusion of
any kind of context. For all four chat modes, we recognize that the model’s
performance is similar. Interestingly, we do observe 1) precision improves as
more context is fed to the model and 2) global performs better than moderator
in terms of recall and F1 score but slightly lower in precision. From these
observations, it confirms with Lu et al. (2020) that a consistent part of the
chat is necessary for the model to perform well and a better scheme for
consistent chat can be further researched.

| Precision | Recall | F1 Score
---|---|---|---
No Context | 77.21 ± 1.30 | 77.91 ± 1.04 | 77.36 ± 1.31
Personal | 79.11 ± 0.66 | 79.82 ± 0.34 | 78.89 ± 0.76
Team | 80.51 ± 0.15 | 80.10 ± 0.16 | 81.30 ± 0.02
Global | 81.60 ± 0.51 | 82.21 ± 0.39 | 81.70 ± 0.46
Moderator | 81.90 ± 0.41 | 81.47 ± 0.37 | 81.68 ± 0.43

Table 9: Filtering Context Impacts Model Performance

####  4.5.2 Chat Speaker Segmentation

For this experiment, we use the global mode for chat history. We compare two
methods of including chat metadata in-line (*) and chat speaker segmentation.
In-line is a very intuitive method to include the metadata since we are
appending it in front of each chat line. We show the impact of each specific
metadata added and the full impact with all three included in Table 10. From a
human annotation perspective, all these information are automatically taken
into consideration, often to understand whether the chat is consistent or not.
In both methods, we can see that the most important feature is the playerID.
Again, this follows our intuition since that is the biggest differentiator for
which player is writing the chat line. The next feature is chat type, whether
the intended audience is for the team or for everyone (often to communicate
with the enemy team). When comparing in-line or chat speaker segmentation, for
each metadata, chat speaker segmentation performs better in terms of
precision, recall and F1 score. We believe that ToxBuster can learn the
correlation of the chat line metadata much better as each token has its own
associated metadata, rather than positioned in front of the chat line.

| Precision | Recall | F1 Score
---|---|---|---
Base | 81.60 ± 0.51 | 82.21 ± 0.39 | 81.70 ± 0.46
w/ teamID* | 81.59 ± 0.54 | 82.08 ± 0.89 | 81.69 ± 0.65
w/ chat type* | 81.82 ± 0.51 | 82.38 ± 0.38 | 81.88 ± 0.43
w/ playerID* | 81.90 ± 0.41 | 82.47 ± 0.37 | 82.01 ± 0.43
w/ full* | 81.95 ± 0.31 | 82.56 ± 0.26 | 82.12 ± 0.30
w/ teamID | 81.61 ± 0.41 | 82.23 ± 0.37 | 81.72 ± 0.43
w/ chat type | 81.91 ± 0.51 | 82.53 ± 0.76 | 82.18 ± 0.62
w/ playerID | 82.36 ± 0.44 | 82.79 ± 0.37 | 82.43 ± 0.41
w/ full | 82.95 ± 0.31 | 83.56 ± 0.26 | 83.25 ± 0.30

Table 10: Model performance with different metadata incorporated. * means it
was added in-line before each chat line rather than chat speaker segmentation.
Full has all metadata included.

##  5 Conclusion

We propose ToxBuster, a simple and scalable model to detect in-game toxicity.
We evaluate our model and state-of-the-art baselines on our dataset. Our
experiments revealed that language between social media and in-game chat
contrast enough to warrant their own dataset. We also demonstrated the
importance of past chat history and chat metadata, alluding to the possibility
of some real-time moderation. Applying our model towards automated post-game
moderation, it can correctly identify toxic chat players, with 82% of chat
reported players being flagged at 90.0% precision level. We also found that it
is transferable from one game to another.

## Limitations

Currently, the model’s dataset is limited to the English language, with the
exceptions of common toxic phrases appearing in in-game chat lines from other
languages. Based on results from Perspective API and Jigsaw, we know that the
methods presented in this paper can be extended from mono-lingual to multi-
lingual.

ToxBuster will make errors. Only existing patterns of toxicity in the dataset
will be detected. Language that closely resemble existing patterns of toxic
language could also be incorrectly flagged. As such, the model without any
active learning is not suitable for a fully automated moderation. The model
also cannot completely replace human moderation.

ToxBuster is intended for in-game chat that will have mentions of in-game
events. Hence, phrases that could be considered toxic (a threat) in normal
everyday language could be scored as neutral or having less probability of
being toxic.

ToxBuster is a step in the right direction towards combatting in-game
toxicity. Many areas can be addressed. In terms of improving toxicity
detection, some directions are performing domain adaption on the base language
model on unlabeled chat data, continuous learning and adversarial training.
The model and dataset can be extended from English to multilingual. Another
area is biases and its mitigation. While we have mitigated some during the
data annotation phase, we still need to measure biases the model has learned
and ways to debias the models without degrading the model performance.
Finally, we can also analyze the causes and impacts of toxicity from a player
and game design perspective.

## Ethics Statement

As with any language models, ToxBuster will propagate any existing biases in
the dataset. We have tried to mitigate biases in the annotation by taking the
diversity of the annotators identities into consideration. In our sessions, we
recognize that it was hard to recruit those that identify as a woman. We had
more success in recruiting those that identified as belonging to marginalized
groups (e.g. LGBTQA1+, BIPOC), where half of the annotators self-identifies as
belonging to at least one of the marginalized group.

We have made efforts to anonymize the data by removing player usernames and
removing any identifying information. Annotators were also warned about the
toxic content they will see. They were given a very lax schedule and allowed
to annotate freely at their own pace over a lengthy time period, allowing many
breaks if needed.

As stated in the limitations, we are in the process of devising methods to
measure bias and debias the model. An adaptation of Kiritchenko and Mohammad
(2018) Equity evaluation corpus (EEC) will be created to test and measure
several categories of social biases such as gender, race, sexual orientation,
etc. Meade et al. (2021) also includes a few benchmarks (Sentence Encoder
Association Test and Word Embedding Association Test May et al. (2019)) and
de-biasing methods. De-biasing methods include counterfactual data
augmentation (CDA), increasing dropout and projection-based techniques. CDA
works by rebalancing the dataset by swapping bias attribute words. As
recommended by Blodgett et al. (2020), we have invited and welcome new
researchers from other disciplinary studies, namely from linguistics and
psychology.

## Acknowledgements

We wish to thank Ubisoft La Forge and Ubisoft Montreal User Research Lab for
providing technical support and insightful comments on this work. We also
acknowledge funding in support of this work from Ubisoft.

## References

  * ADL (2021) ADL. 2021.  Online hate and harassment: The american experience 2021.
  * Almerekhi et al. (2022) Hind Almerekhi, Haewoon Kwak, Joni Salminen, and Bernard J. Jansen. 2022.  Provoke: Toxicity trigger detection in conversations from the top 100 subreddits.  _Data and Information Management_ , page 100019.
  * Blodgett et al. (2020) Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020.  Language (technology) is power: A critical survey of “bias” in NLP.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5454–5476, Online. Association for Computational Linguistics.
  * Ciftci et al. (2017) Tuba Ciftci, Liridona Gashi, René Hoffmann, David Bahr, Aylin Ilhan, and Kaja Fietkiewicz. 2017.  Hate speech on facebook.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: pre-training of deep bidirectional transformers for language understanding.  _CoRR_ , abs/1810.04805.
  * Döring and Mohseni (2020) Nicola Döring and M. Mohseni. 2020.  Gendered hate speech in youtube and younow comments: Results of two content analyses.  _Studies in Communication and Media_ , 9:62–88.
  * FairPlayAlliance and ADL (2020) FairPlayAlliance and ADL. 2020.  [link].
  * Fehn Unsvåg and Gambäck (2018) Elise Fehn Unsvåg and Björn Gambäck. 2018.  The effects of user features on Twitter hate speech detection.  In _Proceedings of the 2nd Workshop on Abusive Language Online (ALW2)_ , pages 75–85, Brussels, Belgium. Association for Computational Linguistics.
  * Gambäck and Sikdar (2017) Björn Gambäck and Utpal Kumar Sikdar. 2017.  Using convolutional neural networks to classify hate-speech.  In _Proceedings of the First Workshop on Abusive Language Online_ , pages 85–90, Vancouver, BC, Canada. Association for Computational Linguistics.
  * Gao and Huang (2017) Lei Gao and Ruihong Huang. 2017.  Detecting online hate speech using context aware models.  In _Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017_ , pages 260–266, Varna, Bulgaria. INCOMA Ltd.
  * Georgakopoulos et al. (2018) Spiros V. Georgakopoulos, Sotiris K. Tasoulis, Aristidis G. Vrahatis, and Vassilis P. Plagianakos. 2018.  Convolutional neural networks for toxic comment classification.
  * Hanu and Unitary team (2020) Laura Hanu and Unitary team. 2020.  Detoxify.  Github. https://github.com/unitaryai/detoxify.
  * Jhaveri et al. (2022) Manan Jhaveri, Devanshu Ramaiya, and Harveen Singh Chadha. 2022.  Toxicity detection for indic multilingual social media content.  _CoRR_ , abs/2201.00598.
  * Kiritchenko and Mohammad (2018) Svetlana Kiritchenko and Saif M. Mohammad. 2018.  Examining gender and race bias in two hundred sentiment analysis systems.
  * Lees et al. (2022) Alyssa Lees, Vinh Q. Tran, Yi Tay, Jeffrey Sorensen, Jai Gupta, Donald Metzler, and Lucy Vasserman. 2022.  A new generation of perspective api: Efficient multilingual character-level transformers.
  * Lewington (2021) Robert Lewington. 2021.  Being ‘targeted’ about content moderation:.
  * Lu et al. (2020) Junyu Lu, Xiancong Ren, Yazhou Ren, Ao Liu, and Zenglin Xu. 2020.  Improving contextual language models for response retrieval in multi-turn conversation.  pages 1805–1808.
  * Maher (2016) Brendan Maher. 2016.  Can a video game company tame toxic behaviour? _Nature News_ , 531:568.
  * May et al. (2019) Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019\.  On measuring social biases in sentence encoders.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 622–628, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Meade et al. (2021) Nicholas Meade, Elinor Poole-Dayan, and Siva Reddy. 2021.  An empirical survey of the effectiveness of debiasing techniques for pre-trained language models.
  * Mohan et al. (2017) Shruthi Mohan, Apala Guha, Michael Harris, Fred Popowich, Ashley Schuster, and Chris Priebe. 2017.  The impact of toxic language on the health of reddit communities.  pages 51–56.
  * Mubarak et al. (2017) Hamdy Mubarak, Kareem Darwish, and Walid Magdy. 2017.  Abusive language detection on Arabic social media.  In _Proceedings of the First Workshop on Abusive Language Online_ , pages 52–56, Vancouver, BC, Canada. Association for Computational Linguistics.
  * Park and Fung (2017) Ji Ho Park and Pascale Fung. 2017.  One-step and two-step classification for abusive language detection on Twitter.  In _Proceedings of the First Workshop on Abusive Language Online_ , pages 41–45, Vancouver, BC, Canada. Association for Computational Linguistics.
  * Pavlopoulos et al. (2020a) John Pavlopoulos, Jeffrey Sorensen, Lucas Dixon, Nithum Thain, and Ion Androutsopoulos. 2020a.  Toxicity detection: Does context really matter? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4296–4305, Online. Association for Computational Linguistics.
  * Pavlopoulos et al. (2020b) John Pavlopoulos, Jeffrey Sorensen, Lucas Dixon, Nithum Thain, and Ion Androutsopoulos. 2020b.  Toxicity detection: Does context really matter?
  * Silva et al. (2020) Bruno Silva, Mirian Tavares, Filipa Cerol, Susana Silva, Paulo Alves, and Beatriz Isca. 2020.  Playing against hate speech -how teens see hate speech in video games and online gaming communities.  pages 34–52.
  * Van Aken et al. (2018) Betty Van Aken, Julian Risch, Ralf Krestel, and Alexander Löser. 2018.  Challenges for toxic comment classification: An in-depth error analysis.
  * Watanabe et al. (2018) Hajime Watanabe, Mondher Bouazizi, and Tomoaki Ohtsuki. 2018.  Hate speech on twitter: A pragmatic approach to collect hateful and offensive expressions and perform hate speech detection.  _IEEE Access_ , 6:13825–13835.
  * Zannettou et al. (2020) Savvas Zannettou, Mai Elsherief, Elizabeth Belding, Shirin Nilizadeh, and Gianluca Stringhini. 2020.  Measuring and characterizing hate speech on news’ websites.  In _12th ACM Conference on Web Science_ , WebSci ’20, page 125–134, New York, NY, USA. Association for Computing Machinery.
  * Zhong et al. (2016) Haoti Zhong, Hao Li, Anna Squicciarini, Sarah Rajtmajer, Christopher Griffin, David Miller, and Cornelia Caragea. 2016.  Content-driven detection of cyberbullying on the instagram social network.

##  Appendix A Annotator Details

For all three sessions, annotators were recruited from social media with
representation in game experience and self-identification with marginalized
groups taken into consideration. Each annotator had to be at least 18 years
old, advanced in English proficiency, reside in the North American time zone
and active in the respective game. We define active as having played the
respective game within the last year for at least 16 hours in the player
versus player (PVP) mode. After the initial recruitment, a pilot test was
conducted to further filter annotators to those that understood the task and
aligned themselves on the common definition of toxicity based on examples
shown. Each annotator was instructed to highlight the minimum span of
contiguous words in a chat line that falls under a toxic category. If a span
of words can fall under more than one toxic category, they were to use the
most severe category. Each chat line was annotated by three annotators, with
full visibility of all previous chat lines of the match available. We
anonymized any player names shown to annotators as numbers. We also did not
show any game related events nor whether the player was reported.

##  Appendix B Detailed Toxicity Definitions

Our dataset does not include any audio or visual data, and therefore,
categories such as cheating, abuse of play, antisocial actions are not within
the scope of this model.

  1. 1.

Hate and Harassment: Identity-based hate or harassment (e.g., racism, sexism,
homophobia) or bullying / mobbing (e.g., a group of players bullying one or
more players).

  2. 2.

Threats: Threats of violence, physical safety to another player, employee or
property, terrorism, or releasing a player’s real-world personal information
(e.g., doxxing).

  3. 3.

Minor Endangerment: Sexual and/or aggressive actions towards minors or
attempts to get minors to perform sexual activities.

  4. 4.

Extremism: Extremist views (e.g., white supremacy), attempts to groom or
recruit for an extremist group or repeated sharing of political, religious, or
social beliefs.

  5. 5.

Scams and Ads: Fraud / scamming (e.g., including phishing, account stealing,
bad trades or theft), posting inappropriate links (e.g., malware, dangerous
websites, advertising exploits, etc ) and advertising of websites, services,
cheats or rival products.

  6. 6.

Insults and Flaming: Insults or attacks on another player or team (not based
on player or team’s real or perceived identity)

  7. 7.

Spam: Excessive sharing of the same or similar words, phrases, emojis or
sharing (e.g., "kdjfklsjafkldjkla").

  8. 8.

Other Offensive Texts: Any other message not covered in the above categories
that is offensive and/or harms a player’s reasonable enjoyment of the game.

##  Appendix C Model Reproduction Details

Our model is implemented with HuggingFace444https://huggingface.co/ and
PyTorch555https://pytorch.org/. Section 3.2.1 and 3.2.2 include explicit
information on the model architecture and preprocessing steps. Our model uses
default values for BERT. The learning rate is 1e-5, chosen from a
hyperparameter search amongst 1e-4, 1e-5 and 1e-6. The learning rate scheduler
is set to linear decay with a warmup step ratio of 5%. Training with GeForce
RTX 2080 took approximately 7 hours with max train epochs set to 100, early
stopping with patience of 5 epochs based on the weighted F1 score.

Figure 3: Precision-Recall per Toxic Category.

##  Appendix D Sentence-level vs. Token-level

For this experiment, we do not include chat speaker segmentation and use
global mode for chat history. We analyze in Table 11 the impact of changing
the max_token_size for the tokenizer and the level of classification
(sentence-level and token-level). Contrary to our beliefs, it would seem that
classifying on the sentence level as opposed the token-level is a slightly
harder task.

| Token | Sentence
---|---|---
64 | 77.06 ± 0.88 | 76.12 ± 1.56
128 | 78.90 ± 0.64 | 79.05 ± 1.39
256 | 81.41 ± 0.42 | 80.45 ± 1.27
512 | 82.09 ± 0.39 | 80.88 ± 0.51
Table 11: Model mean F1 score on different max token length and classification
mode.