# Collaborative Learning to Generate Audio-Video Jointly

###### Abstract

There have been a number of techniques that have demonstrated the generation
of multimedia data for one modality at a time using GANs, such as the ability
to generate images, videos, and audio. However, so far, the task of multi-
modal generation of data, specifically for audio and videos both, has not been
sufficiently well-explored. Towards this, we propose a method that
demonstrates that we are able to generate naturalistic samples of video and
audio data by the joint correlated generation of audio and video modalities.
The proposed method uses multiple discriminators to ensure that the audio,
video, and the joint output are also indistinguishable from real-world
samples. We present a dataset for this task and show that we are able to
generate realistic samples. This method is validated using various standard
metrics such as Inception Score, Frechet Inception Distance (FID) and through
human evaluation.

Index Terms—  Audio-video generation, cross-modal learning

##  1 Introduction

In the real world, different modalities of the same data are highly
correlated. Video-audio pairs are one such example. Recent work has
extensively explored the generation of data such as audio, video (without the
corresponding audio), and text. These methods consider the single modality
generation problem either in the form of unconditional generation or
conditional generation. Meta information provided for conditional generation
can be the class label or one of the other modalities. Cross model generation
focuses on the latter form of conditional generation. The former is known as
single modality generation, such as audio generation or video generation,
which generate the required modality independently. One of the major drawbacks
of these methods is that they do not consider that audio and video streams of
the same data are highly correlated and could be generated jointly both
benefiting each other. Most multimedia synchronization methods developed in
the past have dealt with uni-modal streams and consider only audio data or
video data. Recently, there have been a few works [1, 2] that aim to solve
cross-modal generation. However, they generate a single image frame for an
audio duration and hence fail in real-world video generation as they do not
generate a sequence of image frames (i.e. a real video signal) corresponding
to the synchronized audio. The interaction between audio and vision, despite
seeing traction as of late, is still largely unexplored. In order to solve the
problem effectively, we propose a collaborative learning framework that uses
adversarial models to ensure that there is out of sample generalization.
Specifically, we incorporate multiple discriminators to ensure that the
samples generated individually for each modality are realistic. Figure 1 shows
our proposed framework.

Fig. 1: Illustration of collaborative generation of audio-video pairs using
joint discriminators.

The exact problem that we want to tackle has not been explored precisely and
therefore we did not have a dataset directly applicable to our task. Hence, we
composed two novel datasets - Hand Motion Music Dataset (HMMD) and Sub-URMP-
video dataset (the latter by modifying an existing dataset). These datasets
are primarily related to musical instruments, and through these datasets, we
explore the accurate joint-modal correspondence between audio and video
signals. To summarize, we make the following main contributions in this work:
1) We propose a joint collaborative audio-video generation module that uses
noise as a latent vector. 2) We provide an audio-video multimodal dataset
comprising specifically chosen videos where audio is a direct outcome of the
motion (actions) happening in the video frames. 3) We formulate the joint
audio-video generation training problem in an adversarial framework and
achieve promising results.

##  2 Related work

Generative adversarial networks [3] is one of the unsupervised techniques that
maps low dimensional latent features to high dimensional data. Various
architectures have been suggested for generating images using GANs [4, 5, 6,
7]. These image generation architectures have also been extended for video
generation. MoCoGAN [8] proposes to solve the video generation problem by
decomposing motion and content vector. The work in generation is not only
limited to the visual modality, but it also extended for other modalities such
as music [9, 10] and speech generation [11, 12, 13]. For instance, SEGAN [14]
uses and adversarial framework to generate speech signals. Van et al. [15]
introduced a WaveNet model based on an autoregressive method to predict raw
audio samples. Engel et al. [16] used an auto-encoder model in WaveNet to
generate musical instrument sounds. Other multi-modal generation frameworks
that have been proposed in the space of audio-visual modalities are [1, 2, 17,
18]. Most of the above audio generation works are based on conditional
generation conditioned on an image frame of the corresponding video. Recently,
several approaches that solve for the alignment of various modalities [19, 20,
21, 11, 22, 12, 23, 24] have also been suggested. Music gesture [25] uses the
keypoint-based structured representation to explicitly model the body’s and
finger’s dynamics motion cues for visual sound separation. A few very recent
works have also explored the multimodal generation problem. Gan et al.[26]
synthesized plausible music for a silent video clip of people playing musical
instruments. Another similar work [27] generated music for a given video. [28]
generated visually aligned sounds from videos. In sound2sight [29], future
video frames and motion dynamics are generated by conditioning on audio and a
few past frames. In SA-CMGAN [30] self-attention mechanism is applied to
cross-modal visual-audio generation.

Fig. 2: In our proposed model, audio and video are generated using a latent
noise vector. The joint discriminators are trained on joint distributions of
audio-video pairs.

##  3 Method

The proposed framework consists of three modules - audio generation module,
video generation module and joint discriminators module.

Audio Generation Module: It consists of an audio generator $A_{G}$, that
generates audio from noise $\epsilon_{0}$ and a discriminator [31]. This
generator module is parameterized through a function
$G_{a}(\epsilon_{0},W_{ga})$ where $W_{ga}$ are the weights of the model. We
work with the raw audio waveform, and the generator produces raw waveforms
instead of spectrograms. The generator module uses 1-D deconvolutional layers.
Since there are frequency bands present in each audio waveform, unlike the
image signal where the frequency patterns are not so common, an image
discriminator architecture doesn’t work well for audio generation. The
discriminator easily learns the frequency pattern commonly known as the
checker board effect and discards those waveforms. These artifact frequencies
occur for a particular phase. Therefore, our discriminator needs to learn this
trivial policy to discard such generated examples. To resolve this hurdle, we
employ the skill of phase shuffle operation [31] to prevent our audio
discriminator from learning such artifacts. It is parameterized through a
function $D_{a}(\epsilon_{0},W_{da})$ where $W_{da}$ are the weights of the
model. Generator $G_{a}$ is trained to minimize the following value function,
while $D_{a}$ is trained to maximize it.

$\begin{split}V(D_{a},G_{a})=\mathbb{E}_{\boldsymbol{a}\sim
R_{\boldsymbol{A}}}[\log D_{a}(\boldsymbol{a})]+\\\
\mathbb{E}_{\boldsymbol{\epsilon}\sim
P_{\epsilon}}[\log(1-D_{a}(G_{a}(\boldsymbol{\epsilon_{0}})))]\end{split}\vspace{-0.75em}$
(1)

$\boldsymbol{a}$ is sampled from real data($R_{\boldsymbol{A}}$) and
$\boldsymbol{\epsilon}$ is random noise.

Table 1: Sub-URMP Video dataset

Cat. | Cello | D.Bass | Oboe | Sax | Trumpet | Viola | Bassoon | Clarinet | Horn | Flute | Trombone | Tuba | Violin
---|---|---|---|---|---|---|---|---|---|---|---|---|---
Train | 881 | 123 | 394 | 696 | 100 | 648 | 135 | 716 | 475 | 511 | 787 | 323 | 707
Test | 94 | 114 | 36 | 81 | 49 | 48 | 37 | 92 | 49 | 90 | 77 | 49 | 91

Table 2: Hand Motion Music Dataset(HMMD). Complete videos refer to raw video
clips collected from Youtube for composing our HMMD dataset detailed in
Section 4.1. We had an 80:20 train-test split for the composed dataset.
AG:Acoustic Guitar

Category | Cello | Tabla | Ghatam | Erhu | AG | Xylophone | Violin
---|---|---|---|---|---|---|---
Videos | 15 | 3 | 1 | 12 | 11 | 10 | 25
Frame | 1251 | 317 | 454 | 519 | 1041 | 800 | 1847

Video Generation Module: Our video generation component takes inspiration from
the idea of decomposing videos into a separate motion and content part [8].
Our video clips $v_{n}=[I_{1},\dots,I_{n}]$ are composed of $N$ frames. The
temporal relationship between the video frames is obtained by modeling the
latent space vector $Z_{n}$. The content of the video clips is encapsulated by
introducing a context vector $Z_{C}$ and is combined with every motion vector.
Each input to the video generator is a combination of the latent motion vector
and the latent content vector. Each $Z_{I}$ is an input to the image generator
to generate each frame in the video clip. We use individual image
discriminator $D_{I}$ and a video discriminator $D_{V}$ to play the role of a
judge to provide criticism to video (image) generator $G_{I}$. The learning
problem is formulated as:

$\begin{split}&\min_{G_{I}}\max_{D_{I},D_{V}}(C_{v}(D_{I},D_{V},G_{I}))=\\\
&\mathbb{E}_{v\sim\mathcal{V}}[\log D_{I}(S_{1}(v))]+\mathbb{E}_{\hat{v}\sim
G_{V}}[\log(1-D_{I}(S_{1}(\hat{v})))]+\\\ &\mathbb{E}_{v\sim\mathcal{V}}[\log
D_{V}(S_{T}(v))]+\mathbb{E}_{\hat{v}\sim
G_{V}}[\log(1-D_{V}(S_{T}(\hat{v})))]\vspace{-0.5em}\end{split}$

$S_{1}$ and $S_{T}$ are two sample functions for image discriminator $D_{I}$
and video discriminator $D_{V}$ which takes as input, one frame and T
consecutive frames respectively. The sample functions $S_{1}$, and $S_{T}$ are
sampled from real $\mathcal{V}$ and generated video $\hat{V}$ clip. $\hat{v}$
is the generated video.

Joint Discriminators: The marginal distributions are represented for audio and
video as $p(a)$ and $p(v)$. The joint distribution of audio and video is
denoted by $p(a,v)$. To jointly learn the audio-video generation, we use two
joint discriminators, inspired by [32]. This pair of joint discriminators is
trained to distinguish real data pairs $(a,v)$ and two kinds of generated data
pairs $(a,\hat{v})$, $(\hat{a},v)$. The main objective of this module is to
match the three joint distributions: $p(a,v),p(a,\hat{v})\&p(\hat{a},v)$ such
that the generated fake data pairs $(a,\hat{v})$ and $(\hat{a},v)$ are
indistinguishable from the real data pairs $(a,v)$. We follow an adversarial
framework to match the joint distributions. The overall-cost function for
joint discriminators is defined as:

$\vspace{-1em}\begin{split}\small&\min_{G_{A},G_{I}}\max_{D_{1},D_{2}}C_{M}(D_{1},D_{2},G_{A},G_{I})=\mathbb{E}_{a\sim\mathcal{A},v\sim\mathcal{V}}[\log
D_{1}(a,v)]\\\ &+\mathbb{E}_{\hat{a}\sim
G_{A},v\sim\mathcal{V}}\Big{[}\log\Big{(}(1-D_{1}(\hat{a},v)D_{2}(\hat{a},v)\Big{)}\Big{]}\\\
&+\mathbb{E}_{{a}\sim\mathcal{A},\hat{v}\sim
G_{I}}\Big{[}\log\Big{(}(1-D_{1}(a,\hat{v})(1-D_{2}(a,\hat{v})\Big{)}\Big{]}\end{split}\vspace{-1em}$
(2)

Here, $\mathcal{A}$ and $\mathcal{V}$ are the audio and video data
distributions respectively. The objective of $D_{1}$ is to distinguish whether
a sample pair is from real $p(a,v)$ or not, If the sample pair is not from
$p(a,v)$, we use another discriminator $D_{2}$ to distinguish if the sample
pair is from $p(\hat{a},v)$ or $p(a,\hat{v})$. Both the discriminators
$D_{1}\&D_{2}$ work collectively, and the use of both of them implicitly
define a ternary discriminative function $D$.

##  4 Experiments

###  4.1 Datasets

We compose two novel datasets to train and evaluate our models- Sub-URMP-Video
Dataset, a video version of the Sub-URMP dataset [1] and Hand Motion Music
Dataset (HMMD). To show the effectiveness of our method, we need videos with
significant movement across frames and where audio is directly related of the
motion happening in the video frames. We construct our own dataset (HMMD)
comprising videos from Youtube focusing particularly on percussion instruments
like Tabla, etc. which have significant hand movements across frames. Both the
datasets and the code will be made publicly available.

Sub-URMP-Video Dataset: It contains 13 musical instrument categories. It
contains image-audio pairs with ten images per second and audio of 0.5 seconds
paired with each image. We form 10 frames per second(fps) video with 1 second
of audio and match them to make it suitable for our task. We also manually
remove the videos that contain different speakers. The basic information is
summarized in Table 1.

HMMD: There are seven types of instruments, in total and 77 video clips. The
number of clips ranges from 1 to 25 for various instruments. We preprocess the
dataset to form videos of 1 sec each having a frame rate of 25 per sec. Audio
streams are sampled at 16kHz. The proposed Hand Motion Music Dataset consists
of 6229 video-audio pairs. The details of the dataset are present in Table 2.

###  4.2 Results

We compare our model with the baseline model using both quantitative and
qualitative methods such as inception score, Statistically-Different Bins,
pitch entropy, Frechet Inception Distance (FID) for audio and qualitative
performance, along with a human evaluation for video and audio. We work with
two modalities having separate architecture for generation. The video
generation module is inspired from MoCoGAN [8], and for audio generation
model, it is adapted from WaveGAN [31]. For video generation, we used a
deconvolution layers based generator for producing 64x64 images from a 100-dim
noise vector. To the best of our knowledge, there was no prior work in joint
audio video generation as has been iterated in the paper and therefore the
absence of a precise baseline. Consequently, we remove the joint discriminator
component (c) from our model and consider the remaining model as a baseline
for quantitative evaluations. and compare the effect of joint generation to
isolated generations. Therefore, the baseline model we choose are the single
modality generation components for audio and video respectively. The other
details are provided in project page 111https://delta-lab-iitk.github.io/AVG/.

Table 3: Audio Evaluation Metrics for proposed models.
BL: Baseline, JM:Proposed Joint Model

Dataset-Model |  Human($\uparrow$) |  NDB($\downarrow$) |  FID($\downarrow$) |  IS($\uparrow$) |  PE($\downarrow$)
---|---|---|---|---|---
S-URMP(BL) |  3.2 |  14 |  111.25 |  1.38 |  7.53
S-URMP(JM) |  3.6 |  10 |  97.43 |  1.44 |  6.12
HMMD(BL) |  3.1 |  36 |  93.00 |  1.34 |  7.33
HMMD(JM |  3.4 |  29 |  79.89 |  1.48 |  5.42

Table 4: Video Evaluation Metrics for the proposed models

Dataset | Model | Human($\uparrow$) | IS($\uparrow$)
---|---|---|---
Sub-URMP | Ground Truth | 4.6 | 1.42
Sub-URMP | Baseline | 3.1 | 1.88
Sub-URMP | Joint Model | 3.7 | 1.91
HMMD | Ground Truth | 4.8 | 1.26
HMMD | Baseline | 2.9 | 4.24
HMMD | Joint Model | 3.5 | 4.33

###  4.3  Audio Performance Evaluation method

To validate the proposed model, we provide a quantitative analysis of the
generated audio signals. The audio signals are generated using only noise as a
latent vector, unlike the previously proposed methods [1, 2] that use a
conditional vector for audio generation. We also compare the quality of the
audio signal produced without the joint discriminator model. From this
assessment, we can clearly say that the joint audio-video model helps to
generate better audio samples.

###  4.4 Video Performance Evaluation

We also validate our model using the video generation task. The state-of-art
video generation model [8] is considered as a baseline for our task which is a
component of our joint model. We cannot compare our model with other video
generation methods because they rely on conditional video generation which is
a much easier task. The generated frames are shown in Figure- 3 for Sub-URMP
dataset. We observe that the joint audio-video generation framework helps the
model to learn better video frames. We observe that the inception score
improves 0.3% for Sub-URMP and 0.9% for HMMD dataset.

Fig. 3: Results on Sub-URMP Dataset:In this figure, the first row indicates
the given ground truth video. The second row indicates generated videos for
our baseline model and third row shows generated video through our joint model

##  5 Analysis

The joint model outperforms the baseline models in all our evaluations-in
Table 2 as well as Table 4. We have designated alongside every evaluation
metric with arrows, where up-arrow ($\uparrow$) indicates that a larger value
is better and similarly the down-arrow ($\downarrow$) suggests that a lower
value is better.

Comparison with Deep Cross model generation: In Deep cross-modal generation
[1], audio and image generations are conditional in nature. But in our
proposed model, audio and video are generated from a latent noise vector. The
former also uses the music class label for generation while our work is
completely unsupervised. Another difference is that we are generating videos
(a sequence of image frames ), which is a more challenging task, unlike the
previous work which generates only images.

Comparison with CMC GAN: The CMC-GAN [2] uses the cycle consistency between
generated audio and video samples. However, this work too relies on
conditional generation of audio and images data. It faces the same drawback as
the previous one as it works only on images and not on videos whereas our
model generates audio and video modalities collaboratively without
conditioning on each other.

Comparison with Triangle GAN: The Triangle GAN [32] model too does conditional
generation by joint training of the modalities. Again, our work does not use
any conditional dependence rather it jointly learns the audio-video
generation.

Comparison with independent modality Generation model: The other video
generation models such as MoCoGAN [8] do not consider audio modality for
generating videos. Similarly audio generation models such as WaveGAN [33] also
generate audio without considering the video information. Therefore, to the
best of our knowledge, there are no other models available that can generate
audio and video simultaneously by jointly training the generators.

Comparison with SA-CMGAN: In SA-CMGAN [30] audios and images are generated
using the conditional GAN, while we generate videos and audio, which is more
challenging using the joint learning. They also use the class label while
training the model. In contrast to it, proposed framework is completely
unsupervised.

##  6 Conclusion

This paper proposes a collaborative generation paradigm for audio and video
generation without any conditioning involved. It can also be trained for an
end to end fashion for better convenience. The generated videos and audio are
realistic, as is manifested by numerous experiments as well as human
evaluation. The results clearly depict that collaborative generation
successfully takes advantage of the subtle relationship that video stream has
with the audio stream to generate naturalistic images and improved quality
audio. The proposed two datasets will immensely help the community in audio-
visual research. Using collaborative generation of related modalities to
improve the quality of individual modalities generated is a promising
direction which we have initiated through this work. Hence, we believe that
this paper will act as a catalyst for work in multimodal generation.

## References

  * [1] L. Chen, S. Srivastava, Z. Duan, and C. Xu,  “Deep cross-modal audio-visual generation,”  in ACM Multimedia Workshops, 2017.
  * [2] Wangli Hao, Zhaoxiang Zhang, and He Guan,  “Cmcgan: A uniform framework for cross-modal visual-audio mutual generation,”  in AAAI, 2018.
  * [3] I. Goodfellow, J. P.-Abadie, M. Mirza, B. Xu, S. W.-Farley, D.and Ozair, A. Courville, and Y. Bengio,  “Generative adversarial nets,”  in NIPS, 2014, pp. 2672–2680.
  * [4] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen,  “Progressive growing of gans for improved quality, stability, and variation,”  2018\.
  * [5] A Radford, L Metz, and S Chintala,  “Unsupervised representation learning with deep convolutional generative adversarial networks,”  arXiv:1511.06434, 2015.
  * [6] Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz,  “Multimodal unsupervised image-to-image translation,”  in (ECCV), 2018, pp. 172–189.
  * [7] C. Wan, S.-Po Chuang, and H.-Yi Lee,  “Towards audio to scene image synthesis using generative adversarial network,”  in ICASSP, 2019.
  * [8] Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz,  “Mocogan: Decomposing motion and content for video generation,”  in CVPR, 2018.
  * [9] Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck,  “Music transformer: Generating music with long-term structure,”  2018\.
  * [10] Jesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue, and Adam Roberts,  “Gansynth: Adversarial neural audio synthesis,”  2018\.
  * [11] Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein,  “Looking to listen at the cocktail party: a speaker-independent audio-visual model for speech separation,”  ACM Transactions on Graphics (TOG), 2018.
  * [12] David Harwath, Adria Recasens, Dídac Surís, Galen Chuang, Antonio Torralba, and James Glass,  “Jointly discovering visual objects and spoken words from raw sensory input,”  in ECCV, 2018, pp. 649–665.
  * [13] Chris Donahue, Bo Li, and Rohit Prabhavalkar,  “Exploring speech enhancement with generative adversarial networks for robust speech recognition,”  in (ICASSP), 2018.
  * [14] Santiago Pascual, Antonio Bonafonte, and Joan Serrà,  “Segan: Speech enhancement generative adversarial network,”  Proc. Interspeech 2017, 2017.
  * [15] Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu,  “Wavenet: A generative model for raw audio,”  in 9th ISCA Speech Synthesis Workshop.
  * [16] Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Mohammad Norouzi, Douglas Eck, and Karen Simonyan,  “Neural audio synthesis of musical notes with wavenet autoencoders,”  in ICML, 2017.
  * [17] Yipin Zhou, Zhaowen Wang, Chen Fang, Trung Bui, and Tamara L Berg,  “Visual to sound: Generating natural sound for videos in the wild,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 3550–3558.
  * [18] C. Young Lee, A. Toffy, G. J. Jung, and W.J. Han,  “Conditional wavegan,”  arXiv preprint arXiv:1809.10636, 2018.
  * [19] Relja Arandjelovic and Andrew Zisserman,  “Objects that sound,”  in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 435–451.
  * [20] A. Senocak, Tae-Hyun Oh, J. Kim, Ming-Hsuan Yang, and In So Kweon,  “Learning to localize sound source in visual scenes,”  in CVPR, 2018, pp. 4358–4366.
  * [21] Ruohan Gao, Rogerio Feris, and Kristen Grauman,  “Learning to separate object sounds by watching unlabeled video,”  in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 35–53.
  * [22] Andrew Owens and Alexei A Efros,  “Audio-visual scene analysis with self-supervised multisensory features,”  in ECCV, 2018, pp. 631–648.
  * [23] Tae-Hyun Oh, T. Dekel, C. Kim, I. Mosseri, W. T Freeman, M. Rubinstein, and W. Matusik,  “Speech2face: Learning the face behind a voice,”  in CVPR, 2019, pp. 7539–7548.
  * [24] Andrew Owens, Phillip Isola, Josh McDermott, Antonio Torralba, Edward H Adelson, and William T Freeman,  “Visually indicated sounds,”  in CVPR, 2016, pp. 2405–2413.
  * [25] Chuang Gan, Deng Huang, Hang Zhao, Joshua B Tenenbaum, and Antonio Torralba,  “Music gesture for visual sound separation,”  in CVPR, 2020, pp. 10478–10487.
  * [26] Chuang Gan, Deng Huang, Peihao Chen, Joshua B Tenenbaum, and Antonio Torralba,  “Foley music: Learning to generate music from videos,”  ECCV, 2020.
  * [27] Kun Su, Xiulong Liu, and Eli Shlizerman,  “Audeo: Audio generation for a silent performance video,”  NeurIPS, 2020.
  * [28] Peihao Chen, Yang Zhang, Mingkui Tan, Hongdong Xiao, Deng Huang, and Chuang Gan,  “Generating visually aligned sound from videos,”  IEEE TIP, vol. 29, pp. 8292–8302, 2020.
  * [29] Anoop Cherian, Moitreya Chatterjee, and Narendra Ahuja,  “Sound2sight: Generating visual dynamics from sound and context,”  ECCV, 2020.
  * [30] Huadong Tan, Guang Wu, Pengcheng Zhao, and Yanxiang Chen,  “Spectrogram analysis via self-attention for realizing cross-model visual-audio generation,”  in ICASSP 2020.
  * [31] Chris Donahue, Julian McAuley, and Miller Puckette,  “Adversarial audio synthesis,”  2018\.
  * [32] Zhe Gan, Liqun Chen, Weiyao Wang, Y. Pu, Y. Zhang, Hao Liu, Chunyuan Li, and Lawrence Carin,  “Triangle generative adversarial networks,”  in NIPS, 2017.
  * [33] Chris Donahue, Julian McAuley, and Miller Puckette,  “Synthesizing audio with generative adversarial networks,”  arXiv preprint arXiv:1802.04208, 2018.


# Diverse forms of $\sigma$ bonding in two-dimensional Si allotropes: Nematic
orbitals in the MoS2 structure

Florian Gimbert  Chi-Cheng Lee  Rainer Friedlein  Antoine Fleurence  Yukiko
Yamada-Takamura  School of Materials Science, Japan Advanced Institute of
Science and Technology (JAIST), 1-1 Asahidai, Nomi, Ishikawa 923-1292, Japan
Taisuke Ozaki  School of Materials Science, Japan Advanced Institute of
Science and Technology (JAIST), 1-1 Asahidai, Nomi, Ishikawa 923-1292, Japan
Research Center for Simulation Science, Japan Advanced Institute of Science
and Technology (JAIST), 1-1 Asahidai, Nomi, Ishikawa 923-1292, Japan

###### Abstract

The interplay of $sp^{2}$\- and $sp^{3}$-type bonding defines silicon
allotropes in two- and three-dimensional forms. A novel two-dimensional phase
bearing structural resembleance to a single MoS2 layer is found to possess a
lower total energy than low-buckled silicene and to be stable in terms of its
phonon dispersion relations. A new set of cigar-shaped, nematic orbitals
originating from the Si $sp^{2}$ orbitals realizes bonding with a 6-fold
coordination of the inner Si atoms of the layer. The identification of these
nematic orbitals advocates diverse Si bonding configurations different from
those of C atoms.

###### pacs:

73.22.-f, 71.20.Mq, 61.48.-c

With its forefront runner graphene and its unique and exotic properties, at
present, two-dimensional materials experience an explosion of interest in
scientific and technological aspects Geim07 . While the excellent electronic
properties of graphene are derived from its structural robustness, the same
property makes it a challenging task to engineer the optical and transport
properties. This challenge is stimulating the search for alternative two-
dimensional layered materials that are more flexible in terms of its
structural and electronic properties Cahangirov07 ; Fleurence12 . In this
context, in particular, two new promising two-dimensional materials with a
honeycomb structure made of silicon or germanium atoms have been studied as
theoretical objects since 1994 Takeda94 . Most importantly, in yet
hypothetical, slightly buckled, free-standing forms, silicene and germanene,
as they were later called Guzman07 , exhibit a band structure similar to that
of graphene, merging linear dispersion of $\pi$ and $\pi^{*}$ bands at the
Fermi level to form Dirac cones at the $K$ points Takeda94 ; Guzman07 ;
Cahangirov07 .

Experimentally, it has been shown that two-dimensional Si honeycomb lattices
can be formed epitaxially on Er layers Wetzel94  as well as on the
Ag(111)Voigt12 ; Lin12 ; Jamgotchian12 , ZrB2(0001)Fleurence12  and
Ir(111)Meng13  surfaces. It is established that the interactions with the
substrates have a distinct influence on the structural and electronic
properties of the layers Chen12 ; Chen13 ; Lee13 . No experimental evidence
for the existence of germanene has been reported yet.

As density functional theory (DFT) calculations predict consistently that
slighly or low-buckled (LB) silicene is the most stable form of freestanding
Si allotropes, very recently, it has been shown that the addition of Si
adatoms on pristine silicene results in the formation of a dumbbell structure
with a lower total energy Kaltsas13 ; Ongun13 . Interestingly, a higher
cohesive energy can be achieved towards the complete coverage of the adatoms,
which possesses even a higher cohesive energy than the low-buckled silicene.
In fact, the periodic dumbbells can be recognized as the structure of a well-
known single-layer of MoS2. By considering the 4-fold coordination realized in
the $sp^{3}$ bonding of Si atoms and also for the Si atom connecting the low-
buckled silicene and a dumbbell, it is diffcult to understand why bonding with
a 6-fold coordination could be formed by Si atoms in the MoS2 structure.
Therefore, it is timely and interesting to investigate the properties of this
new Si phase beyond the view related to the introduction of defects or adatoms
to low-buckled silicene.

In this Letter, by first-principles calculations Ozaki03 ; DFT , we
investigate the stability of this new Si phase (MoS2-Si) together with a
possible similar Ge allotrope (MoS2-Ge), whose structures are that of a single
layer of molybdenum disulfide, or MoS2 Radisavljevic11 , in a wide range of
lattice constants and compare it with that of other two-dimensional silicon
structures. The phonon dispersion further demonstrates that MoS2-Si is stable
on the Born-Oppenheimer surface. To understand the 6-fold coordinated bonding
in MoS2-Si, we construct symmetry-respecting Wannier functions Vanderbilt ;
Weng . A new form of $\sigma$ bonding expressed by three cigar-shaped orbitals
co-exists with $\pi$ bands that are formed by three additional reconstructed
$p_{z}$ orbitals. The direction of these orbitals has changed from the typical
in-plane direction of the $sp^{2}$ to the out-of-plane direction to form
cigar-shaped orbitals. In analogy to the nematic electronic structure Chuang ,
the aligned and cigar-shaped orbitals may be called ”nematic”. While as a
common feature in the two-dimensional Si allotropes, the three $\sigma$ bands
show dispersions similar to those of bands in low-buckled silicene, values of
bond lengths and buckling heights vary significantly. Our finding suggests
that the $\sigma$ bonds of Si atoms are more flexible than one could expect
such that diverse forms of $\sigma$ bonding can allow the existence of a
number of Si allotropes.

Figure 1: Top and side view of the MoS2-type single layer for silicon atoms
with the lattice constant (a), the bonding distance (d) and the thickness (t)
indicated.

As shown in Figure 1, a single layer of MoS2-Si crystallizes in an A-B-A
stacking structure. In preserving the honeycomb structure, the A-B layer taken
by itself is exactly freestanding low-buckled silicene. In the MoS2-Si
structure, the primitive unit cell contains three atoms in comparison with two
atoms in silicene. The middle atom is bound to six atoms while the top and
bottom atoms have three neighbors. Clearly, the coordination is very different
from that in graphene-like structures where each atom is bound to three
neighboring atoms.

| $a$ | $t$ | $d$ | $E_{r}$
---|---|---|---|---
silicene | 3.90 | 0.49 | 2.30 | 0.032
MoS2-Si | 3.64 | 2.63 | 2.47 | 0
germanene | 4.07 | 0.74 | 2.47 | 0.140
MoS2-Ge | 3.90 | 2.88 | 2.67 | 0
Table 1: Values of the lattice constant ($a$ in Å), the thickness ($t$ in Å),
the bonding distance ($d$ in Å) and the relative energy ($E_{r}$ in eV/atom)
for silicene and MoS2-Si (upper part), as well as germanene and MoS2-Ge (lower
part).

Next, we compare the total energy per atom and structural parameters of the
phases under consideration as a function of $a$. Since the high-buckled forms
of silicene and germanene are unstable Cahangirov07 , we restrict the
investigations to lattice parameters around to the region of the LB phase. In
Fig. 2(a) is shown the relationship between the total energy per atom $E_{r}$
and the lattice constant $a$, for both LB silicene (empty squares) and MoS2-Si
(filled circles). The evolution of the buckling is plotted in Fig. 2(b). In
order to facillitate comparison between the two phases, for the MoS2
structure, half of the thickness is taken as the value of the buckling. With
the energy minimum occuring at a lattice constant of 3.90 Å, the LB silicene
prefers a buckling of 0.49 Å and the energy minimum occurs at $a=3.90$ Å.
These parameters compare well with values reported previously Cahangirov07 ;
Houssa10 ; Wang13 .

Interestingly, in equilibrium, the total energy of MoS2-Si is lower than that
of LB silicene, stabilized at a shorter lattice constant of $a=3.64$ Å. Due to
the stacking of three atoms, with 2.63 Å, the thickness $t$ of the MoS2-Si
layer is larger than that of silicene. Similar to the Si phases, MoS2-Ge
possess a lower total energy than LB germanene as well. The total energy and
lattice parameters of both MoS2-Si and MoS2-Ge are given in Tab. 1.

Figure 2: (a): Relative energy (in eV/atom) of silicene (empty squares) and
MoS2-Si (filled circles). (b): Buckling of silicene (empty squares) and
MoS2-Si(filled circles). For MoS2-Si, the buckling is defined as the distance
beween two planes, corresponding to half of the thickness. (c): Phonon
dispersion relations of MoS2-Si as obtained by the force-constant method.

With MoS2-Si being more stable than LB silicene, it is relevant to understand
its stability by investigating the Born-Oppenheimer surface. This can be done
by calculating the phonon frequencies in the harmonic approximation. In order
to do so, a dynamical matrix has been constructed by calculating real-space
force constants. Using a $12\times 12$ supercell, the atoms have been chosen
to be displaced by 0.02 Å out of the equilibrium positions. The phonon
dispersion relations of MoS2-Si are shown in Fig. 2(c). The frequencies are
overall lower than those of LB silicene which can be understood from the
elongated bond lengths that represent a weaker bonding. No branch with
imaginary frequencies is found. This suggests that the freestanding MoS2-Si is
a stable phase that is preferentially formed instead of the commonly studied
low-bucked silicene. For MoS2-Ge, on the other hand, the situation is not as
clear since a small amount of computed imaginary phonon frequencies maybe due
to either a possible instability of the structure or artifact derived from
numerical noise in the calculated forces. The following discussion will
therefore focus on MoS2-Si.

Figure 3: (a): Band structure of low-buckled silicene with a lattice parameter
of $a=3.90$ Å, (b): Band structure of MoS2-Si with $a=3.64$ Å. (c): Symmetry-
respecting Wannier functions of MoS2-Si related to the bands labeled 1-6 in
(b). The top, bottom and middle Si atoms are shown with different colors.

The electronic band structure of LB silicene and MoS2-Si are presented in
Figs. 3(a) and (b), respectively. Given that the bond length and the degree of
buckling are larger and that the lattice constants are shorter for MoS2-Si as
compared to silicene, it is surprising that the band structure is not far away
from that of silicene. Major differences observed around the Fermi energy
relate to the disapearance of the Dirac cone at the K point typical for the LB
silicene and to the appearance of two new bands, labeled 5 and 6 in Fig. 3(b).

To further understand similarities and differences between these two phases,
we construct the symmetry-respecting Wannier functions of MoS2-Si.
Technically, in order to do so, the commonly adopted procedure for maximizing
the localization of Wannier functions was not be performed Weng . The energy
window is chosen to allow for a reproduction of the six occupied bands,
labeled 1-6 in Fig. 3(b).

As shown in Fig. 3(c), the respective orbitals represented by Wannier
functions have contributions in different bands and adopt particular shapes.
The orbitals dominating the bands 1-3 originate from the $sp^{2}$ orbitals of
the middle Si atom. Interestingly, in order to accomodate bonding between the
top and bottom Si atoms, these orbitals have a nematic shape. For clarity,
only one of the three symmetric nematic orbitals is shown in Fig. 3(c). The
band dispersions related to the nematic orbitals resemble those of the
$\sigma$ bands of the LB silicene in Fig. 3(a). Without these nematic
orbitals, it is difficult to provide fully occupied $\sigma$ band dispersions
that are similar to the ones in LB silicene. Note that for the silicene
structure of the A-B stacking obtained directly from the equilibrium lattice
parameters of MoS2-Si, the $\sigma$ bands of silicene can only be partially
occupied.

Another interesting finding relates to orbitals with $p_{z}$ contributions: in
particular, the 4th orbital is derived from the $p_{z}$ orbitals of the top
and bottom Si atoms and the $sp^{2}$ orbitals of the middle Si atoms. At the K
point, the corresponding band crosses the 7th band which, however, does not
have any $p_{z}$ character. The crossing can therefore not be considered to be
derived from an original Dirac point of silicene. The 5th and 6th orbitals
form orbitals with $p_{z}$ symmetry having a node at the height of the middle
Si atom. As it can be recognized in Fig. 3(c), while the 5th orbital is mainly
derived from the $p_{z}$ and $s$ orbitals of the top and bottom Si atoms, the
6th orbital stems from the $sp^{2}$ orbitals of the top and bottom Si atoms
such displaying $p_{z}$ symmetry. Although the band dispersions of the $p_{z}$
orbitals bear some resemblance to the $\pi$ and $\pi^{*}$ bands of LB
silicene, no Dirac cones are formed since the orbital nature of the two
$p_{z}$ orbitals is essentially different.

The modification of the electronic properties is also evident from the plot of
the charge density shown in Fig. 4(a). The top and bottom atoms are bound to
the central atom via three of these nematic orbitals which allow for the
coordination of the central atoms with its six neighbors. For comparison, in
Figs. 4(b) and (c) are displayed the charge densities of silicene and of the
corresponding Si layer in the diamond structure, in which atoms are 3- or
4-fold coordinated, respectively. Note that as single layers, these two
structures have a higher total energy per atom than MoS2-Si. This suggests
that for two-dimensional silicon, $\sigma$ bonding of the planar $sp^{2}$ type
is preferred. This symmetry is well respected by the nematic orbital.

In order to allow for an even higher flexibility in the bonding, one can
imagine to twist the nematic orbitals to form a new A-B-C stacking structure.
Such an asymmetric bonding with respect to the planar $sp^{2}$ orbitals is by
180 meV per Si atom energetically unfavourable. In addition, the middle Si
atom of A-B-C stacking structure shares a bonding similar to that of the
6-fold coordinated Si atom in the $\beta$-tin Si phase that can only be
stabilized under a high pressure Betatin . The charge density of the A-B-C
phase is shown in Fig. 4(d).

Figure 4: Valence charge density for different structures composed of Si
atoms. (a): silicon MoS2-type layer, corresponding to A-B-A stacking. (b):
low-buckled silicene. (c): silicon layer with a diamond structure. (d): A-B-C
stacking structure for silicon.

To summarize, a new two-dimensional Si phase structurally equal to a single
MoS2 layer is identified. Within DFT, this phase is stable on the Born-
Oppenheimer surface in terms of the total energy and the phonon frequencies.
Instead of the commonly accepted bonding configurations in silicene or the
diamond structure of silicon that are related to a mixture of $sp^{2}$ and
$sp^{3}$ or pure $sp^{3}$ hybridization, respectively, this phase exhibits
nematic orbitals that allow $\sigma$ bonding with a 6-fold coordination for
the middle atoms in the MoS2 structure. With bond lengths longer than for
silicene, the three nematic orbitals exhibit $\sigma$ band dispersions similar
to those of LB silicene which represents a common feature of $\sigma$ bonding
in two-dimensional Si phases. On the other hand, the reconstructed $p_{z}$
orbitals, or super $p_{z}$ orbitals, are prominently different from those of
low-buckled silicene. Per Si atom, the MoS2-Si phase is lower in total energy
than the low-buckled silicene making it the the most stable two-dimensional Si
allotrope predicted so far. Our study demonstrates that Si atoms are capable
of forming diverse types of $\sigma$ bonds even under ambient pressure
conditions that are by themselves quite different from those formed by its
smaller and larger cousins carbon and germanium. Even more, the presence of an
extended $\pi$ electronic system with properties different to those in
silicene and graphene will lead to properties that still must be explored. In
a wider context, this finding does not only open new opportunities in the
engineering of novel nanostructures to be employed in future applications but
it also leads to intriguing questions for instance related to the Si-based
chemistry.

This work has been supported by the Strategic Programs for Innovative Research
(SPIRE), MEXT, the Computational Materials Science Initiative (CMSI), by
Materials Design through Computics: Complex Correlation and Non-Equilibrium
Dynamics, A Grant in Aid for Scientific Research on Innovative Areas, MEXT,
Japan, and by the Funding Program for Next Generation World-Leading
Researchers (GR046). The calculations have been performed using the Cray XC30
machine at Japan Advanced Institute of Science and Technology (JAIST).

## References

  * (1) A. K. Geim and K. S. Novoselov, Nature Materials 6, 183 (2007).
  * (2) K. Takeda and K.Shiraishi, Phys. Rev. B 50, 14917 (1994).
  * (3) G. G. Guzman-Verri and L. C. Lew Yan Voon, Phys. Rev. B 76, 075131 (2007).
  * (4) S. Cahangirov, M. Topsakal, E. Akturk, H. Sahin, and S. Ciraci, Phys. Rev. Lett. 102, 236804 (2007).
  * (5) M. Houssa, G. Pourtois, V. V. Afanas’ev, and A. Stesmans, Appl. Phys. Lett. 96, 082111 (2010).
  * (6) Y. Wang and Y. Ding, Solid State Communications 155, 6 (2013).
  * (7) P. De Padova, C. Quaresima, B. Olivieri, P. Perfetti, and G. Le Lay, Appl. Phys. Lett. 98, 081909 (2011).
  * (8) P. Wetzel, S. Saintenoy, C. Pirri, D. Bolmont, and G. Gewinner, Phys. Rev. B 50, 10886 (1994).
  * (9) P. Vogt, P. De Padova, C. Quaresima, J. Avila, E. Frantzeskakis, M.C. Asension, A. Resta, B. Ealet, and G. Le Lay, Phys. Rev. Lett. 108, 155501 (2012).
  * (10) C.L. Lin, R. Arafune, K. Kawahara, N. Tsukahara, E. Minamitani, Y. Kim, N. Takagi, and M. Kawai, Appl. Phys. Express 5, 045802 (2012).
  * (11) H. Jamgotchian, Y. Colignon, N. Hamzaoui, B. Ealet, J. Y. Hoarau, B. Aufray, and J. P. Bibeérian, J. Phys.: Condens. Matter 24, 172001 (2012).
  * (12) A. Fleurence, R. Friedlein, T. Ozaki, H. Kawai, Y. Wang, and Y. Yamada-Takamura, Phys. Rev. Lett. 108, 245501 (2012).
  * (13) L. Meng, Y. Wang, L. Zhang, S. Du, R. Wu, L. Li, Y. Zhang, G. Li, H. Zhou, W. A. Hofer, H.-G. Gao, Nano Lett. 13, 685 (2013).
  * (14) L. Chen, H. Li, B. Feng, Z. Ding, J. Qiu, P. Cheng, K. Wu, and S. Meng, Phys. Rev. Lett. 109, 056804 (2012).
  * (15) L. Chen, H. Li, B. Feng, Z. Ding, J. Qiu, P. Cheng, K. Wu, and S. Meng, Phys. Rev. Lett. 110, 085504 (2013).
  * (16) C.C. Lee, A. Fleurence, R. Friedlein, Y. Yamada-Takamura, and T. Ozaki, Phys. Rev. B 88, 165404 (2013).
  * (17) D. Kaltsas, and L. Tsetseris, Phys. Chem. Chem. Phys. 15, 9710 (2013).
  * (18) V. Ongun Özçelik and S. Ciraci, J. Phys. Chem. C 117, 26305 (2013).
  * (19) T. Ozaki, Phys. Rev. B 67, 155108 (2003).
  * (20) The DFT calculations within a generalized gradient approximation [J.P. Perdew, K. Burke, and M. Ernzerhof, Phys. Rev. Lett. 77, 3865 (1996)] have been performed using the OpenMX code [T. Ozaki et al, http://www.openmx-square.org/] which is based on norm-conserving pseudopotentials generated with multi reference energies and optimized pseudoatomic basis functions. The cut-off radius of 7 Bohr has been chosen for all the basis functions. For Si and Ge atoms, $s2p2d1$ configurations have been adopted. The spin-orbit coupling has not been considered in our calculation. In order to avoid interactions, the distance between supercells is larger than 10 Å in the z-direction (direction perpendicular to the plane of layers). The k-mesh has been set to 12x12x1 for the primitive cell. The structures have been fully optimized until the maximum force became less than $10^{-4}$ Hartree/Bohr. All the geometric structures are plotted using the XCrySDen software. The energy curves for the phases shown in Fig. 2 have also been confirmed by calculations with the first principles code WIEN2k. [P. Blaha, K. Schwarz, G. K. H. Madsen, D. Kvasnicka, and J. Luits, Wien2k, An Augmented Plane Wave+Local Orbitals Program for Calculating Crystal Properties (K. Schwarz, Vienna, Austria, 2001)]
  * (21) B. Radisavljevic, A. Radenovic, J. Brivio, V. Giacometti, and A. Kis, Nature Nano. 6, 147 (2011).
  * (22) N. Marzari and D. Vanderbilt, Phys. Rev. B 56, 12847 (1997).
  * (23) H. Weng, T. Ozaki, and K. Terakura, Phys. Rev. B 79, 235118 (2009).
  * (24) T.-M. Chuang, M. P. Allan, J. Lee, Y. Xie, N. Ni, S. L. Bud’ko, G. S. Boebinger, P. C. Canfield, J. C. Davis, Science 327, 181 (2010).
  * (25) A. Mujica, A. Rubio, A. Munoz, and R. J. Needs, Rev. Mod. Phys. 75, 863 (2003).


# Lengths matters, periodically.
(The movie)
Fluid Dynamics Video

M. C. Renoult, S. Ferjani, C. Rosenblatt, P. Carles

Fast Lab., UPMC/Paris 6, FRANCE
Physics Dept., CWRU, Cleveland, OH,USA

The video shows the development of the Rayleigh-Taylor Instability between two
immiscible fluids for four distinct initial single-mode perturbations. To
obtain such controlled initial conditions, a Magnetic Levitation technique is
used (see [1]). The principle is explained in the first part of the animation.
A homogeneous magnetic force is produced in opposition to gravity and allows
the stabilization of the dense fluid (paramagnetic aqueous mixture) above the
less dense fluid (hexadecane). In addition, segments of magnetically permeable
wires are placed on the outside of the cell in a precise configuration -
predicted numerically - to achieve almost any desired initial condition (see
[2]). The initial interface thus is no longer flat and the experiment is
started by turning off the magnetic field and allowing the denser fluid to
fall under gravity. Here the wires are periodically aligned along a row, a few
mm above the initial interface position, producing a small amplitude single
mode perturbation of the same wavelength as the wires (but too small to
image). The video presents the Rayleigh-Taylor Instability for four decreasing
wavelengths (15 mm, 12.5, 10 mm and 7.5 mm). In each case, one observes the
different stages of development of the Rayleigh-Taylor Instability from the
early time linear behavior to the late mixing flow. The second instability
also can be observed.

The link for the video is: Video.

## References

  * [1] Mahajan M.P., Tsige M., Taylor P.L. and Rosenblatt C., Phys. Fluids 10, 2208 (1998)
  * [2] Huang Z., De Luca A., Atherton T.J., Bird M., Rosenblatt C. and Carles P., Phys. Rev. Lett. 204502 (2007)


# DeepPavlov at SemEval-2024 Task 8: Leveraging Transfer Learning for
Detecting Boundaries of Machine-Generated Texts

Anastasia Voznyuk  Vasily Konovalov
Moscow Institute of Physics and Technology
{vozniuk.ae<EMAIL_ADDRESS>

###### Abstract

The Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated
Text Detection shared task in the SemEval-2024 competition aims to tackle the
problem of misusing collaborative human-AI writing. Although there are a lot
of existing detectors of AI content, they are often designed to give a binary
answer and thus may not be suitable for more nuanced problem of finding the
boundaries between human-written and machine-generated texts, while hybrid
human-AI writing becomes more and more popular. In this paper, we address the
boundary detection problem. Particularly, we present a pipeline for augmenting
data for supervised fine-tuning of DeBERTaV3. We receive new best MAE score,
according to the leaderboard of the competition, with this pipeline.

DeepPavlov at SemEval-2024 Task 8: Leveraging Transfer Learning for Detecting
Boundaries of Machine-Generated Texts

Anastasia Voznyuk and Vasily Konovalov Moscow Institute of Physics and
Technology {vozniuk.ae<EMAIL_ADDRESS>

##  1 Introduction

Recently, there has been a rapid development of auto-regressive language
models, for example, GPT-3 Brown et al. (2020), GPT-4 OpenAI (2023), and
LLaMA2 Touvron et al. (2023). These models are trained on enormous amounts of
data and are able to produce coherent texts that can be indistinguishable from
human-written texts Dugan et al. (2022).

The SemEval-2024 Task 8 competition suggests to tackle the problem of
detecting machine-generated texts. This problem has become more relevant
recently due to the release of ChatGPT111https://openai.com/blog/chatgpt, a
model by OpenAI that simplified the access to the large language models (LLM)
and their usage. For example, LLM can be maliciously used to generate fake
news Zellers et al. (2019). There are also some concerns raised among
scientists Ma et al. (2023) and educators Zeng et al. (2023) that the usage of
LLMs will devalue the process of learning and research.

The commonly used approach to formulate the task of detecting machine-
generated texts is a binary classification task Jawahar et al. (2020). In this
case, a text can be attributed to either a human or a LLM. Otherwise, the task
can be formulated as a multiclass classification or an authorship attribution
task Uchendu et al. (2020), where it is needed to determine which one of the
$k$ authors is the real author of the given text. Finally, the trend toward
human-AI collaborative writing is rising, which highlights the importance of
the boundary detection task. In this setup, text contains consecutive chunks
of different authorship, and it is required to detect where the boundaries
between chunks lie and who is the author of every chunk. Due to its
complexity, it is usually assumed the text has a human-written prefix and the
rest of the text is AI-generated Dugan et al. (2022); Cutler et al. (2021);
Kushnareva et al. (2023).
Our main contributions are three-fold:

  1. 1.

We receive the new best MAE score on the task of detecting the boundary
between human-written and machine-generated parts of the text.

  2. 2.

We present a new simple yet effective pipeline of augmenting data for the task
of boundary detection, which allows us to get more data for training and
improve the results of fine-tuning large language models.

  3. 3.

We compare the performance of several fine-tuned models with different
architectures on various amounts of training data.

Additionally, we’ve made the code of augmentation publicly
available.222https://github.com/natriistorm/semeval2024-boundary-detection

##  2 Related Work

Most of approaches Jawahar et al. (2020) for machine-generated text detection
are based on calculating linguistic Fröhling and Zubiaga (2021), stylometric,
and statistical features, as well as on using classical machine learning
methods like logistic regression, random forest, and gradient boosting as
classifiers. Among commonly used features are word and n-gram frequencies
Manjavacas et al. (2017), and tf-idf Solaiman et al. (2019).

An alternative strategy is to use zero-shot techniques based on the internal
metrics of the texts. For example, token-wise log probability can be evaluated
by models like GROVER Zellers et al. (2019) or GPT-2 Solaiman et al. (2019). A
probability threshold is established to distinguish writings produced by
machines from those written by humans. Moreover, rank Gehrmann et al. (2019)
or log-rank Mitchell et al. (2023) can be calculated for each token and
compared for consistency with the prior context.

It’s shown by Ippolito et al. (2020) that feature-based methods are inferior
to methods based on using encoders of pretrained language models like BERT
Devlin et al. (2019) as a basis for fine-tuning on the selected domain.
Representations from auto-regressive language models can be used as input for
the classification head. Such transformer-based methods require supervised
detection examples for further training. Among the models commonly used for
fine-tuning are RoBERTa Liu et al. (2019) and XLNet Yang et al. (2019). Fine-
tuned RoBERTa has shown the state-of-the-art results on the problems of
authorship attribution Uchendu et al. (2021).

To tackle the problem of mixed human-machine writing, Dugan et al. (2022)
introduces the Real Or Fake Text (RoFT) tool, where humans were asked to
detect the sentence where the text transitions from human-written text to
machine-generated text. One of the possible formulations of this task is
multilabel classification Cutler et al. (2021), where the boundary detector
needs to determine the first generated sentence, and the number of this
sentence is considered the label of the text. In that work, each sentence is
processed separately with shallow classification and regression models based
on RoBERTa and SRoBERTa Reimers and Gurevych (2019). That solution perform
well in an in-domain setup, but is limited in an out-of-domain setup.

Any solution for tasks about detecting machine-generated texts should be
robust to domain change. The organisers of the SemEval-2024 Task 8 competition
claimed to have added new domains to the test set for testing the robustness
of participants’ solutions. There are several works on performance of
detecting methods on out-of-domain setup, such as Kushnareva et al. (2023) and
Zeng et al. (2023). Kushnareva et al. (2023) conclude that perplexity-based
and topological features appear to be helpful in case of domain shift.

##  3 Data and Task Description

###  3.1 Task Description

The Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated
Text Detection Wang et al. (2024) is focused on challenging detectors of
machine-generated texts. The dataset, provided by organisers, consists of 3
parts:

  1. 1.

texts of different authorship for subtask A and subtask B;

  2. 2.

texts with collaborative human-AI writing for subtask C.

This paper focuses only on subtask C, suggesting a solution to differentiate a
human-written prefix from the rest of the AI-generated text. The texts for
this subtask are generated in the following way: the language model should
continue the human-written text, which is given as a prompt. Several examples
of texts are presented in Appendix A. The designated evaluation metric for
this task is Mean Absolute Error (MAE), which quantifies the absolute distance
between the predicted word and the actual word where the switch of authorship
between human and LLM occurs.

###  3.2 Data Description

[Distribution of lengths of tokenized texts] [Distribution of boundary
subtoken positions]

Figure 1: Statistics of the texts in the datasets

The dataset for this task is derived from the M4 Wang et al. (2023) dataset,
which contains texts of various domains, various languages, and generators.
The authors show that current detectors tend to misclassify machine-generated
texts as human-written if they’re given a text from a different domain. The
texts in the train and validation datasets are generated from scientific paper
reviews from PeerRead Kang et al. (2018). The test set partially consists of
texts generated from PeerRead. In order to check the robustness of the
solutions to domain shift, texts from Outfox, a dataset of LLM-generated
student essays Koike et al. (2023), are added to the dataset. All data sets
contain English texts only.

###  3.3 Data Analysis

The distribution of lengths of texts in train and test data sets, tokenized by
DeBERTa-V3-base models, is shown in Figure 1. The majority of the texts in the
test set are shorter in length, but there are several texts that exceed 1,000
subtokens. The distribution of positions of a boundary subtoken in texts is
shown in Figure 1. A boundary subtoken is the first subtoken of a boundary
word. In both datasets, there is a distinguishable disproportion in the
position of a boundary subtoken, as most of them have the boundary subtoken
within the first 200 subtokens. Thus, a model trained only on this data will
give limited results when encountering longer texts with longer human prefixes
in them.

##  4 Proposed Method

The provided train dataset consists only of 3,649 texts. It’s well known that
an abundance of in-domain training data is crucial for classifier performance
Konovalov et al. (2016). However, during the competition period, it was
prohibited to use any external data for training, and thus we were limited to
working only with the provided dataset. In this case, getting more training
data with some kind of augmentation plays a crucial role. We designed an
augmentation pipeline and ran all our experiments on two sets of data: the one
provided by the organisers of the competition, described in Section 3, and our
augmented data.

###  4.1 Data Augmentation

The general idea of our augmentation pipeline is to split the text into
distinct sentences and take several consequent sentences from the text with
authorship change. It will make new texts coherent and will not mislead models
during training Ostyakova et al. (2023). In addition, to be useful for
training, each sequence should contain a sentence with an authorship change.

Another nuance about the augmentation process is the need to correctly
determine the boundary label. The boundary label is calculated as a number of
whitespace-split words in the human-written prefix. However, the initial
dataset contains texts where a pair or even a sequence of words is split only
by line breaks or punctuation symbols. Such a sequence of words should be
considered as one word when calculating the boundary label. Thus, we have to
pay a lot of attention to whitespace characters during augmentation and do not
mistakenly append new whitespace characters between words.

We preprocess each text in the dataset for augmentation in the following way:

  1. 1.

Split the text into sentences by punctuation symbols.

  2. 2.

Split the sentences themselves by whitespaces into lists of whitespace-split
words.

  3. 3.

Compare the list of whitespace-split words from previous step with the list of
whitespace-split words obtained by splitting the text itself. In case of
discrepancy, fix it, depending on its type.

See example of preprocessing for the text from train set in Appendix D.

In the third step of preprocessing, there are two main types of discrepancies:
lost whitespace characters and words sequences, originally separated by line
breaks only that were split during sentence split process. The former is
solved by inserting the missing whitespace characters, while for the latter we
merge the split words into one sequence. The last step of preprocessing is
crucial and skipping it will result in the incorrect calculation of the
boundary label for augmented text, as the label directly depends on the number
of whitespace characters in the text.

After preprocessing, we take a number of consecutive sentences to the left and
to the right from the boundary sentence, combine them in a text and determine
the label of the boundary word in this new text.

###  4.2 Model Comparison

We used only the transfer learning approach, where a pretrained transformer-
based model is fine-tuned on our task in a supervised way. We would like the
model to be able to work with long enough sequences because we want the whole
human-written prefix to fit in the encoder. Thus, we’ve determined three
models that showed good results on the task of machine-generated text
detection:

  1. 1.

RoBERTa Liu et al. (2019) has shown good performance in both tasks of boundary
detection Kushnareva et al. (2023) and machine-generated text detection Macko
et al. (2023).

  2. 2.

Longformer Beltagy et al. (2020), which was suggested as a baseline by
organisers of the task. This model is based on pretrained RoBERTa with novel
attention mechanism with a sliding window to long sequences.

  3. 3.

DeBERTa He et al. (2021b), which is the state-of-the-art model for machine-
generated text detection Macko et al. (2023). It overcomes the BERT and
RoBERTa by introducing a disentangled attention for encoding the position and
content of each token separately into two vectors. We decided to test it in
the boundary detection task to understand whether it outperform RoBERTa. In
our experiments, we fine-tuned DeBERTaV3 He et al. (2021a) which is an
enhanced version of DeBERTa.

All three models are fine-tuned for token classification task. For each token,
models predict the probability of being a boundary token and output the most
probable token.

##  5 Experimental Setup

We have used pretrained longformer-4096-base and longformer-4096-large with
default hyperparameters to fine-tune Longformer. For experiments with RoBERTa,
we have chosen two models: roberta-base and roberta-large. The models were
fine-tuned with the set of custom hyperparameters, taken from the original
paper Liu et al. (2019). Finally, for experiments with DeBERTa, we have also
chosen two models: deberta-v3-base and deberta-v3-large. The models were fine-
tuned with the set of custom hyperparameters taken from He et al. (2021a). All
custom hyperparameters are listed in Appendix B.

For all models we set the maximum length of context in tokenizer equal to 512
as there are only few text items in both train and test set with tokenized
text length greater than 512. Additionaly, we’ve used the early stopping
method for all of our experiments to get rid of epochs dependency.

Model | dev | test
---|---|---
RoBERTa-base | 9.04 \ 5.78 | 31.56 \ 30.71
RoBERTa-large | 6.72 \ 4.18 | 25.25 \ 20.66
longformer-base | 5.10 \ 5.67 | 23.16 \ 22.94
longformer-large | 4.54 \ 4.40 | 22.97 \ 20.33
DeBERTaV3-base | 3.66 \ 3.15 | 16.12 \ 13.98
DeBERTaV3-large | 2.38 \ 2.54 | 15.16 \ 13.38
Top-1 Submission | - | 15.68
Table 1: MAE on original \ augmented dataset and comparison with Top-1
submission on the leaderboard. Longformer-base is suggested as a baseline
solution by organisers of competition.

##  6 Results and Discussion

###  6.1 Main Results

In Tables 1, we compare MAE scores of different models from Section 4.2. There
are experiments with models fine-tuned on the original dataset provided by
organisers and on the extended dataset with both augmented and original texts.

All models perform better when they are fine-tuned on the extended dataset
rather than only on original texts. It clearly shows that even such a simple
data augmentation provides a significant boost in performance.

If we compare the results among the models, we will clearly see the dominance
of DeBERTaV3 models. For both setups DeBERTaV3-large has shown the best
performance and the lowest MAE score on the validation and test datasets. On
the setup with the extended dataset, DeBERTaV3-large gets new best MAE score
for the competition, which is equal to 13.375. It improves MAE score of the
top-1 submission by more than 2 points, from 15.683 to 13.38.

###  6.2 Discussion

Results in Section 6.1 show the importance of both the variety of the data in
the dataset and a pretrained model. Leveraging augmented training data
significantly increases performance on the task, because it introduces variety
in lengths of texts and boundary token positions while preserving the
coherence of the texts. We believe that to be the reason why the models
perform better when they are fine-tuned on the dataset with augmented data
rather than on the original dataset.

Apart from various data, it is also important for the pretrained model to have
great generalization capabilities. DeBERTaV3-large has better generalization
capabilities than other tested models He et al. (2021b). The advanced
architecture of DeBERTaV3 helps to significantly improve the MAE score in
comparison with both RoBERTa and Longformer.

For all three models, large vesion of each model outperforms base version on
both the original and the extended dataset. The reason to this is, greater
number of trainable parameters allows models to generalize on training set
better.

###  6.3 Error Analysis

We manually inspected texts from the test set on which the best-performing
model, DeBERTaV3-large, made serious mistakes. We’ve limited our inspection
set to the texts where the distance between the predicted label and the true
label was more than 100 tokens. Thus, we got 276 texts. Only 75 out of these
276 texts were from PeerReview domain, so model made most of its mistakes on
the Outfox domain, texts from which are not present in the train set.

These two domains presented in the test set are very different: they vary in
the style of formatting, punctuation, and text structuring. The second domain
of LLM-generated student essays have a lot of spelling and punctuation
problems and it may confuse the models, as they trained on more formal and
literate texts. It would be interesting to evaluate models on each of these
domains separately. However, because we do not have domain classification
lables in the test set, it is not yet feasible.

###  6.4 Anomalies in Texts

In the majority of texts from the original dataset, the model generates a
coherent continuation of the human-written prefix, and it may be hard for a
human to guess the boundary word without knowing it. However, there are a
number of texts in the data sets that have some flaws in the generated parts.

There are texts in which LLM hallucinated. It either repeated the human prefix
or went into a loop where it generated excessive lists with the same
beginning. See example of it in Appendix A and in Appendix C. Such
hallucinations can be an immediate hint for detector model to put the label
boundary near this anomaly. Also, sometimes machine-generated text can have
some distinguished features that imply the artificial nature of a particular
part of the text. For example, in a number of cases the model begins the
generation with the """ (three double quotation marks). It may also be a hint
for the detector. A list of other common features we’ve encountered while
examining the test set is provided in Appendix C.

##  7 Conclusion

In this paper we describe the system submitted for SemEval2024-Task 8, the
subtask dedicated to hybrid human-machine writing detection. We present a
simple yet effective augmentation pipeline. We explore how adding this
pipeline to the process of fine-tuning can significantly increase the
performance on the task, and provide an analysis of performance of various
models with and without our augmentation pipeline. The best model, which is
DeBERTaV3-large fine-tuned on a large set of augmented data, receives a new
best score according to the leaderboard of the competition. Other fine-tuned
models achieve competitive results, ranking in the upper half of the
leaderboard and beating the organisers’ baseline. As the provided data was
limited to English language only, future work might include training
multilingual boundary detection solution by mixing training data of different
languages and using a multilingual encoder Chizhikova et al. (2023). Such a
system can be used for hybrid AI-writing detection as a standalone solution or
can be integrated into existing NLP frameworks like DeepPavlov Burtsev et al.
(2018).

## References

  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , NIPS’20, Red Hook, NY, USA. Curran Associates Inc.
  * Burtsev et al. (2018) Mikhail Burtsev, Alexander Seliverstov, Rafael Airapetyan, Mikhail Arkhipov, Dilyara Baymurzina, Nickolay Bushkov, Olga Gureenkova, Taras Khakhulin, Yuri Kuratov, Denis Kuznetsov, and Vasily Konovalov. 2018.  Deeppavlov: An open source library for conversational ai.  In _NIPS_.
  * Chizhikova et al. (2023) Anastasia Chizhikova, Vasily Konovalov, and Mikhail Burtsev. 2023.  Multilingual case-insensitive named entity recognition.  In _Advances in Neural Computation, Machine Learning, and Cognitive Research VI_ , pages 448–454, Cham. Springer International Publishing.
  * Cutler et al. (2021) Joseph W. Cutler, Liam Dugan, Shreya Havaldar, and Adam Stein. 2021.  Automatic detection of hybrid human-machine text boundaries.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dugan et al. (2022) Liam Dugan, Daphne Ippolito, Arun Kirubarajan, Sherry Shi, and Chris Callison-Burch. 2022.  Real or fake text?: Investigating human ability to detect boundaries between human-written and machine-generated text.
  * Fröhling and Zubiaga (2021) Leon Fröhling and Arkaitz Zubiaga. 2021.  Feature-based detection of automated language models: tackling gpt-2, gpt-3 and grover.  _PeerJ Computer Science_ , 7.
  * Gehrmann et al. (2019) Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. 2019.  GLTR: Statistical detection and visualization of generated text.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 111–116, Florence, Italy. Association for Computational Linguistics.
  * He et al. (2021a) Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021a.  Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing.
  * He et al. (2021b) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021b.  Deberta: Decoding-enhanced bert with disentangled attention.
  * Ippolito et al. (2020) Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. 2020.  Automatic detection of generated text is easiest when humans are fooled.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1808–1822, Online. Association for Computational Linguistics.
  * Jawahar et al. (2020) Ganesh Jawahar, Muhammad Abdul-Mageed, and Laks Lakshmanan, V.S. 2020.  Automatic detection of machine generated text: A critical survey.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 2296–2309, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Kang et al. (2018) Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. 2018.  A dataset of peer reviews (PeerRead): Collection, insights and NLP applications.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1647–1661, New Orleans, Louisiana. Association for Computational Linguistics.
  * Koike et al. (2023) Ryuto Koike, Masahiro Kaneko, and Naoaki Okazaki. 2023.  Outfox: Llm-generated essay detection through in-context learning with adversarially generated examples.  In _AAAI Conference on Artificial Intelligence_.
  * Konovalov et al. (2016) Vasily Konovalov, Oren Melamud, Ron Artstein, and Ido Dagan. 2016.  Collecting Better Training Data using Biased Agent Policies in Negotiation Dialogues.  In _Proceedings of WOCHAT, the Second Workshop on Chatbots and Conversational Agent Technologies_ , Los Angeles. Zerotype.
  * Kushnareva et al. (2023) Laida Kushnareva, Tatiana Gaintseva, German Magai, Serguei Barannikov, Dmitry Abulkhanov, Kristian Kuznetsov, Irina Piontkovskaya, and Sergey Nikolenko. 2023.  Artificial text boundary detection with topological data analysis and sliding window techniques.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.
  * Ma et al. (2023) Yongqiang Ma, Jiawei Liu, Fan Yi, Qikai Cheng, Yong Huang, Wei Lu, and Xiaozhong Liu. 2023.  Ai vs. human – differentiation analysis of scientific content generation.
  * Macko et al. (2023) Dominik Macko, Robert Moro, Adaku Uchendu, Jason Lucas, Michiharu Yamashita, Matúš Pikuliak, Ivan Srba, Thai Le, Dongwon Lee, Jakub Simko, and Maria Bielikova. 2023.  MULTITuDE: Large-scale multilingual machine-generated text detection benchmark.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 9960–9987, Singapore. Association for Computational Linguistics.
  * Manjavacas et al. (2017) Enrique Manjavacas, Jeroen De Gussem, Walter Daelemans, and Mike Kestemont. 2017.  Assessing the stylistic properties of neurally generated text in authorship attribution.  In _Proceedings of the Workshop on Stylistic Variation_ , pages 116–125, Copenhagen, Denmark. Association for Computational Linguistics.
  * Mitchell et al. (2023) Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. 2023.  Detectgpt: zero-shot machine-generated text detection using probability curvature.  In _Proceedings of the 40th International Conference on Machine Learning_ , ICML’23.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Ostyakova et al. (2023) Lidiia Ostyakova, Veronika Smilga, Kseniia Petukhova, Maria Molchanova, and Daniel Kornev. 2023.  ChatGPT vs. crowdsourcing vs. experts: Annotating open-domain conversations with speech functions.  In _Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 242–254, Prague, Czechia. Association for Computational Linguistics.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Conference on Empirical Methods in Natural Language Processing_.
  * Solaiman et al. (2019) Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, Miles McCain, Alex Newhouse, Jason Blazakis, Kris McGuffie, and Jasmine Wang. 2019.  Release strategies and the social impacts of language models.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _ArXiv_ , abs/2307.09288.
  * Uchendu et al. (2020) Adaku Uchendu, Thai Le, Kai Shu, and Dongwon Lee. 2020.  Authorship attribution for neural text generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8384–8395, Online. Association for Computational Linguistics.
  * Uchendu et al. (2021) Adaku Uchendu, Zeyu Ma, Thai Le, Rui Zhang, and Dongwon Lee. 2021.  TURINGBENCH: A benchmark environment for Turing test in the age of neural text generation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 2001–2016, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Wang et al. (2023) Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Chenxi Whitehouse, Osama Mohammed Afzal, Tarek Mahmoud, Alham Fikri Aji, and Preslav Nakov. 2023.  M4: Multi-generator, multi-domain, and multi-lingual black-box machine-generated text detection.
  * Wang et al. (2024) Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Chenxi Whitehouse, Osama Mohammed Afzal, Tarek Mahmoud, Giovanni Puccetti, Thomas Arnold, Alham Fikri Aji, Nizar Habash, Iryna Gurevych, and Preslav Nakov. 2024.  Semeval-2024 task 8: Multigenerator, multidomain, and multilingual black-box machine-generated text detection.  In _Proceedings of the 18th International Workshop on Semantic Evaluation_ , SemEval 2024, Mexico, Mexico.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Neural Information Processing Systems_.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019.  _Defending against neural fake news_. Curran Associates Inc., Red Hook, NY, USA.
  * Zeng et al. (2023) Zijie Zeng, Lele Sha, Yuheng Li, Kaixun Yang, Dragan Gašević, and Guanliang Chen. 2023.  Towards automatic boundary detection for human-ai collaborative hybrid essay in education.

##  Appendix A Examples of Texts

Table 2 contains three examples of how authorship change occurs in the texts
from the train set. While the first text contains no signs of a flawed
generation, the second and third texts have some flaws. In the second text,
the model begins to generate from the capital letter, and its generation is
incoherent with the human-written prefix. In the third text, the model starts
to repeat the end of the human part, which is a glaring sign of machine-
generated text.

Model
---
I noticed that in Figure 2, the two quantization factors for quantized layers
are missing labels. It would be helpful for the reader to understand which
layers are being quantized in the figure .
Hi Authors, You seem to have submitted two of the same paper? Pls advise Could
you please clarify if this is a mistake or if there are any differences
between the two submitted papers?
There has been prior work on semi-supervised GAN, though this paper is the
first context conditional variant. The novelty of the approach was the novelty
of the approach was leveraging in-painting using an adversarial loss to gene-
rate contextually relevant images.
Table 2: Examples of texts from train set with different quality of LLM
generation and with highlighted human prefix

##  Appendix B Hyperparameters

For fine-tuning DeBERTaV3 we use hyperparameters, listed by model authors in
He et al. (2021a). Table 3 lists these hyperparameters.

Hyperparameters | Large | Base
---|---|---
Optimizer | AdamW | AdamW
Adam $\beta_{1}$, $\beta_{2}$ | 0.9, 0.999 | 0.9, 0.999
Adam $\epsilon$ | 1e-6 | 1e-6
Warm-up step | 50 | 50
Batch size | 4 | 32
Learning rate (LR) | 5e-6 | 2e-6
Learning rate decay | Linear | Linear
Weight decay | 0.01 | 0.01
Gradient clipping | 1.0 | 1.0
Table 3: Hyperparameters for fine-tuning DebertaV3-large and DebertaV3-base

Table 4 contains hyperparameters for fine-tuning RoBERTa, also taken from
original paper Liu et al. (2019).

Hyperparameters | values
---|---
Optimizer | AdamW
Warm-up steps | 50
Batch size | 16
Learning rate (LR) | 5e-6
Learning rate decay | Linear
Weight decay | 0.01
Table 4: Hyperparameters for fine-tuning RoBERTa

##  Appendix C Examples For Error Analysis

See Table 5.

Anomaly | Text Id | Example
---|---|---
Excessive repetition | 613 |  …praying for. No more traffic jams, no more parking nightmares, no more car payments, no more insurance, no more maintenance, no more oil changes, no more tire rotations…
Extremely long lists of items | 8854 |  …nice approach to """ \+ learning skills \+ learning skills in a sample efficient way \+ learning skills in an interpretable way \+ learning skills that can be used on downstream tasks \+ learning skills that are transferable between domains
JSON-structured hallucinations | 5358 |  …Summary of revisions: $\ast\ast\ast$ """, "title": "Diet Networks: Thin Parameters for Fat Genomics", "abstract": "Learning tasks such as…
Bizarre formatting | 5639 |  …Below are my comments: (1) """ The first sentence of the abstract is too long. It should be divided into two sentences (2) """
Table 5: Table with some frequent anomalies in the generated part of texts
from test set. The highlighted part is human-written prefix.

##  Appendix D Augmenting Pipeline Scheme

See Figure 2.

Figure 2: Preprocessing for Augmentation Pipeline