11institutetext: III. Physikalisches Institut, RWTH Aachen University, D-52056
Aachen, Germany 22institutetext: Department of Physics, University of
Adelaide, Adelaide, 5005, Australia 33institutetext: Dept. of Physics and
Astronomy, University of Alaska Anchorage, 3211 Providence Dr., Anchorage, AK
99508, USA 44institutetext: Dept. of Physics, University of Texas at
Arlington, 502 Yates St., Science Hall Rm 108, Box 19059, Arlington, TX 76019,
USA 55institutetext: CTSPS, Clark-Atlanta University, Atlanta, GA 30314, USA
66institutetext: School of Physics and Center for Relativistic Astrophysics,
Georgia Institute of Technology, Atlanta, GA 30332, USA 77institutetext: Dept.
of Physics, Southern University, Baton Rouge, LA 70813, USA 88institutetext:
Dept. of Physics, University of California, Berkeley, CA 94720, USA
99institutetext: Lawrence Berkeley National Laboratory, Berkeley, CA 94720,
USA 1010institutetext: Institut für Physik, Humboldt-Universität zu Berlin,
D-12489 Berlin, Germany 1111institutetext: Fakultät für Physik & Astronomie,
Ruhr-Universität Bochum, D-44780 Bochum, Germany 1212institutetext: Université
Libre de Bruxelles, Science Faculty CP230, B-1050 Brussels, Belgium
1313institutetext: Vrije Universiteit Brussel (VUB), Dienst ELEM, B-1050
Brussels, Belgium 1414institutetext: Department of Physics and Laboratory for
Particle Physics and Cosmology, Harvard University, Cambridge, MA 02138, USA
1515institutetext: Dept. of Physics, Massachusetts Institute of Technology,
Cambridge, MA 02139, USA 1616institutetext: Dept. of Physics and The
International Center for Hadron Astrophysics, Chiba University, Chiba
263-8522, Japan 1717institutetext: Department of Physics, Loyola University
Chicago, Chicago, IL 60660, USA 1818institutetext: Dept. of Physics and
Astronomy, University of Canterbury, Private Bag 4800, Christchurch, New
Zealand 1919institutetext: Dept. of Physics, University of Maryland, College
Park, MD 20742, USA 2020institutetext: Dept. of Astronomy, Ohio State
University, Columbus, OH 43210, USA 2121institutetext: Dept. of Physics and
Center for Cosmology and Astro-Particle Physics, Ohio State University,
Columbus, OH 43210, USA 2222institutetext: Niels Bohr Institute, University of
Copenhagen, DK-2100 Copenhagen, Denmark 2323institutetext: Dept. of Physics,
TU Dortmund University, D-44221 Dortmund, Germany 2424institutetext: Dept. of
Physics and Astronomy, Michigan State University, East Lansing, MI 48824, USA
2525institutetext: Dept. of Physics, University of Alberta, Edmonton, Alberta,
T6G 2E1, Canada 2626institutetext: Erlangen Centre for Astroparticle Physics,
Friedrich-Alexander-Universität Erlangen-Nürnberg, D-91058 Erlangen, Germany
2727institutetext: Physik-department, Technische Universität München, D-85748
Garching, Germany 2828institutetext: Département de physique nucléaire et
corpusculaire, Université de Genève, CH-1211 Genève, Switzerland
2929institutetext: Dept. of Physics and Astronomy, University of Gent, B-9000
Gent, Belgium 3030institutetext: Dept. of Physics and Astronomy, University of
California, Irvine, CA 92697, USA 3131institutetext: Karlsruhe Institute of
Technology, Institute for Astroparticle Physics, D-76021 Karlsruhe, Germany
3232institutetext: Karlsruhe Institute of Technology, Institute of
Experimental Particle Physics, D-76021 Karlsruhe, Germany 3333institutetext:
Dept. of Physics, Engineering Physics, and Astronomy, Queen’s University,
Kingston, ON K7L 3N6, Canada 3434institutetext: Department of Physics &
Astronomy, University of Nevada, Las Vegas, NV 89154, USA 3535institutetext:
Nevada Center for Astrophysics, University of Nevada, Las Vegas, NV 89154, USA
3636institutetext: Dept. of Physics and Astronomy, University of Kansas,
Lawrence, KS 66045, USA 3737institutetext: Centre for Cosmology, Particle
Physics and Phenomenology - CP3, Université catholique de Louvain, Louvain-la-
Neuve, Belgium 3838institutetext: Department of Physics, Mercer University,
Macon, GA 31207-0001, USA 3939institutetext: Dept. of Astronomy, University of
Wisconsin—Madison, Madison, WI 53706, USA 4040institutetext: Dept. of Physics
and Wisconsin IceCube Particle Astrophysics Center, University of
Wisconsin—Madison, Madison, WI 53706, USA 4141institutetext: Institute of
Physics, University of Mainz, Staudinger Weg 7, D-55099 Mainz, Germany
4242institutetext: Department of Physics, Marquette University, Milwaukee, WI
53201, USA 4343institutetext: Institut für Kernphysik, Westfälische Wilhelms-
Universität Münster, D-48149 Münster, Germany 4444institutetext: Bartol
Research Institute and Dept. of Physics and Astronomy, University of Delaware,
Newark, DE 19716, USA 4545institutetext: Dept. of Physics, Yale University,
New Haven, CT 06520, USA 4646institutetext: Columbia Astrophysics and Nevis
Laboratories, Columbia University, New York, NY 10027, USA 4747institutetext:
Dept. of Physics, University of Oxford, Parks Road, Oxford OX1 3PU, United
Kingdom 4848institutetext: Dipartimento di Fisica e Astronomia Galileo
Galilei, Università Degli Studi di Padova, I-35122 Padova PD, Italy
4949institutetext: Dept. of Physics, Drexel University, 3141 Chestnut Street,
Philadelphia, PA 19104, USA 5050institutetext: Physics Department, South
Dakota School of Mines and Technology, Rapid City, SD 57701, USA
5151institutetext: Dept. of Physics, University of Wisconsin, River Falls, WI
54022, USA 5252institutetext: Dept. of Physics and Astronomy, University of
Rochester, Rochester, NY 14627, USA 5353institutetext: Department of Physics
and Astronomy, University of Utah, Salt Lake City, UT 84112, USA
5454institutetext: Dept. of Physics, Chung-Ang University, Seoul 06974, Korea
5555institutetext: Oskar Klein Centre and Dept. of Physics, Stockholm
University, SE-10691 Stockholm, Sweden 5656institutetext: Dept. of Physics and
Astronomy, Stony Brook University, Stony Brook, NY 11794-3800, USA
5757institutetext: Dept. of Physics, Sungkyunkwan University, Suwon 16419,
Republic of Korea 5858institutetext: Institute of Physics, Academia Sinica,
Taipei, 11529, Taiwan 5959institutetext: Dept. of Physics and Astronomy,
University of Alabama, Tuscaloosa, AL 35487, USA 6060institutetext: Dept. of
Astronomy and Astrophysics, Pennsylvania State University, University Park, PA
16802, USA 6161institutetext: Dept. of Physics, Pennsylvania State University,
University Park, PA 16802, USA 6262institutetext: Dept. of Physics and
Astronomy, Uppsala University, Box 516, SE-75120 Uppsala, Sweden
6363institutetext: Dept. of Physics, University of Wuppertal, D-42119
Wuppertal, Germany 6464institutetext: Deutsches Elektronen-Synchrotron DESY,
Platanenallee 6, D-15738 Zeuthen, Germany 6565institutetext: also at Institute
of Physics, Sachivalaya Marg, Sainik School Post, Bhubaneswar 751005, India
6666institutetext: also at Department of Space, Earth and Environment,
Chalmers University of Technology, 412 96 Gothenburg, Sweden
6767institutetext: also at Earthquake Research Institute, University of Tokyo,
Bunkyo, Tokyo 113-0032, Japan 6868institutetext: now at Dept. of Physics and
Astronomy, University of Alabama, Tuscaloosa, AL 35487, USA

Corresponding Author<EMAIL_ADDRESS>
# Citizen Science for IceCube: Name that Neutrino

R. Abbasiloyola  M. Ackermannzeuthen  J. Adamschristchurch  S. K.
Agarwallamadisonpac,bhubaneswar  J. A. Aguilarbrusselslibre  M.
Ahlerscopenhagen  J.M. Alameddinedortmund  N. M. Aminbartol  K.
Andeenmarquette  G. Antonerlangen  C. Argüellesharvard  Y. Ashidautah  S.
Athanasiadouzeuthen  L. Ausbormaachen  S. N. Axanibartol  X. Baisouthdakota
A. Balagopal V.madisonpac  M. Baricevicmadisonpac  S. W. Barwickirvine  V.
Basumadisonpac  R. Bayberkeley  J. J. Beattyohioastro,ohio  J. Becker
Tjusbochum,chalmers  J. Beiseuppsala  C. Bellenghimunich  C. Benningaachen  S.
BenZvirochester  D. Berleymaryland  E. Bernardinipadova  D. Z. Bessonkansas
E. Blaufussmaryland  S. Blotzeuthen  F. Bontempokarlsruhe  J. Y. Bookharvard
C. Boscolo Meneguolopadova  S. Bösermainz  O. Botneruppsala  J. Böttcheraachen
J. Braunmadisonpac  B. Brinsongeorgia  J. Brostean-Kaiserzeuthen  L.
Brusaaachen  R. T. Burleyadelaide  R. S. Bussemunster  D.
Butterfieldmadisonpac  M. A. Campanadrexel  I. Caracasmainz  K. Carloniharvard
J. Carpiolasvegasphysics,lasvegasastro  S. Chattopadhyaymadisonpac,bhubaneswar
N. Chaubrusselslibre  C. Chengeorgia  Z. Chenstonybrook  D. Chirkinmadisonpac
S. Choiskku  B. A. Clarkmaryland  A. Colemanuppsala  G. H. Collinmit  A.
Connollyohioastro,ohio  J. M. Conradmit  P. Coppinbrusselsvrije  R. Corleyutah
P. Correabrusselsvrije  D. F. Cowenpennastro,pennphys  P. Davegeorgia  C. De
Clercqbrusselsvrije  J. J. DeLaunayalabama  D. Delgadoharvard  S. Dengaachen
K. Deoskarstockholmokc  A. Desaimadisonpac  P. Desiatimadisonpac  K. D. de
Vriesbrusselsvrije  G. de Wasseigeuclouvain  T. DeYoungmichigan  A. Diazmit
J. C. Díaz-Vélezmadisonpac  M. Dittmermunster  A. Domierlangen  L. Draperutah
H. Dujmovicmadisonpac  M. A. DuVernoismadisonpac  T. Ehrhardtmainz  A.
Eimererlangen  P. Ellermunich  E. Ellingerwuppertal  S. El Mentawiaachen  D.
Elsässerdortmund  R. Engelkarlsruhe,karlsruheexp  H. Erpenbeckmadisonpac  J.
Evansmaryland  P. A. Evensonbartol  K. L. Fanmaryland  K. Fangmadisonpac  K.
Farragchiba2022  A. R. Fazelysouthern  A. Fedynitchsinica  N. Feiglberlin  S.
Fiedlschustererlangen  C. Finleystockholmokc  L. Fischerzeuthen  D.
Foxpennastro  A. Franckowiakbochum  P. Fürstaachen  J. Gallaghermadisonastro
E. Gansteraachen  A. Garciaharvard  L. Gerhardtlbnl  A. Ghadimialabama  C.
Girard-Carillomainz  C. Glaseruppsala  T. Glüsenkamperlangen,uppsala  J. G.
Gonzalezbartol  S. Goswamilasvegasphysics,lasvegasastro  A. Granadosmichigan
D. Grantmichigan  S. J. Graymaryland  O. Griesaachen  S. Griffinmadisonpac  S.
Griswoldrochester  K. M. Grothcopenhagen  C. Güntheraachen  P. Gutjahrdortmund
C. Hachung-ang-2023  C. Haackerlangen  A. Hallgrenuppsala  R. Hallidaymichigan
L. Halveaachen  F. Halzenmadisonpac  H. Hamdaouistonybrook  M. Ha Minhmunich
M. Handtaachen  K. Hansonmadisonpac  J. Hardinmit  A. A. Harnischmichigan  P.
Hatchqueens  A. Haungskarlsruhe  J. Häußleraachen  K. Helbingwuppertal  J.
Hellrungbochum  J. Hermannsgabneraachen  L. Heuermannaachen  N. Heyeruppsala
S. Hickfordwuppertal  A. Hidvegistockholmokc  C. Hillchiba2022  G. C.
Hilladelaide  K. D. Hoffmanmaryland  S. Horimadisonpac  K.
Hoshinamadisonpac,tokyo  W. Houkarlsruhe  T. Huberkarlsruhe  K.
Hultqviststockholmokc  M. Hünnefelddortmund  R. Hussainmadisonpac  K.
Hymondortmund  A. Ishiharachiba2022  W. Iwakirichiba2022  M.
Jacquartmadisonpac  O. Janikerlangen  M. Janssonstockholmokc  G. S.
Japaridzeatlanta  M. Jeongutah  M. Jinharvard  B. J. P. Jonesarlington  N.
Kampharvard  D. Kangkarlsruhe  W. Kangskku  X. Kangdrexel  A. Kappesmunster
D. Kappessermainz  L. Kardumdortmund  T. Kargzeuthen  M. Karlmunich  A.
Karlemadisonpac  A. Katiledmonton  U. Katzerlangen  M. Kauermadisonpac  J. L.
Kelleymadisonpac  M. Khanalutah  A. Khatee Zathulmadisonpac  A.
Kheirandishlasvegasphysics,lasvegasastro  J. Kirylukstonybrook  S. R.
Kleinberkeley,lbnl  A. Kochockimichigan  R. Koiralabartol  H. Kolanoskiberlin
T. Kontrimasmunich  L. Köpkemainz  C. Koppererlangen  D. J. Koskinencopenhagen
P. Koundalkarlsruhe  M. Kovacevichdrexel  M. Kowalskiberlin,zeuthen  T.
Kozynetscopenhagen  J. Krishnamoorthimadisonpac,bhubaneswar  K.
Kruiswijkuclouvain  E. Krupczakmichigan  A. Kumarzeuthen  E. Kunbochum  N.
Kurahashidrexel  N. Ladzeuthen  C. Lagunas Gualdazeuthen  M.
Lamoureuxuclouvain  M. J. Larsonmaryland  S. Latsevaaachen  F. Lauberwuppertal
J. P. Lazaruclouvain  J. W. Leeskku  K. Leonard DeHoltonpennastro,pennphys  A.
Leszczyńskabartol  M. Lincettobochum  M. Liubarskaedmonton  E. Lohfinkmainz
C. Lovedrexel  C. J. Lozano Mariscalmunster  L. Lumadisonpac  F.
Lucarelligeneva  W. Luszczakohioastro,ohio  Y. Lyuberkeley,lbnl  J.
Madsenmadisonpac  E. Magnusbrusselsvrije  K. B. M. Mahnmichigan  Y.
Makinomadisonpac  E. Manaomunich  S. Mancinamadisonpac,padova  W. Marie
Saintemadisonpac  I. C. Marişbrusselslibre  S. Markacolumbia  Z. Markacolumbia
M. Marseealabama  I. Martinez-Solerharvard  R. Maruyamayale  F. Mayhewmichigan
T. McElroyedmonton  F. McNallymercer  J. V. Meadcopenhagen  K.
Meaghermadisonpac  S. Mechbalzeuthen  A. Medinaohio  M. Meierchiba2022  Y.
Merckxbrusselsvrije  L. Mertenbochum  J. Micallefmichigan  J. Mitchellsouthern
T. Montaruligeneva  R. W. Mooreedmonton  Y. Moriichiba2022  R. Morsemadisonpac
M. Moulaimadisonpac  T. Mukherjeekarlsruhe  R. Naabzeuthen  R. Nagaichiba2022
M. Nakosmadisonpac  U. Naumannwuppertal  J. Neckerzeuthen  A. Negiarlington
M. Neumannmunster  H. Niederhausenmichigan  M. U. Nisamichigan  A. Noellaachen
A. Novikovbartol  S. C. Nowickimichigan  A. Obertacke Pollmannchiba2022  V.
O’Dellmadisonpac  B. Oeyengent  A. Olivasmaryland  R. Orsoemunich  J.
Osbornmadisonpac  E. O’Sullivanuppsala  H. Pandyabartol  N. Parkqueens  G. K.
Parkerarlington  E. N. Paudelbartol  L. Paulsouthdakota  C. Pérez de los
Herosuppsala  T. Pernicezeuthen  J. Petersonmadisonpac  S. Philippenaachen  A.
Pizzutomadisonpac  M. Plumsouthdakota  A. Ponténuppsala  Y. Popovychmainz  M.
Prado Rodriguezmadisonpac  B. Priesmichigan  R. Procter-Murphymaryland  G. T.
Przybylskilbnl  C. Raabuclouvain  J. Rack-Helleismainz  K. Rawlinsanchorage
Z. Rechavmadisonpac  A. Rehmanbartol  P. Reichherzerbochum  E. Resconimunich
S. Reuschzeuthen  W. Rhodedortmund  B. Riedelmadisonpac  A. Rifaieaachen  E.
J. Robertsadelaide  S. Robertsonberkeley,lbnl  S. Rodanskku  G.
Roellinghoffskku  M. Rongenerlangen  A. Rostedchiba2022  C. Rottutah,skku  T.
Ruhedortmund  L. Ruohanmunich  D. Ryckboschgent  I. Safaharvard,madisonpac  J.
Safferkarlsruheexp  D. Salazar-Gallegosmichigan  P. Sampathkumarkarlsruhe  A.
Sandrockwuppertal  M. Santanderalabama  S. Sarkaredmonton  S. Sarkaroxford  J.
Savelbergaachen  P. Savinamadisonpac  M. Schaufelaachen  H. Schielerkarlsruhe
S. Schindlererlangen  B. Schlütermunster  F. Schlüterbrusselslibre  N.
Schmeisserwuppertal  T. Schmidtmaryland  J. Schneidererlangen  F. G.
Schröderkarlsruhe,bartol  L. Schumachererlangen  S. Sclafanimaryland  D.
Seckelbartol  M. Seikhkansas  M. Seoskku  S. Seunarineriverfalls  R.
Shahdrexel  S. Shefalikarlsruheexp  N. Shimizuchiba2022  M. Silvamadisonpac
B. Skrzypekberkeley  B. Smithersarlington  R. Snihurmadisonpac  J.
Soedingreksodortmund  A. Søgaardcopenhagen  D. Soldinutah  P. Soldinaachen  G.
Sommanibochum  C. Spannfellnermunich  G. M. Spiczakriverfalls  C.
Spieringzeuthen  M. Stamatikosohio  T. Stanevbartol  T. Stezelbergerlbnl  T.
Stürwaldwuppertal  T. Stuttardcopenhagen  G. W. Sullivanmaryland  I.
Taboadageorgia  S. Ter-Antonyansouthern  A. Terliukmunich  M. Thiesmeyeraachen
W. G. Thompsonharvard  J. Thwaitesmadisonpac  S. Tilavbartol  K.
Tollefsonmichigan  C. Tönnisskku  S. Toscanobrusselslibre  D. Tosimadisonpac
A. Trettinzeuthen  C. F. Tunggeorgia  R. Turcottekarlsruhe  J. P.
Twagirayezumichigan  M. A. Unland Elorrietamunster  A. K.
Upadhyaymadisonpac,bhubaneswar  K. Upshawsouthern  A. Vaidyanathanmarquette
N. Valtonen-Mattilauppsala  J. Vandenbrouckemadisonpac  N. van
Eijndhovenbrusselsvrije  D. Vannerommit  J. van Santenzeuthen  J. Varamunster
J. Veitch-Michaelismadisonpac  M. Venugopalkarlsruhe  M. Vereeckenuclouvain
S. Verpoestbartol  D. Veskecolumbia  A. Vijaimaryland  E. H. S.
Warrickdrexel,alabamanow  C. Walckstockholmokc  C. Weavermichigan  P.
Weigelmit  A. Weindlkarlsruhe  J. Weldertpennastro,pennphys  A. Y. Wenharvard
C. Wendtmadisonpac  J. Werthebachdortmund  M. Weyrauchkarlsruhe  N.
Whitehornmichigan  C. H. Wiebuschaachen  D. R. Williamsalabama  L.
Witthausdortmund  A. Wolfaachen  M. Wolfmunich  G. Wredeerlangen  X. W.
Xusouthern  J. P. Yanezedmonton  E. Yildizcimadisonpac  S. Yoshidachiba2022
R. Youngkansas  S. Yuutah  T. Yuanmadisonpac  Z. Zhangstonybrook  P.
Zhelninharvard  P. Zilbermanmadisonpac  M. Zimmermanmadisonpac

(Received: date / Accepted: date)

###### Abstract

Name that Neutrino is a citizen science project where volunteers aid in
classification of events for the IceCube Neutrino Observatory, an immense
particle detector at the geographic South Pole. From March 2023 to September
2023, volunteers did classifications of videos produced from simulated data of
both neutrino signal and background interactions. Name that Neutrino obtained
more than 128,000 classifications by over 1,800 registered volunteers that
were compared to results obtained by a deep neural network machine-learning
algorithm. Possible improvements for both Name that Neutrino and the deep
neural network are discussed.

###### Keywords:

neutrino citizen science Zooniverse

††journal: Eur. Phys. J. Plus

##  1 Introduction

Name that Neutrino ntnwebsite  is a citizen science project that seeks input
from the public to aid in classification of neutrino events for the IceCube
Neutrino Observatory. Citizen science provides a powerful tool for advancing
both science and education and outreach. Motivated volunteers learn more about
cutting-edge research and then perform analyses based on visual scans of data.
The ultimate goal is to design, develop and implement an online experience
that allows novices to contribute to ongoing research. Results are aggregated
to reach statistics not possible within the traditional research community,
enabling new insights and discoveries. This paper gives a summary of the
IceCube’s promising first attempt to engage people outside the collaboration
to analyze data using the citizen science approach.

Name that Neutrino is hosted on Zooniverse zooniverse , the largest web-based
research platform of its kind with over a million volunteers world-wide.
Zooniverse has established the power of the citizen science approach. For
example, Galaxy Zoo, a Zooniverse project that works on galaxy shape
classification, has had more than 10,000 volunteers, resulting in over 60
publications Géron et al. (2023). Their work, much like IceCube’s discussed
here, is related to pattern recognition and capitalizes on the keen ability of
humans to see things that currently remain difficult to identify with
computers, even with advances in machine-learning algorithms. However, Name
that Neutrino is one of the few projects on Zooniverse to include videos
rather than static images.

Identifying a research question and developing and implementing the tools
needed for Name that Neutrino has been a long process. The first attempt at an
IceCube citizen science project began in June 2016 at a six-week internship
program for high school students quarknet . This group produced much of the
background material to introduce the IceCube project to the citizen users but
the effort was limited to displaying data in static images at that time. As
described in more detail the next section, the IceCube Neutrino Observatory is
a cubic kilometer array of 5,160 light sensors (Digital Optical Modules or
DOMS) embedded in the South Pole ice. From the amount and time sequence of
recorded light, the energy and direction of the incident particle can be
reconstructed. A video of the time sequence of the data for each event is much
more informative, especially for novices.

The ability to include videos was eventually implemented by Zooniverse, and we
conceived of the idea to look at results from people classifying IceCube
events and see how they compared to current state-of-the-art machine-learning
algorithms. A formal Zooniverse launch was completed in 2023 which included a
rigorous approval process in order to be featured to the Zooniverse community.
Videos for Name that Neutrino were produced from Monte Carlo simulations of
trigger-level IceCube data which included significant noise and many ambiguous
events. Trigger-level refers to the as-collected data before any quality cuts
or noise reduction techniques have been applied. The primary motivation for
using trigger-level data was to inspect and compare the performance of both
citizen users and a deep neural network (DNN) machine-learning algorithm at
the most challenging level.

Figure 1: The IceCube Neutrino Observatory.

##  2 IceCube Neutrino Observatory

The IceCube Neutrino Observatory icecube , located at the geographic South
Pole, has a multi-faceted and growing science scope. Research topics include
astrophysics, particle and fundamental physics, glaciology, and more, with
exciting results identifying the first high-energy neutrino sources Aartsen et
al. (2018); Abbasi et al. (2022, 2023a), leading limits on certain classes of
dark matter candidates Abbasi et al. (2023b) and measurements of neutrino
oscillations at energies beyond those reachable in dedicated reactor and
accelerator experiments Abbasi et al. (2023c). IceCube consists of a hexagonal
array of DOMs that instrument a cubic kilometer of ice at depths between 1.5
to 2.5 kilometers below the surface as shown in Fig. 1.

When a neutrino interacts with with the rock below or the ice near or within
the IceCube array, the resulting high-energy secondary particles emit
Cherenkov light, some of which is detected by the DOMs. The light pattern (or
topology) will depend on the neutrino characteristics (flavor, direction, and
energy) and the type of interaction Aartsen et al. (2017a). In the work
presented here, IceCube cannot distinguish between neutrinos and anti-
neutrinos. Since there are three neutrino types, and two channels of neutrino
interaction relevant to this work, there are in principle six different
options. In practice most of the recorded events fall into two broad
categories depending on the neutrino flavor and how it interacted— either a
charged or neutral current interaction. The two main types of topologies seen
in IceCube are tracks and cascades. Tracks are produced by muons originating
from muon neutrino charged current interactions or cosmic-ray induced air
showers. Muons with sufficient energy can propagate large distances in the ice
resulting in a linear light pattern. Cascades are produced by particle showers
induced by all neutral current neutrino interactions, as well as electron- and
tau charged current neutrino interactions. These particle showers evolve over
distances of approximately 10 meters in the ice, resulting in a roughly
spherical outward going light pattern.

Unfortunately, at least for those only interested in neutrinos, there is an
overwhelming background— a steady rain of muons produced by cosmic-ray
interactions above the detector in the Earth’s atmosphere from the southern
hemisphere. Cosmic-ray interactions in the atmosphere also produce neutrinos,
which are identified at the rate of about one per million background events.
Roughly one in a few hundred of the neutrinos which interact in the detector
is from an astrophysical source rather than a cosmic-ray interaction in the
Earth’s atmosphere. IceCube sees about 3000 events a second, almost entirely
background events from cosmic-ray induced muon tracks Aartsen et al. (2017b).

Figure 2: Examples of the five signal topologies used in this work. The color
indicates the arrival time at individual DOMs with red happening first, then
yellow, green, and blue last. The size of each bubble is related to the light
detected by the DOM.

Many investigations of IceCube data begin by separating events into the two
broad categories, tracks or cascades. The track events can be further
subdivided into groups as shown in Fig. 2. Each colored “bubble” in these
event displays represents a DOM that detected light. The size of each bubble
is related to the total Cherenkov light detected. The colors indicate the
relative time the light was received, with red first and blue last. Through-
going tracks start and end outside of the detector and therefore the bubbles
transverse the volume. Starting tracks begin inside the detector and move
outward, identified by redder bubbles on the interior and bluer bubbles toward
the edge. Starting tracks and cascades are some of the most interesting
topologies for IceCube since they are only produced by neutrino signal events.
However, classification is further complicated since starting tracks are
usually accompanied by a cascade from the hadronic part of the charged current
interaction and starting tracks turn into cascades as the track length
approaches zero. Stopping tracks start outside of the detector and stop
inside, with redder bubbles near the edge and bluer bubbles on the interior.
Skimming tracks are events all energy loss outside of the detector volume; the
detected light will be near the outer volume of the instrumented ice only
making it difficult to reconstruct the energy and direction of the incident
particle.

IceCube has developed a DNN to classify events that returns a probability of
an event being one of the five event topologies: a cascade, or a through-
going, starting, stopping, or skimming track. In general, DNNs are a class of
machine-learning algorithms that attempt to mimic the brain by learning from
known examples. The specific architecture of the DNN used in this work is
described in Ref. Kronmueller and Glauch (2020) and it was inspired by
Google’s InceptionResnet architecture Szegedy et al. (2016). Each event is
interpreted as a 4D image with three spatial detector dimensions and one
feature dimension that contains information about the time and charge recorded
at each DOM.

Training samples for DNNs usually have a majority of examples that are clear
enough for the algorithm to learn. For this particular DNN, the training used
Monte Carlo simulations of events after cuts to the trigger-level data that
suppressed noise, background, and ambiguous events. The Monte Carlo truth
value is used to provide the DNN with the correct characterization for the
training samples; however, the Monte Carlo truth value for trigger-level
events in IceCube is not clear cut. One reason for this is trigger-level
events can contain coincident events where there are two or more separate
signals present, and therefore the Monte Carlo truth is not single-valued.
Furthermore, the aforementioned event topologies have been found by IceCube to
be useful classes in later stages of our data processing pipeline. They are
based on knowledge about the physical processes but they also have some
ambiguity (for example how close an event has to be to the detector boundary
to count as skimming). On trigger-level, these categories may not be optimal
and it was thus a goal of this project to compare the intuition of citizen
users against the DNN.

##  3 Name that Neutrino

Name that Neutrino is available in English, Spanish, and German to anyone in
the world with internet access. After selecting “classify,” first time
visitors are prompted with a brief tutorial on how to perform the task, and a
field guide with frequently asked questions. Once the tutorial is completed,
users have access to a random event from the sample of 4,273 videos. The
“classify” section shown to users can be seen in Fig. 3. They are able to
replay the 7-second video as many times as desired and adjust the playback
speed. Then the users must choose from one of the topologies described
previously and shown in Fig. 2.

The 4,273 simulated trigger-level events chosen for Name that Neutrino were
randomly selected to produce a uniform distribution in the log of the
energy—the number of DOMs detecting light scales with energy. With trigger-
level events, it is expected to have events that are difficult to classify,
especially events with energies close to the detector threshold. An artificial
enhancement of electron neutrino events was also implemented to ensure there
was a variety of topologies. The enhancement was needed due to the lower
trigger rate of electron neutrinos compared to much longer lived muons.
Cascades produced by electron neutrino interactions cannot travel far and
therefore the electron neutrino must interact inside of the detector. Muon
neutrinos on the other hand can interact far from the detector producing long-
lived muons that make it inside the detector.

Videos (10 frames per second) were produced with the IceCube event display
software Steamshovel La Dieu et al. (2024) showing 3D visualizations of each
event with the detector rotating by 5.3 degrees per second. The choices for
the videos were chosen to provide a variety of viewpoints with adequate video
quality after compression to fit within the Zooniverse file limit of 1 MB per
event file. The videos were uploaded as a subject-set to work with a uniquely
designed Zooniverse classification workflow Warrick (2023).

The Zooniverse approval process requires reviews and beta tests; Name that
Neutrino completed one internal Zooniverse review and two beta tests that
provided feedback from citizen users who suggested improvements and assessed
the feasibility of the project. Zooniverse approved and officially launched
Name that Neutrino in March 2023; after 3 months each of the videos were
classified 15 times (by 15 different users) resulting in 64,095
classifications. The video repetition is standard practice for Zooniverse
projects and is called the retirement limit. When an individual user is
classifying events for Name that Neutrino, they will be shown a randomly
chosen video that has not yet met the retirement limit and was not previously
seen by that user. In June 2023, the retirement limit was increased to 20 and
all of the videos were then classified 5 more times. By September 2023, the
new retirement limit of 20 was reached for all videos, resulting in 85,460
classifications. As of December 2023, there are over 128,000 classifications
and over 1,800 registered volunteers for Name that Neutrino who continue to
work toward the higher retirement limits.

Figure 3: The “classify” section of the Name that Neutrino Zooniverse project.
Users are prompted to complete a brief tutorial on their first visit and
presented with a field guide with FAQs.

##  4 Results

After the classification of all 4,273 events, a maximum score is calculated
for each event. The maximum score for either the Zooniverse citizen users or
for the DNN is the probability value of the most-likely category. For the
users, the maximum score is calculated as the fraction of the total votes that
the category with the most votes received. This is a proxy for confidence as
it shows the level of agreement between users. A maximum score of 0.2 is the
minimum possible value and indicates no preference for any classification. A
maximum score of 1 means that every user chose the same classification. For
the DNN, the maximum score is defined to be the highest probability among the
five categories provided by the DNN.

The distribution of maximum scores are shown in Fig. 4 for a retirement limit
of 20 classifications per event for users (left), and for the DNN (right). For
users, the distribution is broadly peaked around 0.53 with the majority of
events achieving a maximum score less than 0.5. The low user scores signify
disagreements between users and are likely related to the difficulty of
classifying trigger-level events. There are no events with a maximum score
below 0.25, indicating that there is always at least a weak preference for a
classification. The distribution of DNN maximum scores peaks sharply at 1,
indicating a high confidence in the classification. It is important to point
out that high confidence does not necessarily correlate with accuracy. The DNN
was applied to data with more information than the training set, and could
preferentially select the same classification for events outside its original
scope.

Figure 4: The distribution of maximum scores for the citizen science users
(left) and the DNN machine-learning algorithm (right). Figure 5: A confusion
matrix showing the agreement along the diagonal between the user and DNN
maximum categories, normalized to the maximum DNN category. Only user events
with a maximum score of 0.55 and above are included.

The confusion matrix shown in Fig. 5 compares the maximum categories chosen by
the DNN to the maximum categories chosen by the citizen science users. To
better compare the broad, low-confidence user distribution with the sharply
peaked, high-confidence DNN results, we applied a cut to remove user data with
a maximum score of below 0.55. The confusion matrix counts the number of
events in each category chosen by users compared to and normalized to the DNN
maximum category. The columns display events in the associated DNN maximum
categories and the rows include events in the corresponding user maximum
categories. The diagonal values represent agreement between the DNN and the
users, with the largest agreement occurring with cascades at 92.1%. The
disagreement between the users and the DNN is represented by off-diagonal
values. For example, in the top right corner of Fig. 5, there were 25 events
(30.1%) where the users chose skimming out of the 83 total stopping tracks as
classified by the DNN. Similarly, users chose another 35 of those 83 events
(42.2%) to be through-going tracks instead of stopping tracks. Some of the
confusion shown in Fig. 5 comes from differences in the training methods for
the users and the DNN. The DNN was trained on around 13 million events with
concrete definitions for each classification and the users were given a
qualitative explanation of each classification and one example image per
classification. Finally, though this confusion matrix is normalized to the DNN
maximum category, this does not imply that is the correct characterization of
the event. The matrix simply shows levels of agreement between users and the
DNN results.

Since the events are trigger level, “expert-by-eye” values were produced by
members of the Name that Neutrino team as a substitute for the Monte Carlo
truth. Specifically, we explored the off-diagonal starting, stopping, and
through-going track events where users and the DNN disagreed on the
classification. The comparison of the user and DNN classifications with the
“expert-by-eye” classifications are shown in the left and right sides of Fig.
6 respectively. Again, diagonal values represent agreement and off-diagonal
values represent disagreement, but now in comparison to the “expert-by-eye”.
Note the columns do not sum to 100% since it is still possible for selected
events to be classified as cascades and skimming tracks. Future work will
explore these scenarios.

Though the number of events has been significantly reduced, there are some
large, noteworthy differences between the performance of the users and the
DNN. For through-going tracks, users agreed with the expert category 78.5% of
the time compared to 19.0% for the DNN. This could be an indication that the
users are better at identifying through-going tracks or that users were more
inclined to pick through-going track because they took that option much more
often than other categories. Alternatively, the DNN agreed with the expert
category more often for the starting and stopping tracks than the users. This
may be due to difficulty in identifying the edges of the detector in the views
available to the users. More work is needed to understand the results shown in
Fig. 6.

Figure 6: Confusion matrices showing both user (left) and DNN (right) results
normalized to the “expert-by-eye” value for through-going tracks, starting
tracks, and stopping tracks. Only the off diagonal values from Fig. 5 were
used in order to explore disagreeing scenarios. Note it is possible for these
events to be classified as cascades or skimming tracks.

##  5 Conclusion and Future Work

After seven months of collecting data, the Name that Neutrino project has
provided insights into data classification for both citizen users and an
IceCube specific DNN machine-learning algorithm. There was more agreement
between the users and the DNN for events classified as cascades and less
consistency for events classified as starting, stopping, or through-going
tracks. The comparison of user and “expert-by-eye” classifications indicates
that additional training is needed for the users. More work should be done in
order to fully understand the DNN results compared to the “expert-by-eye”
classifications. This initial study has demonstrated the feasibility of using
the citizen science approach to classify IceCube data but that more work is
needed to establish the validity of the user results.

Future improvements to Name that Neutrino should include increasing the
rotation and sharpening the edges of the detector to help improve the
identification of starting and stopping tracks by the users. Since the task of
classifying events at the trigger-level was rather challenging, future
iterations of Name that Neutrino could include cleaner events where it would
be possible to compare the Monte Carlo truth values to the user results. Or
alternatively, more work could be done to optimize the Monte Carlo truth
values at trigger-level in order to compare to the results presented here.
Future citizen science projects could involve the use of real data rather than
simulation data to separate coincident events, identify new classes, or search
for possible biases in DNN performance that come from the differences between
simulation and data.

Name that Neutrino demonstrated that citizen science is a powerful tool for
public engagement for IceCube. It was not clear that there would be interest
in looking at the rather abstract data, especially compared to more readily
identifiable astronomical optical telescope images. Engaging with over 1,800
members of the general public with the IceCube project through 128,000
classifications and over 600 discussion board posts certainly counts as a
successful start.

##  6 Acknowledgements

The IceCube collaboration acknowledges significant contributions to this
manuscript from Christina Love and Jim Madsen.

The authors gratefully acknowledge the support from the following agencies and
institutions: USA – U.S. National Science Foundation-Office of Polar Programs,
U.S. National Science Foundation-Physics Division, U.S. National Science
Foundation-EPSCoR, U.S. National Science Foundation-Office of Advanced
Cyberinfrastructure, Wisconsin Alumni Research Foundation, Center for High
Throughput Computing (CHTC) at the University of Wisconsin–Madison, Open
Science Grid (OSG), Partnership to Advance Throughput Computing (PATh),
Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support
(ACCESS), Frontera computing project at the Texas Advanced Computing Center,
U.S. Department of Energy-National Energy Research Scientific Computing
Center, Particle astrophysics research computing center at the University of
Maryland, Institute for Cyber-Enabled Research at Michigan State University,
Astroparticle physics computational facility at Marquette University, NVIDIA
Corporation, and Google Cloud Platform; Belgium – Funds for Scientific
Research (FRS-FNRS and FWO), FWO Odysseus and Big Science programmes, and
Belgian Federal Science Policy Office (Belspo); Germany – Bundesministerium
für Bildung und Forschung (BMBF), Deutsche Forschungsgemeinschaft (DFG),
Helmholtz Alliance for Astroparticle Physics (HAP), Initiative and Networking
Fund of the Helmholtz Association, Deutsches Elektronen Synchrotron (DESY),
and High Performance Computing cluster of the RWTH Aachen; Sweden – Swedish
Research Council, Swedish Polar Research Secretariat, Swedish National
Infrastructure for Computing (SNIC), and Knut and Alice Wallenberg Foundation;
European Union – EGI Advanced Computing for research; Australia – Australian
Research Council; Canada – Natural Sciences and Engineering Research Council
of Canada, Calcul Québec, Compute Ontario, Canada Foundation for Innovation,
WestGrid, and Digital Research Alliance of Canada; Denmark – Villum Fonden,
Carlsberg Foundation, and European Commission; New Zealand – Marsden Fund;
Japan – Japan Society for Promotion of Science (JSPS) and Institute for Global
Prominent Research (IGPR) of Chiba University; Korea – National Research
Foundation of Korea (NRF); Switzerland – Swiss National Science Foundation
(SNSF).

This publication uses data generated via the Zooniverse.org platform,
development of which is funded by generous support, including a Global Impact
Award from Google, and by a grant from the Alfred P. Sloan Foundation.

##  7 Data Availability Statement

The datasets used and/or analysed during the current study are available at
https://zenodo.org/records/10521028 and the videos are available at
https://www.zooniverse.org/projects/icecubeobservatory/name-that-neutrino.

## References

  * (1) Name that Neutrino. Zooniverse Project. https://www.zooniverse.org/projects/icecubeobservatory/name-that-neutrino
  * (2) Zooniverse. https://www.zooniverse.org/about
  * Géron et al. (2023) Géron, T., et al.: Galaxy Zoo: Kinematics of strongly and weakly barred galaxies. Monthly Notices of the Royal Astronomical Society 521(2), 1775–1793 (2023)
  * (4) Quarknet. https://quarknet.org/content/about-quarknet
  * (5) The IceCube Neutrino Observatory. https://icecube.wisc.edu/
  * Aartsen et al. (2018) Aartsen, M.G., et al.: Neutrino emission from the direction of the blazar txs 0506+056 prior to the icecube-170922a alert. Science 361(6398), 147–151 (2018)
  * Abbasi et al. (2022) Abbasi, R., et al.: Evidence for neutrino emission from the nearby active galaxy ngc 1068. Science 378(6619), 538–543 (2022)
  * Abbasi et al. (2023a) Abbasi, R., et al.: Observation of high-energy neutrinos from the galactic plane. Science 380(6652), 1338–1343 (2023)
  * Abbasi et al. (2023b) Abbasi, R., et al.: Searches for connections between dark matter and high-energy neutrinos with icecube. Journal of Cosmology and Astroparticle Physics (10), 003 (2023)
  * Abbasi et al. (2023c) Abbasi, R., et al.: Measurement of atmospheric neutrino mixing with improved icecube deepcore calibration and data processing. Phys. Rev. D 108, 012014 (2023)
  * Aartsen et al. (2017a) Aartsen, M.G., et al.: All-sky search for time-integrated neutrino emission from astrophysical sources with 7 yr of icecube data. The Astrophysical Journal 835(2), 151 (2017)
  * Aartsen et al. (2017b) Aartsen, M.G., et al.: The IceCube Neutrino Observatory: Instrumentation and Online Systems. JINST 12(03), 03012 (2017)
  * Kronmueller and Glauch (2020) Kronmueller, M., Glauch, T.: Application of Deep Neural Networks to Event Type Classification in IceCube. PoS ICRC2019, 937 (2020)
  * Szegedy et al. (2016) Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.: Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning. arXiv e-prints, 1602–07261 (2016) https://doi.org/10.48550/arXiv.1602.07261
  * La Dieu et al. (2024) La Dieu, D., et al.: icecube/icetray-public: v1.9.1. Zenodo (2024). https://www.zenodo.org/doi/10.5281/zenodo.10452620
  * Warrick (2023) Warrick, E.H.S.: Building a Citizen Science Project to Investigate IceCube Data Analysis. MS Thesis, Drexel University (2023). https://doi.org/10.17918/00001705


# An Unsupervised Domain Adaptation Method for Locating Manipulated Region in
Partially Fake Audio

Siding Zeng1, 2, Jiangyan Yi2, Jianhua Tao3, Shan Liang2, Yujie Chen2,
Chenglong Wang2, Yong Ren2, Xiaohui Zhang2

###### Abstract

When the task of locating manipulation regions in partially fake audio (PFA)
involves cross-domain datasets, the performance of deep learning models drops
significantly due to the shift between the source and target domains. To
address this issue, existing approaches often employ data augmentation before
training. However, they overlook the characteristics in the target domain that
are absent in the source domain. Inspired by the mixture-of-experts model, we
propose an unsupervised method named Samples mining with Diversity and Entropy
(SDE). Our method first learns from a collection of diverse experts that
achieve great performance from different perspectives in the source domain,
but with ambiguity on target samples. We leverage these diverse experts to
select the most informative samples by calculating their entropy. Furthermore,
we introduce a label generation method tailored for these selected samples
that are incorporated in the training process in the source domain integrating
the target domain information. We applied our method to a cross-domain
partially fake Audio Detection dataset, ADD2023Track2. By introducing 10% of
samples from the target domain, we achieved an F1 score of 43.84%, which
represents a relative increase of 77.2% compared to the second-best method.

## Introduction

AI generated content technology has witnessed swift progress in recent years,
particularly in speech-related applications such as text-to-speech (TTS) (Wang
et al. 2023a; Huang et al. 2022; Tan et al. 2024) and voice conversion
(VC)(Chan et al. 2022; Patel and Patil 2015; Chen et al. 2021). Although these
technologies have brought about convenience, they have also brought about the
forgery and deception of voices, which pose significant security threats to
social security, privacy security, and property security (Lorenzo-Trueba et
al. 2018b; Kinnunen et al. 2018; Lorenzo-Trueba et al. 2018a). As an important
branch of voice forgery and deception, partially fake audio (PFA) (Yi et al.
2023) has been receiving an increasing focus in recent years. Attackers can
contaminate original voice segments with synthesized or authentic vocal
utterances. Such low-cost modifications can easily alter the semantics of
sentences. For instance, attackers can easily modify a single word such as
time, place, and characters in sentence to dramatically change the semantics.
Furthermore, if attackers have knowledge of phonology, they can manipulate
vowels and even consonants such as “pan”, “pin”, “pen”,in English or “sha”,
“la”, “pa”, in Chinese, which are smaller than the word level. These PFAs
contain a substantial amount of genuine information, making them undetectable
by models designed for authenticating entire segments of speech.

Figure 1: The visualization of the feature distribution of HAD training
dataset(source domain) and ADD2023 test dataset(target domain)

However, compared to PFA detection, which tries to identify the authenticity
of voices containing a substantial amount of genuine content, locating the
manipulated region of PFA is more meaningful. PFA detection focuses on
sentence-level positive detection, determining whether a sentence has been
manipulated, while locating the manipulated region of PFA emphasizes frame-
level positive detection, assessing whether a vowel or even a single frame has
been manipulated, which can be more challenging. Therefore, defending against
such fine-grained partially fake scenarios poses significant challenges for
defenders. Existing research on the localization of manipulated regions has
shown satisfactory performance on certain in-domain partially fake audio
datasets.(Zhang et al. 2022a; Xie et al. 2024). Nevertheless, their efficacy
markedly declines when applied to cross-domain partially fake audio datasets.
This phenomenon, known as out-of-distribution, arises when there is a shift
between the source domain and the target domain. In addressing this issue,
researchers have proposed many methods in the field of anti-spoof tasks for
fake audio. However, domain adaptation for PFA has not yet received widespread
attention. When participants in the ADD2023 competition face cross-domain
challenges, they often resort to data augmentation as a coping
strategy(Martín-Doñas and Álvarez 2023; Li et al. 2023; Cai and Li 2024), such
as adding noise or processing segments through voice conversion models.

However, methods based on data augmentation still only consider information
from the source domain and do not assess the information missing in the target
domain. The urgent problem is that, under the standard empirical risk
minimization (ERM) paradigm, machine learning algorithms tend to choose
solutions based on the most salient features without considering alternatives.
Learning from one distribution and applying it to another can cause model
failure. Any method based on the source domain struggles to be effective when
there is a significant distribution shift between the source and target
domains, because machine learning algorithms have never encountered
characteristic from the target domain. Therefore, it is necessary to
incorporate samples from the target domain for domain adaptation. In such
cases, selecting the highest quality samples becomes crucial. Many methods
proceed through various analyses and then introduce samples of the target
domain. However, they overlook the relativity of information; information
already present in the source domain should not be the focus of learning,
while information absent in the source domain should be emphasized. However,
in the task of PFA, the situation is somewhat different. The normal domain
adaptation task generally refers to adaptation between two domains. However,
as for the partially fake audio dataset, it is much more difficult. Due to the
recording environment, devices, compression methods, and other factors, there
will exhibit distribution differences among the unprocessed utterances.
Moreover, considering the variety of methods for generating manipulated fake
audio, coupled with the manipulation using non-synthetic real audio, the
distribution of audio sources in ”partially fake audio” becomes even more
complex. This complexity makes the task of ”Locating Manipulated Region in
partially fake audio” a more challenging cross-domain issue.

So we propose our method for the task of Locating Manipulated Region in
partially fake audio (PFA) in cross-domain scenarios, structured in three
phases. In the first phase, we employ a reverse knowledge distillation
approach to learn a collection of diverse experts, which is inspired by the
Mixture of Experts model. Although these experts share the same structure, our
reverse distillation process constraints allow them to achieve high
performance on source domain but interpret the source domain from multiple
perspectives. Then, with the assistance of these diverse experts, we aim to
find samples that contain the subset of the symmetric difference between the
collection of information in the source domain by calculating the entropy of
samples relative to diverse experts, which can be viewed as characteristics
present in the target domain but absent in the source domain. Finally, we
transfer information from the target domain by swapping voice active segments
and generating labels, allowing these samples to participate in the training
process. By incorporating 10% the most informative samples from the target
domain into our analysis, we attained an F1 score of 43.84%. This score
signifies a 77.2% relative improvement over the performance of the second-best
method. Furthermore, the results of the Ablation Experiment demonstrate the
unique contribution of each step of SDF, indirectly validating the
effectiveness of our method.

We make contribution as follows:

  * •

we propose a method training for diverse experts based on reverse knowledge
distillation.

  * •

We propose a method that introduces information entropy to evaluate the most
informative samples based on a set of diverse experts.

  * •

We proposed a soft-label generation method tailored for the most informative
samples in PFA tasks.

The remainder of this survey is structured as follows. In Section 2, we
introduced the relevant work and progress of the PFA. In Section 3, we first
defined some notations, and then proposed our method. Experiments, results,
and discussion are reported in Section 4. Finally, we concluded the paper in
Section 5.

## related work

since research on the detection of synthetic speech has already become very
sophisticated(Zhang et al. 2023b, 2024, c; Zhao et al. 2022; Wang et al.
2023b; Zhang et al. 2023a; Wang et al. 2024; Chen et al. 2024), numerous
methods and datasets have been developed for partially fake audio tasks . Yi
et al. (Yi et al. 2021; Khanjani, Watson, and Janeja 2023)constructed the
first partially spoofed audio dataset named HAD, which replaces some nature
segments with synthesized speech segments having different semantic content.
The Audio Deep Synthesis Detection (ADD) challenge 2022 (Yi et al. 2022)
involves a detection track containing partially spoofed audio. The organizers
provided only real and entirely fake data for training, and the task was
target to detect fake at utterance level. In this challenge, notable detection
performance was achieved by fine-tuning pretrained self-supervised learning
(SSL) models with different back-end classifiers (Lv et al. 2022; Liu et al.
2022). Boundary detection task was only introduced as a proxy task to achieve
utterancelevel detection (Cai, Wang, and Li 2023; Wu et al. 2022).

Simultaneously, Zhang et al.(Zhang et al. 2021b) construct a speech database
called ‘Partial Spoof’ designed for Partially scenarios. The above two
datasets are the beginning of the research for PF scenario in ADD task.
Afterward, Zhang et al. (Zhang et al. 2021a) propose the SELCNN network to
enhance the ability of the accuracy of the utterance. Lv et al. (Lv et al.
2022) use Wav2Vec2 (W2V2) (Baevski et al. 2020) as front-end, ECAPA-TDNN
(Desplanques, Thienpondt, and Demuynck 2020) as back-end achieving the first
rank in ADD 2022 Track 2(Yi et al. 2022). Although the above research shows
effectiveness at the utterance level detection in PF, they do not pinpoint
specific segments with precision. Thus, Zhang et al. (Zhang et al. 2022b)
extended the previous utterance-level PF dataset labels to frame-level and
proposed corresponding W2V2-based countermeasures to enhance frame-level
detection capability.and scholars from Inner Mongolia University have proposed
a method to detect high-frequency features at splicing sites using wavelet
functions(Zhang et al. 2022a). Moreover, the ADD 2023 (Yi et al. 2023)
introduced a PSAL track, which has resulted in numerous promising works
(Martín-Doñas and Álvarez 2023; Li et al. 2023; Cai and Li 2024). Among them,
an effective approach was proposed to achieve the best localization score by
integrating the decisions from three countermeasure (CM) systems including
boundary detection, frame-level fake detection, and utterance-level fake
detection (Cai and Li 2024) Recently, an approach incorporating a contrastive
learning module and temporal convolution was proposed to effectively capture
better features for localization (Xie et al. 2024).

Figure 2: The workflow of SDF: On the left is our process of training diverse
experts through reverse knowledge distillation. On the right is the process of
mining the most informative samples in the target domain using diverse
experts. Below is our process for generating labels for these most informative
samples, which ultimately participate in the training.

## proposed method

### Model Always with Ambiguity

Consider a specifically situation, we train a model $f$ that takes input
feature of every single frame as input $x\in\mathcal{X}$ and predicts its
corresponding label $y\in\mathcal{Y}$. We train $f$ using a labeled dataset
$DS=\\{(x_{1},y_{1}),(x_{2},y_{2}),\ldots\\}$ drawn from the data distribution
$p_{S}(x,y)$. The model $f$ learns to adjust its parameters by approximately
minimizing the predictive risk $E_{p_{S}}(x,y)\left[\lVert
f(x)-y\rVert\right]$ on the source data distribution $p_{S}(x,y)$, aiming to
reduce its prediction error for the training dataset labels, thereby achieving
the required performance on the current data distribution. The collection of
parameters of the model that learns under such conditions above is defined as
$\mathcal{F}$, which is also considered as the model’s set of interpretations
for the source domain.Then each $f\in\mathcal{F}$ is evaluated based on its
predictive risk on held-out samples from $p_{S}(x,y)$.

However, this approach can be less than ideal, leading to model failures when
there is a shift in the data distribution, The standard paradigm of empirical
risk minimization performs poorly on cross-domain tasks, as it tends to select
solutions based on the most salient features (i.e., directions in the
parameter space where the loss decreases most rapidly) without considering
other possibilities. This simplicity bias occurs even when training an
ensemble, as machine learning models still tends to choose the easy way to
learn. This lead to that methods concentrate exclusively on one solution,
ignoring the possibility of multiple interpretations. Since the prominent
features in the source domain are not suitable for the target domain, such
decisions frequently lead to failure. Although many recent methods have
improved robustness in distribution shift settings, standard procedures such
as regularization and cross-validation , encourage this generalization, they
fail in the presence of severe under-specification.

In many distribution shift scenarios (Koh et al. 2021), the overall data
distribution can be modeled as a mixture of domains, where each domain
$d\in\mathcal{D}$ corresponds to a fixed data distribution $p_{d}(x,y)$. Even
if a model $f$ generalizes to unseen data sampled from the same distribution
$p_{S}(x,y)$, performance often deteriorates under distribution shift
conditions, when evaluated on target data sampled from a different
distribution $p_{T}(x,y)$.Datasets frequently lacking specific details,
implying that several reasonable assumptions could explain the known data
distribution just as effectively, with the available information not providing
adequate evidence to favor one assumption over the others. This uncertainty
results in machine learning models arbitrarily choosing one potential
explanation for the presented dataset within the parameter space, devoid of
any external guidance. These choices might not be ideal, leading to the
models’ underperformance upon shifts in data distribution, a situation often
encountered in practical applications. This occurs because datasets often lack
detailed information, allowing for multiple reasonable interpretations that
can explain the data equally well. In such instances, the data does not offer
extra evidence to select the correct interpretation over others, thereby
failing to resolve this ambiguity.

### Comprehensive Coverage

We have specific expectations regarding the model’s fit to the dataset. Under
the condition that this anticipated performance is met, all combinations of
parameters adopted by the model in the parameter space constitutes the model’s
complete interpretation of the dataset.In this discussion, we define a
critical concept within the supervised learning framework: the best model Set:
Let $p(x,y)$ be a distribution defined over $X\times Y$, and $\mathcal{F}$ be
the set of models achieving the expectation under $\epsilon$, where the
$\epsilon$-optimal represents the parameter sets of these models. The loss
function $L_{p}:\mathcal{F}\rightarrow\mathbb{R}$ is defined as the risk
associated with the distribution $p(x,y)$. The set $\mathcal{F}$ is defined as

$\mathcal{F}_{\text{expectation}}=\\{f\in\mathcal{F}\mid
L_{p}(f)\leq\epsilon\\}$

Put differently, the $\epsilon$-optimal set encompasses parameter sets of the
model that can achieve expected performance within the distribution $p(x,y)$.
The parameter $\epsilon$ is utilized to modulate the degree of fitting, with a
preference for not employing excessively large values of $\epsilon$. It is
crucial to note that a model’s predictions on samples from
$p_{S}(x,y)$—whether it be the training set $DS$ or an independent validation
collection—cannot serve to distinguish between different nearly optimal
functions with respect to $p_{S}(x,y)$. This is due to the definition that,
for a given expectation value $\epsilon$, the performance of any two models
$f_{1},f_{2}\in\mathcal{F}_{\epsilon}$ on $p_{S}(x,y)$ is nearly identical.
Relying solely on source data, we lack sufficient grounds to favor any
specific interpretation within $\mathcal{F}_{\epsilon}$. Therefore, our belief
system should aim to comprehensively cover $\mathcal{F}_{\epsilon}$, assigning
non-zero significance to a variety of parameter combinations that represent
different interpretative angles. This approach aligns with existing principles
for reasoning under uncertainty, such as the maximum entropy principle (Keynes
2013; Jaynes 1957), advocating for evenly distributing one’s beliefs across
all possibilities consistent with the available information in the absence of
complete knowledge.

### Training Diverse Experts

In most practical situations, rather than meticulously selecting a neural
network, the quality of data is more vital. High-quality data signifies the
provision of the richest information possible to a deep learning model. In
cross-domain tasks, the quality is a relative concept. If the training data
can encompass as much information from the target domain as possible, in
comparison to the current data domain, the model will be able to achieve an
effective fit and transition to the target domain. Specifically, when a source
dataset lacks certain information, the optimal solution is to provide data
containing this missing information rather than supplying a large volume of
well-distributed scientific data. Although such evenly distributed data
includes the information absent in the source domain, it can lead to an
imbalance between samples that should be learned and those that are
unnecessary for deep learning models. Consequently, models may become
overwhelmed by the abundance of extraneous information, resulting in poor
learning outcomes on the scant missing information. When we want to obtain the
most effective information in the target domain relative to the source domain,
we do not directly analyze the target domain, but instead comprehensively
assess the information already existing in the source domain. We cover the
explanations on the source domain as comprehensively as possible, and obtain a
set of models, $f_{1},f_{2},...,f_{n}$ , we ensure they reach the optimal set
$F_{\epsilon}$ on $p_{S}(x,y)$, while also ensuring that
$f_{1},f_{2},...,f_{n}$ on $p_{S}(x,y)$ avoids redundancy. To identify these
pieces of information, we introduce the concept of symmetric difference, which
is the result of the union of information from the source and target domains
minus their intersection. The symmetric difference in our task is defined by
the equation:

$(f_{1},f_{2},...,f_{n})\,\triangle\,\,Target\,\,Domain$

where $\triangle$ represents calculating the symmetric difference. In this
experiment, we primarily focus on the symmetric difference found within the
target domain.

Sample Evaluation Based on the Diverse Hypothesis Collection: This component
utilizes the hypothesis collection to assess which samples are crucial in
reducing ambiguity during the model’s learning process. It entails identifying
samples that can most effectively aid the model in distinguishing between
these hypotheses, ensuring that the model maintains a focus on universally
effective features during the transition from the source domain to the target
domain, while avoiding overfitting to domain-specific salient features. When
all data participates in the training and the model optimizes to meet an
expected $\epsilon$ via the loss function, this collection of parameters of
the model can be considered as the model’s interpretation of the source. In
neural networks, the final stage involves passing a latent vector
$h^{\textit{now}}$through a fully connected layer, which is precisely the best
mapping of these parameters. Ultimately, we propose a method based on the
multi-perspective collection $f_{1},f_{2},...,f_{n}$ derived from the original
dataset, utilizing this collection to assess which samples should be
introduced to eliminate the current ambiguities. This method aims to guide the
model in summarizing what it has learned from various perspectives and
explicitly identifies what the model needs to learn in the target domain.

First, without introducing constraints, we train the first expert:

$L=\frac{1}{N}\sum_{j=1}^{N}\left(\sum_{i=1}^{M}-\left[y_{(j,i)}\log(p_{(j,i)})+(1-y_{(j,i)})\log(1-p_{(j,i)})\right]\right)$
(1)

where$N$ is the total number of samples and$M$denotes the total number of
frames within each sample. $y_{(j,i)}$ is the true label of the $i^{th}$ frame
in the $j^{th}$ sample, where $y_{(j,i)}\in\\{0,1\\}$. $p_{(j,i)}$ is the
probability predicted by the model that the $i^{th}$ frame in the $j^{th}$
sample belongs to the positive class.$\log$ denotes the natural logarithm.

After this, the training process, built upon the foundation of existing
experts, introduces constraints with the objective of ensuring that the new
interpretations sought by the neural network model in the parameter space are
as distinct as possible from the existing interpretations. This is achieved by
ensuring that the latent vector $h^{\textit{now}}$ , which is processed by the
neural network right before its final layer, is as dissimilar as possible from
the latent vectors $h^{\textit{pre}}$ of previous experts. In other words, it
aims to make the current expert’s interpretation of a given input data as
different as possible from those of the existing experts. To facilitate this
process, we introduce an inverse reverse distillation loss below:

$L_{\textit{Dis}}=\frac{1}{n}\sum_{w=0}^{n}y_{o}\cdot\frac{1}{2}\max(0,\cos(h^{\textit{now}},h^{\textit{pre}}_{w})-u)^{2}$
(2)

where, $h^{\textit{now}}$ represents a latent vector before being fed into the
Fully Connected(FC) Layer layer, belonging to the current training process,
while $h^{\textit{pre}}$ represents the latent vector of a previous expert.
$y_{0}$ indicates whether $h^{\textit{now}}$ and $h^{\textit{pre}}_{w}$ are
dissimilar , $\cos(h^{\textit{now}},h^{\textit{pre}}_{w})$ is the cosine
similarity between $h^{\textit{now}}$ and $h^{\textit{pre}}_{w}$, $u$ is the
margin, a threshold beyond which the loss for dissimilar pairs starts
increasing.

in the Given two vectors $h^{\textit{now}}$ and $h^{\textit{pre}}_{w}$, the
cosine similarity between them can be calculated as:

$\cos(h^{\textit{now}},h^{\textit{pre}}_{w})=\frac{\sum_{k=1}^{d}h^{\textit{now}}_{k}h^{\textit{pre}}_{w,k}}{\sqrt{\sum_{k=1}^{d}(h^{\textit{now}}_{k})^{2}}\cdot\sqrt{\sum_{k=1}^{d}(h^{\textit{pre}}_{w,k})^{2}}}$
(3)

where $d$ is the dimensionality of the vectors, $h^{\textit{now}}_{k}$ is the
$k$-th component of the $h^{\textit{now}}$ vector, $h^{\textit{pre}}_{w,k}$ is
the $k$-th component of the $h^{\textit{pre}}_{w}$ vector.

And the total loss function is updated as follows:

$L=\frac{1}{N}\sum_{j=1}^{N}\left(\sum_{i=1}^{M}\left[-y_{(j,i)}\log(p_{(j,i)})+(1-y_{(j,i)})\log(1-p_{(j,i)})\right]+L_{\text{Dis}}\right)$
(4)

### Selecting The Most Informative Samples

Ultimately, we have acquired a series of well-performing, diverse experts,
which represent interpretations from multiple perspectives of the original
domain. Through this array of experts, we assess the informational content of
the available data within the target domain, with the aim of identifying
samples with the highest amount of information. Based on the information
entropy, the total information quantity $I$ of a single sample, which consists
of $M$ frames, can be calculated using the following formula:

$I_{j}=\sum_{i=1}^{M}\left(-P_{i}\log_{2}(P_{i})-(1-P_{i})\log_{2}(1-P_{i})\right)$
(5)

$P_{i}$ is the prediction result of $n$ experts, which represents the
probability of the $i$-th frame being predicted as manipulated.

$P_{i}=\frac{m}{n}$ (6)

where $P_{i}$ represents the probability, calculated as the number of experts
($n$) predicting the frame as manipulated divided by the total number of
experts ($n$)

After calculating the information quantity, we sort the samples based on their
entropy levels. For the required number of samples, we select them from the
highest to lowest information quantity.

### Unsupervised Training

In the processing unlabeled samples from the target dataset for voice activity
detection, When two audio segments are concatenated, the distinct differences
in their amplitude levels, phases, or waveform shapes often result in abrupt
transitions at the juncture. These abrupt transitions introduce instantaneous
high-frequency components into the signal, leading to rapid changes in energy.
Thus, our method concentrates on the absolute energy change from one frame to
its predecessor and then selects the location with the highest energy change
as the cutting point.

$E=\sum_{i=0}^{N-1}|x[i]|^{2}$ (7) $\Delta E[i]=|E[i]-E[i-1]|$ (8)

Algorithm 1 Samples mining with Diversity and Entropy

Input: Training data from different datasets, $h^{\text{now}}$ (the hidden-
vector of the current expert), $h^{\text{pre}}$ (the hidden-vector of the
previous expert)
Parameter: Loss function $L$, parameter $u$ is set to (0,1), $n$ (the number
of existing experts), $z$ (the number introduced from target dataset), $R$
(the number of all samples in target dataset)
Output: Updated dataset with labeled target samples

1: for every sample $x_{(j,i)}$ in source dataset do

2: for every frame $i$ do

3: $L(j,i)=y_{(j,i)}(\log(p(j,i)))+(1-y_{(j,i)})\log(1-p_{(j,i)})$

4: end for

5: if $n=0$ then

6: $L=\sum_{j}\sum_{i}L(j,i)$

7: else

8: for every expert $w$ do

9:
$\cos(h^{\textit{now}}(j),h^{\textit{pre}}_{w}(j))=\frac{\sum_{k=1}^{d}h^{\textit{now}}_{k}h^{\textit{pre}}_{w,k}}{\sqrt{\sum_{k=1}^{d}(h^{\textit{now}}_{k})^{2}}\cdot\sqrt{\sum_{k=1}^{d}(h^{\textit{pre}}_{w,k})^{2}}}$

10:
$L_{\text{Dis}}(j)=\frac{1}{2n}\max(0,\cos(h^{\text{now}},h^{\text{pre}}_{w})-u)^{2}$

11: end for

12: $L=\sum_{j}\sum_{i}L(j,i)+\sum_{j}L_{\text{Dis}}(j)$

13: end if

14: end for

15: for each sample $x_{(j)}$ do

16: for each frame $f_{i}$ do

17: $P_{i}=\frac{m}{n}$ {where $m$ is the number of experts predicting the
frame as manipulated}

18:
$I_{j}\leftarrow-\sum_{i}[P_{i}\log_{2}(P_{i})+(1-P_{i})\log_{2}(1-P_{i})]$

19: end for

20: end for

21: $H=\text{Sort}(I_{1},I_{2},I_{3},\ldots,I_{R})$ {$H[0]\geq
H[1]\geq\ldots\geq H[R-1]$}

22: $H=H[:z]$

23: for every sample $x{(i)}\in H$ do

24: for every frame $f_{i}$ do

25: $E\leftarrow\sum_{i=0}^{N-1}|x[i]|^{2}$

26: $\Delta E[i]\leftarrow|E[i]-E[i-1]|$

27: end for

28: end for

29: Randomly select two groups of start and end

30: Exchange their places and form labels based on the exchanged regions

31: Introduce these labeled target samples to source dataset

where $x[i]$ represents the amplitude of the $i$th frame, and $N$ is the total
number of frames in the samples. This step helps us assess the signal’s
strength, providing a basis for subsequent analysis.

Next, we calculate the zero crossing rate which is an important metric for
assessing changes in signal frequency. By calculating the absolute change in
energy $\Delta E[i]$ between adjacent frames and zero crossing rate, we can
more accurately determine the start and end points of voice activity. During
silent segments, the energy change is minimal; however, at the onset or
conclusion of speech, the energy change is significantly larger. Therefore, by
randomly selecting points where the energy change exceeds a threshold as the
cutting points for active voice segments, and then randomly selecting the
start and end points of two segments from these threshold-exceeding points,
followed by swapping these two segments, we can label the entire audio. Areas
without exchanges are labeled as 1, while exchanged areas are labeled as 0.

## experiment

### Baseline and Dataset

#### Linear Frequency Cepstral Coefficients

LFCC is an acoustic feature extraction technique used in speech processing
that effectively represents the features of speech signals.

#### Light Convolution Neural Network

LCNN consists of convolutions, max-feature-map (MFM) modules, a pooling layer,
and a linear layer. The MFM is a module that only retains the maximum value of
the two channels to reduce CNN architecture. A convolution and an MFM module
are combined to form a group. Several groups are arranged consecutively to
encode the input feature, and then the encoded feature is pooled and sent to
the linear layer to obtain scores. More details about LCNN can be found in
(Lavrentyeva et al. 2017).

#### PAF dataset

In our experiments, we use Half-truth Audio Detection (HAD) dataset (Yi et al.
2021) and ADD2023track2 dataset.The HAD is generated based on the AISHELL-3
corpus (Shi et al. 2020) which consists of 88035 utterances about 85 hours
from 218 native Chinese mandarin speakers. The authors use fake clips
generated by TTS and vocoder to replace named entities or attitudinal words to
generate fake audios. The HAD dataset includes 50000 audios in the training
set, 17823 audios in the validation set, and 50092 audios in the test set.

For the ADD2023track2 dataset, by providing a mix of genuine and manipulated
audio segments,its test set contains genuine and synthetic voices from six
different datasets, including Ai-shell3 Ai-shell1 etc. The ADD test set
includes 53092 audios.The training and validation sets of it are adopted from
those of HAD.

### Training Detail and System Setting

The signals are divided into frames with a frame length of 10ms to extract
LFCC features, and the overlap between adjacent frames is 10ms. To control
variables, all subsequent experimental comparisons will be based on LFCC as
the feature input. The Adam optimizer, with a learning rate of 0.0001 is used
with a batch size of 16 utterances. Note that a segment with 750 frames will
be intercepted from the feature if it has more frames than 750 while smaller
ones are zero-padded to match the size of the longest utterance in the batch.

During the training process, the parameter $u$ of the reverse distillation
loss, which represents the interval between the ten experts, was set to a
value between 0 and 1. And the parameter$y_{0}$ was set to 1. And the
parameter$n$ which is the number of experts was set to 10. Instead of relying
on the Equal Error Rate (EER) for evaluation, Track 2 incorporates multiple
metrics to calculate the final score of a submission. These metrics include
sentence accuracy ($A$) and segment $F1$ score, which is calculated by recall
and precision. More details can be found in(Yi et al. 2023). All of the
experiment are training on HAD training set but testing on ADD2023 test
set,Using LFCC as the base model, the parameter $u$ of $L_{\textit{Dis}}$ is
set to 0.75.

### Fails on Cross-domain Dataset

Table 1: The F1scores (in %) for the SDF and other models, which are trained using the HAD training set, and test on the HAD training set, HAD validation set, and HAD test set. | HAD dev | HAD test | ADD test
---|---|---|---
| Avg. | Best | Avg. | Best | Avg. | Best
RSDM | 98.26 | 99.17 | 96.01 | 97.76 | 04.74 | 09.19
ADD2023baseline | 93.15 | 94.83 | 92.03 | 93.71 | 08.38 | 12.13
SDE (ours) | 96.00 | 96.25 | 87.71 | 88.81 | 43.84 | 47.24
Table 2: Ablation experiment results for the SDF model by introducing 5000 samples from target dataset, trained using the HAD training dataset and tested on the ADD datatest set. Model | F1score(%) | Recall(%) | Precision(%)
---|---|---|---
SDF | 43.84 | 49.12 | 39.58
w/o - diverse experts | 23.59 | 29.31 | 19.73
w/o - evaluating entropy | 11.23 | 9.50 | 13.73
w/o - labeled samples | 15.53 | 18.04 | 13.63

We have evaluated our method with two currently available models for locating
manipulated regions on in-domain datasets, which have already published their
code. RDASDM refers to “a Robust Deep Audio Splicing Detection Method via
Singularity Detection Feature”(Zhang et al. 2022a), and The ADD2023baseline
originates from (Yi et al. 2023). All models utilize the LCNN architecture as
their foundation. We set the shuffle parameter of the dataloader function to
true, conducted two experiments on each of the three models, and reported both
the average and the best values. However, the RDASDM model differs in that it
incorporates wavelet features and Long Short-Term Memory model, making its
structure more deep and complex.

Table 3: The average and best F1score (%) of various mining methods at different data sizes, trained on the HAD training set and tested on the ADD 2023 test set. | 2500 | 5000 | 7500 | 10000 | 20000
---|---|---|---|---|---
Method | Avg. | Best | Avg. | Best | Avg. | Best | Avg. | Best | Avg. | Best
Undersampling | 11.07 | 13.02 | 14.34 | 16.91 | 18.07 | 19.84 | 18.64 | 21.82 | 40.68 | 41.04
Oversampling | 14.07 | 14.69 | 16.14 | 16.67 | 19.07 | 20.78 | 23.64 | 24.40 | 42.68 | 44.98
Negative Mining | 07.54 | 7.65 | 08.41 | 8.63 | 10.57 | 11.07 | 14.93 | 15.11 | 36.82 | 37.45
Multi-Cluster | 23.39 | 24.17 | 24.74 | 26.44 | 32.02 | 33.36 | 32.16 | 33.17 | 49.48 | 55.25
Random | 10.44 | 18.01 | 13.11 | 22.29 | 18.56 | 27.25 | 23.54 | 33.79 | 37.22 | 50.44
SDE (ours) | 28.33 |  29.21 | 43.84 |  43.98 | 51.64 |  51.73 | 56.99 |  57.77 | 67.68 | 69.92

From Table 1, it is evident that when faced with cross-domain datasets two of
the models failed. Additionally, our model with a simple structure shows
slightly poorer fitting on the source dataset than the other two models. We
hypothesize that the probable reason for this is that our network architecture
is not as deep and our neural network is shorter than those of the other two
models.

It’s noteworthy that the F1score is an indicator of model performance and
represents the model’s fit. In in-domain tasks, a higher F1score generally
indicates better performance. However, for cross-domain tasks, a higher
performance in the source domain could potentially lead to failures in the
target domain.

Figure 3: The left side shows the average entropy size for all samples of the
target dataset with different parameter settings of u, and the right side
shows the F1score results for the corresponding u parameter settings when
introducing 5000 samples from target dataset.

### Ablation Experiment

In the ablation experiment, we evaluated the contribution of each method
within the SDF model. The ”without diverse experts” configuration was set to
randomly shuffle the training data without adding any other constraints. The
”without evaluating entropy” configuration was set to randomly introduce
samples from the target dataset. The ”without labeled samples” configuration
was set to not detect voice activity or swap segments, and to train using
randomly generated labels for each introduced sample. The results are shown in
Table 2. When the diverse experts were removed from our method, performance
dropped by 46%. The main reason for this decline is the absence of experts to
assist in sample selection, which hinders the effective selection of samples.
However, due to the shuffling of the dataset, models optimized from different
starting points may still retain some capability. In fact, most of the samples
we selected were those with a prediction probability $P=\frac{1}{10}$. Which
means that at this point, most of the experts are homogeneous. In the scenario
without evaluating entropy, we were only able to select samples randomly.
Consequently, the final experimental results showed a significant decline in
performance, amounting to a 74% drop. when random labels were used,
performance declined by 64%. We speculate that this could be because random
labeling may lead to the misclassification of boundary frames that the model
should learn, thereby causing a decrease in model performance.

### Comparison of Methods for Mining Samples

In the study, we introduce a series of experiments to evaluate the
effectiveness of SDE. We take LFCC as input, use LCNN as the base model, train
on HAD training set but test on ADD2023 test set and employ the F1score as the
evaluation metric. We first experimented with the performance of various
sample mining methods when a certain number of target dataset samples were
introduced. Among them, Multi-Cluster employs the K-means clustering method to
analyze the existing samples in the target dataset. It selects samples based
on their proximity to the cluster centers, with the number of clusters set at
6 in this case. Undersampling, using the same analysis method as Multi-
Cluster, randomly selects samples from sparsely distributed areas. Conversely,
Oversampling also based on the same analysis as Multi-Cluster, randomly
selects samples from densely populated areas. Negative Mining, on the other
hand, intentionally selects samples that the model consistently misclassifies
with the same outcome, representing the samples with the lowest information
entropy. It is noticed that, in order to make the experimental results more
objective, we conducted two or more experiments for each method and recorded
the outcomes. Specifically, we performed five experiments using the random
method. For our method, SDE, we conducted two reverse distillation processes.
Since we set up shuffling in the dataloader, we can consider the experts
generated in the two iterations to be different. The other methods were tested
three times each.

Under-sampling mitigates class imbalance by reducing the number of examples
from the majority class, which simplifies the model training process and
potentially diminishes the risk of overfitting. We have employed under-
sampling specifically on the majority class within the target dataset.
Similarly, multi-cluster sampling ensures diversity in the sampling process
and enhances the model’s adaptability to different data distributions by
partitioning the dataset into several clusters and sampling from each cluster.
We hold great expectations for these two methods. The results, surprisingly,
are clearly evident in Table 2: neither method managed to select high-quality
samples, as seen from the F1score outcomes. The results from random selection
met expectations, showing that the F1score increase is nearly proportional to
the number of samples added.

Furthermore, as indicators based on information entropy, two completely
opposite methods, Negative Mining and our method SDE, demonstrate the
effectiveness of entropy-based sample selection strategies. Firstly, when a
limited number of samples are introduced, the outcomes of our method surpass
those of Negative Mining. Additionally, the data in Table 2 clearly shows that
for our method, the slope of the performance improvement line for the top 10%
of samples is steeper than that for the top 20% of samples. This suggests that
the samples in the top 10%, which have higher information entropy, are of
higher quality than those between 10% and 20%. Our method significantly
improves model performance by addressing the information disparity between the
source and target domains. By selectively choosing samples that offer more
information, we enhance the source domain model’s comprehension when applied
to the target domain. Two key factors contribute to the success of our
approach. First, we employ reverse distillation to minimize shared information
among different experts. This process compels experts to concentrate on
secondary distribution information within the source dataset, enabling them to
interpret the dataset from diverse perspectives. Secondly, by selecting
samples from the target dataset that possess maximum information entropy
relative to the source dataset, we effectively reduce the model’s ambiguity.
Together, these strategies optimize the transfer of relevant information
across domains, leading to notable improvements in model accuracy and
robustness.

### Entropy and Information

To demonstrate the correlation between information entropy and performance, by
setting different values of parameter$u$, we learned multiple sets of experts
with significant viewpoint differences in the source dataset. These experts
evaluated the information entropy of the existing target samples, with the
average results shown in Figure 3. Subsequently, we used these samples with
varying information entropy levels for evaluation, and the experimental
results are presented in Figure 3. The parameter $u$ serves as a constraint on
the minimum distance between latent vectors, essentially acting as a distance
limitation among experts. A sufficiently large and appropriate $u$ value
ensures that the experts interpret from multiple distinct perspectives within
the source dataset, rather than from similar angles. Furthermore, more
importantly, as the average information entropy increases, the F1score also
rises, indicating that the amount of information entropy in the introduced
samples directly impacts the model’s performance. This relationship
underscores the significance of selecting high-entropy samples to enhance the
model’s ability to generalize and perform effectively. We subsequently
evaluated the diversity among the ten experts trained with the $L_{D}is$
parameter $u$ set to 0.75. This evaluation was conducted by calculating the
dissimilarity in their predictions for the samples. The results shown in
Figure 4 indicate that the initial training stage, where the divergence among
experts was the greatest, resulted in the best performance. The clear initial
differentiation among experts enables more effective exploration of diverse
strategies and insights, leading to improved overall model performance. As the
number of experts increases, the parameter space becomes more complex with
multiple divergence points. This complexity can lead to potential conflicts in
optimization directions. Consequently, experts trained later might not diverge
sufficiently from the perspectives of the original experts, leading to
homogenization.

Figure 4: The dissimilarity of predictions among experts

## Conclusion

In this work, we propose an unsupervised domain adaptation method called SDE
to overcome out-of-distribution challenges in the task of partially fake
audio. This method introduces the most informative samples by calculating
their entropy relative to the original domain. Additionally, we have designed
a training method for unlabeled samples for the task of locating manipulation
regions in partially fake audio. The specific operation involves selecting
segmentation points based on the rate of voice activity changes to perform
segment swapping and label generation. The generated labels and the swapped
segment samples are then used for training in the source domain.Our results
indicate that our method outperforms five other sample selection techniques in
domain adaptation for the specific task of identifying manipulation regions in
partially fake audio. In this context, by incorporating the top 10% of samples
from the target domain, we achieved an F1 score of 43.84%. Compared to the
performance of the second-best method, this score represents a relative
enhancement of 77.2%. Additionally, the outcomes of the Ablation Experiment
elucidate the distinct contributions of each phase within the SDF methodology,
thereby providing indirect substantiation of our method’s efficacy. We propose
a method based on entropy for mining the most relatively abundant information,
which has a very broad range of applications. The most direct application
scenario is the localization of tampered regions in images, which will be the
focus of our future experiments.

## References

  * Baevski et al. (2020) Baevski, A.; Zhou, Y.; Mohamed, A.; and Auli, M. 2020.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  _Advances in neural information processing systems_ , 33: 12449–12460.
  * Cai and Li (2024) Cai, Z.; and Li, M. 2024.  Integrating frame-level boundary detection and deepfake detection for locating manipulated regions in partially spoofed audio forgery attacks.  _Computer Speech & Language_, 85: 101597.
  * Cai, Wang, and Li (2023) Cai, Z.; Wang, W.; and Li, M. 2023.  Waveform boundary detection for partially spoofed audio.  In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 1–5. IEEE.
  * Chan et al. (2022) Chan, C. H.; Qian, K.; Zhang, Y.; and Hasegawa-Johnson, M. 2022.  Speechsplit2. 0: Unsupervised speech disentanglement for voice conversion without tuning autoencoder bottlenecks.  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 6332–6336. IEEE.
  * Chen et al. (2024) Chen, Y.; Yi, J.; Xue, J.; Wang, C.; Zhang, X.; Dong, S.; Zeng, S.; Tao, J.; Zhao, L.; and Fan, C. 2024.  RawBMamba: End-to-End Bidirectional State Space Model for Audio Deepfake Detection.  _arXiv preprint arXiv:2406.06086_.
  * Chen et al. (2021) Chen, Y.-H.; Wu, D.-Y.; Wu, T.-H.; and Lee, H.-y. 2021.  Again-vc: A one-shot voice conversion using activation guidance and adaptive instance normalization.  In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 5954–5958. IEEE.
  * Desplanques, Thienpondt, and Demuynck (2020) Desplanques, B.; Thienpondt, J.; and Demuynck, K. 2020.  Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification.  _arXiv preprint arXiv:2005.07143_.
  * Huang et al. (2022) Huang, R.; Zhao, Z.; Liu, H.; Liu, J.; Cui, C.; and Ren, Y. 2022.  Prodiff: Progressive fast diffusion model for high-quality text-to-speech.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , 2595–2605.
  * Jaynes (1957) Jaynes, E. T. 1957.  Information theory and statistical mechanics.  _Physical review_ , 106(4): 620.
  * Keynes (2013) Keynes, J. M. 2013.  _A treatise on probability_.  Courier Corporation.
  * Khanjani, Watson, and Janeja (2023) Khanjani, Z.; Watson, G.; and Janeja, V. P. 2023.  Audio deepfakes: A survey.  _Frontiers in Big Data_ , 5: 1001063.
  * Kinnunen et al. (2018) Kinnunen, T.; Lorenzo-Trueba, J.; Yamagishi, J.; Toda, T.; Saito, D.; Villavicencio, F.; and Ling, Z. 2018.  A spoofing benchmark for the 2018 voice conversion challenge: Leveraging from spoofing countermeasures for speech artifact assessment.  _arXiv preprint arXiv:1804.08438_.
  * Koh et al. (2021) Koh, P. W.; Sagawa, S.; Marklund, H.; Xie, S. M.; Zhang, M.; Balsubramani, A.; Hu, W.; Yasunaga, M.; Phillips, R. L.; Gao, I.; et al. 2021.  Wilds: A benchmark of in-the-wild distribution shifts.  In _International conference on machine learning_ , 5637–5664. PMLR.
  * Lavrentyeva et al. (2017) Lavrentyeva, G.; Novoselov, S.; Malykh, E.; Kozlov, A.; Kudashev, O.; and Shchemelinin, V. 2017.  Audio replay attack detection with deep learning frameworks.  In _Interspeech_ , 82–86.
  * Li et al. (2023) Li, J.; Li, L.; Luo, M.; Wang, X.; Qiao, S.; and Zhou, Y. 2023.  Multi-grained Backend Fusion for Manipulation Region Location of Partially Fake Audio.  In _Proceedings of IJCAI 2023 Workshop on Deepfake Audio Detection and Analysis_ , volume 755.
  * Liu et al. (2022) Liu, X.; Liu, M.; Zhang, L.; Zhang, L.; Zeng, C.; Li, K.; Li, N.; Lee, K. A.; Wang, L.; and Dang, J. 2022.  Deep spectro-temporal artifacts for detecting synthesized speech.  In _Proceedings of the 1st International Workshop on Deepfake Detection for Audio Multimedia_ , 69–75.
  * Lorenzo-Trueba et al. (2018a) Lorenzo-Trueba, J.; Fang, F.; Wang, X.; Echizen, I.; Yamagishi, J.; and Kinnunen, T. 2018a.  Can we steal your vocal identity from the Internet?: Initial investigation of cloning Obama’s voice using GAN, WaveNet and low-quality found data.  _arXiv preprint arXiv:1803.00860_.
  * Lorenzo-Trueba et al. (2018b) Lorenzo-Trueba, J.; Yamagishi, J.; Toda, T.; Saito, D.; Villavicencio, F.; Kinnunen, T.; and Ling, Z. 2018b.  The voice conversion challenge 2018: Promoting development of parallel and nonparallel methods.  _arXiv preprint arXiv:1804.04262_.
  * Lv et al. (2022) Lv, Z.; Zhang, S.; Tang, K.; and Hu, P. 2022.  Fake audio detection based on unsupervised pretraining models.  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 9231–9235. IEEE.
  * Martín-Doñas and Álvarez (2023) Martín-Doñas, J. M.; and Álvarez, A. 2023.  The Vicomtech partial deepfake detection and location system for the 2023 ADD Challenge.  In _Proceedings of IJCAI 2023 Workshop on Deepfake Audio Detection and Analysis_.
  * Patel and Patil (2015) Patel, T. B.; and Patil, H. A. 2015.  Combining evidences from mel cepstral, cochlear filter cepstral and instantaneous frequency features for detection of natural vs. spoofed speech.  In _Interspeech_ , 2062–2066.
  * Shi et al. (2020) Shi, Y.; Bu, H.; Xu, X.; Zhang, S.; and Li, M. 2020.  Aishell-3: A multi-speaker mandarin tts corpus and the baselines.  _arXiv preprint arXiv:2010.11567_.
  * Tan et al. (2024) Tan, X.; Chen, J.; Liu, H.; Cong, J.; Zhang, C.; Liu, Y.; Wang, X.; Leng, Y.; Yi, Y.; He, L.; et al. 2024.  Naturalspeech: End-to-end text-to-speech synthesis with human-level quality.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_.
  * Wang et al. (2023a) Wang, C.; Chen, S.; Wu, Y.; Zhang, Z.; Zhou, L.; Liu, S.; Chen, Z.; Liu, Y.; Wang, H.; Li, J.; et al. 2023a.  Neural codec language models are zero-shot text to speech synthesizers.  _arXiv preprint arXiv:2301.02111_.
  * Wang et al. (2024) Wang, C.; He, J.; Yi, J.; Tao, J.; Zhang, C. Y.; and Zhang, X. 2024.  Multi-Scale Permutation Entropy for Audio Deepfake Detection.  In _ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 1406–1410. IEEE.
  * Wang et al. (2023b) Wang, C.; Yi, J.; Zhang, X.; Tao, J.; Xu, L.; and Fu, R. 2023b.  Low-rank adaptation method for wav2vec2-based fake audio detection.  _arXiv preprint arXiv:2306.05617_.
  * Wu et al. (2022) Wu, H.; Kuo, H.-C.; Zheng, N.; Hung, K.-H.; Lee, H.-Y.; Tsao, Y.; Wang, H.-M.; and Meng, H. 2022.  Partially fake audio detection by self-attention-based fake span discovery.  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 9236–9240. IEEE.
  * Xie et al. (2024) Xie, Y.; Cheng, H.; Wang, Y.; and Ye, L. 2024.  An Efficient Temporary Deepfake Location Approach Based Embeddings for Partially Spoofed Audio Detection.  In _ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 966–970. IEEE.
  * Yi et al. (2021) Yi, J.; Bai, Y.; Tao, J.; Ma, H.; Tian, Z.; Wang, C.; Wang, T.; and Fu, R. 2021.  Half-truth: A partially fake audio detection dataset.  _arXiv preprint arXiv:2104.03617_.
  * Yi et al. (2022) Yi, J.; Fu, R.; Tao, J.; Nie, S.; Ma, H.; Wang, C.; Wang, T.; Tian, Z.; Bai, Y.; Fan, C.; et al. 2022.  Add 2022: the first audio deep synthesis detection challenge.  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 9216–9220. IEEE.
  * Yi et al. (2023) Yi, J.; Tao, J.; Fu, R.; Yan, X.; Wang, C.; Wang, T.; Zhang, C. Y.; Zhang, X.; Zhao, Y.; Ren, Y.; et al. 2023.  Add 2023: the second audio deepfake detection challenge.  _arXiv preprint arXiv:2305.13774_.
  * Zhang et al. (2022a) Zhang, K.; Liang, S.; Nie, S.; He, S.; Pan, J.; Zhang, X.; Ma, H.; and Yi, J. 2022a.  A Robust Deep Audio Splicing Detection Method via Singularity Detection Feature.  In _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2919–2923.
  * Zhang et al. (2022b) Zhang, L.; Wang, X.; Cooper, E.; Evans, N.; and Yamagishi, J. 2022b.  The partialspoof database and countermeasures for the detection of short fake speech segments embedded in an utterance.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 31: 813–825.
  * Zhang et al. (2021a) Zhang, L.; Wang, X.; Cooper, E.; and Yamagishi, J. 2021a.  Multi-task learning in utterance-level and segmental-level spoof detection.  _arXiv preprint arXiv:2107.14132_.
  * Zhang et al. (2021b) Zhang, L.; Wang, X.; Cooper, E.; Yamagishi, J.; Patino, J.; and Evans, N. 2021b.  An initial investigation for detecting partially spoofed audio.  _arXiv preprint arXiv:2104.02518_.
  * Zhang et al. (2023a) Zhang, X.; Yi, J.; Tao, J.; Wang, C.; Xu, L.; and Fu, R. 2023a.  Adaptive Fake Audio Detection with Low-Rank Model Squeezing.  _arXiv preprint arXiv:2306.04956_.
  * Zhang et al. (2023b) Zhang, X.; Yi, J.; Tao, J.; Wang, C.; and Zhang, C. Y. 2023b.  Do you remember? Overcoming catastrophic forgetting for fake audio detection.  In _International Conference on Machine Learning_ , 41819–41831. PMLR.
  * Zhang et al. (2024) Zhang, X.; Yi, J.; Wang, C.; Zhang, C. Y.; Zeng, S.; and Tao, J. 2024.  What to remember: Self-adaptive continual learning for audio deepfake detection.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 38, 19569–19577.
  * Zhang et al. (2023c) Zhang, X.; Yoon, J.; Bansal, M.; and Yao, H. 2023c.  Multimodal representation learning by alternating unimodal adaptation.  _arXiv preprint arXiv:2311.10707_.
  * Zhao et al. (2022) Zhao, Y.; Yi, J.; Tao, J.; Wang, C.; Zhang, X.; and Dong, Y. 2022.  EmoFake: An initial dataset for emotion fake audio detection.  _arXiv preprint arXiv:2211.05363_.