††thanks: now at CERN††thanks: now at CERN††thanks: also at J-PARC, Tokai,
Japan††thanks: also at J-PARC, Tokai, Japan††thanks: also at J-PARC, Tokai,
Japan††thanks: also at J-PARC, Tokai, Japan††thanks: also at J-PARC, Tokai,
Japan††thanks: affiliated member at Kavli IPMU (WPI), the University of Tokyo,
Japan††thanks: affiliated member at Kavli IPMU (WPI), the University of Tokyo,
Japan††thanks: affiliated member at Kavli IPMU (WPI), the University of Tokyo,
Japan††thanks: also at J-PARC, Tokai, Japan††thanks: affiliated member at
Kavli IPMU (WPI), the University of Tokyo, Japan††thanks: also at National
Research Nuclear University ”MEPhI” and Moscow Institute of Physics and
Technology, Moscow, Russia††thanks: also at J-PARC, Tokai, Japan††thanks:
affiliated member at Kavli IPMU (WPI), the University of Tokyo, Japan††thanks:
affiliated member at Kavli IPMU (WPI), the University of Tokyo, Japan††thanks:
also at J-PARC, Tokai, Japan††thanks: also at J-PARC, Tokai, Japan††thanks:
affiliated member at Kavli IPMU (WPI), the University of Tokyo, Japan††thanks:
also at J-PARC, Tokai, Japan††thanks: deceased††thanks: also at J-PARC, Tokai,
Japan††thanks: also at JINR, Dubna, Russia††thanks: also at J-PARC, Tokai,
Japan††thanks: affiliated member at Kavli IPMU (WPI), the University of Tokyo,
Japan††thanks: also at J-PARC, Tokai, Japan††thanks: affiliated member at
Kavli IPMU (WPI), the University of Tokyo, Japan††thanks: also at J-PARC,
Tokai, Japan††thanks: also at J-PARC, Tokai, Japan††thanks: affiliated member
at Kavli IPMU (WPI), the University of Tokyo, Japan††thanks: also at J-PARC,
Tokai, Japan††thanks: affiliated member at Kavli IPMU (WPI), the University of
Tokyo, Japan††thanks: deceased††thanks: also at BMCC/CUNY, Science Department,
New York, New York, U.S.A.††thanks: affiliated member at Kavli IPMU (WPI), the
University of Tokyo, Japan

The T2K Collaboration

# Search for heavy neutrinos with the T2K near detector ND280

K. Abe  University of Tokyo, Institute for Cosmic Ray Research, Kamioka
Observatory, Kamioka, Japan  R. Akutsu  University of Tokyo, Institute for
Cosmic Ray Research, Research Center for Cosmic Neutrinos, Kashiwa, Japan  A.
Ali  INFN Sezione di Padova and Università di Padova, Dipartimento di Fisica,
Padova, Italy  C. Andreopoulos  STFC, Rutherford Appleton Laboratory, Harwell
Oxford, and Daresbury Laboratory, Warrington, United Kingdom  University of
Liverpool, Department of Physics, Liverpool, United Kingdom  L. Anthony
University of Liverpool, Department of Physics, Liverpool, United Kingdom  M.
Antonova  IFIC (CSIC & University of Valencia), Valencia, Spain  S. Aoki  Kobe
University, Kobe, Japan  A. Ariga  University of Bern, Albert Einstein Center
for Fundamental Physics, Laboratory for High Energy Physics (LHEP), Bern,
Switzerland  Y. Ashida  Kyoto University, Department of Physics, Kyoto, Japan
Y. Awataguchi  Tokyo Metropolitan University, Department of Physics, Tokyo,
Japan  Y. Azuma  Osaka City University, Department of Physics, Osaka, Japan
S. Ban  Kyoto University, Department of Physics, Kyoto, Japan  M. Barbi
University of Regina, Department of Physics, Regina, Saskatchewan, Canada
G.J. Barker  University of Warwick, Department of Physics, Coventry, United
Kingdom  G. Barr  Oxford University, Department of Physics, Oxford, United
Kingdom  C. Barry  University of Liverpool, Department of Physics, Liverpool,
United Kingdom  M. Batkiewicz-Kwasniak  H. Niewodniczanski Institute of
Nuclear Physics PAN, Cracow, Poland  F. Bench  University of Liverpool,
Department of Physics, Liverpool, United Kingdom  V. Berardi  INFN Sezione di
Bari and Università e Politecnico di Bari, Dipartimento Interuniversitario di
Fisica, Bari, Italy  S. Berkman  University of British Columbia, Department of
Physics and Astronomy, Vancouver, British Columbia, Canada  TRIUMF, Vancouver,
British Columbia, Canada  R.M. Berner  University of Bern, Albert Einstein
Center for Fundamental Physics, Laboratory for High Energy Physics (LHEP),
Bern, Switzerland  L. Berns  Tokyo Institute of Technology, Department of
Physics, Tokyo, Japan  S. Bhadra  York University, Department of Physics and
Astronomy, Toronto, Ontario, Canada  S. Bienstock  Sorbonne Université,
Université Paris Diderot, CNRS/IN2P3, Laboratoire de Physique Nucléaire et de
Hautes Energies (LPNHE), Paris, France  A. Blondel  University of Geneva,
Section de Physique, DPNC, Geneva, Switzerland  S. Bolognesi  IRFU, CEA
Saclay, Gif-sur-Yvette, France  B. Bourguille  Institut de Fisica d’Altes
Energies (IFAE), The Barcelona Institute of Science and Technology, Campus
UAB, Bellaterra (Barcelona) Spain  S.B. Boyd  University of Warwick,
Department of Physics, Coventry, United Kingdom  D. Brailsford  Lancaster
University, Physics Department, Lancaster, United Kingdom  A. Bravar
University of Geneva, Section de Physique, DPNC, Geneva, Switzerland  C.
Bronner  University of Tokyo, Institute for Cosmic Ray Research, Kamioka
Observatory, Kamioka, Japan  M. Buizza Avanzini  Ecole Polytechnique,
IN2P3-CNRS, Laboratoire Leprince-Ringuet, Palaiseau, France  J. Calcutt
Michigan State University, Department of Physics and Astronomy, East Lansing,
Michigan, U.S.A.  T. Campbell  University of Colorado at Boulder, Department
of Physics, Boulder, Colorado, U.S.A.  S. Cao  High Energy Accelerator
Research Organization (KEK), Tsukuba, Ibaraki, Japan  S.L. Cartwright
University of Sheffield, Department of Physics and Astronomy, Sheffield,
United Kingdom  M.G. Catanesi  INFN Sezione di Bari and Università e
Politecnico di Bari, Dipartimento Interuniversitario di Fisica, Bari, Italy
A. Cervera  IFIC (CSIC & University of Valencia), Valencia, Spain  A. Chappell
University of Warwick, Department of Physics, Coventry, United Kingdom  C.
Checchia  INFN Sezione di Padova and Università di Padova, Dipartimento di
Fisica, Padova, Italy  D. Cherdack  University of Houston, Department of
Physics, Houston, Texas, U.S.A.  N. Chikuma  University of Tokyo, Department
of Physics, Tokyo, Japan  G. Christodoulou  University of Liverpool,
Department of Physics, Liverpool, United Kingdom  J. Coleman  University of
Liverpool, Department of Physics, Liverpool, United Kingdom  G. Collazuol
INFN Sezione di Padova and Università di Padova, Dipartimento di Fisica,
Padova, Italy  D. Coplowe  Oxford University, Department of Physics, Oxford,
United Kingdom  A. Cudd  Michigan State University, Department of Physics and
Astronomy, East Lansing, Michigan, U.S.A.  A. Dabrowska  H. Niewodniczanski
Institute of Nuclear Physics PAN, Cracow, Poland  G. De Rosa  INFN Sezione di
Napoli and Università di Napoli, Dipartimento di Fisica, Napoli, Italy  T.
Dealtry  Lancaster University, Physics Department, Lancaster, United Kingdom
P.F. Denner  University of Warwick, Department of Physics, Coventry, United
Kingdom  S.R. Dennis  University of Liverpool, Department of Physics,
Liverpool, United Kingdom  C. Densham  STFC, Rutherford Appleton Laboratory,
Harwell Oxford, and Daresbury Laboratory, Warrington, United Kingdom  F. Di
Lodovico  Queen Mary University of London, School of Physics and Astronomy,
London, United Kingdom  N. Dokania  State University of New York at Stony
Brook, Department of Physics and Astronomy, Stony Brook, New York, U.S.A.  S.
Dolan  Ecole Polytechnique, IN2P3-CNRS, Laboratoire Leprince-Ringuet,
Palaiseau, France  IRFU, CEA Saclay, Gif-sur-Yvette, France  O. Drapier  Ecole
Polytechnique, IN2P3-CNRS, Laboratoire Leprince-Ringuet, Palaiseau, France
K.E. Duffy  Oxford University, Department of Physics, Oxford, United Kingdom
J. Dumarchez  Sorbonne Université, Université Paris Diderot, CNRS/IN2P3,
Laboratoire de Physique Nucléaire et de Hautes Energies (LPNHE), Paris, France
P. Dunne  Imperial College London, Department of Physics, London, United
Kingdom  S. Emery-Schrenk  IRFU, CEA Saclay, Gif-sur-Yvette, France  A.
Ereditato  University of Bern, Albert Einstein Center for Fundamental Physics,
Laboratory for High Energy Physics (LHEP), Bern, Switzerland  P. Fernandez
IFIC (CSIC & University of Valencia), Valencia, Spain  T. Feusels  University
of British Columbia, Department of Physics and Astronomy, Vancouver, British
Columbia, Canada  TRIUMF, Vancouver, British Columbia, Canada  A.J. Finch
Lancaster University, Physics Department, Lancaster, United Kingdom  G.A.
Fiorentini  York University, Department of Physics and Astronomy, Toronto,
Ontario, Canada  G. Fiorillo  INFN Sezione di Napoli and Università di Napoli,
Dipartimento di Fisica, Napoli, Italy  C. Francois  University of Bern, Albert
Einstein Center for Fundamental Physics, Laboratory for High Energy Physics
(LHEP), Bern, Switzerland  M. Friend  High Energy Accelerator Research
Organization (KEK), Tsukuba, Ibaraki, Japan  Y. Fujii  High Energy Accelerator
Research Organization (KEK), Tsukuba, Ibaraki, Japan  R. Fujita  University of
Tokyo, Department of Physics, Tokyo, Japan  D. Fukuda  Okayama University,
Department of Physics, Okayama, Japan  Y. Fukuda  Miyagi University of
Education, Department of Physics, Sendai, Japan  K. Gameil  University of
British Columbia, Department of Physics and Astronomy, Vancouver, British
Columbia, Canada  TRIUMF, Vancouver, British Columbia, Canada  C. Giganti
Sorbonne Université, Université Paris Diderot, CNRS/IN2P3, Laboratoire de
Physique Nucléaire et de Hautes Energies (LPNHE), Paris, France  F. Gizzarelli
IRFU, CEA Saclay, Gif-sur-Yvette, France  T. Golan  Wroclaw University,
Faculty of Physics and Astronomy, Wroclaw, Poland  M. Gonin  Ecole
Polytechnique, IN2P3-CNRS, Laboratoire Leprince-Ringuet, Palaiseau, France
D.R. Hadley  University of Warwick, Department of Physics, Coventry, United
Kingdom  L. Haegel  University of Geneva, Section de Physique, DPNC, Geneva,
Switzerland  J.T. Haigh  University of Warwick, Department of Physics,
Coventry, United Kingdom  P. Hamacher-Baumann  RWTH Aachen University, III.
Physikalisches Institut, Aachen, Germany  M. Hartz  TRIUMF, Vancouver, British
Columbia, Canada  Kavli Institute for the Physics and Mathematics of the
Universe (WPI), The University of Tokyo Institutes for Advanced Study,
University of Tokyo, Kashiwa, Chiba, Japan  T. Hasegawa  High Energy
Accelerator Research Organization (KEK), Tsukuba, Ibaraki, Japan  N.C.
Hastings  University of Regina, Department of Physics, Regina, Saskatchewan,
Canada  T. Hayashino  Kyoto University, Department of Physics, Kyoto, Japan
Y. Hayato  University of Tokyo, Institute for Cosmic Ray Research, Kamioka
Observatory, Kamioka, Japan  Kavli Institute for the Physics and Mathematics
of the Universe (WPI), The University of Tokyo Institutes for Advanced Study,
University of Tokyo, Kashiwa, Chiba, Japan  A. Hiramoto  Kyoto University,
Department of Physics, Kyoto, Japan  M. Hogan  Colorado State University,
Department of Physics, Fort Collins, Colorado, U.S.A.  J. Holeczek  University
of Silesia, Institute of Physics, Katowice, Poland  N.T. Hong Van  Institute
For Interdisciplinary Research in Science and Education (IFIRSE), ICISE, Quy
Nhon, Vietnam  Institute of Physics (IOP), Vietnam Academy of Science and
Technology (VAST), Hanoi, Vietnam  F. Hosomi  University of Tokyo, Department
of Physics, Tokyo, Japan  F. Iacob  INFN Sezione di Padova and Università di
Padova, Dipartimento di Fisica, Padova, Italy  A.K. Ichikawa  Kyoto
University, Department of Physics, Kyoto, Japan  M. Ikeda  University of
Tokyo, Institute for Cosmic Ray Research, Kamioka Observatory, Kamioka, Japan
T. Inoue  Osaka City University, Department of Physics, Osaka, Japan  R.A.
Intonti  INFN Sezione di Bari and Università e Politecnico di Bari,
Dipartimento Interuniversitario di Fisica, Bari, Italy  T. Ishida  High Energy
Accelerator Research Organization (KEK), Tsukuba, Ibaraki, Japan  T. Ishii
High Energy Accelerator Research Organization (KEK), Tsukuba, Ibaraki, Japan
M. Ishitsuka  Tokyo University of Science, Faculty of Science and Technology,
Department of Physics, Noda, Chiba, Japan  K. Iwamoto  University of Tokyo,
Department of Physics, Tokyo, Japan  A. Izmaylov  IFIC (CSIC & University of
Valencia), Valencia, Spain  Institute for Nuclear Research of the Russian
Academy of Sciences, Moscow, Russia  B. Jamieson  University of Winnipeg,
Department of Physics, Winnipeg, Manitoba, Canada  C. Jesus  Institut de
Fisica d’Altes Energies (IFAE), The Barcelona Institute of Science and
Technology, Campus UAB, Bellaterra (Barcelona) Spain  M. Jiang  Kyoto
University, Department of Physics, Kyoto, Japan  S. Johnson  University of
Colorado at Boulder, Department of Physics, Boulder, Colorado, U.S.A.  P.
Jonsson  Imperial College London, Department of Physics, London, United
Kingdom  C.K. Jung  State University of New York at Stony Brook, Department of
Physics and Astronomy, Stony Brook, New York, U.S.A.  M. Kabirnezhad  Oxford
University, Department of Physics, Oxford, United Kingdom  A.C. Kaboth  Royal
Holloway University of London, Department of Physics, Egham, Surrey, United
Kingdom  STFC, Rutherford Appleton Laboratory, Harwell Oxford, and Daresbury
Laboratory, Warrington, United Kingdom  T. Kajita  University of Tokyo,
Institute for Cosmic Ray Research, Research Center for Cosmic Neutrinos,
Kashiwa, Japan  H. Kakuno  Tokyo Metropolitan University, Department of
Physics, Tokyo, Japan  J. Kameda  University of Tokyo, Institute for Cosmic
Ray Research, Kamioka Observatory, Kamioka, Japan  D. Karlen  University of
Victoria, Department of Physics and Astronomy, Victoria, British Columbia,
Canada  TRIUMF, Vancouver, British Columbia, Canada  T. Katori  Queen Mary
University of London, School of Physics and Astronomy, London, United Kingdom
Y. Kato  University of Tokyo, Institute for Cosmic Ray Research, Kamioka
Observatory, Kamioka, Japan  E. Kearns  Boston University, Department of
Physics, Boston, Massachusetts, U.S.A.  Kavli Institute for the Physics and
Mathematics of the Universe (WPI), The University of Tokyo Institutes for
Advanced Study, University of Tokyo, Kashiwa, Chiba, Japan  M. Khabibullin
Institute for Nuclear Research of the Russian Academy of Sciences, Moscow,
Russia  A. Khotjantsev  Institute for Nuclear Research of the Russian Academy
of Sciences, Moscow, Russia  H. Kim  Osaka City University, Department of
Physics, Osaka, Japan  J. Kim  University of British Columbia, Department of
Physics and Astronomy, Vancouver, British Columbia, Canada  TRIUMF, Vancouver,
British Columbia, Canada  S. King  Queen Mary University of London, School of
Physics and Astronomy, London, United Kingdom  J. Kisiel  University of
Silesia, Institute of Physics, Katowice, Poland  A. Knight  University of
Warwick, Department of Physics, Coventry, United Kingdom  A. Knox  Lancaster
University, Physics Department, Lancaster, United Kingdom  T. Kobayashi  High
Energy Accelerator Research Organization (KEK), Tsukuba, Ibaraki, Japan  L.
Koch  STFC, Rutherford Appleton Laboratory, Harwell Oxford, and Daresbury
Laboratory, Warrington, United Kingdom  T. Koga  University of Tokyo,
Department of Physics, Tokyo, Japan  P.P. Koller  University of Bern, Albert
Einstein Center for Fundamental Physics, Laboratory for High Energy Physics
(LHEP), Bern, Switzerland  A. Konaka  TRIUMF, Vancouver, British Columbia,
Canada  L.L. Kormos  Lancaster University, Physics Department, Lancaster,
United Kingdom  Y. Koshio  Okayama University, Department of Physics, Okayama,
Japan  K. Kowalik  National Centre for Nuclear Research, Warsaw, Poland  H.
Kubo  Kyoto University, Department of Physics, Kyoto, Japan  Y. Kudenko
Institute for Nuclear Research of the Russian Academy of Sciences, Moscow,
Russia  R. Kurjata  Warsaw University of Technology, Institute of
Radioelectronics, Warsaw, Poland  T. Kutter  Louisiana State University,
Department of Physics and Astronomy, Baton Rouge, Louisiana, U.S.A.  M. Kuze
Tokyo Institute of Technology, Department of Physics, Tokyo, Japan  L. Labarga
University Autonoma Madrid, Department of Theoretical Physics, 28049 Madrid,
Spain  J. Lagoda  National Centre for Nuclear Research, Warsaw, Poland  M.
Lamoureux  IRFU, CEA Saclay, Gif-sur-Yvette, France  P. Lasorak  Queen Mary
University of London, School of Physics and Astronomy, London, United Kingdom
M. Laveder  INFN Sezione di Padova and Università di Padova, Dipartimento di
Fisica, Padova, Italy  M. Lawe  Lancaster University, Physics Department,
Lancaster, United Kingdom  M. Licciardi  Ecole Polytechnique, IN2P3-CNRS,
Laboratoire Leprince-Ringuet, Palaiseau, France  T. Lindner  TRIUMF,
Vancouver, British Columbia, Canada  Z.J. Liptak  University of Colorado at
Boulder, Department of Physics, Boulder, Colorado, U.S.A.  R.P. Litchfield
University of Glasgow, School of Physics and Astronomy, Glasgow, United
Kingdom  X. Li  State University of New York at Stony Brook, Department of
Physics and Astronomy, Stony Brook, New York, U.S.A.  A. Longhin  INFN Sezione
di Padova and Università di Padova, Dipartimento di Fisica, Padova, Italy
J.P. Lopez  University of Colorado at Boulder, Department of Physics, Boulder,
Colorado, U.S.A.  T. Lou  University of Tokyo, Department of Physics, Tokyo,
Japan  L. Ludovici  INFN Sezione di Roma and Università di Roma “La Sapienza”,
Roma, Italy  X. Lu  Oxford University, Department of Physics, Oxford, United
Kingdom  T. Lux  Institut de Fisica d’Altes Energies (IFAE), The Barcelona
Institute of Science and Technology, Campus UAB, Bellaterra (Barcelona) Spain
L. Magaletti  INFN Sezione di Bari and Università e Politecnico di Bari,
Dipartimento Interuniversitario di Fisica, Bari, Italy  K. Mahn  Michigan
State University, Department of Physics and Astronomy, East Lansing, Michigan,
U.S.A.  M. Malek  University of Sheffield, Department of Physics and
Astronomy, Sheffield, United Kingdom  S. Manly  University of Rochester,
Department of Physics and Astronomy, Rochester, New York, U.S.A.  L. Maret
University of Geneva, Section de Physique, DPNC, Geneva, Switzerland  A.D.
Marino  University of Colorado at Boulder, Department of Physics, Boulder,
Colorado, U.S.A.  J.F. Martin  University of Toronto, Department of Physics,
Toronto, Ontario, Canada  P. Martins  Queen Mary University of London, School
of Physics and Astronomy, London, United Kingdom  T. Maruyama  High Energy
Accelerator Research Organization (KEK), Tsukuba, Ibaraki, Japan  T. Matsubara
High Energy Accelerator Research Organization (KEK), Tsukuba, Ibaraki, Japan
V. Matveev  Institute for Nuclear Research of the Russian Academy of Sciences,
Moscow, Russia  K. Mavrokoridis  University of Liverpool, Department of
Physics, Liverpool, United Kingdom  W.Y. Ma  Imperial College London,
Department of Physics, London, United Kingdom  E. Mazzucato  IRFU, CEA Saclay,
Gif-sur-Yvette, France  M. McCarthy  York University, Department of Physics
and Astronomy, Toronto, Ontario, Canada  N. McCauley  University of Liverpool,
Department of Physics, Liverpool, United Kingdom  K.S. McFarland  University
of Rochester, Department of Physics and Astronomy, Rochester, New York, U.S.A.
C. McGrew  State University of New York at Stony Brook, Department of Physics
and Astronomy, Stony Brook, New York, U.S.A.  A. Mefodiev  Institute for
Nuclear Research of the Russian Academy of Sciences, Moscow, Russia  C.
Metelko  University of Liverpool, Department of Physics, Liverpool, United
Kingdom  M. Mezzetto  INFN Sezione di Padova and Università di Padova,
Dipartimento di Fisica, Padova, Italy  A. Minamino  Yokohama National
University, Faculty of Engineering, Yokohama, Japan  O. Mineev  Institute for
Nuclear Research of the Russian Academy of Sciences, Moscow, Russia  S. Mine
University of California, Irvine, Department of Physics and Astronomy, Irvine,
California, U.S.A.  M. Miura  University of Tokyo, Institute for Cosmic Ray
Research, Kamioka Observatory, Kamioka, Japan  L. Molina Bueno  ETH Zurich,
Institute for Particle Physics, Zurich, Switzerland  S. Moriyama  University
of Tokyo, Institute for Cosmic Ray Research, Kamioka Observatory, Kamioka,
Japan  J. Morrison  Michigan State University, Department of Physics and
Astronomy, East Lansing, Michigan, U.S.A.  Th.A. Mueller  Ecole Polytechnique,
IN2P3-CNRS, Laboratoire Leprince-Ringuet, Palaiseau, France  S. Murphy  ETH
Zurich, Institute for Particle Physics, Zurich, Switzerland  Y. Nagai
University of Colorado at Boulder, Department of Physics, Boulder, Colorado,
U.S.A.  T. Nakadaira  High Energy Accelerator Research Organization (KEK),
Tsukuba, Ibaraki, Japan  M. Nakahata  University of Tokyo, Institute for
Cosmic Ray Research, Kamioka Observatory, Kamioka, Japan  Kavli Institute for
the Physics and Mathematics of the Universe (WPI), The University of Tokyo
Institutes for Advanced Study, University of Tokyo, Kashiwa, Chiba, Japan  Y.
Nakajima  University of Tokyo, Institute for Cosmic Ray Research, Kamioka
Observatory, Kamioka, Japan  A. Nakamura  Okayama University, Department of
Physics, Okayama, Japan  K.G. Nakamura  Kyoto University, Department of
Physics, Kyoto, Japan  K. Nakamura  Kavli Institute for the Physics and
Mathematics of the Universe (WPI), The University of Tokyo Institutes for
Advanced Study, University of Tokyo, Kashiwa, Chiba, Japan  High Energy
Accelerator Research Organization (KEK), Tsukuba, Ibaraki, Japan  K.D.
Nakamura  Kyoto University, Department of Physics, Kyoto, Japan  Y. Nakanishi
Kyoto University, Department of Physics, Kyoto, Japan  S. Nakayama  University
of Tokyo, Institute for Cosmic Ray Research, Kamioka Observatory, Kamioka,
Japan  T. Nakaya  Kyoto University, Department of Physics, Kyoto, Japan  Kavli
Institute for the Physics and Mathematics of the Universe (WPI), The
University of Tokyo Institutes for Advanced Study, University of Tokyo,
Kashiwa, Chiba, Japan  K. Nakayoshi  High Energy Accelerator Research
Organization (KEK), Tsukuba, Ibaraki, Japan  C. Nantais  University of
Toronto, Department of Physics, Toronto, Ontario, Canada  K. Niewczas  Wroclaw
University, Faculty of Physics and Astronomy, Wroclaw, Poland  K. Nishikawa
High Energy Accelerator Research Organization (KEK), Tsukuba, Ibaraki, Japan
Y. Nishimura  University of Tokyo, Institute for Cosmic Ray Research, Research
Center for Cosmic Neutrinos, Kashiwa, Japan  T.S. Nonnenmacher  Imperial
College London, Department of Physics, London, United Kingdom  P. Novella
IFIC (CSIC & University of Valencia), Valencia, Spain  J. Nowak  Lancaster
University, Physics Department, Lancaster, United Kingdom  H.M. O’Keeffe
Lancaster University, Physics Department, Lancaster, United Kingdom  L.
O’Sullivan  University of Sheffield, Department of Physics and Astronomy,
Sheffield, United Kingdom  K. Okumura  University of Tokyo, Institute for
Cosmic Ray Research, Research Center for Cosmic Neutrinos, Kashiwa, Japan
Kavli Institute for the Physics and Mathematics of the Universe (WPI), The
University of Tokyo Institutes for Advanced Study, University of Tokyo,
Kashiwa, Chiba, Japan  T. Okusawa  Osaka City University, Department of
Physics, Osaka, Japan  W. Oryszczak  University of Warsaw, Faculty of Physics,
Warsaw, Poland  S.M. Oser  University of British Columbia, Department of
Physics and Astronomy, Vancouver, British Columbia, Canada  TRIUMF, Vancouver,
British Columbia, Canada  R.A. Owen  Queen Mary University of London, School
of Physics and Astronomy, London, United Kingdom  Y. Oyama  High Energy
Accelerator Research Organization (KEK), Tsukuba, Ibaraki, Japan  V. Palladino
INFN Sezione di Napoli and Università di Napoli, Dipartimento di Fisica,
Napoli, Italy  J.L. Palomino  State University of New York at Stony Brook,
Department of Physics and Astronomy, Stony Brook, New York, U.S.A.  V. Paolone
University of Pittsburgh, Department of Physics and Astronomy, Pittsburgh,
Pennsylvania, U.S.A.  W.C. Parker  Royal Holloway University of London,
Department of Physics, Egham, Surrey, United Kingdom  P. Paudyal  University
of Liverpool, Department of Physics, Liverpool, United Kingdom  M. Pavin
TRIUMF, Vancouver, British Columbia, Canada  D. Payne  University of
Liverpool, Department of Physics, Liverpool, United Kingdom  L. Pickering
Michigan State University, Department of Physics and Astronomy, East Lansing,
Michigan, U.S.A.  C. Pidcott  University of Sheffield, Department of Physics
and Astronomy, Sheffield, United Kingdom  E.S. Pinzon Guerra  York University,
Department of Physics and Astronomy, Toronto, Ontario, Canada  C. Pistillo
University of Bern, Albert Einstein Center for Fundamental Physics, Laboratory
for High Energy Physics (LHEP), Bern, Switzerland  B. Popov  Sorbonne
Université, Université Paris Diderot, CNRS/IN2P3, Laboratoire de Physique
Nucléaire et de Hautes Energies (LPNHE), Paris, France  K. Porwit  University
of Silesia, Institute of Physics, Katowice, Poland  M. Posiadala-Zezula
University of Warsaw, Faculty of Physics, Warsaw, Poland  A. Pritchard
University of Liverpool, Department of Physics, Liverpool, United Kingdom  B.
Quilain  Kavli Institute for the Physics and Mathematics of the Universe
(WPI), The University of Tokyo Institutes for Advanced Study, University of
Tokyo, Kashiwa, Chiba, Japan  T. Radermacher  RWTH Aachen University, III.
Physikalisches Institut, Aachen, Germany  E. Radicioni  INFN Sezione di Bari
and Università e Politecnico di Bari, Dipartimento Interuniversitario di
Fisica, Bari, Italy  B. Radics  ETH Zurich, Institute for Particle Physics,
Zurich, Switzerland  P.N. Ratoff  Lancaster University, Physics Department,
Lancaster, United Kingdom  E. Reinherz-Aronis  Colorado State University,
Department of Physics, Fort Collins, Colorado, U.S.A.  C. Riccio  INFN Sezione
di Napoli and Università di Napoli, Dipartimento di Fisica, Napoli, Italy  E.
Rondio  National Centre for Nuclear Research, Warsaw, Poland  B. Rossi  INFN
Sezione di Napoli and Università di Napoli, Dipartimento di Fisica, Napoli,
Italy  S. Roth  RWTH Aachen University, III. Physikalisches Institut, Aachen,
Germany  A. Rubbia  ETH Zurich, Institute for Particle Physics, Zurich,
Switzerland  A.C. Ruggeri  INFN Sezione di Napoli and Università di Napoli,
Dipartimento di Fisica, Napoli, Italy  A. Rychter  Warsaw University of
Technology, Institute of Radioelectronics, Warsaw, Poland  K. Sakashita  High
Energy Accelerator Research Organization (KEK), Tsukuba, Ibaraki, Japan  F.
Sánchez  University of Geneva, Section de Physique, DPNC, Geneva, Switzerland
S. Sasaki  Tokyo Metropolitan University, Department of Physics, Tokyo, Japan
E. Scantamburlo  University of Geneva, Section de Physique, DPNC, Geneva,
Switzerland  C.M. Schloesser  ETH Zurich, Institute for Particle Physics,
Zurich, Switzerland  K. Scholberg  Duke University, Department of Physics,
Durham, North Carolina, U.S.A.  J. Schwehr  Colorado State University,
Department of Physics, Fort Collins, Colorado, U.S.A.  M. Scott  Imperial
College London, Department of Physics, London, United Kingdom  Y. Seiya  Osaka
City University, Department of Physics, Osaka, Japan  T. Sekiguchi  High
Energy Accelerator Research Organization (KEK), Tsukuba, Ibaraki, Japan  H.
Sekiya  University of Tokyo, Institute for Cosmic Ray Research, Kamioka
Observatory, Kamioka, Japan  Kavli Institute for the Physics and Mathematics
of the Universe (WPI), The University of Tokyo Institutes for Advanced Study,
University of Tokyo, Kashiwa, Chiba, Japan  D. Sgalaberna  University of
Geneva, Section de Physique, DPNC, Geneva, Switzerland  R. Shah  STFC,
Rutherford Appleton Laboratory, Harwell Oxford, and Daresbury Laboratory,
Warrington, United Kingdom  Oxford University, Department of Physics, Oxford,
United Kingdom  A. Shaikhiev  Institute for Nuclear Research of the Russian
Academy of Sciences, Moscow, Russia  F. Shaker  University of Winnipeg,
Department of Physics, Winnipeg, Manitoba, Canada  D. Shaw  Lancaster
University, Physics Department, Lancaster, United Kingdom  A. Shaykina
Institute for Nuclear Research of the Russian Academy of Sciences, Moscow,
Russia  M. Shiozawa  University of Tokyo, Institute for Cosmic Ray Research,
Kamioka Observatory, Kamioka, Japan  Kavli Institute for the Physics and
Mathematics of the Universe (WPI), The University of Tokyo Institutes for
Advanced Study, University of Tokyo, Kashiwa, Chiba, Japan  A. Smirnov
Institute for Nuclear Research of the Russian Academy of Sciences, Moscow,
Russia  M. Smy  University of California, Irvine, Department of Physics and
Astronomy, Irvine, California, U.S.A.  J.T. Sobczyk  Wroclaw University,
Faculty of Physics and Astronomy, Wroclaw, Poland  H. Sobel  University of
California, Irvine, Department of Physics and Astronomy, Irvine, California,
U.S.A.  Kavli Institute for the Physics and Mathematics of the Universe (WPI),
The University of Tokyo Institutes for Advanced Study, University of Tokyo,
Kashiwa, Chiba, Japan  Y. Sonoda  University of Tokyo, Institute for Cosmic
Ray Research, Kamioka Observatory, Kamioka, Japan  J. Steinmann  RWTH Aachen
University, III. Physikalisches Institut, Aachen, Germany  T. Stewart  STFC,
Rutherford Appleton Laboratory, Harwell Oxford, and Daresbury Laboratory,
Warrington, United Kingdom  P. Stowell  University of Sheffield, Department of
Physics and Astronomy, Sheffield, United Kingdom  Y. Suda  University of
Tokyo, Department of Physics, Tokyo, Japan  S. Suvorov  Institute for Nuclear
Research of the Russian Academy of Sciences, Moscow, Russia  IRFU, CEA Saclay,
Gif-sur-Yvette, France  A. Suzuki  Kobe University, Kobe, Japan  S.Y. Suzuki
High Energy Accelerator Research Organization (KEK), Tsukuba, Ibaraki, Japan
Y. Suzuki  Kavli Institute for the Physics and Mathematics of the Universe
(WPI), The University of Tokyo Institutes for Advanced Study, University of
Tokyo, Kashiwa, Chiba, Japan  A.A. Sztuc  Imperial College London, Department
of Physics, London, United Kingdom  R. Tacik  University of Regina, Department
of Physics, Regina, Saskatchewan, Canada  TRIUMF, Vancouver, British Columbia,
Canada  M. Tada  High Energy Accelerator Research Organization (KEK), Tsukuba,
Ibaraki, Japan  A. Takeda  University of Tokyo, Institute for Cosmic Ray
Research, Kamioka Observatory, Kamioka, Japan  Y. Takeuchi  Kobe University,
Kobe, Japan  Kavli Institute for the Physics and Mathematics of the Universe
(WPI), The University of Tokyo Institutes for Advanced Study, University of
Tokyo, Kashiwa, Chiba, Japan  R. Tamura  University of Tokyo, Department of
Physics, Tokyo, Japan  H.K. Tanaka  University of Tokyo, Institute for Cosmic
Ray Research, Kamioka Observatory, Kamioka, Japan  H.A. Tanaka  SLAC National
Accelerator Laboratory, Stanford University, Menlo Park, California, USA
University of Toronto, Department of Physics, Toronto, Ontario, Canada  T.
Thakore  Louisiana State University, Department of Physics and Astronomy,
Baton Rouge, Louisiana, U.S.A.  L.F. Thompson  University of Sheffield,
Department of Physics and Astronomy, Sheffield, United Kingdom  W. Toki
Colorado State University, Department of Physics, Fort Collins, Colorado,
U.S.A.  C. Touramanis  University of Liverpool, Department of Physics,
Liverpool, United Kingdom  K.M. Tsui  University of Liverpool, Department of
Physics, Liverpool, United Kingdom  T. Tsukamoto  High Energy Accelerator
Research Organization (KEK), Tsukuba, Ibaraki, Japan  M. Tzanov  Louisiana
State University, Department of Physics and Astronomy, Baton Rouge, Louisiana,
U.S.A.  Y. Uchida  Imperial College London, Department of Physics, London,
United Kingdom  W. Uno  Kyoto University, Department of Physics, Kyoto, Japan
M. Vagins  Kavli Institute for the Physics and Mathematics of the Universe
(WPI), The University of Tokyo Institutes for Advanced Study, University of
Tokyo, Kashiwa, Chiba, Japan  University of California, Irvine, Department of
Physics and Astronomy, Irvine, California, U.S.A.  Z. Vallari  State
University of New York at Stony Brook, Department of Physics and Astronomy,
Stony Brook, New York, U.S.A.  D. Vargas  Institut de Fisica d’Altes Energies
(IFAE), The Barcelona Institute of Science and Technology, Campus UAB,
Bellaterra (Barcelona) Spain  G. Vasseur  IRFU, CEA Saclay, Gif-sur-Yvette,
France  C. Vilela  State University of New York at Stony Brook, Department of
Physics and Astronomy, Stony Brook, New York, U.S.A.  T. Vladisavljevic
Oxford University, Department of Physics, Oxford, United Kingdom  Kavli
Institute for the Physics and Mathematics of the Universe (WPI), The
University of Tokyo Institutes for Advanced Study, University of Tokyo,
Kashiwa, Chiba, Japan  V.V. Volkov  Institute for Nuclear Research of the
Russian Academy of Sciences, Moscow, Russia  T. Wachala  H. Niewodniczanski
Institute of Nuclear Physics PAN, Cracow, Poland  J. Walker  University of
Winnipeg, Department of Physics, Winnipeg, Manitoba, Canada  Y. Wang  State
University of New York at Stony Brook, Department of Physics and Astronomy,
Stony Brook, New York, U.S.A.  D. Wark  STFC, Rutherford Appleton Laboratory,
Harwell Oxford, and Daresbury Laboratory, Warrington, United Kingdom  Oxford
University, Department of Physics, Oxford, United Kingdom  M.O. Wascko
Imperial College London, Department of Physics, London, United Kingdom  A.
Weber  STFC, Rutherford Appleton Laboratory, Harwell Oxford, and Daresbury
Laboratory, Warrington, United Kingdom  Oxford University, Department of
Physics, Oxford, United Kingdom  R. Wendell  Kyoto University, Department of
Physics, Kyoto, Japan  M.J. Wilking  State University of New York at Stony
Brook, Department of Physics and Astronomy, Stony Brook, New York, U.S.A.  C.
Wilkinson  University of Bern, Albert Einstein Center for Fundamental Physics,
Laboratory for High Energy Physics (LHEP), Bern, Switzerland  J.R. Wilson
Queen Mary University of London, School of Physics and Astronomy, London,
United Kingdom  R.J. Wilson  Colorado State University, Department of Physics,
Fort Collins, Colorado, U.S.A.  C. Wret  University of Rochester, Department
of Physics and Astronomy, Rochester, New York, U.S.A.  Y. Yamada  High Energy
Accelerator Research Organization (KEK), Tsukuba, Ibaraki, Japan  K. Yamamoto
Osaka City University, Department of Physics, Osaka, Japan  S. Yamasu  Okayama
University, Department of Physics, Okayama, Japan  C. Yanagisawa  State
University of New York at Stony Brook, Department of Physics and Astronomy,
Stony Brook, New York, U.S.A.  G. Yang  State University of New York at Stony
Brook, Department of Physics and Astronomy, Stony Brook, New York, U.S.A.  T.
Yano  University of Tokyo, Institute for Cosmic Ray Research, Kamioka
Observatory, Kamioka, Japan  K. Yasutome  Kyoto University, Department of
Physics, Kyoto, Japan  S. Yen  TRIUMF, Vancouver, British Columbia, Canada  N.
Yershov  Institute for Nuclear Research of the Russian Academy of Sciences,
Moscow, Russia  M. Yokoyama  University of Tokyo, Department of Physics,
Tokyo, Japan  T. Yoshida  Tokyo Institute of Technology, Department of
Physics, Tokyo, Japan  M. Yu  York University, Department of Physics and
Astronomy, Toronto, Ontario, Canada  A. Zalewska  H. Niewodniczanski Institute
of Nuclear Physics PAN, Cracow, Poland  J. Zalipska  National Centre for
Nuclear Research, Warsaw, Poland  K. Zaremba  Warsaw University of Technology,
Institute of Radioelectronics, Warsaw, Poland  G. Zarnecki  National Centre
for Nuclear Research, Warsaw, Poland  M. Ziembicki  Warsaw University of
Technology, Institute of Radioelectronics, Warsaw, Poland  E.D. Zimmerman
University of Colorado at Boulder, Department of Physics, Boulder, Colorado,
U.S.A.  M. Zito  IRFU, CEA Saclay, Gif-sur-Yvette, France  S. Zsoldos  Queen
Mary University of London, School of Physics and Astronomy, London, United
Kingdom  A. Zykova  Institute for Nuclear Research of the Russian Academy of
Sciences, Moscow, Russia

###### Abstract

This paper reports on the search for heavy neutrinos with masses in the range
$140<M_{N}<493$ MeV/c2 using the off-axis near detector ND280 of the T2K
experiment. These particles can be produced from kaon decays in the standard
neutrino beam and then subsequently decay in ND280. The decay modes under
consideration are $N\to\ell^{\pm}_{\alpha}\pi^{\mp}$ and
$N\to\ell^{+}_{\alpha}\ell^{-}_{\beta}\overset{\scalebox{0.3}{(}\raisebox{-1.7pt}[0.0pt][0.0pt]{$-$}\scalebox{0.3}{)}}{\nu}$
($\alpha,\beta=e,\mu$). A search for such events has been made using the Time
Projection Chambers of ND280, where the background has been reduced to less
than two events in the current dataset in all channels. No excess has been
observed in the signal region. A combined Bayesian statistical approach has
been applied to extract upper limits on the mixing elements of heavy neutrinos
to electron-, muon- and tau- flavoured currents ($U_{e}^{2}$, $U_{\mu}^{2}$,
$U_{\tau}^{2}$) as a function of the heavy neutrino mass, e.g.
$U_{e}^{2}<10^{-9}$ at $90\%$ C.L. for a mass of $390$ MeV/c2. These
constraints are competitive with previous experiments.

##  I Introduction

Neutrino oscillations provide strong evidence that neutrinos are massive
particles. Although in the minimal Standard Model they are massless, the most
natural extension to allow non-zero masses compatible with oscillation
experiments results (two different $\Delta m^{2}$) consists in the
introduction of $n\geq 2$ new right-handed (sterile) neutrino fields $\nu_{R}$
with the following mass term Drewes (2013):

$\displaystyle\mathcal{L}_{\text{mass}}=-\frac{1}{2}\left(\begin{array}[]{cc}\bar{\nu}_{L}\
\bar{\nu}_{R}^{c}\end{array}\right)\left(\begin{array}[]{cc}0&m_{D}\\\
m_{D}^{T}&m_{R}\end{array}\right)\left(\begin{array}[]{c}\nu_{L}^{c}\\\
\nu_{R}\end{array}\right)+\text{h.c.},$ (6)

where $m_{D}$ is the $3\times n$ Dirac mass matrix and $m_{R}$ is the $n\times
n$ Majorana mass matrix. If the seesaw condition $m_{D}^{T}m_{D}\ll m_{R}^{2}$
holds (in terms of eigenvalues), diagonalisation of the mass matrix yields
three light Majorana mass eigenstates $\nu_{i}$ ($i=1,2,3$), with masses
$m_{\nu,i}$ of the order of the eigenvalues of $m_{D}m_{R}^{-1}m_{D}^{T}$ and
$n$ heavy Majorana mass eigenstates $N_{I}$ ($I=1,\ldots,n$) (heavy neutrinos,
also called heavy neutral leptons in the literature), with masses $M_{N,I}$ of
the order of the eigenvalues of $m_{R}$. The flavour eigenstates can be
expressed in terms of the mass eigenstates as:

$\nu_{\alpha}=\sum_{i=1}^{3}V_{\alpha i}\nu_{i}+\sum_{I=1}^{n}\Theta_{\alpha
I}N_{I}\quad(\alpha=e,\mu,\tau),$ (7)

where $V$ corresponds to the usual PMNS matrix and $\Theta$ is the active-
heavy mixing matrix. Heavy neutrinos can be produced in leptonic meson decays
$M^{\pm}\to\ell_{\alpha}^{\pm}+N_{I}$ with a branching ratio proportional to
$|\Theta_{\alpha I}|^{2}$, for $M_{N}<m_{\text{meson}}-m_{\ell_{\alpha}}$ and
they can similarly decay via the same mixing element.

If at least two of the heavy neutrinos have a mass between $0.1$ and $100$
GeV/c2, they can generate baryogenesis via leptogenesis without any additional
new physics Drewes  _et al._ (2018). An example of such a model is the
Neutrino Minimal Standard Model ($\nu$MSM) with $n=3$, in which $N_{1}$ has a
mass of $1-100$ keV/c2 and is a warm dark matter candidate, while $N_{2,3}$
are degenerate with GeV-scale masses Asaka and Shaposhnikov (2005); Asaka  _et
al._ (2005).

In the following, we define $U_{\alpha}^{2}\equiv\sum|\Theta_{\alpha I}|^{2}$
summing over the heavy neutrinos that cannot be distinguished experimentally
(such as $N_{2}$ and $N_{3}$ in the $\nu$MSM).

Limits on $U_{\alpha}^{2}$ for $M_{N}<493$ MeV/c2 can be obtained either by
studying heavy neutrino production from kaon decays ($K^{\pm}\to\ell^{\pm}N$)
or by searching for heavy neutrino decays, e.g. to one pion and one charged
lepton ($N\to\ell_{\alpha}^{\pm}\pi^{\mp}$). The best constraints in this mass
range were obtained by the BNL E949 Artamonov  _et al._ (2015) and the CERN
PS191 Bernardi  _et al._ (1986, 1988) experiments with limits of the order of
$10^{-9}-10^{-8}$ on $U_{e}^{2}$, $U_{\mu}^{2}$ and $U_{e}U_{\mu}$ for
$M_{N}=200-450$ MeV/c2. Limits from other experiments are summarised in the
review Drewes and Garbrecht (2017).

This paper presents the search for potential heavy neutrinos produced in the
T2K decay volume and decaying in the T2K Near Detector, ND280, as was
originally suggested in Asaka  _et al._ (2013).

##  II Experimental setup

###  II.1 The T2K beamline

The Tokai-to-Kamioka (T2K) experiment Abe  _et al._ (2011) is a long-baseline
neutrino experiment located in Japan with the primary goal of measuring muon
(anti-)neutrino oscillations using Super-Kamiokande as its far detector. The
T2K neutrino beam is produced at the Japan Proton Accelerator Research Complex
(J-PARC) by colliding $30$ GeV protons on a graphite target. The pions and
kaons produced are focused and selected by charge with magnetic horns and
subsequently decay in flight to neutrinos. Depending on the polarity of the
current in the horns, the experiment can be run either in neutrino or anti-
neutrino mode.

In this analysis, the production of heavy neutrinos from kaon decays in data
taken from November 2010 to May 2017 are considered. This corresponds to a
total exposure of $12.34\times 10^{20}$ protons-on-target (POT) in neutrino
mode and $6.29\times 10^{20}$ POT in anti-neutrino mode, after data quality
cuts.

###  II.2 The off-axis near detector ND280

The off-axis near detector ND280 is located 280 metres from the proton target.
It is composed of several sub-detectors with a $0.2$ T magnet Abe  _et al._
(2011). The central tracker consists of three Time Projection Chambers (TPCs)
Abgrall  _et al._ (2011), two scintillator-based Fine-Grained Detectors (FGDs)
Amaudruz  _et al._ (2012) and one $\pi^{0}$ detector (P0D). It is surrounded
by an Electromagnetic Calorimeter (ECal) and a Side Muon Range Detector
(SMRD). A schematic view of ND280 is shown in Figure 1.

The main goal of ND280 is to detect neutrino interactions in order to
constrain both neutrino flux and cross-section parameters. The TPCs are filled
with a gas mixture based on argon gas and provide excellent track and momentum
reconstruction with a typical resolution of 8% for 1 GeV/c tracks Abe  _et
al._ (2015). This can be combined with energy loss (dE/dx) measurements in
order to perform particle identification (PID) of charged tracks crossing the
TPCs.

Figure 1: An exploded view of the ND280 off-axis near detector labelling each
sub-detector. Adapted from Abe  _et al._ (2011).

The analysis focuses on heavy neutrino decays occurring in the ND280 TPC gas
volumes, which corresponds to a total volume of interest of $6.3$ m3.

##  III Analysis

###  III.1 Simulation

The simulation of heavy neutrino production and decay is performed using the
T2K neutrino flux prediction, which is constrained by the NA61/SHINE
experiment results and by in-situ measurements Abe  _et al._ (2013); Abgrall
_et al._ (2016). We first consider the flux of standard light neutrinos coming
from kaon decays in the beamline and crossing the ND280 TPCs. This flux is
transformed into a flux of heavy neutrinos ($K^{\pm}\to\ell^{\pm}_{\alpha}N$,
$\alpha=e,\mu$) by weighting event-by-event using the appropriate branching
ratios Shrock (1981a, b); Gorbunov and Shaposhnikov (2007) and modified
kinematics. The analysis assumes the heavy neutrino lifetime is long enough to
reach ND280 ($\tau\gg\text{time of flight}\sim 1\,\mu$s), which is consistent
with current limits on the mixing elements. Figure 2 presents the results of
the simulation for different heavy neutrino masses and for both production
modes in neutrino mode. The flux has the same shape for anti-neutrino mode,
although it is a factor of $\sim 3$ lower.

Figure 2: Expected flux of heavy neutrinos crossing the ND280 TPCs from
$K^{\pm}\to\mu^{\pm}N$ and $K^{\pm}\to e^{\pm}N$ for several values of
$M_{N}$, with the T2K beam in neutrino-mode and for $U_{e}^{2}=U_{\mu}^{2}=1$.
The black dotted curve corresponds to the limiting case of a massless neutrino
($N=\nu$); the one for $K\to e\nu$ is not drawn as it is a few orders of
magnitude lower due to helicity suppression.

The heavy neutrino decays are then simulated at a random point along their
trajectories inside ND280. All the possible modes $N\to\ell^{\pm}\pi^{\mp}$
and
$N\to\ell^{\pm}\ell^{\mp}\overset{\scalebox{0.3}{(}\raisebox{-1.7pt}[0.0pt][0.0pt]{$-$}\scalebox{0.3}{)}}{\nu}$
were simulated. Figure 3 shows the allowed production and decay modes as a
function of the heavy neutrino mass. The neutral current decay modes $N\to
e^{+}e^{-}\overset{\scalebox{0.3}{(}\raisebox{-1.7pt}[0.0pt][0.0pt]{$-$}\scalebox{0.3}{)}}{\nu}_{\tau}$
and
$N\to\mu^{+}\mu^{-}\overset{\scalebox{0.3}{(}\raisebox{-1.7pt}[0.0pt][0.0pt]{$-$}\scalebox{0.3}{)}}{\nu}_{\tau}$
are directly sensitive to the mixing element $U_{\tau}^{2}$.

Effects related to heavy neutrino polarisation Levy (2018) and delayed arrival
time (with respect to light neutrinos) are taken into account in the
simulation.

Figure 3: Schematic of all the possible production and decay mode combinations
for heavy neutrinos with $140<M_{N}<493$ MeV/c2. The coloured bars show the
allowed kinematic regions for each decay mode, with the corresponding mixing
element in the right column. A total of 40 production/decay mode combinations
are possible.

###  III.2 Selection

The selection was developed to isolate the signal events listed in Figure 3
from the background expected from standard neutrino interactions with matter.
In order to significantly improve the signal to background ratio, which is
inversely proportional to the density of the medium, only events occurring in
the TPC gas volume are considered for this analysis.

Events are pre-selected by identifying two tracks of opposite charge
originating from a vertex in a TPC. There should be no other tracks in the TPC
itself or in the detector located directly upstream (e.g. P0D for the first
TPC or the first FGD for the second TPC). Particle identification for each
individual track is performed using energy loss in the TPC. Five channels are
then identified: $\mu^{\pm}\pi^{\mp}$, $e^{-}\pi^{+}$, $e^{+}\pi^{-}$,
$e^{+}e^{-}$, $\mu^{+}\mu^{-}$.

In the analysis, we do not define any specific selection for the three-body
decays $N\to e^{\pm}\mu^{\mp}\nu$ because these modes already contribute to
the $e^{\pm}\pi^{\mp}$ selection channels. For the $\mu^{+}\mu^{-}$ channel,
electromagnetic calorimeter information is also used to clearly identify the
two muons.

Several kinematic cuts are then applied to further reject the background:

  * •

invariant mass $m_{\rm inv}$ of the two-track system: in the case of a heavy
neutrino decay, it is expected that $m_{\rm inv}^{\rm true}\leq M_{N}$
($m_{\rm inv}^{\rm true}=M_{N}$ for the two-body decays). The heavy neutrino
is produced in kaon decays so that it is necessarily lighter than $M_{K}=493$
MeV/c2 allowing an upper cut on the reconstructed invariant mass $m_{\rm
inv}^{\rm reco}<700$ MeV/c2 to be applied. The additional margin accounts for
detector resolution effects.

  * •

angle between the two tracks $\Delta\Phi$: the two charged tracks produced in
the decay are boosted forward so that only events with $\Delta\Phi<90\degree$
can be selected without loss of signal efficiency.

  * •

incoming heavy neutrino polar angle $\theta$: the heavy neutrino’s direction
is collinear to the beam, while the products of an active neutrino interaction
are expected to be distributed with a larger angle because of potential missed
tracks or nuclear effects. $\theta$ is reconstructed using the two charged
tracks but can still be used with a good approximation for the three-body
decays. The cut is $\cos\theta>0.992$ for $\mu^{\pm}\pi^{\mp}$ for the
$\mu\pi$ channel and $\cos\theta>0.99$ for the others.

Applying these criteria to the signal simulated in the ND280 TPC gas volumes,
the efficiencies of the signal selection for the different modes were
obtained, as shown in Figure 4. For a given mass, they are quite independent
of the production mode ($K^{\pm}\to\mu^{\pm}N$ or $K^{\pm}\to e^{\pm}N$).
$\mu\pi$ efficiencies are slightly better as muon tracks are easier to
reconstruct in the TPC.

Figure 4: Heavy neutrino signal selection efficiency in neutrino mode as a
function of heavy neutrino mass for some of the decay modes. Error bars
include both statistical and systematic uncertainties.

###  III.3 Signal systematic uncertainties

Two sources of systematic uncertainties on the heavy neutrino signal are
considered:

  * •

flux: uncertainties on the kaon flux used as input to the simulation, as
presented in subsection III.1, are directly transposed into uncertainties on
the flux of heavy neutrinos reaching ND280. The total normalisation
uncertainty has been estimated to be $15\%$, using external data such as those
from the NA61/SHINE experiment Abe  _et al._ (2013).

  * •

signal selection efficiency: detector systematic uncertainties are defined to
cope with any discrepancies between data and simulation of the detector
effects. The dominant uncertainties are related to TPC reconstruction and
particle identification performances and have been computed as in previous
ND280 analyses Abe  _et al._ (2018). The overall effects have been estimated
to be approximately $5\%$.

###  III.4 Background estimation

The background remaining after the selection has first been estimated using
the NEUT 5.3.2 Monte Carlo generator Hayato (2002), before being constrained
using control regions in ND280 data.

One of the dominant background contributions is the neutrino-induced coherent
pion production on argon nuclei in the TPC gas
($\nu_{\mu}+\text{Ar}\to\mu^{-}+\pi^{+}+\text{Ar}$). The NEUT prediction has
been tuned to T2K and MINERvA data Abe  _et al._ (2016); Higuera  _et al._
(2014) with a $30\%$ normalisation uncertainty.

Additional background sources include other types of neutrino interactions in
the gas and interactions outside the gas. An example of the latter is the
conversion of a photon, emitted by a neutrino interaction in a FGD, to an
electron-positron pair.

Data and simulations are compared with two sets of control regions in order to
estimate the model uncertainty on the background. First, a selection of events
similar to the signal events, but where the kinematic cut on the polar angle
$\theta$ is inverted (CR-I), contains mostly resonant pion production and
quasi-elastic processes on argon. Similarly, control regions are identified by
considering events starting in the borders of the TPC (meaning the box
containing the gas) as the volume of interest rather than in the gas itself
(CR-II). These control regions are dominated by photon conversions and other
mis-reconstructed processes.

Table 1 presents the comparison of T2K data and NEUT predictions in the
aforementioned control regions. We have not found any significant
discrepancies between data and Monte Carlo predictions in any of them.
Conservatively, we have assigned to each background source a model uncertainty
equal to the statistical uncertainty of the data in the corresponding control
region.

Table 1: Comparison of number of events in data (D) and corresponding NEUT prediction (with statistical uncertainties) in the control regions used to determine the model uncertainties in the different channels, using the data set presented in subsection II.1. | Neutrino mode | Anti-neutrino mode
---|---|---
| CR-I | CR-II | CR-I | CR-II
Ch. | D | NEUT | D | NEUT | D | NEUT | D | NEUT
$\mu^{\pm}\pi^{\mp}$ | $15$ | $11.4\pm 1.0$ | $36$ | $30.1\pm 1.6$ | $2$ | $2.6\pm 0.5$ | $5$ | $6.5\pm 0.8$
$e^{-}\pi^{+}$ | $2$ | $2.4\pm 0.5$ | $14$ | $13.3\pm 1.1$ | $2$ | $0.7\pm 0.2$ | $7$ | $5.4\pm 0.7$
$e^{+}\pi^{-}$ | $5$ | $3.1\pm 0.5$ | $17$ | $11.7\pm 1.0$ | $1$ | $0.5\pm 0.2$ | $2$ | $3.3\pm 0.6$
$\mu^{+}\mu^{-}$ | $0$ | $0.9\pm 0.3$ | $3$ | $2.5\pm 0.5$ | $0$ | $0.2\pm 0.1$ | $1$ | $0.4\pm 0.2$
$e^{+}e^{-}$ | $4$ | $4.5\pm 0.6$ | $58$ | $53.5\pm 2.2$ | $3$ | $0.8\pm 0.3$ | $13$ | $14.3\pm 1.3$

For a given channel, the number of expected background events is the nominal
value from NEUT and the total uncertainty is the sum of the contributions from
the Monte Carlo statistical uncertainty, the flux and detector systematic
uncertainties and the model uncertainties described above.

Table 2: Summary of the estimated numbers of background events in the different analysis channels in neutrino and anti-neutrino beam modes with the corresponding absolute uncertainties (statistical, flux-related, detector-related, model), using the data set presented in subsection II.1. Mode | Ch. | Expected | Uncertainties
---|---|---|---
background | stat. | flux | det. | model | total
neutrino | $\mu^{\pm}\pi^{\mp}$ | 1.543 | 0.366 | 0.154 | 0.165 | 0.285 | 0.516
$e^{-}\pi^{+}$ | 0.376 | 0.213 | 0.038 | 0.104 | 0.097 | 0.259
$e^{+}\pi^{-}$ | 0.328 | 0.186 | 0.033 | 0.117 | 0.115 | 0.250
$\mu^{+}\mu^{-}$ | 0.216 | 0.107 | 0.022 | 0.045 | 0.062 | 0.133
$e^{+}e^{-}$ | 0.563 | 0.192 | 0.056 | 0.092 | 0.074 | 0.233
anti-neutrino | $\mu^{\pm}\pi^{\mp}$ | 0.384 | 0.161 | 0.038 | 0.058 | 0.100 | 0.202
$e^{-}\pi^{+}$ | 0.018 | 0.018 | 0.002 | 0.005 | 0.005 | 0.020
$e^{+}\pi^{-}$ | 0.219 | 0.155 | 0.022 | 0.140 | 0.122 | 0.243
$\mu^{+}\mu^{-}$ | 0.038 | 0.038 | 0.004 | 0.007 | 0.011 | 0.040
$e^{+}e^{-}$ | 0.015 | 0.015 | 0.002 | 0.001 | 0.004 | 0.016

Table 2 summarises the background in the different analysis channels. The
dominant contribution to its uncertainty comes from the limited statistics of
the samples. The background in the $\mu\pi$ channel is higher than for other
channels, as it is dominated by the irreducible coherent pion production.

###  III.5 Statistical analysis

Two approaches have been considered to constrain the mixing elements
$U_{e}^{2}$, $U_{\mu}^{2}$ and $U_{\tau}^{2}$.

In the first approach, each heavy neutrino production/decay mode is considered
independently and the corresponding analysis channel is used to put limits on
the associated mixing elements. For instance, the $\mu^{\pm}\pi^{\mp}$ channel
as defined in subsection III.2 can constrain:

  * •

either $U_{\mu}^{2}$ by considering only the signal from
$K^{\pm}\to\mu^{\pm}N$, $N\to\mu^{\pm}\pi^{\mp}$,

  * •

or $U_{e}\times U_{\mu}$ by considering only the signal from $K^{\pm}\to
e^{\pm}N$, $N\to\mu^{\pm}\pi^{\mp}$.

Three methods to obtain constraints in this approach have been applied:

  1. ($\mathbf{A}$)

assuming that the background is zero, set conservative upper limits,
independently of background modelling and estimation, on the mixing elements
using the Highland-Cousin method Cousins and Highland (1992);

  2. ($\mathbf{B}$)

the Feldman-Cousins method to define confidence intervals, taking into account
the non-zero background Feldman and Cousins (1998);

  3. ($\mathbf{C}$)

a Bayesian method to define credible intervals, taking into account the non-
zero background.

This “single-channel” approach has the advantage of being straightforward and
is similar to that of the PS191 collaboration Bernardi  _et al._ (1986, 1988).
However, it does not allow the different modes and channels to be combined, so
that the constraints are valid only under strong assumptions of the hierarchy
of $U_{e}^{2}$, $U_{\mu}^{2}$ and $U_{\tau}^{2}$. A “combined” approach was
then defined, in which all the heavy neutrino production and decay modes
(presented in Figure 3) and the ten different analysis channels (five for each
beam mode) are considered simultaneously. For a given analysis channel $A$,
the contribution of a mode $i$ is characterised by:

  * •

the expected number of decays in the detector assuming
$U_{e}^{2}=U_{\mu}^{2}=U_{\tau}^{2}=1$ and 100% selection efficiency, denoted
$\Phi_{i}$ ;

  * •

the selection efficiency of these decays in the current channel
$\varepsilon_{A,i}$ ;

  * •

the actual values of $U_{e}^{2}$, $U_{\mu}^{2}$ and $U_{\tau}^{2}$ via the
factor $f_{i}=U_{\alpha}^{2}\sum U_{\beta_{j}}^{2}$ with
$\alpha,\beta_{j}\in\\{e,\mu,\tau\\}$ where $\alpha$ is the flavour involved
at the production of the heavy neutrino and $\beta_{j}$ are the flavours
involved in its decay (only one for charge current modes, several for neutral
current modes).

The expected number of events $\mathcal{N}_{A}$ in a channel $A$ depends on
the background in this channel $B_{A}$ and the sum of the contributions from
the different production and decay modes:

$\mathcal{N}_{A}=B_{A}+\sum_{i}\varepsilon_{A,i}\times
f_{i}(U_{e}^{2},U_{\mu}^{2},U_{\tau}^{2})\times\Phi_{i}.$ (8)

Only a Bayesian method has been considered in this combined approach. The
likelihood is built using a Poisson function for the observed number of events
$n^{\text{obs}}_{A}$ in each channel $A$, with Poisson parameter
$\mathcal{N}_{A}$:

$\mathcal{L}=\prod_{A}\text{Poisson}\left(n^{\text{obs}}_{A},\mathcal{N}_{A}\right).$
(9)

The uncertainties on the flux and efficiency are taken into account in the
forms of multivariate Gaussian priors $\pi_{\Phi}$ and $\pi_{\varepsilon}$
respectively. The priors on the background $\pi_{B}$ are taken to be log-
normal with means and standard deviations given by the expected background and
its uncertainty in Table 2. The priors on the mixing elements $U_{\alpha}^{2}$
are assumed to be flat.

The marginalised posterior probability $p$ is then defined as the product of
the likelihood $\mathcal{L}$ and the priors, integrating over all the nuisance
parameters (flux, efficiency and background):

$p(U_{e}^{2},U_{\mu}^{2},U_{\tau}^{2})=\int
d\Phi\,d\varepsilon\,dB\times\mathcal{L}\times\pi_{\Phi}\,\pi_{\varepsilon}\,\pi_{B}\,\pi_{U^{2}}.$
(10)

A Markov Chain Monte Carlo method has been implemented using PyMC Patil  _et
al._ (2010) to perform this integration. The output can then be used to define
90% domains, either by profiling or by marginalising over the two other mixing
elements. For instance,

$\displaystyle p_{\text{prof}}(U_{e}^{2})$
$\displaystyle=p(U_{e}^{2},U_{\mu,\text{max}}^{2}U_{\tau,\text{max}}^{2}),$
(11) $\displaystyle p_{\text{marg}}(U_{e}^{2})$ $\displaystyle=\int
p(U_{e}^{2},U_{\mu}^{2},U_{\tau}^{2})dU_{\mu}^{2}dU_{\tau}^{2},$ (12)

where $U_{\mu,\text{max}}^{2}$ and $U_{\tau,\text{max}}^{2}$ are the values
maximising the likelihood.

Limits in 2D/3D parameter space may be obtained as well. Limits on $U_{e}^{2}$
can be computed for $140<M_{N}<493$ MeV/c2, while limits on $U_{\mu}^{2}$ and
$U_{\tau}^{2}$ can only be computed for $140<M_{N}<388$ MeV/c2 due to the
kinematic constraints presented in Figure 3.

##  IV Results

Following the selection from subsection III.2, no events were observed in any
of the different signal regions, which is consistent with the background-only
hypothesis, allowing upper limits on $U_{e}^{2}$, $U_{\mu}^{2}$ and
$U_{\tau}^{2}$ to be placed.

An example of results from the single-channel approach is presented in Figure
5. It shows the comparison of the three methods ($\mathbf{A}$, $\mathbf{B}$,
$\mathbf{C}$), which give similar upper limits with method $\mathbf{A}$ giving
slightly more conservative limits as expected.

Figure 5: 90% upper limits on the mixing element $U_{e}^{2}$ as a function of
heavy neutrino mass using the single-channel approach, considering only the
contribution from $K^{\pm}\to e^{\pm}N,N\to e^{\pm}\pi^{\mp}$, with the three
methods $\mathbf{A}$, $\mathbf{B}$ and $\mathbf{C}$. The limits are compared
to the ones of PS191 experiment Bernardi  _et al._ (1986, 1988).

The results of the combined approach are shown in Figure 6. They provide an
improvement by a factor of 2-3 with respect to the single-channel approach,
thanks to the increased statistical power of the combination.

Figure 6: 90% upper limits on the mixing elements $U_{e}^{2}$ (top),
$U_{\mu}^{2}$ (middle), $U_{\tau}^{2}$ (bottom) as a function of heavy
neutrino mass, obtained with the combined approach. The blue dashed lines
corresponds to the results of the single-channel approach (method C). The blue
solid lines are obtained after marginalisation over the two other mixing
elements. In the top plot, the additional blue dotted line corresponds to the
case where profiling is used ($U_{\mu}^{2}=U_{\tau}^{2}=0$), as explained in
the main text. The limits are compared to the ones of other experiments: PS191
Bernardi  _et al._ (1986, 1988), E949 Artamonov  _et al._ (2015), CHARM Orloff
_et al._ (2002).

The limits are competitive with those of previous experiments such as PS191
Bernardi  _et al._ (1986, 1988), E949 Artamonov  _et al._ (2015) and CHARM
Orloff  _et al._ (2002), especially in the high-mass region (above $300$
MeV/c2). The kinks clearly visible on $U_{\mu}^{2}$ and $U_{\tau}^{2}$ limits
come from the changes in the contributing production and decay modes as
presented in Figure 3.

The limits are obtained after marginalisation over the two other mixing
elements. For $U_{e}^{2}$, the limits after profiling (equation 12) are also
presented, which effectively corresponds to setting
$U_{\mu}^{2}=U_{\tau}^{2}=0$. Indeed, for $M_{N}>388$ MeV/c2, the correlations
between $U_{e}^{2}$ and $U_{\mu}^{2}$ (as seen in Figure 7) would give limits
on $U_{e}^{2}$ outside T2K’s reach. However, profiling leads to a loss in the
sensitivity on $U_{e}^{2}$ with respect to the marginalisation as it
forcefully suppresses the contributions of the decay modes involving
$U_{\mu}^{2}$ or $U_{\tau}^{2}$.

It is worth mentioning that the limits depend on the choice of prior on
$U_{\alpha}^{2}$. The limits on $U_{e}^{2}$ and $U_{\mu}^{2}$ are quite robust
with respect to a change of prior as T2K data are directly sensitive to these
mixing elements (e.g. using $\pi_{U^{2}}(U_{\alpha}^{2})=U_{\alpha}^{2}$
varies the limit by less than $30\%$), while the limit on $U_{\tau}^{2}$ is
strongly affected (more than $50\%$).

It is also possible to define 2D contours, e.g. in the $U_{e}^{2}-U_{\mu}^{2}$
plane, allowing the correlations between the mixing elements to be visualised.
Figure 7 presents a set of such contours for different heavy neutrino masses.
The change of behaviour at $M_{N}=388$ MeV/c2 corresponds to the kinematic
cut-off for $K^{\pm}\to\mu^{\pm}N$ processes as seen in Figure 3.

Figure 7: 2D contours in the $U_{e}^{2}-U_{\mu}^{2}$ plane, after profiling
over $U_{\tau}^{2}$ ($U_{\tau}^{2}=0$). Each line corresponds to a different
heavy neutrino mass hypothesis.

##  V Conclusion

A selection of events with two tracks with opposite charges originating from
the ND280 TPC gas volumes allows heavy neutrino decays to be efficiently
isolated from expected background coming from standard neutrino interactions
with matter. No events are observed in the defined signal regions, which is
consistent with the background-only hypothesis.

Limits on the mixing elements $U_{e}^{2}$, $U_{\mu}^{2}$ and $U_{\tau}^{2}$
are obtained using a combined Bayesian approach. Results apply to any model
with heavy neutrinos with masses between $140$ MeV/c2 and $493$ MeV/c2 such as
Rasmussen and Winter (2016), and can, in particular, be interpreted as
constraints on the sum of $N_{2}$ and $N_{3}$ coupling squared as explained in
the introduction, for the $\nu$MSM.

As the analysis is still statistically limited, results are expected to
further improve by a factor of 2-3 with T2K data up to 2026. Additional data
will also allow the background treatment to be improved by using more
populated control regions.

By considering heavy neutrino production from pion decays, it would also be
possible to extend the phase space down to a few MeV/c2. When combined with a
better understanding of the expected background, this may permit the low-mass
heavy neutrino phase space ($10<M_{N}<493$ MeV/c2 and
$U_{e,\mu}^{2}>10^{-11}-10^{-10}$) to be fully explored.

The results presented in this article are available in the corresponding data
release dat (2018). It contains the signal flux and selection efficiencies for
all modes and masses, the detailed background predictions, the limits
presented in Figures 5-6-7 and the raw output of the MCMC. One can use the
latter to re-compute the limits with different priors on the mixing elements
or with different ways to present the results.

## Acknowledgments

We thank the J-PARC staff for superb accelerator performance. We thank the
CERN NA61/SHINE Collaboration for providing valuable particle production data.
We acknowledge the support of MEXT, Japan; NSERC (Grant No. SAPPJ-2014-00031),
NRC and CFI, Canada; CEA and CNRS/IN2P3, France; DFG, Germany; INFN, Italy;
National Science Centre (NCN) and Ministry of Science and Higher Education,
Poland; RSF, RFBR, and MES, Russia; MINECO and ERDF funds, Spain; SNSF and
SERI, Switzerland; STFC, UK; and DOE, USA. We also thank CERN for the
UA1/NOMAD magnet, DESY for the HERA-B magnet mover system, NII for SINET4, the
WestGrid and SciNet consortia in Compute Canada, and GridPP in the United
Kingdom. In addition, participation of individual researchers and institutions
has been further supported by funds from ERC (FP7), “la Caixa” Foundation (ID
100010434, fellowship code LCF/BQ/IN17/11620050), the European Union’s Horizon
2020 Research and Innovation programme under the Marie Sklodowska-Curie grant
agreement no. 713673 and H2020 Grant No. RISE-GA644294-JENNIFER 2020; JSPS,
Japan; Royal Society, UK; the Alfred P. Sloan Foundation and the DOE Early
Career program, USA; RFBR research project 18-32-00072, Russia.

## References

  * Drewes (2013) M. Drewes, Int.J.Mod.Phys. E22, 1330019 (2013).
  * Drewes  _et al._ (2018) M. Drewes, B. Garbrecht, P. Hernandez, M. Kekic, J. Lopez-Pavon, J. Racker, N. Rius, J. Salvado, and D. Teresi, Int. J. Mod. Phys. A33, 1842002 (2018), arXiv:1711.02862 [hep-ph] .
  * Asaka and Shaposhnikov (2005) T. Asaka and M. Shaposhnikov, Phys. Lett. B620, 17 (2005).
  * Asaka  _et al._ (2005) T. Asaka, S. Blanchet, and M. Shaposhnikov, Phys. Lett. B631, 151 (2005).
  * Artamonov  _et al._ (2015) A. V. Artamonov _et al._ (E949), Phys. Rev. D91, 052001 (2015), [Erratum: Phys. Rev.D91,no.5,059903(2015)].
  * Bernardi  _et al._ (1986) G. Bernardi _et al._ , Phys. Lett. 166B, 479 (1986).
  * Bernardi  _et al._ (1988) G. Bernardi _et al._ , Phys. Lett. B203, 332 (1988).
  * Drewes and Garbrecht (2017) M. Drewes and B. Garbrecht, Nuclear Physics B 921, 250 (2017).
  * Asaka  _et al._ (2013) T. Asaka, S. Eijima, and A. Watanabe, JHEP 03, 125 (2013), arXiv:1212.1062 [hep-ph] .
  * Abe  _et al._ (2011) K. Abe _et al._ (T2K), Nucl. Instrum. Meth. A659, 106 (2011), arXiv:1106.1238 [physics.ins-det] .
  * Abgrall  _et al._ (2011) N. Abgrall _et al._ (T2K ND280 TPC), Nucl. Instrum. Meth. A637, 25 (2011), arXiv:1012.0865 [physics.ins-det] .
  * Amaudruz  _et al._ (2012) P. A. Amaudruz _et al._ (T2K ND280 FGD), Nucl. Instrum. Meth. A696, 1 (2012), arXiv:1204.3666 [physics.ins-det] .
  * Abe  _et al._ (2015) K. Abe _et al._ (T2K), Phys. Rev. D92, 112003 (2015), arXiv:1411.6264 [hep-ex] .
  * Abe  _et al._ (2013) K. Abe _et al._ (T2K Collaboration), Physical Review D 87, 012001 (2013), arXiv:1211.0469v3 [hep-ex] .
  * Abgrall  _et al._ (2016) N. Abgrall _et al._ (NA61/SHINE), Eur. Phys. J. C76, 84 (2016), arXiv:1510.02703 [hep-ex] .
  * Shrock (1981a) R. E. Shrock, Phys. Rev. D24, 1232 (1981a).
  * Shrock (1981b) R. E. Shrock, Phys. Rev. D24, 1275 (1981b).
  * Gorbunov and Shaposhnikov (2007) D. Gorbunov and M. Shaposhnikov, Journal of High Energy Physics 2007, 015 (2007), arXiv:0705.1729v2 [hep-ph] .
  * Levy (2018) J.-M. Levy, (2018), arXiv:1805.06419 [hep-ph] .
  * Abe  _et al._ (2018) K. Abe _et al._ (T2K), Phys. Rev. D98, 012004 (2018), arXiv:1801.05148 [hep-ex] .
  * Hayato (2002) Y. Hayato,  _Proceedings, 1st International Workshop on Neutrino-nucleus interactions in the few GeV region (NuInt 01): Tsukuba, Japan, December 13-16, 2001_ , Nucl. Phys. Proc. Suppl. 112, 171 (2002).
  * Abe  _et al._ (2016) K. Abe _et al._ (T2K), Phys. Rev. Lett. 117, 192501 (2016), arXiv:1604.04406 [hep-ex] .
  * Higuera  _et al._ (2014) A. Higuera _et al._ (MINERvA Collaboration), Phys. Rev. Lett. 113, 261802 (2014).
  * Cousins and Highland (1992) R. D. Cousins and V. L. Highland, Nucl. Instrum. Meth. A320, 331 (1992).
  * Feldman and Cousins (1998) G. J. Feldman and R. D. Cousins, Phys. Rev. D57, 3873 (1998).
  * Patil  _et al._ (2010) A. Patil, D. Huard, and C. Fonnesbeck, Journal of Statistical Software, Articles 35, 1 (2010).
  * Orloff  _et al._ (2002) J. Orloff, A. Rozanov, and C. Santoni, Physics Letters B 550, 8 (2002).
  * Rasmussen and Winter (2016) R. W. Rasmussen and W. Winter, Phys. Rev. D94, 073004 (2016), arXiv:1607.07880 [hep-ph] .
  * dat (2018) http://t2k-experiment.org/results/nd280-heavy-neutrino-search-2018-datarelease/ (2018).


# BONES: a Benchmark fOr Neural Estimation of Shapley values

1st Davide Napolitano  Department of Control and Computer Engineering
Politecnico di Torino
Torino, Italy
<EMAIL_ADDRESS>2nd Luca Cagliero  Department of Control and
Computer Engineering
Politecnico di Torino
Torino, Italy
<EMAIL_ADDRESS>
###### Abstract

Shapley Values are concepts established for eXplainable AI. They are used to
explain black-box predictive models by quantifying the features’ contributions
to the model’s outcomes. Since computing the exact Shapley Values is known to
be computationally intractable on real-world datasets, neural estimators have
emerged as alternative, more scalable approaches to get approximated Shapley
Values estimates. However, experiments with neural estimators are currently
hard to replicate as algorithm implementations, explainer evaluators and
results visualizations are neither standardized nor promptly usable. To bridge
this gap, we present BONES, a new benchmark focused on neural estimation of
Shapley Value. It provides researchers with a suite of state-of-the-art neural
and traditional estimators, a set of commonly used benchmark datasets, ad hoc
modules for training black-box models, as well as specific functions to easily
compute the most popular evaluation metrics and visualize results. The purpose
is to simplify XAI model usage, evaluation, and comparison. In this paper, we
showcase BONES results and visualizations for XAI model benchmarking on both
tabular and image data. The opensource library is available at the following
link: https://github.com/DavideNapolitano/BONES.

###### Index Terms:

Explainable AI, Shapley Values, Neural Shapley Values Estimation, Benchmarking

##  I Introduction

EXplainable Artificial Intelligence (XAI) aims to make AI models more
transparent to end-users [1]. Given a black-box predictive model, XAI
solutions focus on providing explanations for the decision-making process.
Shapley Values (SVs) [2] are concepts rooted in cooperative game theory, which
have become established for XAI. SVs provide end-users with a deep
understanding of each feature’s contribution to the model’s prediction,
thereby enhancing the interpretability and trustworthiness of complex
predictors.

The exact computation of SVs from real-world data is known to be
computationally intractable [3] as the number of feature combinations is
exponential with the dimensionality of the input dataset. Hence, a number of
heuristic methods (e.g., KernelSHAP [4]) have been proposed to generate
approximated SV estimates.

The increasing availability of GPU-equipped hardware and the evolution of Deep
Learning techniques has fostered the study of Neural Network-based approaches
to compute approximated SVs. During the training process, these networks learn
the functional mapping between the input data features and their SVs
attributions. State-of-the-art neural approaches (e.g., FastSHAP [5]) are
currently able to efficiently generate accurate SVs estimates.

As a drawback, neural SVs estimators are currently neither promptly accessible
nor easy to use. Actually, the most popular XAI projects (e.g., Quantus [6],
OpenXAI [7], Compare-xAI [8]) lack neural solutions. Furthermore, there is a
lack of standardization in model testing, evaluation, and comparison. This
limits the applicability of neural approaches compared to more popular
approximated methods such as Monte Carlo sampling [9] and regression
techniques [10].

We present BONES, a Benchmark fOr Neural Estimation of Shapley values, aimed
to foster XAI applications that mainly rely on neural SVs estimators. BONES
consists of

  * •

A suite of Shapley Values estimators, mainly neural and some traditional,
tightly integrated for easy comparison and use;

  * •

A set of benchmark datasets, both in tabular form and images, that are
commonly used for XAI model benchmarking;

  * •

Ad hoc modules to train black-box models and generate reliable ground truth
SVs, whenever not already available, either exact or approximated.

  * •

A set of testing functions implementing the most popular performance
evaluation metrics;

  * •

A set of promptly interactive plots that can be used to visually explore the
main results and compare models’ performance with each other.

BONES simplifies and expedites the use of state-of-the-art neural approaches
and allows end-users to perform accurate model comparisons considering aspects
such as computational efficiency, attribution accuracy, model robustness to
data cardinality and dimensionality. We hope BONES could effectively support
XAI researchers interested in exploring the strengths and limitations of
neural solutions.

##  II Related Works

###  II-A XAI tools

As discussed in [11, 12], the attention of the research community to the
eXplainable AI (XAI) field is ever-increasing. To actively support related
research activities, several XAI benchmarks and tools have been released,
e.g., SHAP [4], Quantus [6], OpenXAI [7], Compare-xAI [8], and Ferret [13].
The purpose is to allow fair and transparent comparisons among different XAI
methods by making available suites of state-of-the-art algorithms, datasets,
evaluation metrics, and visualization techniques. However, existing libraries
do not incorporate the latest neural approaches. Thus, comparing neural
Shapley Value estimators with each other or with traditional approaches
requires additional effort. BONES addresses the above limitation by providing
researchers with promptly usable implementations of state-of-the-art Shapley
Values estimators, both neural and traditional, as well as a testing suite
including benchmark tabular datasets, evaluation metrics, and visualization
tools. Our solution welcomes future extensions towards the integration of new
algorithms, datasets, and standardized evaluation procedures.

###  II-B XAI models

Understanding how AI models make decisions is crucial for augmenting their
transparency and interpretability, especially because most AI predictors act
as black boxes. Feature importance attribution measures how much each input
feature contributes to a model’s predictions. Among existing methods, Shapley
Values are popular due to their solid mathematical basis, as they fairly
distribute the model’s output among the input features based on their
contributions and interactions. However, computing Shapley Values is often
impractical. To address this issue, researchers have developed methods to
approximate Shapley Values and, as alternatives, other techniques to compute
feature relevance, like permutation importance [14], LIME [15], and DeepLIFT
[16]. Empirical studies have compared various Shapley Values approximation
methods, highlighting the trade-offs between accuracy, computational
efficiency, and robustness. Existing analysis [17] provide a comprehensive
evaluation of different Shapley Value approximation methods, showing that
while neural approaches can significantly reduce computation time, their
approximation accuracy varies depending upon the model architecture and
dataset characteristics.

#### Traditional Approaches

Classical methods to compute Shapley Values involve exact computation [4],
Monte Carlo sampling [9], and regression techniques [10, 4]. Exact computation
evaluates the model on all possible subsets of features, providing precise
Shapley Values but at a computationally prohibitive cost. This approach is not
applicable to models trained on many features due to the combinatorial
explosion in the number of candidate subsets. Monte Carlo sampling methods
approximate Shapley Values by averaging over random subsets of features,
reducing computational burden but often requiring a large number of sampling
iterations to achieve accurate results. Regression techniques, such as
KernelSHAP [4] and Unbiased KernelSHAP [10], are also used to approximate
Shapley Values using linear regressions, allowing for improved computational
efficiency.

#### Neural Approaches

Although traditional methods are accurate, they often have computational
problems when scaling up the dataset size, making them impractical, especially
at inference time. Regarding existing neural approaches, DeepExplainer is part
of the SHAP library [4]. It consists in an enhanced version of DeepLift [16],
which recursively attributes the difference in the model’s output between each
input sample and the corresponding background sample back to the input
features, significantly improving the computational efficiency over
traditional methods. GradientExplainer leverages integrated gradient-based
attributions [18] with SHAP values, utilizing the gradients of the output with
respect to the inputs to approximate feature contributions more efficiently.
FastSHAP [5] employs a neural network to learn a mapping from model inputs to
Shapley Values, reducing the computation time, especially on large datasets,
by approximating the complex Shapley Value calculations through a learned
function. DASP (Differentiable Approximation of Shapley values) [19]
introduces a polynomial-time algorithm that leverages neural network
architectures to approximate Shapley Values, enhancing the scalability and
efficiency of the computation process. ViT-Shapley [20], designed mainly for
Vision Transformers (ViTs) [21], adapts the Shapley Value computation to the
unique architecture of ViTs, providing interpretable explanations for image
classification tasks by learning to estimate the contribution of image patches
to the model’s predictions. Other techniques, such as ShapNet [22], focus on
computing Shapley Values from ground truth data, making them unsuitable for
explaining black-box models.

##  III The BONES Benchmark

BONES is a benchmark for neural Shapley Values estimation. It consists of the
following modules:

  * •

Black-Box Models: it generates post-hoc explanations of arbitrary
classification of various types and with various settings.

  * •

XAI Models: it integrates a variety of approaches to approximated SVs
estimations, both neural and not.

  * •

Datasets: it provides access to several benchmark datasets, both tabular and
image data.

  * •

Ground Truth: it supports the computation of both exact SVs [4] and of
regression-based estimations [10] that can be used as alternative ground
truths.

  * •

Evaluation Functions: it allows to quantify the accuracy of the SVs’ estimates
against the ground truth and the efficiency of the estimation process, as well
as to compare different models with each other.

  * •

Visualization: it natively supports the generation of various plots useful to
perform exploratory analysis of the models’ results and of their accuracy-
efficiency ratio.

The design of BONES ensures maximal usability, portability, and extendabily.
The key properties are summarized below.

  * •

Modality-Agnostic. A core strength of our framework is its modality
agnosticism by-design. Shapley Values are potentially applicable across
various data modalities such as image, tabular and text data. Our framework is
designed to support a wide range of approaches and data types, ensuring its
applicability in different input types domains. This broad applicability is
crucial for researchers and practitioners who deal with data in different
modalities and require reliable explainability standards. Currently, BONES
supports tabular and image data. The extension to other modalities is already
planned as a future work.

  * •

Post-Hoc Explanations. Our benchmark allows end-users to explain predictions
of already trained models. This aspect of the framework is particularly
valuable for practical applications, where models are often trained in a
production environment and explanability needs to be retrofitted to provide
insights into model behavior and decision-making processes.

  * •

Opensource, modular framework. To foster collaboration, reproducibility, and
extensibility, our framework is designed with an open and modular
architecture. The open BONES benchmark fosters contributions from the broader
research community, facilitating the integration of new methods, datasets, and
evaluation metrics. Modularity ensures that components of the framework can be
independently developed, tested, and replaced. This flexibility allows users
to customize the framework to suit their specific needs, whether that involves
incorporating new neural architectures, experimenting with alternative Shapley
value estimation techniques, or adapting the benchmark to novel
interpretability challenges.

In the following we detail the characteristics of the BONES components.

###  III-A Datasets

BONES is currently suited to both tabular and image data. The benchmark is
designed to facilitate the seamless integration and utilization of both
proprietary and benchmark datasets such as those available in the UCI
repository [23]. The current list of integrated datasets is given in Table I.

For tabular data, we choose a subset of datasets that are representative of
different cardinality, dimensionality, and density distributions. For image
data we include datasets covering different aspects of visual information and
model explainability. In detail, we integrate ImageNette [24] and Pet [25] by
adopting the same configuration as in Vit-Shapley [20].

TABLE I: Benchmark datasets. |  | Dataset
---
| Source
---
| Train
---
samples
| Validation
---
samples
| Num.
---
features
Tabular | Monks | UCI | 302 | 130 | 6 |
WBC | UCI | 436 | 110 | 9 |
Census | SHAP | 20838 | 5210 | 12 |
Credit | UCI | 19200 | 4800 | 23 |
Magic | UCI | 12172 | 3044 | 10 |
Image | ImageNette | ViT-Shapley | 9469 | 1963 | 224x224 |
Pet | ViT-Shapley | 5879 | 735 | 224x224 |

###  III-B Explainers

BONES provides a comprehensive suite of SVs estimators, both neural and not.
The list of currently available XAI models is reported in Table II, where
column Type differentiates between traditional and neural estimators. We
standardize the integration process to make the module easily extensible with
newly proposed approaches. The implementation currently rely on TensorFlow and
PyTorch for tabular data and on PyTorch for images.

For tabular data, our framework supports several approaches, including SHAP
[4], i.e., the Exact, GradientSHAP, and DeepSHAP versions, ShapleyRegression
[10] with Unbiased KernelSHAP and KernelSHAP, Monte Carlo Sampling [9], DASP
[19]111BONES currently integrates the authors’ implementation relying on
TensorFlow version 1., and FastSHAP [5]. For image data, the framework
currently includes SHAP [4] (i.e., DeepSHAP and GradientSHAP variants),
FastSHAP [5], and ViT-Shapley [20] for Vision Transformers.

TABLE II: Explainers |  | Model
---
| Type
---
| Supported
---
Framework
| Black-Box
---
Type
Tabular | Exact | Traditional | All | All
Unbiased KS | Traditional | All | All
KernelSHAP | Traditional | All | All
MonteCarlo | Traditional | All | All
DeepExplainer | Neural | PT/TF | Neural
GradientExplainer | Neural | PT/TF | Neural
DASP | Neural | TF 1 | Neural
FastSHAP | Neural | PT/TF | All
Image | DeepExplainer | Neural | PT/TF | Neural
GradientExplainer | Neural | PT/TF | Neural
FastSHAP | Neural | PT/TF | All
ViT-Shapley | Neural | PT | ResNet/DeepNet/ViT

###  III-C Black-Box Models

Most neural explainers are suited to explain neural Network-based models only
(see Column Black-Box type in Table II). However, latest approaches (e.g.,
FastSHAP) are compatible with non-neural classifiers as well.

As default black-box models, BONES exploits:

  * •

For tabular data, a Multi-Layer Perceptron classifier with two intermediate
dense layers, each containing 64 units and ReLU activation, interspersed with
dropout layers. The final dense layer has an output corresponding to the
number of classes, followed by a softmax activation. This implementation
relies on Tensorflow;

  * •

For image data, a pre-trained Vision Transformer (ViT) in its tiny version
[21], followed by a linear layer corresponding to the number of classes, is
used for classification.

Thanks to its modularity and extensibility, BONES straightforwardly supports
the integration of traditional non-neural classifiers as well (e.g., the
classifiers available in the Scikit-Learn library [26]).

###  III-D Evaluation functions

#### Estimation error

BONES natively supports evaluation functions suited to quantify the prediction
error made by a SVs estimator against a ground truth. It integrates the L1 and
L2 distances. Furthermore, for tabular data only, it also supports the Kendall
correlation coefficient, which evaluates the consistency in the SVs feature
ranking.

#### Computational cost

To evaluate the efficiency of the XAI models, BONES keeps track of the
explainers’ training and inference times.

#### Comparative analysis

To compare the performance of different explainers with each other, BONES
supports the following performance metric $P$:

$P=1-\frac{d_{i}-d_{min}}{d_{max}-d_{min}}$ (1)

where $d_{i}$ is the distance metric of the $i-th$ explainer, whereas
$d_{min}$ and $d_{max}$ are the minimum and maximum values on the same
distance metric across all analyzed explainers, respectively. This metric
provides a value from 0 to 1, where a higher value highlights better
performances.

To compare the performance of image explainers, BONES also supports the
Inclusion and Exclusion AUC (Area Under the Curve) [5]. Inclusion evaluates
how well an image explainer identifies important regions by measuring the
increase in the model’s prediction score as these regions are progressively
included. Exclusion assesses the impact of removing important regions
identified by the explainer on the model’s prediction score. Their combined
use allows end-users to identify the best-performing image explainer, i.e.,
the model with maximal Inclusion and minimal Exclusion [5].

###  III-E Visualization

BONES offers the following options to visualize the performance results of SVs
estimators and to graphically compare them with each other:

  * •

Bar plot: It displays the local or global per-feature SVs computed by
different explainers. This visualization allows for a direct comparison of the
feature importance assigned by each explainer. This visualization is mainly
intended for tabular data.

  * •

Image plot: For image data only, it graphically shows the mask of Shapley
Values retrieved by different explainer pairs overlaid on the input processed
image. In detail, a 14x14 pixels mask is used for all approaches,
interpolating ones providing pixel-wise explanations.

  * •

AUC curves: It plots the Inclusion and Exclusion AUC for image data only. AUC
shows the predictor accuracy by varying the percentage of Inclusion/Exclusion.

  * •

Quadrant plot: The quadrant plot is computed based on overall times and our
performance metric $P$ (1). It offers a comprehensive view of the
computational efficiency and performance of different explainers, aiding in
selecting the most suitable method for a given task.

  * •

Computational times vs. number of samples plot: This plot visualizes the
model’s computational times by varying the number of samples processed on the
chosen dataset. End-users can vary the total number of samples, the interval
between the tested values, and the sampling techniques applied to the input
dataset. It provides end-users with insights into the explainers’ scalability
of explainers with the dataset cardinality.

  * •

Computational times vs. number of features plot: This plots show the inference
times spent by the XAI model by varying the number of input features.

##  IV Case Study

We showcase the usability and flexibility of BONES using two example case
studies, one on a tabular dataset, i.e., Monks [23], and on a image dataset,
i.e., ImageNette [24].

###  IV-A Tabular Data

⬇

from bones.sv.tabular.explainers import FastSHAPModel, ShapRegModel,
DASPModel, ExactModel

from bones.sv.tabular.datasets import Monks, Census

from bones.sv.tabular.metrics import L1, L2, Kendal

from bones.sv.tabular.evaluation import Banckmark

from bones.sv.tabular.display import TimeSamplePlot, TimeFeaturePlot, BarPlot,
QuadrantPlot

benchmark=Benchmark(explainers=[FastSHAP, ShapReg, DASP], ground_truth=Exact,
dataset=[Monks, Census, Credit], metrics=[L1, L2, Kendal],
num_samples=100).run()

benchmark.print_results(Monks) # table results

TimeSamplePlot(benchmark, dataset=Monks, number_sample=100000, interval=10000,
sample_method="random").plot()

TimeFeaturePlot(benchmark).plot()

BarPlot(bechmark, dataset=Monks).plot()

QuadrantPlot(benchmark, dataset=Monks).plot()

Listing 1: Example of code snippet for tabular data.

The snippet in Listing 1 shows how to efficiently import SVs estimators and
datasets, select the preferred evaluation metric, and generate the
corresponding plots.

Figure 1: Example of bar plot comparing the global Shapley Values estimated by
six explainers on the Monks dataset against the ground truth (i.e., Exact).

Figure 1 shows the resulting bar plot, which compares the outcomes of several
approximated SVs estimators with the Exact SVs estimates [4]. Specifically,
for each explainer, the bars correspond to the Global Shapley Value relative
to a specific feature computed over the whole dataset. The more similar to the
Exact SVs, the better. Besides global evaluations, this visualization supports
local inspection when a dataset sample is specified. Figure 3 shows the
inference times by varying the number of features (upper image) and samples
(lower image), respectively. Unlike all the other approaches, FastSHAP [5] has
an inference time per sample negligble compared to its training time. Hence,
the per-sample variation is flattened. Finally, the quadrant plot in Figure 2
allows a graphical comparison between the tested models in terms of L2
distance-Computational time ratio. In this dataset analysis, Neural approaches
have shown to be consistently better than traditional models (e.g., MonteCarlo
sampling [9], KernelSHAP [10]).

Figure 2: Quadrant plot combining computational times and a L2 distance
metric.

###  IV-B Image Data

Figure 4: Image plot: comparison of the Shapley Values’ masks computed by the
different explainers on a ImageNette sample.

The snippet in Listing 2 shows a similar example tailored to image data. BONES
offers the opportunity to visualize the Inclusion and Exclusion AUC plot (see
Figure 5). For example, the experiments on ImageNette confirm the better
capabilities of FastSHAP to avoid excluding discriminating regions. The Image
plot in Figure 4 allows end-users to select a sample and view the masks of the
Shapley Values calculated by the various methods. This is particularly
interesting for users who would like to quickly perform qualitative analysis.

Figure 3: Examplse of visualizations plots showing the variations of the
computational times with the number of dataset features (upper plot) and the
number of dataset samples (bottom).

⬇

from bones.sv.image.explainers import FastSHAP, ViTShapley, DeepExplainer,
GradientExplainer

from bones.sv.image.datasets import ImageNette

from bones.sv.image.metrics import L1, L2, AUC

from bones.sv.image.evaluation import Banckmark

from bones.sv.image.display import ImagePlot, AUC

benchmark=Benchmark(explainers=[ViTShapley, DeepExplainer, GradientExplainer],
ground_truth=FastSHAP, dataset=[ImageNette], metrics=[L1, L2, AUC],
num_samples=100).run()

# results, TimeSample and Quadrant as for Tabular data

ImagePlot(bechmark, datset=ImageNette, sample=0).plot()

AUC(benchmark, dataset=ImageNette, num_sample=100).plot()

Listing 2: Example of code snippet for image data. |
---|---
Figure 5: AUC Exclusion (left) and Inclusion (right) computed on ImageNette.

##  V Conclusions and future work

The paper presented BONES, a benchmarking library for neural Shapley Values
estimation. The main purpose is to make neural estimators available and easy-
to-use to researchers of the XAI community, leveraging applications to real-
world data and quantitative models’ comparisons. As future work, we plan to
extend BONES to support a broader range of modalities, models, and datasets.
Additionally, we aim to integrate variants of Shapley Values into BONES,
including Shapley Residuals and Interval Shapley Values.

##  VI Acknowledgements

This study was carried out within the FAIR - Future Artificial Intelligence
Research and received funding from the European Union Next-GenerationEU (PNRR
M4C2, INVESTIMENTO 1.3 D.D. 1555 11/10/2022, PE00000013). This manuscript
reflects only the authors’ views and opinions, neither the European Union nor
the European Commission can be considered responsible for them. The research
leading to these results has been partly funded by the SmartData@PoliTO center
for Big Data and Machine Learning technologies.

## References

  * [1] W. Saeed and C. Omlin, “Explainable ai (xai): A systematic meta-survey of current challenges and future opportunities,” _Knowledge-Based Systems_ , vol. 263, p. 110273, 2023.
  * [2] L. S. Shapley, “A value for n-person games,” in _Contributions to the Theory of Games II_ , H. W. Kuhn and A. W. Tucker, Eds. Princeton: Princeton University Press, 1953, pp. 307–317.
  * [3] G. V. den Broeck, A. Lykov, M. Schleich, and D. Suciu, “On the tractability of SHAP explanations,” _J. Artif. Intell. Res._ , vol. 74, pp. 851–886, 2022.
  * [4] S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” in _Advances in Neural Information Processing Systems 30_ , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds. Curran Associates, Inc., 2017, pp. 4765–4774.
  * [5] N. Jethani, M. Sudarshan, I. C. Covert, S. Lee, and R. Ranganath, “Fastshap: Real-time shapley value estimation,” in _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net, 2022.
  * [6] A. Hedström, L. Weber, D. Krakowczyk, D. Bareeva, F. Motzkus, W. Samek, S. Lapuschkin, and M. M.-C. Höhne, “Quantus: An explainable ai toolkit for responsible evaluation of neural network explanations and beyond,” _Journal of Machine Learning Research_ , vol. 24, no. 34, pp. 1–11, 2023.
  * [7] C. Agarwal, S. Krishna, E. Saxena, M. Pawelczyk, N. Johnson, I. Puri, M. Zitnik, and H. Lakkaraju, “Openxai: Towards a transparent evaluation of model explanations,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 15 784–15 799, 2022.
  * [8] M. K. Belaid, R. Bornemann, M. Rabus, R. Krestel, and E. Hüllermeier, “Compare-xai: Toward unifying functional testing methods for post-hoc xai algorithms into a multi-dimensional benchmark,” in _World Conference on Explainable Artificial Intelligence_. Springer, 2023, pp. 88–109.
  * [9] E. Štrumbelj and I. Kononenko, “Explaining prediction models and individual predictions with feature contributions,” _Knowledge and information systems_ , vol. 41, pp. 647–665, 2014.
  * [10] I. Covert and S.-I. Lee, “Improving kernelshap: Practical shapley value estimation via linear regression,” _arXiv preprint arXiv:2012.01536_ , 2020.
  * [11] P. Q. Le, M. Nauta, V. B. Nguyen, S. Pathak, J. Schlötterer, and C. Seifert, “Benchmarking explainable ai: a survey on available toolkits and open challenges,” in _International Joint Conference on Artificial Intelligence_ , 2023.
  * [12] R. Dwivedi, D. Dave, H. Naik, S. Singhal, R. Omer, P. Patel, B. Qian, Z. Wen, T. Shah, G. Morgan, and R. Ranjan, “Explainable ai (xai): Core ideas, techniques, and solutions,” _ACM Comput. Surv._ , vol. 55, no. 9, jan 2023.
  * [13] G. Attanasio, E. Pastor, C. Di Bonaventura, and D. Nozza, “ferret: a framework for benchmarking explainers on transformers,” in _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations_ , D. Croce and L. Soldaini, Eds. Dubrovnik, Croatia: Association for Computational Linguistics, May 2023, pp. 256–266.
  * [14] A. Altmann, L. Toloşi, O. Sander, and T. Lengauer, “Permutation importance: a corrected feature importance measure,” _Bioinformatics_ , vol. 26, no. 10, pp. 1340–1347, 2010.
  * [15] M. T. Ribeiro, S. Singh, and C. Guestrin, “‘’why should i trust you?” explaining the predictions of any classifier,” in _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , 2016, pp. 1135–1144.
  * [16] A. Shrikumar, P. Greenside, and A. Kundaje, “Learning important features through propagating activation differences,” in _International conference on machine learning_. PMLR, 2017, pp. 3145–3153.
  * [17] H. Chen, I. C. Covert, S. M. Lundberg, and S.-I. Lee, “Algorithms to estimate shapley value feature attributions,” _Nature Machine Intelligence_ , vol. 5, no. 6, pp. 590–601, 2023.
  * [18] M. Sundararajan, A. Taly, and Q. Yan, “Axiomatic attribution for deep networks,” in _Proceedings of the 34th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, D. Precup and Y. W. Teh, Eds., vol. 70. PMLR, 06–11 Aug 2017, pp. 3319–3328.
  * [19] M. Ancona, C. Oztireli, and M. Gross, “Explaining deep neural networks with a polynomial time algorithm for shapley value approximation,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 272–281.
  * [20] I. Covert, C. Kim, and S.-I. Lee, “Learning to estimate shapley values with vision transformers,” 2023.
  * [21] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” _ICLR_ , 2021.
  * [22] R. Wang, X. Wang, and D. I. Inouye, “Shapley explanation networks,” _arXiv preprint arXiv:2104.02297_ , 2021.
  * [23] K. Bache and M. Lichman, “UCI machine learning repository,” 2013.
  * [24] J. Howard and S. Gugger, “Fastai: a layered api for deep learning,” _Information_ , vol. 11, no. 2, p. 108, 2020.
  * [25] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar, “Cats and dogs,” in _2012 IEEE conference on computer vision and pattern recognition_. IEEE, 2012, pp. 3498–3505.
  * [26] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,” _Journal of Machine Learning Research_ , vol. 12, pp. 2825–2830, 2011.