Kim580000-0001-7161-2133, E. Kuznetsova59, V. Murzin, V. Oreshkin, I. Smirnov,
D. Sosnov0000-0002-7452-8380, V. Sulimov, L. Uvarov, S. Volkov, A. Vorobyev
Institute for Nuclear Research, Moscow, Russia Yu. Andreev0000-0002-7397-9665,
A. Dermenev, S. Gninenko0000-0001-6495-7619, N. Golubev, A.
Karneyeu0000-0001-9983-1004, D. Kirpichnikov0000-0002-7177-077X, M. Kirsanov,
N. Krasnikov, A. Pashenkov, G. Pivovarov0000-0001-6435-4463, A. Toropin
Institute for Theoretical and Experimental Physics named by A.I. Alikhanov of
NRC ‘Kurchatov Institute’, Moscow, Russia V. Epshteyn, V. Gavrilov, N.
Lychkovskaya, A. Nikitenko60, V. Popov, A. Stepennov, M. Toms, E.
Vlasov0000-0002-8628-2090, A. Zhokin Moscow Institute of Physics and
Technology, Moscow, Russia T. Aushev National Research Nuclear University
’Moscow Engineering Physics Institute’ (MEPhI), Moscow, Russia O. Bychkova, R.
Chistov610000-0003-1439-8390, M. Danilov610000-0001-9227-5164, A. Oskin, P.
Parygin, S. Polikarpov610000-0001-6839-928X P.N. Lebedev Physical Institute,
Moscow, Russia V. Andreev, M. Azarkin, I. Dremin0000-0001-7451-247X, M.
Kirakosyan, A. Terkulov Skobeltsyn Institute of Nuclear Physics, Lomonosov
Moscow State University, Moscow, Russia A. Belyaev, E.
Boos0000-0002-0193-5073, V. Bunichev, M. Dubinin620000-0002-7766-7175, L.
Dudko0000-0002-4462-3192, A. Gribushin, V. Klyukhin0000-0002-8577-6531, N.
Korneeva0000-0003-2461-6419, I. Lokhtin0000-0002-4457-8678, S. Obraztsov, M.
Perfilov, V. Savrin, P. Volkov Novosibirsk State University (NSU),
Novosibirsk, Russia V. Blinov63, T. Dimova63, L. Kardapoltsev63, A. Kozyrev63,
I. Ovtin63, O. Radchenko63, Y. Skovpen630000-0002-3316-0604 Institute for High
Energy Physics of National Research Centre ‘Kurchatov Institute’, Protvino,
Russia I. Azhgirey0000-0003-0528-341X, I. Bayshev, D. Elumakhov, V. Kachanov,
D. Konstantinov0000-0001-6673-7273, P. Mandrik0000-0001-5197-046X, V. Petrov,
R. Ryutin, S. Slabospitskii0000-0001-8178-2494, A. Sobol, S.
Troshin0000-0001-5493-1773, N. Tyurin, A. Uzunian, A. Volkov National Research
Tomsk Polytechnic University, Tomsk, Russia A. Babaev, V. Okhotnikov Tomsk
State University, Tomsk, Russia V. Borshch, V. Ivanchenko0000-0002-1844-5433,
E. Tcherniaev0000-0002-3685-0635 University of Belgrade: Faculty of Physics
and VINCA Institute of Nuclear Sciences, Belgrade, Serbia P.
Adzic640000-0002-5862-7397, M. Dordevic0000-0002-8407-3236, P.
Milenovic0000-0001-7132-3550, J. Milosevic0000-0001-8486-4604 Centro de
Investigaciones Energéticas Medioambientales y Tecnológicas (CIEMAT), Madrid,
Spain M. Aguilar-Benitez, J. Alcaraz Maestre0000-0003-0914-7474, A. Álvarez
Fernández, I. Bachiller, M. Barrio Luna, Cristina F.
Bedoya0000-0001-8057-9152, C.A. Carrillo Montoya0000-0002-6245-6535, M.
Cepeda0000-0002-6076-4083, M. Cerrada, N. Colino0000-0002-3656-0259, B. De La
Cruz, A. Delgado Peris0000-0002-8511-7958, J.P. Fernández
Ramos0000-0002-0122-313X, J. Flix0000-0003-2688-8047, M.C.
Fouz0000-0003-2950-976X, O. Gonzalez Lopez0000-0002-4532-6464, S. Goy
Lopez0000-0001-6508-5090, J.M. Hernandez0000-0001-6436-7547, M.I.
Josa0000-0002-4985-6964, J. León Holgado0000-0002-4156-6460, D. Moran, Á.
Navarro Tobar0000-0003-3606-1780, C. Perez Dengra, A. Pérez-Calero
Yzquierdo0000-0003-3036-7965, J. Puerta Pelayo0000-0001-7390-1457, I.
Redondo0000-0003-3737-4121, L. Romero, S. Sánchez Navas, L. Urda
Gómez0000-0002-7865-5010, C. Willmott Universidad Autónoma de Madrid, Madrid,
Spain J.F. de Trocóniz Universidad de Oviedo, Instituto Universitario de
Ciencias y Tecnologías Espaciales de Asturias (ICTEA), Oviedo, Spain B.
Alvarez Gonzalez0000-0001-7767-4810, J. Cuevas0000-0001-5080-0821, C.
Erice0000-0002-6469-3200, J. Fernandez Menendez0000-0002-5213-3708, S.
Folgueras0000-0001-7191-1125, I. Gonzalez Caballero0000-0002-8087-3199, J.R.
González Fernández, E. Palencia Cortezon0000-0001-8264-0287, C. Ramón Álvarez,
V. Rodríguez Bouza0000-0002-7225-7310, A. Soto Rodríguez, A. Trapote, N.
Trevisani0000-0002-5223-9342, C. Vico Villalba Instituto de Física de
Cantabria (IFCA), CSIC-Universidad de Cantabria, Santander, Spain J.A.
Brochero Cifuentes0000-0003-2093-7856, I.J. Cabrillo, A.
Calderon0000-0002-7205-2040, J. Duarte Campderros0000-0003-0687-5214, M.
Fernandez0000-0002-4824-1087, C. Fernandez Madrazo0000-0001-9748-4336, P.J.
Fernández Manteca0000-0003-2566-7496, A. García Alonso, G. Gomez, C. Martinez
Rivero, P. Martinez Ruiz del Arbol0000-0002-7737-5121, F.
Matorras0000-0003-4295-5668, P. Matorras Cuevas0000-0001-7481-7273, J. Piedra
Gomez0000-0002-9157-1700, C. Prieels, A. Ruiz-Jimeno0000-0002-3639-0368, L.
Scodellaro0000-0002-4974-8330, I. Vila, J.M. Vizan Garcia0000-0002-6823-8854
University of Colombo, Colombo, Sri Lanka M.K. Jayananda, B. Kailasapathy65,
D.U.J. Sonnadara, D.D.C. Wickramarathna University of Ruhuna, Department of
Physics, Matara, Sri Lanka W.G.D. Dharmaratna0000-0002-6366-837X, K. Liyanage,
N. Perera, N. Wickramage CERN, European Organization for Nuclear Research,
Geneva, Switzerland T.K. Aarrestad0000-0002-7671-243X, D. Abbaneo, J.
Alimena0000-0001-6030-3191, E. Auffray, G. Auzinger, J. Baechler, P.
Baillon${}^{\textrm{\textdagger}}$, D. Barney0000-0002-4927-4921, J. Bendavid,
M. Bianco0000-0002-8336-3282, A. Bocci0000-0002-6515-5666, C. Caillol, T.
Camporesi, M. Capeans Garrido0000-0001-7727-9175, G. Cerminara, N.
Chernyavskaya0000-0002-2264-2229, S.S. Chhibra0000-0002-1643-1388, S.
Choudhury, M. Cipriani0000-0002-0151-4439, L. Cristella0000-0002-4279-1221, D.
d’Enterria0000-0002-5754-4303, A. Dabrowski0000-0003-2570-9676, A.
David0000-0001-5854-7699, A. De Roeck0000-0002-9228-5271, M.M.
Defranchis0000-0001-9573-3714, M. Deile0000-0001-5085-7270, M. Dobson, M.
Dünser0000-0002-8502-2297, N. Dupont, A. Elliott-Peisert, F. Fallavollita66,
A. Florent0000-0001-6544-3679, L. Forthomme0000-0002-3302-336X, G.
Franzoni0000-0001-9179-4253, W. Funk, S. Ghosh0000-0001-6717-0803, S. Giani,
D. Gigi, K. Gill, F. Glege, L. Gouskos0000-0002-9547-7471, M.
Haranko0000-0002-9376-9235, J. Hegeman0000-0002-2938-2263, V.
Innocente0000-0003-3209-2088, T. James, P. Janot0000-0001-7339-4272, J.
Kaspar0000-0001-5639-2267, J. Kieseler0000-0003-1644-7678, M.
Komm0000-0002-7669-4294, N. Kratochwil, C. Lange0000-0002-3632-3157, S.
Laurila, P. Lecoq0000-0002-3198-0115, A. Lintuluoto, K.
Long0000-0003-0664-1653, C. Lourenço0000-0003-0885-6711, B. Maier, L.
Malgeri0000-0002-0113-7389, S. Mallios, M. Mannelli, A.C.
Marini0000-0003-2351-0487, F. Meijers, S. Mersi0000-0003-2155-6692, E.
Meschi0000-0003-4502-6151, F. Moortgat0000-0001-7199-0046, M.
Mulders0000-0001-7432-6634, S. Orfanelli, L. Orsini, F.
Pantaleo0000-0003-3266-4357, E. Perez, M. Peruzzi0000-0002-0416-696X, A.
Petrilli, G. Petrucciani0000-0003-0889-4726, A. Pfeiffer0000-0001-5328-448X,
M. Pierini0000-0003-1939-4268, D. Piparo, M. Pitt0000-0003-2461-5985, H.
Qu0000-0002-0250-8655, T. Quast, D. Rabady0000-0001-9239-0605, A. Racz, G.
Reales Gutiérrez, M. Rovere, H. Sakulin, J. Salfeld-Nebgen0000-0003-3879-5622,
S. Scarfi, C. Schäfer, C. Schwick, M. Selvaggi0000-0002-5144-9655, A. Sharma,
P. Silva0000-0002-5725-041X, W. Snoeys0000-0003-3541-9066, P.
Sphicas670000-0002-5456-5977, S. Summers0000-0003-4244-2061, K.
Tatar0000-0002-6448-0168, V.R. Tavolaro0000-0003-2518-7521, D. Treille, P.
Tropea, A. Tsirou, J. Wanczyk68, K.A. Wozniak, W.D. Zeuner Paul Scherrer
Institut, Villigen, Switzerland L. Caminada690000-0001-5677-6033, A.
Ebrahimi0000-0003-4472-867X, W. Erdmann, R. Horisberger, Q. Ingram, H.C.
Kaestli, D. Kotlinski, U. Langenegger, M. Missiroli690000-0002-1780-1344, L.
Noehte69, T. Rohe ETH Zurich - Institute for Particle Physics and Astrophysics
(IPA), Zurich, Switzerland K. Androsov680000-0003-2694-6542, M.
Backhaus0000-0002-5888-2304, P. Berger, A. Calandri0000-0001-7774-0099, A. De
Cosa, G. Dissertori0000-0002-4549-2569, M. Dittmar, M. Donegà, C.
Dorfer0000-0002-2163-442X, F. Eble, K. Gedia, F. Glessgen, T.A. Gómez
Espinosa0000-0002-9443-7769, C. Grab0000-0002-6182-3380, D. Hits, W.
Lustermann, A.-M. Lyon, R.A. Manzoni0000-0002-7584-5038, L.
Marchese0000-0001-6627-8716, C. Martin Perez, M.T. Meinhard, F. Nessi-Tedaldi,
J. Niedziela0000-0002-9514-0799, F. Pauss, V. Perovic, S.
Pigazzini0000-0002-8046-4344, M.G. Ratti0000-0003-1777-7855, M. Reichmann, C.
Reissel, T. Reitenspiess, B. Ristic0000-0002-8610-1130, D. Ruini, D.A. Sanz
Becerra0000-0002-6610-4019, V. Stampf, J. Steggemann680000-0003-4420-5510, R.
Wallny0000-0001-8038-1613 Universität Zürich, Zurich, Switzerland C.
Amsler700000-0002-7695-501X, P. Bärtschi, C. Botta0000-0002-8072-795X, D.
Brzhechko, M.F. Canelli0000-0001-6361-2117, K. Cormier, A. De
Wit0000-0002-5291-1661, R. Del Burgo, J.K. Heikkilä0000-0002-0538-1469, M.
Huwiler, W. Jin, A. Jofrehei0000-0002-8992-5426, B.
Kilminster0000-0002-6657-0407, S. Leontsinis0000-0002-7561-6091, S.P. Liechti,
A. Macchiolo0000-0003-0199-6957, P. Meiring, V.M. Mikuni0000-0002-1579-2421,
U. Molinatti, I. Neutelings, A. Reimers, P. Robmann, S. Sanchez
Cruz0000-0002-9991-195X, K. Schweiger0000-0002-5846-3919, M. Senger, Y.
Takahashi0000-0001-5184-2265 National Central University, Chung-Li, Taiwan C.
Adloff71, C.M. Kuo, W. Lin, A. Roy0000-0002-5622-4260, T.
Sarkar390000-0003-0582-4167, S.S. Yu National Taiwan University (NTU), Taipei,
Taiwan L. Ceard, Y. Chao, K.F. Chen0000-0003-1304-3782, P.H.
Chen0000-0002-0468-8805, P.s. Chen, H. Cheng0000-0001-6456-7178, W.-S.
Hou0000-0002-4260-5118, Y.y. Li, R.-S. Lu, E. Paganis0000-0002-1950-8993, A.
Psallidas, A. Steen, H.y. Wu, E. Yazgan0000-0001-5732-7950, P.r. Yu
Chulalongkorn University, Faculty of Science, Department of Physics, Bangkok,
Thailand B. Asavapibhop0000-0003-1892-7130, C.
Asawatangtrakuldee0000-0003-2234-7219, N. Srimanobhas0000-0003-3563-2959
Çukurova University, Physics Department, Science and Art Faculty, Adana,
Turkey F. Boran0000-0002-3611-390X, S. Damarseckin72, Z.S.
Demiroglu0000-0001-7977-7127, F. Dolek0000-0001-7092-5517, I.
Dumanoglu730000-0002-0039-5503, E. Eskut, Y. Guler740000-0001-7598-5252, E.
Gurpinar Guler740000-0002-6172-0285, C. Isik, O. Kara, A. Kayis Topaksu, U.
Kiminsu0000-0001-6940-7800, G. Onengut, K. Ozdemir75, A. Polatoz, A.E.
Simsek0000-0002-9074-2256, B. Tali76, U.G. Tok0000-0002-3039-021X, S.
Turkcapar, I.S. Zorbakir0000-0002-5962-2221 Middle East Technical University,
Physics Department, Ankara, Turkey G. Karapinar, K.
Ocalan770000-0002-8419-1400, M. Yalvac780000-0003-4915-9162 Bogazici
University, Istanbul, Turkey B. Akgun, I.O. Atakisi0000-0002-9231-7464, E.
Gulmez0000-0002-6353-518X, M. Kaya790000-0003-2890-4493, O. Kaya80, Ö.
Özçelik, S. Tekten81, E.A. Yetkin820000-0002-9007-8260 Istanbul Technical
University, Istanbul, Turkey A. Cakir0000-0002-8627-7689, K.
Cankocak730000-0002-3829-3481, Y. Komurcu, S. Sen830000-0001-7325-1087
Istanbul University, Istanbul, Turkey S. Cerci76, I. Hos84, B. Isildak85, B.
Kaynak, S. Ozkorucuklu, H. Sert0000-0003-0716-6727, D. Sunar
Cerci760000-0002-5412-4688, C. Zorbilmez Institute for Scintillation Materials
of National Academy of Science of Ukraine, Kharkov, Ukraine B. Grynyov
National Scientific Center, Kharkov Institute of Physics and Technology,
Kharkov, Ukraine L. Levchuk0000-0001-5889-7410 University of Bristol, Bristol,
United Kingdom D. Anthony, E. Bhal0000-0003-4494-628X, S. Bologna, J.J.
Brooke0000-0002-6078-3348, A. Bundock0000-0002-2916-6456, E.
Clement0000-0003-3412-4004, D. Cussans0000-0001-8192-0826, H.
Flacher0000-0002-5371-941X, J. Goldstein0000-0003-1591-6014, G.P. Heath, H.F.
Heath0000-0001-6576-9740, L. Kreczko0000-0003-2341-8330, B.
Krikler0000-0001-9712-0030, S. Paramesvaran, S. Seif El Nasr-Storey, V.J.
Smith, N. Stylianou860000-0002-0113-6829, K. Walkingshaw Pass, R. White
Rutherford Appleton Laboratory, Didcot, United Kingdom K.W. Bell, A.
Belyaev870000-0002-1733-4408, C. Brew0000-0001-6595-8365, R.M. Brown, D.J.A.
Cockerill, C. Cooke, K.V. Ellis, K. Harder, S. Harper, M.-L. Holmberg88, J.
Linacre0000-0001-7555-652X, K. Manolopoulos, D.M. Newbold0000-0002-9015-9634,
E. Olaiya, D. Petyt, T. Reis0000-0003-3703-6624, T. Schuh, C.H. Shepherd-
Themistocleous, I.R. Tomalin, T. Williams0000-0002-8724-4678 Imperial College,
London, United Kingdom R. Bainbridge0000-0001-9157-4832, P.
Bloch0000-0001-6716-979X, S. Bonomally, J. Borg0000-0002-7716-7621, S. Breeze,
O. Buchmuller, V. Cepaitis0000-0002-4809-4056, G.S.
Chahal890000-0003-0320-4407, D. Colling, P. Dauncey0000-0001-6839-9466, G.
Davies0000-0001-8668-5001, M. Della Negra0000-0001-6497-8081, S. Fayer, G.
Fedi0000-0001-9101-2573, G. Hall0000-0002-6299-8385, M.H. Hassanshahi, G.
Iles, J. Langford, L. Lyons, A.-M. Magnan, S. Malik, A.
Martelli0000-0003-3530-2255, D.G. Monk, J. Nash900000-0003-0607-6519, M.
Pesaresi, B.C. Radburn-Smith, D.M. Raymond, A. Richards, A. Rose, E.
Scott0000-0003-0352-6836, C. Seez, A. Shtipliyski, A.
Tapper0000-0003-4543-864X, K. Uchida, T. Virdee210000-0001-7429-2198, M.
Vojinovic0000-0001-8665-2808, N. Wardle0000-0003-1344-3356, S.N.
Webb0000-0003-4749-8814, D. Winterbottom Brunel University, Uxbridge, United
Kingdom K. Coldham, J.E. Cole0000-0001-5638-7599, A. Khan, P.
Kyberd0000-0002-7353-7090, I.D. Reid0000-0002-9235-779X, L. Teodorescu, S.
Zahid0000-0003-2123-3607 Baylor University, Waco, Texas, USA S.
Abdullin0000-0003-4885-6935, A. Brinkerhoff0000-0002-4853-0401, B.
Caraway0000-0002-6088-2020, J. Dittmann0000-0002-1911-3158, K.
Hatakeyama0000-0002-6012-2451, A.R. Kanuganti, B. McMaster0000-0002-4494-0446,
N. Pastika, M. Saunders0000-0003-1572-9075, S. Sawant, C. Sutantawibul, J.
Wilson0000-0002-5672-7394 Catholic University of America, Washington, DC, USA
R. Bartek0000-0002-1686-2882, A. Dominguez0000-0002-7420-5493, R.
Uniyal0000-0001-7345-6293, A.M. Vargas Hernandez The University of Alabama,
Tuscaloosa, Alabama, USA A. Buccilli0000-0001-6240-8931, S.I.
Cooper0000-0002-4618-0313, D. Di Croce0000-0002-1122-7919, S.V.
Gleyzer0000-0002-6222-8102, C. Henderson0000-0002-6986-9404, C.U.
Perez0000-0002-6861-2674, P. Rumerio910000-0002-1702-5541, C.
West0000-0003-4460-2241 Boston University, Boston, Massachusetts, USA A.
Akpinar0000-0001-7510-6617, A. Albert0000-0003-2369-9507, D.
Arcaro0000-0001-9457-8302, C. Cosby0000-0003-0352-6561, Z.
Demiragli0000-0001-8521-737X, E. Fontanesi, D. Gastler, S.
May0000-0002-6351-6122, J. Rohlf0000-0001-6423-9799, K.
Salyer0000-0002-6957-1077, D. Sperka, D. Spitzbart0000-0003-2025-2742, I.
Suarez0000-0002-5374-6995, A. Tsatsos, S. Yuan, D. Zou Brown University,
Providence, Rhode Island, USA G. Benelli0000-0003-4461-8905, B.
Burkle0000-0003-1645-822X, X. Coubez22, D. Cutts0000-0003-1041-7099, M.
Hadley0000-0002-7068-4327, U. Heintz0000-0002-7590-3058, J.M.
Hogan920000-0002-8604-3452, T. Kwon, G. Landsberg0000-0002-4184-9380, K.T.
Lau0000-0003-1371-8575, D. Li, M. Lukasik, J. Luo0000-0002-4108-8681, M.
Narain, N. Pervan, S. Sagir930000-0002-2614-5860, F. Simpson, E.
Usai0000-0001-9323-2107, W.Y. Wong, X. Yan0000-0002-6426-0560, D.
Yu0000-0001-5921-5231, W. Zhang University of California, Davis, Davis,
California, USA J. Bonilla0000-0002-6982-6121, C. Brainerd0000-0002-9552-1006,
R. Breedon, M. Calderon De La Barca Sanchez, M. Chertok0000-0002-2729-6273, J.
Conway0000-0003-2719-5779, P.T. Cox, R. Erbacher, G. Haza, F.
Jensen0000-0003-3769-9081, O. Kukral, R. Lander, M.
Mulhearn0000-0003-1145-6436, D. Pellett, B. Regnery0000-0003-1539-923X, D.
Taylor0000-0002-4274-3983, Y. Yao0000-0002-5990-4245, F.
Zhang0000-0002-6158-2468 University of California, Los Angeles, California,
USA M. Bachtis0000-0003-3110-0701, R. Cousins0000-0002-5963-0467, A.
Datta0000-0003-2695-7719, D. Hamilton, J. Hauser0000-0002-9781-4873, M.
Ignatenko, M.A. Iqbal, T. Lam, W.A. Nash, S. Regnard0000-0002-9818-6725, D.
Saltzberg0000-0003-0658-9146, B. Stone, V. Valuev0000-0002-0783-6703
University of California, Riverside, Riverside, California, USA Y. Chen, R.
Clare0000-0003-3293-5305, J.W. Gary0000-0003-0175-5731, M. Gordon, G.
Hanson0000-0002-7273-4009, G. Karapostoli0000-0002-4280-2541, O.R.
Long0000-0002-2180-7634, N. Manganelli, W. Si0000-0002-5879-6326, S. Wimpenny,
Y. Zhang University of California, San Diego, La Jolla, California, USA J.G.
Branson, P. Chang0000-0002-2095-6320, S. Cittolin, S.
Cooperstein0000-0003-0262-3132, N. Deelen0000-0003-4010-7155, D.
Diaz0000-0001-6834-1176, J. Duarte0000-0002-5076-7096, R.
Gerosa0000-0001-8359-3734, L. Giannini0000-0002-5621-7706, J. Guiang, R.
Kansal0000-0003-2445-1060, V. Krutelyov0000-0002-1386-0232, R. Lee, J.
Letts0000-0002-0156-1251, M. Masciovecchio0000-0002-8200-9425, F. Mokhtar, M.
Pieri0000-0003-3303-6301, B.V. Sathia Narayanan0000-0003-2076-5126, V.
Sharma0000-0003-1736-8795, M. Tadel, F. Würthwein0000-0001-5912-6124, Y.
Xiang0000-0003-4112-7457, A. Yagil0000-0002-6108-4004 University of
California, Santa Barbara - Department of Physics, Santa Barbara, California,
USA N. Amin, C. Campagnari0000-0002-8978-8177, M. Citron0000-0001-6250-8465,
G. Collura0000-0002-4160-1844, A. Dorsett, V. Dutta0000-0001-5958-829X, J.
Incandela0000-0001-9850-2030, M. Kilpatrick0000-0002-2602-0566, J.
Kim0000-0002-2072-6082, B. Marsh, H. Mei, M. Oshiro, M.
Quinnan0000-0003-2902-5597, J. Richman, U. Sarica0000-0002-1557-4424, F.
Setti, J. Sheplock, P. Siddireddy, D. Stuart, S. Wang0000-0001-7887-1728
California Institute of Technology, Pasadena, California, USA A.
Bornheim0000-0002-0128-0871, O. Cerri, I. Dutta0000-0003-0953-4503, J.M.
Lawhorn0000-0002-8597-9259, N. Lu0000-0002-2631-6770, J. Mao, H.B.
Newman0000-0003-0964-1480, T.Q. Nguyen0000-0003-3954-5131, M.
Spiropulu0000-0001-8172-7081, J.R. Vlimant0000-0002-9705-101X, C.
Wang0000-0002-0117-7196, S. Xie0000-0003-2509-5731, Z.
Zhang0000-0002-1630-0986, R.Y. Zhu0000-0003-3091-7461 Carnegie Mellon
University, Pittsburgh, Pennsylvania, USA J. Alison0000-0003-0843-1641, S.
An0000-0002-9740-1622, M.B. Andrews, P. Bryant0000-0001-8145-6322, T.
Ferguson0000-0001-5822-3731, A. Harilal, C. Liu, T.
Mudholkar0000-0002-9352-8140, M. Paulini0000-0002-6714-5787, A. Sanchez, W.
Terrill University of Colorado Boulder, Boulder, Colorado, USA J.P.
Cumalat0000-0002-6032-5857, W.T. Ford0000-0001-8703-6943, A. Hassani, G.
Karathanasis, E. MacDonald, R. Patel, A. Perloff0000-0001-5230-0396, C.
Savard, N. Schonbeck, K. Stenson0000-0003-4888-205X, K.A.
Ulmer0000-0001-6875-9177, S.R. Wagner0000-0002-9269-5772, N. Zipper Cornell
University, Ithaca, New York, USA J. Alexander0000-0002-2046-342X, S. Bright-
Thonney0000-0003-1889-7824, X. Chen0000-0002-8157-1328, Y.
Cheng0000-0002-2602-935X, D.J. Cranshaw0000-0002-7498-2129, S. Hogan, J.
Monroy0000-0002-7394-4710, J.R. Patterson0000-0002-3815-3649, D.
Quach0000-0002-1622-0134, J. Reichert0000-0003-2110-8021, M.
Reid0000-0001-7706-1416, A. Ryd, W. Sun0000-0003-0649-5086, J.
Thom0000-0002-4870-8468, P. Wittich0000-0002-7401-2181, R.
Zou0000-0002-0542-1264 Fermi National Accelerator Laboratory, Batavia,
Illinois, USA M. Albrow0000-0001-7329-4925, M. Alyari0000-0001-9268-3360, G.
Apollinari, A. Apresyan0000-0002-6186-0130, A. Apyan0000-0002-9418-6656,
L.A.T. Bauerdick0000-0002-7170-9012, D. Berry0000-0002-5383-8320, J.
Berryhill0000-0002-8124-3033, P.C. Bhat, K. Burkett0000-0002-2284-4744, J.N.
Butler, A. Canepa, G.B. Cerati0000-0003-3548-0262, H.W.K.
Cheung0000-0001-6389-9357, F. Chlebana, K.F. Di Petrillo0000-0001-8001-4602,
J. Dickinson0000-0001-5450-5328, V.D. Elvira0000-0003-4446-4395, Y. Feng, J.
Freeman, Z. Gecse, L. Gray, D. Green, S. Grünendahl0000-0002-4857-0294, O.
Gutsche0000-0002-8015-9622, R.M. Harris0000-0003-1461-3425, R. Heller, T.C.
Herwig0000-0002-4280-6382, J. Hirschauer0000-0002-8244-0805, B.
Jayatilaka0000-0001-7912-5612, S. Jindariani, M. Johnson, U. Joshi, T.
Klijnsma0000-0003-1675-6040, B. Klima0000-0002-3691-7625, K.H.M. Kwok, S.
Lammel0000-0003-0027-635X, D. Lincoln0000-0002-0599-7407, R. Lipton, T. Liu,
C. Madrid, K. Maeshima, C. Mantilla0000-0002-0177-5903, D. Mason, P.
McBride0000-0001-6159-7750, P. Merkel, S. Mrenna0000-0001-8731-160X, S.
Nahn0000-0002-8949-0178, J. Ngadiuba0000-0002-0055-2935, V. Papadimitriou, K.
Pedro0000-0003-2260-9151, C. Pena620000-0002-4500-7930, F.
Ravera0000-0003-3632-0287, A. Reinsvold Hall940000-0003-1653-8553, L.
Ristori0000-0003-1950-2492, E. Sexton-Kennedy0000-0001-9171-1980, N.
Smith0000-0002-0324-3054, A. Soha0000-0002-5968-1192, L. Spiegel, S.
Stoynev0000-0003-4563-7702, J. Strait0000-0002-7233-8348, L.
Taylor0000-0002-6584-2538, S. Tkaczyk, N.V. Tran0000-0002-8440-6854, L.
Uplegger0000-0002-9202-803X, E.W. Vaandering0000-0003-3207-6950, H.A.
Weber0000-0002-5074-0539 University of Florida, Gainesville, Florida, USA P.
Avery, D. Bourilkov0000-0003-0260-4935, L. Cadamuro0000-0001-8789-610X, V.
Cherepanov, R.D. Field, D. Guerrero, B.M. Joshi0000-0002-4723-0968, M. Kim, E.
Koenig, J. Konigsberg0000-0001-6850-8765, A. Korytov, K.H. Lo, K.
Matchev0000-0003-4182-9096, N. Menendez0000-0002-3295-3194, G.
Mitselmakher0000-0001-5745-3658, A. Muthirakalayil Madhu, N. Rawal, D.
Rosenzweig, S. Rosenzweig, K. Shi0000-0002-2475-0055, J.
Wang0000-0003-3879-4873, Z. Wu0000-0003-2165-9501, E.
Yigitbasi0000-0002-9595-2623, X. Zuo Florida State University, Tallahassee,
Florida, USA T. Adams0000-0001-8049-5143, A. Askew0000-0002-7172-1396, R.
Habibullah0000-0002-3161-8300, V. Hagopian, K.F. Johnson, R. Khurana, T.
Kolberg0000-0002-0211-6109, G. Martinez, H. Prosper0000-0002-4077-2713, C.
Schiber, O. Viazlo0000-0002-2957-0301, R. Yohay0000-0002-0124-9065, J. Zhang
Florida Institute of Technology, Melbourne, Florida, USA M.M.
Baarmand0000-0002-9792-8619, S. Butalla, T. Elkafrawy950000-0001-9930-6445, M.
Hohlmann0000-0003-4578-9319, R. Kumar Verma0000-0002-8264-156X, D.
Noonan0000-0002-3932-3769, M. Rahmani, F. Yumiceva0000-0003-2436-5074
University of Illinois at Chicago (UIC), Chicago, Illinois, USA M.R. Adams, H.
Becerril Gonzalez0000-0001-5387-712X, R. Cavanaugh0000-0001-7169-3420, S.
Dittmer, O. Evdokimov0000-0002-1250-8931, C.E. Gerber0000-0002-8116-9021, D.J.
Hofman0000-0002-2449-3845, A.H. Merrit, C. Mills0000-0001-8035-4818, G.
Oh0000-0003-0744-1063, T. Roy, S. Rudrabhatla, M.B. Tonjes0000-0002-2617-9315,
N. Varelas0000-0002-9397-5514, J. Viinikainen0000-0003-2530-4265, X. Wang, Z.
Ye0000-0001-6091-6772 The University of Iowa, Iowa City, Iowa, USA M.
Alhusseini0000-0002-9239-470X, K. Dilsiz960000-0003-0138-3368, L. Emediato,
R.P. Gandrajula0000-0001-9053-3182, O.K. Köseyan0000-0001-9040-3468, J.-P.
Merlo, A. Mestvirishvili97, J. Nachtman, H. Ogul980000-0002-5121-2893, Y.
Onel0000-0002-8141-7769, A. Penzo, C. Snyder, E. Tiras990000-0002-5628-7464
Johns Hopkins University, Baltimore, Maryland, USA O.
Amram0000-0002-3765-3123, B. Blumenfeld0000-0003-1150-1735, L.
Corcodilos0000-0001-6751-3108, J. Davis, A.V. Gritsan0000-0002-3545-7970, S.
Kyriacou, P. Maksimovic0000-0002-2358-2168, J. Roskes0000-0001-8761-0490, M.
Swartz, T.Á. Vámi0000-0002-0959-9211 The University of Kansas, Lawrence,
Kansas, USA A. Abreu, J. Anguiano, C. Baldenegro Barrera0000-0002-6033-8885,
P. Baringer0000-0002-3691-8388, A. Bean0000-0001-5967-8674, Z. Flowers, T.
Isidori, S. Khalil0000-0001-8630-8046, J. King, G.
Krintiras0000-0002-0380-7577, A. Kropivnitskaya0000-0002-8751-6178, M.
Lazarovits, C. Le Mahieu, C. Lindsey, J. Marquez, N.
Minafra0000-0003-4002-1888, M. Murray0000-0001-7219-4818, M. Nickel, C.
Rogan0000-0002-4166-4503, C. Royon, R. Salvatico0000-0002-2751-0567, S.
Sanders, E. Schmitz, C. Smith0000-0003-0505-0528, Q. Wang0000-0003-3804-3244,
Z. Warner, J. Williams0000-0002-9810-7097, G. Wilson0000-0003-0917-4763 Kansas
State University, Manhattan, Kansas, USA S. Duric, A.
Ivanov0000-0002-9270-5643, K. Kaadze0000-0003-0571-163X, D. Kim, Y.
Maravin0000-0002-9449-0666, T. Mitchell, A. Modak, K. Nam Lawrence Livermore
National Laboratory, Livermore, California, USA F. Rebassoo, D. Wright
University of Maryland, College Park, Maryland, USA E. Adams, A. Baden, O.
Baron, A. Belloni0000-0002-1727-656X, S.C. Eno0000-0003-4282-2515, N.J.
Hadley0000-0002-1209-6471, S. Jabeen0000-0002-0155-7383, R.G. Kellogg, T.
Koeth, Y. Lai, S. Lascio, A.C. Mignerey, S. Nabili, C.
Palmer0000-0003-0510-141X, M. Seidel0000-0003-3550-6151, A.
Skuja0000-0002-7312-6339, L. Wang, K. Wong0000-0002-9698-1354 Massachusetts
Institute of Technology, Cambridge, Massachusetts, USA D. Abercrombie, G.
Andreassi, R. Bi, W. Busza0000-0002-3831-9071, I.A. Cali, Y.
Chen0000-0003-2582-6469, M. D’Alfonso0000-0002-7409-7904, J. Eysermans, C.
Freer0000-0002-7967-4635, G. Gomez Ceballos, M. Goncharov, P. Harris, M. Hu,
M. Klute0000-0002-0869-5631, D. Kovalskyi0000-0002-6923-293X, J. Krupa, Y.-J.
Lee0000-0003-2593-7767, C. Mironov0000-0002-8599-2437, C.
Paus0000-0002-6047-4211, D. Rankin0000-0001-8411-9620, C.
Roland0000-0002-7312-5854, G. Roland, Z. Shi0000-0001-5498-8825, G.S.F.
Stephans0000-0003-3106-4894, J. Wang, Z. Wang0000-0002-3074-3767, B.
Wyslouch0000-0003-3681-0649 University of Minnesota, Minneapolis, Minnesota,
USA R.M. Chatterjee, A. Evans0000-0002-7427-1079, J. Hiltbrand, Sh.
Jain0000-0003-1770-5309, M. Krohn, Y. Kubota, J. Mans0000-0003-2840-1087, M.
Revering, R. Rusack0000-0002-7633-749X, R. Saradhy, N.
Schroeder0000-0002-8336-6141, N. Strobbe0000-0001-8835-8282, M.A. Wadud
University of Nebraska-Lincoln, Lincoln, Nebraska, USA K.
Bloom0000-0002-4272-8900, M. Bryson, S. Chauhan0000-0002-6544-5794, D.R.
Claes, C. Fangmeier, L. Finco0000-0002-2630-5465, F. Golf0000-0003-3567-9351,
C. Joo, I. Kravchenko0000-0003-0068-0395, I. Reed, J.E. Siado, G.R.
Snow${}^{\textrm{\textdagger}}$, W. Tabb, A. Wightman, F. Yan, A.G.
Zecchinelli State University of New York at Buffalo, Buffalo, New York, USA G.
Agarwal0000-0002-2593-5297, H. Bandyopadhyay0000-0001-9726-4915, L.
Hay0000-0002-7086-7641, I. Iashvili0000-0003-1948-5901, A. Kharchilava, C.
McLean0000-0002-7450-4805, D. Nguyen, J. Pekkanen0000-0002-6681-7668, S.
Rappoccio0000-0002-5449-2560, A. Williams0000-0003-4055-6532 Northeastern
University, Boston, Massachusetts, USA G. Alverson0000-0001-6651-1178, E.
Barberis, Y. Haddad0000-0003-4916-7752, Y. Han, A. Hortiangtham, A. Krishna,
J. Li0000-0001-5245-2074, J. Lidrych0000-0003-1439-0196, G. Madigan, B.
Marzocchi0000-0001-6687-6214, D.M. Morse0000-0003-3163-2169, V. Nguyen, T.
Orimoto0000-0002-8388-3341, A. Parker, L. Skinnari0000-0002-2019-6755, A.
Tishelman-Charny, T. Wamorkar, B. Wang0000-0003-0796-2475, A. Wisecarver, D.
Wood0000-0002-6477-801X Northwestern University, Evanston, Illinois, USA S.
Bhattacharya0000-0002-0526-6161, J. Bueghly, Z. Chen0000-0003-4521-6086, A.
Gilbert0000-0001-7560-5790, T. Gunter0000-0002-7444-5622, K.A. Hahn, Y. Liu,
N. Odell, M.H. Schmitt0000-0003-0814-3578, M. Velasco University of Notre
Dame, Notre Dame, Indiana, USA R. Band0000-0003-4873-0523, R. Bucci, M.
Cremonesi, A. Das0000-0001-9115-9698, N. Dev0000-0003-2792-0491, R.
Goldouzian0000-0002-0295-249X, M. Hildreth, K. Hurtado
Anampa0000-0002-9779-3566, C. Jessop0000-0002-6885-3611, K.
Lannon0000-0002-9706-0098, J. Lawrence, N. Loukas0000-0003-0049-6918, D.
Lutton, J. Mariano, N. Marinelli, I. Mcalister, T.
McCauley0000-0001-6589-8286, C. Mcgrady, K. Mohrman, C. Moore, Y. Musienko55,
R. Ruchti, A. Townsend, M. Wayne, M. Zarucki0000-0003-1510-5772, L. Zygala The
Ohio State University, Columbus, Ohio, USA B. Bylsma, L.S.
Durkin0000-0002-0477-1051, B. Francis0000-0002-1414-6583, C.
Hill0000-0003-0059-0779, M. Nunez Ornelas0000-0003-2663-7379, K. Wei, B.L.
Winer, B.R. Yates0000-0001-7366-1318 Princeton University, Princeton, New
Jersey, USA F.M. Addesa0000-0003-0484-5804, B. Bonham0000-0002-2982-7621, P.
Das0000-0002-9770-1377, G. Dezoort, P. Elmer0000-0001-6830-3356, A.
Frankenthal0000-0002-2583-5982, B. Greenberg0000-0002-4922-1934, N. Haubrich,
S. Higginbotham, A. Kalogeropoulos0000-0003-3444-0314, G. Kopp, S.
Kwan0000-0002-5308-7707, D. Lange, D. Marlow0000-0002-6395-1079, K.
Mei0000-0003-2057-2025, I. Ojalvo, J. Olsen0000-0002-9361-5762, D.
Stickland0000-0003-4702-8820, C. Tully0000-0001-6771-2174 University of Puerto
Rico, Mayaguez, Puerto Rico, USA S. Malik0000-0002-6356-2655, S. Norberg
Purdue University, West Lafayette, Indiana, USA A.S. Bakshi, V.E.
Barnes0000-0001-6939-3445, R. Chawla0000-0003-4802-6819, S.
Das0000-0001-6701-9265, L. Gutay, M. Jones0000-0002-9951-4583, A.W.
Jung0000-0003-3068-3212, D. Kondratyev0000-0002-7874-2480, A.M. Koshy, M. Liu,
G. Negro, N. Neumeister0000-0003-2356-1700, G. Paspalaki, S.
Piperov0000-0002-9266-7819, A. Purohit, J.F. Schulte0000-0003-4421-680X, M.
Stojanovic17, J. Thieman0000-0001-7684-6588, F. Wang0000-0002-8313-0809, R.
Xiao0000-0001-7292-8527, W. Xie0000-0003-1430-9191 Purdue University
Northwest, Hammond, Indiana, USA J. Dolen0000-0003-1141-3823, N. Parashar Rice
University, Houston, Texas, USA D. Acosta0000-0001-5367-1738, A.
Baty0000-0001-5310-3466, T. Carnahan, M. Decaro, S.
Dildick0000-0003-0554-4755, K.M. Ecklund0000-0002-6976-4637, S. Freed, P.
Gardner, F.J.M. Geurts0000-0003-2856-9090, A. Kumar0000-0002-5180-6595, W. Li,
B.P. Padley0000-0002-3572-5701, R. Redjimi, J. Rotter, W.
Shi0000-0002-8102-9002, A.G. Stahl Leiton0000-0002-5397-252X, S.
Yang0000-0002-2075-8631, L. Zhang100, Y. Zhang0000-0002-6812-761X University
of Rochester, Rochester, New York, USA A. Bodek0000-0003-0409-0341, P. de
Barbaro, R. Demina0000-0002-7852-167X, J.L. Dulemba0000-0002-9842-7015, C.
Fallon, T. Ferbel0000-0002-6733-131X, M. Galanti, A. Garcia-
Bellido0000-0002-1407-1972, O. Hindrichs0000-0001-7640-5264, A.
Khukhunaishvili, E. Ranken, R. Taus, G.P. Van Onsem0000-0002-1664-2337
Rutgers, The State University of New Jersey, Piscataway, New Jersey, USA B.
Chiarito, J.P. Chou0000-0001-6315-905X, A. Gandrakota0000-0003-4860-3233, Y.
Gershtein0000-0002-4871-5449, E. Halkiadakis0000-0002-3584-7856, A. Hart, M.
Heindl0000-0002-2831-463X, O. Karacheban250000-0002-2785-3762, I. Laflotte, A.
Lath0000-0003-0228-9760, R. Montalvo, K. Nash, M. Osherson, S.
Salur0000-0002-4995-9285, S. Schnetzer, S. Somalwar0000-0002-8856-7401, R.
Stone, S.A. Thayil0000-0002-1469-0335, S. Thomas, H. Wang0000-0002-3027-0752
University of Tennessee, Knoxville, Tennessee, USA H. Acharya, A.G.
Delannoy0000-0003-1252-6213, S. Fiorendi0000-0003-3273-9419, S.
Spanier0000-0002-8438-3197 Texas A&M University, College Station, Texas, USA
O. Bouhali1010000-0001-7139-7322, M. Dalchenko0000-0002-0137-136X, A.
Delgado0000-0003-3453-7204, R. Eusebi, J. Gilmore, T. Huang, T. Kamon102, H.
Kim0000-0003-4986-1728, S. Luo0000-0003-3122-4245, S. Malhotra, R. Mueller, D.
Overton, D. Rathjens0000-0002-8420-1488, A. Safonov0000-0001-9497-5471 Texas
Tech University, Lubbock, Texas, USA N. Akchurin, J. Damgov, V. Hegde, S.
Kunori, K. Lamichhane, S.W. Lee0000-0002-3388-8339, T. Mengke, S.
Muthumuni0000-0003-0432-6895, T. Peltola0000-0002-4732-4008, I. Volobouev, Z.
Wang, A. Whitbeck Vanderbilt University, Nashville, Tennessee, USA E.
Appelt0000-0003-3389-4584, S. Greene, A. Gurrola0000-0002-2793-4052, W. Johns,
A. Melo, K. Padeken0000-0001-7251-9125, F. Romeo0000-0002-1297-6065, P.
Sheldon0000-0003-1550-5223, S. Tuo, J. Velkovska0000-0003-1423-5241 University
of Virginia, Charlottesville, Virginia, USA M.W. Arenton0000-0002-6188-1011,
B. Cardwell, B. Cox0000-0003-3752-4759, G. Cummings0000-0002-8045-7806, J.
Hakala0000-0001-9586-3316, R. Hirosky0000-0003-0304-6330, M.
Joyce0000-0003-1112-5880, A. Ledovskoy0000-0003-4861-0943, A. Li, C.
Neu0000-0003-3644-8627, C.E. Perez Lara0000-0003-0199-8864, B.
Tannenwald0000-0002-5570-8095, S. White0000-0002-6181-4935 Wayne State
University, Detroit, Michigan, USA N. Poudyal0000-0003-4278-3464 University of
Wisconsin - Madison, Madison, WI, Wisconsin, USA S. Banerjee, K.
Black0000-0001-7320-5080, T. Bose0000-0001-8026-5380, S.
Dasu0000-0001-5993-9045, I. De Bruyn0000-0003-1704-4360, P.
Everaerts0000-0003-3848-324X, C. Galloni, H. He, M.
Herndon0000-0003-3043-1090, A. Herve, U. Hussain, A. Lanaro, A. Loeliger, R.
Loveless, J. Madhusudanan Sreekala0000-0003-2590-763X, A. Mallampalli, A.
Mohammadi, D. Pinna, A. Savin, V. Shang, V. Sharma0000-0003-1287-1471, W.H.
Smith0000-0003-3195-0909, D. Teague, S. Trembath-Reichert, W.
Vetens0000-0003-1058-1163 †: Deceased
1: Also at TU Wien, Wien, Austria
2: Also at Institute of Basic and Applied Sciences, Faculty of Engineering,
Arab Academy for Science, Technology and Maritime Transport, Alexandria, Egypt
3: Also at Université Libre de Bruxelles, Bruxelles, Belgium
4: Also at Universidade Estadual de Campinas, Campinas, Brazil
5: Also at Federal University of Rio Grande do Sul, Porto Alegre, Brazil
6: Also at The University of the State of Amazonas, Manaus, Brazil
7: Also at University of Chinese Academy of Sciences, Beijing, China
8: Also at Department of Physics, Tsinghua University, Beijing, China
9: Also at UFMS, Nova Andradina, Brazil
10: Also at Nanjing Normal University Department of Physics, Nanjing, China
11: Now at The University of Iowa, Iowa City, Iowa, USA
12: Also at Institute for Theoretical and Experimental Physics named by A.I.
Alikhanov of NRC ‘Kurchatov Institute’, Moscow, Russia
13: Also at Joint Institute for Nuclear Research, Dubna, Russia
14: Also at Suez University, Suez, Egypt
15: Now at British University in Egypt, Cairo, Egypt
16: Now at Cairo University, Cairo, Egypt
17: Also at Purdue University, West Lafayette, Indiana, USA
18: Also at Université de Haute Alsace, Mulhouse, France
19: Also at Ilia State University, Tbilisi, Georgia
20: Also at Erzincan Binali Yildirim University, Erzincan, Turkey
21: Also at CERN, European Organization for Nuclear Research, Geneva,
Switzerland
22: Also at RWTH Aachen University, III. Physikalisches Institut A, Aachen,
Germany
23: Also at University of Hamburg, Hamburg, Germany
24: Also at Isfahan University of Technology, Isfahan, Iran
25: Also at Brandenburg University of Technology, Cottbus, Germany
26: Also at Forschungszentrum Jülich, Juelich, Germany
27: Also at Physics Department, Faculty of Science, Assiut University, Assiut,
Egypt
28: Also at Karoly Robert Campus, MATE Institute of Technology, Gyongyos,
Hungary
29: Also at Institute of Physics, University of Debrecen, Debrecen, Hungary
30: Also at Institute of Nuclear Research ATOMKI, Debrecen, Hungary
31: Now at Universitatea Babes-Bolyai - Facultatea de Fizica, Cluj-Napoca,
Romania
32: Also at MTA-ELTE Lendület CMS Particle and Nuclear Physics Group, Eötvös
Loránd University, Budapest, Hungary
33: Also at Wigner Research Centre for Physics, Budapest, Hungary
34: Also at IIT Bhubaneswar, Bhubaneswar, India
35: Also at Institute of Physics, Bhubaneswar, India
36: Also at Punjab Agricultural University, Ludhiana, India
37: Also at Shoolini University, Solan, India
38: Also at University of Hyderabad, Hyderabad, India
39: Also at University of Visva-Bharati, Santiniketan, India
40: Also at Indian Institute of Science (IISc), Bangalore, India
41: Also at Indian Institute of Technology (IIT), Mumbai, India
42: Also at Department of Electrical and Computer Engineering, Isfahan
University of Technology, Isfahan, Iran
43: Also at Department of Physics, Isfahan University of Technology, Isfahan,
Iran
44: Also at Sharif University of Technology, Tehran, Iran
45: Also at Department of Physics, University of Science and Technology of
Mazandaran, Behshahr, Iran
46: Now at INFN Sezione di Bari, Università di Bari, Politecnico di Bari,
Bari, Italy
47: Also at Italian National Agency for New Technologies, Energy and
Sustainable Economic Development, Bologna, Italy
48: Also at Centro Siciliano di Fisica Nucleare e di Struttura Della Materia,
Catania, Italy
49: Also at Scuola Superiore Meridionale, Università di Napoli Federico II,
Napoli, Italy
50: Also at Università di Napoli ’Federico II’, Napoli, Italy
51: Also at Consiglio Nazionale delle Ricerche - Istituto Officina dei
Materiali, Perugia, Italy
52: Also at Riga Technical University, Riga, Latvia
53: Also at Consejo Nacional de Ciencia y Tecnología, Mexico City, Mexico
54: Also at IRFU, CEA, Université Paris-Saclay, Gif-sur-Yvette, France
55: Also at Institute for Nuclear Research, Moscow, Russia
56: Now at National Research Nuclear University ’Moscow Engineering Physics
Institute’ (MEPhI), Moscow, Russia
57: Also at Institute of Nuclear Physics of the Uzbekistan Academy of
Sciences, Tashkent, Uzbekistan
58: Also at St. Petersburg Polytechnic University, St. Petersburg, Russia
59: Also at University of Florida, Gainesville, Florida, USA
60: Also at Imperial College, London, United Kingdom
61: Also at P.N. Lebedev Physical Institute, Moscow, Russia
62: Also at California Institute of Technology, Pasadena, California, USA
63: Also at Budker Institute of Nuclear Physics, Novosibirsk, Russia
64: Also at Faculty of Physics, University of Belgrade, Belgrade, Serbia
65: Also at Trincomalee Campus, Eastern University, Sri Lanka, Nilaveli, Sri
Lanka
66: Also at INFN Sezione di Pavia, Università di Pavia, Pavia, Italy
67: Also at National and Kapodistrian University of Athens, Athens, Greece
68: Also at Ecole Polytechnique Fédérale Lausanne, Lausanne, Switzerland
69: Also at Universität Zürich, Zurich, Switzerland
70: Also at Stefan Meyer Institute for Subatomic Physics, Vienna, Austria
71: Also at Laboratoire d’Annecy-le-Vieux de Physique des Particules,
IN2P3-CNRS, Annecy-le-Vieux, France
72: Also at Şırnak University, Sirnak, Turkey
73: Also at Near East University, Research Center of Experimental Health
Science, Nicosia, Turkey
74: Also at Konya Technical University, Konya, Turkey
75: Also at Piri Reis University, Istanbul, Turkey
76: Also at Adiyaman University, Adiyaman, Turkey
77: Also at Necmettin Erbakan University, Konya, Turkey
78: Also at Bozok Universitetesi Rektörlügü, Yozgat, Turkey
79: Also at Marmara University, Istanbul, Turkey
80: Also at Milli Savunma University, Istanbul, Turkey
81: Also at Kafkas University, Kars, Turkey
82: Also at Istanbul Bilgi University, Istanbul, Turkey
83: Also at Hacettepe University, Ankara, Turkey
84: Also at Istanbul University - Cerrahpasa, Faculty of Engineering,
Istanbul, Turkey
85: Also at Ozyegin University, Istanbul, Turkey
86: Also at Vrije Universiteit Brussel, Brussel, Belgium
87: Also at School of Physics and Astronomy, University of Southampton,
Southampton, United Kingdom
88: Also at Rutherford Appleton Laboratory, Didcot, United Kingdom
89: Also at IPPP Durham University, Durham, United Kingdom
90: Also at Monash University, Faculty of Science, Clayton, Australia
91: Also at Università di Torino, Torino, Italy
92: Also at Bethel University, St. Paul, Minnesota, USA
93: Also at Karamanoğlu Mehmetbey University, Karaman, Turkey
94: Also at United States Naval Academy, Annapolis, Maryland, USA
95: Also at Ain Shams University, Cairo, Egypt
96: Also at Bingol University, Bingol, Turkey
97: Also at Georgian Technical University, Tbilisi, Georgia
98: Also at Sinop University, Sinop, Turkey
99: Also at Erciyes University, Kayseri, Turkey
100: Also at Institute of Modern Physics and Key Laboratory of Nuclear Physics
and Ion-beam Application (MOE) - Fudan University, Shanghai, China
101: Also at Texas A&M University at Qatar, Doha, Qatar
102: Also at Kyungpook National University, Daegu, Korea


of stars is evident, clustered in a region with $\text{Z}_{\text{g}}\approx
15$ kpc and between a Galactic radius of 5-10 kpc, as illustrated in Figure
10. We argue that this is likely a portion of the well-known Virgo over-
density (Jurić et al., 2008). Its counterpart at negative Z is the Hercules-
Aquila Cloud, also discernible in the Figure, on the other side of the Galaxy,
beyond the bulge.

  5. 5.

The RRL density distribution is stratified on ellipsoidal surfaces flattened
along the direction perpendicular to the Galactic disc. The density decreases
regularly following a power-law, without evidence of an inner-core or an
change of slope within the radial range accessible, as demonstrated in Figure
8. For the first time, we are able to show the evolution of the halo
flattening with radius directly in the data (see Figure 9). The Milky Way’s
stellar halo is rather flat in the inner parts but becomes more spherical at
$\text{R}\sim 20$ kpc. A similar property has been glimpsed in the sample of
RGB halo stars considered by Liu et al. (2017).

  6. 6.

To mitigate possible biases associate with the low-latitude excess of the RRL
candidates, we extracted a subset of 13713 stars from the original sample. We
applied to this “clean” sample a maximum likelihood fitting method, testing a
large variety of halo models. The parameters of our final best-model are in
good agreement with the results obtained by the non-parametric analysis of the
RRLs distribution. The halo is mildly triaxial with its major and intermediate
axes in the Galactic pane ($\text{p}=1.27$) and a significant flattening along
the Galactic Z-axis which varies from $\text{q}\approx 0.57$ in the centre to
$\text{q}\approx 0.75$ at the edge of the radial range analysed. The halo’s
major axis is rotated in the anticlockwise direction by a angle
$\gamma\approx-21^{\circ}$ with respect to the direction of the Galactic
Y-axis. The density slope is well approximated by a single power-law with
exponent $\alpha=-2.96$.

We have demonstrated the power of the Gaia data for the Galactic stellar halo
exploration. The wealth of the information available in GDR1 is unexpected and
signals that a paradigm shift in the halo studies is imminent. One only has to
wait for the GDR2 to be unleashed.

## Acknowledgments

The authors wish to thank the members of the Cambridge Streams club for
stimulating discussions and the anonymous referee for useful comments that
improved this manuscript.

This work has made use of data from the European Space Agency (ESA) mission
Gaia (http://www.cosmos.esa.int/gaia), processed by the Gaia Data Processing
and Analysis Consortium (DPAC,
http://www.cosmos.esa.int/web/gaia/dpac/consortium). Funding for the DPAC has
been provided by national institutions, in particular the institutions
participating in the Gaia Multilateral Agreement.

The research leading to these results has received funding from the European
Research Council under the European Union’s Seventh Framework Programme
(FP/2007-2013) / ERC Grant Agreement n. 308024. V.B., D.E. and S.K.
acknowledge financial support from the ERC. S.K. also acknowledges the support
from the STFC (grant ST/N004493/1).

## References

  * Abadi et al. (2010) Abadi M. G., Navarro J. F., Fardal M., Babul A., Steinmetz M., 2010, MNRAS, 407, 435
  * Abazajian et al. (2009) Abazajian K. N., et al., 2009, ApJS, 182, 543
  * Akhter et al. (2012) Akhter S., Da Costa G. S., Keller S. C., Schmidt B. P., 2012, ApJ, 756, 23
  * Amorisco (2017a) Amorisco N. C., 2017a, preprint,  (arXiv:1701.02741)
  * Amorisco (2017b) Amorisco N. C., 2017b, MNRAS, 464, 2882
  * Antoja et al. (2014) Antoja T., et al., 2014, A&A, 563, A60
  * Bell et al. (2008) Bell E. F., et al., 2008, ApJ, 680, 295
  * Belokurov et al. (2007) Belokurov V., et al., 2007, ApJ, 657, L89
  * Belokurov et al. (2017) Belokurov V., Erkal D., Deason A. J., Koposov S. E., De Angeli F., Evans D. W., Fraternali F., Mackey D., 2017, MNRAS, 466, 4711
  * Benjamin et al. (2005) Benjamin R. A., et al., 2005, ApJ, 630, L149
  * Bovy et al. (2012) Bovy J., et al., 2012, ApJ, 759, 131
  * Bowden et al. (2016) Bowden A., Evans N. W., Williams A. A., 2016, MNRAS, 460, 329
  * Bullock & Johnston (2005) Bullock J. S., Johnston K. V., 2005, ApJ, 635, 931
  * Cabré et al. (2012) Cabré A., Vikram V., Zhao G.-B., Jain B., Koyama K., 2012, J. Cosmology Astropart. Phys., 7, 034
  * Carollo et al. (2007) Carollo D., et al., 2007, Nature, 450, 1020
  * Catelan (2009) Catelan M., 2009, Ap&SS, 320, 261
  * Chan et al. (2015) Chan T. K., Kereš D., Oñorbe J., Hopkins P. F., Muratov A. L., Faucher-Giguère C.-A., Quataert E., 2015, MNRAS, 454, 2981
  * Crane et al. (2003) Crane J. D., Majewski S. R., Rocha-Pinto H. J., Frinchaboy P. M., Skrutskie M. F., Law D. R., 2003, ApJ, 594, L119
  * Das & Binney (2016) Das P., Binney J., 2016, MNRAS, 460, 1725
  * Das et al. (2016) Das P., Williams A., Binney J., 2016, MNRAS, 463, 3169
  * Davé et al. (2001) Davé R., Spergel D. N., Steinhardt P. J., Wandelt B. D., 2001, ApJ, 547, 574
  * De Propris et al. (2010) De Propris R., Harrison C. D., Mares P. J., 2010, ApJ, 719, 1582
  * Deason et al. (2011) Deason A. J., Belokurov V., Evans N. W., 2011, MNRAS, 416, 2903
  * Deason et al. (2013) Deason A. J., Belokurov V., Evans N. W., Johnston K. V., 2013, ApJ, 763, 113
  * Deason et al. (2016) Deason A. J., Mao Y.-Y., Wechsler R. H., 2016, ApJ, 821, 5
  * Deason et al. (2017) Deason A. J., Belokurov V., Erkal D., Koposov S. E., Mackey D., 2017, MNRAS, 467, 2636
  * Drake et al. (2013) Drake A. J., et al., 2013, ApJ, 763, 32
  * Drake et al. (2014) Drake A. J., et al., 2014, ApJS, 213, 9
  * Duffau et al. (2006) Duffau S., Zinn R., Vivas A. K., Carraro G., Méndez R. A., Winnick R., Gallart C., 2006, ApJ, 636, L97
  * Duffy et al. (2010) Duffy A. R., Schaye J., Kay S. T., Dalla Vecchia C., Battye R. A., Booth C. M., 2010, MNRAS, 405, 2161
  * Einasto (1965) Einasto J., 1965, Trudy Astrofizicheskogo Instituta Alma-Ata, 5, 87
  * Fermani & Schönrich (2013) Fermani F., Schönrich R., 2013, MNRAS, 432, 2402
  * Fitzpatrick (1999) Fitzpatrick E. L., 1999, PASP, 111, 63
  * Foreman-Mackey et al. (2013) Foreman-Mackey D., Hogg D. W., Lang D., Goodman J., 2013, PASP, 125, 306
  * Francis & Anderson (2014) Francis C., Anderson E., 2014, MNRAS, 441, 1105
  * Gaia Collaboration et al. (2016a) Gaia Collaboration et al., 2016a, A&A, 595, A1
  * Gaia Collaboration et al. (2016b) Gaia Collaboration et al., 2016b, A&A, 595, A2
  * Gnedin et al. (2004) Gnedin O. Y., Kravtsov A. V., Klypin A. A., Nagai D., 2004, ApJ, 616, 16
  * Goodman & Weare (2010) Goodman J., Weare J., 2010, Comm. App. Math. Comp. Sci., 5, 65
  * Governato et al. (2004) Governato F., et al., 2004, ApJ, 607, 688
  * Graham et al. (2006) Graham A. W., Merritt D., Moore B., Diemand J., Terzić B., 2006, AJ, 132, 2701
  * Hawkins (1984) Hawkins M. R. S., 1984, MNRAS, 206, 433
  * Helmi (2008) Helmi A., 2008, A&ARv, 15, 145
  * Helmi et al. (2011) Helmi A., Cooper A. P., White S. D. M., Cole S., Frenk C. S., Navarro J. F., 2011, ApJ, 733, L7
  * Helmi et al. (2017) Helmi A., Veljanoski J., Breddels M. A., Tian H., Sales L. V., 2017, A&A, 598, A58
  * Hernitschek et al. (2016) Hernitschek N., et al., 2016, ApJ, 817, 73
  * Ivezić et al. (2000) Ivezić Ž., et al., 2000, AJ, 120, 963
  * Jeans (1915) Jeans J. H., 1915, MNRAS, 76, 70
  * Jordi et al. (2010) Jordi C., et al., 2010, A&A, 523, A48
  * Jurcsik & Kovacs (1996) Jurcsik J., Kovacs G., 1996, A&A, 312, 111
  * Jurić et al. (2008) Jurić M., et al., 2008, ApJ, 673, 864
  * Karim & Mamajek (2017) Karim M. T., Mamajek E. E., 2017, MNRAS, 465, 472
  * Kauffmann et al. (1993) Kauffmann G., White S. D. M., Guiderdoni B., 1993, MNRAS, 264, 201
  * Kazantzidis et al. (2004) Kazantzidis S., Kravtsov A. V., Zentner A. R., Allgood B., Nagai D., Moore B., 2004, ApJ, 611, L73
  * Kinman et al. (1966) Kinman T. D., Wirtanen C. A., Janes K. A., 1966, ApJS, 13, 379
  * Koposov et al. (2017) Koposov S. E., Belokurov V., Torrealba G., 2017, preprint,  (arXiv:1702.01122)
  * Lepage (1978) Lepage G. P., 1978, J. Comput. Phys, 27, 192
  * Lindegren et al. (2012) Lindegren L., Lammers U., Hobbs D., O’Mullane W., Bastian U., Hernández J., 2012, A&A, 538, A78
  * Liu et al. (2017) Liu C., et al., 2017, preprint,  (arXiv:1701.07831)
  * López-Corredoira & Molgó (2014) López-Corredoira M., Molgó J., 2014, A&A, 567, A106
  * Lovell et al. (2014) Lovell M. R., Frenk C. S., Eke V. R., Jenkins A., Gao L., Theuns T., 2014, MNRAS, 439, 300
  * Mateu et al. (2011) Mateu C., Vivas A. K., Downes J. J., Briceño C., 2011, in Revista Mexicana de Astronomia y Astrofisica Conference Series. pp 245–246
  * McMillan & Binney (2010) McMillan P. J., Binney J. J., 2010, MNRAS, 402, 934
  * Miceli et al. (2008) Miceli A., et al., 2008, ApJ, 678, 865
  * Milgrom (1983) Milgrom M., 1983, ApJ, 270, 371
  * Morganson et al. (2016) Morganson E., et al., 2016, ApJ, 825, 140
  * Newberg et al. (2002) Newberg H. J., et al., 2002, ApJ, 569, 245
  * Pérez-Villegas et al. (2017) Pérez-Villegas A., Portail M., Gerhard O., 2017, MNRAS, 464, L80
  * Pietrukowicz et al. (2015) Pietrukowicz P., et al., 2015, ApJ, 811, 113
  * Pila-Díez et al. (2015) Pila-Díez B., de Jong J. T. A., Kuijken K., van der Burg R. F. J., Hoekstra H., 2015, A&A, 579, A38
  * Pillepich et al. (2014) Pillepich A., et al., 2014, MNRAS, 444, 237
  * Posti et al. (2015) Posti L., Binney J., Nipoti C., Ciotti L., 2015, MNRAS, 447, 3060
  * Robin et al. (2012) Robin A. C., Marshall D. J., Schultheis M., Reylé C., 2012, A&A, 538, A106
  * Saha (1984) Saha A., 1984, ApJ, 283, 580
  * Schlegel et al. (1998) Schlegel D. J., Finkbeiner D. P., Davis M., 1998, ApJ, 500, 525
  * Schönrich (2012) Schönrich R., 2012, MNRAS, 427, 274
  * Schwarz (1978) Schwarz G., 1978, Annals of Statistics, 6, 461
  * Sesar et al. (2007) Sesar B., et al., 2007, AJ, 134, 2236
  * Sesar et al. (2010) Sesar B., et al., 2010, ApJ, 708, 717
  * Sesar et al. (2011) Sesar B., Jurić M., Ivezić Ž., 2011, ApJ, 731, 4
  * Sesar et al. (2013) Sesar B., et al., 2013, AJ, 146, 21
  * Sesar et al. (2014) Sesar B., et al., 2014, ApJ, 793, 135
  * Sesar et al. (2017) Sesar B., et al., 2017, AJ, 153, 204
  * Shanks et al. (2000) Shanks T., Boyle B. J., Croom S., Loaring N., Miller L., Smith R. J., 2000, in Mazure A., Le Fèvre O., Le Brun V., eds, Astronomical Society of the Pacific Conference Series Vol. 200, Clustering at High Redshift. p. 57 (arXiv:astro-ph/0003206)
  * Simion et al. (2014) Simion I. T., Belokurov V., Irwin M., Koposov S. E., 2014, MNRAS, 440, 161
  * Skrutskie et al. (2006) Skrutskie M. F., et al., 2006, AJ, 131, 1163
  * Sollima et al. (2011) Sollima A., Valls-Gabaud D., Martinez-Delgado D., Fliri J., Peñarrubia J., Hoekstra H., 2011, ApJ, 730, L6
  * Sommer-Larsen et al. (2003) Sommer-Larsen J., Götz M., Portinari L., 2003, ApJ, 596, 47
  * Soszyński et al. (2014) Soszyński I., et al., 2014, Acta Astron., 64, 177
  * Soszyński et al. (2016) Soszyński I., et al., 2016, Acta Astron., 66, 131
  * Stokes et al. (2000) Stokes G. H., Evans J. B., Viggh H. E. M., Shelly F. C., Pearce E. C., 2000, Icarus, 148, 21
  * Torrealba et al. (2015) Torrealba G., et al., 2015, MNRAS, 446, 2251
  * Vivas & Zinn (2006) Vivas A. K., Zinn R., 2006, AJ, 132, 714
  * Vivas et al. (2004) Vivas A. K., et al., 2004, AJ, 127, 1158
  * Vivas et al. (2016) Vivas A. K., Zinn R., Farmer J., Duffau S., Ping Y., 2016, ApJ, 831, 165
  * Watkins et al. (2009) Watkins L. L., et al., 2009, MNRAS, 398, 1757
  * Wetterer & McGraw (1996) Wetterer C. J., McGraw J. T., 1996, AJ, 112, 1046
  * Williams & Evans (2015a) Williams A. A., Evans N. W., 2015a, MNRAS, 448, 1360
  * Williams & Evans (2015b) Williams A. A., Evans N. W., 2015b, MNRAS, 454, 698
  * Xu et al. (2017) Xu Y., et al., 2017, preprint,  (arXiv:1706.08650)
  * Xue et al. (2015) Xue X.-X., Rix H.-W., Ma Z., Morrison H., Bovy J., Sesar B., Janesh W., 2015, ApJ, 809, 144
  * Yanny et al. (2003) Yanny B., et al., 2003, ApJ, 588, 824
  * Yanny et al. (2009) Yanny B., et al., 2009, AJ, 137, 4377
  * de Boer et al. (2017) de Boer T. J. L., Belokurov V., Koposov S. E., 2017, preprint,  (arXiv:1706.09468)
  * van Leeuwen et al. (2016) van Leeuwen F., et al., 2016, preprint,  (arXiv:1612.02952)

##  Appendix A Jacobian

The Jacobian needed to go from the set of Cartesian coordinate (X, Y, Z) to
the observed one ($m$, l, b) is the determinant of the following matrix (see
Sec. 4.3.1):

$\text{J}=\left[\begin{array}[]{ccc}\frac{\partial\text{X}}{\partial
m}&\frac{\partial\text{X}}{\partial\text{l}}&\frac{\partial\text{X}}{\partial\text{b}}\\\
\frac{\partial\text{Y}}{\partial
m}&\frac{\partial\text{Y}}{\partial\text{l}}&\frac{\partial\text{Y}}{\partial\text{b}}\\\
\frac{\partial\text{Z}}{\partial
m}&\frac{\partial\text{Z}}{\partial\text{l}}&\frac{\partial\text{Z}}{\partial\text{b}}\end{array}\right].$

Therefore, the intrinsic and observed coordinates are related as

$\left\\{\begin{array}[]{lll}\text{X}=\text{D}_{\odot}\cos\text{l}\cos\text{b}\\\
\text{Y}=\text{D}_{\odot}\sin\text{l}\cos\text{b}\\\
\text{Z}=\text{D}_{\odot}\sin\text{b}\end{array}\right.$ (32)

where $\text{D}_{\odot}=10^{\frac{m-M}{5}-2}\ \text{kpc}$ is the distance of
the object from the Sun. The Jacobian matrix becomes

$\text{J}=\left[\begin{array}[]{ccc}\text{dD}_{\odot}\cos\text{l}\cos\text{b}&-\text{D}_{\odot}\sin\text{l}\cos\text{b}&-\text{D}_{\odot}\cos\text{l}\sin\text{b}\\\
\text{dD}_{\odot}\sin\text{l}\cos\text{b}&\text{D}_{\odot}\cos\text{l}\cos\text{b}&-\text{D}_{\odot}\sin\text{l}\sin\text{b}\\\
\text{dD}_{\odot}\sin\text{b}&0&\text{D}_{\odot}\cos\text{b}\end{array}\right].$

The determinant is

$|\text{J}|=\text{D}^{2}_{\odot}\text{dD}_{\odot}\cos\text{b}=\frac{\ln
10}{5}\text{D}^{3}_{\odot}\cos\text{b}.$ (33)

The result of Eq. 33 has been obtained for a frame of reference centered at
the Sun, however it is also valid for a Galactocentric frame of reference. The
two coordinate system differ only by an additive constant (the distance of the
Sun from the Galactic center) in the first row of the system 32., so the
determinant is the same in the two cases.

##  Appendix B Exploration of the parameters space

The exploration of the parameter space is performed evaluating the likelihood
in Eq. 29, which is a very time-consuming step. The bottleneck of the process
is the calculation of the normalisation integral $\text{V}_{c}$ in Eq. 27.
Given the presence of the selection function $W$, the integrand function is
not continuous and shows abrupt decreases to 0 in some regions of the
integration domain. For this reason, the classical multi-dimensional
quadrature methods are not able to give robust results, so we decided to make
use of a Monte Carlo integration technique. In particular, we used the vegas
algorithm (Lepage, 1978) through its Python
implementation666https://github.com/gplepage/vegas. The final estimate of the
integral comes from the average of $N_{\text{it}}$ vegas runs with
$N_{\text{eval}}$ integrand evaluations. The values of $N_{\text{it}}$ and
$N_{\text{eval}}$ have been chosen after comparing the results of vegas and an
adaptive quadrature technique777https://github.com/saullocastro/cubature
setting $W$=1 in the integrand (Eq. 27). In particular we used two settings:

  * •

$fast$ with $N_{\text{eval}}=10^{5}$ and $N_{\text{it}}=20$;

  * •

$robust$ with $N_{\text{eval}}=5\cdot 10^{5}$ and $N_{\text{it}}=100$.

In principle, we can also divide our integration domain into multiple regions
where $W=1$ and use the faster quadrature methods. However, given our cut in
$\theta$, the integral extremes $G$, l and b become non-trivially
interconnected and further time is expended on their calculation. Moreover, we
also note that the time spent by a quadrature integral solver is roughly
independent of the dimension of the sample space, therefore N different
evaluations increase the computational time by about N times. In conclusion,
using this approach we do not obtain a significant decrease of the
computational time.

In order to have the best compromise between computational time and good
sampling of the parameter space we adopt a two-stage procedure.

  1. 1.

In the first “explorative” stage we sample the parameter space using 200
walkers with 300 steps each after 100 “burn-in” steps and using the $fast$
estimate of the normalisation integral (see Sec. 4.3.3). The initial positions
of the walkers are randomly extracted from their prior distributions (Tab. 2).
Given the wide prior ranges (see Tab. 2), a large number of walkers is needed
to have an unbiased sampling of the posterior distributions.

  2. 2.

In the second stage we use 48 walkers with 300 steps each after 100 “burn-in”
steps using the $robust$ estimate of the normalisation integral (See sec.
4.3.3). The walkers are initially placed in a small region around the
parameters for which we obtained the maximum value of the likelihood in the
first stage.


# Causal Campbell-Goodhart’s law and Reinforcement Learning

Hal Ashton 1
1Computer Science, University College London, UK
<EMAIL_ADDRESS>
https://orcid.org/0000-0002-1780-9127

###### Abstract

Campbell-Goodhart’s law relates to the causal inference error whereby
decision-making agents aim to influence variables which are correlated to
their goal objective but do not reliably cause it. This is a well known error
in Economics and Political Science but not widely labelled in Artificial
Intelligence research. Through a simple example, we show how off-the-shelf
deep Reinforcement Learning (RL) algorithms are not necessarily immune to this
cognitive error. The off-policy learning method is tricked, whilst the on-
policy method is not. The practical implication is that naive application of
RL to complex real life problems can result in the same types of policy errors
that humans make. Great care should be taken around understanding the causal
model that underpins a solution derived from Reinforcement Learning.

##  1 INTRODUCTION

In many learning tasks, the learning agent has an impact on a stochastic state
variable through its previous actions. If when undisturbed, this variable
predicts but does not reliably cause something of interest to the agent, the
agent might choose actions to target this variable thereby causing suboptimal
performance. This is related to Campbell-Goodhart’s law in social science
which is described as: ”When a measure becomes a target, it ceases to be a
good measure” (Strathern, 1997). Manheim Garrabrant (2018) describe it
occurring ”When optimization causes a collapse of the statistical relationship
between a goal which the optimizer intends and the proxy used for that goal”.

In such a situation it is important to have a causal model for the learning
task so that the agent is able to correctly separate the effect of its actions
on the world, from those which are caused by some other mechanism. According
to Pearl Mackensie (2018), reasoning based on correlations alone is not
sufficient to solve certain causal problems. So-called level 2 and 3 problems
on their inference ladder require concepts of causality, intervention and
counterfactual reasoning.

Recently ’Deep’ Reinforcement Learning (RL) has had great success in the
automated mastery of learning optimal policies to hard problems such as Chess
and Go (Silver ., 2017) and a range of computer games starting with Atari
(Mnih ., 2015) through to more advanced games like StarCraft (Vinyals ., 2019)
through the use of Neural networks. All of these problems have environments
where there is an effect to the optimizer’s actions and yet RL has
traditionally avoided discussing causality at all111For example the canonical
text in Reinforcement Learning: Sutton Barto (2018) makes no explicit
reference to causality throughout the book.. Are the successes stated above
made possible because the problems have straightforward causal dependencies?
Perhaps inside the black boxes derived during training, a causal inference
technique is found automatically. Else, are RL methods easily confounded by
simple causal problems? It seems useful to know for anyone wishing to use RL
for real world applications like finance. The inability to explain AI coupled
with claims as to its superhuman abilities is termed ’enchanted determinism’
in Campolo Crawford (2020). Over-confidence and an accountability shield are
two ill-effects of this phenomenon.

In this paper I present a toy-problem where an agent is able to alter (or
intervene on) a variable that can be otherwise used to predict its reward.
This problem is classified as Causal Goodhart by Mannheim and Garrabant
(Specifically metric manipulation). It is deliberately simple and it can be
solved either analytically or using a number of learning methods without
involving neural networks. The motivation is to see whether problems that have
an interesting causal structure can be solved through a naive application of
existing, off the shelf RL learning algorithms which have had success solving
complex problems.

##  2 THE DOG BAROMETER PROBLEM

There is a dog living in a house in Scotland that wants to go for a walk. The
dog can observe current weather through a window but really needs to know
future weather when it is walking.

The capricious weather of Scotland is either Rain or Sunshine and depends on
recent barometric pressure only.

The barometric pressure (henceforth just pressure) is either high or low.
Future pressure depends only on past pressure. High pressure causes sunshine
more often and low pressure causes rain more often.

The dog would like to wear its smart coat when it is raining, but not wear it
when it is sunny. Once it has committed to its sartorial choice, the dog
leaves its house and experiences the weather during its walk. This marks the
end of the decision problem for the dog.

Within the house there is a barometer which measures current pressure and has
two states high and low. The dog can see the barometer. The barometer also has
a button which the dog can press. The effect is to set its reading to
high.222The barometer use is inspired from a lecture given by Prof Ricardo
Silva at UCL

###  2.1 Causal structure

High pressure ($P_{t}=1$) causes a high barometer reading ($B_{t}=1$) and a
high chance of sunshine period ($W_{t+1}=1$).

Conversely Low pressure ($P_{t}=1$) causes a low barometer reading ($B_{t}=1$)
and a higher chance of rain next period ($W_{t+1}=1$).

Touching the barometer causes a high barometer reading ($B_{t+1}=1$) next
period regardless of the pressure

This is summarised by the DAG in Figure 1 limited to three periods.
Alternatively it can be written with the following independence statement:

$\mathbb{P}(P_{t},B_{t},W_{t}|P_{t-1},B_{t-1},W_{t-1},A_{t-1})=\\\
\mathbb{P}(P_{t}|P_{t-1}).\mathbb{P}(B_{t}|P_{t},A_{t-1}).\mathbb{P}(W_{t}|P_{t-1})$
(1)

In the simplest case, Pressure has no autocorrelation:
$\mathbb{P}(P_{t}|P_{t-1})=\mathbb{P}(P_{t})$. This also has the effect of
making the Weather variable useless to the dog.

The effect of the dog pressing the button and setting the Barometer variable
$B_{t}$ is akin to that of an atomic intervention (Pearl, 2000). This removes
any arcs from parental nodes leading to $B_{t}$, meaning that inference about
the state of $P_{t}$ is impossible.

Figure 1: Causal diagram for dog barometer problem shown for 3 periods.
$A_{t}$ variables represent actions, $B_{t}$ pressure readings from the
barometer, $P_{t}$ the actual pressure, and $W_{t}$ the current weather.
Pressure is hidden to the (canine) observer. The button press makes an
intervention on the state of $B_{i}$, accordingly all incoming arrows into
$B_{i}$ should be deleted, thereby making it useless as an indicator for
$P_{i}$.

###  2.2 Problem as a MDP

We will model this problem as a MDP (Markov Decision Process)333Assuming that
pressure is not observable, it is better modelled as a Partially Observable
Markov Decision Process (POMDP), but we will proceed naively ’à la mode’ and
ignore the hidden variable.. Time is discretised and indexed by $t$. An MDP is
a tuple $(\mathcal{S},\mathcal{A},R,\mathcal{T},\mathcal{S}_{0},\gamma)$
where:

  1. 1.

$\mathcal{S}$ is the set of states. The binary random weather variable be W,
with rain denoted $W=0$ and sunshine $W=1$. Pressure variable is P, with a
high state denoted $P=1$ and a low state $P=0$. Similarly for Barometer
reading $B$.

  2. 2.

$\mathcal{A}$ is the set of actions. The dog is able to do one of four things
at any time period.

    1. (a)

Do nothing and wait $a_{t}=w$

    2. (b)

Touch the barometer $a_{t}=m$

    3. (c)

Put on a coat and leave their kennel $a_{t}=c$

    4. (d)

Leave the kennel without a coat. $a_{t}=n$

  3. 3.

$R:\mathcal{S}\times\mathcal{A}:\rightarrow\mathbb{R}$ is the reward function.
The dog prefers to go outside in the sun ($r_{nS}$) with no coat, quite likes
going outside with a coat in the rain ($r_{cR}$) but dislikes wearing a coat
in the sun ($r_{cS}$) and going out in the rain without a coat ($r_{nR}$).
These four rewards are ordered $r_{nS}\geq r_{cR}\geq r_{cS}\geq r_{nR}$. An
example assignment is shown in table 1.

| Rain | Sun
---|---|---
Coat | $r_{cR}=4$ | $r_{cS}=-8$
No Coat | $r_{nR}=-8$ | $r_{nS}=8$
Table 1: Rewards for the dog

There is also an optional penalty $r_{wait}=-1$ for the dog when it chooses
actions $w$ and $m$, which involve it waiting in the house. Whilst the
discount factor would seem to have a similar effect, such a penalty is often
used in practice to avoid sparse reward signals and help the learning
algorithm.

  4. 4.

$\mathcal{T}(s,a,s^{\prime})=\mathbb{P}(s^{\prime}|s,a)$ is the transition
function determining the probability of transitioning from state $s$ to
$s^{\prime}$ after taking action $a$. It is shown in tables 2, 3 and 4.

$\mathbb{P}(P_{t}|P_{t-1})$ | $P_{t-1}=Low$ | $P_{t-1}=High$
---|---|---
$P_{t}=Low$ | $\rho_{LL}$ | $\rho_{LH}=1-\rho_{HH}$
$P_{t}=High$ | $\rho_{HL}=1-\rho_{LL}$ | $\rho_{HH}$
Table 2: $\mathbb{P}(P_{t}|P_{t-1})$. Pressure transition: When
$\rho_{HH}=\rho_{LL}=0.5$ there is no autocorrelation.

$\mathbb{P}(B_{t}|P_{t},A_{t-1}$) | $A_{t-1}=0$ | $A_{t-1}=1$
---|---|---
| $P_{t}=Low$ | $P_{t}=High$ | $P_{t}=Low$ | $P_{t}=High$
$B_{t}=Low$ | $\alpha_{L}=0.9$ | $1-\alpha_{H}$ | 0 | 0
$B_{t}=High$ | $1-\alpha_{L}$ | $\alpha_{H}=0.9$ | 1 | 1

Table 3: $\mathbb{P}(B_{t}|P_{t-1},A_{t-1})$ If the button on the barometer has been touched in the previous period ($A_{t-1}=1$) the probability of a high reading is 1. The $\alpha_{.}$ coefficients correspond to the accuracy of the barometer. $\mathbb{P}(W_{t}|P_{t-1})$ | $P_{t-1}=Low$ | $P_{t-1}=High$
---|---|---
$W_{t}=Rain$ | $\omega_{RL}=0.9$ | $1-\omega_{SH}$
$W_{t}=Sun$ | $1-\omega_{RL}$ | $\omega_{SH}=0.9$
Table 4: $\mathbb{P}(W_{t}|P_{t-1})$. The $\omega_{..}$ coefficients
correspond to the capriciousness of the weather.

  5. 5.

$S_{0}\in\mathbb{P}(\mathcal{S})$ is a distribution over states that the
process begins at. For the initial states we draw $P_{-1}=H$ with probability
0.5 and generate $P_{0}$, $B_{0}$ and $W_{0}$ according to the conditional
distributions in tables 2,3 and 4 respectively.

  6. 6.

$\gamma=0.95$ is a discount factor which reflects how much less the dog values
future rewards from present ones.

The dog must choose a policy function $\pi:\mathcal{S}:\rightarrow\mathcal{A}$
to maximise the following discounted sum of rewards:

$\arg_{\pi}\max E\big{[}\sum_{t}\gamma^{t}R(s_{t},a_{t})\big{|}\pi\big{]}$ (2)

##  3 Method

I implemented the Dog Barometer problem in Open AI gym
444https://gym.openai.com/. I then tested two Deep RL learning algorithms on
this problem using the StableBaselines 3 module which provides a number of
implementations of state of the art deep RL algorithms
555https://github.com/DLR-RM/stable-baselines3. The code for the Dog Barometer
environment and tests is available online 666Environment available at
https://github.com/yetiminer/dogbarometer/.

For each algorithm I tested the case when pressure is (somehow) visible to the
dog as a baseline and when it is not. In both cases the algorithm was trained
separately 10 times.

The first learning algorithm tested was DQN, which was shown to be successful
learning how to play Atari games in Mnih . (2015). It learns through
Q-learning (Sutton Barto, 2018) and approximates the State-action function
(aka Q-function) through a neural network. The optimal policy is then the
action with the highest value for any state A memory of experiences (termed
experience replay) is built up to allow batch updates of the neural networks.
This algorithm is classified as off-policy learning, since the algorithm
estimates the value of an optimal policy without having to follow that policy
during exploration.

The second learning algorithm I tested was A2C which evolved from Mnih .
(2016). This is an actor-critic method which estimates value and actions
functions through neural-networks. It is an on-policy learning method, that is
to say the algorithm seeks to improve the policy it is currently following.

In both cases, I used the default parameters according to StableBaselines 3.
In particular all the neural networks were two layered, feed-forward
perceptrons of 64 neurons each with $Tanh$ activations.

The A2C algorithm was trained for 20,000 episodes and the DQN algorithm, being
less efficient was trained for 100,000. These figures were chosen for
sufficient convergence properties. Default settings from the StableBaselines3
777https://github.com/DLR-RM/stable-baselines3 module were used for both
algorithms. The resultant strategies were evaluated over 10,000 episodes.

##  4 Results

In Experiment 1, pressure has no auto-correlation: $\rho_{LL}=\rho_{HH}=0.5$.
This makes the strategy of waiting for high pressure or a high barometer
reading the most efficient. We denote this strategy $\Pi_{nw}$. Table 5 shows
the results of the different training methods. When Pressure is visible, the
optimal strategy is recovered by both algos though some of the time A2C
converges on $\Pi_{nc}$ \- wear or don’t wear coat according to barometer.
When pressure is hidden, the A2C algorithm successfully finds the optimal
strategy $\Pi_{nw}$ on every occasion. The DQN algorithm always converges on
$\Pi_{nb}$; the naive strategy that involves pressing the barometer button if
the barometer is initially low and going outside without a coat if the
barometer is high.

Exp series | Pressure Hidden | Mean Reward | Strategy count
---|---|---|---
|  |  | $\Pi_{nc}$ | $\Pi_{nw}$ | $\Pi_{nb}$
A2C |  | 4.80 | 3 | 7 |
A2CH | TRUE | 4.15 |  | 10 |
DQN |  | 5.39 |  | 10 |
DQNH | TRUE | 2.05 |  |  | 10

Table 5: DQN trained dogs are consistently fooled into pressing the barometer
- strategy $\Pi_{nb}$ whilst the A2C algorithm correctly recovers the optimal
strategy - $\Pi_{nc}$

Exp series | Pressure Hidden | Mean Reward | Strategy count
---|---|---|---
|  |  | $\Pi_{nc}$ | $\Pi_{nb}$ | $\Pi_{nwc}$ | $\Pi_{nbb}$
A2C E2 |  | 4.58 | 10 |  |  |
A2C E2 H | TRUE | 3.58 |  |  | 10 |
DQN E2 |  | 4.60 | 10 |  |  |
DQN E2 H | TRUE | 0.87 |  | 8 |  | 2

Table 6: Strategies found when weather is auto-correlated

In Experiment 2 $\rho_{LL}=\rho_{HH}=0.75$. That is to say pressure has auto-
correlation - the probability of maintaining the same level between periods is
0.75. This now makes waiting for high pressure less desirable. It also makes
the weather variable useful as an indicator independent to the barometer for
the previous state of pressure. Table 6 shows the results of the different
training methods. The A2C algorithm successfully finds the optimal strategy
$\Pi_{nwc}$ on every occasion. This is the strategy where the Dog exits the
house with or without a coat depending on the barometer unless the barometer
reads low and the weather is fine, in which case the dog will wait a period.
This balances the chance of a misreading from the barometer, the penalty of
going out with a coat when the weather is sunny. The DQN algorithm mostly
converges on $\Pi_{nb}$; the sub-optimal strategy that involves pressing the
barometer button if the barometer is initially low but also $\Pi_{nbb}$ which
involves pressing the barometer on every occasion except when the barometer
and the weather agree on a high/sun reading. This is an improvement on
$\Pi_{nb}$ since the weather is being used as an indicator though is still not
efficient.

##  5 Discussion

In our experiments we saw that the DQN method of training consistently led to
the naive strategy of pressing the barometer to ’cause’ high pressure which
would cause desirable sunny weather. In contrast A2C avoids this pitfall and
consistently finds an optimal strategy.

I hypothesise that this could be due to two related features of DQN;
Experience replay and Off-policy updating. Experience replay consists of the
chunking experience which is subsequently sampled in batches to update the
neural network that estimates the state-action value (Q-value) of each state.
Because prior-actions are not saved in this memory, the distribution of
rewards is not separated between those where the button has been pressed and
those where it has not. High reward signals from not wearing a coat after
reading a legitimately high barometer reading are mixed with the disappointing
ones of pressing the barometer and exiting without a coat. Bareinboim Pearl
(2016) call this the data-fusion problem. Secondly DQN is an off-policy
learning method - all policies are updated during learning not just the policy
that the learner is currently following. Again this would seem to mean that
the feedback from optimal policies where the button is not pressed is also
credited to policies where the button is pressed.

In contrast A2C does successfully navigate the Dog-Barometer. This is a
surprise given the inadequacy of the state signal in its ability to show when
the barometer is behaving properly. On reflection I think this might be
because this method of learning is _’on-policy’_. Learning only occurs on a
policy which is currently being used by the learner. Since the dynamics of
this environment are dependent on the action-history, and in this example, the
policy encodes action history, the learner is not tripped up as easily.

In all cases I used the default 2 layer 64 neuron feed forward MLP. It would
be useful to try a recurrent neural network like an LSTM Goodfellow . (2016)
to see how the results changed.

##  6 Related work

Causality within Reinforcement Learning is beginning to receive mainstream
attention. An introduction to the subject is given by Bareinboim (2020) and
the associated website888https://crl.causalai.net/ Accessed August 2020.. Guo
. (2018) provide a more general survey of learning causality from data. The
general issues surrounding the replay-memory of DQN erroneously mixing data
generated under different policies are explored in Bareinboim Pearl (2016).

Motivated from biological/psychological perspectives Gershman (2015) places
causal knowledge in model-based and model free RL and provides a novel, simple
taxonomy to identify where causal considerations can come into RL. Gershman
highlights research that suggest model and model-free reasoning exists within
the brain and discusses how they interact with reference to the Dyna
architecture of Sutton and BartoSutton (1990).

Buesing . (2019) present a counterfactual learning technique in the context of
POMDPS (Partially observable MDPs). They show how a model based RL approach to
POMDPs can be cast in the language of a SCM (Structural Causal Model - see
Pearl (2000)). This is important since SCMs are the principal tool through
which causal research has progressed. Counterfactual reasoning is strongly
related with off-policy learning methods since it considers the results of
actions not taken under the policy used to generate the experience. In
contrast to their use of an SCM to generate counter-factual data to search for
better policies, for a model-free setting, they observe that sampling from
experience can in certain circumstances be high variance or even useless when
evaluating different policies. This echoes the poor performance of DQN in our
experiment.

The subject of unobserved confounding in MDPs (termed MDPUCs) is studied in
Zhang Bareinboim (2016). The authors point out that MDPUCs are quite separate
from POMDPS. They observe that, to date, MDP learning algorithms do not
differentiate between passive data collection and data collection after
actions (interventions). The authors go on to show that standard MDP
techniques as used in RL are not guaranteed to converge to optimal policies
and present a method using counterfactual analysis to improve upon existing
learning algorithms.

Model based Reinforcement Learning (MBRL) is an area of RL research typically
separate from the RL mainstream which is model-free. In it an agent builds a
representation of the world from observational data in order to predict future
observations and thereby plan future actions and optimise a policy. Rezende .
(2020) study the problem of causal errors arising from partially modelled
environments in RL and show why they occur and propose a way of mitigating
them. They illustrate the problem with a simple MDP called ’FuzzyBear’ and
explicitly relate it to causal reasoning’s concepts of interventions,
backdoors and frontdoors taken from Pearl (2000).

Goodhart’s Law in economics (Goodhart, 1984) is analogous to Campbell’s law
(Campbell, 1979) in social science. Since both were originally communicated at
around the same point in time, I thought it was important to combine the two
to aid communication, as espoused in Rodamar (2018). This article is concerned
with one variant of Goodhart-Campbell termed ’causal’ in Manheim Garrabrant
(2018). Whilst there is a growing knowledge base on the failure cases of AI
(see for example Lehman . (2020)) the relevant ones are most often examples of
what Manheim and Garrabant term ’Shared-Cause’ Campbell-Goodhart, which is an
example of objective misalignment - the AI learns to maximise the metric but
not the ultimate goal of the programmer. To my knowledge, AI research has not
explicitly identified Causal-Goodhart effects when discussing failure cases.
One potential example of Causal Campbell-Goodhart is to be found in Ha
Schmidhuber (2018), where the AI which learns an internal model of Doom - a
computer game. On occasion it would learn strategies that would work in its
model, but would fail when used on the actual computer game.

##  7 Conclusion

Reinforcement learning has had great success learning optimal policies in a
variety of game settings, often exceeding human competence in Go, Chess, Atari
games etc. It is tempting to apply this method to more complex problems where
there are hidden variables, stochastic outcomes and non-trivial causal
structures and hope for the best. This may lead to disappointment and more
seriously, bad policies being enacted. To date, RL Research has given very
little consideration to causal issues, often because the causal mechanisms in
the canonical test cases are straightforward. By applying RL as is to more
complex problems, there is an implicit assumption that the neural networks
used in deep RL, are able to figure out the problem of understanding causality
just as well as they can learn to decode the raw vision data fed to them in
Atari Games as in Mnih et al. and turn it into winning strategies. Pearl
(2000) argues that certain problems cannot be solved using correlation based
statistics alone; to progress to the second rung of his causation ladder,
interventions need to be made. RL is a learning framework which naturally
performs interventions on the environment it seeks to learn about yet its
tools do automatically account for the mathematical implications of making
interventions.

The dog barometer problem that I present here is deliberately simple and the
state-space given to the learner is not ideal for a learning algorithm. RL
Methods do exist for settings with hidden variables and I have not used them
here. Expanding the state space to include previous state values and actions
may solve the problem. However I do think that the problems in RL raised by
dog barometer are not simply the result of a straw-man argument. The presence
of hidden variables in real life learning applications is almost certain as is
the existence of non-trivial causal structures whose effect may linger over
arbitrarily long timescales thereby negating the efficacy of adding more
history. Every model of a real problem will be misspecified to some extent; it
is important to understand when and why this matters. The fact that the
cognitive error is sufficiently common in social science to be named
Goodhart’s Law is a good indicator that this is a policy failure case which is
likely to appear again and again in real life applications of RL. In defence
of RL, the performance of the A2C algorithm even in the face of such
misspecification is very promising and warrants further investigation to see
whether this is consistent or an artefact of the environment.

Finally, I would like to continue to build an open library of causal problems
which new RL algorithms can be benchmarked against. Such an approach using the
OpenAI interface has already benefited RL and I think such a library will help
widen the audience of Causal RL to general RL researchers. In parallel it
would be useful to begin to build a taxonomy of cognitive errors that AI
suffers from, starting by investigating whether others, similar to Campbell-
Goodhart can be recreated with RL.

## ACKNOWLEDGEMENTS

This work is supported by an EPSRC PhD studentship.

## REFERENCES

  * Bareinboim (2020) Bareinboim2020TowardsCRLBareinboim, E.  2020\.  Towards Causal Reinforcement Learning (CRL) Towards Causal Reinforcement Learning (CRL).  Thirty-seventh International Conference on Machine Learning (ICML2020). Thirty-seventh international conference on machine learning (icml2020).
  * Bareinboim Pearl (2016) Bareinboim2016CausalProblemBareinboim, E. Pearl, J.  2016\.  Causal inference and the data-fusion problem Causal inference and the data-fusion problem.  Proceedings of the National Academy of Sciences of the United States of America113277345–7352.  doi:10.1073/pnas.1510507113
  * Buesing . (2019) Buesing2019WouldaSearchBuesing, L., Weber, T., Zwols, Y., Racaniere, S., Guez, A., Lespiau, JB. Heess, N.  2019\.  Woulda, Coulda, Shoulda: Counterfactually-guided policy search Woulda, Coulda, Shoulda: Counterfactually-guided policy search.  ICLR. Iclr.
  * Campbell (1979) Campbell1979AssessingChangeCampbell, DT.  1979\.  Assessing the impact of planned social change Assessing the impact of planned social change.  Evaluation and Program Planning2167–90.  doi:10.1016/0149-7189(79)90048-X
  * Campolo Crawford (2020) Campolo2020EnchantedIntelligenceCampolo, A. Crawford, K.  2020\.  Enchanted Determinism: Power without Responsibility in Artificial Intelligence Enchanted Determinism: Power without Responsibility in Artificial Intelligence.  Engaging Science, Technology, and Society61.  doi:10.17351/ests2020.277
  * Gershman (2015) Gershman2015ReinforcementModelsGershman, SJ.  2015\.  Reinforcement learning and causal models Reinforcement learning and causal models.  Oxford Handbook of Causal Reasoning1–32.
  * Goodfellow . (2016) GoodfellowBook2016Goodfellow, I., Bengio, Y. Courville, A.  2016\.  Deep Learning Deep Learning.
  * Goodhart (1984) Goodhart1984ProblemsExperienceGoodhart, CAE.  1984\.  Problems of Monetary Management: The UK Experience Problems of Monetary Management: The UK Experience.  Monetary Theory and Practice91–121.  doi:10.1007/978-1-349-17295-5_4
  * Guo . (2018) Guo2018AMethodsGuo, R., Cheng, L., Li, J., Hahn, PR. Liu, H.  2018\.  A Survey of Learning Causality with Data: Problems and Methods A Survey of Learning Causality with Data: Problems and Methods.  ACM Computing Surveys (CSUR)5341–37.  doi:10.1145/3397269
  * Ha Schmidhuber (2018) Ha2018WorldModelsHa, D. Schmidhuber, J.  2018\.  World Models. World Models.  doi:10.1016/b978-0-12-295180-0.50030-6
  * Lehman . (2020) Lehman2020TheCommunitiesLehman, J., Clune, J. Misevic, D.  2020\.  The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research communities The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research communities.  Artificial Life262274–306.  doi:10.1162/artl_a_00319
  * Manheim Garrabrant (2018) Manheim2018CategorizingLawManheim, D. Garrabrant, S.  2018\.  Categorizing Variants of Goodhart’s Law. Categorizing Variants of Goodhart’s Law.  http://arxiv.org/abs/1803.04585
  * Mnih . (2016) Mnih2016AsynchronousLearningMnih, V., Badia, AP., Mirza, L., Graves, A., Harley, T., Lillicrap, TP.Kavukcuoglu, K.  2016\.  Asynchronous methods for deep reinforcement learning Asynchronous methods for deep reinforcement learning.  33rd International Conference on Machine Learning, ICML 201642850–2869.
  * Mnih . (2015) Mnih2015Human-levelLearningMnih, V., Kavukcuoglu, K., Silver, D., … Hassabis, D.  2015\.  Human-level control through deep reinforcement learning Human-level control through deep reinforcement learning.  Nature5187540529–533.  doi:10.1038/nature14236
  * Pearl (2000) Pearl2000Causality:InferencePearl, J.  2000\.  Causality: Models, reasoning and inference Causality: Models, reasoning and inference.  Cambridge University Press.
  * Pearl Mackensie (2018) Pearl2018TheEffectPearl, J. Mackensie, D.  2018\.  The Book of Why: The new science of cause and effect The Book of Why: The new science of cause and effect.  Basic Books.
  * Rezende . (2020) Rezende2020CausallyLearningRezende, DJ., Danihelka, I., Papamakarios, G., … Buesing, L.  2020\.  Causally Correct Partial Models for Reinforcement Learning. Causally Correct Partial Models for Reinforcement Learning.  http://arxiv.org/abs/2002.02836
  * Rodamar (2018) Rodamar2018ThereGoodhartRodamar, J.  2018\.  There ought to be a law! Campbell versus Goodhart There ought to be a law! Campbell versus Goodhart.  Significance1569.  doi:10.1111/j.1740-9713.2018.01205.x
  * Silver . (2017) Silver2017MasteringKnowledgeSilver, D., Schrittwieser, J., Simonyan, K., … Hassabis, D.  2017\.  Mastering the game of Go without human knowledge Mastering the game of Go without human knowledge.  Nature5507676354–359.  doi:10.1038/nature24270
  * Strathern (1997) Strathern1997ImprovingSystemStrathern, M.  19977\.  ‘Improving ratings’: audit in the British University system ‘Improving ratings’: audit in the British University system.  European Review53305–321.  doi:10.1002/(SICI)1234-981X(199707)5:33.0.CO;2-4
  * Sutton (1990) Sutton1990Integrated7thSutton, RS.  1990\.  Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Proceedings of the 7th Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Proceedings of the 7th.  International Conference on Machine Learningpages1987216–224.
  * Sutton Barto (2018) Sutton2018ReinforcementIntroductionSutton, RS. Barto, AG.  2018\.  Reinforcement Learning: An Introduction Reinforcement Learning: An Introduction (2nd ).  MIT Press.
  * Vinyals . (2019) Vinyals2019GrandmasterLearningVinyals, O., Babuschkin, I., Czarnecki, WM., … Silver, D.  2019\.  Grandmaster level in StarCraft II using multi-agent reinforcement learning Grandmaster level in StarCraft II using multi-agent reinforcement learning.  Nature5757782350–354.  doi:10.1038/s41586-019-1724-z
  * Zhang Bareinboim (2016) Zhang2016MarkovApproachZhang, J. Bareinboim, E.  2016\.  Markov Decision Processes with Unobserved Confounders: A Causal Approach. Markov Decision Processes with Unobserved Confounders: A Causal Approach.  https://causalai.net/mdp-causal.pdf