* [195] E. Cooper-Balis and B. Jacob, “Fine-Grained Activation for Power Reduction in DRAM,” _IEEE Micro_ , 2010.
  * [196] A. N. Udipi, N. Muralimanohar _et al._ , “Rethinking DRAM Design and Organization for Energy-Constrained Multi-Cores,” in _ISCA_ , 2010.
  * [197] T. Zhang, K. Chen _et al._ , “Half-DRAM: A High-Bandwidth and Low-Power DRAM Architecture from the Rethinking of Fine-Grained Activation,” in _ISCA_ , 2014.
  * [198] H. Ha, A. Pedram _et al._ , “Improving Energy Efficiency of DRAM by Exploiting Half Page Row Access,” in _MICRO_ , 2016.
  * [199] Y. Lee, H. Kim _et al._ , “Partial Row Activation for Low-Power DRAM System,” in _HPCA_ , 2017.
  * [200] A. Olgun, F. Bostanci _et al._ , “Sectored DRAM: An Energy-Efficient High-Throughput and Practical Fine-Grained DRAM Architecture,” arXiv:2207.13795 [cs.AR], 2022.
  * [201] J. M. O’Connor, “Energy Efficient High Bandwidth DRAM for Throughput Processors,” Ph.D. dissertation, The University of Texas at Austin, 2021.
  * [202] M. O’Connor, N. Chatterjee _et al._ , “Fine-Grained DRAM: Energy-Efficient DRAM for Extreme Bandwidth Systems,” in _MICRO_ , 2017.
  * [203] JEDEC, _JESD79-4: DDR4 SDRAM_ , 2012.
  * [204] N. Firasta, M. Buxton _et al._ , “Intel AVX: New Frontiers in Performance Improvements and Energy Efficiency,” Intel Corp. White Paper, 2008.
  * [205] B. J. Smith, “A Pipelined, Shared Resource MIMD Computer,” in _ICPP_ , 1978.
  * [206] M. J. Flynn, “Very High-Speed Computing Systems,” _Proc. IEEE_ , 1966.
  * [207] J. E. Thornton, “Parallel Operation in the Control Data 6600,” in _AFIPS_ , 1964.
  * [208] R. M. Yoo, A. Romano, and C. Kozyrakis, “Phoenix Rebirth: Scalable MapReduce on a Large-Scale Shared-Memory System,” in _IISWC_ , 2009.
  * [209] L.-N. Pouchet, “PolyBench: The Polyhedral Benchmark Suite,” https://www.cs.colostate.edu/~pouchet/software/polybench/.
  * [210] S. Che, M. Boyer _et al._ , “Rodinia: A Benchmark Suite for Heterogeneous Computing,” in _IISWC_ , 2009.
  * [211] Standard Performance Evaluation Corp., “SPEC CPU2017,” https://t.ly/_ibe2.
  * [212] D. Lee, Y. Kim _et al._ , “Adaptive-Latency DRAM: Optimizing DRAM Timing for the Common-Case,” in _HPCA_ , 2015.
  * [213] R. H. Dennard, “Field-Effect Transistor Memory,” U.S. Patent 3,387,286, 1968.
  * [214] B. Keeth, R. J. Baker _et al._ , _DRAM Circuit Design: Fundamental and High-Speed Topics_. John Wiley & Sons, 2007.
  * [215] A. G. Yağlikçi, A. Olgun _et al._ , “HiRA: Hidden Row Activation for Reducing Refresh Latency of Off-the-Shelf DRAM Chips,” in _MICRO_ , 2022.
  * [216] H. Luo, A. Olgun _et al._ , “RowPress: Amplifying Read Disturbance in Modern DRAM Chips,” in _ISCA_ , 2023.
  * [217] B. Keeth and R. J. Baker, _DRAM Circuit Design: A Tutorial_ , 2001.
  * [218] J. Kim, M. Patel _et al._ , “Solar-DRAM: Reducing DRAM Access Latency by Exploiting the Variation in Local Bitlines,” in _ICCD_ , 2018.
  * [219] D. Lee, S. Khan _et al._ , “Design-Induced Latency Variation in Modern DRAM Chips: Characterization, Analysis, and Latency Reduction Mechanisms,” in _SIGMETRICS_ , 2017.
  * [220] C. Kim, D. Burger, and S. W. Keckler, “An Adaptive, Non-Uniform Cache Structure for Wire-Delay Dominated On-Chip Caches,” in _ASPLOS_ , 2002.
  * [221] A. Pohl, B. Cosenza, and B. Juurlink, “Cost Modelling for Vectorization on ARM,” in _CLUSTER_ , 2018.
  * [222] K. Trifunovic, D. Nuzman _et al._ , “Polyhedral-Model Guided Loop-Nest Auto-Vectorization,” in _PACT_ , 2009.
  * [223] R. M. Sokulski, S. R. dos Santos, and M. A. Alves, “On the SPEC-CPU 2017 Opportunities for Dynamic Vectorization Possibilities on PIM Architectures,” in _WSCAD_ , 2022.
  * [224] C. Lattner, “LLVM and Clang: Next Generation Compiler Technology,” in _BSDCan_ , 2008.
  * [225] S. Sarda and M. Pandey, _LLVM Essentials_. Packt Publishing Ltd, 2015.
  * [226] B. C. Lopes and R. Auler, _Getting Started with LLVM Core Libraries_. Packt Publishing Ltd, 2014.
  * [227] A. Sampson, “LLVM for Grad Students,” https://www.cs.cornell.edu/~asampson/blog/llvm.html.
  * [228] IITH-Compilers, “LLVM-Loop-Profiler,” https://github.com/IITH-Compilers/LLVM-Loop-Profiler.
  * [229] Y. Wang, L. Orosa _et al._ , “FIGARO: Improving System Performance via Fine-Grained In-DRAM Data Relocation and Caching,” in _MICRO_ , 2020.
  * [230] S. Lee, K.-D. Kang _et al._ , “GreenDIMM: OS-Assisted DRAM Power Management for DRAM with a Sub-array Granularity Power-Down State,” in _MICRO_ , 2021.
  * [231] JEDEC, _JESD79-4C: DDR4 SDRAM_ , 2017.
  * [232] JEDEC, _JESD79-5: DDR5 SDRAM Standard_ , 2020.
  * [233] M. R. Garey, R. L. Graham, and J. D. Ullman, “Worst-Case Analysis of Memory Allocation Algorithms,” in _STOC_ , 1972.
  * [234] LLVM Project, “Auto-Vectorization in LLVM,” https://llvm.org/docs/Vectorizers.html.
  * [235] H. Ahmed, P. C. Santos _et al._ , “A Compiler for Automatic Selection of Suitable Processing-in-Memory Instructions,” in _DATE_ , 2019.
  * [236] A. Devic, S. B. Rai _et al._ , “To PIM or Not for Emerging General Purpose Processing in DDR Memory Systems,” in _ISCA_ , 2022.
  * [237] T. H. Cormen, C. E. Leiserson _et al._ , _Introduction to Algorithms_. MIT Press, 2022.
  * [238] R. E. Tarjan, “Edge-Disjoint Spanning Trees and Depth-First Search,” _Acta Informatica_ , 1976.
  * [239] L. Dagum and R. Menon, “OpenMP: An Industry Standard API for Shared-Memory Programming,” _IEEE CSE_ , 1998.
  * [240] E. Lindholm, J. Nickolls _et al._ , “NVIDIA Tesla: A Unified Graphics and Computing Architecture,” _IEEE Micro_ , 2008.
  * [241] NVIDIA, “NVIDIA;s Next Generation CUDA Compute Architecture: Fermi,” White Paper, 2009.
  * [242] B. Lopes, R. Auler _et al._ , “ISA Aging: A X86 Case Study,” in _WIVOSCA_ , 2013.
  * [243] B. C. Lopes, R. Auler _et al._ , “SHRINK: Reducing the ISA Complexity via Instruction Recycling,” in _ISCA_ , 2015.
  * [244] M. Doblas, O. Lostes-Cazorla _et al._ , “GMX: Instruction Set Extensions for Fast, Scalable, and Efficient Genome Sequence Alignment,” in _MICRO_ , 2023.
  * [245] R. Razdan and M. D. Smith, “A High-Performance Microarchitecture with Hardware-Programmable Functional Units,” in _MICRO_ , 1994.
  * [246] Intel Corp., _Intel® 64 and IA-32 Architectures Software Developer’s Manual, Vol. 3_ , 2016.
  * [247] ARM Ltd., _Cortex-A8 Technical Reference Manual_ , 2010.
  * [248] The Kernel Development Community, “Linux and the Devicetree — The Linux Kernel Documentation,” https://t.ly/uC69W.
  * [249] J. S. Kim, M. Patel _et al._ , “Revisiting RowHammer: An Experimental Analysis of Modern DRAM Devices and Mitigation Techniques,” in _ISCA_ , 2020.
  * [250] L. Orosa, A. G. Yaglikci _et al._ , “A Deeper Look into RowHammer’s Sensitivities: Experimental Analysis of Real DRAM Chips and Implications on Future Attacks and Defenses",” in _MICRO_ , 2021.
  * [251] A. G. Yağlıkçı, H. Luo _et al._ , “Understanding RowHammer Under Reduced Wordline Voltage: An Experimental Study Using Real DRAM Devices,” in _DSN_ , 2022.
  * [252] K. Loughlin, J. Rosenblum _et al._ , “Siloz: Leveraging DRAM Isolation Domains to Prevent Inter-VM Rowhammer,” in _SOSP_ , 2023.
  * [253] D. S. Johnson, “Near-Optimal Bin Packing Algorithms,” Ph.D. dissertation, Massachusetts Institute of Technology, 1973.
  * [254] K. C. Knowlton, “A Programmer’s Description of L6,” _CACM_ , 1966.
  * [255] N. Binkert, B. Beckmann _et al._ , “The gem5 Simulator,” _Comput. Archit. News_ , 2011.
  * [256] Intel Corp., “6th Generation Intel Core Processor Family Datasheet,” http://www.intel.com/content/www/us/en/processors/core/.
  * [257] NVIDIA, “NVIDIA A100 Tensor Core GPU Architecture,” https://t.ly/rMUgA, 2020.
  * [258] M. Hähnel, B. Döbel _et al._ , “Measuring Energy Consumption for Short Code Paths Using RAPL,” _SIGMETRICS_ , 2012.
  * [259] NVIDIA, “NVIDIA Management Library (NVML),” https://developer.nvidia.com/nvidia-management-library-nvml.
  * [260] P. Shivakumar and N. P. Jouppi, “CACTI 3.0: An Integrated Cache Timing, Power, and Area Model,” Compaq Computer Corporation, Tech. Rep. 2001/2, 2001.
  * [261] N. Muralimanohar, R. Balasubramonian, and N. Jouppi, “Optimizing NUCA Organizations and Wiring Alternatives for Large Caches with CACTI 6.0,” in _MICRO_ , 2007.
  * [262] O. Mutlu and T. Moscibroda, “Stall-Time Fair Memory Access Scheduling for Chip Multiprocessors,” in _MICRO_ , 2007.
  * [263] W. K. Zuravleff and T. Robinson, “Controller for a Synchronous DRAM That Maximizes Throughput by Allowing Memory Requests and Commands to Be Issued Out of Order,” U.S. Patent 5 630 096, 1997.
  * [264] Standard Performance Evaluation Corp., “SPEC CPU2006 Benchmarks,” http://www.spec.org/cpu2006/.
  * [265] J. A. Stratton, C. Rodrigues _et al._ , “Parboil: A Revised Benchmark Suite for Scientific and Commercial Throughput Computing,” Univ. of Illinois at Urbana–Champaign, IMPACT Research Group, Tech. Rep. IMPACT-12-01, 2012.
  * [266] S. C. Woo, M. Ohara _et al._ , “The SPLASH-2 Programs: Characterization and Methodological Considerations,” in _ISCA_ , 1995.
  * [267] SAFARI Research Group, “DAMOV Benchmark Suite and Simulation Framework,” https://github.com/CMU-SAFARI/DAMOV.
  * [268] K. Vadivel, L. Chelini _et al._ , “TDO-CIM: Transparent Detection and Offloading for Computation In-Memory,” in _DATE_ , 2020.
  * [269] V. Iskandar, M. A. Abd El Ghany, and D. Goehringer, “NDP-RANK: Prediction and Ranking of NDP Systems Performance using Machine Learning,” _Microprocessors and Microsystems_ , 2023.
  * [270] A. Yazdanbakhsh, D. Mahajan _et al._ , “AxBench: A Multiplatform Benchmark Suite for Approximate Computing,” _IEEE Design & Test_, 2016.
  * [271] N.-M. Ho, E. Manogaran _et al._ , “Efficient Floating Point Precision Tuning for Approximate Computing,” in _ASP-DAC_ , 2017.
  * [272] A. Snavely and D. M. Tullsen, “Symbiotic Jobscheduling for a Simultaneous Multithreaded Processor,” in _ASPLOS_ , 2000.
  * [273] S. Eyerman and L. Eeckhout, “System-Level Performance Metrics for Multiprogram Workloads,” _IEEE Micro_ , 2008.
  * [274] P. Michaud, “Demystifying Multicore Throughput Metrics,” _CAL_ , 2012.
  * [275] K. Luo, J. Gummaraju, and M. Franklin, “Balancing Thoughput and Fairness in SMT Processors,” in _ISPASS_ , 2001.
  * [276] Y. Kim, M. Papamichael _et al._ , “Thread Cluster Memory Scheduling: Exploiting Differences in Memory Access Behavior,” in _MICRO_ , 2010.
  * [277] Y. Kim, D. Han _et al._ , “ATLAS: A Scalable and High-Performance Scheduling Algorithm for Multiple Memory Controllers,” in _HPCA_ , 2010.
  * [278] L. Subramanian, D. Lee _et al._ , “The Blacklisting Memory Scheduler: Achieving High Performance and Fairness at Low Cost,” in _ICCD_ , 2014.
  * [279] L. Subramanian, D. Lee _et al._ , “BLISS: Balancing Performance, Fairness and Complexity in Memory Access Scheduling,” _TPDS_ , 2016.
  * [280] L. Subramanian, V. Seshadri _et al._ , “MISE: Providing Performance Predictability and Improving Fairness in Shared Main Memory Systems,” in _HPCA_ , 2013.
  * [281] L. Subramanian, V. Seshadri _et al._ , “The Application Slowdown Model: Quantifying and Controlling the Impact of Inter-Application Interference at Shared Caches and Main Memory,” in _MICRO_ , 2015.
  * [282] E. Ebrahimi, C. J. Lee _et al._ , “Fairness via Source Throttling: A Configurable and High-Performance Fairness Substrate for Multi-Core Memory Systems,” in _ASPLOS_ , 2010.
  * [283] E. Ebrahimi, C. J. Lee _et al._ , “Prefetch-Aware Shared Resource Management for Multi-Core Systems,” in _ISCA_ , 2011.
  * [284] R. Das, O. Mutlu _et al._ , “Application-Aware Prioritization Mechanisms for On-Chip Networks,” in _MICRO_ , 2009.
  * [285] R. Das, R. Ausavarungnirun _et al._ , “Application-to-Core Mapping Policies to Reduce Memory System Interference in Multi-Core Systems,” in _HPCA_ , 2013.
  * [286] O. Mutlu and T. Moscibroda, “Parallelism-Aware Batch Scheduling: Enhancing Both Performance and Fairness of Shared DRAM Systems,” in _ISCA_ , 2008.
  * [287] Y. Kim, W. Yang, and O. Mutlu, “Ramulator: A Fast and Extensible DRAM Simulator,” _CAL_ , 2016.
  * [288] C. J. Lee, V. Narasiman _et al._ , “Improving Memory Bank-Level Parallelism in the Presence of Prefetching,” in _MICRO_ , 2009.
  * [289] M. Xie, D. Tong _et al._ , “Improving System Throughput and Fairness Simultaneously in Shared Memory CMP Systems via Dynamic Bank Partitioning,” in _HPCA_ , 2014.
  * [290] G. F. Oliveira, A. Olgun _et al._ , “MIMDRAM: An End-to-End Processing-Using-DRAM System for High-Throughput, Energy-Efficient and Programmer-Transparent Multiple-Instruction Multiple-Data Processing,” arXiv, 2024.
  * [291] O. Leitersdorf, D. Leitersdorf _et al._ , “AritPIM: High-Throughput In-Memory Arithmetic,” _IEEE Trans. Emerg. Topics Comput._ , 2023.
  * [292] Synopsys, Inc., “Design Compiler.” [Online]. Available: https://www.synopsys.com/implementation-and-signoff/rtl-synthesis-test/design-compiler-graphical.html
  * [293] A. Stillmaker and B. Baas, “Scaling Equations for the Accurate Prediction of CMOS Device Performance from 180 Nm to 7 Nm,” _Integration_ , 2017.
  * [294] G. Dai, Z. Zhu _et al._ , “DIMMining: Pruning-Efficient and Parallel Graph Mining on Near-Memory-Computing,” in _ISCA_ , 2022.
  * [295] Y. Zhang, N. Zhang _et al._ , “SARA: Scaling a Reconfigurable Dataflow Accelerator,” in _ISCA_ , 2021.
  * [296] R. Zhou, A. Roohi _et al._ , “FlexiDRAM: A Flexible In-DRAM Framework to Enable Parallel General-Purpose Computation,” in _ISLPED_ , 2022.
  * [297] J. Cheng, M. Grossman, and T. McKercher, _Professional CUDA C Programming_. John Wiley & Sons, 2014.
  * [298] OpenACC Organization, “The OpenACC® Application Programming Interface, Version 3.1,” 2020.


Institute of Nuclear Physics, Lomonosov Moscow State University, Moscow,
Russia
A. Belyaev, E. Boos, M. Dubinin46, L. Dudko, A. Ershov, A. Gribushin, V.
Klyukhin, O. Kodolova, I. Lokhtin, S. Obraztsov, S. Petrushanko, V. Savrin, A.
Snigirev Novosibirsk State University (NSU), Novosibirsk, Russia
A. Barnyakov47, V. Blinov47, T. Dimova47, L. Kardapoltsev47, Y. Skovpen47
Institute for High Energy Physics of National Research Centre ‘Kurchatov
Institute’, Protvino, Russia
I. Azhgirey, I. Bayshev, S. Bitioukov, V. Kachanov, D. Konstantinov, P.
Mandrik, V. Petrov, R. Ryutin, S. Slabospitskii, A. Sobol, S. Troshin, N.
Tyurin, A. Uzunian, A. Volkov National Research Tomsk Polytechnic University,
Tomsk, Russia
A. Babaev, A. Iuzhakov, V. Okhotnikov Tomsk State University, Tomsk, Russia
V. Borchsh, V. Ivanchenko, E. Tcherniaev University of Belgrade: Faculty of
Physics and VINCA Institute of Nuclear Sciences
P. Adzic48, P. Cirkovic, M. Dordevic, P. Milenovic, J. Milosevic, M.
Stojanovic Centro de Investigaciones Energéticas Medioambientales y
Tecnológicas (CIEMAT), Madrid, Spain
M. Aguilar-Benitez, J. Alcaraz Maestre, A. Álvarez Fernández, I. Bachiller, M.
Barrio Luna, CristinaF. Bedoya, J.A. Brochero Cifuentes, C.A. Carrillo
Montoya, M. Cepeda, M. Cerrada, N. Colino, B. De La Cruz, A. Delgado Peris,
J.P. Fernández Ramos, J. Flix, M.C. Fouz, O. Gonzalez Lopez, S. Goy Lopez,
J.M. Hernandez, M.I. Josa, D. Moran, Á. Navarro Tobar, A. Pérez-Calero
Yzquierdo, J. Puerta Pelayo, I. Redondo, L. Romero, S. Sánchez Navas, M.S.
Soares, A. Triossi, C. Willmott Universidad Autónoma de Madrid, Madrid, Spain
C. Albajar, J.F. de Trocóniz, R. Reyes-Almanza Universidad de Oviedo,
Instituto Universitario de Ciencias y Tecnologías Espaciales de Asturias
(ICTEA), Oviedo, Spain
B. Alvarez Gonzalez, J. Cuevas, C. Erice, J. Fernandez Menendez, S. Folgueras,
I. Gonzalez Caballero, J.R. González Fernández, E. Palencia Cortezon, V.
Rodríguez Bouza, S. Sanchez Cruz Instituto de Física de Cantabria (IFCA),
CSIC-Universidad de Cantabria, Santander, Spain
I.J. Cabrillo, A. Calderon, B. Chazin Quero, J. Duarte Campderros, M.
Fernandez, P.J. Fernández Manteca, A. García Alonso, G. Gomez, C. Martinez
Rivero, P. Martinez Ruiz del Arbol, F. Matorras, J. Piedra Gomez, C. Prieels,
T. Rodrigo, A. Ruiz-Jimeno, L. Russo49, L. Scodellaro, I. Vila, J.M. Vizan
Garcia University of Colombo, Colombo, Sri Lanka
D.U.J. Sonnadara University of Ruhuna, Department of Physics, Matara, Sri
Lanka
W.G.D. Dharmaratna, N. Wickramage CERN, European Organization for Nuclear
Research, Geneva, Switzerland
D. Abbaneo, B. Akgun, E. Auffray, G. Auzinger, J. Baechler, P. Baillon, A.H.
Ball, D. Barney, J. Bendavid, M. Bianco, A. Bocci, P. Bortignon, E. Bossini,
E. Brondolin, T. Camporesi, A. Caratelli, G. Cerminara, E. Chapon, G.
Cucciati, D. d’Enterria, A. Dabrowski, N. Daci, V. Daponte, A. David, O.
Davignon, A. De Roeck, M. Deile, M. Dobson, M. Dünser, N. Dupont, A. Elliott-
Peisert, N. Emriskova, F. Fallavollita50, D. Fasanella, S. Fiorendi, G.
Franzoni, J. Fulcher, W. Funk, S. Giani, D. Gigi, K. Gill, F. Glege, L.
Gouskos, M. Gruchala, M. Guilbaud, D. Gulhan, J. Hegeman, C. Heidegger, Y.
Iiyama, V. Innocente, T. James, P. Janot, O. Karacheban21, J. Kaspar, J.
Kieseler, M. Krammer1, N. Kratochwil, C. Lange, P. Lecoq, C. Lourenço, L.
Malgeri, M. Mannelli, A. Massironi, F. Meijers, S. Mersi, E. Meschi, F.
Moortgat, M. Mulders, J. Ngadiuba, J. Niedziela, S. Nourbakhsh, S. Orfanelli,
L. Orsini, F. Pantaleo18, L. Pape, E. Perez, M. Peruzzi, A. Petrilli, G.
Petrucciani, A. Pfeiffer, M. Pierini, F.M. Pitters, D. Rabady, A. Racz, M.
Rieger, M. Rovere, H. Sakulin, J. Salfeld-Nebgen, C. Schäfer, C. Schwick, M.
Selvaggi, A. Sharma, P. Silva, W. Snoeys, P. Sphicas51, J. Steggemann, S.
Summers, V.R. Tavolaro, D. Treille, A. Tsirou, G.P. Van Onsem, A. Vartak, M.
Verzetti, W.D. Zeuner Paul Scherrer Institut, Villigen, Switzerland
L. Caminada52, K. Deiters, W. Erdmann, R. Horisberger, Q. Ingram, H.C.
Kaestli, D. Kotlinski, U. Langenegger, T. Rohe, S.A. Wiederkehr ETH Zurich -
Institute for Particle Physics and Astrophysics (IPA), Zurich, Switzerland
M. Backhaus, P. Berger, N. Chernyavskaya, G. Dissertori, M. Dittmar, M.
Donegà, C. Dorfer, T.A. Gómez Espinosa, C. Grab, D. Hits, W. Lustermann, R.A.
Manzoni, M.T. Meinhard, F. Micheli, P. Musella, F. Nessi-Tedaldi, F. Pauss, G.
Perrin, L. Perrozzi, S. Pigazzini, M.G. Ratti, M. Reichmann, C. Reissel, T.
Reitenspiess, B. Ristic, D. Ruini, D.A. Sanz Becerra, M. Schönenberger, L.
Shchutska, M.L. Vesterbacka Olsson, R. Wallny, D.H. Zhu Universität Zürich,
Zurich, Switzerland
T.K. Aarrestad, C. Amsler53, C. Botta, D. Brzhechko, M.F. Canelli, A. De Cosa,
R. Del Burgo, B. Kilminster, S. Leontsinis, V.M. Mikuni, I. Neutelings, G.
Rauco, P. Robmann, K. Schweiger, C. Seitz, Y. Takahashi, S. Wertz, A.
Zucchetta National Central University, Chung-Li, Taiwan
T.H. Doan, C.M. Kuo, W. Lin, A. Roy, S.S. Yu National Taiwan University (NTU),
Taipei, Taiwan
P. Chang, Y. Chao, K.F. Chen, P.H. Chen, W.-S. Hou, Y.y. Li, R.-S. Lu, E.
Paganis, A. Psallidas, A. Steen Chulalongkorn University, Faculty of Science,
Department of Physics, Bangkok, Thailand
B. Asavapibhop, C. Asawatangtrakuldee, N. Srimanobhas, N. Suwonjandee Çukurova
University, Physics Department, Science and Art Faculty, Adana, Turkey
A. Bat, F. Boran, A. Celik54, S. Damarseckin55, Z.S. Demiroglu, F. Dolek, C.
Dozen56, I. Dumanoglu, G. Gokbulut, EmineGurpinar Guler57, Y. Guler, I. Hos58,
C. Isik, E.E. Kangal59, O. Kara, A. Kayis Topaksu, U. Kiminsu, G. Onengut, K.
Ozdemir60, S. Ozturk61, A.E. Simsek, U.G. Tok, S. Turkcapar, I.S. Zorbakir, C.
Zorbilmez Middle East Technical University, Physics Department, Ankara, Turkey
B. Isildak62, G. Karapinar63, M. Yalvac Bogazici University, Istanbul, Turkey
I.O. Atakisi, E. Gülmez, M. Kaya64, O. Kaya65, Ö. Özçelik, S. Tekten, E.A.
Yetkin66 Istanbul Technical University, Istanbul, Turkey
A. Cakir, K. Cankocak67, Y. Komurcu, S. Sen68 Istanbul University, Istanbul,
Turkey
S. Cerci69, B. Kaynak, S. Ozkorucuklu, D. Sunar Cerci69 Institute for
Scintillation Materials of National Academy of Science of Ukraine, Kharkov,
Ukraine
B. Grynyov National Scientific Center, Kharkov Institute of Physics and
Technology, Kharkov, Ukraine
L. Levchuk University of Bristol, Bristol, United Kingdom
E. Bhal, S. Bologna, J.J. Brooke, D. Burns70, E. Clement, D. Cussans, H.
Flacher, J. Goldstein, G.P. Heath, H.F. Heath, L. Kreczko, B. Krikler, S.
Paramesvaran, B. Penning, T. Sakuma, S. Seif El Nasr-Storey, V.J. Smith, J.
Taylor, A. Titterton Rutherford Appleton Laboratory, Didcot, United Kingdom
K.W. Bell, A. Belyaev71, C. Brew, R.M. Brown, D.J.A. Cockerill, J.A. Coughlan,
K. Harder, S. Harper, J. Linacre, K. Manolopoulos, D.M. Newbold, E. Olaiya, D.
Petyt, T. Reis, T. Schuh, C.H. Shepherd-Themistocleous, A. Thea, I.R. Tomalin,
T. Williams Imperial College, London, United Kingdom
R. Bainbridge, P. Bloch, J. Borg, S. Breeze, O. Buchmuller, A. Bundock,
GurpreetSingh CHAHAL72, D. Colling, P. Dauncey, G. Davies, M. Della Negra, R.
Di Maria, P. Everaerts, G. Hall, G. Iles, M. Komm, L. Lyons, A.-M. Magnan, S.
Malik, A. Martelli, V. Milosevic, A. Morton, J. Nash73, V. Palladino, M.
Pesaresi, D.M. Raymond, A. Richards, A. Rose, E. Scott, C. Seez, A.
Shtipliyski, M. Stoye, T. Strebler, A. Tapper, K. Uchida, T. Virdee18, N.
Wardle, D. Winterbottom, A.G. Zecchinelli, S.C. Zenz Brunel University,
Uxbridge, United Kingdom
J.E. Cole, P.R. Hobson, A. Khan, P. Kyberd, C.K. Mackay, I.D. Reid, L.
Teodorescu, S. Zahid Baylor University, Waco, USA
A. Brinkerhoff, K. Call, B. Caraway, J. Dittmann, K. Hatakeyama, C. Madrid, B.
McMaster, N. Pastika, C. Smith Catholic University of America, Washington, DC,
USA
R. Bartek, A. Dominguez, R. Uniyal, A.M. Vargas Hernandez The University of
Alabama, Tuscaloosa, USA
A. Buccilli, S.I. Cooper, C. Henderson, P. Rumerio, C. West Boston University,
Boston, USA
A. Albert, D. Arcaro, Z. Demiragli, D. Gastler, C. Richardson, J. Rohlf, D.
Sperka, D. Spitzbart, I. Suarez, L. Sulak, D. Zou Brown University,
Providence, USA
G. Benelli, B. Burkle, X. Coubez19, D. Cutts, Y.t. Duh, M. Hadley, U. Heintz,
J.M. Hogan74, K.H.M. Kwok, E. Laird, G. Landsberg, K.T. Lau, J. Lee, M.
Narain, S. Sagir75, R. Syarif, E. Usai, W.Y. Wong, D. Yu, W. Zhang University
of California, Davis, Davis, USA
R. Band, C. Brainerd, R. Breedon, M. Calderon De La Barca Sanchez, M. Chertok,
J. Conway, R. Conway, P.T. Cox, R. Erbacher, C. Flores, G. Funk, F. Jensen, W.
Ko${}^{\textrm{\textdagger}}$, O. Kukral, R. Lander, M. Mulhearn, D. Pellett,
J. Pilot, M. Shi, D. Taylor, K. Tos, M. Tripathi, Z. Wang, F. Zhang University
of California, Los Angeles, USA
M. Bachtis, C. Bravo, R. Cousins, A. Dasgupta, A. Florent, J. Hauser, M.
Ignatenko, N. Mccoll, W.A. Nash, S. Regnard, D. Saltzberg, C. Schnaible, B.
Stone, V. Valuev University of California, Riverside, Riverside, USA
K. Burt, Y. Chen, R. Clare, J.W. Gary, S.M.A. Ghiasi Shirazi, G. Hanson, G.
Karapostoli, O.R. Long, M. Olmedo Negrete, M.I. Paneva, W. Si, L. Wang, S.
Wimpenny, B.R. Yates, Y. Zhang University of California, San Diego, La Jolla,
USA
J.G. Branson, P. Chang, S. Cittolin, S. Cooperstein, N. Deelen, M. Derdzinski,
R. Gerosa, D. Gilbert, B. Hashemi, D. Klein, V. Krutelyov, J. Letts, M.
Masciovecchio, S. May, S. Padhi, M. Pieri, V. Sharma, M. Tadel, F. Würthwein,
A. Yagil, G. Zevi Della Porta University of California, Santa Barbara -
Department of Physics, Santa Barbara, USA
N. Amin, R. Bhandari, C. Campagnari, M. Citron, V. Dutta, M. Franco Sevilla,
J. Incandela, B. Marsh, H. Mei, A. Ovcharova, H. Qu, J. Richman, U. Sarica, D.
Stuart, S. Wang California Institute of Technology, Pasadena, USA
D. Anderson, A. Bornheim, O. Cerri, I. Dutta, J.M. Lawhorn, N. Lu, J. Mao,
H.B. Newman, T.Q. Nguyen, J. Pata, M. Spiropulu, J.R. Vlimant, S. Xie, Z.
Zhang, R.Y. Zhu Carnegie Mellon University, Pittsburgh, USA
M.B. Andrews, T. Ferguson, T. Mudholkar, M. Paulini, M. Sun, I. Vorobiev, M.
Weinberg University of Colorado Boulder, Boulder, USA
J.P. Cumalat, W.T. Ford, E. MacDonald, T. Mulholland, R. Patel, A. Perloff, K.
Stenson, K.A. Ulmer, S.R. Wagner Cornell University, Ithaca, USA
J. Alexander, Y. Cheng, J. Chu, A. Datta, A. Frankenthal, K. Mcdermott, J.R.
Patterson, D. Quach, A. Ryd, S.M. Tan, Z. Tao, J. Thom, P. Wittich, M. Zientek
Fermi National Accelerator Laboratory, Batavia, USA
S. Abdullin, M. Albrow, M. Alyari, G. Apollinari, A. Apresyan, A. Apyan, S.
Banerjee, L.A.T. Bauerdick, A. Beretvas, D. Berry, J. Berryhill, P.C. Bhat, K.
Burkett, J.N. Butler, A. Canepa, G.B. Cerati, H.W.K. Cheung, F. Chlebana, M.
Cremonesi, J. Duarte, V.D. Elvira, J. Freeman, Z. Gecse, E. Gottschalk, L.
Gray, D. Green, S. Grünendahl, O. Gutsche, J. Hanlon, R.M. Harris, S.
Hasegawa, R. Heller, J. Hirschauer, B. Jayatilaka, S. Jindariani, M. Johnson,
U. Joshi, T. Klijnsma, B. Klima, M.J. Kortelainen, B. Kreis, S. Lammel, J.
Lewis, D. Lincoln, R. Lipton, M. Liu, T. Liu, J. Lykken, K. Maeshima, J.M.
Marraffino, D. Mason, P. McBride, P. Merkel, S. Mrenna, S. Nahn, V. O’Dell, V.
Papadimitriou, K. Pedro, C. Pena, G. Rakness, F. Ravera, A. Reinsvold Hall, L.
Ristori, B. Schneider, E. Sexton-Kennedy, N. Smith, A. Soha, W.J. Spalding, L.
Spiegel, S. Stoynev, J. Strait, N. Strobbe, L. Taylor, S. Tkaczyk, N.V. Tran,
L. Uplegger, E.W. Vaandering, C. Vernieri, R. Vidal, M. Wang, H.A. Weber
University of Florida, Gainesville, USA
D. Acosta, P. Avery, D. Bourilkov, L. Cadamuro, V. Cherepanov, F. Errico, R.D.
Field, S.V. Gleyzer, D. Guerrero, B.M. Joshi, M. Kim, J. Konigsberg, A.
Korytov, K.H. Lo, K. Matchev, N. Menendez, G. Mitselmakher, D. Rosenzweig, K.
Shi, J. Wang, S. Wang, X. Zuo Florida International University, Miami, USA
Y.R. Joshi Florida State University, Tallahassee, USA
T. Adams, A. Askew, S. Hagopian, V. Hagopian, K.F. Johnson, R. Khurana, T.
Kolberg, G. Martinez, T. Perry, H. Prosper, C. Schiber, R. Yohay, J. Zhang
Florida Institute of Technology, Melbourne, USA
M.M. Baarmand, M. Hohlmann, D. Noonan, M. Rahmani, M. Saunders, F. Yumiceva
University of Illinois at Chicago (UIC), Chicago, USA
M.R. Adams, L. Apanasevich, R.R. Betts, R. Cavanaugh, X. Chen, S. Dittmer, O.
Evdokimov, C.E. Gerber, D.A. Hangal, D.J. Hofman, C. Mills, T. Roy, M.B.
Tonjes, N. Varelas, J. Viinikainen, H. Wang, X. Wang, Z. Wu The University of
Iowa, Iowa City, USA
M. Alhusseini, B. Bilki57, K. Dilsiz76, S. Durgut, R.P. Gandrajula, M.
Haytmyradov, V. Khristenko, O.K. Köseyan, J.-P. Merlo, A. Mestvirishvili77, A.
Moeller, J. Nachtman, H. Ogul78, Y. Onel, F. Ozok79, A. Penzo, C. Snyder, E.
Tiras, J. Wetzel Johns Hopkins University, Baltimore, USA
B. Blumenfeld, A. Cocoros, N. Eminizer, A.V. Gritsan, W.T. Hung, S. Kyriacou,
P. Maksimovic, J. Roskes, M. Swartz The University of Kansas, Lawrence, USA
C. Baldenegro Barrera, P. Baringer, A. Bean, S. Boren, J. Bowen, A. Bylinkin,
T. Isidori, S. Khalil, J. King, G. Krintiras, A. Kropivnitskaya, C. Lindsey,
D. Majumder, W. Mcbrayer, N. Minafra, M. Murray, C. Rogan, C. Royon, S.
Sanders, E. Schmitz, J.D. Tapia Takaki, Q. Wang, J. Williams, G. Wilson Kansas
State University, Manhattan, USA
S. Duric, A. Ivanov, K. Kaadze, D. Kim, Y. Maravin, D.R. Mendis, T. Mitchell,
A. Modak, A. Mohammadi Lawrence Livermore National Laboratory, Livermore, USA
F. Rebassoo, D. Wright University of Maryland, College Park, USA
A. Baden, O. Baron, A. Belloni, S.C. Eno, Y. Feng, N.J. Hadley, S. Jabeen,
G.Y. Jeng, R.G. Kellogg, A.C. Mignerey, S. Nabili, F. Ricci-Tam, M. Seidel,
Y.H. Shin, A. Skuja, S.C. Tonwar, K. Wong Massachusetts Institute of
Technology, Cambridge, USA
D. Abercrombie, B. Allen, R. Bi, S. Brandt, W. Busza, I.A. Cali, M. D’Alfonso,
G. Gomez Ceballos, M. Goncharov, P. Harris, D. Hsu, M. Hu, M. Klute, D.
Kovalskyi, Y.-J. Lee, P.D. Luckey, B. Maier, A.C. Marini, C. Mcginn, C.
Mironov, S. Narayanan, X. Niu, C. Paus, D. Rankin, C. Roland, G. Roland, Z.
Shi, G.S.F. Stephans, K. Sumorok, K. Tatar, D. Velicanu, J. Wang, T.W. Wang,
B. Wyslouch University of Minnesota, Minneapolis, USA
R.M. Chatterjee, A. Evans, S. Guts${}^{\textrm{\textdagger}}$, P. Hansen, J.
Hiltbrand, Sh. Jain, Y. Kubota, Z. Lesko, J. Mans, M. Revering, R. Rusack, R.
Saradhy, N. Schroeder, M.A. Wadud University of Mississippi, Oxford, USA
J.G. Acosta, S. Oliveros University of Nebraska-Lincoln, Lincoln, USA
K. Bloom, S. Chauhan, D.R. Claes, C. Fangmeier, L. Finco, F. Golf, R.
Kamalieddin, I. Kravchenko, J.E. Siado, G.R. Snow${}^{\textrm{\textdagger}}$,
B. Stieger, W. Tabb State University of New York at Buffalo, Buffalo, USA
G. Agarwal, C. Harrington, I. Iashvili, A. Kharchilava, C. McLean, D. Nguyen,
A. Parker, J. Pekkanen, S. Rappoccio, B. Roozbahani Northeastern University,
Boston, USA
G. Alverson, E. Barberis, C. Freer, Y. Haddad, A. Hortiangtham, G. Madigan, B.
Marzocchi, D.M. Morse, T. Orimoto, L. Skinnari, A. Tishelman-Charny, T.
Wamorkar, B. Wang, A. Wisecarver, D. Wood Northwestern University, Evanston,
USA
S. Bhattacharya, J. Bueghly, A. Gilbert, T. Gunter, K.A. Hahn, N. Odell, M.H.
Schmitt, K. Sung, M. Trovato, M. Velasco University of Notre Dame, Notre Dame,
USA
R. Bucci, N. Dev, R. Goldouzian, M. Hildreth, K. Hurtado Anampa, C. Jessop,
D.J. Karmgard, K. Lannon, W. Li, N. Loukas, N. Marinelli, I. Mcalister, F.
Meng, Y. Musienko40, R. Ruchti, P. Siddireddy, G. Smith, S. Taroni, M. Wayne,
A. Wightman, M. Wolf, A. Woodard The Ohio State University, Columbus, USA
J. Alimena, B. Bylsma, L.S. Durkin, B. Francis, C. Hill, W. Ji, A. Lefeld,
T.Y. Ling, B.L. Winer Princeton University, Princeton, USA
G. Dezoort, P. Elmer, J. Hardenbrook, N. Haubrich, S. Higginbotham, A.
Kalogeropoulos, S. Kwan, D. Lange, M.T. Lucchini, J. Luo, D. Marlow, K. Mei,
I. Ojalvo, J. Olsen, C. Palmer, P. Piroué, D. Stickland, C. Tully University
of Puerto Rico, Mayaguez, USA
S. Malik, S. Norberg Purdue University, West Lafayette, USA
A. Barker, V.E. Barnes, R. Chawla, S. Das, L. Gutay, M. Jones, A.W. Jung, A.
Khatiwada, B. Mahakud, D.H. Miller, G. Negro, N. Neumeister, C.C. Peng, S.
Piperov, H. Qiu, J.F. Schulte, N. Trevisani, F. Wang, R. Xiao, W. Xie Purdue
University Northwest, Hammond, USA
T. Cheng, J. Dolen, N. Parashar Rice University, Houston, USA
A. Baty, U. Behrens, S. Dildick, K.M. Ecklund, S. Freed, F.J.M. Geurts, M.
Kilpatrick, Arun Kumar, W. Li, B.P. Padley, R. Redjimi, J. Roberts, J. Rorie,
W. Shi, A.G. Stahl Leiton, Z. Tu, A. Zhang University of Rochester, Rochester,
USA
A. Bodek, P. de Barbaro, R. Demina, J.L. Dulemba, C. Fallon, T. Ferbel, M.
Galanti, A. Garcia-Bellido, O. Hindrichs, A. Khukhunaishvili, E. Ranken, R.
Taus Rutgers, The State University of New Jersey, Piscataway, USA
B. Chiarito, J.P. Chou, A. Gandrakota, Y. Gershtein, E. Halkiadakis, A. Hart,
M. Heindl, E. Hughes, S. Kaplan, I. Laflotte, A. Lath, R. Montalvo, K. Nash,
M. Osherson, H. Saka, S. Salur, S. Schnetzer, S. Somalwar, R. Stone, S. Thomas
University of Tennessee, Knoxville, USA
H. Acharya, A.G. Delannoy, S. Spanier Texas A&M University, College Station,
USA
O. Bouhali80, M. Dalchenko, M. De Mattia, A. Delgado, R. Eusebi, J. Gilmore,
T. Huang, T. Kamon81, H. Kim, S. Luo, S. Malhotra, D. Marley, R. Mueller, D.
Overton, L. Perniè, D. Rathjens, A. Safonov Texas Tech University, Lubbock,
USA
N. Akchurin, J. Damgov, F. De Guio, V. Hegde, S. Kunori, K. Lamichhane, S.W.
Lee, T. Mengke, S. Muthumuni, T. Peltola, S. Undleeb, I. Volobouev, Z. Wang,
A. Whitbeck Vanderbilt University, Nashville, USA
S. Greene, A. Gurrola, R. Janjam, W. Johns, C. Maguire, A. Melo, H. Ni, K.
Padeken, F. Romeo, P. Sheldon, S. Tuo, J. Velkovska, M. Verweij University of
Virginia, Charlottesville, USA
M.W. Arenton, P. Barria, B. Cox, G. Cummings, J. Hakala, R. Hirosky, M. Joyce,
A. Ledovskoy, C. Neu, B. Tannenwald, Y. Wang, E. Wolfe, F. Xia Wayne State
University, Detroit, USA
R. Harr, P.E. Karchin, N. Poudyal, J. Sturdy, P. Thapa University of Wisconsin
- Madison, Madison, WI, USA
T. Bose, J. Buchanan, C. Caillol, D. Carlsmith, S. Dasu, I. De Bruyn, L. Dodd,
C. Galloni, H. He, M. Herndon, A. Hervé, U. Hussain, A. Lanaro, A. Loeliger,
K. Long, R. Loveless, J. Madhusudanan Sreekala, A. Mallampalli, D. Pinna, T.
Ruggles, A. Savin, V. Sharma, W.H. Smith, D. Teague, S. Trembath-reichert †:
Deceased
1: Also at Vienna University of Technology, Vienna, Austria
2: Also at IRFU, CEA, Université Paris-Saclay, Gif-sur-Yvette, France
3: Also at Universidade Estadual de Campinas, Campinas, Brazil
4: Also at Federal University of Rio Grande do Sul, Porto Alegre, Brazil
5: Also at UFMS, Nova Andradina, Brazil
6: Also at Universidade Federal de Pelotas, Pelotas, Brazil
7: Also at Université Libre de Bruxelles, Bruxelles, Belgium
8: Also at University of Chinese Academy of Sciences, Beijing, China
9: Also at Institute for Theoretical and Experimental Physics named by A.I.
Alikhanov of NRC ‘Kurchatov Institute’, Moscow, Russia
10: Also at Joint Institute for Nuclear Research, Dubna, Russia
11: Also at Suez University, Suez, Egypt
12: Now at British University in Egypt, Cairo, Egypt
13: Also at Zewail City of Science and Technology, Zewail, Egypt
14: Also at Purdue University, West Lafayette, USA
15: Also at Université de Haute Alsace, Mulhouse, France
16: Also at Tbilisi State University, Tbilisi, Georgia
17: Also at Erzincan Binali Yildirim University, Erzincan, Turkey
18: Also at CERN, European Organization for Nuclear Research, Geneva,
Switzerland
19: Also at RWTH Aachen University, III. Physikalisches Institut A, Aachen,
Germany
20: Also at University of Hamburg, Hamburg, Germany
21: Also at Brandenburg University of Technology, Cottbus, Germany
22: Also at Institute of Physics, University of Debrecen, Debrecen, Hungary,
Debrecen, Hungary
23: Also at Institute of Nuclear Research ATOMKI, Debrecen, Hungary
24: Also at MTA-ELTE Lendület CMS Particle and Nuclear Physics Group, Eötvös
Loránd University, Budapest, Hungary, Budapest, Hungary
25: Also at IIT Bhubaneswar, Bhubaneswar, India, Bhubaneswar, India
26: Also at Institute of Physics, Bhubaneswar, India
27: Also at G.H.G. Khalsa College, Punjab, India
28: Also at Shoolini University, Solan, India
29: Also at University of Hyderabad, Hyderabad, India
30: Also at University of Visva-Bharati, Santiniketan, India
31: Also at Isfahan University of Technology, Isfahan, Iran
32: Now at INFN Sezione di Bari a, Università di Bari b, Politecnico di Bari
c, Bari, Italy
33: Also at Italian National Agency for New Technologies, Energy and
Sustainable Economic Development, Bologna, Italy
34: Also at Centro Siciliano di Fisica Nucleare e di Struttura Della Materia,
Catania, Italy
35: Also at Scuola Normale e Sezione dell’INFN, Pisa, Italy
36: Also at Riga Technical University, Riga, Latvia, Riga, Latvia
37: Also at Malaysian Nuclear Agency, MOSTI, Kajang, Malaysia
38: Also at Consejo Nacional de Ciencia y Tecnología, Mexico City, Mexico
39: Also at Warsaw University of Technology, Institute of Electronic Systems,
Warsaw, Poland
40: Also at Institute for Nuclear Research, Moscow, Russia
41: Now at National Research Nuclear University ’Moscow Engineering Physics
Institute’ (MEPhI), Moscow, Russia
42: Also at St. Petersburg State Polytechnical University, St. Petersburg,
Russia
43: Also at University of Florida, Gainesville, USA
44: Also at Imperial College, London, United Kingdom
45: Also at P.N. Lebedev Physical Institute, Moscow, Russia
46: Also at California Institute of Technology, Pasadena, USA
47: Also at Budker Institute of Nuclear Physics, Novosibirsk, Russia
48: Also at Faculty of Physics, University of Belgrade, Belgrade, Serbia
49: Also at Università degli Studi di Siena, Siena, Italy
50: Also at INFN Sezione di Pavia a, Università di Pavia b, Pavia, Italy,
Pavia, Italy
51: Also at National and Kapodistrian University of Athens, Athens, Greece
52: Also at Universität Zürich, Zurich, Switzerland
53: Also at Stefan Meyer Institute for Subatomic Physics, Vienna, Austria,
Vienna, Austria
54: Also at Burdur Mehmet Akif Ersoy University, BURDUR, Turkey
55: Also at Şırnak University, Sirnak, Turkey
56: Also at Department of Physics, Tsinghua University, Beijing, China,
Beijing, China
57: Also at Beykent University, Istanbul, Turkey, Istanbul, Turkey
58: Also at Istanbul Aydin University, Application and Research Center for
Advanced Studies (App. & Res. Cent. for Advanced Studies), Istanbul, Turkey
59: Also at Mersin University, Mersin, Turkey
60: Also at Piri Reis University, Istanbul, Turkey
61: Also at Gaziosmanpasa University, Tokat, Turkey
62: Also at Ozyegin University, Istanbul, Turkey
63: Also at Izmir Institute of Technology, Izmir, Turkey
64: Also at Marmara University, Istanbul, Turkey
65: Also at Kafkas University, Kars, Turkey
66: Also at Istanbul Bilgi University, Istanbul, Turkey
67: Also at Near East University, Research Center of Experimental Health
Science, Nicosia, Turkey
68: Also at Hacettepe University, Ankara, Turkey
69: Also at Adiyaman University, Adiyaman, Turkey
70: Also at Vrije Universiteit Brussel, Brussel, Belgium
71: Also at School of Physics and Astronomy, University of Southampton,
Southampton, United Kingdom
72: Also at IPPP Durham University, Durham, United Kingdom
73: Also at Monash University, Faculty of Science, Clayton, Australia
74: Also at Bethel University, St. Paul, Minneapolis, USA, St. Paul, USA
75: Also at Karamanoğlu Mehmetbey University, Karaman, Turkey
76: Also at Bingol University, Bingol, Turkey
77: Also at Georgian Technical University, Tbilisi, Georgia
78: Also at Sinop University, Sinop, Turkey
79: Also at Mimar Sinan University, Istanbul, Istanbul, Turkey
80: Also at Texas A&M University at Qatar, Doha, Qatar
81: Also at Kyungpook National University, Daegu, Korea, Daegu, Korea


# Long-form Question Answering: An Iterative Planning-Retrieval-Generation
Approach

Pritom Saha Akash1 Kashob Kumar Roy1 Lucian Popa 2 Kevin Chen-Chuan Chang1
1University of Illinois at Urbana-Champaign, USA
2IBM Research, USA
{pakash2, kkroy2<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Long-form question answering (LFQA) poses a challenge as it involves
generating detailed answers in the form of paragraphs, which go beyond simple
yes/no responses or short factual answers. While existing QA models excel in
questions with concise answers, LFQA requires handling multiple topics and
their intricate relationships, demanding comprehensive explanations. Previous
attempts at LFQA focused on generating long-form answers by utilizing relevant
contexts from a corpus, relying solely on the question itself. However, they
overlooked the possibility that the question alone might not provide
sufficient information to identify the relevant contexts. Additionally,
generating detailed long-form answers often entails aggregating knowledge from
diverse sources. To address these limitations, we propose an LFQA model with
iterative Planning, Retrieval, and Generation. This iterative process
continues until a complete answer is generated for the given question. From an
extensive experiment on both an open domain and a technical domain QA dataset,
we find that our model outperforms the state-of-the-art models on various
textual and factual metrics for the LFQA task.

##  1 Introduction

Question answering (QA) is a computational task that involves providing a
relevant and accurate response to a question expressed in natural language. A
considerable amount of progress has been made in open-domain question
answering, specifically in settings where questions are answerable with short
phrases and entities. For example, significant advancements have been made in
factoid question-answering (QA) research, which has yielded impressive results
with the creation of comprehensive datasets like SQuAD Rajpurkar et al. (2018)
and MS MARCO Nguyen et al. (2016), as well as the utilization of transformer-
based models such as ALBERT Lan et al. (2019). In many cases, these models
have even demonstrated the ability to outperform human performance. However,
while short-form question answering has proven to be effective for simple
factual questions, it often falls short when it comes to complex and nuanced
questions that require more comprehensive and detailed responses.

One of the main challenges of LFQA is that there is not much data available
for this task. One prominent dataset used for this purpose is the ELI5 dataset
Fan et al. (2019). It comprises questions asked on the "Explain Like I’m Five"
Reddit forum, along with corresponding answers in paragraph form. However, the
questions in ELI5 tend to be broad (e.g., "How do animals see different
colors?"), and multiple valid approaches can be taken to answer them. This
multiplicity makes it difficult to establish objective criteria for assessing
the quality of answers. In a study by Krishna et al. (2021), various hurdles
in effectively leveraging this dataset for meaningful advancements in modeling
were highlighted, including the absence of reliable evaluation metrics.

In addition to the limitation of available datasets, the existing models for
LFQA fall short in performance. For example, the KILT benchmark, recently
introduced by Petroni et al. (2020), is a framework that evaluates retrieval-
augmented models on various knowledge-intensive tasks, such as ELI5. It
assesses LFQA models based on the quality of their generated answers (measured
using ROUGE-L against reference answers) as well as the relevance of retrieved
documents (measured using R-precision against human-annotated relevant
documents). However, the utilization of retrieved contexts, such as passages
or documents by models on the KILT leaderboard, is found to be minimal,
according to the investigation conducted by Krishna et al. (2021). This lack
of utilization poses a challenge for retrieval-augmented models aiming to
enhance their performance in LFQA tasks. More specifically, the retrieved
contexts fail to contribute significantly to the acquisition of new
information necessary for generating comprehensive answers, impeding the
models’ progress.

One of the main reasons for the above problem is that previous models retrieve
relevant contexts from a provided knowledge source or corpus, relying solely
on the question itself. They overlooked the possibility that the question
alone might not provide sufficient information to identify the relevant
contexts within the corpus. Additionally, generating detailed long-form
answers often entails aggregating knowledge snippets from diverse sources to
provide a comprehensive explanation. To overcome these limitations, we propose
a Long-Form Question Answering (LFQA) model with iterative planning, retrieve,
and generation (IPRG) approach. The idea behind our model is that even if the
question does not have enough information for a complete answer generation, we
may use that as an initial query to construct a plan, retrieve contexts, and
generate a preliminary answer. This preliminary answer can then provide new
hints for gathering further information by the next step of planning and
retrieving. Consequently, it will generate more detailed answers for the LFQA
task.

To evaluate the performance of the proposed models, we conduct both
quantitative and qualitative evaluations on two datasets from two different
domains. As specified before the limitation of the ELI5 dataset, we use two
datasets with more definitive long-form answers from both an open domain
(i.e., wikiHow) and a specific technical domain (i.e., apple exchange
dataset).

##  2 Methodology

Figure 1: IPRG: At i-th iteration, KPG produces a set of keywords from the
question and pretext. Then, Retriever retrieves k passages by using the
question, pretext, and plan. Finally, the i-th sentence in the answer
paragraph generated by AG is appended to OA. The OA serves as pretext at the
next iteration.

The proposed IPRG model addresses the task of LFQA where it takes as input a
question $q$ and a corpus $\mathcal{C}$. The corpus can be obtained through
methods such as web search or by using a static corpus like Wikipedia. The
model consists of three modules: 1) A Keyword Plan Generator
$p(w_{i}|[q;y_{1:i-1}])$ that generates a keyword plan $w_{i}$ consisting of a
set of keywords for the next answer sentence given the concatenation of the
question $q$ and already generated answer sentences $y_{1:i-1}$, 2) A
Retriever $p(C_{i}|[q;w_{i}])$ that retrieves top $k$ passages as contexts
$C_{i}$ from $\mathcal{C}$ for supporting the next answer sentence generation
based on the question $q$ and current keyword plan $w_{i}$, and 3) an Answer
Generator $p(y_{i}|[q;k_{i};C_{i}])$ that generate next sentence for answer
given the question, keyword plan and retrieved passages. In the following
subsections, we will describe each of these modules, followed by how they are
combined and trained for the LFQA task. The overview of the proposed
architecture is shown in Figure 1.

###  2.1 Keyword Plan Generator

A long-form answer demands detailed information. However, the conventional
text generation models generally can generate short text with informativeness
and coherence and tend to hallucinate or be repetitive if we force them to
generate longer texts. One possible solution for this can be iteratively
generating short text (i.e., sentence) at each iteration rather than
generating the whole output at once. However, it still need to solve the
problem of generating repetitive sentences. To solve this issue, we predict
some key points that will be used to guide the generation of the next answer
sentence at each iteration. In other words, we plan to generate some future
keywords that will be discussed in the next round of answer sentences. And
these keywords also help find relevant contexts from the given corpus in the
subsequent step. To do so, we frame this as a text-to-keyword generation (or
prediction) problem.

To generate an answer at each iteration, we first plan by finding what key
points need to be discussed in the next sentence of the answer, and these key
points also help find relevant contexts from the given corpus. We define this
as a keyword generation task, where given a prompt, it generates a set of
keywords. Initially, the prompt is the question itself. In the subsequent
iterations, the generated answers portions get concatenated with the question
to make the next prompt for generating the next set of keywords. More
specifically, the Keyword Plan Generator $p(w_{i}|[q;y_{1:i-1}])$ that
generates a keyword plan $w_{i}$ consisting of a set of keywords for the next
answer sentence given the concatenation of the question $q$ and already
generated answer sentences $y_{1:i-1}$.

To train the keyword plan generator, we convert each of the QA pairs in the
training dataset into some texts to keyword-set pairs. For example, a
question: “How to put or move downloaded files in different folders depending
on file type?" and the first sentence of the ground truth answer is “I don’t
know of any Safari extension doing this but you may use Automator to create a
folder action attached to your preferred download folder sorting files
according to their extension or kind to various folders.” To convert the above
pair into a training sample of the keyword plan generator, we first use an
existing keyword extraction model to first extract important keywords from the
first sentence of the ground truth answer, which is, for instance, Keys =
[’preferred download folder’, ’safari extension’, ’various folders’, ’folder
action’, ’use’]. We use these extracted Keys as ground truth keywords for the
Question. So the task in training is, given the Question, can we generate the
Keys? In other words, starting with the Question itself, we generate the
keywords of the first answer sentence using a seq2seq model. Subsequently, we
add each sentence of the answer one after another as input to predict the
keywords of the next answer sentence. We initialized with pretrained BART
Lewis et al. (2019) for the text-to-keyword generation task.

###  2.2 Retriever

At each step of the iterative process, given the question and generated
keyword plan, we retrieve relevant contexts (e.g., passages or sentences) from
the given corpus. Specifically, we concatenated the question, pretext, and
generated keywords as a query to retrieve the top $k$ contexts from the
corpus. We use an existing dense passage retrieval (DPR) Karpukhin et al.
(2020) model for this purpose.

###  2.3 Answer Generator

In this module, we ensemble the question, pretext, generated keywords, and
retrieved contexts as input for the final answer generation model, which is
another seq2seq model initialized using BART Lewis et al. (2019). Now, for the
answer generation, one possible approach can be only generating the next
sentence rather than the whole answer at a time. However, this may make the
generated sentences disconnected from each other and which may result in the
repetition of sentences in the generated answer. To solve this problem, we
generate the paragraph length instead of a single sentence and append the
first new sentence in the final output sentence. To train this seq2seq model,
we use the answer sentences up to the next answer sentences as ground-truth
answers.

##  3 Experiments

WikiHowQA dataset:  We prepare a novel long-form question-answering dataset,
WikiHowQA, based on the WikiHow knowledge base
111https://www.wikihow.com/Main-Page. Each article title is a “How to”
question. Field experts have written these articles and provided a coherent
paragraph summary for each one. Unlike summarization datasets Cohen et al.
(2021); Koupaee and Wang (2018), we design this dataset for open-domain long-
form question answering tasks, where the paragraph summaries of articles are
served as the long-form answers to the title questions (as shown in Figure 2),
and Wikipedia dump is used as knowledge corpus to retrieve relevant contexts.
More details can be found in Appendix A.2.

Apple Exchange Dataset:  To evaluate the proposed method in a technical
domain, we use the Apple Exchange dataset adopted from a large COALA dataset
Rücklé et al. (2019). The answers to these technical questions require deeper
technical knowledge compared to WikiHowQA. Thus Wikipedia dumps are not
sufficiently informative to answer these questions. We retrieve the top 10
sentences relevant to each ground truth answer sentence in Google search,
excluding sentences from the StackExchange website, and compile them into a
knowledge corpus. This corpus can be easily updated online. For comparison
purposes, we create a static corpus by crawling the web-search results.

###  3.1 Comparison with Baselines

Baselines: We compare our models with i) pretrained models such as GPT2-XL
Radford et al. (2019) and T5-3b Raffel et al. (2020), ii) Sequence-to-Sequence
BART Lewis et al. (2019) model, retrieval-augmented generation DPR+BART
Petroni et al. (2020) model, Fusion in Decoder (FiD) model Izacard and Grave
(2020) and MDR Xiong et al. (2021). We fine-tune both BART, DPR+BART, FiD, and
MDR on each target datasets and report results in Table 1.

IRG: A variant of IPRG by excluding Keyword Plan Generator in which both
Retriever and Answer Generator do not take any sequence of keywords as input.

Models | WikiHow QA | Apple Exchange
---|---|---
Rouge-1 | Rouge-L | Rouge-1 | Rouge-L
Recall | F1 | Recall | F1 | Recall | F1 | Recall | F1
GPT2 | 4.66 | 8.19 | 4.41 | 7.75 | 0.27 | 0.47 | 0.27 | 0.47
T5 | 4.02 | 6.68 | 3.78 | 6.26 | 0.30 | 0.49 | 0.30 | 0.49
BART | 28.24 | 32.49 | 26.40 | 30.39 | 11.81 | 16.23 | 10.45 | 14.36
FiD | 25.19 | 31.59 | 23.72 | 29.75 | 6.31 | 9.48 | 5.79 | 8.61
DPR + BART | 28.71 | 32.74 | 26.78 | 30.54 | 12.88 | 17.22 | 11.32 | 15.16
MDR | 33.50 | 33.30 | 31.01 | 30.40 | 22.74 | 21.50 | 20.76 | 19.56
IRG | 33.68 | 3.29 | 31.30 | 30.99 | 23.88 | 21.78 | 21.70 | 19.75
IPRG | 35.36 | 33.65 | 32.88 | 31.25 | 24.73 | 22.13 | 23.63 |  20.42
Table 1: Comparison with baselines (Best and 2nd best score)

Results: Both IPRG and IRG consistently outperform all the baselines in all
metrics. Specifically, they outperform in recall scores (R-1 & R-L) by a large
margin. In LFQA, the answers demand a larger coverage of ground truth
information. In other words, the higher recall values imply more detailed
answers with accurate information.

IPRG generates more Entailed less Contradictory answer. Not only comparing
performances by measuring the text overlapping with references, but we also
compute entailment scores to evaluate how much the model’s generated answers
are logically aligned to the ground truth. We leverage a pretrained BART-
large-mnli model (available on
Huggingface222https://huggingface.co/facebook/bart-large-mnli) where the
generated answers are considered as hypothesis and ground-truth as the NLI
premise. This model calculates the scores indicating the logical entailment,
contradiction, or neutrality of a hypothesis with respect to the premise. As
reported in Table 2, IPRG achieves more entailment scores as well as less
contradictory scores in both datasets. Whereas, DPR+BART generated answers
contain more contradiction. This happens because the contexts retrieved using
only questions might lack relevant important information.

Models | WikiHow QA | Apple Exchange
---|---|---
NLI | NLI
Entailed | Contradict | Entailed | Contradict
DPR + BART | 7.29 | 20.37 | 10.00 | 20.49
MDR | 6.07 | 12.48 | 9.63 | 20.77
IRG | 7.09 | 14.86 | 9.52 | 19.56
IPRG | 8.34 | 10.98 | 12.74 | 13.88

Table 2: Factual Consistency Comparison

Planning enhances relevant Contexts Retrieval resulting in Comprehensive
Answers. DPR+BART performs per BART, indicating that question-only retrieved
contexts in a single pass have minimal impacts. On the other hand, iterative
retrieval refines the contexts resulting in the promising performance of the
IRG model. Furthermore, the performance of the IPRG model is boosted by the
two-fold advantages: i) keyword planning helps the retrieval module to
identify and find out important information and the answer generator module to
efficiently employ the current retrieved knowledge into answers at each
iteration; ii) the understanding and knowledge about questions are getting
enriched and refined by iterative characteristics of planning-retrieval-
generation process. Thus, the performance improvement is reflected in both
Rouge scores and entailment scores. Please refer to Section A.3 in appendix
for a detailed walkthrough of our whole process.

Figure 2: Comprehensiveness of answers w.r.t. three aspects: Strengthening,
Interval training, Stretching.

A Qualitative Case Study: Our IPRG model can effectively capture relevant
aspects to generate an elaborative answer for the target question. For
instance, in Figure 2, the ground-truth answer mentions three ways to improve
through: i) Strength training, ii) Interval training, and iii) Stretching. A
well-detailed answer should include all aspects. We can see that both IPRG and
IRG variants showed superior performance in capturing aspects. DPR+BART only
includes details about strength training but failed to capture the other two.
On the other hand, IPRG perfectly captures all three aspects while stretching
aspect is absent in IRG’s answer.

###  3.2 Related Works

Open domain question answering is the task of answering questions by utilizing
a knowledge base or corpus. In this setup, passage retrieval is a key step to
retrieving supporting documents before answering the question. A plethora of
research has been conducted on retrieval-augmented question answering Lewis et
al. (2020); Petroni et al. (2020); Mao et al. (2021); Izacard and Grave
(2021); Nguyen et al. (2016). Existing works mostly focus on answering factoid
questions.

However, due to the recent emergent of generative models Radford et al.
(2019); Lewis et al. (2019); Raffel et al. (2020), long-form question
answering has become an active research area crowded by lots of unavoidable
challenges such as factual hallucination, retrieving relevant context, and
fluent and logically consistent answer generation. RAG Lewis et al. (2020), an
end-to-end retrieval-based generative model, which incorporates DPR Karpukhin
et al. (2020) to retrieve supporting passages and then employ BART Lewis et
al. (2019) to generate long-form answers from questions concatenated with
evidence documents. Instead of concatenating document texts, FID Izacard and
Grave (2021) encodes first retrieved documents independently and then fuses
representations into a decoder to generate answers. RBG Su et al. (2022)
combines outputs from fusion-in-decoder module and machine reading
comprehension module by following the mechanism of pointer-generator network
See et al. (2017). To improve the contexts, Re2G Glass et al. (2022) augment
RAG architecture by adding a learnable reranker module to select top k
passages from a pool of passages from multiple retrievers. It is to be noted
that all of these models perform their retrieval process once using questions
as queries and generate answers in a single hop.

Recently, several multi-hop question-answering models have been introduced in
Xiong et al. (2020); Wang et al. (2022); Yavuz et al. (2022), but are limited
to factoid question answering.

##  4 Conclusion

This paper introduces a new approach for long-form question-answering tasks by
iteratively planning content through keywords, retrieving contexts from a
corpus, and generating answers using all the available information. Unlike
existing long-form question-answering models that suffer from underutilization
of retrieved contexts, our model demonstrates better retrieval of relevant
contexts from diverse sources feeding the generated answer with iterative
refinement. Experiments on multiple datasets show the superiority of our model
over other state-of-the-art models.

##  5 Limitation

Firstly, the proposed model uses a keyword planning module that depends on
existing keyword extraction methods for training the module. Therefore, the
error from the keyword extraction method may propagate to the other modules,
resulting inconsistent and irrelevant answer generation. Developing a better
task-dependent keyword planning generation that does not rely on the existing
keyword extraction method can be a future research direction to improve the
current model. Moreover, not all kinds of keywords are important as content
plans for future answer generation. Therefore, automatically identifying
informative keywords, in this case, can also improve the quality of the
result.

Secondly, while training, the proposed IPRG model separately deals with each
module which also makes it susceptible to error propagation. There is no scope
to learn from one module’s error to refine the model to another. Therefore,
developing a joint end-end model can be another future research direction in
this regard.

## References

  * Cohen et al. (2021) Nachshon Cohen, Oren Kalinsky, Yftah Ziser, and Alessandro Moschitti. 2021.  WikiSum: Coherent summarization dataset for efficient human-evaluation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 212–219, Online. Association for Computational Linguistics.
  * Coleman and Liau (1975) Meri Coleman and Ta Lin Liau. 1975.  A computer readability formula designed for machine scoring.  _Journal of Applied Psychology_ , 60(2):283.
  * Dale and Chall (1948) Edgar Dale and Jeanne S Chall. 1948.  A formula for predicting readability: Instructions.  _Educational research bulletin_ , pages 37–54.
  * Fan et al. (2019) Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019.  Eli5: Long form question answering.  _arXiv preprint arXiv:1907.09190_.
  * Farr et al. (1951) James N Farr, James J Jenkins, and Donald G Paterson. 1951.  Simplification of flesch reading ease formula.  _Journal of applied psychology_ , 35(5):333.
  * Glass et al. (2022) Michael Glass, Gaetano Rossiello, Md Faisal Mahbub Chowdhury, Ankita Naik, Pengshan Cai, and Alfio Gliozzo. 2022.  Re2G: Retrieve, rerank, generate.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2701–2715, Seattle, United States. Association for Computational Linguistics.
  * Gunning et al. (1952) Robert Gunning et al. 1952.  Technique of clear writing.
  * Izacard and Grave (2020) Gautier Izacard and Edouard Grave. 2020.  Leveraging passage retrieval with generative models for open domain question answering.  _arXiv preprint arXiv:2007.01282_.
  * Izacard and Grave (2021) Gautier Izacard and Edouard Grave. 2021.  Leveraging passage retrieval with generative models for open domain question answering.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 874–880, Online. Association for Computational Linguistics.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  _arXiv preprint arXiv:2004.04906_.
  * Koupaee and Wang (2018) Mahnaz Koupaee and William Yang Wang. 2018.  Wikihow: A large scale text summarization dataset.  _arXiv preprint arXiv:1810.09305_.
  * Krishna et al. (2021) Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021.  Hurdles to progress in long-form question answering.  _arXiv preprint arXiv:2103.06332_.
  * Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019.  Albert: A lite bert for self-supervised learning of language representations.  _arXiv preprint arXiv:1909.11942_.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ , 33:9459–9474.
  * Mao et al. (2021) Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. 2021.  Rider: Reader-guided passage reranking for open-domain question answering.  _arXiv preprint arXiv:2101.00294_.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  Ms marco: A human generated machine reading comprehension dataset.  _choice_ , 2640:660.
  * Petroni et al. (2020) Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, et al. 2020.  Kilt: a benchmark for knowledge intensive language tasks.  _arXiv preprint arXiv:2009.02252_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know what you don’t know: Unanswerable questions for squad.  _arXiv preprint arXiv:1806.03822_.
  * Rücklé et al. (2019) Andreas Rücklé, Nafise Sadat Moosavi, and Iryna Gurevych. 2019.  Coala: A neural coverage-based approach for long answer selection with small data.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 6932–6939.
  * See et al. (2017) Abigail See, Peter J Liu, and Christopher D Manning. 2017.  Get to the point: Summarization with pointer-generator networks.  _arXiv preprint arXiv:1704.04368_.
  * Senter and Smith (1967) RJ Senter and Edgar A Smith. 1967.  Automated readability index.  Technical report, Cincinnati Univ OH.
  * Su et al. (2022) Dan Su, Xiaoguang Li, Jindi Zhang, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. 2022.  Read before generate! faithful long form question answering with machine reading.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 744–756, Dublin, Ireland. Association for Computational Linguistics.
  * Wang et al. (2022) Siyuan Wang, Zhongyu Wei, Zhihao Fan, Qi Zhang, and Xuanjing Huang. 2022.  Locate then ask: Interpretable stepwise reasoning for multi-hop question answering.  _arXiv preprint arXiv:2208.10297_.
  * Xiong et al. (2021) Wenhan Xiong, Xiang Li, Srini Iyer, Jingfei Du, Patrick Lewis, William Yang Wang, Yashar Mehdad, Scott Yih, Sebastian Riedel, Douwe Kiela, and Barlas Oguz. 2021.  Answering complex open-domain questions with multi-hop dense retrieval.  In _International Conference on Learning Representations_.
  * Xiong et al. (2020) Wenhan Xiong, Xiang Lorraine Li, Srini Iyer, Jingfei Du, Patrick Lewis, William Yang Wang, Yashar Mehdad, Wen-tau Yih, Sebastian Riedel, Douwe Kiela, et al. 2020.  Answering complex open-domain questions with multi-hop dense retrieval.  _arXiv preprint arXiv:2009.12756_.
  * Yavuz et al. (2022) Semih Yavuz, Kazuma Hashimoto, Yingbo Zhou, Nitish Shirish Keskar, and Caiming Xiong. 2022.  Modeling multi-hop question answering as single sequence prediction.  _arXiv preprint arXiv:2205.09226_.

##  Appendix A Appendix

###  A.1 Related Works

Open domain question answering is the task of answering questions by utilizing
a knowledge base or corpus. In this setup, passage retrieval is a key step to
retrieving supporting documents before answering the question. A plethora of
research has been conducted on retrieval-augmented question answering Lewis et
al. (2020); Petroni et al. (2020); Mao et al. (2021); Izacard and Grave
(2021); Nguyen et al. (2016). Existing works mostly focus on answering factoid
questions.

However, due to the recent emergent of generative models Radford et al.
(2019); Lewis et al. (2019); Raffel et al. (2020), long-form question
answering has become an active research area crowded by lots of unavoidable
challenges such as factual hallucination, retrieving relevant context, and
fluent and logically consistent answer generation. RAG Lewis et al. (2020), an
end-to-end retrieval-based generative model, which incorporates DPR Karpukhin
et al. (2020) to retrieve supporting passages and then employ BART Lewis et
al. (2019) to generate long-form answers from questions concatenated with
evidence documents. Instead of concatenating document texts, FID Izacard and
Grave (2021) encodes first retrieved documents independently and then fuses
representations into a decoder to generate answers. RBG Su et al. (2022)
combines outputs from fusion-in-decoder module and machine reading
comprehension module by following the mechanism of pointer-generator network
See et al. (2017). To improve the contexts, Re2G Glass et al. (2022) augment
RAG architecture by adding a learnable reranker module to select top k
passages from a pool of passages from multiple retrievers. It is to be noted
that all of these models perform their retrieval process once using questions
as queries and generate answers in a single hop.

Recently, several multi-hop question-answering models have been introduced in
Xiong et al. (2020); Wang et al. (2022); Yavuz et al. (2022), but are limited
to factoid question answering.

###  A.2 WikiHowQA dataset

We collect article titles that have coherent paragraph summaries by filtering
out those which have no paragraph-styled summaries or incomplete summaries.
Consequently, the WikiHowQA dataset comprises 37,815 question-answer pairs.
One of the key attributes of this dataset is that the answers are expert-
written in plain English, which requires less world knowledge to understand
and evaluate the answers. In order to measure the readability scores, we use
classical readability metrics such as FKGL Farr et al. (1951), GFI Gunning et
al. (1952), ARI Senter and Smith (1967), CLI Coleman and Liau (1975), DCR Dale
and Chall (1948). All metrics’ generated scores indicate the number of years
of formal education required for a native English speaker to understand the
answer text. As reported in Table 3, the scores for WikiHowQA answers are
significantly smaller than that of the widely used ELI5 dataset.

Datasets | Readability
---|---
FKGL | DCR | ARI | CLI | GFI
WikiHowQA | 8.70 | 7.78 | 9.12 | 7.58 | 11.37
ELI5 | 9.81 | 8.38 | 10.22 | 8.97 | 12.70
Table 3: Readability scores for reference answers. Smaller value is more Readable. Query |  Retrieved top two Passages
---|---
Question: How to Improve Your Running Speed and Endurance? Pretext: "" Keywords: walking lunges, run, endurance, stretch, cramps |  P1: including 2–3 days of core strengthening. In the base phase the athlete now works on building cardiovascular endurance by having several long runs staying in heart rate zone 1-2 every week and each week adding slightly more mileage (using 10% rule for safely increasing the mileage). Core strengthening is continued in the base period. Once the base phase is complete and the athlete has sufficient endurance, the build period is needed to give the athlete the ability to hold a faster pace for the race duration. The build phase is where duration of runs is traded for intensity. P2: Stretching does not appear to reduce the risk of injury during exercises, except perhaps a dynamic warm-up for runners. While running places extreme stress loads on the joints, static stretching can help to improve joint flexibility. However, this has not been proven to reduce risk of injury in the runners. A dynamic (stretching) warm up has been shown to help overall running performance. One should avoid overtraining to prevent cramps. The calf muscles slowly respond to
Question: How to Improve Your Running Speed and Endurance? Pretext: To improve your running speed and endurance, perform a "stretch"—walking lunges, or a "run-stretch" where you stand up straight and push yourself up to avoid cramps. Keywords: stretch, arms, muscles |  P1: leg muscles. Unlike the previous exercises, these are outdoor activities. The calf exercises should be performed every 3–4 days. They can be alternated and interchanged to avoid getting used to the load. General Workout Tips Before any serious activity, including doing calf-building exercises, the muscles and joints should be properly warmed up. Aerobic exercises should be performed at the beginning of a workout. A workout should end with calf strengthening exercises to stimulate their growth, and stretching that involves taking a wide step backward, placing the heel on the floor, and bending the torso forward. Each leg should be stretched for 10–20. P2: must return to, and pause in, the correct starting position before continuing. If you rest on the ground or raise either hand or foot from the ground, your performance will be terminated. You may reposition your hands and/or feet during the event as long as they remain in contact with the ground at all times. Correct performance is important. You will have two minutes in which to do as many push-ups as you can." Sit-up "The sit-up event measures the endurance of the abdominal and hip-flexor muscles. On the command ’get set,’ assume the starting position by lying on your.
Question: How to Improve Your Running Speed and Endurance? Pretext: To improve your running speed and endurance, perform a "stretch"—walking lunges, or a "run-stretch" where you stand up straight and push yourself up to avoid cramps. It involves stretching your arms and legs over your body. Keywords: sprinting, repeat, process, minutes, times |  P1: in losing weight, staying in shape and improving body composition. Research suggests that the person of average weight will burn approximately 100 calories per mile run. Running increases one’s metabolism, even after running; one will continue to burn an increased level of calories for a short time after the run. Different speeds and distances are appropriate for different individual health and fitness levels. For new runners, it takes time to get into shape. The key is consistency and a slow increase in speed and distance. While running, it is best to pay attention to how one’s body feels. P2: Many training programs last a minimum of five or six months, with a gradual increase in the distance run and finally, for recovery, a period of tapering in the one to three weeks preceding the race. For beginners wishing to merely finish a marathon, a minimum of four months of running four days a week is recommended. Many trainers recommend a weekly increase in mileage of no more than 10%.
Question: How to Improve Your Running Speed and Endurance? Pretext: To improve your running speed and endurance, perform a "stretch"—walking lunges, or a "run-stretch" where you stand up straight and push yourself up to avoid cramps. It involves stretching your arms and legs over your body. Start by sprinting for 5 to 10 minutes to build up your endurance, then take a short break and repeat the process 4 to 5 times to warm up your muscles. Keywords: run, repeat, muscles, reps, recover |  P1: a base for more intense workouts by strengthening the heart and increasing the muscles’ ability to use oxygen, and to recover between hard workouts. Daniels recommends that most training miles are performed in E pace. Typical E runs include continuous runs up to about an hour. Marathon (M) pace At 80-85% $HR_{max}$, this intensity is primarily aimed towards runners training for the marathon. The pace is one at which the runner hopes to compete. The pace can be included in other programs for a more intense workout, especially if the runner feels fresh and there is enough time to recover afterwards. P2: in losing weight, staying in shape and improving body composition. Research suggests that the person of average weight will burn approximately 100 calories per mile run. Running increases one’s metabolism, even after running; one will continue to burn an increased level of calories for a short time after the run. Different speeds and distances are appropriate for different individual health and fitness levels. For new runners, it takes time to get into shape. The key is consistency and a slow increase in speed and distance. While running, it is best to pay attention to how one’s body feels.
Question: How to Improve Your Running Speed and Endurance? Pretext: To improve your running speed and endurance, perform a "stretch"—walking lunges, or a "run-stretch" where you stand up straight and push yourself up to avoid cramps. It involves stretching your arms and legs over your body. Start by sprinting for 5 to 10 minutes to build up your endurance, then take a short break and repeat the process 4 to 5 times to warm up your muscles. Repeat these exercises 4 times in the middle of your run, which will give your muscles time to rest and recover. Keywords: stretch, arms, legs, run, repeat, times |  P1: must return to, and pause in, the correct starting position before continuing. If you rest on the ground or raise either hand or foot from the ground, your performance will be terminated. You may reposition your hands and/or feet during the event as long as they remain in contact with the ground at all times. Correct performance is important. You will have two minutes in which to do as many push-ups as you can." Sit-up "The sit-up event measures the endurance of the abdominal and hip-flexor muscles. On the command ’get set,’ assume the starting position by lying on your P2: in losing weight, staying in shape and improving body composition. Research suggests that the person of average weight will burn approximately 100 calories per mile run. Running increases one’s metabolism, even after running; one will continue to burn an increased level of calories for a short time after the run. Different speeds and distances are appropriate for different individual health and fitness levels. For new runners, it takes time to get into shape. The key is consistency and a slow increase in speed and distance. While running, it is best to pay attention to how one’s body feels.
Table 4: Walk through IPRG model workflow with an example Query |  Retrieved top two Passages
---|---
Question: How to Improve Your Running Speed and Endurance? Pretext: To improve your running speed and endurance, perform a "stretch"—walking lunges, or a "run-stretch" where you stand up straight and push yourself up to avoid cramps. It involves stretching your arms and legs over your body. Start by sprinting for 5 to 10 minutes to build up your endurance, then take a short break and repeat the process 4 to 5 times to warm up your muscles. Repeat these exercises 4 times in the middle of your run, which will give your muscles time to rest and recover. After your run is over, run for 2 to 3 minutes to let your muscles cool down. Keywords: running, stamina, interval training, muscles, help |  P1: cope with the intensity, and to train for longer periods of time, this training is performed as interval training, hence the name. The interval between each workout should be a little less than the time of the work bout. Optimum intervals are 3–5 minutes long. There is no benefit to exceeding 5 minutes at this pace, under Daniels’ theory, which means that despite the popularity of mile-repeats in many running groups, Daniels discourages them for people whose pace is slower than about 5:00/mile, preferring shorter intervals such as 1200 meters. P2: (a minimum of 45 minutes). The development of aerobic and anaerobic capacities, and the adaptability of fartlek - to mimic running during specific sports - are characteristics it shares with other types of interval training. Sprint interval training "Walk-back sprinting" is one example of interval training for runners, in which one sprints a short distance (anywhere from 100 to 800 metres), then walks back to the starting point (the recovery period), to repeat the sprint a certain number of times. To add challenge to the workout, each of these sprints may start at predetermined time intervals - e.g. 200 metre. One should avoid overtraining to prevent cramps. The calf muscles slowly respond to
Question: How to Improve Your Running Speed and Endurance? Pretext: To improve your running speed and endurance, perform a "stretch"—walking lunges, or a "run-stretch" where you stand up straight and push yourself up to avoid cramps. It involves stretching your arms and legs over your body. Start by sprinting for 5 to 10 minutes to build up your endurance, then take a short break and repeat the process 4 to 5 times to warm up your muscles. Repeat these exercises 4 times in the middle of your run, which will give your muscles time to rest and recover. After your run is over, run for 2 to 3 minutes to let your muscles cool down. You should also use some interval training to help build your stamina. Keywords: run, sprint, muscles, time |  P1: workouts the day after interval sessions. Finally the race phase of the periodization approach is where the duration of the workouts decreases but intense workouts remain so as to keep the high lactate threshold that was gained in the build phase. In Ironman training, the race phase is where a long "taper" occurs of up to 4 weeks for highly trained Ironman racers. A final phase is designated transition and is a period of time, where the body is allowed to recover from the hard race effort and some maintenance endurance training is performed so the high fitness level. P2:(a minimum of 45 minutes). The development of aerobic and anaerobic capacities, and the adaptability of fartlek - to mimic running during specific sports - are characteristics it shares with other types of interval training. Sprint interval training "Walk-back sprinting" is one example of interval training for runners, in which one sprints a short distance (anywhere from 100 to 800 metres), then walks back to the starting point (the recovery period), to repeat the sprint a certain number of times. To add challenge to the workout, each of these sprints may start at predetermined time intervals - e.g. 200 metre.
Question: How to Improve Your Running Speed and Endurance? Answer: To improve your running speed and endurance, perform a "stretch"—walking lunges, or a "run-stretch" where you stand up straight and push yourself up to avoid cramps. It involves stretching your arms and legs over your body. Start by sprinting for 5 to 10 minutes to build up your endurance, then take a short break and repeat the process 4 to 5 times to warm up your muscles. Repeat these exercises 4 times in the middle of your run, which will give your muscles time to rest and recover. After your run is over, run for 2 to 3 minutes to let your muscles cool down. You should also use some interval training to help build your stamina. Sprint for 10 seconds on a hard surface, then 5 minutes on a softer surface to work your core muscles. |
Table 5: Walk through IPRG model workflow with an example (as the continuation
of Table 4)

###  A.3 Model Workflow with an Example

In this section, we illustrate our model workflow using an example step-by-
step iteratively shown in Figure 4 and 5. Each row of these tables contains
the question, current pretext, and generated keywords at each iteration step
in the first column. For convenience, we display the top 2 retrieved passages
in the second column.