* Srivastava et al. (2022) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartłomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Chris Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martínez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch-López, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Kocoń, Jana Thompson, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Omondi, Kory Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Colón, Luke Metz, Lütfi Kerem Şenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramírez Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael A. Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michał Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Miłkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramón Risco Delgado, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan LeBras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Debnath Shyamolima, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven T. Piantadosi, Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Timothy Telleen-Lawton, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2022.  URL https://arxiv.org/abs/2206.04615.
  * Steinegger and Söding (2017) Martin Steinegger and Johannes Söding.  MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets.  _Nature Biotechnology_ , 35(11):1026–1028, October 2017.  doi: 10.1038/nbt.3988.  URL https://doi.org/10.1038/nbt.3988.
  * Suzgun et al. (2022) Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei.  Challenging big-bench tasks and whether chain-of-thought can solve them, 2022.  URL https://arxiv.org/abs/2210.09261.
  * Taboureau et al. (2011) Olivier Taboureau, Sonny Kim Nielsen, Karine Audouze, Nils Weinhold, Daniel Edsgärd, Francisco S Roque, Irene Kouskoumvekaki, Alina Bora, Ramona Curpan, Thomas Skøt Jensen, Søren Brunak, and Tudor I Oprea.  ChemProt: a disease chemical biology database.  _Nucleic Acids Res._ , 39(Database issue):D367–72, January 2011.
  * Talmor et al. (2018) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant.  Commonsenseqa: A question answering challenge targeting commonsense knowledge.  _CoRR_ , abs/1811.00937, 2018.  URL http://arxiv.org/abs/1811.00937.
  * Tay et al. (2022a) Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q. Tran, Dani Yogatama, and Donald Metzler.  Scaling laws vs model architectures: How does inductive bias influence scaling?, 2022a.  URL https://arxiv.org/abs/2207.10551.
  * Tay et al. (2022b) Yi Tay, Jason Wei, Hyung Won Chung, Vinh Q. Tran, David R. So, Siamak Shakeri, Xavier Garcia, Huaixiu Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery, Denny Zhou, Donald Metzler, Slav Petrov, Neil Houlsby, Quoc V. Le, and Mostafa Dehghani.  Transcending scaling laws with 0.12022b.  URL https://arxiv.org/abs/2210.11399.
  * Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le.  Lamda: Language models for dialog applications, 2022.  URL https://arxiv.org/abs/2201.08239.
  * V et al. (2021) Venktesh V, Mukesh K. Mohania, and Vikram Goyal.  Tagrec: Automated tagging of questions with hierarchical learning taxonomy.  _CoRR_ , abs/2107.10649, 2021.  URL https://arxiv.org/abs/2107.10649.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _CoRR_ , abs/1706.03762, 2017.  URL http://arxiv.org/abs/1706.03762.
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le.  Finetuned language models are zero-shot learners, 2021.  URL https://arxiv.org/abs/2109.01652.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou.  Chain of thought prompting elicits reasoning in large language models, 2022.  URL https://arxiv.org/abs/2201.11903.
  * Weininger (1988) David Weininger.  Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules.  _Journal of Chemical Information and Computer Sciences_ , 28(1):31–36, 1988.  doi: 10.1021/ci00057a005.  URL https://doi.org/10.1021/ci00057a005.
  * Welbl et al. (2017) Johannes Welbl, Nelson F. Liu, and Matt Gardner.  Crowdsourcing multiple choice science questions.  _ArXiv_ , abs/1707.06209, 2017.
  * Wheeler (1990) John Wheeler.  Information, physics, quantum: The search for links.  _Zurek, W.H., Ed., Complexity, Entropy, and the Physics of Information_ , 1990.
  * Wigner (1959) Eugene Wigner.  The unreasonable effectiveness of mathematics in the natural sciences.  _Communications on Pure and Applied Mathematics_ , 1959.
  * Wu et al. (2017) Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay Pande.  Moleculenet: A benchmark for molecular machine learning, 2017.  URL https://arxiv.org/abs/1703.00564.
  * Xu et al. (2017) Yichong Xu, Jingjing Liu, Jianfeng Gao, Yelong Shen, and Xiaodong Liu.  Towards human-level machine reading comprehension: Reasoning and inference with multiple strategies.  _CoRR_ , abs/1711.04964, 2017.  URL http://arxiv.org/abs/1711.04964.
  * Yasunaga et al. (2022) Michihiro Yasunaga, Jure Leskovec, and Percy Liang.  Linkbert: Pretraining language models with document links, 2022.  URL https://arxiv.org/abs/2203.15827.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer.  Opt: Open pre-trained transformer language models, 2022.  URL https://arxiv.org/abs/2205.01068.
  * Zhou et al. (2022) Gengmo Zhou, Zhifeng Gao Gao, Qiankun Ding, Hang Zheng, Wei Xu, Hongteng, Linfeng Zhang, and Guolin Ke.  Uni-mol: A universal 3d molecular representation learning framework, 2022\.  URL https://chemrxiv.org/engage/chemrxiv/article-details/628e5b4d5d948517f5ce6d72.

##  Appendix A Appendix

###  A.1 Dataset Components

We cover the various components of the corpus in this section.

####  A.1.1 Papers

We source scientific papers from preprint servers such as arXiv, PMC and other
sources; see Table 22.

We also use the Semantic Scholar full text dataset (S2) to capture the long
tail of science (Lo et al., 2019a). We apply several quality filters,
including excluding papers from journals with certain keywords, and also
excluding papers with a low journal impact factor. Details of the filters we
used are contained in the Appendix.

We source abstracts where full texts are not open access. In total the full
dataset contains 48 million papers, abstract and full-text, up to July 2022.

Data source | Documents | Tokens
---|---|---
arXiv | 2 million | 35 billion
PMC | 3 million | 23 billion
Semantic Scholar | 3 million | 18 billion
PubMed Abstracts | 21 million | 5 billion
Semantic Scholar Abstracts | 19 million | 4 billion
bioRxiv | 128,059 | 1 billion
OSF | 54,905 | 428 million
medRxiv | 24,019 | 176 million
ACL | 25,518 | 150 million
PubAg Abstracts | 308,235 | 105 million
ChemRxiv | 7,617 | 67 million
Total | 48 million | 88 billion
Table 22: Paper sources used in our corpus

We use a modified version of the GROBID library for converting PDFs to text,
as well as obtaining titles, authors and citations (GROBID, 2008–2022). Where
mathematical LaTeX is available, for example in arXiv, we make sure to combine
the GROBID results with LaTeX source to recover mathematical content.

The final paper documents are stored in a markdown format, as opposed to full
LaTeX. We use markdown as the standard format for all documents in the corpus
to support knowledge blending between sources. Papers are citation processed,
following the title-based approach of Section 2.2.

####  A.1.2 Reference Material

We source encyclopedias, textbooks and educational material to create a base
of reference material that the model can learn from. The details are covered
in Table 23.

Data source | Documents | Tokens
---|---|---
Wikipedia | 6 million | 5 billion
StackExchange | 1.6 million | 1 billion
LibreText | 95,113 | 185 million
Wikibooks | 74,705 | 110 million
Open Textbooks | 647 | 94 million
MIT OCW | 25,640 | 90 million
Wikiversity | 38,138 | 52 million
ProofWiki | 32,389 | 12 million
Khan Academy | 3,075 | 7 million
Papers with Code | 13,430 | 4 million
IUPAC Goldbook | 6,788 | 1 million
Total | 8 million | 7 billion
Table 23: Reference material used in our corpus

We apply source specific processing for several of the datasets, specifically:

  * •

For StackExchange, we take questions from scientific sites; see the Appendix
for the subset used.

  * •

For Papers with Code and IUPAC Goldbook we apply data augmentation in the form
of prompt randomization. Sometimes we pose sections as questions/answers; for
example a section explaining a machine learning method is sometimes posed as
"Question: What is [method]?".

  * •

For KhanAcademy articles, we add `<work>` tokens for step-by-step reasoning
examples, which we explain shortly in Section 2.4.

We make an effort to preserve mathematical LaTeX and capture citations,
including hyperlinks to papers.

####  A.1.3 Knowledge Bases

We source fine-grained knowledge from scientific knowledge bases. The details
are covered in Table 24.

Data source | Documents | Tokens
---|---|---
PubChem Compound | 1.7 million | 1 billion
UniProt | 551,837 | 0.6 billion
RefSeq Genome | 69 | 0.1 billion
OEIS | 350,833 | 0.07 billion
Ribosome | 9,950 | 0.05 billion
LIPID MAPS | 45,273 | 0.03 billion
Reactome | 156 | 0.01 billion
NASA Exoplanet | 5,021 | 0.01 billion
Total | 2 million | 2 billion
Table 24: Knowledge bases used in our corpus

For the chemistry and biology datasets, we wrap modalities like SMILES and
protein sequences with their specialized tokens (see Section 2.1). For UniProt
we apply data augmentation to the document format:

  * •

Order Randomization \- with probability $0.5$ the protein sequence starts at
beginning of the document, else the end of document. This ensures we can learn
from $\text{seq}\rightarrow\text{property}$ and
$\text{property}\rightarrow\text{seq}$.

  * •

Format Randomization \- with probability $\frac{1}{3}$ we replace a
description, e.g. "The function of protein is…", with a Q&A, e.g. "Question:
What is the function of the protein? Answer: The function is…".

For NASA Exoplanet we apply order randomization to the exoplanet
characteristics.

For chemical and biological sequences, we take a small subset of available
entities. This is to ensure the model is not overly biased towards learning
natural sequences over natural language. Specifically:

  * •

For PubChem Compound, we take a small, random subset ($2$ million) of total
compounds ($110$ million).

  * •

For UniProt, we take reviewed Swiss-Prot proteins; a small subset ($0.5$
million) of total ($227$ million).

  * •

For RefSeq Genome, we take reference sequences, which is a small subset of
available nucleotide sequences. For the human genome, we only include the
protein-coding genes.

This is a constraint we can relax in future work, enabling for much larger
corpus. In this work, we focus on the first step of investigating whether a
single model can learn effectively in this multi-modal setting.

####  A.1.4 Common Crawl

We source academic and scientific content via a highly-filtered subset of
CommonCrawl. The details are covered in Table 25.

Data source | Documents | Tokens
---|---|---
ScientificCC | 0.8 million | 0.7 billion
AcademicCC | 0.05 million | 0.4 billion
Total | 0.9 million | 1.1 billion
Table 25: CommonCrawl material used in our corpus

For Scientific Common Crawl, we train a fasttext classifier to identify Common
Crawl webpages with scientific content (Joulin et al., 2016) using a noisy set
of 600 domains. We then manually annotated the domains predicted by fasttext
as scientific to assemble a list of 200 high-quality scientific and reference
domains.

For Academic Common Crawl, we assemble a list of academic domains, such as
university websites. We take PDFs from these domains, based on the Common
Crawl index, and process these using GROBID.

We do not LaTeX-process pages from these sources.

We found the quality of extracted text in CommonCrawl generally quite poor,
which is why we applied stringent filters. We suspect this could be an
important area for future work in order to capture more base scientific
knowledge.

####  A.1.5 Code

We source academic GitHub repositories from the Papers with Code index for
machine learning, physics, mathematics, statistics and astronomy. The index
does not explicitly cover sciences such as biology and chemistry, but many of
these repositories are captured as part of the general machine learning index.
We exclude repositories that do not have a license or copyright file.

####  A.1.6 <work> Datasets

For KhanProblems, we used the problems from AMPS and converted to a <work>
format (Hendrycks et al., 2021). Where possible we tried to include more
tedious steps to reduce errors from a single pass, but this annotation was
fairly incomplete and we suspect bigger gains are possible with more cleaning.

For GSM8k we use the provided training dataset and convert so the calculator
steps are performed by writing a Python program, following the <work> format
(Cobbe et al., 2021). In general, we found when the model went into this
prompt style, it was more error-prone. We think this is because the prompt
style made the model write too many programs within <work>, rather than
getting things ready to run in a single program. In general we found longer
<work> answers led to a higher chance of a mistake on the reasoning path.

For OneSmallStep, we made 50 problem set question templates, and randomized
the variables in the problem to get more prompt examples. We summarize the
fields we made prompts below.

Field | Templates
---|---
Astronomy | 2
Chemistry | 7
Electronics | 10
Mathematics | 15
Physics | 14
Statistics | 2
Total | 50

As we can see the diversity was not very large, and so further gains are
likely with more annotation.

Lastly we wrote 921 examples, based off internet examples, in a <work> format
for Workout. This was our highest quality dataset, and had reasonable
diversity across fields: mathematics, chemistry, biology, astronomy, physics,
geology, history. This is the type of dataset we would look to scale in future
work.

###  A.2 Dataset Deduplication

We use the following procedure for deduplicating the corpus:

  * •

We identify identical spans of 100 bytes or more (of utf-8 text) across the
whole corpus, except for some explicitly excluded data sources. We do this
using the repository from Lee et al. (2022).

  * •

We process corpus files in a predetermined order to prioritize some sources.
From a set of spans representing the exact same content across files, we
remove the span in the first file. If the same content repeats across a single
file and it was not found in the files before, all its occurrences are kept.

  * •

We merge duplicated spans separated by at most 4 bytes.

  * •

We narrow down the resulting spans to paragraph boundaries (i.e. "`\n\n`").

  * •

We remove the content from files corresponding to the spans.

###  A.3 Citation Identifier Ablations

We report ablations for the citation identifier ablations below, where we test
title-based identifiers versus alphanumeric identifiers.

Specifically, we set up an evaluation set of dataset and method names from
Papers with Code. The task is to predict the citation given the method or
dataset name, e.g. `ResNet [START_REF]`, where the target is `Deep Residual
Learning for Image Recognition, He`. We train a 6.7bn model on both types of
processing for the ablation. Method and dataset results are shown below.

| Citation Processing
---|---
| (a) Titles | (b) IDs
Method citations | Correct | Hallucinated | Incorrect | Correct | Hallucinated | Incorrect
$k=1$ | 13.8% | 54.5% | 31.7% | 1.8% | 3.5% | 94.7%
2 $\leq k<5$ | 30.4% | 38.6% | 31.1% | 9.3% | 4.0% | 86.7%
5 $\leq k<10$ | 36.3% | 29.5% | 34.2% | 17.9% | 0.0% | 82.1%
10 $\leq k<25$ | 43.0% | 15.8% | 41.2% | 38.8% | 3.0% | 58.2%
25 $\leq k<50$ | 53.4% | 8.7% | 37.9% | 43.7% | 0.0% | 56.3%
50 $\leq k<100$ | 64.8% | 9.9% | 25.3% | 60.6% | 1.4% | 38.0%
100 $\leq k<500$ | 64.6% | 8.3% | 27.1% | 63.5% | 1.0% | 35.4%
$\geq 500$ | 78.6% | 0.0% | 21.4% | 78.6% | 0.0% | 21.4%
Table 26: Citation Processing Ablation. We predict citations for the PWC Methods dataset using 6.7 billion size models. Papers are bucketed according to the number of citations (mentions) in the dataset. The title processing model has a higher accuracy, but greater risk of hallucination. There are 1,705 methods in this evaluation dataset. | Citation Processing
---|---
| (a) Titles | (b) IDs
Dataset citations | Correct | Hallucinated | Incorrect | Correct | Hallucinated | Incorrect
$k=1$ | 1.4% | 62.5% | 36.1% | 0.5% | 11.5% | 88.1%
$2\leq k<5$ | 5.0% | 59.2% | 35.8% | 0.6% | 10.2% | 89.2%
$5\leq k<10$ | 15.4% | 49.7% | 34.8% | 2.6% | 6.2% | 91.1%
$10\leq k<25$ | 25.7% | 36.8% | 37.5% | 8.3% | 4.8% | 86.9%
$25\leq k<50$ | 44.6% | 27.4% | 28.0% | 22.9% | 7.0% | 70.0%
$50\leq k<100$ | 58.6% | 17.7% | 23.6% | 41.4% | 7.7% | 50.9%
$100\leq k<500$ | 65.5% | 6.7% | 27.8% | 62.4% | 3.1% | 34.5%
$\geq 500$ | 81.8% | 6.1% | 12.1% | 81.8% | 3.0% | 15.2%
Table 27: Citation Processing Ablation. We predict citations for the PWC
Datasets dataset using 6.7 billion capacity models. There are 4,735 datasets
in this evaluation dataset.

###  A.4 120B Validation Loss Per Source

Figure 23: Validation Loss Per Source. Validation loss falls through training
for all dataset categories. Results are shown for the 120B model above.

###  A.5 Chain-of-Thought vs <work>

We used the recent results by Chung et al. (2022) of PaLM 540B on the MMLU
validation set (Hendrycks et al., 2020) for comparison. While use of reasoning
degrades performance versus direct prompting for both approaches, the `<work>`
token appears more robust.

Chain-of-Thought versus <work>
---
Subject | Examples | PaLM 540B CoT | GAL 30B <work> | GAL 120B <work>
Abstract Algebra | 11 | 9.1% | 27.3% | 27.3%
Astronomy | 16 | 7.1% | 43.8% | 25.0%
College Chemistry | 8 | 12.5% | 37.5% | 37.5%
College Computer Science | 11 | 9.1% | 45.5% | 54.6%
College Mathematics | 11 | 0.0% | 36.4% | 18.2%
College Physics | 11 | 36.4% | 36.4% | 45.5%
Econometrics | 11 | 33.3% | 33.3% | 33.3%
Electrical Engineering | 16 | 18.8% | 37.5% | 56.3%
Elementary Mathematics | 41 | 24.4% | 53.7% | 58.5%
Formal Logic | 9 | 0.0% | 21.4% | 21.4%
High School Chemistry | 22 | 22.7% | 27.3% | 36.4%
High School Computer Science | 9 | 33.3% | 44.4% | 44.4%
High School Mathematics | 29 | 24.1% | 31.0% | 51.7%
High School Physics | 17 | 11.8% | 23.5% | 29.4%
High School Statistics | 23 | 26.1% | 39.1% | 56.5%
Machine Learning | 11 | 18.2% | 9.1% | 27.3%
Overall | 261 | 19.1% | 35.9% | 42.4%
Table 28: <work> vs Chain-of-Thought. PaLM is evaluated with CoT 5-shot.
Galactica with the <work> token included in pre-training. Results here are on
MMLU dev for comparability with PaLM. BIG-bench
---
Benchmark | OPT 30B | OPT 175B | BLOOM 176B | GAL 30B | GAL 120B
Anachronisms | 47.4% | 49.1% | 1.3% | 47.0% | 48.7%
Analogical Similarity | 12.7% | 19.8% | 19.2% | 17.0% | 23.5%
Analytic Entailment | 40.0% | 52.9% | 48.6% | 47.1% | 51.3%
Causal Judgment | 53.7% | 55.3% | 54.7% | 49.5% | 51.1%
Crash Blossom | 42.1% | 36.8% | 47.4% | 42.1% | 42.1%
Crass AI | 20.5% | 34.1% | 31.8% | 40.9% | 52.3%
Dark Humor Detection | 46.3% | 48.8% | 51.3% | 48.8% | 46.3%
Date Understanding | 15.5% | 21.1% | 12.2% | 11.4% | 16.8%
Disambiguation QA | 39.5% | 44.6% | 44.2% | 46.9% | 43.0%
Empirical Judgments | 38.4% | 52.5% | 56.6% | 50.5% | 54.6%
English Proverbs | 26.5% | 20.6% | 26.5% | 26.5% | 17.7%
Entailed Polarity | 87.8% | 88.5% | 89.2% | 89.2% | 85.8%
Epistemic Reasoning | 43.4% | 43.5% | 61.2% | 40.1% | 53.0%
Evaluating Information Essentiality | 32.4% | 19.1% | 29.4% | 25.0% | 22.1%
Fantasy Reasoning | 67.7% | 69.2% | 65.2% | 66.7% | 52.7%
Figure of Speech Detection | 10.2% | 13.6% | 22.0% | 13.6% | 15.3%
General Knowledge | 51.4% | 78.6% | 80.0% | 68.6% | 74.3%
GRE Reading Comprehension | 6.5% | 12.9% | 22.6% | 16.1% | 35.5%
Hindu Knowledge | 32.6% | 42.3% | 48.6% | 36.6% | 49.7%
Human Organs Senses | 45.2% | 57.1% | 59.5% | 71.4% | 73.8%
Identify Odd Metaphor | 27.7% | 21.3% | 19.2% | 19.2% | 27.7%
Implicatures | 44.3% | 49.6% | 53.7% | 59.4% | 69.9%
Implicit Relations | 22.4% | 35.3% | 28.2% | 16.5% | 25.9%
Intent Recognition | 66.2% | 79.2% | 89.5% | 87.8% | 89.5%
Irony Identification | 50.5% | 49.5% | 63.6% | 60.6% | 59.6%
Known Unknowns | 50.0% | 52.2% | 50.0% | 50.0% | 41.3%
Logic Grid Puzzle | 32.7% | 31.6% | 31.1% | 35.8% | 39.4%
Logical Args | 18.8% | 34.4% | 25.0% | 34.4% | 43.8%
Logical Fallacy Detection | 50.9% | 54.9% | 54.5% | 54.1% | 55.1%
Logical Sequence | 38.5% | 46.2% | 30.8% | 25.6% | 43.6%
Mathematical Induction | 60.9% | 55.1% | 52.2% | 44.9% | 58.0%
Metaphor Boolean | 51.1% | 57.5% | 61.5% | 63.4% | 49.1%
Misconceptions | 56.1% | 57.5% | 54.8% | 51.6% | 58.0%
Moral Permissibility | 50.6% | 54.4% | 57.0% | 52.3% | 49.7%
Movie Recommendation | 6.4% | 52.6% | 49.4% | 31.6% | 36.8%
Navigate | 49.3% | 49.8% | 51.1% | 50.9% | 51.8%
Nonsense Words Grammar | 28.0% | 46.0% | 48.0% | 38.0% | 48.0%
Novel Concepts | 9.4% | 12.5% | 15.6% | 6.3% | 9.4%
Odd One Out | 30.2% | 26.7% | 22.1% | 12.8% | 19.8%
Penguins in a Table | 29.5% | 32.9% | 28.2% | 40.9% | 36.9%
Phrase Relatedness | 45.0% | 51.0% | 55.0% | 53.0% | 64.0%
Physical Intuition | 39.5% | 42.0% | 37.0% | 55.6% | 58.0%
Physics | 39.3% | 42.8% | 54.2% | 55.9% | 65.5%
Presuppositions as NLI | 36.6% | 36.2% | 39.6% | 34.0% | 28.0%
Question Selection | 39.8% | 42.1% | 5.2% | 41.1% | 42.7%
Reasoning about Colored Objects | 33.9% | 38.7% | 40.5% | 45.8% | 55.0%
Riddle Sense | 40.8% | 57.1% | 44.9% | 46.9% | 42.9%
Ruin Names | 19.4% | 20.8% | 12.5% | 24.1% | 33.0%
Sentence Ambiguity | 63.3% | 60.0% | 65.0% | 60.0% | 66.7%
Similarities Abstraction | 21.1% | 22.4% | 27.6% | 21.1% | 13.2%
Snarks | 42.0% | 41.4% | 47.0% | 48.1% | 48.6%
Sports Understanding | 50.0% | 48.8% | 54.5% | 52.0% | 51.8%
StrategyQA | 56.1% | 58.5% | 57.1% | 53.9% | 53.7%
Temporal Sequences | 31.4% | 28.4% | 20.5% | 26.4% | 21.2%
Timedial | 15.3% | 22.2% | 24.4% | 39.9% | 40.8%
Understanding Fables | 20.1% | 19.6% | 24.9% | 28.0% | 20.1%
Winowhy | 37.2% | 39.7% | 38.0% | 56.5% | 56.4%
Average (weighted) | 39.6% | 43.4% | 42.6% | 46.6% | 48.7%
Average (unweighted) | 32.8% | 42.7% | 42.2% | 42.7% | 45.3%
Table 29: BIG-bench Results. Galactica exceeds the performance of general
models, even at lower scales.

###  A.6 Prompt Pre-training Datasets

We report the prompt datasets we included in pre-training below.

Data source | Split | Prompts | Tokens
---|---|---|---
MedMCQA (Pal et al., 2022) | train | 180,894 | 13,311,290
RACE (Xu et al., 2017) | train | 29,502 | 12,160,390
Quoref (Dasigi et al., 2019) | train | 19,206 | 10,361,335
ROPES (Lin et al., 2019) | train | 10,815 | 2,672,195
BioASQ7 task b (Nentidis et al., 2021) | train | 2,676 | 1,288,462
TQA (Kembhavi et al., 2017) | train | 8,566 | 1,856,473
BoolQ (Clark et al., 2019) | train | 9,333 | 1,224,335
SciQ (Welbl et al., 2017) | train | 10,346 | 1,397,668
QASC (Khot et al., 2020) | train | 8,053 | 930,414
CommonSenseQA (Talmor et al., 2018) | train | 9,644 | 660,750
OpenBookQA (Mihaylov et al., 2018) | train | 4,908 | 324,995
QCScience (V et al., 2021) | train | 2,417 | 209,803
PubMedQA (Jin et al., 2019) | train | 495 | 186,304
QASPER (Dasigi et al., 2021) | train | 606 | 105,985
UChallenge (new) | train | 346 | 29,308
TrueOrFalse (new) | train | 107 | 2,854
Table 30: Question answering prompts used in Naturebook Data source | Split | Prompts | Tokens
---|---|---|---
JNLPBA (Kim et al., 2004) | train | 91,213 | 5,262,723
BC4CHEMD (Krallinger et al., 2004) | train | 30,234 | 1,756,929
ChemProt (Taboureau et al., 2011) | train | 3,030 | 1,286,816
BC2GM (Smith et al., 2008) | train | 12,375 | 704,357
S800 (Pafilis et al., 2013) | train | 5,318 | 281,448
BC5CDR Chem (Li et al., 2016) | train | 4,503 | 241,729
BC5CDR Disease (Li et al., 2016) | train | 4,498 | 231,322
MethodNet (new) | train | 659 | 167,904
Scientific Entities (new) | train | 305 | 97,935
Table 31: Entity extraction prompts used in Naturebook Data source | Split | Prompts | Tokens
---|---|---|---
PWC Desc (new) | train | 3,586 | 9,663,419
SciTail (Khot et al., 2018) | train | 23,361 | 1,383,614
Fragmented Glass (new) | train | 718 | 867,985
SciTLDR (Cachola et al., 2020) | train | 1,973 | 472,169
Table 32: Summarization prompts used in Naturebook Data source | Split | Prompts | Tokens
---|---|---|---
Wizard of Wikipedia (Dinan et al., 2018) | train | 18,246 | 4,466,113
Advising (Gunasekara et al., 2019) | train | 495 | 147,793
Table 33: Dialog prompts used in Naturebook Data source | Split | Prompts | Tokens
---|---|---|---
BACE Classification | train | 1,198 | 122,699
BACE Regression | train | 1,198 | 154,656
BBBP | train | 1,613 | 115,916
ClinTox | train | 1,171 | 100,955
Delaney | train | 893 | 62,083
FreeSolv | train | 508 | 29,542
HIV | train | 32,572 | 2,308,966
HOPV | train | 2,217 | 333,620
Lipo | train | 3,327 | 362,342
PCBA | train | 714,277 | 553,645,656
QM7 | train | 5,416 | 320,199
QM8 | train | 275,569 | 27,163,516
QM9 | train | 1,259,090 | 128,427,073
SAMPL | train | 508 | 1,259,090
SIDER | train | 30,499 | 2,741,904
Thermosol | train | 1,396 | 139,481
Tox21 | train | 73,883 | 54,224,093
Table 34: Chemical property prediction prompts used in Naturebook

####  A.6.1 Chemical Property Prediction

We set up a prediction task for chemical and physical properties with our
validation set of 17,052 compounds. We use the PubChem document structure to
design a prompt. We show an example for XLogP in Figure 24.

Canonical SMILES `[START_SMILES]CC(=O)OC1=CC=CC=C1C(=O)O[END_SMILES]`
Computed Properties | `Property Name` | `Property Value` | `XLogP3-AA Log P` | Figure 24: Chemical Property Prompt. We design a prompt based on the PubChem document format. Using this prompt style, we test the model’s ability to learn chemical and physical properties from the SMILES sequence.

We report results in Table 35. The error decreases fairly smoothly with scale,
suggesting self-supervised learning is occurring within-document from SMILES
towards the chemical and physical properties. But it tails off for 120B which
suggests more molecule data might be needed.

Chemical and Physical Property Prediction
---
Model | Param (bn) | Mol. Weight | XLogP | Rotatable Bond # | Topological PSA
GAL 125M | 0.1 | 101.43 | 1.638 | 4.389 | 36.63
GAL 1.3B | 1.3 | 101.05 | 1.413 | 3.930 | 41.11
GAL 6.7B | 6.7 | 81.76 | 1.197 | 2.932 | 30.01
GAL 30B | 30 | 77.46 | 1.101 | 3.534 | 29.54
GAL 120B | 120 | 86.57 | 1.131 | 3.474 | 28.84
Table 35: Chemical and physical property prediction. All results reported as
RMSE. Prediction error generally decreases with scale, indicating Galactica
can infer properties from SMILES.

####  A.6.2 Docking Regression

We looked briefly at the docking score regression task (García-Ortegón et al.,
2022). Here the task is to predict a docking score based on an target and a
ligand. In the case of Galactica, we use a text format to represent this
information. An example is shown in Figure 25. We report results in Table 36.

`[START_AMINO]MLEICLKLVGCKSKKGLSSSSSCYLEEALQRPVASDFEPQGLSEAARWNSKE...[END_AMINO]`
`[START_I_SMILES]O1[C@@H]([C@@H](O)[C@@H](O)[C@@H]1N2C(=O)NC(=O)C=C2)...[END_I_SMILES]`
Question: What will be the docking score of this compound against the protein?
Answer: -8.8 Figure 25: DockSTRING Format. To construct the training set, we
take the protein target and ligand sequences, pose a natural language
question, and have the docking score as the answer. Docking Regression
---
Model | Param (bn) | ESR2 | F2 | KIT | PARP1 | PGR
GAL 125M | 0.1 | -12.4 | -6.09 | -6.73 | -1.69 | -12.4
GAL 1.3B | 1.3 | -0.293 | 0.591 | 0.063 | 0.728 | -1.72
GAL 6.7B | 6.7 | -0.216 | 0.694 | 0.290 | 0.681 | -0.894
GAL 30B | 30 | -0.186 | 0.679 | 0.313 | 0.732 | -0.468
GAL 120B | 120 | -0.564 | 0.626 | 0.249 | 0.732 | -0.960
Table 36: DockSTRING Results. Metric shown is $R^{2}$.

For three of the targets, Galactica is able to infer from looking at the
sequences alone, and performance scales from 1.3B parameters onwards. However,
Galactica does not solve the two harder targets ESR2 and PGR. This hints at a
limitation with the text representation, and may point to more geometrical
information being needed to solve the task with reasonable data-efficiency.

####  A.6.3 Rest of MMLU

We report social sciences and results for other fields below:

Subject | OPT | BLOOM | Gopher | Chinchilla | GAL 30B | GAL 120B
---|---|---|---|---|---|---
Anatomy | 28.9% | 37.0% | 56.3% | 70.4% | 54.1% | 58.5%
Business Ethics | 31.0% | 36.0% | 70.0% | 72.0% | 42.0% | 48.0%
Clinical Knowledge | 21.9% | 29.8% | 67.2% | 75.1% | 57.7% | 59.2%
Computer Security | 32.0% | 34.0% | 65.0% | 76.0% | 65.0% | 67.0%
Conceptual Physics | 34.9% | 36.6% | 49.4% | 67.2% | 43.4% | 50.6%
Global Facts | 23.0% | 32.0% | 38.0% | 39.0% | 32.0% | 35.0%
High School European History | 6.7% | 4.8% | 72.1% | 78.8% | 60.6% | 67.3%
High School Geography | 26.3% | 38.9% | 76.8% | 86.4% | 58.1% | 63.6%
High School Gov. & Politics | 32.6% | 30.6% | 83.9% | 91.2% | 58.5% | 61.7%
High School Macroeconomics | 36.2% | 23.1% | 65.1% | 70.5% | 40.5% | 46.4%
High School Microeconomics | 32.8% | 27.3% | 66.4% | 77.7% | 49.2% | 55.9%
High School Psychology | 25.5% | 36.9% | 81.8% | 86.6% | 68.8% | 74.3%
High School US History | 9.3% | 11.8% | 78.9% | 83.3% | 51.5% | 58.3%
High School World History | 30.0% | 29.1% | 75.1% | 85.2% | 63.7% | 71.7%
Human Aging | 35.0% | 34.5% | 66.4% | 77.6% | 55.2% | 59.2%
Human Sexuality | 26.0% | 33.6% | 67.2% | 86.3% | 56.5% | 58.8%
International Law | 33.1% | 41.3% | 77.7% | 90.9% | 64.4% | 71.1%
Jurisprudence | 0.0% | 0.0% | 71.3% | 79.6% | 47.2% | 53.7%
Logical Fallacies | 28.2% | 28.2% | 72.4% | 80.4% | 47.2% | 59.5%
Management | 25.2% | 27.2% | 77.7% | 82.5% | 60.2% | 63.1%
Marketing | 32.5% | 41.0% | 83.3% | 89.7% | 70.5% | 76.5%
Miscellaneous | 31.5% | 37.7% | 75.7% | 84.5% | 54.0% | 63.9%
Moral Disputes | 28.2% | 32.7% | 66.8% | 77.5% | 50.3% | 56.6%
Moral Scenarios | 25.4% | 24.4% | 40.2% | 36.5% | 24.1% | 24.2%
Nutrition | 30.4% | 32.4% | 69.9% | 77.1% | 63.1% | 67.3%
Philosophy | 29.9% | 31.5% | 68.8% | 79.4% | 52.4% | 54.7%
Prehistory | 36.7% | 36.1% | 67.6% | 81.2% | 52.2% | 59.6%
Professional Accounting | 29.8% | 28.7% | 44.3% | 52.1% | 31.2% | 40.0%
Professional Law | 30.3% | 25.5% | 44.5% | 56.5% | 34.6% | 36.0%
Professional Medicine | 27.9% | 25.4% | 64.0% | 75.4% | 52.2% | 59.6%
Professional Psychology | 32.7% | 33.3% | 68.1% | 75.7% | 50.5% | 56.5%
Public Relations | 34.5% | 30.0% | 71.8% | 73.6% | 44.5% | 53.6%
Security Studies | 35.1% | 29.8% | 64.9% | 75.9% | 46.5% | 57.1%
Sociology | 26.4% | 29.9% | 84.1% | 91.0% | 65.7% | 72.6%
US Foreign Policy | 44.0% | 37.0% | 81.0% | 92.0% | 64.0% | 75.0%
Virology | 30.7% | 28.3% | 47.0% | 53.6% | 44.6% | 48.2%
World Religion | 43.9% | 41.5% | 84.2% | 87.7% | 44.4% | 64.9%
Table 37: Rest of MMLU. The corpus delta effects are more evidence with non-
STEM subjects in particular, where Galactica lags the performance of
Chinchilla and Gopher.

###  A.7 Further Training Dataset Details

####  A.7.1 FragmentedGlass

We compile a list of scientific entities, retrieve fragments for each one, and
write a description of the entity based on the retrieved fragments. This can
be considered a summarization task. We also write ground-truth descriptions
without any retrieved fragments.

####  A.7.2 MethodNet

We compile machine learning abstracts and predict the new method that was
introduced in the paper.

####  A.7.3 PWC Desc

For a list of dataset and methods in machine learning, we retrieve fragments
for each one from the introducing paper, and write a summary description based
on the retrieved fragments.

####  A.7.4 Ribosome

We use Expasy666https://web.expasy.org/translate/ to create a paired
translation set between nucleotide sequences from the protein coding part of
the human genome and protein sequences.

####  A.7.5 S2

Papers from certain fields are ignored due to quality concerns: psychology,
business, art, economics, geography, history, political science, philosophy
and sociology. Papers from journals with words like "law", "history",
"politics", "business", "religion" were also ignored. For S2, we also exclude
papers from low impact journals. The approximate impact factor of each journal
in the S2 dataset was computed, by counting the number of papers in that
journal and the number of citations that these papers received. If the
approximate impact factor $<1$, the papers from that journal are ignored. Non-
English papers are ignored. Some of these constraints can likely be relaxed in
future work.

####  A.7.6 ScientificEntities

For a random sample of academic paper abstracts, we predict the scientific
entities that were mentioned in the abstract.

####  A.7.7 StackExchange

We include question and answers from the following sources: academic, ai,
arduino, astronomy, aviation, bioinformatics, biology, chemistry, chess,
cogsci, computergraphics, cs, cseducators, cstheory, datascience, dsp,
earthscience, economics, electronics, engineering, hardwarerecs, health, hsm,
math, matheducators, mathematica, mathoverflow, /mechanics,
networkengineering, or, physics, puzzling, quant, quantumcomputing,
retrocomputing, reverseengineering, robotics, scicomp, softwareengineering,
softwarerecs, sound, space, stats.

####  A.7.8 TrueOrFalse

We include 107 True or False questions to improve zero-shot performance for
this type of question.

####  A.7.9 UChallenge

We include 346 free-form question and answers of university-level questions
about science; this is a form of closed-book QA (and not multiple-choice).

###  A.8 Evaluation Dataset Examples

####  A.8.1 AminoProbe

Prompt Question: Does peptide bond cleavage occur on the carbonyl side or the
amino side for trypsin? Answer: carbonyl

####  A.8.2 Galaxy Clusters

Prompt Abell 370 is a galaxy cluster located in the constellation of Correct
Completion: Cetus

####  A.8.3 Mineral Groups

Prompt Fayalite is a silicate mineral from the major group Correct Completion:
Nesosilicates

####  A.8.4 Deduplication Results

One of our concerns from reading the literature was the lack of data leakage
analysis for results on MMLU, given the massive corpuses being used. Following
from previous work of Brown et al. (2020), we search for n-gram matches
between the training and test set. We chose to remove any 13-gram matches from
the test set that appear in the training set and we report the scores before
and after removal of these clashing examples. Results are shown overleaf.

| score_before | score_after | count_before | count_after
---|---|---|---|---
abstract_algebra | 33.0% | 32.32% | 100 | 99
anatomy | 58.52% | 58.95% | 135 | 134
astronomy | 65.13% | 64.67% | 152 | 150
business_ethics | 48.0% | 48.0% | 100 | 100
clinical_knowledge | 59.24% | 59.24% | 265 | 265
college_biology | 68.75% | 69.23% | 144 | 143
college_chemistry | 46.0% | 46.46% | 100 | 99
college_computer_science | 49.0% | 48.98% | 100 | 98
college_mathematics | 43.0% | 45.26% | 100 | 95
college_medicine | 57.23% | 57.74% | 173 | 168
college_physics | 42.16% | 42.27% | 102 | 97
computer_security | 67.0% | 67.35% | 100 | 98
conceptual_physics | 50.64% | 50.85% | 235 | 234
econometrics | 42.11% | 42.11% | 114 | 114
electrical_engineering | 62.76% | 62.76% | 145 | 145
elementary_mathematics | 38.10% | 38.10% | 378 | 378
formal_logic | 32.54% | 32.54% | 126 | 126
global_facts | 35.0% | 35.05% | 100 | 97
high_school_biology | 69.35% | 69.61% | 310 | 306
high_school_chemistry | 47.78% | 47.78% | 203 | 203
high_school_computer_science | 70.0% | 70.0% | 100 | 100
high_school_european_history | 67.27% | 66.17% | 165 | 133
high_school_geography | 63.63% | 63.63% | 198 | 198
high_school_government_and_politics | 61.66% | 61.46% | 193 | 192
high_school_macroeconomics | 46.41% | 46.53% | 390 | 389
high_school_mathematics | 32.59% | 32.58% | 270 | 267
high_school_microeconomics | 55.88% | 55.88% | 238 | 238
high_school_physics | 33.77% | 33.77% | 151 | 151
high_school_psychology | 74.31% | 74.26% | 545 | 544
high_school_statistics | 41.20% | 41.20% | 216 | 216
high_school_us_history | 58.33% | 58.59% | 204 | 99
high_school_world_history | 71.73% | 72.04% | 237 | 186
human_aging | 59.19% | 59.19% | 223 | 223
human_sexuality | 58.78% | 58.78% | 131 | 131
international_law | 71.07% | 71.07% | 121 | 121
jurisprudence | 53.70% | 53.70% | 108 | 108
logical_fallacies | 59.51% | 59.26% | 163 | 162
machine_learning | 38.39% | 36.54% | 112 | 104
management | 63.11% | 63.11% | 103 | 103
marketing | 76.50% | 76.50% | 234 | 234
medical_genetics | 68.0% | 67.68% | 100 | 99
miscellaneous | 63.86% | 63.81% | 783 | 782
moral_disputes | 56.65% | 56.52% | 346 | 345
moral_scenarios | 24.24% | 24.24% | 895 | 895
nutrition | 67.32% | 67.32% | 306 | 306
philosophy | 54.66% | 54.52% | 311 | 310
prehistory | 59.57% | 59.57% | 324 | 324
professional_accounting | 40.07% | 39.86% | 282 | 281
professional_law | 36.05% | 36.05% | 1534 | 1534
professional_medicine | 59.56% | 59.63% | 272 | 270
professional_psychology | 56.54% | 56.54% | 612 | 612
public_relations | 53.63% | 53.63% | 110 | 110
security_studies | 57.14% | 57.14% | 245 | 245
sociology | 72.64% | 73.0% | 201 | 200
us_foreign_policy | 75.0% | 75.76% | 100 | 99
virology | 48.19% | 48.48% | 166 | 165
world_religions | 64.91% | 64.91% | 171 | 171

####  A.8.5 Example Wikipedia Article Written by Galactica

Multi-Head Attention Multi-Head Attention is an attention mechanism used in
the Transformer architecture for neural networks (Vaswani et al., 2017).
Background In natural language processing, an attention mechanism is a model
that maps a query and a set of key-value pairs to an output, where the query,
keys, values, and output are all vectors. The output is computed as a weighted
sum of the values, where the weight assigned to each value is computed by a
compatibility function of the query with the corresponding key (Vaswani et
al., 2017). The scaled dot-product attention is an attention mechanism used in
the Transformer architecture. It is defined as mapping a query and a set of
key-value pairs to an output, where the query, keys, values, and output are
all vectors. The output is computed as a weighted sum of the values, where the
weight assigned to each value is computed by a compatibility function of the
query with the corresponding key (Vaswani et al., 2017). The multi-head
attention is an extension of the scaled dot-product attention. The multi-head
attention mechanism is defined as several scaled dot-product attention stacked
together, where the input is linearly projected h times with different,
learned linear projections to $d_{k}$, $d_{k}$, and $d_{v}$ dimensions,
respectively (Vaswani et al., 2017). Formal definition Let the query matrix be
$Q$, the key matrix be $K$, and the value matrix be $V$. The matrices are of
shape $d_{k}\times d_{q}$, $d_{k}\times d_{k}$, and $d_{v}\times d_{k}$
respectively. The multi-head attention mechanism is defined as:
$\text{MultiHead}\left(Q,K,V\right)=\text{Concat}\left(\text{head}_{1},\dots,\text{head}_{h}\right)W^{O}$
where:
$\text{head}_{i}=\text{Attention}\left(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V}\right)$
where $W_{i}^{Q}$, $W_{i}^{K}$, and $W_{i}^{V}$ are the learned linear
projections to the $d_{k}$, $d_{k}$, and $d_{v}$ dimensions, respectively.
$W^{O}$ is the linear projection from $d_{v}h$ to $d_{v}$ dimensions (Vaswani
et al., 2017). Applications Multi-head attention is widely used in natural
language processing and other domains. For example, in the BERT model, multi-
head attention is used to learn the context-aware representation of the input
sequence (Devlin et al., 2019). References 1\. Vaswani, Ashish, Noam Shazeer,
Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and
Illia Polosukhin. “Attention Is All You Need.” ArXiv abs/1706.03762 (2017).
2\. Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. “BERT:
Pre-Training of Deep Bidirectional Transformers for Language Understanding.”
ArXiv abs/1810.04805 (2019).

####  A.8.6 Example Literature Survey Written by Galactica

Self-Supervised Learning, A Survey Abstract: In this paper we review self-
supervised learning, a method of learning features from unlabelled data. We
discuss the historical motivation behind self-supervised learning and review
some of the current state of the art methods. We also look at how self-
supervised learning has been used to solve a wide range of problems, including
image classification, object detection, video analysis and robotics. Finally
we discuss some of the challenges and future research directions for self-
supervised learning. 1 Introduction Deep learning has been very successful at
solving many problems in machine learning, however these methods are reliant
on large amounts of labelled data. In many real world applications, it is very
expensive or impossible to obtain large amounts of labelled data. Self-
supervised learning has emerged as a way of overcoming this issue. Self-
supervised learning is the process of learning a task that can be
automatically generated from unlabelled data. In this way, the data is
automatically labelled by exploiting some form of structure in the data. Once
the self-supervised task has been solved, the features learned by the network
are used to solve the original task. In this paper we will discuss the
historical motivation behind self-supervised learning and review some of the
current state of the art methods. We will look at how self-supervised learning
has been used to solve a wide range of problems, including image
classification, object detection, video analysis and robotics. Finally we will
discuss some of the challenges and future research directions for self-
supervised learning. 2 Historical Motivation The first wave of deep learning
was dominated by supervised learning methods. These methods require large
amounts of labelled data, which is very expensive to obtain in many domains.
To overcome this issue, unsupervised learning methods were introduced. These
methods don’t require labelled data, and so can be applied to a wide range of
domains. The downside of these methods is that the learned features are not as
good as those learned by supervised learning methods. To bridge the gap
between supervised and unsupervised learning, self-supervised learning was
introduced. The idea is to learn a supervised task from unlabelled data. Once
this task has been learned, the features learned can be transferred to other
tasks. The hope is features learned by self-supervised learning will be better
than those learned by unsupervised learning, and that self-supervised learning
can be applied to a wide range of domains. The first method of self-supervised
learning was autoencoders (Hinton et al., 1993). Autoencoders are a type of
neural network that learn to reconstruct the input. This can be thought of as
learning a self-supervised task of predicting the input from the input. The
features learned by autoencoders have been shown to be useful for solving
other tasks (Hinton et al., 2006). Another method of self-supervised learning
is multi-task learning (Caruana, 2017). In multi-task learning, multiple tasks
are learned simultaneously. The idea is that by learning multiple tasks, the
network will learn better features. In this way, multi-task learning can be
thought of as a form of self-supervised learning. 3 Learning Methods (cont)
3.1 Pretext Tasks The first method of self-supervised learning that we will
discuss is the use of pretext tasks. In this method, a supervised task is
learned from unlabelled data. This task is called a pretext task, and the
labels for this task are automatically generated from the data. Once the
pretext task has been learned, the features learned by the network are used to
solve the original task. (cont)

####  A.8.7 Example Lecture Notes Written by Galactica

Lecture 1: Hartree-Fock Theory Introduction In this lecture, we will consider
the simplest quantum mechanical method for approximating the ground state
energy of a many-electron system. The Hartree-Fock method is also known as the
self-consistent field method, and is an approximation to the exact many-
electron wavefunction known as the Slater determinant. The Slater determinant
is an anti-symmetric product of one-electron functions (orbitals), and is the
simplest wavefunction that satisfies the Pauli exclusion principle. The
Hartree-Fock method is an iterative method for finding the best possible set
of orbitals that minimizes the energy of the Slater determinant. In the
Hartree-Fock method, the electron-electron repulsion energy is approximated as
a classical Coulomb interaction between two charge distributions, one for each
electron. This approximation neglects the correlation between the electrons,
which is an important part of the total electron-electron repulsion energy.
The resulting method is a mean-field theory in which each electron moves in an
average field due to the other electrons. The resulting equations are known as
the Hartree-Fock equations, and can be solved self-consistently to obtain the
best possible orbitals. The resulting ground state energy is known as the
Hartree-Fock energy, and the Hartree-Fock wavefunction is a single Slater
determinant built from these orbitals. The Hartree-Fock method is also known
as the Self-Consistent Field (SCF) method, because the equations for the
orbitals are solved self-consistently. The Hartree-Fock equations can be
derived variationally by minimizing the energy of the Slater determinant with
respect to the orbitals. The resulting orbitals are known as the canonical
Hartree-Fock orbitals, and are not necessarily localized in space. The
Hartree-Fock energy is invariant to unitary transformations of the canonical
orbitals, and therefore there are an infinite number of orbitals that yield
the same Hartree-Fock energy. These orbitals are known as non-canonical
orbitals, and can be localized in space by appropriate unitary
transformations. Single-Electron Approximation In this section, we will review
the basics of quantum mechanics for a single particle. This is useful for
understanding the single-electron approximation used in Hartree-Fock theory.
The time-independent Schrödinger equation for a particle in a potential $V(r)$
is given by: $\bar{H}\psi(r)=E\psi(r)$ where the Hamiltonian is
$\bar{H}=-\frac{\hbar}{2m}\nabla^{2}+V(r)$ The time-independent Schrödinger
equation is an eigenvalue equation for the Hamiltonian operator, where the
eigenvalues are the allowed energies of the system. The Hamiltonian is a sum
of two operators, one corresponding to the kinetic energy of the particle, and
the other corresponding to the potential energy. The potential energy operator
acts on the wavefunction by multiplying by the potential $V(r)$. The kinetic
energy operator is the Laplacian operator $\nabla^{2}$, which is the
divergence of the gradient of the wavefunction. The Laplacian operator is a
second derivative with respect to the position of the particle. (cont)

####  A.8.8 I’m sorry Frank, I think you missed it

If AI is going to help us explore the universe, we need it to have basic chess
abilities to alleviate boredom - given the impossibility of faster-than-light
travel.

The BIG-bench task suite of Srivastava et al. (2022) has a benchmark for
checkmate-in-one detection. For fun, we made a dataset of 20,000 public chess
games and converted them to ASCII chess using the python-chess
library777https://python-chess.readthedocs.io/en/latest/. We included 19,426
games in our pre-training corpus (rest for validation). We also recorded the
ELO ratings of players. An example document looks like below:

# A Chess Game ## Player Information White ELO: 2286 Black ELO: 2586 ## The
Game Begins r n b q k b n r p p p p p p p p . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . P P P P P P P P R N B Q K B N R White (ELO:
2286) plays e4 r n b q k b n r p p p p p p p p . . . . . . . . . . . . . . . .
. . . . P . . . . . . . . . . . P P P P . P P P R N B Q K B N R (cont)

For evaluation, we converted the checkmate-in-one boards to ASCII and prompted
for a move. Results are shown below.

Model | Accuracy
---|---
GAL 125M | 0.54%
GAL 1.3B | 0.43%
GAL 6.7B | 1.77%
GAL 30B | 1.29%
GAL 120B | 3.03%
Table 38: Checkmate-in-one Results. Metric shown is Accuracy.

While this represents the state-of-the-art over other large language
models888https://github.com/google/BIG-
bench/tree/main/bigbench/benchmark_tasks/checkmate_in_one, it is clear that
more work is needed on this problem.


# Autoencoders, Kernels, and Multilayer Perceptrons for Electron Micrograph
Restoration and Compression

Jeffrey M. Ede

<EMAIL_ADDRESS>

Abstract: We present 14 autoencoders, 15 kernels and 14 multilayer perceptrons
for electron micrograph restoration and compression. These have been trained
for transmission electron microscopy (TEM), scanning transmission electron
microscopy (STEM) and for both (TEM+STEM). TEM autoencoders have been trained
for 1$\times$, 4$\times$, 16$\times$ and 64$\times$ compression, STEM
autoencoders for 1$\times$, 4$\times$ and 16$\times$ compression and TEM+STEM
autoencoders for 1$\times$, 2$\times$, 4$\times$, 8$\times$, 16$\times$,
32$\times$ and 64$\times$ compression. Kernels and multilayer perceptrons have
been trained to approximate the denoising effect of the 4$\times$ compression
autoencoders. Kernels for input sizes of 3, 5, 7, 11 and 15 have been fitted
for TEM, STEM and TEM+STEM. TEM multilayer perceptrons have been trained with
1 hidden layer for input sizes of 3, 5 and 7 and with 2 hidden layers for
input sizes of 5 and 7. STEM multilayer perceptrons have been trained with 1
hidden layer for input sizes of 3, 5 and 7. TEM+STEM multilayer perceptrons
have been trained with 1 hidden layer for input sizes of 3, 5, 7 and 11 and
with 2 hidden layers for input sizes of 3 and 7. Our code, example usage and
pre-trained models are available at https://github.com/Jeffrey-Ede/Denoising-
Kernels-MLPs-Autoencoders.

Keywords: machine learning, data compression, electron microscopy, noise
removal

##  1 Introduction

Many imaging modes in electron microscopy are limited by noise or generate
large amounts of data that needs to be compressed. Nevertheless, there has
been little development of electron microscopy specific denoising
autoencoders, kernels and multilayer perceptrons (MLPs) for image restoration
and compression.

In this paper, we present a process to train noise removal kernels and MLPs
for specific or general electron microscopy imaging modes. This is done by
training denoising autoencoders for domains of interest and then training
kernels and MLPs to approximate them. This has a number of advantages over
just using an autoencoder, including:

Figure 1:  Denoising autoencoder with a 20$\times$20$\times$$x$ latent space. | Latent Depth, $x$
---|---
Dataset |  1 |  2 |  4 |  8 |  16 |  32 |  64
TEM |  ✓ |  ✗ |  ✓ |  ✗ |  ✓ |  ✗ |  ✓
STEM |  ✗ |  ✗ |  ✓ |  ✗ |  ✓ |  ✗ |  ✓
TEM+STEM |  ✓ |  ✓ |  ✓ |  ✓ |  ✓ |  ✓ |  ✓
Table 1:  Autoencoders with 20$\times$20$\times$$x$ latent spaces trained for
TEM, STEM and TEM+STEM.

  * •

Lower susceptibility to overfitting as a result of lower capacities.

  * •

Higher computational efficiency.

  * •

Improved resistance to checkerboard artefacts[1].

We have used our process to train autoencoders, kernels and MLPs for
transmission electron microscopy (TEM), scanning transmission electron
microscopy (STEM) and for both modes together (TEM+STEM). To be clear, in this
paper TEM refers to all electron micrographs created through transmission;
rather than rastering, and includes diffraction patterns.

Altogether, we present 14 autoencoders, 15 kernels and 14 MLPs for TEM, STEM
and TEM+STEM restoration and compression. Autoencoders are presented in
section 2. The kernels and MLPs trained to approximate 4$\times$ compression
autoencoders are presented in section 3. Performances are compared in section
4. Finally, some of our kernels and example applications of each of the
denoising autoencoders, kernels and MLPs are provided as an appendix.

##  2 Autoencoders

We have trained denoising autoencoders for TEM, STEM and TEM+STEM restoration
and compression. These are simple convolution banks that compress images into
latent spaces that are a fraction of their original size. These latent spaces
can be stored as compressed data or decoded with a denoising effect.

Figure 2:  Autoencoder MSE learning curves for TEM, STEM and TEM+STEM. TEM
MSEs have the lowest mean and variance, STEM MSEs have the highest mean and
itermediate variance and TEM+STEM MSEs means are intermediate and have the
highest variance. Learning curves are moving averages of 5000 batches.

###  2.1 Architecture

Our autoencoder architecture is shown in fig. 1. Each convolutional layer is
followed by batch normalization, then ReLU[2] activation. This is except for
the last 2 layers: the second to last layer is not batch normalized and the
last layer has no weights, biases, batch normalization or activation.

All our autoencoders encode 20$\times$20$\times$$x$ latent spaces, with 64:$x$
compression ratios. Autoencoders trained for TEM, STEM and TEM+STEM with
different latent depths are tabulated in table 1.

###  2.2 Learning Policy

In this subsection, we present our autoencoder training hyperparameters and
learning protocol for the learning curves shown in fig. 2.

Loss metric: Our autoencoders were trained to minimize the Huberised[3] mean
squared error (MSE), $L_{\mathrm{auto}}$, between their outputs and inputs:

$L_{\mathrm{auto}}=\begin{cases}\textit{MSE},&\textit{MSE}<1.0\\\
\textit{MSE}^{\frac{1}{2}},&\textit{MSE}\geq 1.0\end{cases}$ (1)

The loss is Huberized to prevent autoencoders from being too disturbed by
large errors in the early stages of training. Otherwise, it did not
significantly affect training as most MSEs were of the order $10^{-1}$ or
less.

To make all the 160$\times$160 micrograph crops in our training datasets of
similar importance, their minima were subtracted and they were divided by
their means. Dividing by means put differences between outputs and inputs on
the same scale so that MSEs would be a meaningful loss metric. Minima
subtraction was applied to increase the similarity of input intensity
distributions, making it easier for the autoencoders to learn. It also removes
any systematic intensity offsets that may be present as a result of flat
background subtractions being applied as part of some acquisitions.

Figure 3:  Architecture of kernels and MLPs with 1 and 2 hidden layers for an
input size, $w$. Kernels weight the input and then fully connect it to the
output. MLPs also have additional biases, sigmoid activation, a fully
connected layer and weights for each hidden layer. Every weights and biases
tensor has $w^{2}$ parameters, Fully connected layers connect to $w^{2}$
nodes, except for the last which connects to a single node.

Optimization: The ADAM solver was used with the parameters recommended in [4]
for 60000 batches. The learning rate was quadratically stepped down from
$\eta_{0}~{}=~{}0.01$ to
$\eta=(1-\textit{iter}/\textit{max\\_iter})^{2}\eta_{0}$ after every 5000
iterations.

Batch normalization: Batch normalization layers from [5] were trained with a
decay rate of 0.999 and batch size 32. They are used after every convolutional
layer before activation, except before and after the last.

Activation: Outputs of each layer are ReLU[2] activated, except for the last
layer which is unactivated.

Initialization: All weights were Xavier[6] initialized and all biases were
zero initialized. The last convolutional layer has no weights or biases.

TEM |  |  |  |  |
---|---|---|---|---|---
| Input Size
|  3 |  5 |  7 |  11 |  15
Kernel |  ✓ |  ✓ |  ✓ |  ✓ |  ✓
1 Hidden Layer |  ✓ |  ✓ |  ✓ |  ✗ |  ✗
2 Hidden Layers |  ✗ |  ✓ |  ✓ |  ✗ |  ✗
|  |  |  |  |
STEM |  |  |  |  |
| Input Size
|  3 |  5 |  7 |  11 |  15
Kernel |  ✓ |  ✓ |  ✓ |  ✓ |  ✓
1 Hidden Layer |  ✓ |  ✓ |  ✓ |  ✗ |  ✗
2 Hidden Layers |  ✗ |  ✗ |  ✗ |  ✗ |  ✗
|  |  |  |  |
TEM+STEM |  |  |  |  |
| Input Size
|  3 |  5 |  7 |  11 |  15
Kernel |  ✓ |  ✓ |  ✓ |  ✓ |  ✓
1 Hidden Layer |  ✓ |  ✓ |  ✓ |  ✓ |  ✗
2 Hidden Layers |  ✓ |  ✗ |  ✓ |  ✗ |  ✗
Table 2:  Kernels and MLPs trained to approximate latent depth 16 TEM, STEM
and TEM+STEM autoencoders.

##  3 Kernels and Multilayer Perceptrons

Figure 4:  Kernel and MLP learning curves for TEM, STEM and TEM+STEM. Larger
input sizes and more hidden layers are correlated with lower MSEs. TEM MSEs
are lowest, STEM MSEs are the highest and TEM+STEM MSEs are intermediate.
Similarities in the graphs are a result of sets of kernels and MLPs being
shown training data in the same order. Learning curves are moving averages of
500 iterations.

Our latent depth 16 autoencoders have been used to train TEM, STEM and
TEM+STEM kernels and MLPs. Many of these have higher computational
efficiencies than the autoencoders. Their lower capacities also make them less
susceptible to overfitting and checkerboard artefacts[1].

| TEM | STEM | TEM+STEM
---|---|---|---
|  Mean |  Std Dev |  Mean |  Std Dev |  Mean |  Std Dev
Autoencoder, Latent Depth 1 |  0.026422 |  0.000033 |  - |  - |  0.275182 |  0.002885
Autoencoder, Latent Depth 2 |  - |  - |  - |  - |  0.248378 |  0.003710
Autoencoder, Latent Depth 4 |  0.018571 |  0.000005 |  0.387636 |  0.001898 |  0.236593 |  0.003277
Autoencoder, Latent Depth 8 |  - |  - |  - |  - |  0.219717 |  0.003008
Autoencoder, Latent Depth 16 |  0.014360 |  0.000004 |  0.314286 |  0.002043 |  0.212957 |  0.003008
Autoencoder, Latent Depth 32 |  - |  - |  - |  - |  0.191258 |  0.001654
Autoencoder, Latent Depth 64 |  0.013034 |  0.000008 |  0.219523 |  0.001777 |  0.192866 |  0.001395
Kernel, Input Size 3 |  0.010822 |  0.000288 |  0.181633 |  0.002732 |  0.014224 |  0.000436
Kernel, Input Size 5 |  0.009728 |  0.000398 |  0.177021 |  0.002536 |  0.014071 |  0.000518
Kernel, Input Size 7 |  0.008032 |  0.000410 |  0.178340 |  0.002297 |  0.012350 |  0.000599
Kernel, Input Size 11 |  0.008217 |  0.000191 |  0.181634 |  0.002163 |  0.011783 |  0.000908
Kernel, Input Size 15 |  0.000929 |  0.000522 |  0.190376 |  0.003822 |  0.015055 |  0.001629
MLP, 1 Hidden Layer, Input Size 3 |  0.007167 |  0.000188 |  0.057006 |  0.000905 |  0.011494 |  0.000271
MLP, 1 Hidden Layer, Input Size 5 |  0.006934 |  0.000128 |  0.054407 |  0.001163 |  0.010921 |  0.000334
MLP, 1 Hidden Layer, Input Size 7 |  0.005917 |  0.000113 |  0.050175 |  0.001688 |  0.010340 |  0.000308
MLP, 1 Hidden Layer, Input Size 11 |  - |  - |  - |  - |  0.009965 |  0.000342
MLP, 2 Hidden Layers, Input Size 3 |  - |  - |  - |  - |  0.011027 |  0.000095
MLP, 2 Hidden Layers, Input Size 5 |  0.006590 |  0.000263 |  - |  - |  - |  -
MLP, 2 Hidden Layers, Input Size 7 |  0.006614 |  0.000247 |  - |  - |  0.009252 |  0.000292
Table 3:  Means and standard deviations of autoencoder, kernel and MLP MSEs in
their last 500 training batches for TEM, STEM and TEM+STEM. The highest
performing autoencoders and kernel or MLP are emboldened for each column.

###  3.1 Architecture

Our kernel and MLP architecture is shown in fig. 3. Kernels weight their input
then fully connect it to their outputs. MLPs also have additional biases,
sigmoid activation, a fully connected layer and weights for each hidden layer.
For a $w$$\times$$w$ input, every weights and biases tensor has $w^{2}$
parameters and hidden layers have $w^{2}$ nodes. Our fully trained kernels and
MLPs are summarized in table 2.

###  3.2 Learning Policy

In this subsection, we present our kernel and MLP training hyperparameters and
learning protocol for the learning curves shown in fig. 4.

Batched training: Sets of kernels and MLPs were trained together and shown
training data in the same order. This is why some learning curves look
similar. The sets were: TEM kernels, TEM MLPs with 1 hidden layer, TEM MLPs
with 2 hidden layers, STEM kernels, STEM MLPs with 1 hidden layer, STEM MLPs
with 2 hidden layers, TEM+STEM kernels, TEM+STEM MLPs with 1 hidden layer and
TEM+STEM MLPs with 2 hidden layers.

Loss metric: Our kernels and MLPs were trained to minimize MSEs between single
$d$$\times$$d$ crops from TEM, STEM or TEM+STEM images and the respective
outputs from latent depth 16 TEM, STEM and TEM+STEM autoencoders. Crop sizes
were at least $d=w_{\mathrm{max}}+5$, where $w_{\mathrm{max}}$ is the largest
input size, $w$, in a set of kernels or autoencoders being trained together.
To avoid edge artifacts, MSEs were calculated using pixels at least
$(w~{}-~{}1)/2$ from the edges of the crops.

Optimization: The ADAM solver was used with the parameters recommended in [4]
for 10000 batches. The learning rate was quadratically stepped down from
$\eta_{0}~{}=~{}0.01$ to
$\eta=(1-\textit{iter}/\textit{max\\_iter})^{2}\eta_{0}$ after each iteration.

Activation: All neurons are sigmoid activated.

Initialization: All weights were Xavier[6] initialized. All biases were zero
initialized. The last convolutional layer has no weights or biases.

##  4 Performance

Autoencoder, kernel and MLP MSE performances for TEM, STEM and TEM+STEM
training are summarized in table 3 for comparison. They are for the last 500
training batches of the learning curves in fig. 2 and fig. 4. The performance
of each autoencoder, kernel and MLP for the domain it was trained for is
exemplified by figs 5-12 in the appendix.

Autoencoder MSE means and standard deviations are negatively correlated with
latent depth and are lowest for either the highest or second highest latent
depths we trained. This is a result of larger latent spaces encoding more
information, making decoding easier.

Kernel and MLP MSE means and standard deviations are negatively correlated
with input sizes and the number of hidden layers (a kernel is a 0 hidden layer
MLP). This is a result of kernels and MLPs with higher input sizes and more
hidden layers having more trainable parameters. Since their successive layers
are fully connected, they have higher capacities and were therefore able to
more accurately approximate the 4$\times$ compression autoencoders.

STEM MSEs are systematically higher than TEM MSEs, with TEM+STEM MSEs
inbetween. In part, this is because local variations in intensity are much
higher in STEM images, making them harder to autoencode. This meant that STEM
autoencoder learning curves did not plateau in our 60000 batch training
schedule, resulting in STEM MSEs being above their convergence limit.
Nevertheless, this has not had a large qualitative effect on the ability of
kernels and MLPs to restore STEM images, as shown by the examples in the
appendix.

##  5 Summary

  * •

We have presented TEM, STEM and TEM+STEM denoising autoencoders for a range of
compression ratios between 1$\times$ and 64$\times$.

  * •

Kernels and MLPs with input sizes between 3 and 15 have been trained
approximate the restorations of 4$\times$ compression TEM, STEM and TEM+STEM
autoencoders.

  * •

Fully trained autoencoders, kernels and MLPs have been made publicly available
with example usage: https://github.com/Jeffrey-Ede/Denoising-Kernels-MLPs-
Autoencoders.

  * •

Our autoencoder, kernel and MLP architectures, training hyperparameters and
learning protocols are detailed.

  * •

Example applications of all our autoencoders, kernels and MLPs are provided in
the appendix. Some kernels are also presented.

## References

  * [1] Y. Sugawara, S. Shiota, and H. Kiya, “Super-resolution using convolutional neural networks without any checkerboard artifacts,” arXiv preprint arXiv:1806.02658, 2018.
  * [2] V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807–814, 2010.
  * [3] P. J. Huber, “Robust estimation of a location parameter,” The annals of mathematical statistics, pp. 73–101, 1964.
  * [4] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  * [5] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” arXiv preprint arXiv:1502.03167, 2015.
  * [6] X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249–256, 2010.

##  6 Appendix

In this appendix, we show example applications of each of our kernels, MLPs
and autoencoders to the domains they were trained for. These are followed by
our 3$\times$3, 5$\times$5, 7$\times$7 and 11$\times$11 kernels for TEM, STEM
and TEM+STEM.

TEM
---

$\left(\begin{array}[]{ccc}0.064&0.135&0.064\\\ 0.135&0.218&0.135\\\
0.064&0.135&0.064\end{array}\right)$

$\left(\begin{array}[]{ccccc}-0.084&-0.001&0.030&-0.001&-0.084\\\
-0.001&0.107&0.157&0.107&-0.001\\\ 0.030&0.157&0.220&0.157&0.030\\\
-0.001&0.107&0.157&0.107&-0.001\\\
-0.084&-0.001&0.030&-0.001&-0.084\end{array}\right)$

$\left(\begin{array}[]{ccccccc}-0.039&-0.038&-0.025&-0.018&-0.025&-0.038&-0.039\\\
-0.038&-0.009&0.035&0.058&0.035&-0.009&-0.038\\\
-0.025&0.035&0.114&0.153&0.114&0.035&-0.025\\\
-0.018&0.058&0.153&0.204&0.153&0.058&-0.018\\\
-0.025&0.035&0.114&0.153&0.114&0.035&-0.025\\\
-0.038&-0.009&0.035&0.058&0.035&-0.009&-0.038\\\
-0.039&-0.038&-0.025&-0.018&-0.025&-0.038&-0.039\end{array}\right)$

$\left(\begin{array}[]{ccccccccccc}-0.017&0.000&0.001&-0.005&-0.006&-0.006&-0.006&-0.005&0.001&0.000&-0.017\\\
0.000&0.009&-0.001&-0.013&-0.015&-0.014&-0.015&-0.013&-0.001&0.009&0.000\\\
0.001&-0.001&-0.015&-0.022&-0.010&-0.001&-0.010&-0.022&-0.015&-0.001&0.001\\\
-0.005&-0.013&-0.022&-0.007&0.035&0.059&0.035&-0.007&-0.022&-0.013&-0.005\\\
-0.006&-0.015&-0.010&0.035&0.111&0.152&0.111&0.035&-0.010&-0.015&-0.006\\\
-0.006&-0.014&-0.001&0.059&0.152&0.202&0.152&0.059&-0.001&-0.014&-0.006\\\
-0.006&-0.015&-0.010&0.035&0.111&0.152&0.111&0.035&-0.010&-0.015&-0.006\\\
-0.005&-0.013&-0.022&-0.007&0.035&0.059&0.035&-0.007&-0.022&-0.013&-0.005\\\
0.001&-0.001&-0.015&-0.022&-0.010&-0.001&-0.010&-0.022&-0.015&-0.001&0.001\\\
0.000&0.009&-0.001&-0.013&-0.015&-0.014&-0.015&-0.013&-0.001&0.009&0.000\\\
-0.017&0.000&0.001&-0.005&-0.006&-0.006&-0.006&-0.005&0.001&0.000&-0.017\end{array}\right)$

STEM
---

$\left(\begin{array}[]{ccc}0.108&0.111&0.108\\\ 0.111&0.109&0.111\\\
0.108&0.111&0.108\end{array}\right)$

$\left(\begin{array}[]{ccccc}0.004&0.026&0.040&0.026&0.004\\\
0.026&0.057&0.089&0.057&0.026\\\ 0.040&0.089&0.089&0.089&0.040\\\
0.026&0.057&0.089&0.057&0.026\\\
0.004&0.026&0.040&0.026&0.004\end{array}\right)$

$\left(\begin{array}[]{ccccccc}-0.016&-0.004&0.007&0.012&0.007&-0.004&-0.016\\\
-0.004&0.007&0.026&0.035&0.026&0.007&-0.004\\\
0.007&0.026&0.057&0.071&0.057&0.026&0.007\\\
0.012&0.035&0.071&0.089&0.071&0.035&0.012\\\
0.007&0.026&0.057&0.071&0.057&0.026&0.007\\\
-0.004&0.007&0.026&0.035&0.026&0.007&-0.004\\\
-0.016&-0.004&0.007&0.012&0.007&-0.004&-0.016\end{array}\right)$

$\left(\begin{array}[]{ccccccccccc}0.012&0.003&-0.001&-0.002&0.000&0.002&0.000&-0.002&-0.001&0.003&0.012\\\
0.003&-0.006&-0.009&-0.007&-0.001&0.005&-0.001&-0.007&-0.009&-0.006&0.003\\\
-0.001&-0.009&-0.010&-0.001&0.007&0.013&0.007&-0.001&-0.010&-0.009&-0.001\\\
-0.002&-0.007&-0.001&0.012&0.029&0.038&0.029&0.012&-0.001&-0.007&-0.002\\\
0.000&-0.001&0.007&0.029&0.055&0.070&0.055&0.029&0.007&-0.001&0.000\\\
0.002&0.005&0.013&0.038&0.070&0.089&0.070&0.038&0.013&0.005&0.002\\\
0.000&-0.001&0.007&0.029&0.055&0.070&0.055&0.029&0.007&-0.001&0.000\\\
-0.002&-0.007&-0.001&0.012&0.029&0.038&0.029&0.012&-0.001&-0.007&-0.002\\\
-0.001&-0.009&-0.010&-0.001&0.007&0.013&0.007&-0.001&-0.010&-0.009&-0.001\\\
0.003&-0.006&-0.009&-0.007&-0.001&0.005&-0.001&-0.007&-0.009&-0.006&0.003\\\
0.012&0.003&-0.001&-0.002&0.000&0.002&0.000&-0.002&-0.001&0.003&0.012\end{array}\right)$

TEM+STEM
---

$\left(\begin{array}[]{ccc}0.093&0.124&0.093\\\ 0.124&0.149&0.124\\\
0.093&0.124&0.093\end{array}\right)$

$\left(\begin{array}[]{ccccc}-0.061&0.016&0.042&0.016&-0.061\\\
0.016&0.091&0.116&0.091&0.016\\\ 0.042&0.116&0.142&0.116&0.042\\\
0.016&0.091&0.116&0.091&0.016\\\
-0.061&0.016&0.042&0.016&-0.061\end{array}\right)$

$\left(\begin{array}[]{ccccccc}-0.077&-0.037&-0.008&0.001&-0.008&-0.037&-0.077\\\
-0.037&0.016&0.052&0.063&0.052&0.016&-0.037\\\
-0.008&0.052&0.095&0.110&0.095&0.052&-0.008\\\
0.001&0.063&0.110&0.127&0.110&0.063&0.001\\\
-0.008&0.052&0.095&0.110&0.095&0.052&-0.008\\\
-0.037&0.016&0.052&0.063&0.052&0.016&-0.037\\\
-0.077&-0.037&-0.008&0.001&-0.008&-0.037&-0.077\end{array}\right)$

$\left(\begin{array}[]{ccccccccccc}0.005&-0.003&-0.015&-0.022&-0.019&0.017&-0.019&-0.022&-0.015&-0.003&0.005\\\
-0.003&-0.008&-0.013&-0.013&-0.004&0.001&-0.004&-0.013&-0.013&-0.008&1\\\
-0.015&-0.013&-0.011&-0.001&0.017&0.025&0.017&-0.001&-0.011&-0.013&-0.015\\\
-0.022&-0.013&-0.001&0.021&0.050&0.062&0.050&0.021&-0.001&-0.013&-0.022\\\
-0.019&-0.004&0.017&0.050&0.088&0.105&0.088&0.050&0.017&-0.004&-0.019\\\
0.017&0.001&0.025&0.062&0.105&0.123&0.105&0.062&0.025&0.001&0.017\\\
-0.019&-0.004&0.017&0.050&0.088&0.105&0.088&0.050&0.017&-0.004&-0.019\\\
-0.022&-0.013&-0.001&0.021&0.050&0.062&0.050&0.021&-0.001&-0.013&-0.022\\\
-0.015&-0.013&-0.011&-0.001&0.017&0.025&0.017&-0.001&-0.011&-0.013&-0.015\\\
-0.003&-0.008&-0.013&-0.013&-0.004&0.001&-0.004&-0.013&-0.013&-0.008&-0.003\\\
0.005&-0.003&-0.015&-0.022&-0.019&0.017&-0.019&-0.022&-0.015&-0.003&0.005\end{array}\right)$

Figure 5:  Example applications of denoising autoencoders, kernels and MLPs.
Figure 6:  More example applications of denoising autoencoders, kernels and
MLPs.  Figure 7:  More example applications of denoising autoencoders, kernels
and MLPs.  Figure 8:  More example applications of denoising autoencoders,
kernels and MLPs.  Figure 9:  More example applications of denoising
autoencoders, kernels and MLPs.  Figure 10:  More example applications of
denoising autoencoders, kernels and MLPs.  Figure 11:  More example
applications of denoising autoencoders, kernels and MLPs.  Figure 12:  More
example applications of denoising autoencoders, kernels and MLPs.