# MatSci-NLP: Evaluating Scientific Language Models on Materials Science
Language Tasks Using Text-to-Schema Modeling

Yu Song1, Santiago Miret2,11footnotemark: 1 Bang Liu1,
1University of Montreal / Mila - Quebec AI, 2Intel Labs
{yu.song<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
Equal contribution. Corresponding author. Canada CIFAR AI Chair.

###### Abstract

We present MatSci-NLP, a natural language benchmark for evaluating the
performance of natural language processing (NLP) models on materials science
text. We construct the benchmark from publicly available materials science
text data to encompass seven different NLP tasks, including conventional NLP
tasks like named entity recognition and relation classification, as well as
NLP tasks specific to materials science, such as synthesis action retrieval
which relates to creating synthesis procedures for materials. We study various
BERT-based models pretrained on different scientific text corpora on MatSci-
NLP to understand the impact of pretraining strategies on understanding
materials science text. Given the scarcity of high-quality annotated data in
the materials science domain, we perform our fine-tuning experiments with
limited training data to encourage the generalize across MatSci-NLP tasks. Our
experiments in this low-resource training setting show that language models
pretrained on scientific text outperform BERT trained on general text.
MatBERT, a model pretrained specifically on materials science journals,
generally performs best for most tasks. Moreover, we propose a unified text-
to-schema for multitask learning on MatSci-NLP and compare its performance
with traditional fine-tuning methods. In our analysis of different training
methods, we find that our proposed text-to-schema methods inspired by
question-answering consistently outperform single and multitask NLP fine-
tuning methods. The code and datasets are publicly
available111https://github.com/BangLab-UdeM-Mila/NLP4MatSci-ACL23.

##  1 Introduction

Materials science comprises an interdisciplinary scientific field that studies
the behavior, properties and applications of matter that make up materials
systems. As such, materials science often requires deep understanding of a
diverse set of scientific disciplines to meaningfully further the state of the
art. This interdisciplinary nature, along with the great technological impact
of materials advances and growing research work at the intersection of machine
learning and materials science (Miret et al., ; Pilania, 2021; Choudhary et
al., 2022), makes the challenge of developing and evaluating natural language
processing (NLP) models on materials science text both interesting and
exacting.

The vast amount of materials science knowledge stored in textual format, such
as journal articles, patents and technical reports, creates a tremendous
opportunity to develop and build NLP tools to create and understand advanced
materials. These tools could in turn enable faster discovery, synthesis and
deployment of new materials into a wide variety of application, including
clean energy, sustainable manufacturing and devices.

Understanding, processing, and training language models for scientific text
presents distinctive challenges that have given rise to the creation of
specialized models and techniques that we review in Section 2. Additionally,
evaluating models on scientific language understanding tasks, especially in
materials science, often remains a laborious task given the shortness of high-
quality annotated data and the lack of broad model benchmarks. As such, NLP
research applied to materials science remains in the early stages with a
plethora of ongoing research efforts focused on dataset creation, model
training and domain specific applications.

The broader goal of this work is to enable the development of pertinent
language models that can be applied to further the discovery of new material
systems, and thereby get a better sense of how well language models understand
the properties and behavior of existing and new materials. As such, we propose
MatSci-NLP, a benchmark of various NLP tasks spanning many applications in the
materials science domain described in Section 3. We utilize this benchmark to
analyze the performance of various BERT-based models for MatSci-NLP tasks
under distinct textual input schemas described in Section 4. Concretely,
through this work we make the following research contributions:

  * •

MatSci-NLP Benchmark: We construct the first broad benchmark for NLP in the
materials science domain, spanning several different NLP tasks and materials
applications. The benchmark contents are described in Section 3 with a general
summary and data sources provided in Table 1. The processed datasets and code
will be released after acceptance of the paper for reproducibility.

  * •

Text-to-Schema Multitasking: We develop a set of textual input schemas
inspired by question-answering settings for fine-tuning language models. We
analyze the models’ performance on MatSci-NLP across those settings and
conventional single and multitask fine-tuning methods. In conjunction with
this analysis, we propose a new Task-Schema input format for joint multitask
training that increases task performance for all fine-tuned language models.

  * •

MatSci-NLP Analysis: We analyze the performance of various BERT-based models
pretrained on different scientific and non-scientific text corpora on the
MatSci-NLP benchmark. This analysis help us better understand how different
pretraining strategies affect downstream tasks and find that MatBERT (Walker
et al., 2021), a BERT model trained on materials science journals, generally
performs best reinforcing the importance of curating high-quality pretraining
corpora.

We centered our MatSci-MLP analysis on exploring the following questions:

  1. Q1

_How does in-domain pretraining of language models affect the downstream
performance on MatSci-NLP tasks?_ We investigate the performance of various
models pretrained on different kinds of domain-specific text including
materials science, general science and general language (BERT (Devlin et al.,
2018)). We find that MatBERT generally performs best and that language models
pretrained on diverse scientific texts outperform a general language BERT.
Interestingly, SciBERT (Beltagy et al., 2019) often outperforms materials
science language models, such as MatSciBERT (Gupta et al., 2022) and
BatteryBERT (Huang and Cole, 2022).

  2. Q2

_How do in-context data schema and multitasking affect the learning efficiency
in low-resource training settings?_ We investigate how several input schemas
shown in Figure 1 that contain different kinds of information affect various
domain-specific language models and propose a new Task-Schema method. Our
experiments show that our proposed Task-Schema method mostly performs best
across all models and that question-answering inspired schema outperform
single task and multitask fine-tuning settings.

##  2 Background

The advent of powerful NLP models has enabled the analysis and generation of
text-based data across a variety of domains. BERT (Devlin et al., 2018) was
one of the first large-scale transformer-based models to substantially advance
the state-of-the-art by training on large amounts of unlabeled text data in a
self-supervised way. The pretraining procedure was followed by task-specific
fine-tuning, leading to impressive results on a variety of NLP task, such as
named entity recognition (NER), question and answering (QA), and relation
classification (Hakala and Pyysalo, 2019; Qu et al., 2019; Wu and He, 2019). A
significant collection of large language models spanning millions to billions
of parameters followed the success of BERT adopting a similar approach of
pretraining on vast corpora of text with task-specific fine-tuning to push the
state-of-the-art for in natural language processing and understanding (Raffel
et al., 2020; Brown et al., 2020; Scao et al., 2022).

###  2.1 Scientific Language Models

The success of large language models on general text motivated the development
of domain-specific language models pretrained on custom text data, including
text in the scientific domain: SciBERT (Beltagy et al., 2019), ScholarBERT
(Hong et al., 2022) and Galactica (Taylor et al., 2022) are pretrained on
general corpus of scientific articles; BioBERT (Lee et al., 2020), PubMedBERT
(Gu et al., 2021), BioMegatron (Shin et al., 2020) and Sci-Five (Phan et al.,
2021) are pretrained on various kinds of biomedical corpora; MatBERT (Walker
et al., 2021), MatSciBERT (Gupta et al., 2022) are pretrained on materials
science specific corpora; and BatteryBERT (Huang and Cole, 2022) is pretrained
on a corpus focused on batteries. Concurrently, several domain-specific NLP
benchmarks were established to assess language model performance on domain-
specific tasks, such as QASPER (Dasigi et al., 2021) and BLURB (Gu et al.,
2021) in the scientific domain, as well as PubMedQA (Jin et al., 2019), BioASQ
(Balikas et al., 2015), and Biomedical Language Understanding Evaluation
(BLUE) (Peng et al., 2019) in the biomedical domain.

###  2.2 NLP in Materials Science

The availability of openly accessible, high-quality corpora of materials
science text data remains highly restricted in large part because data from
peer-reviewed journals and scientific documents is usually subject to
copyright restrictions, while open-domain data is often only available in
difficult-to-process PDF formats (Olivetti et al., 2020; Kononova et al.,
2021). Moreover, specialized scientific text, such as materials synthesis
procedures containing chemical formulas and reaction notation, require
advanced data mining techniques for effective processing (Kuniyoshi et al.,
2020; Wang et al., 2022b). Given the specificity, complexity, and diversity of
specialized language in scientific text, effective extraction and processing
remain an active area of research with the goal of building relevant and
sizeable text corpora for pretraining scientific language models (Kononova et
al., 2021).

Nonetheless, materials science-specific language models, including MatBERT
(Walker et al., 2021), MatSciBERT (Gupta et al., 2022), and BatteryBERT (Huang
and Cole, 2022), have been trained on custom-built pretraining dataset curated
by different academic research groups. The pretrained models and some of the
associated fine-tuning data have been released to the public and have enabled
further research, including this work.

The nature of NLP research in materials science to date has also been highly
fragmented with many research works focusing on distinct tasks motivated by a
given application or methodology. Common ideas among many works include the
prediction and construction of synthesis routes for a variety of materials
(Mahbub et al., 2020; Karpovich et al., 2021; Kim et al., 2020), as well as
the creation of novel materials for a given application (Huang and Cole, 2022;
Georgescu et al., 2021; Jensen et al., 2021), both of which relate broader
challenges in the field of materials science.

Figure 1: Example of different question-answering inspired textual input
schemas (Task-Schema , Potential Choices, Example) applied on MatSci-NLP. The
input of the language model includes the shared text (green) along with
relevant task details (blue for NER and orange for event extraction). The
shared text can contain relevant information for multiple tasks and be part of
the language model input multiple times.

##  3 MatSci-NLP Benchmark

Through the creation of MatSci-NLP, we aim to bring together some of the
fragmented data across multiple research works for a wide-ranging materials
science NLP benchmark. As described in Section 2, the availability of
sizeable, high-quality and diverse datasets remain a major obstacle in
applying modern NLP to advance materials science in meaningful ways. This is
primarily driven by a high cost of data labeling and the heterogeneous nature
of materials science. Given those challenges, we created MatSci-NLP by
unifying various publicly available, high-quality, smaller-scale datasets to
form a benchmark for fine-tuning and evaluating modern NLP models for
materials science applications. MatSci-NLP consists of seven NLP tasks shown
in Table 1, spanning a wide range of materials categories including fuel cells
(Friedrich et al., 2020), glasses (Venugopal et al., 2021), inorganic
materials (Weston et al., 2019; MatSciRE, 2022), superconductors (Yamaguchi et
al., 2020), and synthesis procedures pertaining to various kinds of materials
(Mysore et al., 2019; Wang et al., 2022a). Some tasks in MatSci-NLP had
multiple source components, meaning that the data was curated from multiple
datasets (e.g. NER), while many were obtained from a single source dataset.

Task Size (# Samples) Meta-Dataset Components Named Entity Recognition 112,191
4 Relation Classification 25,674 3 Event Argument Extraction 6,566 2 Paragraph
Classification 1,500 1 Synthesis Action Retrieval 5,547 1 Sentence
Classification 9,466 1 Slot Filling 8,253 1

Table 1: Collection of NLP tasks in the meta-dataset of the MatSci-NLP
Benchmark drawn from Weston et al. (2019); Friedrich et al. (2020); Mysore et
al. (2019); Yamaguchi et al. (2020); Venugopal et al. (2021); Wang et al.
(2022a); MatSciRE (2022).

The data in MatSci-NLP adheres to a standard JSON-based data format with each
of the samples containing relevant text, task definitions, and annotations.
These can in turn be refactored into different input schemas, such as the ones
shown in Figure 1 consisting of 1) _Input_ : primary text jointly with task
descriptions and instructions, and 2) _Output_ : query and label, which we
perform in our text-to-schema modeling described in Section 4. Next, we
describe the tasks in MatSci-NLP in greater detail:

  * •

Named Entity Recognition (NER): The NER task requires models to extract
summary-level information from materials science text and recognize entities
including materials, descriptors, material properties, and applications
amongst others. The NER task predicts the best entity type label for a given
text span $s_{i}$ with a non-entity span containing a “null” label. MatSci-NLP
contains NER task data adapted from Weston et al. (2019); Friedrich et al.
(2020); Mysore et al. (2019); Yamaguchi et al. (2020).

  * •

Relation Classification:  In the relation classification task, the model
predicts the most relevant relation type for a given span pair
$(s_{i},s_{j})$. MatSci-NLP contains relation classification task data adapted
from Mysore et al. (2019); Yamaguchi et al. (2020); MatSciRE (2022).

  * •

Event Argument Extraction:  The event argument extraction task involves
extracting event arguments and relevant argument roles. As there may be more
than a single event for a given text, we specify event triggers and require
the language model to extract corresponding arguments and their roles. MatSci-
NLP contains event argument extraction task data adapted from Mysore et al.
(2019); Yamaguchi et al. (2020).

  * •

Paragraph Classification:  In the paragraph classification task adapted from
Venugopal et al. (2021), the model determines whether a given paragraph
pertains to glass science.

  * •

Synthesis Action Retrieval (SAR):  SAR is a materials science domain-specific
task that defines eight action terms that unambiguously identify a type of
synthesis action to describe a synthesis procedure. MatSci-NLP adapts SAR data
from Wang et al. (2022a) to ask language models to classify word tokens into
pre-defined action categories.

  * •

Sentence Classification:  In the sentence classification task, models identify
sentences that describe relevant experimental facts based on data adapted from
Friedrich et al. (2020).

  * •

Slot Filling:  In the slot-filling task, models extract slot fillers from
particular sentences based on a predefined set of semantically meaningful
entities. In the task data adapted from Friedrich et al. (2020), each sentence
describes a single experiment frame for which the model predicts the slots in
that frame.

The tasks contained in MatSci-NLP were selected based on publicly available,
high-quality annotated materials science textual data, as well as their
relevance to applying NLP tools to materials science. Conventional NLP tasks
(NER, Relation Classification, Event Argument Extraction, Paragraph
Classification, Sentence Classification) enable materials science researchers
to better process and understand relevant textual data. Domain specific tasks
(SAR, Slot Filling) enable materials science research to solve concrete
challenges, such as finding materials synthesis procedures and real-world
experimental planning. In the future, we aim to augment to current set of
tasks with additional data and introduce novel tasks that address materials
science specific challenges with NLP tools.

##  4 Unified Text-to-Schema Language Modeling

As shown in Figure 1, a given piece of text can include multiple labels across
different tasks. Given this multitask nature of the MatSci-NLP benchmark, we
propose a new and unified Task-Schema  multitask modeling method illustrated
in Figure 2 that covers all the tasks in the MatSci-NLP dataset. Our approach
centers on a unified text-to-schema modeling approach that can predict
multiple tasks simultaneously through a unified format. The underlying
language model architecture is made up of modular components, including a
domain-specific encoder model (e.g. MatBERT, MatSciBERT, SciBERT), and a
generic transformer-based decoder, each of which can be easily exchanged with
different pretrained domain-specific NLP models. We fine-tune these pretrained
language models and the decoder with collected tasks in MatSci-NLP using the
procedure described in Section 4.3.

The unified text-to-schema provides a more structured format to training and
evaluating language model outputs compared to seq2seq and text-to-text
approaches (Raffel et al., 2020; Luong et al., 2015). This is particularly
helpful for the tasks in MatSci-NLP given that many tasks can be reformulated
as classification problems. NER and Slot Filling, for example, are
classifications at the token-level, while event arguments extraction entails
the classification of roles of certain arguments. Without a predefined schema,
the model relies entirely on unstructured natural language to provide the
answer in a seq2seq manner, which significantly increases the complexity of
the task and also makes it harder to evaluate performance. The structure
imposed by text-to-schema method also simplifies complex tasks, such as event
extraction, by enabling the language model to leverage the structure of the
schema to predict the correct answer. We utilize the structure of the schema
in decoding and evaluating the output of the language models, as described
further in Section 4.3 in greater detail.

Moreover, our unified text-to-schema approach alleviates error propagation
commonly found in multitask scenarios (Van Nguyen et al., 2022; Lu et al.,
2021), enables knowledge sharing across multiple tasks and encourages the
fine-tuned language model to generalize across a broader set of text-based
instruction scenarios. This is supported by our results shown in Section 5.2
showing text-to-schema outperforming conventional methods.

Figure 2: Unified text-to-schema method for MatSci-NLP text understanding
applied across the seven tasks. The language model includes a domain specific
encoder, which can be exchanged in a modular manner, as well as a general
language pretrained transformer decoder.

###  4.1 Language Model Formulation

The general purpose of our model is to achieve multitask learning by a mapping
function ($f$) between input ($x$), output ($y$), and schema ($s$), i.e.,
$f(x,s)=y$. Due to the multitasking nature of our setting, both inputs and
outputs can originate from different tasks n, i.e.
$x=[x_{t}1,x_{t}2,...x_{t}n]$ and $y=[y_{t}1,y_{t}2,...y_{t}n]$, all of which
fit under a common schema ($s$). Given the presence of domain-specific
materials science language, our model architecture includes a domain-specific
BERT encoder and a transformer decoder. All BERT encoders and transformer
decoders share the same general architecture, which relies on a self-attention
mechanism: Given an input sequence of length $N$, we compute a set of
attention scores, $A=\rm{softmax}(QT^{K}/(\sqrt{d_{k}}))$. Next, we compute
the weighted sum of the value vectors, $O=AV$, where $Q$, $K$, and $V$ are the
query, key, and value matrices, and $d_{k}$ is the dimensionality of the key
vectors.

Additionally, the transformer based decoder differ from the domain specific
encoder by: 1) Applying masking based on the schema applied to ensure that it
does not attend to future positions in the output sequence. 2) Applying both
self-attention and encoder-decoder attention to compute attention scores that
weigh the importance of different parts of the output sequence and input
sequence. The output of the self-attention mechanism ($O_{1}$) and the output
of the encoder-decoder attention mechanism ($O_{2}$) are concatenated and
linearly transformed to obtain a new hidden state,
$H=\rm{tanh}(W_{o}[O_{1};O_{2}]+b_{o})$ with $W_{o}$ and $b_{o}$ being the
weight and biases respectively. The model then applies a _softmax_ to $H$ to
generate the next element in the output sequence
$P=\rm{softmax}(W_{p}H+b_{p})$ , where $P$ is a probability distribution over
the output vocabulary.

###  4.2 Text-To-Schema Modeling

As shown in Figure 1, our schema structures the text data based on four
general components: text, description, instruction options, and the predefined
answer schema.

  * •

Text specifies raw text from the literature that is given as input to the
language model.

  * •

Description describes the task for a given text according to a predefined
schema containing the task name and the task arguments.

  * •

Instruction Options contains the core explanation related to the task with
emphasis on three different types: 1) Potential choices of answers; 2) Example
of an input/output pair corresponding to the task; 3) Task-Schema : our
predefined answer schema illustrated in Figure 2.

  * •

Answer describes the correct label of each task formatted as a predefined
answer schema that can be automatically generated based on the data structure
of the task.

###  4.3 Language Decoding & Evaluation

Evaluating the performance of the language model on MatSci-NLP requires
determining if the text generated by the decoder is valid and meaningful in
the context of a given task. To ensure consistency in evaluation, we apply a
constrained decoding procedure consisting of two steps: 1) Filtering out
invalid answers through the predefined answer schema shown in Figure 2 based
on the structure of the model’s output; 2) Match the model’s prediction with
the most similar valid class given by the annotation for the particular task.
For example, if for the NER task shown in Figure 1 the model’s predicted token
is “BaCl2 2H2O materials”, it will be matched with the NER label of
“material”, which is then used as the final prediction for computing losses
and evaluating performance. This approach essentially reformulates each task
as a classification problem where the classes are provided based on the labels
from the tasks in MatSci-NLP. We then apply a cross-entropy loss for model
fine-tuning based on the matched label from the model output. The matching
procedure simplifies the language modeling challenge by not requiring an exact
match of the predicted tokens with the task labels. This in turns leads to a
more comprehensible signal in the fine-tuning loss function.

NLP Model Named Entity Recognition Relation Classification Event Argument
Extraction Paragraph Classification Synthesis Action Retrieval Sentence
Classification Slot Filling Overall (All Tasks) MatSciBERT (Gupta et al.,
2022) 0.707±0.076 0.470±0.092 0.791±0.046 0.507±0.073 0.436±0.066 0.251±0.075
0.719±0.116 0.623±0.183 0.692±0.179 0.484±0.254 0.914±0.008 0.660±0.079
0.436±0.142 0.194±0.062 0.671±0.060 0.456±0.042 MatBERT (Walker et al., 2021)
0.875±0.015 0.630±0.047 0.804±0.071 0.513±0.138 0.451±0.091 0.288±0.066
0.756±0.073 0.691±0.188 0.717±0.040 0.549±0.091 0.909±0.009 0.614±0.134
0.548±0.058 0.273±0.051 0.722±0.023 0.517±0.041 BatteryBERT (Huang and Cole,
2022) 0.786±0.113 0.472±0.150 0.801±0.081 0.466±0.111 0.457±0.024 0.277±0.034
0.633±0.075 0.610±0.046 0.614±0.128 0.419±0.149 0.912±0.015 0.684±0.095
0.520±0.057 0.224±0.073 0.663±0.038 0.456±0.048 SciBERT (Beltagy et al., 2019)
0.734±0.079 0.497±0.091 0.819±0.067 0.545±0.119 0.451±0.077 0.276±0.080
0.696±0.094 0.546±0.243 0.701±0.138 0.516±0.217 0.911±0.017 0.617±0.143
0.481±0.144 0.224±0.010 0.685±0.056 0.460±0.044 ScholarBERT (Hong et al.,
2022) 0.168±0.067 0.101±0.034 0.428±0.148 0.274±0.110 0.489±0.083 0.356±0.109
0.663±0.032 0.433±0.122 0.322±0.260 0.178±0.051 0.906±0.007 0.478±0.008
0.296±0.085 0.109±0.044 0.468±0.028 0.276±0.024 BioBERT (Wada et al., 2020)
0.715±0.031 0.459±0.055 0.797±0.092 0.465±0.134 0.488±0.036 0.274±0.049
0.675±0.144 0.578±0.102 0.647±0.140 0.446±0.231 0.915±0.021 0.686±0.098
0.452±0.114 0.191±0.045 0.670±0.061 0.442±0.057 BERT (Devlin et al., 2018)
0.657±0.077 0.461±0.058 0.782±0.056 0.494±0.061 0.418±0.053 0.225±0.091
0.665±0.057 0.532±0.194 0.656±0.099 0.515±0.067 0.910±0.017 0.633±0.133
0.520±0.019 0.257±0.022 0.658±0.030 0.439±0.021

Table 2: Low-resource fine-tuning results applying unified Task-Schema setting
for various BERT-based encoder models pretrained on different domain specific
text data. For each model, the top line represents the micro-F1 score and the
bottom line represents the macro-F1 score. We report the mean across 5
experiments with a confidence interval of two standard deviations. We denote
the best performing encoder model and those that outperform the general
language BERT according to the micro-f1 with orange shading with MatBERT and
SciBERT performing best on most tasks and ScholarBERT and general language
BERT generally performing worst.

##  5 Evaluation and Results

NLP Model Single Task Single Task Prompt MMOE No Explanations Potential
Choices Examples Task-Schema MatSciBERT (Gupta et al., 2022) 0.501±0.057
0.320±0.078 0.485±0.043 0.238±0.017 0.457±0.021 0.228±0.038 0.651±0.045
0.438±0.052 0.670±0.036 0.435±0.061 0.688±0.045 0.463±0.040 0.671±0.060
0.456±0.042 MatBERT (Walker et al., 2021) 0.537±0.036 0.330±0.063 0.523±0.021
0.267±0.014 0.557±0.010 0.301±0.006 0.721±0.033 0.514±0.045 0.699±0.020
0.478±0.032 0.705±0.025 0.470±0.029 0.722±0.023 0.517±0.041 BatteryBERT (Huang
and Cole, 2022) 0.469±0.050 0.288±0.055 0.488±0.011 0.241±0.009 0.431±0.044
0.200±0.022 0.660±0.013 0.450±0.031 0.622±0.069 0.423±0.039 0.660±0.033
0.416±0.054 0.663±0.038 0.456±0.048 SciBERT (Beltagy et al., 2019) 0.500±0.055
0.300±0.080 0.502±0.030 0.248±0.015 0.504±0.052 0.275±0.031 0.680±0.066
0.458±0.060 0.660±0.042 0.435±0.061 0.686±0.039 0.460±0.042 0.685±0.056
0.460±0.044 ScholarBERT (Hong et al., 2022) 0.472±0.137 0.234±0.094
0.429±0.258 0.250±0.142 0.367±0.075 0.165±0.044 0.461±0.016 0.271±0.022
0.513±0.041 0.295±0.055 0.467±0.019 0.260±0.018 0.468±0.028 0.276±0.024
BioBERT (Wada et al., 2020) 0.487±0.059 0.281±0.026 0.488±0.032 0.238±0.017
0.360±0.007 0.151±0.002 0.663±0.044 0.442±0.079 0.587±0.022 0.365±0.018
0.632±0.040 0.404±0.046 0.670±0.061 0.442±0.057 BERT (Devlin et al., 2018)
0.498±0.051 0.266±0.044 0.488±0.043 0.239±0.011 0.394±0.009 0.166±0.008
0.670±0.020 0.440±0.052 0.601±0.046 0.382±0.039 0.636±0.052 0.394±0.051
0.658±0.030 0.439±0.021 Overall (All Models) 0.493±0.064 0.288±0.063
0.486±0.062 0.246±0.032 0.439±0.003 0.212±0.022 0.644±0.034 0.430±0.049
0.622±0.035 0.402±0.049 0.639±0.044 0.410±0.043 0.688±0.046 0.435±0.039

Table 3: Consolidated results among all MatSci-NLP tasks on different training
settings for various BERT-based encoder models pretrained on different domain
specific text data. For each model, the top line represents the micro-F1 score
and the bottom line represents the macro-F1 score. We report the mean across 5
experiments with a confidence interval of two standard deviations. We
highlight the performance of different schema according to heatmap ranging
from best and worst. The concentration of red hues on right side indicates
that the question-answering inspiring schema generally outperform conventional
fine-tuning method. Our proposed Task-Schema generally outperforms all other
schemas across most enconder models.

Our analysis focuses on the questions outlined in Section 1: 1) Studying the
effectiveness of domain-specific language models as encoders, and 2) Analyzing
the effect of different input schemas in resolving MatSci-NLP tasks.
Concretely, we study the performance of the language models and language
schema in a low resource setting where we perform fine-tuning on different
pretrained BERT models with limited data from the MatSci-NLP benchmark. This
low-resource setting makes the learning problem harder given that the model
has to generalize on little amount of data. Moreover, this setting
approximates model training with very limited annotated data, which is
commonly found in materials science as discussed in Section 2. In our
experiments, we split the data in MatSci-NLP into 1% training subset and a 99%
testing subset for evaluation. None of the evaluated encoder models were
exposed to the fine-tuning data in advance of our experiments and therefore
have to rely on the knowledge acquired during their respective pretraining
processes. We evaluate the results of our experiments using micro-F1 and
macro-F1 scores of the language model predictions on the test split of the
MatSci-NLP that were not exposed during fine-tuning.

###  5.1 How does in-domain pretraining of language models affect the
downstream performance on MatSci-NLP tasks? (Q1)

Based on the results shown in Table 2, we can gather the following insights:

First, domain-specific pretraining affects model performance. We perform fine-
tuning on various models pretrained on domain-specific corpora in a low-
resource setting and observe that: i) MatBert, which was pretrained on textual
data from materials science journals, generally performs best for most tasks
in the MatSci-NLP benchmark with SciBERT generally performing second best. The
high performance of MatBERT suggests that materials science specific
pretraining does help the language models acquire relevant materials science
knowledge. Yet, the underperformance of MatSciBERT compared to MatBERT and
SciBERT indicates that the curation of pretraining data does significantly
affect performance. ii) The importance of the pretraining corpus is further
reinforced by the difference in performance between SciBERT and ScholarBERT,
both of which were trained on corpora of general scientific text, but show
vastly different results. In fact, ScholarBERT underperforms all other models,
including the general language BERT, for all tasks except event argument
extraction where ScholarBERT performs best compared to all other models. iii)
The fact that most scientific BERT models outperform BERT pretrained on
general language suggests that pretraining on high-quality scientific text is
beneficial for resolving tasks involving materials science text and
potentially scientific texts from other domains. This notion of enhanced
performance on MatSci-NLP when pretraining on scientific text is further
reinforced by the performance of BioBERT by Wada et al. (2020). BioBERT
outperforms BERT on most tasks even though it was trained on text from the
biomedical domain that has minor overlap with the materials science domain.
This strongly indicates that scientific language, regardless of the domain,
has a significant distribution shift from general language that is used to
pretrain common language models.

Second, imbalanced datasets in MatSci-NLP skew performance metrics: We can see
from Table 2 that the micro-F1 scores are significantly higher than the
macro-f1 across all tasks. This indicates that the datasets used in the
MatSci-NLP are consistently imbalanced, including in the binary classification
tasks, and thereby push the micro-F1 higher compared to the macro-F1 score. In
the case of paragraph classification, for example, the number of positive
examples is 492 compared with the total number of 1500 samples. As such, only
models with a micro-F1 score above 0.66 and macro-F1 above 0.5 can be
considered to have semantically meaningful understanding of the task. This is
even more pronounced for sentence classification where only $876/9466\approx
10\%$ corresponds to one label. All models except ScholarBERT outperform a
default guess of the dominant class for cases. While imbalanced datasets may
approximate some real-world use cases of materials science text analysis, such
as extracting specialized materials information, a highly imbalanced can be
misguiding in evaluating model performance.

To alleviate the potentially negative effects of imbalanced data, we suggest
three simple yet effective methods: 1) Weighted loss functions: This involves
weighting the loss function to give higher weights to minority classes. Focal
loss Lin et al. (2017), for example, is a loss function that dynamically
modulates the loss based on the prediction confidence, with greater emphasis
on more difficult examples. As such, Focal loss handles class imbalance well
due to the additional attention given to hard examples of the minority
classes. 2) Class-balanced samplers: Deep learning frameworks, such as
Pytorch, have class-balanced batch samplers that can be used to oversample
minority classes within each batch during training, which can help indirectly
address class imbalance. 3) Model architecture tweaks: The model architecture
and its hyper-parameters can be adjusted to place greater emphasis on minority
classes. For example, one can apply separate prediction heads for minority
classes or tweak L2 regularization and dropout to behave differently for
minority and majority classes.

###  5.2 How do in-context data schema and multitasking affect the learning
efficiency in low-resource training settings? (Q2)

To assess the efficacy of the proposed textual schemas shown in Figure 1, we
evaluate four different QA-inspired schemas: 1) No Explanations \- here the
model receives only the task description; 2) Potential Choices \- here the
model receives the class labels given by the task; 3) Examples \- here the
model receives an example of a correct answer, 4) Task-Schema \- here the
model receives our proposed textual schema. We compare the schemas to three
conventional fine-tuning methods: 1) Single Task \- the traditional method to
solve each task separately using the language model and a classification head;
2) Single Task Prompt \- here we change the format of the task to the same QA-
format as “No Explanations”, but train each task separately; 3) MMOE by Ma et
al. (2018) uses multiple encoders to learn multiple hidden embeddings, which
are then weighed by a task-specific gate unit and aggregated to the final
hidden embedding using a weighted sum for each task. Next, a task-specific
classification head outputs the label probability distribution for each task.

Based on the results shown in Table 3, we gather the following insights:

First, Text-to-Schema methods perform better for all language models. Overall,
the Task-Schema method we proposed performs best across all tasks in the
MatSci-NLP benchmark. The question-answering inspired schema (“No
Explanations”, “Potential Choices”, “Examples”, “Task-Schema ”) perform better
than fine-tuning in a traditional single task setting, single task prompting,
as well as fine-tuning using the MMOE multitask method. This holds across all
models for all the tasks in MatSci-NLP showing the efficacy of structured
language modeling inspired by question-answering.

Second, schema design affects model performance. The results show that both
the pretrained model and the input format affect performance. This can be seen
by the fact that while all scientific models outperform general language BERT
using the Task-Schema method, BERT outperforms some models, mainly ScholarBERT
and BioBERT, in the other text-to-schema settings and the conventional
training settings. Nevertheless, BERT underperforms the stronger models
(MatBERT, SciBERT, MatSciBERT) across all schema settings for all tasks in
MatSci-NLP, further emphasizing the importance of domain-specific model
pretraining for materials science language understanding.

##  6 Conclusion and Future Works

We proposed MatSci-NLP, the first broad benchmark on materials science
language understanding tasks constructed from publicly available data. We
further proposed text-to-schema multitask modeling to improve the model
performance in low-resource settings. Leveraging MatSci-NLP and text-to-schema
modeling, we performed an in-depth analysis of the performance of various
scientific language models and compare text-to-schema language modeling
methods with other input schemas, guided by (Q1) addressing the pretrained
models and (Q2) addressing the textual schema. Overall, we found that the
choice of pretrained models matters significantly for downstream performance
on MatSci-NLP tasks and that pretrained language models on scientific text of
any kind often perform better than pretrained language models on general text.
MatBERT generally performed best, highlighting the benefits of pretraining
with high-quality domain-specific language data. With regards to the textual
schema outlined in (Q2), we found that significant improvements can be made by
improving textual schema showcasing the potential of fine-tuning using
structured language modeling.

The proposed encoder-decoder architecture, as well as the proposed multitask
schema, could also be useful for additional domains in NLP, including both
scientific and non-scientific domains. The potential for open-domain
transferability of our method is due to: 1) Our multitask training method and
associated schemas do not depend on any domain-specific knowledge, allowing
them to be easily transferred to other domains. 2) The encoder of our proposed
model architecture can be exchanged in a modular manner, which enables our
model structure to be applied across multiple domains. 3) If the fine-tuning
data is diverse across a wide range of domains, our method is likely to learn
general language representations for open-domain multitask problems. Future
work could build upon this paper by applying the model and proposed schema to
different scientific domains where fine-tuning data might be sparse, such as
biology, physics and chemistry. Moreover, future work can build upon the
proposed schema by suggesting novel ways of modeling domain-specific or
general language that lead to improvements in unified multi-task learning.

## Limitations

One of the primary limitations of NLP modeling in materials science, including
this work, is the low quantity of available data as discussed in Section 2.
This analysis is affected by this limitation as well given that our
evaluations were performed in a low-data setting within a dataset that was
already limited in size. We believe that future work can improve upon this
study by applying larger datasets, both in the number of samples and in the
scope of tasks, to similar problem settings. The small nature of the datasets
applied in this study also presents the danger that some of the models may
have memorized certain answers instead of achieving a broader understanding,
which could be mitigated by enlarging the datasets and making the tasks more
complex.

Moreover, we did not study the generalization of NLP models beyond the
materials science domain, including adjacent domains such as chemistry and
physics. This targeted focus was intentional but imposes limitations on
whether the proposed techniques and insights we gained from our analysis are
transferable to other domains, including applying NLP models for scientific
tasks outside of materials science.

Another limitation of our study is the fact that we focused on BERT-based
models exclusively and did not study autoregressive models, including large
language models with billions of parameters highlighted in the introduction.
The primary reason for focusing on BERT-based models was the diversity of
available models trained on different scientific text corpora. Large
autoregressive models, on the other hand, are mostly trained on general text
corpora with some notable exceptions, such as Galactica (Taylor et al., 2022).
We believe that future work analyzing a greater diversity of language models,
including large autoregressive models pretrained on different kinds of text,
would significantly strengthen the understanding surrounding the ability of
NLP models to perform text-based tasks in materials science.

While the results presented in this study indicate that domain-specific
pretraining can lead to noticeable advantages in downstream performance on
text-based materials science tasks, we would like to highlight the associated
risks and costs of pretraining a larger set of customized language models for
different domains. The heavy financial and environmental costs associated with
these pretraining procedures merit careful consideration of what conditions
may warrant expensive pretraining and which ones may not. When possible, we
encourage future researchers to build upon existing large models to mitigate
the pretraining costs.

## Broader Impacts and Ethics Statement

Our MatSci-NLP benchmark can help promote the research on NLP for material
science, an important and growing research field. We expect that the
experience we gained from the material science domain can be transferred to
other domains, such as biology, health, and chemistry. Our Text-to-Schema also
helps with improving NLP tasks’ performance in low-resource situations, which
is a common challenge in many fields.

Our research does not raise major ethical concerns.

## Acknowlegments

This work is supported by the Mila internal funding - Program P2-V1: Industry
Sponsored Academic Labs (project number: 10379), the Canada CIFAR AI Chair
Program, and the Canada NSERC Discovery Grant (RGPIN-2021-03115).

## References

  * Balikas et al. (2015) Georgios Balikas, Anastasia Krithara, Ioannis Partalas, and George Paliouras. 2015\.  Bioasq: A challenge on large-scale biomedical semantic indexing and question answering.  In _International Workshop on Multimodal Retrieval in the Medical Domain_ , pages 26–39. Springer.
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  Scibert: A pretrained language model for scientific text.  _arXiv preprint arXiv:1903.10676_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Choudhary et al. (2022) Kamal Choudhary, Brian DeCost, Chi Chen, Anubhav Jain, Francesca Tavazza, Ryan Cohn, Cheol Woo Park, Alok Choudhary, Ankit Agrawal, Simon JL Billinge, et al. 2022.  Recent advances and applications of deep learning methods in materials science.  _npj Computational Materials_ , 8(1):1–26.
  * Dasigi et al. (2021) Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. 2021.  A dataset of information-seeking questions and answers anchored in research papers.  _arXiv preprint arXiv:2105.03011_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Friedrich et al. (2020) Annemarie Friedrich, Heike Adel, Federico Tomazic, Johannes Hingerl, Renou Benteau, Anika Maruscyk, and Lukas Lange. 2020.  The sofc-exp corpus and neural approaches to information extraction in the materials science domain.  _arXiv preprint arXiv:2006.03039_.
  * Georgescu et al. (2021) Alexandru B Georgescu, Peiwen Ren, Aubrey R Toland, Shengtong Zhang, Kyle D Miller, Daniel W Apley, Elsa A Olivetti, Nicholas Wagner, and James M Rondinelli. 2021.  Database, features, and machine learning model to identify thermally driven metal–insulator transition compounds.  _Chemistry of Materials_ , 33(14):5591–5605.
  * Gu et al. (2021) Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021.  Domain-specific language model pretraining for biomedical natural language processing.  _ACM Transactions on Computing for Healthcare (HEALTH)_ , 3(1):1–23.
  * Gupta et al. (2022) Tanishq Gupta, Mohd Zaki, NM Krishnan, et al. 2022.  Matscibert: A materials domain language model for text mining and information extraction.  _npj Computational Materials_ , 8(1):1–11.
  * Hakala and Pyysalo (2019) Kai Hakala and Sampo Pyysalo. 2019.  Biomedical named entity recognition with multilingual bert.  In _Proceedings of the 5th workshop on BioNLP open shared tasks_ , pages 56–61.
  * Hong et al. (2022) Zhi Hong, Aswathy Ajith, Gregory Pauloski, Eamon Duede, Carl Malamud, Roger Magoulas, Kyle Chard, and Ian Foster. 2022.  Scholarbert: Bigger is not always better.  _arXiv preprint arXiv:2205.11342_.
  * Huang and Cole (2022) Shu Huang and Jacqueline M Cole. 2022.  Batterybert: A pretrained language model for battery database enhancement.  _Journal of Chemical Information and Modeling_.
  * Jensen et al. (2021) Zach Jensen, Soonhyoung Kwon, Daniel Schwalbe-Koda, Cecilia Paris, Rafael Gómez-Bombarelli, Yuriy Román-Leshkov, Avelino Corma, Manuel Moliner, and Elsa A Olivetti. 2021.  Discovering relationships between osdas and zeolites through data mining and generative neural networks.  _ACS central science_ , 7(5):858–867.
  * Jin et al. (2019) Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. 2019.  Pubmedqa: A dataset for biomedical research question answering.  _arXiv preprint arXiv:1909.06146_.
  * Karpovich et al. (2021) Christopher Karpovich, Zach Jensen, Vineeth Venugopal, and Elsa Olivetti. 2021.  Inorganic synthesis reaction condition prediction with generative machine learning.  _arXiv preprint arXiv:2112.09612_.
  * Kim et al. (2020) Edward Kim, Zach Jensen, Alexander van Grootel, Kevin Huang, Matthew Staib, Sheshera Mysore, Haw-Shiuan Chang, Emma Strubell, Andrew McCallum, Stefanie Jegelka, et al. 2020.  Inorganic materials synthesis planning with literature-trained neural networks.  _Journal of chemical information and modeling_ , 60(3):1194–1201.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Kononova et al. (2021) Olga Kononova, Tanjin He, Haoyan Huo, Amalie Trewartha, Elsa A Olivetti, and Gerbrand Ceder. 2021.  Opportunities and challenges of text mining in materials research.  _Iscience_ , 24(3):102155.
  * Kuniyoshi et al. (2020) Fusataka Kuniyoshi, Kohei Makino, Jun Ozawa, and Makoto Miwa. 2020.  Annotating and extracting synthesis process of all-solid-state batteries from scientific literature.  _arXiv preprint arXiv:2002.07339_.
  * Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , 36(4):1234–1240.
  * Lin et al. (2017) Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017\.  Focal loss for dense object detection.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2980–2988.
  * Lu et al. (2021) Yaojie Lu, Hongyu Lin, Jin Xu, Xianpei Han, Jialong Tang, Annan Li, Le Sun, Meng Liao, and Shaoyi Chen. 2021.  Text2event: Controllable sequence-to-structure generation for end-to-end event extraction.  _arXiv preprint arXiv:2106.09232_.
  * Luong et al. (2015) Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2015\.  Multi-task sequence to sequence learning.  _arXiv preprint arXiv:1511.06114_.
  * Ma et al. (2018) Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. 2018.  Modeling task relationships in multi-task learning with multi-gate mixture-of-experts.  In _Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining_, pages 1930–1939.
  * Mahbub et al. (2020) Rubayyat Mahbub, Kevin Huang, Zach Jensen, Zachary D Hood, Jennifer LM Rupp, and Elsa A Olivetti. 2020.  Text mining for processing conditions of solid-state battery electrolytes.  _Electrochemistry Communications_ , 121:106860.
  * MatSciRE (2022) MatSciRE. 2022.  Material science relation extraction (matscire).
  * (28) Santiago Miret, Marta Skreta, Benjamin Sanchez-Lengelin, Shyue Ping Ong, Zamyla Morgan-Chan, and Alan Aspuru-Guzik.  Ai4mat - neurips 2022.
  * Mysore et al. (2019) Sheshera Mysore, Zach Jensen, Edward Kim, Kevin Huang, Haw-Shiuan Chang, Emma Strubell, Jeffrey Flanigan, Andrew McCallum, and Elsa Olivetti. 2019.  The materials science procedural text corpus: Annotating materials synthesis procedures with shallow semantic structures.  _arXiv preprint arXiv:1905.06939_.
  * Olivetti et al. (2020) Elsa A Olivetti, Jacqueline M Cole, Edward Kim, Olga Kononova, Gerbrand Ceder, Thomas Yong-Jin Han, and Anna M Hiszpanski. 2020.  Data-driven materials research enabled by natural language processing and information extraction.  _Applied Physics Reviews_ , 7(4):041317.
  * Peng et al. (2019) Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019.  Transfer learning in biomedical natural language processing: An evaluation of bert and elmo on ten benchmarking datasets.  In _Proceedings of the 2019 Workshop on Biomedical Natural Language Processing (BioNLP 2019)_.
  * Phan et al. (2021) Long N Phan, James T Anibal, Hieu Tran, Shaurya Chanana, Erol Bahadroglu, Alec Peltekian, and Grégoire Altan-Bonnet. 2021.  Scifive: a text-to-text transformer model for biomedical literature.  _arXiv preprint arXiv:2106.03598_.
  * Pilania (2021) Ghanshyam Pilania. 2021.  Machine learning in materials science: From explainable predictions to autonomous design.  _Computational Materials Science_ , 193:110360.
  * Qu et al. (2019) Chen Qu, Liu Yang, Minghui Qiu, W Bruce Croft, Yongfeng Zhang, and Mohit Iyyer. 2019\.  Bert with history answer embedding for conversational question answering.  In _Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval_ , pages 1133–1136.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21(140):1–67.
  * Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022.  Bloom: A 176b-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_.
  * Shin et al. (2020) Hoo-Chang Shin, Yang Zhang, Evelina Bakhturina, Raul Puri, Mostofa Patwary, Mohammad Shoeybi, and Raghav Mani. 2020.  Biomegatron: Larger biomedical domain language model.  _arXiv preprint arXiv:2010.06060_.
  * Taylor et al. (2022) Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022\.  Galactica: A large language model for science.  _arXiv preprint arXiv:2211.09085_.
  * Van Nguyen et al. (2022) Minh Van Nguyen, Bonan Min, Franck Dernoncourt, and Thien Nguyen. 2022.  Joint extraction of entities, relations, and events via modeling inter-instance and inter-label dependencies.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4363–4374.
  * Venugopal et al. (2021) Vineeth Venugopal, Sourav Sahoo, Mohd Zaki, Manish Agarwal, Nitya Nand Gosvami, and NM Anoop Krishnan. 2021.  Looking through glass: Knowledge discovery from materials science literature using natural language processing.  _Patterns_ , 2(7):100290.
  * Wada et al. (2020) Shoya Wada, Toshihiro Takeda, Shiro Manabe, Shozo Konishi, Jun Kamohara, and Yasushi Matsumura. 2020.  A pre-training technique to localize medical bert and enhance biobert.
  * Walker et al. (2021) Nicholas Walker, Amalie Trewartha, Haoyan Huo, Sanghoon Lee, Kevin Cruse, John Dagdelen, Alexander Dunn, Kristin Persson, Gerbrand Ceder, and Anubhav Jain. 2021\.  The impact of domain-specific pre-training on named entity recognition tasks in materials science.  _Available at SSRN 3950755_.
  * Wang et al. (2022a) Zheren Wang, Kevin Cruse, Yuxing Fei, Ann Chia, Yan Zeng, Haoyan Huo, Tanjin He, Bowen Deng, Olga Kononova, and Gerbrand Ceder. 2022a.  Ulsa: Unified language of synthesis actions for the representation of inorganic synthesis protocols.  _Digital Discovery_.
  * Wang et al. (2022b) Zheren Wang, Olga Kononova, Kevin Cruse, Tanjin He, Haoyan Huo, Yuxing Fei, Yan Zeng, Yingzhi Sun, Zijian Cai, Wenhao Sun, et al. 2022b.  Dataset of solution-based inorganic materials synthesis procedures extracted from the scientific literature.  _Scientific Data_ , 9(1):1–11.
  * Weston et al. (2019) Leigh Weston, Vahe Tshitoyan, John Dagdelen, Olga Kononova, Amalie Trewartha, Kristin A Persson, Gerbrand Ceder, and Anubhav Jain. 2019.  Named entity recognition and normalization applied to large-scale information extraction from the materials science literature.  _Journal of chemical information and modeling_ , 59(9):3692–3702.
  * Wu and He (2019) Shanchan Wu and Yifan He. 2019.  Enriching pre-trained language model with entity information for relation classification.  In _Proceedings of the 28th ACM international conference on information and knowledge management_ , pages 2361–2364.
  * Yamaguchi et al. (2020) Kyosuke Yamaguchi, Ryoji Asahi, and Yutaka Sasaki. 2020.  Sc-comics: a superconductivity corpus for materials informatics.  In _Proceedings of The 12th Language Resources and Evaluation Conference_ , pages 6753–6760.

## Appendix

##  Appendix A Experimental Details

We performed fine-tuning experiments using a single GPU with a learning rate
was 2e-5, the hidden size of the encoders being 768, except ScholarBERT which
is 1024, using the Adam Kingma and Ba (2014) optimizer for a max number of 20
training epochs with early stopping. All models are implemented with Python
and PyTorch, and repeated five times to report the average performance. The
full set of hyperparameters will be provided in our code release upon
publication.

##  Appendix B Additional Text-to-Schema Experiments

To arrive at our data presented in Table 3, we conducted experiments for all
the language models across all tasks in MatSci-NLP. The results for seven
tasks in MatSci-NLP are shown in subsequent tables:

  * •

Named Entity Recognition in Table 4.

  * •

Relation Classification in Table 5.

  * •

Event Argument Extraction in Table 6.

  * •

Paragraph Classification in Table 7.

  * •

Synthesis Action Retrieval in Table 8.

  * •

Sentence Classification in Table 9.

  * •

Slot Filling in Table 10.

The experimental results summarized in the aforementioned tables reinforce the
conclusions in our analysis of (Q2) in Section 5.2 with the text-to-schema
based fine-tuning method generally outperforming the conventional single and
multitask methods across all tasks and all language models.

NLP Model Single Task Single Task Prompt MMOE No Explanations Potential
Choices Examples Text2Schema MatSciBERT (Gupta et al., 2022) 0.690±0.018
0.403±0.029 0.707±0.089 0.445±0.071 0.451±0.114 0.188±0.065 0.655±0.066
0.410±0.051 0.732±0.048 0.480±0.087 0.753±0.060 0.505±0.066 0.707±0.076
0.470±0.092 MatBERT (Walker et al., 2021) 0.705±0.011 0.469±0.037 0.796±0.029
0.558±0.044 0.691±0.060 0.400±0.070 0.805±0.018 0.574±0.061 0.756±0.071
0.524±0.088 0.778±0.015 0.547±0.039 0.798±0.031 0.569±0.055 BatteryBERT (Huang
and Cole, 2022) 0.690±0.014 0.464±0.018 0.673±0.029 0.407±0.045 0.439±0.185
0.168±0.110 0.733±0.026 0.483±0.049 0.607±0.169 0.369±0.140 0.743±0.015
0.497±0.015 0.722±0.045 0.470±0.043 SciBERT (Beltagy et al., 2019) 0.686±0.015
0.464±0.035 0.754±0.029 0.493±0.063 0.598±0.027 0.298±0.048 0.708±0.115
0.465±0.115 0.724±0.045 0.471±0.069 0.754±0.054 0.509±0.064 0.734±0.079
0.497±0.091 ScholarBERT (Hong et al., 2022) 0.206±0.350 0.069±0.131
0.179±0.088 0.108±0.057 0.109±0.142 0.018±0.033 0.134±0.036 0.071±0.023
0.263±0.109 0.122±0.073 0.168±0.044 0.098±0.045 0.168±0.067 0.101±0.034
BioBERT (Wada et al., 2020) 0.665±0.018 0.403±0.030 0.708±0.119 0.431±0.115
0.204±0.114 0.019±0.000 0.723±0.075 0.474±0.071 0.455±0.114 0.188±0.065
0.725±0.024 0.452±0.044 0.715±0.031 0.459±0.055 BERT (Devlin et al., 2018)
0.606±0.009 0.304±0.024 0.636±0.034 0.382±0.041 0.235±0.069 0.055±0.040
0.670±0.056 0.441±0.060 0.455±0.138 0.267±0.089 0.664±0.047 0.418±0.046
0.657±0.079 0.416±0.058

Table 4: Results of named entity recognition task among seven tasks on
different schema settings for various BERT models pre-trained on different
domain specific text data. For each model, the top line represents the
micro-F1 score and the bottom line represents the macro-F1 score. We report
the mean across 5 experiments with a confidence interval of two standard
deviations. We highlight the best performing method.

NLP Model Single Task Single Task Prompt MMOE No Explanations Potential
Choices Examples Text2Schema MatSciBERT (Gupta et al., 2022) 0.671±0.083
0.439±0.137 0.545±0.102 0.219±0.035 0.490±0.139 0.218±0.073 0.747±0.128
0.461±0.190 0.800±0.058 0.482±0.064 0.818±0.137 0.530±0.203 0.791±0.046
0.507±0.073 MatBERT (Walker et al., 2021) 0.714±0.023 0.487±0.075 0.644±0.050
0.310±0.078 0.591±0.267 0.297±0.143 0.871±0.020 0.623±0.035 0.804±0.071
0.513±0.138 0.848±0.045 0.569±0.019 0.875±0.015 0.630±0.047 BatteryBERT (Huang
and Cole, 2022) 0.594±0.085 0.359±0.075 0.592±0.084 0.297±0.025 0.423±0.097
0.167±0.074 0.823±0.073 0.553±0.074 0.801±0.081 0.466±0.111 0.854±0.029
0.592±0.066 0.786±0.113 0.472±0.150 SciBERT (Beltagy et al., 2019) 0.699±0.105
0.495±0.099 0.585±0.125 0.267±0.042 0.643±0.088 0.311±0.098 0.799±0.139
0.527±0.204 0.783±0.085 0.474±0.099 0.814±0.125 0.528±0.180 0.819±0.067
0.545±0.119 ScholarBERT (Hong et al., 2022) 0.603±0.179 0.178±0.186
0.619±0.248 0.384±0.154 0.243±0.351 0.078±0.139 0.416±0.013 0.334±0.006
0.543±0.060 0.252±0.062 0.367±0.080 0.236±0.119 0.428±0.148 0.274±0.110
BioBERT (Wada et al., 2020) 0.692±0.105 0.458±0.087 0.538±0.108 0.243±0.029
0.306±0.032 0.079±0.017 0.743±0.199 0.442±0.215 0.674±0.093 0.323±0.092
0.666±0.220 0.324±0.118 0.797±0.092 0.465±0.134 BERT (Devlin et al., 2018)
0.564±0.130 0.357±0.076 0.626±0.103 0.306±0.075 0.368±0.112 0.100±0.018
0.792±0.056 0.533±0.041 0.696±0.046 0.382±0.039 0.636±0.094 0.382±0.043
0.782±0.056 0.494±0.061

Table 5: Results of relation classification task among seven tasks on
different schema settings for various BERT models pre-trained on different
domain specific text data. For each model, the top line represents the
micro-F1 score and the bottom line represents the macro-F1 score. We report
the mean across 5 experiments with a confidence interval of two standard
deviations. We highlight the best performing method.

NLP Model Single Task Single Task Prompt MMOE No Explanations Potential
Choices Examples Text2Schema MatSciBERT (Gupta et al., 2022) 0.108±0.062
0.041±0.020 0.148±0.182 0.050±0.071 0.280±0.127 0.122±0.063 0.448±0.091
0.251±0.075 0.498±0.045 0.310±0.036 0.484±0.015 0.292±0.052 0.436±0.066
0.251±0.075 MatBERT (Walker et al., 2021) 0.152±0.093 0.029±0.021 0.160±0.169
0.033±0.033 0.341±0.006 0.174±0.027 0.453±0.108 0.274±0.087 0.483±0.063
0.298±0.037 0.515±0.040 0.288±0.064 0.451±0.091 0.288±0.066 BatteryBERT (Huang
and Cole, 2022) 0.149±0.072 0.030±0.039 0.162±0.166 0.036±0.029 0.232±0.196
0.104±0.088 0.397±0.105 0.233±0.086 0.438±0.063 0.298±0.037 0.443±0.023
0.250±0.068 0.457±0.024 0.277±0.034 SciBERT (Beltagy et al., 2019) 0.152±0.123
0.041±0.068 0.160±0.189 0.033±0.032 0.312±0.015 0.159±0.024 0.449±0.079
0.259±0.072 0.442±0.135 0.264±0.103 0.484±0.042 0.287±0.075 0.451±0.077
0.276±0.080 ScholarBERT (Hong et al., 2022) 0.349±0.102 0.250±0.101
0.444±0.091 0.253±0.103 0.262±0.062 0.102±0.108 0.454±0.094 0.312±0.131
0.454±0.095 0.264±0.102 0.431±0.081 0.296±0.144 0.489±0.083 0.356±0.109
BioBERT (Wada et al., 2020) 0.119±0.080 0.030±0.011 0.160±0.170 0.034±0.032
0.054±0.000 0.013±0.000 0.489±0.058 0.305±0.090 0.491±0.027 0.295±0.059
0.473±0.034 0.268±0.061 0.488±0.036 0.274±0.049 BERT (Devlin et al., 2018)
0.198±0.041 0.042±0.055 0.160±0.170 0.033±0.033 0.232±0.002 0.049±0.008
0.400±0.017 0.194±0.025 0.414±0.064 0.214±0.092 0.451±0.074 0.265±0.104
0.418±0.053 0.225±0.091

Table 6: Results of event argument extraction task among seven tasks on
different schema settings for various BERT models pre-trained on different
domain specific text data. For each model, the top line represents the
micro-F1 score and the bottom line represents the macro-F1 score. We report
the mean across 5 experiments with a confidence interval of two standard
deviations. We highlight the best performing method.

NLP Model Single Task Single Task Prompt MMOE No Explanations Potential
Choices Examples Text2Schema MatSciBERT (Gupta et al., 2022) 0.685±0.074
0.588±0.152 0.673±0.003 0.402±0.001 0.607±0.277 0.386±0.150 0.706±0.013
0.633±0.115 0.694±0.041 0.524±0.175 0.686±0.158 0.583±0.226 0.719±0.116
0.623±0.183 MatBERT (Walker et al., 2021) 0.753±0.031 0.730±0.016 0.671±0.002
0.402±0.001 0.673±0.001 0.404±0.004 0.727±0.089 0.601±0.212 0.776±0.059
0.722±0.076 0.649±0.039 0.509±0.155 0.756±0.073 0.691±0.188 BatteryBERT (Huang
and Cole, 2022) 0.663±0.088 0.585±0.156 0.672±0.001 0.402±0.000 0.672±0.002
0.402±0.001 0.621±0.160 0.564±0.180 0.626±0.113 0.574±0.092 0.672±0.031
0.540±0.129 0.633±0.075 0.610±0.046 SciBERT (Beltagy et al., 2019) 0.690±0.074
0.605±0.150 0.673±0.002 0.402±0.001 0.568±0.289 0.370±0.089 0.703±0.041
0.598±0.204 0.711±0.076 0.598±0.203 0.662±0.169 0.562±0.202 0.696±0.094
0.546±0.243 ScholarBERT (Hong et al., 2022) 0.620±0.161 0.386±0.150
0.603±0.271 0.371±0.122 0.658±0.029 0.407±0.010 0.672±0.003 0.482±0.001
0.662±0.144 0.534±0.260 0.668±0.016 0.405±0.007 0.663±0.032 0.433±0.122
BioBERT (Wada et al., 2020) 0.629±0.041 0.507±0.033 0.672±0.002 0.402±0.001
0.671±0.001 0.401±0.001 0.658±0.211 0.588±0.258 0.709±0.033 0.651±0.081
0.680±0.193 0.622±0.226 0.675±0.144 0.578±0.102 BERT (Devlin et al., 2018)
0.709±0.090 0.585±0.093 0.672±0.001 0.468±0.283 0.672±0.003 0.402±0.001
0.685±0.050 0.562±0.221 0.727±0.102 0.602±0.238 0.629±0.291 0.468±0.283
0.665±0.057 0.532±0.194

Table 7: Results of paragraph classification task among seven tasks on
different schema settings for various BERT models pre-trained on different
domain specific text data. For each model, the top line represents the
micro-F1 score and the bottom line represents the macro-F1 score. We report
the mean across 5 experiments with a confidence interval of two standard
deviations. We highlight the best performing method.

NLP Model Single Task Single Task Prompt MMOE No Explanations Potential
Choices Examples Text2Schema MatSciBERT (Gupta et al., 2022) 0.383±0.024
0.082±0.009 0.334±0.004 0.063±0.001 0.424±0.249 0.169±0.096 0.676±0.071
0.505±0.094 0.631±0.081 0.445±0.153 0.741±0.157 0.549±0.179 0.692±0.179
0.484±0.254 MatBERT (Walker et al., 2021) 0.346±0.006 0.067±0.004 0.334±0.001
0.063±0.000 0.549±0.087 0.300±0.045 0.792±0.073 0.653±0.184 0.669±0.061
0.497±0.086 0.744±0.010 0.557±0.082 0.717±0.040 0.549±0.091 BatteryBERT (Huang
and Cole, 2022) 0.280±0.004 0.118±0.041 0.334±0.001 0.063±0.000 0.311±0.062
0.073±0.028 0.670±0.046 0.496±0.117 0.558±0.179 0.358±0.149 0.492±0.181
0.282±0.184 0.614±0.128 0.419±0.149 SciBERT (Beltagy et al., 2019) 0.281±0.009
0.052±0.027 0.334±0.001 0.063±0.001 0.455±0.081 0.207±0.095 0.727±0.114
0.564±0.137 0.623±0.069 0.456±0.135 0.740±0.133 0.533±0.160 0.701±0.138
0.516±0.217 ScholarBERT (Hong et al., 2022) 0.437±0.104 0.193±0.076
0.489±0.105 0.266±0.105 0.330±0.007 0.070±0.015 0.389±0.001 0.190±0.000
0.492±0.165 0.308±0.156 0.389±0.001 0.191±0.001 0.322±0.260 0.178±0.051
BioBERT (Wada et al., 2020) 0.300±0.015 0.073±0.002 0.324±0.001 0.062±0.000
0.334±0.062 0.073±0.027 0.662±0.060 0.426±0.078 0.561±0.128 0.346±0.133
0.545±0.157 0.347±0.128 0.647±0.140 0.446±0.231 BERT (Devlin et al., 2018)
0.348±0.047 0.091±0.020 0.334±0.001 0.063±0.000 0.313±0.083 0.073±0.037
0.668±0.061 0.495±0.058 0.593±0.059 0.424±0.086 0.594±0.081 0.371±0.103
0.656±0.099 0.515±0.067

Table 8: Results of synthesis action retrieval task among seven tasks on
different schema settings for various BERT models pre-trained on different
domain specific text data. For each model, the top line represents the
micro-F1 score and the bottom line represents the macro-F1 score. We report
the mean across 5 experiments with a confidence interval of two standard
deviations. We highlight the best performing method.

NLP Model Single Task Single Task Prompt MMOE No Explanations Potential
Choices Examples Text2Schema MatSciBERT (Gupta et al., 2022) 0.888±0.093
0.602±0.151 0.908±0.001 0.476±0.001 0.907±0.001 0.493±0.069 0.908±0.010
0.601±0.159 0.903±0.019 0.573±0.135 0.905±0.020 0.616±0.150 0.914±0.008
0.660±0.079 MatBERT (Walker et al., 2021) 0.908±0.011 0.441±0.038 0.908±0.001
0.476±0.001 0.907±0.000 0.476±0.000 0.906±0.016 0.645±0.025 0.910±0.012
0.561±0.135 0.903±0.018 0.600±0.089 0.909±0.009 0.614±0.134 BatteryBERT (Huang
and Cole, 2022) 0.908±0.012 0.452±0.045 0.907±0.000 0.475±0.001 0.908±0.000
0.476±0.000 0.895±0.050 0.679±0.080 0.890±0.036 0.685±0.074 0.907±0.002
0.519±0.144 0.912±0.015 0.684±0.095 SciBERT (Beltagy et al., 2019) 0.896±0.080
0.421±0.159 0.907±0.000 0.469±0.004 0.825±0.218 0.535±0.079 0.908±0.009
0.586±0.166 0.902±0.017 0.596±0.161 0.902±0.020 0.623±0.130 0.911±0.017
0.617±0.143 ScholarBERT (Hong et al., 2022) 0.805±0.020 0.458±0.099
0.839±0.268 0.477±0.004 0.908±0.001 0.485±0.000 0.908±0.000 0.476±0.000
0.900±0.019 0.509±0.093 0.907±0.001 0.476±0.001 0.906±0.007 0.478±0.008
BioBERT (Wada et al., 2020) 0.908±0.001 0.476±0.001 0.907±0.001 0.478±0.001
0.907±0.001 0.503±0.005 0.910±0.012 0.614±0.175 0.899±0.047 0.610±0.078
0.908±0.015 0.638±0.089 0.915±0.021 0.686±0.098 BERT (Devlin et al., 2018)
0.911±0.010 0.475±0.036 0.907±0.000 0.476±0.000 0.907±0.001 0.476±0.000
0.906±0.007 0.549±0.086 0.905±0.010 0.581±0.153 0.892±0.035 0.563±0.136
0.910±0.016 0.633±0.133

Table 9: Results of sentence classification task among seven tasks on
different schema settings for various BERT models pre-trained on different
domain specific text data. For each model, the top line represents the
micro-F1 score and the bottom line represents the macro-F1 score. We report
the mean across 5 experiments with a confidence interval of two standard
deviations. We highlight the best performing method.

NLP Model Single Task Single Task Prompt MMOE No Explanations Potential
Choices Examples Text2Schema MatSciBERT (Gupta et al., 2022) 0.083±0.047
0.087±0.045 0.086±0.072 0.010±0.011 0.043±0.023 0.016±0.005 0.419±0.074
0.182±0.043 0.433±0.121 0.169±0.069 0.428±0.187 0.169±0.075 0.436±0.142
0.194±0.062 MatBERT (Walker et al., 2021) 0.179±0.074 0.087±0.030 0.151±0.121
0.024±0.022 0.148±0.148 0.057±0.067 0.547±0.050 0.276±0.047 0.493±0.078
0.230±0.067 0.502±0.034 0.221±0.011 0.548±0.058 0.273±0.051 BatteryBERT (Huang
and Cole, 2022) 0.093±0.074 0.009±0.012 0.073±0.033 0.008±0.011 0.032±0.031
0.008±0.009 0.540±0.092 0.270±0.108 0.433±0.155 0.211±0.056 0.506±0.065
0.236±0.072 0.520±0.057 0.262±0.073 SciBERT (Beltagy et al., 2019) 0.098±0.054
0.020±0.021 0.099±0.075 0.013±0.018 0.125±0.073 0.047±0.016 0.469±0.112
0.207±0.066 0.432±0.106 0.183±0.061 0.446±0.167 0.179±0.071 0.481±0.144
0.224±0.010 ScholarBERT (Hong et al., 2022) 0.286±0.042 0.110±0.009
0.289±0.044 0.111±0.019 0.063±0.007 0.005±0.004 0.323±0.058 0.111±0.027
0.276±0.080 0.076±0.024 0.338±0.053 0.117±0.015 0.296±0.085 0.109±0.044
BioBERT (Wada et al., 2020) 0.096±0.171 0.023±0.020 0.094±0.118 0.015±0.024
0.042±0.024 0.004±0.001 0.517±0.031 0.241±0.082 0.319±0.059 0.110±0.048
0.424±0.145 0.177±0.119 0.452±0.114 0.191±0.045 BERT (Devlin et al., 2018)
0.086±0.032 0.011±0.005 0.082±0.065 0.012±0.018 0.034±0.026 0.005±0.006
0.566±0.042 0.306±0.073 0.421±0.137 0.204±0.078 0.476±0.079 0.225±0.066
0.520±0.019 0.257±0.022

Table 10: Results of slot filling task among seven tasks on different schema
settings for various BERT models pre-trained on different domain specific text
data. For each model, the top line represents the micro-F1 score and the
bottom line represents the macro-F1 score. We report the mean across 5
experiments with a confidence interval of two standard deviations. We
highlight the best performing method.


# Skilful Precipitation Nowcasting Using NowcastNet

Ajitabh Kumar
Enrflo
Noida, UP 201301 India
<EMAIL_ADDRESS>

###### Abstract

Designing early warning system for precipitation requires accurate short-term
forecasting system. Climate change has led to an increase in frequency of
extreme weather events, and hence such systems can prevent disasters and loss
of life. Managing such events remain a challenge for both public and private
institutions. Precipitation nowcasting can help relevant institutions to
better prepare for such events as they impact agriculture, transport, public
health and safety, etc. Physics-based numerical weather prediction (NWP) is
unable to perform well for nowcasting because of large computational turn-
around time. Deep-learning based models on the other hand are able to give
predictions within seconds. We use recently proposed NowcastNet, a physics-
conditioned deep generative network, to forecast precipitation for different
regions of Europe using satellite images. Both spatial and temporal transfer
learning is done by forecasting for the unseen regions and year. Model makes
realistic predictions and is able to outperform baseline for such a prediction
task.

##  1 Introduction

Precipitation forecasting is important for planning day-to-day activities as
well as mitigating disasters. Climate change brings new set of challenges for
government as extreme weather events become more frequent. Nowcasting for 0-8
hour range can help planners navigate these challenges but traditional NWP
based approaches have relatively large turn around time. State-of-the-art NWP
methods typically advect precipitation fields with radar-based wind estimates,
and fail to capture convective initiations. Recently deep learning based
approaches have been used in nowcasting as they can yield predictions within
seconds using available satellite and radar images. Different studies have
shown that deep learning based approaches can even outperform NWP in
nowcasting problem.

Weather4Cast $2023$ challenge is organized in conjunction with NeurIPS $2023$
conference for developing and comparing different deep learning approaches for
solving such a nowcasting problem. Training data is available for seven
different regions of Europe for two years (i.e. $2019$ and $2020$). Goal of
challenge is to predict the precipitation for next $8$ hours given satellite
images for last $1$ hour. Next, spatial transfer learning problem involves
forecasting precipitation for another three regions of Europe for which no
training data is provided. Finally, temporal transfer learning problem
involves predicting precipitation for all ten regions for the third year (i.e.
$2021$).

Precipitation forecasting is challenging as it is highly heterogeneous event
and requires dense prediction. Smooth and blurry prediction can easily
diminish forecast usability, and so rain intensities must be captured. Core
challenge of this competition is to create rain movie at $15$ minute interval
for the next $8$ hour. NowcastNet incorporates both physics and data to make
realistic precipitation forecast, and the code repository is available at
$https://github.com/ajikmr/w4c\\_2023$.

##  2 Method

In this section, we first discuss available training data and then we discuss
the deep learning architecture used.

###  2.1 Dataset

Ground-radar reflectivity measurements are used to calculate pan-European
composite rainfall rates by the Operational Program for Exchange of Weather
Radar Information (OPERA) radar network. Even though these are more precise,
accurate, and of higher resolution than satellite data, they are expensive to
obtain and not available in many parts of the world. Hence, goal of this
competition is to develop deep learning based models to predict rain rates
from radiations measured by geostationary satellites, which are operated by
the European Organisation for the Exploitation of Meteorological Satellites
(EUMETSAT).

Satellite based images with $11$ different channels are available for larger
context region of area $3024\times 3024$ km. Each image contains data point
for every grid size of $12\times 12$ km, and hence these images are of
$252\times 252$ dimensions. Channels denote slightly noisy satellite radiance
covering so-called $2$ visible (VIS), $2$ water vapor (WV), and $7$ infrared
(IR) bands. Precipitation data on the other hand is available for the central
$504\times 504$ km area with a grid size of $2\times 2$ km. Thus, even
precipitation images are of $252\times 252$ dimension. Figure 1 shows the
larger context area with satellite data and the central area with OPERA based
precipitation data for one of the regions. Input data is cleaned in the pre-
processing step in which all unavailable values are replaced with $0$. Input
values are normalized so that different channels could be used together.

Figure 1: Central area of interest within larger context area for a region
under study

###  2.2 NowcastNet

Deep learning core task is to predict high-resolution ($2\times 2$ km)
precipitation values using low resolution ($12\times 12$ km) satellite images.
NowcastNet has two main modules: an evolution network which learns underlying
physics using neural operators, and a generative network which makes final
prediction using context from the evolution network (Figure 2). Noise
projector is added in generative network which transforms the latent vector
into required spatial size. Thus, the model includes a deterministic evolution
network parameterized by $\phi$ and a stochastic generative network
parameterized by $\theta$. Nowcasting procedure is thus physics-conditional
generation from latent random vectors z, given as:

$P(\hat{X}_{1:T}|Y_{-T_{0}:0},\phi;\theta)=\int
P(\hat{X}_{1:T}|Y_{-T_{0}:0},\phi(Y_{-T_{0}:0}),z;\theta)P(z)dz$ (1)

where $Y_{-T_{0}:0}$ denotes given past satellite images and $\hat{X}_{1:T}$
denotes predicted precipitation image.

Evolution network is a two path U-Net with shared evolution encoder for
learning context representation (Figure 3). Motion decoder learns the two-
dimensional velocities for x- and y-directions, while intensity decoder learns
intensity residuals for the desired times. Spectral normalization technique is
applied for each convolutional layer. Input and output values are concatenated
on temporal dimension for skip connection.

Generative network conditions on the evolution network predictions
$X^{{}^{\prime\prime}}_{1:T}$, and takes low-resolution past satellite images
as input $X_{-T_{0}:0}$. It generates final precipitation field from latent
random vectors $z$. The backbone of this network is also a U-Net encoder-
decoder structure. The encoder has same structure as the evolution encoder,
and takes as input concatenation of $X_{-T_{0}:0}$ and
$X^{{}^{\prime\prime}}_{1:T}$. The decoder takes contextual input from
encoder, along with the transformation of the latent Gaussian vector $z$.
Noise projector takes this latent vector $z$ as input and transforms it to the
same spatial size as the encoder context.

Figure 2: NowcastNet architecture: a physics conditional deep generative
network Figure 3: Evolution network and operator

##  3 Results

Original input data consists of low-resolution satellite image with $11$
different channels for a larger context area, while high-resolution prediction
is done for a smaller central area. We use Adam optimizer and cosine annealing
learning rate with linear warm-up. Mean squared error is used as loss function
in the initial epochs, which is changed to weighted mean absolute error in the
later epochs. Pixel-wise weight is calculated as $min(1+y,24)$ where $y$ is
the target precipitation rate. This helps in learning less frequent but high
value precipitation events, and ensuring that network does not only fit to
dominant medium to low value precipitation events.

For the core challenge, input data consists of $4$ past values at $15$-minute
interval (i.e. past $1$ hour), while prediction is made for next $32$ future
times at $15$-minute interval (i.e. next $8$ hours). Next for the nowcasting
and transfer-learning challenge, input data consists of $4$ past values at
$15$-minute interval (i.e. past $1$ hour), while prediction is made only for
the next $16$ future times at $15$-minute interval (i.e. next $4$ hours).
First, we use the previously learnt model and fine-tune it for the $16$ output
values. Next we also train a model from scratch with only $16$ output values.

All different approaches yield similar result and some sample prediction
images are shown next. Figures 4 and 5 show the actual and predicted rainfall
values for two cases. As we can see, nowcastnet based model is able to capture
rain events. More improvement could be made in result by pre-processing data
appropriately to capture logarithmic trend and non-stationarity in data.
Nowcastnet performs well and captures both advective and convective effects
using evolution and generative networks.

Figure 4: Actual and predicted rainfall values for different prediction times
(case 1) Figure 5: Actual and predicted rainfall values for different
prediction times (case 2)

##  4 Conclusions

NowcastNet architecture based neural network model is trained for
precipitation nowcasting. Evolution network of the model learns underlying
physics using neural operators, while generative network which makes final
prediction using context from the evolution network. Model is able to predict
rainfall events using satellite based images.

## Acknowledgments and Disclosure of Funding

This research is made possible by generous cloud credits provided by OVH
Cloud.

## References

[1] Asperti, A., Merizzi, F., Paparella, A., Pedrazzi, G., Angelinelli, M. &
Colamonaco, S. (2023) Precipitation nowcasting with generative diffusion
models. arXiv preprint arXiv:2308.06733.

[2] Zhang, Y., Long, M., Chen, K.Xing, L, Jin, R., Jordan, M.I. & Wang, J.
(2023) Skilful nowcasting of extreme precipitation with NowcastNet. Nature
619:526–532.

[3] Ravuri, S., Lenc, K., Willson, M. et al. (2021) Skilful precipitation
nowcasting using deep generative models of radar. Nature 597:672–677.

[4] Pihrt, J., Raevskiy, R., Šimánek, P. & Choma, M. (2022) WeatherFusionNet:
Predicting precipitation from satellite data. arXiv preprint arXiv:2211.16824.

[5] Pulkkinen, S., Nerini, D., Pérez Hortal, A.A., Velasco-Forero, C., Seed,
A., Germann, U. & Foresti, L. (2019) Pysteps: an open-source Python library
for probabilistic precipitation nowcasting. Geoscientific Model Development
12(10):4185-4219.