# To FP8 and Back Again: Quantifying the Effects of Reducing Precision on LLM
Training Stability

Joonhyung Lee
NAVER Cloud
&Jeongin Bae
NAVER Cloud
Byeongwook Kim
NAVER Cloud
&Se Jung Kwon
NAVER Cloud
&Dongsoo Lee
NAVER Cloud
Correspondence to: Joonhyung Lee<EMAIL_ADDRESS>
###### Abstract

The massive computational costs associated with large language model (LLM)
pretraining have spurred great interest in reduced-precision floating-point
representations to accelerate the process. As a result, the BrainFloat16
(BF16) precision has become the de facto standard for LLM training, with
hardware support included in recent accelerators. This trend has gone even
further in the latest processors, where FP8 has recently been introduced.
However, prior experience with FP16, which was found to be less stable than
BF16, raises concerns as to whether FP8, with even fewer bits than FP16, can
be a cost-effective option for LLM training. We argue that reduced-precision
training schemes must have similar training stability and hyperparameter
sensitivities to their higher-precision counterparts in order to be cost-
effective. However, we find that currently available methods for FP8 training
are not robust enough to allow their use as economical replacements. This
prompts us to investigate the stability of reduced-precision LLM training in
terms of robustness across random seeds and learning rates. To this end, we
propose new evaluation techniques and a new metric for quantifying loss
landscape sharpness in autoregressive language models. By simulating
incremental bit reductions in floating-point representations, we analyze the
relationship between representational power and training stability with the
intent of aiding future research into the field.

##  1 Introduction

Conversational large language models (LLMs), such as ChatGPT [1], Gemini [2,
3], Claude [4], Llama [5, 6], and HyperCLOVA [7], have captured the
imagination of both academics and the public at large with their ability to
communicate fluently with humans in natural language. However, these models
require unprecedented amounts of computation to train, which has engendered
interest in methods to improve training efficiency.

A popular method of improving computational performance is to reduce the bit
count of the floating-point representations used for training [8, 9, 10]. This
is because reading memory is the main bottleneck in modern processors, a
problem known as the “memory wall” [11, 12]. Reducing the number of bits that
each floating-point number uses can, therefore, accelerate the computation in
proportion to the amount of memory reduced. For example, in processors that
support it, computations in BrainFloat16 (BF16) [13] can have double the
maximum throughput of single precision FP32. Furthermore, the FP32 data type,
the highest precision data type used in deep learning, has only half the bits
of FP64, the most widely used floating-point data type in scientific
computing. The current best practice for LLM training is to use BF16 for most
of the LLM training computation, with some sensitive portions, such as layer
normalization [14], carried out in FP32.

As a natural extension of this development, 8-bit floating-point (FP8) [8, 15,
16, 10] and even 4-bit floating-point [9] data formats have been proposed to
accelerate training even further. However, the naïve application of FP8 to LLM
training is unstable and requires additional techniques to become viable.
While several methods have been proposed to stabilize training LLMs with FP8,
relatively little attention has been paid to quantifying the decrease in
training stability compared to the standard mixed-precision BF16 training.

As the motivation behind the use of FP8 and other reduced-precision training
schemes is cost reduction, our concern here is not whether LLM pretraining
with FP8 is possible but whether it is profitable. For cost savings to be
realized, the time per training step must be reduced while the number of
training steps is kept to a similar number. Training stability is thus a
crucial factor for cost-effective LLM training, considering that additional
hyperparameter searches and restarts from training failures can outweigh any
gains from the acceleration in raw computation. Therefore, for the newly
proposed reduced-precision training schemes to be economical, the models
trained on them must be similarly robust to hyperparameter choice and
stochastic noise as models trained using higher precision.

Previous experience with training LLMs in FP16 raises further concerns. Teams
that have trained LLMs have found that even when gradient scaling and other
techniques are applied, the FP16 data type, which has five exponent bits, is
much less stable for LLM training than BF16, which has eight exponent bits as
in FP32. This raises doubts as to whether FP8, which has even fewer bits than
FP16, is a practical option for real-world LLM training.

We motivate our line of inquiry with some surprising findings from experiments
on the nanoGPT [17] codebase, an open-source implementation of GPT-2
pretraining, where we found that even the current best practice of mixed-
precision BF16 can introduce training instabilities. When we compared BF16 and
TensorFloat32 (TF32) runs, where we ran training for 5% of the dataset for
each run, we found that the BF16 models diverged for 18 of 188 runs, or
approximately 10% of all cases, despite using the same configurations as the
default run. In contrast, no cases of loss divergence were found for the 70
TF32 models that were trained using different random seeds. We compare against
the TF32 data type because NVIDIA GPUs do not offer tensor cores in FP32.

This is a surprising finding in light of the fact that most recent LLMs are
trained with mixed precision BF16 without a comparison with training on TF32,
which has three additional mantissa bits. However, a loss divergence rate of
approximately 10% at only 5% of training indicates that even standard BF16
mixed-precision training may add non-trivial instability. If even mixed-
precision BF16 can cause instabilities, the effects of using even fewer bits
should be investigated further.

We make the following contributions in our work.

  * •

We analyze the effects of incremental bit reduction on training stability by
simulating intermediate-bit representations of floating-point numbers. As
observed in Kalamkar et al. [13], we confirm that reducing exponent bits has a
greater impact on model performance than reducing mantissa bits, finding that
removing even a single exponent bit on all layers precludes training
altogether for Llama models.

  * •

We propose a metric for quantifying the loss landscape sharpness of models
that can predict when training divergence will occur. As even the removal of
mantissa bits has a destabilizing effect on LLM training, we use our metric to
predict loss divergences even when the loss curve itself has not yet diverged.

Figure 1: We show three cases of loss divergence on nanoGPT when using the
same configurations as the default run except for the random seed. The blue
lines indicate the average losses obtained for eight training runs that did
not diverge. Of the 188 random seeds that were tested, 18 were found to
diverge. As full pretraining requires over 4 days on a single node with 8 A100
GPUs, even for BF16, we perform early stopping at 30K steps, or 5% of the
original training runs, requiring approximately 4 hours for a BF16 run and 8
hours for a TF32 run per A100 node with 8 GPUs. Because we only use 5% of the
training data, we suspect that the measured divergence rate of approximately
10% underestimates the true rate of training loss divergence.

##  2 Related work

###  2.1 Training stability

Analyzing training instability in LLM pretraining directly is impractical due
to the massive costs involved. Instead, smaller language models must be used
as proxies. Wortsman et al. [18] explore the robustness of smaller language
models as a proxy for LLM pretraining instability. They find that small models
using a larger learning rate show similar instability patterns as larger
models, showing unstable behavior, such as the growth of attention layer
logits and the divergence of the output logits from the log probabilities, can
be reproduced in smaller Transformers. They also explore both the causes of
numerical instability in LLM training and mitigating strategies such as
applying query/key (QK) normalization [19].

Keskar et al. [20] explores the sharpness of loss landscapes during large-
batch neural network training, finding that larger batch sizes prevent the
model from reaching flat regions of the loss landscape and causing performance
degradation due to the inability to escape local minima. Of most significance
to our work, they propose a metric for calculating loss landscape sharpness,
which we adapt for LLMs.

###  2.2 Reduced-precision processors

To improve throughput on computationally intensive matrix multiplication
tasks, recently developed processors have been equipped with specialized
hardware units such as systolic arrays for TPUs [21] and tensor cores [22],
which serve a similar purpose, for NVIDIA GPUs. These features can improve
throughput by an order of magnitude. For example, for BF16, the maximum
throughput on tensor cores is 312 TFLOPS compared to 39 TFLOPS using CUDA
cores [22].

However, the number of multiplexer circuits required for the barrel shifter of
an $n$-bit floating-point unit is $n\log_{2}n$ [23], which incentivizes the
use of smaller floating-point representations. As a result, many mixed-
precision techniques perform computationally intensive matrix multiplication
tasks in BF16 while preserving sensitive portions of the model, such as the
weights and residual path activations, in FP32. Alternatively, Henry et al.
[24] developed a technique to approximate FP32 matrix multiplication using
only BF16 by representing a single FP32 value as three BF16 values, which
could accelerate FP32 matrix multiplication without requiring FP32 circuits.

###  2.3 Hybrid FP8

The adoption of the hybrid E5M2/E4M3 formats for neural networks [16] in
recent generations of processors, such as the NVIDIA Hopper series of GPUs and
the Intel Gaudi v2, has spurred interest in stable FP8 training. The hybrid
FP8 format, where E4M3 is used for the forward pass for its greater resolution
and E5M2 is used for the backward pass due to its greater expressive range,
was first proposed by Wang et al. [8] as a means to accelerate both the
training and inference stages of neural networks.

Sun et al. [15] built on this work to propose various techniques for
stabilizing training, such as stochastic rounding, chunk-based accumulation,
and Kahan summation during the optimizer update. However, the number of
techniques that can be used in practice is limited by whether the technique in
question can be applied without slowing the computation. As current NVIDIA
GPUs do not support these techniques natively, the overhead caused by the
software-based implementations cancels out any gains from the reduced
precision.

###  2.4 MS-AMP

Introduced in Peng et al. [10], MS-AMP is an automatic mixed-precision package
to utilize FP8 that offers multiple optimization levels to allow for differing
model sensitivities when applying reduced precision to the computations and
communications of neural network training. Our experiments use the O1
optimization level of MS-AMP, which performs GEMM computations in FP8 while
reducing the weights to half-precision and uses FP8 states for the weight
gradients and all-reduce communications. MS-AMP offers additional
optimizations for the optimizer buffer in level O2 optimization and for
distributed communication in level O3 optimization, but we choose to use only
the most basic optimization scheme so as to verify the effects of the least
invasive modifications.

##  3 Methods

We seek to answer whether sub-BF16 training is competitive with standard
mixed-precision BF16 training from a cost-effectiveness point of view. In
order to be cost-effective, reduced-precision training schemes must have
minimal increases in training instability and changes to hyperparameters. To
better analyze the effect of reduced precision on training stability, we aim
to quantify the effects of gradually reducing floating-point representations
bit by bit for both the exponent and the mantissa. Hopefully, analyzing the
intermediate bit representations will better illuminate the interaction
between bit width and training stability.

Our intermediate-bit floating-point experiments use the TinyLlama [25]
repository, which implements the widely used Llama [5, 6] architecture.
TinyLlama is an open-source implementation of Llama pretraining that uses a
mix of the SlimPajama [26] and StarCoder [27] datasets. It also includes
performance optimizations such as Flash Attention v2 [28]. We use the default
learning rate of $lr=4e-4$, global batch size 512, and the same learning rate
schedule as in the original code. The 120M models use a sequence length of
2048, while the 7B models use a sequence length of 4096.

###  3.1 Sharpness metric

Figure 2: Loss landscape diagrams for Llama 120M E8M3 at 5K steps (left) and
10K steps (right). Even during loss divergence, the loss landscape visualized
using the method in [29] appears smooth, motivating our introduction of a new
loss landscape sharpness metric. The value of the validation loss is included
at each point of the loss landscape.

To better investigate the state of the model when loss divergence occurs, we
attempted to visualize the loss landscape of the Llama models with the method
proposed by Li et al. [29] in Figure 2. However, we found that even when the
model is clearly in the process of loss divergence, the generated
visualizations do not show signs of sharpness. Because of this, we propose a
modified loss landscape sharpness metric that is more suitable for
autoregressive models based on the one proposed in Keskar et al. [20]. The
main difference with our version is that we use the logit of the last token
instead of the model input for the calculation. This is because adding noise
to the embeddings in a language model has different implications compared to
adding noise to input images in a computer vision model. Instead of searching
the input space as in Keskar et al. [20], we apply the search algorithm to the
logit space of the last token. Searching the logit space has the additional
advantage that the forward pass of the model need only be performed once for
each measurement, greatly reducing the computational cost. The logit of the
last token is chosen because it is not computationally feasible to perform an
optimization on the entire output space. Also, due to the autoregressive
character of decoder-only generative language models, the last token is the
only one to receive inputs from all other tokens.

#### Definition

Let $\boldsymbol{y}\in\mathbb{R}^{s\times v}$ be the output logit for an
autoregressive model of sequence length $s$ and vocabulary size $v$. Then, for
$\boldsymbol{y}_{i}$, the output logit at sequence position
$i\in\\{1,2,...,s\\}$, and one-vector $\mathbf{1}_{v}\in\mathbb{R}^{v}$, we
define a constraint set $\mathcal{C}_{\epsilon}$ at $i=s$ such that

$\mathcal{C}_{\epsilon}\in\\{\boldsymbol{z}_{s}\in\mathbb{R}^{v}:-\epsilon(|\boldsymbol{y}_{s}|+\mathbf{1}_{v})\leq\boldsymbol{z}_{s}\leq\epsilon(|\boldsymbol{y}_{s}|+\mathbf{1}_{v})\\}.$
(1)

Given $\boldsymbol{y}_{s}\in\mathbb{R}^{v},\epsilon>0$, and noise vector
$z_{s}$, the loss landscape sharpness $\phi_{\epsilon}$ for loss function $f$
can be defined as

$\phi_{\epsilon}:=\frac{max_{\boldsymbol{z}_{s}\in\mathcal{C}_{\epsilon}}f(\boldsymbol{y}_{s}+\boldsymbol{z}_{s})-f(\boldsymbol{y}_{s})}{1+f(\boldsymbol{y}_{s})}\times
100.$ (2)

The proposed metric can best be thought of as the relative magnitude of the
largest loss spike on the logit within the provided bounds. The bounds are set
to be the logit magnitudes plus one multiplied by $\epsilon$. The largest
spike in the vicinity of the logits is found using the L-BFGS-B algorithm
[30], using the SciPy [31] implementation with the output logit set as the
starting point of the search. We use $\epsilon=5e\mathrm{-4}$ for all our
experiments. Also, loss sharpness is measured with validation data that has
not been used for training. In addition, we do not apply random projection as
in Keskar et al. [20] to reduce the stochasticity of the measurement.

###  3.2 Masking

Figure 3:  Diagram showing the precisions used in a Llama decoder block (best
seen in color). The activations in the path of the residual connection are
kept in FP32, as are the model weights and embeddings. The LayerNorm and RoPE
layers use FP32 internally for their computations. The Flash Attention kernel
uses BF16 with no reduction in precision. All other layers use reduced-
precision matrix multiplication that emulates low-precision computation with a
high-precision accumulator.  Algorithm 1 PyTorch-like pseudocode for the
reduced-precision forward pass.

def forward(input, weight):

save_for_backward(input, weight)

masked_input = reduce_precision(input)

masked_weight = reduce_precision(weight)

output = F.linear(masked_input, masked_weight)

masked_output = reduce_precision(output)

return masked_output

Figure 4: Exponent masking implemented by clamping values that cannot be
expressed with the allowed number of exponent bits.

Our experiments use a simplified method of reducing floating-point precision
to achieve reasonable throughput. To simulate removing exponent bits, we
threshold the values to the minimum and maximum absolute values possible with
the given number of exponent bits. Figure 4 depicts the exponent masking
process. To remove the mantissa bits, a bitmask is applied to remove the
unrepresentable bits. The resulting method is an imperfect approximation of
reducing the bit count of floating-point numbers. However, it has the
advantage of being fast at run time, causing at most a doubling of the time
per each training step.

Reduced precision operations are applied only on the matrix multiplication
computations of the model, excluding the attention computation, which uses the
Flash Attention v2 kernel. Following existing FP8 libraries such as
TransformerEngine [32], we separate the effects of reducing the computation’s
precision from reducing the data’s precision in storage. As a result, the
activations and model weights are kept at their original precision while the
inputs and outputs of matrix multiplication are dynamically masked to emulate
reduced precision computation with a high-precision accumulator. All states
are kept in their original precision, and all operations other than matrix
multiplication are performed in their original precision. In Figure 3, we
include a diagram indicating the precision of the states and computations in a
Llama decoder block.

##  4 Results

###  4.1 MS-AMP experiments

We first analyze the effect of real-world FP8 training by applying the MS-AMP
[10] library (version 0.3.0) to the nanoGPT codebase. We run all experiments
for MS-AMP on an H100 node with 8 GPUs to ensure hardware availability of FP8.
However, despite only using the O1 optimization level for MS-AMP, we find that
the resulting models show non-trivial performance degradations. These results
indicate either that MS-AMP cannot converge to the same loss as BF16 training
or that it requires more training steps. In either case, the justification for
deploying FP8 training as a cost-saving measure becomes weaker.

Figure 5: Comparison between training with BF16 and with MS-AMP level O1.
Experiments on MS-AMP were conducted on the nanoGPT codebase with only the
model architecture as a variable. In both cases, even level O1 optimization of
MS-AMP underperforms BF16 training noticeably. We use a sequence length of
1024 and a global batch size of 480 on a single node with eight H100 GPUs. The
nanoGPT codebase was used because we were unable to integrate TinyLlama with
MS-AMP.

###  4.2 Bit reduction experiments

Figure 6: TinyLlama 120M models trained until loss divergence occurred. E8M3,
E8M4, and E8M5 models trained for 16K, 20K, and 100K steps, respectively. The
dotted black line in each figure indicates the loss landscape sharpness of the
model. A global batch size of 512 was used with a sequence length of 2048.

Figure 7: Llama 7B model training loss curves for different mantissa bit
masks. The x-axis represents training steps, while the y-axis represents the
training loss.

Steps | E8M3 | E8M4 | E8M5 | E8M6 | E8M7
---|---|---|---|---|---
1K | 0.209 | 0.205 | 0.191 | 0.191 | 0.192
2K | 0.488 | 0.363 | 0.265 | 0.221 | 0.200
3K | 1.306 | 0.734 | 0.352 | 0.229 | 0.200
4K | 2.006 | 1.125 | 0.475 | 0.237 | 0.207
5K | 1.927 | 1.439 | 0.628 | 0.248 | 0.215

Table 1: Loss landscape sharpness values at $\epsilon=5e\mathrm{-4}$ for Llama
v2 7B models trained with TinyLlama for 5,000 steps in Figure 7. Training used
a global batch size of 512 and a sequence length of 4096.

We first attempt to identify the points where training instability becomes
visible. The emulated reduced-precision representations are denoted using the
number of exponent and explicit mantissa bits used. For example, standard BF16
is referred to as E8M7, while a floating-point number with its exponent
clamped to seven bits and mantissa clamped to six bits is referred to as E7M6.

We found that removing even a single exponent bit prevents training
altogether, resulting in the model failing to progress with any learning using
E7M7, confirming previous findings [24] that neural network training is more
sensitive to exponent bits than mantissa bits. To analyze the cause, we first
conduct an ablation on the clamping mechanism by either removing only the
inner or outer exponent range, as depicted in Figure 4. We find that models
with only their inner exponent ranges clamped train normally while models with
only their outer exponent ranges clamped could not engage in training,
indicating that the inability to represent large values is the cause of
failure for E7M7.

We also found that specific regions of the model are more sensitive to reduced
precision than others. For the Llama models, removing the masking from the LM
head was sufficient to restore training, although the resulting model was much
more unstable. This result supports the conclusion that certain parts of the
model are more sensitive to reduced precision than others, as no training
occurs when the entire model is trained with E7M7.

###  4.3 Loss landscape sharpness

To further uncover the relationship between bit width and training robustness,
we use Equation 2 to quantify the degree of training instability increase by
measuring the loss landscape instability of Llama 7B models in Table 1 and
Llama 120M models in Figure 6.

In Figure 6, we show Llama 120M models trained until training loss divergence
occurs, as well as plotting the loss landscape sharpness values of the models
in E8M3, E8M4, and E8M5. Although the points of divergence are different for
each of the models, we can see a general trend of increasing sharpness until
the model diverges sharply, after which it cannot revert to its original
training trajectory. Training required approximately 2 hours per 10,000
training steps on a node with eight A100 GPUs.

To verify that similar behavior occurs in larger models, we include a
comparison of the training losses of Llama v2 models with 7B parameters
trained for 5,000 steps in Figure 7. We apply early stopping at 5,000 training
steps because training a Llama 7B model for 5K steps requires approximately
one week on a node with eight A100 GPUs. From these experiments, we can see
that loss divergence is visible in the E8M3 and E8M4 models, while it has yet
to emerge in the E8M5 model. However, from Table 1, we can see that the loss
landscape sharpness continues to increase for the E8M5 model, even though no
signs of instability are yet visible.

The E8M3 and E8M4 models show much higher sharpness values and both
accordingly diverge early in training. In contrast, there is only a gradual
increase in the loss-landscape sharpness for the E8M7 runs. As seen in Figures
7 and 6, we observe that models gradually increase in sharpness until a
threshold is reached, although the threshold value may differ depending on the
configurations. These results suggest that models with fewer mantissa bits are
entering regions of ever greater instability during training, even when these
instabilities are not visible in the loss curve. We believe that, in the
future, such analysis of loss landscape sharpness can be used to identify when
the model is at risk of loss divergence.

###  4.4 Robustness to learning rate changes

We further attempt to identify hidden instability in E8M5, which did not
diverge during the initial training stages in Figure 6. Inspired by Wortsman
et al. [18], we analyze the robustness of Llama 120M models to changes in the
learning rate by comparing training at BF16 with that for E8M5. As seen in
Figure 8, the E8M5 training runs have more frequent loss spikes during
training, especially when the learning rate is increased to $4e\mathrm{-3}$.
Although no cases of loss divergence were found, we believe that the higher
frequency of loss spikes indicates greater instability during training,
supporting our claim that the loss landscape is sharper for E8M5 even before
loss divergence occurs.

Figure 8: Comparison between Llama 120M models trained using E8M5 masked
training (left) and standard BF16 training (right) for $lr=4e\mathrm{-4}$ (the
default learning rate) and $lr=4e\mathrm{-3}$. Using 18 random seeds per
configuration, the E8M5 runs show more frequent loss spikes, especially at the
higher learning rate, indicating greater training instability.

##  5 Discussion

This work aims to propose quantitative evaluations and analyses of training
instabilities that occur when reducing floating-point precision. By clamping
unrepresentable exponent values and masking out the mantissa before the
computation, we can approximate matrix multiplication with reduced-precision
computation and high-precision accumulation. Our experiments have shown that
reducing the precision of floating-point computations destabilizes LLM
training and that existing mechanisms to stabilize FP8 training do not offer
sufficient robustness as to allow their cost-effective use.

Again, we emphasize that our work does not aim to disprove the viability of
FP8 training. We merely argue that FP8 training, in its current form, is
uneconomical, not impractical. Even if FP8 training is effective for carefully
selected hyperparameters under specific conditions, our assertion is that the
costs of finding such conditions and the risks involved in training a less
stable model outweigh the benefits of using FP8 for accelerated computation.
Instead, we propose methods to evaluate the stability and robustness of
reduced-precision training, which is key for FP8 or other reduced-precision
training schemes to be viable for LLM training in the real world.

Also, we would like to preempt misunderstanding by making clear that we are
not questioning the usefulness of FP8 for inference. Several recent works [33,
34, 35] have shown that it is even possible to quantize LLM weights to below 4
bits without sacrificing much accuracy. Xia et al. [36] has also shown that,
even for the A100, using FP6 for inference is a viable option. We believe that
dedicated FP8 and FP4 processors can make LLM inference simpler to implement
and faster to compute.

We note that, from our experiments, several methods naturally suggest
themselves as possible stabilization techniques. First, the initial stages of
training could be conducted in higher precision, similar to how smaller batch
sizes may be used during the initial stages of training [20]. Increasing the
precision when the loss landscape becomes too sharp and returning to lower
precision when the model is in a smoother region may also provide a tradeoff
between training speed and stability.

Second, the layers that are more sensitive to reduced precision may be kept at
high precision, while only the less sensitive model layers are computed with
reduced precision. For example, during our experiments, we found that removing
masking from the LM head of a Llama model was sufficient to enable E7M7
training, although the resulting model was less stable. For GPT models, we
found that increasing the precision of the first two decoder blocks to TF32
was sufficient to prevent loss divergence. However, as such compensatory
techniques depend on the model architecture, training data, and other aspects
of the training environment, we leave their investigation to future work.

##  6 Limitations

A limitation of this work is that it focuses on the initial stages of pre-
training when many instabilities are known to arise only later in training
[37]. For example, Wortsman et al. [18] show that the logits of the outputs
diverge from zero only at the later stages of training. To this, we argue that
our studies likely underestimate the instabilities that FP8 or other reduced
precision training schemes will face, further strengthening our case that
reduced-precision training methods are too unstable to be profitably utilized
in their current form.

Second, despite finding that the exponent bits are of greater importance to
LLM training than mantissa bits, we were unable to experiment by increasing
the number of exponent bits. This was because matrix multiplication in FP64 is
over an order of magnitude slower than BF16 on A100 and H100 GPUs when using
tensor cores. Experiments using representations such as E11M4, created by
removing 48 mantissa bits from FP64, may be illuminating, but we found it
impractical to train models with a greater number of exponent bits.

Finally, our experiments are limited in that they only the training loss is
used as an evaluation metric instead of real-world natural language tasks such
as MMLU [38] scores. However, while lower perplexity is no guarantee of
superior performance on downstream tasks, we believe that the divergence of
the training loss is sufficient as an indicator of training failure.

##  7 Societal impact

As the costs of training or even fine-tuning LLMs continue to climb with the
ever-growing size of the largest models and datasets, we believe that FP8 can
be a useful tool in reducing the costs of LLMs to manageable levels. The
recent introduction of FP8 in the NVIDIA Ada and Hopper [39] series of GPUs
and the even more recent announcement of FP4 support in the Blackwell [40]
series of GPUs is placing these low-precision data types within the reach of
industry and academia, both of which groups are always in need of ever more
compute.

Although we have focused on the difficulties of using FP8, with the
electricity consumption of data centers projected to double within the next
few years [41], mostly due to increased artificial intelligence workloads, we
believe that any improvement in the training efficiency of LLMs would have a
noticeable impact on global electricity consumption. As such, it is our hope
that future work uses our work as a guide to improving the ways in which low-
precision computations can be used.

##  8 Conclusion

We demonstrate that the training stability of LLMs decreases incrementally
with the reduction of floating-point bit widths used for training models of up
to 7B parameters. Using our proposed loss landscape sharpness metric, we
measure the gradual increase of instability that leads to loss divergence,
shedding light on a phenomenon with potentially large financial and
environmental costs.

## References

  * OpenAI [2024] OpenAI.  Gpt-4 technical report, 2024.
  * Team [2024a] Gemini Team.  Gemini: A family of highly capable multimodal models, 2024a.
  * Team [2024b] Gemini Team.  Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024b.
  * Anthropic [2024] Anthropic.  The claude 3 model family: Opus, sonnet, haiku, 2024.  URL https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf.
  * Touvron et al. [2023a] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models, 2023a.
  * Touvron et al. [2023b] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models, 2023b.
  * Yoo et al. [2024] Kang Min Yoo, Jaegeun Han, Sookyo In, Heewon Jeon, Jisu Jeong, Jaewook Kang, Hyunwook Kim, Kyung-Min Kim, Munhyong Kim, Sungju Kim, Donghyun Kwak, Hanock Kwak, Se Jung Kwon, Bado Lee, Dongsoo Lee, Gichang Lee, Jooho Lee, Baeseong Park, Seongjin Shin, Joonsang Yu, Seolki Baek, Sumin Byeon, Eungsup Cho, Dooseok Choe, Jeesung Han, Youngkyun Jin, Hyein Jun, Jaeseung Jung, Chanwoong Kim, Jinhong Kim, Jinuk Kim, Dokyeong Lee, Dongwook Park, Jeong Min Sohn, Sujung Han, Jiae Heo, Sungju Hong, Mina Jeon, Hyunhoon Jung, Jungeun Jung, Wangkyo Jung, Chungjoon Kim, Hyeri Kim, Jonghyun Kim, Min Young Kim, Soeun Lee, Joonhee Park, Jieun Shin, Sojin Yang, Jungsoon Yoon, Hwaran Lee, Sanghwan Bae, Jeehwan Cha, Karl Gylleus, Donghoon Ham, Mihak Hong, Youngki Hong, Yunki Hong, Dahyun Jang, Hyojun Jeon, Yujin Jeon, Yeji Jeong, Myunggeun Ji, Yeguk Jin, Chansong Jo, Shinyoung Joo, Seunghwan Jung, Adrian Jungmyung Kim, Byoung Hoon Kim, Hyomin Kim, Jungwhan Kim, Minkyoung Kim, Minseung Kim, Sungdong Kim, Yonghee Kim, Youngjun Kim, Youngkwan Kim, Donghyeon Ko, Dughyun Lee, Ha Young Lee, Jaehong Lee, Jieun Lee, Jonghyun Lee, Jongjin Lee, Min Young Lee, Yehbin Lee, Taehong Min, Yuri Min, Kiyoon Moon, Hyangnam Oh, Jaesun Park, Kyuyon Park, Younghun Park, Hanbae Seo, Seunghyun Seo, Mihyun Sim, Gyubin Son, Matt Yeo, Kyung Hoon Yeom, Wonjoon Yoo, Myungin You, Doheon Ahn, Homin Ahn, Joohee Ahn, Seongmin Ahn, Chanwoo An, Hyeryun An, Junho An, Sang-Min An, Boram Byun, Eunbin Byun, Jongho Cha, Minji Chang, Seunggyu Chang, Haesong Cho, Youngdo Cho, Dalnim Choi, Daseul Choi, Hyoseok Choi, Minseong Choi, Sangho Choi, Seongjae Choi, Wooyong Choi, Sewhan Chun, Dong Young Go, Chiheon Ham, Danbi Han, Jaemin Han, Moonyoung Hong, Sung Bum Hong, Dong-Hyun Hwang, Seongchan Hwang, Jinbae Im, Hyuk Jin Jang, Jaehyung Jang, Jaeni Jang, Sihyeon Jang, Sungwon Jang, Joonha Jeon, Daun Jeong, Joonhyun Jeong, Kyeongseok Jeong, Mini Jeong, Sol Jin, Hanbyeol Jo, Hanju Jo, Minjung Jo, Chaeyoon Jung, Hyungsik Jung, Jaeuk Jung, Ju Hwan Jung, Kwangsun Jung, Seungjae Jung, Soonwon Ka, Donghan Kang, Soyoung Kang, Taeho Kil, Areum Kim, Beomyoung Kim, Byeongwook Kim, Daehee Kim, Dong-Gyun Kim, Donggook Kim, Donghyun Kim, Euna Kim, Eunchul Kim, Geewook Kim, Gyu Ri Kim, Hanbyul Kim, Heesu Kim, Isaac Kim, Jeonghoon Kim, Jihye Kim, Joonghoon Kim, Minjae Kim, Minsub Kim, Pil Hwan Kim, Sammy Kim, Seokhun Kim, Seonghyeon Kim, Soojin Kim, Soong Kim, Soyoon Kim, Sunyoung Kim, Taeho Kim, Wonho Kim, Yoonsik Kim, You Jin Kim, Yuri Kim, Beomseok Kwon, Ohsung Kwon, Yoo-Hwan Kwon, Anna Lee, Byungwook Lee, Changho Lee, Daun Lee, Dongjae Lee, Ha-Ram Lee, Hodong Lee, Hwiyeong Lee, Hyunmi Lee, Injae Lee, Jaeung Lee, Jeongsang Lee, Jisoo Lee, Jongsoo Lee, Joongjae Lee, Juhan Lee, Jung Hyun Lee, Junghoon Lee, Junwoo Lee, Se Yun Lee, Sujin Lee, Sungjae Lee, Sungwoo Lee, Wonjae Lee, Zoo Hyun Lee, Jong Kun Lim, Kun Lim, Taemin Lim, Nuri Na, Jeongyeon Nam, Kyeong-Min Nam, Yeonseog Noh, Biro Oh, Jung-Sik Oh, Solgil Oh, Yeontaek Oh, Boyoun Park, Cheonbok Park, Dongju Park, Hyeonjin Park, Hyun Tae Park, Hyunjung Park, Jihye Park, Jooseok Park, Junghwan Park, Jungsoo Park, Miru Park, Sang Hee Park, Seunghyun Park, Soyoung Park, Taerim Park, Wonkyeong Park, Hyunjoon Ryu, Jeonghun Ryu, Nahyeon Ryu, Soonshin Seo, Suk Min Seo, Yoonjeong Shim, Kyuyong Shin, Wonkwang Shin, Hyun Sim, Woongseob Sim, Hyejin Soh, Bokyong Son, Hyunjun Son, Seulah Son, Chi-Yun Song, Chiyoung Song, Ka Yeon Song, Minchul Song, Seungmin Song, Jisung Wang, Yonggoo Yeo, Myeong Yeon Yi, Moon Bin Yim, Taehwan Yoo, Youngjoon Yoo, Sungmin Yoon, Young Jin Yoon, Hangyeol Yu, Ui Seon Yu, Xingdong Zuo, Jeongin Bae, Joungeun Bae, Hyunsoo Cho, Seonghyun Cho, Yongjin Cho, Taekyoon Choi, Yera Choi, Jiwan Chung, Zhenghui Han, Byeongho Heo, Euisuk Hong, Taebaek Hwang, Seonyeol Im, Sumin Jegal, Sumin Jeon, Yelim Jeong, Yonghyun Jeong, Can Jiang, Juyong Jiang, Jiho Jin, Ara Jo, Younghyun Jo, Hoyoun Jung, Juyoung Jung, Seunghyeong Kang, Dae Hee Kim, Ginam Kim, Hangyeol Kim, Heeseung Kim, Hyojin Kim, Hyojun Kim, Hyun-Ah Kim, Jeehye Kim, Jin-Hwa Kim, Jiseon Kim, Jonghak Kim, Jung Yoon Kim, Rak Yeong Kim, Seongjin Kim, Seoyoon Kim, Sewon Kim, Sooyoung Kim, Sukyoung Kim, Taeyong Kim, Naeun Ko, Bonseung Koo, Heeyoung Kwak, Haena Kwon, Youngjin Kwon, Boram Lee, Bruce W. Lee, Dagyeong Lee, Erin Lee, Euijin Lee, Ha Gyeong Lee, Hyojin Lee, Hyunjeong Lee, Jeeyoon Lee, Jeonghyun Lee, Jongheok Lee, Joonhyung Lee, Junhyuk Lee, Mingu Lee, Nayeon Lee, Sangkyu Lee, Se Young Lee, Seulgi Lee, Seung Jin Lee, Suhyeon Lee, Yeonjae Lee, Yesol Lee, Youngbeom Lee, Yujin Lee, Shaodong Li, Tianyu Liu, Seong-Eun Moon, Taehong Moon, Max-Lasse Nihlenramstroem, Wonseok Oh, Yuri Oh, Hongbeen Park, Hyekyung Park, Jaeho Park, Nohil Park, Sangjin Park, Jiwon Ryu, Miru Ryu, Simo Ryu, Ahreum Seo, Hee Seo, Kangdeok Seo, Jamin Shin, Seungyoun Shin, Heetae Sin, Jiangping Wang, Lei Wang, Ning Xiang, Longxiang Xiao, Jing Xu, Seonyeong Yi, Haanju Yoo, Haneul Yoo, Hwanhee Yoo, Liang Yu, Youngjae Yu, Weijie Yuan, Bo Zeng, Qian Zhou, Kyunghyun Cho, Jung-Woo Ha, Joonsuk Park, Jihyun Hwang, Hyoung Jo Kwon, Soonyong Kwon, Jungyeon Lee, Seungho Lee, Seonghyeon Lim, Hyunkyung Noh, Seungho Choi, Sang-Woo Lee, Jung Hwa Lim, and Nako Sung.  Hyperclova x technical report, 2024.
  * Wang et al. [2018] Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yu Chen, and Kailash Gopalakrishnan.  Training deep neural networks with 8-bit floating point numbers.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc., 2018.  URL https://proceedings.neurips.cc/paper_files/paper/2018/file/335d3d1cd7ef05ec77714a215134914c-Paper.pdf.
  * Sun et al. [2020] Xiao Sun, Naigang Wang, Chia-Yu Chen, Jiamin Ni, Ankur Agrawal, Xiaodong Cui, Swagath Venkataramani, Kaoutar El Maghraoui, Vijayalakshmi (Viji) Srinivasan, and Kailash Gopalakrishnan.  Ultra-low precision 4-bit training of deep neural networks.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 1796–1807. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper_files/paper/2020/file/13b919438259814cd5be8cb45877d577-Paper.pdf.
  * Peng et al. [2023] Houwen Peng, Kan Wu, Yixuan Wei, Guoshuai Zhao, Yuxiang Yang, Ze Liu, Yifan Xiong, Ziyue Yang, Bolin Ni, Jingcheng Hu, Ruihang Li, Miaosen Zhang, Chen Li, Jia Ning, Ruizhe Wang, Zheng Zhang, Shuguang Liu, Joe Chau, Han Hu, and Peng Cheng.  Fp8-lm: Training fp8 large language models, 2023.
  * Wulf and McKee [1995] Wm A Wulf and Sally A McKee.  Hitting the memory wall: Implications of the obvious.  _ACM SIGARCH computer architecture news_ , 23(1):20–24, 1995.
  * Kim et al. [2023a] Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael Mahoney, and Kurt Keutzer.  Squeezellm: Dense-and-sparse quantization.  _arXiv_ , 2023a.
  * Kalamkar et al. [2019] Dhiraj D. Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradeep Dubey.  A study of BFLOAT16 for deep learning training.  _CoRR_ , abs/1905.12322, 2019.  URL http://arxiv.org/abs/1905.12322.
  * Ba et al. [2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton.  Layer normalization, 2016.
  * Sun et al. [2019] Xiao Sun, Jungwook Choi, Chia-Yu Chen, Naigang Wang, Swagath Venkataramani, Vijayalakshmi (Viji) Srinivasan, Xiaodong Cui, Wei Zhang, and Kailash Gopalakrishnan.  Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.  URL https://proceedings.neurips.cc/paper_files/paper/2019/file/65fc9fb4897a89789352e211ca2d398f-Paper.pdf.
  * Micikevicius et al. [2022] Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, Naveen Mellempudi, Stuart Oberman, Mohammad Shoeybi, Michael Siu, and Hao Wu.  Fp8 formats for deep learning, 2022.
  * Karpathy [2022] Andrej Karpathy.  The simplest, fastest repository for training/finetuning medium-sized GPTs., 2022.  URL https://github.com/karpathy/nanoGPT.
  * Wortsman et al. [2024] Mitchell Wortsman, Peter J Liu, Lechao Xiao, Katie E Everett, Alexander A Alemi, Ben Adlam, John D Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl-Dickstein, Kelvin Xu, Jaehoon Lee, Justin Gilmer, and Simon Kornblith.  Small-scale proxies for large-scale transformer training instabilities.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=d8w0pmvXbZ.
  * Dehghani et al. [2023] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnab, Xiao Wang, Carlos Riquelme Ruiz, Matthias Minderer, Joan Puigcerver, Utku Evci, Manoj Kumar, Sjoerd Van Steenkiste, Gamaleldin Fathy Elsayed, Aravindh Mahendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Collier, Alexey A. Gritsenko, Vighnesh Birodkar, Cristina Nader Vasconcelos, Yi Tay, Thomas Mensink, Alexander Kolesnikov, Filip Pavetic, Dustin Tran, Thomas Kipf, Mario Lucic, Xiaohua Zhai, Daniel Keysers, Jeremiah J. Harmsen, and Neil Houlsby.  Scaling vision transformers to 22 billion parameters.  In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 7480–7512. PMLR, 23–29 Jul 2023.  URL https://proceedings.mlr.press/v202/dehghani23a.html.
  * Keskar et al. [2017] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang.  On large-batch training for deep learning: Generalization gap and sharp minima.  In _International Conference on Learning Representations_ , 2017.  URL https://openreview.net/forum?id=H1oyRlYgg.
  * Jouppi et al. [2017] Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, Rick Boyle, Pierre-luc Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami, Rajendra Gottipati, William Gulland, Robert Hagmann, C. Richard Ho, Doug Hogberg, John Hu, Robert Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alek Jaworski, Alexander Kaplan, Harshit Khaitan, Daniel Killebrew, Andy Koch, Naveen Kumar, Steve Lacy, James Laudon, James Law, Diemthu Le, Chris Leary, Zhuyuan Liu, Kyle Lucke, Alan Lundin, Gordon MacKean, Adriana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi Narayanaswami, Ray Ni, Kathy Nix, Thomas Norrie, Mark Omernick, Narayana Penukonda, Andy Phelps, Jonathan Ross, Matt Ross, Amir Salek, Emad Samadiani, Chris Severn, Gregory Sizikov, Matthew Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory Thorson, Bo Tian, Horia Toma, Erick Tuttle, Vijay Vasudevan, Richard Walter, Walter Wang, Eric Wilcox, and Doe Hyun Yoon.  In-datacenter performance analysis of a tensor processing unit.  _SIGARCH Comput. Archit. News_ , 45(2):1–12, jun 2017.  ISSN 0163-5964.  doi: 10.1145/3140659.3080246.  URL https://doi.org/10.1145/3140659.3080246.
  * NVIDIA [2020] NVIDIA.  Nvidia a100 tensor core gpu architecture, 2020.  URL https://resources.nvidia.com/en-us-genomics-ep/ampere-architecture-white-paper.
  * Kroening and Strichman [2008] Daniel Kroening and Ofer Strichman.  _Decision Procedures_.  Springer, 2008.
  * Henry et al. [2019] Greg Henry, Ping Tak Peter Tang, and Alexander Heinecke.  Leveraging the bfloat16 artificial intelligence datatype for higher-precision computations.  In _2019 IEEE 26th Symposium on Computer Arithmetic (ARITH)_ , pages 69–76, 2019.  doi: 10.1109/ARITH.2019.00019.
  * Zhang et al. [2024] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu.  Tinyllama: An open-source small language model, 2024.
  * Soboleva et al. [2023] Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey.  SlimPajama: A 627B token cleaned and deduplicated version of RedPajama.  https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama, 2023.  URL https://huggingface.co/datasets/cerebras/SlimPajama-627B.
  * Li et al. [2023] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries.  Starcoder: may the source be with you!, 2023.
  * Dao [2023] Tri Dao.  FlashAttention-2: Faster attention with better parallelism and work partitioning, 2023.
  * Li et al. [2018] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein.  Visualizing the loss landscape of neural nets.  In _Neural Information Processing Systems_ , 2018.
  * Liu and Nocedal [1989] Dong C Liu and Jorge Nocedal.  On the limited memory BFGS method for large scale optimization.  _Mathematical Programming_ , 45(1):503–528, August 1989.
  * Virtanen et al. [2020] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors.  SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.  _Nature Methods_ , 17:261–272, 2020.  doi: 10.1038/s41592-019-0686-2.
  * NVIDIA [2023] NVIDIA.  TransformerEngine, 2023.  URL https://github.com/NVIDIA/TransformerEngine.
  * Lee et al. [2023] Jung Hyun Lee, Jeonghoon Kim, Se Jung Kwon, and Dongsoo Lee.  Flexround: Learnable rounding based on element-wise division for post-training quantization.  In _ICML_ , pages 18913–18939, 2023.  URL https://proceedings.mlr.press/v202/lee23h.html.
  * Kwon et al. [2022] Se Jung Kwon, Jeonghoon Kim, Jeongin Bae, Kang Min Yoo, Jin-Hwa Kim, Baeseong Park, Byeongwook Kim, Jung-Woo Ha, Nako Sung, and Dongsoo Lee.  AlphaTuning: Quantization-aware parameter-efficient adaptation of large-scale pre-trained language models.  In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 3288–3305, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.findings-emnlp.240.  URL https://aclanthology.org/2022.findings-emnlp.240.
  * Kim et al. [2023b] Jeonghoon Kim, Jung Hyun Lee, Sungdong Kim, Joonsuk Park, Kang Min Yoo, Se Jung Kwon, and Dongsoo Lee.  Memory-efficient fine-tuning of compressed large language models via sub-4-bit integer quantization.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023b.  URL https://openreview.net/forum?id=2jUKhUrBxP.
  * Xia et al. [2024] Haojun Xia, Zhen Zheng, Xiaoxia Wu, Shiyang Chen, Zhewei Yao, Stephen Youn, Arash Bakhtiari, Michael Wyatt, Donglin Zhuang, Zhongzhu Zhou, Olatunji Ruwase, Yuxiong He, and Shuaiwen Leon Song.  Fp6-llm: Efficiently serving large language models through fp6-centric algorithm-system co-design, 2024.
  * Bekman [2023] Stas Bekman.  Machine learning: Llm/vlm training and engineering by stas bekman, 2023.  URL https://stasosphere.com/machine-learning/.
  * Hendrycks et al. [2021] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2021.
  * NVIDIA [2022] NVIDIA.  Nvidia h100 tensor core gpu architecture overview, 2022.  URL https://resources.nvidia.com/en-us-tensor-core.
  * NVIDIA [2024] NVIDIA.  Nvidia dgx b200, 2024.  URL https://resources.nvidia.com/en-us-dgx-systems/dgx-b200-datasheet.
  * IEA [2024] IEA.  Nvidia h100 tensor core gpu architecture overview, 2024.  URL https://www.iea.org/reports/electricity-2024.


# Extensively Matching for Few-shot Learning Event Detection

Viet Dac Lai1, Franck Dernoncourt2 and Thien Huu Nguyen1
1Department of Computer and Information Science,
University of Oregon, Eugene, Oregon, USA
2Adobe Research, San Jose, CA, USA
{vietl<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Current event detection models under supervised learning settings fail to
transfer to new event types. Few-shot learning has not been explored in event
detection even though it allows a model to perform well with high
generalization on new event types. In this work, we formulate event detection
as a few-shot learning problem to enable to extend event detection to new
event types. We propose two novel loss factors that matching examples in the
support set to provide more training signals to the model. Moreover, these
training signals can be applied in many metric-based few-shot learning models.
Our extensive experiments on the ACE-2005 dataset (under a few-shot learning
setting) show that the proposed method can improve the performance of few-shot
learning.

##  1 Introduction

Event Detection (ED) is an important task in Information Extraction (IE) in
Natural Language Processing (NLP). Event Detection is the task to detect event
triggers from a given text (e.g. a sentence) and classify it into one of the
event types of interest. The following sentence is an example of ED:

In 1997, the company hired John D. Idol to take over as chief executive.

In this example, an ideal event detection system should detect the word hired
as an event, and classify it to class of Personnel:Start-Position, assuming
that Personnel:Start-Position is in the set of interested classes.

The current works in ED typically employ traditional supervised learning based
on feature engineering Li et al. (2014); Chen et al. (2017) and neural
networks Nguyen et al. (2016a); Chen et al. (2018); Lu and Nguyen (2018). The
main problem with supervised learning models is that they can not perform well
on unseen classes (e.g. training a model to classify daily events, then run
this model to classify laboratory operations). As a result, supervised
learning ED can not extend to unseen event types. A trivial solution is to
annotate more data for unseen event types, then retraining the model with
newly annotated data. However, this method is usually impractical because of
the extremely high cost of annotation Liu et al. (2019).

A human can learn about a new concept with limited supervision e.g. one can
detect and classify events with 3-5 examples Grishman et al. (2005). This
motivates the setting we aim for event detection: few-shot learning (FSL). In
FSL, a trained model rapidly learns a new concept from a few examples while
keeping great generalization from observed examples Vinyals et al. (2016).
Hence, if we need to extend event detection into a new domain, a few examples
are needed to activate the system in the new domain without retraining the
model. By formulating ED as FSL, we can significantly reduce the annotation
cost and training cost while maintaining highly accurate results.

In a few shot learning iteration, the model is given a support set and a query
instance. The support set consists of examples from a small set of classes. A
model needs to predict the label of the query instance in accordance with the
set of classes appeared in the support set. Typical methods employ a neural
network to embed the samples into a low-dimension vector space Vinyals et al.
(2016); Snell et al. (2017), then, classification is done by matching those
vectors based on vector distances Vinyals et al. (2016); Snell et al. (2017);
Sung et al. (2018). One potential problem of prior FSL methods is that the
model relies solely on training signals between query instance and the support
set Vinyals et al. (2016); Snell et al. (2017); Sung et al. (2018). Thus, the
matching information between samples in the support set has not been exploited
yet. We believe that this is not an efficient use of training data because
dataset in ED is very small Grishman et al. (2005). Therefore, in this study,
we propose to train an ED model using matching information (1) between query
instance and the support set and (2) between the samples in the support
themselves. This is implemented by adding two auxiliary factors into the loss
function to constrain the learning process.

We apply the proposed training signals to different FSL models on the
benchmark event detection dataset Grishman et al. (2005). The experiments show
that the training signal can improve the performance of the examined FSL
models. To summarize, our contributions to this work include:

  * •

We formulate event detection as a few-shot learning problem to extend ED to
new event types and provide a baseline for this new research direction. To our
best knowledge, this is a new branch of research that has not been explored.

  * •

We propose two novel training signals for FSL. These signals can remarkably
improve the performance of existing FSL models. As these signals do not
require any additional information (e.g. dependency tree or part-of-speech),
they can be applied in any metric-based FSL models.

##  2 Related work

Early studies in event detection mainly address feature engineering for
statistical models Ahn (2006); Ji and Grishman (2008); Hong et al. (2011); Li
et al. (2014, 2015) including semantic features and syntactic features.
Recently, due to the advances with deep learning, many neural network
architectures have been presented for ED, e.g. convolutional neural networks
(CNN) (Chen et al., 2015; Nguyen and Grishman, 2015, 2016; Nguyen et al.,
2016b), recurrent neural networks (RNN) (Liu et al., 2017; Chen et al., 2018;
Nguyen et al., 2016a; Nguyen and Nguyen, 2018) and graph convolutional neural
networks (GCN) Nguyen and Grishman (2018); Pouran Ben Veyseh et al. (2019).
These methods formulate ED as a supervised learning problem which usually
fails to predict the labels of new event types.

By transitioning the symbolic event types to descriptive event types in the
form of bags of keywords Bronstein et al. (2015); Peng et al. (2016); Lai and
Nguyen (2019), the adaptibility of event detection can be formed as a
supervised-learning problem. However, these studies have not examined FSL as
we do in this work. One can also address this problem in zero-shot learning
with data generated from abstract meaning representation Huang et al. (2018)
or two-stage pipeline ( trigger identification and few-shot event
classification) based on dynamic memory network Deng et al. (2020). A recent
study has employed few-shot learning for event classification Lai et al.
(2020). Our work is similar in terms of formulation, however, we consider it
in a larger extent of event detection where the NULL event is also included.

Few-shot learning has been studied early in the literature Thrun (1996).
Before the era of the deep neural network, FSL approaches focused on building
generative models that can transfer priors across classes. However, these
methods are hard to apply to real applications because they require a subject-
dedicated design such as handwritten characters Lake et al. (2013); Wong and
Yuille (2015). As a result, they cannot capture the nature of the distribution
Salimans et al. (2016). Later studies, based on deep neural network, proposed
metric learning to model the distribution of distance among classes, Koch et
al. (2015) with many incremental improvements in distance functions such as
cosine similarity Vinyals et al. (2016), Euclidean distance Snell et al.
(2017) and learnable distance function Sung et al. (2018). Metric-based FSL
presents its advantages in two dimensions. First, it is based on the well-
studied theory in distance functions. Second, the simplicity in architecture
and training processes can encourage its application in practice. Recently,
meta-learning with parameter update strategy is also proposed to enable the
models to learn quickly in few training iterations Santoro et al. (2016); Finn
et al. (2017).

##  3 Methodology

Our goal in this work is to formulate ED as a FSL problem, which has not been
done in prior work. In order to achieve this, this section is divided into
three parts. In the section 3.1 we present the overall framework that
formulate Event Detection as an Few-Shot Learning problem. Then, we present
popular models for FSL in the prior work and common sentence encoders which
have been widely used in ED in section 3.2. Finally, we present two novel
reguarlization technique to further improve the FSL model for ED in section
3.3.

###  3.1 Event Detection as Few-shot Learning

In few-shot learning, models learn to predict the label of a query instance
$x$ given a support set $S$ (a set of well-classified instances) and a set of
classes $C$, which appears in the support set $S$. Prior studies in FSL employ
$N$-way $K$-shot setting, in which there are $N$ clusters, which represent $N$
classes, each cluster contains $K$ data points (i.e., examples).

However, this setting is designed for problems that do not involve the “NULL”
class (e.g., image classification and event classification). In event
detection, the systems need to predict whether a query instance is an event
(positive event type) or not (negative event type – the “NULL” type) before it
is further classified into one of the classes of interest. To this end, we
propose to extend the N-way K-shot setting to be N+1-way K-shot setting. In
this setting, the support set contains $N$ clusters representing $N$ positive
event types and 1 cluster representing the NULL event type. The support set is
denoted as follows:

$\displaystyle S=$
$\displaystyle\\{(s^{1}_{1},a^{1}_{1},t_{1}),\ldots,(s^{K}_{1},a^{K}_{1},t_{1}),$
$\displaystyle\ldots$
$\displaystyle(s^{1}_{N},a^{1}_{N},t_{N}),\ldots,(s^{K}_{N},a^{K}_{N},t_{N}),$
$\displaystyle(s^{1}_{N+1},a^{1}_{N+1},t_{null}),\ldots,(s^{K}_{1},a^{K}_{N+1},t_{null})\\}$

where:

  * •

$\\{t_{1},t_{2},\cdots t_{N}\\}$ is the set of positive labels, which indicate
an event

  * •

$t_{null}$ a special label for non-event.

  * •

$(s^{j}_{i},a^{j}_{i},t_{i})$ indicates that the $a^{j}_{i}$-th word in the
sentence $s^{j}_{i}$ is the trigger word of an event mention with the event
type $t_{i}$

###  3.2 Framework

Follow prior studies in FSL Gao et al. (2019), we employ the metric-based FSL
framework with three components: instance encoder, prototype encoder, and
classification module.

####  3.2.1 Instance Encoder

Given a sentence of $L$ words $\\{w_{1},w_{2},\cdots,w_{L}\\}$ and the event
mention $w_{a}$, which is the $a$-th word of the sentence, we first map
discrete words to a continuous high dimensional vector space to facilitate
neural network using both pre-trained word embedding and position embedding as
follow:

  * •

In order to capture the syntactic and semantic of the word itself, we map each
word in the sentence to a single vector using pre-trained word embedding,
following previous studies in ED Nguyen and Grishman (2015). After this step,
we derive a sequence of vectors $\\{e_{1},e_{2},\cdots,e_{L}\\}$ where
$e_{i}\in R^{u}$.

  * •

To provide a sense of the relative position of a word regarding the position
of the anchor word, we further provide position embedding. It is mapped from
the relative distance, $i-a$, of the $i$-th word with respect to the anchor
word, $a$-th word to a single vector $p_{i}\in R^{v}$. We randomly initialize
this word embedding and update the embedding during the training process.

  * •

Following previous work (Nguyen and Grishman, 2015), the final embedding of a
word $w_{i}$ is derived by concatenating word embedding and position embedding
$m_{i}=[e_{i},p_{i}]\in R^{u+v}$.

Once we get the embedding for the whole sentence
$E(s)=\\{m_{1},m_{2},\cdots,m_{L}\\}$, we employ a neural network, denoted as
$f$, to encode the information of an instance $(s,a)$ of the anchor $w_{a}$
under the context in the sentence $s$ into a single vector $v=f(E(s),a)$. In
this work, consider the three following neural network architectures for this
encoding purpose:

  * •

Convolution Neural Network (CNN) Kim (2014) encodes the sentence by
convolution operation on $k$ consecutive vectors representing $k$-gram. Follow
Nguyen and Grishman (2015), we use multiple kernel sizes $k\in\\{2,3,4,5\\}$
to cover the context with 150 filters for each kernel size. To squeeze the
information of the sentence, we apply max pooling to the top convolution layer
to get a pooled vector $p$. We also introduce local embedding $e_{[a-w,a+w]}$
with window size $w=2$. We concatenate pooled vector and local embeddings, and
feed them through multiple dense layer to get the final representation:

$v=W[p,e_{[a-w,a+w]}]$

  * •

Long Short-Term Memory (LSTM) Hochreiter and Schmidhuber (1997), at each step
$i$, computes a hidden vector $h_{i}$ from the hidden vector of the previous
step $h_{i-1}$ and the current input vector $e_{i}$. To capture the context
from both sides a word in the sentence, we employ two separate LSTMs running
on forward and backward directions. Eventually, we can obtain two sequence of
hidden vector $\\{h^{forward}_{i},\cdots,h^{forward}_{L}\\}$ and
$\\{h^{backward}_{i},\cdots,h^{backward}_{L}\\}$. Finally, we concatenate the
$a$-th vectors, at the position of the anchor, to form the representation of
the instance:

$v=\text{concat}(h^{forward}_{a},h^{backward}_{a})$

  * •

Graph Convolutional Neural Network features graph convolution Kipf and Welling
(2017) on syntactic dependency graph, which allows the model to access to the
nonconsecutive words based on the connection on the syntactic dependency tree.
Following Nguyen and Grishman (2018), we transform the dependency tree into a
syntactic graph by making it an undirected graph and adding node loops. The
hidden vectors $h^{l}_{i}$ of the $l$-th vector is obtained by feeding hidden
vectors of the $l-1$-th layer through a GCN layer Kipf and Welling (2017). The
final representation is the hidden vector in the top layer at the position of
the trigger $h_{a}^{L}$ where $L=2$ is the number of GCN layers.

####  3.2.2 Prototype Encoder

This module computes a representative vector, called prototype, for each class
$t\in T$ in the support set $S$ from its instances’ vectors. We employ two
variants of prototype computation.

The first version, proposed in the original Prototypical Network Snell et al.
(2017), considers all representation vectors are equally important. To
calculate the prototype for a class $t_{i}$, it aggregates all the
representation vectors of the instance of class $t_{i}$, and then perform
averaging over all vectors :

$\textbf{c}_{i}=\frac{1}{K}\sum_{(s^{j}_{i},a^{j}_{i},t_{i})\in
S}f(E(s^{j}_{i}),a^{j}_{i})$ (1)

On the other hand, it was claimed that the supporting vectors are
conditionally important with respect to the query $(q,p)$. Thus, the second
version computes the prototype as a weighted sum of the supporting vectors.
The weights are obtained by attention mechanism according to the
representational vector of the query as follow:

$\displaystyle\textbf{c}_{i}$
$\displaystyle=\sum_{(s^{j}_{i},a^{j}_{i},t_{i})\in
S}\alpha_{ij}f(E(s^{j}_{i}),a^{j}_{i})$ (2)

where

$\displaystyle\alpha_{ij}$
$\displaystyle=\frac{\text{exp}(b_{ij})}{\sum_{(s^{k}_{i},a^{k}_{i},t_{i})\in
S}\text{exp}(b_{ik})}$ $\displaystyle b_{ij}$
$\displaystyle=\sum\Big{[}\sigma(f(E(s^{j}_{i}),a^{j}_{i})\odot
f(E(q),p))\Big{]}$ $\displaystyle\odot$ denotes the element-wise product.

Model | 5+1-way 5-shot | 10+1-way 10-shot
---|---|---
Encoder | CNN | LSTM | GCNN | CNN | LSTM | GCNN
Proto | 70.85 | 68.77 | 71.30 | 61.43 | 57.89 | 62.36
Proto+Att | 71.23 | 69.32 | 72.76 | 63.50 | 59.56 | 65.08
Relation | 54.36 | 68.33 | 58.37 | 41.37 | 62.85 | 44.43
Matching | 34.71 | 49.40 | 32.49 | 23.05 | 33.84 | 21.51
Table 1: F1-score (micro) of models using CNN, LSTM and GCN encoders without
proposed losses.

####  3.2.3 Classification Module

This module computes the distribution on all the event types $T$ of a query
instance $x=(q,p)$ using a distance/similarity function $d:R\leftarrow R^{d}$.

$P(y=t_{i}|x,S)=\frac{\text{exp}(-d(v,\textbf{c}^{i}))}{\sum^{N}_{j=1}\text{exp}(-d(v,\textbf{c}^{j}))}$
(3)

where $d$ is a distance/similarity function, $v=f(q,p)$ is the representation
vector of the given query instance and $\textbf{c}^{i}$ and $\textbf{c}^{j}$
are the prototype vectors obtained in either Equation (1) or Equation (2) from
the support set $S$.

In this paper, we examine three kinds of distance/similarity function with
prototype module to form 4 model as follow:

  * •

Cosine similarity with averaging prototype as Matching network Vinyals et al.
(2016).

  * •

Euclidean distance with averaging prototype as Proto network Snell et al.
(2017).

  * •

Euclidean distance with weighted sum prototype as Proto+Att network Gao et al.
(2019).

  * •

Learnable distance function with averaging prototype as Relation network Sung
et al. (2018).

###  3.3 Training Objectives

In the literature, a metric-based FSL model is typically trained by minimizing
the negative log-likelihood as follow:

$L_{query}(x,S)=-\log P(y=t|x,S)$ (4)

where $x$, $t$, $S$ are query instance, ground truth label, and support set,
respectively.

This loss function exploits the signal of matching information between the
query instance and the supporting instances. It can work efficiently in
computer vision because the number of samples in computer vision datasets are
typically huge. However, in NLP tasks, the dataset is commonly relatively much
smaller (e.g. ACE 2005 contains 4000 positive examples). So using this loss
function is not enough to deliver a good system.

Therefore, providing more training signals is crucial to the problem which
involves a small dataset. Fortunately, the support set is a well-classified
set of instances with $K$ examples per class in a total of $N$ classes. In
this paper, we proposed two ways to exploit this resourceful set as follow:

  * •

Intra-cluster matching: We argue that the representational vectors in the same
class should be close to each other. Therefore, we minimize the distance
between instance in the same class.

$L_{intra}=\sum^{N}_{i=1}\sum^{K}_{k=1}\sum^{K}_{j=k+1}\text{mse}(v^{j}_{i},v^{k}_{i})$
(5)

  * •

Inter-cluster information: We also argue that the clusters should distribute
far away from each other. Hence, their prototypes are also distant from the
other. Hence, we maximize the distances between pairs of prototypes.

$L_{inter}=1-\sum^{N}_{i=1}\sum^{N}_{j=i+1}\text{cosine}(c_{i},c_{j})$ (6)

In this work, we train our model using a combination of the loss functions in
equations 4, 5,6. We control the contribution of the additional losses by two
hyperparameters $\beta$ and $\gamma$ as follow:

$L=L_{query}+\beta\hat{L}_{intra}+\gamma\hat{L}_{inter}$ (7)

where $\hat{L}_{intra}$ and $\hat{L}_{inter}$ are scaled losses with respect
to $L_{query}$, and $\beta$ and $\gamma$ are the trade-off parameters.

##  4 Experiments

Encoder | Model | 5+1-way 5-shot | 10+1-way 10-shot
---|---|---|---
Original | \+ $L_{inter}+L_{intra}$ | Original | $+L_{inter}+L_{intra}$
CNN | Proto | 70.85 | 72.07 | 61.43 | 62.84
LSTM | Proto | 68.77 | 78.09 | 57.89 | 72.78
GCN | Proto | 71.30 | 71.82 | 62.36 | 63.49
CNN | Proto+Att | 71.23 | 72.46 | 63.5 | 64.38
LSTM | Proto+Att | 69.32 | 78.44 | 59.56 | 72.94
GCN | Proto+Att | 72.76 | 72.92 | 65.08 | 66.10
Table 2: F1-score (micro) of models using CNN, LSTM, and GCN. Original columns
show the models without additional training signal. $L_{inter}+L_{intra}$
columns demonstrate the models with additional inter and intra loss functions.

###  4.1 Data

We use the ACE-2005 dataset to evaluate all of the models in this study.
ACE-2005 is a benchmark dataset in event detection with 33 positive event
subtypes, which are grouped into 8 event types Business, Contact, Conflict,
Justice, Life, Movement, Personnel, and Transaction. Although the dataset is
split into training, development, and testing sets, we cannot use these splits
directly because, in FSL, the set of event types in the training set and
testing sets are disjoint. Therefore, we further split these datasets to
satisfy three conditions for FSL:

  * •

The set of event types in the training set $T^{train}$ are disjoint to those
in the development and the testing set:

$T^{dev}\equiv T^{test};T^{train}\cap T^{test}=\emptyset$;

  * •

In order to run FSL with the 10-way 10-shot setting, the set of event subtypes
should contain at least 10 subtypes.

  * •

The training set should contain as many samples as possible.

Based on these criteria, we use all samples belonging to 4 event types:
Business, Contact, Conflict and Justice as the training set. While the rest
(Life, Movement, Personnel and Transaction) are used for the development and
testing sets. We split the sample by ratio 50:50 in every subtype to ensure
the balance of the development and the testing set. Finally, since there are
event types that have less than 15 examples, we eliminate all of these from
the training, development, and testing set.

###  4.2 Hyper-parameters

We evaluate using 5+1-way 5-shot and 10+1-way 10-shot FSL settings. Although
it was seen that the higher number of classes we have during the training
time, the better performance on testing Snell et al. (2017), we avoid feeding
all event types in every iteration during training time. We manage to sample
20 positive classes (over 21 in the training set) in each training iteration.

We initialize the embedding vectors with 300-dimension GLoVe embedding,
trained from 6 billion tokens. We use 50-dimension position embedding and
initialize it randomly. These embedding vectors are updated during training
time.

We train Proto, Proto+Att, and Matching using Stochastic Gradient Decent (SGD)
optimizer while Relation is trained with AdaDelta optimizer because SGD hardly
converges with Relation network. The learning rate is initialized to 0.03 and
decays after every 500 iterations. We trained our models in 2500 iterations
and evaluation at every 200 iterations.

In order to find the best set of $\beta$ and $\gamma$, we do grid search with
with $(\beta,\gamma)\in\\{0.0,0.1,0.2,0.3\\}^{2}$.

###  4.3 Result

Encoder | FSL Model | Original | +Inter | +Intra | +Intra+Inter
---|---|---|---|---|---
CNN | Proto | 67.92 | 68.78 | 68.83 | 69.37
LSTM | Proto | 65.94 | 65.28 | 72.07 | 77.56
GCN | Proto | 69.28 | 70.05 | 69.49 | 70.11
CNN | Proto+Att | 69.90 | 70.23 | 70.06 | 70.43
LSTM | Proto+Att | 67.26 | 67.48 | 72.00 | 77.81
GCN | Proto+Att | 71.65 | 71.75 | 71.56 | 71.18
Table 3: Ablation study: F1-score (micro) of Prototypical-based models on dev
set with 5+1-way 5-shot FSL setting

In this section, we perform our experiment in three steps:(1) find the best
FSL models among Proto, Proto+Att, Matching, Relation models; (2) evaluate the
proposed additional training factors and (3) analyze the effectiveness of each
training factor in an ablation study.

Table 1 shows the F-scores of four models using three kinds of sentence
encoders on the ACE-2005 dataset under 5+1-way 5-shot and 10+1-way 10-shot FSL
settings without our proposed losses. As can be seen from the Table 1, the
performance of the models on 5+1-way 5-shot is always better than 10+1-way
10-shot because the number of classes needs to be classified in the 10+1-way
setting is almost twice as much of in 5+1-way setting. Second, we can see that
Prototypical-based (Proto and Proto+Att) models outperform the Matching
network and the Relation network on both FSL settings. Among Prototypical
network models, Proto+Att is slightly better than Proto with a 0.8%
performance gap in the 10+1-way 10-shot setting.

Most importantly, Table 2 presents the F-scores of Proto and Proto+A with the
proposed loss functions (i.e., $L_{intra}$, $L_{inter}$). As we can see from
the table, the proposed loss functions can significantly improve the
performance of Proto and Proto+Att models over different encoders (i.e., CNN,
LSTM, and GCN), clearly demonstrating the benefits of the intra and inter-
similarity constraints in this work.

###  4.4 Ablation Study

In this study, we introduce two penalization factors, presented in Equations 5
and 6.

Besides the FSL formulation for event detection, a major contribution in this
work involves the two loss functions $L_{intra}$ and $L_{inter}$ to improve
the representation vectors for the models. To evaluate the contribution of
these terms, Table 3 shows the performance of the FSL models with different
combinations of loss functions on the development set. In particular, we focus
on the prototypical-based FSL model on the 5+1-way 5-shot setting in this
analysis (although the similar trends of the performance are also observed for
the other models and settings). The “Original” column corresponds to the
models where both $L_{inter}$ and $L_{intra}$ are not applied. The other
columns, on the other hand, report the performance of the models when the
combinations $L_{inter}$, $L_{intra}$, and $L_{inter}+L_{intra}$ of the loss
terms are introduced.

It is clear from the table that both loss terms are important for the FSL
models for ED as eliminating any of them would significantly hurt the
performance excepting the Proto+Att model with GCN encoder. The best
performance is achieved with both loss terms are applied, thus testifying to
the benefits of the proposed regularization techniques in this work.

##  5 Conclusion

In this paper, we address the problem of extending event detection to unseen
event types through few-shot learning. We investigate four metric-based few-
shot learning models with different encoder types (CNN, LSTM, and GCN).
Moreover, we propose two novel loss functions to provide more training signals
to the model exploiting domain-matching information in the support set. Our
extensive experiments show that our method increases the efficiency of using
training data, resulting in better classification performance. Our ablation
study shows that both intra-cluster matching and inter-cluster matching
contributes to the improvement.

## Acknowledgments

This research is based upon work supported in part by the Office of the
Director of National Intelligence (ODNI), Intelligence Advanced Research
Projects Activity (IARPA), via IARPA Contract No. 2019-19051600006 under the
Better Extraction from Text Towards Enhanced Retrieval (BETTER) Program. The
views and conclusions contained herein are those of the authors and should not
be interpreted as necessarily representing the official policies, either
expressed or implied, of ODNI, IARPA, the Department of Defense, or the U.S.
Government. The U.S. Government is authorized to reproduce and distribute
reprints for governmental purposes notwithstanding any copyright annotation
therein. This document does not contain technology or technical data
controlled under either the U.S. International Traffic in Arms Regulations or
the U.S. Export Administration Regulations.

## References

  * Ahn (2006) David Ahn. 2006.  The stages of event extraction.  In _Proceedings of the Workshop on Annotating and Reasoning about Time and Events_.
  * Bronstein et al. (2015) Ofer Bronstein, Ido Dagan, Qi Li, Heng Ji, and Anette Frank. 2015.  Seed-based event trigger labeling: How far can event descriptions get us?  In _ACL-IJCNLP_.
  * Chen et al. (2017) Yubo Chen, Shulin Liu, Xiang Zhang, Kang Liu, and Jun Zhao. 2017.  Automatically labeled data generation for large scale event extraction.  In _ACL_.
  * Chen et al. (2015) Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015.  Event extraction via dynamic multi-pooling convolutional neural networks.  In _ACL-IJCNLP_.
  * Chen et al. (2018) Yubo Chen, Hang Yang, Kang Liu, Jun Zhao, and Yantao Jia. 2018.  Collective event detection via a hierarchical and bias tagging networks with gated multi-level attention mechanisms.  In _EMNLP_.
  * Deng et al. (2020) Shumin Deng, Ningyu Zhang, Jiaojian Kang, Yichi Zhang, Wei Zhang, and Huajun Chen. 2020.  Meta-learning with dynamic-memory-based prototypical network for few-shot event detection.  In _Proceedings of the 13th International Conference on Web Search and Data Mining_ , pages 151–159.
  * Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _ICML_.
  * Gao et al. (2019) Tianyu Gao, Xu Han, Zhiyuan Liu, and Maosong Sun. 2019.  Hybrid attention-based prototypical networks for noisy few-shot relation classification.  In _AAAI_.
  * Grishman et al. (2005) Ralph Grishman, David Westbrook, and Adam Meyers. 2005.  Nyu’s english ace 2005 system description.  In _ACE 2005 Evaluation Workshop_.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997.  Long short-term memory.  In _Neural Computation_.
  * Hong et al. (2011) Yu Hong, Jianfeng Zhang, Bin Ma, Jianmin Yao, Guodong Zhou, and Qiaoming Zhu. 2011\.  Using cross-entity inference to improve event extraction.  In _ACL_.
  * Huang et al. (2018) Lifu Huang, Heng Ji, Kyunghyun Cho, and Clare R Voss. 2018.  Zero-shot transfer learning for event extraction.  In _ACL_ , pages 2160–2170.
  * Ji and Grishman (2008) Heng Ji and Ralph Grishman. 2008.  Refining event extraction through cross-document inference.  In _ACL_.
  * Kim (2014) Yoon Kim. 2014.  Convolutional neural networks for sentence classification.  In _EMNLP_.
  * Kipf and Welling (2017) Thomas N. Kipf and Max Welling. 2017.  Semi-supervised classification with graph convolutional networks.  In _ICLR_.
  * Koch et al. (2015) Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. 2015.  Siamese neural networks for one-shot image recognition.  In _ICML deep learning workshop_ , volume 2.
  * Lai et al. (2020) Viet Dac Lai, Franck Dernoncourt, and Thien Huu Nguyen. 2020.  Exploiting the matching information in the support set for few shot event classification.  In _PAKDD_.
  * Lai and Nguyen (2019) Viet Dac Lai and Thien Huu Nguyen. 2019.  Extending event detection to new types with learning from keywords.  In _Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)_.
  * Lake et al. (2013) Brenden M Lake, Ruslan R Salakhutdinov, and Josh Tenenbaum. 2013.  One-shot learning by inverting a compositional causal process.  In _NIPS_.
  * Li et al. (2014) Qi Li, Heng Ji, Yu Hong, and Sujian Li. 2014.  Constructing information networks using one single model.  In _EMNLP_.
  * Li et al. (2015) Xiang Li, Thien Huu Nguyen, Kai Cao, and Ralph Grishman. 2015.  Improving event detection with Abstract Meaning Representation.  In _Proceedings of the First Workshop on Computing News Storylines_.
  * Liu et al. (2017) Shulin Liu, Yubo Chen, Kang Liu, and Jun Zhao. 2017.  Exploiting argument information to improve event detection via supervised attention mechanisms.  In _ACL_.
  * Liu et al. (2019) Shulin Liu, Yang Li, Feng Zhang, Tao Yang, and Xinpeng Zhou. 2019.  Event detection without triggers.  In _NAACL:HLT_ , pages 735–744.
  * Lu and Nguyen (2018) Weiyi Lu and Thien Huu Nguyen. 2018.  Similar but not the same: Word sense disambiguation improves event detection via neural representation matching.  In _EMNLP_.
  * Nguyen and Grishman (2018) Thien Nguyen and Ralph Grishman. 2018.  Graph convolutional networks with argument-aware pooling for event detection.  In _AAAI_.
  * Nguyen et al. (2016a) Thien Huu Nguyen, Kyunghyun Cho, and Ralph Grishman. 2016a.  Joint event extraction via recurrent neural networks.  In _NAACL_.
  * Nguyen et al. (2016b) Thien Huu Nguyen, Lisheng Fu, Kyunghyun Cho, and Ralph Grishman. 2016b.  A two-stage approach for extending event detection to new types via neural networks.  In _Proceedings of the 1st ACL Workshop on Representation Learning for NLP (RepL4NLP)_.
  * Nguyen and Grishman (2015) Thien Huu Nguyen and Ralph Grishman. 2015.  Event detection and domain adaptation with convolutional neural networks.  In _ACL-IJCNLP_.
  * Nguyen and Grishman (2016) Thien Huu Nguyen and Ralph Grishman. 2016.  Modeling skip-grams for event detection with convolutional neural networks.  In _EMNLP_.
  * Nguyen and Nguyen (2018) Trung Minh Nguyen and Thien Huu Nguyen. 2018.  One for all: Neural joint modeling of entities and events.  In _AAAI_.
  * Peng et al. (2016) Haoruo Peng, Yangqiu Song, and Dan Roth. 2016.  Event detection and co-reference with minimal supervision.  In _EMNLP_.
  * Pouran Ben Veyseh et al. (2019) Amir Pouran Ben Veyseh, Thien Huu Nguyen, and Dejing Dou. 2019.  Graph based neural networks for event factuality prediction using syntactic and semantic structures.  In _ACL_.
  * Salimans et al. (2016) Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. 2016.  Improved techniques for training gans.  In _NIPS_.
  * Santoro et al. (2016) Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. 2016.  Meta-learning with memory-augmented neural networks.  In _ICML_.
  * Snell et al. (2017) Jake Snell, Kevin Swersky, and Richard Zemel. 2017.  Prototypical networks for few-shot learning.  In _NIPS_.
  * Sung et al. (2018) Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. 2018.  Learning to compare: Relation network for few-shot learning.  In _CVPR_.
  * Thrun (1996) Sebastian Thrun. 1996.  Is learning the n-th thing any easier than learning the first?  In _NIPS_.
  * Vinyals et al. (2016) Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. 2016.  Matching networks for one shot learning.  In _NIPS_.
  * Wong and Yuille (2015) Alex Wong and Alan L Yuille. 2015.  One shot learning via compositions of meaningful patches.  In _ICCV_.


* [114] Bryan Lim and Stefan Zohren  “Time-series forecasting with deep learning: A survey”  In _Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences_ 379.2194, 2021, pp. 20200209  DOI: 10.1098/rsta.2020.0209
  * [115] Spyros Makridakis, Evangelos Spiliotis and Vassilios Assimakopoulos  “The M5 competition: Background, organization, and implementation”  In _International Journal of Forecasting_ , 2021  DOI: 10.1016/j.ijforecast.2021.07.007
  * [116] Chen Liang and Yifeng Lu  “Using AutoML for time series forecasting” accessed: 2022-01-25  In _Google AI Blog_ Google Research, 2020  URL: http://ai.googleblog.com/2020/12/using-automl-for-time-series-forecasting.html