# ISQuant: apply squant to the real deployment
*Note: Sub-titles are not captured in Xplore and should not be used ††thanks: Identify applicable funding agency here. If none, delete this.

Dezan Zhao

###### Abstract

The model quantization technique of deep neural networks has garnered
significant attention and has proven to be highly useful in compressing model
size, reducing computation costs, and accelerating inference. Many researchers
employ fake quantization for analyzing or training the quantization process.
However, fake quantization is not the final form for deployment, and there
exists a gap between the academic setting and real-world deployment.
Additionally, the inclusion of additional computation with scale and zero-
point makes deployment a challenging task.

In this study, we first analyze why the combination of quantization and
dequantization is used to train the model and draw the conclusion that fake
quantization research is reasonable due to the disappearance of weight
gradients and the ability to approximate between fake and real quantization.
Secondly, we propose ISQuant as a solution for deploying 8-bit models. ISQuant
is fast and easy to use for most 8-bit models, requiring fewer parameters and
less computation. ISQuant also inherits the advantages of SQuant, such as not
requiring training data and being very fast at the first level of
quantization. Finally We conduct some experiments and found the results is
acceptable.our code is available at https://github.com/

###### Index Terms:

quantization

##  I Introduction

Model compression[1] now is nessesary for model acceleration and deployment.
Althoght the computation ability and memory of hardware become bigger and
stronger, The neural network become bigger and bigger. the deployment is still
a problem. To get faster rate and less enermy need, Model compression is still
needed. The model compression technique includes prune [2] CHEX [3] Ps and Qs
[4] Once for All [5] Any-precision [6], quantization, distill, the light
nerual network design, and nas. the quantization often is the Compatible
technique for other compression technique. For example, we can prune and
quantize the same time, etc. the deep compression [7] do quantization after
the prune, the Apq[8] Joint search for network architecture. After we do the
distil we can quantize. What’s more, we can do the distill to propose the
accuray after the quantization.inference of neural networks. To reduce the
memory bandwidth and foot print, model size is always needed to compression.
The reason why we can do model compression is that the models is usually over-
parameterized. After the quantization, less FLOPS and less memory footprint is
needed, and the inference’s rate is faster.

The Quantization has achieve great progress recent years. The bits become
lower and accuracy become higher. From BWN [9], TWN [10], Xnor-net [9], we
must take the whole network and finetune, the AdaQuant [11] ACIQ[12], QDrop
[13] only use some train and not requires the label. What’s more, train data
is not nessesary For some work. The work squant has pushes the accuracy and
processing time of data free quantization to a new frontier. AdaRound [14]
research found that the how round affect the accuracy.DFQ [15] can do the
quantization without the train data. SQuant [15] is also without the train
data and can calibrate faster.

However, most work focus on the fake quantization accuracy, which is not the
last form to deployment.[16] do the real deployment on hardware different
platforms, found that there is a nonnegligible accuracy gap between the real
deployment and academic setting.

a very hot directions is dummed as extreme bits quantization which can train
the model into bnn, or bwn. 4 bits also can be see n good performance[17][18].
But the performance may drop sharply. we can also use the fixed-precision
which for different layer allocate different bits num. But many devices can
only support 8bits acceleration. So we adpot 8bits as our objective bits num.
The 8bits still is the common bits choice for application. Because the lower
bits, the lower accuracy. What’s more, many hardware device supports 8bits
gemm acceleration. So We take the 8bits as our object bits.

we can carry on 8bits quantization with QAT, etc google’s earlier work[19],
F8Net[20], also can do it with PTQ, such as EasyQuant[21].

because the privacy of data, some work need to do quantization without train
data. The famous work is DFQ[15], Squant[22]

Some works focus on the special nerual network architectures.do quantization
with transformer[23].

The Hessian-based[24] method is a promising direction, related work is
Hawq[25],Hawq-v3[26], Squant[22]

Here are our contribution below:

  * •

we explain why use the fake quantization to analyse and train.

  * •

we develop the squant to isquant, with less parameters and less computation
cost.

  * •

we conduct experiments to prove the ISQuant and cause negligible relative
performance drop, inherit the squant’s fast quantization rate and no need for
train data.

Deploment work is expected to get some inspiration from this work.

##  II PRELIMINARIES

###  II-A notations

number set: a set of numbers which are the same type. The number set can be a
tensor or a part of tensor. For example, we can think the whole Filters of a
convolution is a number set, or we can think that every filter of convolution
is a different number set. The function of the tensor can be ignored, the
distribution characteristic of data is considered.

signed number set: the numbers in the number set is signed, some numbers are
negative and some numbers are positive, the others are zero. For example, The
weight tensor is signed number.

unsigned number set: the numbers in the number set is all positive and zero.
There is no negative number is in the set. For example, the activation tensor
after the relu operation is a unsigned number set.

Transform Zone: from a number set of one type to a number set of another type.
We can classify the zone into there zones:the real zone, the quantized zone
and the dequantized zone, respectively are float type, integer type and float
type. We can call them the real number set, the quantized number set and the
dequantized number set.

we can describe the quantization as two transformation.

quantize: Transpose a number set from the real zone to the quantized zone.

deqauntize: Transpose a number set from the quantized zone to the dequantized
zone.

quantization error : the error between the real number set and the dequantized
number set.

###  II-B taxonomy

we can devide quantization technique into different categories according
different norm.

  * •

uniform quantization and non-uniform quantization. google’s
whitepaper[27].thinking two number sets $A$ and $B$: we may set the range of A
is $[a_{0},a_{1}]$, and the range of B is $[b_{0},b_{1}]$. The uniform
quantization depends on special hardware. This paper only consider the uniform
quantization.

  * •

symmetric quantization and asymmetric quantization.

  * •

static quantization and dynamic quantization. For dynamic quantization, the
zero point and the scale factor is computed for every input. The static
quantization zero point and scale factor is computed once, then input that
follow can share this zero point and scale factor. In this paper, we use the
static quantization. The dynamic quantization theoretically can achieve higher
accuraccy because the quantization parameter is computed for the specific
data. But it will increase the computation cost.

  * •

weight quantization and activation quantization. The quantized number set is
weight tensor of conv or fc, called weight quantization. The quantized number
set is activation tensor, we call activation quantization. If we only quantize
the weight, called weight only quantization. To reduce static memory footprint
of model size not during inference. ususally weight quantization and
activation quantization is done at the same time.

  * •

post training quantization(PTQ) and quantization-aware training(QAT). PTQ
means quantization during the training, it requires all the train dataset just
like train the float model. PTQ means do quantization after the training, it
does not need the train label but some train datasets.

  * •

per-channel quantization and per-tensor quantization. The per-channel means
every filter of one convolution has one scale coresponding to activation per-
channel. per-tensor means the filtes, as a tensor, share a scale factor.

  * •

extremely low bit and mixed-precision quantization we call 1bits quantization
and 2bits is extremely low bit quantization. The extremely low bit
quantization causes the nonnegligible accuracy drop. Now 4bits can get a
result which is almost equivalent to original accuracy by QAT methods, but the
real deployment accuracy also drops according to [16].To get a trade-off
performance between accuracy and bits number, mixed-precision quantization is
applied. The mixed-precision decides bits of a layer according to its
sensitivity to bits number. academic research is always make the first layer
of model unquantized or allocate a big bits number, because generally
speaking, the first layer is every sensitive to the bits number. Hawq-v2[28]
shows that it is a good sensitivity metric to compute the average of all of
the Hessian eigenvalues.

we can also classify according to the theory. the DSQ classify the
quantization solutions into four levels:

  * •

no data and no backpropogation

  * •

requires data and no backpropagation

  * •

requires data and requires backpropagation but just need fineting

  * •

requires data and need backpropagation but needs to be trained from scratch

The first level, also need data, but the data is random data, like squant or
generated from some rules. The second level, often named Post-training
quantization, named as PTQ, but some work, like Adaround[14], QDrop[13],
Brecq[29], still needs backprogation, but needs some train data, and use the
original model’s output as the label. So we describe the second levels as
requires some train data and no train label.The three level, named as
quantization-awre training, abbre as QAT,like Pact[30],LSQ[31], and LSQ+[32]
and so on, has push the quantization training to new frontier. so we depict
the quantization level as the four categories more precisely. The proposed
ISQuant belongs to the first level.

  * •

no train data and no train label.

  * •

need some train data and no train label.

  * •

need whole train data and train label but finetune.

  * •

need whole train data and train label needs to be trained from scratch.

The paper F8Net[20] classify the quantization into three categories: simulated
quant, Inter-only quant and Fixed quant, this paper we concern the simulated
quant, because the simulated quant is an intermidiate stage, with the
convolution operation is integer, and can extend to other quant. We hope our
work can give people some thought.

###  II-C squant

squant’s knowledge. squant adpot the Hessian-based approach which is one of
most promising analysis methods to improve the performance of quantization.The
optimization is squant has three goals.firstly, make very weight element
quantized, at the goal absolute quantization error smaller than 0.5, called
EQ. This is the primitive nearest-round method. secondly, make absolute sum of
error(ASE) is smaller than 0.5, called KQ. Thirdly, called CQ.

##  III Methodology

as [33] depicted, the quantization can be classified into the affine
quantization and scale quantization.

$\displaystyle s=\frac{2^{b}-1}{\alpha-\beta}$ (1) $\displaystyle
z=-round(\beta*s)-2^{b-1}$ (2)

quantized zone’s data range size is $2^{b}-1$ for b bits quantization, and the
real zone’s data range size is $\alpha-\beta$. Here s means the scale factor
and the z means the zero-point, zero-point is the integer number in the
quantized zone that the zero number in the real zone maps to.

for the scale quantization

$s=\frac{2^{b}-1}{\alpha}$ (3)

for signed number set,

$\begin{split}x_{q}&=quantize(x,b,s)\\\
&=clip(round(s,x),-2^{b-1}+1,2^{b-1}-1)\end{split}$ (4)

for unsigned number set,

$\begin{split}x_{q}&=quantize(x,b,s)\\\ &=clip(round(s,x),0,2^{b})\end{split}$
(5)

the scale quantization is considered as the special situation where $\beta=0$.

the picture and be shown about the training process and the real: as [16]
depicted, the QAT can be approximated as

$y_{ij}=\sum_{k=1}^{n}\hat{w}_{ik}\hat{x}_{kj}=s_{w}s_{x}\sum_{k=1}^{n}(\overline{w}_{ik}\overline{x}_{kj}-z_{w}\overline{x}_{kj}-z_{x}\overline{w}_{ik}+z_{w}z_{x})$
(6)

the computation is complex and destroy original convolution operation because
the additional operation. consider the simple convolution example as Fig. 1.

Figure 1: Example of a Convolution.

take this for example.

$\begin{split}y_{0}&=\hat{x}_{0}*\hat{w}_{0}+\hat{x}_{1}*\hat{w}_{1}+\hat{x}_{3}*\hat{w}_{2}+\hat{x}_{4}*\hat{w}_{3}\\\
&=({x}_{0}-z_{x})*s_{x}*({w}_{0}-z_{w})*s_{w}+\\\
&\quad(x_{1}-z_{x})*s_{x}*({w}_{1}-z_{w})*s_{w}+\\\
&\quad(x_{3}-z_{x})*s_{x}*(w_{2}-z_{w})*s_{w}+\\\
&\quad(x_{4}-z_{x})*s_{x}*(w_{3}-z_{w})*s_{w}\\\
&=s_{x}s_{w}(p_{0}+p_{1}+p_{2}+p_{3})\end{split}$ (7)

where

$\displaystyle p_{0}=x_{0}w_{0}+x_{1}w_{1}+x_{3}w_{2}+x_{4}w_{2}$ (8)
$\displaystyle p_{1}=z_{w}(x_{0}+x_{1}+x_{3}+x_{4})$ (9) $\displaystyle
p_{2}=z_{x}(w_{0}+w_{1}+w_{2}+w_{3})$ (10) $\displaystyle p_{3}=4z_{x}z_{w}$
(11)

the item $p_{1},p_{2},p_{3}$ destroy the original convolution structure.

it is hard to deploy because the zero-point, which incurs additional
computation.the example is this. for the 3*3 input and 2*2 conv, and the
output, the picture is like below.

$conv_{O}ut=sdfdf$ (12)

Figure 2: Example of a figure caption.

Figure 3: Example of a figure caption.

why we consider the fake quantization to analyse the quantization problem. we
can explain like this.

  * •

approximate is good, we can use the tailor to.

  * •

can train. if we take the real deployment runtime of quantization, when the
loss backward,

$y^{l}=\frac{\overline{y}_{l}}{s_{w}*s_{l}}$ (13)

and

$\frac{\partial loss}{\partial\overline{y}_{l}}=\frac{\partial loss}{\partial
y_{l}}*\frac{\partial y_{l}}{\partial\overline{y}_{l}}$ (14)

then

$\frac{\partial loss}{\partial\overline{y}_{l}}=\frac{\partial loss}{\partial
y_{l}}*\frac{1}{s_{w}*s_{l}}$ (15)

$s_{w}*s_{l}$ is always a big number, and because the grad disappear, the
$\frac{\partial loss}{\partial y_{l}}$ is ususally a small value, which leads
to the $\frac{\partial loss}{\partial\overline{y}_{l}}$ tend to zero, the
weight will not update. with the fake quantization We can use STE[34] to do
the backpropagation.

however, the item with zero point is always hard to computer, so we relax the
equation to as the zero-point to real 0, then the equation will become

$y_{ij}=\sum_{k=1}^{n}\hat{w}_{ik}\hat{x}_{kj}=s_{w}s_{x}\sum_{k=1}^{n}(\overline{w}_{ik}\overline{x}_{kj})$
(16)

the equation is easier to do and can be used.

we develop Squant as ISquant, which is faster to quantize and can do the
quantization with free data.

we use 8bit as the destination goal is because that it is the widely used
bits. and lower bits may lead to big drop, which is sensitive to the symmetric
and per-layer.

our scheme can be describe like this:

  * •

use symmetric not asymmetric. Zero-point is still set to zero.

  * •

use per-tensor. per-tensor is less some parameters that per-channel, but
reduce many computation.

  * •

use 8bits.

  * •

ues squant. based on squant enable us to do quantization very fast and gain
good performance without train dataset.

the advantage of our scheme is like this:

  * •

easy to deploy, can use the original structure of conv

  * •

less parameters and less computation. ISQuant change the perchannel scheme to
per-tensor scheme, and change the asymmetric to symmetric form, reduce the
paramzer

  * •

fast, squant’s advantage.

Folding Bn: [16]introduces four strategies for Batch Normlization Fold.
Because our work does not need train, so we just fold the BN layers use the
naive style.

$\displaystyle w_{fold}=w\frac{\gamma}{\sqrt{\sigma^{2}+\epsilon}}$ (17)
$\displaystyle
b_{fold}=\beta+(b-\mu)\frac{\gamma}{\sqrt{\sigma^{2}+\epsilon}}$ (18)

$w_{fold}$ means the folded weight,$b_{fold}$means the folded bias, $w$means
the original weight, $b$ means the original bias, $u$ is the running mean,
$\sigma^{2}$ is variance, $gamma$ is the affine weight, $\beta$ is the affine
bias, $\epsilon$ is a small value for the numerical stability. After the
operation, BN layers is removed. We do the fuse operation before the
quantization as a pre-process.

##  IV Results

TABLE I: 8 bits quantization of different methods | PerLayer | symmetric | FBN | Conv Int | No BP | No SD | resnet18 | resnet50 | inceptionv3 | SqueezeNext
---|---|---|---|---|---|---|---|---|---|---
Baseline |  |  |  |  |  |  | 71.47 | 77.74 | 78.81 | 69.38
DFQ | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | 69.70 | 77.65 | - | -
ZeroQ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | 71.43 | 77.68 | 78.78 | 68.18
GDFQ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | 70.68 | 77.51 | 78.62 | 68.22
SQuant | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | 71.47 | 77.71 | 78.79 | 69.22
ISQuant | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 71.046 | 77.140 | 78.688 | 68.342
FBN mean fold batch norm, No BP means no backprogation, No SD means no
synthetic data, Conv Int means convolution computation use integer.

We firstly study the accuracy of the different algorithms on the dataset
imagenet on 8 bits number setting. From Table I, we can see that even our
ISQuant satisfy all the six conditions, the relative error still drop
negligibly. The resnet18 is on the $>=71\%$, resnet50 $>77\%$, inceptionv3
$>=78\%$, and the SqueezeNext[35] is $>=68\%$. The SqueezeNext drop about
$0.9\%$ compare the squant, that’s because network architecture of SqueezeNext
is the light network, which is more sensitive to the zero-point and per-layer
settings. Note the ISQuant has fuse the batchnorm into convolution, and the
convolution operate with the int8. we can see that although our ISQuant use
convolution int, still get good performance.

TABLE II: other bits performance of ISQuant | resnet18 | resnet34 | resnet50 | resnet101 | VGG19 | InceptionV3 | InceptionV4 | SqueezeNext
---|---|---|---|---|---|---|---|---
Baseline | 71.47 | 75.178 | 77.74 | 78.102 | 73.810 | 78.81 | 79.392 | 69.38
8bit | 71.046(0.424) | 75.144(0.034) | 77.140(0.599) | 77.974(0.128) | 73.102(0.707) | 78.688(0.121) | 79.120(0.272) | 68.342(1.038)
7bit | 70.786(0.684) | 74.904(0.274) | 76.846(0.893) | 77.872(0.23) | 73.108(0.702) | 78.478(0.332) | 78.968(0.423) | 67.390(1.99)
6bit | 68.726(2.744) | 73.684(1.494) | 75.204(2.536) | 77.016(1.085) | 70.132(3.678) | 77.662(1.148) | 78.086(1.306) | 64.614(4.766)

From tabel II we can see that the tha 7bits setting still get good performance
for many classic models, the error drop is within $1\%$. However we also can
see that the SqueezeNext is more sensitive to bits. it shows that the more
light the neural network is, the more sensitive to the schema that symmetric
and per-tensor. Also see that the 6bits setting drops more. Maybe need some
train. The conclusion is:

  * •

ISQuant can get good accuraccy within $1\%$ drop for many big model.

  * •

ISQuant is not very suitable for the light model architecture.

The conclusion shows the limitation of our methods, but the feature of free
data and fast rate of ISQuant is suitable for some situation. more work should
be done.

The rate of ISQuant is supposed to have the same time as squant, the table is
like table III.

TABLE III: rate of ISQuant model | resnet18 | resnet50 | InceptionV3 | SqueezeNext
---|---|---|---|---
Layers | 21 | 54 | 95 | 112
ISQuant time(ms) | 84 | 188 | 298 | 272
ZeroQ time(s) | 38 | 92 | 136 | 109
GDFQ Time(hour) | 1.7 | 3.1 | 5.7 | 4.8

##  V Conclusions

A easy deployment schema called ISQuant is proposed in this paper, which
replace the quantized convolution’s fake quantization with integer
computation. The ISQuant can use original convolution computation without
additional operation. It achieve good performance in 8bits setting and still
have acceptable performance in 7bits setting with less parameters. What’s more
it is free of data and very fast to do quantization. ISQuant can be applied to
many big models with negligible performance drop within $1\%$ at 8bits without
the requirement for train data.

## Acknowledgment

## References

  * [1] Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer.  A survey of quantization methods for efficient neural network inference.  arXiv preprint arXiv:2103.13630, 2021.
  * [2] Yann LeCun, John Denker, and Sara Solla.  Optimal brain damage.  Advances in neural information processing systems, 2, 1989.
  * [3] Zejiang Hou, Minghai Qin, Fei Sun, Xiaolong Ma, Kun Yuan, Yi Xu, Yen-Kuang Chen, Rong Jin, Yuan Xie, and Sun-Yuan Kung.  Chex: Channel exploration for cnn model compression.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12287–12298, 2022.
  * [4] B Hawks et al.  Ps and qs: Quantization-aware pruning for efficient low latency neural network inference.(2021).  arXiv preprint arXiv:2102.11289, 2021.
  * [5] Han Cai, Chuang Gan, and Song Han.  Once for all: Train onenetwork and specialize it for efficient deployment. arxivpreprint.  arXiv preprint arXiv:1908.09791, 2, 2019.
  * [6] Haichao Yu, Haoxiang Li, Humphrey Shi, Thomas S Huang, and Gang Hua.  Any-precision deep neural networks.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10763–10771, 2021.
  * [7] Song Han, Huizi Mao, and William J Dally.  Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding.  arXiv preprint arXiv:1510.00149, 2015.
  * [8] Tianzhe Wang, Kuan Wang, Han Cai, Ji Lin, Zhijian Liu, Hanrui Wang, Yujun Lin, and Song Han.  Apq: Joint search for network architecture, pruning and quantization policy.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2078–2087, 2020.
  * [9] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi.  Xnor-net: Imagenet classification using binary convolutional neural networks.  In European conference on computer vision, pages 525–542. Springer, 2016.
  * [10] Fengfu Li, Bo Zhang, and Bin Liu.  Ternary weight networks.  arXiv preprint arXiv:1605.04711, 2016.
  * [11] Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry.  Improving post training neural quantization: Layer-wise calibration and integer programming.  arXiv preprint arXiv:2006.10518, 2020.
  * [12] R Banner, Y Nahshan, E Hoffer, and D Soudry.  Aciq: Analytical clipping for integer quantization of neural networks. arxiv 2018.  arXiv preprint arXiv:1810.05723.
  * [13] Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, and Fengwei Yu.  Qdrop: Randomly dropping quantization for extremely low-bit post-training quantization.  In International Conference on Learning Representations, 2021.
  * [14] Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort.  Up or down? adaptive rounding for post-training quantization.  In International Conference on Machine Learning, pages 7197–7206. PMLR, 2020.
  * [15] Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling.  Data-free quantization through weight equalization and bias correction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1325–1334, 2019.
  * [16] Yuhang Li, Mingzhu Shen, Jian Ma, Yan Ren, Mingxin Zhao, Qi Zhang, Ruihao Gong, Fengwei Yu, and Junjie Yan.  Mqbench: Towards reproducible and deployable model quantization benchmark.  In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
  * [17] Ron Banner, Yury Nahshan, and Daniel Soudry.  Post training 4-bit quantization of convolutional networks for rapid-deployment.  Advances in Neural Information Processing Systems, 32, 2019.
  * [18] Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, et al.  $\\{$TVM$\\}$: An automated $\\{$End-to-End$\\}$ optimizing compiler for deep learning.  In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pages 578–594, 2018.
  * [19] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko.  Quantization and training of neural networks for efficient integer-arithmetic-only inference.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2704–2713, 2018.
  * [20] Qing Jin, Jian Ren, Richard Zhuang, Sumant Hanumante, Zhengang Li, Zhiyu Chen, Yanzhi Wang, Kaiyuan Yang, and Sergey Tulyakov.  F8net: Fixed-point 8-bit only multiplication for network quantization.  In International Conference on Learning Representations, 2021.
  * [21] Di Wu, Qi Tang, Yongle Zhao, Ming Zhang, Ying Fu, and Debing Zhang.  Easyquant: Post-training quantization via scale optimization.  arXiv preprint arXiv:2006.16669, 2020.
  * [22] G Cong et al.  Squant: On-the-fly data-free quantization via diagonal hessian approximation, 2022.
  * [23] Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao.  Post-training quantization for vision transformer.  Advances in Neural Information Processing Systems, 34:28092–28103, 2021.
  * [24] Zhewei Yao, Amir Gholami, Qi Lei, Kurt Keutzer, and Michael W Mahoney.  Hessian-based analysis of large batch training and robustness to adversaries.  Advances in Neural Information Processing Systems, 31, 2018.
  * [25] Zhen Dong, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer.  Hawq: Hessian aware quantization of neural networks with mixed-precision.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 293–302, 2019.
  * [26] Zhewei Yao, Zhen Dong, Zhangcheng Zheng, Amir Gholami, Jiali Yu, Eric Tan, Leyuan Wang, Qijing Huang, Yida Wang, Michael Mahoney, et al.  Hawq-v3: Dyadic neural network quantization.  In International Conference on Machine Learning, pages 11875–11886. PMLR, 2021.
  * [27] Raghuraman Krishnamoorthi.  Quantizing deep convolutional networks for efficient inference: A whitepaper.  arXiv preprint arXiv:1806.08342, 2018.
  * [28] Zhen Dong, Zhewei Yao, Daiyaan Arfeen, Amir Gholami, Michael W Mahoney, and Kurt Keutzer.  Hawq-v2: Hessian aware trace-weighted quantization of neural networks.  Advances in neural information processing systems, 33:18518–18529, 2020.
  * [29] Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu.  Brecq: Pushing the limit of post-training quantization by block reconstruction.  arXiv preprint arXiv:2102.05426, 2021.
  * [30] Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan.  Pact: Parameterized clipping activation for quantized neural networks.  arXiv preprint arXiv:1805.06085, 2018.
  * [31] Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S Modha.  Learned step size quantization.  arXiv preprint arXiv:1902.08153, 2019.
  * [32] Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak.  Lsq+: Improving low-bit quantization through learnable offsets and better initialization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 696–697, 2020.
  * [33] Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev, and Paulius Micikevicius.  Integer quantization for deep learning inference: Principles and empirical evaluation.  arXiv preprint arXiv:2004.09602, 2020.
  * [34] Yoshua Bengio, Nicholas Léonard, and Aaron Courville.  Estimating or propagating gradients through stochastic neurons for conditional computation.  arXiv preprint arXiv:1308.3432, 2013.
  * [35] Amir Gholami, Kiseok Kwon, Bichen Wu, Zizheng Tai, Xiangyu Yue, Peter Jin, Sicheng Zhao, and Kurt Keutzer.  Squeezenext: Hardware-aware neural network design.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 1638–1647, 2018.


# PRiSM: Enhancing Low-Resource Document-Level Relation Extraction with
Relation-Aware Score Calibration

Minseok Choi Hyesu Lim Jaegul Choo
KAIST AI
{minseok.choi, hyesulim<EMAIL_ADDRESS>
###### Abstract

Document-level relation extraction (DocRE) aims to extract relations of all
entity pairs in a document. A key challenge in DocRE is the cost of annotating
such data which requires intensive human effort. Thus, we investigate the case
of DocRE in a low-resource setting, and we find that existing models trained
on low data overestimate the `NA` (“no relation”) label, causing limited
performance. In this work, we approach the problem from a calibration
perspective and propose PRiSM, which learns to adapt logits based on relation
semantic information. We evaluate our method on three DocRE datasets and
demonstrate that integrating existing models with PRiSM improves performance
by as much as 26.38 F1 score, while the calibration error drops as much as 36
times when trained with about 3% of data. The code is publicly available at
https://github.com/brightjade/PRiSM.

##  1 Introduction

Document-level relation extraction (DocRE) is a fundamental task in natural
language understanding, which aims to identify relations between entities that
exist in a document. A major challenge in DocRE is the cost of annotating such
documents, requiring annotators to consider relations of all possible entity
combinations Yao et al. (2019); Zaporojets et al. (2021); Tan et al. (2022b).
However, there is a lack of ongoing studies investigating the low-resource
setting in DocRE Zhou et al. (2023), and we discover that most of the current
DocRE models show subpar performance when trained with a small set of data. We
argue that the reason is two-fold. First, the long-tailed distribution of
DocRE data encourages models to be overly confident in predicting frequent
relations and less sure about infrequent ones Du et al. (2022); Tan et al.
(2022a). Out of the 96 relations in DocRED Yao et al. (2019), a widely-used
DocRE dataset, the 7 most frequent relations account for 55% of the total
relation triples. Under the low-resource setting, chances to observe
infrequent relations become much harder. Second, DocRE models predict the `NA`
(“no relation”) label if an entity pair does not express any relation. In
DocRED, about 97% of all entity pairs have the `NA` label. With limited data,
there is a much less signal for ground-truth (GT) labels during training,
resulting in models overpredicting the `NA` label instead.

High confidence in common relations and the `NA` label and low confidence in
rare relations suggest that models may be miscalibrated. We hypothesize that
lowering the former and raising the latter would improve the overall RE
performance. At a high level, we wish to penalize logits of frequent labels
(including `NA`) and supplement logits of infrequent labels such that models
are able to predict them without seeing them much during training. To
implement such behavior, we leverage relation semantic information, which has
proved to be effective in low-resource sentence-level RE Yang et al. (2020);
Dong et al. (2021); Zhang and Lu (2022).

Figure 1: An overview of our proposed method. Top represents the original
DocRE framework. PRiSM (bottom) leverages relation descriptions to compute
scores for each relation triple. These scores are then used to reweight the
prediction logits.

In this work, we propose the Pair-Relation Similarity Module (PRiSM) that
learns to adapt logits by exploiting semantic information from label
descriptions, as depicted in Figure 1. Specifically, we compute a similarity
function for each entity pair embedding, constructed from two entities of
interest, with relation embeddings, built from corresponding label
descriptions. PRiSM then learns relation representations to output adaptive
scores for each relation triple. Note that previous work mostly utilized
relation representations for self-supervised learning Dong et al. (2021); Du
et al. (2022); Zhou et al. (2023), whereas PRiSM uses them to directly adjust
logits, which brings a calibration effect. To elaborate further, let us say
that classification logits are statistical scores and similarities are
semantic scores. We have four scenarios: 1) relation is common and GT, 2)
relation is common but not GT, 3) relation is uncommon but GT, and 4) relation
is uncommon and not GT. In Cases 1 and 4, both statistical and semantic scores
are either high or low, and thus, appending PRiSM mostly would not affect the
original RE predictions. In Case 2, the statistical score is high, but the
semantic score is low, possibly negative to penalize the statistical score.
This is the case of PRiSM decreasing the confidence of common relations and
`NA` label. In Case 3, the statistical score is low, but the semantic score is
high, which is the case of PRiSM increasing the confidence of uncommon
relations. As such, PRiSM incorporates both statistical and semantic scores
such that the confidence is adjusted regardless of the relation frequency.

Our technical contributions are three-fold. First, we propose PRiSM, a
relation-aware calibration technique that improves model performance and
adjusts model confidence on low-resource DocRE. Second, we demonstrate the
performance improvement across various state-of-the-art models integrated with
PRiSM. Third, we validate the effectiveness of our method on widely-used long-
tailed DocRE datasets and calibration metrics.

##  2 Methodology

###  2.1 Problem Formulation

Given a document $d$, a set of $n$ annotated entities
$\mathcal{E}=\\{e_{i}\\}_{i=1}^{n}$, and a pre-defined set of relations
$\mathcal{R}\cup\\{$NA$\\}$, the task of DocRE is to extract the relation
triple set
$\\{(e_{h},r,e_{t})|e_{h}\in\mathcal{E},r\in\mathcal{R},e_{t}\in\mathcal{E}\\}\subseteq\mathcal{E}\times\mathcal{R}\times\mathcal{E}$
from all possible relation triples, where $(e_{h},r,e_{t})$ denotes that a
relation $r$ holds between head entity $e_{h}$ and tail entity $e_{t}$. An
entity $e_{i}$ may appear $k$ times in the document in which we denote
corresponding instances as entity mentions $\\{m_{ij}\\}_{j=1}^{k}$. A
relation $r$ exists between an entity pair $(e_{h},e_{t})$ if any pair of
their mentions express the relation, and if they do not express any relation,
the entity pair is then labeled as NA.

###  2.2 Document-Level Relation Extraction

Given a document $d$ as an input token sequence $\bm{x}=[x_{t}]_{t=1}^{l}$,
where $l$ is the length of the token sequence, we explicitly locate the
position of entity mentions by inserting a special token “*” before and after
each mention. The presence of the entity marker has proved to be effective
from previous studies (Zhang et al., 2017; Shi and Lin, 2019; Baldini Soares
et al., 2019). The entity-marked document is then fed into a pre-trained
language model (PLM) encoder, which outputs the contextual embeddings:
$[\bm{h}_{1},\bm{h}_{2},...,\bm{h}_{l}]=\text{Encoder}(\bm{x})$. We take the
embedding of “*” at the start of each mention as its mention-level
representation $\bm{h}_{m_{ij}}$ of the entity $e_{i}$. For extracting the
entity-level representation, we apply the logsumexp pooling over all mentions
$\\{m_{ij}\\}_{j=1}^{k}$ of the entity $e_{i}$:

$\bm{h}_{e_{i}}=\log\sum\limits_{j=1}^{k}\exp\left(\bm{h}_{m_{ij}}\right).$
(1)

The logsumexp pooling is a smooth version of max pooling and has been shown to
accumulate weak signals from each different mention representation, which
results in a better performance (Jia et al., 2019). We pass the embeddings of
head and tail entities through a linear layer followed by non-linear
activation to obtain the hidden representations:
$\bm{z}_{h}=\tanh(\bm{W}_{h}\bm{h}_{e_{h}}+\bm{b}_{h})$ and
$\bm{z}_{t}=\tanh(\bm{W}_{t}\bm{h}_{e_{t}}+\bm{b}_{t})$, where
$\bm{W}_{h},\bm{W}_{t},\bm{b}_{h},\bm{b}_{t}$ are learnable parameters. Then
we calculate a score for relation $r$ between entities $h$ and $t$ by taking a
bilinear function:

$s_{(h,r,t)}=\bm{z}_{h}^{\top}\bm{W}_{r}\bm{z}_{t}+\bm{b}_{r},$ (2)

where $\bm{W}_{r},\bm{b}_{r}$ are learnable parameters.

DocRED Re-DocRED Dev Test Dev Test Model Ign $F_{1}$ $F_{1}$ Ign $F_{1}$
$F_{1}$ Ign $F_{1}$ $F_{1}$ Ign $F_{1}$ $F_{1}$ 3% training examples $(N=100)$
BERTBASE $10.27\pm 1.82$ $10.44\pm 1.90$ $11.36$ $11.50$ $28.65\pm 2.87$
$29.40\pm 3.19$ $28.77\pm 3.34$ $29.44\pm 3.67$ BERTBASE \+ PRiSM
$\mathbf{35.06}\pm 0.94$ $\mathbf{37.02}\pm 0.88$ $\mathbf{35.79}$
$\mathbf{37.88}$ $\mathbf{47.39}\pm 0.79$ $\mathbf{49.09}\pm 0.90$
$\mathbf{46.90}\pm 1.59$ $\mathbf{48.57}\pm 1.73$ RoBERTaBASE $20.70\pm 1.91$
$21.31\pm 1.87$ $21.74$ $22.25$ $39.66\pm 2.25$ $40.74\pm 1.89$ $39.42\pm
2.80$ $40.53\pm 2.43$ RoBERTaBASE \+ PRiSM $\mathbf{32.40}\pm 0.85$
$\mathbf{34.49}\pm 0.76$ $\mathbf{32.20}$ $\mathbf{34.32}$ $\mathbf{47.71}\pm
1.03$ $\mathbf{49.40}\pm 1.14$ $\mathbf{47.31}\pm 0.96$ $\mathbf{49.04}\pm
1.05$ SSAN-BERTBASE $10.92\pm 0.88$ $11.18\pm 0.89$ $11.93$ $12.16$ $28.89\pm
1.68$ $29.01\pm 1.69$ $28.64\pm 1.89$ $29.29\pm 1.94$ SSAN-BERTBASE \+ PRiSM
$\mathbf{32.86}\pm 2.35$ $\mathbf{34.76}\pm 2.50$ $\mathbf{34.00}$
$\mathbf{36.03}$ $\mathbf{46.49}\pm 1.16$ $\mathbf{48.11}\pm 1.40$
$\mathbf{46.51}\pm 1.77$ $\mathbf{48.11}\pm 2.00$ ATLOP-BERTBASE $38.99\pm
2.30$ $40.50\pm 2.07$ $40.88$ $42.37$ $49.45\pm 2.09$ $50.60\pm 1.95$
$49.24\pm 2.25$ $50.32\pm 2.13$ ATLOP-BERTBASE \+ PRiSM $\mathbf{40.59}\pm
0.68$ $\mathbf{42.09}\pm 0.66$ $\mathbf{40.94}$ $\mathbf{42.43}$
$\mathbf{50.10}\pm 0.53$ $\mathbf{51.12}\pm 0.64$ $\mathbf{50.15}\pm 1.11$
$\mathbf{51.14}\pm 1.17$ 10% training examples $(N=305)$ BERTBASE $39.84\pm
0.92$ $41.55\pm 0.99$ $40.98$ $42.98$ $52.34\pm 0.66$ $53.54\pm 0.80$
$52.34\pm 0.68$ $53.54\pm 0.84$ BERTBASE \+ PRiSM $\mathbf{46.01}\pm 0.12$
$\mathbf{48.02}\pm 0.13$ $\mathbf{45.52}$ $\mathbf{47.83}$ $\mathbf{58.10}\pm
0.31$ $\mathbf{59.86}\pm 0.27$ $\mathbf{57.75}\pm 0.65$ $\mathbf{59.53}\pm
0.51$ RoBERTaBASE $43.42\pm 1.09$ $45.20\pm 1.09$ $43.78$ $45.63$ $54.82\pm
1.85$ $56.10\pm 1.80$ $55.36\pm 2.18$ $56.67\pm 2.06$ RoBERTaBASE \+ PRiSM
$\mathbf{46.60}\pm 0.20$ $\mathbf{48.57}\pm 0.29$ $\mathbf{47.02}$
$\mathbf{49.22}$ $\mathbf{59.51}\pm 0.36$ $\mathbf{61.19}\pm 0.32$
$\mathbf{59.08}\pm 0.61$ $\mathbf{60.80}\pm 0.52$ SSAN-BERTBASE $40.00\pm
1.62$ $41.65\pm 1.63$ $41.11$ $43.03$ $53.57\pm 0.83$ $54.86\pm 0.81$
$53.67\pm 1.55$ $54.94\pm 1.52$ SSAN-BERTBASE \+ PRiSM $\mathbf{46.14}\pm
0.15$ $\mathbf{48.18}\pm 0.09$ $\mathbf{45.48}$ $\mathbf{47.72}$
$\mathbf{58.47}\pm 0.39$ $\mathbf{60.17}\pm 0.36$ $\mathbf{58.21}\pm 0.31$
$\mathbf{59.93}\pm 0.19$ ATLOP-BERTBASE $49.93\pm 1.11$ $51.61\pm 1.16$
$50.04$ $51.85$ $60.38\pm 0.46$ $61.52\pm 0.29$ $60.46\pm 0.55$ $61.54\pm
0.29$ ATLOP-BERTBASE \+ PRiSM $\mathbf{50.20}\pm 0.68$ $\mathbf{51.83}\pm
0.64$ $\mathbf{50.29}$ $\mathbf{52.17}$ $\mathbf{60.58}\pm 0.18$
$\mathbf{61.68}\pm 0.17$ $\mathbf{60.90}\pm 0.37$ $\mathbf{61.97}\pm 0.40$
100% training examples $(N=3053)$ BERTBASE $57.15\pm 0.17$ $59.18\pm 0.05$
$57.02$ $59.35$ $71.70\pm 0.61$ $73.17\pm 0.55$ $71.01\pm 0.88$ $72.48\pm
0.78$ BERTBASE \+ PRiSM $\mathbf{57.82}\pm 0.10$ $\mathbf{59.93}\pm 0.15$
$\mathbf{57.17}$ $\mathbf{59.52}$ $\mathbf{72.92}\pm 0.07$ $\mathbf{74.25}\pm
0.07$ $\mathbf{72.35}\pm 0.07$ $\mathbf{73.69}\pm 0.11$ RoBERTaBASE $58.24\pm
0.36$ $60.19\pm 0.38$ $58.00$ $60.10$ $74.00\pm 0.20$ $75.20\pm 0.20$
$73.56\pm 0.04$ $74.75\pm 0.04$ RoBERTaBASE \+ PRiSM $\mathbf{58.73}\pm 0.09$
$\mathbf{60.70}\pm 0.02$ $\mathbf{58.36}$ $\mathbf{60.51}$ $\mathbf{74.50}\pm
0.09$ $\mathbf{75.71}\pm 0.06$ $\mathbf{74.17}\pm 0.10$ $\mathbf{75.38}\pm
0.10$ SSAN-BERTBASE $57.59\pm 0.35$ $59.62\pm 0.24$ $57.71$ $59.79$ $72.59\pm
0.15$ $74.01\pm 0.15$ $71.95\pm 0.11$ $73.37\pm 0.11$ SSAN-BERTBASE \+ PRiSM
$\mathbf{58.20}\pm 0.20$ $\mathbf{60.27}\pm 0.14$ $\mathbf{58.02}$
$\mathbf{60.27}$ $\mathbf{73.22}\pm 0.10$ $\mathbf{74.65}\pm 0.07$
$\mathbf{72.37}\pm 0.19$ $\mathbf{73.80}\pm 0.18$ ATLOP-BERTBASE $59.22\pm
0.17$ $61.18\pm 0.10$ $\mathbf{58.99}$ $\mathbf{61.08}$ $72.78\pm 0.46$
$73.73\pm 0.37$ $72.60\pm 0.41$ $73.51\pm 0.38$ ATLOP-BERTBASE \+ PRiSM
$\mathbf{59.51}\pm 0.09$ $\mathbf{61.31}\pm 0.05$ $58.80$ $60.77$
$\mathbf{72.85}\pm 0.29$ $\mathbf{73.80}\pm 0.35$ $\mathbf{72.61}\pm 0.59$
$\mathbf{73.53}\pm 0.53$

Table 1: Performance (%) on DocRED and Re-DocRED. Better scores between with
and without PRiSM are in bold. The test results for DocRED are obtained by
submitting the best dev model predictions to
CodaLab222https://codalab.lisn.upsaclay.fr/.

###  2.3 PRiSM

Model Macro Macro@500 Macro@200 Macro@100 $N=100$ BERTBASE $0.36\pm 0.05$ $0$
$0$ $-$ BERTBASE \+ PRiSM $\bm{7.77}\pm 1.87$ $\bm{4.08}\pm 0.43$
$\bm{0.44}\pm 0.33$ $-$ RoBERTaBASE $1.18\pm 0.28$ $0$ $0$ $-$ RoBERTaBASE \+
PRiSM $\bm{6.41}\pm 0.77$ $\bm{2.31}\pm 0.82$ $\bm{0.38}\pm 0.28$ $-$ $N=305$
BERTBASE $9.31\pm 1.59$ $3.70\pm 1.46$ $0.29\pm 0.17$ $0$ BERTBASE \+ PRiSM
$\bm{20.19}\pm 0.70$ $\bm{14.91}\pm 0.64$ $\bm{7.73}\pm 0.17$ $\bm{2.19}\pm
1.16$ RoBERTaBASE $14.80\pm 0.51$ $9.13\pm 0.61$ $3.74\pm 0.35$ $0.83\pm 0.89$
RoBERTaBASE \+ PRiSM $\bm{21.03}\pm 0.27$ $\bm{15.69}\pm 0.48$ $\bm{8.37}\pm
0.16$ $\bm{2.63}\pm 1.25$ $N=3053$ BERTBASE $38.31\pm 0.39$ $34.06\pm 0.45$
$26.07\pm 0.72$ $\bm{19.73}\pm 0.96$ BERTBASE \+ PRiSM $\bm{38.89}\pm 0.52$
$\bm{34.57}\pm 0.59$ $\bm{26.51}\pm 0.65$ $19.57\pm 0.71$ RoBERTaBASE
$38.67\pm 1.12$ $34.28\pm 1.22$ $26.14\pm 1.44$ $18.69\pm 1.70$ RoBERTaBASE \+
PRiSM $\bm{39.12}\pm 0.57$ $\bm{34.72}\pm 0.69$ $\bm{26.45}\pm 1.01$
$\bm{19.23}\pm 1.55$

Table 2: Dev performance (%) on low-frequency relations in DocRED. Test
results cannot be reported because the labels are not accessible.

Following previous work Zhang and Lu (2022), we feed relation descriptions to
a PLM encoder to obtain the relation embedding $\bm{z}_{r}$ for relation $r$.
The details of the relation descriptions used can be found in Appendix A.4. We
then construct the entity pair-level representation $\bm{z}_{(h,t)}$ by
mapping the head and tail embeddings to a linear layer followed by non-linear
activation:
$\bm{z}_{(h,t)}=\tanh(\bm{W}_{(h,t)}[\bm{z}_{h};\bm{z}_{t}]+\bm{b}_{(h,t)})$,
where $\bm{z}_{h}$, $\bm{z}_{t}$ are concatenated and $\bm{W}_{(h,t)}$,
$\bm{b}_{(h,t)}$ are learnable parameters. An adaptive score for relation $r$
between entities $h$ and $t$ is computed by taking a similarity function
between the entity pair embedding and relation embedding:
$s^{\prime}_{(h,r,t)}=sim(\bm{z}_{(h,t)},\bm{z}_{r})$, where $sim(\cdot)$ is
cosine similarity. Formally, the probability of relation $r$ between entities
$h$ and $t$ is simply an addition of two scores followed by sigmoid
activation:

$P(r\mid e_{h},e_{t})=\sigma(s_{(h,r,t)}+\lambda s^{\prime}_{(h,r,t)}),$ (3)

where $\lambda$ is the scale factor. Finally, we optimize our model with the
binary cross-entropy (BCE) loss:

$\mathcal{L}=-\frac{1}{T}\sum_{<h,t>}\sum_{r}\verb|BCE|(P(r|e_{h},e_{t}),\bar{y}_{(h,r,t)}),$
(4)

where $\bar{y}$ is the target label and $T$ is the total number of relation
triples.

##  3 Experiments

###  3.1 Dataset

We evaluate our framework on three public DocRE datasets. DocRED (Yao et al.,
2019) is a widely-used human-annotated DocRE dataset constructed from
Wikipedia and Wikidata. Re-DocRED (Tan et al., 2022b) is a revised dataset
from DocRED, addressing the incomplete annotation problem. DWIE (Zaporojets et
al., 2021) is a multi-task document-level information extraction dataset
consisting of news articles collected from Deutsche Welle. Dataset statistics
are shown in Table 5.

Figure 2: Reliability diagram for BERTBASE when trained with 3% of DocRED
data.

###  3.2 Implementation Details

Our framework is built on PyTorch and Huggingface’s Transformers library (Wolf
et al., 2020). We use the cased BERT (Devlin et al., 2019) and RoBERTa (Liu et
al., 2019) for encoding the text and optimize their weights with AdamW
(Loshchilov and Hutter, 2019). We tune our hyperparameters to maximize the
$F_{1}$ score on the development set. The additional implementation details
are included in Appendix B. During inference, we predict all relation triples
that have probabilities higher than the F1-maximizing threshold found in the
development set. We conduct our experiments with three different random seeds
and report the averaged results. Following Yao et al. (2019), all models are
evaluated on $F_{1}$ and Ign $F_{1}$, where Ign $F_{1}$ excludes the relations
shared by the training and development/test sets. Moreover, we measure Macro,
which computes the average of per-class $F_{1}$, and Macro@500, Macro@200, and
Macro@100, targeting rare relations where the frequency count in the training
dataset is less than 500, 200, and 100, respectively.

Dev Test Model $F_{1}$ Macro $F_{1}$ Macro $N=100$ BERTBASE $11.97\pm 1.78$
$1.79\pm 0.27$ $12.41\pm 2.29$ $1.87\pm 0.32$ BERTBASE \+ PRiSM $\bm{45.20}\pm
1.60$ $\bm{10.34}\pm 1.91$ $\bm{44.31}\pm 1.47$ $\bm{9.47}\pm 1.46$
RoBERTaBASE $50.27\pm 1.57$ $8.55\pm 0.56$ $48.29\pm 1.74$ $8.95\pm 0.86$
RoBERTaBASE \+ PRiSM $\bm{55.51}\pm 1.11$ $\bm{12.76}\pm 2.03$ $\bm{54.23}\pm
1.24$ $\bm{13.80}\pm 0.64$ $N=305$ BERTBASE $52.98\pm 0.76$ $15.68\pm 1.71$
$52.05\pm 0.60$ $14.80\pm 0.63$ BERTBASE \+ PRiSM $\bm{58.23}\pm 0.40$
$\bm{24.62}\pm 0.59$ $\bm{57.05}\pm 0.23$ $\bm{22.43}\pm 0.88$ RoBERTaBASE
$65.45\pm 1.94$ $21.72\pm 1.29$ $62.39\pm 1.29$ $20.39\pm 0.76$ RoBERTaBASE \+
PRiSM $\bm{71.18}\pm 1.98$ $\bm{28.36}\pm 1.53$ $\bm{67.12}\pm 2.02$
$\bm{25.82}\pm 0.34$ $N=587$ BERTBASE $62.06\pm 0.33$ $25.17\pm 0.37$
$60.78\pm 0.25$ $22.93\pm 0.40$ BERTBASE \+ PRiSM $\bm{66.81}\pm 0.56$
$\bm{28.17}\pm 0.54$ $\bm{66.53}\pm 0.52$ $\bm{29.31}\pm 1.13$ RoBERTaBASE
$76.23\pm 0.72$ $31.71\pm 0.13$ $74.07\pm 0.77$ $28.72\pm 1.54$ RoBERTaBASE \+
PRiSM $\bm{78.43}\pm 0.12$ $\bm{32.85}\pm 0.37$ $\bm{78.13}\pm 0.61$
$\bm{33.66}\pm 1.24$

Table 3: Performance (%) on the DWIE dataset.

###  3.3 Experimental Results

To simulate the low-data setting, we reduce the number of training documents
$N$ to 100 and 305, which is about 3% and 10% of the original data. To create
each of the settings, we repeat random sampling until the label distribution
resembles that of the full data. As shown in Table 2, we observe that
performance increases consistently across different models when appended with
PRiSM. Particularly, PRiSM improves performance by a large margin when trained
with just 3% of data, as much as 24.43 Ign $F_{1}$ and 26.38 $F_{1}$ on the
test set of DocRED for BERTBASE. We also test PRiSM on RoBERTaBASE and two
state-of-the-art models SSAN Xu et al. (2021) and ATLOP Zhou et al. (2021) and
notice a similar trend, indicating that our method is effective on various
existing models. We additionally evaluate PRiSM using macro metrics in Table 2
and observe that adding PRiSM improves performance on infrequent relations,
especially in the low-data setting. Lastly, we validate our method on a
different dataset DWIE, as illustrated in Table 3.

###  3.4 Calibration Evaluation

We measure model calibration on two metrics: expected calibration error (ECE)
Naeini et al. (2015) and adaptive calibration error (ACE) Nixon et al. (2019).
ECE partitions predictions into a fixed number of bins and computes a weighted
average of the difference between accuracy and confidence over the bins, while
ACE puts the same number of predictions in each bin. We compare with general
calibration methods such as temperature scaling (TS) Guo et al. (2017) and
class-distribution-aware TS (CDA-TS) Islam et al. (2021). As reported in Table
4, PRiSM outperforms other methods in both metrics, while also maintaining a
comparable RE performance. We also visualize with a reliability diagram
DeGroot and Fienberg (1983); Niculescu-Mizil and Caruana (2005) in Figure 2.
We observe that PRiSM effectively lowers the confidence of the `NA` label and
raises the confidence of low-frequency relations (bottom 89). For high-
frequency relations (top 7), confidence is adjusted in both ways. In any case,
PRiSM displays the most stable, closest line to the perfect calibration (blue
line).

$N=100$ $N=305$ Method $F_{1}$($\uparrow$) ECE($\downarrow$) ACE($\downarrow$)
$F_{1}$($\uparrow$) ECE($\downarrow$) ACE($\downarrow$) Uncalibrated 10.82
0.359% 0.379% 42.56 0.137% 0.164% TS 38.19 0.144% 0.173% 48.49 0.053% 0.062%
CDA-TS 37.82 0.139% 0.167% 48.54 0.057% 0.078% PRiSM (ours) 37.84 0.010%
0.020% 48.10 0.023% 0.020%

Table 4: Comparison of calibration errors (with 10 bins) under a low-resource
setting of DocRED.

##  4 Related Work

With the introduction of DocRED (Yao et al., 2019), many approaches were
proposed to extract relations from a document (Wang et al., 2019; Ye et al.,
2020; Zhang et al., 2021; Xu et al., 2021; Zhou et al., 2021; Xie et al.,
2022). The long-tailed data problem of DocRE has been addressed in some
studies (Du et al., 2022; Tan et al., 2022a), as well as low-resource DocRE
Zhou et al. (2023); however, most require additional pretraining, which is
compute- and cost-intensive, while PRiSM only requires adjusting logits in
existing models. Low-resource RE has been extensively studied at the sentence
level, and we specifically focus on leveraging label information Yang et al.
(2020); Dong et al. (2021); Zhang and Lu (2022) in which PRiSM applies it to
the document level. In contrast to prior work in calibration Guo et al.
(2017); Islam et al. (2021), our approach is relation-aware, updating logits
at a much finer granularity.

##  5 Conclusion and Future Work

In this work, we propose a simple modular framework PRiSM, which exploits
relation semantics to update logits. We empirically demonstrate that our
method effectively improves and calibrates DocRE models where the data is
long-tailed and the `NA` label is overestimated. For future work, we can apply
PRiSM to more tasks such as event extraction and dialogue state tracking,
which also enclose long-tailed data and overestimation of “null” labels.

## Limitations

Although our approach is resilient to data scarcity, quite a few annotated
documents are still required for the model to learn the pattern. The ultimate
goal of DocRE is undoubtedly to build a model that is able to perform well on
zero-shot, but we believe our approach takes a step toward that direction.
Moreover, we process the long documents (> 512 tokens) in a very naive way, as
described in Appendix A.3, and we think that exploration of long-sequence
modeling on longer document data could further enrich the field of DocRE.

## Acknowledgements

This work was supported by Institute of Information & communications
Technology Planning & Evaluation (IITP) grant funded by the Korea government
(MSIT) (No.2019-0-00075, Artificial Intelligence Graduate School Program
(KAIST)), the National Supercomputing Center with supercomputing resources
including technical support (KSC-2022-CRE-0312), and Samsung Electronics Co.,
Ltd. We thank the anonymous reviewers for their constructive feedback.

## References

  * Baldini Soares et al. (2019) Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019\.  Matching the blanks: Distributional similarity for relation learning.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2895–2905, Florence, Italy. Association for Computational Linguistics.
  * DeGroot and Fienberg (1983) Morris H DeGroot and Stephen E Fienberg. 1983.  The comparison and evaluation of forecasters.  _Journal of the Royal Statistical Society: Series D (The Statistician)_ , 32(1-2):12–22.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dong et al. (2021) Manqing Dong, Chunguang Pan, and Zhipeng Luo. 2021.  MapRE: An effective semantic mapping approach for low-resource relation extraction.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 2694–2704, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Du et al. (2022) Yangkai Du, Tengfei Ma, Lingfei Wu, Yiming Wu, Xuhong Zhang, Bo Long, and Shouling Ji. 2022.  Improving long tailed document-level relation extraction via easy relation augmentation and contrastive learning.  _arXiv preprint arXiv:2205.10511_.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017.  On calibration of modern neural networks.  In _International conference on machine learning_ , pages 1321–1330. PMLR.
  * Islam et al. (2021) Mobarakol Islam, Lalithkumar Seenivasan, Hongliang Ren, and Ben Glocker. 2021.  Class-distribution-aware calibration for long-tailed visual recognition.  _arXiv preprint arXiv:2109.05263_.
  * Jia et al. (2019) Robin Jia, Cliff Wong, and Hoifung Poon. 2019.  Document-level n-ary relation extraction with multiscale representation learning.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3693–3704, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_.
  * Naeini et al. (2015) Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015.  Obtaining well calibrated probabilities using bayesian binning.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 29.
  * Niculescu-Mizil and Caruana (2005) Alexandru Niculescu-Mizil and Rich Caruana. 2005.  Predicting good probabilities with supervised learning.  In _Proceedings of the 22nd international conference on Machine learning_ , pages 625–632.
  * Nixon et al. (2019) Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. 2019.  Measuring calibration in deep learning. In _CVPR workshops_ , volume 2.
  * OpenAI (2023) OpenAI. 2023.  ChatGPT March 23 Version.
  * Shi and Lin (2019) Peng Shi and Jimmy Lin. 2019.  Simple bert models for relation extraction and semantic role labeling.  _arXiv preprint arXiv:1904.05255_.
  * Tan et al. (2022a) Qingyu Tan, Ruidan He, Lidong Bing, and Hwee Tou Ng. 2022a.  Document-level relation extraction with adaptive focal loss and knowledge distillation.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 1672–1681, Dublin, Ireland. Association for Computational Linguistics.
  * Tan et al. (2022b) Qingyu Tan, Lu Xu, Lidong Bing, Hwee Tou Ng, and Sharifah Mahani Aljunied. 2022b.  Revisiting DocRED - addressing the false negative problem in relation extraction.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 8472–8487, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Wang et al. (2019) Hong Wang, Christfried Focke, Rob Sylvester, Nilesh Mishra, and William Wang. 2019\.  Fine-tune bert for docred with two-step process.  _arXiv preprint arXiv:1909.11898_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Xie et al. (2022) Yiqing Xie, Jiaming Shen, Sha Li, Yuning Mao, and Jiawei Han. 2022.  Eider: Empowering document-level relation extraction with efficient evidence extraction and inference-stage fusion.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 257–268, Dublin, Ireland. Association for Computational Linguistics.
  * Xu et al. (2021) Benfeng Xu, Quan Wang, Yajuan Lyu, Yong Zhu, and Zhendong Mao. 2021.  Entity structure within and throughout: Modeling mention dependencies for document-level relation extraction.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 35, pages 14149–14157.
  * Yang et al. (2020) Kaijia Yang, Nantao Zheng, Xinyu Dai, Liang He, Shujian Huang, and Jiajun Chen. 2020\.  Enhance prototypical network with text descriptions for few-shot relation classification.  In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_, pages 2273–2276.
  * Yao et al. (2019) Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019.  DocRED: A large-scale document-level relation extraction dataset.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 764–777, Florence, Italy. Association for Computational Linguistics.
  * Ye et al. (2020) Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Peng Li, Maosong Sun, and Zhiyuan Liu. 2020.  Coreferential Reasoning Learning for Language Representation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7170–7186, Online. Association for Computational Linguistics.
  * Zaporojets et al. (2021) Klim Zaporojets, Johannes Deleu, Chris Develder, and Thomas Demeester. 2021.  Dwie: An entity-centric dataset for multi-task document-level information extraction.  _Information Processing & Management_, 58(4):102563.
  * Zhang et al. (2021) Ningyu Zhang, Xiang Chen, Xin Xie, Shumin Deng, Chuanqi Tan, Mosha Chen, Fei Huang, Luo Si, and Huajun Chen. 2021.  Document-level relation extraction as semantic segmentation.  In _Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21_ , pages 3999–4006. International Joint Conferences on Artificial Intelligence Organization.  Main Track.
  * Zhang and Lu (2022) Peiyuan Zhang and Wei Lu. 2022.  Better few-shot relation extraction with label prompt dropout.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 6996–7006, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Zhang et al. (2017) Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017.  Position-aware attention and supervised data improve slot filling.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 35–45, Copenhagen, Denmark. Association for Computational Linguistics.
  * Zhou et al. (2021) Wenxuan Zhou, Kevin Huang, Tengyu Ma, and Jing Huang. 2021.  Document-level relation extraction with adaptive thresholding and localized context pooling.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 14612–14620.
  * Zhou et al. (2023) Wenxuan Zhou, Sheng Zhang, Tristan Naumann, Muhao Chen, and Hoifung Poon. 2023.  Continual contrastive finetuning improves low-resource relation extraction.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 13249–13263, Toronto, Canada. Association for Computational Linguistics.

## Appendix

##  Appendix A Additional Dataset Details

###  A.1 Data Statistics

We report the statistics for the datasets in Table 5. The test set of DocRED
is not included in calculating % NA due to its inaccessibility. 14 documents
in DWIE are filtered out because of missing labels, and 1 document is removed
because the annotated entities did not exist in the input document.

Statistics DocRED Re-DocRED DWIE # Train 3,053 3,053 587 # Dev 1,000 500 100 #
Test 1,000 500 100 # Relation Types 97 97 66 % NA 97.05% 94.02% 97.87%

Table 5: Dataset statistics. # Relation Types includes the NA label. % NA
indicates a ratio of entity pairs having the NA label over all entity pairs.

###  A.2 Class Distribution

We count the number of ground-truth relations in the train sets and visualize
their class distribution in Figure 3. We observe that the re-annotation of Re-
DocRED further skewed the class distribution, and the DWIE dataset seems to
demonstrate a relatively less imbalanced distribution. Nevertheless, a few
classes still exhibit high frequency, which PRiSM can handle effectively.

Figure 3: Dataset class distribution.

###  A.3 Processing Long Document

For DocRED and Re-DocRED, most of the documents contain less than 512 tokens,
and thus we follow the previous work and truncate all of the inputs to 512
tokens, which is the maximum sequence length of BERT. However, we notice that
the DWIE dataset mostly contains documents much longer than 512 tokens (as
shown in Figure 4) in which the truncation hurts the performance
significantly. Therefore, we choose the most naive way of splitting the input
document into multiple chunks of length 512 and passing them through the
encoder multiple times. The performance improvement over the truncation method
is demonstrated in Table 6.

Figure 4: Lengths of input documents in training sets.

Method Dev $F_{1}$ Test $F_{1}$ $N=100$ Truncation $36.53\pm 2.09$ $35.39\pm
0.71$ Chunking (ours) $\bm{45.20}\pm 1.60$ $\bm{44.31}\pm 1.47$ $N=305$
Truncation $53.65\pm 0.63$ $51.30\pm 0.62$ Chunking (ours) $\bm{58.23}\pm
0.40$ $\bm{57.05}\pm 0.23$ $N=587$ Truncation $61.34\pm 0.52$ $59.17\pm 1.86$
Chunking (ours) $\bm{66.81}\pm 0.56$ $\bm{66.53}\pm 0.52$

Table 6: Performance comparison of long document processing methods. The model
is fixed with BERTBASE \+ PRiSM evaluating on the DWIE dataset.

###  A.4 Relation Descriptions

We provide a small set of relations and their descriptions in DocRED and DWIE
in Table 7 and 8. For DocRED, a full list can be found either in their paper
Yao et al. (2019) or link333https://www.wikidata.org/wiki/Wikidata:Main_Page.
For DWIE, a full list is not available publicly; however, we were able to
obtain a draft of the annotation documentation from the author. Unannotated
relation descriptions were crafted with the help of a large language model
OpenAI (2023).

Relation Name Description head of government head of the executive power of
this town, city, municipality, state, country, or other governmental body
country sovereign state of this item; don’t use on humans place of birth most
specific known (e.g. city instead of country, or hospital instead of city)
birth location of a person, animal or fictional character country of
citizenship the object is a country that recognizes the subject as its citizen
member of sports team sports teams or clubs that the subject currently
represents or formerly represented

Table 7: DocRED relation descriptions.

Relation Name Description based_in0 Relations between organizations and the
countries they are based in in0 Relations between geographic locations and the
countries they are located in citizen_of Relations between people and the
country they are citizens of based_in0-x Relations between organizations and
the nominal variations of the countries they are based in citizen_of-x
Relations between people and the nominal variations of the countries they are
citizens of

Table 8: DWIE relation descriptions.

##  Appendix B Additional Details for PRiSM

###  B.1 Detailed Experimental Setup

#### Device.

For all our experiments, we trained the networks on a single NVIDIA TITAN RTX
GPU with 24GB of memory.

#### Model Size.

PRiSM shares parameters with the PLM used when learning relation
representations. The only additional parameter weights come from a linear
layer constructing pair representations and an extra embedding space
initialized for relation tokens. The number of trainable parameters for each
model is illustrated in Table 9.

Model # Parameters BERTBASE 108,310,272 BERTBASE-DocRE 114,259,297 BERTBASE-
DocRE + PRiSM 115,514,209 RoBERTaBASE 124,645,632 RoBERTaBASE-DocRE
130,594,657 RoBERTaBASE-DocRE + PRiSM 131,849,569

Table 9: Comparison of model parameters. DocRE includes a bilinear layer and
two linear layers for constructing head and tail representations.

#### GPU Hours.

Adding PRiSM takes a slightly longer computation time than the existing DocRE
models due to having to pass the PLM twice. Note that PRiSM is built for a
low-resource setting in which the computation time does not seem to differ as
much. The comparison of GPU hours is reported in Table 10.

Model Training Hours 3% training examples BERTBASE 0.8 BERTBASE \+ PRiSM 0.8
10% training examples BERTBASE 0.8 BERTBASE \+ PRiSM 1.0 100% training
examples BERTBASE 2.7 BERTBASE \+ PRiSM 3.5

Table 10: GPU hours for DocRED training. Time for evaluating and saving the
model every epoch is included.

#### Hyperparameters.

We perform a grid search on finding the best hyperparameter configuration and
report the tuning range used for our experiments in Table 11. The evaluation
on the validation set is performed for every epoch and the tolerance increases
by 1 when the validation $F_{1}$ is worse than the previous evaluation. The
training stops early when the count reaches the max tolerance.

Model Dataset Hyperparameter Range Best BERTBASE, RoBERTaBASE (Re-)DocRED
batch size { 4, 8 } 4 learning rate { 1e-5, 2e-5, 3e-5, 4e-5, 5e-5, 1e-4 }
3e-5 warmup ratio { 0, 0.06, 0.1 } 0.06 $\lambda$ { 1, 5, 10, 100 } 10 max
grad norm { 1.0 } 1.0 max tolerance { 5 } 5 epoch { 30 } 30 BERTBASE,
RoBERTaBASE DWIE batch size { 4, 8 } 4 learning rate { 1e-5, 2e-5, 3e-5, 4e-5,
5e-5, 1e-4 } 5e-5 warmup ratio { 0, 0.06, 0.1 } 0.06 $\lambda$ { 1, 5, 10, 100
} 10 max grad norm { 1.0 } 1.0 max tolerance { 5 } 5 epoch { 30 } 30

Table 11: Hyperparameter tuning range and best values used in the experiments.
Figure 5: Reliability diagram for BERTBASE when trained with 10% of DocRED
data. Figure 6: Reliability diagram for BERTBASE when trained with 100% of
DocRED data.

###  B.2 Evaluation Details

We elaborate on the details of calculating calibration errors. We utilize two
metrics in our paper. ECE Naeini et al. (2015) divides the probability
interval into a fixed number of bins, calculates the difference between the
accuracy of the predictions and the mean of the probabilities (confidence) in
each bin, and computes a weighted average over the bins. Formally, the
equation can be written as

$\text{ECE}=\sum_{b=1}^{B}\frac{n_{b}}{T}|\text{acc}(b)-\text{conf}(b)|,$ (5)

where $n_{b}$ is the number of predictions in bin $b$, $B$ is a hyperparameter
for the total number of bins, and $T$ is the total number of samples. On the
other hand, ACE Nixon et al. (2019) divides up the probability interval by
having the same number of predictions in each bin, thereby mitigating the
issue of only calibrating the most confident samples. The equation is written
as

$\text{ACE}=\frac{1}{KR}\sum_{k=1}^{K}\sum_{r=1}^{R}|\text{acc}(r,k)-\text{conf}(r,k)|,$
(6)

where $\text{acc}(r,k)$ and $\text{conf}(r,k)$ are the accuracy and confidence
of adaptive calibration range $r$ for class label $k$, respectively.

###  B.3 Additional Calibration Results

We visualize the calibration of the rest of the data setting (i.e., 10% and
100% training data) with reliability diagrams in Figure 5 and 6. We notice
that PRiSM is still effective with 10% of training data, but with full data,
the performance gain is minimal; that is, the line barely moves toward the
perfect calibration line.

We understand that calibration results on models other than the BERTBASE may
be important in demonstrating the effectiveness of PRiSM. As shown in Table
12, we find that RoBERTaBASE and SSAN-BERTBASE follow the same trend as
BERTBASE, showing the lowest calibration error when PRiSM is appended. We also
observe a similar pattern with the Re-DocRED data. We do not report results
for ATLOP because the calibration errors for ATLOP must be computed
differently, as it does not use probabilities (confidence) for prediction.

$N=100$ $N=305$ Method $F_{1}$($\uparrow$) ECE($\downarrow$) ACE($\downarrow$)
$F_{1}$($\uparrow$) ECE($\downarrow$) ACE($\downarrow$) DocRED results BERT
10.82 0.359% 0.379% 42.56 0.137% 0.164% BERT + PRiSM 37.84 0.010% 0.020% 48.10
0.023% 0.020% RoBERTa 22.55 0.671% 0.691% 45.83 0.237% 0.259% RoBERTa + PRiSM
35.10 0.015% 0.025% 48.70 0.022% 0.020% SSAN-BERT 11.93 0.368% 0.390% 42.82
0.128% 0.152% SSAN-BERT + PRiSM 36.96 0.019% 0.019% 48.28 0.023% 0.023% Re-
DocRED results BERT 32.75 0.367% 0.407% 54.44 0.185% 0.190% BERT + PRiSM 49.90
0.038% 0.048% 60.17 0.056% 0.036% RoBERTa 42.40 0.722% 0.718% 57.61 0.191%
0.208% RoBERTa + PRiSM 50.34 0.055% 0.051% 61.55 0.047% 0.033% SSAN-BERT 30.28
0.362% 0.407% 55.60 0.125% 0.148% SSAN-BERT + PRiSM 49.04 0.050% 0.053% 60.57
0.051% 0.036%

Table 12: Comparison of calibration errors (with 10 bins) of different models
under a low-resource setting of DocRED and Re-DocRED.