# Improved Multi-step FCS-MPCC with Disturbance Compensation for PMSM Drives –
Methods and Experimental Validation

Hai Yang
College of Information Engineering
Zhejiang University of Technology
Hangzhou, China.
<EMAIL_ADDRESS>
&Yibin Liu
College of Information Engineering
Zhejiang University of Technology
Hangzhou, China.
<EMAIL_ADDRESS>
&Junxiao Wang
College of Information Engineering
Zhejiang University of Technology
Hangzhou, China.
<EMAIL_ADDRESS>
&Jun Yang
Department of Aeronautical and Automotive Engineering
Loughborough University
Loughborough LE11 3TU, U.K.
<EMAIL_ADDRESS>

###### Abstract

In this paper, an improved multi-step finite control set model predictive
current control (FCS-MPCC) strategy with speed loop disturbance compensation
is proposed for permanent magnet synchronous machine (PMSM) drives system. A
multi-step prediction mechanism is beneficial to significantly improve the
steady-state performance of the motor system. While the conventional multi-
step prediction has the defect of heavy computational burden, an improved
multi-step finite control set model predictive current control (IM MPCC)
strategy is proposed by developing a new multi-step prediction mechanism.
Furthermore, in order to improve the dynamic response of the system, a
disturbance compensation (DC) mechanism based on an extended state observer
(ESO) is proposed to estimate and compensate the total disturbance in the
speed loop of the PMSM system. Both simulation and experimental results
validate the effectiveness of the proposed control strategy.

_K_ eywords Finite Control Set Model Predictive Current Control (FCS-MPCC)
$\cdot$ Multi-step Prediction $\cdot$ PMSM $\cdot$ Extended State Observer
(ESO)

##  1 Introduction

Permanent magnet synchronous motor (PMSM) has the advantages of small size,
high power density, high ratio of moment of inertia and high operating
efficiency, which admit it widely used in electric vehicles, ships, railway
transportation and industrial robot and other fields [1]-[2]. However, PMSM
drive system is essentially a complex nonlinear multivariable coupling system,
which is quite sensitive to system parameter uncertainty and external
disturbance, and thus make it difficult to meet the increasing control
requirements with respect to higher-precision serving [3]. Traditional linear
control strategies, having a slow response time under parameter changes and
external disturbances, would lead to unsatisfactory dynamic performance of the
system [4]-[5]. In order to solve these problems, many new nonlinear control
technologies have been investigated and applied to the PMSM drives system,
such as sliding model control [6], model reference adaptive control [7],
active disturbance rejection control [8], model predictive control (MPC) [9],
etc.

Because of simple implementation, straightforward handling of nonlinearities
and constraints, and good dynamic response, the MPC has attracted more
attention in recent years. MPC can be divided into continuous control set
model predictive control (CCS-MPC) [10]-[11] and finite control set model
predictive control (FCS-MPC) [12]. Compared with CCS-MPC, FCS-MPC dose not
require complex pulse width modulation, which directly manipulates the
inverter with switching signals, delivering much faster dynamic performance of
power converters and motor drives [13].

Since FCS-MPC is based upon a certain number of control sets, there must be a
defect that large ripples lead to unsatisfactory steady-state performance
[14]. In order to reduce the torque and magnetic flux ripple, in [15], two
non-zero vectors are applied in one control cycle to obtain better steady-
state performance. In [16], a passivity-based model predictive control
strategy is proposed, which improves the steady-state performance of the
system while reducing the computational burden. T. Geyer et al. [17] propose a
modified sphere decoding algorithm, which can solve the optimization problem
underlying MPC for a longer prediction horizon. However, when it comes to the
case of multi-level converters, the spherical decoding algorithm still shows
its limitation in computational efficiency. Therefore, an improved spherical
decoding algorithm is proposed in [18], which makes direct MPC with long
prediction horizon an attractive and computationally affordable control
scheme. It should be highlighted that the complexity of this algorithm still
brings a barrier for practical implementation. These methods are improved the
computational efficiency as compared with the baseline multi-step FCS-MPC, but
the control system is still vulnerable to the system parameters uncertainty
and external disturbances [19].

In order to suppress the system parameters uncertainty and external
disturbances for better dynamic response of the system, the disturbance
estimation technology is gradually introduced into the field of high
performance control of PMSM drives [20]-[22]. In [1], the predictive function
control method (PFC) is combined with the extended state observer (ESO) to
obtain the optimal control rate by minimizing the quadratic performance index,
which has certain anti-disturbance effect. But in the case of strong
disturbance, the result is not satisfactory.

Motivated by the above-mentioned challenges, an improved multi-step FCS-MPCC
strategy with speed loop disturbance compensation is proposed to achieve
improved steady-state performance and dynamic response for PMSM system. The
design philosophy of the proposed strategy is explicitly demonstrated in the
following two phases. First, based on a single-step prediction, the optimal
voltage vector and the sub-optimal voltage vector calculated from the cost
function are utilized as the vector waiting to be applied. The vector applied
to the inverter is determined according to the range of the optimal voltage
vector which predicted in the second step. Second, in order to improve the
dynamic response of the system, a disturbance compensation mechanism based on
an ESO is added to estimate and compensate the total disturbance for the speed
loop.

The structure of this paper is as follows: Section 2 introduces the
mathematical model of PMSM and the conventional multi-step model. Section 2
explains the proposed control strategy in this paper. The simulation and
experimental results are shown in Section 4 and Section 5 respectively. The
conclusion of proposed strategy are drawn in Section 6.

##  2  Conventional FCS-MPCC

###  2.1 PMSM Model

The mathematical model of PMSM can be described as follows:

$\displaystyle u_{d}$
$\displaystyle={R_{s}}{i_{d}}+{L_{d}}\frac{{d{i_{d}}}}{{dt}}-{\omega_{re}}{L_{q}}{i_{q}}$
(1) $\displaystyle u_{q}$
$\displaystyle={\omega_{re}}{L_{d}}{i_{d}}{\rm{+}}{R_{s}}{i_{q}}+{L_{q}}\frac{{d{i_{q}}}}{{dt}}+{\omega_{re}}{\psi_{f}}$
(2) $\displaystyle J\frac{{d\omega}}{{dt}}{\rm{}}$
$\displaystyle=\frac{3}{2}{p_{n}}({\psi_{f}}{i_{q}}+({L_{d}}-{L_{q}}){i_{d}}{i_{q}})-B\omega-{T_{L}}$
(3)

where$\ u_{d}$,$\ i_{d}$,$\ L_{d}$ represent d axis voltage, current,
inductance respectively. $\ u_{q}$,$\ i_{q}$,$\ L_{q}$ represent q axis
voltage, current, inductance respectively.$\ R_{s}$,$\ p_{n}$,$\ \psi_{f}$
represent the stator resistance, the number of pole pairs, rotor flux
respectively. $\ \omega_{re}$, $\ \omega$, $\ J$, $\ B$ and$\ T_{L}$ represent
electrical angular velocity, mechanical angular velocity, moment of inertia,
friction coefficient, load torque respectively.

###  2.2 Single-step FCS-MPCC

Compared with the system which adopts FOC strategy, the cost function which is
based on the error is introduced to the current internal loop instead of PI
controller. The numbers of the voltage vectors in the finite control set
depend on the selected inverter, and the optimal voltage vector is obtained by
minimize the cost function, which can be applied to the inverter directly.

According to the model (1),(2) the differential form of the current is
obtained by discretizing the current state equation by the forward Euler
method.

$\displaystyle\begin{array}[]{l}{i_{d}}(k+1)=\left({1-{R_{s}}G}\right){i_{d}}(k)+{L_{q}}G{\omega_{re}}(k){i_{q}}(k)\\\
\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;+G{u_{d}}(k)\end{array}$ (6)
$\displaystyle\begin{array}[]{l}{i_{q}}(k+1)=-{L_{d}}H{\omega_{re}}(k){i_{d}}(k){\rm{+}}\left({1-{R_{s}}H}\right){i_{q}}(k)\\\
\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;+H{u_{q}}(k)-{\psi_{f}}H{\omega_{re}}(k)\end{array}$
(9)

where $\ T_{s}$ is the sample time, $\ G=\frac{{{T_{s}}}}{{{L_{d}}}}$,$\
H=\frac{{{T_{s}}}}{{{L_{q}}}}$.

Because there exists the computation delay in the actual system, the optimal
voltage vector at the present sampling period will be applied to the inverter
at the next sampling period,which will influence the system performance.It is
necessary to make one-step delay compensation for the system. Equation (4),(5)
can be used to compensate one step before the model prediction,then the
predicted current value and cost function value can be obtained by putting
diferent voltage vector from the finite voltage vector control set $\
({u_{fv}})$ into the following formular:

$\displaystyle\begin{array}[]{l}i_{d}^{pre}(k+1)=\left({1-{R_{s}}G}\right){i_{d}}(k+1)+Gu_{fv}^{d}\\\
\begin{array}[]{*{20}{c}}{}\hfil&{}\hfil&{\begin{array}[]{*{20}{c}}{}\hfil&{}\hfil\end{array}}\end{array}+{L_{q}}G{\omega_{re}}(k+1){i_{q}}(k+1)\end{array}$
(14)
$\displaystyle\begin{array}[]{l}i_{q}^{pre}(k+1)=-{L_{d}}H{\omega_{re}}(k+1){i_{d}}(k+1)+Hu_{fv}^{q}\\\
\begin{array}[]{*{20}{c}}{\begin{array}[]{*{20}{c}}{}\hfil&{}\hfil\end{array}}&{}\hfil&{}\hfil\end{array}{\rm{+}}\left({1-{R_{s}}H}\right){i_{q}}(k+1)-{\psi_{f}}H{\omega_{re}}(k+1)\end{array}$
(19)
$\displaystyle\begin{array}[]{l}g=\left|{i_{d}^{*}-i_{d}^{pre}(k+1)}\right|+\left|{i_{q}^{*}-i_{q}^{pre}(k+1)}\right|\\\
\;\;\;\;\;\;\;\;\;\;\;\;+\lim(i_{d,q}^{pre}(k+1))\end{array}$ (22)

where, ${i^{*}}$ is current reference. In formula (8), the first term and
second term evaluate the predicted current error, the last term is the current
constraint condition, which can be expressed as

$\displaystyle\lim(i_{d,q}^{pre}(k+1))=\left\\{\begin{array}[]{l}0,\;if\;i_{d}^{pre}(k+1)\leq{i_{\max}}\\\
\;\;\;\;and\;i_{q}^{pre}(k+1)\leq{i_{\max}}\\\
\infty,if\;i_{d}^{pre}(k+1)>{i_{\max}}\\\
\;\;\;\;or\;i_{q}^{pre}(k+1)>{i_{\max}}\end{array}\right.$ (27)

where $i_{\max}$ is the maximum instantaneous current.

###  2.3 Conventional Multi-step FCS-MPCC

The prediction time domain can be extended to N step on the basis of single-
step FCS-MPCC, the following equations (10),(11),(12) can be obtained:

$\displaystyle\begin{array}[]{*{20}{c}}{i_{d}^{pre}(k+N)=\left({1-{R_{s}}G}\right){i_{d}}(k+N)+Gu_{fv}^{d}}\\\
{\begin{array}[]{*{20}{c}}{\begin{array}[]{*{20}{c}}{}\hfil&{}\hfil\end{array}}&{}\hfil&{}\hfil\end{array}+{L_{q}}G{\omega_{re}}(k+N){i_{q}}(k+N)}\end{array}$
(32)
$\displaystyle\begin{array}[]{l}i_{q}^{pre}(k+N)=-{L_{d}}H{\omega_{re}}(k+N){i_{d}}(k+N)+Hu_{fv}^{q}\\\
\begin{array}[]{*{20}{c}}{\begin{array}[]{*{20}{c}}{}\hfil&{}\hfil&{}\hfil&{}\hfil\end{array}}&{}\hfil\end{array}{\rm{+}}\left({1-{R_{s}}H}\right){i_{q}}(k+N)-{\psi_{f}}H{\omega_{re}}(k+N)\end{array}$
(37) $\displaystyle
g=\sum\limits_{n=1}^{N}{\left[{\begin{array}[]{*{20}{l}}{\left|{i_{d}^{*}-i_{d}^{pre}(k+n)}\right|+\left|{i_{q}^{*}-i_{q}^{pre}(k+n)}\right|}\\\
{+\lim(i_{d,q}^{pre}(k+n)}\end{array}}\right]}$ (40)
$\displaystyle\lim(i_{d,q}^{pre}(k+n))=\left\\{\begin{array}[]{l}0,\;if\;i_{d}^{pre}(k+n)\leq{i_{\max}}\\\
\;\;\;and\;i_{q}^{pre}(k+n)\leq{i_{\max}}\\\
\infty,if\;i_{d}^{pre}(k+n)>{i_{\max}}\\\
\;\;\;\;or\;i_{q}^{pre}(k+n)>{i_{\max}}\end{array}\right.$ (45)

where$\ {n=1,2,\cdots,N}$,${N}$ is the predicted step size, the current limit
condition is expressed as (13).

Figure 1: The finite voltage vector control set.

Multi-step prediction needs to take the number of calculations into account,
for example, when$\ {N=2}$, the controller has to calculate at least 72 times;
with the increase of step size, the amount of calculation will increase
exponentially, and when$\ {N=3}$, the controller needs at least 584
operations. Although the long prediction step size may bring better steady-
state performance to the system, the algorithm is difficult to implement
because of the limited processing capacity of the processor in the actual
system.

##  3 Improved Multi-step FCS-MPCC with speed disturbance compensation

The voltage vector selected by the traditional FCS-MPC method is only optimal
for one control cycle, and it is not optimal for multiple control cycles,
which makes the steady-state performance of the system unsatisfactory. As
shown in Fig. 2, the current value after one step delay compensation at time
$\ (k+1)T$, the current prediction value of the finite control set is obtained
through the prediction model at time $\ (k+1)T_{pre}$, and the second step
prediction is performed at time $\ (k+2)T_{pre}$. It is easy to find that the
control quantity closest to the reference value obtained at time $\
(k+1)T_{pre}$ is not optimal at time $\ (k+2)T_{pre}$. This problem
illustrates that single-step prediction may make the system have
unsatisfactory steady-state performance. The proposed strategy of IM MPCC in
the paper can improve these shortcomings and has less computational burden.

Figure 2: Multi-step FCS-MPCC.

Furthermore, a disturbance compensation(DC) mechanism based on extended state
observer (ESO) has been added to estimate and compensate the total disturbance
of the speed loop to improve the dynamic response of the system. The proposed
strategy is shown in Fig.3.

###  3.1 Improved Two-step FCS-MPCC

On the basis of single-step prediction, the eight voltage vectors are
calculated by the prediction model equation (6, 7) and the cost function
equation (8) to obtain the voltage vectors that minimize and subminimize the
cost function. We call it the optimal output voltage vector $\ u_{out_{min1}}$
and the sub-optimal output voltage vector $\ u_{out_{min2}}$ at time $\
(k+1)T_{pre}$.

$\displaystyle u_{ou{t_{\min 1}}}=\left[{\begin{array}[]{*{20}{c}}{{S_{a\min
1}}}\\\ {{S_{b\min 1}}}\\\ {{S_{c\min 1}}}\end{array}}\right],\left\\{{\min
1({g_{i}}),i=0,1...7}\right\\}$ (49) $\displaystyle u_{ou{t_{\min
2}}}=\left[{\begin{array}[]{*{20}{c}}{{S_{a\min 2}}}\\\ {{S_{b\min 2}}}\\\
{{S_{c\min 2}}}\end{array}}\right],\left\\{{\min
2({g_{i}}),i=0,1...7}\right\\}$ (53)

Calculate the optimal and sub-optimal current prediction values at time $\
(k+2)T_{pre}$ through the prediction model, and express the prediction model
as follow

$\displaystyle\begin{array}[]{l}i_{{d_{\min
1}}}^{pre}(k+2)=\left({1-{R_{s}}G}\right)i_{{d_{\min
1}}}^{pre}(k+1)+Gu_{fv}^{d}\\\
\begin{array}[]{*{20}{c}}{\begin{array}[]{*{20}{c}}{\begin{array}[]{*{20}{c}}{}\hfil&{}\hfil\end{array}}&{}\hfil\end{array}}&{}\hfil\end{array}+{L_{q}}G{\omega_{re}}(k+2)i_{{q_{\min
1}}}^{pre}(k+1)\end{array}$ (59) $\displaystyle\begin{array}[]{l}i_{{q_{\min
2}}}^{pre}(k+2)=-{L_{d}}H{\omega_{re}}(k+2)i_{{d_{\min
2}}}^{pre}(k+1)+Hu_{fv}^{q}\\\
\begin{array}[]{*{20}{c}}{\begin{array}[]{*{20}{c}}{}\hfil&{}\hfil\end{array}}&{}\hfil&{}\hfil\end{array}{\rm{+}}\left({1-{R_{s}}H}\right)i_{{q_{\min
2}}}^{pre}(k+1)-{\psi_{f}}H{\omega_{re}}(k+2)\end{array}$ (64)

Compared with the conventional two-step prediction, the proposed improved two-
step prediction will obtain 16 cost function values, reducing at least 48
model and cost function calculations. The new cost function select act voltage
vector applied to inverter at time $\ (k+1)T_{pre}$ based on the range of
optimal voltage vector predicted at time $\ (k+2)T_{pre}$.

$\displaystyle{g_{sum}}=\left[{\begin{array}[]{*{20}{c}}A\\\
B\end{array}}\right]=\left[{\begin{array}[]{*{20}{c}}{g_{\min 1}^{0}}&{g_{\min
1}^{1}}&\ldots&{g_{\min 1}^{7}}\\\ {g_{\min 2}^{0}}&{g_{\min
2}^{1}}&\ldots&{g_{\min 2}^{7}}\end{array}}\right]$ (69)
$\displaystyle{V_{out}}{\rm{=}}\left\\{{\begin{array}[]{*{20}{c}}{{u_{ou{t_{\min
1}}}},if\begin{array}[]{*{20}{c}}{\min({g_{sum}})\in A}\end{array}}\\\
{{u_{ou{t_{\min 2}}}},if\begin{array}[]{*{20}{c}}{\min({g_{sum}})\in
B}\end{array}}\end{array}}\right.$ (74)

where, A and B respectively represent the set of cost function values at time
$\ (k+2)T_{pre}$ calculated based on the optimal voltage vector and the sub-
optimal voltage vector, ${g_{sum}}$ is the set of all predicted cost function
values at time $\ (k+2)T_{pre}$, and ${V_{out}}$ is the voltage applied to the
inverter in the end. If the minimum cost function value in ${g_{sum}}$ belongs
to A set, the optimal voltage vector at time $\ (k+1)T_{pre}$ is applied; if
the minimum cost function value in ${g_{sum}}$ belongs to B set, the
suboptimal voltage vector at time $\ (k+1)T_{pre}$ is applied.

Figure 3: IM MPCC strategy with speed loop DC.

This strategy can be topologically applied to the long prediction step size
model. Compared with the conventional FCS-MPCC cost optimization method, this
strategy has obvious advantages in less computation but similar steady-state
performance. When the prediction steps ${N=3}$, the number of calculations of
the model and cost function is only 56 times. The final selection function in
multi-step prediction can be described as

$\displaystyle
g_{sum}={\left[{\underbrace{\begin{array}[]{*{20}{c}}A&B&C&\ldots\end{array}}_{{2^{N-1}}}}\right]^{\rm{T}}}$
(76) $\displaystyle V_{out}=\left\\{{\begin{array}[]{*{20}{c}}{{u_{ou{t_{\min
1}}}},if\begin{array}[]{*{20}{c}}{\min({g_{sum}})\in{{\left[{\overbrace{\begin{array}[]{*{20}{c}}A&B&\ldots\end{array}}^{[1,\frac{{{2^{N-1}}}}{2}]}}\right]}^{T}}}\end{array}}\\\
{{u_{ou{t_{\min
2}}}},if\begin{array}[]{*{20}{c}}{\min({g_{sum}})\in{{\left[{\overbrace{\begin{array}[]{*{20}{c}}E&F&\ldots\end{array}}^{(\frac{{{2^{N-1}}}}{2},{2^{N-1}}]}}\right]}^{T}}}\end{array}}\end{array}}\right.$
(83)

If the minimum cost function value in ${g_{sum}}$ belongs to set before the $\
\frac{{{2^{N-1}}}}{2}$th, the optimal voltage vector at time $\ (k+1)T_{pre}$
is applied; if the minimum cost function value in ${g_{sum}}$ belongs to other
set in $\ \frac{{{2^{N-1}}}}{2}$th to $\ 2^{N-1}$th, the suboptimal voltage
vector at time $\ (k+1)T_{pre}$ is applied. Since the final voltage vector
acting on the inverter is always the voltage vector at time $\ (k+1)T_{pre}$,
the effect is not as obvious when the prediction step is too long. so, the
final step size is two.

###  3.2 Speed Loop Disturbance Compensation

Considering the uncertainty of system parameter uncertainty and external
disturbance, the state variable
${d_{\omega}}=-\frac{{B{\omega}}}{J}-\frac{{{T_{L}}}}{J}+\frac{3}{2}p{\psi_{f}}(\frac{{{i_{q}}}}{J}-\frac{{{i_{q}}^{*}}}{{{J_{n}}}})$
is introduced, ${J_{n}}$ is the moment of inertia of the rotor.If the sampling
time is small enough, ${d_{\omega}}$ can be regarded as approximately
unchanged, so let $\ {{\dot{d}}_{\omega}}=0$.

From this, the new equation of state can be deduced as

$\displaystyle\left\\{{\begin{array}[]{*{20}{l}}{{{\dot{\omega}}}=\frac{{{i_{q}}^{*}}}{k}+{d_{\omega}}}\\\
{{{\dot{d}}_{\omega}}=0}\end{array}}\right.$ (86)

According to the formula (22), the extended state observer (ESO) is designed
as

$\displaystyle\left\\{{\begin{array}[]{*{20}{l}}{{{\dot{z}}_{1}}=\frac{{{i_{q}}^{*}}}{k}+{z_{2}}-{\beta_{1}}({z_{1}}-{\omega})}\\\
{{{\dot{z}}_{2}}=-{\beta_{2}}({z_{1}}-{\omega})}\end{array}}\right.$ (89)

As long as the parameters $\ \beta_{1}$ and $\ \beta_{2}$ of the observer are
adjusted appropriately, the system speed estimation value $\ \ z_{1}$ and the
total disturbance estimation value $\ z_{2}$ can be obtained through the state
observer. It can be seen from equation (23) that if the system $\ d_{\omega}$
is observed and compensated, the PMSM speed control system can be approximated
as a first-order integral system.

The linear state error feedback control law is adopted for the input $\
u_{0}$, so the control law based on disturbance compensation controller can be
obtained as

$\displaystyle\left\\{\begin{array}[]{l}{u_{0}}={k_{p}}(\omega^{*}-{z_{1}})\\\
{i_{q}}^{*}={u_{0}}-k{z_{2}}\end{array}\right.$ (92)

where $k=\frac{{2J}}{{3p{\psi_{f}}}}$,$\ k_{p}$ is the magnification factor,$\
\omega^{*}$ is the reference speed.

##  4 Simulation and analysis

In order to test the effectiveness of the control strategy proposed in Fig.3,
some simulation results are given on the Matlab/simulink. The following three
groups of simulation results are carried out. The first is to show the steady-
state performance of IM MPCC for PMSM by comparing with conventional MPCC. The
second is to verify that the proposed strategy can improve the dynamic
response of the system by comparing with proposed IM MPCC. The last is to
reflect the high-order harmonic reduction effect of the current control
strategy by total harmonic distortion (THD). The specific simulation
parameters of the surface-mounted permanent magnet synchronous motor are shown
in Table 1.

Table 1: Parameter Values of PMSM Descriptions | Parameters | Nominal Values
---|---|---
DC link Voltage | $V_{dc}$ | 311 (V)
Stator Resistance | $R_{s}$ | 1.3 ($\Omega$)
Stator Inductance | $L_{s}$ | 0.0085 (H)
Flux | $\psi_{f}$ | 0.175 (Wb)
Pole Pairs | $P$ | 4.0
Inertia | $J$ | 0.008 (Kg$\cdot$ m2)

###  4.1 Steady-state Performance

In order to test the control strategy has improved the steady-state
performance of the system, the single-step model predictive current control
(MPCC) strategy is compared with the proposed improved multi-step model
predictive current control (IM MPCC) strategy in this part. The simulation
results are shown in Fig 4. Except for the controller differences mentioned
above, these two simulations have exactly the same structure and parameters.
It can be seen that ripple of three-phase current (take A phase current $\
i_{a}$ as an example), torque ($\ T_{e}$) and speed ($\ \omega$) are all
reduced.

(a)

(b)

(c)

(d)

Figure 4:  (Simulation). Steady-state performance comparison between
conventional MPCC and IM MPCC: (a) Speed; (b) Torque; (c) Phase-current under
zero load; and (d) Phase current under 5N$\cdot$m load.

###  4.2 Dynamic Response

In order to fairly test the dynamic response of the proposed control strategy,
the improved multi-step model predictive control strategy with speed loop
disturbance compensation (IM MPCC+DC) is compared with the control strategy
without disturbance compensation. Except for this differences mentioned above,
these two simulations have exactly the same structure and parameters. The
dynamic response of the system results from the sudden change of load
disturbance. The load disturbance is suddenly increased , and then suddenly
reduced to 0 (load torque from 0 to 5 N$\cdot$m, and then jumps back to 0),
the effect is shown in Fig 5. It can be seen that dynamic response of IM MPCC
with speed loop DC is better than the strategy without DC.

(a)

(b)

Figure 5: (Simulation). Response curves of the IM MPCC strategy with and
without disturbance compensation: (a) Speed; and (b) Torque.

###  4.3 THD

By comparing the high-frequency component with the basic component of the
three-phase current, we can see that the three-phase current THD data of the
three strategies are shown in Table 2. It can be observed that compared with
the initial single-step prediction measurement, the THD of the three-phase
current of the proposed control strategy is reduced by 27.18$\ \rm{\%}$ on
average.

Table 2: Three-phase Current THD Comparison (Simulation) Control | Three-phase Current THD
---|---
Strategy | $\ i_{a}$ | $\ i_{b}$ | $\ i_{c}$
MPCC | 1.034$\ \rm{\%}$ | 1.046$\ \rm{\%}$ | 1.040 $\ \rm{\%}$
IM MPCC | 0.773$\ \rm{\%}$ | 0.796$\ \rm{\%}$ | 0.792$\ \rm{\%}$
IM MPCC+DC | 0.737$\ \rm{\%}$ | 0.776$\ \rm{\%}$ | 0.759$\ \rm{\%}$

##  5 Experiment and analysis

To further verify the effectiveness of the proposed control strategy, an
experiment platform is set up, which is shown by Fig. 6. The experimental
platform consists of a dSPACE controller board, a three-phase inverter, a DC
link power supply, current sensors, a PMSM and a load motor. The dSPACE 1103
controller board used in this experimental test setup integrates a core
processor based on TMS320F240 DSP. The dSPACE mainly implements signal
acquisition, signal processing and controller calculation for output, and the
minimum reliable current measurement time is set to 30 $\mu s$. The three-
phase inverter is built by a 3 kW FSBB30CH60C Smart Power Module (SPM) with a
rated current of 10 A. To ensure the accuracy of the feedback current, LEM
LAH25-NP is employed as the current sensor. The maximum voltage of the DC link
power supply is 300 V.

Figure 6: Experimental platform.

###  5.1 Steady-state Performance

In order to test the performance of the two control strategies fairly, it is
necessary to ensure that the parameters in the prediction model are
consistent. The parameters of the two control policies are selected as
follows. The traditional PI control is aodpted in the speed loop, where the
parameters are designed as $K_{p}=0.003$ and $K_{i}=0.001$. The reference
speed is set as 1000 rpm. The experimental results of the MPCC and IMMPCC
strategies in steady state are shown in Fig. 7. The maximum speed and torque
offsets of the IM MPCC strategy were 2.31 rpm and 0.19 N$\cdot$m,
respectively, which improved the control performance as compared with these of
4.35 rpm and 0.28 N$\cdot$m of the MPCC strategy.

(a)

(b)

Figure 7: (Experiment). Response curves for steady-state performance
evaluation: (a) the traditional MPCC; and (b) the proposed IM MPCC.

###  5.2 Dynamic Response

Fig. 8 shows the experimental results of IM MPCC strategy when the system is
subjected to a load torque disturbance. The load torque was suddenly increased
to 2 N$\cdot$m, and the reference speed was set at 1000 rpm. Two sets of
experimental results that with or without disturbance compensation (DC)
mechanism based on ESO are carried out for comparisons. The parameters of ESO
are designed as $\beta_{1}=1200$ and $\beta_{2}=4000$. In order to
quantitatively analyze the experimental results, the index parameter
reflecting the dynamic performance was given in Table 3. The experimental
results show that the proposed IM MPCC strategy with disturbance compensation
mechanism has much faster dynamic response and disturbance rejection ability.

(a)

(b)

Figure 8: (Experiment). Response curves for dynamic performance evaluation under load disturbance torques: (a) IM MPCC without DC; and (b) IM MPCC with DC. Table 3: Comparison of Performance indicators (Experiment) Control Strategy | Performance indicators
---|---
| $\ e_{max}$ | $\ t_{c}$
| speed | torque | speed | torque
IM MPCC | -8.32$rpm$ | - | 0.23$s$ | 0.045$s$
IM MPCC+DC | -4.19$rpm$ | - | 0.15$s$ | 0.022$s$

###  5.3 THD

The THD values of three-phase current with the fundamental wave frequency of
50 Hz are compared in Table 4. As compared with MPCC, the THD of IM MPCC
strategy decreased by 22.03% in average, which indicates that IM MPCC strategy
has much better steady-state performance.

Table 4: Three-phase Current THD Comparison Data (Experiment) Control | Three-phase Current THD
---|---
Strategy | $\ i_{a}$ | $\ i_{b}$ | $\ i_{c}$
MPCC | 10.552$\ \rm{\%}$ | 10.519$\ \rm{\%}$ | 10.538 $\ \rm{\%}$
IM MPCC | 8.012$\ \rm{\%}$ | 8.133$\ \rm{\%}$ | 8.087$\ \rm{\%}$

##  6 CONCLUSION

In this paper, an improved multi-step FCS-MPCC strategy with speed loop
disturbance compensation for PMSM drives has been investigated. In order to
improve the steady-state performance and reduce computation burden, an IM MPCC
strategy is proposed. To further improve the dynamic response of the closed-
loop system, a disturbance compensation (DC) mechanism based on extended state
observer (ESO) has been added to estimate and compensate the total disturbance
of the speed loop for PMSM system before the IM MPCC controller. The
simulation and experimental results show that the steady-state performance and
dynamic response are both improved in the proposed strategy. Therefore, the
proposed control strategy in this paper has a good application prospect for
high-precision PMSM drives and easier to implement.

## References

  * [1] H. Liu and S. Li, “Speed control for PMSM servo system using predictive functional control and extended state observer," IEEE Transactions on Industrial Electronics, vol. 59, no. 2, pp. 1171-1183, Feb. 2012.
  * [2] J. Holtz, “Pulsewidth modulation for electronic power conversion," Proceedings of the IEEE, vol. 82, no. 8, pp. 1194-1214, Aug. 1994.
  * [3] S. Li, M. Zhou and X. Yu, “Design and implementation of terminal sliding mode control method for PMSM speed regulation system," IEEE Transactions on Industrial Informatics, vol. 9, no. 4, pp. 1879-1891, Nov. 2013.
  * [4] J. Wang, F. Wang, G. Wang, S. Li and L. Yu, “Generalized proportional integral observer based robust finite control set predictive current control for induction motor systems with time-varying disturbances," in IEEE Transactions on Industrial Informatics, vol. 14, no. 9, pp. 4159-4168, Sept. 2018.
  * [5] J. Wang, F. Wang, Z. Zhang, S. Li and J. Rodriguez, “Design and implementation of disturbance compensation-based enhanced robust finite control set predictive torque control for induction motor systems," IEEE Transactions on Industrial Informatics, vol. 13, no. 5, pp. 2645-2656, Oct. 2017.
  * [6] H. Kim, J. Son and J. Lee, “A high-speed sliding-mode observer for the sensorless speed control of a PMSM," IEEE Transactions on Industrial Electronics, vol. 58, no. 9, pp. 4069-4077, Sept. 2011.
  * [7] S. Li and H. Gu. “Fuzzy adaptive internal model control schemes for PMSM speed-regulation system." IEEE Transactions on Industrial Informatics, vol. 8, no. 4, pp. 767-779, 2012.
  * [8] H. Sira-Ramirez, J. Linares-Flores, C. Garcia-Rodriguez and M. A. Contreras-Ordaz, “On the control of the permanent magnet synchronous motor: An active disturbance rejection control approach," IEEE Transactions on Control Systems Technology, vol. 22, no. 5, pp. 2056-2063, Sept. 2014.
  * [9] J. Rodriguez,et al., “Predictive control of three-phase inverter," Electronics Letters, vol. 40, no. 9, pp. 561-563, 29 April 2004.
  * [10] F. Wang, L. He and J. Rodriguez, “FPGA-based continuous control set model predictive current control for PMSM system using multistep error tracking technique," IEEE Transactions on Power Electronics, vol. 35, no. 12, pp. 13455-13464, Dec. 2020.
  * [11] F. Wang, J. Wang, R. M. Kennel and J. Rodriguez, “Fast speed control of AC machines without the proportional-integral controller: Using an extended high-gain state observer," IEEE Transactions on Power Electronics, vol. 34, no. 9, pp. 9006-9015, Sept. 2019.
  * [12] S. Kouro, P. Cortes, R. Vargas, U. Ammann and J. Rodriguez, “Model predictive control: A simple and powerful method to control power converters," IEEE Transactions on Industrial Electronics, vol. 56, no. 6, pp. 1826-1838, June 2009.
  * [13] A. A. Ahmed, B. K. Koh and Y. I. Lee, “A comparison of finite control set and continuous control set model predictive control schemes for speed control of induction motors," IEEE Transactions on Industrial Informatics, vol. 14, no. 4, pp. 1334-1346, April 2018.
  * [14] W. Xie, “Finite-control-set model predictive torque control with a deadbeat solution for PMSM drives," IEEE Transactions on Industrial Electronics, vol. 62, no. 9, pp. 5402-5410, Sept. 2015.
  * [15] Y. Zhang and H. Yang, “Two-vector-based model predictive torque control without weighting factors for induction motor drives," IEEE Transactions on Power Electronics, vol. 31, no. 2, pp. 1381-1390, Feb. 2016.
  * [16] F. Wang, G. Lin and Y. He, “Passivity-based model predictive control of three-level inverter-fed induction motor," IEEE Transactions on Power Electronics, vol. 36, no. 2, pp. 1984-1993, Feb. 2021.
  * [17] T. Geyer and D. E. Quevedo, “Multistep finite control set model predictive control for power electronics," IEEE Transactions on Power Electronics, vol. 29, no. 12, pp. 6836-6846, Dec. 2014.
  * [18] T. Geyer and D. E. Quevedo, “Performance of Multistep finite control set model predictive control for power electronics," IEEE Transactions on Power Electronics, vol. 30, no. 3, pp. 1633-1644, March 2015.
  * [19] R. Baidya, “Enabling Multistep model predictive control for transient operation of power converters," IEEE Open Journal of the Industrial Electronics Society, vol. 1, pp. 284-297, 2020.
  * [20] S. Li, M. Zhou and X. Yu, “Design and implementation of terminal sliding mode control method for PMSM speed regulation system," IEEE Transactions on Industrial Informatics, vol. 9, no. 4, pp. 1879-1891, Nov. 2013.
  * [21] L. He, F. Wang, J. Wang and J. Rodriguez, “Zynq implemented Luenberger disturbance observer based predictive control scheme for PMSM drives," IEEE Transactions on Power Electronics, vol. 35, no. 2, pp. 1770-1778, Feb. 2020.
  * [22] S. Li, J. Yang, W.H. Chen and X. Chen, Disturbance Observer-Based Control: Methods and Applications, CRC press, 2014.


# CUED at ProbSum 2023: Hierarchical Ensemble of Summarization Models

Potsawee Manakul, Yassir Fathullah, Adian Liusie,
Vyas Raina, Vatsal Raina, Mark Gales
ALTA Institute, Engineering Department, University of Cambridge
<EMAIL_ADDRESS>

###### Abstract

In this paper, we consider the challenge of summarizing patients’ medical
progress notes in a limited data setting. For the Problem List Summarization
(shared task 1A) at the BioNLP Workshop 2023, we demonstrate that Clinical-T5
fine-tuned to 765 medical clinic notes outperforms other extractive,
abstractive and zero-shot baselines, yielding reasonable baseline systems for
medical note summarization. Further, we introduce Hierarchical Ensemble of
Summarization Models (HESM), consisting of token-level ensembles of diverse
fine-tuned Clinical-T5 models, followed by Minimum Bayes Risk (MBR) decoding.
Our HESM approach lead to a considerable summarization performance boost, and
when evaluated on held-out challenge data achieved a ROUGE-L of 32.77, which
was the best-performing system at the top of the shared task
leaderboard.111Our code is available at
https://github.com/potsawee/hierarchical_ensemble_summ.

##  1 Introduction

Summarization is a common natural language generation (NLG) task with growing
recent interest El-Kassas et al. (2021). The 1A shared task of BioNLP 2023
considers medical problem list summarization Gao et al. (2023), where patient
notes are summarized to assist medical diagnosis applications. There are
several challenges faced in designing systems for this task: First, the
challenge is low-resource, with only 765 examples available for
training/validation. Second, a high-stake application with specialized medical
terms requires systems that can deal with domain-specific terms and find
relevant diagnoses from patient documents.

Figure 1: Hierarchical ensemble of summarization models where each individual
model is a fine-tuned Clinical-T5.

This paper introduces Hierarchical Ensemble of Summarization Models (HESM), an
approach that is composed of two sequential ensembling layers, of token-level
ensembles, followed by Minimum Bayes Risk (MBR) decoding, as shown in Figure
1. Ensembling methods combine the predictions of various models Fort et al.
(2020) and have been effective in NLG tasks, such as in summarization Manakul
and Gales (2020). For low-resource settings, this method allows outputs to be
composed from multiple different ensemble members, which can reduce the
influence of noise and eliminate spurious signals, reducing the chance of
medically inaccurate summaries. We demonstrate that using HESM with
Clinical-T5 models Lehman et al. (2023) leads to systems that have a good
grasp of medical knowledge, and that are able to generate outputs that are
consistently closer to ground-truth summaries. Our proposed HESM method was
submitted to the BioNLP problem list summarization challenge, and achieved the
top position of the shared task leaderboard, out of 9 teams.

##  2 Background and Related Work

###  2.1 Existing Pre-Trained Language Models

A common approach for current NLP applications has been the pre-train and
fine-tuning paradigm, where pre-trained models are fine-tuned to specific
target tasks. The community has open-sourced a variety of pre-trained
backbones of different sizes and architectures, including encoder-only such as
BERT Devlin et al. (2019), decoder-only such as GPT-2 Radford et al. (2019),
and encoder-decoder such as BART Lewis et al. (2020) and T5 Raffel et al.
(2020). Medical domain versions of language models have also been created,
including ClinicalBERT Huang et al. (2019), BioBERT Lee et al. (2020), BioMed-
RoBERTa Gururangan et al. (2020), BioGPT Luo et al. (2022), and Clinical-T5
Lehman et al. (2023).

###  2.2 Summarization Methods

The two main summarization approaches are extractive methods, which select
relevant words/phrases present in the input for the summary, and abstractive
methods, which can freely generate text (even text that may not be present in
the source). Previous work in medical list summarization demonstrated
significant gain from adapting BART and T5 (both are abstractive models) to
the medical domain and by fine-tuning them for summarization Gao et al.
(2022). For long-input summarization, a preliminary stage of selecting the
most relevant input sentences before summarization was shown to be effective
Manakul and Gales (2021). Alternatively, large language models have recently
shown success in zero-shot summarization Brown et al. (2020).

##  3 Hierarchical Ensemble of Summarization Models

When working with a small dataset, individual models are prone to overfit
specific aspects of the data due to the limited number of training samples. By
training multiple models on the same dataset, each model can potentially
capture different aspects of the data that are generalizable and not prone to
overfitting. Combining these diverse models together can then create a more
robust and accurate prediction model Sim et al. (2007).

Various approaches can be used to create diverse individual systems. A simple
approach is to use different weights’ initialization Lakshminarayanan et al.
(2017) for different seeds. Alternatively for pre-trained systems (as
considered in this work), one can set different random seeds, which will
influence training dropout and stochastic gradient descent batch creation,
resulting in variability in the final models’ weights. One can also use a form
of data bagging, where a different subset of the data is used to train each
model Galar et al. (2011). For example for the clinical notes, one model can
be trained using only the assessment section of the notes, whilst another can
be trained using the assessment+subjective sections.

Given an ensemble of diverse models, one may then combine them for a more
robust ensemble system. A possible model combination method is weight
averaging. Although weight averaging across training runs has shown success in
image classification Wortsman et al. (2022), weight averaging across different
training runs is expected to work only when individual runs operate in similar
weight spaces. This limits the types of combinations for weight averaging to
only models with the same architecture and the same input format. As a result,
we focus instead on two other methods of combination: token-level ensembling
and Minimum Bayes Risk decoding Rosti et al. (2007a, b).

#### Token-level Ensemble

Token-level ensemble (also known as product-of-expectations) is a technique to
improve the performance of sequence-to-sequence models by combining
predictions from multiple models at the token-level Sennrich et al. (2015);
Freitag et al. (2017); Malinin and Gales (2021); Fathullah et al. (2021). Let
us consider $M$ different models, where we want to generate an output
sequence, $\mathbf{y}=y_{0},y_{1},\ldots,$ from an input sequence
$\mathbf{x}$. In the standard decoding setup, we can generate each token
sequentially:

$p(\mathbf{y}|\mathbf{x})=\prod_{i}p(y_{i}|\mathbf{x},y_{<i})$ (1)

In token ensembling, each token’s probability is the average probability of
the individual models:

$p(y_{i}|\mathbf{x},y_{<i})=\frac{1}{M}\sum_{m}p_{m}(y_{i}|\mathbf{x},y_{<i})$
(2)

#### Minimum Bayes Risk (MBR) Decoding

Given all possible output sequences, $\mathcal{Y}$, standard decoding
(inference) strategies such as beam search are used to select the sequence
with the greatest likelihood:

$\mathbf{y}^{*}=\operatorname*{arg\,max}_{{\mathbf{y}}\in\mathcal{Y}}\left\\{p({\mathbf{y}}|\mathbf{x})\right\\}$
(3)

However, the above method is not well aligned with the final reward metric,
$\mathcal{R}$, used to assess the quality of samples (e.g. ROUGE-L). Following
MBR decoding Kumar and Byrne (2004); Sim et al. (2007), we can seek to select
the most average sample, $\mathbf{y}^{*}\in\mathcal{Y}$, as per our desired
reward metric:

$\mathbf{y}^{*}=\operatorname*{arg\,max}_{{\mathbf{y}}\in\mathcal{Y}}\left\\{\mathbb{E}_{p(\tilde{\mathbf{y}}|x)}[\mathcal{R}(\mathbf{y},\tilde{\mathbf{y}})]\right\\}$
(4)

where $\mathbf{y}^{*}$ is expected to be the most representative of all
generated samples. In practice, with only access to $N$ sequences,
$\mathcal{Y}=\\{\mathbf{y}_{1},\ldots,\mathbf{y}_{N}\\}$, sourced from
different model structures, there may not be available sensible or meaningful
posterior distributions, $p(\mathbf{y}|\mathbf{x})$ and hence we approximate
the expectation as a simple average, where each observed output $\mathbf{y}$
is taken to be equiprobable:

$\mathbf{y}^{*}\approx\operatorname*{arg\,max}_{\mathbf{y}\in\mathcal{Y}}\left\\{\sum_{n=1}^{N}\mathcal{R}(\mathbf{y},\tilde{\mathbf{y}}_{n})\right\\}$
(5)

The selection of $\mathbf{y}^{*}$ can also be viewed as a method to
automatically reject the anomalous samples and thus improve overall
performance. Previous work showed that MBR decoding improves machine
translation Rosti et al. (2007a, b), and the highest evaluation score is
obtained when $\mathcal{R}$ matches the evaluation metric Freitag et al.
(2022). Thus, we use ROUGE-L as the reward metric $\mathcal{R}$.

We note that MBR decoding is applied at inference time, and it can be applied
to any set of models regardless of their architectures or training techniques,
but it is expected to be effective when there is diversity in the models’
outputs.

#### Hierarchical Ensembling

Finally, hierarchical ensembling is a method that aims to combine the above
two approaches. Multiple output sequences, $\mathcal{Y}$, can be generated by
performing token ensembling over different sets of individual models.
Subsequently, MBR decoding can be used to select the most representative
sample from these different output sequences to give a single output sequence,
$\mathbf{y}^{*}$. This hierarchical structure is depicted in Figure 1.

##  4 Experiments

###  4.1 Experimental Setup

Data. Training data consists of 765 progress notes along with output medical
summaries, which were sourced from MIMIC-III. Due to the small amount of data
available, systems were initially evaluated using 5-fold cross-validation. The
test (held-out competition) data consists of 237 progress notes, where for the
competition evaluation we submitted generated summaries onto an online
platform where the ROUGE-L was calculated. ROUGE-1, ROUGE-2 and ROUGE-L Lin
(2004) were all computed during cross-fold validation.

The medical reports have three fields available: ‘assessment’ {A}, ‘objective’
{O}, and ‘subjective’ {S}, with word statistics shown in Table 1. We also
consider different permutations by concatenating fields, separated by special
tokens.

Field | {O} | {S} | {A} | Summary
---|---|---|---|---
#words | 304.7$\pm$83.4 | 85.5$\pm$54.8 | 33.7$\pm$17.1 | 10.5$\pm$7.5
Table 1: Medical report and summary statistics.

Models. For abstractive summarization, we consider T5 and Clinical-T5 as the
backbone. Clinical-T5222https://huggingface.co/xyla/Clinical-T5-Large. was
initialized from scratch and pre-trained on the union of MIMIC-III and MIMIC-
IV databases Lehman et al. (2023). The models are downloaded through
HuggingFace; we finetune models with teacher forcing on our training data and
use beam search during inference. More details about training and inference
are provided in Appendix A.

###  4.2 Baseline Selection

We start our investigation by comparing zero-shot, extractive, and abstractive
summarization methods. Note that we provide the results of zero-shot
summarization based on open-sourced large language models in Appendix B.1.

#### Extractive Summarization

To obtain an empirical upper bound, we compute ROUGE-1 between each input
sentence against its ground-truth summary. The input sentences are ordered by
the sections {A}, {S}, {O}.

We consider two oracle options: (1) All-overlap, which concatenates all input
sentences where ROUGE-1 recall is positive to the generated summary; and (2)
Greedy-best, which uses a greedy algorithm to obtain extractive sentences
similar to Nallapati et al. (2017). This greedy-best method iteratively adds
sentences one at a time to the generated summary, where the added sentence is
the one which yields the highest ROUGE-1 (F1) score. This process is repeated
until the ROUGE-1 (F1) of the generated summary does not improve. Our results
in Table 2 show that even the oracle (greedy-best) approach achieves lower
scores than fine-tuned T5 models (in Table 3).

Method | R1 | R2 | RL
---|---|---|---
Oracle (All-overlap) | 20.37 | 6.41 | 15.44
Oracle (Greedy-best) | 29.14 | 11.09 | 22.74
Table 2: Empirical upper bounds of extractive summarization methods on the
training data.

#### Abstractive Summarization

Our first experiment is to determine the best transformer backbone for
abstractive summarization. Table 3 shows T5 performance when fine-tuned using
the assessment field only. We find that T5-Large significantly outperforms
T5-Small, but that performance does not further scale with size with T5-Large
and T5-XL performing similarly.

Pre-Trained Model | R1 | R2 | RL
---|---|---|---
T5-Base | 26.77 | 10.33 | 24.82
T5-Large | 29.56 | 12.01 | 27.88
T5-XL | 29.46 | 12.72 | 27.58
Clinical-T5-Base | 26.62 | 12.11 | 24.96
Clinical-T5-Large | 32.22 | 14.30 | 30.15
Table 3: ROUGE-scores of various T5/Clinical-T5 models on cross-validation,
with inputs being {A} only.

We further find that domain adaptation can lead to an additional boost, with
Clinical-T5-Large showing ROUGE scores 2% higher than T5-Large. Domain
adaptation, however, was not helpful for our low-capacity model, with
Clinical-T5-Base performing similarly to T5-Base. We, therefore, use
Clinical-T5-Large, the best-performing system during cross-validation, as our
backbone transformer for all further systems.

The next experiments consider which input fields are most useful for
generating the summary. Table 4 shows that the assessment field, {A}, contains
the key information for the patient’s problem summary with ROUGE scores below
20 when any other field is used alone. We further observe better performance
when {A} is augmented with {S}.

Inputs | R1 | R2 | RL
---|---|---|---
$\emptyset$ |  9.61 |  2.93 |  9.24
{O} | 14.09 | 4.99 | 13.27
{S} | 17.96 | 7.00 | 16.81
{A} | 32.22 | 14.30 | 30.02
{A}+{S} | 33.46 | 15.07 | 31.03
Table 4: Comparison of Clinical-T5-Large performance when using different
inputs on training data (using cross-validation). The empty input baseline
$\emptyset$ is trained to generate summaries without the input report.

###  4.3 Ensemble Methods

To maximize the performance, we apply ensemble methods to fine-tuned
Clinical-T5-Large models. For the simplicity of notation, we use $\theta_{\tt
A}$ and $\theta_{\tt AS}$ to denote the system where the input is {A} and
{A}+{S}, respectively. We train nine $\theta_{\tt A}$ individual models where
all models are initialized from Clinical-T5-Large weights, and have the same
training hyperparameters except random seeds for data batching. In Table 5, we
compare different methods for combining nine $\theta_{\tt A}$ models and the
results show that: 1) weight averaging results in a slightly worse system; 2)
both token-level ensemble and MBR decoding yield better performance than
single models. In addition, we observe a similar trend when combining
$\theta_{{\tt AS}}$ models as shown in Table 6.

Method | ROUGE-L
---|---
F1 | Prec | Rec
Individual | 29.84$\pm$0.69 | 40.11$\pm$1.40 | 27.68$\pm$0.76
Weight Avg. | 29.39 | 39.68 | 27.50
Tok. Ensemble | 30.50 | 41.09 | 28.37
MBR Decoding | 30.72 | 40.96 | 28.91
Table 5: ROUGE-L on the test data. This table compares combination methods of nine $\theta_{{\tt A}}$ models. Method | ROUGE-L
---|---
F1 | Prec | Rec
Individual | 29.44$\pm$0.45 | 37.57$\pm$0.69 | 28.33$\pm$0.65
Weight Avg. | 28.00 | 35.65 | 27.16
Tok. Ensemble | 30.04 | 36.35 | 29.96
MBR Decoding | 30.30 | 38.39 | 28.92
Table 6: ROUGE-L on the test data. This table compares combination methods of
nine $\theta_{{\tt AS}}$ models.

Hierarchical Ensemble. We explore combining $\theta_{\tt A}$ and $\theta_{\tt
AS}$ models in a token-level ensemble followed by MBR decoding to form a
hierarchical ensemble. Based on nine $\theta_{\tt A}$ models and nine
$\theta_{\tt AS}$ models, Table 7 provides the results of hierarchical
combination in different setups.

The first block shows the performance when combining one $\theta_{\tt A}$ and
one $\theta_{\tt AS}$ in a token-level ensemble, followed by an MBR
combination stage over 9 of these ensembles. Similarly, the second block shows
the performance when combining three $\theta_{\tt A}$ and three $\theta_{\tt
AS}$ each in a token-level ensemble fashion followed by an MBR decoding stage
over 3 of these ensembles.

Name | Ensemble | ROUGE-L
---|---|---
Token | MBR | F1 | Prec | Rec
$\theta_{\tt A}$+$\theta_{\tt AS}$ | (1, 1) | ✗ | 31.17$\pm$0.67 | 39.51$\pm$1.30 | 29.66$\pm$1.02
HESM | (1, 1) | 9 | 32.31 | 41.16 | 30.16
$\theta_{\tt A}$+$\theta_{\tt AS}$ | (3, 3) | ✗ | 31.50$\pm$0.42 | 39.74$\pm$0.79 | 29.97$\pm$0.57
HESM | (3, 3) | 3 | 31.87 | 39.63 | 30.24
(3, 3) | 9 | 31.88 | 40.07 | 30.17
Table 7: ROUGE-L of HESM on the test data. ($a$, $b$) denotes token-level
ensemble consisting of $a\theta_{\tt A}$ models and $b\theta_{\tt AS}$ models.
MBR=$c$ denotes the outputs of $c$ token-level ensembles combined using MBR
decoding. For HESM(3,3) w/ MBR=3, ensembles with non-overlap members are
chosen.

###  4.4 Evaluation System

This section discusses the specific nature of our HESM systems submitted to
the shared task. Given the flexibility in an MBR combination, members of HESMs
are not limited to token-level ensembles. Hence, during the competition we
made use of previously submitted systems to build the final HESM. As a result,
our HESM consists of six systems: best-performing $\theta_{{\tt A}}$; weight
averaging of 3$\theta_{{\tt A}}$; token-level ensemble of 3$\theta_{{\tt A}}$
with $\mathcal{L}_{\text{RL}}$; 2$\times$token-level ensemble of
9$\theta_{{\tt A}}$; and token-level ensemble of 9$\theta_{{\tt AS}}$. The
results of the HESM’s members are provided in Table 11 in the appendix.

We further consider combining this HESM with the token-level ensembles of
3$\theta_{{\tt A}}$+3$\theta_{{\tt AS}}$ investigated in Section 4.3. The
first ensemble (v1) is obtained by selecting three $\theta_{{\tt A}}$ and
three $\theta_{{\tt AS}}$ (out of the nine $\theta_{{\tt A}}$ and nine
$\theta_{{\tt AS}}$) with the lowest cross-entropy training losses. The second
ensemble (v2) is obtained by training variants of the three $\theta_{{\tt A}}$
and three $\theta_{{\tt AS}}$ in the first ensemble using different
hyperparameters to increase diversity. The results of these two token-level
ensembles are reported in Table 8.

Ultimately, we combine the above HESM with these two ensembles using MBR
decoding, and this combined system can be viewed as a higher level of HESM as
it consists of HESM as a member of the MBR combination. This final combination
sets the state-of-the-art performance of the task, achieving the ROUGE-L score
of 32.77.

System | ROUGE-L
---|---
F1 | Prec | Rec
HESM | 31.86 | 43.52 | 28.90
TokEns(3$\theta_{\tt A}$+3$\theta_{\tt AS}$)-v1 | 32.03 | 41.01 | 30.16
TokEns(3$\theta_{\tt A}$+3$\theta_{\tt AS}$)-v2 | 32.19 | 39.59 | 30.88
\+ MBR Combination† | 32.77 | 41.69 | 30.51
Table 8: ROUGE-L on the test data. †This system attains the top position on
the shared task leaderboard.

##  5 Conclusions

In low-resource and medical-domain summarization, our work has demonstrated
that abstractive summarization outperforms extractive and zero-shot methods.
Furthermore, both token-level ensemble and MBR decoding improve the overall
performance. Our HESM, which utilizes both ensembling techniques, achieves
state-of-the-art performance with the highest ROUGE-L score in the BioNLP
2023’s shared task 1A leaderboard.

##  6 Limitations

The limitations of this work are mainly that there is a small amount of data
available for inference to test the models. ROUGE-L is used as an assessment
metric and n-gram overlap metrics are notably not optimal for abstractive
summarization assessment Zhang* et al. (2020); Deutsch (2022).

##  7 Ethics Statement

The study used de-identified health data to develop a system that overcomes
biases in medical decision-making. However, social biases in language models
need to be addressed to ensure fairness in model training. Therefore, before
deploying any pre-trained language model, fairness audits are necessary to
ensure an ethical and trustworthy model for all stakeholders. Note, doctors
should not rely on automated summarization systems for diagnoses in the
interest of patient care.

## Acknowledgements

This paper reports on research supported by Cambridge University Press &
Assessment (CUP&A), a department of The Chancellor, Masters, and Scholars of
the University of Cambridge. This research is further supported by the EPSRC
(The Engineering and Physical Sciences Research Council) Doctoral Training
Partnership (DTP) PhD studentship, the Cambridge International & St John’s
College scholarship, and the Gates Cambridge Scholarship.

## References

  * Black et al. (2022) Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022.  GPT-NeoX-20B: An open-source autoregressive language model.  In _Proceedings of the ACL Workshop on Challenges & Perspectives in Creating Large Language Models_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Deutsch (2022) Daniel Deutsch. 2022.  _Methods for Text Summarization Evaluation_.  Ph.D. thesis, University of Pennsylvania.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * El-Kassas et al. (2021) Wafaa S. El-Kassas, Cherif R. Salama, Ahmed A. Rafea, and Hoda K. Mohamed. 2021\.  Automatic text summarization: A comprehensive survey.  _Expert Systems with Applications_ , 165:113679.
  * Fathullah et al. (2021) Yassir Fathullah, Mark J.F. Gales, and Andrey Malinin. 2021.  Ensemble distillation approaches for grammatical error correction.  In _ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 2745–2749.
  * Fort et al. (2020) Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. 2020.  Deep ensembles: A loss landscape perspective.
  * Freitag et al. (2017) Markus Freitag, Yaser Al-Onaizan, and Baskaran Sankaran. 2017.  Ensemble distillation for neural machine translation.  _arXiv preprint arXiv:1702.01802_.
  * Freitag et al. (2022) Markus Freitag, David Grangier, Qijun Tan, and Bowen Liang. 2022.  High quality rather than high model probability: Minimum Bayes risk decoding with neural metrics.  _Transactions of the Association for Computational Linguistics_ , 10:811–825.
  * Galar et al. (2011) Mikel Galar, Alberto Fernandez, Edurne Barrenechea, Humberto Bustince, and Francisco Herrera. 2011.  A review on ensembles for the class imbalance problem: bagging-, boosting-, and hybrid-based approaches.  _IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)_ , 42(4):463–484.
  * Gao et al. (2022) Yanjun Gao, Dmitriy Dligach, Timothy Miller, Dongfang Xu, Matthew M. M. Churpek, and Majid Afshar. 2022.  Summarizing patients’ problems from hospital progress notes using pre-trained sequence-to-sequence models.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 2979–2991, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Gao et al. (2023) Yanjun Gao, Dmitry Dligach, Timothy Miller, Matthew M. Churpek, and Majid Afshar. 2023.  Overview of the problem list summarization (probsum) 2023 shared task on summarizing patients’ active diagnoses and problems from electronic health record progress notes.  In _Proceedings of the 22nd Workshop on Biomedical Language Processing_ , Toronto, Canada. Association for Computational Linguistics.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020.  Don’t stop pretraining: Adapt language models to domains and tasks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8342–8360, Online. Association for Computational Linguistics.
  * Huang et al. (2019) Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. 2019.  Clinicalbert: Modeling clinical notes and predicting hospital readmission.  _arXiv:1904.05342_.
  * Iyer et al. (2022) Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Dániel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. 2022.  Opt-iml: Scaling language model instruction meta learning through the lens of generalization.  _arXiv preprint arXiv:2212.12017_.
  * Kumar and Byrne (2004) Shankar Kumar and William Byrne. 2004.  Minimum Bayes-risk decoding for statistical machine translation.  In _Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004_ , pages 169–176, Boston, Massachusetts, USA. Association for Computational Linguistics.
  * Lakshminarayanan et al. (2017) Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017.  Simple and scalable predictive uncertainty estimation using deep ensembles.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , 36(4):1234–1240.
  * Lehman et al. (2023) Eric Lehman, Evan Hernandez, Diwakar Mahajan, Jonas Wulff, Micah J Smith, Zachary Ziegler, Daniel Nadler, Peter Szolovits, Alistair Johnson, and Emily Alsentzer. 2023.  Do we still need clinical language models?  _arXiv preprint arXiv:2302.08091_.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Luo et al. (2022) Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. 2022.  BioGPT: generative pre-trained transformer for biomedical text generation and mining.  _Briefings in Bioinformatics_ , 23(6).  Bbac409.
  * Malinin and Gales (2021) Andrey Malinin and Mark Gales. 2021.  Uncertainty estimation in autoregressive structured prediction.  In _International Conference on Learning Representations_.
  * Manakul and Gales (2020) Potsawee Manakul and Mark Gales. 2020.  CUED_SPEECH at TREC 2020 podcast summarisation track.  _arXiv preprint arXiv:2012.02535_.
  * Manakul and Gales (2021) Potsawee Manakul and Mark Gales. 2021.  Long-span summarization via local attention and content selection.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6026–6041, Online. Association for Computational Linguistics.
  * Nallapati et al. (2017) Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017.  Summarunner: A recurrent neural network based sequence model for extractive summarization of documents.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 31.
  * Paulus et al. (2018) Romain Paulus, Caiming Xiong, and Richard Socher. 2018.  A deep reinforced model for abstractive summarization.  In _International Conference on Learning Representations_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551.
  * Rosti et al. (2007a) Antti-Veikko Rosti, Necip Fazil Ayan, Bing Xiang, Spyros Matsoukas, Richard Schwartz, and Bonnie Dorr. 2007a.  Combining outputs from multiple machine translation systems.  In _Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference_ , pages 228–235, Rochester, New York. Association for Computational Linguistics.
  * Rosti et al. (2007b) Antti-Veikko Rosti, Spyros Matsoukas, and Richard Schwartz. 2007b.  Improved word-level system combination for machine translation.  In _Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics_ , pages 312–319, Prague, Czech Republic. Association for Computational Linguistics.
  * Sennrich et al. (2015) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015.  Improving neural machine translation models with monolingual data.  _arXiv preprint arXiv:1511.06709_.
  * Sim et al. (2007) K. C. Sim, W. J. Byrne, M. J. F. Gales, H. Sahbi, and P. C. Woodland. 2007.  Consensus network decoding for statistical machine translation system combination.  In _2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP ’07_ , volume 4, pages IV–105–IV–108.
  * Wang and Komatsuzaki (2021) Ben Wang and Aran Komatsuzaki. 2021.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax.
  * Wortsman et al. (2022) Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. 2022.  Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.  In _International Conference on Machine Learning_ , pages 23965–23998. PMLR.
  * Zhang* et al. (2020) Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.

##  Appendix A More details about experiments

###  A.1 Inference Hyperparameters

num_beams = 4, length_penalty = 0.6, min_length = 5, max_length = 256,
no_repeat_ngram_size = 4.

###  A.2 RL training

We follow Paulus et al. (2018) in using reinforcement learning (RL) based
loss:

$\mathcal{L}_{\text{RL}}=(\mathcal{R}(\bar{\mathbf{y}},\mathbf{y})-\mathcal{R}(\hat{\mathbf{y}},\mathbf{y}))\log
P(\hat{\mathbf{y}}|\mathbf{x})$ (6)

where $\bar{\mathbf{y}}$ is the sequence obtained by greedy search,
$\hat{\mathbf{y}}$ is the sequence obtained by sampling, and
$\hat{\mathbf{y}}$ is the ground-truth sequence. To improve the stability of
training, we initialize the model using the weights from the maximum
likelihood training (cross-entropy loss), and we use a combined loss:
$\mathcal{L}=\gamma\mathcal{L}_{\text{RL}}+(1-\gamma)\mathcal{L}_{\text{ML}}$
where $\gamma$ = 0.9 and $\mathcal{L}_{\text{ML}}$ is the standard cross
entropy loss. The results are provided in Table 11, showing that a marginal
gain can be achieved from RL training.

##  Appendix B Additional Results

###  B.1 Zero-shot Summarization

Since state-of-the-art LLMs such as GPT-3 or ChatGPT are only available via
API services, using them would violate the MIMIC data use agreement. Instead,
we use open-source LLMs. We use the following text input to large language
models (LLMs),

        {prompt}:
        {clinical_note}

where clinical_note is the assessment section, and we consider two prompts:

  * •

P1: Summarize these clinical notes.

  * •

P2: Give a one or two word summary for these clinical notes.

We use open-source LLMs including OPT-IML Iyer et al. (2022), GPT-J Wang and
Komatsuzaki (2021), and GPT-NeoX Black et al. (2022). We provide the results
on training data in Table 9. The poor performance could be attributed to the
small size of LLMs, and larger systems such as GPT-3 or ChatGPT could
potentially perform much better.

LLM | prompt | R1 | R2 | RL
---|---|---|---|---
OPT-IML-1.3B | P1 | 4.97 | 0.80 | 4.37
| P2 | 4.46 | 0.65 | 4.05
OPT-IML-30B | P1 | 2.76 | 0.44 | 2.51
| P2 | 2.07 | 0.36 | 1.96
GPT-J-6B | P1 | 4.13 | 0.58 | 3.68
| P2 | 4.66 | 0.73 | 4.29
GPT-NeoX-20B | P1 | 2.41 | 0.38 | 2.21
| P2 | 3.04 | 0.59 | 2.79
Table 9: Zero-shot Summarization performance on training data of LLMs with
different user prompts.

###  B.2 More Analysis

Table 10 shows our post-evaluation studies on the performance using different
input fields, and the results suggest that it is possible to improve the
performance further by using {A}+{S}+{O} in addition to {A} and {A}+{S} as the
input.

Inputs | R1 | R2 | RL
---|---|---|---
{A} | 32.22 | 14.30 | 30.02
{A}+{O} | 32.50 | 13.81 | 30.31
{A}+{S} | 33.46 | 15.07 | 31.03
{A}+{S}+{O} | 33.80 | 15.38 | 31.28
Table 10: Comparison of Clinical-T5-large performance when using different
inputs on training data (using cross-validation).

###  B.3 Submitted Systems

In Table 11, we present other approaches that were submitted to the shared
task, including model weight averaging, and RL-based training. These models
also formed components of the final HESM model submitted.

System | ROUGE-L
---|---
F1 | Prec | Rec
Weight Avg. of 3$\theta_{{\tt A}}$ | 30.26 | 42.51 | 27.31
TokEns 3$\theta_{{\tt A}}$ w/ $\mathcal{L}_{\text{RL}}^{\dagger}$ | 30.40 | 43.78 | 27.10
Best-performing $\theta_{{\tt A}}$ | 30.56 | 39.97 | 28.95
TokEns 9$\theta_{{\tt A}}$-v1 | 30.74 | 42.14 | 27.93
TokEns 9$\theta_{{\tt A}}$-v2 | 30.50 | 41.09 | 28.37
TokEns 9$\theta_{{\tt AS}}$ | 30.04 | 36.35 | 29.96
Table 11: ROUGE-L scores on test data of the members of HESM.
${}^{\dagger}\mathcal{L}_{\text{RL}}$ is described in Appendix A.2.

##  Appendix C Post-evaluation Ablation Study

The results in Table 8 found that a higher-level Hierarchical Ensemble (HESM)
model had the best performance. This model performs MBR decoding over the
output from an existing shallower HESM model (we will refer to as HESM-
shallow) and 2$\times$token-level ensemble of 3$\theta_{{\tt
A}}$+3$\theta_{{\tt AS}}$. Table 12 explores the impact on performance with
unpacking the token level-ensemble systems and performing MBR decoding over
all individual systems. The system labelled with unpack-1 performs MBR
decoding over the 6 ensemble systems that form HESM-shallow and the 12
individual systems used to make the two token level ensemble systems; i.e. MBR
decoding is performed over 18 system output sequences. The unpack-2 system
considers further unpacking the 6 ensemble systems used for HESM-shallow, such
that MBR decoding is now performed over a total of 34 unique individual
systems.

System | ROUGE-L
---|---
F1 | Prec | Rec
HESM (final) | 32.77 | 41.69 | 30.51
HESM-unpack-1 | 32.38 | 42.98 | 29.91
HESM-unpack-2 | 32.26 | 43.18 | 29.62
Table 12: ROUGE-L scores on the test data. This table considers the impact of
performing MBR decoding on the individual systems after unpacking token-level
ensemble systems used as components for the final higher-level HESM model
submitted in the competition in Table 8.


EXCESS SPECIFIC HEAT OF PTFE AND PCTFE AT LOW TEMPERATURES: APPROXIMATION
DETAILS

Nina B. Bogdanova$\,{}^{a)}$, B.M. Terziyska$\,{}^{b)}$

$\,{}^{a)}$Inst.of Nuclear Research,BAS, 72 Tzarigradsko choussee, 1784 Sofia,
Bulgaria<EMAIL_ADDRESS>
$\,{}^{b)}$Inst.of Solid State Physics,BAS, 72 Tzarigradsko choussee, 1784
Sofia, Bulgaria<EMAIL_ADDRESS>
###### Abstract

Approximation of the previously estimated excess specific heat
$C^{excess}/T^{5}$ of two fluoropolymers, $PTFE$ and $PCTFE$, is presented
using Orthonormal Polynomial Expansion Method (OPEM). The new type of
weighting functions in OPEM involves the experimental errors in every point of
the studied thermal characteristic. The investigated temperature dependence of
the function $C^{excess}/T^{5}$ is described in the whole temperature ranges
$0.4\div 8K$ and $2.5\div 7K$ respectively for PTFE and PCTFE as well as in
two subintervals $(0.4\div 2)K$, $(2.5\div 8)K$ for PTFE. Numerical results of
the deviations between the evaluated $C^{excess}/T^{5}$ data and their
approximating values are given. The usual polynomial coefficients obtained by
orthonormal ones in our OPEM approach and the calculated in every point
absolute, relative and specific sensitivities of the studied thermal
characteristic are proposed too. The approximation parameters of this type
thermal characteristic are shown in Figures and Tables.

Key words: orthonormal and usual polynomial approximation, low-temperature
excess specific heat of two fluoropolymers - PTFE and PCTFE

I.INTRODUCTION

The unusual thermal properties concerning the low-temperature specific
features of the heat capacities of the polytetrafluoroethylene (PTFE) and
polyclorothrifluoroethylene (PCTFE) were considered in an earlier paper [1].
The estimated there excess specific heat over Debye contribution below 10 K of
these fluoroplasts was discussed in the frame of the recently developed Soft-
Potential Model (SPM).
The present study is devoted to the mathematical description of the low-
temperature excess specific heat of these semi-crystalline polymers applying
Orthonormal Polynomial Expansion Method (OPEM) [2].

II.EXCESS SPECIFIC HEAT DATA

The previous work [1] clarifies several points concerning the specific heat
peculiarities observed in $C_{p}/T^{3}$ temperature dependencies of PTFE and
PCTFE, a maximum appearing around $T_{max}$= 4 K for both polymers and a found
shallow minimum for PTFE centered at $T_{min}=1~{}K$. Following the Soft-
Potential Model (SPM) [3, 4, 5, 6], supposing a coexistence of acoustic
phonons with quasi-localized low-frequency (soft) modes in glasses and
successfully applied by us to some chalcogenide glasses [7, 8], the specific
heat data, taken from the available calorimetric measurements at low-
temperatures ($T~{}<10$ K) for the studied fluoroplasts [9, 10], were
described in a paper [1]. The $C_{p}$ components are:
i) $C_{p}^{TLS}$ \- a linear contribution, described by double-well
potentials, conditioned by the thermal exitations of the tunneling state
(TLS); for PTFE it was established to predominate at $T~{}\leq 0.2~{}K$.
ii) $(C_{p}^{D})^{acou}$ \- a cubic Debye contribution, evaluated and
discussed in details in a paper [1].
iii) $C_{p}^{exc(SM)}$ \- an excess specific heat (a soft mode) contribution
of the quasi-harmonic exitations, described by single-well potentials.
$C_{p}^{exc(SM)}$ component was evaluated [1] by difference between the
measured specific heat $C_{p}$ [9, 10] and the sum ($C_{p}^{TLS}$ \+
$(C_{p}^{D})^{acou}$) as follows.

$C_{p}^{exc(SM)}=C_{p}-[C_{p}^{TLS}+(C_{p}^{D})^{acou}],$ (1)

The temperature dependencies of the three specific heat components in Eq. 1
are:

$C_{p}^{TLS}=C_{TLS}T,\hskip 8.5359ptC_{p}^{D}=(C_{D})^{acou}T^{3},\hskip
8.5359ptC_{p}^{exc}=C^{exc(SM)}T^{5}.$ (2)

The abbreviations of Two Level State, Debye and excess specific heat (Soft
Modes) are marked with $TLS$, $D$, and $exc(SM)$, respectively. Here,
$C_{TLS}$ was determined by the $C_{p}$ experimental data of Nittke et al [9];
$(C_{D})^{acou},$ the true elastic coefficient, was calculated [1] by the
macroscopic parameters of the investigated materials and the average sound
velocity $v_{s}$, evaluated [1] for PTFE (PCTFE) by available measurements of
transversely and longitudinally polarized 10 (5) MHz ordinary sound waves
between 4.2 and 140 (180) K. Note that the average sound velocity $v_{s}$ for
both polymers changes its value about 0.3 % within 0 and 10 K. This fact
allows us to accept up to 10 K a constant Debye coefficient of the Debye
specific heat contribution by acoustic measurements .
The quantities $C_{D}$ up to 10 K and $C_{TLS}$ are independent of the
temperature. In accordance with the SPM, the softening of the lattice
vibrations leads to increasing of the density of states of the harmonic
oscillators with rising of their energy $E$. This increase appears to be
proportional to the energy as $E^{4}$ leading to proportional to $T^{5}$ term
in $C_{p}(T)$ at enough low-temperatures, i.e. the quantity $C^{exc(SM)}$ is
temperature independent only in a narrow low-temperature range,
$0<T<1.5T_{min}$. It is worth to mention that the quantity $C^{exc(SM)}$
evaluated for some glasses [11, 12, 13] confirms this prediction of the SPM.
For the studied polymers [1] $T_{min}$ was estimated to be very close to 1 K.
But in wider low-temperature range the temperature dependencies of the
quantities $C_{p}^{exc(SM)}/T^{5}$ and the excess specific heat
$C_{p}^{exc(SM)}$ presented, respectively in Fig.1 and in the inset of the
Fig.1, are temperature dependent. As it can be seen in the inset of Fig.1, the
Soft Modes (SM) delocalized at T = 8 K for PTFE and at T = 7 K for PCTFE.

Figure 1:  Temperature dependencies of the excess specific heat presented as
$C^{exc}/T^{5}$; inset: Temperature dependencies of the excess specific heat
$C^{exc}$ $[mJ/gK]$ of the PTFE and PCTFE

.

It is important to note that $C^{exc}/T^{5}$ function changes its temperature
behavior just at $T=(3/2)T_{min}$.

III. MATHEMATICAL APPROACH

Our Orthonormal Polynomial Expansion Method(OPEM) and its applications in
cryogenic thermometry are presented in papers [2, 14, 15, 16]. Some important
features of OPEM concerning cryogenic thermometry at the approximation of
thermometric characteristics of different type low- temperature sensors are
protected by a patent for an invention [15]. Our OPEM is defined on Forsythe
[16] three-term relation for constructing orthogonal polynomials over discrete
point set with arbitrary weights in the term of the least square method. The
one-dimensional recurrence for generation of orthonormal polynomials
$\\{\Psi_{k}^{(0)}$, $k=1,2,\ldots\\}$ and their derivatives
$\\{\Psi_{k}^{(m)},m=0,1,2,\ldots\\}$, in OPEM is:

$\Psi_{k+1}^{(m)}(q)=\gamma_{k+1}[(q-\alpha_{k+1})\Psi_{k}^{(m)}(q)-\\\
(1-\delta_{k0})\beta_{k}\Psi_{k-1}^{(m)}(q)+m\Psi_{k}^{(m-1)}(q)]$

OPEM is a development of the Forsythe approach for receiving derivatives and
integrals with fourth term in the Eq. (3). The polynomials
$\\{\Psi_{k}^{(0)}\\}$ satisfy the orthogonality relations:

$\sum_{i=1}^{M}w_{i}\Psi_{k}^{(0)}(q_{i})\Psi_{l}^{(0)}(q_{i})=\delta_{kl}$

over the point set $\\{q_{i},i=1,2,\ldots M\\}$ with weights
$w_{i}=1/\sigma_{i}^{2}$ , depending on errors $\sigma_{i}$ in every point.
The approximating values $f^{appr}$ of the function and its $m$-th derivative
$f^{(m)appr}$,$\\{m=0,1,..\\}$ are calculated by

$f^{(m)appr}(q)=\sum_{k=0}^{N}a_{k}\Psi_{k}^{(m)}(q)=\sum_{k=0}^{N}c_{k}q^{k}.$
(3)

The optimal degree $N$ of the approximating polynomials in Eq.(4) is selected
by the algorithm, combining the following two criteria. First, the fitting
curve should lie in the error corridor of the dependent variable
$(q_{j},f_{j}^{exp}\pm{\sigma_{j}},j=1,...M)$.

$(f_{j}^{appr}-f_{j}^{exp})^{2}w_{j}\leq{1}.$ (4)

Second, the minimum $\chi^{2}$ should be reached.

$\sum_{j=1}^{M}{w_{j}(f_{j}^{appr}-f_{j}^{exp})^{2}}/(M-N-1)\rightarrow{min}.$
(5)

When the first criterion is satisfied, the search of the minimum $\chi^{2}$
stops. The development of the algorithm in the biophysics with the total
variance formula for involving the errors in independent variable was
published in a paper [17]. The last version with obtaining of usual $c_{k}$
coefficients from orthogonal ones $a_{k}$ from Eq. 3 is developed in our work,
RSI-2005 [2].

IV. APPROXIMATION RESULTS
A. Orthonormal expansion

The temperature dependence of the $C^{exc}/T^{5}$ function is described by
orthonormal polynomials in the whole temperature range $0.4\div 8$ $[K]$ or in
two subintervals $0.4\div 2$ $[K]$, $2.5\div 8$ $[K]$ for PTFE, and in the
temperature range $2.5\div 7$ $[K]$ for PCTFE, using the new type of weights,
$W^{C^{exc}/T^{5}}$. The studied thermal characteristic $C^{exc}/T^{5}$ and
its sensitivities, absolute $d(C^{exc}/T^{5})/dT$, relative
$[1/(C^{exc}/T^{5})]d(C^{exc}/T^{5})/dT$ and specific,
$[T/(C^{exc}/T^{5})]d(C^{exc}/T^{5})/dT$, evaluated in every point, are shown
in linear-linear plot in Figs.2 and 3 for PTFE and PCTFE, respectively. It is
worth to note that the subintervals for PTFE are chosen by the temperature
behavior of the specific sensitivity of this polymer (see Fig.2).

Figure 2: Temperature dependencies of the relative
$[1/(C^{exc}/T^{5})]d(C^{exc}/T^{5})/dT$ $[K^{-1}]$ and specific
$[T/(C^{exc}/T^{5})]d(C^{exc}/T^{5})/dT$ $[-]$ sensitivities of the PTFE;
inset: Temperature dependencies of the absolute sensitivity
$d(C^{exc}/T^{5})/dT$.

Figure 3:  Temperature dependencies of the relative
$[1/(C^{exc}/T^{5})]d(C^{exc}/T^{5})/dT$ $[K^{-1}]$ and specific
$[T/(C^{exc}/T^{5})]d(C^{exc}/T^{5})/dT$ $[-]$ sensitivities of the PCTFE;
inset: Temperature dependencies of the absolute sensitivity
$d(C^{exc}/T^{5})/dT$ $[mJ/gK^{7}]$.

By definition the weighting function $W^{C^{exc}/T^{5}}$ is $1/\sigma^{2}$,
where $\sigma^{2}$ is a variance of the thermal characteristic $C^{exc}/T^{5}$
versus temperature $T$. In our investigation this variance is accepted to be,
correspondingly square of the absolute heat capacity resolution $\Delta
C^{exc}_{acr}$, determined by the experimental specific heat accuracy as
follows: $(\Delta C^{exc}_{acr})_{i}=0.025(C^{exc}/T^{5})_{i}$ $[mJ/gK^{6}]$
for the first approximating interval of PTFE and $(\Delta
C^{exc}_{acr})_{i}=0.1(C^{exc}/T^{5})_{i}$ $[mJ/gK^{6}]$ for both, the whole
PCTFE approximating interval and the second approximating interval of PTFE.
Here the weights, $W^{C^{exc}/T^{5}}$ are expressed by the relations:

$(W^{C^{exc}/T^{5}})_{i}=1/(\Delta
C^{exc}/T^{5}_{acr})_{i}=1600/(C^{exc}/T^{5})^{2}_{i}~{}~{}[mJ/gK^{6}]^{-2}$
(6)

for the first approximating interval of PTFE, or

$(W^{C^{exc}/T^{5}})_{i}=1/(\Delta
C^{exc}_{acr})_{i}=100/(C^{exc}/T^{5})^{2}_{i}~{}~{}[mJ/gK^{6}]^{-2}$ (7)

for the PCTFE and the second approximating interval of PTFE. The deviations
$\Delta(C^{exc}/T^{5})_{i}$ between experimental and approximating values of
the excess specific heat are estimated in every point by the expression:

$\Delta(C^{exc}/T^{5})_{i}=(C^{exc}/T^{5})_{i}^{exp}-(C^{exc}/T^{5})_{i}^{appr}$
(8)

The temperature behavior of the calculated differences
$\Delta(C^{exc}/T^{5})_{i}$, the root mean square deviations $RMS^{C}$ and the
mean absolute deviations $MAD^{C}$ respectively for PTFE and PCTFE excess
specific heat approximations are shown in the Figs.4,5 and 6.

Figure 4: Temperature dependencies of $\Delta(C^{exc}/T^{5})$,
$(\Delta(C^{exc}/T^{5})_{acr}$, $MAD^{C^{exc}/T^{5}}$ and
$RMS^{C^{exc}/T^{5}}$ of the PTFE for the whole approximated temperature
range;inset:Temperature dependence of the weighting function $w(T)$ for FTPE .

Figure 5: Temperature dependencies of $\Delta(C^{exc}/T^{5})$,
$(\Delta(C^{exc}/T^{5})_{acr}$, $MAD^{C^{exc}/T^{5}}$ and
$RMS^{C^{exc}/T^{5}}$ of the PTFE for two interval approximation.

Figure 6: Temperature dependencies of $\Delta(C^{exc}/T^{5})$,
$\Delta(C^{exc})_{acr}$, $MAD^{C^{exc}/T^{5}}$ and $RMS^{C^{exc}/T^{5}}$ of
the PCTFE for the whole approximated temperature range.

The RMS deviation is more popular. It is given in another our paper [2]. The
characteristics MAD is defined as follows:

$MAD^{C}=\frac{1}{M}\sum_{i}|{\Delta(C^{exc}/T^{5})_{i}-\overline{\Delta(C^{exc}/T^{5})}|},$
(9)

where
$\overline{\Delta(C^{exc}/T^{5})}=\frac{1}{M}\sum_{i}{\Delta(C^{exc}/T^{5})_{i}}$
and $\Delta(C^{exc}/T^{5})_{i}$ are calculated from Eq.(9).
Following the cited criteria in Eqs.(5,6) the deviations
$\Delta(C^{exc}/T^{5})_{i}$ are in the error corridor (see Eq.5). As a result
of our approximation, the polynomial degree for temperature range of PTFE
subintervals is lower than for the whole temperature range. For
$C^{exc}/T^{5}$ vs. $T$ approximation the optimal degree $N$ as well as some
main characteristics including the overall approximation characteristics: RMS
and MAD of $\Delta C^{exc}/T^{5}$ vs. $T$ approximation, the weighting
function $W^{C}$ and a goodness of fit $\chi^{2}$ are presented in the TABLE
1.

B.Usual expansion obtained by orhonormal one

The next step in our approximation is done here. The OPEM is extended by
calculation of usual coefficients $\\{{c}_{k}\\}$ from orthonormal ones
$\\{{a}_{k}\\}$ using Eq.(4). This is carried out for the investigated
intervals. The calculations are made for the $C^{exc}/T^{5}=f(T)$
descriptions, in two runs: first - in an interval [-1,1], and second - in the
input intervals. Optimal values of usual polynomial degrees are chosen using
two new criteria.

Table 1:  OPEM approximations of $C^{excess}/T^{5}$ for $PTFE$ and $PCTFE$ T | C | N | $W^{C}$ | $RMS^{C}$ | $MAD^{C}$ | $\chi^{2}$
---|---|---|---|---|---|---
$[K]$ | $[\mu J/gK^{6}]$ | - | $[mJ/gK^{6}]$ | $[mJ/gK^{6}]$ |  - |
$0.4\div 2$ | $152\div 1$ | 6 | $6.91\times{10}^{4}\div 1.60\times{10}^{9}$ | $0.121\times 10^{-2}$ | $0.828\times 10^{-3}$ | 1.38
$2.5\div 8$ | $1\div 0.015$ | 4 | $2.65\times 10^{8}\div 4.44\times 10^{11}$ | $0.265\times 10^{-4}$ | $0.208\times 10^{-4}$ | 0.57
$0.4\div 8$ | $152\div 0.015$ | 8 | $6.91\times 10^{4}\div 4.44\times 10^{11}$ | $0.109\times 10^{-2}$ | $0.326\times 10^{-3}$ | 0.95
$2.5\div 7$ | $1.75\div 0.031$ | 6 | $3.26\times 10^{7}\div 1.04\times 10^{11}$ | $0.428\times 10^{-4}$ | $0.306\times 10^{-4}$ | 0.58
Table 2: Usual coefficients for $C^{excess}/T^{5}$ $[10^{-3}mJ/gK^{6}]$ vs. $T$ approximation $material$ | $PTFE$ | $PCTFE$
---|---|---
$RangeT(K)$ | $0.4\div 2$ | $2.5\div 8$ | $0.4\div 8$ | $2.5\div 7$
$Range(C^{exc.}/T^{5})$ | $152\div 1$ | $1\div 0.015$ | $152\div 0.015$ | $1.75\div 0.031$
$c_{0}$ | .022854 | .003118 | .016067 | .182459
$c_{1}$ | -.001420 | -.009158 | -.063937 | -.667996
$c_{2}$ | .105876 | .011825 | .103755 | .997805
$c_{3}$ | -.582978 | -.006389 | -.070355 | -.769121
$c_{4}$ | -.390617 | .001190 | -.008661 | .320591
$c_{5}$ | .845833 | - | .048421 | -.068068
$c_{6}$ | .696453 | - | -.033867 | .005687
$c_{7}$ | - | - | .010411 | -
$c_{8}$ | - | - | -.001230 | -

The first criterion is:

$\max|f_{i}^{exp}-f_{i}^{appr,u}|/f_{i}^{exp}\rightarrow{min},$ (10)

where $f^{appr,u}$ is the approximating function defined with usual expansion.
The second criterion is:

$\Delta\\{c_{k}\\}_{1}^{N}\rightarrow{min},$ (11)

where $\\{\Delta c_{k}\\}$ are inherited errors in usual coefficients, defined
in OPEM, discussed in a paper[18]. In the TABLE 2 the usual coefficients for
$C^{exc}/T^{5}$ vs. $T$ approximation are presented.

In conclusion, the heat capacity components of the studied fluoroplasts are
well defined earlier [1] using the SPM. In this study, the temperature
dependencies of the excess specific heat of PTFE and PCTFE below 10 K
presented as $C^{exc(SM)}/T^{5}$ are described by an Orthonormal Polynomial
Expansion Method (OPEM). An approximation of the above mentioned thermal
characteristic by usual polynomials, obtained by orthonormal ones, is given
too. The numerical results for approximation characteristics as root mean
square deviation, mean absolute deviation and normalized $\chi$ square assure
good accuracy for calculated values of physical quantity in the input
temperature intervals. Our approach for description of this characteristic
proposes possibility for the future analyzing the low-temperature excess
specific heat of the materials with glass-like behavior.

## References

  * [1] B. Terziyska, H. Madge, Some special feature of low-temperature specific heat of PTFE and PCTFE analyzed within the Soft Potential Model, to be published.
  * [2] N.Bogdanova, B. Terzijska, Thermometric characteristics approximation of Germanium film temperature microsensors by orthonormal polynomials, Rev. Sci. Instrum. 68 (10), 3766-3771 (2005).
  * [3] M.A.Il’in, Karpov V.G., and Parshin D.A., Parameters of soft atomic potentials in glasses, Zh. Exper. Teor. Fiz. 92, 291 (1987).
  * [4] V.G.Karpov , Klinger M.I., and Ignat’ev F.N., Theory of low-temperature anomalies in the thermal properties of amorphic structures, Zh. Exper. Teor. Fiz. 84 (2), 760-775 (1983).
  * [5] D.A.Parshin, Interactions of soft atomic potentials and universality of low- temperature properties of glasses, Phys. Rev. B 49 (14), 9400-9418 (1994).
  * [6] D.A.Parshin, Liu X., Brand O., and Lohneysen H.v., Analysis of the low-temperature specific heat of amorphous AsxSe1-x within the soft potential model, Z. Phys. B 93 (1), 57-62 (1993).
  * [7] B. Terziyska, H. Misiorek, E. Vateva, A. Jezovski, and D.Arsova, Low-temperature thermal conductivity of GexAs40-xS60 glasses, SSC 134, 349 (2005).
  * [8] B. Terziyska, A. Czopnik, E. Vateva, D. Arsova, and R. Czopnik, Low-temperature specific heat of Ge-As-S glasses, Phil. Mag. Letters, 85, 145-150 (2005).
  * [9] A. Nittke, P. Esquinazi, H.C. Semmelhack, A.L. Burin, A.Z. Patashinskii, Eur. Phys. J. B8, 19 (1999).
  * [10] B. Terziyska, H. Madge, and V. Lovtchinov, Specific heat of PTFE and PCTFE within the temperatute range 2.5-20 K, Journal of Thermal Analysis 20, 33 (1981).
  * [11] M.A.Ramos, Talon C, and Vieira S.,The Boson peak in structural and oriental glasses of simple alcohols: specific heat at low temperatures, J. Non-Cryst. Sol. 307-310, 80-86 (2002) and references therein.
  * [12] C.Talon , Ramos M.A., and Vieira S.,Low-temperature specific heat of amorphous, orientational glass, and crystal phases of ethanol, Physical Review B 66, 012201(4)(2002).
  * [13] M.A.Ramos, Talon C., Jimenez-Rioboo RJ, and Vieira S. Low-temperature specific heat of structural and orientational glasses of simple alcohols, J. Phys.:Condens. Matter 15, S1007-S1018 (2003).
  * [14] N.Bogdanova, B. Terzijska, A novel approach to the Rh-Fe thermometric characteristic approximation, Commun. JINR, Dubna, E11–97-396 (1997).
  * [15] B. Terzijska, N.Bogdanova, Weighted Orthonormal Polynomial Method in Cryogenic Thermometry, A Patent for an Invention, Bulgaria, No. 62582 (2000).
  * [16] G.Forsythe, Generation and Use of Orthogonal Polynomials, J. Soc. Ind. Appl. Math. 5, 74-88 (1957).
  * [17] N.Bogdanova, S. Todorov, Fitting of water Hydrogen bond energy data with uncertainties in both variables by help of orthogonal polynomials, Int. J. Modern Physics C 12, 1 (2001).
  * [18] N.Bogdanova, Orthonormal Polynomial Expansion Method with errors in variables, E11-98-3, Communication JINR, Dubna (1998).