$\displaystyle\int_{\Theta}\mathrm{d}^{k}\theta\,p_{\mathrm{f}}(\bm{\theta})\ln\left(\frac{p_{\mathrm{f}}(\bm{\theta})}{p_{\mathrm{s}}(\bm{\theta})}\right)$
$\displaystyle=$
$\displaystyle\int_{\Theta}\mathrm{d}^{k}\theta\,p_{\mathrm{f}}(\bm{\theta})\left[\ln\left(\sqrt{\frac{|{\mathbf{F}_{\mathrm{s}}}^{-1}|}{|\mathbf{F}_{\mathrm{f}}^{-1}|}}\right)+\tfrac{1}{2}\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}(\mathbf{F}_{\mathrm{s}}-\mathbf{F}_{\mathrm{f}})\bm{\Delta}\right]$
$\displaystyle=$
$\displaystyle\ln\left(m^{-k/2}\sqrt{\frac{|{\mathbf{F}^{\prime}}_{\mathrm{s}}^{-1}|}{|\mathbf{F}_{\mathrm{f}}^{-1}|}}\right)+\tfrac{1}{2}\int_{\Theta}\mathrm{d}^{k}\theta\,p_{\mathrm{f}}(\bm{\theta})\cdot\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}(m\mathbf{F}^{\prime}_{\mathrm{s}}-\mathbf{F}_{\mathrm{f}})\bm{\Delta}\,,$
(74)

since the integral of the posterior (which we take to be normalised) over
parameter space is equal to 1. In the last line, we also used the definition
of the FIM in equation (15) and
$\mathbb{C}_{\mathrm{s}}^{-1}\\!=\\!m\mathbf{M}$ to find that

$F_{\mathrm{s},ij}=h_{k,i}(\mathbb{C}_{\mathrm{s}}^{-1})_{kl}h_{l,j}=mh_{k,i}M_{kl}h_{l,j}=mF^{\prime}_{\mathrm{s},ij}\,,$
(75)

or $\mathbf{F}_{\mathrm{s}}=m\mathbf{F}^{\prime}_{\mathrm{s}}$, from which we
obtain

$|\mathbf{F}_{\mathrm{s}}|=m^{k}|\mathbf{F}^{\prime}_{\mathrm{s}}|\,.$ (76)

The KL divergence is a function of $m$, with the minimum being the $m$ at
which the derivative with respect to $m$ vanishes (it is clear, given our
restriction to considering Gaussians, that there will always be one extremum
only, which is a minimum). The minimum is given by the $m=m_{\mathrm{KL}}$
solving
$\frac{\mathrm{d}}{\mathrm{d}m}D_{\mathrm{KL}}(p_{\mathrm{f}}(\bm{\theta})\,||\,p_{\mathrm{s}}(\bm{\theta}))=0$,
which is

$\displaystyle m_{\mathrm{KL}}$
$\displaystyle=k\left(\int_{\Theta}\mathrm{d}^{k}\theta\,p_{\mathrm{f}}(\bm{\theta})\cdot\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}\mathbf{F}^{\prime}_{\mathrm{s}}\bm{\Delta}\right)^{-1}\,.$ (77)

To compute the integral, let us change coordinates first by a translation
$\theta_{\mu}\to{\theta}^{\star}_{\mu}$ that takes
$\bm{\hat{\theta}}\to{\bm{\hat{\theta}}}^{\star}=\mathbf{0}$, centering the
peak of the posterior at the origin, so that
$\bm{\Delta}\to{\bm{\Delta}}^{\star}={\bm{\theta}}^{\star}$. Then perform a
rotation ${\theta}^{\star}_{\mu}\to\widetilde{\theta}_{\mu}$ such that
$\mathbf{F}_{\mathrm{f}}^{\star}\to\widetilde{\mathbf{F}}_{\mathrm{f}}$, where
the matrix $\widetilde{\mathbf{F}}_{\mathrm{f}}$ is diagonal, i.e., aligning
the coordinate axes with the ellipsoid axes. We denote all objects in the
final coordinates with a tilde accent. Both transformations are unitary
resulting in a Jacobian determinant equal to 1, and if we further suppose that
$\bm{\Theta}\sim\mathbb{R}^{k}$ then also
$\widetilde{\bm{\Theta}}\sim\mathbb{R}^{k}$, so we can write the integral in
(77) as

$\displaystyle\int_{\Theta}\mathrm{d}^{k}\theta\,p_{\mathrm{f}}(\bm{\theta})\cdot\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}\mathbf{F}^{\prime}_{\mathrm{s}}\bm{\Delta}$
$\displaystyle=\int_{\widetilde{\Theta}}\mathrm{d}^{k}\widetilde{\theta}\,p_{\mathrm{f}}(\widetilde{\bm{\theta}})\cdot\widetilde{\bm{\theta}}^{\mathrm{\scriptscriptstyle
T}}\widetilde{\mathbf{F}}^{\prime}_{\mathrm{s}}\widetilde{\bm{\theta}}$
$\displaystyle=\left[(2\pi)^{k}\left|\widetilde{\mathbf{F}}_{\mathrm{f}}^{-1}\right|\right]^{-1/2}\int_{\widetilde{\Theta}}\mathrm{d}^{k}\widetilde{\theta}\,\exp\left[-\tfrac{1}{2}\widetilde{\bm{\theta}}^{\mathrm{\scriptscriptstyle
T}}\widetilde{\mathbf{F}}_{\mathrm{f}}\widetilde{\bm{\theta}}\right]\cdot\widetilde{\bm{\theta}}^{\mathrm{\scriptscriptstyle
T}}\widetilde{\mathbf{F}}^{\prime}_{\mathrm{s}}\widetilde{\bm{\theta}}$
$\displaystyle=\left[(2\pi)^{k}\left|\widetilde{\mathbf{F}}_{\mathrm{f}}^{-1}\right|\right]^{-1/2}\int_{\widetilde{\Theta}}\mathrm{d}^{k}\widetilde{\theta}\,\exp\left[-\tfrac{1}{2}\sum_{l=1}^{k}\widetilde{F}_{\mathrm{f},ll}\widetilde{\theta}_{l}^{2}\right]\sum_{i,j=1}^{k}\widetilde{\theta}_{i}\widetilde{F}^{\prime}_{\mathrm{s},ij}\widetilde{\theta}_{j}$
$\displaystyle=\left[(2\pi)^{k}\left|\widetilde{\mathbf{F}}_{\mathrm{f}}^{-1}\right|\right]^{-1/2}\sum_{i,j=1}^{k}\widetilde{F}^{\prime}_{\mathrm{s},ij}\int_{\widetilde{\Theta}}\mathrm{d}^{k}\widetilde{\theta}\,\prod_{l=1}^{k}\exp\left[-\tfrac{1}{2}\widetilde{F}_{\mathrm{f},ll}\widetilde{\theta}_{l}^{2}\right]\widetilde{\theta}_{i}\widetilde{\theta}_{j}\,.$

Noting the well-known integral formulas:

$\int_{-\infty}^{\infty}\mathrm{d}x\;e^{-ax^{2}}=\sqrt{\pi/a}\,,\;\;\;\;\;\;\;\;\;\int_{-\infty}^{\infty}\mathrm{d}x\;x\,e^{-ax^{2}}=0\,,\;\;\;\;\;\;\;\;\;\int_{-\infty}^{\infty}\mathrm{d}x\;x^{2}\,e^{-ax^{2}}=\frac{\sqrt{2\pi}}{(2a)^{3/2}}\,,$
(78)

we find that the only surviving terms in the $i,j$ sum are the $i=j$ terms.
The integral becomes

$\displaystyle\int_{\Theta}\mathrm{d}^{k}\theta\,p_{\mathrm{f}}(\bm{\theta})\cdot\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}\mathbf{F}^{\prime}_{\mathrm{s}}\bm{\Delta}$
$\displaystyle=\left[(2\pi)^{k}\left|\widetilde{\mathbf{F}}_{\mathrm{f}}^{-1}\right|\right]^{-1/2}\sum_{i=1}^{k}\widetilde{F}^{\prime}_{\mathrm{s},ii}\int_{-\infty}^{\infty}\mathrm{d}\widetilde{\theta}_{i}\,\widetilde{\theta}_{i}^{2}\,e^{-\frac{1}{2}\widetilde{F}_{\mathrm{f},ii}\widetilde{\theta}_{i}^{2}}\prod_{j\neq
i}\int_{-\infty}^{\infty}\mathrm{d}\widetilde{\theta}_{j}\,e^{-\frac{1}{2}\widetilde{F}_{\mathrm{f},jj}\widetilde{\theta}_{j}^{2}}$
$\displaystyle=\left[(2\pi)^{k}\left|\widetilde{\mathbf{F}}_{\mathrm{f}}^{-1}\right|\right]^{-1/2}\sum_{i=1}^{k}\widetilde{F}^{\prime}_{\mathrm{s},ii}\sqrt{\frac{2\pi}{\widetilde{F}^{3}_{\mathrm{f},ii}}}\;\prod_{j\neq
i}\sqrt{\frac{2\pi}{\widetilde{F}_{\mathrm{f},jj}}}$
$\displaystyle=\sum_{i=1}^{k}\frac{\widetilde{F}^{\prime}_{\mathrm{s},ii}}{\widetilde{F}_{\mathrm{f},ii}}\,.$
(79)

If the matrix $\mathbf{G}$ diagonalises $\mathbf{F}_{\mathrm{f}}$, that is, if
$\mathbf{F}_{\mathrm{f}}\to\widetilde{\mathbf{F}}_{\mathrm{f}}=\mathbf{G}^{-1}{\mathbf{F}}_{\mathrm{f}}\mathbf{G}$
for diagonal $\widetilde{\mathbf{F}}_{\mathrm{f}}$, then the integral may be
written

$\int_{\Theta}\mathrm{d}^{k}\theta\,p_{\mathrm{f}}(\bm{\theta})\cdot\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}\mathbf{F}^{\prime}_{\mathrm{s}}\bm{\Delta}=\sum_{i=1}^{k}\frac{G^{-1}_{ij}F^{\prime}_{\mathrm{s},jl}G_{li}}{G^{-1}_{ij}F_{\mathrm{f},jl}G_{li}}\,,$
(80)

or for general symmetric matrices $\mathbf{A}$ and $\mathbf{B}$, and with
$\mathbf{P}$ diagonalising $\mathbf{A}$, we have

$\int_{\Theta}\mathrm{d}^{k}\theta\,e^{-\frac{1}{2}\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}\mathbf{A}\bm{\Delta}}\cdot\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}\mathbf{B}\bm{\Delta}=\sqrt{\frac{(2\pi)^{k}}{|\mathbf{A}|}}\sum_{i=1}^{k}\frac{P^{-1}_{ij}B_{jl}P_{li}}{P^{-1}_{ij}A_{jl}P_{li}}\,.$
(81)

Therefore our factor $m_{\mathrm{KL}}$, using (77) is

$\displaystyle m_{\mathrm{KL}}$
$\displaystyle=k\left(\sum_{i=1}^{k}\frac{G^{-1}_{ij}F^{\prime}_{\mathrm{s},jl}G_{li}}{G^{-1}_{ij}F_{\mathrm{f},jl}G_{li}}\right)^{-1}\,.$
(82)

This in particular minimises
$D_{\mathrm{KL}}(p_{\mathrm{f}}\,||\,p_{\mathrm{s}})$, which should be noted
is not equal to $D_{\mathrm{KL}}(p_{\mathrm{s}}\,||\,p_{\mathrm{f}})$: the KL
divergence not symmetric. Let us now minimise the symmetric Jeffreys’
divergence. With equation (73), this can be written as

$\displaystyle D_{\mathrm{J}}(p_{\mathrm{f}}\,||\,p_{\mathrm{s}})$
$\displaystyle\equiv\int_{\Theta}\mathrm{d}^{k}\theta\,(p_{\mathrm{f}}-p_{\mathrm{s}})\cdot\ln\left(\frac{p_{\mathrm{f}}}{p_{\mathrm{s}}}\right)$
$\displaystyle=\int_{\Theta}\mathrm{d}^{k}\theta\,(p_{\mathrm{f}}-p_{\mathrm{s}})\left[\ln\left(\sqrt{\frac{|{\mathbf{F}_{\mathrm{s}}}^{-1}|}{|\mathbf{F}_{\mathrm{f}}^{-1}|}}\right)+\tfrac{1}{2}\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}(\mathbf{F}_{\mathrm{s}}-\mathbf{F}_{\mathrm{f}})\bm{\Delta}\right]$
$\displaystyle=\tfrac{1}{2}\int_{\Theta}\mathrm{d}^{k}\theta\,(p_{\mathrm{f}}-p_{\mathrm{s}})\cdot\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}(m\mathbf{F}^{\prime}_{\mathrm{s}}-\mathbf{F}_{\mathrm{f}})\bm{\Delta}\,,$
(83)

again since the integral of both $p_{\mathrm{f}}$ and $p_{\mathrm{s}}$ over
the whole space must be equal to 1. Writing out $p_{\mathrm{f}}$ and
$p_{\mathrm{s}}$ in full we have

$\displaystyle D_{\mathrm{J}}(p_{\mathrm{f}}\,||\,p_{\mathrm{s}})$
$\displaystyle=\tfrac{1}{2}\int_{\Theta}\mathrm{d}^{k}\theta\,\left[\sqrt{\frac{|\mathbf{F}_{\mathrm{f}}|}{(2\pi)^{k}}}e^{-\frac{1}{2}\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}\mathbf{F}_{\mathrm{f}}\bm{\Delta}}-\sqrt{\frac{m^{k}|{\mathbf{F}^{\prime}}_{\mathrm{s}}|}{(2\pi)^{k}}}e^{-\frac{1}{2}m\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}\mathbf{F}^{\prime}_{\mathrm{s}}\bm{\Delta}}\right]\cdot\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}(m\mathbf{F}^{\prime}_{\mathrm{s}}-\mathbf{F}_{\mathrm{f}})\bm{\Delta}$
$\displaystyle=C\int_{\Theta}\mathrm{d}^{k}\theta\,\left[C^{\prime}e^{-\frac{1}{2}\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}\mathbf{F}_{\mathrm{f}}\bm{\Delta}}-m^{k/2}e^{-\frac{1}{2}m\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}\mathbf{F}^{\prime}_{\mathrm{s}}\bm{\Delta}}\right]\cdot\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}(m\mathbf{F}^{\prime}_{\mathrm{s}}-\mathbf{F}_{\mathrm{f}})\bm{\Delta}\,,$

where
$C=\tfrac{1}{2}\sqrt{\frac{|{\mathbf{F}^{\prime}}_{\mathrm{s}}|}{(2\pi)^{k}}}$
and
$C^{\prime}=\sqrt{\frac{|\mathbf{F}_{\mathrm{f}}|}{|{\mathbf{F}^{\prime}}_{\mathrm{s}}|}}$.
Now differentiate with respect to $m$:

$\displaystyle\frac{\mathrm{d}}{\mathrm{d}m}D_{\mathrm{J}}(p_{\mathrm{f}}\,||\,p_{\mathrm{s}})$
$\displaystyle=C\int_{\Theta}\mathrm{d}^{k}\theta\,\left[C^{\prime}e^{-\frac{1}{2}\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}\mathbf{F}_{\mathrm{f}}\bm{\Delta}}-m^{k/2}e^{-\frac{1}{2}m\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}\mathbf{F}^{\prime}_{\mathrm{s}}\bm{\Delta}}\right]\cdot\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}\mathbf{F}^{\prime}_{\mathrm{s}}\bm{\Delta}$ $\displaystyle\hskip
45.0pt-\left[\tfrac{k}{2}m^{k/2-1}e^{-\frac{1}{2}m\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}\mathbf{F}^{\prime}_{\mathrm{s}}\bm{\Delta}}-\tfrac{1}{2}m^{k/2}\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}\mathbf{F}^{\prime}_{\mathrm{s}}\bm{\Delta}e^{-\frac{1}{2}m\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}\mathbf{F}^{\prime}_{\mathrm{s}}\bm{\Delta}}\right]\cdot\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}(m\mathbf{F}^{\prime}_{\mathrm{s}}-\mathbf{F}_{\mathrm{f}})\bm{\Delta}$
$\displaystyle=C\int_{\Theta}\mathrm{d}^{k}\theta\,C^{\prime}e^{-\frac{1}{2}\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}\mathbf{F}_{\mathrm{f}}\bm{\Delta}}\cdot\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}\mathbf{F}^{\prime}_{\mathrm{s}}\bm{\Delta}$ $\displaystyle\hskip
45.0pt-m^{k/2}e^{-\frac{1}{2}m\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}\mathbf{F}^{\prime}_{\mathrm{s}}\bm{\Delta}}\left\\{\left[1+\tfrac{k}{2}-\tfrac{1}{2}m\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}\mathbf{F}^{\prime}_{\mathrm{s}}\bm{\Delta}\right]\cdot\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}\mathbf{F}^{\prime}_{\mathrm{s}}\bm{\Delta}\right.$ $\displaystyle\hskip
150.0pt\left.-\left[\tfrac{k}{2m}-\tfrac{1}{2}\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}\mathbf{F}^{\prime}_{\mathrm{s}}\bm{\Delta}\right]\cdot\bm{\Delta}^{\mathrm{\scriptscriptstyle
T}}\mathbf{F}_{\mathrm{f}}\bm{\Delta}\right\\}$
$\displaystyle=CC^{\prime}\int_{{\Theta}}\mathrm{d}^{k}{\theta}\,e^{-\frac{1}{2}\bm{{\Delta}}^{\mathrm{\scriptscriptstyle
T}}{\mathbf{F}}_{\mathrm{f}}\bm{{\Delta}}}\cdot\bm{{\Delta}}^{\mathrm{\scriptscriptstyle
T}}{\mathbf{F}}^{\prime}_{\mathrm{s}}\bm{{\Delta}}$ $\displaystyle\hskip
45.0pt-
Cm^{k/2}\int_{{\Theta}}\mathrm{d}^{k}{\theta}\,e^{-\frac{1}{2}m\bm{{\Delta}}^{\mathrm{\scriptscriptstyle
T}}{\mathbf{F}}^{\prime}_{\mathrm{s}}\bm{{\Delta}}}\cdot\bm{{\Delta}}^{\mathrm{\scriptscriptstyle
T}}\left(\left[1+\tfrac{k}{2}\right]{\mathbf{F}}^{\prime}_{\mathrm{s}}-\tfrac{k}{2m}{\mathbf{F}}_{\mathrm{f}}\right)\bm{{\Delta}}$
$\displaystyle\hskip
45.0pt+\tfrac{1}{2}Cm^{k/2}\int_{{\Theta}}\mathrm{d}^{k}{\theta}\,e^{-\frac{1}{2}m\bm{{\Delta}}^{\mathrm{\scriptscriptstyle
T}}{\mathbf{F}}^{\prime}_{\mathrm{s}}\bm{{\Delta}}}\left\\{m(\bm{{\Delta}}^{\mathrm{\scriptscriptstyle
T}}{\mathbf{F}}^{\prime}_{\mathrm{s}}\bm{{\Delta}})^{2}-\bm{{\Delta}}^{\mathrm{\scriptscriptstyle
T}}{\mathbf{F}}^{\prime}_{\mathrm{s}}\bm{{\Delta}}\cdot\bm{{\Delta}}^{\mathrm{\scriptscriptstyle
T}}{\mathbf{F}}_{\mathrm{f}}\bm{{\Delta}}\right\\}\,.$ (84)

The first two integrals of the last line are computed easily using (81) by
making the relevant matrix substitutions;

$\int_{\bar{\Theta}}\mathrm{d}^{k}{\theta}\,e^{-\frac{1}{2}\bm{{\Delta}}^{\mathrm{\scriptscriptstyle
T}}{\mathbf{F}}_{\mathrm{f}}\bm{{\Delta}}}\cdot\bm{{\Delta}}^{\mathrm{\scriptscriptstyle
T}}{\mathbf{F}}^{\prime}_{\mathrm{s}}\bm{{\Delta}}=\sqrt{\frac{(2\pi)^{k}}{|\mathbf{F}_{\mathrm{f}}|}}\sum_{i=1}^{k}\frac{D^{-1}_{\mathrm{f},ij}F^{\prime}_{\mathrm{s},jl}D_{\mathrm{f},li}}{D^{-1}_{\mathrm{f},ij}F_{\mathrm{f},jl}D_{\mathrm{f},li}}$
(85)

for diagonal
$\mathbf{D}^{-1}_{\mathrm{f}}\mathbf{F}_{\mathrm{f}}\mathbf{D}_{\mathrm{f}}$,
and

$\displaystyle\int_{{\Theta}}\mathrm{d}^{k}{\theta}\,e^{-\frac{1}{2}m\bm{{\Delta}}^{\mathrm{\scriptscriptstyle
T}}{\mathbf{F}}^{\prime}_{\mathrm{s}}\bm{{\Delta}}}\cdot\bm{{\Delta}}^{\mathrm{\scriptscriptstyle
T}}$
$\displaystyle\left(\left[1+\tfrac{k}{2}\right]{\mathbf{F}}^{\prime}_{\mathrm{s}}-\tfrac{k}{2m}{\mathbf{F}}_{\mathrm{f}}\right)\bm{{\Delta}}$
$\displaystyle=\sqrt{\frac{(2\pi)^{k}}{|m\mathbf{F}^{\prime}_{\mathrm{s}}|}}\sum_{i=1}^{k}\frac{D^{-1}_{\mathrm{s},ij}\left(\left[1+\tfrac{k}{2}\right]F^{\prime}_{\mathrm{s}}-\tfrac{k}{2m}F_{\mathrm{f}}\right)_{jl}D_{\mathrm{s},li}}{D^{-1}_{\mathrm{s},ij}mF^{\prime}_{\mathrm{s},jl}D_{\mathrm{s},li}}$
$\displaystyle=km^{-\frac{k+4}{2}}\sqrt{\frac{(2\pi)^{k}}{|\mathbf{F}^{\prime}_{\mathrm{s}}|}}\left(\left[1+\tfrac{k}{2}\right]m-\tfrac{1}{2}\sum_{i=1}^{k}\frac{D^{-1}_{\mathrm{s},ij}F_{\mathrm{f},jl}D_{\mathrm{s},li}}{D^{-1}_{\mathrm{s},ij}F^{\prime}_{\mathrm{s},jl}D_{\mathrm{s},li}}\right)$
(86)

for diagonal
$\mathbf{D}^{-1}_{\mathrm{s}}\mathbf{F}^{\prime}_{\mathrm{s}}\mathbf{D}_{\mathrm{s}}$.

For the final integral in (84) we again diagonalise the matrix in the exponent
to make the integration simple. Similarly to how we transformed earlier, with
a translation and a rotation, let us take
$\theta_{\mu}\to\check{\theta}_{\mu}$ so that
${\mathbf{F}}^{\prime}_{\mathrm{s}}\to\check{\mathbf{F}}^{\prime}_{\mathrm{s}}$
where $\check{\mathbf{F}}^{\prime}_{\mathrm{s}}$ is diagonal, and such that
$\bm{\check{\Delta}}=\bm{\check{\theta}}$. Then we can rewrite the integral as

$\displaystyle I$
$\displaystyle=\int_{{\Theta}}\mathrm{d}^{k}{\theta}\,e^{-\frac{1}{2}m\bm{{\Delta}}^{\mathrm{\scriptscriptstyle
T}}{\mathbf{F}}^{\prime}_{\mathrm{s}}\bm{{\Delta}}}\left\\{m(\bm{{\Delta}}^{\mathrm{\scriptscriptstyle
T}}{\mathbf{F}}^{\prime}_{\mathrm{s}}\bm{{\Delta}})^{2}-\bm{{\Delta}}^{\mathrm{\scriptscriptstyle
T}}{\mathbf{F}}^{\prime}_{\mathrm{s}}\bm{{\Delta}}\cdot\bm{{\Delta}}^{\mathrm{\scriptscriptstyle
T}}{\mathbf{F}}_{\mathrm{f}}\bm{{\Delta}}\right\\}$
$\displaystyle=\int_{\check{\Theta}}\mathrm{d}^{k}\check{\theta}\,e^{-\frac{1}{2}m\bm{\check{\theta}}^{\mathrm{\scriptscriptstyle
T}}\check{\mathbf{F}}^{\prime}_{\mathrm{s}}\bm{\check{\theta}}}\left\\{m(\bm{\check{\theta}}^{\mathrm{\scriptscriptstyle
T}}\check{\mathbf{F}}^{\prime}_{\mathrm{s}}\bm{\check{\theta}})^{2}-\bm{\check{\theta}}^{\mathrm{\scriptscriptstyle
T}}\check{\mathbf{F}}^{\prime}_{\mathrm{s}}\bm{\check{\theta}}\cdot\bm{\check{\theta}}^{\mathrm{\scriptscriptstyle
T}}\check{\mathbf{F}}_{\mathrm{f}}\bm{\check{\theta}}\right\\}$
$\displaystyle=\int_{\check{\Theta}}\mathrm{d}^{k}\check{\theta}\,e^{-\frac{1}{2}m\bm{\check{\theta}}^{\mathrm{\scriptscriptstyle
T}}\check{\mathbf{F}}^{\prime}_{\mathrm{s}}\bm{\check{\theta}}}\left\\{m\sum_{i=1}^{k}\check{F}^{\prime}_{\mathrm{s},ii}\check{\theta}_{i}^{2}\sum_{j=1}^{k}\check{F}^{\prime}_{\mathrm{s},jj}{\check{\theta}}_{j}^{2}-\sum_{i=1}^{k}\check{F}^{\prime}_{\mathrm{s},ii}{\check{\theta}}_{i}^{2}\sum_{j=1}^{k}\check{F}_{\mathrm{f},jj}{\check{\theta}}_{j}^{2}\right\\}$
$\displaystyle=\int_{\check{\Theta}}\mathrm{d}^{k}\check{\theta}\,e^{-\frac{1}{2}m\bm{\check{\theta}}^{\mathrm{\scriptscriptstyle
T}}\check{\mathbf{F}}^{\prime}_{\mathrm{s}}\bm{\check{\theta}}}\sum_{i=1}^{k}\check{F}^{\prime}_{\mathrm{s},ii}\check{\theta}_{i}^{2}\sum_{j=1}^{k}(m\check{F}^{\prime}_{\mathrm{s},jj}-\check{F}_{\mathrm{f},jj}){\check{\theta}}_{j}^{2}$
$\displaystyle=\sum_{i=1}^{k}\check{F}^{\prime}_{\mathrm{s},ii}\sum_{j=1}^{k}(m\check{F}^{\prime}_{\mathrm{s},jj}-\check{F}_{\mathrm{f},jj})\int_{\check{\Theta}}\mathrm{d}^{k}\check{\theta}\,\prod_{l=1}^{k}e^{-\frac{1}{2}m\check{F}^{\prime}_{\mathrm{s},ll}\check{\theta}_{l}^{2}}\check{\theta}_{i}^{2}{\check{\theta}}_{j}^{2}$
$\displaystyle=\sum_{i=1}^{k}(m(\check{F}^{\prime}_{\mathrm{s},ii})^{2}-\check{F}^{\prime}_{\mathrm{s},ii}\check{F}_{\mathrm{f},ii})\int_{-\infty}^{\infty}\mathrm{d}\check{\theta}_{i}\,e^{-\frac{1}{2}m\check{F}^{\prime}_{\mathrm{s},ii}\check{\theta}_{i}^{2}}\cdot\check{\theta}_{i}^{4}\;\prod_{j\neq
i}\int_{-\infty}^{\infty}\mathrm{d}^{k}\check{\theta}\,e^{-\frac{1}{2}m\check{F}^{\prime}_{\mathrm{s},jj}\check{\theta}_{j}^{2}}$
$\displaystyle\hskip
20.0pt+\sum_{i=1}^{k}\check{F}^{\prime}_{\mathrm{s},ii}\sum_{j\neq
i}(m\check{F}^{\prime}_{\mathrm{s},jj}-\check{F}_{\mathrm{f},jj})\prod_{l\in\\{i,j\\}}\int_{-\infty}^{\infty}\mathrm{d}\check{\theta}_{l}\,e^{-\frac{1}{2}m\check{F}^{\prime}_{\mathrm{s},ll}\check{\theta}_{l}^{2}}\cdot\check{\theta}_{l}^{2}\prod_{h\not\in\\{i,j\\}}\int_{-\infty}^{\infty}\mathrm{d}\check{\theta}_{h}\,e^{-\frac{1}{2}m\check{F}^{\prime}_{\mathrm{s},hh}\check{\theta}_{h}^{2}}\,,$

where in the third line terms relating to the off-diagonal elements of
$\check{\mathbf{F}}_{\mathrm{f}}$ drop out since integrands of those terms are
odd functions. In the last step, we split the sum into an $i=j$ part (first
line) and a $i\neq j$ part (second line) for the
$\check{\theta}_{i}^{2}{\check{\theta}}_{j}^{2}$ factor in the integrand.
Along with (78) we require the following well-known identity

$\int_{-\infty}^{\infty}\mathrm{d}x\;x^{4}e^{-ax^{2}}=\frac{3}{4}\sqrt{\frac{\pi}{a^{5}}}\,.$
(87)

Then we have that

$\displaystyle I$
$\displaystyle=\sum_{i=1}^{k}(m(\check{F}^{\prime}_{\mathrm{s},ii})^{2}-\check{F}^{\prime}_{\mathrm{s},ii}\check{F}_{\mathrm{f},ii})\frac{3}{4}\sqrt{\frac{\pi}{(\frac{1}{2}m\check{F}^{\prime}_{\mathrm{s},ii})^{5}}}\;\prod_{j\neq
i}\sqrt{\frac{\pi}{\frac{1}{2}m\check{F}^{\prime}_{\mathrm{s},jj}}}$
$\displaystyle\hskip
20.0pt+\sum_{i=1}^{k}\check{F}^{\prime}_{\mathrm{s},ii}\sum_{j\neq
i}(m\check{F}^{\prime}_{\mathrm{s},jj}-\check{F}_{\mathrm{f},jj})\prod_{l\in\\{i,j\\}}\sqrt{\frac{2\pi}{(m\check{F}^{\prime}_{\mathrm{s},ll})^{3}}}\prod_{h\not\in\\{i,j\\}}\sqrt{\frac{\pi}{\frac{1}{2}m\check{F}^{\prime}_{\mathrm{s},hh}}}$
$\displaystyle=3m^{-\frac{k+4}{2}}\sum_{i=1}^{k}(m(\check{F}^{\prime}_{\mathrm{s},ii})^{2}-\check{F}^{\prime}_{\mathrm{s},ii}\check{F}_{\mathrm{f},ii})(\check{F}^{\prime}_{\mathrm{s},ii})^{-2}\;\prod_{j=1}^{k}\sqrt{\frac{2\pi}{\check{F}^{\prime}_{\mathrm{s},jj}}}$
$\displaystyle\hskip
20.0pt+m^{-\frac{k+4}{2}}\sum_{i=1}^{k}\check{F}^{\prime}_{\mathrm{s},ii}\sum_{j\neq
i}(m\check{F}^{\prime}_{\mathrm{s},jj}-\check{F}_{\mathrm{f},jj})\prod_{l\in\\{i,j\\}}(\check{F}^{\prime}_{\mathrm{s},ll})^{-1}\prod_{h=1}^{k}\sqrt{\frac{2\pi}{\check{F}^{\prime}_{\mathrm{s},hh}}}$
$\displaystyle=3m^{-\frac{k+4}{2}}\sqrt{\frac{(2\pi)^{k}}{|\check{\mathbf{F}}^{\prime}_{\mathrm{s}}|}}\sum_{i=1}^{k}\left(m-\frac{\check{F}_{\mathrm{f},ii}}{\check{F}^{\prime}_{\mathrm{s},ii}}\right)+m^{-\frac{k+4}{2}}\sqrt{\frac{(2\pi)^{k}}{|\check{\mathbf{F}}^{\prime}_{\mathrm{s}}|}}\sum_{i=1}^{k}\sum_{j\neq
i}\left(m-\frac{\check{F}_{\mathrm{f},jj}}{\check{F}^{\prime}_{\mathrm{s},jj}}\right)$
$\displaystyle=m^{-\frac{k+4}{2}}\sqrt{\frac{(2\pi)^{k}}{|\check{\mathbf{F}}^{\prime}_{\mathrm{s}}|}}\left[2\sum_{i=1}^{k}\left(m-\frac{\check{F}_{\mathrm{f},ii}}{\check{F}^{\prime}_{\mathrm{s},ii}}\right)+k\sum_{j=1}^{k}\left(m-\frac{\check{F}_{\mathrm{f},jj}}{\check{F}^{\prime}_{\mathrm{s},jj}}\right)\right]$
$\displaystyle=m^{-\frac{k+4}{2}}\sqrt{\frac{(2\pi)^{k}}{|\check{\mathbf{F}}^{\prime}_{\mathrm{s}}|}}\left[(2+k)\left(km-\sum_{i=1}^{k}\frac{\check{F}_{\mathrm{f},ii}}{\check{F}^{\prime}_{\mathrm{s},ii}}\right)\right]\,.$
(88)

Substituting (85), (86), and (88) into (84), we have

$\displaystyle\frac{\mathrm{d}}{\mathrm{d}m}D_{\mathrm{J}}(p_{\mathrm{f}}\,||\,p_{\mathrm{s}})$
$\displaystyle=CC^{\prime}\sqrt{\frac{(2\pi)^{k}}{|\mathbf{F}_{\mathrm{f}}|}}\sum_{i=1}^{k}\frac{D^{-1}_{\mathrm{f},ij}F^{\prime}_{\mathrm{s},jl}D_{\mathrm{f},li}}{D^{-1}_{\mathrm{f},ij}F_{\mathrm{f},jl}D_{\mathrm{f},li}}$
$\displaystyle\hskip 45.0pt-
Cm^{k/2}km^{-\frac{k+4}{2}}\sqrt{\frac{(2\pi)^{k}}{|\mathbf{F}^{\prime}_{\mathrm{s}}|}}\left(\left[1+\tfrac{k}{2}\right]m-\tfrac{1}{2}\sum_{i=1}^{k}\frac{D^{-1}_{\mathrm{s},ij}F_{\mathrm{f},jl}D_{\mathrm{s},li}}{D^{-1}_{\mathrm{s},ij}F^{\prime}_{\mathrm{s},jl}D_{\mathrm{s},li}}\right)$
$\displaystyle\hskip
45.0pt+\tfrac{1}{2}Cm^{k/2}m^{-\frac{k+4}{2}}\sqrt{\frac{(2\pi)^{k}}{|\check{\mathbf{F}}^{\prime}_{\mathrm{s}}|}}\left[(2+k)\left(km-\sum_{i=1}^{k}\frac{\check{F}_{\mathrm{f},ii}}{\check{F}^{\prime}_{\mathrm{s},ii}}\right)\right]$
$\displaystyle=C\sqrt{\frac{(2\pi)^{k}}{|\mathbf{F}^{\prime}_{\mathrm{s}}|}}\left\\{\sum_{i=1}^{k}\frac{D^{-1}_{\mathrm{f},ij}F^{\prime}_{\mathrm{s},jl}D_{\mathrm{f},li}}{D^{-1}_{\mathrm{f},ij}F_{\mathrm{f},jl}D_{\mathrm{f},li}}-km^{-2}\left(\left[1+\tfrac{k}{2}\right]m-\tfrac{1}{2}\sum_{i=1}^{k}\frac{D^{-1}_{\mathrm{s},ij}F_{\mathrm{f},jl}D_{\mathrm{s},li}}{D^{-1}_{\mathrm{s},ij}F^{\prime}_{\mathrm{s},jl}D_{\mathrm{s},li}}\right)\right.$
$\displaystyle\hskip
80.0pt\left.+m^{-2}\left[(1+\tfrac{k}{2})\left(km-\sum_{i=1}^{k}\frac{\check{F}_{\mathrm{f},ii}}{\check{F}^{\prime}_{\mathrm{s},ii}}\right)\right]\right\\}$
$\displaystyle=\tfrac{1}{2}\left[\sum_{i=1}^{k}\frac{D^{-1}_{\mathrm{f},ij}F^{\prime}_{\mathrm{s},jl}D_{\mathrm{f},li}}{D^{-1}_{\mathrm{f},ij}F_{\mathrm{f},jl}D_{\mathrm{f},li}}-m^{-2}\sum_{i=1}^{k}\frac{D^{-1}_{\mathrm{s},ij}F_{\mathrm{f},jl}D_{\mathrm{s},li}}{D^{-1}_{\mathrm{s},ij}F^{\prime}_{\mathrm{s},jl}D_{\mathrm{s},li}}\right]\,.$

We want the $m=m_{\mathrm{J}}$ such that this derivative vanishes, so finally
we see that the factor minimising the Jeffreys’ divergence is given by:

$m_{\mathrm{J}}=\sqrt{\sum_{i=1}^{k}\frac{D^{-1}_{\mathrm{s},ij}F_{\mathrm{f},jl}D_{\mathrm{s},li}}{D^{-1}_{\mathrm{s},ij}F^{\prime}_{\mathrm{s},jl}D_{\mathrm{s},li}}\Bigg{/}\sum_{i=1}^{k}\frac{D^{-1}_{\mathrm{f},ij}F^{\prime}_{\mathrm{s},jl}D_{\mathrm{f},li}}{D^{-1}_{\mathrm{f},ij}F_{\mathrm{f},jl}D_{\mathrm{f},li}}}\;,$
(89)

where recall that $\mathbf{D}_{\mathrm{s}}$ diagonalises
$\mathbf{F}^{\prime}_{\mathrm{s}}$ and $\mathbf{D}_{\mathrm{f}}$ diagonalises
$\mathbf{F}_{\mathrm{f}}$. This is similar to $m_{\mathrm{KL}}$ in form but
clearly balanced between the distributions.

##  Appendix C Numerical Marginalisation of Phase in Time-Domain

For the latter problem, we can remove uninteresting (or _nuisance_) parameters
from the parameter space by marginalisation. The coalescence phase is one such
parameter. The phase marginalised probability is given by

$p^{\prime}(\bm{\theta}^{\prime}\,|\,\textbf{d})=\int_{0}^{2\pi}\mathrm{d}\phi_{\mathrm{c}}\,p(\bm{\theta}\,|\,\textbf{d})=\int_{0}^{2\pi}\mathrm{d}\phi_{\mathrm{c}}\,\frac{p(\bm{\theta})\,p(\textbf{d}\,|\,\bm{\theta})}{p(\mathbf{d})}\,,$
(90)

where $\bm{\theta}^{\prime}$ is the tuple of parameters excluding the
coalescence phase, $\phi_{\mathrm{c}}$. If the prior on $\phi_{\mathrm{c}}$ is
$(2\pi)^{-1}$, then the prior on $\bm{\theta}$ can be written

$p(\bm{\theta})=\frac{p^{\prime}(\bm{\theta}^{\prime})}{2\pi}\,,$

where $p^{\prime}(\bm{\theta}^{\prime})$ is the prior on
$\bm{\theta}^{\prime}$. Thus

$p^{\prime}(\bm{\theta}^{\prime}\,|\,\textbf{d})=\frac{p^{\prime}(\bm{\theta}^{\prime})}{p(\mathbf{d})}\int_{0}^{2\pi}\mathrm{d}\phi_{\mathrm{c}}\,\frac{p(\textbf{d}\,|\,\bm{\theta})}{2\pi}\,.$
(91)

Therefore, one can obtain the marginal probability simply by substituting a
marginalised likelihood. Now since we can write the waveform in the form
$h(t)=A(t)\cos(\phi(t)+\phi_{\mathrm{c}})$, then from equations (2) and (3),
we have that

$\displaystyle p(\textbf{d}\,|\,\bm{\theta})$
$\displaystyle=\left[(2\pi)^{N}\det(\mathbf{C})\right]^{-1/2}\exp\left[-\tfrac{1}{2}\;\mathbf{\Big{(}d-h(\bm{\theta})\Big{)}}^{\mathrm{\scriptscriptstyle
T}}\mathbf{C}^{-1}\mathbf{\Big{(}d-h(\bm{\theta})\Big{)}}\right]$
$\displaystyle=\left[(2\pi)^{N}\det(\mathbf{C})\right]^{-1/2}\exp\left[-\tfrac{1}{2}\;\mathbf{\Big{(}d-\mathbf{A}\cos(\bm{\phi}^{\prime}+\phi_{\mathrm{c}})\Big{)}}^{\mathrm{\scriptscriptstyle
T}}\mathbf{C}^{-1}\mathbf{\Big{(}d-\mathbf{A}\cos(\bm{\phi}^{\prime}+\phi_{\mathrm{c}})}\right]\,,$

where $\bm{\phi}^{\prime}=\bm{\phi}^{\prime}(\bm{\theta}^{\prime})$. By
trigonometric identities, we have

$\displaystyle\mathbf{\Big{(}d-h(\bm{\theta})\Big{)}}^{\mathrm{\scriptscriptstyle
T}}\mathbf{C}^{-1}\mathbf{\Big{(}d-h(\bm{\theta})\Big{)}}$
$\displaystyle=\mathbf{\Big{(}d-\mathbf{A}\cos(\bm{\phi}^{\prime})\cos(\phi_{\mathrm{c}})+\mathbf{A}\sin(\bm{\phi}^{\prime})\sin(\phi_{\mathrm{c}})\Big{)}}^{\mathrm{\scriptscriptstyle
T}}$ $\displaystyle\hskip
20.0pt\times\mathbf{C}^{-1}\mathbf{\Big{(}d-\mathbf{A}\cos(\bm{\phi}^{\prime})\cos(\phi_{\mathrm{c}})+\mathbf{A}\sin(\bm{\phi}^{\prime})\sin(\phi_{\mathrm{c}})\Big{)}}$
$\displaystyle=A\cos^{2}(\phi_{\mathrm{c}})+B\cos(\phi_{\mathrm{c}})\sin(\phi_{\mathrm{c}})+C\cos(\phi_{\mathrm{c}})$
$\displaystyle\hskip
20.0pt+D\sin^{2}(\phi_{\mathrm{c}})+E\sin(\phi_{\mathrm{c}})+F\,,$

where

$\begin{aligned}
A&=\Big{(}\mathbf{A}\cos(\bm{\phi}^{\prime})\Big{)}^{\mathrm{\scriptscriptstyle
T}}\mathbf{C}^{-1}\mathbf{A}\cos(\bm{\phi}^{\prime})\\\
B&=-2\Big{(}\mathbf{A}\cos(\bm{\phi}^{\prime})\Big{)}^{\mathrm{\scriptscriptstyle
T}}\mathbf{C}^{-1}\mathbf{\mathbf{A}\sin(\bm{\phi}^{\prime})}\\\
C&=-2\mathbf{d}^{\mathrm{\scriptscriptstyle
T}}\mathbf{C}^{-1}\mathbf{A}\cos(\bm{\phi}^{\prime})\end{aligned}\hskip
60.0pt\begin{aligned}
D&=\mathbf{\Big{(}\mathbf{A}\sin(\bm{\phi}^{\prime})\Big{)}}^{\mathrm{\scriptscriptstyle
T}}\mathbf{C}^{-1}\mathbf{A}\sin(\bm{\phi}^{\prime})\\\
E&=2\mathbf{d}^{\mathrm{\scriptscriptstyle
T}}\mathbf{C}^{-1}\mathbf{A}\sin(\bm{\phi}^{\prime})\\\
F&=\mathbf{d}^{\mathrm{\scriptscriptstyle
T}}\mathbf{C}^{-1}\mathbf{d}\,.\end{aligned}$

Then the marginalised likelihood is

$\int_{0}^{2\pi}\mathrm{d}\phi_{\mathrm{c}}\,\frac{p(\textbf{d}\,|\,\bm{\theta})}{2\pi}=\int_{0}^{2\pi}\mathrm{d}\phi_{\mathrm{c}}\,\frac{e^{-\frac{1}{2}\;\left(A\cos^{2}(\phi_{\mathrm{c}})+B\cos(\phi_{\mathrm{c}})\sin(\phi_{\mathrm{c}})+C\cos(\phi_{\mathrm{c}})+D\sin^{2}(\phi_{\mathrm{c}})+E\sin(\phi_{\mathrm{c}})+F\right)}}{2\pi\left[(2\pi)^{N}\det(\mathbf{C})\right]^{1/2}}\,.$
(92)

There does not appear to be a straightforward method to obtain a closed form
expression of this integral, but a numerical integration can be achieved
easily and performed very cheaply, as we shall show. The computationally
expensive part is in evaluating $\bm{\phi}^{\prime}$. There is a moderately
expensive operation in computing inner products in the definition of the
constants (given some $\bm{\theta}^{\prime}$) $A$ to $F$. However, notice that
all the constants are easily computed by various inner products of the vectors
$\mathbf{C}^{(-1/2)}\mathbf{d}$,
$\mathbf{C}^{(-1/2)}\mathbf{\mathbf{A}\sin(\bm{\phi}^{\prime})}$, and
$\mathbf{C}^{(-1/2)}\mathbf{A}\cos(\bm{\phi}^{\prime})$, so only these three
matrix operations are required, and the downsampling significantly reduces the
cost of these matrix operations. For many likelihood evaluations, store the
sines and cosines (and their squares) of a range of $\phi_{\mathrm{c}}$ values
in memory so as not to reevaluate them on each likelihood function call.

One can now approximate the integral by a discrete sum. We tested numerically
‘pre-marginalised’ posteriors against ‘post-marginalised’ posteriors (that is,
we acquired the full posterior from the non-marginalised model, and
marginalised this) and found that the number of terms in the sum must be
decreased to around 20 before the approximation begins to fail. The likelihood
evaluation time only begins to increase noticeably (by a few percent) when the
number of terms in the sum reaches around $10^{4}$. We used $10^{3}$ terms,
giving an accurate marginalised likelihood with negligible likelihood
evaluation time increase. The PE convergence time, however, with the phase
parameter removed, is significantly decreased by a factor of around three.

##  Appendix D Downsampling visualisation with corner plots

Figure 4 shows convergence of (representatives of) posteriors as the number of
time domain datapoints is increased.

Figure 4: Downsampled posteriors of the fiducial system with $\Delta
f/f_{\mathrm{max}}=0.9$, $N_{\mathrm{f}}=10^{6}$ (see Section 4.4.1 and Table
4.4.2). The posteriors converge as the number of samples is increased.
Selected posteriors have been included to prevent a congested plot. There is
almost no discernible difference between corner plots with $N_{\mathrm{s}}\geq
256$. As $N_{\mathrm{s}}$ decreases, we can clearly see the posterior ‘breaks
down’, the few samples remaining being unable to constrain the parameters in
the same way as the higher $N_{\mathrm{s}}$ posteriors.


# Jacob’s ladders and new synergetic formula generating infinite set of
$\zeta$-cooperative three-parametric invariants

Jan Moser  Department of Mathematical Analysis and Numerical Mathematics,
Comenius University, Mlynska Dolina M105, 842 48 Bratislava, SLOVAKIA
<EMAIL_ADDRESS>
###### Abstract.

In this paper we obtain new $\zeta$-synergetic formula namely an exact
secondary complete hybrid formula. This one is generated by some set of
trigonometric and power functions together with the square of module of the
Riemann’s zeta-function on the critical line. By means of this new formula, we
define a two-parametric continuum set of a three-parametric invariants.

###### Key words and phrases:

Riemann zeta-function

##  1\. Introduction

### 1.1.

This paper is based also on a set of new notions and methods we have
introduced into the theory of the Riemann’s zeta-function in our series of 47
papers concerning Jacob’s ladders. These can be found in arXiv[math.CA]
starting with the paper [1], (2009).

Here we use especially the following notions: Jacob’s ladder,
$\zeta$-disconnected sets generated by the Jacob’s ladder, algorithm for
generating the $\zeta$-factorization formulas, complete hybrid formula (exact
and asymptotic). Short survey of these notions can be found in our papers [5],
[8].

Next, we have introduced also the following notions (see [8]): secondary
crossbreeding, secondary complete hybrid formula and secondary $\zeta$-family
of basic elements.

### 1.2.

In this paper we obtain some new results in the direction mentioned in 1.1.
For example, we obtain as a simple variant of complete result, the following
one: the set

$\begin{split}&\\{\sin^{2}t,\cos^{2}t,(t-\pi L)^{1/3},(t-\pi L)^{1/5}\\},\\\
&t\in[\pi L,\pi L+U],\ U\in(0,\pi/2),\ L\in\mathbb{N}\end{split}$

of elementary functions generates the following exact secondary complete
hybrid formula

(1.1) $\begin{split}&\left(\frac{\alpha_{0}^{3,k_{2}}-\pi
L}{\alpha_{0}^{4,k_{2}}-\pi
L}\right)^{1/2}\left\\{\prod_{r=1}^{k_{2}}\frac{|\tilde{Z}(\alpha_{r}^{2,k_{2}})|^{2}|\tilde{Z}(\alpha_{r}^{3,k_{2}})|^{3}}{|\tilde{Z}(\alpha_{r}^{4,k_{2}})|^{5}}\right\\}\cos^{2}(\alpha_{0}^{2,k_{2}})+\\\
&+\left(\frac{\alpha_{0}^{3,k_{1}}-\pi L}{\alpha_{0}^{4,k_{1}}-\pi
L}\right)^{1/2}\left\\{\prod_{r=1}^{k_{1}}\frac{|\tilde{Z}(\alpha_{r}^{1,k_{1}})|^{2}|\tilde{Z}(\alpha_{r}^{3,k_{1}})|^{3}}{|\tilde{Z}(\alpha_{r}^{4,k_{1}})|^{5}}\right\\}\cos^{2}(\alpha_{0}^{1,k_{1}})=\\\
&=\frac{81}{250}\sqrt{10},\ \forall\\-L\geq L_{0}>0,\end{split}$

where

(1.2) $1\leq k_{1},k_{2},k_{3},k_{4}\leq k_{0},$

and we fix arbitrary $k_{0}\in\mathbb{N}$ and $L_{0}\in\mathbb{N}$ being
sufficiently big,

(1.3)
$\tilde{Z}^{2}(t)=\frac{|\zeta\left(\frac{1}{2}+it\right)|^{2}}{\omega(t)},\
\omega(t)=\left\\{1+\mathcal{O}\left(\frac{\ln\ln t}{\ln t}\right)\right\\}\ln
t,$

(see [2], (6.1), (6.7), (7.7), (7.8), (9.1)), next

(1.4)
$\begin{split}&\alpha_{0}^{1,k_{1}},\alpha_{0}^{3,k_{1}},\alpha_{0}^{4,k_{1}},\alpha_{0}^{2,k_{2}},\alpha_{0}^{3,k_{2}},\alpha_{0}^{4,k_{2}}\in(\pi
L,\pi L+U),\\\
&\alpha_{r}^{1,k_{1}},\alpha_{r}^{3,k_{1}},\alpha_{r}^{4,k_{1}}\in(\overset{r}{\wideparen{\pi
L}},\overset{r}{\wideparen{\pi L+U}}),\ r=1,\dots,k_{1},\\\
&\alpha_{r}^{2,k_{2}},\alpha_{r}^{3,k_{2}},\alpha_{r}^{4,k_{2}}\in(\overset{r}{\wideparen{\pi
L}},\overset{r}{\wideparen{\pi L+U}}),\ r=1,\dots,k_{2},\end{split}$

and the segment

$[\overset{r}{\wideparen{\pi L}},\overset{r}{\wideparen{\pi L+U}}]$

is the $r$-th reverse iteration by means of the Jacob’s ladder (see [3]) of
the basic segment

$[\pi L,\pi L+U]=[\overset{0}{\wideparen{\pi L}},\overset{0}{\wideparen{\pi
L+U}}].$

### 1.3.

Let us notice explicitly that the following holds true: consecutive components
of the main $\zeta$-disconnected set

(1.5) $\Delta(\pi
L,U,\bar{k})=\bigcup_{r=0}^{\bar{k}}[\overset{r}{\wideparen{\pi
L}},\overset{r}{\wideparen{\pi L+U}}]$

where

$\bar{k}=\max\\{k_{1},k_{2}\\},\ \bar{k}\leq k_{0}$

are separated each from other by gigantic distances $\rho$:

(1.6) $\begin{split}&\rho\left\\{[\overset{r}{\wideparen{\pi
L}},\overset{r}{\wideparen{\pi L+U}}];[\overset{r+1}{\wideparen{\pi
L}},\overset{r+1}{\wideparen{\pi L+U}}]\right\\}\sim(1-c)\pi(\pi L)\sim\\\
&\sim\pi(1-c)\frac{L}{\ln L}\to\infty\ \mbox{as}\ L\to\infty,\
r=0,1,\dots\bar{k}-1,\end{split}$

where $c$ is the Euler’s constant and $\pi(x)$ stands for the prime-counting
function.

### 1.4.

Next (see (1.1), (1.3)) the following asymptotic secondary hybrid formula

(1.7) $\begin{split}&\left(\frac{\alpha_{0}^{3,k_{2}}-\pi
L}{\alpha_{0}^{4,k_{2}}-\pi
L}\right)^{1/2}\left\\{\prod_{r=1}^{k_{2}}\frac{|\zeta(\frac{1}{2}+i\alpha_{r}^{2,k_{2}})|^{2}|\zeta(\frac{1}{2}+i\alpha_{r}^{3,k_{3}})|^{3}}{|\zeta(\frac{1}{2}+i\alpha_{r}^{4,k_{4}})|^{5}}\right\\}\cos^{2}(\alpha_{0}^{2,k_{2}})+\\\
&+\left(\frac{\alpha_{0}^{3,k_{1}}-\pi L}{\alpha_{0}^{4,k_{1}}-\pi
L}\right)^{1/2}\left\\{\prod_{r=1}^{k_{1}}\frac{|\zeta(\frac{1}{2}+i\alpha_{r}^{1,k_{1}})|^{2}|\zeta(\frac{1}{2}+i\alpha_{r}^{3,k_{1}})|^{3}}{|\zeta(\frac{1}{2}+i\alpha_{r}^{4,k_{1}})|^{5}}\right\\}\sin^{2}(\alpha_{0}^{1,k_{1}})\sim\\\
&\sim\frac{81}{250}\sqrt{10},\\\ &1\leq k_{1},k_{2}\leq k_{0},\
L\to\infty\end{split}$

holds true.

### 1.5.

Now, let us notice something about meaning of the the formula (1.1) as well as
the meaning of the formula (1.7) introduced in the paper [8]. Both mentioned
formulas are synergetic ones follow as result of interactions controlled by
the Jacob’s ladder $\varphi_{1}(t)$ between following continuum sets

$\left\\{\left|\zeta\left(\frac{1}{2}+it\right)\right|^{2}\right\\},\\{\sin^{2}t\\},\\{\cos^{2}t\\},\\{(t-\pi
L)^{1/3}\\},\\{(t-\pi L)^{1/5}\\},\ t\geq L_{0}.$

Let us call these interactions as the $\zeta$-chemical reaction as an analogue
to the classical Belousov-Zhabotinski chemical oscillations.

######  Remark 1.

The result of the above mentioned $\zeta$-chemical reaction (the
$\zeta$-chemical compound) is the synergetic formula (see (1.1) and (1.7))
that contains a finite set of the arguments (see (1.4) - for every fixed and
admissible $U,L,k_{1},k_{2}$, of course) that are distributed within
disconnected set (1.5) which components are separated by gigantic distances
according to (1.6).

### 1.6.

Let us notice explicitly about another result that is contained in the formula
(1.1).

######  Remark 2.

Namely, an invariant relatively to arbitrary sampling is defined by the left-
hand side of the formula (1.1):

  * (a)

of the continuous parameter $U\in(0,\pi/2)$

  * (b)

of the discrete parameter $L\geq L_{0}$

  * (c)

of the subset

$\\{k_{1},k_{2}\\}\subset\\{1,\dots,k_{0}\\}$

######  Remark 3.

It is clear that the synergetic (cooperative) formula (1.1) as well as (1.7)
represents completely new type of result in the theory of the Riemann’s zeta-
function and, simultaneously, in the theory of real continuous functions.

##  2\. The first exact complete hybrid formula

### 2.1.

By making use of our algorithm for generating the $\zeta$-factorization
formulas (see [5], (3.1) – (3.11), comp. [4]) we obtain the following result
(comp. [6], (2.1) – (2.9)):

######  Lemma 1.

For the function

(2.1) $f_{1}(t)=\sin^{2}t\in\tilde{C}_{0}[\pi L,\pi L+U],\ U\in(0,\pi/2)$

there are vector-valued functions

(2.2)
$(\alpha_{0}^{1,k_{1}},\alpha_{1}^{1,k_{1}},\dots,\alpha_{k_{1}}^{1,k_{1}},\beta_{1}^{k_{1}},\dots,\beta_{k_{1}}^{k_{1}}),\
1\leq k_{1}\leq k_{0}$

such that the following exact $\zeta$-factorization formula

(2.3)
$\prod_{r=1}^{k_{1}}\frac{\tilde{Z}^{2}(\alpha_{r}^{1,k_{1}})}{\tilde{Z}^{2}(\beta_{r}^{k_{1}})}=\frac{1}{2}\left(1-\frac{\sin
2U}{2U}\right)\frac{1}{\sin^{2}(\alpha_{0}^{1,k_{1}})},\ \forall\\-L\geq
L_{0}>0$

holds true, (comp. [5], (3.7)), where

(2.4) $\begin{split}&\alpha_{r}^{1,k_{1}}=\alpha_{r}(U,L,k_{1};f_{1}),\
r=0,1,\dots,k_{1},\\\ &\beta_{r}^{k_{1}}=\beta_{r}(U,L,k_{1}),\
r=1,\dots,k_{1},\\\ &\alpha_{0}^{1,k_{1}}\in(\pi L,\pi L+U),\
\alpha_{r}^{1,k_{1}},\beta_{r}^{k_{1}}\in(\overset{r}{\wideparen{\pi
L}},\overset{r}{\wideparen{\pi L+U}}),\ r=1,\dots,k_{1}.\end{split}$

######  Lemma 2.

For the function

(2.5) $f_{2}(t)=\cos^{2}t\in\tilde{C}_{0}[\pi L,\pi L+U],\ U\in(0,\pi/2)$

there are vector-valued functions

(2.6)
$(\alpha_{0}^{k_{2}},\alpha_{1}^{2,k_{2}},\dots,\alpha_{k_{2}}^{2,k_{2}},\beta_{1}^{k_{2}},\dots,\beta_{k_{2}}^{k_{2}}),\
1\leq k_{2}\leq k_{0}$

such that the following exact $\zeta$-factorization formula

(2.7)
$\prod_{r=1}^{k_{2}}\frac{\tilde{Z}^{2}(\alpha_{r}^{2,k_{2}})}{\tilde{Z}^{2}(\beta_{r}^{k_{2}})}=\frac{1}{2}\left(1+\frac{\sin
2U}{2U}\right)\frac{1}{\cos^{2}(\alpha_{0}^{2,k_{2}})},\ \forall\\-L\geq
L_{0}>0$

holds true, where

(2.8) $\begin{split}&\alpha_{r}^{2,k_{2}}=\alpha_{r}(U,L,k_{2};f_{2}),\
r=0,1,\dots,k_{2},\\\ &\beta_{r}^{k_{2}}=\beta_{r}(U,L,k_{2}),\
r=1,\dots,k_{2},\\\ &\alpha_{0}^{2,k_{2}}\in(\pi L,\pi L+U),\
\alpha_{r}^{2,k_{2}},\beta_{r}^{k_{2}}\in(\overset{r}{\wideparen{\pi
L}},\overset{r}{\wideparen{\pi L+U}}),\ r=1,\dots,k_{2}.\end{split}$

### 2.2.

Now, we obtain by the crossbreeding (see [6], (2.9)) between
$\zeta$-factorization formulae (2.3) and (2.7) the following

######  Exact Complete Hybrid Formula 1.

(2.9)
$\begin{split}&\left\\{\prod_{r=1}^{k_{2}}\frac{\tilde{Z}^{2}(\alpha_{r}^{2,k_{2}})}{\tilde{Z}^{2}(\beta_{r}^{k_{2}})}\right\\}\cos^{2}(\alpha_{0}^{2,k_{2}})+\left\\{\prod_{r=1}^{k_{1}}\frac{\tilde{Z}^{2}(\alpha_{r}^{1,k_{1}})}{\tilde{Z}^{2}(\beta_{r}^{k_{1}})}\right\\}\sin^{2}(\alpha_{0}^{1,k_{1}})=1,\\\
&\forall\\-L\geq L_{0}>0,\ 1\leq k_{1},k_{2}\leq k_{0}.\end{split}$

##  3\. The second exact complete hybrid formula

### 3.1.

Further, we have the following exact $\zeta$-factorization formula (see [6],
(3.1) – (3.4), $L\to\pi L$).

######  Lemma 3.

For the function

(3.1) $\bar{f}_{\Delta}(t)=\bar{f}(t;L,\Delta)=(t-\pi
L)^{\Delta}\in\tilde{C}_{0}[\pi L,\pi L+U],\ U\in(0,\pi/2),\ \Delta>0$

there are vector-valued functions

(3.2)
$(\alpha_{0}^{\Delta,\bar{k}_{\Delta}},\alpha_{1}^{\Delta,\bar{k}_{\Delta}},\dots,\alpha_{\bar{k}_{\Delta}}^{\Delta,\bar{k}_{\Delta}},\beta_{1}^{\bar{k}_{\Delta}},\dots,\beta_{\bar{k}_{\Delta}}^{\bar{k}_{\Delta}}),\
1\leq\bar{k}_{\Delta}\leq k_{0},\ \bar{k}_{\Delta}\in\mathbb{N}$

such that the following exact $\zeta$-factorization formula

(3.3)
$\prod_{r=1}^{\bar{k}_{\Delta}}\frac{\tilde{Z}^{2}(\alpha_{r}^{\Delta,\bar{k}_{\Delta}})}{\tilde{Z}^{2}(\beta_{r}^{\bar{k}_{\Delta}})}=\frac{1}{1+\Delta}\left(\frac{U}{\alpha_{0}^{\Delta,\bar{k}_{\Delta}}-\pi
L}\right)^{\Delta},\ \forall\\-L\geq L_{0}>0$

holds true, where

(3.4)
$\begin{split}&\alpha_{r}^{\Delta,\bar{k}_{\Delta}}=\alpha_{r}(U,L,\bar{k}_{\Delta};\bar{f}_{\Delta}),\
r=0,1,\dots,\bar{k}_{\Delta},\\\
&\beta_{r}^{\bar{k}_{\Delta}}=\beta_{r}(U,L,\bar{k}_{\Delta}),\
r=1,\dots,\bar{k}_{\Delta},\\\ &\alpha_{0}^{\Delta,\bar{k}_{\Delta}}\in(\pi
L,\pi L+U),\
\alpha_{r}^{\Delta,\bar{k}_{\Delta}},\beta_{r}^{\bar{k}_{\Delta}}\in(\overset{r}{\wideparen{\pi
L}},\overset{r}{\wideparen{\pi L+U}}),\
r=1,\dots,\bar{k}_{\Delta}.\end{split}$

### 3.2.

Next, we have the following set of two exact $\zeta$-factorization formulae
(see (3.3))

(3.5) $\begin{split}&(1+\Delta_{3})(\alpha_{0}^{3,k_{3}}-\pi
L)^{\Delta_{3}}\prod_{r=1}^{k_{3}}\frac{\tilde{Z}^{2}(\alpha_{r}^{3,k_{3}})}{\tilde{Z}^{2}(\beta_{r}^{k_{3}})}=U^{\Delta_{3}},\\\
&(1+\Delta_{4})(\alpha_{0}^{4,k_{4}}-\pi
L)^{\Delta_{4}}\prod_{r=1}^{k_{4}}\frac{\tilde{Z}^{2}(\alpha_{r}^{4,k_{4}})}{\tilde{Z}^{2}(\beta_{r}^{k_{4}})}=U^{\Delta_{4}},\\\
&\Delta_{3}\not=\Delta_{4},\end{split}$

where

(3.6) $\alpha_{r}^{\Delta_{3},\bar{k}(\Delta_{3})}=\alpha_{r}^{3,k_{3}},\
\dots$

for the brevity. Now, we obtain, by crossbreeding on the set (3.5) (i.e. by
elimination of the external variable $U$ in (3.5)) the following

######  Exact Complete Hybrid Formula 2.

(3.7) $\begin{split}&(1+\Delta_{3})^{1/\Delta_{3}}(\alpha_{0}^{3,k_{3}}-\pi
L)\left\\{\prod_{r=1}^{k_{3}}\frac{\tilde{Z}^{2}(\alpha_{r}^{3,k_{3}})}{\tilde{Z}^{2}(\beta_{r}^{k_{3}})}\right\\}^{1/\Delta_{3}}=\\\
&=(1+\Delta_{4})^{1/\Delta_{4}}(\alpha_{0}^{4,k_{4}}-\pi
L)\left\\{\prod_{r=1}^{k_{4}}\frac{\tilde{Z}^{2}(\alpha_{r}^{4,k_{4}})}{\tilde{Z}^{2}(\beta_{r}^{k_{4}})}\right\\}^{1/\Delta_{4}},\\\
&\forall\\-L\geq L_{0}>0,\ \Delta_{3},\Delta_{4}>0,\
\Delta_{3}\not=\Delta_{4},\ 1\leq k_{3},k_{4}\leq k_{0}.\end{split}$

##  4\. The first variant of secondary exact complete hybrid formula

Now, we make use of our operation of the the secondary crossbreeding (see [8])
on the set of two exact complete hybrid formulae (2.9) and (3.7) as follows.

### 4.1.

Firstly, we put into (3.7)

(4.1) $k_{3}=k_{4}=k;\ 1\leq k\leq k_{0}$

that gives us the following formula

(4.2) $\begin{split}&\prod_{r=1}^{k}\tilde{Z}^{2}(\beta_{r}^{k})=\\\
&=\left[\frac{(1+\Delta_{3})^{1/\Delta_{3}}}{(1+\Delta_{4})^{1/\Delta_{4}}}\right]^{\frac{\Delta_{3}\Delta_{4}}{\Delta_{4}-\Delta_{3}}}\left(\frac{\alpha_{0}^{3,k}-\pi
L}{\alpha_{0}^{4,k}-\pi
L}\right)^{\frac{\Delta_{3}\Delta_{4}}{\Delta_{4}-\Delta_{3}}}\left\\{\prod_{r=1}^{k}\tilde{Z}^{2}(\alpha_{r}^{3,k})\right\\}^{\frac{\Delta_{4}}{\Delta_{4}-\Delta_{3}}}\times\\\
&\times\left\\{\prod_{r=1}^{k}\tilde{Z}^{2}(\alpha_{r}^{4,k})\right\\}^{-\frac{\Delta_{3}}{\Delta_{4}-\Delta_{3}}}.\end{split}$

### 4.2.

Secondly, we put consecutively in (4.2) $k=k_{1},k_{2}$ and the corresponding
results we substitute into the formula (2.9). Hence, we obtain the following

######  Theorem 1.

The set

(4.3) $\begin{split}&\\{\sin^{2}t,\cos^{2}t,(t-\pi L)^{\Delta_{3}},(t-\pi
L)^{\Delta_{4}}\\},\\\ &t\in[\pi L,\pi L+U],\ U\in(0,\pi/2),\
\Delta_{3},\Delta_{4}>0,\Delta_{3}\not=\Delta_{4}\end{split}$

of real continuous functions generates the following secondary complete hybrid
formula

(4.4)
$\begin{split}&\frac{\prod_{r=1}^{k_{2}}\tilde{Z}^{2}(\alpha_{r}^{2,k_{2}})[\tilde{Z}^{2}(\alpha_{r}^{3,k_{2}})]^{\frac{\Delta_{4}}{\Delta_{3}-\Delta_{4}}}[\tilde{Z}^{2}(\alpha_{r}^{4,k_{2}})]^{\frac{-\Delta_{3}}{\Delta_{3}-\Delta_{4}}}}{\left(\frac{\alpha_{0}^{4,k_{2}}-\pi
L}{\alpha_{0}^{3,k_{2}}-\pi
L}\right)^{\frac{\Delta_{3}\Delta_{4}}{\Delta_{3}-\Delta_{4}}}}\cos^{2}(\alpha_{0}^{2,k_{2}})+\\\
&+\frac{\prod_{r=1}^{k_{1}}\tilde{Z}^{2}(\alpha_{r}^{1,k_{1}})[\tilde{Z}^{2}(\alpha_{r}^{3,k_{1}})]^{\frac{\Delta_{4}}{\Delta_{3}-\Delta_{4}}}[\tilde{Z}^{2}(\alpha_{r}^{4,k_{1}})]^{\frac{-\Delta_{3}}{\Delta_{3}-\Delta_{4}}}}{\left(\frac{\alpha_{0}^{4,k_{1}}-\pi
L}{\alpha_{0}^{3,k_{1}}-\pi
L}\right)^{\frac{\Delta_{3}\Delta_{4}}{\Delta_{3}-\Delta_{4}}}}\sin^{2}(\alpha_{0}^{1,k_{1}})=\\\
&=\left[\frac{(1+\Delta_{4})^{1/\Delta_{4}}}{(1+\Delta_{3})^{1/\Delta_{3}}}\right]^{\frac{\Delta_{3}\Delta_{4}}{\Delta_{3}-\Delta_{4}}},\
\forall\\-L\geq L_{0}>0,\ 1\leq k_{1},k_{2}\leq k_{0}\end{split}$

for arbitrary fixed $k_{0}\in\mathbb{N}$ and sufficiently big
$L_{0}\in\mathbb{N}$.

######  Remark 4.

We list here main properties of the formula (4.4):

  * (A)

it is the synergetic (cooperative) formula (see sect. 1.5 and paper [8]),

  * (B)

continuum set of the invariants relatively to arbitrary sampling of:

    * (a)

of the continuum parameter $U\in(0,\pi/2)$,

    * (b)

of discrete parameter $L\in[L_{0},\infty)$,

    * (c)

of the subset (as the set-parameter)

$\\{k_{1},k_{2}\\}\subset\\{1,\dots,k_{0}\\},$

is defined by its left-hand side for arbitrary

$\Delta_{3},\Delta_{4}>0,\ \Delta_{3}\not=\Delta_{4}.$

Consequently, we have defined two-parametric continuum set of three-parametric
invariants.

######  Remark 5.

Formula (1.1) corresponds to the values

$\Delta_{3}=\frac{1}{3},\ \Delta_{4}=\frac{1}{5}$

in (4.4).

##  5\. Second variant of secondary exact complete hybrid formula

Here, we make again use of the secondary crossbreeding on the set of the
formulae (2.9) and (3.7) however in the reverse direction as follows.

### 5.1.

First, we put in (2.9)

(5.1) $k_{1}=k_{2}=k;\ 1\leq k\leq k_{0}$

that gives us in result

(5.2)
$\prod_{r=1}^{k}\tilde{Z}^{2}(\beta_{r}^{k})=\left\\{\prod_{r=1}^{k}\tilde{Z}^{2}(\alpha_{r}^{2,k})\right\\}\cos^{2}(\alpha_{0}^{2,k})+\left\\{\prod_{r=1}^{k}\tilde{Z}^{2}(\alpha_{r}^{1,k})\right\\}\sin^{2}(\alpha_{0}^{1,k}).$

### 5.2.

Secondly, we put consecutively in (5.2)

$k=k_{3},k_{4}$

and we put the corresponding result into (3.7). The final result is listed
bellow as theorem.

######  Theorem 2.

The set

(5.3) $\begin{split}&\\{\sin^{2}t,\cos^{2}t,(t-\pi L)^{\Delta_{3}},(t-\pi
L)^{\Delta_{4}}\\},\\\ &t\in[\pi L,\pi L+U],\ U\in(0,\pi/2),\
\Delta_{3},\Delta_{4}>0,\Delta_{3}\not=\Delta_{4}\end{split}$

generates the following secondary complete hybrid formula

(5.4) $\begin{split}&\frac{\alpha_{0}^{3,k_{3}}-\pi
L}{\alpha_{0}^{3,k_{3}}-\pi L}\times\\\
&\times\left(\frac{\prod_{r=1}^{k_{3}}\tilde{Z}^{2}(\alpha_{r}^{3,k_{3}})}{[\prod_{r=1}^{k_{3}}\tilde{Z}^{2}(\alpha_{r}^{2,k_{3}})]\cos^{2}(\alpha_{0}^{2,k_{3}})+[\prod_{r=1}^{k_{3}}\tilde{Z}^{2}(\alpha_{r}^{1,k_{3}})]\sin^{2}(\alpha_{0}^{1,k_{3}})}\right)^{\frac{1}{\Delta_{3}}}\times\\\
&\times\left(\frac{\prod_{r=1}^{k_{4}}\tilde{Z}^{2}(\alpha_{r}^{4,k_{4}})}{[\prod_{r=1}^{k_{4}}\tilde{Z}^{2}(\alpha_{r}^{2,k_{4}})]\cos^{2}(\alpha_{0}^{2,k_{4}})+[\prod_{r=1}^{k_{4}}\tilde{Z}^{2}(\alpha_{r}^{1,k_{4}})]\sin^{2}(\alpha_{0}^{1,k_{4}})}\right)^{-\frac{1}{\Delta_{4}}}=\\\
&=\frac{(1+\Delta_{4})^{1/\Delta_{4}}}{(1+\Delta_{3})^{1/\Delta_{3}}},\
\forall\\-L\geq L_{0}>0,\ 1\leq k_{3},k_{4}\leq k_{0}.\end{split}$

######  Remark 6.

The case

$\Delta_{3}=\Delta_{4}$

is excluded (see (5.3)) since this one implies trivially that $1=1$.

######  Remark 7.

Main properties of the formula (5.4) are characterized by the Remark 4.

##  6\. On notion of exact ternary complete hybrid formula

Now, we are able to proceed in our hierarchy of types of crossbreeding.
Namely, we can introduce notion of the exact ternary complete hybrid formula.
Of course, it is sufficient to give an example.

We have obtained the set of two exact secondary hybrid formulae (4.4), (5.4).
Every of these formulae contains the functions (this is the point to be
generalized)

$\frac{(1+\Delta_{4})^{1/\Delta_{4}}}{(1+\Delta_{3})^{1/\Delta_{3}}},\
\Delta_{3},\Delta_{4}>0,\ \Delta_{3}\not=\Delta_{4}.$

The elimination of this function from the set $\\{(\ref{4.4}),(\ref{5.4})\\}$
(that is the ternary crossbreeding) gives the following

######  Exact Ternary Complete Hybrid Formula .

(6.1)
$\begin{split}&\frac{\prod_{r=1}^{k_{2}}\tilde{Z}^{2}(\alpha_{r}^{2,k_{2}})[\tilde{Z}^{2}(\alpha_{r}^{3,k_{2}})]^{\frac{\Delta_{4}}{\Delta_{3}-\Delta_{4}}}[\tilde{Z}^{2}(\alpha_{r}^{4,k_{2}})]^{\frac{-\Delta_{3}}{\Delta_{3}-\Delta_{4}}}}{\left(\frac{\alpha_{0}^{4,k_{2}}-\pi
L}{\alpha_{0}^{3,k_{2}}-\pi
L}\right)^{\frac{\Delta_{3}\Delta_{4}}{\Delta_{3}-\Delta_{4}}}}\cos^{2}(\alpha_{0}^{2,k_{2}})+\\\
&+\frac{\prod_{r=1}^{k_{1}}\tilde{Z}^{2}(\alpha_{r}^{1,k_{1}})[\tilde{Z}^{2}(\alpha_{r}^{3,k_{1}})]^{\frac{\Delta_{4}}{\Delta_{3}-\Delta_{4}}}[\tilde{Z}^{2}(\alpha_{r}^{4,k_{1}})]^{\frac{-\Delta_{3}}{\Delta_{3}-\Delta_{4}}}}{\left(\frac{\alpha_{0}^{4,k_{1}}-\pi
L}{\alpha_{0}^{3,k_{1}}-\pi
L}\right)^{\frac{\Delta_{3}\Delta_{4}}{\Delta_{3}-\Delta_{4}}}}\sin^{2}(\alpha_{0}^{1,k_{1}})=\\\
&\left(\frac{\alpha_{0}^{3,k_{3}}-\pi L}{\alpha_{0}^{3,k_{3}}-\pi
L}\right)^{\frac{\Delta_{3}\Delta_{4}}{\Delta_{3}-\Delta_{4}}}\times\\\
&\times\left(\frac{\prod_{r=1}^{k_{3}}\tilde{Z}^{2}(\alpha_{r}^{3,k_{3}})}{[\prod_{r=1}^{k_{3}}\tilde{Z}^{2}(\alpha_{r}^{2,k_{3}})]\cos^{2}(\alpha_{0}^{2,k_{3}})+[\prod_{r=1}^{k_{3}}\tilde{Z}^{2}(\alpha_{r}^{1,k_{3}})]\sin^{2}(\alpha_{0}^{1,k_{3}})}\right)^{\frac{\Delta_{4}}{\Delta_{3}-\Delta_{4}}}\times\\\
&\times\left(\frac{\prod_{r=1}^{k_{4}}\tilde{Z}^{2}(\alpha_{r}^{4,k_{4}})}{[\prod_{r=1}^{k_{4}}\tilde{Z}^{2}(\alpha_{r}^{2,k_{4}})]\cos^{2}(\alpha_{0}^{2,k_{4}})+[\prod_{r=1}^{k_{4}}\tilde{Z}^{2}(\alpha_{r}^{1,k_{4}})]\sin^{2}(\alpha_{0}^{1,k_{4}})}\right)^{-\frac{\Delta_{3}}{\Delta_{3}-\Delta_{4}}},\\\
&\forall\\-L\geq L_{0}>0,\ 1\leq k_{1},k_{2},k_{3},k_{4}\leq
k_{0}.\end{split}$

## References

  * [1] J. Moser, ‘Jacob’s ladders and almost exact asymptotic representation of the Hardy-Littlewood integral‘, Math. Notes 88, (2010), 414-422, arXiv: 0901.3937.
  * [2] J. Moser, ‘Jacob’s ladders, structure of the Hardy-Littlewood integral and some new class of nonlinear integral equations‘, Proc. Steklov Inst. 276 (2011), 208-221, arXiv: 1103.0359.
  * [3] J. Moser, ‘Jacob’s ladders, reverse iterations and new infinite set of $L_{2}$-orthogonal systems generated by the Riemann zeta-function, arXiv: 1402.2098.
  * [4] J. Moser, ‘Jacob’s ladders, factorization and metamorphoses as an appendix to the Riemann functional equation for $\zeta(s)$ on the critical line‘, Proc. Steklov Inst. 296 (2017), pp. 92-102, arXiv: 1506.00442v1.
  * [5] J. Moser, ’Jacob’s ladders, interactions between $\zeta$-oscillating systems and $\zeta$-analogue of an elementary trigonometric identity’, arXiv: 1609.09293v1, Proc. Steklov Inst. 299, 189-204, 2017.
  * [6] J. Moser, ‘Jacob’s ladders, crossbreeding in the set of $\zeta$-factorization formulas and the selection of families of $\zeta$-kindred real continuous functions‘, arXiv: 1710.04428 .
  * [7] J. Moser, ‘Jacob ladders and new families of $\zeta$-kindred real continuous functions‘, arXiv: 1801.09425v1.
  * [8] J. Moser, ‘Jacob ladders, crossbreeding, secondary crossbreeding and synergetic phenomena generated by the Riemann’s zeta-function and some elementary functions on disconnected sets of the critical line‘, arXiv: 1806.07095v1.


# Fact-level Extractive Summarization with
Hierarchical Graph Mask on BERT

Ruifeng Yuan Zili Wang Wenjie Li
Department of Computing, The Hong Kong Polytechnic University, Hong Kong
Xidian University
{csryuan<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

Most current extractive summarization models generate summaries by selecting
salient sentences. However, one of the problems with sentence-level extractive
summarization is that there exists a gap between the human-written gold
summary and the oracle sentence labels. In this paper, we propose to extract
fact-level semantic units for better extractive summarization. We also
introduce a hierarchical structure, which incorporates the multi-level of
granularities of the textual information into the model. In addition, we
incorporate our model with BERT using a hierarchical graph mask. This allows
us to combine BERT’s ability in natural language understanding and the
structural information without increasing the scale of the model. Experiments
on the CNN/DaliyMail dataset show that our model achieves state-of-the-art
results. 00footnotetext: The code is available at https://github.com/Ruifeng-
paper/FactExsum-coling2020

00footnotetext: This work is licensed under a Creative Commons Attribution 4.0
International License. License details:
http://creativecommons.org/licenses/by/4.0/.

##  1 Introduction

The aim of text summarization is to generate a compressed shorter highlight of
a given document. The generated summaries must conform to natural language
constraints and cover the most important information conveyed in the source
text. In this paper, we focus on extractive summarization which generates
summaries by selecting salient sentences or other semantic units from the
source text.

As we know, sentences are regarded as the basic textual units making up the
final summaries in most extractive summarization models. One of the problems
with sentence-level extractive summarization is that there exists a gap
between the human-written gold summary and the training objective. Since most
datasets do not contain sentence-level labels, oracle summaries are normally
generated by a greedy algorithm, which maximizes the ROUGE score between the
sentences in the source text and in the gold summary [Nallapati et al., 2017,
Liu and Lapata, 2019]. As pointed out by [Narayan et al., 2018], sentences
with high ROUGE scores may not necessarily lead to an optimal summary. Such
discrepancy may be due to the overlapping contents and over extraction.
Therefore, some researches try to perform extraction at a lower level such as
words or phrases [Bui et al., 2016, Cheng and Lapata, 2016, Gehrmann et al.,
2018]. Although these models are able to learn which phrases or words are more
important directly from the gold summary, it is hard to maintain semantic
integrity when information is scattered.

Inspired by [Cao et al., 2018], we propose to extract facts, a granularity
between phrases and sentences, as the primary textual units to generate
summaries. In this work, we first develop a heuristic algorithm to split a
sentence into several smaller units, where each unit is considered as a single
fact description. Then we apply a one-to-one strategy to match each fact in
the gold summary to one fact in the source text to obtain the oracle label. In
this way, we smooth the gap between the gold summary and oracle labels and
further reduce redundancy and over-extraction. Meanwhile, fact units can still
well keep semantic integrity and thus are able to provide faithful summaries.
Considering facts are relatively independent semantic units, in order to
maintain rich contextual information and to capture potential relationships
among facts, we believe the sentence-level information still plays an
important supporting role. Moreover, based on the assumption that the selected
facts should be semantically close to the majority of source documents, we
also import the document-level information as part of our model. We will show
that such a fact-sentence-document hierarchical text modeling facilitates to
capture more structured contextual information for effective fact extraction.

Recent works [Liu and Lapata, 2019, Bae et al., 2019, Zhang et al., 2019] have
demonstrated that it is highly beneficial to apply pretrained language models
such as BERT to extractive summarization models. Following this trend, we
adopt a BERT-based encoder to generate contextualized representations for
further extraction. It is challenging to impose the structure information when
fine-tuning the BERT model in a downstream task. [Fang et al., 2019, Zhang et
al., 2019] separately encode representations for each granularity with BERT
and then capture the structure information with a graph network stacked upon
the encoder. However, this method not only results in a large complex model
but also cannot fully utilize the advantage of pretrained language models.
Based on the idea that BERT can be regarded as a full connected graph, we
propose to directly impose the structure information on BERT with a graph-
based mask to jointly learn contextual representations of different text
granularities within a single BERT. On the one hand, we significantly reduce
the time cost of running BERT several times plus running a graph neural
network to running BERT once. On the other hand, we are able to utilize the
complex structure of BERT to capture the structure information.

We conduct experiments on the CNN/DailyMail dataset, a well-known extractive
summarization benchmark dataset. The results show that using facts as
extractive units improves the summarization quality and our model achieves
better performance than the state-of-the-art model. The contributions of our
work can be summarized as follows. (1) We propose to extract fact-level
semantic units for extractive summarization to fill the gap between the gold
summary and oracle labels, reducing redundancy and over-extraction. (2) We
import a hierarchical structure to remedy the information loss after splitting
sentences into facts. (3) We propose a graph-based mask algorithm to impose
the structure information on BERT directly. To the best of our knowledge, we
are the first to combine the pretrained language model and the structure
information without increasing the scale of the model, which provides a new
idea for fine-tuning pretrained models on downstream tasks.

##  2 Related Work

###  2.1 Extractive Summarization

Extractive summarization attempts to select the most important sentences from
the source text and subsequently concatenate them as a summary. With neural
network models, researchers usually formulate it as a sentence binary
classification problem. SummaRuNNer [Nallapati et al., 2017], one of the
earliest neural summarization models, applies an RNN-based encoder to generate
sentence representations and a neural classifier to determine which sentences
should be included in the summary. [Narayan et al., 2018] further extends
SummaRuNNer with a reinforcement model, which optimizes the summary-level
ROUGE metric. Some other works achieve better performance through more complex
models. [Zhou et al., 2018] proposes a decoder that jointly learns to score
and select sentences, while [Zhang et al., 2018] presents a latent variable
extractive summarization model, which directly maximizes the likelihood of
human summaries. In recent works, models based on pretrained models [Liu and
Lapata, 2019, Bae et al., 2019, Zhang et al., 2019], especially BERT, have
made a step forward.

###  2.2 Graph Based Summarization

Graph-based summarization methods aim to utilize the structure information of
the text. Based on the assumption that important sentences are often linked
with each other, unsupervised models like TextRank [Mihalcea and Tarau, 2004]
and LexRank [Erkan and Radev, 2004] show great ability to identify the salient
information. [Liu et al., 2018] proposes an abstractive summarization
framework based on the Abstract Meaning Representation (AMR) graph, which
captures the propositional semantic information. [Koncel-Kedziorski et al.,
2019] presents a graph transformer to generate one-sentence summaries from a
knowledge graph. Meanwhile, other researches focus on learning latent tree
structures while optimizing summarization models. [Williams et al., 2018]
regards the tree structure learning problem as a reinforcement learning
problem and [Liu et al., 2019] generates a multi-root dependency tree where
each root is a summary sentence.

###  2.3 Pretrained Language Models

Pretrained language models [Radford et al., 2018, Devlin et al., 2018] have
been proved to be extremely successful for making great progress in various
nature language tasks. These models are originated from the idea of word
embeddings [Pennington et al., 2014] and further extend it by pretraining a
sentence encoder on the huge unlabeled corpus using language modeling
objectives. Bidirectional Encoder Representations from Transformers (BERT)
[Devlin et al., 2018], one of the state-of-art language models, is trained
with a masked language model and a next-sentence-predicting task. Recently,
pretrained language models have been widely used to improve performance in
language understanding tasks [Dong et al., 2019]. As for the extractive
summarization task, it provides the powerful sentence embeddings and the
contextualized information among sentences [Zhong et al., 2019], which have
been proven to be critical to extractive summarization.

##  3 Extraction of Facts and Alignment with Gold Summary

We propose to explicitly align a gold summary with its corresponding facts
descriptions in the source text. For this purpose, we develop a heuristic
algorithm to break up the source text and the summary into smaller
granularities as introduced below.

When splitting sentences into smaller semantic units, we need to ensure each
unit has a proper size while maintaining the integrity of information. An
intuitive idea might be to split sentences with commas and other conjunctions.
But this straightforward strategy is not capable of handling complicated
sentences in documents. Therefore, we leverage the dependency parser to handle
this issue. To begin with, we adopt a dependency parser to convert a sentence
into the labeled tuples in the form of (word1, word2, label), where the label
denotes a grammatical relation between a pair of words. After that, the
sentence is split using the labels representing punctuation marks,
conjunctions and clauses, including punct (punctuation marks), cc
(coordination relationship) and mark (finite clause). To acquire a more
complete semantic unit, we merge units that are connected by some special
labels such as the relative clause modifier (acl:relcl), the adverbial clause
modifier (advcl), the appositional modifier (appos) and the clausal complement
(ccomp). Furthermore, we use the conjunct (conj) to identify whether a
conjunction connects two sentences or two phrases. A conjunct is a relation
between two elements connected by a coordinating conjunction. When the
distance between the two elements is less than a threshold, we regard the two
coordinated elements are words or phrases rather than clauses. If so, the two
split parts are merged back as one unit.

Moreover, we predefine a minimum unit length and a maximum clause length. If
the length of a unit is smaller than the minimum unit length, this unit will
be merged with the preceding unit. If a clause is longer than the maximum
clause length, we regard it as an independent semantic unit. Take the
dependency trees in Figure 1 as an example. The sentence is split into five
parts based on the above-mentioned rules. We merge the first three parts back
together since they are connected by the appos label while not exceeding the
threshold length for being an independent clause. As for the last two parts,
although there is a coordination relationship, it turns out that they are two
coordinate entities based on the conj label. Hence, we also merge the last two
parts as a fact.

Figure 1: A dependency tree example. We extract the following two fact
descriptions: Ahmadinejad essentially called Yukiya Amano, the director
general of the IAEA, a U.S. puppet $|||$ said the U.N.A has no jurisdiction in
Iran and Irap

In the experiments, we adopt the well-known NLP pipeline Stanford CoreNLP
[Hermann et al., 2015] for dependency parsing. Table 1 presents the statistics
on the train set of CNN/DaliyMail. On average, a sentence contains 1.6 facts.
It should be noted that some sentences remain as single facts, while others
are split into 2 or 3 facts.

granularity | num | len
---|---|---
Sentence | 34.3 | 24.7
Fact | 51.5 | 14.8
Table 1: The average unit number and unit length in train set of CNN/DM
dataset .

Once facts are extracted, the next step is to obtain oracle labels for them
from the gold summary. We match each fact in the summary with one
corresponding fact that has the maximum ROUGE score from original text. Such
an alignment allows each part of the summary to be accurately mapped to a
semantic unit in the source text.

##  4 Summarization Model

###  4.1 Model Framework

As illustrated in Figure 2, our model consists of three components: a BERT-
based encoder, a hierarchical graph mask, which incorporates the structure
constraint on BERT and a classifier taking the multi-granularity information
as input to extract salient facts to form the summary.

###  4.2 BERT-based Encoder

We use a BERT-based encoder to generate contextualized representations of the
semantic units with different granularities. Since the outputs of BERT are
grounded to tokens, we adapt the strategy that is similar to [Liu and Lapata,
2019] to modify the embeddings and the input sequence of BERT.

To obtain representations of the facts, the sentences and the document, we add
a special token at the begining of each semantic unit. As shown in Figure 2,
[cls_d], [cls_s] and [cls_f] are inserted at the beginning of the document,
the head of each sentence and the fact representing the embeddings of
different granularities, respectively. Initially, these three kinds of [cls]
tokens share the same pre-trained [cls] embedding. We also add a [seq] token
at the end of each fact to separate the smallest semantic units. In addition,
segment embeddings are used to identify multiple granularity levels within a
document. For the $i$th granularity level, we assign the segment embedding
$E_{A}$ or $E_{B}$ conditioned on whether $i$ is odd or even. For example, in
Figure 2, we first assign the [cls_d] segment embedding $E_{A}$ and then for
the next granularity level, sentence, we allocate all [cls_s] with $E_{B}$.
With two different segment embeddings that separate adjacent granularity
levels, the model is aware of the hierarchical structure among different
granularity levels.

Figure 2: The framework of the summarization model

###  4.3 Hierarchical Graph Mask on BERT

A document is composed of a collection of sentences, while each sentence may
contain multiple facts. Such a hierarchical structure motivates us to generate
representations for different granularity levels with hierarchical constrains
on BERT. We propose a hierarchical graph mask to restrict information
dissemination in BERT. To build a hierarchical graph, we add three kinds of
edges to a token in the input sequence, including the bi-directional edges
connecting to all other tokens at the same granularity level, the edges from
the tokens belonging to them at the smaller granularity level and the edges to
their corresponding tokens at the larger granularity level. For example, as
shown in Figure 2, a fact token [cls_f1] can disseminate its information to
other fact tokens and the sentence token [cls_s1], while receiving the
information from the word tokens “fact”, “one” and [seq]. The tokens at each
granularity level thus capture semantics from different sources. As a result,
the hierarchical graph can effectively utilize all levels of structural
information with different granularities.

To implement this graph structure on BERT, we assign a mask vector to each
token based on the hierarchical graph $H$. Given an input sequence with $n$
tokens $[t_{1},t_{2},...,t_{n}]$, the mask vector of $t_{i}$ is denoted by
$[h_{i1},h_{i2},...,h_{in}]$, where $h_{ij}$ refers to whether there is a
directed edge from the token $j$ to the token $i$ in the graph $H$. Taking
[cls_f1] in Figure 2 as an example, the mask vector of this token is
$[0,0,1,1,1,1,1,0,0,0,0,1,0,0,0]$. After obtaining the mask vector for each
token, we stack them to form a $n\times n$ mask matrix $M$ and calculate
attentions with the equation below. For simplification, we write it in the
one-head form.

$Attention(Q,K,V)=Softmax(\frac{QK^{T}M}{\sqrt{d^{k}}})V$

where $Q$ refers to the query matrix, $K$ refers to the key matrix, $V$ refers
to the value matrix and $\sqrt{d^{k}}$ represents a scaling factor.

###  4.4 Classifier

After applying the hierarchical graph mask on BERT from words to document, we
obtain representations of all levels of granularities. Although our focus is
to extract the salient facts from a given document, we believe that the
sentence-level information also contributes to identifying the importance
among facts. Considering facts are rather independent semantic units, the
sentence information is able to build up connections from a higher-level
perspective and help locate the important facts more accurately. Moreover, we
postulate that the selected facts should be semantically representative of the
source document. Therefore, we also import the document representation as part
of input to the classifier.

Given a fact representation $f_{i}$ from the last layer of BERT, we
concatenate it with the document representation $d$ and its corresponding
sentence representation $s_{j}$. Then we employ a sigmoid classifier to
predict whether or not it should be selected into the summary:

$y^{{}^{\prime}}_{i}=\sigma(W_{c}(d||s_{j}||f_{i})+b_{c})$

where $||$ represents concatenation, $W_{c}$ and $b_{c}$ are the parameters of
the classifier.

###  4.5 Learning

We apply a binary classification entropy of prediction $y^{{}^{\prime}}_{i}$
against gold label $y_{i}$ as the training loss. Similar to [Liu and Lapata,
2019, Vaswani et al., 2017], we use the Adam optimizer with $lr=2e^{-5}$,
$\beta_{1}=0.9$ and $\beta_{2}=0.999$, and adopt a learning rate schedule with
warming-up=10000:

$lr=2e^{-3}\cdot min(step^{-0.5},step\cdot warmup^{-1.5})$

For testing, we first predict the score $y_{i}$ for each fact using our model
and then rank these facts with their scores to select the top-$4$ facts as the
summary, considering the average number of facts contained in gold summaries
is four. We also apply the Trigram Blocking algorithm [Paulus et al., 2017] to
reduce redundancy. In particular, given the current summary $s$, a candidate
fact $c$ will not be included in $s$ if there is a trigram overlapping between
$c$ and $s$. In this way, we minimize the similarity between the current
summary and the candidate sentence to increase information diversity and
richness.

##  5 Experiments

###  5.1 Dataset and Evaluation Metric

We conduct experiments on the non-anonymized version of the CNN/DailyMail
dataset [Hermann et al., 2015]. The dataset is composed of news articles and
their corresponding highlights that give brief overviews of articles. We apply
the standard splits for training, validation, and testing (CNN:
90,266/1,220/1,093 and DailyMail: 196,961/12,148/10,397). Due to the length
limitation of BERT, we follow the common practice to truncate all input
documents to 512 tokens.

To evaluate the summarization quality, we apply ROUGE [Lin, 2004] for
automatic evaluation. Following the convention, we report ROUGE-1 (unigram),
ROUGE-2 (bi-gram) and ROUGE-L (longest common subsequence) F1-score in the
following experiments. Moreover, we manually evaluate the fact-level recall,
precision and F1-score between system-generally summaries and human-written
gold summaries.

###  5.2 Details

We build our model using PyTorch and the BERT-base-uncased version of BERT.
All the input documents are tokenized by BERT’s sub-words tokenizer. We train
the model for at most 50000 steps and the batch size in each step is 32. After
training for 10000 steps, the model is saved and evaluated for every 1500
steps. With the three best checkpoints on the validation dataset, we record
best model on the test dataset among the three.

###  5.3 Automatic Evaluation

We first examine the ROUGE results of the oracle summaries matched with
different granularities. As shown in Table 2, by aligning facts with gold
summaries it achieves significant improvement on all three ROUGE metrics. It
shows that our idea of building oracle summaries based on facts is able to
generate more accurate oracle labels for training.

Granularity | R-1 | R-2 | R-L
---|---|---|---
Fact | 57.96 | 34.93 | 54.69
Sentence | 52.59 | 31.24 | 48.87
Table 2: Results on oracle summary on CNN/DM test set.

Table 3 summarizes the results of a variety of models on the CNN/DailyMail
test set. The first section in the table presents two lead-3 baselines that
select the first three sentences as summaries. One is our own implantation,
while the other one is copied from [Liu and Lapata, 2019]. The slight
discrepancy on ROUGE scores may be because of the difference in data
preprocessing.

The second section in the table displays performances of the existing
extractive models, including the baseline model SummaRunner, two SOTA models
with no pretraining and several models based on BERT. All the results are
directly taken from their respective papers.

The last section reports the results of our own implementations. In order to
avoid interference of other factors, we implement the SOTA model BERTSUM in
our training environment. Meanwhile, for better comparison, we propose a fact-
extraction BERTSUM baseline, which simply changes the primary textual unit for
extraction from sentences to facts under the same architecture as BERTSUM.
Compared to BERTSUM, using facts as extraction units improves the performance
on ROUGE-2 and ROUGE-L, since it provides more accurate alignment on the
salient information and reduces the proportion of meaningless words in
summaries. After imposing the structure information on BERT, we further
develop the potential of fact-level extractive summarization. All three
variants of our model achieve improvement on ROUGE scores. Among the variants,
the model with additional sentence-level information performs best. This is
not surprising at all. In fact, sentence-level relationships contain the rich
contextual information. Unexpectedly, we find that it does not lead to the
best result by combining all three levels of information. We suspect that it
may be due to that the document-level information is not effective enough for
single document summarization.

Model | ROUGE-1 | ROUGE-2 | ROUGE-L
---|---|---|---
Lead-3 (our) | 40.25 | 17.40 | 36.41
Lead-3 [Liu and Lapata, 2019] | 40.43 | 17.62 | 36.67
SummaRunner [Nallapati et al., 2017] | 39.60 | 16.20 | 35.30
BANDITSUM [Dong et al., 2018] | 41.50 | 18.70 | 37.60
NEUSUM [Zhou et al., 2018] | 41.59 | 19.01 | 37.98
HIBERT [Zhang et al., 2019] | 42.37 | 19.95 | 38.83
PNBERT+RL [Zhong et al., 2019] | 42.69 | 19.60 | 38.85
BERTEXT+RL [Bae et al., 2019] | 42.76 | 19.87 | 39.11
BERTSUM [Liu and Lapata, 2019] | 43.23 | 20.22 | 39.60
BERTSUM (our) | 42.78 | 19.79 | 39.03
BERTSUM+fact | 42.81 | 19.93 | 39.84
Our d+f | 42.97 | 20.09 | 39.98
Our s+f | 43.10 | 20.17 | 40.10
Our d+s+f | 43.01 | 20.07 | 40.02
Table 3: Results on CNN/DM test set. BERTSUM+fact refers to the BERTSUM
baseline which simply changes the primary textual unit for extraction from
sentences to facts. The last three models represent three variants of our
method. For example, d+f means that we use document-level information and
fact-level information for summarization

###  5.4 Human Evaluation

As we know, although the ROUGE metric has long been regarded as a classical
evaluation metric in summarization, it does not always reflect the true
quality of summaries. Hence, we also conduct human evaluation to check the
fact-level recall, precision and F1-score from 50 random samples. As shown in
Table 4, compared with sentence extraction, precision of fact extraction
increases from 32% to 39%, which demonstrates the ability of our model to
reduce the over-extraction problem. As for recall, we also gain some degree of
improvement.

Model | Recall | Precision | F1
---|---|---|---
BERTSUM+fact | 41.79 | 38.78 | 39.52
BERTSUM | 40.45 | 31.79 | 34.98
Table 4: Human evaluation results on CNN/DM test set.

###  5.5 Model Analysis

#### Position of Extracted Facts:

In addition to evaluating the models through ROUGE, we further look into the
details of our model. As we know, in a document the sentences at the beginning
tend to be more important and are much more likely to be included in
summaries. That is the reason why lead-3 is such a strong baseline in the
CNN/DailyMail dataset. It turns out that current models may rely too much on
the positional information. We feel it is important to analyze the position of
the selected units in the source text. To ensure fairness in comparison, we
convert all sentences into their corresponding fact units in this experiment.
Table 5 shows the proportion of the selected facts that appear in the source
text at different fact positions. 1-5 refers to the first five positions, 6-10
represents the positions 6 to 10, and “Rest” means the rest of the positions
except 1-15. The results are obtained from the oracle summaries generated by
sentence extraction, the oracle summaries generated by facts, those generated
by BERTSUM (truncated to four facts) and our s+f. We find that the oracle
summaries are distributed smoothly across documents , while the summaries
generated by both models highly bias towards the beginning text. The
difference is that our model shows a flatter curve on the first 10 facts,
indicating that our model can achieve better diversity.

Model | 1-5 | 6-10 | 11-15 | Rest
---|---|---|---|---
Oracle s | 33.11 | 23.15 | 11.79 | 31.95
Oracle f | 29.34 | 21.81 | 11.89 | 36.96
BERTSUM | 59.39 | 26.65 | 6.12 | 7.84
Our s+f | 55.66 | 30.33 | 8.15 | 5.86
Table 5: Proportion of extracted facts according to their position in the
original text.

#### Ablation Study:

Since the transformer-based model provides us an effective way to disentangle
the information from different sources, we are allowed to design an experiment
to investigate what roles these types of information play in our model. As
shown in Table 6, with our s+f as a base model, we report the ROUGE score
after removing segment embeddings or positional embeddings from it. We can
observe that the performance of the model without segment embeddings decreases
to the same level as BERTSUM+Fact, since it is challenging for a model to
understand the hierarchical structure when there is no difference among the
information from different sources. We believe that our segmentation strategy
is essential for learning the structural information. As for positional
embeddings, it is not a surprise that the performance drops by a large margin.
According to [Zhong et al., 2019], current extractive models heavily rely on
positional information. The bottom half of Table 6 contains the performance of
other models without positional information, including Transformer from [Zhong
et al., 2019] and BERTSUM from our experiments. Compared to these models, the
hierarchical structure leads to notable improvement, demonstrating that our
model has good potential to learn more semantic information rather than simply
relying on the positional information.

Model | R-1 | R-2 | R-L
---|---|---|---
Original | 43.10 | 20.17 | 40.10
\- Segment | 42.91 | 19.98 | 39.87
\- Position | 39.55 | 17.37 | 36.52
Transformer | 37.90 | 15.69 | 34.31
BERTSUM | 37.97 | 15.93 | 34.66
Table 6: Results for disentangling test on CNN/DM test set.

###  5.6 Case Study

Table 7 illustrates an example to show why facts are more effective than
sentences in extractive summarization. Here, we use a semicolon to separate
different facts in one sentence. The words in italics refer to the sentences
selected by the greedy algorithm used in previous models and the colored text
represents the facts selected by fact-level alignment, where each selected
fact corresponds to one fact in a gold summary. Based on our observation, one
sentence may contain several facts but only one of them is included in
summary. Meanwhile, although the first sentence selected by the greedy
algorithm has a high ROUGE score against the gold summary, it does not
correlate with any important facts. On the contrary, our model proves its
ability to align the gold summary to its correct location in the source text.
The last part of the table contains a summary generated by our model. We can
see that our model successfully extracts two gold facts out of four.As a
contrast, we also mark the sentences generated by BERTSUM using underscore. It
is observed that BERTSUM tends to focus on several leading sentences but only
one of them actually contains a gold fact.

Document
---
Virtual reality may still seem like a hobby reserved for hardcore gamers; but
as headsets drop in price it is on the verge of becoming mainstream. One firm
helping to fuel this trend is immerse. It has created a virtual reality
headset that works with any android and ios phone; is compatible with hundreds
of virtual reality apps from the respective stores. The immerse virtual
reality headset is available from firebox for 29. It works with any android
and ios phone that can run virtual reality apps from the respective stores and
play any 3d movie. The maximum size of compatible devices is 3;which means it
will work with the iphone 6 ; not the iphone 6 plus , for example. … Immerse
calls itself an affordable alternative to rivals such as oculus rift; which is
expected to launch a consumer version soon with prices ranging from between
200 and 400. Immerse is available to buy from firebox and can be shipped
internationally.
Gold summary
1: The immerse virtual reality headset is available from firebox for 29.
2: It works with android and ios phones via virtual reality apps and 3d films.
3: The maximum size of the device must be 3.
4: It calls itself an affordable alternative to rivals such as oculus rift.
Our s+f
The immerse virtual reality headset is available from firebox for 29.
The maximum size of the device must be 3.
Is compatible with hundreds of virtual reality apps from the respective
stores.
Virtual reality may still seem like a hobby reserved for hardcore gamers.
Table 7: Example from the CNN/DailMail test dataset

##  6 Conclusion

In this paper, we propose to extract fact-level semantic units for extractive
summarization. Without increasing the scale of the model, we propose a
hierarchical graph mask on BERT to fully utilize the structural information
among different semantic levels. Experiments on the CNN/DaliyMail dataset
shows that our model achieves state-of-the-art results. In the future, we
would like to use our current model to guide abstractive summarization.

## Acknowledgements

The work described in this paper was supported by and Research Grants Council
of Hong Kong (15203617 and 5210919) and National Natural Science Foundation of
China (61672445).

## References

  * [Bae et al., 2019]  Sanghwan Bae, Taeuk Kim, Jihoon Kim, and Sang-goo Lee.  2019\.  Summary level training of sentence rewriting for abstractive summarization.  arXiv preprint arXiv:1909.08752.
  * [Bui et al., 2016]  Duy Duc An Bui, Guilherme Del Fiol, John F Hurdle, and Siddhartha Jonnalagadda.  2016\.  Extractive text summarization system to aid data extraction from full text in systematic review development.  Journal of biomedical informatics, 64:265–272.
  * [Cao et al., 2018]  Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li.  2018\.  Faithful to the original: Fact aware neural abstractive summarization.  In Thirty-Second AAAI Conference on Artificial Intelligence.
  * [Cheng and Lapata, 2016]  Jianpeng Cheng and Mirella Lapata.  2016\.  Neural summarization by extracting sentences and words.  arXiv preprint arXiv:1603.07252.
  * [Devlin et al., 2018]  Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  2018\.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805.
  * [Dong et al., 2018]  Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung.  2018\.  Banditsum: Extractive summarization as a contextual bandit.  arXiv preprint arXiv:1809.09672.
  * [Dong et al., 2019]  Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon.  2019\.  Unified language model pre-training for natural language understanding and generation.  In Advances in Neural Information Processing Systems, pages 13042–13054.
  * [Erkan and Radev, 2004]  Günes Erkan and Dragomir R Radev.  2004\.  Lexrank: Graph-based lexical centrality as salience in text summarization.  Journal of artificial intelligence research, 22:457–479.
  * [Fang et al., 2019]  Yuwei Fang, Siqi Sun, Zhe Gan, Rohit Pillai, Shuohang Wang, and Jingjing Liu.  2019\.  Hierarchical graph network for multi-hop question answering.  arXiv preprint arXiv:1911.03631.
  * [Gehrmann et al., 2018]  Sebastian Gehrmann, Yuntian Deng, and Alexander M Rush.  2018\.  Bottom-up abstractive summarization.  arXiv preprint arXiv:1808.10792.
  * [Hermann et al., 2015]  Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom.  2015\.  Teaching machines to read and comprehend.  In Advances in neural information processing systems, pages 1693–1701.
  * [Koncel-Kedziorski et al., 2019]  Rik Koncel-Kedziorski, Dhanush Bekal, Yi Luan, Mirella Lapata, and Hannaneh Hajishirzi.  2019\.  Text generation from knowledge graphs with graph transformers.  arXiv preprint arXiv:1904.02342.
  * [Lin, 2004]  Chin-Yew Lin.  2004\.  ROUGE: A package for automatic evaluation of summaries.  In Text Summarization Branches Out, pages 74–81, Barcelona, Spain, July. Association for Computational Linguistics.
  * [Liu and Lapata, 2019]  Yang Liu and Mirella Lapata.  2019\.  Text summarization with pretrained encoders.  arXiv preprint arXiv:1908.08345.
  * [Liu et al., 2018]  Fei Liu, Jeffrey Flanigan, Sam Thomson, Norman Sadeh, and Noah A Smith.  2018\.  Toward abstractive summarization using semantic representations.  arXiv preprint arXiv:1805.10399.
  * [Liu et al., 2019]  Yang Liu, Ivan Titov, and Mirella Lapata.  2019\.  Single document summarization as tree induction.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1745–1755.
  * [Mihalcea and Tarau, 2004]  Rada Mihalcea and Paul Tarau.  2004\.  Textrank: Bringing order into text.  In Proceedings of the 2004 conference on empirical methods in natural language processing, pages 404–411.
  * [Nallapati et al., 2017]  Ramesh Nallapati, Feifei Zhai, and Bowen Zhou.  2017\.  Summarunner: A recurrent neural network based sequence model for extractive summarization of documents.  In Thirty-First AAAI Conference on Artificial Intelligence.
  * [Narayan et al., 2018]  Shashi Narayan, Shay B Cohen, and Mirella Lapata.  2018\.  Ranking sentences for extractive summarization with reinforcement learning.  arXiv preprint arXiv:1802.08636.
  * [Paulus et al., 2017]  Romain Paulus, Caiming Xiong, and Richard Socher.  2017\.  A deep reinforced model for abstractive summarization.  arXiv preprint arXiv:1705.04304.
  * [Pennington et al., 2014]  Jeffrey Pennington, Richard Socher, and Christopher D Manning.  2014\.  Glove: Global vectors for word representation.  In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543.
  * [Radford et al., 2018]  Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.  2018\.  Improving language understanding by generative pre-training.  URL https://s3-us-west-2. amazonaws. com/openai-assets/researchcovers/languageunsupervised/language understanding paper. pdf.
  * [Vaswani et al., 2017]  Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  2017\.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008.
  * [Williams et al., 2018]  Adina Williams, Andrew Drozdov*, and Samuel R Bowman.  2018\.  Do latent tree learning models identify meaningful structure in sentences?  Transactions of the Association for Computational Linguistics, 6:253–267.
  * [Zhang et al., 2018]  Xingxing Zhang, Mirella Lapata, Furu Wei, and Ming Zhou.  2018\.  Neural latent extractive document summarization.  arXiv preprint arXiv:1808.07187.
  * [Zhang et al., 2019]  Xingxing Zhang, Furu Wei, and Ming Zhou.  2019\.  Hibert: Document level pre-training of hierarchical bidirectional transformers for document summarization.  arXiv preprint arXiv:1905.06566.
  * [Zhong et al., 2019]  Ming Zhong, Pengfei Liu, Danqing Wang, Xipeng Qiu, and Xuanjing Huang.  2019\.  Searching for effective neural extractive summarization: What works and what’s next.  arXiv preprint arXiv:1907.03491.
  * [Zhou et al., 2018]  Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao.  2018\.  Neural document summarization by jointly learning to score and select sentences.  arXiv preprint arXiv:1807.02305.