# Quantized Innovations Bayesian Filtering

Chun-Chia Huang<EMAIL_ADDRESS>Robert R.Bitmead<EMAIL_ADDRESS>Department of Mechanical & Aerospace Engineering, University of California,
San Diego, 9500 Gilman Drive, La Jolla CA, 92093-0411, USA.

###### Abstract

The paper provides simple formulas of Bayesian filtering for the exact
recursive computation of state conditional probability density functions given
quantized innovations signal measurements of a linear stochastic system. This
is a topic of current interest because the innovations signal should be white
and therefore efficient in its use of channel capacity and in the design and
optimization of the quantizer. Earlier approaches, which we reexamine and
characterize here, have relied on assumptions concerning densities or
approximations to yield recursive solutions, which include the sign-of-
innovations Kalman filter and a Particle filtering technique. Our approach
uses the Kalman filter innovations at the transmitter side and provides a
point of comparison for the other methods, since it is based on the Bayesian
filter. Computational examples are provided.

##  1 Introduction

Efficient transmission of signals over a limited capacity digital channel
requires coding and quantization. Here we examine the recently explored
problem of linear state estimation based on quantized innovations signals to
reconstruct the conditional probability density function of the predicted and
filtered system state given these measurements. We draw on [2, 9, 5] for
algorithmic approaches to this problem, which in turn are based on Bayesian
filtering formulations. The Bayesian filter is a recursive approach for the
computation of predicted and filtered state conditional probability density
functions (pdfs) given a sequence of measurements. It is an approach to
nonlinear filtering. Our contribution is to develop for linear stochastic
systems a signal transmission system based on the quantized Kalman filter
innovations at the transmitter and an associated explicit Bayesian filtering
solution at the receiver. We also draw comparisons to other works through the
provision of a unifying framework from which to view the development. The
advantage of this new approach is that it can benefit from the whiteness of
the Kalman filter innovations and its zero-mean and known covariance
properties to realize optimal coding and Lloyd-Max optimal quantization [11].
We also postulate that it has improved error recovery properties because of
the reconvergent behavior of the Kalman filter.

An innovations signal, $\iota_{k}$, is formed by subtracting from a measured
signal, $y_{k},$ its one-step-ahead prediction, $\hat{y}_{k|k-1}.$ In signal
processing, such calculations are performed in order to whiten a signal prior
to transmission over a bandlimited channel and thereby to achieve more
efficient use of the channel capacity. In the formal context of Kalman
filtering for linear Gaussian systems, the innovations formed from the state
prediction is statistically white; a property that we shall apply. The
prediction can equally be computed as a reconstruction of a signal available
at the receiver, which is achieved at the transmitter by using the computed
receiver signal $\bar{\iota}_{k}$ and control $u_{k}$ as the inputs to the
predictor. Such a scheme forms the basis of the ITU-T G.726 adaptive
differential pulse-coded modulation speech codec standard. The topic of
investigation in this paper is the collection of methods using quantized
innovations signals for the construction of a state state conditional pdf at
the receiver. This is depicted in Figure 1, where the reconstruction is
performed by a Bayesian filter, which we derive in the sequel. We presume that
the input signal, $\\{u_{k}\\}$, is available to both ends; a property that
occurs in remotely located sensing or in filtering.

Figure 1: Block diagram of quantized innovations system.

We use an $m$-level quantizer, $\mathcal{Q}_{k}(\cdot),$ consisting of a
collection of $m$ intervals $(z_{k,l},z_{k,u}],$ which form a disjoint
covering of the real line and a corresponding rule,
$\mathcal{Q}_{k}^{-1}(\cdot),$ for dequantizing the received signal into a
real number — for a $p$-vector channel quantizer, we take a scalar quantizer
in each channel. When the innovations signal $\iota_{k}$ lies within the range
$(z_{k,l},z_{k,u}]$ the output of the quantizer is transmitted as one of $m$
symbols which is then dequantized at the receiver as $\bar{\iota}_{k}$ as one
of $m$ distinct values, which we take also to be a value within the range
$(z_{k,l},z_{k,u}]$ including for the two edge saturation levels. A $3$-bit
uniform quantizer-dequantizer cascade is shown in Figure 2. Evidently,
$\bar{\iota}_{k}=\mathcal{Q}_{k}^{-1}\mathcal{Q}_{k}(\iota_{k})$. The input is
the innovations signal, $\iota_{k}$, and the output is the recovered
innovations signal, $\bar{\iota}_{k}$. The values $\pm\zeta$ denote the upper
and lower saturation limits of the quantizer, which we take for simplicity to
be symmetric. Our results apply equally to any well defined quantizer-
dequantizer pair without reliance on symmetry or uniformity – this is
important because it admits optimized quantizer designs.

Figure 2: Three-bit ($m=8$) uniform quantizer-dequantizer pair input-output
relation with $\zeta=0.6222$ and $L=2\zeta/2^{3}=0.1555$. The methods of the
paper apply equally to non-symmetric and non-uniform quantizers.

The receiver reconstructs a conditional pdf of the plant state $x_{k}$ given
the received quantized innovations data,
$\\{\bar{\iota}_{k},\bar{\iota}_{k-1},\dots,\bar{\iota}_{0}\\}$. We shall
consider three distinct constructions of this measurement signal and
corresponding receiver pdfs, distinguished where necessary by superscripts.
The detailed algorithms will be presented in Section 3.

Method K:

A Kalman filter is operated at the transmitter using signals $\\{y_{k}\\}$ and
$\\{u_{k}\\}$ to produce Kalman output prediction
$\hat{y}^{\text{K}}_{k|k-1},$ and Kalman innovations signal $\epsilon_{k}$,
which is then quantized, transmitted and dequantized at the receiver.

$\displaystyle\bar{\epsilon}_{k}$
$\displaystyle=\mathcal{Q}_{k}^{-1}\mathcal{Q}_{k}(\epsilon_{k})=\mathcal{Q}_{k}^{-1}\mathcal{Q}_{k}(y_{k}-\hat{y}_{k|k-1}^{\text{K}}),$
(1)

The receiver conducts its own processing of this data using a K-Bayesian
filter developed in Section 3. This is depicted in Figure 3.

Figure 3: Block diagram of Method K showing calculation of the innovations
signal, $\epsilon_{k}$. The K-Bayesian filter is detailed in Section 3.

Method R:

The transmitter computes the received signal $\bar{\iota}_{k}$ and uses a copy
of the receiver’s R-Bayesian filter to construct
$p^{\text{R}}(x_{k}\Big{|}\bar{I}^{\text{R}}_{k-1})$, its own version of
receiver’s conditional state prediction pdf with conditional mean value
$\hat{x}^{\text{R}}_{k|k-1}$. Then

$\displaystyle\bar{\iota}^{\text{R}}_{k}$
$\displaystyle=\mathcal{Q}^{-1}_{k}\mathcal{Q}_{k}(y_{k}-C_{k}\hat{x}^{\text{R}}_{k|k-1}).$
(2)

This is illustrated in Figure 4.

Figure 4: Block diagram of Method R showing calculation of the innovations
signal, $\iota^{\text{R}}_{k}$. The R-Bayesian filter is detailed in Section
3.

Method S:

A modified Kalman filter recursion replaces the R-Bayesian filter of Method R
at both transmitter and receiver, yielding synchronized: predicted and
filtered state estimates $\hat{x}^{\text{S}}_{k|k-1}$ and
$\hat{x}^{\text{S}}_{k-1|k-1}$, output prediction
$\hat{y}^{\text{S}}_{k|k-1}=C_{k}\hat{x}^{\text{S}}_{k|k-1}$, using data
$\bar{I}^{\text{S}}_{k-1}.$ At the transmitter,

$\displaystyle\bar{\iota}^{\text{S}}_{k}$
$\displaystyle=\mathcal{Q}^{-1}_{k}\mathcal{Q}_{k}(y_{k}-C_{k}\hat{x}^{\text{S}}_{k|k-1}).$
(3)

The Bayesian filter and its progeny Kalman filter are presented in brief
detail in Section 2.1 for reference. Section 3 is then devoted to the
explication of the three methods in terms of their algorithms based on the
Bayesian and Kalman filter recursions. Computational comparisons are made in
Section 4 and conclusions are then provided. The central contribution of the
paper is to recognize the transmitter-side signal model underpinning Method K
and its corresponding complete Bayesian filter for the receiver side. The
subsequent reinterpretation of Methods R and S and their comparison are
separate contributions enabled by the new Bayesian computation. In the paper
we denote: $p(\cdot)$ is a general pdf; $\mathcal{N}(\mu,\Sigma)$ is the
Gaussian pdf with mean value $\mu$ covariance matrix $\Sigma$.

##  2 Background and other literature

###  2.1 The Bayesian Filter and Kalman Filter

Suppose we have a state-space system as follows.

$\displaystyle x_{k+1}$ $\displaystyle=f_{k}(x_{k},w_{k}),$ (4) $\displaystyle
z_{k}$ $\displaystyle=h_{k}(x_{k},v_{k}),$ (5)

where process noise $\\{w_{k}\\}$ and measurement noise $\\{v_{k}\\}$ are
white and of known densities and joint density. The functions
$f_{k}(\cdot,\cdot)$ and $h_{k}(\cdot,\cdot)$ are not necessarily linear time-
varying system equations. Denoting $Z_{k}=\\{z_{0},z_{1}\ldots z_{k}\\}$, the
goal of the Bayesian filter is to compute the predicted state conditional
probability density function, $p(x_{k}\Big{|}Z_{k-1}),$ and the filtered pdf,
$p(x_{k}\Big{|}Z_{k}),$ from an initial density
$p(x_{0})=p(x_{0}\Big{|}Z_{-1}).$

The Bayesian filter recursion [6] is

$\displaystyle p(x_{k}\Big{|}Z_{k-1})$
$\displaystyle=\int_{x_{k-1}}p(x_{k}\Big{|}x_{k-1})p(x_{k-1}\Big{|}Z_{k-1})dx_{k-1},$
(6) $\displaystyle p(x_{k}\Big{|}Z_{k})$
$\displaystyle=\frac{p(z_{k}\Big{|}x_{k})p(x_{k}\Big{|}Z_{k-1})}{\int_{x_{k}}p(z_{k}\Big{|}x_{k})p(x_{k}\Big{|}Z_{k-1})dx_{k}}.$
(7)

(6)

is expressed in terms of $p(x_{k-1}|Z_{k-1})$, the filtered conditional pdf
from the other half of the recursion, and $p(x_{k}|x_{k-1})$ derived from (4)
using knowledge of function $f$ and the distribution of $w_{k}$.

(7)

is expressed in terms of $p(x_{k}|Z_{k-1})$, the predicted conditional pdf
from the other half of the recursion, and $p(z_{k}|x_{k},Z_{k-1})$ derived
from (5) using knowledge of function $h$ and the distribution of $v_{k}$.

For linear Gaussian systems, the Bayesian filter yields the explicit
conditional pdf calculation of the Kalman filter. The Kalman filter, however,
is much more efficient in that it propagates the Gaussian conditional
densities via conditional means and conditional covariances only. For the
linear system,

$\displaystyle x_{k+1}$ $\displaystyle=A_{k}x_{k}+B_{k}u_{k}+w_{k},$ (8)
$\displaystyle y_{k}$ $\displaystyle=C_{k}x_{k}+v_{k},$ (9)

with noises $\\{w_{k}\\}\sim\mathcal{N}(0,Q_{k})$,
$\\{v_{k}\\}\sim\mathcal{N}(0,R_{k})$ and
$x_{0}\sim\mathcal{N}(\hat{x}_{0|-1},\Sigma_{0|-1})$ all mutually independent,
the Kalman filter recursion is

$\displaystyle L_{k}$
$\displaystyle=\Sigma^{\text{K}}_{k|k-1}C_{k}^{T}(C_{k}\Sigma^{\text{K}}_{k|k-1}C_{k}^{T}+R_{k})^{-1},$
$\displaystyle\epsilon_{k}$
$\displaystyle=y_{k}-C_{k}\hat{x}^{\text{K}}_{k|k-1}=y_{k}-\hat{y}^{\text{K}}_{k|k-1},$
$\displaystyle\hat{x}_{k|k}^{\text{K}}$
$\displaystyle=\hat{x}_{k|k-1}^{\text{K}}+L_{k}\epsilon_{k},$
$\displaystyle\Sigma_{k|k}^{\text{K}}$
$\displaystyle=(I-L_{k}C_{k})\Sigma_{k|k-1}^{\text{K}},$
$\displaystyle\Sigma_{k+1|k}^{\text{K}}$
$\displaystyle=A_{k}\Sigma_{k|k}^{\text{K}}A_{k}+Q_{k},$
$\displaystyle\hat{x}_{k+1|k}^{\text{K}}$
$\displaystyle=A_{k}\hat{x}_{k|k}^{\text{K}}+B_{k}u_{k}.$

Therefore, we see the predicted state estimation error signal,
$\tilde{x}^{\text{K}}_{k+1|k}=x_{k+1}-\hat{x}^{\text{K}}_{k+1|k}$, follows the
recursion:

$\displaystyle K_{k}$ $\displaystyle=A_{k}L_{k},$
$\displaystyle\tilde{x}_{k+1|k}^{\text{K}}$
$\displaystyle=(A_{k}-K_{k}C_{k})\tilde{x}_{k|k-1}^{\text{K}}+w_{k}-K_{k}v_{k},$
(10) $\displaystyle\epsilon_{k}$ $\displaystyle=C_{k}\tilde{x}_{k|k-1}+v_{k}.$

We appeal later to the following properties of the Kalman filter.

######  Lemma 1.

[1] The Kalman filter innovations is white and

$\epsilon_{k}\sim\mathcal{N}(0,C_{k}\Sigma^{\text{K}}_{k|k-1}C^{T}_{k}+R_{k}).$

######  Lemma 2.

[3] Subject to uniform complete detectability of $[A_{k},C_{k}]$, uniform
complete stabilizability of $[A_{k},Q_{k}]$, and the existence of $R>0$ such
that $R_{k}\geq R$, the discrete Kalman filter is exponentially asymptotically
stable.

This section has been devoted to a brief presentation of the Bayesian and
Kalman filters based on signal models (4-5) and (8-9) respectively. In the
following sections, algorithms will be presented in which the state process,
$\\{x_{k}\\}$, and measurement sequence, $\\{z_{k}\\}=\\{\bar{\iota}_{k}\\}$
or $\\{\bar{\epsilon}_{k}\\}$, will be identified with different signals on
the transmitter and receiver sides respectively. In particular, we point out
that the signal model for Method K possesses a state, which we denote
$\mathcal{Z}_{k},$ consisting of the concatenation of the plant state and the
Kalman filter state at the transmitter. It is this observation which admits
the Bayesian filter derivation in this paper.

###  2.2 Other literature

The sign-of-innovations Kalman filter (SOI-KF) was proposed by Ribeiro et al.
[2] and yields a precursor to Method S, applicable in the case of 1-bit
quantization of the innovations. Indeed, [2] appears to have been the
progenitor for this Bayesian approach to quantized innovations filtering,
hence the nomenclature for Method R derived in Section 3. The development of
the 1-bit Method S proceeds from a careful study of the Bayesian filter
leading to the statement of Method R as above but not its precise
implementation. We draw on the observations of [2] to make Method R explicit
before proceeding with the analysis of Method S for the SOI-KF and its
extension in the multi-level quantizer Kalman filter (MLQ-KF) of [10, 9]. The
advantage of the SOI-KF and MLQ-KF is that a variant of the Kalman filter
recursion is computed for approximate conditional means and covariances in
place of calculation of the entire pdf. Such reduced calculations suffice
because the state predicted conditional pdf is assumed to be Gaussian. This
assumption is evaluated (and shown to be somewhat wanting) in the
computational examples of Section 4.

Sukhavasi and Hassibi in [5] consider the Method K signal model (8-9) for
plant state $x_{k}$ and demonstrate that the conditional state density given
quantized measurements at the receiver is the density of a sum of two
independent random variables. The first random variable is the transmitter-
side Kalman filter state conditional mean estimate and the second possesses a
multivariate truncated Gaussian density involving the complete history of
previous quantized measurements. Based on this decomposition, the authors
provide a recursive estimation algorithm called Kalman-like particle filter.
The decomposition is non-recursive and the algorithm requires approximation
for implementation. The authors empirically establish its performance relative
to the SOI-KF of [2].

For the Method K problem, [4] observe that, while the predicted pdf,
$p(x_{k}|\bar{E}_{k-1}),$ can be computed from the traditional Bayesian
filtering formula (6), the filtered pdf can be computed by the following
formula.

$\displaystyle p(x_{k}\Big{|}\bar{E}_{k})$
$\displaystyle=\int_{E_{k}}p(x_{k}\Big{|}E_{k})p(E_{k}\Big{|}\bar{E}_{k})dE_{k},$
(11)

where $p(x_{k}|E_{k})$ is a Gaussian probability density function and
$p(E_{k}|\bar{E}_{k})$ is a truncated Gaussian function whose dimension
increases with time. That is, this is not a recursive formula. The integration
operation in (11) is replaced by a recursive formula based on a mid-point
approximation to the integrals.

###  2.3 Contributions of this paper

The earlier works either do not yield recursive solutions to the state
estimation problem or rely on approximation of the densities to achieve
recursion. Our approach, Method K, is both recursive and exact, subject to the
accuracy of computation of the integrals of Bayesian filtering. By the same
token, Method K relies on the computation of these integrals, while Method S
is Kalman-filter-like in its computational demands, which are significantly
more modest. The properties described in Lemmata 1 and 2 imply advantages of
Method K.

Property 1: Since $\\{\epsilon_{k}\\}$ and $\\{\bar{\epsilon}_{k}\\}$ are both
white sequences, they are well coded from the perspective that there is no
predictable component correlated with past or future data.
Property 2: Building on the coding property, since the innovations sequence is
Gaussian and of a priori known variance
$C_{k}\Sigma^{\text{K}}_{k|k-1}C_{k}+R_{k}$, the quantizer
$\mathcal{Q}_{k}(\cdot)$ can be designed to optimize performance using, for
example, the Lloyd-Max optimal quantization rule [11]. This yields maximum
entropy coding of the transmitted data.
Property 3: Under the conditions of Lemma 2, the transmitter-side Kalman
filter forgets exponentially its initial conditions. That is, the dependence
of the Kalman filter and of the receiver-side Bayesian filter on initial
conditions disappears over time. This property has been studied further for
nonlinear filters by van Handel [8, 7]. In the context here, this property
suggests that, subject to well-posedness of the Kalman filtering signal model
at the transmitter, the Method K receiver should eventually resynchronize
after an isolated error in reception. Similarly, the Bayesian filter integral
recursion should exhibit some robustness to inaccuracies in the evaluation of
the integrals.

It is not immediately evident that Method R and its descendent Method S enjoy
these same properties.

##  3 Analysis of techniques

In this section, we analyze each method under the following common
assumptions.

  1. A.1

The plant is linear and described by (8-9).

  2. A.2

Process and measurement noises $\\{w_{k}\\}\sim\mathcal{N}(0,Q_{k})$ and
$\\{v_{k}\\}\sim\mathcal{N}(0,R_{k})$ are independent and white.

  3. A.3

Initial state $x_{0}\sim\mathcal{N}(\hat{x}_{0|-1},\Sigma_{0|-1})$ is
independent from $\\{w_{k}\\}$ and $\\{v_{k}\\}$.

  4. A.4

The receiver knows: $\hat{x}_{0|-1}$, $\Sigma_{0|-1}$, and $A_{k}$, $B_{k}$,
$u_{k}$, $C_{k}$, $Q_{k}$, $R_{k}$, $\mathcal{Q}_{k}(\cdot)$,
$\mathcal{Q}^{-1}_{k}(\cdot)$, for every value of $k\geq 0.$

###  3.1 Method K of this paper

From (10), the receiver side of Method K possesses a state-space and
measurement description

$\displaystyle\mathcal{Z}_{k+1}$ $\displaystyle=\begin{bmatrix}x_{k+1}\\\
\tilde{x}_{k+1|k}^{K}\end{bmatrix}=\begin{bmatrix}A_{k}&0\\\
0&A_{k}-K_{k}C_{k}\end{bmatrix}\begin{bmatrix}x_{k}\\\
\tilde{x}_{k|k-1}^{K}\end{bmatrix}+$ $\displaystyle\hskip
71.13188pt\begin{bmatrix}B_{k}\\\ 0\end{bmatrix}u_{k}+\begin{bmatrix}I&0\\\
I&-K_{k}\end{bmatrix}\begin{bmatrix}w_{k}\\\ v_{k}\end{bmatrix},$ (12)
$\displaystyle\bar{\epsilon}_{k}$
$\displaystyle=\mathcal{Q}_{k}^{-1}\mathcal{Q}_{k}\left(\begin{bmatrix}0&C_{k}\end{bmatrix}\mathcal{Z}_{k}+\begin{bmatrix}0&I\end{bmatrix}\begin{bmatrix}w_{k}\\\
v_{k}\end{bmatrix}\right)$ (13)

By comparison to (4-5), this immediately invites the application of the
Bayesian filter (6-7) with the substitutions $\mathcal{Z}_{k}\mapsto x_{k},$
$\bar{\epsilon}_{k}\mapsto z_{k},$ and $\bar{E}_{k}\mapsto Z_{k}.$

$\displaystyle p^{\text{K}}(\mathcal{Z}_{k}\Big{|}\bar{E}_{k-1})$
$\displaystyle=\int_{\mathcal{Z}_{k-1}}{p(\mathcal{Z}_{k}\Big{|}\mathcal{Z}_{k-1})p^{\text{K}}(\mathcal{Z}_{k-1}\Big{|}\bar{E}_{k-1})d\mathcal{Z}_{k-1}},$
(14) $\displaystyle p^{\text{K}}(\mathcal{Z}_{k}\Big{|}\bar{E}_{k})$
$\displaystyle=\frac{p(\bar{\epsilon}_{k}\Big{|}\mathcal{Z}_{k})p^{\text{K}}(\mathcal{Z}_{k}\Big{|}\bar{E}_{k-1})}{\int_{\mathcal{Z}_{k}}{p(\bar{\epsilon}_{k}\Big{|}\mathcal{Z}_{k})p^{\text{K}}(\mathcal{Z}_{k}\Big{|}\bar{E}_{k-1})d\mathcal{Z}_{k}}},$
$\displaystyle=\frac{p(\bar{\epsilon}_{k}\Big{|}\mathcal{Z}_{k})p^{\text{K}}(\mathcal{Z}_{k}\Big{|}\bar{E}_{k-1})}{p(\bar{\epsilon}_{k}\Big{|}\bar{E}_{k-1})},$
$\displaystyle=\frac{p(\bar{\epsilon}_{k}\Big{|}\mathcal{Z}_{k})p^{\text{K}}(\mathcal{Z}_{k}\Big{|}\bar{E}_{k-1})}{p(\bar{\epsilon}_{k})}.$
(15)

The above equations form a recursive algorithm to compute the predicted pdf
and filtered pdf for each time. We show the detail of each term:

  1. 1.

In (14),

$\displaystyle\hskip-14.22636ptp(\mathcal{Z}_{k}\Big{|}\mathcal{Z}_{k-1})$
$\displaystyle=p\left(w_{k-1}=x_{k}-A_{k-1}x_{k-1}-B_{k-1}u_{k-1}\right)$
$\displaystyle\hskip-48.36967pt\times
p\Big{(}K_{k-1}v_{k-1}=A_{k-1}(\tilde{x}^{\text{K}}_{k-1|k-2}-x_{k-1})-B_{k-1}u_{k-1}$
$\displaystyle\hskip-28.45274pt-
K_{k-1}C_{k-1}\tilde{x}^{\text{K}}_{k-1|k-2}-(\tilde{x}^{\text{K}}_{k|k-1}-x_{k-1})\Big{)}.$
(16)

This probability can be computed from the joint Gaussian distribution of
$w_{k-1}$ and $v_{k-1}$, corresponding to Assumption A.2.

  2. 2.

In (15), the whiteness of the innovations signal $\\{\epsilon_{k}\\},$ and
therefore of $\\{\bar{\epsilon}_{k}\\},$ admits

$\displaystyle
p(\bar{\epsilon}_{k}\Big{|}\bar{E}_{k-1})=p(\bar{\epsilon}_{k})=\int_{z_{k,l}}^{z_{k,u}}p(\epsilon_{k})d\epsilon_{k},$
$\displaystyle=\int_{z_{k,l}}^{z_{k,u}}\mathcal{N}(0,C_{k}\Sigma_{k|k-1}^{\text{K}}C_{k}^{T}+R_{k})d\epsilon_{k},$
(17)

where $(z_{k,l},z_{k,u}]$ is a quantization interval at time $k$.

  3. 3.

The term $p(\bar{\epsilon}_{k}\Big{|}\mathcal{Z}_{k})$ can be computed as
follows:

$\displaystyle
p(\bar{\epsilon}_{k}\Big{|}\mathcal{Z}_{k})=\int_{z_{k,l}}^{z_{k,u}}p(\epsilon_{k}\Big{|}\mathcal{Z}_{k})d\epsilon_{k},$
$\displaystyle=\int_{z_{k,l}}^{z_{k,u}}p(\epsilon_{k}=C_{k}\tilde{x}^{\text{K}}_{k|k-1}+v_{k})d\epsilon_{k},$
$\displaystyle=\int_{z_{k,l}}^{z_{k,u}}\mathcal{N}(C_{k}\tilde{x}^{\text{K}}_{k|k-1},R_{k})d\epsilon_{k}.$
(18)

  4. 4.

The initial probability density function at the receiver

$\displaystyle p(\mathcal{Z}_{0}\Big{|}\mathcal{Z}_{-1})$
$\displaystyle=p(\mathcal{Z}_{0})=p\left(\begin{bmatrix}x_{0}\\\
\tilde{x}_{0|-1}\end{bmatrix}\right),$
$\displaystyle=\mathcal{N}\left(\begin{bmatrix}\hat{x}_{0|-1}\\\
0\end{bmatrix},\begin{bmatrix}\Sigma_{0|-1}&\Sigma_{0|-1}\\\
\Sigma_{0|-1}&\Sigma_{0|-1}\end{bmatrix}\right).$ (19)

The second equality corresponds to Assumption A.4 that the transmitter and
receiver are synchronized at time zero.

We summarize the quantized Kalman filter innovations Bayesian filter, Method
K, as follows.

Method K algorithm at the receiver.
---
Kalman filter at the transmitter
Required function: $p^{\text{K}}(\mathcal{Z}_{k-1}\Big{|}\bar{E}_{k-1}).$
1\. Compute $p(\mathcal{Z}_{k}\Big{|}\mathcal{Z}_{k-1})$ from (1).
2\. Compute predicted pdf $p^{\text{K}}(\mathcal{Z}_{k}\Big{|}\bar{E}_{k-1})$
from (14).
3\. Receive
$\bar{\epsilon}_{k}=\mathcal{Q}^{-1}_{k}\mathcal{Q}_{k}\left[y_{k}-C_{k}\hat{x}^{\text{K}}_{k|k-1}\right]$.
4\. Compute $p(\bar{\epsilon}_{k})$ from (17) and
$p(\bar{\epsilon}_{k}\Big{|}\mathcal{Z}_{k})$ from (18).
5\. Compute filtered pdf $p^{\text{K}}(\mathcal{Z}_{k}\Big{|}\bar{E}_{k})$
from (15).

###  3.2 Method R of Ribeiro et al. [2]

Method R of [2] uses the Bayesian filter (6-7) with measurement sequence
$\bar{I}^{\text{R}}_{k-1}=\\{\bar{\iota}^{\text{R}}_{k-1},\dots,\bar{\iota}^{\text{R}}_{0}\\}$
followed by a conditional mean calculation.

$\displaystyle p^{\text{R}}(x_{k}\Big{|}\bar{I}^{\text{R}}_{k-1})$
$\displaystyle=\int_{x_{k}}p(x_{k}\Big{|}x_{k-1})p^{\text{R}}(x_{k-1}\Big{|}\bar{I}^{\text{R}}_{k-1})dx_{k},$
(20) $\displaystyle p^{\text{R}}(x_{k}\Big{|}\bar{I}^{\text{R}}_{k})$
$\displaystyle=\frac{p(\bar{\iota}^{\text{R}}_{k}\Big{|}x_{k},\bar{I}^{\text{R}}_{k-1})p^{\text{R}}(x_{k}\Big{|}\bar{I}^{\text{R}}_{k-1})}{\int_{x_{k}}p(\bar{\iota}^{\text{R}}_{k}\Big{|}x_{k},\bar{I}^{\text{R}}_{k-1})p^{\text{R}}(x_{k}\Big{|}\bar{I}^{\text{R}}_{k-1})dx_{k}},$
(21) $\displaystyle\hat{x}_{k|k-1}^{R}$
$\displaystyle=E[x_{k}\Big{|}\bar{I}^{\text{R}}_{k-1}]=\int_{x_{k}}x_{k}p^{\text{R}}(x_{k}\Big{|}\bar{I}^{\text{R}}_{k-1})dx_{k}.$
(22)

System state equation (4) admits the substitution into (20)

$\displaystyle
p(x_{k}\Big{|}x_{k-1})=p(w_{k-1}=x_{k}-A_{k-1}x_{k-1}-B_{k-1}u_{k-1}),$ (23)

which forms part of (1) in Method K. Additionally for Method R, from (2) and
the quantizer intervals $(z_{k,l},z_{k,u}]$,

$\displaystyle
p(\bar{\iota}_{k}^{\text{R}}\Big{|}x_{k},\bar{I}^{\text{R}}_{k-1})$
$\displaystyle=p\left(\mathcal{Q}_{k}^{-1}\mathcal{Q}_{k}[C_{k}(x_{k}-\hat{x}^{\text{R}}_{k|k-1})+v_{k}]\Big{|}x_{k},\bar{I}^{\text{R}}_{k-1}\right),$
$\displaystyle=\int_{z_{k,l}}^{z_{k,u}}p\left(C_{k}(x_{k}-\hat{x}_{k|k-1}^{\text{R}})+v_{k}\Big{|}x_{k},\bar{I}^{\text{R}}_{k-1}\right)d\iota^{\text{R}}_{k},$
$\displaystyle=\int_{z_{k,l}}^{z_{k,u}}\mathcal{N}\left(C_{k}(x_{k}-\hat{x}_{k|k-1}^{\text{R}}),R_{k}\right)dv_{k}.$
(24) Method R of Ribeiro et al. [2] algorithm at transmitter
---
and receiver
Required function: $p^{\text{R}}(x_{k-1}\Big{|}\bar{I}_{k-1}^{\text{R}})$
1\. Compute $p(x_{k}\Big{|}x_{k-1})$ from (23).
2\. Compute predicted pdf $p^{\text{R}}(x_{k}\Big{|}\bar{I}_{k-1}^{\text{R}})$
from (20).
3\. Compute the conditional mean $\hat{x}_{k|k-1}^{\text{R}}$ from (22).
4\. Compute at the transmitter
$\bar{\iota}^{\text{R}}_{k}=\mathcal{Q}^{-1}_{k}\mathcal{Q}_{k}\left[y_{k}-C_{k}\hat{x}^{\text{R}}_{k|k-1}\right],$
and receive $\bar{\iota}^{\text{R}}_{k}$ at the receiver.
5\. Compute
$p^{\text{R}}(\bar{\iota}_{k}^{\text{R}}\Big{|}x_{k},\bar{I}^{\text{R}}_{k-1})$
from (24).
6\. Compute filtered pdf $p^{\text{R}}(x_{k}\Big{|}\bar{I}^{\text{R}}_{k})$
from (21).

###  3.3 Method S of Ribeiro et al [2] and You et al. [10]

In [2], Ribeiro et al. derive from their Method R a Kalman-filter-like
recursion based on the one-bit quantization (signum) of the associated
innovations signal, $\\{\iota^{\text{R}}_{k}\\}$ depicted in Figure 4. This is
called the sign of innovations Kalman filter (SOI-KF) and the algorithm
operates at both the transmitter and receiver. You et al. [10] extend these
ideas from one-bit quantization of SOI-KF to the multiple-level quantized
innovations Kalman filter (MLQ-KF), which we refer to as Method S. The
derivations of both works operate under the assumption that the predicted pdf
$p^{\text{S}}(x_{k+1}|\bar{I}_{k}^{\text{S}})$ is Gaussian.

Denote the normal density function and $\sqrt{\pi}/2\,\text{erfc}(x)$ as

$\phi(x)=\frac{1}{\sqrt{2\pi}}\exp(-\frac{x^{2}}{2}),\,\,\,\,\alpha_{z}=\int_{z}^{\infty}\phi(x)dx.$

Further, denote the $n^{\text{th}}$ positive level of a symmetric, mid-rise,
$N$-level, scalar quantizer by $(z_{k,l}^{n},z_{k,u}^{n}]$. Then the Kalman-
filter-like recursion of Method S/MLQ-KF for

$|\bar{\iota}_{k}|\in(z_{k,l}^{n},z_{k,u}^{n}]$

is

$\displaystyle\hat{x}_{k|k-1}^{S}$ $\displaystyle=A_{k}\hat{x}_{k-1|k-1}^{S},$
(25) $\displaystyle\Sigma_{k|k-1}^{S}$
$\displaystyle=A_{k}\Sigma_{k-1|k-1}^{S}A_{k}^{T}+Q_{k},$ (26)
$\displaystyle\hat{x}_{k|k}^{S}$
$\displaystyle=\hat{x}_{k|k-1}^{S}+\text{sgn}(\bar{\iota}_{k}^{\text{S}})\frac{\phi(z_{k,l}^{n})-\phi(z_{k,u}^{n})}{\alpha_{z_{k,l}^{n}}-\alpha_{z_{k,u}^{n}}},$
$\displaystyle\hskip
42.67912pt\times\frac{\Sigma_{k|k-1}^{S}C_{k}^{T}}{\sqrt{C_{k}\Sigma_{k|k-1}^{S}C_{k}^{T}+R_{k}}},$
(27) $\displaystyle\Sigma_{k|k}^{S}$
$\displaystyle=\Sigma_{k|k-1}^{S}-\sum_{n=1}^{N}\frac{[\phi(z_{k,l}^{n})-\phi(z_{k,u}^{n})]^{2}}{|\alpha_{z_{k,l}^{n}}-\alpha_{z_{k,u}^{n}}|},$
$\displaystyle\hskip
42.67912pt\times\frac{\Sigma_{k|k-1}^{S}C_{k}^{T}C_{k}\Sigma_{k|k-1}^{\text{S}}}{C_{k}\Sigma_{k|k-1}^{S}C_{k}^{T}+R_{k}}.$
(28)

where $\text{sgn}(\cdot)$ is the signum (or sign) function

$\displaystyle\text{sgn}(\bar{\iota}_{k}^{\text{S}})=\left\\{\begin{array}[]{ll}+1&\mbox{if
}\bar{\iota}^{\text{S}}_{k}\geq 0\\\ -1&\mbox{if
}\bar{\iota}^{\text{S}}_{k}<0\end{array}\right.$ Method S of Ribeiro et al.
[2] and You et al. [10]
---
algorithm at transmitter and receiver
Required data: $\hat{x}_{k-1|k-1}^{\text{S}}$, $\Sigma_{k-1|k-1}^{\text{S}}$
1\. Compute predicted values $\hat{x}_{k|k-1}^{\text{S}}$ and
$\Sigma_{k|k-1}^{\text{S}}$
from time-update (25-26).
2\. Compute at the transmitter
$\bar{\iota}^{\text{S}}_{k}=\mathcal{Q}^{-1}_{k}\mathcal{Q}_{k}\left[y_{k}-C_{k}\hat{x}^{\text{S}}_{k|k-1}\right],$
and receive $\bar{\iota}^{\text{S}}_{k}$ at the receiver.
3\. Compute filtered values $\hat{x}_{k|k}^{\text{S}}$ and covariance
$\Sigma_{k|k}^{\text{S}}$
from measurement update (27-28).

##  4 Comparison and examples

In this section we present a few simple simulation comparisons between the
methods, noting that Method K yields the precise conditional densities.

###  4.1 Examples

Case 1: Almost fixed state, Method K: The first example has a state which is
almost constant and well measured. It is provided to demonstrate that Method K
is sound. The system has scalar state which obeys

$\displaystyle x_{k+1}$ $\displaystyle=x_{k}+w_{k},$ $\displaystyle y_{k}$
$\displaystyle=x_{k}+v_{k},$

with $w_{k}\sim\mathcal{N}(0,0.0001),$ $v_{k}\sim\mathcal{N}(0,0.00001),$
$x_{0}\sim\mathcal{N}(0,.02).$ The 3-bit quantizer is that depicted in Figure
2.

At the transmitter Kalman filter, the innovations value is
$\epsilon_{0}=0.1160$, leading to a received quantized innovations value
$\bar{\epsilon}_{0}\in(0,0.1555]$. The reconstructed conditional pdfs are
close to truncated Gaussians with slight smoothing at the edges according to
the narrow densities of $w_{k}$ in the predictor and $v_{k}$ in the filter.
Figure 5 is the result calculated in matlab using 101 sample points for the
integrals. These computed pdfs make sense. The next few steps do not alter the
figures significantly.

Figure 5: Example 1 predicted and filtered pdfs at the receiver computed using
Bayesian filtering Method K with true innovations $\epsilon_{0}=0.1160$ and
received quantized innovations $\bar{\epsilon}_{0}\in(0,0.1555]$ via the 3-bit
quantizer in Figure 2.

Case 2: Comparison of methods: Consider the system

$\displaystyle x_{k+1}$ $\displaystyle=0.95x_{k}+w_{k},$ $\displaystyle y_{k}$
$\displaystyle=x_{k}+v_{k}.$

with $w_{k}\sim\mathcal{N}(0,0.01),$ $v_{k}\sim\mathcal{N}(0,0.01),$
$x_{0}\sim\mathcal{N}(0,.02)$ and the same $3$-bit quantizer shown in Figure
2. We compare the sequence of predicted and filtered pdfs computed using
Method K and Method R, and the predicted Gaussian pdf only from Method S/MLQ-
KF using the computed mean and covariance. These are contrasted with the
transmitter-side Kalman filter innovations Gaussian pdf. The results are
displayed in Figures 6-10.

The signals are are follows.
Time 0:

$\displaystyle x_{0}$ $\displaystyle=-0.0319,\;v_{0}=0.1117,$ $\displaystyle
w_{0}$ $\displaystyle=-0.1089,\;x_{1}=-0.1392,$
$\displaystyle\hat{x}^{\text{K}}_{0|-1}$
$\displaystyle=\hat{x}^{\text{R}}_{0|-1}=\hat{x}^{\text{S}}_{0|-1}=0,$
$\displaystyle\epsilon_{0}$
$\displaystyle=\iota_{0}^{\text{R}}=\iota^{\text{S}}_{0}=0.0798,$
$\displaystyle\bar{\epsilon}_{0}$
$\displaystyle=\bar{\iota}_{0}^{\text{R}}=\bar{\iota}^{S}_{0}\in(0,0.1555],$
$\displaystyle\hat{x}_{1|0}^{\text{S}}$
$\displaystyle=0.0085,\;\Sigma_{1|0}^{\text{S}}=0.0181.$

Time 1:

$\displaystyle x_{1}$ $\displaystyle=-0.1392,\;v_{1}=0.0033,$ $\displaystyle
w_{1}$ $\displaystyle=0.0553,\;x_{2}=-0.0770,$ $\displaystyle\epsilon_{1}$
$\displaystyle=-0.1898,\;\iota_{1}^{\text{R}}=-0.1853,$
$\displaystyle\bar{\epsilon}_{1}$
$\displaystyle=\bar{\iota}^{\text{R}}_{1}\in(-0.3111,-0.1555],$
$\displaystyle\hat{x}_{2|1}^{\text{S}}$
$\displaystyle=0.0161,\;\Sigma_{2|1}^{\text{S}}=0.0177$
$\displaystyle\hat{x}_{2|1}^{\text{K}}$
$\displaystyle=-0.0630,\;\Sigma_{2|1}^{\text{K}}=0.0156.$

Time 2:

$\displaystyle x_{2}$ $\displaystyle=-0.0770,\;v_{2}=0.1101,$ $\displaystyle
w_{2}$ $\displaystyle=0.1544,\;x_{3}=0.0813,$ $\displaystyle\epsilon_{2}$
$\displaystyle=-0.0140,\;\iota_{2}^{\text{R}}=-0.1230,$
$\displaystyle\bar{\epsilon}_{2}$
$\displaystyle=\bar{\iota}^{\text{R}}_{2}\in(-0.1555,0],$
$\displaystyle\hat{x}_{3|2}^{\text{S}}$
$\displaystyle=0.0231,\;\Sigma_{3|2}^{\text{S}}=0.0176,$
$\displaystyle\hat{x}_{3|2}^{\text{K}}$
$\displaystyle=-0.0679,\;\Sigma_{3|2}^{\text{K}}=0.0155.$

Time 3:

$\displaystyle x_{3}$ $\displaystyle=0.0813,\;v_{3}=0.0086,$ $\displaystyle
w_{3}$ $\displaystyle=-0.1492,\;x_{4}=-0.0720,$ $\displaystyle\epsilon_{3}$
$\displaystyle=0.1492,\;\iota_{3}^{\text{R}}=0.0352,$
$\displaystyle\bar{\epsilon}_{3}$
$\displaystyle=\bar{\iota}^{\text{R}}_{3}\in(0,0.1555],$
$\displaystyle\hat{x}_{4|3}^{\text{S}}$
$\displaystyle=0.0297,\;\Sigma_{4|3}^{\text{S}}=0.0175,$
$\displaystyle\hat{x}_{4|3}^{\text{K}}$
$\displaystyle=0.0216,\;\Sigma_{4|3}^{\text{K}}=0.0155.$

Time 4:

$\displaystyle x_{4}$ $\displaystyle=-0.0720,\;v_{4}=-0.0742,$
$\displaystyle\epsilon_{4}$
$\displaystyle=-0.0936,\;\iota_{4}^{\text{R}}=-0.1180,$
$\displaystyle\bar{\epsilon}_{4}$
$\displaystyle=\bar{\iota}^{\text{R}}_{4}\in(-0.1555,0],$
$\displaystyle\hat{x}_{5|4}^{\text{S}}$
$\displaystyle=0.0360,\;\Sigma_{5|4}^{\text{S}}=0.0175,$
$\displaystyle\hat{x}_{5|4}^{\text{K}}$
$\displaystyle=-0.0335,\;\Sigma_{5|4}^{\text{K}}=0.0155.$

Figure 6: Time 0: upper figure filtered state conditional pdfs for Method K
(blue) and R (red) and for the transmitter-side Kalman filter (dashed blue);
and lower figure predicted state conditional pdfs for Method K (blue) and R
(red) and Gaussian pdfs for the transmitter Kalman filter (dashed blue) and
Method S/MLQ-KF (dot-dash green).  Figure 7: Time 1: state conditional pdfs.
Figure 8: Time 2: state conditional pdfs.  Figure 9: Time 3: state conditional
pdfs.  Figure 10: Time 4: state conditional pdfs.

This second example demonstrates that the methods are workable. Although the
explicit computation of pdfs is numerically demanding compared with the moment
recursions. However, we note from this sequence of figures that Methods K and
R yield pdfs which circumscribe the transmitter-side Kalman filter pdfs. So
the actual state pdf is feasible under the estimated pdf. This appears not the
be the case for MLQ-KF at times. Methods K and R appear comparable and do
indeed produce, with these unsaturated quantized data, densities which are
roughly gaussian. Example 1 demonstrates that this is not always true,
however. If the quantizer were chosen to be Lloyd-Max optimal for Method K,
then one may conjecture that its reconstructed conditional density of the
state is itself optimal in some sense. One might then interpret Method R as
producing an approximation of the conditional pdf from Method K.

##  5 Conclusions

We have developed via Bayesian filtering an exact and recursive approach to
the reconstruction from quantized Kalman filer innovations signals of
predicted and filtered state conditional pdfs. The key observation was to
recognize the role played by the Kalman filter state in describing the
transmitter signal model. Comparisons were made to a candidate Bayesian
approach underpinning the SOI-KF derivation and to the MLQ-KF, indicating some
issues with the latter methods. Some specific advantages of the new method of
this paper were identified in terms of coding performance, optimal
quantization, and recovery from error.

## References

  * [1] B.D.O. Anderson and J.B. Moore.  Optimal Filtering.  Prentice-Hall, Englewood Cliffs, New Jersey, 1979.
  * [2] A.Ribeiro, G.B. Giannakis, and S.I. Roumeliotis.  SOI-KF: Distributed Kalman filtering with low-cost communications using the sign of innovations.  IEEE Transactions on Signal Processing, 54:4782–4795, 2006.
  * [3] J.J. Deyst and C.F. Price.  Conditions for asymptotic stability of the discrete minimum variance linear estimator.  IEEE Transactions on Automatic Control, 13(6):702–705, 1968.
  * [4] Zhansheng Duan, V. P. Jilkov, and X. Rong Li.  State estimation with quantized measurements: Approximate MMSE approach.  In 11th IEEE International Conference on Information Fusion, pages 1–6, Cologne, 2008.
  * [5] R.Sukhavasi and B. Hassibi.  The Kalman-like particle filter: Optimal estimation with quantized innovations/measurements.  IEEE Transactions on Signal Processing, 61(1), January 2013.
  * [6] D. Simon.  Optimal State Estimation: Kalman, H∞, and nonlinear approaches.  John Wiley & Sons, Hoboken NJ, 2006.
  * [7] R. van Handel.  Observability and nonlinear filtering.  Probability Theory and Related Fields, 145(35-74), 2009.
  * [8] R. van Handel.  Uniform observability of Hidden Markov Models and filter stability for unstable signals.  Annals of Applied Probability, 19(1172-1199), 2009.
  * [9] Keyou You, Lihua Xie, Shuli Sun, and Wendong Xiao.  Quantized filtering of linear stochastic systems.  Transaction of the Institute of Measurement and Control, 33(6):683–698, 2011.
  * [10] Keyou You, Linhue Xie, Shuli Sun, and Wendong Xiao.  Mutli-level quantized innovation Kalman filter.  In 17th IFAC World Congress, pages 1420–1425, Seoul, Korea, 2008\.
  * [11] Yuli You.  Audio Coding Theory and Applications.  Springer Verlag, New York, 2010.


11institutetext: Intelligent Healthcare Unit, Baidu Inc.
11email<EMAIL_ADDRESS>
22institutetext: Purdue University

# SeATrans: Learning Segmentation-Assisted diagnosis model via Transformer

Junde Wu 11 Huihui Fang 11 Fangxin Shang 11 Dalu Yang 11 Zhaowei Wang 11 Jing
Gao 22 Yehui Yang 11 Yanwu Xu (✉) 11

###### Abstract

Clinically, the accurate annotation of lesions/tissues can significantly
facilitate the disease diagnosis. For example, the segmentation of optic
disc/cup (OD/OC) on fundus image would facilitate the glaucoma diagnosis, the
segmentation of skin lesions on dermoscopic images is helpful to the melanoma
diagnosis, etc. With the advancement of deep learning techniques, a wide range
of methods proved the lesions/tissues segmentation can also facilitate the
automated disease diagnosis models. However, existing methods are limited in
the sense that they can only capture static regional correlations in the
images. Inspired by the global and dynamic nature of Vision Transformer, in
this paper, we propose Segmentation-Assisted diagnosis Transformer (SeATrans)
to transfer the segmentation knowledge to the disease diagnosis network.
Specifically, we first propose an asymmetric multi-scale interaction strategy
to correlate each single low-level diagnosis feature with multi-scale
segmentation features. Then, an effective strategy called SeA-block is adopted
to vitalize diagnosis feature via correlated segmentation features. To model
the segmentation-diagnosis interaction, SeA-block first embeds the diagnosis
feature based on the segmentation information via the encoder, and then
transfers the embedding back to the diagnosis feature space by a decoder.
Experimental results demonstrate that SeATrans surpasses a wide range of
state-of-the-art (SOTA) segmentation-assisted diagnosis methods on several
disease diagnosis tasks.

###### Keywords:

Segmentation-assisted diagnosisTransformer Classification.

##  1 Introduction

Clinically, the disease diagnosis is usually conducted based on critical
biomarkers derived from an analysis of the images. For example, on fundus
images, the vertical Cup-to-Disc Ratio (vCDR) parameter computed from the
optic cup/disc (OD/OC) masks is one of the most important clinical parameters
for the glaucoma diagnosis [12]. In melanoma diagnosis, an unusual shape of
the skin lesions is a major biomarker indicating melanoma [11]. In order to
derive these important biomarkers, an essential step is to identify lesions or
tissues in an image and segment these areas of interest from the rest of the
image [18, 9, 34].

Motivated by this observation, methods have been proposed to utilize
segmentation information to facilitate the automated disease diagnosis [10,
36, 32, 19, 4, 31, 33]. The common practices include region of interest (ROI)
extraction [10, 4], input concatenation, channel attention [19, 36], and
transfer learning [32]. These methods have two main limitations. First, the
methods proposed for specific medical tasks are not general enough. They are
often inapplicable or have unsatisfactory performance on other medical tasks.
Second, most methods simply assume that the segmentation and diagnosis
features are regional correlated, which is an invalid assumption in most
cases. Traditional techniques they applied, like convolution layers and
channel attentions are difficult to model this non-regional feature
interaction, since these tools are largely local-focused. With the rise of
vision transformer [7], such a research gap can be possibly addressed by its
global and dynamic nature [22].

In this paper, we propose a novel transformer-based model to better capture
the interaction of segmentation and diagnosis features. In order to address
the scale-level discrepancy between segmentation and diagnosis features, we
propose asymmetric multi-scale interaction to correlate multi-scale
segmentation features with each single low-level diagnosis feature. A one-to-
one coarse interaction and a one-against-rest fine-grain interaction are fused
to produce the final feature. An effective approach, called SeA-block, is
proposed to model the segmentation-diagnosis interaction, which is constructed
by an encoder-decoder pair. The encoder first embeds the diagnosis feature
through the calculated segmentation affinity map. Then a decoder maps the
embeddings back to the diagnosis feature space through the calculated
diagnosis affinity map. Through SeA-block, diagnosis features can be vitalized
by the correlated segmentation information.

In brief, we have made three major contributions. First, we propose a general
segmentation-assisted diagnosis model, named SeATrans, for integrating
segmentation and diagnosis based on medical images. Thanks to the global and
dynamic nature of transformer mechanism, SeATrans can achieve superior and
robust performance comparing with state-of-the-art counterparts. Second, we
propose asymmetric multi-scale interaction to correlate each low-level
diagnosis feature with multi-scale segmentation features. In this way,
diagnosis feature can be vitalized by both coarse and fine-grain segmentation
information. Last but not the least, we propose a new strategy, i.e., SeA-
block, for the segmentation-diagnosis interaction. A transformer-based
encoder-decoder architecture is constructed to learn across the segmentation
and diagnosis feature space. The experimental results show SeATrans
outperforms previous best method by at least a 2 % AUC over three different
disease diagnosis tasks. Meanwhile, it shows competitive robustness to the
domain shift of segmentation model.

##  2 Methodology

In this paper, we propose a general segmentation-assisted diagnosis framework.
Given a raw image $x$ and its lesions/tissues segmentation features $f_{m}$
extracted from a segmentation network (joint-trained or pre-trained), our goal
is to predict the disease $y$ (0 for benign, 1 for malignant) of the image.
Our basic idea is to integrate the segmentation information into diagnosis
model on the feature level. The interaction module and diagnosis model are
jointly optimized to predict the correct diagnosis. An illustration of the
overflow is shown in Fig. 1 (a). Raw fundus image $x$ is first sent into a
UNet to obtain the deep segmentation embedding. The segmentation features in
the UNet decoder are used to interact with the diagnosis features of a disease
diagnosis network. In the diagnosis model, convolution layer and SeA-block
based Interaction alternatively abstracts and vitalizes the features. The
final disease probability is supervised by the binary disease label through
binary cross-entropy (BCE) loss function.

Figure 1: An illustration of SeATrans framework, which starts from (a) an
overview of the processing pipeline, and continues with zoomed-in diagrams of
individual modules, including (b) the Asymmetric Multi-scale Interaction and
(c) the SeA-block.

###  2.1 Asymmetric Multi-scale Interaction

Note that the diagnosis network abstracts the low-level structure features to
the deep semantic features, while the segmentation model abstracts multi-scale
structural features. In order to align the diagnosis and segmentation
features, we correlate multi-scale segmentation features to each single low-
level diagnosis feature. As shown in Fig. 1 (b), stacked multi-scale
segmentation features are collected for a single low-level diagnosis feature.
The segmentation feature with the largest scale will first interact with the
target diagnosis feature. As large-scale feature contains more specific but
artifact structure information [29], this one-to-one interaction will produce
a coarse vitalized diagnosis feature. Other segmentation features with smaller
scales are fused together for the interaction with the diagnosis feature.
Since these features contain more fine-grained and abstract features, this
interaction will produce a fine-grained vitalized diagnosis feature. The
coarse and fine-grained features are fused by $1\times 1$ convolution layer to
produce the final result.

In practice, the second and third layers of the diagnosis model will interact
with the multi-scale features in UNet decoder. Consider the deep segmentation
feature and diagnosis feature are $f_{m}$ and $f_{d}$. To instill segmentation
information into $f^{i}_{d}\in\mathbb{R}^{\frac{H}{r}\times\frac{W}{r}\times
C}$ ($i$ is the index of layer, $H$,$W$,$r$,$C$ are height, width, down-sample
rate and channel number respectively), stacked multi-scale segmentation
features $[f^{j}_{m}]_{j=i}^{j=N}$ ($N$ is the number of layers) are collected
for the interaction. First,
$f^{i}_{m}\in\mathbb{R}^{\frac{H}{r}\times\frac{W}{r}\times C}$ will interact
with $f^{i}_{d}$ by SeA-block for coarse vitalization. Then the subsequent
segmentation features $[f^{j}_{m}]_{j=i+1}^{j=N}$ will be rearranged by pixel
shuffle [26] to the scale of $\frac{H}{r}\times\frac{W}{r}$ and concatenated
together. Then it will interact with diagnosis feature $f^{i}_{d}$ for the
fine-grained interaction. The fine-grained feature and coarse feature are
integrated by $1\times 1$ convolution kernel to obtain the final vitalized
diagnosis feature with shape $\frac{H}{r}\times\frac{W}{r}\times C$. Then a
residual convolution block [16] with pooling layer is connected to abstract
the next feature
$f_{d}^{i+1}\in\mathbb{R}^{\frac{H}{2r}\times\frac{W}{2r}\times 2C}$.

###  2.2 SeA-block

SeA-block is adopted for the segmentation-diagnosis feature interaction. The
architecture of SeA-block is shown in Fig. 1 (c). The proposed SeA-block
contains an encoder and a decoder. The encoder embeds the diagnosis feature
according to its affinity with segmentation feature, which is implemented with
the multi-head dot-product attention mechanism (MHA) [28]. Formally, consider
encoding a diagnosis feature $f_{d}$ with segmentation feature $f_{m}$, we use
$f_{m}$ as query and $f_{d}$ as key and value of the attention, which can be
formulated as:

${\rm MHA}(q,k,v)={\rm MHA}(f_{m}+E_{m},f_{d}+E_{d},f_{d}),$ (1)

where $E_{m},E_{d}$ are positional encodings [5] for segmentation feature and
diagnosis feature respectively. The features are all reshaped into a sequence
of flattened patches following ViT. In this attention mechanism, the
normalized affinity weights is first calculated between query and key to
reflect the correlation between diagnosis and segmentation feature globally.
Then the affinity weights are used to select and reinforce the diagnosis
feature through the dot production of value. After the attention, the Layer
Normalization [3] with residual connection is applied before and after the MLP
layer. The embedded diagnosis feature, which we denoted as $f_{md}$, is
outputted with the same shape as the inputs.

A decoder is connected after the encoder to map $f_{md}$ back to diagnosis
feature space. There are two inputs for the decoder, diagnosis embedding
$f_{md}$ and original diagnosis feature $f_{d}$. Being symmetrical to the
encoder, decoder is implemented by the multi-head attention with diagnosis
feature $f_{d}$ as $query$ and diagnosis embedding $f_{md}$ as $key$ and
$value$, which can be formulated as:

${\rm MHA}(q,k,v)={\rm MHA}(f_{d}+E_{d},f_{md}+E_{md},f_{md}),$ (2)

where $E_{d},E_{md}$ are positional encodings for diagnosis feature and
diagnosis embedding respectively. The decoder transfers $f_{md}$ to a
diagnosis feature by enhancing its affinity with $f_{d}$. A self-attention
block is connected after the decoder to refine the representations. The
obtained sequence will be reshaped back as a vitalized diagnosis feature
$\hat{f}_{d}$ with the same shape as $f_{d}$.

##  3 Experiment

###  3.1 Diagnosis Tasks

We evaluate SeATrans on three different disease diagnosis tasks: glaucoma
diagnosis, thyroid cancer diagnosis and melanoma diagnosis. Glaucoma is
predicted from fundus images and is assisted by OD/OC segmentation. Thyroid
cancer is predicted from ultrasound images and is assisted by the thyroid
nodule segmentation. Melanoma is predicted from dermoscopic images and is
assisted by skin lesions segmentation. The experiments of glaucoma, thyroid
cancer and melanoma diagnosis are conducted on REFUGE-2 dataset [8], TNMIX
dataset [13, 27] and ISIC dataset [15], which contain 1200, 8046, 1600
samples, respectively. The datasets are publicly available with both
segmentation and diagnosis labels. Train/validation/test sets are split
following the default settings of the dataset.

###  3.2 Experimental Settings

In our experiments, the main framework utilizes the UNet [24] architecture as
the segmentation model and ResNet50 [16] as the diagnosis model. The
segmentation network is pre-trained on heterologous data distribution. All the
experiments are implemented with the PyTorch platform and trained/tested on 4
Tesla P40 GPU with 24GB of memory. All images are uniformly resized to the
dimension of 256$\times$256 pixels. The networks are trained in an end-to-end
manner using Adam optimizer with a mini-batch of 16 for 80 epochs. The
learning rate is initially set to 1 $\times 10^{-4}$. The detailed
configurations can be found in the code.

To verify the effectiveness of SeATrans, we compare it with several baselines.
The vanilla baseline is a standard classification model implemented by
ResNet50 with no segmentation mask provided. Three other baselines are
implemented by commonly used segmentation-assisted diagnosis techniques [2],
which are denoted as ’Base-cat’, ’Base-multi’, and ’Base-ROI’, respectively.
’Base-cat’ concatenates the estimated masks with the raw images as the input
of the diagnosis model. ’Base-multi’ learns a single network for both
segmentation and diagnosis. ’Base-ROI’ crops the region of interest (ROI)
based on the estimated segmentation masks.

In order to verify the generalization of the models, we train segmentation
network on homologous (-homo) and heterologous (-hetero) data, respectively.
’-homo’ means segmentation and diagnosis network are trained on the same
source of data. ’-hetero’ means segmentation model is trained on an external
dataset, which is RIGA [1], DDIT [23] and PH2 [21] for glaucoma, thyroid
cancer and melanoma diagnosis, respectively.

###  3.3 Main Results

Comparing SeATrans with baselines in Table 1, we can see significant
improvement on all three diagnosis tasks. Concretely, comparing with the best
baseline by AUC, SeATrans improves 6.56%, 6.78% and 8.14% on glaucoma, thyroid
cancer and melanoma diagnosis respectively, indicating SeATrans can gain
general and considerable improvement comparing with the present commonly used
techniques. SeATrans also achieves the highest sensitivity with competitive
accuracy and specificity, indicating it is more applicable to the real
clinical scenarios, since sensitivity is commonly of great concern in clinical
scenes.

Comparing vanilla baseline with the other methods, we can see except ’Base-
multi’, the segmentation more or less improves the diagnosis performance. It
demonstrates the segmentation information of lesions/tissues is definitely
useful for the automated diagnosis models. However, the improvement it can
bring depends largely on the way we use it. Multi-task learning based methods
seemed to be invalid according to our experimental results. This may be due to
the large discrepancy between segmentation and diagnosis features. The
segmentation encoder extract the low-level structure features while the
diagnosis needs the high-level semantic features, it is thus hard to learn the
universal features in one encoder. SeATrans fuses the multi-scale segmentation
features to first few layers of the diagnosis model. In this way, these
structure-focused layers are enhanced by the awareness of lesions/tissues
structures, and the later layers can still abstract the high-level diagnosis
feature. As a result, SeATrans outperforms the other segmentation-assisted
diagnosis methods by a large margin.

Table 1: Comparing with the baselines. Accuracy, specificity, sensitivity and
AUC (%) are measured on three different diagnosis tasks. Segmentation model
performance measured by Dice score (%) is also reported.

Tasks | Glaucoma | Thyroid Cancer | Melanoma
---|---|---|---
Metrics | $\mathcal{D}_{disc}$ | $\mathcal{D}_{cup}$ | ACC | SPE | SEN | AUC | Dice | ACC | SPE | SEN | AUC | Dice | ACC | SPE | SEN | AUC
No Mask | Baseline | - | - | 82.95 | 94.06 | 37.97 | 77.29 | - | 79.29 | 93.75 | 68.62 | 77.08 | - | 78.53 | 92.23 | 22.22 | 72.49
-homo | Base-cat | 94.73 | 81.77 | 78.95 | 82.18 | 65.82 | 81.91 | 86.76 | 81.74 | 95.42 | 70.76 | 80.06 | 82.35 | 77.72 | 87.84 | 36.11 | 76.42
Base-multi | 94.73 | 81.77 | 82.20 | 92.50 | 40.50 | 74.73 | 86.76 | 77.25 | 84.58 | 65.75 | 72.37 | 82.35 | 80.98 | 95.27 | 22.22 | 71.72
Base-ROI | 94.73 | 81.77 | 75.18 | 77.18 | 67.08 | 79.88 | 86.76 | 83.25 | 90.26 | 74.92 | 77.50 | 82.35 | 79.35 | 90.88 | 31.94 | 72.70
SeATrans | 94.73 | 81.77 | 86.96 | 90.93 | 70.88 | 88.47 | 86.76 | 85.54 | 91.75 | 78.84 | 86.84 | 82.35 | 85.56 | 93.77 | 62.74 | 84.56
-hetero | Base-cat | 94.60 | 78.31 | 82.95 | 92.19 | 50.63 | 80.70 | 85.17 | 82.13 | 88.93 | 73.58 | 80.15 | 80.07 | 79.89 | 92.91 | 26.39 | 74.41
Base-multi | 94.60 | 78.31 | 76.19 | 90.31 | 55.70 | 72.40 | 85.17 | 81.35 | 94.81 | 67.79 | 72.21 | 80.07 | 80.98 | 95.27 | 22.22 | 68.70
Base-ROI | 94.60 | 78.31 | 83.20 | 94.06 | 39.24 | 77.52 | 85.17 | 82.39 | 87.74 | 78.65 | 76.73 | 80.07 | 72.28 | 77.03 | 52.78 | 72.22
SeATrans | 94.60 | 78.31 | 80.20 | 80.62 | 78.48 | 87.61 | 85.17 | 84.60 | 90.26 | 80.45 | 86.23 | 80.07 | 84.42 | 87.20 | 57.35 | 83.16

To verify the generalization of the methods, we also conduct the experiment on
heterologous data, where the segmentation model is pre-trained on external
dataset. Due to the domain shift, the segmentation masks/features would be
inferior to ’-homo’, thus disturb the diagnosis models. Comparing ’-homo’ with
’-hetero’, we can see a drop on the AUC performance over all of the methods.
But SeATrans shows very competitive generalization ability, dropping only
about 1% AUC on ’-hetero’.

###  3.4 Comparing with SOTA

To demonstrate the advantage of SeATrans, we compare it with SOTA methods for
segmentation-assisted diagnosis. Table 2 quantitatively compare SeATrans with
nine SOTA segmentation-assisted diagnosis methods.

SeATrans vs Transformers. Present SOTA transformer-based diagnosis
architectures: ConViT [6] and Swin Transformer [20] are involved for the
comparison. Segmentation masks are concatenated as the inputs of the models.
It shows SeATrans clearly outperforms these transformer architectures,
increases about 5.60%, 5.82% and 7.10% AUC on glaucoma, thyroid and melanoma,
respectively. It demonstrates a large proportion of the improvement comes from
the proposed feature fusion strategy, but not the transformer-like
architecture.

SeATrans vs ROI. We compare SeATrans with ROI based segmentation-assisted
diagnosis methods: DualStage [4] and DENet [10]. It shows [4] only gains
marginal improvement compared with vanilla baseline. Although [10] achieves
better performance, it is only applicable on glaucoma diagnosis. SeATrans
outperforms ROI based methods by an average 4% AUC on a range of tasks.

SeATrans vs Channel Attention. We also compare SeATrans with SOTA channel
attention based segmentation-assisted diagnosis methods: AGCNN [19] and ColNet
[36], who adopted channel-attention to enhance the diagnosis feature by the
segmentation masks/features. We observe that SeATrans can surpass AGCNN and
ColNet by 6.31% and 3.10% AUC on glaucoma, 3.99% and 2.40% on thyroid
cancer,and 4.39% and 3.84% on melanoma diagnosis, indicating the superiority
of SeATrans comparing with regional-correlated channel attention.

SeATrans vs Multi-task. Multi-task learning methods MagNet [14] and CMSNET
[37] are involved for the comparison. SeATrans consistently outperforms both
methods, especially on thyroid cancer diagnosis, which outperforms MagNet and
CMSVNET by 11.16% and 10.13% AUC respectively.

SeATrans vs Transfer-learning. L2T-KT [32] uniquely processed the task by
teacher-student based transfer learning and achieved competitive performance.
Comparing the AUC, SeATrans outperforms L2T-KT by 2.23%, 2.55% and 2.66% on
glaucoma, thyroid cancer and melanoma diagnosis, respectively. SeATrans also
achieves better sensitivity-speficity trade-off than L2T-KT. For example,
SeATrans achieves 79.66% F1 score which surpasses 77.43% F1 score of L2T-KT on
glaucoma diagnosis.

Heterologous data Generalization. Comparing with ’-homo’ and ’-hetero’, we can
see ROI-based methods (Dual-stage, DENet) show the best generalization, since
they used less segmentation information than the others. SeATrans and
Transformer-based methods (ConViT, Swin) also show competitive generalization
capability, which drop only about 1% AUC on a range of tasks. Channel-
attention based methods (AGCNN, ColNet) are more sensitive since their
regional correlated assumption is vulnerable to the domain shift. Thanks to
the dynamic and global nature of SeATrans, it gains high performance with very
competitive generalization ability comparing with the other methods.

Table 2: Comparing with SOTA segmentation-assisted diagnosis methods.
Accuracy, specificity, sensitivity and AUC (%) are measured on three different
diagnosis tasks.

|  | Glaucoma | Thyroid Cancer | Melanoma
---|---|---|---|---
|  | ACC | SPE | SEN | AUC | ACC | SPE | SEN | AUC | ACC | SPE | SEN | AUC
-homo | ConViT [6] | 80.45 | 86.56 | 55.69 | 82.87 | 80.85 | 90.31 | 64.67 | 81.02 | 79.89 | 90.87 | 34.72 | 77.46
Swin [20] | 81.95 | 91.56 | 43.03 | 82.32 | 82.76 | 85.70 | 73.82 | 80.34 | 80.76 | 89.11 | 47.29 | 76.75
DualStage [4] | 80.20 | 90.31 | 39.24 | 80.37 | 79.56 | 85.64 | 70.93 | 77.15 | 78.26 | 87.84 | 38.89 | 72.34
DENet [10] | 80.04 | 85.00 | 59.49 | 84.70 | - | - | - | - | - | - | - | -
AGCNN [19] | 81.20 | 89.68 | 41.77 | 82.16 | 84.78 | 88.69 | 71.05 | 82.85 | 82.60 | 92.20 | 43.83 | 80.17
ColNet [36] | 79.69 | 79.69 | 79.74 | 85.36 | 87.60 | 94.08 | 72.47 | 84.43 | 85.21 | 98.31 | 30.98 | 80.72
MagNet [14] | 83.20 | 94.06 | 39.24 | 77.52 | 78.91 | 86.71 | 69.25 | 75.68 | 75.54 | 81.08 | 52.77 | 71.77
CMSNET [37] | 64.16 | 73.41 | 61.85 | 80.86 | 74.52 | 87.03 | 67.32 | 76.71 | 82.88 | 98.32 | 17.14 | 78.55
L2T-KT [32] | 80.20 | 80.62 | 78.48 | 86.24 | 81.49 | 90.31 | 75.59 | 84.29 | 79.34 | 84.69 | 58.10 | 81.90
SeATrans | 86.96 | 90.93 | 70.88 | 88.47 | 85.54 | 91.75 | 78.84 | 86.84 | 85.59 | 93.77 | 62.74 | 84.56
-hetero | ConViT [6] | 80.45 | 91.56 | 35.44 | 82.37 | 77.46 | 93.75 | 60.11 | 81.02 | 79.34 | 85.42 | 54.79 | 76.55
Swin [20] | 72.18 | 83.54 | 69.37 | 81.85 | 80.34 | 83.45 | 76.18 | 80.34 | 79.89 | 87.11 | 50.68 | 75.28
DualStage [4] | 75.18 | 67.08 | 77.18 | 80.22 | 80.70 | 88.69 | 75.71 | 77.15 | 81.79 | 92.56 | 37.50 | 72.63
DENet [10] | 78.94 | 84.81 | 77.50 | 84.12 | - | - | - | - | - | - | - | -
AGCNN [19] | 66.16 | 62.81 | 79.74 | 80.94 | 82.29 | 93.41 | 70.58 | 82.85 | 76.08 | 78.98 | 64.38 | 77.38
ColNet [36] | 61.40 | 97.46 | 52.50 | 82.78 | 84.93 | 91.19 | 73.82 | 84.43 | 82.33 | 93.91 | 34.72 | 77.95
MagNet [14] | 70.42 | 70.31 | 70.88 | 75.44 | 78.15 | 87.74 | 77.36 | 75.68 | 79.34 | 90.50 | 33.33 | 69.67
CMSNET [37] | 60.15 | 77.21 | 55.93 | 78.17 | 81.75 | 84.60 | 78.14 | 76.71 | 72.82 | 73.73 | 69.01 | 76.39
L2T-KT [32] | 79.95 | 85.00 | 59.49 | 84.98 | 83.89 | 94.04 | 76.18 | 84.29 | 82.06 | 88.51 | 55.56 | 80.75
SeATrans | 80.70 | 80.62 | 78.48 | 87.61 | 84.60 | 90.26 | 80.45 | 86.23 | 84.42 | 87.20 | 57.35 | 83.16

###  3.5 Ablation study

Ablation studies are performed over each component of SeATrans, including
multi-scale, asymmetric interaction and SeA-Block, as listed in Table 3. The
experiments are conducted on glaucoma diagnosis task. Feature concatenation is
adopted to replace SeA-block. In Table 3, as we sequentially adding the
proposed modules on vanilla baseline, the model performance is gradually
improved. First, by applying multi-scale segmentation-diagnosis integration,
the AUC value is increased by 2% on homologous data while only 0.6% on
heterologous data. This indicates that multi-scale integration can improve the
diagnosis performance with limited generalization. Then, the asymmetric multi-
scale interaction is applied to further focus the integration on the low-level
features, which boosts the AUC by a 3.53 % and a 3.42% on ’-homo’ and
’-hetero’ respectively. Finally, SeA-Block is utilized for the segmentation-
diagnosis interaction. It can be observed the diagnosis performance is
remarkably improved, which gains 5.09% and 6.34% AUC improvement on ’-homo’
and ’-hetero’, respectively. It indicates SeA-block gains significant and
general improvement by its dynamic and global interaction.

Table 3: Ablation study on glaucoma diagnosis task. The diagnosis performance is measured by AUC (%) Multi-scale | Asymmetric | SeA-block | -homo | -hetero
---|---|---|---|---
|  |  | 77.29 | 77.29
✓ |  |  | 79.85 | 77.84
✓ | ✓ |  | 83.38 | 81.27
✓ | ✓ | ✓ | 88.47 | 87.61

##  4 Conclusion

In this work, we proposed SeATrans to overcome the shortcomings of existing
segmentation-assisted diagnosis models. In SeATrans, asymmetric multi-scale
interaction is proposed to address the segmentation-diagnosis scale level
discrepancy. Then SeA-block is constructed for the global and dynamic feature
interaction between segmentation and diagnosis space. Extensive empirical
experiments demonstrated the general and superior performance of the proposed
SeATrans on a range of medical image diagnosis tasks.

## References

  * [1] Almazroa, A., Alodhayb, S., Osman, E., Ramadan, E., Hummadi, M., Dlaim, M., Alkatee, M., Raahemifar, K., Lakshminarayanan, V.: Agreement among ophthalmologists in marking the optic disc and optic cup in fundus images. International ophthalmology 37(3), 701–717 (2017)
  * [2] Anwar, S.M., Majid, M., Qayyum, A., Awais, M., Alnowami, M., Khan, M.K.: Medical image analysis using convolutional neural networks: a review. Journal of medical systems 42(11), 1–13 (2018)
  * [3] Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv preprint arXiv:1607.06450 (2016)
  * [4] Bajwa, M.N., Malik, M.I., Siddiqui, S.A., Dengel, A., Shafait, F., Neumeier, W., Ahmed, S.: Two-stage framework for optic disc localization and glaucoma classification in retinal fundus images using deep learning. BMC medical informatics and decision making 19(1), 1–16 (2019)
  * [5] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: European Conference on Computer Vision. pp. 213–229. Springer (2020)
  * [6] d’Ascoli, S., Touvron, H., Leavitt, M., Morcos, A., Biroli, G., Sagun, L.: Convit: Improving vision transformers with soft convolutional inductive biases. arXiv preprint arXiv:2103.10697 (2021)
  * [7] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * [8] Fang, H., Li, F., Fu, H., Sun, X., Cao, X., Son, J., Yu, S., Zhang, M., Yuan, C., Bian, C., et al.: Refuge2 challenge: Treasure for multi-domain learning in glaucoma assessment. arXiv preprint arXiv:2202.08994 (2022)
  * [9] Fu, H., Cheng, J., Xu, Y., Wong, D.W.K., Liu, J., Cao, X.: Joint optic disc and cup segmentation based on multi-label deep network and polar transformation. IEEE transactions on medical imaging 37(7), 1597–1605 (2018)
  * [10] Fu, H., Cheng, J., Xu, Y., Zhang, C., Wong, D.W.K., Liu, J., Cao, X.: Disc-aware ensemble network for glaucoma screening from fundus image. IEEE transactions on medical imaging 37(11), 2493–2501 (2018)
  * [11] Gachon, J., Beaulieu, P., Sei, J.F., Gouvernet, J., Claudel, J.P., Lemaitre, M., Richard, M.A., Grob, J.J.: First prospective study of the recognition process of melanoma in dermatological practice. Archives of dermatology 141(4), 434–438 (2005)
  * [12] Garway-Heath, D.F., Ruben, S.T., Viswanathan, A., Hitchings, R.A.: Vertical cup/disc ratio in relation to optic disc size: its value in the assessment of the glaucoma suspect. British Journal of Ophthalmology 82(10), 1118–1124 (1998)
  * [13] Gong, H., Chen, G., Wang, R., Xie, X., Mao, M., Yu, Y., Chen, F., Li, G.: Multi-task learning for thyroid nodule segmentation with thyroid region prior. In: 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI). pp. 257–261. IEEE (2021)
  * [14] Gupta, S., Punn, N.S., Sonbhadra, S.K., Agarwal, S.: Mag-net: Multi-task attention guided network for brain tumor segmentation and classification. In: International Conference on Big Data Analytics. pp. 3–15. Springer (2021)
  * [15] Gutman, D., Codella, N.C., Celebi, E., Helba, B., Marchetti, M., Mishra, N., Halpern, A.: Skin lesion analysis toward melanoma detection: A challenge at the international symposium on biomedical imaging (isbi) 2016, hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv:1605.01397 (2016)
  * [16] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [17] Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., Madry, A.: Adversarial examples are not bugs, they are features. arXiv preprint arXiv:1905.02175 (2019)
  * [18] Ji, W., Yu, S., Wu, J., Ma, K., Bian, C., Bi, Q., Li, J., Liu, H., Cheng, L., Zheng, Y.: Learning calibrated medical image segmentation via multi-rater agreement modeling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12341–12351 (2021)
  * [19] Li, L., Xu, M., Wang, X., Jiang, L., Liu, H.: Attention based glaucoma detection: A large-scale database and cnn model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10571–10580 (2019)
  * [20] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10012–10022 (2021)
  * [21] Mendonça, T., Ferreira, P.M., Marques, J.S., Marcal, A.R., Rozeira, J.: Ph 2-a dermoscopic image database for research and benchmarking. In: 2013 35th annual international conference of the IEEE engineering in medicine and biology society (EMBC). pp. 5437–5440. IEEE (2013)
  * [22] Naseer, M.M., Ranasinghe, K., Khan, S.H., Hayat, M., Shahbaz Khan, F., Yang, M.H.: Intriguing properties of vision transformers. Advances in Neural Information Processing Systems 34 (2021)
  * [23] Pedraza, L., Vargas, C., Narváez, F., Durán, O., Muñoz, E., Romero, E.: An open access thyroid ultrasound image database. In: 10th International Symposium on Medical Information Processing and Analysis. vol. 9287, p. 92870W. International Society for Optics and Photonics (2015)
  * [24] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. pp. 234–241. Springer (2015)
  * [25] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision. pp. 618–626 (2017)
  * [26] Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1874–1883 (2016)
  * [27] Shusharina, M., Heinrich, R., Huang: Segmentation, classification, and registration of multi-modality medical imaging data (2020)
  * [28] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in neural information processing systems. pp. 5998–6008 (2017)
  * [29] Wojna, Z., Ferrari, V., Guadarrama, S., Silberman, N., Chen, L.C., Fathi, A., Uijlings, J.: The devil is in the decoder. In: British Machine Vision Conference 2017, BMVC 2017. pp. 1–13. BMVA Press (2017)
  * [30] Wu, J.: Generating adversarial examples in the harsh conditions. arXiv preprint arXiv:1908.11332 (2019)
  * [31] Wu, J., Fang, H., Li, F., Fu, H., Lin, F., Li, J., Huang, L., Yu, Q., Song, S., Xu, X., et al.: Gamma challenge: glaucoma grading from multi-modality images. arXiv preprint arXiv:2202.06511 (2022)
  * [32] Wu, J., Yu, S., Chen, W., Ma, K., Fu, R., Liu, H., Di, X., Zheng, Y.: Leveraging undiagnosed data for glaucoma classification with teacher-student learning. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 731–740. Springer (2020)
  * [33] Yang, Y., Fangxin, S., Binghong, W., Dalu, Y., Lei, W., Xu, Y., Zhang, W., Zhang, T.: Robust collaborative learning of patch-level and image-level annotations for diabetic retinopathy grading from fundus image. IEEE Transactions on Cybernetics pp. 1–11 (2021)
  * [34] Yuan, Y., Chao, M., Lo, Y.C.: Automatic skin lesion segmentation using deep fully convolutional networks with jaccard distance. IEEE transactions on medical imaging 36(9), 1876–1886 (2017)
  * [35] Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L., Jordan, M.: Theoretically principled trade-off between robustness and accuracy. In: International conference on machine learning. pp. 7472–7482. PMLR (2019)
  * [36] Zhou, Y., He, X., Huang, L., Liu, L., Zhu, F., Cui, S., Shao, L.: Collaborative learning of semi-supervised segmentation and classification for medical images. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2079–2088 (2019)
  * [37] Zhou, Y., Chen, H., Li, Y., Liu, Q., Xu, X., Wang, S., Yap, P.T., Shen, D.: Multi-task learning for segmentation and classification of tumors in 3d automated breast ultrasound images. Medical Image Analysis 70, 101918 (2021)

##  5 Supplementary material

In order to further analysis the interrelation of the segmentation and the
diagnosis. We adopt the network explanation techniques on the models to
visualize the discriminative features. Grad-CAM[25] is a commonly used
explanation tool that produces visual explanations for model decisions. It
visualize the gradients of the loss function with pixel-wise weighted feature
maps. We compare the Grad-CAM produced visualization results on an glaucoma
diagnosis example in Fig. 2.

Figure 2: Visualization results compared with the other methods on fundus
images based glaucoma diagnosis. Grad-CAM is adopted to show the attentive
regions for the diagnosis.

We can see ROI based methods (DENet and DualStage) and Transformer based
methods (Swin and ConViT) show less attention on the clinical focused region,
like optic cup. It may because these methods impose the segmentation
enhancement on the model inputs rather than the deep features. Although the
explanation is not so good, some of these models with sophisticated network
structures still achieve fine diagnosis performance, like Swin, ConViT and
DENet. Some of the recent literature also show that the sophisticated networks
will show stronger capability but inferior explanation[35, 30, 17], since they
are prone to learn some features that discriminative to the networks while
meaningless to the human[17]. Multi-task based methods (MagNet and CMSNET) and
channel attention based methods (AGCNN and ColNet) mainly focus on the optic-
cup region, which is important for the clinical glaucoma diagnosis. But most
of them are not implemented with sufficient learnable parameters, which cause
they show inferior diagnosis performance. Transfer-learning based method
(L2T-KT) and proposed SeATrans pay more attention on the optic-cup region.
Besides optic-cup region, SeATrans also focuses on the gap between OC and OD
boundary, which is another important parameter indicating glaucoma suspect
clinically. Such visualization results demonstrate SeATrans can reach superior
diagnosis performance with clear and reasonable explanation.


# New Proof of the Equation  $\sum_{k=1}^{\infty}\frac{\mu(k)}{k}=0.$

Edmund Landau
Translated by Michael J. Coons
Preliminary Version: September 2007

[Neuer Beweis der Gleichung $\sum\mu(k)/k=0$,
Inaugural-Dissertation, Berlin, 1899.]

Translation © M. J. Coons 2007.

New Proof of the Equation

$\sum_{k=1}^{\infty}\frac{\mu(k)}{k}=0.$

Inaugural–Dissertation
for the

acquisition of the title of Doctor from the Faculty of Philosophy

of

Friedrich–Wilhelms University in Berlin
defended publicly and approved together with the attached theses

on 15 July, 1899

by

Edmund Landau

of Berlin

Opponents:

Mr. Rudolf Zeigel, Student of Mathematics

Mr. Fritz Hartoge, Student of Mathematics

Ernst Steinitz, Ph.D. , Privatdozent at the Royal Technical high school in
Charlottenburg.

Berlin 1899.

For my dear parents.

The function $\mu(k)$ is normally defined as the number–theoretic function for
which

  1. 1.

$\mu(1)=1$,

  2. 2.

$\mu(k)=0$ when $k>1$ is divisible by a square,

  3. 3.

$\mu(k)=(-1)^{r}$ when $k$ is the product of $r$ distinct primes.

This statement was first expressed by Euler, that

$\sum_{k=1}^{\infty}\frac{\mu(k)}{k}=0$

holds; that is, $\lim_{x\to\infty}\sum_{k=1}^{x}\frac{\mu(k)}{k}$ exists and
equals $0$, the recent proof of which is due to von Mangoldt111) “Beweis der
Gleichung $\sum_{k=1}^{\infty}\frac{\mu(k)}{k}=0$”; Proceedings of the Royal
Prussian Academy of Science of Berlin, 1897, pp. 835–852.). The same goes for
the investigations of Hadamard and de la Vallée Poussin over the Riemann
$\zeta$–function, and it seems also, that without the use of these works, the
present means of analysis is not enough to give a proof of Euler’s statement.
However, if one expects the results of those investigations to be in agreement
with those of von Mangoldt, then one, as will be executed in the following,
can arrive at the target along a quite short path.

The proof, which forms the contents of this dissertation, uses first the
theorem222) This theorem is proven without the use of von Mangoldt’s proof.)
of Hadamard333) Bulletin de la société mathématique de France, Volume 24,
1896, p. 217.) and de la Vallée Poussin444) Annales de la société scientifique
de Bruxelles, Volume 20, Part 2, p. 251.):

“If $\vartheta(x):=\sum_{p\leq x}\log p$, then
$\lim_{x\to\infty}\frac{\vartheta(x)}{x}=1.$” However, apart from the use of
this theorem, it is as elementary as can be for such a transcendent statement.

## 1

Denote by $g(x)$ the sum $\sum_{k=1}^{[x]}\frac{\mu(k)}{k}$, where $[x]$
denotes the greatest integer less than or equal to $x$; more simply we write

$g(x)=\sum_{k=1}^{x}\frac{\mu(k)}{k},$ (1)

where $k$ ranges over all positive integers less than or equal to $x$. The sum
has meaning only for $x\geq 1$; thus, for $x<1$, set $g(x)=0$.

With the above notation, we read the two lemmas, which von Mangoldt proves in
a simple way at the start of his paper555) 1. c., pp. 837–839.) and which are
also applied in the following one, as:

For all $x$

$|g(x)|\leq 1\ )$ (2)

and for all $x\geq 1$

$\left|\log x\cdot g(x)-\sum_{k=1}^{x}\frac{\mu(k)\log k}{k}\right|\leq
3+\gamma$ (3)

where $\gamma$ denotes Euler’s constant.

The inequality (3), which von Mangoldt only derived in order to apply it in a
certain place in his proof777) 1. c., p. 843.), serves in the following one as
the basis of the whole investigation.

Concerning the sum $\sum_{k=1}^{x}\frac{\mu(k)\log k}{k}$, Möbuis888) “Über
eine besondere Art von Umkehrung Reihen,” Journal für die reine und angewandte
Mathematik, Volume 9, p. 122.) believed he had proved that for sufficiently
large $x$, its difference from $-1$ is arbitrarily small: however, his proof
is not sound. Although new writers consider it probable999) E.g., Mertens
proved, which the general validity of the inequality condition assumes:
$\left|\sum_{k=1}^{x}\mu(k)\right|\leq\sqrt{x},$ that this theorem is correct,
if this relation is generally fulfilled (Proceedings of the Vienna Academy,
math.-nat. Kl., Volume 106, Dept. 2a, p. 774.)) that

$\lim_{x=\infty}\sum_{k=1}^{x}\frac{\mu(k)\log k}{k}$

exists and equals $-1$, it has yet to be proven that for all $x$,
$\sum_{k=1}^{x}\frac{\mu(k)\log k}{k}$ is contained between two finite
boundaries. Now since (3) yields

$\left|g(x)-\frac{1}{\log x}\sum_{k=1}^{x}\frac{\mu(k)\log
k}{k}\right|\leq\frac{3+\gamma}{\log x},$

it follows, with use of the Euler–v. Mangoldt Theorem, that

$\lim_{x=\infty}g(x)=0$

so that

$\frac{1}{\log x}\sum_{k=1}^{x}\frac{\mu(k)\log k}{k}$

approaches $0$ as $x\to\infty$.

If, in reverse, it was successfully proven that

$\lim_{x=\infty}\frac{1}{\log x}\sum_{k=1}^{x}\frac{\mu(k)\log k}{k}$

exists and equals 0, then one would trivially have that

$\sum_{k=1}^{x}\frac{\mu(k)}{k}=0,$

since for any $\delta$ there is a $G$ such that for all $x\geq G$

$\left|\frac{1}{\log x}\sum_{k=1}^{x}\frac{\mu(k)\log
k}{k}\right|\leq\frac{\delta}{2}$

and

$0<\frac{3+\gamma}{\log x}\leq\frac{\delta}{2},$

thus it follows for $x\geq G$:

$\displaystyle|g(x)|$ $\displaystyle=\left|\left(g(x)-\frac{1}{\log
x}\sum_{k=1}^{x}\frac{\mu(k)\log k}{k}\right)+\frac{1}{\log
x}\sum_{k=1}^{x}\frac{\mu(k)\log k}{k}\right|$
$\displaystyle\leq\left|\left(g(x)-\frac{1}{\log
x}\sum_{k=1}^{x}\frac{\mu(k)\log k}{k}\right)\right|+\left|\frac{1}{\log
x}\sum_{k=1}^{x}\frac{\mu(k)\log k}{k}\right|$
$\displaystyle\leq\frac{3+\gamma}{\log x}+\left|\frac{1}{\log
x}\sum_{k=1}^{x}\frac{\mu(k)\log k}{k}\right|$
$\displaystyle\leq\frac{\delta}{2}+\frac{\delta}{2}=\delta,$

also

$\lim_{x=\infty}g(x)=0.$

The proof, that for

$f(x)=\sum_{k=1}^{x}\frac{\mu(k)\log k}{k},$ (4)

one has

$\lim_{x=\infty}\frac{f(x)}{\log x}=0$ (5)

will be supplied in what follows.

In order not to have to interrupt the course of the investigation, we note the
following simple lemma, which was already known to Gram101010) 1. c., p. 197,
where separately for all valid $r$ in equation (43) set $r=1$.): it is

$\sum_{\nu=1}^{x}\frac{1}{\nu}g\left(\frac{x}{\nu}\right)=g(x)+\frac{1}{2}g\left(\frac{x}{2}\right)+\frac{1}{3}g\left(\frac{x}{3}\right)+\cdots+\frac{1}{[x]}g\left(\frac{x}{[x]}\right)=1.$
(6)

The $\nu$–th summand $\frac{1}{\nu}g\left(\frac{x}{\nu}\right)$ contains the
sum of the terms

$\frac{1}{\nu}\frac{\mu(1)}{1}=\frac{\mu(1)}{\nu},\
\frac{1}{\nu}\frac{\mu(2)}{2}=\frac{\mu(2)}{2\nu},\cdots,\
\frac{1}{\nu}\frac{\mu(n)}{n}=\frac{\mu(n)}{n\nu},\cdots,\
\frac{1}{\nu}\frac{\mu\left[\frac{x}{\nu}\right]}{\left[\frac{x}{\nu}\right]}=\frac{\mu\left[\frac{x}{\nu}\right]}{\left[\frac{x}{\nu}\right]\nu};$

the sum $\sum_{\nu=1}^{x}\frac{1}{\nu}g\left(\frac{x}{\nu}\right)$ consists of
terms of the form $\frac{\mu(n)}{t}$, where $n$ is a divisor of $t$, and $t$
runs through the integers from 1 to $[x]$; that is,

$\sum_{\nu=1}^{x}\frac{1}{\nu}g\left(\frac{x}{\nu}\right)=\sum_{t=1}^{x}\frac{1}{t}\sum_{n|t}\mu(n);$

now since $\sum_{n|t}\mu(n)$ is 1 for $t=1$ and $0$ otherwise, we have

$\sum_{\nu=1}^{x}\frac{1}{\nu}g\left(\frac{x}{\nu}\right)=1.$

## 2

If one lets $x=p_{1}p_{2}\cdots p_{r}$ in the defining equation (4) of $f(x)$,
replaces111111) The $\log k$ is multiplied by the factor $\frac{\mu(k)}{k}$,
which occurs only for $k$ that are the product of distinct primes.) $\log(x)$
by $\log p_{1}+\cdots+\log p_{r}$ and gathers like terms in which the
logarithm is applied to the same prime number, then (4) becomes an equation of
the form

$f(x)=\sum F(p,x)\log p$

where the sum extends over all prime numbers $p\leq x$. As was easily given by
von Mangoldt121212) 1. c., p. 840.) for another purpose,

$F(p,x)=-\left(\frac{1}{p}\sum_{k=1}^{\frac{x}{p}}\frac{\mu(k)}{k}+\frac{1}{p^{2}}\sum_{k=1}^{\frac{x}{p^{2}}}\frac{\mu(k)}{k}+\frac{1}{p^{3}}\sum_{k=1}^{\frac{x}{p^{3}}}\frac{\mu(k)}{k}+\cdots\right),$

a series which has only a finite number of non-zero summands, since the
summation index of the $i$–th sum runs from 1 to
$\left[\frac{x}{p^{i}}\right]$, so that $p^{i}\leq x$ gives $i\leq\frac{\log
x}{\log p}$. Therefore

$f(x)=-\sum_{p\leq x}\log
p\left(\frac{1}{p}g\left(\frac{x}{p}\right)+\frac{1}{p^{2}}g\left(\frac{x}{p^{2}}\right)+\frac{1}{p^{3}}g\left(\frac{x}{p^{3}}\right)+\cdots\right),$
$f(x)=-\sum_{p\leq x}\frac{\log p}{p}g\left(\frac{x}{p}\right)-\sum_{p\leq
x}\log
p\left(\frac{1}{p^{2}}g\left(\frac{x}{p^{2}}\right)+\frac{1}{p^{3}}g\left(\frac{x}{p^{3}}\right)+\cdots\right).$
(7)

According to (2), for all $y$

$|g(y)|\leq 1,$

so that the absolute value of the second sum in (7) is

$\displaystyle\left|\sum_{p\leq x}\log
p\left(\frac{1}{p^{2}}g\left(\frac{x}{p^{2}}\right)+\frac{1}{p^{3}}g\left(\frac{x}{p^{3}}\right)+\cdots\right)\right|$
$\displaystyle\leq\sum_{p\leq x}\log
p\left(\frac{1}{p^{2}}\left|g\left(\frac{x}{p^{2}}\right)\right|+\frac{1}{p^{3}}\left|g\left(\frac{x}{p^{3}}\right)\right|+\cdots\right)$
$\displaystyle\leq\sum_{p\leq x}\log
p\left(\frac{1}{p^{2}}+\frac{1}{p^{3}}+\frac{1}{p^{4}}+\cdots+\frac{1}{p^{n}}+\cdots\right)$
$\displaystyle\leq\sum_{p\leq x}\log
p\left(\frac{1}{p^{2}}+\frac{1}{2p^{2}}+\frac{1}{2^{2}p^{2}}+\cdots+\frac{1}{2^{n-2}p^{2}}+\cdots\right)$
$\displaystyle\leq\sum_{p\leq x}\frac{\log
p}{p^{2}}\left(1+\frac{1}{2}+\frac{1}{4}+\frac{1}{8}+\cdots\right)$
$\displaystyle\leq 2\sum_{p\leq x}\frac{\log p}{p^{2}}$
$\displaystyle<2\sum_{\nu=1}^{\infty}\frac{\log\nu}{\nu^{2}}.$

It is well known that $\sum_{\nu=1}^{\infty}\frac{\log\nu}{\nu^{2}}$ is
convergent; thus as $x\to\infty$ the sum on the right-hand side of (7)
approaches either a certain bound, or its value oscillates between two finite
uncertain bounds. In either case, the quotient with $\log x$ approaches $0$ as
$x\to\infty$.

Denote by $h(x)$, the function defined by

$h(x)=\sum_{p\leq x}\frac{\log p}{p}g\left(\frac{x}{p}\right).$ (8)

If $\lim_{x=\infty}\frac{h(x)}{\log x}$ exists and equals $0$, then according
to (7)

$\frac{f(x)+h(x)}{\log x}=\frac{-1}{\log x}\sum_{p\leq x}\log
p\left(\frac{1}{p^{2}}g\left(\frac{x}{p^{2}}\right)+\frac{1}{p^{3}}g\left(\frac{x}{p^{3}}\right)+\cdots\right),$

and as we just saw, as $x$ gets large the right-hand side approaches $0$, from
which the correctness of statement (5) follows.

The proof of Euler’s statement, that

$\sum_{k=1}^{x}\frac{\mu(k)}{k}=0,$

thus depends on the proof of the statement

$\lim_{x=\infty}\frac{h(x)}{\log x}=0,$

which will be furnished in the following section.

## 3

Recall that the function $\vartheta(x)$ 131313) See the theorem of Hadamard
and de la Vallée Poussin in the introduction.) is defined for all positive
$\nu$ by

$\vartheta(\nu)-\vartheta(\nu-1)=\begin{cases}\log\nu&\mbox{ if $\nu$ is
prime,}\\\ 0&\mbox{ if $\nu$ is composite or 1},\\\ \log\nu=0&\mbox{ if
$\nu=1$.}\end{cases}$

And

$h(x)=\sum_{\nu=1}^{x}\frac{\vartheta(\nu)-\vartheta(\nu-1)}{\nu}g\left(\frac{x}{\nu}\right),$

where the sum ranges over the integers between $1$ and $x$.

In the place of the function $\vartheta(x)$, use

$\vartheta(x)=x\\{1+\varepsilon(x)\\},$ (9)

where the function $\varepsilon(x)$ takes only non-negative values of $x$, and
$\varepsilon(0)=0$. We note the following properties of $\varepsilon(x)$:

  1. 1.

Since by definition, $\vartheta(x)$ is never negative, then always

$\varepsilon(x)\geq-1.$

  2. 2.

As shown by Mertens141414) “Ein Beitrag zur analytischen Zahlentheorie,”
Journal für die reine und angewandte Mathematik, Volume 78, p. 48.), for all
$x$

$\vartheta(x)<2x,$

so that always

$\varepsilon(x)<1;$

therefore, we gain the inequality

$|\varepsilon(x)|\leq 1.$ (10)

  3. 3.

The theorem cited in the introduction151515) See the theorem of Hadamard and
de la Vallée Poussin in the introduction.), that

$\lim_{x=\infty}\frac{\vartheta(x)}{x}=1,$

gives

$\lim_{x=\infty}\varepsilon(x)=0.$ (11)

The introduction of the function $\varepsilon(x)$ yields, for $h(x)$,

$\displaystyle h(x)$
$\displaystyle=\sum_{\nu=1}^{x}\frac{\nu+\nu\varepsilon(\nu)-(\nu-1)-(\nu-1)\varepsilon(\nu-1)}{\nu}g\left(\frac{x}{\nu}\right)$
$\displaystyle=\sum_{\nu=1}^{x}\left\\{\frac{1}{\nu}g\left(\frac{x}{\nu}\right)+\left(\varepsilon(\nu)-\frac{\nu-1}{v}\varepsilon(\nu-1)\right)g\left(\frac{x}{\nu}\right)\right\\}$
$\displaystyle=\sum_{\nu=1}^{x}\frac{1}{\nu}g\left(\frac{x}{\nu}\right)+\sum_{\nu=1}^{x}\left(\varepsilon(\nu)-\varepsilon(\nu-1)+\frac{1}{\nu}\varepsilon(\nu-1)\right)g\left(\frac{x}{\nu}\right).$

Using Eqs. (8) and (9),

$\sum_{\nu=1}^{x}\frac{1}{\nu}g\left(\frac{x}{\nu}\right)=1;$

yielding

$h(x)-1=\sum_{\nu=1}^{x}\left(\varepsilon(\nu)-\varepsilon(\nu-1)\right)g\left(\frac{x}{\nu}\right)+\sum_{\nu=1}^{x}\frac{1}{\nu}\varepsilon(\nu-1)g\left(\frac{x}{\nu}\right).$
(12)

For the first of the two sums in (12) we get

$\displaystyle\sum_{\nu=1}^{x}(\varepsilon(\nu)$
$\displaystyle-\varepsilon(\nu-1))g\left(\frac{x}{\nu}\right)$
$\displaystyle=\sum_{\nu=1}^{x}\varepsilon(x)\left(g\left(\frac{x}{\nu}\right)-g\left(\frac{x}{\nu+1}\right)\right)+\varepsilon([x])g\left(\frac{x}{[x]+1}\right)$
$\displaystyle=\sum_{\nu=1}^{x}\varepsilon(\nu)\left(g\left(\frac{x}{\nu}\right)-g\left(\frac{x}{\nu+1}\right)\right),$

where $x<[x]+1$ so that $g\left(\frac{x}{[x]+1}\right)=0$.

If in the second sum in (12) we write $\nu+1$ in place of $\nu$, we have

$\sum_{\nu=0}^{x-1}\frac{1}{\nu+1}\varepsilon(\nu)g\left(\frac{x}{\nu+1}\right)=\sum_{\nu=1}^{x-1}\frac{1}{\nu+1}\varepsilon(\nu)g\left(\frac{x}{\nu+1}\right)$

and so

$h(x)-1=\sum_{\nu=1}^{x}\varepsilon(\nu)\left(g\left(\frac{x}{\nu}\right)-g\left(\frac{x}{\nu+1}\right)\right)+\sum_{\nu=1}^{x-1}\frac{1}{\nu+1}\varepsilon(\nu)g\left(\frac{x}{\nu+1}\right).$
(13)

Let $\delta$ be an arbitrary small positive quantity. Then by (11), there is a
$G$ such that for all $\nu\geq G$

$|\varepsilon(\nu)|\leq\frac{\delta}{3}.$ (14)

This yields

$\left|\sum_{\nu=1}^{x}\varepsilon(\nu)\left(g\left(\frac{x}{\nu}\right)-g\left(\frac{x}{\nu+1}\right)\right)\right|\leq\left|\sum_{\nu=1}^{G-1}\varepsilon(\nu)\left(g\left(\frac{x}{\nu}\right)-g\left(\frac{x}{\nu+1}\right)\right)\right|$
$+\left|\sum_{\nu=G}^{x}\varepsilon(\nu)\left(g\left(\frac{x}{\nu}\right)-g\left(\frac{x}{\nu+1}\right)\right)\right|$
$\leq\sum_{\nu=1}^{G-1}|\varepsilon(\nu)|\left|\left(g\left(\frac{x}{\nu}\right)-g\left(\frac{x}{\nu+1}\right)\right)\right|+\sum_{\nu=G}^{x}|\varepsilon(\nu)|\left|\left(g\left(\frac{x}{\nu}\right)-g\left(\frac{x}{\nu+1}\right)\right)\right|.$

As the right-hand side is increased, $|\varepsilon(\nu)|$ goes to 1 in the
first sum (by (10)), and goes to $\delta/3$ in the second sum (by (14)),
yielding

$\left|\sum_{\nu=1}^{x}\varepsilon(\nu)\left(g\left(\frac{x}{\nu}\right)-g\left(\frac{x}{\nu+1}\right)\right)\right|\leq\sum_{\nu=1}^{G-1}\left|\left(g\left(\frac{x}{\nu}\right)-g\left(\frac{x}{\nu+1}\right)\right)\right|$
(15)
$+\frac{\delta}{3}\sum_{\nu=G}^{x}\left|\left(g\left(\frac{x}{\nu}\right)-g\left(\frac{x}{\nu+1}\right)\right)\right|.$

Now

$\displaystyle\sum_{\nu=1}^{G-1}\left|\left(g\left(\frac{x}{\nu}\right)-g\left(\frac{x}{\nu+1}\right)\right)\right|$
$\displaystyle\leq\sum_{\nu=1}^{G-1}\left\\{\left|\left(g\left(\frac{x}{\nu}\right)\right|-\left|g\left(\frac{x}{\nu+1}\right)\right)\right|\right\\}$
$\displaystyle=\sum_{\nu=1}^{G-1}(1+1)\ \mbox{ (by (2))},$

so that

$\sum_{\nu=1}^{G-1}\left|\left(g\left(\frac{x}{\nu}\right)-g\left(\frac{x}{\nu+1}\right)\right)\right|\leq
2(G-1)$ (16)

and

$\displaystyle\sum_{\nu=G}^{x}\left|\left(g\left(\frac{x}{\nu}\right)-g\left(\frac{x}{\nu+1}\right)\right)\right|$
$\displaystyle\leq\sum_{\nu=1}^{x}\left|\left(g\left(\frac{x}{\nu}\right)-g\left(\frac{x}{\nu+1}\right)\right)\right|$
$\displaystyle=\sum_{\nu=1}^{x}\left|\sum_{k=1}^{\frac{x}{\nu}}\frac{\mu(k)}{k}-\sum_{k=1}^{\frac{x}{\nu+1}}\frac{\mu(k)}{k}\right|$
$\displaystyle=\sum_{\nu=1}^{x}\left|\sum\frac{\mu(k)}{k}\right|,$

where $k$ ranges over all integers in the interval
$\left(\frac{x}{\nu+1},\frac{x}{\nu}\right]$. Therefore

$\sum_{\nu=G}^{x}\left|\left(g\left(\frac{x}{\nu}\right)-g\left(\frac{x}{\nu+1}\right)\right)\right|\leq\sum_{\nu=1}^{x}\sum_{\frac{x}{\nu}\geq
k>\frac{x}{\nu+1}}\frac{|\mu(k)|}{k}\leq\sum_{\nu=1}^{x}\sum_{\frac{x}{\nu}\geq
k>\frac{x}{\nu+1}}\frac{1}{k}$ $=\sum_{x\geq
k>\frac{x}{2}}\frac{1}{k}+\sum_{\frac{x}{2}\geq
k>\frac{x}{3}}\frac{1}{k}+\sum_{\frac{x}{3}\geq
k>\frac{x}{4}}\frac{1}{k}+\cdots+\sum_{\frac{x}{[x]-1}\geq
k>\frac{x}{[x]}}\frac{1}{k}$ $+\sum_{\frac{x}{[x]}\geq
k>1}\frac{1}{k}=\sum_{k=1}^{x}\frac{1}{k},$

and since always

$\sum_{k=1}^{x}\frac{1}{k}\leq\log x+1,$

we have

$\sum_{\nu=G}^{x}\left|\left(g\left(\frac{x}{\nu}\right)-g\left(\frac{x}{\nu+1}\right)\right)\right|\leq\log
x+1.$ (17)

Replacing both sums of the right-hand side of inequality (15) by the results
gained in (16) and (17) yields

$\left|\sum_{\nu=1}^{x}\varepsilon(x)\left(g\left(\frac{x}{\nu}\right)-g\left(\frac{x}{\nu+1}\right)\right)\right|\leq
2(G-1)+\frac{\delta}{3}(\log x+1).$ (18)

The handling of the second sum in (13) is somewhat simpler. We have

$\displaystyle\left|\sum_{\nu=1}^{x-1}\frac{1}{\nu+1}\varepsilon(\nu)g\left(\frac{x}{\nu+1}\right)\right|$
$\displaystyle\leq\sum_{\nu=1}^{x-1}\frac{1}{\nu+1}|\varepsilon(\nu)|\left|g\left(\frac{x}{\nu+1}\right)\right|$
$\displaystyle\leq\sum_{\nu=1}^{x-1}\frac{|\varepsilon(\nu)|}{\nu+1}$
$\displaystyle=\sum_{\nu=1}^{G-1}\frac{|\varepsilon(\nu)|}{\nu+1}+\sum_{\nu=G}^{x-1}\frac{|\varepsilon(\nu)|}{\nu+1}$
$\displaystyle\leq\sum_{\nu=1}^{G-1}\frac{1}{\nu+1}+\sum_{\nu=G}^{x-1}\frac{1}{\nu+1}\frac{\delta}{3}$
$\displaystyle\leq\sum_{\nu=1}^{G-1}1+\frac{\delta}{3}\sum_{\nu=1}^{x-1}\frac{1}{\nu},$

so that

$\left|\sum_{\nu=1}^{x-1}\frac{1}{\nu+1}\varepsilon(\nu)g\left(\frac{x}{\nu+1}\right)\right|\leq
G-1+\frac{\delta}{3}(\log x+1).$ (19)

With help from the inequalities (18) and (19), (13) becomes

$|h(x)|\leq 1+2G-2+\frac{\delta}{3}\log
x+\frac{\delta}{3}+G-1+\frac{\delta}{3}\log x+\frac{\delta}{3}$
$=3G-2+\frac{2}{3}\delta+\frac{2}{3}\delta\log x,$

thus for all

$x\geq e^{\frac{3G-2+\frac{2}{3}\delta}{\frac{1}{2}\delta}},$

we have

$3G-2+\frac{2}{3}\delta\leq\frac{1}{3}\delta\log x,$

so that

$|h(x)|\leq\frac{1}{3}\delta\log x+\frac{2}{3}\delta\log x=\delta\log x,$

which yields

$\left|\frac{h(x)}{\log x}\right|\leq\delta.$ (20)

For such a $\delta$ there is a $\xi$ assignable, such that for all $x\geq\xi,$
(20) is fulfilled; therefore the $\lim_{x=\infty}\frac{h(x)}{\log x}$ exists
and equals 0. Thus all the results shown in the first two paragraphs of this
work are valid; that is, the $\lim_{x=\infty}\sum_{k=1}^{x}\frac{\mu(k)}{k}$
exists and equals 0, and thus the correctness of the equation named in the
title, briefly

$\sum_{k=1}^{x}\frac{\mu(k)}{k}=0.$

## 4

If we define161616) von Mangoldt, 1. c., p. 850.)

$M(x)=\sum_{k=1}^{x}\mu(k),$

then with help of the proven result,

$\lim_{x=\infty}g(x)=0,$

we have

$\lim_{x=\infty}\frac{M(x)}{x}=0.$

Von Mangoldt171717) 1. c., pp. 849–851.) proved this indirectly by use of the
identity

$g(x)=\sum_{k=1}^{x}\frac{\mu(k)}{k}=\sum_{k=1}^{x}(M(k)-M(k-1))\frac{1}{k}.$

It can be furnished as follows directly. From the equation

$M(x)=\sum_{k=1}^{x}\mu(k)=\sum_{k=1}^{x}\frac{\mu(k)}{k}\cdot
k=\sum_{k=1}^{x}(g(k)-g(k-1))\cdot k,$

it follows that

$\displaystyle M(x)$ $\displaystyle=\sum_{k=1}^{x-1}g(k)(k-(k+1))+g(x)[x]$
$\displaystyle=-\sum_{k=1}^{x-1}g(k)+g(x)[x],$

so that since for $\delta>0$ there is a $G$ such that for all $k\geq G$

$|g(k)|\leq\frac{\delta}{3}$

for all $x\geq G$

$\displaystyle|M(x)|$
$\displaystyle\leq\sum_{k=1}^{G-1}|g(k)|+\sum_{k=G}^{x-1}|g(k)|+|g(x)|\cdot x$
$\displaystyle\leq G-1+\frac{\delta}{3}([x]-G)+\frac{\delta}{3}x,$

so that

$\left|\frac{M(x)}{x}\right|\leq\frac{G-1-\frac{\delta}{3}G}{x}+\frac{2}{3}\delta,$

then for

$x\geq\frac{G-1-\frac{\delta}{3}G}{\frac{\delta}{3}}$

and at the same time greater than or equal to $G$,

$\left|\frac{M(x)}{x}\right|\leq\frac{1}{3}\delta+\frac{2}{3}\delta=\delta,$

with which the statement

$\lim_{x=\infty}\frac{M(x)}{x}=0$

is proved.

## Theses

  1. 1.

It is desirable during every existence proof of a mathematical quantity to be
led, at the same time on the way to the result, to the actual existing
quantity.

  2. 2.

A boundary between arithmetic and analytic areas of mathematics cannot be
drawn.

  3. 3.

The concept of the semiconvergent series is a relative concept.

  4. 4.

Out of the impossibility of perpetual motion of second kind comes the proof of
the second law of thermodynamics.

  5. 5.

It did not succeed, the justifying of psychology on an exactly mathematical
basis.