When ${\cal
N}({\mathcal{X}}\setminus({\mathcal{X}}_{1}\cup\\{x\\}))=\emptyset$,
${\mathcal{X}}_{*}$ is one of subsets
$\\{{\mathcal{X}}\setminus{\mathcal{X}}_{1}\\}_{{\mathcal{X}}_{1}\in{\cal
B}^{k+1}}$.

When ${\cal
N}({\mathcal{X}}\setminus({\mathcal{X}}_{1}\cup\\{x\\}))\neq\emptyset$,
$\hat{C}({\mathcal{X}}\setminus({\mathcal{X}}_{1}\cup\\{x\\})\geq\hat{C}({\mathcal{X}}_{*})$
and $\hat{C}({\mathcal{X}}_{*})$ equals the capacity. Since
${\mathcal{X}}_{*}$ is not contained in ${\cal B}^{k}$, we have
$\hat{C}({\mathcal{X}}_{*})>C^{k}$. Hence, we have
$\hat{C}({\mathcal{X}}\setminus({\mathcal{X}}_{1}\cup\\{x\\})>C^{k}$. Thus,
${\mathcal{X}}_{1}\cup\\{x\\}\in{\cal A}_{k+1}$. Therefore,
${\mathcal{X}}_{*}$ is contained in one of sets
$\\{{\mathcal{X}}\setminus{\mathcal{X}}_{1}\\}_{{\mathcal{X}}_{1}\in{\cal
A}_{k+1}}\subset\\{{\mathcal{X}}\setminus{\mathcal{X}}_{1}\\}_{{\mathcal{X}}_{1}\in{\cal
A}_{k+1}\cup{\cal B}_{k+1}}$.

Algorithm 7 Non-iterative algorithm for classical channel capacity in the
general case

We apply Algorithm 5 to the input set ${\mathcal{X}}$. If ${\cal
N}({\mathcal{X}})$ is the empty set, we output $\hat{C}({\mathcal{X}})$ as the
capacity. Otherwise, we define the family ${\cal A}_{1}$ of subsets of
${\mathcal{X}}$ with cardinality 1 as $\\{\\{x\\}\\}_{x\in{\cal
N}({\mathcal{X}})}$. Set $j=1$;

repeat

We define families ${\cal A}_{j+1}$, ${\cal B}_{j+1}$, ${\cal B}^{j+1}$ of
subsets of ${\mathcal{X}}$ and the positive number $C^{j+1}$ by using
Algorithm 5 as follows.  $\displaystyle{\cal A}_{j+1}$
$\displaystyle:=\left\\{{\mathcal{X}}_{1}\cup\\{x\\}\left|\begin{array}[]{l}{\mathcal{X}}_{1}\in{\cal
A}_{j},x\in{\cal N}({\mathcal{X}}\setminus{\mathcal{X}}_{1}),{\cal
N}({\mathcal{X}}\setminus({\mathcal{X}}_{1}\cup\\{x\\}))\neq\emptyset,\\\
\hat{C}({\mathcal{X}}\setminus({\mathcal{X}}_{1}\cup\\{x\\}))>C^{j}\end{array}\right\\}\right.$
(193) $\displaystyle{\cal B}_{j+1}$
$\displaystyle:=\\{{\mathcal{X}}_{1}\cup\\{x\\}|{\mathcal{X}}_{1}\in{\cal
A}_{j},x\in{\mathcal{X}}\setminus{\mathcal{X}}_{1},{\cal
N}({\mathcal{X}}\setminus({\mathcal{X}}_{1}\cup\\{x\\}))=\emptyset\\}$ (194)
$\displaystyle{\cal B}^{j+1}$ $\displaystyle:={\cal B}^{j}\cup{\cal B}_{j+1}$
(195) $\displaystyle C^{j+1}$ $\displaystyle:=\max_{{\mathcal{X}}_{1}\in{\cal
B}^{j+1}}\hat{C}({\mathcal{X}}\setminus{\mathcal{X}}_{1}).$ (196)

until ${\cal A}_{j+1}$ is empty. When this stopping condition holds, we denote
$j+1$ by $j_{0}$.

We output $C^{j_{0}}$ as the channel capacity.

###### Remark 3

Muroga Muroga  also considered the calculation method of the classical channel
capacity. In (Muroga, , Section 1), he derived an analytical calculation
method when $n_{1}=n_{2}$. In this special case, our method is slightly
different from his method as follows. While his method needs to calculate the
inverse matrix of an $n_{1}\times n_{1}$ matrix, our method needs only to
calculate the inverse matrix of an $(n_{1}-1)\times(n_{1}-1)$ matrix. Hence,
our method is slightly better than his method. When $n_{2}>n_{1}$, he
presented his calculation method in (Muroga, , Section 2). His calculation
method requires to solve nonlinear characteristic equations (Muroga, , (28)).
Although he did not explain how to solve the characteristic equations, the
solution can be characterized by the minimizer of a certain convex function of
$n_{2}-n_{1}$ variables in a similar way to (181). Also, his calculation
method requires to calculate the determinants of $n_{1}(n_{2}-n_{1})+1$
$n_{1}\times n_{1}$-matrices while our method needs to calculate $f_{j}(y)$,
which can be calculated by the inverse matrix of
$(n_{1}-1)\times(n_{1}-1)$-matrix. The calculation of the inverse matrix of
size $n_{1}-1$ is easier than the determinants of $(n_{1}-1)^{2}$
$(n_{1}-2)\times(n_{1}-2)$-matrices and one
$(n_{1}-1)\times(n_{1}-1)$-matrices due to Cramer’s formula of the inverse
matrix. Hence, our method is slightly easier than his method.

###  5.7 Application of non-iterative method

This section aims to demonstrate the advantage of our method over the method
in exact . That is, applying Algorithm 7, we make a numerical calculation of
the classical channel capacity with the following channel of $n_{1}=n_{2}=4$;

$\displaystyle W_{1}:=\left(\begin{array}[]{c}0.05\\\ 0.9-t\\\ 0.05\\\
t\end{array}\right),~{}W_{2}:=\left(\begin{array}[]{c}0.05\\\ 0.05\\\ 0.9-t\\\
t\end{array}\right),~{}W_{3}:=\left(\begin{array}[]{c}0.9\\\ 0.05\\\ 0.05\\\
0\end{array}\right),~{}W_{4}:=\left(\begin{array}[]{c}0.05\\\ 0.05\\\ 0.05\\\
0.85\end{array}\right).$ (213)

In this channel (213), according to Algorithm 7, we apply Algorithm 5 to the
input set $\\{1,2,3,4\\}$. As a result, we found that the optimal input
distribution has the support $\\{1,2,3,4\\}$ when $0\leq t\leq 0.18$. However,
when $t\geq 0.18$, it does not have a positive probability at $X=4$, i.e.,
${\cal N}({\mathcal{X}})$ is not the empty set. This case cannot be covered by
Algorithm 1 in the reference exact . Hence, in this case, as the next step, we
apply Algorithm 5 with ${\cal X}=\\{1,2,3\\}$, where we need to make the
minimization (181) with one free parameter. Its numerical calculation is done
as Figs. 2 and 3.

Figure 2: Capacity of the channel (213). For $0\leq t\leq 0.18$, the capacity
is calculated by Algorithm 5 with ${\cal X}=\\{1,2,3,4\\}$. For $0.76\geq
t\geq 0.18$, the capacity is calculated by Algorithm 5 with ${\cal
X}=\\{1,2,3\\}$. Figure 3: Input distribution realizing capacity: Due to the
symmetry, $P_{X}(1)=P_{X}(2)$ in the optimal input distribution. Green curve
shows $P_{X}(1)$ of the optimal input distribution. Black curve shows
$P_{X}(3)$ of the optimal input distribution. Red curve shows $P_{X}(4)$ of
the optimal input distribution. This value is zero for $t\geq 0.18$.

##  6 Classical Secrecy Capacity

###  6.1 Problem setting

Let ${\mathcal{X}}:=\\{1,\ldots,n_{1}\\}$, ${\cal Z}:=\\{1,\ldots,n_{2}\\}$,
and ${\mathcal{Y}}:=\\{1,\ldots,n_{3}\\}$ be finite sets. We call a map
$W:{\mathcal{X}}\rightarrow{\cal P}_{{\mathcal{Z}}\times{\mathcal{Y}}}$ be a
channel from ${\mathcal{X}}$ to ${\mathcal{Z}}\times{\mathcal{Y}}$. In this
section, we use the subscript
$\mathsf{a},\mathsf{b},\mathsf{c},\mathsf{d},\mathsf{e},\mathsf{f},\mathsf{g},\mathsf{h}$
to express elements of $\mathbb{R}^{n_{1}-1}$,
$\mathbb{R}^{n_{2}(n_{1}+n_{3})-n_{1}}$, $\mathbb{R}^{n_{2}(n_{1}+n_{3})-1}$,
$\mathbb{R}^{n_{1}n_{2}n_{3}-n_{1}}$,
$\mathbb{R}^{n_{2}(n_{1}+n_{3})-2n_{1}+1}$, $\mathbb{R}^{n_{2}n_{1}-n_{1}}$,
$\mathbb{R}^{n_{2}(n_{3}-1)}$, $\mathbb{R}^{n_{2}(n_{3}-1)-n_{1}+1}$,
respectively, as Tables 4 and 4.

Table 3: Summary of subscripts for Section 6 (1)

Subscript | $\mathsf{a}$ | $\mathsf{b}$ | $\mathsf{c}$ | $\mathsf{d}$
---|---|---|---|---
Vector space | $\mathbb{R}^{n_{1}-1}$ | $\mathbb{R}^{n_{2}(n_{1}+n_{3})-n_{1}}$ | $\mathbb{R}^{n_{2}(n_{1}+n_{3})-1}$ | $\mathbb{R}^{n_{1}n_{2}n_{3}-n_{1}}$
Examples | $\Theta_{{\cal M}},\Theta_{{\cal M}}$ | $\Theta_{{\cal E},\mathsf{b}},\Theta_{{\cal E},\mathsf{b}}$ | $\Theta_{{\cal E}},\Theta_{{\cal E}}$ |
$\Theta_{{\cal E},\mathsf{a}},\Theta_{{\cal E},\mathsf{a}}$ |

In Section 6, there are more types of vector spaces than in Section 4. This
table and the next table show the relation between a vector space appearing in
Section 6 and the subscript.

Table 4: Summary of subscripts for Section 6 (2)

Subscript | $\mathsf{e}$ | $\mathsf{f}$ | $\mathsf{g}$ | $\mathsf{h}$
---|---|---|---|---
Vector space | $\mathbb{R}^{n_{2}(n_{1}+n_{3})-2n_{1}+1}$ | $\mathbb{R}^{n_{2}n_{1}-n_{1}}$ | $\mathbb{R}^{n_{2}(n_{3}-1)}$ | $\mathbb{R}^{n_{2}(n_{3}-1)-n_{1}+1}$

For a conditional distribution $P_{Y|Z}$ and a joint distribution $P_{X,Z}$,
we define the joint distribution $P_{Y|Z}\times P_{X,Z}$ on
${\mathcal{X}}\times{\mathcal{Z}}\times{\mathcal{Y}}$ as $P_{Y|Z}\times
P_{X,Z}(x,z,y):=P_{Y|Z}(y|z)P_{X,Z}(x,z)$. We use the notations
$W_{x}(z,y):=W(z,y|x)$, $W_{Z}(z|x):=\sum_{y^{\prime}}W(z,y^{\prime}|x)$, and
$W_{x,z}(y):=\frac{W(z,y|x)}{W_{Z}(z|x)}$. For $q\in{\cal P}_{{\mathcal{X}}}$
and $r\in{\cal P}_{{\mathcal{Y}}}$, $(W\times q)_{Y|Z}$ is defined by
$(W\times q)_{Y|Z}(y|z):=\frac{(W\cdot q)(z,y)}{(W_{Z}\cdot q)(z)}$. Hence,
$((W\times q)_{Y|Z}\times(W_{Z}\times q))(x,z,y)=(W\times
q)_{Y|Z}(y|z)(W_{Z}\times q)(x,z)$.

When the channel $W$ satisfies Markov chain $X-Y-Z$, the secrecy capacity of
the wire-tap channel $W$ is given by Wyner ; CK79

$\displaystyle\max_{q\in{{\cal P}_{{\mathcal{X}}}}}D(W_{Y}\times
q\|(W_{Y}\cdot q)\times q)-D(W_{Z}\times q\|(W_{Z}\cdot q)\times q)$
$\displaystyle=$ $\displaystyle\max_{q\in{{\cal P}_{{\mathcal{X}}}}}D(W\times
q\|(W\times q)_{Y|Z}\times(W_{Z}\times q))$ $\displaystyle=$
$\displaystyle\max_{q\in{{\cal P}_{{\mathcal{X}}}}}\min_{Q\in{{\cal
P}_{X-Z-Y}(W_{Z|X})}}D(W\times q\|Q).$ (214)

We define the set of distributions ${{\cal P}_{X-Z-Y}}$ on ${\cal
X}\times{\cal Z}\times{\cal Y}$ to satisfy the Markov chain $X-Z-Y$.

As proven in Subsection 6.4, the set ${{\cal P}_{X-Z-Y}}$ forms an exponential
subfamily ${\cal E}$ and the set of $W\times q$ forms a mixture subfamily
${\cal M}$. Hence, the maximization problem (214) is a special case of the
maximization (47) with $k=n_{1}-1$, $l=n_{1}n_{2}-1+n_{2}(n_{3}-1)$, and
$d=n_{1}n_{2}n_{3}-1$. In the following, we apply Algorithm 4. For this aim,
we need to choose a suitable coordinate to satisfy conditions (B1), (B3),
(B4), and check Condition (B2).

###  6.2 Constructions of vectors
$u_{1},\ldots,u_{n_{1}n_{2}n_{3}-1},v_{1},\ldots,v_{n_{2}(n_{1}+n_{3}-1)-1}$

To choose a suitable coordinate to satisfy conditions (B3), (B4), and (B5), we
need to choose suitable vectors
$u_{1},\ldots,u_{n_{1}n_{2}},v_{1},\ldots,v_{n_{1}+n_{2}-1}$, which form the
matrices $U$ and $V$. For this aim, we define various functions on ${\cal Y}$
and ${\cal X}\times{\cal Y}$. Given $z=1,\ldots,n_{2}$, we choose $n_{3}-1$
independent functions $f_{j,z}$ on ${\cal Y}$ with $j=1,\ldots,n_{3}-1$ to
satisfy the condition that

$\displaystyle\sum_{y\in{\cal Y}}f_{j,z}(y)W_{n_{1},z}(y)=0$ (215)

and the linear space spanned by $f_{1,z},\ldots,f_{n_{2}-1,z}$ does not
contain a constant function. As a typical case, $f_{j,z}$ can be chosen as
follows.

$\displaystyle
f_{j,z}(y):=\left\\{\begin{array}[]{ll}W_{n_{1},z}(j+1)&\hbox{when }y=j\\\
-W_{n_{1},z}(j)&\hbox{when }y=j+1\\\ 0&\hbox{otherwise}.\end{array}\right.$
(219)

Then, we define the functions $\xi_{1},\ldots,\xi_{n_{1}n_{2}n_{3}-1}$ on
${\cal X}\times{\cal Z}\times{\cal Y}$ as follows;

$\displaystyle\xi_{i}(x,z,y):=$ $\displaystyle\delta_{i}(x)$
$\displaystyle\xi_{n_{1}-1+(i-1)(n_{2}-1)+j^{\prime}}(x,z,y):=$
$\displaystyle(\delta_{j^{\prime}}(z)-W_{Z,i}(j^{\prime}))\delta_{i}(x)$
$\displaystyle\xi_{n_{1}n_{2}-1+(n_{1}-1)n_{2}(n_{3}-1)+(z^{\prime}-1)(n_{3}-1)+j}(x,z,y):=$
$\displaystyle f_{j,z^{\prime}}(y)\delta_{n_{1}}(x)\delta_{z^{\prime}}(z),$

and

$\displaystyle\xi_{n_{1}n_{2}-1+(i-1)n_{2}(n_{3}-1)+(z^{\prime}-1)(n_{3}-1)+j}(x,z,y)$
$\displaystyle:=$
$\displaystyle(f_{j,z^{\prime}}(y)\\!-\\!h_{i,z^{\prime}(n_{3}-1)+j})\delta_{i}(x)\delta_{z^{\prime}}(z)$
(220)

for $i=1,\ldots,n_{1}-1$, $z^{\prime}=1,\ldots,n_{2}$,
$j^{\prime}=1,\ldots,n_{2}-1$, and $j=1,\ldots,n_{3}-1$, where

$\displaystyle
h_{i,z^{\prime}(n_{3}-1)+j}:=\sum_{y}f_{j,z^{\prime}}(y)W_{i}(z^{\prime},y).$
(221)

Then, we define the $C^{\infty}-$strictly convex function $F$ on
$\mathbb{R}^{n_{1}n_{2}n_{3}-1}$ as

$\displaystyle
F(\theta):=\log\sum_{x,z,y}e^{\sum_{i=1}^{n_{1}n_{2}n_{3}-1}\theta^{i}\xi_{i}(x,z,y)}.$
(222)

That is, we consider the Bregman divergence system $(\mathbb{R}^{d},F,D^{F})$.
We define the distribution
$P_{\theta},P_{\theta,XZ},P_{\theta,X},P_{\theta,Z},P_{\theta,Y|Z}$ as

$\displaystyle P_{\theta}(x,z,y)$
$\displaystyle:=e^{\sum_{i=1}^{n_{1}n_{2}n_{3}-1}\theta^{i}\xi_{i}(x,z,y)-F(\theta)},$
(223) $\displaystyle P_{\theta,XZ}(xz)$
$\displaystyle:=\sum_{y}e^{\sum_{i=1}^{n_{1}n_{2}n_{3}-1}\theta^{i}\xi_{i}(x,z,y)-F(\theta)},$
(224) $\displaystyle P_{\theta,X}(x)$
$\displaystyle:=\sum_{z,y}e^{\sum_{i=1}^{n_{1}n_{2}n_{3}-1}\theta^{i}\xi_{i}(x,z,y)-F(\theta)},$
(225) $\displaystyle P_{\theta,Z}(z)$
$\displaystyle:=\sum_{x,y}e^{\sum_{i=1}^{n_{1}n_{2}n_{3}-1}\theta^{i}\xi_{i}(x,z,y)-F(\theta)},$
(226) $\displaystyle P_{\theta,Y|Z}(y|z)$
$\displaystyle:=\frac{\sum_{x}e^{\sum_{i=1}^{n_{1}n_{2}n_{3}-1}\theta^{i}\xi_{i}(x,z,y)-F(\theta)}}{P_{\theta,Z}(z)}.$
(227)

Then, as a special case of (40), we have

$\displaystyle
D^{F}(\theta\|\theta^{\prime})=D(P_{\theta}\|P_{\theta^{\prime}}).$ (228)

Next, we choose the matrix $U$ as the identity matrix, and
$u_{1},\ldots,u_{n_{1}n_{2}n_{3}-1}$ are chosen as its $n_{1}n_{2}n_{3}-1$
column vectors. Then, we define vector
$v_{1},\ldots,v_{n_{2}(n_{1}+n_{3}-1)-1}$ as follows, whereas
$V=(v_{1},\ldots,v_{n_{2}(n_{1}+n_{3}-1)-1})$.

$\displaystyle v_{i}$ $\displaystyle:=u_{i}$ (229) $\displaystyle
v_{n_{1}n_{2}-1+j}$
$\displaystyle:=\sum_{i=1}^{n_{1}}u_{n_{1}n_{2}-1+(n_{1}-1)n_{2}(n_{3}-1)+j}+\sum_{i=1}^{n_{1}-1}h_{i,j}u_{i}$
(230)

for $i=1,\ldots,n_{1}n_{2}-1$ and $j=1,\ldots,n_{2}(n_{3}-1)$. We define
$g_{j}(x,z,y)$ for $j=1,\ldots,n_{2}(n_{1}+n_{3}-1)-1$ as

$\displaystyle g_{i}(x,z,y):=$ $\displaystyle\delta_{i}(x)$ (231)
$\displaystyle g_{n_{1}-1+(i-1)(n_{2}-1)+j^{\prime}}(x,z,y):=$
$\displaystyle(\delta_{j^{\prime}}(z)-W_{Z,i}(j^{\prime}))\delta_{i}(x)$ (232)
$\displaystyle g_{n_{1}n_{2}-1+(z^{\prime}-1)(n_{3}-1)+j}(x,z,y):=$
$\displaystyle f_{j,z^{\prime}}(y)\delta_{z^{\prime}}(z),$ (233)

for $i=1,\ldots,n_{1}-1$, $z^{\prime}=1,\ldots,n_{2}$,
$j^{\prime}=1,\ldots,n_{2}-1$, and $j=1,\ldots,n_{3}-1$. Then, we have

$\displaystyle\sum_{i=1}^{n_{2}(n_{1}+n_{3}-1)-1}\xi_{i}(x,z,y)v_{j}^{i}=g_{j}(x,z,y).$
(234)

###  6.3 Parameterizations of ${\cal E}$ and ${\cal M}$

We define the exponential subfamily ${\cal E}$ by the generator
$v_{1},\ldots,v_{n_{2}(n_{1}+n_{3}-1)-1}$ at the point $0$. Since the set
$\\{v_{1},\ldots,v_{n_{2}(n_{1}+n_{3}-1)-1}\\}$ spans the function space
spanned by functions of ${\mathcal{X}}$ and ${\mathcal{Z}}$ and functions of
${\mathcal{Y}}$ and ${\mathcal{Z}}$, the exponential subfamily ${\cal E}$ is
the inner of ${\cal P}_{X-Z-Y}$.

We define the mixture family ${\cal M}$ by the constraint
$\sum_{i=1}^{n_{1}n_{2}-1}u^{i}_{n_{1}-1+j^{\prime}}\partial_{i}F(\theta)=0$
for $j^{\prime}=1,\ldots,n_{1}(n_{2}-1)$. This constraint is equivalent to

$\displaystyle\sum_{z,y}(\delta_{j}(z)-W_{Z,i}(j))P_{\theta}(i,z,y)$
$\displaystyle=0$ (235)
$\displaystyle\sum_{z,y}(f_{j,z^{\prime}}(y)-h_{i,z^{\prime}(n_{3}-1)+j})\delta_{z^{\prime}}(z)P_{\theta}(i,z,y)$
$\displaystyle=0$ (236)
$\displaystyle\sum_{z,y}f_{j,z}(y)\delta_{z^{\prime}}(z)P_{\theta}(n_{1},z,y)$
$\displaystyle=0$ (237)

for $i=1,\ldots,n_{1}-1$, $z^{\prime}=1,\ldots,n_{2}$, $j=1,\ldots,n_{3}-1$.
For $\theta_{1}\in\mathbb{R}^{n_{1}-1}$ and
$\theta_{\mathsf{b}}\in\mathbb{R}^{n_{2}(n_{1}+n_{3}-1)-n_{1}}$, the function
$F_{{\cal E}}$ is given as

$\displaystyle F_{{\cal
E}}(\theta_{\mathsf{a}},\theta_{\mathsf{b}})=F(V(\theta_{\mathsf{a}},\theta_{\mathsf{b}})^{T}).$
(238)

The mixture family ${\cal M}$ is composed of distributions with the form
$W\times q$. Thus, the problem (214) is written as the problem (47) with the
above defined ${\cal E}$ and ${\cal M}$. Since the conditional probability
$P_{\theta,ZY|X}(z,y|i)=\frac{P_{\theta,XZY}(i,z,y)}{\sum_{z^{\prime},y^{\prime}}P_{\theta,XZY}(i,z^{\prime},y^{\prime})}$
depends only on $(\theta^{j})_{j=n_{1}}^{n_{2}(n_{1}+n_{3})-1}$ for
$i=1,\ldots,n_{1}$, we choose
$\theta_{\mathsf{d}}^{\dagger}=(\theta^{n_{1},\dagger},\ldots,\theta^{n_{1}n_{2}n_{3}-1,\dagger})$
as

$\displaystyle
W_{i}(z,y)=\frac{e^{\sum_{j=n_{1}}^{n_{1}n_{2}n_{3}-1}\theta^{j,\dagger}\xi_{j}(i,z,y)}}{\sum_{z^{\prime},y^{\prime}}e^{\sum_{j=n_{1}}^{n_{1}n_{2}n_{3}-1}\theta^{j,\dagger}\xi_{j}(i,z^{\prime},y^{\prime})}}.$
(239)

Since (220) and (221) guarantee the relation

$\displaystyle\sum_{z^{\prime},y^{\prime}}\xi_{j^{\prime}}(i,z^{\prime},y^{\prime})e^{\sum_{j=n_{1}}^{n_{1}n_{2}n_{3}-1}\theta^{j,\dagger}\xi_{j}(i,z^{\prime},y^{\prime})}=0$
(240)

for $j^{\prime}=n_{1},\ldots,n_{1}n_{2}n_{3}-1$, we have

$\displaystyle H(W_{i})$ $\displaystyle=$
$\displaystyle-\sum_{z,y}\bigg{(}\sum_{j=n_{1}}^{n_{1}n_{2}n_{3}-1}\theta^{j,\dagger}\xi_{j}(i,z,y)-\log\Big{(}\sum_{z^{\prime},y^{\prime}}e^{\sum_{j=n_{1}}^{n_{1}n_{2}n_{3}-1}\theta^{j,\dagger}\xi_{j}(i,z^{\prime},y^{\prime})}\Big{)}\bigg{)}W_{i}(z,y)$
$\displaystyle=$
$\displaystyle\log\bigg{(}\sum_{z^{\prime},y^{\prime}}e^{\sum_{j=n_{1}}^{n_{1}n_{2}n_{3}-1}\theta^{j,\dagger}\xi_{j}(i,z^{\prime},y^{\prime})}\bigg{)}$
(241)

Due to (239), ${\cal M}$ is written as
$\\{(\theta_{\mathsf{a}},\theta_{\mathsf{d}}^{\dagger})|\theta_{1}\in\mathbb{R}^{n_{1}-1}\\}$
because the matrix $U$ is the identity matrix. That is, ${\cal M}$ forms an
exponential subfamily generated by $u_{1},\ldots,u_{n_{1}-1}$ at
$(0,\theta_{\mathsf{d}}^{\dagger})$. For $\theta_{1}\in\mathbb{R}^{n_{1}-1}$,
the function $F_{{\cal M}}$ is given as

$\displaystyle F_{{\cal
M}}(\theta_{\mathsf{a}})=F(\theta_{\mathsf{a}},\theta_{\mathsf{d}}^{\dagger}).$
(242)

In addition, the maximization (214) is rewritten as

$\displaystyle\max_{q\in{{\cal P}_{{\mathcal{X}}}}}\min_{Q\in{{\cal
P}_{X-Z-Y}(W_{Z|X})}}D(W\times q\|Q)=$
$\displaystyle\max_{\theta\in\mathcal{M}}D^{F}(\theta\|\mathop{\Gamma}\nolimits^{(m),F}_{\mathcal{E}}(\theta))$
$\displaystyle=$
$\displaystyle\max_{\theta\in\mathcal{M}}\min_{\theta^{\prime}\in\mathcal{E}}D^{F}(\theta\|\theta^{\prime}).$
(243)

###  6.4 Check of Conditions (B2), (B3), and (B4)

Lemma 7 guarantees Condition (B1). We define the $(n_{1}-1)\times
n_{2}(n_{3}-1)$ matrix $H:=(h_{i,j})$, where $h_{i,j}$ is defined in (221).
Then, (230) guarantees that the $(n_{1}-1)\times(n_{2}(n_{1}+n_{3}-1)-1)$
matrix $V_{1}$ is $(I,0_{n_{1}-1,n_{1}(n_{2}-1)},H)$, where
$0_{n_{1}-1,n_{1}(n_{2}-1)}$ is the $(n_{1}-1)\times n_{1}(n_{2}-1)$ zero
matrix. That is, $(n_{1}-1)\times(n_{1}(n_{2}-1)+n_{2}(n_{3}-1))$ matrix
$V_{3}$ is $(0_{n_{1}-1,n_{1}(n_{2}-1)},H)$. Hence, conditions (B3) and (B4)
hold. Therefore, we can apply Algorithm 4. However, in this example, the
condition (B5) does not hold, in general.

As the relation

$\displaystyle P_{\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta)}=P_{\theta,Y|Z}\times P_{\theta,XZ}$ (244)

holds for any $\theta$, we have

$\displaystyle D^{F}(\theta^{\prime}\|\theta)=$ $\displaystyle
D(P_{\theta}\|P_{\theta^{\prime}})=D(P_{\theta,XZ}\|P_{\theta^{\prime},XZ})$
$\displaystyle\leq$ $\displaystyle
D(P_{\theta,XZ}\|P_{\theta^{\prime},XZ})+\sum_{z}P_{\theta,Z}(z)D(P_{\theta,Y|Z=z}\|P_{\theta^{\prime},Y|Z=z})$
$\displaystyle=$ $\displaystyle D(P_{\theta,Y|Z}\times
P_{\theta,XZ}\|P_{\theta^{\prime},Y|Z}\times
P_{\theta^{\prime},XZ})=D(P_{\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta^{\prime})}\|(P_{\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta)})$ $\displaystyle=$ $\displaystyle
D^{F}(\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta^{\prime})\|\mathop{\Gamma}\nolimits^{(m),F}_{{\cal E}}(\theta))$
(245)

for $\theta,\theta^{\prime}\in{\cal M}$. Thus, condition (B2) holds.
Therefore, Theorem 4.1 guarantees the global convergence. When $\theta^{(1)}$
is the uniform distribution on ${\cal X}$, in the same way as (164), we can
show that the supremum
$\sup_{\theta\in\mathcal{M}}D^{F}(\theta\|\theta^{(1)})$ equals $\log n_{1}$.
Therefore, when Theorem 4.1 is applied, we obtain the precision (54) with
$\frac{\log n_{1}}{\epsilon}$ iterations.

###  6.5 Conversion to em-problem

We define the following functions;

$\displaystyle\bar{g}_{i}(x)$ $\displaystyle:=g_{i}(x,z,y)$ (246)
$\displaystyle\bar{g}_{j}(x,z)$ $\displaystyle:=g_{j}(x,z,y)$ (247)
$\displaystyle\bar{g}_{j^{\prime}}(z,y)$
$\displaystyle:=g_{j^{\prime}}(x,z,y)$ (248)

for $i=1,\ldots,n_{1}-1$, $j=n_{1},\ldots,n_{1}n_{2}-1$, and
$j^{\prime}=n_{1}n_{2},\ldots,n_{2}(n_{1}+n_{3}-1)-1$.

For
$\theta_{\mathsf{a}}=(\theta^{1},\ldots,\theta^{n_{1}-1})\in\mathbb{R}^{n_{1}-1}$
and
$\theta_{\mathsf{f}}=(\theta^{n_{1}},\ldots,\theta^{n_{1}n_{2}-1})\in\mathbb{R}^{n_{1}(n_{2}-1)}$,
we define $\hat{P}_{\theta_{\mathsf{a}},X}$,
$\hat{P}_{\theta_{\mathsf{a}},\theta_{\mathsf{f}},XZ}$, and
$\hat{P}_{\theta_{\mathsf{a}},\theta_{\mathsf{f}},Z|X}$ as

$\displaystyle\hat{P}_{\theta_{\mathsf{a}},X}(x)$
$\displaystyle:=\frac{e^{\sum_{j=1}^{n_{1}-1}\theta^{j}\bar{g}_{j}(x)}}{\sum_{x^{\prime}}e^{\sum_{j=1}^{n_{1}-1}\theta^{j}\bar{g}_{j}(x^{\prime})}}$
(249) $\displaystyle\hat{P}_{\theta_{\mathsf{a}},\theta_{\mathsf{f}},XZ}(x,z)$
$\displaystyle:=\frac{e^{\sum_{j=1}^{n_{1}-1}\theta^{j}\bar{g}_{j}(x)+\sum_{j=n_{1}}^{n_{1}n_{2}-1}\theta^{j}\bar{g}_{j}(x,z)}}{\sum_{x^{\prime},z^{\prime}}e^{\sum_{j=1}^{n_{1}-1}\theta^{j}\bar{g}_{j}(x^{\prime})+\sum_{j=n_{1}}^{n_{1}n_{2}-1}\theta^{j}\bar{g}_{j}(x^{\prime},z^{\prime})}}$
(250) $\displaystyle\hat{P}_{\theta_{\mathsf{f}},Z|X}(z|x)$
$\displaystyle:=\frac{e^{\sum_{j=n_{1}}^{n_{1}n_{2}-1}\theta^{j}\bar{g}_{j}(x,z)}}{\sum_{z^{\prime}}e^{\sum_{j=n_{1}}^{n_{1}n_{2}-1}\theta^{j}\bar{g}_{j}(x,z^{\prime})}}.$
(251)

Then, we have

$\displaystyle\hat{P}_{\theta_{\mathsf{a}},\theta_{\mathsf{f}},Z|X}(z|x)=\frac{\hat{P}_{\theta_{\mathsf{a}},\theta_{\mathsf{f}},XZ}(x,z)}{\sum_{z^{\prime}}\hat{P}_{\theta_{\mathsf{a}},\theta_{\mathsf{f}},XZ}(x,z^{\prime})}.$
(252)

For
$\theta_{\mathsf{f}}=(\theta^{n_{1}},\ldots,\theta^{n_{1}n_{2}-1})\in\mathbb{R}^{n_{1}(n_{2}-1)}$,
we choose
$\psi_{\mathsf{a}}(\theta_{\mathsf{f}})=(\psi^{1},\ldots,\psi^{n_{1}-1})\in\mathbb{R}^{n_{1}-1}$
and $C_{X}(\theta_{\mathsf{f}})$ as

$\displaystyle\sum_{j=1}^{n_{1}-1}\psi^{j}\bar{g}_{j}(x)=-\log\sum_{z^{\prime}}e^{\sum_{j=n_{1}}^{n_{1}n_{2}-1}\theta^{j}\bar{g}_{j}(x,z^{\prime})}+C_{X}(\theta_{\mathsf{f}}).$
(253)

Also, we define
$\theta_{\mathsf{a}}^{\dagger}=(\theta^{1,\dagger},\ldots,\theta^{n_{1}-1,\dagger})\in\mathbb{R}^{n_{1}-1}$
and $C_{X,\dagger}$ as

$\displaystyle\sum_{j=1}^{n_{1}-1}\theta^{j,\dagger}\bar{g}_{j}(x)=-H(W_{x})+C_{X,\dagger}.$
(254)

The relation (254) gives the unique definition of $\theta_{1}^{\dagger}$
because the functions $\\{\bar{g}_{j}\\}_{j}$ and the constant form a basis of
the function space over ${\cal X}$.

###### Lemma 17

We have the following relations

$\displaystyle\hat{P}_{\theta_{1},X}(x)=$
$\displaystyle\sum_{z}\hat{P}_{\theta_{\mathsf{a}}+\psi_{1}(\theta_{\mathsf{f}}),\theta_{\mathsf{f}},XZ}(x,z)$
(255) $\displaystyle\hat{P}_{\theta_{\mathsf{a}},X}(x)=$ $\displaystyle
P_{\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\dagger},\theta_{\mathsf{d}}^{\dagger},X}(x).$
(256)

###### Proof

The relation (255) is shown as follows;

$\displaystyle\sum_{z}\hat{P}_{\theta_{\mathsf{a}}+\psi_{\mathsf{a}}(\theta_{\mathsf{f}}),\theta_{\mathsf{f}},XZ}(x,z)$
$\displaystyle=$
$\displaystyle\sum_{z}\frac{e^{\sum_{j=1}^{n_{1}-1}(\theta^{j}+\psi^{j})\bar{g}_{j}(x)+\sum_{j=n_{1}}^{n_{1}n_{2}-1}\theta^{j}\bar{g}_{j}(x,z)}}{\sum_{x^{\prime},z^{\prime}}e^{\sum_{j=1}^{n_{1}-1}(\theta^{j}+\psi^{j})\bar{g}_{j}(x^{\prime})+\sum_{j=n_{1}}^{n_{1}n_{2}-1}\theta^{j}\bar{g}_{j}(x^{\prime},z^{\prime})}}$
$\displaystyle\overset{(a)}{=}$
$\displaystyle\frac{e^{\sum_{j=1}^{n_{1}-1}\theta^{j}\bar{g}_{j}(x)+C_{X}(\theta_{\mathsf{f}})}}{\sum_{x^{\prime}}e^{\sum_{j=1}^{n_{1}-1}\theta^{j}\bar{g}_{j}(x^{\prime})+C_{X}(\theta_{\mathsf{f}})}}$
$\displaystyle=$
$\displaystyle\frac{e^{\sum_{j=1}^{n_{1}-1}\theta^{j}\bar{g}_{j}(x)}}{\sum_{x^{\prime}}e^{\sum_{j=1}^{n_{1}-1}\theta^{j}\bar{g}_{j}(x^{\prime})}}=\hat{P}_{\theta_{\mathsf{a}},X}(x),$
(257)

where $(a)$ follows from (253).

The relation (256) is shown as follows;

$\displaystyle
P_{\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\dagger},\theta_{\mathsf{d}}^{\dagger},X}(x)=$
$\displaystyle\frac{e^{\sum_{j=1}^{n_{1}-1}(\theta^{j}+\theta^{j,\dagger})\bar{g}_{j}(x)}\sum_{z,y}e^{\sum_{j=n_{1}}^{n_{1}n_{2}n_{3}-1}\theta^{j,\dagger}\xi_{j}(x,z,y)}}{\sum_{x^{\prime}}e^{\sum_{j=1}^{n_{1}-1}(\theta^{j}+\theta^{j,\dagger})\bar{g}_{j}(x^{\prime})}\sum_{z,y}e^{\sum_{j=n_{1}}^{n_{1}n_{2}n_{3}-1}\theta^{j,\dagger}\xi_{j}(x^{\prime},z,y)}}$
$\displaystyle\overset{(a)}{=}$
$\displaystyle\frac{e^{\sum_{j=1}^{n_{1}-1}(\theta^{j}+\theta^{j,\dagger})\bar{g}_{j}(x)+H(W_{x})}}{\sum_{x^{\prime}}e^{\sum_{j=1}^{n_{1}-1}(\theta^{j}+\theta^{j,\dagger})\bar{g}_{j}(x^{\prime})+H(W_{x^{\prime}})}}$
$\displaystyle\overset{(b)}{=}$
$\displaystyle\frac{e^{\sum_{j=1}^{n_{1}-1}\theta^{j}\bar{g}_{j}(x)+C_{X,\dagger}}}{\sum_{x^{\prime}}e^{\sum_{j=1}^{n_{1}-1}\theta^{j}\bar{g}_{j}(x^{\prime})+C_{X,\dagger}}}=\frac{e^{\sum_{j=1}^{n_{1}-1}\theta^{j}\bar{g}_{j}(x)}}{\sum_{x^{\prime}}e^{\sum_{j=1}^{n_{1}-1}\theta^{j}\bar{g}_{j}(x^{\prime})}}=\hat{P}_{\theta_{\mathsf{a}},X}(x),$
(258)

where $(a)$ and $(b)$ follows from (241) and (254), respectively.

To check Condition (B6), we define
$\theta_{\mathsf{a}}^{\ddagger}=(\theta^{1,\ddagger},\ldots,\theta^{n_{1}-1,\ddagger})$,
$C_{X,\ddagger}$, and
$\theta_{\mathsf{f}}^{\ddagger}=(\theta^{n_{1},\ddagger},\ldots,\theta^{n_{2}n_{1}-1,\ddagger})$
as

$\displaystyle\sum_{j=1}^{n_{1}-1}\theta^{j,\ddagger}\bar{g}_{j}(x)$
$\displaystyle=-H(W_{Z|x})+C_{X,\ddagger}$ (259) $\displaystyle W_{Z|i}(z)$
$\displaystyle=\frac{e^{\sum_{j=1}^{n_{2}-1}\theta^{n_{1}-1+(i-1)(n_{2}-1)+j,\ddagger}\bar{g}_{(i-1)(n_{2}-1)+j}(i,z)}}{\sum_{z^{\prime}}e^{\sum_{j=1}^{n_{2}-1}\theta^{n_{1}-1+(i-1)(n_{2}-1)+j,\ddagger}\bar{g}_{(i-1)(n_{2}-1)+j}(i,z^{\prime})}}$
(260)

for $i=1,\ldots,n_{1}$. Since the function $\bar{g}_{j}$ is defined by (232)
and (247), (260) is rewritten as

$\displaystyle
W_{Z|x}(z)=\frac{e^{\sum_{j=n_{1}}^{n_{1}n_{2}-1}\theta^{j,\ddagger}\bar{g}_{j}(x,z)}}{\sum_{z^{\prime}}e^{\sum_{j=n_{1}}^{n_{1}n_{2}-1}\theta^{j,\ddagger}\bar{g}_{j}(x,z^{\prime})}}.$
(261)

###### Lemma 18

We have the following relations.

$\displaystyle\hat{P}_{\theta_{\mathsf{f}}^{\ddagger},Z|X}(z|x)=$
$\displaystyle W_{Z|x}(z)$ (262)
$\displaystyle\theta_{\mathsf{a}}^{\ddagger}=$
$\displaystyle\psi_{\mathsf{a}}(\theta_{\mathsf{f}}^{\ddagger})$ (263)
$\displaystyle\hat{P}_{(\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\ddagger},\theta_{\mathsf{f}}^{\ddagger}),XZ}=$
$\displaystyle
P_{(\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\dagger},\theta_{\mathsf{d}}^{\dagger}),XZ}.$
(264)

###### Proof

The relation (262) follows from (261) and (251). The combination of (253),
(259), and (261) yields (263).

The relation (263) is shown as follows; Similar to (241), since (232) and
(247) guarantee $\sum_{z}\bar{g}_{j}(x,z)W_{Z|x}(z)=0$, (261) implies

$\displaystyle
H(W_{Z|x})=-\log\sum_{z^{\prime}}e^{\sum_{j=n_{1}}^{n_{1}n_{2}-1}\theta^{j}\bar{g}_{j}(x,z^{\prime})}.$
(265)

The combination of (253), (259), and (265) yields

$\displaystyle\sum_{j=1}^{n_{1}-1}\psi^{j}\bar{g}_{j}(x)-C_{X}(\theta_{\mathsf{f}})=\sum_{j=1}^{n_{1}-1}\theta^{j,\ddagger}\bar{g}_{j}(x)-C_{X,\ddagger}.$
(266)

Since the functions $\\{\bar{g}_{j}\\}_{j}$ and the constant are linearly
independent, we obtain (263).

The relation (264) is shown as follows;

$\displaystyle
P_{\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\dagger},\theta_{\mathsf{d}}^{\dagger},XZ}(x)$
$\displaystyle=$
$\displaystyle\frac{e^{\sum_{j=1}^{n_{1}-1}(\theta^{j}+\theta^{j,\dagger})\bar{g}_{j}(x)}\sum_{y}e^{\sum_{j=n_{1}}^{n_{1}n_{2}n_{3}-1}\theta^{j,\dagger}\xi_{j}(x,z,y)}}{\sum_{x^{\prime},y^{\prime},z^{\prime}}e^{\sum_{j=1}^{n_{1}-1}(\theta^{j}+\theta^{j,\dagger})\bar{g}_{j}(x^{\prime})}e^{\sum_{j=n_{1}}^{n_{1}n_{2}n_{3}-1}\theta^{j,\dagger}\xi_{j}(x^{\prime},z^{\prime},y^{\prime})}}$
$\displaystyle=$
$\displaystyle\frac{e^{\sum_{j=1}^{n_{1}-1}(\theta^{j}+\theta^{j,\dagger})\bar{g}_{j}(x)}\sum_{z^{\prime},y^{\prime}}e^{\sum_{j=n_{1}}^{n_{1}n_{2}n_{3}-1}\theta^{j,\dagger}\xi_{j}(x,z^{\prime},y^{\prime})}}{\sum_{x^{\prime},y^{\prime},z^{\prime}}e^{\sum_{j=1}^{n_{1}-1}(\theta^{j}+\theta^{j,\dagger})\bar{g}_{j}(x^{\prime})}e^{\sum_{j=n_{1}}^{n_{1}n_{2}n_{3}-1}\theta^{j,\dagger}\xi_{j}(x^{\prime},z^{\prime},y^{\prime})}}$
$\displaystyle\cdot\frac{\sum_{y}e^{\sum_{j=n_{1}}^{n_{1}n_{2}n_{3}-1}\theta^{j,\dagger}\xi_{j}(x,z,y)}}{\sum_{z^{\prime},y^{\prime}}e^{\sum_{j=n_{1}}^{n_{1}n_{2}n_{3}-1}\theta^{j,\dagger}\xi_{j}(x,z^{\prime},y^{\prime})}}$
$\displaystyle\overset{(a)}{=}$
$\displaystyle\hat{P}_{\theta_{\mathsf{a}},X}(x)\sum_{y}W_{x}(z,y)=\hat{P}_{\theta_{\mathsf{a}},X}(x)W_{Z|x}(z)\overset{(b)}{=}\hat{P}_{\theta_{\mathsf{a}},X}(x)\hat{P}_{\theta_{\mathsf{f}}^{\ddagger},Z|X}(z|x)$
$\displaystyle\overset{(c)}{=}$
$\displaystyle\frac{e^{\sum_{j=1}^{n_{1}-1}\theta^{j}\bar{g}_{j}(x)}}{\sum_{x^{\prime}}e^{\sum_{j=1}^{n_{1}-1}\theta^{j}\bar{g}_{j}(x^{\prime})}}\frac{e^{\sum_{j=n_{1}}^{n_{1}n_{2}-1}\theta^{j,\ddagger}\bar{g}_{j}(x,z)}}{\sum_{z^{\prime}}e^{\sum_{j=n_{1}}^{n_{1}n_{2}-1}\theta^{j,\ddagger}\bar{g}_{j}(x,z^{\prime})}}$
$\displaystyle\overset{(d)}{=}$
$\displaystyle\frac{e^{\sum_{j=1}^{n_{1}-1}(\theta^{j}+\psi_{1}^{j}(\theta_{\mathsf{f}}^{\ddagger}))\bar{g}_{j}(x)}e^{\sum_{j=n_{1}}^{n_{1}n_{2}-1}\theta^{j,\ddagger}\bar{g}_{j}(x,z)}}{\sum_{x^{\prime}}e^{\sum_{j=1}^{n_{1}-1}\theta^{j}\bar{g}_{j}(x^{\prime})}e^{C_{X}(\theta_{\mathsf{f}}^{\ddagger})}}$
$\displaystyle\overset{(e)}{=}$
$\displaystyle\frac{e^{\sum_{j=1}^{n_{1}-1}(\theta^{j}+\theta^{j,\ddagger})\bar{g}_{j}(x)+\sum_{j=n_{1}}^{n_{1}n_{2}-1}\theta^{j,\ddagger}\bar{g}_{j}(x,z)}}{\sum_{x^{\prime},z^{\prime}}e^{\sum_{j=1}^{n_{1}-1}(\theta^{j}+\theta^{j,\ddagger})\bar{g}_{j}(x^{\prime})+\sum_{j=n_{1}}^{n_{1}n_{2}-1}\theta^{j,\ddagger}\bar{g}_{j}(x^{\prime},z^{\prime})}}$
$\displaystyle=$
$\displaystyle\hat{P}_{(\theta_{1}+\theta_{1}^{\ddagger},\theta_{2}^{\ddagger}),XZ}(x,z),$
(267)

where each step can be shown as follows. $(a)$ follows from (239) and (264).
$(b)$ follows from (262). $(c)$ follows from (249) and (251). $(d)$ follows
from (254). $(e)$ follows from (263) and the fact that the denominator
$\sum_{x^{\prime}}e^{\sum_{j=1}^{n_{1}-1}\theta^{j}\bar{g}_{j}(x^{\prime})}e^{C_{X}(\theta_{\mathsf{f}}^{\ddagger})}$
is a constant that does not depend on $x,z$.

For any element $\theta_{\mathsf{a}}=$ $(\theta^{1},\ldots,$
$\theta^{n_{1}-1})^{T}\in\Theta_{{\cal M}}$, we define
$\phi_{\mathsf{g}}(\theta_{\mathsf{a}})=(\phi^{n_{1}n_{2}},\ldots,$
$\phi^{n_{2}(n_{1}+n_{3}-1)-1})^{T}$ as follows.

$\displaystyle
P_{(\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\dagger},\theta_{\mathsf{d}}^{\dagger}),Y|Z}(y|z)=\frac{e^{\sum_{j=n_{1}n_{2}}^{n_{2}(n_{1}+n_{2}-1)-1}\phi^{j}\bar{g}_{j}(z,y)}}{\sum_{y^{\prime}\in{\mathcal{Y}}}e^{\sum_{j=n_{1}n_{2}}^{n_{2}(n_{1}+n_{2}-1)-1}\phi^{j}\bar{g}_{j}(z,y^{\prime})}}$
(268)

for $z\in{\mathcal{Z}},y\in{\mathcal{Y}}$. Then, for any element
$\theta_{\mathsf{a}}=(\theta^{1},\ldots,\theta^{n_{1}-1})^{T}\in\Theta_{{\cal
M}}$, we choose
$\phi_{\mathsf{f}}(\theta_{\mathsf{a}})=(\phi^{n_{1}},\ldots,\phi^{n_{1}n_{2}-1})^{T}$
and $C_{XZ}(\theta_{\mathsf{a}})$ such that

$\displaystyle\sum_{j=n_{1}}^{n_{1}n_{2}-1}\phi^{j}\bar{g}_{j}(x,z)$
$\displaystyle=$
$\displaystyle-\log\Big{(}\sum_{y^{\prime}\in{\mathcal{Y}}}e^{\sum_{j=n_{1}n_{2}}^{n_{2}(n_{1}+n_{2}-1)-1}\phi^{j}\bar{g}_{j}(z,y^{\prime})}\Big{)}+C_{XZ}(\theta_{1})$
(269)

for $x\in{\mathcal{X}},z\in{\mathcal{Z}}$. Then, we prepare the following
lemma.

###### Lemma 19

The relation

$\displaystyle
P_{(\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\dagger},\theta_{\mathsf{d}}^{\dagger}),Y|Z}\times
P_{(\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\dagger},\theta_{\mathsf{f}}^{\dagger}),XZ}=P_{V(\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\ddagger},\theta_{\mathsf{f}}^{\ddagger}+\phi_{\mathsf{f}}(\theta_{\mathsf{a}}),\phi_{\mathsf{g}}(\theta_{\mathsf{a}}))^{T}}$
(270)

holds for $\theta_{\mathsf{a}}\in\Theta_{{\cal M}}$.

###### Proof

We have

$\displaystyle
P_{V(\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\ddagger},\theta_{\mathsf{f}}^{\ddagger}+\phi_{\mathsf{f}}(\theta_{\mathsf{a}}),\phi_{\mathsf{g}}(\theta_{\mathsf{a}}))^{T},XZ}(x,z)$
$\displaystyle\overset{(a)}{=}$
$\displaystyle\frac{e^{\sum_{j=1}^{n_{1}-1}(\theta^{j}+\theta^{j,\ddagger})\bar{g}_{j}(x)+\sum_{j=n_{1}}^{n_{1}n_{2}-1}(\theta^{j,\ddagger}+\phi^{j})\bar{g}_{j}(x,z)}\sum_{y}e^{\sum_{j=n_{1}n_{2}}^{n_{2}(n_{1}+n_{2}-1)-1}\phi^{j}\bar{g}_{j}(z,y)}}{\sum_{x^{\prime},z^{\prime}}e^{\sum_{j=1}^{n_{1}-1}(\theta^{j}+\theta^{j,\ddagger})\bar{g}_{j}(x^{\prime})+\sum_{j=n_{1}}^{n_{1}n_{2}-1}(\theta^{j,\ddagger}+\phi^{j})\bar{g}_{j}(x^{\prime},z^{\prime})}\sum_{y^{\prime}}e^{\sum_{j=n_{1}n_{2}}^{n_{2}(n_{1}+n_{2}-1)-1}\phi^{j}\bar{g}_{j}(z^{\prime},y^{\prime})}}$
$\displaystyle\overset{(b)}{=}$
$\displaystyle\frac{e^{\sum_{j=1}^{n_{1}-1}(\theta^{j}+\theta^{j,\ddagger})\bar{g}_{j}(x)+\sum_{j=n_{1}}^{n_{1}n_{2}-1}\theta^{j,\ddagger}\bar{g}_{j}(x,z)+C_{XZ}(\theta_{1})}}{\sum_{x^{\prime},z^{\prime}}e^{\sum_{j=1}^{n_{1}-1}(\theta^{j}+\theta^{j,\ddagger})\bar{g}_{j}(x^{\prime})+\sum_{j=n_{1}}^{n_{1}n_{2}-1}\theta^{j,\ddagger}\bar{g}_{j}(x^{\prime},z^{\prime})+C_{XZ}(\theta_{1})}}$
$\displaystyle=$
$\displaystyle\frac{e^{\sum_{j=1}^{n_{1}-1}(\theta^{j}+\theta^{j,\ddagger})\bar{g}_{j}(x)+\sum_{j=n_{1}}^{n_{1}n_{2}-1}\theta^{j,\ddagger}\bar{g}_{j}(x,z)}}{\sum_{x^{\prime},z^{\prime}}e^{\sum_{j=1}^{n_{1}-1}(\theta^{j}+\theta^{j,\ddagger})\bar{g}_{j}(x^{\prime})+\sum_{j=n_{1}}^{n_{1}n_{2}-1}\theta^{j,\ddagger}\bar{g}_{j}(x^{\prime},z^{\prime})}}$
$\displaystyle\overset{(c)}{=}$
$\displaystyle\hat{P}_{\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\ddagger},\theta_{2}^{\ddagger},XZ}(x,z),$
(271)

where $(a)$, $(b)$, and $(c)$ follow from (234), (269), and (250),
respectively. The combination of (264) and (271) yields that

$\displaystyle
P_{V(\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\ddagger},\theta_{\mathsf{f}}^{\ddagger}+\phi_{\mathsf{f}}(\theta_{\mathsf{a}}),\phi_{\mathsf{g}}(\theta_{\mathsf{a}}))^{T},XZ}=P_{(\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\dagger},\theta_{\mathsf{d}}^{\dagger}),XZ}.$
(272)

In the same way as $(a)$ of (271), we have

$\displaystyle
P_{V(\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\ddagger},\theta_{\mathsf{f}}^{\ddagger}+\phi_{\mathsf{f}}(\theta_{\mathsf{a}}),\phi_{\mathsf{g}}(\theta_{\mathsf{a}}))^{T},ZY}(z,y)$
$\displaystyle=$
$\displaystyle\frac{\sum_{x}e^{\sum_{j=1}^{n_{1}-1}(\theta^{j}+\theta^{j,\ddagger})\bar{g}_{j}(x)+\sum_{j=n_{1}}^{n_{1}n_{2}-1}(\theta^{j,\ddagger}+\phi^{j})\bar{g}_{j}(x,z)}e^{\sum_{j=n_{1}n_{2}}^{n_{2}(n_{1}+n_{2}-1)-1}\phi^{j}\bar{g}_{j}(z,y)}}{\sum_{x^{\prime},z^{\prime},y^{\prime}}e^{\sum_{j=1}^{n_{1}-1}(\theta^{j}+\theta^{j,\ddagger})\bar{g}_{j}(x^{\prime})+\sum_{j=n_{1}}^{n_{1}n_{2}-1}(\theta^{j,\ddagger}+\phi^{j})\bar{g}_{j}(x^{\prime},z^{\prime})}e^{\sum_{j=n_{1}n_{2}}^{n_{2}(n_{1}+n_{2}-1)-1}\phi^{j}\bar{g}_{j}(z^{\prime},y^{\prime})}}$
$\displaystyle=$
$\displaystyle\tau(z)e^{\sum_{j=n_{1}n_{2}}^{n_{2}(n_{1}+n_{2}-1)-1}\phi^{j}\bar{g}_{j}(z,y)},.$
(273)

where $\tau(z)$ is a constant that depends only on $z$. Hence,

$\displaystyle
P_{V(\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\ddagger},\theta_{\mathsf{f}}^{\ddagger}+\phi_{\mathsf{f}}(\theta_{\mathsf{a}}),\phi_{\mathsf{g}}(\theta_{\mathsf{a}}))^{T},Y|Z}(y|z)$
$\displaystyle=$
$\displaystyle\frac{e^{\sum_{j=n_{1}n_{2}}^{n_{2}(n_{1}+n_{2}-1)-1}\phi^{j}\bar{g}_{j}(z,y)}}{\sum_{y^{\prime}\in{\mathcal{Y}}}e^{\sum_{j=n_{1}n_{2}}^{n_{2}(n_{1}+n_{2}-1)-1}\phi^{j}\bar{g}_{j}(z,y^{\prime})}}.$
(274)

Thus, (268) and (274) yield that

$\displaystyle
P_{V(\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\ddagger},\theta_{\mathsf{f}}^{\ddagger}+\phi_{\mathsf{f}}(\theta_{\mathsf{a}}),\phi_{\mathsf{g}}(\theta_{\mathsf{a}}))^{T},Y|Z}=P_{(\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\dagger},\theta_{\mathsf{d}}^{\dagger}),Y|Z}.$
(275)

Therefore, the combination of (275) and (272) implies (270).

For $i=1,\ldots,n_{1}-1$, we have

$\displaystyle\nabla_{i}^{(e)}[F](V(\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\ddagger},\theta_{\mathsf{f}}^{\ddagger}+\phi_{\mathsf{f}}(\theta_{\mathsf{a}}),\phi_{\mathsf{g}}(\theta_{\mathsf{a}}))^{T})$
$\displaystyle=$
$\displaystyle\sum_{x,y,z}g_{i}(x,y,z)P_{V(\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\ddagger},\theta_{\mathsf{f}}^{\ddagger}+\phi_{\mathsf{f}}(\theta_{\mathsf{a}}),\phi_{\mathsf{g}}(\theta_{\mathsf{a}}))^{T}}(x,y,z)$
$\displaystyle=$
$\displaystyle\sum_{x,y,z}\delta_{i}(x)P_{V(\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\ddagger},\theta_{\mathsf{f}}^{\ddagger}+\phi_{\mathsf{f}}(\theta_{\mathsf{a}}),\phi_{\mathsf{g}}(\theta_{\mathsf{a}}))^{T}}(x,y,z)$
$\displaystyle\overset{(a)}{=}$
$\displaystyle\sum_{x,y,z}\delta_{i}(x)P_{(\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\dagger},\theta_{\mathsf{d}}^{\dagger}),Y|Z}(y|z)P_{(\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\dagger},\theta_{\mathsf{f}}^{\dagger}),XZ}(x,z)$
$\displaystyle=$
$\displaystyle\sum_{x,z}\delta_{i}(x)P_{\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\dagger},\theta_{\mathsf{d}}^{\dagger},XZ}(x,z)$
$\displaystyle=$
$\displaystyle\sum_{x,y,z}g_{i}(x,y,z)P_{\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\dagger},\theta_{\mathsf{d}}^{\dagger}}(x,y,z)=\nabla_{i}^{(e)}[F_{{\cal
M}}](\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\dagger}),$ (276)

where $(a)$ follows from Lemma 19.

For $i^{\prime}=n_{1},\ldots,n_{2}(n_{1}+n_{3}-1)-1$, we have

$\displaystyle\nabla_{i^{\prime}}^{(e)}[F](V(\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\ddagger},\theta_{\mathsf{f}}^{\ddagger}+\phi_{\mathsf{f}}(\theta_{\mathsf{a}}),\phi_{\mathsf{g}}(\theta_{\mathsf{a}}))^{T})$
$\displaystyle=$
$\displaystyle\sum_{x,y,z}g_{i^{\prime}}(x,y,z)P_{V(\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\ddagger},\theta_{\mathsf{f}}^{\ddagger}+\phi_{\mathsf{f}}(\theta_{\mathsf{a}}),\phi_{\mathsf{g}}(\theta_{\mathsf{a}}))^{T}}(x,y,z)=0.$
(277)

Combining (238), (276), and (277), for $j=1,\ldots,n_{2}(n_{1}+n_{3}-1)-1$, we
have

$\displaystyle\nabla_{j}^{(e)}[F_{{\cal
E}}](\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\ddagger},\theta_{\mathsf{f}}^{\ddagger}+\phi_{\mathsf{f}}(\theta_{\mathsf{a}}),\phi_{\mathsf{g}}(\theta_{\mathsf{a}}))$
$\displaystyle=$
$\displaystyle\sum_{i=1}^{n_{2}(n_{1}+n_{3}-1)-1}\nabla_{i}^{(e)}[F](V(\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\ddagger},\theta_{\mathsf{f}}^{\ddagger}+\phi_{\mathsf{f}}(\theta_{\mathsf{a}}),\phi_{\mathsf{g}}(\theta_{\mathsf{a}}))^{T})v_{j}^{i}$
$\displaystyle=$ $\displaystyle\sum_{i=1}^{n_{1}-1}\nabla_{i}^{(e)}[F_{{\cal
M}}](\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\dagger})v_{j}^{i}=(\nabla^{(e)}[F_{{\cal
M}}](\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\dagger})V_{1})_{j}.$ (278)

When $\eta_{\mathsf{a}}=\nabla^{(e)}[F_{{\cal
M}}](\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\dagger})$, (278) guarantees
that $\eta_{\mathsf{a}}V_{1}=\nabla^{(e)}[F_{{\cal
E}}](\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\ddagger},\theta_{\mathsf{f}}^{\ddagger}+\phi_{\mathsf{f}}(\theta_{\mathsf{a}}),\phi_{\mathsf{g}}(\theta_{\mathsf{a}}))$,
which implies that $\nabla_{\mathsf{a}}^{(m)}[F_{{\cal
E}}^{*}](\eta_{\mathsf{a}}V_{1})=\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\ddagger}$.
Thus, we have

$\displaystyle\nabla^{(m)}[F_{{\cal
M}}^{*}](\eta_{\mathsf{a}})-\nabla_{\mathsf{a}}^{(m)}[F_{{\cal
E}}^{*}](\eta_{\mathsf{a}}V_{1})=\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\dagger}-(\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\ddagger})=\theta_{\mathsf{a}}^{\dagger}-\theta_{\mathsf{a}}^{\ddagger},$
(279)

which implies Condition (B6).

In the following, we assume that the rank of $H$ is $n_{1}-1$. We choose the
parameter
$\theta_{\mathsf{b}}^{\ddagger}\in\mathbb{R}^{n_{1}(n_{2}-1)+n_{2}(n_{3}-1)}$
such that
$(0_{n_{1}-1,n_{1}(n_{2}-1)},H)\theta_{\mathsf{b}}^{\ddagger}=\theta_{1}^{\ddagger}$.
We choose
$(n_{1}(n_{2}-1)+n_{2}(n_{3}-1))\times(n_{1}(n_{2}-2)+n_{2}(n_{3}-1)+1)$
matrix $G$ such that $\mathop{\rm Im}G=\mathop{\rm
Ker}(0_{n_{1}-1,n_{1}(n_{2}-1)},H)$. Then, $\overline{\cal E}$ and
$\overline{\cal M}$ defined in (113) and (114) are written as

$\displaystyle\overline{\cal E}=$
$\displaystyle\\{(\theta_{\mathsf{a}},\theta_{\mathsf{b}}^{\ddagger}+G\theta_{\mathsf{e}})^{T}|\theta_{1}\in\mathbb{R}^{n_{1}-1},\theta_{\mathsf{e}}\in\mathbb{R}^{n_{1}(n_{2}-2)+n_{2}(n_{3}-1)+1}\\}$
(280) $\displaystyle\overline{\cal M}=$
$\displaystyle\left\\{(\theta_{\mathsf{a}},\theta_{\mathsf{b}})^{T}\left|\begin{array}[]{l}(\eta_{\mathsf{a}},\eta_{\mathsf{b}})=\nabla^{(e)}[F_{{\cal
E}}]((\theta_{\mathsf{a}},\theta_{\mathsf{b}})^{T}),\\\
\eta^{1}(0_{n_{1}-1,n_{1}(n_{2}-1)},H)-\eta_{\mathsf{b}}=0\end{array}\right.\right\\}.$
(283)

We choose the parameter
$\theta_{\mathsf{g}}^{\ddagger}\in\mathbb{R}^{n_{2}(n_{3}-1)}$ such that
$H\theta_{\mathsf{g}}^{\ddagger}=\theta_{1}^{\ddagger}$. We choose
$n_{2}(n_{3}-1)\times(n_{2}(n_{3}-1)+1-n_{1})$ matrix $G$ such that
$\mathop{\rm Im}G=\mathop{\rm Ker}H$. Then, $\overline{\cal E}$ and
$\overline{\cal M}$ defined in (251) and (114) are written as

$\displaystyle\overline{\cal E}=$
$\displaystyle\\{(\theta_{\mathsf{a}},\theta_{\mathsf{f}},\theta_{\mathsf{g}}^{\ddagger}+G\theta_{\mathsf{h}})^{T}|\theta_{\mathsf{a}}\in\mathbb{R}^{n_{1}-1},\theta_{\mathsf{f}}\in\mathbb{R}^{n_{1}(n_{2}-1)},\theta_{\mathsf{h}}\in\mathbb{R}^{n_{2}(n_{3}-1)-n_{1}+1}\\}$
(284) $\displaystyle\overline{\cal M}=$
$\displaystyle\\{(\theta_{\mathsf{a}},\theta_{\mathsf{f}},\theta_{\mathsf{g}})^{T}|(\eta_{\mathsf{a}},\eta_{\mathsf{f}},\eta_{\mathsf{g}})=\nabla^{(e)}[F_{{\cal
E}}]((\theta_{\mathsf{a}},\theta_{\mathsf{f}},\theta_{\mathsf{g}})^{T}),~{}\eta_{\mathsf{f}}=0,~{}\eta_{\mathsf{a}}H=\eta_{\mathsf{g}}\\}.$
(285)

Therefore, when the intersection $\overline{\cal E}\cap\overline{\cal M}$ is
not empty, due to Corollary 4, the maximization (47) is written by using an
element $(\theta_{\mathsf{a},*},\theta_{\mathsf{b},*})^{T}\in\overline{\cal
E}\cap\overline{\cal M}$ as

$\displaystyle C_{\sup}({\cal M},{\cal E})=D^{F}(\phi^{(e)}_{{\cal
M}}(\Gamma_{{\cal
M}}^{(m),F}((\bar{\theta}_{\mathsf{a},*},\bar{\theta}_{\mathsf{b},*})^{T})\|\phi^{(e)}_{{\cal
E}}((\bar{\theta}_{\mathsf{a},*},\bar{\theta}_{\mathsf{b},*})^{T}).$ (286)

##  7 Capacity of classical-quantum channel

###  7.1 Problem setting

Next, we discuss a classical-quantum channel from the classical system ${\cal
C}:=\\{1,\ldots,n_{1}\\}$ to the quantum system ${\cal H}_{A}$ with dimension
$n_{2}$, which is given as a set of density matrices
$\\{W_{j}\\}_{j=1}^{n_{1}}$. Under this classical-quantum channel, given an
input probability distribution $(p_{j})$ on the classical system ${\cal C}$,
we define the classical-quantum state
$\rho[p]:=\sum_{j=1}^{n_{1}}p_{j}|j\rangle\langle j|\otimes W_{j}$ on ${\cal
H}_{A}\otimes{\cal H}_{C}$, where ${\cal H}_{C}$ is spanned by
$\\{|i\rangle\\}_{i=1}^{n_{1}}$. Then, we denote the partial trace for ${\cal
C}$ and ${\cal H}_{A}$ by $\operatorname{\mathrm{Tr}}_{C}$ and
$\operatorname{\mathrm{Tr}}_{A}$, respectively. The Hovelo quantity is defined
as

$\displaystyle\sum_{j=1}^{k}p_{j}D\bigg{(}W_{j}\bigg{\|}\sum_{j^{\prime}=1}^{k}p_{j^{\prime}}W_{j^{\prime}}\bigg{)}=D\big{(}\rho[p]\big{\|}(\operatorname{\mathrm{Tr}}_{C}\rho[p])\otimes(\operatorname{\mathrm{Tr}}_{A}\rho[p])\big{)}.$
(287)

The capacity of the classical-quantum channel $\\{W_{j}\\}_{j=1}^{k}$ is
defined as the maximum

$\displaystyle\max_{p\in{\cal P}_{{\cal
C}}}\sum_{j=1}^{k}p_{j}D\bigg{(}W_{j}\bigg{\|}\sum_{j^{\prime}=1}^{k}p_{j^{\prime}}W_{j^{\prime}}\bigg{)}=\max_{p\in{\cal
P}_{{\cal C}}}\min_{\rho\in{\cal S}({\cal H}_{A}),q\in{\cal P}_{{\cal
C}}}D(\rho[p]\|\rho\otimes q),$ (288)

where the distribution $q$ is identified with the state
$\sum_{i=1}^{n_{1}}q_{i}|i\rangle\langle i|$. This quantity expresses the
maximum transmission rate of classical information when we employ the
classical-quantum channel $\\{W_{j}\\}_{j=1}^{k}$ Holevo ; SW .

The set of $\rho^{\prime\prime}\otimes q$ forms an exponential subfamily
${\cal E}$ and the set of $\rho[p]$ forms a mixture subfamily ${\cal M}$.
Hence, the maximization problem (288) is a special case of the maximization
(47) with $k=n_{1}-1$, $l=n_{1}+n_{2}^{2}-2$, and $d=n_{1}n_{2}^{2}-1$. As
shown as Lemma 8, condition (B1) holds. In the following, we apply Algorithm
4.

###  7.2 Constructions of vectors
$u_{1},\ldots,u_{n_{1}n_{2}^{2}-1},v_{1},\ldots,v_{n_{1}+n_{2}^{2}-2}$

For this aim, we need to choose a suitable coordinate to satisfy conditions
(B3), (B4), and (B5) and check Condition (B2). For this aim, we choose
$n_{2}^{2}-1$ linearly independent Hermitian matrices $X_{j}$ on ${\cal
H}_{A}$ for $j=1,\ldots,n_{2}^{2}-1$ to satisfy the condition that

$\displaystyle\operatorname{\mathrm{Tr}}X_{j}W_{n_{1}}=0$ (289)

and the linear space spanned by $X_{1},\ldots,X_{n_{2}^{2}-1}$ does not
contain the identity matrix. Then, we define the Hermitian matrices
$\xi_{1},\ldots,\xi_{n_{1}n_{2}^{2}-1}$ on ${\cal H}_{A}\otimes{\cal H}_{C}$
as follows. We define $\xi_{i}:=I_{A}\otimes|i\rangle\langle i|$ for
$i=1,\ldots,n_{1}-1$. We define
$\xi_{n_{1}-1+(i-1)(n_{2}-1)+j}:=(X_{j}-h_{i,j}I_{A})\otimes|i\rangle\langle
i|$ for $i=1,\ldots,n_{1}-1$ and $j=1,\ldots,n_{2}-1$, where
$h_{i,j}:=\operatorname{\mathrm{Tr}}X_{j}W_{i}$. We define
$\xi_{(n_{1}-1)n_{2}+j}(x,y):=X_{j}\otimes|n_{1}\rangle\langle n_{1}|$ for
$j=1,\ldots,n_{2}-1$. Then, we define the $C^{\infty}-$strictly convex
function $F$ on $\mathbb{R}^{n_{1}n_{2}-1}$ as

$\displaystyle
F(\theta):=\log\operatorname{\mathrm{Tr}}\exp\Big{(}\sum_{i=1}^{n_{1}n_{2}^{2}-1}\theta^{i}\xi_{i}\Big{)}.$
(290)

We define the density matrices $\rho_{\theta},\rho_{\theta,C},\rho_{\theta,A}$
as

$\displaystyle\rho_{\theta}$
$\displaystyle:=e^{\sum_{i=1}^{n_{1}n_{2}^{2}-1}\theta^{i}\xi_{i}-F(\theta)},$
(291) $\displaystyle\rho_{\theta,C}$
$\displaystyle:=\operatorname{\mathrm{Tr}}_{A}e^{\sum_{i=1}^{n_{1}n_{2}^{2}-1}\theta^{i}\xi_{i}-F(\theta)},$
(292) $\displaystyle\rho_{\theta,A}$
$\displaystyle:=\operatorname{\mathrm{Tr}}_{C}e^{\sum_{i=1}^{n_{1}n_{2}^{2}-1}\theta^{i}\xi_{i}-F(\theta)}.$
(293)

Then, as a special case of (43), we have

$\displaystyle
D^{F}(\theta\|\theta^{\prime})=D(\rho_{\theta}\|\rho_{\theta^{\prime}}).$
(294)

Next, we choose the Matrix $U$ as the identity matrix, and
$u_{1},\ldots,u_{n_{1}n_{2}^{2}-1}$ are chosen as its $n_{1}n_{2}^{2}-1$
column vectors. Then, we define vector $v_{1},\ldots,v_{n_{1}+n_{2}^{2}-2}$ as
follows, whereas $V=(v_{1},\ldots,v_{n_{1}+n_{2}^{2}-2})$. We define
$v_{i}:=u_{i}$ for $i=1,\ldots,n_{1}-1$. We define
$v_{n_{1}-1+j}:=\sum_{i=1}^{n_{1}}u_{n_{1}-1+(i-1)(n_{2}^{2}-1)+j}+\sum_{i=1}^{n_{1}-1}h_{i,j}u_{i}$
for $j=1,\ldots,n_{2}^{2}-1$. Then, we have

$\displaystyle\sum_{i=1}^{n_{1}-1}\xi_{i}v_{j}^{i}=\left\\{\begin{array}[]{ll}I_{A}\otimes|j\rangle\langle
j|&\hbox{when }j=1,\ldots,n_{1}-1\\\ X_{j-n_{1}+1}\otimes I_{C}&\hbox{when
}j=n_{1},\ldots,n_{1}+n_{2}^{2}-1.\end{array}\right.$ (297)

###  7.3 Parameterizations of ${\cal E}$ and ${\cal M}$

Using $F_{{\cal
E},\mathsf{a}}(\theta^{1},\ldots,\theta^{n_{1}-1}):=\log\sum_{x}e^{\sum_{i=1}^{n_{1}-1}{\theta}^{i}\delta_{i}(x)}$
and $F_{{\cal
E},\mathsf{b}}(\theta^{n_{1}},\ldots,\theta^{n_{1}+n_{2}^{2}-2}):=\log\operatorname{\mathrm{Tr}}e^{\sum_{j=1}^{n_{2}^{2}-1}{\theta}^{n_{1}-1+j}X_{j}}$,
we define the distributions on ${\mathcal{X}}$ and ${\mathcal{Y}}$ as

$\displaystyle\bar{\rho}_{\theta_{\mathsf{a}},C}:=$ $\displaystyle
e^{\sum_{i=1}^{n_{1}-1}{\theta}^{i}|i\rangle\langle i|-F_{{\cal
E},\mathsf{a}}(\theta_{\mathsf{a}})}$ (298)
$\displaystyle\bar{\rho}_{\theta_{\mathsf{b}},A}:=$ $\displaystyle
e^{\sum_{j=1}^{n_{2}-1}{\theta}^{n_{1}-1+j}X_{j}-F_{{\cal
E},\mathsf{b}}(\theta_{\mathsf{b}})}$ (299)

for $\theta_{\mathsf{a}}:=(\theta^{1},\ldots,\theta^{n_{1}-1})$ and
$\theta_{\mathsf{b}}:=(\theta^{n_{1}},\ldots,\theta^{n_{1}+n_{2}^{2}-2})$.
Then, we have

$\displaystyle\rho_{\sum_{j=1}^{n_{1}-1}\theta_{\mathsf{a}}^{j}v_{j}+\sum_{j^{\prime}=1}^{n_{2}^{2}-1}\theta_{\mathsf{b}}^{j}v_{n_{1}-1+j}}=\bar{\rho}_{\theta_{\mathsf{a}},C}\otimes\bar{\rho}_{\theta_{\mathsf{b}},A}.$
(300)

Hence, the set of product states is written as the exponential subfamily
${\cal E}:=\\{\rho_{\sum_{j=1}^{n_{1}+n_{2}^{2}-2}{\theta}^{j}v_{j}}\\}$
generated by $v_{1},\ldots,v_{n_{1}+n_{2}-2}$ at the point $(0,\ldots,0)$.

We define the mixture family ${\cal M}$ by the constraint
$\sum_{i=1}^{n_{1}n_{2}^{2}-1}u^{i}_{n_{1}-1+j^{\prime}}\partial_{i}F(\theta)=0$
for $j^{\prime}=1,\ldots,n_{1}(n_{2}^{2}-1)$. This constraint is equivalent to

$\displaystyle\operatorname{\mathrm{Tr}}\big{(}(X_{j}-h_{i,j}I)\otimes|i\rangle\langle
i|\big{)}\rho_{\theta}=0,\quad\operatorname{\mathrm{Tr}}\big{(}X_{j}\otimes|n_{1}\rangle\langle
n_{1}|\big{)}\rho_{\theta}=0$ (301)

for $i=1,\ldots,n_{1}-1$ and $j=1,\ldots,n_{2}^{2}-1$. Hence, the mixture
family ${\cal M}$ is composed of density matrices with the form $W\times q$.
Thus, the problem (129) is written as the problem (47) with the above defined
${\cal E}$ and ${\cal M}$.

we choose
$\theta_{\mathsf{b}}^{\dagger}=(\theta^{n_{1},\dagger},\ldots,\theta^{n_{1}n_{2}^{2}-1,\dagger})$
as

$\displaystyle
W_{i}=\frac{e^{\sum_{j=1}^{n_{2}^{2}-1}\theta^{n_{1}-1+(i-1)(n_{2}^{2}-1)+j,\dagger}(X_{j}-h_{i,j})}}{\operatorname{\mathrm{Tr}}e^{\sum_{j^{\prime}=1}^{n_{2}^{2}-1}\theta^{n_{1}-1+(i-1)(n_{2}^{2}-1)+j^{\prime},\dagger}(X_{j^{\prime}}-h_{i,j^{\prime}})}}.$
(302)

In this choice, we have

$\displaystyle\log\operatorname{\mathrm{Tr}}e^{\sum_{j^{\prime}=1}^{n_{2}^{2}-1}\theta^{n_{1}-1+(i-1)(n_{2}^{2}-1)+j^{\prime},\dagger}(X_{j^{\prime}}-h_{i,j^{\prime}})}=H(W_{i}).$
(303)

Then, ${\cal M}$ is written as
$\\{(\theta_{\mathsf{a}},\theta_{\mathsf{b}}^{\dagger})|\theta_{1}\in\mathbb{R}^{n_{1}-1}\\}$.
That is, ${\cal M}$ forms an exponential subfamily generated by
$u_{1},\ldots,u_{n_{1}-1}$. Hence, the maximization (288) is rewritten as

$\displaystyle\max_{p\in{\cal P}_{{\cal C}}}\min_{\rho\in{\cal S}({\cal
H}_{A}),q\in{\cal P}_{{\cal C}}}D(\rho[p]\|\rho\otimes q)=$
$\displaystyle\max_{\theta\in\mathcal{M}}D^{F}(\theta\|\mathop{\Gamma}\nolimits^{(m),F}_{\mathcal{E}}(\theta))$
$\displaystyle=$
$\displaystyle\max_{\theta\in\mathcal{M}}\min_{\theta^{\prime}\in\mathcal{E}}D^{F}(\theta\|\theta^{\prime}).$
(304)

###  7.4 Check of Conditions (B2), (B3), (B4), and (B5)

We define the $(n_{1}-1)\times(n_{2}^{2}-1)$ matrix $H:=(h_{i,j})$. Then, we
find that the $(n_{1}-1)\times(n_{1}+n_{2}^{2}-2)$ matrix $V_{1}$ is $(I,H)$.
That is, the $(n_{1}-1)\times(n_{2}^{2}-1)$ matrix $V_{3}$ is $H$. Hence,
conditions (B3) and (B4) hold. In the exponential family ${\cal E}$, we have
$F_{{\cal
E}}(\bar{\theta})=\log\sum_{x}e^{\sum_{i=1}^{n_{1}-1}{\theta}^{i}\delta_{i}(x)}+\log\operatorname{\mathrm{Tr}}e^{\sum_{j=1}^{n_{2}^{2}-1}{\theta}^{n_{1}-1+j}X_{j}}$.
Hence, the condition (B5) holds. Therefore, we can apply Algorithm 4 with
Condition (B5).

Since we have

$\displaystyle\rho_{\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta)}=\rho_{\theta,C}\otimes\rho_{\theta,A}.$ (305)

for any $\theta$, we have

$\displaystyle D^{F}(\theta^{\prime}\|\theta)=$ $\displaystyle
D(\rho_{\theta}\|\rho_{\theta^{\prime}})=D(\rho_{\theta,C}\|\rho_{\theta^{\prime},C})\leq
D(\rho_{\theta,C}\|\rho_{\theta^{\prime},C})+D(\rho_{\theta,A}\|\rho_{\theta^{\prime},A})$
$\displaystyle=$ $\displaystyle
D(\rho_{\theta,A}\otimes\rho_{\theta,C}\|\rho_{\theta^{\prime},A}\otimes\rho_{\theta^{\prime},C})=D(\rho_{\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta^{\prime})}\|(\rho_{\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta)})$ $\displaystyle=$ $\displaystyle
D^{F}(\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta^{\prime})\|\mathop{\Gamma}\nolimits^{(m),F}_{{\cal E}}(\theta))$
(306)

for $\theta,\theta^{\prime}\in{\cal M}$. Thus, the condition (B2) holds.
Therefore, Theorem 4.1 guarantees the global convergence. When $\theta^{(1)}$
is the uniform distribution on ${\cal X}$, in the same way as (164), we can
show that the supremum
$\sup_{\theta\in\mathcal{M}}D^{F}(\theta\|\theta^{(1)})$ equals $\log n_{1}$.
Therefore, when Theorem 4.1 is applied, we obtain the precision (54) with
$\frac{\log n_{1}}{\epsilon}$ iterations.

Now, with the above choice of $\theta^{(1)}$, we consider the case when the
density matrices $\\{W_{x}\\}_{x}$ are linearly independent. In the same way
as Section 5, there exists $\alpha>0$ to satisfy the condition (B2+). Hence,
we can apply Theorem 4.2 instead of Theorem 4.1. When $\theta^{(1)}$ is the
uniform distribution on ${\cal X}$, we obtain the precision (57) with
$\frac{\log\log n_{1}-\log\epsilon}{\log(1+\alpha)}$ iterations.

###  7.5 Non-iterative method

Next, we characterize the maximization (288) without an iterative method. To
check Condition (B6+), we prepare the following lemmas.

###### Lemma 20

The relation

$\displaystyle\rho_{(\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\dagger},\theta_{\mathsf{b}}^{\dagger}),C}=\bar{\rho}_{\theta_{\mathsf{a}},C}.$
(307)

holds, where
$\theta_{\mathsf{a}}^{\dagger}=(\theta^{1,\dagger},\ldots,\theta^{n_{1}-1,\dagger})$
is defined as $\theta^{i,\dagger}:=-H(W_{i})+H(W_{n_{1}})$ for
$i=1\ldots,n_{1}-1$.

###### Proof

We define
$\theta_{(i)}^{\dagger}=(\theta^{n_{1}-1+(i-1)(n_{2}^{2}-1)+1,\dagger},\ldots,\theta^{n_{1}-1+i(n_{2}^{2}-1),\dagger})\in\mathbb{R}^{n_{1}-1}$.
Since $W_{i}=\bar{\rho}_{\theta_{(i)}^{\dagger},A}$, we have
$W_{i}=e^{\sum_{j=1}^{n_{2}^{2}-1}\theta^{n_{1}-1+(i-1)(n_{2}^{2}-1)+j,\dagger}(X_{j}-h_{i,j})-F_{{\cal
E},\mathsf{b}}(\theta_{(i)}^{\dagger})}$. Because

$\displaystyle e^{F_{{\cal
E},\mathsf{b}}(\theta_{(i)}^{\dagger})}=\operatorname{\mathrm{Tr}}e^{\sum_{j=1}^{n_{2}^{2}-1}\theta^{n_{1}-1+(i-1)(n_{2}^{2}-1)+j,\dagger}(X_{j}-h_{i,j})},$
(308)

(303) implies the relation

$\displaystyle H(W_{i})=-\operatorname{\mathrm{Tr}}W_{i}\log W_{i}=F_{{\cal
E},\mathsf{b}}(\theta_{(i)}^{\dagger})$ (309)

for $i=1,\ldots,n_{1}$.

Now, we choose $\theta_{1}^{\prime}\in\mathbb{R}^{n_{1}-1}$ such that

$\displaystyle\rho_{(\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\dagger},\theta_{\mathsf{b}}^{\dagger}),C}=\bar{\rho}_{\theta_{\mathsf{a}}^{\prime},C}.$
(310)

Since we have

$\displaystyle\langle
n_{1}|\rho_{(\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\dagger},\theta_{\mathsf{b}}^{\dagger}),C}|n_{1}\rangle=e^{F_{{\cal
E},\mathsf{a}}(\theta_{(n_{1})}^{\dagger})-F_{{\cal
M}}(\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\dagger},\theta_{\mathsf{b}}^{\dagger})},$
(311)

the relation
$P_{(\theta_{\mathsf{a}}+\theta_{\mathsf{b}}^{\dagger},\theta_{\mathsf{b}}^{\dagger}),X}(n_{1})=\bar{P}_{\theta_{1}^{\prime},X}(n_{1})$
yields

$\displaystyle e^{F_{{\cal E},\mathsf{b}}(\theta_{(n_{1})}^{\dagger})-F_{{\cal
M}}(\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\dagger},\theta_{\mathsf{b}}^{\dagger})}=e^{-F_{{\cal
E},\mathsf{a}}(\theta_{\mathsf{a}}^{\prime})}.$ (312)

For $x\neq n_{1}$, we have

$\displaystyle\langle
x|\bar{\rho}_{\theta_{\mathsf{a}}^{\prime},C}|x\rangle=\langle
x|\rho_{(\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\dagger},\theta_{\mathsf{b}}^{\dagger}),C}|x\rangle=e^{\theta_{\mathsf{a}}^{x}+\theta_{\mathsf{a}}^{x,\dagger}+F_{{\cal
E},\mathsf{b}}(\theta_{(x)}^{\dagger})-F_{{\cal
M}}(\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\dagger},\theta_{\mathsf{b}}^{\dagger})}$
$\displaystyle\overset{(a)}{=}$ $\displaystyle
e^{\theta_{\mathsf{a}}^{x}+\theta_{\mathsf{a}}^{x,\dagger}+F_{{\cal
E},\mathsf{b}}(\theta_{(x)}^{\dagger})-F_{{\cal
E},\mathsf{b}}(\theta_{(n_{1})}^{\dagger})-F_{{\cal
E},\mathsf{a}}(\theta_{1}^{\prime})}\overset{(b)}{=}e^{\theta^{x}-F_{{\cal
E},\mathsf{a}}(\theta_{\mathsf{a}}^{\prime})},$ (313)

where $(a)$ and $(b)$ follow from (312) and the pair of (309) and the
definition of $\theta_{\mathsf{a}}^{x,\dagger}$, respectively. This relation
shows (307).

In the same way as the end of the previous subsection, we assume that the
distributions $\\{W_{x}\\}_{x}$ are linearly independent. Then, the rank of
$H$ is $n_{1}-1$. The combination of this fact and Lemma 13 guarantees

$\displaystyle\nabla^{(m)}[F_{{\cal M}}^{*}](\eta^{1})-\nabla^{(m)}[F_{{\cal
E},1}^{*}](\eta^{1})=\theta_{\mathsf{a}}+\theta_{\mathsf{a}}^{\dagger}-\theta_{\mathsf{a}}=\theta_{\mathsf{a}}^{\dagger},$
(314)

which implies Condition (B6+). We choose the parameter
$\theta_{\mathsf{b}}^{\ddagger}\in\mathbb{R}^{n_{2}^{2}-1}$ such that
$H\theta_{\mathsf{b}}^{\ddagger}=\theta_{\mathsf{a}}^{\dagger}$. We choose
$(n_{2}^{2}-1)\times(n_{2}^{2}-n_{1})$ matrix $G$ such that $\mathop{\rm
Im}G=\mathop{\rm Ker}H$. Then, ${\cal E}_{\mathsf{b}}$ and ${\cal
M}_{\mathsf{b}}$ defined in (249) and (250) are written as

$\displaystyle{\cal E}_{\mathsf{b}}$
$\displaystyle=\\{\theta_{\mathsf{b}}^{\ddagger}+G\theta_{\mathsf{e}}|\theta_{\mathsf{e}}\in\mathbb{R}^{n_{2}^{2}-n_{1}}\\},$
(315) $\displaystyle{\cal M}_{\mathsf{b}}$
$\displaystyle=\\{\theta_{\mathsf{b}}\in\mathbb{R}^{n_{2}^{2}-1}|\nabla^{(e)}[F_{{\cal
E},\mathsf{b}}](\theta_{2})G=0\\}.$ (316)

As explained in Subsection 4.7, the intersection ${\cal
E}_{\mathsf{b}}\cap{\cal M}_{\mathsf{b}}$ is composed of a unique element. As
the solution of the following minimization (317), we choose
$\theta_{\mathsf{e}}^{\ddagger}$ as

$\displaystyle\bar{\theta}_{\mathsf{e}}^{\ddagger}:=\mathop{\rm
argmin}\limits_{\theta_{\mathsf{e}}}F_{{\cal
E},\mathsf{b}}(\theta_{\mathsf{b}}^{\ddagger}+G\theta_{\mathsf{e}}).$ (317)

Then, we set
$\bar{\theta}_{\mathsf{b}}^{\ddagger}:=\theta_{\mathsf{b}}^{\ddagger}+G\theta_{\mathsf{e}}^{\ddagger}\in{\cal
E}_{\mathsf{b}}\cap{\cal M}_{\mathsf{b}}$. Due to Theorem 4.7, when there
exists $\theta_{\mathsf{a}}^{\ddagger}\in\mathbb{R}^{n_{1}-1}$ such that
$\nabla^{(e)}[F_{{\cal
E},\mathsf{b}}](\bar{\theta}_{\mathsf{b}}^{\ddagger})=\nabla^{(e)}[F_{{\cal
E},\mathsf{a}}](\theta_{\mathsf{a}}^{\ddagger})H$, which is equivalent to

$\displaystyle
W\cdot\bar{\rho}_{\theta_{\mathsf{a}}^{\ddagger},C}=\bar{\rho}_{\bar{\theta}_{\mathsf{b}}^{\ddagger},A}.$
(318)

the maximizer in (304) is
$(\theta_{\mathsf{a}}^{\ddagger}+\theta_{\mathsf{a}}^{\dagger},\theta_{\mathsf{b}}^{\ddagger})\in{\cal
M}$. In addition, the maximum (304) is

$\displaystyle
D(\rho_{(\theta_{\mathsf{a}}^{\ddagger}+\theta_{\mathsf{a}}^{\dagger},\theta_{\mathsf{b}}^{\dagger})}\|\bar{\rho}_{\theta_{\mathsf{a}}^{\ddagger},C}\times\bar{\rho}_{\bar{\theta}_{\mathsf{b}}^{\ddagger},A})=-H(W_{n_{1}})+F_{{\cal
E},\mathsf{b}}(\bar{\theta}_{\mathsf{b}}^{\ddagger}),$ (319)

because

$\displaystyle
D(W_{x}\|\bar{\rho}_{\bar{\theta}_{\mathsf{b}}^{\ddagger},A})=\operatorname{\mathrm{Tr}}W_{x}(\log
W_{x}-\log\bar{\rho}_{\bar{\theta}_{\mathsf{b}}^{\ddagger},A})$
$\displaystyle=$
$\displaystyle-H(W_{x})-\operatorname{\mathrm{Tr}}W_{x}\Big{(}\sum_{j=1}^{n_{2}^{2}-1}\bar{\theta}^{n_{1}-1+j,\ddagger}X_{j}-F_{{\cal
E},\mathsf{b}}(\bar{\theta}_{\mathsf{b}}^{\ddagger})\Big{)}$ $\displaystyle=$
$\displaystyle-H(W_{x})-\sum_{j=1}^{n_{2}^{2}-1}\bar{\theta}^{n_{1}-1+j,\ddagger}h_{x,j}+F_{{\cal
E},\mathsf{b}}(\bar{\theta}_{\mathsf{b}}^{\ddagger})$ $\displaystyle=$
$\displaystyle-H(W_{x})-\sum_{j=1}^{n_{2}^{2}-1}\theta^{n_{1}-1+j,\ddagger}h_{x,j}+F_{{\cal
E},2\mathsf{b}}(\bar{\theta}_{\mathsf{b}}^{\ddagger})$ $\displaystyle=$
$\displaystyle-H(W_{x})-\theta^{x,\dagger}+F_{{\cal
E},\mathsf{b}}(\bar{\theta}_{\mathsf{b}}^{\ddagger})$ $\displaystyle=$
$\displaystyle-H(W_{x})-(-H(W_{x})+H(W_{n_{1}}))+F_{{\cal
E},\mathsf{b}}(\bar{\theta}_{\mathsf{b}}^{\ddagger})$ $\displaystyle=$
$\displaystyle-H(W_{n_{1}})+F_{{\cal
E},\mathsf{b}}(\bar{\theta}_{\mathsf{b}}^{\ddagger}).$ (320)

When $n_{1}=n_{2}^{2}$, we have $l=n_{1}-1=n_{2}^{2}-1$, which enables us to
apply Corollary 6. In this case, as another typical case, we can choose the
matrices $X_{j}$ such that $(\operatorname{\mathrm{Tr}}X_{j}W_{i})_{1\leq
i,j\leq n_{2}^{2}-1}$ is is the identity matrix Under this choice, the
calculation of the maximization (129) based on Corollary 6 is done by
Algorithm 1 in the reference exact . Therefore, the method based on Theorem
4.7 can be considered as a generalization of Algorithm 2 in the reference
exact .

However, there is a case that no distribution $P_{X}$ on ${\mathcal{X}}$
satisfies (176). In this case, instead of a distribution on ${\mathcal{X}}$,
there exists a function $f_{X}$ on ${\mathcal{X}}$ such that

$\displaystyle\sum_{x\in{\mathcal{X}}}f_{X}(x)W_{x}=\bar{\rho}_{\bar{\theta}_{\mathsf{b}}^{\ddagger},A},~{}\sum_{x\in{\mathcal{X}}}f_{X}(x)=1.$
(321)

Also, there does not exist the maximum in (304), and the maximum (288) is
achieved in the boundary of ${\mathcal{P}}_{\mathcal{X}}$. When we remove an
element $x\in{\mathcal{X}}$, we have a subset
$P_{{\mathcal{X}}\setminus\\{x\\}}$ of the boundary. That is, the boundary is
composed of this type of subsets. Hence, to obtain the maximum (288), we need
to apply the method in this subsection to the case when the channel is defined
in the above type of subset.

In summary, in the same way as the capacity of the classical channel, the
capacity of the classical-quantum channel can be calculated with an algorithm
similar to Algorithm 5.

##  8 Conclusion

In our study, we have tackled the reverse em-problem within the general
framework of Bregman divergence. We have formulated this problem as the
maximization of the minimum divergence between a mixture family and an
exponential family, and proposed various methods to address it.

Our first method involves the development of the reverse em-algorithm using
Bregman divergence. We have shown the convergence of this algorithm to the
true value and analyzed its convergence speed under conditions that align with
information-theoretical problem settings. We have applied this approach to
problems related to channel capacity, including quantum settings. This method
was initially proposed by Toyota in the context of calculating the classical
channel capacity Shoji . However, Toyota’s work did not establish the
existence of the inverse map of the map
$\mathop{\Gamma}\nolimits^{(e),F}{{\cal
M}}\circ\mathop{\Gamma}\nolimits^{(m),F}{{\cal E}}|_{{\cal M}}$. In Theorem
4.3, we have shown that the inverse map uniquely exists under our Condition
(B3) within the general framework of Bregman divergence. Furthermore, in
Section 5, we have shown that the case of classical channel capacity satisfies
our Condition (B3). Consequently, we have successfully solved the problem
originally proposed by Toyota Shoji . Theorem 4.3 also provides the form of
the inverse map through the minimization of a convex function. Moreover, in
Section 4.6, we have derived a simpler form of the inverse map under
additional conditions.

In the second method, we have successfully transformed the reverse em-problem
into em-problems by imposing the conditions introduced earlier. In this
method, the reverse em-problem is converted to finding the intersection
between an exponential family and a mixture family. The intersection is
characterized by solving the em-problem between the exponential family and the
mixture family.

In the third method, we have strengthened the conditions and achieved an even
more simplified approach. Under these stronger conditions, the reverse em-
problem is converted into a convex minimization problem. The convex function
involved in this minimization is a part of the function used to define the
exponential family. Importantly, this convex function is simpler compared to
the objective function
$D^{F}(\theta|\mathop{\Gamma}\nolimits^{(m),F}_{\mathcal{E}}(\theta))$ that
needs to be maximized in the original reverse em-problem. Notably, in specific
cases where the original reverse em-problem satisfies certain conditions, this
problem can be solved without requiring the additional minimization step. When
applied to the classical channel capacity, this special case coincides with
the algorithm proposed in the recent paper exact . Consequently, this method
can be regarded as a generalization of the approach presented in that paper
exact .

In the subsequent sections, we have shown that various concrete models,
including those in the quantum setting, satisfy the conditions introduced in
Section 4. Furthermore, we have established that these models also fulfill
several conditions presented in this paper. Additionally, we have provided a
detailed algorithm for calculating the classical channel capacity, which
serves as a generalization of the method proposed in the recent paper exact .
Moreover, we have performed numerical calculations using this algorithm for
cases that cannot be handled by the existing method exact .

As an additional contribution, in Subsection 4.5, we have introduced the
quadratic approximation in each iteration of our proposed algorithm, Algorithm
2. However, we have not extensively discussed the convergence speed or
computational complexity in various applications. This analysis is a topic for
future research, which includes comparing our method with existing approaches.

The results obtained illustrate the effectiveness of information geometry as a
conversion method for optimization problems. A key aspect of information
geometry lies in the choice of parameterization associated with an exponential
family and a mixture family. By leveraging this structure, we have
successfully derived alternative characterizations of the original problems.
Consequently, we can anticipate that the application of information geometry
will lead to further valuable conversions in important optimization problems.
In this way, our findings shed light on this novel application of information
geometry, expanding its potential uses. For example, we can consider the
application of our result to the channel capacity of channels with Markovian
memory. This topic was studied in the preceding studies Kavcic ; Vontobel ; Wu
. Since information geometry of Markovian process can be handled as a special
case of Bgregman divergence system Nakagawa ; Nagaoka ; HW , our method can be
expected to applied this topic.

## Acknowledgments

The author was supported in part by the National Natural Science Foundation of
China (Grant No. 62171212). The author is very grateful to Mr. Shoji Toyota
for helpful discussions and for explaining the achievements of the reference
Shoji . In particular, he explained to the author what problems were not
solved in the reference Shoji . In addition, he pointed out that the secrecy
capacity can be written as the reverse em-algorithm in a similar way as the
channel capacity Toyota .

## Data availability

Data sharing is not applicable to this article as no datasets were generated
or analyzed during the current study.

## Conflict of interest

There are no competing interests.

##  Appendix A Proof of Theorem 4.1

Let $\theta_{(t)}$ be $\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta^{(t)})$. For any $\epsilon_{1}>0$, we choose an element
$\theta(\epsilon_{1})$ of ${\cal M}$ such that
$D^{F}(\theta(\epsilon_{1})\|\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta(\epsilon_{1})))\geq
C_{\sup}(\mathcal{M},\mathcal{E})-\epsilon_{1}$. Also, let
$\theta(\epsilon_{1})_{*}$ be $\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta(\epsilon_{1}))$.

Figure 4:  Algorithms 4 and 2: This figure shows the topological relation
among $\theta(\epsilon_{1})_{*}$, $\theta(\epsilon_{1})$, $\theta_{(t+1)}$,
$\theta^{(t+1)}$, and $\theta^{(t)}$, which is used in the application of
Pythagorean theorem (Proposition 1).
$\mathcal{M}_{\theta(\epsilon_{1})\to\mathcal{E}}$ and
$\mathcal{M}_{\theta^{(t+1)}\to\mathcal{E}}$ are the mixture subfamilies to
project $\theta(\epsilon_{1})$ and $\theta^{(t+1)}$ to the exponential
subfamily $\mathcal{E}$, respectively.
$\mathcal{E}_{\theta_{(t+1)}\to\mathcal{M}}$ is the exponential subfamily to
project $\theta_{(t+1)}$ to the mixture subfamily $\mathcal{M}$.

As explained in Fig. 4, Pythagorean theorem (Proposition 1) guarantees that
the divergence $D^{F}(\theta(\epsilon_{1})\|\theta_{(t+1)})$ can be written in
the following two ways;

$\displaystyle
D^{F}(\theta(\epsilon_{1})\|\theta^{(t)})+D^{F}(\theta^{(t)}\|\theta_{(t+1)})=D^{F}(\theta(\epsilon_{1})\|\theta_{(t+1)})$
$\displaystyle=$ $\displaystyle
D^{F}(\theta(\epsilon_{1})\|\theta(\epsilon_{1})_{*})+D^{F}(\theta(\epsilon_{1})_{*}\|\theta_{(t+1)}).$
(322)

Hence,

$\displaystyle
C_{\sup}(\mathcal{M},\mathcal{E})-\epsilon_{1}-D^{F}(\theta^{(t)}\|\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta^{(t)}))$ $\displaystyle=$ $\displaystyle
D^{F}(\theta(\epsilon_{1})\|\theta(\epsilon_{1})_{*})-D^{F}(\theta^{(t)}\|\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta^{(t)}))$ $\displaystyle=$ $\displaystyle
D^{F}(\theta(\epsilon_{1})\|\theta(\epsilon_{1})_{*})-D^{F}(\theta^{(t)}\|\theta_{(t+1)})$
$\displaystyle=$ $\displaystyle
D^{F}(\theta(\epsilon_{1})\|\theta^{(t)})-D^{F}(\theta(\epsilon_{1})_{*}\|\theta_{(t+1)})$
$\displaystyle=$ $\displaystyle
D^{F}(\theta(\epsilon_{1})\|\theta^{(t)})-D^{F}(\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta(\epsilon_{1}))\|\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta^{(t+1)}))$ $\displaystyle\leq$ $\displaystyle
D^{F}(\theta(\epsilon_{1})\|\theta^{(t)})-D^{F}(\theta(\epsilon_{1})\|\theta^{(t+1)}),$
(323)

where the final inequality follows from condition (B2). Thus,

$\displaystyle\sum_{i=1}^{t}C_{\sup}(\mathcal{M},\mathcal{E})-\epsilon_{1}-D^{F}(\theta^{(i)}\|\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta^{(i)}))$ $\displaystyle\leq$
$\displaystyle\sum_{i=1}^{t}D^{F}(\theta(\epsilon_{1})\|\theta^{(i)})-D^{F}(\theta(\epsilon_{1})\|\theta^{(i+1)})$
$\displaystyle=$ $\displaystyle
D^{F}(\theta(\epsilon_{1})\|\theta^{(1)})-D^{F}(\theta(\epsilon_{1})\|\theta^{(t+1)})\leq
D^{F}(\theta(\epsilon_{1})\|\theta^{(1)})$ $\displaystyle\leq$
$\displaystyle\sup_{\theta\in\mathcal{M}}D^{F}(\theta\|\theta^{(1)}).$ (324)

Taking the limit $\epsilon_{1}\to 0$, we have

$\displaystyle\sum_{i=1}^{t}C_{\sup}(\mathcal{M},\mathcal{E})-D^{F}(\theta^{(i)}\|\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta^{(i)}))\leq\sup_{\theta\in\mathcal{M}}D^{F}(\theta\|\theta^{(1)}).$
(325)

Since

$\displaystyle D^{F}(\theta^{(i+1)}\|\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta^{(i)}))=D^{F}(\theta^{(i+1)}\|\theta_{(i+1)})$ $\displaystyle\geq$
$\displaystyle D^{F}(\theta^{(i)}\|\theta_{(i+1)})\geq
D^{F}(\theta^{(i)}\|\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta^{(i)})),$ (326)

for $i\leq t$, we have

$\displaystyle
C_{\sup}(\mathcal{M},\mathcal{E})-D^{F}(\theta^{(t)}\|\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta^{(t)}))\leq
C_{\sup}(\mathcal{M},\mathcal{E})-D^{F}(\theta^{(i)}\|\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta^{(i)})).$ (327)

The combination of (325) and (327) implies that

$\displaystyle
C_{\sup}(\mathcal{M},\mathcal{E})-D^{F}(\theta^{(t)}\|\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta^{(t)}))\leq\frac{1}{t}\sup_{\theta\in\mathcal{M}}D^{F}(\theta\|\theta^{(1)}),$
(328)

which implies (54).

When the inequality

$\displaystyle
C_{\sup}(\mathcal{M},\mathcal{E})-D^{F}(\theta^{(t)}\|\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta^{(t)}))\geq c(\frac{1}{t})$ (329)

holds with a constant $c>0$, (325) yields

$\displaystyle\infty=\sum_{t=1}^{\infty}c(\frac{1}{t})\leq\sup_{\theta\in\mathcal{M}}D^{F}(\theta\|\theta^{(1)}),$
(330)

which implies the contradiction. Hence, we obtain (53).

Indeed, when the maximum in (47) exists, i.e.,
$\theta^{*}(\mathcal{M},\mathcal{E})$ exists, the supremum
$\sup_{\theta\in\mathcal{M}}D^{F}(\theta\|\theta^{(1)})$ in the above
evaluation is replaced by
$D^{F}(\theta^{*}(\mathcal{M},\mathcal{E})\|\theta^{(1)})$ because
$\theta(\epsilon_{1})$ is replaced by $\theta^{*}(\mathcal{M},\mathcal{E})$.

##  Appendix B Proof of Theorem 4.2

We use the same notation as the proof of Theorem 4.1. We denote
$\theta^{*}(\mathcal{M},\mathcal{E})$ and
$\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta^{*}(\mathcal{M},\mathcal{E}))$ by $\theta^{*}$ and $\theta_{*}$,
respectively. Also, set $\alpha:=\alpha(\theta_{(1)})$. Using (323) with
$\epsilon_{1}=0$, we have

$\displaystyle 0\leq$ $\displaystyle
C_{\sup}(\mathcal{M},\mathcal{E})-D^{F}(\theta^{(t)}\|\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta^{(t)}))$ $\displaystyle=$ $\displaystyle
D^{F}(\theta^{*}\|\theta^{(t)})-D^{F}(\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta^{*})\|\mathop{\Gamma}\nolimits^{(m),F}_{{\cal E}}(\theta^{(t+1)}))$
(331) $\displaystyle\leq$ $\displaystyle
D^{F}(\theta^{*}\|\theta^{(t)})-D^{F}(\theta^{*}\|\theta^{(t+1)}),$ (332)

which implies $D^{F}(\theta^{*}\|\theta^{(t+1)})\leq
D^{F}(\theta^{*}\|\theta^{(1)})$. Thus, the condition (B2+) implies
$(1+\alpha)D^{F}(\theta^{*}\|\theta^{(t+1)})\leq
D^{F}(\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta^{*})\|\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta^{(t+1)}))$. Combining (331), we have $0\leq
D^{F}(\theta^{*}\|\theta^{(t)})-(1+\alpha)D^{F}(\theta^{*}\|\theta^{(t+1)}).$
Thus, we have

$\displaystyle
D^{F}(\theta^{*}\|\theta^{(t+1)})\leq(1+\alpha)^{-1}D^{F}(\theta^{*}\|\theta^{(t)}),$
(333)

which implies that

$\displaystyle
D^{F}(\theta^{*}\|\theta^{(t)})\leq(1+\alpha)^{-(t-1)}D^{F}(\theta^{*}\|\theta^{(1)}).$
(334)

Using (323) with $\epsilon_{1}=0$, we have

$\displaystyle
C_{\sup}(\mathcal{M},\mathcal{E})-D^{F}(\theta^{(t)}\|\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta^{(t)}))$ $\displaystyle\leq$ $\displaystyle
D^{F}(\theta^{*}\|\theta^{(t)})-D^{F}(\theta^{*}\|\theta^{(t+1)})\leq
D^{F}(\theta^{*}\|\theta^{(t)})\leq(1+\alpha)^{-(t-1)}D^{F}(\theta^{*}\|\theta^{(1)}).$
(335)

Hence, we obtain (56).

##  Appendix C Proofs of Lemmas 10 and 11

We show Lemmas 10 and 11.

Proof of Lemma 10:  Eq. (63) follows from the following relation.

$\displaystyle\eta_{\mathsf{a}}V_{1}=\nabla_{\mathsf{a}}^{(m)}[F](\psi_{\mathcal{M}}^{(m)}(\eta_{\mathsf{a}}))V_{1}=\nabla^{(m)}[F](\psi_{\mathcal{M}}^{(m)}(\eta_{\mathsf{a}}))V$
$\displaystyle\stackrel{{\scriptstyle(a)}}{{=}}$
$\displaystyle\nabla^{(m)}[F](\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\psi_{\mathcal{M}}^{(m)}(\eta_{\mathsf{a}})))V$ $\displaystyle=$
$\displaystyle\nabla^{(m)}[F](V(\psi_{\mathcal{E}}^{(e)})^{-1}(\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\psi_{\mathcal{M}}^{(m)}(\eta_{\mathsf{a}}))))V$
$\displaystyle\stackrel{{\scriptstyle(b)}}{{=}}$
$\displaystyle\nabla^{(m)}[F_{\cal
E}]((\psi_{\mathcal{E}}^{(e)})^{-1}(\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\psi_{\mathcal{M}}^{(m)}(\eta_{\mathsf{a}}))))$ $\displaystyle=$
$\displaystyle(\psi_{\mathcal{E}}^{(m)})^{-1}(\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\psi_{\mathcal{M}}^{(m)}(\eta_{\mathsf{a}})))=\mathop{\Gamma}\nolimits^{(m),F}_{\Xi_{{\cal
M}}\to\Xi_{\cal E}}(\eta_{\mathsf{a}}),$ (336)

where $(a)$ follows from Lemma 3, and $(b)$ follows from $F_{\cal
E}(\eta_{\mathsf{c}})=F(V\theta_{\mathsf{c}})$. ∎

Proof of Lemma 11:  To show the equivalence between (i) and (ii), we apply the
condition (C1) of Lemma 4. Hence, (i) means that
$\psi_{\mathcal{E}}^{(m)}(\eta_{\mathsf{a}}V_{1})\in{\cal E}$ and
$\psi_{\mathcal{M}}^{(e)}(\theta_{\mathsf{a}})\in{\cal M}$ belong to the same
exponential family generated by $u_{k+1},\ldots,u_{d}$. That is, these two
elements have the same $k$ coefficients on the $k$ vectors
$u_{1},\ldots,u_{k}$. The $k$ coefficients of
$\psi_{\mathcal{M}}^{(e)}(\theta_{\mathsf{a}})\in{\cal M}$ is
$\theta_{\mathsf{a}}$. The $k$ coefficients of
$\psi_{\mathcal{E}}^{(e)}(\theta_{\mathsf{c}})\in{\cal E}$ is
$V_{1}\theta_{\mathsf{c}}$. That is, the intersection between ${\cal E}$ and
the above exponential family is the exponential subfamily
$\psi_{\mathcal{E}}^{(e)}(\\{\theta_{\mathsf{c}}\in\Theta_{\mathcal{E}}|\theta_{\mathsf{a}}=V_{1}\theta_{\mathsf{c}}\\})$.
Hence, the condition (i) is equivalent to
$\psi_{\mathcal{E}}^{(m)}(\eta_{\mathsf{a}}V_{1})\in\psi_{\mathcal{E}}^{(e)}(\\{\theta_{\mathsf{c}}\in\Theta_{\mathcal{E}}|\theta_{\mathsf{a}}=V_{1}\theta_{\mathsf{c}}\\})$,
i.e., the condition (ii).

Since
$\nabla^{(m)}[F_{\mathcal{E}}^{*}](\eta_{\mathsf{a}}V_{1})=\phi_{\mathcal{E}}^{(e)}\circ\psi_{\mathcal{E}}^{(m)}(\eta_{\mathsf{a}}V_{1})$,
the condition (ii) is equivalent to

$\displaystyle\theta_{\mathsf{a}}=V_{1}\nabla^{(m)}[F_{\mathcal{E}}^{*}](\eta_{\mathsf{a}}V_{1}).$
(337)

Since the relation (13) guarantees that

$\displaystyle V_{1}\nabla^{(m)}[F_{\mathcal{E}}^{*}](\eta_{\mathsf{a}}V_{1})$
$\displaystyle=$ $\displaystyle
L[V_{1}]\circ\nabla^{(m)}[F_{\mathcal{E}}^{*}]\circ
R[V_{1}](\eta_{\mathsf{a}})$ $\displaystyle=$
$\displaystyle\nabla^{(m)}[F_{\mathcal{E}}^{*}\circ
R[V_{1}]](\eta_{\mathsf{a}}),$ (338)

the conditions (ii) and (iii) are equivalent. The relation between (7) and
(10) guarantees the equivalence between the conditions (iii) and (iv).

Since $F_{\mathcal{E}}^{*}\circ R[V_{1}]$ is a convex function, the condition
(iv) is equivalent to the condition

$\displaystyle\eta_{\mathsf{a}}=\mathop{\rm
argmin}\limits_{\eta_{\mathsf{a}}^{\prime}\in\mathbb{R}^{k}}F_{\mathcal{E}}^{*}\circ
R[V_{1}](\eta_{\mathsf{a}}^{\prime})-\langle\eta_{\mathsf{a}}^{\prime},\theta_{\mathsf{a}}\rangle.$
(339)

Since $F_{{\cal E}}^{*}({\eta}_{\mathsf{a}}V_{1})=F_{\mathcal{E}}^{*}\circ
R[V_{1}](\eta_{\mathsf{a}})$, the condition (iv) is equivalent to the
condition (v). ∎

##  Appendix D Proof of Theorem 4.3

Proof of Theorem 4.3:  To show the statement (i) of Theorem 4.3, we choose two
elements $\theta^{(t)},\theta^{(t+1)}\in\mathcal{E}$ as
$\theta^{(t)}=\mathop{\Gamma}\nolimits^{(e),F}_{{\cal
M}}\circ\mathop{\Gamma}\nolimits^{(m),F}_{{\cal E}}(\theta^{(t+1)})$. the
input element is characterized by the mixture parameter
$\hat{\eta}_{\mathsf{a}}(\theta^{(t+1)})$ and the output element is
characterized by the natural parameter
$\hat{\theta}_{\mathsf{a}}(\theta^{(t)})$ with respect to $\mathcal{M}$. Then,
$\mathop{\Gamma}\nolimits^{(m),F}_{{\cal E}}(\theta^{(t+1)})$ has the mixture
parameter $\hat{\eta}_{\mathsf{a}}(\theta^{(t+1)})V_{1}$ with respect to
${\cal E}$ due to Lemma 3. Due to the equivalence between the conditions (i)
and (iv) of Lemma 11, the mixture parameter
$\hat{\eta}_{\mathsf{a}}(\theta^{(t+1)})$ and the natural parameter
$\hat{\theta}_{\mathsf{a}}(\theta^{(t)})$ satisfies the following condition;

$\displaystyle\hat{\eta}_{\mathsf{a}}(\theta^{(t+1)})=\nabla^{(e)}[(F_{\cal
E}^{*}\circ R[V_{1}])^{*}](\hat{\theta}_{\mathsf{a}}(\theta^{(t)})).$ (340)

Since ${\cal M}$ is also an exponential subfamily, the function $F_{\cal M}$
is defined. Hence, the relation (340) is rewritten with the natural parameter
in ${\cal M}$ as

$\displaystyle\hat{\theta}_{\mathsf{a}}(\theta^{(t+1)})=\nabla^{(m)}[F_{\cal
M}^{*}]\circ\nabla^{(e)}[(F_{\cal E}^{*}\circ
R[V_{1}])^{*}](\hat{\theta}_{\mathsf{a}}(\theta^{(t)})).$ (341)

The condition (341) is equivalent to the condition that
$\theta^{(t)}=\mathop{\Gamma}\nolimits^{(e),F}_{{\cal
M}}\circ\mathop{\Gamma}\nolimits^{(m),F}_{{\cal M}\to{\cal
E}}(\theta^{(t+1)})$ for $\theta^{(t)},\theta^{(t+1)}\in{\cal M}$. Hence, for
any $\theta^{(t)}\in\mathcal{M}$, there uniquely exists an element
$\theta^{(t+1)}\in{\cal M}$ to satisfy the condition
$\theta^{(t)}=\mathop{\Gamma}\nolimits^{(e),F}_{{\cal
M}}\circ\mathop{\Gamma}\nolimits^{(m),F}_{{\cal M}\to{\cal
E}}(\theta^{(t+1)})$. Thus, $\nabla^{(m)}[F_{\cal
M}^{*}]\circ\nabla^{(e)}[(F_{\cal E}^{*}\circ R[V_{1}])^{*}]$ is the unique
inverse map of $\mathop{\Gamma}\nolimits^{(e),F}_{{\cal
M}}\circ\mathop{\Gamma}\nolimits^{(m),F}_{{\cal M}\to{\cal E}}$, and is
defined in ${\cal M}$. Hence, $\mathop{\Gamma}\nolimits^{(e),F}_{{\cal
M}}\circ\mathop{\Gamma}\nolimits^{(m),F}_{{\cal M}\to{\cal E}}$ is a bijective
map from ${\cal M}$ to ${\cal M}$. The statement (i) is obtained.

The statement (ii) follows from the equivalence between the conditions (iv)
and (v) of Lemma 11. ∎

The key point of the above proof is the following; Since ${\cal M}$ is an
exponential subfamily as well as a mixture subfamily, the natural parameter is
written as the Legendre transform of the mixture parameter, which is stated as
(341).

##  Appendix E Proof of Theorem 4.4

(Step 0) We prepare several relations that are used in this proof. In this
proof, we use the notation $\hat{\eta}_{\mathsf{a}}^{(t+1),*}:=\mathop{\rm
argmin}\limits_{\hat{\eta}_{\mathsf{a}}\in\mathbb{R}^{k}}F_{{\cal
E}}^{*}(\hat{\eta}_{\mathsf{a}}V_{1})-\langle\hat{\eta}_{\mathsf{a}},\hat{\theta}^{(t)}\rangle$.
We define elements $\theta_{*}:=\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta^{*}),\theta_{(t+1)}:=\psi_{\mathcal{E}}^{(m)}(\hat{\eta}_{\mathsf{a}}^{(t+1)}V_{1}),\theta_{(t+1),*}:=\psi_{\mathcal{E}}^{(m)}(\hat{\eta}_{\mathsf{a}}^{(t+1),*}V_{1})\in\mathcal{E}$.

Figure 5:  Algorithm 3: This figure shows the topological relation among
$\theta_{*}$, $\theta^{*}$, $\theta_{(t+1)}$, $\theta^{(t+1)}$,
$\theta_{(t+1),*}$, $\theta^{(t+1),*}$, and $\theta^{(t)}$, which is used in
the application of Pythagorean theorem (Proposition 1).
$\mathcal{M}_{\theta^{*}\to\mathcal{E}}$,
$\mathcal{M}_{\theta^{(t+1)}\to\mathcal{E}}$, and
$\mathcal{M}_{\theta^{(t+1),*}\to\mathcal{E}}$ are the mixture subfamilies to
project $\theta(\epsilon_{1})$, $\theta^{(t+1),*}$, and $\theta^{(t+1)}$ to
the exponential subfamily $\mathcal{E}$, respectively.
$\mathcal{E}_{\theta_{(t+1),*}\to\mathcal{M}}$ is the exponential subfamily to
project $\theta_{(t+1),*}$ to the mixture subfamily $\mathcal{M}$.

(Step 1): The aim of the first step is to show the inequality

$\displaystyle D^{F}(\theta^{(t+1)}\|\theta^{(t+1),*})\leq\epsilon.$ (342)

We define the mixture subfamily

$\displaystyle\bar{\mathcal{M}}:=\\{\psi_{{\cal
E}}^{(m)}(\hat{\eta}_{\mathsf{a}}V_{1})\in\mathcal{E}|\hat{\eta}_{\mathsf{a}}\in\mathbb{R}^{k}\\}.$
(343)

In this mixture subfamily $\bar{\mathcal{M}}$, we employ the mixture parameter
$\hat{\eta}_{\mathsf{a}}$. That is, we have
$\phi_{\bar{\mathcal{M}}}^{(m)}\circ\psi_{{\cal
E}}^{(m)}(\hat{\eta}_{\mathsf{a}}V_{1})=\hat{\eta}_{\mathsf{a}}$. Hence, we
choose $F_{\bar{\mathcal{M}}}^{*}$ as

$\displaystyle
F_{\bar{\mathcal{M}}}^{*}(\hat{\eta}_{\mathsf{a}})=F_{\mathcal{E}}^{*}(\hat{\eta}_{\mathsf{a}}V_{1})=F_{\mathcal{E}}^{*}\circ
R[V_{1}](\hat{\eta}_{\mathsf{a}}).$ (344)

Since $\mathop{\Gamma}\nolimits^{(e),F}_{{\cal
M}}(\theta_{(t+1),*})=\psi_{\mathcal{M}}^{(e)}(\hat{\theta}^{(t)}_{\mathsf{a}})$,
we have

$\displaystyle\nabla^{(m)}[F_{\bar{\mathcal{M}}}^{*}](\hat{\eta}^{(t+1),*}_{\mathsf{a}})\stackrel{{\scriptstyle(a)}}{{=}}\nabla^{(m)}[F_{\mathcal{E}}^{*}\circ
R[V_{1}]](\hat{\eta}^{(t+1),*}_{\mathsf{a}})\stackrel{{\scriptstyle(b)}}{{=}}\hat{\theta}^{(t)}_{\mathsf{a}},$
(345)

where $(a)$ and $(b)$ follow from (344) and the equivalence between the
conditions (i) and (iii) of Lemma 11, respectively. Hence, using (70), we have

$\displaystyle
F_{\bar{\mathcal{M}}}^{*}(\hat{\eta}^{(t+1)}_{\mathsf{a}})-\langle\hat{\eta}_{\mathsf{a}}^{(t+1)},\hat{\theta}_{\mathsf{a}}^{(t)}\rangle=F_{{\mathcal{E}}}^{*}(\hat{\eta}_{\mathsf{a}}^{(t+1)}V_{1})-\langle\hat{\eta}_{\mathsf{a}}^{(t+1)},\hat{\theta}_{\mathsf{a}}^{(t)}\rangle$
$\displaystyle\leq$ $\displaystyle
F_{{\mathcal{E}}}^{*}(\hat{\eta}_{\mathsf{a}}^{(t+1),*}V_{1})-\langle\hat{\eta}_{\mathsf{a}}^{(t+1),*},\hat{\theta}_{\mathsf{a}}^{(t)}\rangle+\epsilon=F_{\bar{\mathcal{M}}}^{*}(\hat{\eta}_{\mathsf{a}}^{(t+1),*})-\langle\hat{\eta}_{\mathsf{a}}^{(t+1),*},\hat{\theta}_{\mathsf{a}}^{(t)}\rangle+\epsilon.$
(346)

Therefore, we have

$\displaystyle
D^{F}(\theta^{(t+1)}\|\theta^{(t+1),*})\stackrel{{\scriptstyle(a)}}{{=}}D^{F_{\mathcal{E}}^{*}}(\eta^{(t+1),*}V_{1}\|\eta^{(t+1)}V_{1})$
$\displaystyle\stackrel{{\scriptstyle(b)}}{{=}}$ $\displaystyle
D^{F_{\bar{\mathcal{M}}}^{*}}(\eta^{(t+1),*}\|\eta^{(t+1)})$
$\displaystyle\stackrel{{\scriptstyle(c)}}{{=}}$
$\displaystyle\langle\nabla^{(m)}[F_{\bar{\mathcal{M}}}^{*}](\hat{\eta}_{\mathsf{a}}^{(t+1),*}),(\hat{\eta}_{\mathsf{a}}^{(t+1),*}-\hat{\eta}_{\mathsf{a}}^{(t+1)})\rangle-
F_{\bar{\mathcal{M}}}^{*}(\hat{\eta}_{\mathsf{a}}^{(t+1),*})+F_{\bar{\mathcal{M}}}^{*}(\hat{\eta}_{\mathsf{a}}^{(t+1)})$
$\displaystyle\stackrel{{\scriptstyle(d)}}{{=}}$
$\displaystyle\langle\hat{\theta}_{\mathsf{a},(t),*},(\hat{\eta}_{\mathsf{a}}^{(t+1),*}-\hat{\eta}_{\mathsf{a}}^{(t+1)})\rangle-
F_{\bar{\mathcal{M}}}^{*}(\hat{\eta}_{\mathsf{a}}^{(t+1),*})+F_{\bar{\mathcal{M}}}^{*}(\hat{\eta}_{\mathsf{a}}^{(t+1)})$
$\displaystyle\stackrel{{\scriptstyle(e)}}{{\leq}}$ $\displaystyle\epsilon,$
(347)

where $(a)$, $(b)$, $(c)$, $(d)$, and $(e)$ follow from the combination of
(32) and (33), the application of (33) to the substitution of
$F_{\mathcal{E}}^{*}$ and $F_{\bar{\mathcal{M}}}^{*}=F_{\mathcal{E}}^{*}\circ
R[V_{1}]$ into $F$ and $F_{\mathcal{E}}$, (28), (345), and (346),
respectively. Hence, we obtain (342).

(Step 2): The aim of this step is showing

$\displaystyle
D^{F}(\theta^{*}\|\theta^{(t),*})-D^{F}(\theta^{*}\|\theta^{(t+1),*})\geq 0$
(348)

for $t=2,\ldots,t_{0}$, by induction when we assume that $t_{0}$ satisfies the
following condition with for $t=2,\ldots,t_{0}$;

$\displaystyle
D^{F}(\theta^{*}\|\theta_{*})-D^{F}(\theta^{(t)}\|\theta_{(t+1),*})\geq
2\gamma\sqrt{D^{F}(\theta_{*,-}\|\theta_{(1)})\epsilon}+\gamma\epsilon.$ (349)

Applying the Pythagorean Theorem (Proposition 1) to
$D^{F}(\theta^{*}\|\theta_{(t+1),*})$, for $t=1,\ldots,t_{0}$ we have

$\displaystyle
D^{F}(\theta^{*}\|\theta_{(t+1),*})=D^{F}(\theta^{*}\|\theta_{*})+D^{F}(\theta_{*}\|\theta_{(t+1),*})=D^{F}(\theta^{*}\|\theta^{(t)})+D^{F}(\theta^{(t)}\|\theta_{(t+1),*}).$
(350)

Thus, we have

$\displaystyle
D^{F}(\theta^{*}\|\theta^{(t)})-D^{F}(\theta_{*}\|\theta_{(t+1),*})=D^{F}(\theta^{*}\|\theta_{*})-D^{F}(\theta^{(t)}\|\theta_{(t+1),*})$
$\displaystyle\geq$ $\displaystyle
D^{F}(\theta^{*}\|\theta_{*})-D^{F}(\theta^{(t+1),*}\|\theta_{(t+1),*})\geq
0.$ (351)

Due to the assumption of induction, we have

$\displaystyle D^{F}(\theta^{*}\|\theta^{(t),*})\leq
D^{F}(\theta^{*}\|\theta^{(2),*})\stackrel{{\scriptstyle(a)}}{{\leq}}D^{F}(\theta^{*}\|\theta^{(1)}),$
(352)

where $(a)$ follows from (351) with $t=1$.

Since the set $\mathcal{M}_{0}$ is a star subset for $\theta^{*}$, we can
apply Theorem 2.1 to the set $\mathcal{M}_{0}$ as a star subset of
$\mathcal{M}$ for $\theta_{*,-}$. Hence, using the above relations, for
$t=2,\ldots,t_{0}$, we have

$\displaystyle
D^{F}(\theta^{*}\|\theta_{*})-D^{F}(\theta^{(t)}\|\theta_{(t+1),*})\stackrel{{\scriptstyle(a)}}{{=}}D^{F}(\theta^{*}\|\theta^{(t)})-D(\theta_{*}\|\theta_{(t+1),*})$
$\displaystyle\stackrel{{\scriptstyle(b)}}{{\leq}}$ $\displaystyle
D^{F}(\theta^{*}\|\theta^{(t),*})+2\gamma\sqrt{D^{F}(\theta^{*}\|\theta^{(t),*})D^{F}(\theta^{(t)}\|\theta^{(t),*})}$
$\displaystyle+\gamma
D^{F}(\theta^{(t)}\|\theta^{(t),*})-D^{F}(\theta_{*}\|\theta_{(t+1),*})$
$\displaystyle\stackrel{{\scriptstyle(c)}}{{\leq}}$ $\displaystyle
D^{F}(\theta^{*}\|\theta^{(t),*})+2\gamma\sqrt{D^{F}(\theta^{*}\|\theta^{(1)})\epsilon}+\gamma\epsilon-D^{F}(\theta_{*}\|\theta_{(t+1),*})$
$\displaystyle\stackrel{{\scriptstyle(d)}}{{\leq}}$ $\displaystyle
D^{F}(\theta^{*}\|\theta^{(t),*})+2\gamma\sqrt{D^{F}(\theta^{*}\|\theta^{(1)})\epsilon}+\gamma\epsilon-D^{F}(\theta^{*}\|\theta^{(t+1),*}),$
(353)

where $(a)$, $(b)$, $(c)$, and $(d)$ follow from (350), Theorem 2.1, the
combination of (352) and (342), and the condition (B2), respectively.

Thus,

$\displaystyle
D^{F}(\theta^{*}\|\theta_{*})-D^{F}(\theta^{(t)}\|\theta_{(t+1),*})-2\gamma\sqrt{D^{F}(\theta^{*}\|\theta^{(1)})\epsilon}-\gamma\epsilon$
$\displaystyle{\leq}$ $\displaystyle
D^{F}(\theta^{*}\|\theta^{(t),*})-D^{F}(\theta^{*}\|\theta^{(t+1),*}).$ (354)

The combination of (349) and (354) implies the relation (348).

(Step 3): The aim of this step is showing

$\displaystyle
D^{F}(\theta^{*}\|\theta_{*})-D^{F}(\theta_{f}^{(t_{1})}\|\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta_{f}^{(t_{1})}))$ $\displaystyle\leq$
$\displaystyle\max\Big{(}\frac{D^{F}(\theta^{*}\|\theta^{(1)})}{t_{1}-1}+2\gamma\sqrt{D^{F}(\theta^{*}\|\theta^{(1)})\epsilon}+(\gamma+1)\epsilon,$
$\displaystyle\qquad
2\gamma\sqrt{D^{F}(\theta^{*}\|\theta^{(1)})\epsilon}+(\gamma+1)\epsilon\Big{)}.$
(355)

To this aim, it sufficient to show

$\displaystyle
D^{F}(\theta^{*}\|\theta_{*})-D^{F}(\theta_{f}^{(t_{1})}\|\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta_{f}^{(t_{1})}))$ $\displaystyle\leq$
$\displaystyle\frac{D^{F}(\theta^{*}\|\theta^{(1)})}{t_{1}-1}+2\gamma\sqrt{D^{F}(\theta^{*}\|\theta^{(1)})\epsilon}+(\gamma+1)\epsilon$
(356)

under the assumption

$\displaystyle
D^{F}(\theta^{*}\|\theta_{*})-D^{F}(\theta_{f}^{(t_{1})}\|\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta_{f}^{(t_{1})}))\geq
2\gamma\sqrt{D^{F}(\theta^{*}\|\theta^{(1)})\epsilon}+(\gamma+1)\epsilon.$
(357)

The assumption (357) implies that

$\displaystyle
D^{F}(\theta^{*}\|\theta_{*})-D^{F}(\theta^{(t)}\|\theta_{(t)})\geq
2\gamma\sqrt{D^{F}(\theta^{*}\|\theta^{(1)})\epsilon}+(\gamma+1)\epsilon.$
(358)

for $t=2,\ldots,t_{1}$. We have the following relation with
$t=1,\ldots,t_{1}-1$;

$\displaystyle
D^{F}(\theta^{(t+1)}\|\theta_{(t+1)})+\epsilon\stackrel{{\scriptstyle(a)}}{{\geq}}D^{F}(\theta^{(t+1)}\|\theta_{(t+1)})+D^{F}(\theta^{(t+1)}\|\theta_{(t+1),*})$
$\displaystyle\stackrel{{\scriptstyle(b)}}{{=}}$ $\displaystyle
D^{F}(\theta^{(t+1)}\|\theta_{(t+1),*})\stackrel{{\scriptstyle(c)}}{{\geq}}D^{F}(\theta^{(t)}\|\theta_{(t+1),*}),$
(359)

where $(a)$, $(b)$, and $(c)$ follow from (342), Pythagorean theorem
(Proposition 1), and the fact that
$\theta^{(t)}=\mathop{\Gamma}\nolimits^{(e),F}_{{\cal M}}(\theta_{(t+1),*})$,
respectively. The combination of (358) and (359) implies the condition (349)
with $t=2,\ldots,t_{1}-1$. Due to the conclusion of (Step 2), we have (348)
for $t=2,\ldots,t_{1}-1$. Since (Step 2) derived the relation (353) with the
same condition, the relation (353) holds with $t=2,\ldots,t_{1}-1$. Therefore,
we have

$\displaystyle
D^{F}(\theta^{*}\|\theta_{*})-D^{F}(\theta^{(t+1)}\|\theta_{(t+1)})$
$\displaystyle\stackrel{{\scriptstyle(a)}}{{\leq}}$ $\displaystyle
D^{F}(\theta^{*}\|\theta_{*})-D^{F}(\theta^{(t)}\|\theta_{(t+1),*})+\epsilon$
$\displaystyle\stackrel{{\scriptstyle(b)}}{{\leq}}$ $\displaystyle
D^{F}(\theta^{*}\|\theta^{(t),*})+2\gamma\sqrt{D^{F}(\theta^{*}\|\theta^{(1)})\epsilon}+(\gamma+1)\epsilon-D^{F}(\theta^{*}\|\theta^{(t+1),*}),$
(360)

where $(a)$ and $(b)$ follow from (359) and (353), respectively.

Taking the sum for (360), we have

$\displaystyle(t_{1}-1)\Big{(}D^{F}(\theta^{*}\|\theta_{*})-D^{F}(\theta_{f}^{(t_{1})}\|\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta_{f}^{(t_{1})}))\Big{)}$
$\displaystyle\stackrel{{\scriptstyle(a)}}{{\leq}}$
$\displaystyle\sum_{t=1}^{t_{1}-1}\Big{(}D^{F}(\theta^{*}\|\theta_{*})-D^{F}(\theta^{(t+1)}\|\theta_{(t+1)})\Big{)}$
$\displaystyle=$ $\displaystyle
D^{F}(\theta^{*}\|\theta_{*})-D^{F}(\theta^{(2)}\|\theta_{(2)})+\sum_{t=2}^{t_{1}-1}\Big{(}D^{F}(\theta^{*}\|\theta_{*})-D^{F}(\theta^{(t+1)}\|\theta_{(t+1)})\Big{)}$
$\displaystyle\stackrel{{\scriptstyle(b)}}{{\leq}}$ $\displaystyle
D^{F}(\theta^{*}\|\theta^{(1)})-D^{F}(\theta^{*}\|\theta^{(2),*})$
$\displaystyle+\sum_{t=2}^{t_{1}-1}\Big{(}D^{F}(\theta^{*}\|\theta^{(t),*})+2\gamma\sqrt{D(\theta^{*}\|\theta^{(1)})\epsilon}+(\gamma+1)\epsilon-D^{F}(\theta^{*}\|\theta^{(t+1),*})\Big{)}$
$\displaystyle=$ $\displaystyle
D^{F}(\theta^{*}\|\theta^{(1)})-D^{F}(\theta^{*}\|\theta^{(t_{1}),*})+2(t_{1}-2)\gamma\sqrt{D^{F}(\theta^{*}\|\theta^{(1)})\epsilon}+(t_{1}-2)(\gamma+1)\epsilon$
$\displaystyle\leq$ $\displaystyle
D^{F}(\theta^{*}\|\theta^{(1)})+2(t_{1}-2)\gamma\sqrt{D^{F}(\theta^{*}\|\theta^{(1)})\epsilon}+(t_{1}-2)(\gamma+1)\epsilon,$
(361)

where $(a)$ and $(b)$ follow from the relation
$D^{F}(\theta_{f}^{(t_{1})}\|\mathop{\Gamma}\nolimits^{(m),F}_{{\cal
E}}(\theta_{f}^{(t_{1})}))\leq D^{F}(\theta^{(t+1)}\|\theta_{(t+1)})$ and
(360), respectively. Hence, we have (356).

(Step 4): Finally, we show (72) from (71). The condition
$t_{1}-1\geq\frac{2D^{F}(\theta^{*}\|\theta^{(1)})}{\epsilon^{\prime}}$
implies
$\frac{D^{F}(\theta^{*}\|\theta^{(1)})}{t_{1}-1}\leq\frac{\epsilon^{\prime}}{2}$.
The condition
$\epsilon\leq\frac{{\epsilon^{\prime}}^{2}}{4(3\gamma+1)^{2}{D^{F}(\theta^{*}\|\theta^{(1)})}}$
implies
$(3\gamma+1)\sqrt{D^{F}(\theta^{*}\|\theta^{(1)})\epsilon}\leq\frac{\epsilon^{\prime}}{2}$.
Since $D^{F}(\theta^{*}\|\theta^{(1)})\geq\epsilon$, we have
$2\gamma\sqrt{D^{F}(\theta^{*}\|\theta^{(1)})\epsilon}+(\gamma+1)\epsilon\leq\frac{\epsilon^{\prime}}{2}$.
Hence, we have (72).

##  Appendix F Proof of Theorem 4.5

To characterize $\nabla^{(e)}[(F_{\cal E}^{*}\circ
R[V_{1}])^{*}](\hat{\theta})$ for $\hat{\theta}_{\mathsf{a}}\in\Theta_{{\cal
M}}$, we apply (14). For
$(\theta_{\mathsf{a}},\theta_{\mathsf{b}})^{T}\in\Theta_{{\cal M}}$ with
$\theta_{\mathsf{a}}\in\mathbb{R}^{k},\theta_{\mathsf{b}}\in\mathbb{R}^{l-k}$,
the condition
$\hat{\theta}_{\mathsf{a}}=(I,V_{3})(\theta_{\mathsf{a}},\theta_{\mathsf{b}})^{T}$
is equivalent to
$\theta_{\mathsf{a}}=\hat{\theta}_{\mathsf{a}}-V_{3}\theta_{\mathsf{b}}$.
Hence, (14) implies that

$\displaystyle(F_{{\cal E}}^{*}\circ
R[(I,V_{3})])^{*}(\hat{\theta}_{\mathsf{a}})=\min_{\theta_{\mathsf{b}}}F_{\cal
E}(\hat{\theta}_{\mathsf{a}}-V_{3}\theta_{\mathsf{b}},\theta_{\mathsf{b}}).$
(362)

The element $\theta_{\mathsf{b}}^{*}=\mathop{\rm
argmin}\limits_{\theta_{\mathsf{b}}}F_{\cal
E}(\hat{\theta}_{\mathsf{a}}-V_{3}\theta_{\mathsf{b}},\theta_{\mathsf{b}})$
satisfies the following;

$\displaystyle\nabla^{(e)}[F_{\cal
E}](\hat{\theta}_{\mathsf{a}}-V_{3}\theta_{\mathsf{b}},\theta_{\mathsf{b}})\left(\begin{array}[]{c}-V_{3}\\\
I_{l-k}\end{array}\right)=0.$ (365)

That is, when the element $\theta_{\mathsf{b}}$ satisfying (365) is written as
$\theta_{\mathsf{b}}^{*}(\hat{\theta}_{\mathsf{a}})$, we have

$\displaystyle(F_{{\cal E}}^{*}\circ
R[(I,V_{3})])^{*}(\hat{\theta}_{\mathsf{a}})=F_{\cal
E}(\hat{\theta}_{\mathsf{a}}-V_{3}\theta_{\mathsf{b}}^{*}(\hat{\theta}_{\mathsf{a}}),\theta_{\mathsf{b}}^{*}(\hat{\theta}_{\mathsf{a}})).$
(366)

Taking the derivative for $\hat{\theta}_{\mathsf{a}}$ in (366) and using the
relation (365), we have

$\displaystyle\nabla^{(e)}[(F_{\cal E}^{*}\circ
R[V_{1}])^{*}](\hat{\theta}_{\mathsf{a}})=$ $\displaystyle\nabla^{(e)}[F_{\cal
E}](\bar{\theta}_{\mathsf{c}})\left(\begin{array}[]{c}I_{k}\\\
0\end{array}\right),$ (369)

which implies (80).

## References

  * (1) S. Amari, “Information geometry of the EM and em-algorithms for neural networks,” Neural Networks 8(9): 1379 – 1408 (1995).
  * (2) Y. Fujimoto and N. Murata, “A modified EM algorithm for mixture models based on Bregman divergence,” Annals of the Institute of Statistical Mathematics, vol. 59, 3 – 25 (2007).
  * (3) S. Allassonnière and J. Chevallier, “A New Class of EM Algorithms. Escaping Local Minima and Handling Intractable Sampling,”. Computational Statistics & Data Analysis, Elsevier, vol. 159(C). (2019)
  * (4) S. Amari and H. Nagaoka, Methods of Information Geometry (AMS and Oxford, 2000).
  * (5) S. Amari, “$\alpha$-Divergence Is Unique, Belonging to Both f-Divergence and Bregman Divergence Classes,” IEEE Trans. Inform. Theory, vol. 55, 4925 – 4931 (2009).
  * (6) S. Toyota,“Geometry of Arimoto algorithm,” Information Geometry, vol. 3, 183 – 198 (2020)
  * (7) C. E. Shannon,“A Mathematical Theory of Communication,” Bell System Technical Journal, vol.27, 379–423 and 623–656 (1948).
  * (8) S. Arimoto, “An algorithm for computing the capacity of arbitrary discrete memoryless channels,” IEEE Trans. Inform. Theory, vol. 18, no. 1, 14 – 20 (1972).
  * (9) R. Blahut, “Computation of channel capacity and rate-distortion functions,” IEEE Trans. Inform. Theory, vol. 18, no. 4, 460 – 473 (1972).
  * (10) G. Matz and P. Duhamel, “Information geometric formulation and interpretation of accelerated Blahut-Arimoto-Type algorithms,” in Proc. Information Theory Workshop, 24–29, San Antonio, Texas, October, (2004).
  * (11) Yaming Yu, “Squeezing the Arimoto-Blahut algorithm for faster convergence,” IEEE Trans. Inform. Theory, vol. 56, 3149 – 3157 (2010).
  * (12) T. Sutter, D. Sutter, P. Mohajerin Esfahani, and J. Lygeros, “Efficient Approximation of Channel Capacities,” IEEE Trans. Inform. Theory, vol. 61, no. 4, 1649 – 1666 (2015).
  * (13) K. Nakagawa, K. Watabe, and T. Sabu, “On the Search Algorithm for the Output Distribution That Achieves the Channel Capacity,” IEEE Trans. Inform. Theory, vol. 63, 1043 – 1062 (2017).
  * (14) M. Hayashi, “Analytical algorithm for capacities of classical and classical-quantum channels,” IEEE Trans. Inform. Theory, vol. 69, 1680 – 1694 (2023). arXiv:2201.02450 (2022).
  * (15) S. Muroga, “On the Capacity of a Discrete Channel. I Mathematical expression of capacity of a channel which is disturbed by noise in its every one symbol and expressible in one state diagram,” Journal of the Physical Society of Japan, 8, 484-494 (1953).
  * (16) A. D. Wyner, “The wire-tap channel,” Bell. Sys. Tech. Jour., 54 1355 – 1387 (1975).
  * (17) I. Csiszár and J. Körner, “Broadcast channels with confidential messages,” IEEE Trans. Inform. Theory, vol. 24, no. 3, 339 – 348 (1978).
  * (18) A.S. Holevo, “The capacity of the quantum channel with general signal states,” IEEE Trans. Inform. Theory, vol. 44, 269 (1998)
  * (19) B. Schumacher, and M.D. Westmoreland, “Sending classical information via noisy quantum channelsm” Phys. Rev. A vol. 56, 131 (1997)
  * (20) K. Yasui, T. Suko, and T. Matsushima, “An Algorithm for Computing the Secrecy Capacity of Broadcast Channels with Confidential Messages,” Proc. 2007 IEEE Int. Symp. Information Theory (ISIT 2007), Nice, France, 24-29 June 2007, pp. 936 – 940.
  * (21) H. Nagaoka, “Algorithms of Arimoto-Blahut type for computing quantum channel capacity,” Proc. 1998 IEEE Int. Symp. Information Theory (ISIT 1998), Cambridge, MA, USA, 16-21 Aug. 1998, pp. 354.
  * (22) F. Dupuis, W. Yu, and F. Willems, “Blahut-Arimoto algorithms for computing channel capacity and rate-distortion with side information,” Proc. 2014 IEEE Int. Symp. Information Theory (ISIT 2014), Chicago, IL, USA, 27 June-2 July 2004, pp. 179.
  * (23) D. Sutter, T. Sutter, P. M. Esfahani, and R. Renner, “Efficient approximation of quantum channel capacities,” IEEE Trans. Inform. Theory, vol. 62, 578 – 598 (2016).
  * (24) H. Li and N. Cai, “A Blahut-Arimoto Type Algorithm for Computing Classical-Quantum Channel Capacity,” Proc. 2019 IEEE Int. Symp. Information Theory (ISIT 2019), Paris, France, 7-12 July 2019, pp. 255–259.
  * (25) N. Ramakrishnan, R. Iten. V. B. Scholz, and M. Berta, “Computing Quantum Channel Capacities,” IEEE Trans. Inform. Theory, vol. 67, 946 – 960 (2021).
  * (26) M. Hayashi, “Bregman divergence based em-algorithm and its application to classical and quantum rate distortion theory,” IEEE Trans. Inform. Theory, vol. 69, no. 6, 3460 – 3492 (2023).
  * (27) M. Hayashi, Quantum Information Theory: Mathematical Foundation, Graduate Texts in Physics, Springer-Verlag, (2017).
  * (28) S. Toyota, Private communication (2019).
  * (29) A. Kavčić, “On the capacity of Markov sources over noisy channels,” in Proc. IEEE Global Telecommun. Conf., San Antonio, TX, USA, Nov. 2001, pp. 2997 – 3001.
  * (30) P. O. Vontobel, A. Kavčić, D. M. Arnold, and H.-A. Loeliger, “A generalization of the Blahut–Arimoto algorithm to finite-state channels,” IEEE Trans. Inform. Theory, vol. 54, no. 5, pp. 1887 – 1918, May 2008.
  * (31) C. Wu, G. Han, V. Anantharam and B. Marcus, “A Deterministic Algorithm for the Capacity of Finite-State Channels,” IEEE Trans. Inform. Theory, vol. 68, no. 3, pp. 1465 – 1479, March 2022,
  * (32) K. Nakagawa and F. Kanaya, “On the converse theorem in statistical hypothesis testing for Markov chains,” IEEE Trans. Inform. Theory, vol. 39, 629 – 633 (1993).
  * (33) H. Nagaoka “The exponential family of Markov chains and its information geometry,” In Proceedings of the 28th Symposium on Information Theory and Its Applications (SITA2005), Okinawa, Japan (2005).
  * (34) M. Hayashi and S. Watanabe, “ Information Geometry Approach to Parameter Estimation in Markov Chains,” Annals of Statistics, vol. 44, no. 4, 1495 – 1535 (2016).