28

# Proof of Kaneko–Tsumura Conjecture
on Triple $T$-Values

Sasha Berger, Aarav Chandra, Jasper Jain, Daniel Xu
Department of Mathematics, The Bishop’s School, La Jolla, CA 92037, USA
sasha.berger.23<EMAIL_ADDRESS>
jasper.jain.23<EMAIL_ADDRESS>

Ce Xu
School of Math. and Statistics, Anhui Normal University, Wuhu 241000, P.R.
China
<EMAIL_ADDRESS>

Jianqiang Zhao
Department of Mathematics, The Bishop’s School, La Jolla, CA 92037, USA
<EMAIL_ADDRESS>
Abstract Many $\mathbb{Q}$-linear relations exist between multiple zeta
values, the most interesting of which are the weighted sum formulas. In this
paper, we generalized these to Euler sums and Kaneko and Tsumura’s multiple
$T$-values, a variant of the multiple zeta values, by considering generating
functions of the Euler sums. Through this approach we are able to re-prove a
few known formulas, confirm a conjecture of Kaneko and Tsumura on triple
$T$-values, and discover many new identities.

Keywords: Euler sums, multiple zeta values, weighted sum formulas, multiple
$T$-value.

AMS Subject Classifications (2020): 11M32, 11M06, 11G55.

##  1 Introduction

In their seminal work [4], Kaneko and Tsumura defined the multiple $T$-values
(MTVs), a variation of the multiple zeta values as follows: for any positive
integers $s_{1},\dotsc,s_{d}$ with $s_{1}>1$, let

$\displaystyle T(s_{1},\dotsc,s_{d}):=$
$\displaystyle\,\displaystyle\sum_{\begin{subarray}{c}m_{1}>\dotsm>m_{d}>0\\\
m_{j}\equiv
d-j+1\pmod{2}\end{subarray}}\displaystyle\frac{2^{d}}{m_{1}^{s_{1}}\dotsm
m_{d}^{s_{d}}}$ $\displaystyle=$
$\displaystyle\,\displaystyle\sum_{n_{1}>\dotsm>n_{d}>0}\displaystyle\frac{(1+(-1)^{n_{1}+d})\dotsm(1+(-1)^{n_{d-1}})(1-(-1)^{n_{d}})}{n_{1}^{s_{1}}\dotsm
n_{d}^{s_{d}}}.$ (1.1)

They derived many interesting facts concerning these values with analytical
methods and posed some conjectures when the depth $d=3$. In this short note,
we shall prove the following result which was stated in [4, Conjecture 4.6].

######  Theorem 1.1.

_(=Corollary 6.2)_ For all $w\geq 4$

$\displaystyle\sum_{a+b+c=w}2^{b}(3^{a-1}-1)T(a,b,c)=\displaystyle\frac{2}{3}(w-1)(w-2)T(w).$

Observe that all MTVs can be decomposed as linear combinations of Euler sums.
Our strategy to prove Theorem 1.1 is to derive similar weighted sum formulas
for each Euler sum involved by using their generating functions. In the
process, we also discover a few other types of weighted sum formulas for Euler
sums. We remark that the idea of proving identities among MZVs using
generating functions first appeared in [1]. We also notice that Machide
already worked on the MZV case [5, 6] using multiple polylogarithms when depth
$d\leq 4$. Yuan and the last author also studied these directly by using only
finite double shuffle relations in [8] when $d=3$.

##  2 Multiple zeta values and Euler sums

The classical _multiple zeta values_ (abbr. MZVs) are defined by (see [2, 9])

$\displaystyle\zeta(s_{1},\dotsc,s_{d}):=\displaystyle\sum\limits_{m_{1}>\dotsb>m_{d}>0}\displaystyle\frac{1}{m_{1}^{s_{1}}\dotsm
m_{d}^{s_{d}}},$

for positive integers $s_{1},\dotsc,s_{d}$ with $s_{1}>1$. We call
$s_{1}+\dotsb+s_{d}$ and $d$ the _weight_ and _depth_ , respectively. A
composition $(s_{1},\dotsc,s_{d})$ is called _admissible_ if $s_{1}>1$.

Euler sums, also known as alternating MZVs, are defined as follows: for ${\bf
s}=(s_{1},\dotsc,s_{d})\in\mathbb{N}^{d}$, and ${\bf
z}=(z_{1},\dotsc,z_{d})\in\\{\pm 1\\}^{d}$,

$\zeta(s_{1},\dotsc,s_{d};z_{1},\dotsc,z_{d}):=\displaystyle\sum_{n_{1}>\cdots>n_{d}>0}\displaystyle\frac{z_{1}^{n_{1}}\dots
z_{r}^{n_{r}}}{n_{1}^{k_{1}}\dots n_{d}^{k_{d}}}$ (2.2)

which converges if and only if $(s_{1},z_{1})\neq(1,1)$, in which case we call
$({\bf s};{\bf z})$ _admissible_. It is well-known (see, for e.g., [10, p.
17]) that it has an iterated integral expression

$\zeta(s_{1},\dotsc,s_{d};z_{1},\dots,z_{d})=\displaystyle\int_{0}^{1}\left(\displaystyle\frac{dt}{t}\right)^{s_{1}-1}\displaystyle\frac{dt}{a_{1}-t}\cdots\left(\displaystyle\frac{dt}{t}\right)^{s_{d}-1}\displaystyle\frac{dt}{a_{d}-t}.$
(2.3)

where $a_{k}=\prod_{j=1}^{k}z_{j}$ for all $k\leq d$. To save space, we put a
bar on top of $s_{j}$ if $z_{j}=-1$. For example,

$\zeta(\bar{2},3,\bar{1},4)=\zeta(2,3,1,4;-1,1,-1,1).$

It is clear that MZVs are just the special case of Euler sums when all
$z_{j}=1$; namely, no bars can appear.

One of the most important features of Euler sums is that they satisfy many
standard relations (see, for example, [10, §13.3]). Among them, the
(regularized) double shuffle relations, which are consequences of the two
types of expressions in (2.2) and (2.3), play a key role in the study of the
$\mathbb{Q}$-linear relations among these values.

##  3 Regularizations of Euler sums and their generating functions

Let $M$ be a large positive integer and ${\varepsilon}>0$ be a very small
number. For any composition ${\bf s}=(s_{1},\dots,s_{d})$ and
$z_{1},\dotsc,z_{d}=\pm 1$, we consider two variations of the Euler sums
defined by (2.2) and (2.3), respectively. First, set

$\zeta^{(M)}({\bf s};{\bf z}):=\displaystyle\sum_{M\geq
n_{1}>\cdots>n_{d}>0}\displaystyle\frac{z_{1}^{n_{1}}\dots
z_{r}^{n_{r}}}{n_{1}^{s_{1}}\dots n_{d}^{s_{d}}}$ (3.4)

and

$I^{({\varepsilon})}({\bf s};{\bf
z}):=\displaystyle\int_{0}^{1-{\varepsilon}}\left(\displaystyle\frac{dt}{t}\right)^{s_{1}-1}\displaystyle\frac{dt}{a_{1}-t}\cdots\left(\displaystyle\frac{dt}{t}\right)^{s_{d}-1}\displaystyle\frac{dt}{a_{d}-t}$
(3.5)

where $a_{i}=\prod_{j=1}^{i}z_{j}^{-1}$. Then $\zeta^{(M)}$ satisfies the
stuffle product

$\zeta^{(M)}(s_{1};z_{1})\zeta^{(M)}(s_{2};z_{2})=\zeta^{(M)}(s_{1}+s_{2};z_{1}z_{2})+\zeta^{(M)}(s_{1},s_{2};z_{1},z_{2})+\zeta^{(M)}(s_{2},s_{1};z_{2},z_{1}).$
(3.6)

On the other hand, by shuffle product of the iterated integrals we see that

$\displaystyle
I^{({\varepsilon})}(s_{1};z_{1})I^{({\varepsilon})}(s_{2};z_{2})=\displaystyle\sum_{\begin{subarray}{c}t_{1},t_{2}\geq
1\\\
t_{1}+t_{2}=s_{1}+s_{2}\end{subarray}}\Bigg{[}\binom{t_{1}-1}{s_{2}-1}I^{({\varepsilon})}_{t_{1},t_{2}}\Big{(}z_{2},\displaystyle\frac{z_{1}}{z_{2}}\Big{)}+\binom{t_{1}-1}{s_{1}-1}I^{({\varepsilon})}_{t_{1},t_{2}}\Big{(}z_{1},\displaystyle\frac{z_{2}}{z_{1}}\Big{)}\Bigg{]}.$
(3.7)

It is well-known that for admissible $({\bf s};{\bf z})$ we have

$\zeta({\bf s};{\bf z})=\displaystyle\lim_{M\to\infty}\zeta^{(M)}({\bf s};{\bf
z})=\displaystyle\lim_{{\varepsilon}\to 0^{+}}I^{({\varepsilon})}({\bf s};{\bf
z}).$

Therefore one can derive the so-called double shuffle relations using the two
different product structures; namely the stuffle and shuffle products. Then by
the usual regularization process one can discover the extremely useful
regularized double shuffle relation. Briefly speaking, for every admissible
and non-admissible $({\bf s};{\bf z})$ of level $N$ there are two polynomials
of $T$, denoted by $\zeta_{*}({\bf s};{\bf z})$ ($*$-regularzed) and
$\zeta_{\shuffle}({\bf s};{\bf z})$ ($\shuffle$-regularzed), such that

  1. (DBSF1)

$\zeta_{*}({\bf s};{\bf z})\zeta_{*}({\bf s}^{\prime};{\bf z}^{\prime})$ can
be expressed as a $\mathbb{Q}$-linear combination of $*$-regularized colored
MZVs of weight $|{\bf s}|+|{\bf s}^{\prime}|$ and level $N$ using the stuffle
product.

  2. (DBSF2)

$\zeta_{\shuffle}({\bf s};{\bf z})\zeta_{\shuffle}({\bf s}^{\prime};{\bf
z}^{\prime})$ can be expressed as a $\mathbb{Q}$-linear combination of
$\shuffle$-regularized colored MZVs of weight $|{\bf s}|+|{\bf s}^{\prime}|$
and level $N$ using the shuffle product.

  3. (DBSF3)

There is an explicitly defined $\mathbb{R}$-linear map $\rho$ such that
$\rho\circ\zeta_{*}=\zeta_{\shuffle}$ satisfying $\rho(T)=T$ and
$\rho(T^{2})=T^{2}+\zeta(2).$

We will not go into the details of this theory, instead, we would like to
refer the interested reader to §13.3.1 of the book [10].

For any fixed alternating signs ${\bf z}=(z_{1},\dotsc,z_{d})$ we set
$\operatorname*{{sgn}}({\bf
z})=(\operatorname*{{sgn}}(z_{1}),\dotsc,\operatorname*{{sgn}}(z_{d}))$ and
define the generating functions

$F_{\sharp}^{\operatorname*{{sgn}}({\bf z})}({\bf
x}):=\displaystyle\sum_{s_{1},\dotsc,s_{d}\geq 1}\zeta_{\sharp}({\bf s};{\bf
z})x_{1}^{s_{1}-1}\dotsm x_{d}^{s_{d}-1},$ (3.8)

where $\sharp=*$ or $\shuffle$, ${\bf x}=(x_{1},\dotsc,x_{d})$ and
$\zeta_{\sharp}$ denotes the $\sharp$-regularized value.

##  4 Depth 2 weighted sum formulas

By multiplying $x^{s_{1}-1}y^{s_{2}-1}$ on (3.6) and (3.7), taking the sum for
all $s_{1},s_{2}\in\mathbb{N}$, and specializing at
$(z_{1},z_{2})=(1,1),(1,-1)$ and $(-1,-1)$, respectively, we get after
applying the regularization process:

$\displaystyle\zeta_{\sharp}(a)\zeta_{\sharp}(b):$
$\displaystyle\,F_{\shuffle}^{+,+}(x+y,y)+F_{\shuffle}^{+,+}(x+y,x)=F_{*}^{+,+}(x,y)+F_{*}^{+,+}(y,x)+\displaystyle\frac{F_{*}^{+}(x)-F_{*}^{+}(y)}{x-y},$
$\displaystyle\zeta_{\sharp}(a)\zeta(\bar{b}):$
$\displaystyle\,F_{\shuffle}^{+,-}(x+y,y)+F_{\shuffle}^{-,-}(x+y,x)=F_{*}^{+,-}(x,y)+F_{*}^{-,+}(y,x)+\displaystyle\frac{F_{*}^{-}(x)-F_{*}^{-}(y)}{x-y},$
$\displaystyle\zeta(\bar{a})\zeta(\bar{b}):$
$\displaystyle\,F_{\shuffle}^{-,+}(x+y,y)+F_{\shuffle}^{-,+}(x+y,x)=F_{*}^{-,-}(x,y)+F_{*}^{-,-}(y,x)+\displaystyle\frac{F_{*}^{+}(x)-F_{*}^{+}(y)}{x-y}.$

Here we need to remark that $\zeta_{\shuffle}(s)=\zeta_{*}(s)$ for all
$s\in\mathbb{N}$. Now, replacing $(x,y)$ by $(xt,yt)$ and then comparing the
coefficient for $t^{w-2}$ ($w\geq 3$) we immediately derive the following
results. To save space, we set

$\displaystyle\sum=\displaystyle\sum_{a+b=w,a,b\in\mathbb{N}}\quad\text{or}\quad\displaystyle\sum=\displaystyle\sum_{a+b+c=w,a,b,c\in\mathbb{N}}.$

On the other hand, $\displaystyle\sum{}^{\prime}$ means that we remove all
those terms with $a=1$, i.e.,

$\displaystyle\sum{}^{\prime}=\displaystyle\sum_{a+b=w,a\geq
2}\quad\text{or}\quad\displaystyle\sum{}^{\prime}=\displaystyle\sum_{a+b+c=w,a\geq
2}.$

######  Proposition 4.1.

For any fixed $w\geq 3$, set
$f_{w}(x,y)=\displaystyle\sum_{j=0}^{w-2}x^{j}y^{w-2-j}$. Then we have

$\displaystyle\displaystyle\sum\zeta_{\shuffle}(a,b)(x+y)^{{\tilde{a}}}(y^{{\tilde{b}}}+x^{{\tilde{b}}})=$
$\displaystyle\,\displaystyle\sum\zeta_{*}(a,b)(x^{{\tilde{a}}}y^{{\tilde{b}}}+y^{{\tilde{a}}}x^{{\tilde{b}}})+\zeta(w)f_{w}(x,y),$
$\displaystyle\displaystyle\sum(x+y)^{{\tilde{a}}}\big{(}\zeta_{\shuffle}(a,\bar{b})y^{{\tilde{b}}}+\zeta(\bar{a},\bar{b})x^{{\tilde{b}}}\big{)}=$
$\displaystyle\,\displaystyle\sum\big{(}\zeta_{*}(a,\bar{b})x^{{\tilde{a}}}y^{{\tilde{b}}}+\zeta(\bar{a},b)y^{{\tilde{a}}}x^{{\tilde{b}}}\big{)}+\zeta(\bar{w})f_{w}(x,y),$
$\displaystyle\displaystyle\sum\zeta(\bar{a},b)(x+y)^{{\tilde{a}}}(y^{{\tilde{b}}}+x^{{\tilde{b}}})=$
$\displaystyle\,\displaystyle\sum\zeta(\bar{a},\bar{b})(x^{{\tilde{a}}}y^{{\tilde{b}}}+y^{{\tilde{a}}}x^{{\tilde{b}}})+\zeta(w)f_{w}(x,y).$

Here and in the rest of this paper, we set ${{\tilde{a}}}=a-1$,
${{\tilde{b}}}=b-1$.

######  Theorem 4.2.

Let $w\geq 3$ and $v=w-1$. Then we have

$\displaystyle\displaystyle\sum{}^{\prime}\zeta(a,b)=$
$\displaystyle\,\zeta(w),$
$\displaystyle\displaystyle\sum{}^{\prime}\zeta(\bar{a},\bar{b})=$
$\displaystyle\,\zeta(\bar{1},v)-\zeta(\bar{1},\bar{v})+\zeta(\bar{w}),$
$\displaystyle\displaystyle\sum{}^{\prime}\zeta(\bar{a},b)=$
$\displaystyle\,\zeta(\bar{v},\bar{1})+\zeta(\bar{1},\bar{v})-\zeta(\bar{v},1)-\zeta(\bar{1},v)+\zeta(w),$
$\displaystyle\displaystyle\sum{}^{\prime}\zeta(a,\bar{b})=$
$\displaystyle\,\zeta(\bar{v},1)-\zeta(\bar{v},\bar{1})+\zeta(\bar{w}).$

###### Proof.

We can prove these by taking $x=1,y=0$ and $x=0,y=1$ in Proposition 4.1. ∎

This immediately implies the following corollary about double $T$-values.

######  Corollary 4.3.

Let $w\geq 3$ and $v=w-1$. Then we have

$\displaystyle\displaystyle\sum_{a+b=w,a\geq 2}T(a,b)=$
$\displaystyle\,2\Big{(}\zeta(w)-\zeta(\bar{w})+\zeta(\bar{v},\bar{1})+\zeta(\bar{1},\bar{v})-\zeta(\bar{v},1)-\zeta(\bar{1},v)\Big{)}.$
(4.9)

###### Proof.

We have

$\displaystyle\displaystyle\sum{}^{\prime}T(a,b)=\displaystyle\sum{}^{\prime}\Big{(}\zeta(a,b)+\zeta(\bar{a},b)-\zeta(a,\bar{b})-\zeta(\bar{a},\bar{b})\Big{)}$

which reduces to (4.9) quickly by Theorem 4.2. ∎

With this approach we can now easily recover [4, Thm. 3.2]. Notice that in the
proof, we find the depth 2 weighted sum formulas for Euler sums which are
analogs of the MZV formula first discovered by Ohno and Zulidin [7] and later
generalized to arbitrary depth by Guo and Xie [3].

######  Theorem 4.4.

For all $w\geq 3$ we have

$\displaystyle\displaystyle\sum_{a+b=w,a\geq 2}2^{a-1}T(a,b)=$
$\displaystyle\,(w-1)(\zeta(w)-\zeta(\bar{w}))=(w-1)T(w).$

###### Proof.

Set $\displaystyle\sum{}^{\prime}=\displaystyle\sum{}_{a+b=w,a\geq 2}$. Taking
$x=1,y=1$ in Proposition 4.1 we get

$\displaystyle\displaystyle\sum{}^{\prime}2^{{\tilde{a}}}\zeta(a,b)=$
$\displaystyle\,\displaystyle\sum{}^{\prime}\zeta(a,b)+\displaystyle\frac{w-1}{2}\zeta(w)=\displaystyle\frac{w+1}{2}\zeta(w),$
(4.10)
$\displaystyle\displaystyle\sum{}^{\prime}2^{{\tilde{a}}}\big{(}\zeta(a,\bar{b})+\zeta(\bar{a},\bar{b})\big{)}=$
$\displaystyle\,\displaystyle\sum{}^{\prime}\big{(}\zeta(a,\bar{b})+\zeta(\bar{a},b)\big{)}+(w-1)\zeta(\bar{w})=\zeta(w)+w\zeta(\bar{w}),$
$\displaystyle\displaystyle\sum{}^{\prime}2^{{\tilde{a}}}\zeta(\bar{a},b)=$
$\displaystyle\,\displaystyle\sum{}^{\prime}\zeta(\bar{a},\bar{b})+\displaystyle\frac{w-1}{2}\zeta(w)=\zeta(\bar{w})+\displaystyle\frac{w-1}{2}\zeta(w),$

using the sum formulas in Theorem 4.2. The theorem follows immediately. ∎

##  5 Depth 3 weighted sum formulas, Part A

As in the depth 2 case, we may derive many identities by the generating
functions of the triple Euler sums. For depth 3, there are two possible ways
to produce functional equations of $F_{\sharp}^{\operatorname*{{sgn}}({\bf
z})}({\bf x})$. We may consider either

  1. (A)

products of double logarithms with single logarithms, denoted by
$\zeta^{\sharp}_{s_{1},s_{2}}(z_{1},z_{2})\zeta^{\sharp}_{s_{3}}(z_{3})$, or

  2. (B)

products of three logarithms,
$\zeta^{\sharp}_{s_{1}}(z_{1})\zeta^{\sharp}_{s_{2}}(z_{2})\zeta^{\sharp}_{s_{3}}(z_{3})$.

We start by dealing with case (A) in this section. Observe that

$\zeta^{(M)}(s_{1},s_{2};z_{1},z_{2})\zeta^{(M)}(s_{3};z_{3})=\zeta^{(M)}(s_{1}+s_{3},s_{2};z_{1}z_{3},z_{2})+\zeta^{(M)}(s_{1},s_{2}+s_{3};z_{1},z_{2}z_{3})\\\
\zeta^{(M)}(s_{1},s_{2},s_{3};z_{1},z_{2},z_{3})+\zeta^{(M)}(s_{1},s_{3},s_{2};z_{1},z_{3},z_{2})+\zeta^{(M)}(s_{3},s_{1},s_{2};z_{3},z_{1},z_{2})$
(5.11)

On the other hand, by shuffle product of iterated integrals we see that

$\displaystyle
I^{({\varepsilon})}(s_{1},s_{2};z_{1},z_{2})I^{({\varepsilon})}(s_{3};z_{3})=$
$\displaystyle\,\displaystyle\sum_{\begin{subarray}{c}t_{1}\geq 2,t_{2}\geq
1\\\
t_{1}+t_{2}=s_{1}+s_{3}\end{subarray}}\binom{t_{1}-1}{s_{3}-1}I^{({\varepsilon})}\Big{(}t_{1},t_{2},s_{2};z_{3},\displaystyle\frac{z_{1}}{z_{3}},z_{2}\Big{)}$
(5.12) $\displaystyle+$
$\displaystyle\,\displaystyle\sum_{\begin{subarray}{c}t_{1}\geq
2,t_{2},t_{3}\geq 1\\\ t_{1}+t_{2}+t_{3}\\\
=s_{1}+s_{2}+s_{3}\end{subarray}}\binom{t_{1}-1}{s_{1}-1}\binom{t_{2}-1}{s_{2}-t_{3}}I^{({\varepsilon})}\Big{(}t_{1},t_{2},t_{3};z_{1},\displaystyle\frac{z_{3}}{z_{1}},\displaystyle\frac{z_{1}z_{2}}{z_{3}}\Big{)}$
(5.13) $\displaystyle+$
$\displaystyle\,\displaystyle\sum_{\begin{subarray}{c}t_{1}\geq
2,t_{2},t_{3}\geq 1\\\ t_{1}+t_{2}+t_{3}\\\
=s_{1}+s_{2}+s_{3}\end{subarray}}\binom{t_{1}-1}{s_{1}-1}\binom{t_{2}-1}{s_{2}-1}I^{({\varepsilon})}\Big{(}t_{1},t_{2},t_{3};z_{1},z_{2},\displaystyle\frac{z_{3}}{z_{2}}\Big{)}.$
(5.14)

To save space, we set $\zeta_{\sharp}\Big{(}{{\bf s}\atop{\bf
z}}\Big{)}:=\zeta_{\sharp}({\bf s};{\bf z})$. By the usual regularization
process (5.11) and (5.12) easily lead to the following functional equations
for any weight $w\geq 3$ in view of Lemma 5.1.

$\displaystyle\displaystyle\frac{\delta}{2}\zeta(2)\zeta_{*}\Big{(}{{w-2}\atop{z_{3}}}\Big{)}z^{w-3}+\displaystyle\sum_{b+c=w}\Bigg{[}\left(\displaystyle\frac{x^{{\tilde{b}}}-z^{{\tilde{b}}}}{x-z}\right)y^{{\tilde{c}}}\zeta_{*}\Big{(}{{\
\,b\,\
,\,c\,}\atop{z_{1}z_{3},z_{2}}}\Big{)}+x^{{\tilde{b}}}\left(\displaystyle\frac{y^{{\tilde{c}}}-z^{{\tilde{c}}}}{y-z}\right)\zeta_{*}\Big{(}{{b\,,\
\,c\,\ }\atop{z_{1},z_{2}z_{3}}}\Big{)}\Bigg{]}$
$\displaystyle+\displaystyle\sum_{a+b+c=w}\Bigg{[}x^{{\tilde{a}}}y^{{\tilde{b}}}z^{{\tilde{c}}}\zeta_{*}\Big{(}{{\,a\,,\,b\,,\,c\,}\atop{z_{1},z_{2},z_{3}}}\Big{)}+x^{{\tilde{a}}}z^{{\tilde{b}}}y^{{\tilde{c}}}\zeta_{*}\Big{(}{{\,a\,,\,b\,,\,c\,}\atop{z_{1},z_{3},z_{2}}}\Big{)}+z^{{\tilde{a}}}x^{{\tilde{b}}}y^{{\tilde{c}}}\zeta_{*}\Big{(}{{\,a\,,\,b\,,\,c\,}\atop{z_{3},z_{1},z_{2}}}\Big{)}\Bigg{]}$
$\displaystyle=\displaystyle\sum_{a+b+c=w}\Bigg{[}(x+z)^{{\tilde{a}}}(y+z)^{{\tilde{b}}}y^{{\tilde{c}}}\zeta_{\shuffle}\Big{(}{{\,a\,,\
\,b\,\ ,\ \,c\,\ }\atop{z_{1},z_{3}/z_{1},z_{1}z_{2}/z_{3}}}\Big{)}$
$\displaystyle\hskip
56.9055pt+(x+z)^{{\tilde{a}}}(y+z)^{{\tilde{b}}}z^{{\tilde{c}}}\zeta_{\shuffle}\Big{(}{{\,a\,,\,b\,,\
\,c\,\
}\atop{z_{1},z_{2},z_{3}/(z_{1}z_{2})}}\Big{)}+(x+z)^{{\tilde{a}}}x^{{\tilde{b}}}y^{{\tilde{c}}}\zeta_{\shuffle}\Big{(}{{\,a\,,\
\,b\,\ ,\,c\,}\atop{z_{3},z_{1}/z_{3},z_{2}}}\Big{)}\Bigg{]},$ (5.15)

where $\delta=1$ if $z_{1}=z_{2}=1$, and $\delta=0$ otherwise. This yields
eight cases by different combinations of $z_{1},z_{2},z_{3}=\pm 1$.

The following lemma will help us to simplify the above formula when
regularized values appear.

######  Lemma 5.1.

Suppose $a,b,c\in{\mathbb{D}}$ with $|a|+|b|+|c|=w\geq 4$. Then for all
$(a,b)\neq(1,1)$ we have

$\zeta_{\shuffle}(a,b,c)=\zeta_{*}(a,b,c).$

Further, for $c=w-2$ or $c=\overline{w-2}$

$\zeta_{\shuffle}(1,1,c)=\zeta_{*}(1,1,c)+\displaystyle\frac{1}{2}\zeta(2)\zeta(c).$

###### Proof.

We know the regularized values $\zeta_{\shuffle}(a,b,c)$ and
$\zeta_{*}(a,b,c)$ are either constants or linear polynomials of $T$ if
$(a,b)\neq(1,1)$ since the weight is at least 4. Hence
$\zeta_{\shuffle}(a,b,c)=\zeta_{*}(a,b,c)$ as $\rho(T)=T$ by (DBSF3) on page
(DBSF3). On the other hand, for all $c\in{\mathbb{D}}\setminus\\{1\\}$,

$\zeta_{\shuffle}(1,1,c)=\rho(\zeta_{*}(1,1,c))=\rho(\zeta_{*}(1,1)\zeta(c)-f(T))$

for some linear polynomial $f(T)$. Thus, by (DBSF3)

$\zeta_{\shuffle}(1,1,c)=\rho\left(\displaystyle\frac{T^{2}-\zeta(2)}{2}\right)\zeta(c)-f(T)=\displaystyle\frac{T^{2}}{2}\zeta(c)-f(T)=\zeta_{*}(1,1,c)+\displaystyle\frac{1}{2}\zeta(2)\zeta(c)$

as desired. ∎

######  Theorem 5.2.

Let $w\geq 4$, $u=w-2$ and $v=w-1$. Then we have

$\displaystyle\displaystyle\sum{}^{\prime}\zeta(a,b,1)=$
$\displaystyle\,\zeta(v,1)+\zeta(u,2),$
$\displaystyle\displaystyle\sum{}^{\prime}\zeta(a,\bar{b},1)=$
$\displaystyle\,\zeta(\overline{v},1)+\zeta(\bar{u},2)+2\zeta(\bar{u},1,1)-\zeta(\bar{u},\bar{1},\bar{1})-\zeta(\bar{u},1,\bar{1}),$
$\displaystyle\displaystyle\sum{}^{\prime}\zeta(\bar{a},b,1)=$
$\displaystyle\,\zeta(v,1)+\zeta(\bar{u},\bar{2})+\zeta(\bar{u},1,\bar{1})+\zeta(\bar{u},\bar{1},1)+\zeta(\bar{1},\bar{u},1)-2\zeta(\bar{u},1,1)-\zeta(\bar{1},u,1),$
$\displaystyle\displaystyle\sum{}^{\prime}\zeta(\bar{a},\bar{b},1)=$
$\displaystyle\,\zeta(\overline{v},1)+\zeta(u,\bar{2})+\zeta(u,\bar{1},1)+\zeta(\bar{1},u,1)-\zeta(u,\bar{1},\bar{1})-\zeta(\bar{1},\bar{u},1),$
$\displaystyle\displaystyle\sum{}^{\prime}\zeta(\bar{a},\bar{b},\bar{1})=$
$\displaystyle\,\zeta(v,\bar{1})+\zeta(\bar{u},2)+\zeta(\bar{u},\bar{1},\bar{1})-\zeta(\bar{u},1,\bar{1}),$
$\displaystyle\displaystyle\sum{}^{\prime}\zeta(a,b,\bar{1})=$
$\displaystyle\,\zeta(v,\bar{1})+\zeta(u,\bar{2})+\zeta(u,\bar{1},1)-\zeta(u,\bar{1},\bar{1}),$
$\displaystyle\displaystyle\sum{}^{\prime}\zeta(a,\bar{b},\bar{1})=$
$\displaystyle\,\zeta(\overline{v},\bar{1})+\zeta(\bar{u},\bar{2})+\zeta(\bar{u},1,\bar{1})-\zeta(\bar{u},\bar{1},1),$
$\displaystyle\displaystyle\sum{}^{\prime}\zeta(\bar{a},b,\bar{1})=$
$\displaystyle\,\zeta(v,\bar{1})+\zeta(\bar{u},2)+\zeta(\bar{u},\bar{1},\bar{1})+\zeta(\bar{1},\bar{u},\bar{1})-\zeta(\bar{u},1,\bar{1})-\zeta(\bar{1},u,\bar{1}).$

###### Proof.

Taking $x=1,y=z=0$ in (5.15) we obtain the formulas in the theorem immediately
using Lemma 5.1. ∎

######  Corollary 5.3.

Let $w\geq 4$, $u=w-2$ and $v=w-1$. Then we have

$\displaystyle\displaystyle\sum{}^{\prime}T(a,b,1)=$
$\displaystyle\,2T(u,2)+4\Big{(}\zeta(u,\bar{1},\bar{1})+\zeta(\bar{u},1,1)-\zeta(\bar{u},1,\bar{1})-\zeta(u,\bar{1},1)\Big{)}.$

###### Proof.

These follow from direct computation using Theorem 5.2 and the two identities:

$\zeta(\bar{1},\bar{1})-\zeta(\bar{1},1)=\zeta(\bar{2})\quad\text{and}\quad
2\zeta(\bar{2})=-\zeta(2).$ (5.16)

See, for example, [10, Prop. 14.2.5]. ∎

######  Theorem 5.4.

Let $w\geq 4$, $u=w-2$ and $v=w-1$. Set
$\displaystyle\sum{}=\displaystyle\sum{}_{b+c=w}$. Then we have

$\displaystyle\displaystyle\sum{}\zeta_{\shuffle}(1,b,c)=$
$\displaystyle\,\zeta(2,u)+\zeta_{*}(1,v)+\zeta_{*}(1,1,u)-\displaystyle\frac{1}{2}\zeta(2)\zeta(u),$
$\displaystyle\displaystyle\sum{}\zeta_{\shuffle}(1,\bar{b},c)=$
$\displaystyle\,\zeta(\bar{2},\bar{u})+\zeta_{*}(1,v)+\zeta_{*}(1,\bar{1},\bar{u})+\zeta_{*}(1,\bar{u},\bar{1})+\zeta(\bar{1},1,\bar{u})-\zeta(\bar{1},\bar{1},\bar{u})-\zeta_{*}(1,\bar{u},1),$
$\displaystyle\displaystyle\sum{}\zeta(\bar{1},b,c)=$
$\displaystyle\,\zeta(2,u)+\zeta(\bar{1},\bar{v})+\zeta(\bar{1},u,\bar{1})+2\zeta(\bar{1},\bar{1},u)-\zeta(\bar{1},1,u)-\zeta(\bar{1},u,1),$
$\displaystyle\displaystyle\sum{}\zeta(\bar{1},\bar{b},c)=$
$\displaystyle\,\zeta(\bar{2},\bar{u})+\zeta(\bar{1},\bar{v})+\zeta(\bar{1},1,\bar{u}),$
$\displaystyle\displaystyle\sum{}\zeta(\bar{1},\bar{b},\bar{c})=$
$\displaystyle\,\zeta(\bar{2},u)+\zeta(\bar{1},v)+\zeta(\bar{1},u,1)+\zeta(\bar{1},1,u)-\zeta(\bar{1},u,\bar{1}),$
$\displaystyle\displaystyle\sum{}\zeta_{\shuffle}(1,b,\bar{c})=$
$\displaystyle\,\zeta(2,\bar{u})+\zeta_{*}(1,\bar{v})+\zeta_{*}(1,\bar{u},1)+\zeta_{*}(1,1,\bar{u})-\zeta_{\shuffle}(1,\bar{u},\bar{1})-\displaystyle\frac{1}{2}\zeta(2)\zeta(\bar{u}),$
$\displaystyle\displaystyle\sum{}\zeta_{\shuffle}(1,\bar{b},\bar{c})=$
$\displaystyle\,\zeta(\bar{2},u)+\zeta_{*}(1,\bar{v})+\zeta_{*}(1,\bar{1},u)+\zeta(\bar{1},1,u)-\zeta(\bar{1},\bar{1},u),$
$\displaystyle\displaystyle\sum{}\zeta(\bar{1},b,\bar{c})=$
$\displaystyle\,\zeta(2,\bar{u})+\zeta(\bar{1},v)+2\zeta(\bar{1},\bar{1},\bar{u})-\zeta(\bar{1},1,\bar{u}).$

###### Proof.

Taking $y=1,x=z=0$ in (5.15) we obtain the formulas in the theorem immediately
by Lemma 5.1. ∎

Utilizing the same ideas as above, we may now derive the sum formulas for all
triple Euler sums and therefore the sum formula for triple $T$-values.

######  Theorem 5.5.

Let $w\geq 4$ and $u=w-2$. Then we have

$\displaystyle\displaystyle\sum{}^{\prime}T(a,b,c)=$
$\displaystyle\,\displaystyle\frac{2}{3}T(2)T(u)-2T(u,2)+4\Big{(}\zeta(u,\bar{1},1)-\zeta({\bar{u}},1,1)+\zeta({\bar{u}},1,\bar{1})-\zeta(u,\bar{1},\bar{1})\Big{)}.$

###### Proof.

Taking $x=y=0,z=1$ in (5.15) and using Lemma 5.1 we can find the sum formulas
for all triple Euler sums as follows:

$\displaystyle\displaystyle\sum{}^{\prime}\zeta(a,b,c)=$
$\displaystyle\,\zeta(w),$
$\displaystyle\displaystyle\sum{}^{\prime}\zeta(a,\bar{b},c)=$
$\displaystyle\,2\zeta(\bar{u},1,\bar{1})-2\zeta(\bar{u},1,1)-\zeta(\bar{1},1,\bar{u})+\zeta(\bar{1},\bar{1},\bar{u})$
$\displaystyle+\zeta(\bar{v},\bar{1})-\zeta(\bar{v},1)-\zeta(\bar{u},2)-\zeta(\bar{2},\bar{u}),$
$\displaystyle\displaystyle\sum{}^{\prime}\zeta(\bar{a},b,c)=$
$\displaystyle\,\zeta(\bar{u},1,1)-\zeta(\bar{u},1,\bar{1})+\zeta(\bar{1},u,1)-\zeta(\bar{1},u,\bar{1})$
$\displaystyle+\zeta(\bar{1},1,\bar{u})+\zeta(\bar{1},1,u)-2\zeta(\bar{1},\bar{1},u)-\zeta(\bar{u},\bar{2})-\zeta(2,u)$
$\displaystyle\displaystyle\sum{}^{\prime}\zeta(\bar{a},\bar{b},c)=$
$\displaystyle\,\zeta(\bar{1},1,u)-\zeta(\bar{1},1,\bar{u})+\zeta(\bar{1},v)-\zeta(\bar{1},\bar{v})+\zeta(\bar{2},u)-\zeta(\bar{2},\bar{u})+\zeta(\bar{w}),$
$\displaystyle\displaystyle\sum{}^{\prime}\zeta(\bar{a},b,\bar{c})=$
$\displaystyle\,\zeta(u,\bar{1},1)-\zeta(u,\bar{1},\bar{1})+\zeta(\bar{1},1,\bar{u})-2\zeta(\bar{1},\bar{1},\bar{u})+\zeta(\bar{1},\bar{1},u)$
$\displaystyle+\zeta(\bar{1},\bar{v})-\zeta(\bar{1},v)-\zeta(u,2)-\zeta(2,\bar{u})-\zeta(u)\zeta(\bar{2}),$
$\displaystyle\displaystyle\sum{}^{\prime}\zeta(a,\bar{b},\bar{c})=$
$\displaystyle\,\zeta(u,\bar{1},\bar{1})-\zeta(u,\bar{1},1)+\zeta(\bar{1},\bar{1},u)-\zeta(\bar{1},1,u)-\zeta(u,\bar{2})-\zeta(\bar{2},u),$
$\displaystyle\displaystyle\sum{}^{\prime}\zeta(a,b,\bar{c})=$
$\displaystyle\,\zeta(\bar{u},1,1)-\zeta(\bar{u},1,\bar{1})+\zeta(\bar{v},1)-\zeta(\bar{v},\bar{1})-\zeta(\bar{u},\bar{2})+\zeta(\bar{u},2)+\zeta(\bar{w}),$
$\displaystyle\displaystyle\sum{}^{\prime}\zeta(\bar{a},\bar{b},\bar{c})=$
$\displaystyle\,\zeta(\bar{1},u,\bar{1})-\zeta(\bar{1},u,1)+\zeta(\bar{1},\bar{1},\bar{u})-\zeta(\bar{1},1,u)-\zeta(\bar{u},2)-\zeta(\bar{2},u).$

In the process we need Theorems 5.2 and 5.4. The theorem follows immediately
from the decomposition of the triple $T$-values in terms of the triple Euler
sums. ∎

From the above we can derive [4, Thm. 3.3] as a corollary.

######  Corollary 5.6.

For all $w\geq 4$ we have

$\displaystyle\sum_{a+b+c=w,a\geq 2}T(a,b,c)+\displaystyle\sum_{a+b=w,a\geq
2}T(a,b,1)=\displaystyle\frac{2}{3}T(2)T(w-2).$

###### Proof.

Let $u=w-2$. We only need to remove the non-admissible terms from those eight
equations in the proof of Theorem 5.5 by Theorem 5.4. ∎

The following restricted sum formulas for triple Euler sums will be needed in
the future.

######  Theorem 5.7.

Let $w\geq 4$, $u=w-2$ and $v=w-1$. Set
$\displaystyle\sum{}=\displaystyle\sum{}_{a+c=w-1}$. Then we have

$\displaystyle\displaystyle\sum{}\zeta_{*}(a,1,c)$
$\displaystyle\,=\zeta_{*}(1,1,u)+\zeta(v,1)+\zeta(2,u),$
$\displaystyle\displaystyle\sum{}\zeta_{*}(a,\bar{1},\bar{c})$
$\displaystyle\,=\zeta_{*}(1,\bar{1},\bar{u})+\zeta(\bar{1},\bar{u},\bar{1})-\zeta(\bar{1},\bar{u},1)+\zeta(v,\bar{1})+\zeta(\bar{2},\bar{u}),$
$\displaystyle\displaystyle\sum{}\zeta(\bar{a},\bar{1},c)$
$\displaystyle\,=\zeta(\bar{1},\bar{1},\bar{u})+\zeta(2,\bar{u})+\zeta(\bar{v},\bar{1}),$
$\displaystyle\displaystyle\sum{}\zeta(\bar{a},1,\bar{c})$
$\displaystyle\,=\zeta(\bar{1},u,1)+\zeta(\bar{1},1,u)-\zeta(\bar{1},u,\bar{1})+\zeta(\bar{v},1)+\zeta(\bar{2},u),$
$\displaystyle\displaystyle\sum{}\zeta(\bar{a},1,c)$
$\displaystyle\,=\zeta(\bar{1},\bar{1},\bar{u})+2\zeta(\bar{v},1)+\zeta(\bar{1},v)-\zeta(\bar{1},\bar{v})-\zeta(\bar{v},\bar{1})-\zeta(\bar{u},2)-\zeta(w),$
$\displaystyle\displaystyle\sum{}\zeta_{*}(a,1,\bar{c})$
$\displaystyle\,=\zeta(\bar{u},\bar{1},\bar{1})-\zeta(\bar{u},\bar{1},1)+\zeta_{*}(1,1,\bar{u})-\zeta(\bar{v},\bar{1})$
$\displaystyle\,+\zeta(v,\bar{1})+2\zeta(\bar{v},\bar{1})-\zeta(\bar{v},1)-\zeta(\bar{u},\bar{2})+\zeta(2)\zeta(\bar{u})-\zeta(\bar{w}),$
$\displaystyle\displaystyle\sum{}\zeta_{*}(a,\bar{1},c)$
$\displaystyle\,=2\zeta(u,\bar{1},\bar{1})-2\zeta(u,\bar{1},1)+\zeta(\bar{1},1,u)+\zeta_{*}(1,\bar{1},u)-\zeta(\bar{1},\bar{1},u)+\zeta(\bar{v},\bar{1})$
$\displaystyle\,+\zeta(v,\bar{1})+\zeta(\bar{1},v)-\zeta(\bar{v},1)-\zeta(\bar{1},\bar{v})-\zeta(2,u)-2\zeta(u,\bar{2})-\zeta(\bar{w})-\zeta(w),$
$\displaystyle\displaystyle\sum{}\zeta(\bar{a},\bar{1},\bar{c})$
$\displaystyle\,=3\zeta(u,\bar{1},1)-3\zeta(u,\bar{1},\bar{1})+\zeta(\bar{1},\bar{1},u)+2\zeta(\bar{v},1)-\zeta(\bar{v},\bar{1})+\zeta(u,\bar{2})+\zeta(2,u)-\zeta(u,2).$

###### Proof.

Taking $x=y=1,z=0$ in (5.15) we obtain the formulas in the theorem using Lemma
5.1, Theorems 4.2, 5.2, 5.4, and the sum formulas in the proof of Theorem 5.5.
∎

The following result will play a key role in the proof of Theorem 1.1.

######  Theorem 5.8.

Let $w\geq 4$, $u=w-2$ and $v=w-1$. Then we have

$\displaystyle\displaystyle\sum{}2\cdot
2^{\tilde{b}}\zeta_{\shuffle}(a,b,c)=2\zeta_{*}(1,1,u)+w\zeta_{*}(1,v)-2\zeta(u,2)-\zeta(w),$
$\displaystyle\displaystyle\sum{}2\cdot
2^{\tilde{b}}\zeta_{\shuffle}(a,\bar{b},c)=4\zeta(u,\bar{1},\bar{1})-4\zeta(u,\bar{1},1)-2\zeta(\bar{1},\bar{1},u)+2\zeta(\bar{1},1,u)+2\zeta_{*}(1,\bar{1},u)$
$\displaystyle\ \hskip
42.67912pt+2\zeta(\bar{v},\bar{1})-2\zeta(\bar{v},1)+u\zeta_{*}(1,v)+2\zeta_{*}(1,\bar{v})+4\zeta(\bar{2},u)+2\zeta(u,2)+\zeta(w)+2\zeta(\bar{w}),$
$\displaystyle\displaystyle\sum{}2\cdot
2^{\tilde{b}}\zeta(\bar{a},b,c)=2\zeta(\bar{1},\bar{1},\bar{u})+w\zeta(\bar{1},\bar{v})+2\zeta(\bar{v},\bar{1})-2\zeta(\bar{v},1)$
$\displaystyle\ \hskip
42.67912pt+\zeta(\bar{2},\bar{u})+\zeta(2,\bar{u})+2\zeta(w)-\zeta(\bar{w})+\zeta(2)\zeta(\bar{u}),$
$\displaystyle\displaystyle\sum{}2\cdot
2^{\tilde{b}}\zeta(\bar{a},\bar{b},c)=2\zeta(\bar{1},\bar{1},\bar{u})+u\zeta(\bar{1},\bar{v})+2\zeta(\bar{1},v)+\zeta(\bar{2},\bar{u})+\zeta(2,\bar{u})+\zeta(\bar{w}),$
$\displaystyle\displaystyle\sum{}2^{\tilde{b}}\Big{(}\zeta_{\shuffle}(\bar{a},\bar{b},\bar{c})+\zeta_{\shuffle}(\bar{a},b,\bar{c})\Big{)}=2\zeta(u,\bar{1},1)+2\zeta(\bar{1},\bar{1},u)-2\zeta(u,\bar{1},\bar{1})$
$\displaystyle\ \hskip
42.67912pt+v\zeta(\bar{1},v)+\zeta(\bar{v},1)+\zeta(\bar{1},\bar{v})-\zeta(\bar{v},\bar{1})+\zeta(2,u)-\zeta(u,2),$
$\displaystyle\displaystyle\sum{}2^{\tilde{b}}\Big{(}\zeta_{\shuffle}(a,b,\bar{c})+\zeta_{\shuffle}(a,\bar{b},\bar{c})\Big{)}=\zeta_{*}(1,\bar{1},\bar{u})+\zeta(\bar{1},1,\bar{u})-\zeta(\bar{1},\bar{1},\bar{u})+\zeta_{*}(1,1,\bar{u})$
$\displaystyle\ \hskip
42.67912pt-\zeta(\bar{v},\bar{1})+\zeta(\bar{v},1)+\zeta_{*}(1,v)+v\zeta_{*}(1,\bar{v})+\zeta(\bar{2},\bar{u})+\zeta(2,\bar{u})+\zeta(\bar{w})+3\zeta(\bar{2})\zeta(\bar{u}).$

###### Proof.

Taking $x=0,y=z=1$ in (5.15) we obtain the formulas in the theorem using Lemma
5.1, Theorems 4.2, 5.2, 5.4, and 5.7.. ∎

##  6 Depth 3 weighted sum formulas, Part B

Similarly, by considering case (B)
$\zeta_{\sharp}(s_{1})\zeta_{\sharp}(s_{2})\zeta_{\sharp}(s_{3})$ ($\sharp=*$
or $\shuffle$, $s_{j}\in{\mathbb{D}}$) we may arrive at the following, where
$\displaystyle\sum=\displaystyle\sum_{a+b+c=w,a,b,c\geq 1}$ as before and
$\displaystyle\sum{}^{\prime}=\displaystyle\sum_{k+c=w,k\geq 2}$:

$\displaystyle\displaystyle\sum{}\Big{[}(x+y+z)^{{\tilde{a}}}(z+y)^{{\tilde{b}}}y^{{\tilde{c}}}\zeta_{\shuffle}\Big{(}{{\,a\,,\
\,b\,\ ,\ \,c\,\
}\atop{z_{1},z_{3}/z_{1},z_{2}/z_{3}}}\Big{)}+(x+y+z)^{{\tilde{a}}}(z+y)^{{\tilde{b}}}z^{{\tilde{c}}}\zeta_{\shuffle}\Big{(}{{\,a\,,\
\,b\,\ ,\ \,c\,\ }\atop{z_{1},z_{2}/z_{1},z_{3}/z_{2}}}\Big{)}$
$\displaystyle+(x+y+z)^{{\tilde{a}}}(z+x)^{{\tilde{b}}}x^{{\tilde{c}}}\zeta_{\shuffle}\Big{(}{{\,a\,,\
\,b\,\ ,\ \,c\,\
}\atop{z_{2},z_{3}/z_{2},z_{1}/z_{3}}}\Big{)}+(x+y+z)^{{\tilde{a}}}(z+x)^{{\tilde{b}}}z^{{\tilde{c}}}\zeta_{\shuffle}\Big{(}{{\,a\,,\
\,b\,\ ,\ \,c\,\ }\atop{z_{2},z_{1}/z_{2},z_{3}/z_{1}}}\Big{)}$
$\displaystyle+(x+y+z)^{{\tilde{a}}}(x+y)^{{\tilde{b}}}y^{{\tilde{c}}}\zeta_{\shuffle}\Big{(}{{\,a\,,\
\,b\,\ ,\ \,c\,\
}\atop{z_{3},z_{1}/z_{3},z_{2}/z_{1}}}\Big{)}+(x+y+z)^{{\tilde{a}}}(x+y)^{{\tilde{b}}}x^{{\tilde{c}}}\zeta_{\shuffle}\Big{(}{{\,a\,,\
\,b\,\ ,\ \,c\,\ }\atop{z_{3},z_{2}/z_{3},z_{1}/z_{2}}}\Big{)}\Big{]}$
$\displaystyle=\displaystyle\sum{}\Big{[}x^{{\tilde{a}}}y^{{\tilde{b}}}z^{{\tilde{c}}}\zeta_{*}\Big{(}{{\,a\,,\,b\,,\,c\,}\atop{z_{1},z_{2},z_{3}}}\Big{)}+x^{{\tilde{a}}}z^{{\tilde{b}}}y^{{\tilde{c}}}\zeta_{*}\Big{(}{{\,a\,,\,b\,,\,c\,}\atop{z_{1},z_{3},z_{2}}}\Big{)}+y^{{\tilde{a}}}x^{{\tilde{b}}}z^{{\tilde{c}}}\zeta_{*}\Big{(}{{\,a\,,\,b\,,\,c\,}\atop{z_{2},z_{1},z_{3}}}\Big{)}$
$\displaystyle+y^{{\tilde{a}}}z^{{\tilde{b}}}x^{{\tilde{c}}}\zeta_{*}\Big{(}{{\,a\,,\,b\,,\,c\,}\atop{z_{2},z_{3},z_{1}}}\Big{)}+z^{{\tilde{a}}}x^{{\tilde{b}}}y^{{\tilde{c}}}\zeta_{*}\Big{(}{{\,a\,,\,b\,,\,c\,}\atop{z_{3},z_{1},z_{2}}}\Big{)}+z^{{\tilde{a}}}y^{{\tilde{b}}}x^{{\tilde{c}}}\zeta_{*}\Big{(}{{\,a\,,\,b\,,\,c\,}\atop{z_{3},z_{2},z_{1}}}\Big{)}\Big{]}$
$\displaystyle+\displaystyle\sum{}^{\prime}\Bigg{[}\Big{(}\zeta_{*}\Big{(}{{\
\,k\,\ ,\,c\,}\atop{z_{1}z_{2},z_{3}}}\Big{)}+\zeta_{*}\Big{(}{{\,c\,,\ \,k\,\
}\atop{z_{3},z_{1}z_{2}}}\Big{)}\Big{)}\displaystyle\sum_{a+b=k}x^{{\tilde{a}}}y^{{\tilde{b}}}z^{{\tilde{c}}}+\Big{(}\zeta_{*}\Big{(}{{\
\,k\,\ ,\,c\,}\atop{z_{3}z_{1},z_{2}}}\Big{)}+\zeta_{*}\Big{(}{{\,c\,,\ \,k\,\
}\atop{z_{2},z_{3}z_{1}}}\Big{)}\Big{)}\displaystyle\sum_{a+b=k}x^{{\tilde{a}}}z^{{\tilde{b}}}y^{{\tilde{c}}}$
$\displaystyle+\Big{(}\zeta_{*}\Big{(}{{\ \,k\,\
,\,c\,}\atop{z_{2}z_{3},z_{1}}}\Big{)}+\zeta_{*}\Big{(}{{\,c\,,\ \,k\,\
}\atop{z_{1},z_{2}z_{3}}}\Big{)}\Big{)}\displaystyle\sum_{a+b=k}y^{{\tilde{a}}}z^{{\tilde{b}}}x^{{\tilde{c}}}\Bigg{]}+\zeta\Big{(}{{\
\ w\ \ }\atop{z_{1}z_{2}z_{3}}}\Big{)}\displaystyle\sum
x^{{\tilde{a}}}y^{{\tilde{b}}}z^{{\tilde{c}}}$ (6.17)

which leads to essentially four different cases using various combinations of
$z_{1},z_{2},z_{3}=\pm 1$.

######  Theorem 6.1.

Let $w\geq 4$, $u=w-2$ and $v=w-1$. Then we have

$\displaystyle\ 2\displaystyle\sum
3^{\tilde{a}}2^{\tilde{b}}\zeta_{\shuffle}(a,b,c)=\left(\displaystyle\frac{1}{3}\binom{v}{2}+u\right)\zeta(w)+2\zeta_{*}(1,1,u)-2\zeta(u,2)+w\zeta_{*}(1,v),$
$\displaystyle\ 2\displaystyle\sum
3^{\tilde{a}}2^{\tilde{b}}\Big{(}\zeta_{\shuffle}(a,\bar{b},c)+\zeta(\bar{a},b,\bar{c})+\zeta(\bar{a},\bar{b},\bar{c})\Big{)}=2\zeta(\bar{1},1,u)+2\zeta_{*}(1,\bar{1},u)+2\zeta(\bar{1},\bar{1},u)+\binom{v}{2}\zeta(w)$
$\displaystyle\hskip
28.45274pt+2\zeta(\bar{1},v)+2\zeta_{*}(1,\bar{v})+2\zeta(\bar{1},\bar{v})-2\zeta(u,2)-4\zeta(u,\bar{2})+u\Big{(}2\zeta(\bar{1},v)+2\zeta(\bar{w})+\zeta(w)+\zeta_{*}(1,v)\Big{)},$
$\displaystyle\ 2\displaystyle\sum
3^{\tilde{a}}2^{\tilde{b}}\Big{(}\zeta_{\shuffle}(a,\bar{b},\bar{c})+\zeta_{\shuffle}(a,b,\bar{c})+\zeta(\bar{a},\bar{b},c)\Big{)}=2\zeta_{*}(1,1,\bar{u})+2\zeta_{*}(1,\bar{1},\bar{u})+2\zeta(\bar{1},1,\bar{u})$
$\displaystyle\hskip
28.45274pt+2\zeta_{*}(1,\bar{v})+2\zeta(\bar{1},\bar{v})+2\zeta_{*}(1,v)+\zeta(\bar{2},\bar{u})-\zeta(2,\bar{u})-4\zeta(\bar{u},2)-2\zeta(\bar{u},\bar{2})$
$\displaystyle\hskip
28.45274pt+u\Big{(}2\zeta_{*}(1,\bar{v})+\zeta(\bar{1},\bar{v})+\zeta(w)+2\zeta(\bar{w})\Big{)}+\binom{v}{2}\zeta(\bar{w})+\zeta(w)-\zeta(\bar{w}),$
$\displaystyle\ 2\displaystyle\sum
3^{\tilde{a}}2^{\tilde{b}}\zeta(\bar{a},b,c)=2\zeta(\bar{1},\bar{1},\bar{u})+u\zeta(\bar{1},\bar{v})+2\zeta(\bar{1},v)$
$\displaystyle\hskip
28.45274pt+\zeta(2,\bar{u})-2\zeta(\bar{u},\bar{2})-\zeta(\bar{2},\bar{u})+\left(\displaystyle\frac{1}{3}\binom{v}{2}+1\right)\zeta(\bar{w})+(w-3)\zeta(w).$

###### Proof.

Taking $x=y=z=1$ in (6.17) we get the following four equations:

$\displaystyle 6\displaystyle\sum
3^{\tilde{a}}2^{\tilde{b}}\zeta_{\shuffle}(a,b,c)=\binom{v}{2}\zeta(w)+6\displaystyle\sum\zeta_{*}(a,b,c)+3\displaystyle\sum{}{\tilde{k}}\big{[}\zeta_{*}(k,c)+\zeta_{*}(c,k)\big{]},$
$\displaystyle 2\displaystyle\sum
3^{\tilde{a}}2^{\tilde{b}}\Big{(}\zeta_{\shuffle}(a,\bar{b},c)+\zeta_{\shuffle}(\bar{a},b,\bar{c})+\zeta_{\shuffle}(\bar{a},\bar{b},\bar{c})\Big{)}=2\displaystyle\sum\Big{(}\zeta(\bar{a},b,\bar{c})+\zeta(\bar{a},\bar{b},c)+\zeta(\bar{a},b,\bar{c})\Big{)}$
$\displaystyle\
\quad+\binom{v}{2}\zeta(w)+\displaystyle\sum{}{\tilde{k}}\Big{(}\big{[}2\zeta(\bar{k},\bar{c})+2\zeta(\bar{c},\bar{k})\big{]}+\big{[}\zeta_{*}(k,c)+\zeta_{*}(c,k)\big{]}\Big{)},$
$\displaystyle 2\displaystyle\sum
3^{\tilde{a}}2^{\tilde{b}}\Big{(}\zeta_{\shuffle}(a,\bar{b},\bar{c})+\zeta_{\shuffle}(a,b,\bar{c})+\zeta_{\shuffle}(\bar{a},\bar{b},c)\Big{)}=2\displaystyle\sum\Big{(}\zeta_{*}(a,b,\bar{c})+\zeta_{*}(a,\bar{b},c)+\zeta(\bar{a},b,c)\Big{)}$
$\displaystyle\
\quad+\binom{v}{2}\zeta(\bar{w})+\displaystyle\sum{}{\tilde{k}}\Big{(}2\big{[}\zeta_{*}(k,\bar{c})+\zeta_{*}(c,\bar{k})\big{]}+\big{[}\zeta(\bar{c},k)+\zeta(\bar{k},c)\big{]}+\zeta(\bar{k},c)-\zeta_{*}(k,\bar{c})\Big{)},$
$\displaystyle 6\displaystyle\sum
3^{\tilde{a}}2^{\tilde{b}}\zeta_{\shuffle}(\bar{a},b,c)=\binom{v}{2}\zeta(\bar{w})+6\displaystyle\sum\zeta(\bar{a},\bar{b},\bar{c})+3\displaystyle\sum{}{\tilde{k}}\Big{(}\big{[}\zeta(\bar{k},c)+\zeta(\bar{c},k)\big{]}+\zeta_{*}(k,\bar{c})-\zeta(\bar{k},c)\Big{)}.$

Here we have used the fact that $\displaystyle\sum_{k\geq
2}{\tilde{k}}(\cdots)=\displaystyle\sum_{k\geq 1}{\tilde{k}}(\cdots)$.
Exchanging indices $k$ and $c$ for the second term in each of the square
brackets and then setting $(k,c)\to(a,b)$, we find that

$\displaystyle 2\displaystyle\sum
3^{\tilde{a}}2^{\tilde{b}}\zeta_{\shuffle}(a,b,c)=\binom{v}{2}\displaystyle\frac{\zeta(w)}{3}+2\displaystyle\sum\zeta_{*}(a,b,c)+(w-2)\displaystyle\sum{}\zeta_{*}(a,b),$
$\displaystyle 2\displaystyle\sum
3^{\tilde{a}}2^{\tilde{b}}\Big{(}\zeta_{\shuffle}(a,\bar{b},c)+\zeta(\bar{a},b,\bar{c})+\zeta(\bar{a},\bar{b},\bar{c})\Big{)}=2\displaystyle\sum\Big{(}\zeta(\bar{a},b,\bar{c})+\zeta(\bar{a},\bar{b},c)+\zeta(a,\bar{b},\bar{c})\Big{)}$
$\displaystyle\
\qquad+\binom{v}{2}\zeta(w)+(w-2)\displaystyle\sum{}\Big{(}2\zeta(\bar{a},\bar{b})+\zeta_{*}(a,b)\Big{)},$
$\displaystyle 2\displaystyle\sum
3^{\tilde{a}}2^{\tilde{b}}\Big{(}\zeta_{\shuffle}(a,\bar{b},\bar{c})+\zeta_{\shuffle}(a,b,\bar{c})+\zeta(\bar{a},\bar{b},c)\Big{)}=2\displaystyle\sum\Big{(}\zeta_{*}(a,b,\bar{c})+\zeta_{*}(a,\bar{b},c)+\zeta(\bar{a},b,c)\Big{)}$
$\displaystyle\
\qquad+\binom{v}{2}\zeta(\bar{w})+(w-2)\displaystyle\sum{}\Big{(}2\zeta_{*}(a,\bar{b})+\zeta(\bar{a},b)\Big{)}+\displaystyle\sum{}^{\prime}{\tilde{a}}\Big{(}\zeta_{*}(\bar{a},b)-\zeta(a,\bar{b})\Big{)},$
$\displaystyle 2\displaystyle\sum
3^{\tilde{a}}2^{\tilde{b}}\zeta(\bar{a},b,c)=\binom{v}{2}\displaystyle\frac{\zeta(\bar{w})}{3}+2\displaystyle\sum\zeta(\bar{a},\bar{b},\bar{c})+(w-2)\displaystyle\sum{}\zeta_{*}(\bar{a},b)+\displaystyle\sum{}^{\prime}{\tilde{a}}\Big{(}\zeta_{*}(a,\bar{b})-\zeta(\bar{a},b)\Big{)}.$

The theorem follows easily from Theorems 4.2, 5.4 and 5.5. ∎

Finally, we are able to prove Kaneko–Tsumura’s conjecture on triple
$T$-values.

######  Corollary 6.2.

For all $w\geq 4$

$\displaystyle\sum_{a+b+c=w}2^{b}(3^{a-1}-1)T(a,b,c)=\displaystyle\frac{2}{3}(w-1)(w-2)T(w).$

###### Proof.

Set $u=w-2$ and $v=w-1$ as before. By Theorems 5.8 and 6.1 we see that

$\displaystyle\displaystyle\sum_{a+b+c=w}2^{b}(3^{a-1}-1)T(a,b,c)=\displaystyle\frac{2}{3}(w-1)(w-2)\Big{(}\zeta(w)-\zeta(\bar{w})\Big{)}-2\zeta(w)+2\zeta(\bar{w})$
$\displaystyle\hskip
28.45274pt+2\zeta(2)\zeta(u)-3\zeta(2)\zeta(\bar{u})-2\zeta(\bar{u},\bar{1},\bar{1})-2\zeta(\bar{1},\bar{1},\bar{u})-2\zeta(\bar{1},\bar{u},\bar{1})-2\zeta(\bar{1},v)-2\zeta(v,\bar{1})$
$\displaystyle\hskip
28.45274pt+2\zeta(\bar{1},1,\bar{u})+2\zeta(\bar{u},\bar{1},1)+2\zeta(\bar{1},\bar{u},1)+6\zeta(\bar{1},u,1)+6\zeta(u,\bar{1},1)+6\zeta(\bar{1},1,u)+6\zeta(\bar{1},v)$
$\displaystyle\hskip
28.45274pt-6\zeta(\bar{1},\bar{1},u)-6\zeta(\bar{1},u,\bar{1})-6\zeta(u,\bar{1},\bar{1})-6\zeta(\bar{v},\bar{1})-4\zeta(\bar{1},\bar{v})+2\zeta(v,1)+6\zeta(\bar{v},1)$
$\displaystyle\hskip
28.45274pt-2\zeta(u,\bar{2})-2\zeta(\bar{2},u)+4\zeta(\bar{u},2)+4\zeta(\bar{2},\bar{u})+4\zeta(\bar{u},\bar{2})+4\zeta(2,\bar{u})-6\zeta(2,u)-6\zeta(u,2).$

Each of the last four lines can be simplified further by stuffle relations so
that we get

$\displaystyle\displaystyle\sum_{a+b+c=w}2^{b}(3^{a-1}-1)T(a,b,c)=\displaystyle\frac{4}{3}\binom{v}{2}\Big{(}\zeta(w)-\zeta(\bar{w})\Big{)}-4\zeta(2)\zeta(u)+\zeta(2)\zeta(\bar{u})$
$\displaystyle\hskip
28.45274pt+4\zeta(\bar{2})\zeta(\bar{u})-2\zeta(\bar{2})\zeta(u)-2\zeta(\bar{u})\big{(}\zeta(\bar{1},\bar{1})-\zeta(\bar{1},1)\big{)}+6\zeta(u)\big{(}\zeta(\bar{1},1)-\zeta(\bar{1},\bar{1})\big{)}.$

So the corollary follows immediately from (5.16). ∎

Acknowledgments. Ce Xu is supported by the Scientific Research Foundation for
Scholars of Anhui Normal University.

## References

  * [1] H. Gangl, M. Kaneko, D. Zagier, Double zeta values and modular forms, in: Automorphic Forms and Zeta Functions, S. Böcherer, T. Ibukiyama, M. Kaneko and F. Sato eds., World Scientific Publ., Hackensack, NJ (2006), pp. 71–106.
  * [2] M. E. Hoffman, Multiple harmonic series, _Pacific J. Math._ 152 (1992), pp. 275–290.
  * [3] L. Guo and B. Xie, Weighted sum formula for multiple zeta values, _J. Number Theory_ 129 (11) (2009), pp. 2747–2765.
  * [4] M. Kaneko and H. Tsumura, On a variant of multiple zeta values of level two, to appear in _Tsukuba J. Math._ arXiv: 1903.03747v2.
  * [5] T. Machide, Extended double shuffle relations and generating function of triple zeta values of any fixed weight, _Kyushu J. Math._ , 67 (2013), pp. 281–307.
  * [6] T. Machide, A generating function to generalize the sum formula for quadruple zeta values. _Tokyo J. Math._ 42 (2019), pp. 329–355.
  * [7] Y. Ohno and W. Zudilin, Zeta stars, _Commun. Number Theory Phys._ 2 (2008), pp. 325–347.
  * [8] H. Yuan and J. Zhao, New families of weighted sum formulas for multiple zeta values, _Rocky Mountain J. Math._ 45 (2015), 2065-2096
  * [9] D. Zagier, Values of zeta functions and their applications, First European Congress of Mathematics, Volume II, Birkhauser, Boston, 120 (1994), pp. 497–512.
  * [10] J. Zhao, _Multiple zeta functions, multiple polylogarithms and their special values_ , Series on Number Theory and its Applications, Vol. 12, World Scientific Publishing Co. Pte. Ltd., Hackensack, NJ, 2016.


11institutetext: EPFL
11email<EMAIL_ADDRESS>
# Flexible Channel Dimensions for Differentiable Architecture Search

Ahmet Caner Yüzügüler  Nikolaos Dimitriadis  Pascal Frossard

###### Abstract

Finding optimal channel dimensions (i.e., the number of filters in DNN layers)
is essential to design DNNs that perform well under computational resource
constraints. Recent work in neural architecture search aims at automating the
optimization of the DNN model implementation. However, existing neural
architecture search methods for channel dimensions rely on fixed search
spaces, which prevents achieving an efficient and fully automated solution. In
this work, we propose a novel differentiable neural architecture search method
with an efficient dynamic channel allocation algorithm to enable a flexible
search space for channel dimensions. We show that the proposed framework is
able to find DNN architectures that are equivalent to previous methods in task
accuracy and inference latency for the CIFAR-10 dataset with an improvement of
$1.3-1.7\times$ in GPU-hours and $1.5-1.7\times$ in the memory requirements
during the architecture search stage. Moreover, the proposed frameworks do not
require a well-engineered search space a priori, which is an important step
towards fully automated design of DNN architectures.

###### Keywords:

Deep neural networks, neural architecture search, differentiable channel
masking

##  1 Introduction

Deep neural networks (DNN) have become ubiquitous in numerous application
domains such as computer vision and natural language processing. However, the
performance and computational requirements of DNN models are highly dependent
on their channel dimensions, namely a set of hyperparameters that define the
number of filters in DNN layers. Finding the right set of channel dimensions
for a DNN model plays a crucial role in achieving high performance under tight
computational resource constraints, but it poses a challenging task for
developers and engineers. Therefore, researchers have developed neural
architecture search frameworks that automatically look for optimal channel
dimensions of a DNN model[12, 14].

The early versions of neural architecture search frameworks have resorted to
reinforcement learning [10, 12, 17, 18], evolutionary algorithms [9, 11], and
Bayesian optimization [2] to search for optimal DNN architectures.
Unfortunately, these methods have time and space complexities that increase
combinatorially with the number of options that are defined in the search
space, requiring excessive amounts of computational resources. To reduce the
computational complexity of channel dimension search, Wan et al. [14] proposed
the differentiable channel masking method, which mimics various channel
dimensions by simply passing the feature maps through a set of binary masks.
While the differentiable channel masking method significantly reduces the
computational complexity of channel dimension search, it requires a search
space that is carefully designed and tuned prior to the search phase, which
hinders its practicality and usability in realistic scenarios.

In this work, we propose FlexCHarts, which utilizes a flexible search space
that does not need to be defined a priori. We reformulate the problem of
differentiable channel dimension search such that FlexCHarts does not only
search for the optimal channel dimensions but also modifies the boundaries of
the search space on-the-fly to add further flexibility and reach optimal
channel dimensions. The proposed method relaxes the requirement of an a priori
well-designed search space for channel dimension optimization and enables
finding the optimal channel dimensions only with loosely-defined initial
conditions of a search space, improving the practicality and usability of
neural architecture search.

The rest of this paper is organized as follows: We first discuss the related
work on neural architecture search, then we give a background information on
the differentiable channel masking method, which forms the basis of our work.
We then elaborate on the proposed FlexCHarts method as well as our novel
dynamic channel allocation mechanism. Finally, we give the details of our
experiments and discuss the results.

##  2 Related Work

Building DNN architectures that achieve high performance at low minimal
hardware performance is a challenging task for developers and engineers. Thus,
researchers have put considerable effort into developing effective neural
architecture search methods to help automate the design of DNNs. Early work on
neural architecture search adopted frameworks like reinforcement learning [10,
12, 17, 18], evolutionary algorithms [9, 11], and Bayesian optimization [2].
Because these methods operate on a discrete search space and need to perform
many trials while searching for an optimal architecture in an exponentially-
increasing hyperparameter space, they require thousands of GPU-hours to find
optimal DNN architectures, which greatly limits their applicability.

To mitigate the prohibitive cost of architecture search, techniques such as
weight-sharing [10] and one-shot search [1] have been proposed. While these
techniques reduce the cost of each trial by allowing to reuse trained
parameters, they still require many trials to find the optimal DNN
architectures. To further improve the efficiency of neural architecture
search, Liu et al. [8] proposed a differentiable neural architecture search,
which employs a gradient descent optimizer to efficiently search for optimal
DNN architectures. Furthermore, Wu et al. [15] proposed a hardware-aware
differentiable neural architecture search, which enables to co-optimize both
task accuracy and hardware metrics (e.g., latency). While these improvements
drastically improved the computational efficiency of neural architecture
search and the performance of the resulting DNN architectures, they did not
address the challenges of searching for optimal channel dimensions, which is
critical to the performance of DNNs.

To that end, Wan et al. [14] proposed differentiable channel masking method,
which significantly improves the computational efficiency of neural
architecture search with a search space that includes large numbers of options
for channel dimensions. Unfortunately, the effectiveness of the proposed
method is highly sensitive to the design of search space due to its
requirements of a fixed search space. Figure 1 shows the search space of an
existing differentiable neural architecture search framework for channel
dimensions [14]. We observe that the channel ranges in this search space are
narrowed down to a few options out of a wide range using heuristics, while
allowing only a limited degree of freedom for the optimizer. Unfortunately,
determining a priori the channel range of each layer is nontrivial as it
requires expert knowledge on DNN architectures. Also, the resulting search
space is specific to the problem that it is designed for and it may not be
transferable to the settings with different objectives and constraints.
Moreover, when we investigate the channel dimensions found within this search
space, as shown by red circles in Figure 1, we observe that many channel
dimensions are located at the boundary of the search space, which indicates
that the optimal channel dimensions may lie outside of the engineered search
space, resulting in a DNN architecture that does not correspond to the optimal
solution.

Few prior works have addressed the issues inherent to the fixed search spaces.
Liu et al. [7] proposed the progressive neural architecture search, which
gradually increases the complexity of the search space during optimization.
Similarly, Ci et al. [3] proposed the neural search space evolution technique,
which enables adding new operations to the search space as the architecture
search progresses. Unfortunately, these methods are not applicable to the
neural architecture search with channel dimension search. As a result,
efficient search of channel dimensions without the restrictions of a fixed
search space remains an open research question. Our work addresses these
challenges and introduces a flexible search space for channel dimensions by
proposing a novel differentiable neural architecture search to optimize for
channel dimensions efficiently.

Figure 1: Prior work’s search space for channel dimensions [14]. Rows
correspond to the channel range (between 0 and 200) of the layers in
FBNetv2-F4 [14]. Ticks denote the options for channel dimensions and red
circles represent the channel dimensions found.

##  3 Differentiable Channel Masking

The input and output channel dimensions of a convolutional kernel have a
significant impact on its performance and computational complexity. Increasing
the number of channels typically leads to improved performance due to higher
number of parameters at the expense of computational complexity, and vice
versa. Thus, the channel dimensions must be carefully tuned while designing
DNNs to obtain the required performance while attaining low computational
complexity. With the number of kernels in modern DNNs exceeding hundreds, the
task of finding the optimal channel dimensions for each kernel in a DNN is
overwhelmingly time-consuming and costly. Therefore, researchers have
developed automatic search methods for channel dimensions.

Figure 2: Illustration of a convolutional operation followed by channel
masking to simulate various output channel dimensions.

In standard search methods, each candidate channel dimension requires an
additional convolutional kernel, which increases the computational cost and
memory requirements of the search linearly with the number of channel options.
The channel masking method, on the other hand, simulates various candidate
channel dimensions on a single overparameterized kernel, incurring only
minimal computational overhead. Because of its computational efficiency, we
use the channel masking method as a basis for the proposed framework. In this
section, we explain how the channel masking method works in more detail.

Let us consider a convolutional kernel with a number of output channels $f$,
which is to be chosen from a set of possible channel dimensions $S$, denoted
as below:

$S=\\{f_{k}\mid f_{k}\in\mathbb{Z},k\in\mathbb{Z},1\leq k\leq K\\}$ (1)

where $f_{k}$ represents the possible channel dimensions and $K$ is the size
of the set of possible channel dimensions. To evaluate the likelihood of
$f_{k}$ being the optimal channel dimension for the given task, the channel
masking method defines a set of trainable parameters
$\\{\alpha_{k}\mid\alpha_{k}\in\mathbb{R},1\leq k\leq K\\}$, where higher
values of $\alpha_{k}$ indicate that $f_{k}$ is more likely to be the optimal
channel dimension.

Evaluating the impact of each $f_{k}$ on a DNN’s overall performance and
computational cost requires instantiating and training a separate kernel for
each option, which increases the memory footprint and computational complexity
of the search stage linearly with the number of options. To efficiently search
for optimal channel dimensions with large numbers of options, the channel
masking method instantiates and trains a single kernel and simulates various
channel dimensions masking out a fraction of the channels in this kernel.

Figure 2 illustrates how the channel masking method simulates various channel
dimensions using a single kernel. Let $\mathbf{X}$, $\mathbf{W}$ and
$\mathbf{Y}$ be the input activation, weight, and output activation of a
convolutional kernel. $\mathbf{X}$ and $\mathbf{Y}$ have channel dimensions of
$C$ and $F$,respectively. The channel masking method exploits the fact that
any convolutional kernel with an output channel dimension $f_{k}$ that is
smaller than $F$ can be obtained simply by selecting $f_{k}$ channels from
$\mathbf{Y}$ and masking out the rest. For this purpose, the channel masking
method instantiates a set of masks $\displaystyle\\{m_{k}\mid
k\in\mathbb{Z},1\leq k\leq K\\}$, where
$m_{k}=(1)_{i=1}^{f_{k}}\cup(0)_{i=f_{k}+1}^{F}$. In other words, the first
$f_{k}$ elements of $m_{k}$ are set to one whereas the remaining elements are
zero; thus, the $m_{k}$ allows selecting the first $f_{k}$ channels of $Y$ and
zeroes out the channels that are greater than $f_{k}$.

During the search phase, the DMaskingNAS method multiplies the output
activation $\mathbf{Y}$ by the masks $m_{k}$ and calculate $\mathbf{\hat{Y}}$,
which is the weighted sum of the output of simulated layers with various
channel dimension using the following formula:

$\hat{Y}=\sum_{k=1}^{K}{g_{\tau}(\alpha_{k})m_{k}Y}$ (2)

where $g_{\tau}$ is a Gumbel softmax function with the temperature constant
$\tau$ that maps the $\alpha$ values between 0 and 1 [14]. The expression in
Equation 2 can be simplified by taking $\mathbf{Y}$ out of the summation,
reducing the overhead of masking to only a weighted sum of low-dimensional
masks. Therefore, the channel masking method simulates multiple channel
dimensions with negligible computational overhead.

During the search phase, the channel masking method updates the $\alpha$
values with a gradient descent optimizer by minimizing the following loss
function:

$\min_{\alpha}\min_{W}\mathcal{L}_{acc}(\mathcal{N}_{\alpha,W}(x),y)+\lambda\mathcal{L}_{latency}(\mathcal{N}_{\alpha,W})$
(3)

where $\mathcal{N}_{\alpha,W}$ represents the network, and $x$ and $y$
represent training samples and ground-truth, and $\mathcal{L}_{acc}$ and
$\mathcal{L}_{latency}$ represent the loss functions for classification
accuracy and latency, respectively. The coefficient $\lambda$ controls the
trade-off between accuracy and latency. As suggested by the prior work [8],
the loss function is minimized by calculating the gradients using a first-
order approximation in order to reduce the computational cost of the search.
At the end of the search phase, the final channel dimensions are selected as
the channel dimensions that correspond to the maximum $\alpha_{i}$, where
$\displaystyle i=\underset{k}{\arg\max}\alpha_{k}$.

The channel masking method permits to search for channel dimensions among
various options with minimal computational overhead. However, the standard
channel masking methods proposed in prior work can search only within a fixed
range of channel dimensions [14], which hinders its effectiveness and
practicality. Thus, in the next section, we introduce FlexCHarts, which is a
differentiable channel masking method that searches for channel dimensions in
a flexible range.

##  4 FlexCHarts

To enable searching for channel dimensions in a flexible range, we first
introduce the flexible channel masking method, which redefines the $\alpha$
variables so as to permit to change the range of channel dimensions as the
search progresses. Then, we elaborate on our dynamic channel allocation
mechanism, which modifies the kernels to accommodate for the changes in the
channel dimension range during the search.

(a)

(b)

Figure 3: An example of $\alpha$ values in vanilla channel masking method
versus the proposed FlexCHarts methods between the first and last epoch of a
search.

###  4.1 Flexible channel masking

FlexCHarts redefines the $\alpha$ variables in such a way that the channel
dimension range can be changed on-the-fly during the search while still
benefiting from the computational efficiency of the vanilla channel masking
method. Instead of defining each $\alpha_{k}$ as an independent variables
[14], we define $\alpha_{k}$ as a smooth function of the channel dimension
that it corresponds to, where it has the highest value at the center and close
to zero at the edges of the range. While various smooth functions would be
equally applicable, without loss of generality, we opt for the normal
distribution to define $\alpha_{k}$ in this work due to its simplicity:

$\alpha_{k}=\exp(-\frac{1}{2}(\frac{f_{k}-\mu}{\sigma})^{2})$ (4)

where $\mu$ and $\sigma$ are the mean and standard deviation of the
exponential function, respectively. The formulation given above differs from
the vanilla channel masking method in the sense that $\alpha$ is no longer a
trainable variable. Instead, the proposed flexible channel masking method
defines $\mu$ as the trainable variable and derives the $\alpha$ values from
the exponential function given in Equation 4. This formulation smoothly adapts
the $\alpha$ values to different channel ranges based on the information on
the gradients of $\mu$.

Figure 3 shows an example of how $\alpha$ values change in vanilla channel
masking and in FlexCHarts, between the first and last epochs of a dimension
search process. In vanilla channel masking method, as shown in
LABEL:fig:dmaskalphas, $\alpha$ values are initialized randomly. During the
search, the $\alpha$ values are updated independently and eventually in the
last epoch, the $\alpha$ value that corresponds to the optimal channel
dimension becomes significantly higher than the others, setting the channel
dimension for the final architecture. In contrast, the $\alpha$ values in
FlexCHarts, as shown in LABEL:fig:flexchartsalphas, are taken from the
exponential function given in Equation 4 (shown with red dashed line in the
figure). The mean value of the exponential function is updated at every step
of the search phase, gradually shifting the $\alpha$ values towards the
optimal channel dimension.

The proposed reformulation of $\alpha$ variables in Equation 4 has two main
advantages. First, the given expression is already differentiable; thus, it
eliminates the need for tuning additional optimization hyperparameters such as
temperature and noise in Gumbel softmax to make the optimization amenable to
solution with gradient descent as in the prior work [14]. Second, thanks to
$\alpha_{k}$ values that are close to zero at the edges of the range, shifting
the range to larger or smaller values gently introduces the new parameters to
the kernels, preventing abrupt changes in the loss value that would otherwise
be detrimental to the search process. In short, the proposed formulation of
$\alpha$ variables is more efficient, flexible and easier to tune than the
vanilla channel masking method. Thus, it provides a better solution when
searching for optimal channel dimensions in DNN architectures.

###  4.2 Dynamic channel allocation

In vanilla channel masking methods, the kernels do not require any dimension
changes during the search phase as the channel range remains constant.
However, with the reformulation of the $\alpha$ variables in FlexCHarts that
permits to change the channel dimension range as the search progresses, the
dimensions of the kernels must also be efficiently adapted during the search
phase. We now elaborate on how we modify the kernels for dynamic channel
dimension range.

Kernels in channel masking methods must have a number of channels equal to or
greater than the maximum of the channel dimension range. Therefore, when the
range of the channel dimensions shifts to higher values, we need to
instantiate a larger kernel. Likewise, when the range of channel dimensions
shifts to smaller values, a part of the kernel becomes redundant due to
multiplication with an $\alpha_{k}$ value that is close to zero and is
therefore no longer needed. As such, we can reduce the memory footprint and
improve the computational efficiency of the search by switching to a smaller
kernel. To adjust the dimensions of the kernels based on the changes in the
channel dimension range, we introduce the dynamic channel allocation
algorithm.

There are two critical design considerations for the dynamic channel
allocation algorithm. First, changing the kernel dimensions should have
minimal impact on the on-going search to prevent loss of progress. Second,
allocating new channels should incur only an insignificant computational
overhead. To achieve the first condition, when the proposed dynamic channel
allocation algorithm changes the dimensions of a kernel, it transfers the
trained weights of the old kernel to the new one where applicable, which
preserves the progress made in earlier training steps. For the latter
objective, the proposed algorithm shall not react to changes in $\alpha_{k}$
values in every step. Instead, it waits until the end of an epoch to perform
the changes to the kernels to reduce the computational overhead. Altogether,
the proposed dynamic channel allocation algorithm enables changing the kernel
dimensions with minimal impact on the search process and negligible
computational overhead.

Supernet _$\mathcal{N}_{\alpha,W}$ with parameters $W$ and $\alpha$;_

Optimizer _weight optimizer $G_{w}$, arch optimizer $G_{a}$;_

Dataset _training: $D_{t}$, search: $D_{s}$;_

Initialize;

for  _all epoch $e$_ do

for  _all steps $s$_ do

Read training batch $b_{t}\leftarrow D_{t}$;

Backpropagate $\mathcal{N}_{\alpha,W}$ with $b_{t}$;

Update $W\leftarrow G_{w}$;

Read search batch $b_{s}\leftarrow D_{s}$;

Backpropagate $\mathcal{N}_{\alpha,W}$ with $b_{s}$;

Update $\alpha\leftarrow G_{a}$;

end for

Update channel dimensions of $\mathcal{N}_{\alpha,W}$;

end for

Algorithm 1 FlexCHarts algorithm for channel dimension search with dynamic
channel allocation.

In more details, the dynamic channel allocation proceeds as described in
Algorithm 1. The algorithm takes a Supernet $\mathcal{N}_{\alpha,W}$ as input,
which consists of kernels with weights $\mathbf{W}$ and parameters $\alpha$,
as well as the gradient descent optimizers $G_{w}$ and $G_{a}$ to update the
weights and channel parameters, respectively. It also takes three datasets as
inputs: equally sized $D_{t}$ and $D_{s}$ to train the weights and channel
parameters, respectively. In each step of the algorithm, it reads a batch of
samples from $D_{t}$, performs a backpropagation on the supernet, and updates
the weights. Then, it repeats the same steps for channel parameters by reading
a batch from $D_{s}$, performing a backpropagation, and updating them. When
the same operations are performed for all samples in datasets $D_{t}$ and
$D_{s}$, it updates the channel dimensions of the supernet based on the
changes made to $\alpha$ values. We repeat the same operations for a
predefined number of epochs.

In short, the proposed FlexCHarts algorithm permits to search for optimal
channel dimensions in a flexible channel dimension range while automatically
managing the changes in the supernet with minimal computational overhead.

##  5 Experiments

We now show the effectiveness of the proposed method through a number of
experiments. In this section, we first give details about the search space,
datasets, and hyperparameters that we use in the experiments, then we compare
the proposed FlexCHarts method against the baseline methods through extensive
experiments and discuss the results.

###  5.1 Experimental setup

We perform experiments on a widely used image classification dataset, namely
CIFAR10 [6] with a preprocessing pipeline for training that consists of a
random crop of the input image with a size of $32$ and padding of $4$, random
horizontal flip, normalization, and a cut-out with a length of 16 [4]. For the
validation and test phases, we use only a normalization layer in the
preprocessing pipeline. We use a batch size of $96$ for both the search and
training phases. We perform all the experiments on an NVIDIA Tesla V100-SXM2
GPU with a 32GB memory.

For the search phase, we randomly split the training data set into two equally
sized subsets to train the weights and channel parameters separately. We use a
stochastic gradient descent (SGD) optimizer with a momentum coefficient of
$0.9$, a weight decay of $3e-4$, a gradient clip of $5$ to train the weights.
We initialize the learning rate of the SGD optimizer to $0.025$ and anneal it
every step with a cosine annealing scheduler down to $0$ at the end of the
last step. We use an Adam optimizer [5] with a learning rate of $0.1$, running
average coefficients of $0.5$ and $0.999$, and a weight decay of $0$ to train
the channel parameters. We use a dropout with a probability that starts at $0$
and linearly increases to $0.2$ until the end of the last step. The search
phase takes 50 epochs to complete. After the search phase is completed, we
train the DNN architecture with the discovered channel dimensions from scratch
for 100 epochs to obtain its final accuracy. We also use an SGD optimizer in
the training phase with the same hyperparameters as for the search phase.

As widely adopted by the community [8, 15, 14], we use a fixed stem and head
stages at the beginning and end of our DNN architectures while we are
searching for the optimal channel dimensions for the intermediate stages. The
stem and head stages consist of convolutional blocks with kernel sizes of
$3\times 3$ and $1\times 1$ and channel dimensions of 108 and 256,
respectively. The intermediate block consists of 20 stages, each with a
microarchitecture identical to the DARTS architecture [8]. Each stage may have
different channel dimensions as a result of the search phase while all the
layers in a stage share the same channel dimension. For the training phase, we
also use an auxiliary head that consists of three fully-connected layers with
an auxiliary weight of 0.4 as proposed by the prior work [8].

###  5.2 Performance of the differentiable channel search

To evaluate the effectiveness of the proposed FlexCHarts method, we first
compare it against the DMaskingNAS method, which has a fixed search space.
Because the effectiveness and efficiency of DMaskingNAS method is highly
sensitive to their predefined range of channel dimensions, we create two
baselines that represent DMaskingNAS methods with small and large range of
channel dimensions, which we simply refer to as DMask-small and DMask-large.
The details of these search spaces are deferred to the Appendix.

To mimic target inference platforms with different resource constraints, we
perform our experiments under low- and high-resource scenarios. For the low-
resource scenario, we prioritize the computational requirements of the
searched DNN architectures and target an inference latency under 0.3
millisecond per sample. In contrast, for the high-resource scenario, we
prioritize the accuracy and aim for DNN architectures that achieve a top-1
test accuracy higher than 96% on CIFAR10. We adjust the latency coefficients
(i.e., $\lambda$ in Equation 3) to fulfill these accuracy and computational
complexity requirements.

Table 1: Results of the DMaskingNAS and FlexCHarts methods targeting low and high-resource scenarios. Check and cross marks indicate whether the requirement is satisfied or not. | Evaluation | Search
---|---|---
Scenario | Search | Top-1 acc. | Latency | FLOPS | Search time | Search
algorithm | (%) | (ms) | $\times 10^{9}$ | (GPU-hours) | memory (GB)
Low-resource ($<$0.3ms latency) | DMask-small | 95.62 | 0.366 ( ) | 0.258 | 2.63 | 10.3
DMask-large | 93.40 | 0.288 ( ) | 0.095 | 5.75 | 28.3
FlexCHarts | 94.10 | 0.287 ( ) | 0.093 | 3.36 | 16.6
High-resource ($>$96% accuracy) | DMask-small | 95.67 ( ) | 0.452 | 0.433 | 2.74 | 12.0
DMask-large | 96.06 ( ) | 0.606 | 0.736 | 5.78 | 28.3
FlexCHarts | 96.04 ( ) | 0.654 | 0.773 | 4.37 | 19.1

Table 1 summarizes the results of our experiments with the FlexCHarts, DMask-
small, and DMask-large methods under the low- and high-resource scenarios. For
the low-resource scenario, while the DMask-small method has the highest top-1
accuracy and lowest search time and memory, it fails to find a DNN
architecture that achieves the target of 0.3 millisecond inference latency per
batch due to its limited channel range. The DMask-large and FlexCHarts methods
succeed to find DNN architectures that achieve the given inference latency
target. However, the DMask-large method requires 5.75 GPU-hours and 28.3 GB of
memory to find the DNN architecture as it needs to train a larger supernet
whereas the FlexCHarts method finds an equivalent architecture only in 3.36
GPU-hours and using 16.6 GB of memory.

The FlexCHarts method also outperforms the DMaskingNAS method in the high-
resource scenario. Due to its limited range of channel dimensions, DMask-small
fails to find a DNN architecture that is large enough to achieve a top-1
accuracy greater than 96%. In contrast, both DMask-large and FlexCHarts
methods are able to find DNN architectures with the target accuracy
requirements. Similarly to the low-resource scenario, the DMask-large method
requires 5.78 GPU-hours and 28.3 GB of memory to achieve this goal whereas the
FlexCHarts method finds an equivalent DNN architecture in 4.37 GPU-hours and
using 19.1 GB of memory.

These experiments clearly show that the proposed FlexCHarts method can find
the channel dimensions that meet the requirements of varying resource
constraints and optimization goals without the restrictions of a fixed search
space. Moreover, it does not require to train a redundantly large supernet,
thus it searches for the channel dimensions efficiently with lower GPU-hours
and memory requirements than the DMaskingNAS methods.

###  5.3 Comparison with other dimension adaptation methods

We now proceed with the experiments that compare the proposed FlexCHarts
method against other channel dimension scaling methods. For this purpose, we
use the following baselines: WideResnet architectures [16], which proposes to
scale DNN architectures by simply multiplying its channel dimensions by a
predetermined coefficient, and EfficientNet architectures [13], which proposes
compound scaling, in which the depth and width are scaled uniformly by a
coefficient.

In these experiments, we use WideResnet with a depth of 22 and width factors
of $1$, $2$, and $4$ and EfficientNet with its four largest variants, namely
B6, B7, B8, and L2, which requires a similar range of FLOPS with the DNN
architectures found by FlexCHarts. To eliminate the performance discrepancies
caused by differences in their implementations, we compare their computational
complexity in terms of FLOPS. For different search runs with FlexCHarts, we
used the latency coefficients ($\lambda$) varying between $1e-3$ and $1e-1$.
We train all DNN architectures using the same training parameters given in
subsection 5.1.

Figure 4: Accuracy on CIFAR10 versus computational complexity in terms of
FLOPS for various channel dimensions found by FlexCHarts as well as the
baseline WideResnet and EfficientNet architectures.

Figure 4 shows the top-1 accuracy and FLOPs requirements of the DNN
architectures found by the proposed FlexCHarts method as well as the baseline
methods. Because FlexCHarts can automatically search the optimal channel
dimensions within a flexible range, the architectures found by FlexCHarts
achieve between 0.5-1% and 2-4% better accuracy than WideResnet and
EfficientNet for similar FLOPS requirements, respectively. Moreover, the
effectiveness of the channel scaling methods used in WideResnet and
EfficientNet is highly sensitive to the design of initial DNN architectures,
which requires heuristics and manual development efforts. In contrast, the
proposed FlexCHarts method finds channel dimensions that achieve better
accuracy and/or FLOPS and it does so in a completely automatic fashion.

##  6 Conclusion

In this work, we addressed the limitations of neural architecture search
methods that have fixed search space for channel dimensions. We redefined the
architectural variables in the differentiable channel masking method to enable
searching for channel dimensions in a flexible and smooth way. We also
introduced a new dynamic channel allocation mechanism that allows changing the
kernel dimensions efficiently during the search in order to dynamically adapt
to the target channel dimensions.

Through extensive experiments, we demonstrated that the proposed FlexCHarts
framework finds optimal channel dimensions for DNN architectures under various
resource constraints and performance objectives without the limitations of the
existing methods that use fixed search spaces. Moreover, it searches the
optimal channel dimensions faster and with reduced memory requirements than
the existing methods.

## References

  * [1] Bender, G., Kindermans, P., Zoph, B., Vasudevan, V., Le, Q.V.: Understanding and simplifying one-shot architecture search. In: Proceedings of the 35th International Conference on Machine Learning, ICML. vol. 80, pp. 549–558 (2018)
  * [2] Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems. pp. 2546–2554 (2011)
  * [3] Ci, Y., Lin, C., Sun, M., Chen, B., Zhang, H., Ouyang, W.: Evolving search space for neural architecture search. In: 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021. pp. 6639–6649 (2021)
  * [4] DeVries, T., Taylor, G.W.: Improved regularization of convolutional neural networks with cutout. arXiv (2017)
  * [5] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: 3rd International Conference on Learning Representations, ICLR (2015)
  * [6] Krizhevsky, A.: Learning multiple layers of features from tiny images pp. 32–33 (2009)
  * [7] Liu, C., Zoph, B., Neumann, M., Shlens, J., Hua, W., Li, L., Fei-Fei, L., Yuille, A.L., Huang, J., Murphy, K.: Progressive neural architecture search. In: Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part I. vol. 11205, pp. 19–35 (2018)
  * [8] Liu, H., Simonyan, K., Yang, Y.: DARTS: differentiable architecture search. In: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 (2019)
  * [9] Marchisio, A., Massa, A., Mrazek, V., Bussolino, B., Martina, M., Shafique, M.: Nascaps: A framework for neural architecture search to optimize the accuracy and hardware efficiency of convolutional capsule networks. In: IEEE/ACM International Conference On Computer Aided Design, ICCAD. pp. 114:1–114:9 (2020)
  * [10] Pham, H., Guan, M.Y., Zoph, B., Le, Q.V., Dean, J.: Efficient neural architecture search via parameter sharing. In: Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018. vol. 80, pp. 4092–4101 (2018)
  * [11] Real, E., Aggarwal, A., Huang, Y., Le, Q.V.: Regularized evolution for image classifier architecture search. In: The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019. pp. 4780–4789 (2019)
  * [12] Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., Le, Q.V.: MnasNet: Platform-aware neural architecture search for mobile. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. pp. 2820–2828. Computer Vision Foundation / IEEE (2019)
  * [13] Tan, M., Le, Q.V.: EfficientNet: Rethinking model scaling for convolutional neural networks. In: Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA. vol. 97, pp. 6105–6114 (2019)
  * [14] Wan, A., Dai, X., Zhang, P., He, Z., Tian, Y., Xie, S., Wu, B., Yu, M., Xu, T., Chen, K., Vajda, P., Gonzalez, J.E.: Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. pp. 12962–12971 (2020)
  * [15] Wu, B., Dai, X., Zhang, P., Wang, Y., Sun, F., Wu, Y., Tian, Y., Vajda, P., Jia, Y., Keutzer, K.: Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019\. pp. 10734–10742 (2019)
  * [16] Zagoruyko, S., Komodakis, N.: Wide residual networks. In: Proceedings of the British Machine Vision Conference 2016, BMVC (2016)
  * [17] Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings (2017)
  * [18] Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable architectures for scalable image recognition. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018. pp. 8697–8710 (2018)

##  Appendix 0.A Appendix

Table 2: Channel ranges of DMask-small and Dmask-large baselines for the experiments in subsection 5.2. | DMask-small | DMask-large | DMask-systolic
---|---|---|---
Cell id | start | end | step | start | end | step | start | end | step
0 | 24 | 32 | 8 | 16 | 160 | 16 | 16 | 200 | 8
1 | 24 | 32 | 8 | 16 | 160 | 16 | 16 | 200 | 8
2 | 24 | 32 | 8 | 16 | 160 | 16 | 16 | 200 | 8
3 | 24 | 32 | 8 | 16 | 160 | 16 | 16 | 200 | 8
4 | 24 | 32 | 8 | 16 | 160 | 16 | 16 | 200 | 8
5 | 24 | 32 | 8 | 16 | 160 | 16 | 16 | 200 | 8
6 | 48 | 64 | 8 | 16 | 160 | 16 | 16 | 200 | 8
7 | 48 | 64 | 8 | 16 | 160 | 16 | 16 | 200 | 8
8 | 48 | 64 | 8 | 16 | 160 | 16 | 16 | 200 | 8
9 | 48 | 64 | 8 | 16 | 160 | 16 | 16 | 200 | 8
10 | 48 | 64 | 8 | 16 | 160 | 16 | 16 | 200 | 8
11 | 48 | 64 | 8 | 16 | 160 | 16 | 16 | 200 | 8
12 | 48 | 64 | 8 | 16 | 160 | 16 | 16 | 200 | 8
13 | 96 | 160 | 16 | 16 | 160 | 16 | 16 | 200 | 8
14 | 96 | 160 | 16 | 16 | 160 | 16 | 16 | 200 | 8
15 | 96 | 160 | 16 | 16 | 160 | 16 | 16 | 200 | 8
16 | 96 | 160 | 16 | 16 | 160 | 16 | 16 | 200 | 8
17 | 96 | 160 | 16 | 16 | 160 | 16 | 16 | 200 | 8
18 | 96 | 160 | 16 | 16 | 160 | 16 | 16 | 200 | 8
19 | 96 | 160 | 16 | 16 | 160 | 16 | 16 | 200 | 8