# Two-level Group Convolution

Youngkyu Lee<EMAIL_ADDRESS>Jongho Park<EMAIL_ADDRESS>Chang-Ock
Lee<EMAIL_ADDRESS>Department of Mathematical Sciences, KAIST, Daejeon
34141, Korea  Natural Science Research Institute, KAIST, Daejeon 34141, Korea

###### Abstract

Group convolution has been widely used in order to reduce the computation time
of convolution, which takes most of the training time of convolutional neural
networks. However, it is well known that a large number of groups
significantly reduce the performance of group convolution. In this paper, we
propose a new convolution methodology called “two-level” group convolution
that is robust with respect to the increase of the number of groups and
suitable for multi-GPU parallel computation. We first observe that the group
convolution can be interpreted as a one-level block Jacobi approximation of
the standard convolution, which is a popular notion in the field of numerical
analysis. In numerical analysis, there have been numerous studies on the two-
level method that introduces an intergroup structure that resolves the
performance degradation issue without disturbing parallel computation.
Motivated by these, we introduce a coarse-level structure which promotes
intergroup communication without being a bottleneck in the group convolution.
We show that all the additional work induced by the coarse-level structure can
be efficiently processed in a distributed memory system. Numerical results
that verify the robustness of the proposed method with respect to the number
of groups are presented. Moreover, we compare the proposed method to various
approaches for group convolution in order to highlight the superiority of the
proposed method in terms of execution time, memory efficiency, and
performance.

###### keywords:

Group convolution, Parallel computation, Block Jacobi approximation, Two-level
method

††journal: Neural Networks

##  1 Introduction

Modern convolutional neural networks (CNNs) have become deeper and larger to
improve their performance [1, 2, 3, 4, 5]. Almost all successful CNNs consist
of hundreds of layers and thousands of channels [2, 6, 7], which require
significant computational cost to handle them. Increased computational cost
has led to increased training time, and numerous research has been conducted
to shorten the training time by modifying the convolutions in CNNs; see, e.g.,
[8, 9, 10, 11].

Group convolution is a simple and basic modification of convolution. It was
first proposed in AlexNet [3] for distributed computing of convolutions in CNN
over multiple GPUs. It was shown in [7, 8, 9, 12] that the group convolution
is very effective on reducing both the number of parameters and the training
time of CNN. However, it has a drawback that the accuracy of CNN equipped with
group convolutions severely decreases when the number of groups is large. This
is because groups in the group convolution do not communicate with each other,
which greatly affects the representation capacity of the CNN.

Recently, several group convolution techniques that add intergroup
communication have been considered to prevent the decrease of performance. A
channel shuffling technique [11] adds a permutation step among the output
channels of the group convolution in order to make data exchange among groups.
A fully learnable group structure [10] optimizes the group arrangement of the
input and output channels of the group convolution. Depthwise separable
convolutions proposed in [12] improve computational performance and reduce the
number of parameters by distributing the channelwise and spatialwise
computations of the standard convolution into two convolutions. Although these
works successfully resolved the performance degradation issue explained above,
the parallel structure of the group convolution is disturbed by the additional
structures, hence, parallel implementation of these methods in a distributed
memory system becomes difficult. Therefore, it is important to develop a novel
group convolution that resolves the performance degradation issue while
preserving the parallel structure. This paper is devoted to such a novel group
convolution called two-level group convolution.

In view of numerical analysis, the group convolution can be regarded as a
block Jacobi approximation of the standard convolution. Block Jacobi method in
numerical analysis is one of the basic parallel algorithms for scientific
computing (see, e.g., [13, 14]) and shares the same drawback as the group
convolution. That is, as the number of groups increases, the quality of block
Jacobi approximation decreases. However, there is a popular remedy called two-
level correction for the disadvantage of the block Jacobi method [15, 16]. In
two-level methods, a coarse-level grid is added for the purpose of intergroup
communication. Usually, it covers the whole domain of the target problem but
with a less number of degrees of freedom so that it provides data exchange
among the whole domain with marginal computational cost. Due to its small
computational cost, it does not harm the parallel structure of the algorithm.
Such a two-level idea has been successfully applied to various problems in
numerical analysis; see [15, 16].

Motivated by such two-level methods, we present an additional structure to the
group convolution that grants communication among groups with minimal
computational cost and without breaking the parallel structure. We call the
group convolution plus such an additional structure two-level group
convolution. In the two-level group convolution, a single channel
representating each group is generated through groupwise computation, and then
communicates so that all groups share the information of all representatives.
After that, the information of such representative channels are added to the
output of the group convolution in an appropriate manner. Compared to existing
approaches, the proposed two-level group convolution maintains the parallel
structure in the sense that no communication of parameters is required; only a
small number of data communication is required. Numerical results ensure that
the proposed two-level group convolution provides better performance compared
to existing works in terms of suitability for parallel computation and
robustness with respect to the increase of the number of groups.

The rest of this paper is organized as follows. In Section 2, we introduce an
abstract block Jacobi framework and express the group convolution as a block
Jacobi approximation of the standard convolution. We present the proposed two-
level group convolution motivated by a two-level extension of the block Jacobi
method in Section 3. Improved classification accuracy of the proposed two-
level group convolution applied to WideResNet [17] and MobileNetV2 [18] with
various datasets are presented in Section 4. We conclude this paper with
remarks in Section 5.

##  2 Group convolution as block Jacobi approximation

In this section, we introduce an algebraic framework of block Jacobi
approximation for linear operators. Then we represent group convolution in the
framework of block Jacobi approximation. In this perspective, we will adopt
concepts from numerical analysis in order to improve group convolution in the
next section.

###  2.1 General block Jacobi approximation

Let $V$ and $W$ be real vector spaces and $N$ be a positive integer. We assume
that there exist a vector space $V_{k}$ and a linear operator $R_{k}\colon
V\rightarrow V_{k}$ such that

$V=\sum_{k=1}^{N}R_{k}^{T}V_{k}.$ (2.1)

Similarly, we assume that a vector space $W_{k}$ and a linear operator
$\widetilde{R}_{k}\colon W\rightarrow W_{k}$ satisfy

$W=\sum_{k=1}^{N}\widetilde{R}_{k}^{T}W_{k}.$ (2.2)

We construct a block Jacobi approximation of a linear operator $A\colon
V\rightarrow W$ under the space decomposition settings (2.1) and (2.2). A
local operator $A_{k}\colon V_{k}\rightarrow W_{k}$ is defined by

$A_{k}=\widetilde{R}_{k}AR_{k}^{T},\quad 1\leq k\leq N.$ (2.3)

Here, $R_{k}^{T}$ plays a role of prolongation that embeds an element of the
local space $V_{k}$ into the global space $V$, while $\widetilde{R}_{k}$
restricts an element of the global space $W$ to the local space $W_{k}$. The
block Jacobi approximation $M$ of $A$ with local operators (2.3) is given by

$M=\sum_{k=1}^{N}\widetilde{R}_{k}^{T}A_{k}R_{k}.$ (2.4)

Operators of the form (2.4) are typical in the field of numerical analysis;
see, e.g., [13, 14]. As a descriptive example, we consider the case when the
decompositions (2.1) and (2.2) are of direct sums, i.e.,

$V=\bigoplus_{k=1}^{N}V_{k}\quad\textrm{ and }\quad
W=\bigoplus_{k=1}^{N}W_{k}.$ (2.5)

In view of domain decomposition methods, space decompositions in (2.5) are
called one-level since subspaces $V_{k}$ and $W_{k}$ correspond to subdomains
when $V$ and $W$ are appropriate function spaces defined on entire domains;
see [16].

For the block matrix representation

$A=\left[A_{ij}\right]_{1\leq i,j\leq
N}=\begin{bmatrix}A_{11}&A_{12}&\dots&A_{1N}\\\ A_{22}&A_{22}&\dots&A_{2N}\\\
\vdots&\vdots&\ddots&\vdots\\\ A_{N1}&A_{N2}&\dots&A_{NN}\end{bmatrix},$ (2.6)

with respect to the decompositions (2.5), the corresponding block matrix
representation of the block Jacobi approximation $M$ is written as

$M=\operatorname{diag}\left(\left[A_{ii}\right]_{i=1}^{N}\right)=\begin{bmatrix}A_{11}&0&\dots&0\\\
0&A_{22}&\dots&0\\\ \vdots&\vdots&\ddots&\vdots\\\
0&0&\dots&A_{NN}\end{bmatrix},$ (2.7)

where $A_{ij}\colon V_{j}\rightarrow W_{i}$ is defined by
$A_{ij}=\widetilde{R}_{i}AR_{j}^{T}$. That is, $M$ is the block-diagonal part
of $A$. Comparing (2.6) and (2.7), it is clear that as $N$ increases, the
quality of approximation of $M$ with one-level space decompositions decreases
significantly; see [16].

###  2.2 Group convolution

Now, we show that the group convolution proposed in [3] fits the framework of
the block Jacobi approximation. For convenience, it is assumed that all images
considered in this section are of the same size, with height and width of $h$
and $w$, respectively. Let $\mathbb{P}$ be the vector space consisting of all
grayscale images, i.e., $\mathbb{P}=\mathbb{R}^{h\times w}$. For a positive
integer $n$, we may write the space of all $n$-channel images as
$\mathbb{P}^{n}$. In the following, let
$C_{d}\colon\mathbb{P}\rightarrow\mathbb{P}$ denote a generic $d\times d$
convolution on $\mathbb{P}$.

The full $d\times d$ convolution
$A\colon\mathbb{P}^{n}\rightarrow\mathbb{P}^{m}$ from $n$-channel images to
$m$-channel images can be written in a block form

$A=\left[C_{d}\right]_{m\times
n}=\vphantom{\begin{matrix}\overbrace{dummy}^{dummy}\\\ \\\ \\\
\end{matrix}}\begin{bmatrix}\mathrlap{\smash{\overbrace{\phantom{\begin{matrix}C_{d}&C_{d}&\dots&C_{d}\end{matrix}}}^{\mbox{$\scriptstyle{n}\textrm{
times}$}}}}C_{d}&C_{d}&\dots&C_{d}\\\ C_{d}&C_{d}&\dots&C_{d}\\\
\vdots&\vdots&\ddots&\vdots\\\
C_{d}&C_{d}&\dots&C_{d}\end{bmatrix}\begin{matrix}\left.\vphantom{\begin{matrix}x\\\
x\\\ y\\\ y\end{matrix}}\right\\}\scriptstyle{m}\textrm{ times}\end{matrix}.$
(2.8)

Now, let $N$ be a common divisor of $m$ and $n$. We consider a $d\times d$
group convolution
$A_{\mathrm{gp}}\colon\mathbb{P}^{n}\rightarrow\mathbb{P}^{m}$ with $N$ groups
so that $n$ input and $m$ output channels are partitioned into $N$ groups of
$n/N$ and $m/N$ channels, respectively. Then $N$ convolution operations from
$n/N$-channel images to $m/N$-channel images are done separately. If we
express the group convolution $A_{\mathrm{gp}}$ in a compact form, we have

$A_{\mathrm{gp}}=\operatorname{diag}\left(\left[\left[C_{d}\right]_{m/N\times
n/N}\right]_{k=1}^{N}\right).$ (2.9)

Note that, by assembling block matrices appropriately, the standard
convolution in (2.8) is rewritten as

$A=\left[\left[C_{d}\right]_{m/N\times n/N}\right]_{N\times N}.$ (2.10)

Observing that (2.9) and (2.10) have the same forms as (2.7) and (2.6),
respectively, we can say that the group convolution $A_{\mathrm{gp}}$ is
indeed a one-level block Jacobi approximation of the standard convolution $A$.

Equation (2.9) can be written in a more abstract fashion. We set
$V=\mathbb{P}^{n}$, $W=\mathbb{P}^{m}$, $V_{k}=\mathbb{P}^{n/N}$, and
$W_{k}=\mathbb{P}^{m/N}$. Let $R_{k}\colon V\rightarrow V_{k}$ and
$\widetilde{R}_{k}\colon W\rightarrow W_{k}$ be the restriction operators onto
the $k$th groups. Then it is straightforward to observe that $A_{\mathrm{gp}}$
agrees with the corresponding block Jacobi approximation $M$ in (2.4), i.e.,

$A_{\mathrm{gp}}=\sum_{k=1}^{N}\widetilde{R}_{k}^{T}A_{k}R_{k},$ (2.11)

where the local operator $A_{k}\colon V_{k}\rightarrow W_{k}$ defined in (2.3)
is given by

$A_{k}=[C_{d}]_{m/N\times n/N}.$ (2.12)

We note that entries of $R_{k}$ and $\widetilde{R}_{k}$ consist of 0’s and 1’s
only, while $A_{k}$ is composed of parameters to be determined by training.

##  3 Two-level group convolution

As we observed in Section 2, the group convolution $A_{\mathrm{gp}}$ is an
instance of one-level block Jacobi approximations of the standard convolution
$A$. Since the quality of approximation of one-level block Jacobi
approximation is deteriorated when $N$ becomes large, it can be expected that
the performance of neural networks using group convolution heavily depends on
the number of groups $N$; see [10] for relevant numerical results. There have
been several works [10, 11] to address these shortcomings of group
convolution, but all of these harm the parallel structure, another desirable
property of group convolution.

###  3.1 Algebraic framework

On the other hand, in the numerical analysis literature, there is a standard
treatment called a two-level method to compensate for the above-mentioned
problem of the one-level method without disturbing the parallel structure;
see, e.g., [15, 16]. In two-level methods, an additional subspace $V_{0}$
called coarse space is added in the decomposition (2.1) so that we get

$V=R_{0}^{T}V_{0}+\sum_{k=1}^{N}R_{k}^{T}V_{k},$ (3.1)

where $R_{0}\colon V\rightarrow V_{0}$ is a suitable linear operator. Usually,
the basis of $V_{0}$ contains $O(N)$ elements, where each element plays a role
of a representative of the subspace $V_{k}$. Therefore, the coarse space
$V_{0}$ promotes communication between local subspaces $V_{k}$’s. While
computations regarding the coarse space $V_{0}$ cannot be parallelized due to
its global nature, they do not disturb the parallel structure of the method
severely since the dimension of $V_{0}$ is $O(N)$, which is almost negligible
compared to the dimension of the entire space $V$. The same idea applies to
the space $W$, so we have

$W=\widetilde{R}_{0}^{T}W_{0}+\sum_{k=1}^{N}\widetilde{R}_{k}^{T}W_{k},$

where $\widetilde{R}_{0}\colon W\rightarrow W_{0}$ is a linear operator.

Figure 1: A graphical description of the coarse space $V_{0}$ for $N=3$. The
entire space $V$ is decomposed into subspace $V_{i}$ for $i=1,2,3$ satisfying
$V=\bigoplus_{i=1}^{3}V_{i}$. The elements of the coarse space $V_{0}$ are
representatives of each subspace $V_{i}$.

Motivated by the two-level method, we propose a novel group convolution called
two-level group convolution, which preserves the parallel structure without
significantly affecting the performance by increasing the number of groups.
Beginning with (3.1), let $V_{0}$ be the space of $N$-channel images in which
each channel plays a role of a representative of the group $V_{k}$. Likewise,
we define $W_{0}$ as the space of $N$-channel images; see Figure 1 for a
graphical description of the coarse spaces. For suitable operators
$A_{0}\colon V_{0}\rightarrow W_{0}$, $R_{0}\colon V\rightarrow V_{0}$, and
$\widetilde{R}_{0}\colon W\rightarrow W_{0}$ that will be defined later, a
prototype $A_{2\mathrm{L},\mathrm{proto}}$ for the two-level group convolution
is constructed by adding a coarse correction term
$\widetilde{R}_{0}^{T}A_{0}R_{0}$ to (2.11) as follows:

$A_{2\mathrm{L},\mathrm{proto}}=\widetilde{R}_{0}^{T}A_{0}R_{0}+\sum_{k=1}^{N}\widetilde{R}_{k}^{T}A_{k}R_{k}.$
(3.2)

While the restriction operators $R_{0}$ and $\widetilde{R}_{0}$ are naturally
defined according to the geometry of subspaces in the case of classical domain
decomposition methods [16, Sect. 3.3], we have to consider carefully how to
define these operators in the case of group convolution since there are no
underlying geometries for the coarse spaces $V_{0}$ and $W_{0}$. In the
following, we specify the coarse space $V_{0}$ and the operators $A_{0}$,
$R_{0}$, and $\widetilde{R}_{0}$ in the coarse correction term of (3.2).

As mentioned above, the operator $R_{0}$ must be designed so that, for any
$x\in V$, the $k$th channel of $R_{0}x\in V_{0}$ contains information of the
whole $V_{k}$. In this sense, we define $R_{0}$ so that the $k$th channel
$(R_{0}x)_{k}$ of $R_{0}x$ is constructed by the convolution applied to
$R_{k}x$ consisting of $(n/N)$ channels, i.e.,

$(R_{0}x)_{k}=[C_{d_{0}}]_{1\times n/N}R_{k}x,\quad x\in V,$ (3.3)

where $d_{0}$ is a positive integer. As we will see in Section 3.2, a suitable
choice for the kernel size $d_{0}$ is $d$, the size of the kernel $C_{d}$ in
(2.8). It is observed from (3.3) that each channel $(R_{0}x)_{k}$ can be
computed independently.

The main purpose of the coarse operator $A_{0}$ is data communication between
groups. It is desired that, for any $x_{0}\in V_{0}$, each channel of
$A_{0}x_{0}\in W_{0}$ is constructed by using all channels of $x_{0}$. At the
same time, the computational cost of $A_{0}$ should be as cheap as possible to
avoid becoming a bottleneck for parallel computation; it is also an important
issue to reduce the size of coarse structures in classical domain
decomposition methods [19, 20]. A reasonable choice for $A_{0}$ that meets
both of the above requirements is the full $1\times 1$ convolution from
$V_{0}$ to $W_{0}$:

$A_{0}=[C_{1}]_{N\times N}.$

The operator $\widetilde{R}_{0}^{T}$ distributes each channel $(y_{0})_{k}$ of
$y_{0}\in W_{0}$ to $W_{k}$ in an appropriate manner. We simply set
$\widetilde{R}_{0}^{T}$ as the combination of $1\times 1$ convolutions from
each channel $(y_{0})_{k}$ of $y_{0}$ to $m/N$ channels in $W_{k}$. That is,
we set

$\widetilde{R}_{0}^{T}y_{0}=\sum_{k=1}^{N}\widetilde{R}_{k}^{T}[C_{1}]_{m/N\times
1}(y_{0})_{k},\quad y_{0}\in W_{0}.$

Now, $A_{0}$, $R_{0}$, and $\widetilde{R}_{0}^{T}$ in
$A_{2\mathrm{L},\mathrm{proto}}$ are all specified. In addition, by
simplifying the coarse structure, we construct the final version
$A_{2\mathrm{L}}$ of two-level group convolution. Note that $1\times 1$
convolutions are linear combinations of channels. Therefore, for any $x_{0}\in
V_{0}$, every channel of $A_{0}x_{0}\in W_{0}$ is a linear combination of all
channels of $x_{0}$. Similarly, all channels in the $k$th group of
$\widetilde{R}_{0}^{T}A_{0}x_{0}\in W$ are scalar multiples of the $k$th
channel of $A_{0}x_{0}$. Therefore, we can see that
$\widetilde{R}_{0}^{T}A_{0}x_{0}$ is composed of linear combinations of all
channels of $x_{0}$. In this perspective, the proposed two-level group
convolution $A_{2\mathrm{L}}$ is made by replacing
$\widetilde{R}_{0}^{T}A_{0}$ in the coarse structure of
$A_{2\mathrm{L},\mathrm{proto}}$ by the full $1\times 1$ convolution from
$V_{0}$ to $W$, say $\bar{R}_{0}^{T}\colon V_{0}\rightarrow W$ such that

$\bar{R}_{0}^{T}=\sum_{k=1}^{N}\widetilde{R}_{k}^{T}[C_{1}]_{m/N\times
N}=\sum_{k=1}^{N}\widetilde{R}_{k}^{T}S_{k},$ (3.4)

where $S_{k}\colon V_{0}\rightarrow W_{k}$ is a $1\times 1$ standard
convolution from $V_{0}$ to $W_{k}$. There are two advantages of replacing
$\widetilde{R}_{0}^{T}A_{0}$ by $\bar{R}_{0}^{T}$. The first one is obvious;
the coarse structure of $A_{2\mathrm{L}}$ is simpler to implement than
$A_{2\mathrm{L},\mathrm{proto}}$. Moreover,
$\operatorname{ran}\bar{R}_{0}^{T}$ is larger than
$\operatorname{ran}\widetilde{R}_{0}^{T}A_{0}$ in general. As a result, the
performance improvement can be expected as the number of parameters increases.

Note that each component $\tilde{R}_{k}^{T}[C_{1}]_{m/N\times N}$ in (3.4) can
be processed in parallel if every processor possesses the whole $N$ channels
of an element of $V_{0}$. Since the $k$th processor owns the $k$th channel of
$V_{0}$ after parallel computation of $R_{0}$, data communication among
processors is required before applying $\bar{R}_{0}^{T}$; details will be
given in Section 3.2.

Finally, we summarize the proposed two-level group convolution as follows:

$A_{2\mathrm{L}}=\bar{R}_{0}^{T}R_{0}+\sum_{k=1}^{N}\widetilde{R}_{k}^{T}A_{k}R_{k}=\bar{R}_{0}^{T}R_{0}+A_{\mathrm{gp}}.$
(3.5)

###  3.2 Implementation issues

We discuss several issues on efficient parallel implementation of the proposed
two-level group convolution. First, we evaluate how many parameters are dealt
with by a single processor in parallel implementation of the two-level group
convolution using $N$ processors. In (3.5), the two-level group convolution
$A_{2\mathrm{L}}$ consists of two parts: the group convolution
$A_{\mathrm{gp}}=\sum_{k=1}^{N}\widetilde{R}_{k}^{T}A_{k}R_{k}$ and the coarse
structure $\bar{R}_{0}^{T}R_{0}$. Since $R_{k}$ and $\widetilde{R}_{k}^{T}$
consist of 0’s and 1’s and do not contain any parameters, it suffices to
consider the number of parameters in $A_{k}$, $R_{0}$, and $\bar{R}_{0}^{T}$
only. Note that both $R_{k}$ and $\widetilde{R}_{k}^{T}$ do not need to be
explicitly assembled in implementation. The group convolution
$A_{\mathrm{gp}}$ is composed of $N$ parallel blocks of
$A_{k}=[C_{d}]_{m/N\times n/N}$ as shown in (2.9) and (2.12). Similarly,
$R_{0}$ and $\bar{R}_{0}^{T}$ are composed of $N$ blocks of
$[C_{d_{0}}]_{1\times n/N}$ and $[C_{1}]_{m/N\times N}$ as shown in (3.3) and
(3.4), respectively. If each processor holds the parameters of
$[C_{d}]_{m/N\times n/N}$ and $[C_{d_{0}}]_{1\times n/N}$, then no
communication of parameters among processors are required. That is, the number
of parameters per processor of the proposed two-level group convolution is
given by

$d^{2}\cdot\frac{m}{N}\cdot\frac{n}{N}+d_{0}^{2}\cdot\frac{n}{N}=\frac{d^{2}mn}{N^{2}}+\frac{d_{0}^{2}n}{N}.$

On the other hand, the number of parameters of the coarse structure
$\bar{R}_{0}^{T}$ is $mN$. During the two-level computation, data
communication between processors is required even though the distributed
parameters do not need to be communicated. Since parallel computation of
$R_{0}$ is done in a way that the $k$th processor deals with the channels of
$V_{k}$ to generate the $(R_{0}x)_{k}$, each processor has only a single
channel of $R_{0}x$ just after an application of $R_{0}$. After that, a
collective interprocess operation gathers $(R_{0}x)_{k}$ from each processor
and then apply $\bar{R}_{0}^{T}$ to $R_{0}x\in V_{0}$. In the application of
coarse computation $\bar{R}_{0}^{T}$, we may assign a single dedicated
processor to handle whole $mN$ parameters of $\bar{R}_{0}^{T}$ or use all
processors where $\widetilde{R}_{k}^{T}[C_{1}]_{m/N\times N}$ is applied
separately. In the latter, each processor deals with additional $m$ parameters
so that we can avoid the coarse structure being a bottleneck when $N$ is
large. Also it makes the number of parameters per processor of the two-level
group convolution slightly more than that of group convolution. We chose the
latter for our implementation.

Next, we discuss why the choice $d_{0}=d$ of the kernel size of $R_{0}$ in
(3.3) is suitable. In the implementation of the two-level group convolution,
applications of $A_{k}$ and $R_{0}$ can be done simultaneously since $R_{k}$
has no computational cost. If $d_{0}=d$, $A_{k}$ and the restriction
$R_{0}|_{V_{k}}$ of $R_{0}$ onto $V_{k}$ can be implemented in combination as

$\begin{bmatrix}A_{k}\\\ R_{0}|_{V_{k}}\end{bmatrix}=[C_{d}]_{(m/N+1)\times
n/N}.$

That is, $A_{\mathrm{gp}}$ and $R_{0}$ can be implemented simultaneously by a
convolution from $n/N$ channels to $(m/N+1)$ channels in each processor if
$d_{0}=d$. Therefore, we choose $d_{0}=d$ for efficient parallel
implementation. We also note that, in general, a combined implementation of
two convolutions is more efficient than separate implementations since
convolutions are implemented by utilizing fast Fourier transform in libraries
for scientific computing; see, e.g., [21]. The overall procedure of the
proposed two-level group convolution is depicted in Figure 2.

Figure 2: Overall structure of the proposed two-level group convolution in the
implementation perspective when $N=3$. Note that $S_{k}=[C_{1}]_{m/N\times
N}$. Each processor has no parameter communication and only the representative
channels of the groups communicate. The dotted lines denotes such
communications.

##  4 Applications

In this section, we present numerical results of the proposed two-level group
convolution embedded into existing CNNs: WideResNet [17] and MobileNetV2 [18].
In the experiements with WideResNet, we discuss the classification performance
and parallel efficiency of the two-level group convolution based on the
results applied to $3\times 3$ convolutions. On the other hand, in the
experiments with MobileNetV2, based on the results applied to $1\times 1$
convolutions, we show that the number of network parameters is effectively
reduced while minimizing the performance degradation compared to the group
convolution and its variants [10, 11].

We implemented our method on datasets CIFAR-10, CIFAR-100 [22], and ImageNet
ILSVRC 2012 [23]. Details on the datasets we used are as follows. The
CIFAR-$m$ ($m=10,100$) dataset consists of $32\times 32$ colored natural
images including 50,000 training and 10,000 test samples with $m$ classes.
ImageNet is a dataset which contains 1,000 classes of $224\times 224$ colored
natural images. It includes 1,280,000 training and 50,000 test samples.

For CIFAR datasets, a data augmentation technique in [24] was adopted for
training; four pixels are padded on each side of images and $32\times 32$
random crops are sampled from the padded images and their horizontal flips.
All neural networks for CIFAR datasets in this section were trained using the
stochastic gradient descent with the batch size 128, weight decay $0.0005$,
momentum $0.9$, total epoch 200, and weights initialized as in [25]. The
initial learning rate was set to $0.1$ and was reduced to its one tenth in the
60th, 120th, and 160th epochs.

For ImageNet dataset, the input image is $224\times 224$ randomly cropped from
a resized image using the scale and aspect ratio augmentation of [5]
implemented by [26]. Hyperparameter settings are the same as the CIFAR case
except the followings: the weight decay $0.0001$, total epoch $90$, and the
learning rate reduced by a factor of 10 in the 30th and 60th epochs.

All networks were implemented in Python with PyTorch [27] and all computations
were performed on a cluster equipped with Intel Xeon Gold 5515 (2.4GHz, 20C)
CPUs, NVIDIA Titan RTX GPUs, and the operating system Ubuntu 18.04 64bit.

###  4.1 WideResNet

Figure 3: The structure of RU in WideResNet. Conv means the standard
convolution.

WideResNet [17] is a variant of ResNet [1] with improved performance; its
improved performance is due to the increased number of channels in each layer.
Each layer of WideResNet called residual unit (RU) consists of convolutions
and skip connections only (see Figure 3 for the structure of RU), so that the
classification performance of the WideResNet highly depends on the performance
of the convolution used in RU. Moreover, WideResNet is suitable for applying
group convolution techniques since the number of convolutional channels of
each layer is very large. Hence, for numerical experiments that highlight the
superiority of the proposed two-level group convolution compared to other
group convolution variants, WideResNet is a good choice as a model CNN. In
what follows, we denote WideResNet with hyperparameters $l$ and $w$ by
WideResNet-$l$-$w$, where $l$ and $w$ are the number of layers and the
channel-widening factor for convolutions in the network, respectively. We use
WideResNet-28-10 and WideResNet-34-2 for the CIFAR and ImageNet datasets,
respectively.

####  4.1.1 Ablation study

In order to highlight the effect of the coarse structure in the proposed two-
level group convolution, experiments for ablation study were conducted. We
present numerical results of several WideResNet implementations using the
standard convolution (SC), (one-level) group convolution (GC), and the
proposed two-level group convolution (GC-2L).

(a) WideResNet-28-10 (b) WideResNet-34-2

Figure 4: Training loss of WideResNet equipped with the standard convolution
(SC), the group convolution (GC), and the proposed two-level group convolution
(GC-2L): (a) WideResNet-28-10 for the CIFAR-10 dataset, (b) WideResNet-34-2
for the ImageNet dataset.

Figure 4 shows the decay of the training loss of WideResNet equipped with
various kinds of convolutions for the CIFAR-10 and ImageNet datasets. We
observe that the training loss of GC-2L converges to a smaller value than GC
in both cases. This phenomenon can be explained mathematically; since GC-2L
strictly contains GC in the sense that it reduces to GC when the coarse
structure is removed, its minimum loss must be smaller than GC. Moreover, the
loss decay curve of GC-2L lies between those of SC and GC. It means that the
representation capacity of GC-2L is bigger than GC, keeping the parallel
structure.

Table 1: The error rates, parameters per processor of WideResNet equipped with
the standard convolution (SC), the group convolution (GC), and the proposed
two-level group convolution (GC-2L) for the CIFAR and ImageNet datasets, for
which WideResNet-28-10 and WideResNet-34-2 were used, respectively.

WideResNet-28-10
---
Type | $N$ |  Parameters per processor | Error rate (%)
CIFAR-10 | CIFAR-100
SC | 1 | 36.48M | 4.19 | 18.95
GC | 2 | 18.25M | 4.61 | 20.77
GC-2L | 2 | 18.35M | 4.12 | 19.80
GC | 4 | 9.14M | 5.32 | 24.31
GC-2L | 4 | 9.25M | 4.34 | 20.90
GC | 8 | 4.58M | 6.54 | 26.89
GC-2L | 8 | 4.74M | 4.79 | 22.32
GC | 16 | 2.30M | 8.33 | 30.81
GC-2L | 16 | 2.54M | 4.95 | 23.49
WideResNet-34-2
---
Type | $N$ |  Parameters per processor | Error rate (%)
ImageNet
SC | 1 | 86.03M | 24.10
GC | 2 | 43.55M | 26.75
GC-2L | 2 | 43.71M | 25.36
GC | 4 | 22.31M | 30.48
GC-2L | 4 | 22.50M | 27.17
GC | 8 | 11.69M | 36.36
GC-2L | 8 | 11.95M | 28.43
GC | 16 | 6.38M | 43.18
GC-2L | 16 | 6.78M | 29.31

Table 1 summarizes numerical results of WideResNet equipped with various kinds
of convolutions applied to the CIFAR-10, CIFAR-100 and ImageNet datasets: the
number of parameters per processor and error rate for each convolution
according to $N$. The number of parameters per processor of GC-2L is similar
to that of GC, but the error rate is significantly reduced compared to GC. For
example, when $N=16$, WideResNet-34-2 had a $29.31\%$ error rate when equipped
with GC-2L, much less than $43.18\%$ of the one-level case.

####  4.1.2 Performance of parallel computation

Table 2: Numerical results of WideResNet-28-$w$ equipped with SC ($N=1$) and GC-2L ($N=2,4$) for the CIFAR-10 dataset, where $N$ denotes the number of groups. $w$ | $N$ | Parameters | Wall-clock time | Error rate (%)
---|---|---|---|---
40 | 1 | 583.1M | 49h 42m 57s | 3.64
2 | 292.0M | 17h 47m 18s | 3.58
4 | 146.3M | 8h 39m 47s | 3.73
60 | 1 | 1311.7M | 89h 49m 09s | 3.51
2 | 656.5M | 30h 18m 17s | 3.68
4 | 328.7M | 11h 48m 37s | 3.91
80 | 1 | 2331.8M | 255h 01m 27s | 3.77
2 | 1166.8M | 73h 28m 59s | 3.75
4 | 584.0M | 20h 11m 39s | 4.08

To check the performance of our two-level group convolution for parallel
computation, we conducted experiments while increasing the channel-widening
factor $w$ of WideResNet-28-$w$ for CIFAR-10 dataset. Table 2 shows the error
rates and wall-clock times of SC ($N=1$) and GC-2L ($N=2,4$) applied to
WideResNet-28-$w$ with $w=40,60$ and $80$. Note that in the case of $w=60$ and
$80$, it was impossible to train WideResNet-28-$w$ equipped with SC with one
GPU due to memory problems, so we trained using the naive model parallelism
[28] with $2$ and $4$ GPUs, respectively. Also, in the case of GC-2L, the
results of $N=2$ and $4$ are computed by utilizing $2$ and $4$ GPUs,
respectively.

First, looking at the wall-clock time with $w=40$, the training time of
WideResNet-28-$w$ was reduced by about half as $N$ doubled. In other cases,
the training time was reduced by almost a third as $N$ doubled. In fact,
doubling $N$ cuts the number of parameters by half, so the ideal training time
is reduced by quarter. However, it takes more than the ideal time due to the
calculation of the coarse structure and communication time. In addition, the
reason why the case of $w=40$ is different from the cases of $w=60,80$ is that
there is a difference in the GPU utilization rate because the number of
parameters processed by one GPU is relatively small. It is natural that as $N$
increases, the number of parameters decreases and the error rate increases
accordingly. But the increase in error rate is very moderate compared to the
number of decreased parameters. This means that GC-2L can effectively reduce
the training time with a small increase in error rate when performing parallel
computation of convolutions dealing with very large channels.

####  4.1.3 Comparison with existing approaches

Lastly, we verify that the performance of GC-2L is comparable to existing
state-of-the-art variants of group convolution; we compare the performance of
GC-2L to two variants of group convolution: ShuffleNet convolution (Shuffle)
[11] and Fully Learnable Group Convolution (FLGC) [10]. Shuffle randomly
permutes the channels of the output of a group convolution, resulting in data
exchange among groups in the output. On the other hand, FLGC dynamically
determines and updates the input channels and filters for each group according
to the overall loss of the network. Both Shuffle and FLGC lose the parallel
structure of group convolution due to large data exchange among groups so that
it may not be efficient to implement them in a distributed memory system
because of communication bottlenecks. Also, the authors of Shuffle and FLGC
did not apply the proposed convolution to all convolutional layers of the
network, but only a specific part. However, for fair comparison, we applied
the proposed convolutions to all convolutional layers and tested them. In the
following, we present numerical results executed on a single GPU in order to
highlight the performance of our proposed method compared to Shuffle and FLGC.

Table 3: The error rates and total number of parameters of WideResNet equipped
with the group convolution (GC), two-level group convolution(GC-2L), Shuffle,
Fully Learnable Group Convolution (FLGC) for the CIFAR and ImageNet datasets,
where $N$ denotes the number of groups.

WideResNet-28-10
---
Type | $N$ | Parameters | Error rate (%)
CIFAR-10 | CIFAR-100
GC | 2 | 18.25M | 4.61 | 20.77
Shuffle | 2 | 18.25M | 4.00 | 19.51
FLGC | 2 | 18.25M | 4.62 | 21.68
GC-2L | 2 | 18.35M | 4.12 | 19.80
GC | 4 | 9.14M | 5.32 | 24.31
Shuffle | 4 | 9.14M | 4.21 | 20.38
FLGC | 4 | 9.14M | 6.16 | 26.33
GC-2L | 4 | 9.25M | 4.34 | 20.90
GC | 8 | 4.58M | 6.54 | 26.89
Shuffle | 8 | 4.58M | 4.73 | 21.91
FLGC | 8 | 4.58M | 9.81 | 30.78
GC-2L | 8 | 4.74M | 4.79 | 22.32
GC | 16 | 2.30M | 8.33 | 30.81
Shuffle | 16 | 2.30M | 5.28 | 24.20
FLGC | 16 | 2.30M | 11.26 | 37.49
GC-2L | 16 | 2.54M | 4.95 | 23.49
WideResNet-34-2
---
Type | $N$ | Parameters | Error rate (%)
ImageNet
GC | 2 | 43.55M | 26.75
Shuffle | 2 | 43.55M | 25.12
FLGC | 2 | 43.55M | 30.43
GC-2L | 2 | 43.71M | 25.36
GC | 4 | 22.31M | 30.48
Shuffle | 4 | 22.31M | 26.84
FLGC | 4 | 22.31M | 38.30
GC-2L | 4 | 22.50M | 27.17
GC | 8 | 11.69M | 36.36
Shuffle | 8 | 11.69M | 29.03
FLGC | 8 | 11.69M | 45.54
GC-2L | 8 | 11.95M | 28.43
GC | 16 | 6.38M | 43.18
Shuffle | 16 | 6.38M | 32.27
FLGC | 16 | 6.38M | 62.02
GC-2L | 16 | 6.78M | 29.31

Table 3 shows numerical results of all convolutions mentioned above applied to
WideResNet-28-10 with the CIFAR-10, CIFAR-100 datasets and WideResNet-34-2
with the ImageNet dataset. The number of total parameters of GC-2L is slightly
larger than those of Shuffle and FLGC; such a difference is due to the coarse
structure of GC-2L. The classification performance was recorded in terms of
the error rate. The error rate of GC-2L is less than $5\%$ for CIFAR-10,
$24\%$ for CIFAR-100 and $30\%$ for ImageNet. The error rates of GC-2L are
generally similar to or even lower than other variants.

(a) WideResNet-28-10 (b) WideResNet-34-2

Figure 5: The error rate graph of WideResNet equipped with SC, GC, GC-2L,
FLGC, and Shuffle: (a) WideResNet-28-10 for the CIFAR-10 dataset, (b)
WideResNet-34-2 for the ImageNet dataset.

Moreover, Figure 5 shows that GC-2L has the lowest slope of error rate
increase even as the number of groups increases. Also, in both cases, Shuffle
and GC-2L showed similar performance until $N\leq 8$, but when $N>8$, GC-2L
showed better performance. What this means is that the channel shuffling
technique showed good performance in resolving the lack of inter-group
communication when $N$ is small. However, when $N$ is large, a suitable coarse
structure is required rather than simple shuffling. On the other hand, FLGC
showed the poor performance compared to others in our experiments; FLGC was
originally proposed with $1\times 1$ convolution and it seems that FLGC do not
show the good performance with convolutions of large kernel size.

###  4.2 MobileNetV2

Figure 6: The structure of an inverted residual block in MobileNetV2. Here,
Conv and DwConv mean standard convolution and depthwise convolution,
respectively.

MobileNetV2 [18] is a CNN architecture designed for use in smartphones; its
number of parameters is relatively small so that it is suitable to lightweight
computer architectures such as mobile devices. The core structure of
MobileNetV2 is an inverted residual block. It uses $1\times 1$ standard
convolutions to increase or decrease the number of channels. Figure 6 shows
the structure of the inverted residual block in MobileNetV2. The main
computational cost of the inverted residual block is due to two $1\times 1$
standard convolutions. Hence, similar to the case of $3\times 3$ convolutions
in WideResNet, group convolution and its variants can be utilized to reduce
the computational cost of $1\times 1$ standard convolutions in the inverted
residual block. In particular, we note that Shuffle and FLGC were originally
proposed for application to $1\times 1$ convolutions [10, 11]. Based on this
inverted residual block architecture, we replace the $1\times 1$ convolution
layers with GC, GC-2L, Shuffle, and FLGC.

Table 4: The error rates and total number of parameters of MobileNetV2
equipped with the standard $1\times 1$ convolution (SC), group convolution
(GC), Shuffle, Fully Learnable Group Convolution (FLGC), and the proposed two-
level group convolution (GC-2L) for the ImageNet dataset, where $N$ denotes
the number of groups. MobileNetV2
---
Type | $N$ | Parameters | Error rate (%)
ImageNet
SC | 1 | 3.50M | 30.61
GC | 2 | 2.44M | 36.96
Shuffle | 2 | 2.44M | 36.67
FLGC | 2 | 2.44M | 37.98
GC-2L | 2 | 2.47M | 33.56
GC | 4 | 1.91M | 42.63
Shuffle | 4 | 1.91M | 38.14
FLGC | 4 | 1.91M | 44.37
GC-2L | 4 | 1.95M | 36.88
GC | 8 | 1.65M | 50.36
Shuffle | 8 | 1.65M | 42.75
FLGC | 8 | 1.65M | 49.28
GC-2L | 8 | 1.72M | 39.16
Figure 7: The error rate graph of MobileNetV2 equipped with SC, GC, GC-2L,
FLGC, and Shuffle for the ImageNet dataset.

Numerical results in Table 4 ensure that the proposed two-level group
convolution effectively reduces the computational cost of $1\times 1$
convolution without severe performance degradation. The number of parameters
of GC-2L are slightly greater than those of other group convolution variants,
but still considerably less than that of standard convolution. In view of the
classification performance, we can observe that the error rate of GC-2L is the
lowest among all group convolution variants. In particular, the error rate of
GC-2L is less than $40\%$ even if the number of groups increases to $8$; this
highlights the robustness of GC-2L for multi-GPU computing compared to other
state-of-the-art group convolution variants. Figure 7 shows the increase rate
of the error of each convolution as the number of groups increases. One can
readily observe that the graph corresponding to GC-2L has the smallest slope
among all the graphs of the group convolution variants. Hence, we can expect
that the greater the numeber of groups, the greater the difference between the
error rate of GC-2L and other error rates. We conclude that adding an
appropriate coarse structure to the group convolution is a more effective
strategy than shuffling channels in the sense of both performance issue and
distributed memory computing.

##  5 Conclusion

In this paper, we proposed a novel approach for group convolution that is
suitable for distributed memory computing and robust with respect to the large
number of groups. Motivated by two-level methods in numerical analysis, a
coarse structure that promotes intergroup communication was added to the group
convolution. All parameters of the coarse structure can be distributed among
multiple GPUs so that the proposed two-level group convolution is suitable for
parallel computation. We presented how the additional coarse structure helped
the improvement of the performance of the group convolution through an
ablation study. We also observed that the two-level group convolution
performed similarly or better than conventional variants of group convolution.
We expect that the proposed two-level group convolution can be efficiently
utilized for training CNNs with a large number of channels, such as
WideResNet, through multiple GPUs. Additionally, the two-level group
convolution is expected to be effective for parameter pruning of small
networks for smartphones such as MobileNetV2.

## Declaration of Competing Interest

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

## Acknowledgments

This work was supported in part by the National Research Foundation (NRF) of
Korea grant funded by the Korea government (No. 2020R1A2C1A01004276), and in
part by Basic Science Research Program through NRF funded by the Ministry of
Education (No. 2019R1A6A1A10073887).

## References

  * [1] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.
  * [2] K. He, X. Zhang, S. Ren, J. Sun, Identity mappings in deep residual networks, in: European Conference on Computer Vision, Springer, 2016, pp. 630–645.
  * [3] A. Krizhevsky, I. Sutskever, G. E. Hinton, ImageNet classification with deep convolutional neural networks, in: Advances in Neural Information Processing Systems, 2012, pp. 1097–1105.
  * [4] J. Long, E. Shelhamer, T. Darrell, Fully convolutional networks for semantic segmentation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3431–3440.
  * [5] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich, Going deeper with convolutions, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 1–9.
  * [6] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, Z. Wojna, Rethinking the inception architecture for computer vision, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2818–2826.
  * [7] S. Xie, R. Girshick, P. Dollár, Z. Tu, K. He, Aggregated residual transformations for deep neural networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 1492–1500.
  * [8] G. Huang, S. Liu, L. Van der Maaten, K. Q. Weinberger, Condensenet: An efficient densenet using learned group convolutions, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 2752–2761.
  * [9] Y. Ioannou, D. Robertson, R. Cipolla, A. Criminisi, Deep roots: Improving CNN efficiency with hierarchical filter groups, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 1231–1240.
  * [10] X. Wang, M. Kan, S. Shan, X. Chen, Fully learnable group convolution for acceleration of deep neural networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 9049–9058.
  * [11] X. Zhang, X. Zhou, M. Lin, J. Sun, ShuffleNet: An extremely efficient convolutional neural network for mobile devices, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 6848–6856.
  * [12] F. Chollet, Xception: Deep learning with depthwise separable convolutions, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 1251–1258.
  * [13] C.-O. Lee, J. Park, Fast nonoverlapping block Jacobi method for the dual Rudin–Osher–Fatemi model, SIAM Journal on Imaging Sciences 12 (4) (2019) 2009–2034.
  * [14] Y. Saad, Iterative Methods for Sparse Linear Systems, SIAM, Philadelphia, 2003.
  * [15] V. Dolean, P. Jolivet, F. Nataf, An introduction to domain decomposition methods: algorithms, theory, and parallel implementation, SIAM, Philadelphia, 2015\.
  * [16] A. Toselli, O. Widlund, Domain Decomposition Methods – Algorithms and Theory, Springer, Berlin, 2005.
  * [17] S. Zagoruyko, N. Komodakis, Wide residual networks, in: Proceedings of the British Machine Vision Conference (BMVC), BMVA Press, 2016, pp. 87.1–87.12.
  * [18] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, L.-C. Chen, MobileNetV2: Inverted residuals and linear bottlenecks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4510–4520.
  * [19] C. R. Dohrmann, O. B. Widlund, On the design of small coarse spaces for domain decomposition algorithms, SIAM Journal on Scientific Computing 39 (4) (2017) A1466–A1488.
  * [20] A. Klawonn, P. Radtke, O. Rheinbach, FETI-DP methods with an adaptive coarse space, SIAM Journal on Numerical Analysis 53 (1) (2015) 297–320.
  * [21] S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, E. Shelhamer, cuDNN: Efficient primitives for deep learning, arXiv:1410.0759 (Oct. 2014).
  * [22] A. Krizhevsky, V. Nair, G. Hinton, CIFAR-10 and CIFAR-100 datasets, https://www.cs.toronto.edu/kriz/cifar.html 6.
  * [23] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, ImageNet: A large-scale hierarchical image database, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 248–255.
  * [24] C.-Y. Lee, S. Xie, P. Gallagher, Z. Zhang, Z. Tu, Deeply-supervised nets, in: Artificial Intelligence and Statistics, 2015, pp. 562–570.
  * [25] K. He, X. Zhang, S. Ren, J. Sun, Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification, in: Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 1026–1034.
  * [26] S. Gross, M. Wilber, Training and investigating residual nets, Facebook AI Research 6.
  * [27] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, S. Chintala, PyTorch: An imperative style, high-performance deep learning library, in: Advances in Neural Information Processing Systems, 2019, pp. 8024–8035.
  * [28] S. Lee, J. K. Kim, X. Zheng, Q. Ho, G. A. Gibson, E. P. Xing, On model parallelization and scheduling strategies for distributed machine learning, in: Advances in Neural Information Processing Systems, 2014, pp. 2834–2842.


observer $p$ at the point $q=\gamma(\tau_{0}\,=\,-\,L(z))$ defined by the
given length scale $L(z)$. For the reference FLRW the analogous set up is
associated to the constant-time slicing of the FLRW spacetime
$(M,\widehat{g})$ considered. The corresponding 3-dimensional region
$\widehat{V}_{L}^{3}(p)$, embedded in $\mathcal{J}^{-}_{{L}}(p,\widehat{g})$,
has the surface $(\widehat{\Sigma}_{z},\,\hat{h})$ as its boundary. The FLRW
observer $\widehat{\gamma}(\widehat{\tau})$ will intersect
$\widehat{V}_{L}^{3}(p)$ at the point
$\widehat{q}=\widehat{\gamma}(\widehat{\tau}_{0}\,=\,-\,L(z))$. By introducing
geodesic normal coordinates $\\{X^{i}\\}$ in $\mathcal{J}^{-}_{{L}}(p,{g})$
and $\\{Y^{k}\\}$ in $\mathcal{J}^{-}_{{L}}(p,\widehat{g})$, respectively
based at the point $q$ and $\widehat{q}$, we can pull back the metric tensors
$g$ and $\widehat{g}$ to $T_{q}M$ and $T_{\widehat{q}}M$, and obtain the
classical normal coordinate development of the metrics $g$ and $\widehat{g}$
valid in a sufficiently small convex neighborhood of $q$ and $\widehat{q}$.
Explicitly, for the (more relevant case of the) metric $g$, we have (see e. g.
Lemma 3.4 (p. 210) of [49] or [47])

$\displaystyle\left((\mathrm{exp}_{q})^{*}\,g\right)_{ef}\,=\,\eta_{ef}\,-\,\frac{1}{3}\,\mathrm{R}_{eabf}|_{q}X^{a}X^{b}\,-\,\frac{1}{6}\,\nabla_{c}\mathrm{R}_{eabf}|_{q}X^{a}X^{b}X^{c}$
$\displaystyle+\,\left(-\,\frac{1}{20}\,\nabla_{c}\nabla_{d}\mathrm{R}_{eabf}\,+\,\frac{2}{45}\,\mathrm{R}_{eabm}\,\mathrm{R}^{m}_{fcd}\right)_{q}\,X^{a}X^{b}X^{c}X^{d}\,+\,\ldots\;,$

where $\mathrm{R}_{abcd}$ is the Riemann tensor of the metric $g$ (evaluated
at the point $q$). The induced expansion in the pulled-back measure
$\left((\mathrm{exp}_{s(\eta)})^{*}d\mu_{g}\right)$ provides the Lorentzian
analog of the familiar Bertrand-Puiseux formulas associated with the
geometrical interpretation of the sectional, Ricci and scalar curvature for a
Riemannian manifold in terms of the length, area, and volume measures of small
geodesic balls. In the Lorentzian case the relevant formulas are more delicate
to derive, [3], [25], [26], [43]. This asymptotics provides [25], to leading
order in $L(z)$, the following expressions for the area of
$({\Sigma}_{z},\,{g}_{(z)})$ and $(\widehat{\Sigma}_{z},\,\widehat{g}_{(z)})$,

(134)
$A\left({\Sigma}_{z}\right)\,=\,{\pi}\,L^{2}(z)\,\left(1\,-\,\frac{1}{72}\,L^{2}(z)\,\mathrm{R}(q)\,+\,\ldots\right)\;,$

and

(135)
$A\left(\widehat{\Sigma}_{z}\right)\,=\,{\pi}\,L^{2}(z)\,\left(1\,-\,\frac{1}{72}\,L^{2}(z)\,\widehat{\mathrm{R}}(\widehat{q})\,+\,\ldots\right)\;,$

Introducing these expressions in (133) we eventually get

(136)
$\widehat{\mathrm{R}}(\hat{q})\,=\,{\mathrm{R}}({q})\,+\,\frac{72}{\pi}\frac{d_{(z)}\left[\widehat{\Sigma}_{z},\,{\Sigma}_{z}\right]}{L^{4}(z)}\,+\,\frac{144}{\pi
L^{4}(z)}\,\int_{\widehat{\Sigma}_{z}}\,\left[\frac{\left|\mathrm{Jac}\left(\zeta_{(z)}\right)\right|^{\frac{1}{2}}\,D(\zeta_{(z)})\,-\,\widehat{D}(\widehat{r}(z))}{\widehat{D}(\widehat{r}(z))}\right]\,d\mu_{\widehat{g}_{(z)}}\,+\,\ldots\,.$

Notice that the integral is the average value over the sky section
$(\widehat{\Sigma}_{z},\widehat{g}_{(z)})$, of the fluctuations of
$\left|\mathrm{Jac}\left(\zeta_{(z)}\right)\right|^{\frac{1}{2}}\,D(\zeta_{(z)})$
with respect to $\widehat{D}(\widehat{r}(z))$, average that for notational
ease we write as

(137)
$\left\langle\left.D(\zeta_{(z)})\right|\widehat{D}(\widehat{r}(z))\right\rangle_{\widehat{\Sigma}_{z}}\,:=\,A^{-1}(\widehat{\Sigma}_{z})\,\int_{\widehat{\Sigma}_{z}}\,\left[\frac{\left|\mathrm{Jac}\left(\zeta_{(z)}\right)\right|^{\frac{1}{2}}\,D(\zeta_{(z)})\,-\,\widehat{D}(\widehat{r}(z))}{\widehat{D}(\widehat{r}(z))}\right]\,d\mu_{\widehat{g}_{(z)}}\,,$

while, as we have already stressed, the distance functional is (up to the
$A(\widehat{\Sigma}_{z})$ normalization) the square mean deviation of this
average, _i. e._ ,

$\displaystyle\left\langle\left(\left.D(\zeta_{(z)})\right|\widehat{D}(\widehat{r}(z))\right)^{2}\right\rangle_{\widehat{\Sigma}_{z}}\,:\,$
$\displaystyle=$
$\displaystyle\,A^{-1}(\widehat{\Sigma}_{z})\,\int_{\widehat{\Sigma}_{z}}\,\left[\frac{\left|\mathrm{Jac}\left(\zeta_{(z)}\right)\right|^{\frac{1}{2}}\,D(\zeta_{(z)})\,-\,\widehat{D}(\widehat{r}(z))}{\widehat{D}(\widehat{r}(z))}\right]^{2}\,d\mu_{\widehat{g}_{(z)}}$
$\displaystyle=$
$\displaystyle\,A^{-1}(\widehat{\Sigma}_{z})\,d_{(z)}\left[\widehat{\Sigma}_{z},\,{\Sigma}_{z}\right]\,.$

To put these results at work, let us assume the conservative and quite a
reasonable scenario where the fluctuations in the area distance
$D(\zeta_{(z)})$, even if locally large in the various celestial coordinates
bins, average out to zero over $\widehat{\Sigma}_{z}$. However, the
corresponding square mean deviation of the fluctuations
$\left\langle\left(\left.D(\zeta_{(z)})\right|\widehat{D}(\widehat{r}(z))\right)^{2}\right\rangle_{\widehat{\Sigma}_{z}}\,=\,A^{-1}(\widehat{\Sigma}_{z})\,d_{(z)}\left[\widehat{\Sigma}_{z},\,{\Sigma}_{z}\right]$
can be significantly different from zero, and from (136) we get

(139)
$\widehat{\mathrm{R}}(\hat{q})\,=\,{\mathrm{R}}({q})\,+\,\frac{72}{\pi}\frac{d_{(z)}\left[\widehat{\Sigma}_{z},\,{\Sigma}_{z}\right]}{L^{4}(z)}\,+\,\ldots\,.$

The physical scalar curvature we measure (hard to!) in such a scenario is
${\mathrm{R}}({q})$, and if we decide to modeling with a FLRW solution a
cosmological spacetime, homogeneous on large scale but highly inhomogeneous at
smaller scale, then (139) shows that we cannot identify ${\mathrm{R}}({q})$
with the corresponding FLRW scalar curvature $\widehat{\mathrm{R}}(\hat{q})$.
Such an identification is possible, with a rigorous level of scale dependence
precision, only if we take into account the term

(140)
$\frac{72}{\pi}\frac{d_{(z)}\left[\widehat{\Sigma}_{z},\,{\Sigma}_{z}\right]}{L^{4}(z)}\,.$

According to Theorem 1, this term vanishes once $L(z)$ probes the homogeneity
scales, conversely, it is clear from (139) that in pre-homogeneity region its
presence is forced on us and plays the role of a scale-dependent effective
positive contribution to the cosmological constant. As long as the local
inhomogeneities give rise to significant fluctuations in the area distance
$D(\zeta_{(z)})$, this contribution cannot be considered a priori negligible
in high-precision cosmology.

## References

  * [1] J. Beem, P. Ehrlich, K. Easley, Global Lorentzian Geometry, Monographs and Textbooks in Pure and Applied Mathematics Vol. 202, 2nd ed., Dekker, New York, (1996).
  * [2] M. S. Berger, On Riemannian structures of prescribed Gaussian curvature for compact 2-manifolds, J. Diff. Geo. 5 325-332 (1971).
  * [3] C. Berthiere, G. Gibbons, and S. N. Solodukhin, Comparison theorems for causal diamonds, Phys. Rev. D 92, 064036 (2015).
  * [4] T. Buchert, M. Carfora, Regional averaging and scaling in relativistic cosmology, Class. Quantum Grav. 19 6109 (2002) DOI 10.1088/0264-9381/19/23/314.
  * [5] T. Buchert, M. Carfora, Cosmological Parameters Are Dressed, Phys. Rev. Lett. 90, 031101, (2003).
  * [6] T. Buchert, M. Carfora, On the curvature of the present-day Universe, Class.Quant.Grav.25:195001, (2008) https://doi.org/10.1088/0264-9381/25/19/195001, arXiv:0803.1401v2 [gr-qc].
  * [7] T. Buchert, H. van Elst, A. Heinesen The averaging problem on the past null cone in inhomogeneous dust cosmologies, arXiv:2202.10798v1 [gr-qc].
  * [8] M. Carfora, A. Marzuoli, Smoothing Out Spatially Closed Cosmologies, Phys. Rev. Lett. 53, 2445 (1984) https://doi.org/10.1103/PhysRevLett.53.2445.
  * [9] M. Carfora, K. Piotrkowska, Renormalization group approach to relativistic cosmology, Phys. Rev. D 52, 4393 (1995) https://doi.org/10.1103/PhysRevD.52.4393.
  * [10] M. Carfora, A. Marzuoli, Einstein Constraints and Ricci Flow: A Geometrical Averaging of Initial Data Sets, Mathematical Physics Studies (MPST), Springer (2023) ISBN: 978-981-19-8539-3.
  * [11] T. Buchert, M. Carfora, G. F. R. Ellis, E. W. Kolb, M. MacCallum, J. Ostrowski, S. Rasanen, B. Roukema, L. Andersson, A. Coley, D. Wiltshire, Is there proof that backreaction of inhomogeneities is irrelevant in cosmology?. Classical and Quantum Gravity 32 (21), (2015).
  * [12] M. Carfora, F. Familiari, A comparison theorem for cosmological lightcones, Letters in Mathematical Physics, 111:53 (2021), https://doi.org/10.1007/s11005-021-01393-2
  * [13] B.-L. Chen and P. LeFloch, Injectivity Radius of Lorentzian Manifolds, Commun. Math. Phys. 278 (2008)
  * [14] Y. Choquet-Bruhat, P. T. Chrusciel, J. M. Martín-García, The light-cone theorem, Class. Quantum Grav. 26, 135011 (22pp) (2009).
  * [15] Y. Choquet-Bruhat, General relativity and the Einstein Equations, Oxford Mathematical Monographs, Oxford University Press, Oxford, (2009).
  * [16] Y. Choquet-Bruhat, P. T. Chrusciel, J. M. Martín-García, The Cauchy Problem on a Characteristic Cone for the Einstein Equations in Arbitrary Dimensions, Annales Henri Poincaré 12, 419482 (2011)
  * [17] P. Chrusciel and J. Grant, On Lorentzian causality with continuous metrics, Class. Quantum Grav. 29 (2012)
  * [18] G.F.R. Ellis, The Domain of Cosmology and the Testing of Cosmological Theories, pp. 3-39 in The Philosophy of Cosmology, Eds. K. Chamcham, J. Silk, J. D. Barrow, S. Saunders, Cambridge Univ. Press (2017).
  * [19] G.F.R. Ellis and H. van Elst, Deviation of geodesics in FLRW spacetime geometries, p. 203-225 in A. Harvey ed.,On Einstein’s path-Essays in Honor of Engelbert Schücking, Springer, New York, (1999).
  * [20] G.F.R. Ellis, S.D. Nel, R. Maartens, W.R. Stoeger, and A.P. Whitman, Ideal observational cosmology, Phys. Reports 124, 315-417 (1985).
  * [21] G.F.R. Ellis, R. Maartens and M. A. H. MacCallum, Relativistic Cosmology, Cambridge Univ. Press (2012).
  * [22] J. Eells and L. Lemaire, A report on harmonic maps, Bull. London Math. Soc., 10, 168 (1978).
  * [23] L. C. Evans, R. F. Gariepy, Measure Theory and Fine Properties of Functions, Studies in Advanced Mathematics, CRC Press (Boca Raton, Florida) (1992).
  * [24] M. Gasperini, G. Marozzi3, F. Nugier and G. Veneziano, Light-cone averaging in cosmology: formalism and applications, Journal of Cosmology and Astroparticle Physics JCAP07(2011)008, DOI 10.1088/1475-7516/2011/07/008.
  * [25] G. W. Gibbons and S. N. Solodukhin, The geometry of small causal diamonds, Phys. Lett. B 649 (2007) 317 [hep-th/0703098]
  * [26] G. W. Gibbons and S. N. Solodukhin, The geometry of large causal diamonds and the No-Hair property of asymptotically DeSitter spacetimes, Phys. Lett. B 652 (2007) 103 [arXiv: 0706.0603]
  * [27] J. R. Gott, III, M. Juric, D. Schlegel, F. Hoyle, M. Vogeley, M. Tegmark, N. Bahcall, J. Brinkmann, A Map of the Universe, Astrophys.J.624:463,(2005) (arXiv:astro-ph/0310571).
  * [28] X. Gu, S.-T. Yau, Computing conformal structure of surfaces, Communications in Information and Systems, Vol. 2 (2002) 121-146.
  * [29] J. Hass and P. Koehl, Comparing shapes of genus-zero surfaces, Journal of Applied and Computational Topology, Vol. 1, (2017) 5787.
  * [30] A. Heinesen, Multipole decomposition of the general luminosity distance ’Hubble law’ – a new framework for observational cosmology, Journal of Cosmology and Astroparticle Physics 05(2021)008 https://doi.org/10.1088/1475-7516/2021/05/008, arXiv:2010.06534v2 [astro-ph.CO].
  * [31] A. Heinesen, Redshift drift cosmography for model-independent cosmological inference, Phys. Rev. D 104 (2021), 123527 https://doi.org/10.1103/PhysRevD.104.123527, arXiv:2107.08674v2 [astro-ph.CO].
  * [32] F. Hélein and J. C. Wood, Harmonic maps, in Handbook of Global Analysis, Elsevier (2007).
  * [33] D. W. Hogg, Distance measures in cosmology, arXiv: astro-ph/9905116v4 (2000).
  * [34] D. W. Hogg, D. J. Eisenstein, M. R. Blanton, N. A. Bahcall, J. Brinkmann, J. E. Gunn, and D. P. Schneider, Cosmic Homogeneity Demonstrated with Luminous Red Galaxies, The Astrophysical Journal, Vol. 624, (2005) 54-58.
  * [35] M. Jin, Y. Wang, S.-T. Yau, X. Gu, Optimal global conformal surface parametrization for visualization, Communications in Information and Systems, (2005) 117-134.
  * [36] J. Jost, Riemannian geometry and geometric analysis, 2nd ed. Springer Universitext, Springer–Verlag (1998).
  * [37] S. Klainerman and I. Rodnianski, On the radius of injectivity of null hypersurfaces, Journal of the American Mathematical Society, Vol. 21 (2008), 775795, S 0894-0347(08)00592-4
  * [38] M. Kunzinger, R. Steinbauer, J. A. Vickers, The Penrose singularity theorem in regularity $C^{1,\,1}$, Classical and Quantum Gravity, 32 (15), 1-13, [155010]. (doi:10.1088/0264-9381/32/15/155010) (2015).
  * [39] E. W. Kolb, V. Marra, S. Matarrese, Cosmological background solutions and cosmological backreactions, Gen.Rel.Grav. 42 (2010) 1399-1412.
  * [40] M. Kunzinger, R. Steinbauer, M. Stojkovic, The exponential map of a $C^{1,1}-$metric, Differential Geometry and its Applications 34 (2014) 14-24.
  * [41] R. Maartens, Is the Universe homogeneous? Philosophical Transactions of the Royal Society, A 369 (2011), 5115-37.
  * [42] E. Minguzzi, Convex neighborhoods for Lipschitz connections and sprays, Monatsh. Math. 177, 569-625 (2015), https://doi.org/10.1007/s00605-014-0699-y. See also arXiv:1308.6675v2 [math.DG].
  * [43] J. Myrheim, Statistical Geometry, Report No. CERN-TH-2538, (1978) (unpublished).
  * [44] D. Ntalampekos and M. Romney, Polyhedral approximation and uniformization for non-length surfaces, arXiv:2206.01128v1[math. MG] (2022).
  * [45] V. Perlick, Gravitational Lensing from a Spacetime Perspective, Living Reviews in Relativity, Livingreviews.org/lrr-2004-9.
  * [46] R. Penrose and W. Rindler, Spinors and Space Time Vol I, Cambridge Monographs on Mathematical Physics, Cambridge Univ. Press (1984).
  * [47] P. Petersen, Riemannian Geometry, Graduate Text in Mathematics 171 Springer (1998).
  * [48] J. Rosenberg, Applications of Analysis on Lipschitz Manifolds, Proc. Centre Math. Appl., 16: 349pp. (1987) 269-283, Miniconference on Harmonic Analysis and Operator Algebras (August 5-8 and December 2-3, 1987, Australian National University, Canberra). Eds. M. Cowling, C. Meaney, W. Moran.
  * [49] R. Schoen, S-T. Yau, Lectures on differental geometry, Vol.I, International Press, Cambridge, MA, (1994).
  * [50] M. Scrimgeour M et al., The WiggleZ Dark Energy Survey: the transition to large-scale cosmic homogeneity Mon. Not. R. Astr. Soc. 425 (2012) 116 [arXiv:1205.6812].
  * [51] J. M. M. Senovilla, Singularity Theorems and Their Consequences, Gen. Rel. and Gravitation, 30 (1998) 701-848, (this is available with updates in arXiv:1801.04912v1 [gr-qc] https://doi.org/10.48550/arXiv.1801.04912).
  * [52] J.-P. Uzan, Emergent Structures of Effective Field Theories, pp. 109-135 in The Philosophy of Cosmology, Eds. K. Chamcham, J. Silk, J. D. Barrow, S. Saunders, Cambridge Univ. Press (2017).
  * [53] D. L., Wiltshire, P. R. Smale, T. Mattsson, and R. Watkins, Hubble flow variance and the cosmic rest frame, Physical Review D, 88(8), (2013), doi:10.1103/physrevd.88.083529
  * [54] D.Wiltshire, Comment on ”Hubble flow variations as a test for inhomogeneous cosmology”, Atronomy and Astrophysics 624 (2019) A12.
  * [55] P. Zhang and A. Stebbins, Confirmation of the Copernican principle through the anisotropic kinetic Sunyaev Zel’dovich effect, Philosophical Transactions of the Royal Society, A 369 (2011), 5138-45.