$g^{\prime}(F({\bm{w}},{\bm{p}}^{1},{\bm{p}}^{2}))={\bm{G}}=\begin{bmatrix}g_{11}&g_{12}&\cdots&g_{1s_{2}}\\\
g_{21}&\ddots&&g_{2s_{2}}\\\ \vdots&&\ddots&\vdots\\\
g_{s_{1}1}&g_{s_{1}2}&\cdots&g_{s_{1}s_{2}}\\\ \end{bmatrix}$ (35)

As in (11), (12) and (13), by applying the chain rule we obtain:

$\forall i\in\llbracket 1\ ..\ m\rrbracket\colon$

$\displaystyle\frac{\partial loss}{\partial
w_{i}}=g^{\prime}(F({\bm{w}},{\bm{p}}^{1},{\bm{p}}^{2}))\underset{\text{F}}{\times}\frac{\partial
F({\bm{w}},{\bm{p}}^{1},{\bm{p}}^{2})}{\partial w_{i}}$ (36)
$\displaystyle\frac{\partial loss}{\partial
p_{i}^{1}}=g^{\prime}(F({\bm{w}},{\bm{p}}^{1},{\bm{p}}^{2}))\underset{\text{F}}{\times}\frac{\partial
F({\bm{w}},{\bm{p}}^{1},{\bm{p}}^{2})}{\partial p_{i}^{1}}$ (37)
$\displaystyle\frac{\partial loss}{\partial
p_{i}^{2}}=g^{\prime}(F({\bm{w}},{\bm{p}}^{1},{\bm{p}}^{2}))\underset{\text{F}}{\times}\frac{\partial
F({\bm{w}},{\bm{p}}^{1},{\bm{p}}^{2})}{\partial p_{i}^{2}}\ $ (38)

with

$\begin{split}g^{\prime}(F({\bm{w}},{\bm{p}}^{1},{\bm{p}}^{2}))=&\frac{\partial
loss}{\partial\displaystyle{\bm{K}}}=\frac{\partial loss}{\partial
F({\bm{w}},{\bm{p}}^{1},{\bm{p}}^{2})}\end{split}$ (39)

Let us put this time

$g^{\prime}(F({\bm{w}},{\bm{p}}^{1},{\bm{p}}^{2}))={\bm{G}}=\begin{bmatrix}g_{11}&g_{12}&\cdots&g_{1s_{2}}\\\
g_{21}&\ddots&&g_{1s_{2}}\\\ \vdots&&\ddots&\vdots\\\
g_{s_{1}1}&g_{12}&\cdots&g_{s_{1}s_{2}}\\\ \end{bmatrix}$ (40)

Using the definition of $F$, and by substituting the last equation in (36),
(37) and (38), we have:

$\displaystyle\frac{\partial loss}{\partial
w_{i}}={\bm{G}}\underset{\text{F}}{\times}\frac{\partial\sum_{i=1}^{m}f(w_{i},p^{1}_{i},p^{2}_{i})}{\partial
w_{i}}$ (41) $\displaystyle\frac{\partial loss}{\partial
p_{i}^{1}}={\bm{G}}\underset{\text{F}}{\times}\frac{\partial\sum_{i=1}^{m}f(w_{i},p^{1}_{i},p^{2}_{i})}{\partial
p_{i}^{1}}$ (42) $\displaystyle\frac{\partial loss}{\partial
p_{i}^{2}}={\bm{G}}\underset{\text{F}}{\times}\frac{\partial\sum_{i=1}^{m}f(w_{i},p^{1}_{i},p^{2}_{i})}{\partial
p_{i}^{2}}\ $ (43)

And we know that:

$\forall(i,j)\in\llbracket 1\ ..\ m\rrbracket^{2}\colon$

$i\neq j\implies\frac{\partial f(w_{j},p^{1}_{j},p^{2}_{j})}{\partial
w_{i}}=\frac{\partial f(w_{j},p^{1}_{j},p^{2}_{j})}{\partial
p_{i}^{1}}=\frac{\partial f(w_{j},p^{1}_{j},p^{2}_{j})}{\partial p_{i}^{2}}=0$

which simplifies equations (41), (42) and (43) to

$\forall i\in\llbracket 1\ ..\ m\rrbracket\colon$

$\displaystyle\frac{\partial loss}{\partial
w_{i}}={\bm{G}}\underset{\text{F}}{\times}\frac{\partial
f(w_{i},p^{1}_{i},p^{2}_{i})}{\partial w_{i}}$ (44)
$\displaystyle\frac{\partial loss}{\partial
p_{i}^{1}}={\bm{G}}\underset{\text{F}}{\times}\frac{\partial
f(w_{i},p^{1}_{i},p^{2}_{i})}{\partial p_{i}^{1}}$ (45)
$\displaystyle\frac{\partial loss}{\partial
p_{i}^{2}}={\bm{G}}\underset{\text{F}}{\times}\frac{\partial
f(w_{i},p^{1}_{i},p^{2}_{i})}{\partial p_{i}^{2}}\ $ (46)

We can notice that we are brought back to the scalar case, and we deduce from
(30), (31) and (32) the gradients of the loss function with respect to weights
and positions in the general case:

$\forall i\in\llbracket 1\ ..\ m\rrbracket\colon$

$\displaystyle\begin{split}\left(\frac{\partial
loss}{\partial{\bm{w}}}\right)_{i}=\ &(1-r_{i}^{1})\ (1-r_{i}^{2})\ g_{\lfloor
p_{i}^{1}\rfloor\lfloor p_{i}^{2}\rfloor}\\\ &+r_{i}^{1}\ (1-r_{i}^{2})\
g_{\lfloor p_{i}^{1}\rfloor+1\lfloor p_{i}^{2}\rfloor}\\\ &+(1-r_{i}^{1})\
r_{i}^{2}\ g_{\lfloor p_{i}^{1}\rfloor\lfloor p_{i}^{2}\rfloor+1}\\\
&+r_{i}^{1}\ r_{i}^{2}\ g_{\lfloor p_{i}^{1}\rfloor+1\lfloor
p_{i}^{2}\rfloor+1}\end{split}$ (47)
$\displaystyle\begin{split}\left(\frac{\partial
loss}{\partial{\bm{p}}^{1}}\right)_{i}=\ w_{i}\ [&-(1-r_{i}^{2})\ g_{\lfloor
p_{i}^{1}\rfloor\lfloor p_{i}^{2}\rfloor}\\\ &+(1-r_{i}^{2})\ g_{\lfloor
p_{i}^{1}\rfloor+1\lfloor p_{i}^{2}\rfloor}\\\ &-r_{i}^{2}\ g_{\lfloor
p_{i}^{1}\rfloor\lfloor p_{i}^{2}\rfloor+1}\\\ &+r_{i}^{2}\ g_{\lfloor
p_{i}^{1}\rfloor+1\lfloor p_{i}^{2}\rfloor+1}]\end{split}$ (48)
$\displaystyle\begin{split}\left(\frac{\partial
loss}{\partial{\bm{p}}^{2}}\right)_{i}=\ w_{i}\ [&-(1-r_{i}^{1})\ g_{\lfloor
p_{i}^{1}\rfloor\lfloor p_{i}^{2}\rfloor}\\\ &-r_{i}^{1}\ g_{\lfloor
p_{i}^{1}\rfloor+1\lfloor p_{i}^{2}\rfloor}\\\ &+(1-r_{i}^{1})\ g_{\lfloor
p_{i}^{1}\rfloor\lfloor p_{i}^{2}\rfloor+1}\\\ &+r_{i}^{1}\ g_{\lfloor
p_{i}^{1}\rfloor+1\lfloor p_{i}^{2}\rfloor+1}]\end{split}$ (49)

The results found for the general case are nothing but a component-wise
application of the result obtained in the scalar case. In addition, we show in
Appendix 7.4, the extension to the 1D and 3D convolution cases.

###  7.4 1D-DCLS, 3D-DCLS

We denote respectively by
$s_{1},s_{2},s_{3}\in\mathbb{N}^{*}\times\mathbb{N}^{*}\times\mathbb{N}^{*}$,
the sizes of the constructed kernel along the x-axis, y-axis and the z-axis.
Moreover, the $n\times p\times q$ tensor space of third dimension is denoted
$\mathbb{R}^{n\times p\times q}$.

The function $f$ defined in (7) could be adapted in order to construct a
suitable kernel for the 1D convolution in the scalar weight case as follows:

$\displaystyle\begin{split}f_{1D}\colon\mathbb{R}\times\mathbb{R}&\to\mathbb{R}^{s}\\\
w,p&\mapsto\displaystyle{\bm{k}}\end{split}$ (50)

where $\forall i\in\llbracket 1\ ..\ s\rrbracket\colon$

$\displaystyle{\bm{k}}_{i}=\left\\{\begin{array}[]{cl}w\ (1-r)&\text{if
}i=\lfloor p\rfloor\\\ w\ r&\text{if }i=\lfloor p\rfloor+1\\\ 0&\text{else
}\end{array}\right.$ (51)

and where the fractional part is:

$\begin{split}r=\\{p\\}=p-\lfloor p\rfloor\end{split}$ (52)

Following the same construction in Appendix 7.3 we can show that the gradients
of the loss function with respect to weights and positions in the general 1D
case are:

$\forall i\in\llbracket 1\ ..\ m\rrbracket\colon$

$\displaystyle\begin{split}\left(\frac{\partial
loss}{\partial{\bm{w}}}\right)_{i}=\ &(1-r_{i})\ g_{\lfloor
p_{i}\rfloor}+r_{i}\ g_{\lfloor p_{i}\rfloor+1}\end{split}$ (53)
$\displaystyle\begin{split}\left(\frac{\partial
loss}{\partial{\bm{p}}}\right)_{i}=\ &w_{i}\ (g_{\lfloor p_{i}\rfloor+1}-\
g_{\lfloor p_{i}\rfloor})\end{split}$ (54)

Furthermore, we define the function $f_{3D}$, the suitable kernel construction
function in the 3D convolution case, as such:

$\displaystyle\begin{split}f\colon\mathbb{R}\times\mathbb{R}\times\mathbb{R}\times\mathbb{R}&\to\mathbb{R}^{s_{1}\times
s_{2}\times s_{3}}\\\
w,p^{1},p^{2},p^{3}&\mapsto{\bm{\mathsfit{K}}}\end{split}$ (55)

where $\forall i\in\llbracket 1\ ..\ s_{1}\rrbracket$, $\forall j\in\llbracket
1\ ..\ s_{2}\rrbracket,\forall l\in\llbracket 1\ ..\ s_{3}\rrbracket\colon$

${\bm{\mathsfit{K}}}_{ijl}=\left\\{\begin{array}[]{cl}w\ (1-r^{1})\ (1-r^{2})\
(1-r^{3})&\text{if }i=\lfloor p^{1}\rfloor,\ j=\lfloor p^{2}\rfloor,\
l=\lfloor p^{3}\rfloor\\\ w\ r^{1}\ (1-r^{2})\ (1-r^{3})&\text{if }i=\lfloor
p^{1}\rfloor+1,\ j=\lfloor p^{2}\rfloor,\ l=\lfloor p^{3}\rfloor\\\ w\
(1-r^{1})\ r^{2}\ (1-r^{3})&\text{if }i=\lfloor p^{1}\rfloor,\ j=\lfloor
p^{2}\rfloor+1,\ l=\lfloor p^{3}\rfloor\\\ w\ r^{1}\ r^{2}\ (1-r^{3})&\text{if
}i=\lfloor p^{1}\rfloor{+}1,\ j=\lfloor p^{2}\rfloor{+}1,\ l=\lfloor
p^{3}\rfloor\\\ w\ (1-r^{1})\ (1-r^{2})\ r^{3}&\text{if }i=\lfloor
p^{1}\rfloor,\ j=\lfloor p^{2}\rfloor,\ l=\lfloor p^{3}\rfloor+1\\\ w\ r^{1}\
(1-r^{2})\ r^{3}&\text{if }i=\lfloor p^{1}\rfloor+1,\ j=\lfloor p^{2}\rfloor,\
l=\lfloor p^{3}\rfloor+1\\\ w\ (1-r^{1})\ r^{2}\ r^{3}&\text{if }i=\lfloor
p^{1}\rfloor,\ j=\lfloor p^{2}\rfloor+1,\ l=\lfloor p^{3}\rfloor+1\\\ w\
r^{1}\ r^{2}\ r^{3}&\text{if }i=\lfloor p^{1}\rfloor{+}1,\ j=\lfloor
p^{2}\rfloor{+}1,\ l=\lfloor p^{3}\rfloor+1\\\ 0&\text{else
}\end{array}\right.$ (56)

and where the fractional parts are:

$\begin{split}r^{1}=\\{p^{1}\\}=p^{1}-\lfloor p^{1}\rfloor\\\
r^{2}=\\{p^{2}\\}=p^{2}-\lfloor p^{2}\rfloor\\\
r^{3}=\\{p^{3}\\}=p^{3}-\lfloor p^{3}\rfloor\end{split}$ (57)

We can show that the gradients of the loss function with respect to weights
and positions in the general 3D case are:

$\forall i\in\llbracket 1\ ..\ m\rrbracket\colon$

$\displaystyle\begin{split}\left(\frac{\partial
loss}{\partial{\bm{w}}}\right)_{i}=\ &(1-r_{i}^{1})\ (1-r_{i}^{2})\
(1-r_{i}^{3})\ g_{\lfloor p_{i}^{1}\rfloor\lfloor p_{i}^{2}\rfloor\lfloor
p_{i}^{3}\rfloor}\\\ &+r_{i}^{1}\ (1-r_{i}^{2})\ (1-r_{i}^{3})\ g_{\lfloor
p_{i}^{1}\rfloor+1\lfloor p_{i}^{2}\rfloor\lfloor p_{i}^{3}\rfloor}\\\
&+(1-r_{i}^{1})\ r_{i}^{2}\ (1-r_{i}^{3})\ g_{\lfloor p_{i}^{1}\rfloor\lfloor
p_{i}^{2}\rfloor+1\lfloor p_{i}^{3}\rfloor}\\\ &+r_{i}^{1}\ r_{i}^{2}\
(1-r_{i}^{3})\ g_{\lfloor p_{i}^{1}\rfloor+1\lfloor p_{i}^{2}\rfloor+1\lfloor
p_{i}^{3}\rfloor}\\\ &+(1-r_{i}^{1})\ (1-r_{i}^{2})\ r_{i}^{3}\ g_{\lfloor
p_{i}^{1}\rfloor\lfloor p_{i}^{2}\rfloor\lfloor p_{i}^{3}\rfloor+1}\\\
&+r_{i}^{1}\ (1-r_{i}^{2})\ r_{i}^{3}\ g_{\lfloor p_{i}^{1}\rfloor+1\lfloor
p_{i}^{2}\rfloor\lfloor p_{i}^{3}\rfloor+1}\\\ &+(1-r_{i}^{1})\ r_{i}^{2}\
r_{i}^{3}\ g_{\lfloor p_{i}^{1}\rfloor\lfloor p_{i}^{2}\rfloor+1\lfloor
p_{i}^{3}\rfloor+1}\\\ &+r_{i}^{1}\ r_{i}^{2}\ r_{i}^{3}\ g_{\lfloor
p_{i}^{1}\rfloor+1\lfloor p_{i}^{2}\rfloor+1\lfloor
p_{i}^{3}\rfloor+1}\end{split}$ (58)
$\displaystyle\begin{split}\left(\frac{\partial
loss}{\partial{\bm{p}}^{1}}\right)_{i}=\ w_{i}\ [\ &-\ (1-r_{i}^{2})\
(1-r_{i}^{3})\ g_{\lfloor p_{i}^{1}\rfloor\lfloor p_{i}^{2}\rfloor\lfloor
p_{i}^{3}\rfloor}\\\ &+\ (1-r_{i}^{2})\ (1-r_{i}^{3})\ g_{\lfloor
p_{i}^{1}\rfloor+1\lfloor p_{i}^{2}\rfloor\lfloor p_{i}^{3}\rfloor}\\\ &-\
r_{i}^{2}\ (1-r_{i}^{3})\ g_{\lfloor p_{i}^{1}\rfloor\lfloor
p_{i}^{2}\rfloor+1\lfloor p_{i}^{3}\rfloor}\\\ &+\ r_{i}^{2}\ (1-r_{i}^{3})\
g_{\lfloor p_{i}^{1}\rfloor+1\lfloor p_{i}^{2}\rfloor+1\lfloor
p_{i}^{3}\rfloor}\\\ &-\ (1-r_{i}^{2})\ r_{i}^{3}\ g_{\lfloor
p_{i}^{1}\rfloor\lfloor p_{i}^{2}\rfloor\lfloor p_{i}^{3}\rfloor+1}\\\ &+\
(1-r_{i}^{2})\ r_{i}^{3}\ g_{\lfloor p_{i}^{1}\rfloor+1\lfloor
p_{i}^{2}\rfloor\lfloor p_{i}^{3}\rfloor+1}\\\ &-\ r_{i}^{2}\ r_{i}^{3}\
g_{\lfloor p_{i}^{1}\rfloor\lfloor p_{i}^{2}\rfloor+1\lfloor
p_{i}^{3}\rfloor+1}\\\ &+\ r_{i}^{2}\ r_{i}^{3}\ g_{\lfloor
p_{i}^{1}\rfloor+1\lfloor p_{i}^{2}\rfloor+1\lfloor
p_{i}^{3}\rfloor+1}]\end{split}$ (59)
$\displaystyle\begin{split}\left(\frac{\partial
loss}{\partial{\bm{p}}^{2}}\right)_{i}=\ w_{i}\ [&-\ (1-r_{i}^{1})\
(1-r_{i}^{3})\ g_{\lfloor p_{i}^{1}\rfloor\lfloor p_{i}^{2}\rfloor\lfloor
p_{i}^{3}\rfloor}\\\ &-r_{i}^{1}\ (1-r_{i}^{3})\ g_{\lfloor
p_{i}^{1}\rfloor+1\lfloor p_{i}^{2}\rfloor\lfloor p_{i}^{3}\rfloor}\\\
&+(1-r_{i}^{1})\ (1-r_{i}^{3})\ g_{\lfloor p_{i}^{1}\rfloor\lfloor
p_{i}^{2}\rfloor+1\lfloor p_{i}^{3}\rfloor}\\\ &+r_{i}^{1}\ (1-r_{i}^{3})\
g_{\lfloor p_{i}^{1}\rfloor+1\lfloor p_{i}^{2}\rfloor+1\lfloor
p_{i}^{3}\rfloor}\\\ &-(1-r_{i}^{1})\ r_{i}^{3}\ g_{\lfloor
p_{i}^{1}\rfloor\lfloor p_{i}^{2}\rfloor\lfloor p_{i}^{3}\rfloor+1}\\\
&-r_{i}^{1}\ r_{i}^{3}\ g_{\lfloor p_{i}^{1}\rfloor+1\lfloor
p_{i}^{2}\rfloor\lfloor p_{i}^{3}\rfloor+1}\\\ &+(1-r_{i}^{1})\ r_{i}^{3}\
g_{\lfloor p_{i}^{1}\rfloor\lfloor p_{i}^{2}\rfloor+1\lfloor
p_{i}^{3}\rfloor+1}\\\ &+r_{i}^{1}\ r_{i}^{3}\ g_{\lfloor
p_{i}^{1}\rfloor+1\lfloor p_{i}^{2}\rfloor+1\lfloor
p_{i}^{3}\rfloor+1}]\end{split}$ (60)
$\displaystyle\begin{split}\left(\frac{\partial
loss}{\partial{\bm{p}}^{3}}\right)_{i}=\ w_{i}\ [\ &-(1-r_{i}^{1})\
(1-r_{i}^{2})\ \ g_{\lfloor p_{i}^{1}\rfloor\lfloor p_{i}^{2}\rfloor\lfloor
p_{i}^{3}\rfloor}\\\ &-r_{i}^{1}\ (1-r_{i}^{2})\ \ g_{\lfloor
p_{i}^{1}\rfloor+1\lfloor p_{i}^{2}\rfloor\lfloor p_{i}^{3}\rfloor}\\\
&-(1-r_{i}^{1})\ r_{i}^{2}\ \ g_{\lfloor p_{i}^{1}\rfloor\lfloor
p_{i}^{2}\rfloor+1\lfloor p_{i}^{3}\rfloor}\\\ &-r_{i}^{1}\ r_{i}^{2}\ \
g_{\lfloor p_{i}^{1}\rfloor+1\lfloor p_{i}^{2}\rfloor+1\lfloor
p_{i}^{3}\rfloor}\\\ &+(1-r_{i}^{1})\ (1-r_{i}^{2})\ g_{\lfloor
p_{i}^{1}\rfloor\lfloor p_{i}^{2}\rfloor\lfloor p_{i}^{3}\rfloor+1}\\\
&+r_{i}^{1}\ (1-r_{i}^{2})\ g_{\lfloor p_{i}^{1}\rfloor+1\lfloor
p_{i}^{2}\rfloor\lfloor p_{i}^{3}\rfloor+1}\\\ &+(1-r_{i}^{1})\ r_{i}^{2}\
g_{\lfloor p_{i}^{1}\rfloor\lfloor p_{i}^{2}\rfloor+1\lfloor
p_{i}^{3}\rfloor+1}\\\ &+r_{i}^{1}\ r_{i}^{2}\ g_{\lfloor
p_{i}^{1}\rfloor+1\lfloor p_{i}^{2}\rfloor+1\lfloor
p_{i}^{3}\rfloor+1}]\end{split}$ (61)

##  8 Appendix: The 2D-DCLS kernel construction algorithm

In the following, we describe with pseudocode the forward and backward passes
for kernel construction used in 2D-DCLS. In practice, ${\bm{\mathsfit{W}}}$,
${\bm{\mathsfit{P}}}^{1}$ and ${\bm{\mathsfit{P}}}^{2}$ are 3-D tensors of
size (channels_out, channels_in // groups, K_count), but the algorithms
presented here are easily extended to this case by applying them channel-wise.

0: ${\bm{\mathsfit{W}}}$, ${\bm{\mathsfit{P}}}^{1}$, ${\bm{\mathsfit{P}}}^{2}$
: vectors of dimension $m$

0: ${\bm{\mathsfit{K}}}$ : the constructed kernel, of size ($s_{1}\times
s_{2}$)

1: ${\bm{\mathsfit{K}}}\leftarrow 0$

2: ${\bm{p}}^{1}\leftarrow\lfloor{\bm{\mathsfit{P}}}^{1}\rfloor$;
$\quad{\bm{p}}^{2}\leftarrow\lfloor{\bm{\mathsfit{P}}}^{2}\rfloor$

3: ${\bm{\mathsfit{R}}}^{1}\leftarrow{\bm{\mathsfit{P}}}^{1}-{\bm{p}}^{1}$;
$\quad{\bm{\mathsfit{R}}}^{2}\leftarrow{\bm{\mathsfit{P}}}^{2}-{\bm{p}}^{2}$

4: save_for_backward
(${\bm{p}}^{1},{\bm{p}}^{2},{\bm{\mathsfit{R}}}^{1},{\bm{\mathsfit{R}}}^{2}$)

5: for $i=0\rightarrow m-1$ do

6:
${\bm{\mathsfit{K}}}[{\bm{p}}^{1}_{i},{\bm{p}}^{2}_{i}]\mathrel{+}={\bm{\mathsfit{W}}}_{i}*(1-{\bm{\mathsfit{R}}}^{1}_{i})*(1-{\bm{\mathsfit{R}}}^{2}_{i})$

7:
${\bm{\mathsfit{K}}}[{\bm{p}}^{1}_{i}+1,{\bm{p}}^{2}_{i}]\mathrel{+}={\bm{\mathsfit{W}}}_{i}*({\bm{\mathsfit{R}}}^{1}_{i})*(1-{\bm{\mathsfit{R}}}^{2}_{i})$

8:
${\bm{\mathsfit{K}}}[{\bm{p}}^{1}_{i},{\bm{p}}^{2}_{i}+1]\mathrel{+}={\bm{\mathsfit{W}}}_{i}*(1-{\bm{\mathsfit{R}}}^{1}_{i})*({\bm{\mathsfit{R}}}^{2}_{i})$

9:
${\bm{\mathsfit{K}}}[{\bm{p}}^{1}_{i}+1,{\bm{p}}^{2}_{i}+1]\mathrel{+}={\bm{\mathsfit{W}}}_{i}*({\bm{\mathsfit{R}}}^{1}_{i})*({\bm{\mathsfit{R}}}^{2}_{i})$

10: end for

Algorithm 1 2D-DCLS kernel construction forward pass

0: $GradK=\frac{\partial Loss}{\partial K}$ : matrix of dimension
($s_{1}\times s_{2}$)

0: $\frac{\partial Loss}{\partial W}$, $\frac{\partial Loss}{\partial P^{1}}$,
$\frac{\partial Loss}{\partial P^{2}}$ : vectors of dimension $m$

1: $\frac{\partial Loss}{\partial W}\leftarrow 0$, $\frac{\partial
Loss}{\partial P^{1}}\leftarrow 0$, $\frac{\partial Loss}{\partial
P^{2}}\leftarrow 0$

2:
${\bm{p}}^{1},{\bm{p}}^{2},{\bm{\mathsfit{R}}}^{1},{\bm{\mathsfit{R}}}^{2}\leftarrow\text{load\\_saved
( \ )}$

3: for $i=0\rightarrow m-1$ do

4: $\displaystyle\frac{\partial Loss}{\partial W}[i]\mathrel{+}=\frac{\partial
Loss}{\partial
K}[{\bm{p}}^{1}_{i},{\bm{p}}^{2}_{i}]*(1{-}{\bm{\mathsfit{R}}}^{1}_{i})*(1{-}{\bm{\mathsfit{R}}}^{2}_{i})+\frac{\partial
Loss}{\partial
K}[{\bm{p}}^{1}_{i}+1,{\bm{p}}^{2}_{i}]*{\bm{\mathsfit{R}}}^{1}_{i}*(1{-}{\bm{\mathsfit{R}}}^{2}_{i})$
$\displaystyle+\frac{\partial Loss}{\partial
K}[{\bm{p}}^{1}_{i},{\bm{p}}^{2}_{i}+1]*(1{-}{\bm{\mathsfit{R}}}^{1}_{i})*{\bm{\mathsfit{R}}}^{2}_{i}+\frac{\partial
Loss}{\partial
K}[{\bm{p}}^{1}_{i}+1,{\bm{p}}^{2}_{i}+1]*{\bm{\mathsfit{R}}}^{1}_{i}*{\bm{\mathsfit{R}}}^{2}_{i}$

5: $\displaystyle\frac{\partial Loss}{\partial
P^{1}}[i]\mathrel{+}={\bm{\mathsfit{W}}}_{i}*[-\frac{\partial Loss}{\partial
K}[{\bm{p}}^{1}_{i},{\bm{p}}^{2}_{i}]*(1-{\bm{\mathsfit{R}}}^{2}_{i})+\frac{\partial
Loss}{\partial
K}[{\bm{p}}^{1}_{i}+1,{\bm{p}}^{2}_{i}]*(1-{\bm{\mathsfit{R}}}^{2}_{i})$
$\displaystyle-\frac{\partial Loss}{\partial
K}[{\bm{p}}^{1}_{i},{\bm{p}}^{2}_{i}+1]*{\bm{\mathsfit{R}}}^{2}_{i}+\frac{\partial
Loss}{\partial
K}[{\bm{p}}^{1}_{i}+1,{\bm{p}}^{2}_{i}+1]*{\bm{\mathsfit{R}}}^{2}_{i}]$

6: $\displaystyle\frac{\partial Loss}{\partial
P^{2}}[i]\mathrel{+}={\bm{\mathsfit{W}}}_{i}*[-\frac{\partial Loss}{\partial
K}[{\bm{p}}^{1}_{i},{\bm{p}}^{2}_{i}]*(1-{\bm{\mathsfit{R}}}^{1}_{i})-\frac{\partial
Loss}{\partial
K}[{\bm{p}}^{1}_{i}+1,{\bm{p}}^{2}_{i}]*{\bm{\mathsfit{R}}}^{1}_{i}$
$\displaystyle+\frac{\partial Loss}{\partial
K}[{\bm{p}}^{1}_{i},{\bm{p}}^{2}_{i}+1]*(1-{\bm{\mathsfit{R}}}^{1}_{i})+\frac{\partial
Loss}{\partial
K}[{\bm{p}}^{1}_{i}+1,{\bm{p}}^{2}_{i}+1]*{\bm{\mathsfit{R}}}^{1}_{i}]$

7: end for

Algorithm 2 2D-DCLS kernel construction backward pass

The ‘for’ loops in the algorithms are fully parallelized using GPU threads.
The 2D-DCLS convolution with kernel construction is then obtained by applying
the classical 2D-convolution provided natively by PyTorch or any other method
such as the depthwise implicit gemm convolution method Ding et al. (2022)
using the constructed kernel.

When considering a concurrent execution of this pseudocode, the additions may
result, in case of overlapping, in a replacement of the overlapped values
instead of the desired accumulation. This problem can be addressed by using
atomic addition operations.

##  9 Appendix: The DCLS kernel construction algorithm in native PyTorch

In the following, we describe with PyTorch code the DCLS construction module
for 1D, 2D and 3D versions. The DCLS convolution method is obtained by using
the constructed kernel as a weight for the native torch.nn.Conv{1,2,3}d, or
another convolution method such as depthwise implicit gemm Megvii (2020).

⬇

1class ConstructKernel1d(Module):

2 def __init__(self, out_channels,

3 in_channels, groups, kernel_count, dilated_kernel_size):

4 super().__init__()

5 self.out_channels = out_channels

6 self.in_channels = in_channels

7 self.groups = groups

8 self.dilated_kernel_size = dilated_kernel_size

9 self.kernel_count = kernel_count

10 I = torch.arange(0, dilated_kernel_size[0])

11 I = I.expand(out_channels,

12 in_channels//groups, kernel_count,-1).permute(3,0,1,2)

13 self.I = Parameter(I, requires_grad=False)

14

15 self.lim = torch.zeros(1)

16 self.lim[0] = dilated_kernel_size[0]

17 self.lim = self.lim.expand(out_channels, in_channels//groups,

18 kernel_count, -1).permute(3,0,1,2)

19 self.lim = Parameter(self.lim, requires_grad=False)

20

21 def forward(self, W, P):

22 P = P + self.lim // 2

23 Pr = P

24 P = P.floor()

25 R = (Pr - P).expand(self.dilated_kernel_size[0],-1,-1,-1,-1)

26 R1 = R.select(1,0); P1 = P.select(0,0)

27 cond1 = (self.I == P1)

28 cond2 = (self.I == P1+1)

29 W1 = torch.where(cond1, 1.0, 0.0)

30 W2 = torch.where(cond2, 1.0, 0.0)

31

32 K = W1 + R1 * (W2 - W1)

33 K = W * K

34 K = K.sum(3)

35 K = K.permute(1,2,0)

36 return K

37

38class ConstructKernel2d(Module):

39 def __init__(self, out_channels, in_channels, groups,

40 kernel_count, dilated_kernel_size):

41 super().__init__()

42 self.out_channels = out_channels

43 self.in_channels = in_channels

44 self.groups = groups

45 self.dilated_kernel_size = dilated_kernel_size

46 self.kernel_count = kernel_count

47 J = torch.arange(0,

48 dilated_kernel_size[0]).expand(dilated_kernel_size[1],-1)

49 I = torch.arange(0,

50 dilated_kernel_size[1]).expand(dilated_kernel_size[0],-1)

51 I = I.expand(out_channels,

52 in_channels//groups, kernel_count,-1,-1).permute(3,4,0,1,2)

53 J = J.expand(out_channels,

54 in_channels//groups, kernel_count,-1,-1).permute(4,3,0,1,2)

55

56 self.I = Parameter(I, requires_grad=False)

57 self.J = Parameter(J, requires_grad=False)

58 self.lim = torch.zeros(2)

59 self.lim[0] = dilated_kernel_size[0]

60 self.lim[1] = dilated_kernel_size[1];

61 self.lim = self.lim.expand(out_channels, in_channels//groups,

62 kernel_count, -1).permute(3,0,1,2)

63 self.lim = Parameter(self.lim, requires_grad=False)

64

65 def forward(self, W, P):

66 P = P + self.lim // 2

67 Pr = P

68 P = P.floor()

69 R = (Pr - P).expand(self.dilated_kernel_size[0],

70 self.dilated_kernel_size[1],-1,-1,-1,-1)

71 R1 = R.select(2,0); P1 = P.select(0,0)

72 R2 = R.select(2,1); P2 = P.select(0,1)

73 R1R2 = R1*R2

74 cond1 = (self.I == P1)

75 cond2 = (self.J == P2)

76 cond3 = (self.I == P1+1)

77 cond4 = (self.J == P2+1)

78 W1 = torch.where(cond1*cond2, 1.0, 0.0)

79 W2 = torch.where(cond1*cond4, 1.0, 0.0)

80 W3 = torch.where(cond3*cond2, 1.0, 0.0)

81 W4 = torch.where(cond3*cond4, 1.0, 0.0)

82 K = W1 + R1R2*(W1 - W2 - W3 + W4) + R1*(W3 - W1) + R2*(W2-W1)

83 K = W * K

84 K = K.sum(4)

85 K = K.permute(2,3,0,1)

86 return K

87

88class ConstructKernel3d(Module):

89 def __init__(self, out_channels, in_channels, groups,

90 kernel_count, dilated_kernel_size):

91 super().__init__()

92 self.out_channels = out_channels

93 self.in_channels = in_channels

94 self.groups = groups

95 self.dilated_kernel_size = dilated_kernel_size

96 self.kernel_count = kernel_count

97 L = torch.arange(0,

98 dilated_kernel_size[0]).expand(dilated_kernel_size[1],

99 dilated_kernel_size[2],-1)

100 J = torch.arange(0,

101 dilated_kernel_size[1]).expand(dilated_kernel_size[0],

102 dilated_kernel_size[2],-1)

103 I = torch.arange(0,

104 dilated_kernel_size[2]).expand(dilated_kernel_size[0],

105 dilated_kernel_size[1],-1)

106 L = L.expand(out_channels,

107 in_channels//groups,kernel_count,-1,-1,-1).permute(5,3,4,0,1,2)

108 I = I.expand(out_channels,

109 in_channels//groups,kernel_count,-1,-1,-1).permute(3,4,5,0,1,2)

110 J = J.expand(out_channels,

111 in_channels//groups,kernel_count,-1,-1,-1).permute(3,5,4,0,1,2)

112 self.L = Parameter(L, requires_grad=False)

113 self.I = Parameter(I, requires_grad=False)

114 self.J = Parameter(J, requires_grad=False)

115 self.lim = torch.zeros(3)

116 self.lim[0] = dilated_kernel_size[0]

117 self.lim[1] = dilated_kernel_size[1]

118 self.lim[2] = dilated_kernel_size[2]

119 self.lim = self.lim.expand(out_channels, in_channels//groups,

120 kernel_count, -1).permute(3,0,1,2)

121 self.lim = Parameter(self.lim, requires_grad=False)

122

123 def forward(self, W, P):

124 P = P + self.lim // 2

125 Pr = P

126 P = P.floor()

127 R = (Pr - P).expand(self.dilated_kernel_size[0],

128 self.dilated_kernel_size[1], self.dilated_kernel_size[2],-1,-1,-1,-1)

129 R1 = R.select(3,0); P1 = P.select(0,0)

130 R2 = R.select(3,1); P2 = P.select(0,1)

131 R3 = R.select(3,2); P3 = P.select(0,2)

132

133 cond1 = (self.L == P1)

134 cond2 = (self.I == P2)

135 cond3 = (self.J == P3)

136 cond4 = (self.L == P1+1)

137 cond5 = (self.I == P2+1)

138 cond6 = (self.J == P3+1)

139 W1 = torch.where(cond1*cond2*cond3, 1.0, 0.0)

140 W2 = torch.where(cond4*cond2*cond3, 1.0, 0.0)

141 W3 = torch.where(cond1*cond5*cond3, 1.0, 0.0)

142 W4 = torch.where(cond4*cond5*cond3, 1.0, 0.0)

143 W5 = torch.where(cond1*cond2*cond6, 1.0, 0.0)

144 W6 = torch.where(cond4*cond2*cond6, 1.0, 0.0)

145 W7 = torch.where(cond1*cond5*cond6, 1.0, 0.0)

146 W8 = torch.where(cond4*cond5*cond6, 1.0, 0.0)

147 # needs a better computing

148 K = W1 * (1 - R1) * (1 - R2) * (1 - R3)

149 K += W2 * R1 * (1 - R2) * (1 - R3)

150 K += W3 * (1 - R1) * R2 * (1 - R3)

151 K += W4 * R1 * R2 * (1 - R3)

152 K += W5 * (1 - R1) * (1 - R2) * R3

153 K += W6 * R1 * (1 - R2) * R3

154 K += W7 * (1 - R1) * R2 * R3

155 K += W8 * R1 * R2 * R3

156 K = W * K

157 K = K.sum(5)

158 K = K.permute(3,4,0,1,2)

159 return K

##  10 Appendix: Histograms of positions

In the following, we show as histograms over training epochs, the distribution
of the four kernel positions for the 2D-DCLS convolutions of the ConvNeXt-T-
dcls model. Note that there is no agglutination or edge effect around the
kernel limits, and that the distributions are relatively stable, with a higher
concentration around the center of the kernel. Individual positions, however,
are constantly moving; see animation at:

https://github.com/K-H-Ismail/Dilated-Convolution-with-Learnable-Spacings-
PyTorch/blob/main/figs/animation.gif.

Figure 3: The distribution over epochs of kernel positions for the stage 0 of
the ConvNeXt-T-dcls model.

Figure 4: Idem for stage 1.

Figure 5: Idem for stage 2.

Figure 6: Idem for stage 3.

##  11 Appendix: Speed curves and learning rate schedule

Here, we plot the average speed curves of the four position tensors for the
2D-DCLS convolutions $V_{\bm{\mathsfit{P}}}$ of the ConvNeXt-T-dcls model as
functions of the training epochs. The general formula for the average speed of
a DCLS position tensor of size $(cout,cin,m)\in\mathbb{N^{*}}^{3}$, at epoch
$t$, is as follows:

$\forall t\in\llbracket 1\ ..\ t_{max}\rrbracket\colon\quad
V_{\bm{\mathsfit{P}}}(t)=\frac{1}{cout\cdot cin\cdot
m}\sum\limits_{k=1}^{cout}\sum\limits_{j=1}^{cin}\sum\limits_{i=1}^{m}|P^{t}_{ijk}-P^{t-1}_{ijk}|$

Figure 7: The average speed of the four position tensors for the ConvNeXt-T-
dcls model as function of epochs. Figure 8: The learning schedule used for
training ConvNeXt-T-dcls model.

At epoch 0, we can notice that the average speed is abnormally high, this is
due to the fact that in the beginning, the positions are initialized randomly
and we arbitrarily considered that $V_{{\bm{\mathsfit{P}}}}(0)=0$, thus the
large speed gap at initialization. Another peak can be seen at epoch 20, this
one is due to the introduction of the repulsive loss (Thomas et al., 2019) at
this precise epoch during training. This last causes a momentary increase in
the average speed of the DCLS positions. In general, we can say that over
epochs, the average DCLS positions follow the shape of the scheduler used in
training.

##  12 Appendix: Effective receptive fields comparison

In the following, we show the effective receptive fields (ERF) calculated
respectively for ConvNeXt-T-dcls, ConvNeXt-T with a standard dilated kernel of
rate 2 and ConvNeXt-T models. The input crops used here are of size
$1024\times 1024$ and the heatmaps are normalized (between 0 and 1). These
ERFs are obtained by finding (via backpropagation) the input pixels that most
impact the activations of a pretrained model using samples of the dataset. We
observe that the ERF of ConvNeXt-T-dcls has a particular shape that resembles
a square with more prominent diagonals and medians. The ERF of ConvNeXt-T with
a standard dilated kernel is larger but with gridding artifacts. In all plots,
it seems that the center has more importance.

Figure 9: The effective receptive field (ERF) of the ConvNeXt-T-dcls model.

Figure 10: The effective receptive field of the ConvNeXt model with dilated
kernels (dil rate 2) instead of the dense $7\times 7$ ones.

Figure 11: The effective receptive field of the ConvNeXt-T model.

##  13 Appendix: Time measurements

We notice that when using the depthwise implicit gemm algorithm, DCLS forward
is slightly slower than the PyTorch native Conv2d forward pass, but the DCLS
backward pass is faster, in particular when increasing the model size.

Figure 12: For the 3 ConvNeXt variants (tiny, small and base), we measure the
elapsed time in ms for 1 forward + backward pass with a fixed batch size 128
and inputs of size (3,224,224) using DCLS2d convolution accelerated by the
depthwise implicit gemm algorithm. Measures were carried using a single
A100-80gb gpu. We also compare those timings to the 3 ConvNeXt baselines.

When not using the depthwise implicit gemm algorithm but the PyTorch native
Conv2d in DCLS, the forward and backward are about twice slower.

Figure 13: For the 3 ConvNeXt variants (tiny, small and base), we measure the
elapsed time in ms for 1 forward + backward pass with a fixed batch size 128
and inputs of size (3,224,224) using DCLS2d convolution with the PyTorch
native 2D convolution algorithm. Measures were carried using a single
A100-80gb gpu. We compare those timings to the 3 ConvNeXt baselines. We also
compare those timings to the 3 ConvNeXt baselines.

We notice that when using the depthwise implicit gemm algorithm, the DCLS
construction algorithm is negligible in time compared to the convolution
method for large input map sizes. For small map sizes (example 7x7), the DCLS
construction algorithm takes as much time as the convolution.

Figure 14: For different dilated kernel sizes (ranging from 7 to 31), and 4
different map sizes, we measure the elapsed time in ms for 1 forward +
backward pass with a fixed batch size 128 and a fixed kernel count 34 using a
single DCLS2d construct module with a 2D convolution accelerated by the
depthwise implicit gemm algorithm. Measures were carried using a single Quadro
RTX 8000 gpu.

When not using the depthwise implicit gemm algorithm but the PyTorch native
Conv2d in DCLS, the construction algorithm time is significantly lower than
the convolution time even for small input map sizes. This is due to the fact
the convolution time with the PyTorch native Conv2d is significantly higher
than the one with the depthwise implicit gemm algorithm.

Figure 15: For different dilated kernel sizes (ranging from 7 to 31), and 4
different map sizes / channels, we measure the elapsed time in ms for 1
forward + backward pass with a fixed batch size 128 and a fixed kernel count
34 using a single DCLS2d construct module with the PyTorch native 2D
convolution algorithm. Measures were carried using a single Quadro RTX 8000
gpu.


11institutetext: †Dalian University of Technology ‡ The University of Adelaide

# HyperFusion-Net: Densely Reflective Fusion for Salient Object Detection

Pingping Zhang†‡ Huchuan Lu† Chunhua Shen‡
Prof. Lu is the corresponding author.

###### Abstract

Salient object detection (SOD), which aims to find the most important region
of interest and segment the relevant object/item in that area, is an important
yet challenging vision task. This problem is inspired by the fact that human
seems to perceive main scene elements with high priorities. Thus, accurate
detection of salient objects in complex scenes is critical for human-computer
interaction. In this paper, we present a novel feature learning framework for
SOD, in which we cast the SOD as a pixel-wise classification problem. The
proposed framework utilizes a densely hierarchical feature fusion network,
named _HyperFusion-Net_ , automatically predicts the most important area and
segments the associated objects in an end-to-end manner. Specifically,
inspired by the human perception system and image reflection separation, we
first decompose input images into reflective image pairs by content-preserving
transforms. Then, the complementary information of reflective image pairs is
jointly extracted by an interweaved convolutional neural network (ICNN) and
hierarchically combined with a hyper-dense fusion mechanism. Based on the
fused multi-scale features, our method finally achieves a promising way of
predicting SOD. As shown in our extensive experiments, the proposed method
consistently outperforms other state-of-the-art methods on seven public
datasets with a large margin.

###### Keywords:

Salient Object Detection $\cdot$ Image Reflection Separation $\cdot$ Multiple
Feature Fusion $\cdot$ Convolutional Neural Network

##  1 Introduction

Salient object detection (SOD) aims to detect and segment the attractive
objects to human observers in an image, without any prior knowledge of image
content. It is widely used as a fundamental and useful pre-processing method
for numerous object-related applications, including image compression [1],
information retrieval [2, 3], semantic segmentation [4] and photo editing [5].

In the past decades, a large amount of SOD methods have been proposed [6].
Most of these methods adopt handcrafted visual features in detection. Color
feature is explored in various means, such as color contrast and correlation,
because human vision system is highly sensitive to color information [7].
Location cue, especially center-bias, is also frequently used to improve
saliency detection performance, for people prefer to locate the salient
objects near the center position when taking a photo [8].

|  |  |  |  |
---|---|---|---|---|---
Figure 1: Examples of pixel-wise saliency prediction. (a) Input image. (b)
Ground-truth. (c) Color cue [9]. (d) Location cue [10]. (e) Deep feature [11].
(f) Hyper-Fusion feature.

Recently, with the advances of deep learning, learned features as saliency
cues are frequently used for SOD, since learned features have strong ability
to successfully avoid the drawbacks of handcrafted features. However, using a
single cue only provides partial information of salient objects, which may
lead to inaccurate detection results. As shown in Fig. 1 (c)-(e), one can find
that the saliency maps generated by a single cue may omit some salient regions
(Fig. 1 (c)-(d)) or bring in insignificant regions (Fig. 1 (e)). Hence, it is
reasonable to combine multiple cues to improve SOD results (Fig. 1 (f)).

Recent works [11, 12, 13, 14, 15] also show that SOD with multi-scale features
generally achieves better performance than that with single-scale one.
Different features can represent different characteristics of salient objects,
and utilizing different features effectively will have positive effects on
SOD. Meanwhile, the advances of deep convolutional neural networks (CNN)
enable researchers to develop various SOD methods to cooperate with multiple
features. However, even if these methods achieve very encouraging
performances, there still exist some intrinsic problems. Firstly, these
methods directly encode the multi-scale features over the original input
images, by which way human perception information is ignored and the SOD
performance can be compromised. Moreover, when the training data increases in
number, the jointly-encoding process can be very time-consuming. Thirdly,
these methods ignore some semantic relationships among the features, which can
boost the SOD performance. Thus, coarsely utilizing all the features not only
adds extra computation burden, but also prevents further improvement.

To address above issues, we cast SOD as a pixel-wise classification task, and
propose to solve complementary feature extraction and saliency region
classification within a unified framework, as illustrated in Fig. 2. We fuse
the multi-scale features into a more preferable presentation, which is more
compact and discriminative for better SOD performance. Specifically, inspired
by the human perception system and image reflection separation, we first
decompose input images into reflective image pairs by content-preserving
transforms. Then, we design an interweaved CNN (ICNN) which consists of two
weight-stitching branches and one hyper-fusion branch. The complementary
features of reflective image pairs are jointly extracted by the proposed ICNN
and hierarchically combined with a hyper-dense fusion mechanism. Based on the
fused multi-scale features, our method finally achieves a promising way of
predicting SOD in an end-to-end manner. In this manner, our proposed model
sufficiently captures the clear boundaries and spatial contexts of salient
objects, hence significantly boosts the performance of SOD. We evaluate our
model by comparing it with other state-of-the-art approaches on seven public
benchmarks, and the experimental results demonstrate the effectiveness of our
approach.

In summary, our contributions are three folds:

  * •

We present a novel network architecture, _i.e., HyperFusion-Net_ , which is
specifically designed to learn complementary visual features in a fusing view
and predict accurate saliency maps with human perception mechanism.

  * •

We propose a hyper-dense fusion method to diversify the contributions of
multi-scale features from global and local perspectives. This fusion method is
able to to learn clear object boundaries and spatially consistent saliency.

  * •

Extensive experiments on seven large-scale saliency benchmarks demonstrate
that the proposed approach achieves superior performance and outperforms the
very recent state-of-the-art methods by a large margin.

Figure 2: An overview of our SOD approach based on the VGG-16 model [16].
Bottom: The weight-stitching branch for the reflected image. Top: The weight-
stitching branch for the transmitted image. Middle: The hyper-fusion branch to
densely fuse the multi-level features. More details can be found in the main
text.

##  2 Related Work

###  2.1 Salient Object Detection

Over the past two decades, a large mount of SOD methods have been developed.
The majority of existing methods are based on hand-crafted features. A
complete survey of these methods is beyond the scope of this paper and we
refer the readers to a recent survey paper [6] for details. Here, we mainly
focus on discussing recent methods based on deep learning architectures.

Recent years, deep learning based methods have achieved solid performance
improvements in SOD. For example, Wang _et al_ [17] integrate both local pixel
estimation and global proposal search for SOD by training two deep neural
networks. Zhao _et al_ [18] propose a multi-context deep CNN framework to
benefit from the local context and global context of salient objects. Li _et
al_ [19] employ multiple deep CNNs to extract multi-scale features for
saliency prediction. Then they propose a deep contrast network to combine a
pixel-level stream and segment-wise stream for saliency estimation [20].
Inspired by the great success of fully convolutional networks (FCNs) [21],
Wang _et al_ [11] develop a recurrent FCN to incorporate saliency priors for
more accurate saliency map inference. Liu _et al_ [13] also design a deep
hierarchical network to learn a coarse global estimation and then refine the
saliency map hierarchically and progressively. Then, Hou _et al_ [15]
introduce dense short connections to the skip-layers within the holistically-
nested edge detection (HED) architecture [22] to get rich multi-scale features
for SOD. Zhang _et al_ [14] propose a bidirectional learning framework to
aggregate multi-level convolutional features for SOD. And they also develop a
novel dropout to learn the deep uncertain convolutional features to enhance
the robustness and accuracy of saliency detection [23]. Wang _et al_ [24]
provide a stage-wise refinement framework to gradually get accurate saliency
detection results. Despite these approaches employ powerful CNNs and make
remarkable success in SOD, there still exist some obvious problems. For
example, most existing methods are based on the direct supervised learning and
ignore human perception mechanism. And the fusing strategies of multiple
features are sparse and insufficient. As a result, there is still a large
space for performance improvements. We argue that a dense fusion framework
with diversified fusion points and more adaptive fusion paths is in demand,
which not only facilitates the gradient-based optimization process, but also
provides a platform for incorporating a multi-scale understanding into the
fusion process.

Figure 3: Different network structures for deep feature fusion.

###  2.2 Deep Feature Fusion

Recently, deep CNNs have been successfully applied to various computer vision
tasks due to its power in exploring multi-level representations. Encouraged by
its strengths, researchers [11, 12, 13, 17, 18, 19, 20, 25] start to leverage
CNNs to fuse multi-level or multi-cue features automatically for performance
improvement. The information from different sources is typically combined with
the input fusion or early fusion or late fusion stage (shown in Fig. 3 (a),
(b), (c), respectively) via a single fusion point. Some more ad-hoc fusion
methods [14, 15] (shown in Fig. 3 (d)) have also been introduced by
considering the relationship between different scales and levels.
Unfortunately, they do not go beyond the traditional philosophy for feature
fusion, which means applying existing standard methods to multiple features
separately and then fusing their results in the decision stage. To sum up,
though encouraging results have been achieved, the fusion methods in previous
models are typically focalized in sparse points, which may be deficient to
merge all the useful information from multiple sources. As a result, the
fusion process is brute-force and insufficient. Different from previous
methods, we argue that dense points (shown in Fig. 3 (e)) can be applied to
the feature fusion problem to enrich the fusion process, while few works take
this fact into account. Moreover, we observe that human vision system
comprehends a scene in a coarse-to-fine way [26], which includes coarse
understanding for identifying the location and shape of the target object, and
fine capturing for exploring its detailed parts. Similarly, the feature fusion
also needs the collaborations of coarse and fine perspectives. Thus, in this
paper we fuse the multi-scale features in the coarse-to-fine manner.

##  3 Proposed Model

Fig. 2 illustrates the overall flowchart of our SOD method. Inspired by human
vision system, we first convert an input RGB image into a reflective image
pair, _i.e._ , the transmitted image (T-Input) and the reflected image
(R-Input), by utilizing content-preserving transforms. Then the image pair is
fed into the weight-stitching branches of our proposed ICNN, extracting multi-
level deep features. Afterwards, the hyper-fusion branch hierarchically
integrates the complementary features into the same resolution of input
images. Finally, the saliency map is predicted by exploiting integrated
features. In the following subsections, we will elaborate the proposed image
separation, ICNN architecture and the hyper-fusion method in detail.

###  3.1 Content-preserving Image Separation

Essentially, human vision system understands environments from 3D perception.
Image separation plays an important role in the perception process [27, 28,
29]. When image scenes are separated adequately, existing computer vision
algorithms can better understand image contents since other irrelevant
backgrounds are decreased. Motivated by this fact, we resolve the SOD problem
in the simple human perception and image separation views. To be specific, we
pose the image separation as a content-preserving image transformation task,
for which we transform an input image into different visual domains. We first
convert the original RGB image $X_{O}\in R^{W\times H\times 3}$ to a
reflective image pair by the following specular reflection function,

$\displaystyle Sep(X_{O},k)$ $\displaystyle=(X_{O}-E,\phi(X_{O}-E,k))),$ (1)
$\displaystyle=(X_{O}-E,-k(X_{O}-E))$ (2) $\displaystyle=(X_{T},X^{k}_{R}).$
(3)

where $\phi$ is a content-preserving transformer, $k$ is a hyperparameter to
control the reflection scale and $E\in R^{W\times H\times 3}$ is the mean of
an image or image dataset. To reduce the computation, in this paper we use
$k=1$ and the mean of the ImageNet dataset [30]. From above equations, one can
see that the converted image pair, _i.e._ , $X_{T}$ and $X^{k}_{R}$, is
reciprocal with a reflection plane. In detail, the reflection scheme is a
pixel-wise negation operator, allowing the given images to be reflected in
both positive and negative directions while maintaining the same content of
images, as shown in Fig. 2. In the proposed reflection, we use the scale
operator to implement the reflection, however, it is not the only feasible
method. For example, the reflection can be other non-linear operators, such as
quadratic form, exponential transform and logarithmic transform, to add more
diversity. By transforming images, the proposed algorithm makes a key
difference from previous SOD methods as plausible reflected scenes can be
obtained, which are based on the optical aberration and human perception. In
addition, different from previous image separation methods [28, 29, 31], our
method does not rely on a certain approximation model of the reflection as it
may restrict the algorithm to a specific case. Instead, we leverage the fact
that an observed image contains contents of the transmitted scene and
reflected scene. It leads us to model an observed image using a feature space
instead of a pixel-level combination. The network can also be trained in a
multi-source manner by taking transmitted and reflected images as input.

###  3.2 Joint Feature Extraction by ICNN

In order to extract the complementary information from the separated views, we
propose an interweaved CNN which consists of two weight-stitching branches to
extract multi-level features and one hyper-fusion branch to combine them. More
specifically, we build the two weight-stitching branch, following the VGG-16
model [16]. Each weight-stitching branch has 13 convolutional layers (kernel
size = $3\times 3$, stride size = 1) and 4 max pooling layers (pooling size =
$2\times 2$, stride = 2). For notational simplicity, we refer to the ConvNet
as a function $f_{CNN}(X;\theta)$, that takes $X$ as input and $\theta$ as
parameters. The ICNN output multi-level feature maps with different sizes as
the representations of the input image pair generated from above content-
preserving transforms. We denote the joint feature extraction process as
follows:

$\displaystyle\\{f^{l}_{T},f^{l}_{R}\\}$
$\displaystyle=\\{f^{l}_{ICNN}(X_{T};\theta^{conv}_{ws},\theta^{bn}_{T}),f^{l}_{ICNN}(X_{R};\theta^{conv}_{ws},\theta^{bn}_{R})\\},$
(4)

where $f^{l}_{T}$ and $f^{l}_{R}$ denote the $l$-layer feature representation
of images $X_{T}$ and $X_{R}$, respectively. {$\cdot$,$\cdot$} is the
concatenation operator in channel-wise. $\theta^{conv}_{ws}$ are the shared
parameters of the convolutional layers in the two weight-stitching branches.
Note that, the weight-stitching branches are designed to share weights in
convolutional layers, but with the adaptive batch normalization (AdaBN) [32].
In other words, we keep the weights of corresponding convolutional layers of
the two weight-stitching branches the same, while use different learnable BN
(_i.e._ , $\theta^{bn}_{T}$ and $\theta^{bn}_{T}$) between the convolution and
ReLU operators [14]. The main reason of this design is that after performing
the content-preserving transform, the reflective images have different image
domains. Domain related knowledge heavily affects the statistics of BN layers.
In order to learn domain invariant features, it is beneficial for each domain
to keep its own BN statistics in each layers. Through the two weight-stitching
branches, our model learns two complementary groups of features that we
successively leverage for the hyper-dense feature fusion. In addition,
according to the philosophy introduced in [33], the proposed architecture
learns two sets of complementary features more discriminative thanks to the
different transmitted and reflected modalities.

###  3.3 Hyper-Densely Hierarchical Fusion

Inspired by the recent success of DenseNets [34], we leverage a novel hyper-
densely connected pattern to address the feature fusion problem. To be
specific, we propose a hyper-dense architecture, named H-Fusion (Hyper-Densely
Hierarchical Fusion), for the multi-level features of image pairs, as shown in
Fig. 3 (e). In DenseNets, connectivity in each block follows a pattern that
iteratively concatenates all feature outputs in a feed-forward manner, _i.e._
,

$\displaystyle f^{l}$
$\displaystyle=g(\\{f^{1}_{CNN}(X;\theta),f^{2}_{CNN}(X;\theta),...,f^{l-2}_{CNN}(X;\theta),f^{l-1}_{CNN}(X;\theta)\\}),$
(5)

where $g$ is the fusion function, typically a convolution followed by a non-
linear activation function. Unlike DenseNets, where dense connections are
employed through all the layers in a single stream, we exploit the concept of
dense connectivity in a multi-source image setting. Each information source is
integrated through its dedicated module and the extracted descriptors are then
concatenated to perform the final classification. In this scenario, dense
connections occur not only between layers within the same path, but also
between layers in different paths. Formally, the hyper-dense fusion
architecture is defined by

$\hat{f}^{l}=\left\\{\begin{aligned}
&g(\\{f^{l}_{T},\hat{f}^{l+1},f^{l}_{R}\\};\theta_{hf}),~{}\underline{L}_{m}\leq
l<\overline{L}_{m}\\\
&g(\\{f^{l}_{T},f^{l}_{R}\\};\theta_{hf}),~{}l=\overline{L}_{m}\end{aligned}\right.$
(6) $\displaystyle\tilde{f}^{l}$
$\displaystyle=h(\\{\hat{f}^{\underline{L}_{1}},\hat{f}^{\underline{L}_{2}},...,\hat{f}^{\underline{L}_{m-1}},\hat{f}^{\underline{L}_{m}};\theta_{hf}\\}),m\in
M,$ (7)

where $\hat{f}^{l}$ and $\tilde{f}^{l}$ are integrated features at the $l$-th
layer with the same and different resolution, respectively. $\theta_{hf}$ is
the parameter of the hyper-fusion branch. $\underline{L}_{m}$ and
$\overline{L}_{m}$ are the layer bound of the $m$-th block. $h$ denotes the
integration operator, which is a $1\times 1$ convolutional layer followed by a
deconvolutional layer, to ensure the same resolution. Setting up the fusion
process in Equ. 6, which takes both data sources into account at the same
time, ensures that we can merge complementary and useful features for SOD. In
other words, our H-Fusion considers a more sophisticated connectivity pattern
that also links the output from layers in different streams, each one
associated with a different image modality. In addition, to preserve the
spatial structure and enhance the contextual information, we integrate the
multi-level reflection features in a hierarchical manner, quite different from
the DenseNets.

Based on the fused features, we adhere an additional convolutional layer to
the ICNN for the saliency map prediction. The numbers in Fig. 2 illustrate the
detailed filter setting in each convolutional layer.

###  3.4 Network Training and Testing

Given a training dataset $S=\\{(X_{n},Y_{n})\\}^{N}_{n=1}$ with $N$ training
pairs, where $X_{n}=\\{x^{n}_{i},i=1,...,T\\}$ and
$Y_{n}=\\{y^{n}_{i},i=1,...,T\\}$ are the input image and the binary ground-
truth image with $T$ pixels, respectively. $y^{n}_{i}=1$ denotes the
foreground pixel and $y^{n}_{i}=0$ denotes the background pixel. For notional
simplicity, we subsequently drop the subscript $n$ and consider each image
independently.

In most of existing deep learning based SOD methods, the loss function used to
train the network is the standard pixel-wise binary cross-entropy (BCE) loss:

$\displaystyle\mathcal{L}_{bce}=-\sum_{i\in
Y_{+}}\text{log~{}Pr}(y_{i}=1|X;\theta)-\sum_{i\in
Y_{-}}\text{log~{}Pr}(y_{i}=0|X;\theta).$ (8)

where $\theta$ is the parameter of the overall network.
Pr$(y_{i}=1|X;\theta)\in[0,1]$ is the confidence score of the network
prediction that measures how likely the pixel belong to the foreground.
$Y_{+}$ and $Y_{-}$ denote the foreground and background pixel sets in ground
truth, respectively.

However, for a typical natural image, the class distribution of salient/non-
salient pixels is heavily imbalanced: most of the pixels in the ground truth
are non-salient. To automatically balance the loss between positive/negative
classes, we introduce a class-balancing weight $\beta$ on a per-pixel term
basis, following [22]. Specifically, we define the following weighted cross-
entropy loss function,

$\displaystyle\mathcal{L}_{wbce}=-\beta\sum_{i\in
Y_{+}}\text{log~{}Pr}(y_{i}=1|X;\theta)-(1-\beta)\sum_{i\in
Y_{-}}\text{log~{}Pr}(y_{i}=0|X;\theta).$ (9)

The loss weight $\beta=|Y_{-}|/|Y|$, and $|Y_{+}|$ and $|Y_{-}|$ denote the
foreground and background pixel number, respectively. In addItion, it is also
crucial to preserve the overall spatial structure of salient objects. Thus, we
also minimize the structure perceptual (SP) loss [35],

$\displaystyle\mathcal{L}_{sp}=\sum_{l=1}^{L}\lambda_{l}||\phi_{l}(Y;w)-\phi_{l}(\hat{Y};w)||_{2},$
(10)

where $\phi_{l}$ denotes the output of the $l$-th convolutional layer in a
CNN, $\hat{Y}$ is the overall prediction, $w$ is the parameter of a pre-
trained CNN and $\lambda_{l}$ is the trade-off parameter, controlling the
influence of the loss in the $l$-th layer. In this work, we use the first four
convolutional layers of the VGG-16 model to calculate the SP loss between the
ground-truth and the prediction. The proposed loss
($\mathcal{L}_{wbce}+\mu\mathcal{L}_{sp}$) is continuously differentiable, so
we can use the standard stochastic gradient descent (SGD) method to obtain the
optimal parameters.

For saliency inference, we take the two paired images as input. The saliency
map is computed based on the output probabilities ($\textbf{s}_{0}$ and
$\textbf{s}_{1}$) of each pixel with the softmax activation, which is denoted
as:

$\displaystyle\text{Pr}(y_{j}=1|X;\theta)=\frac{exp(\textbf{s}_{1})}{exp(\textbf{s}_{0})+exp(\textbf{s}_{1})}.$
(11)

##  4 Experiments

In this section, we first introduce the experimental setups, including
datasets and evaluation metrics. Then we present the implementation details of
our proposed approach. Finally, we perform a series of experiments to
thoroughly investigate the performance and impact of our proposed methods.

###  4.1 Experimental Setups

Datasets.

To train our model, we follow previous works [11, 14] and adopt the MSRA10K
[6] dataset, which has 10,000 training images with high quality pixel-wise
saliency annotations. Most of images in this dataset have a single salient
object. To make the model robust to the image translation variation and combat
the over-fitting, we augment this dataset by random cropping and mirror
reflection, producing 120,000 training image pairs totally.

For the performance evaluation, we evaluate the proposed method and compare
our results with other state-of-the-art approaches on seven public datasets,
described as follows: DUT-OMRON [36] dataset has 5,168 high quality images.
Each image in this dataset has one or more objects with relatively complex
backgrounds. DUTS-TE dataset is the test set of currently largest saliency
detection benchmark (DUTS) [37]. It contains 5,019 images with high quality
pixel-wise annotations. ECSSD [38] dataset contains 1,000 natural images, in
which many semantically meaningful and complex structures are included. HKU-
IS-TE [19] dataset has 1,447 images with pixel-wise annotations. Images of
this dataset are well chosen to include multiple disconnected objects or
objects touching the image boundary. PASCAL-S [39] dataset is generated from
the PASCAL VOC [40] dataset and contains 850 natural images with segmentation-
based masks. SED [41] dataset has two non-overlapped subsets, _i.e._ , SED1
and SED2. SED1 has 100 images each containing only one salient object, while
SED2 has 100 images each containing two salient objects. SOD [42] dataset has
300 images, in which many images contain multiple objects either with low
contrast or touching the image boundary.

Evaluation Metrics.

To evaluate the performance of varied SOD algorithms, we adopt four metrics,
including the widely used precision-recall (PR) curves, F-measure, mean
absolute error (MAE) [6] and recently proposed S-measure [43]. The PR curve of
a specific dataset exhibits the mean precision and recall of saliency maps at
different thresholds. The F-measure is a weighted mean of average precision
and average recall, calculated by

$F_{\eta}=\frac{(1+\eta^{2})\times Precision\times Recall}{\eta^{2}\times
Precision\times Recall}.$ (12)

We set $\eta^{2}$ to be 0.3 to weigh precision more than recall as suggested
in [6].

For fair comparison on non-salient regions, we also calculate the mean
absolute error (MAE) by

$MAE=\frac{1}{W\times H}\sum_{x=1}^{W}\sum_{y=1}^{H}|S(x,y)-G(x,y)|,$ (13)

where $W$ and $H$ are the width and height of the input image. $S(x,y)$ and
$G(x,y)$ are the pixel values of the saliency map and the binary ground truth
at $(x,y)$, respectively.

To evaluate the spatial structure similarities of saliency maps, we also
calculate the S-measure [43], defined as

$S_{\lambda}=\lambda*S_{o}+(1-\lambda)*S_{r},$ (14)

where $\lambda\in[0,1]$ is the balance parameter, and is set as 0.5 typically.
$S_{o}$ and $S_{r}$ are the object-aware and region-aware structural
similarity, respectively.

###  4.2 Implementation Details

The proposed model is implemented on the widely used deep learning framework,
the Caffe toolbox [44], with the MATLAB 2016 platform. We train and test our
methods in a quad-core PC machine with an NVIDIA TITAN 1070 GPU (with 8G
memory) and an i5-6600 CPU. Following [14, 23], we perform training with the
augmented training images from the MSRA10K dataset. And we do not use any
validation sets and train the model until its training loss converges. The
input image is uniformly resized into $384\times 384\times 3$ pixels and
subtracted the ImageNet mean [30]. The weights of weight-stitching branches
are initialized from the VGG-16 model [16]. For the fusing branch, we
initialize the weights by the “msra” method. During the training, we use
standard SGD method for updating the weights of the network, with a batch size
12, momentum 0.9 and weight decay 0.0005. We set the base learning rate to
1e-8 and decrease the learning rate by 10% when training loss reaches a flat.
In addition, we set $\mu=0.01$ to optimize the loss function for our
experiments without further tuning. The training process converges after 8
epoches. When testing, our proposed SOD algorithm runs at about 6.7 fps. The
source code will be made publicly available.

###  4.3 Comparisons with the State of the Art

To fully evaluate the detection performance, we compare our proposed method
with other 14 state-of-the-art ones, including 10 deep learning based
algorithms (AMU [14], DCL [20], DHS [13], DS [12], ELD [25], LEGS [17], MCDL
[18], MDF [19], RFCN [11], UCF [23]) and 4 outstanding conventional algorithms
(BL [45], BSCA [9], DRFI [42], DSR [10]). For fair comparison, we use the
detection results or original codes provided by authors with default setting.
And we report the results in Tab. 1-2 and Fig. 4-5.

Quantitative Results.

As illustrated in Tab. 1, Tab. 2 and Fig. 4, our model achieves the best
performance on most datasets. Deep learning based methods achieve much better
performance than traditional methods.

| DUT-OMRON | DUTS-TE | ECSSD | HKU-IS-TE
---|---|---|---|---
Methods | $F_{\eta}$ | $MAE$ | $S_{\lambda}$ | $F_{\eta}$ | $MAE$ | $S_{\lambda}$ | $F_{\eta}$ | $MAE$ | $S_{\lambda}$ | $F_{\eta}$ | $MAE$ | $S_{\lambda}$
Ours | 0.701 | 0.084 | 0.784 | 0.722 | 0.075 | 0.812 | 0.886 | 0.050 | 0.903 | 0.880 | 0.037 | 0.912
AMU | 0.647 | 0.098 | 0.771 | 0.682 | 0.085 | 0.796 | 0.868 | 0.059 | 0.894 | 0.843 | 0.050 | 0.886
DCL | 0.684 | 0.157 | 0.743 | 0.714 | 0.150 | 0.785 | 0.829 | 0.149 | 0.863 | 0.853 | 0.136 | 0.859
DHS | – | – | – | 0.724 | 0.066 | 0.809 | 0.872 | 0.060 | 0.884 | 0.854 | 0.053 | 0.869
DS | 0.603 | 0.120 | 0.741 | 0.632 | 0.091 | 0.790 | 0.826 | 0.122 | 0.821 | 0.787 | 0.077 | 0.854
ELD | 0.611 | 0.092 | 0.743 | 0.628 | 0.098 | 0.749 | 0.810 | 0.080 | 0.839 | 0.776 | 0.072 | 0.823
LEGS | 0.592 | 0.133 | 0.701 | 0.585 | 0.138 | 0.687 | 0.785 | 0.118 | 0.787 | 0.732 | 0.118 | 0.745
MCDL | 0.625 | 0.089 | 0.739 | 0.594 | 0.105 | 0.706 | 0.796 | 0.101 | 0.803 | 0.760 | 0.091 | 0.786
MDF | 0.644 | 0.092 | 0.703 | 0.673 | 0.100 | 0.723 | 0.807 | 0.105 | 0.776 | 0.802 | 0.095 | 0.779
RFCN | 0.627 | 0.111 | 0.752 | 0.712 | 0.090 | 0.784 | 0.834 | 0.107 | 0.852 | 0.838 | 0.088 | 0.860
UCF | 0.621 | 0.120 | 0.748 | 0.635 | 0.112 | 0.777 | 0.844 | 0.069 | 0.884 | 0.823 | 0.061 | 0.874
BL | 0.499 | 0.239 | 0.625 | 0.490 | 0.238 | 0.615 | 0.684 | 0.216 | 0.714 | 0.666 | 0.207 | 0.702
BSCA | 0.509 | 0.190 | 0.652 | 0.500 | 0.196 | 0.633 | 0.705 | 0.182 | 0.725 | 0.658 | 0.175 | 0.705
DRFI | 0.550 | 0.138 | 0.688 | 0.541 | 0.175 | 0.662 | 0.733 | 0.164 | 0.752 | 0.726 | 0.145 | 0.743
DSR | 0.524 | 0.139 | 0.660 | 0.518 | 0.145 | 0.646 | 0.662 | 0.178 | 0.731 | 0.682 | 0.142 | 0.701

Table 1: Quantitative comparison with 15 methods on 4 large-scale datasets.
The best three results are shown in red, green and blue, respectively. “–”
means corresponding methods are trained on that dataset. Our method ranks
first or second.

| PASCAL-S | SED1 | SED2 | SOD
---|---|---|---|---
Methods | $F_{\eta}$ | $MAE$ | $S_{\lambda}$ | $F_{\eta}$ | $MAE$ | $S_{\lambda}$ | $F_{\eta}$ | $MAE$ | $S_{\lambda}$ | $F_{\eta}$ | $MAE$ | $S_{\lambda}$
Ours | 0.784 | 0.100 | 0.813 | 0.921 | 0.045 | 0.911 | 0.875 | 0.046 | 0.874 | 0.793 | 0.121 | 0.778
AMU | 0.768 | 0.098 | 0.820 | 0.892 | 0.060 | 0.893 | 0.830 | 0.062 | 0.852 | 0.745 | 0.144 | 0.753
DCL | 0.714 | 0.181 | 0.791 | 0.855 | 0.151 | 0.845 | 0.795 | 0.157 | 0.760 | 0.741 | 0.194 | 0.748
DHS | 0.777 | 0.095 | 0.807 | 0.888 | 0.055 | 0.894 | 0.822 | 0.080 | 0.796 | 0.775 | 0.129 | 0.750
DS | 0.659 | 0.176 | 0.739 | 0.845 | 0.093 | 0.859 | 0.754 | 0.123 | 0.776 | 0.698 | 0.189 | 0.712
ELD | 0.718 | 0.123 | 0.757 | 0.872 | 0.067 | 0.864 | 0.759 | 0.103 | 0.769 | 0.712 | 0.155 | 0.705
LEGS | – | – | – | 0.854 | 0.103 | 0.828 | 0.736 | 0.124 | 0.716 | 0.683 | 0.196 | 0.657
MCDL | 0.691 | 0.145 | 0.719 | 0.878 | 0.077 | 0.855 | 0.757 | 0.116 | 0.742 | 0.677 | 0.181 | 0.650
MDF | 0.709 | 0.146 | 0.692 | 0.842 | 0.099 | 0.833 | 0.800 | 0.101 | 0.772 | 0.721 | 0.165 | 0.674
RFCN | 0.751 | 0.132 | 0.799 | 0.850 | 0.117 | 0.832 | 0.767 | 0.113 | 0.784 | 0.743 | 0.170 | 0.730
UCF | 0.735 | 0.115 | 0.806 | 0.865 | 0.063 | 0.896 | 0.810 | 0.068 | 0.846 | 0.738 | 0.148 | 0.762
BL | 0.574 | 0.249 | 0.647 | 0.780 | 0.185 | 0.783 | 0.713 | 0.186 | 0.705 | 0.580 | 0.267 | 0.625
BSCA | 0.601 | 0.223 | 0.652 | 0.805 | 0.153 | 0.785 | 0.706 | 0.158 | 0.714 | 0.584 | 0.252 | 0.621
DRFI | 0.618 | 0.207 | 0.670 | 0.807 | 0.148 | 0.797 | 0.745 | 0.133 | 0.750 | 0.634 | 0.224 | 0.624
DSR | 0.558 | 0.215 | 0.594 | 0.791 | 0.158 | 0.736 | 0.712 | 0.141 | 0.715 | 0.596 | 0.234 | 0.596

Table 2: Quantitative comparison with 15 methods on 4 complex scene image
datasets. The best three results are shown in red, green and blue,
respectively. “–” means corresponding methods are trained on that dataset. Our
method ranks first or second.

Models | (a) ICNN-hf+$\mathcal{L}_{bce}$ | (b) ICNN+$\mathcal{L}_{bce}$ | (c) ICNN+$\mathcal{L}_{wbce}$ | (d) ICNN+$\mathcal{L}_{bce}$+$\mathcal{L}_{sp}$ | The proposed
---|---|---|---|---|---
$F_{\eta}$ | 0.832 | 0.854 | 0.876 | 0.871 | 0.886
$MAE$ | 0.098 | 0.076 | 0.054 | 0.068 | 0.050
$S_{\lambda}$ | 0.845 | 0.862 | 0.871 | 0.886 | 0.903

Table 3: Results with different loss functions on the ECSSD dataset. The best
three results are shown in red, green and blue, respectively.

From these results, we have other notable observations: (1) Compared with the
existing state-of-the-art methods, our method outperforms other competing ones
(except DHS) with a large margin on the four large-scale datasets, especially
on DUT-OMRON, ECSSD and HKU-IS-TE. (2) Our method achieves higher S-measure on
complex scene datasets, _e.g._ , the DUT-OMRON, SED and SOD datasets. We
attribute this result to our image separation method.

|  |  |
---|---|---|---
(a) DUT-OMRON | (b) DUTS-TE | (c) ECSSD | (d) HKU-IS
|  |  |
(e) PASCAL-S | (f) SED1 | (g) SED2 | (h) SOD

Figure 4: The PR curves of compared methods. Our method denotes as DRF.

(3) Without segmentation pre-training and any post-processing, such as CRF or
superpixel refinement, our method still achieves better results than DCL, ELD,
MCDL and RFCN, especially on the HKU-IS, SED and SOD datasets. In average, our
method achieves about 4% performance leap of F-measure and around 2%
improvement of S-measure, as well as around 4% decrease in MAE compared with
existing best methods. (4) Compared to the top-ranked methods, _i.e._ , AMU
and DHS, our method is inferior on the DUTS-TE and PASCAL-S datasets under
several metrics. However, our method ranks at the second place and is still
very comparable.

|  |  |  |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---|---|---
|  |  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |  |
(a) | (b) | (c) | (d) | (e) | (f) | (g) | (h) | (i) | (j) | (k) | (l)

Figure 5: Comparison of saliency maps. (a) Input images; (b) Ground truth; (c)
Ours; (d) AMU [14]; (e) DCL [20]; (f) DHS [13]; (g) DS [12]; (h) ELD [25]; (i)
MCDL [18]; (j) MDF [19]; (k) RFCN [11]; (l) UCF [23]. The results of LEGS
[17], BL [45], BSCA [9], DRFI [42] and DSR [10] can be found in the
supplemental material.

|  |  |  |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---|---|---
|  |  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |  |
(a) | (b) | (c) | (d) | (e) | (f) | (g) | (h) | (i) | (j) | (k) | (l)

Figure 6: Failure examples. (a) Input images; (b) Ground truth; (c) Ours. Top-
ranked results: (d) AMU [14]; (e) DCL [20]; (f) DHS [13]; (g) DS [12]; (h) ELD
[25]; (i) MCDL [18]; (j) MDF [19]; (k) RFCN [11]; (l) UCF [23].

Quantitative Results.

Fig. 5 provides several visual examples for qualitative comparisons. In
various challenging conditions, our method consistently outperforms other
compared methods. For example, salient objects contain inconsistent regions
(the 1th-3th row), salient objects touches the image boundaries (the 1th-4th
row), the background is complex and confusing (the 1th-2th, 4th-5th row) and
multiple salient objects (the 4th, 7th row). Our model can accurately locate
the salient objects and simultaneously capture clear object boundaries,
generating coherent and precise saliency maps effectively. In addition, we
observe that our model can highlight the salient objects under multi-contrast
and shadow cases (the 1-2th, 6th row). By contrast, other models tend to be
ineffective on these challenging conditions due to the lack of image
reflection and multi-source fusion strategies. Fig. 6 shows some failure
examples. When the salient objects have scattered details (the 1th row), our
method may detect the bulks as the salient object, but human can easily locate
the real objects. When the salient objects have varied saliency (the 2th row),
our method and other approaches simultaneously fail to detect the objects. In
addition, our method may be effected by cluttered light (the 3th row).

###  4.4 Ablation Studies

Models | Input Fusion | Early Fusion | Late Fusion | Ad-hoc Fusion | HyperFusion-Net+RGB | HyperFusion-Net+TR
---|---|---|---|---|---|---
$F_{\eta}$ | 0.804 | 0.821 | 0.855 | 0.863 | 0.852 | 0.886
$MAE$ | 0.140 | 0.129 | 0.121 | 0.074 | 0.069 | 0.050
$S_{\lambda}$ | 0.794 | 0.814 | 0.849 | 0.860 | 0.872 | 0.903

Table 4: Results with different fusion methods w/wo image reflection on the
ECSSD dataset. The best three results are shown in red, green and blue,
respectively.

With different model settings, we also evaluate the performance of main
components in our model. All models are trained on the augmented MSRA10K
dataset and share the same hyper-parameters described in subsection 4.2. Due
to the limitation of space, we only show the results on the ECSSD dataset.
Other datasets have the similar performance trend.

The effect of different losses.

Tab. 3 shows the experimental results with different losses. From the results,
we can see that the ICNN only using the channel concatenation operator without
H-Fusion (model (a)) has achieved comparable performance to most deep learning
methods. This confirms the effectiveness of reflection features. With the
H-Fusion, the resulting ICNN (model (b)) improves the performance by a large
margin. The main reason is that the fusion method introduces more
complementary information, which helps to locate the salient objects. In
addition, it is no wonder that training with the $\mathcal{L}_{wbce}$ loss
achieves better results than $\mathcal{L}_{bce}$. With the structure
perceptual loss $\mathcal{L}_{sp}$, the model achieves better performance in
terms of S-measure. When taking them together, the model achieves best results
in all evaluation metrics.

The effect of fusion methods.

To verify the benefits of fusion methods, we also compare our fusion strategy
with the methods described in Subsection 2.2. For the input fusion, we
concatenate the image pair in channel-wise and use the SegNet [46] for SOD.
For other fusion methods, we follow the practice of previous works [17, 18,
19, 20, 14], and use the VGG-16 model to build the fusion models. More details
are listed in the supplementary material. Tab. 4 shows the quantitative
results. From the results, we can see that adding fusing points can
consistently improve the SOD performance. This fact confirms our motivations
and claims.

The effect of image separation.

Tab. 4 also provides the results of the models with/without image reflection
separation. The results indicates the benefits of image reflection separation
across all the evaluation metrics. Besides, image reflection separation shows
significant improvements in S-measure. The main reason is that our reflection
separation is capable of: 1) retaining the main content and structure of RGB
images and disregarding local salient distractions; 2) highlighting local
details of the target salient object with the reflected views.

##  5 Conclusion and Future Work

In this work, we introduce a novel end-to-end feature learning framework for
SOD. Our method utilizes a densely hierarchical feature fusion network, named
_HyperFusion-Net_ , to predict the most important area and segment the
associated objects. Inspired by the human perception system, we first
decompose input images into reflective image pairs by content-preserving
transforms. Then, the complementary features of reflective image pairs can be
jointly extracted by an interweaved CNN and hierarchically combined with a
hyper-dense fusion mechanism. Based on the fused multi-scale features, our
method finally achieves a promising way of predicting SOD. Extensive
experiments demonstrate that the proposed method achieves significant
improvement over the baseline with a large margin, and performs better than
other state-of-the-art methods.

Based on the superior performance and flexibility, we plan to apply the
framework to other multi-modal applications, for example, RGB-D SOD. It is
also promising to leverage this framework to fuse other modalities, such as
image and text for image capturing, image and audio for video classification
and so on.

## References

  * [1] Hadizadeh, H., Bajic, I.V.:  Saliency-aware video compression.  TIP 23(1) (2014) 19–33
  * [2] He, J., Feng, J., Liu, X., Cheng, T., Lin, T.H., Chung, H., Chang, S.F.:  Mobile product search with bag of hash bits and boundary reranking.  In: CVPR. (2012) 3005–3012
  * [3] Gao, Y., Wang, M., Tao, D., Ji, R., Dai, Q.:  3-d object retrieval and recognition with hypergraph analysis.  IEEE TIP 21(9) (2012) 4290–4303
  * [4] Donoser, M., Urschler, M., Hirzer, M., Bischof, H.:  Saliency driven total variation segmentation.  In: ICCV. (2009) 817–824
  * [5] Chen, Y., Pan, Y., Song, M., Wang, M.:  Improved seam carving combining with 3d saliency for image retargeting.  Neurocomputing 151 (2015) 645–653
  * [6] Borji, A., Cheng, M.M., Jiang, H., Li, J.:  Salient object detection: A benchmark.  IEEE TIP 24(12) (2015) 5706–5722
  * [7] Cheng, M.M., Mitra, N.J., Huang, X., Torr, P.H., Hu, S.M.:  Global contrast based salient region detection.  TPAMI 37(3) (2015) 569–582
  * [8] Ren, T., Ju, R., Liu, Y., Wu, G.:  How important is location in saliency detection.  In: ACM ICIMCS. (2014) 10
  * [9] Qin, Y., Lu, H., Xu, Y., Wang, H.:  Saliency detection via cellular automata.  In: CVPR. (2015) 110–119
  * [10] Li, X., Lu, H., Zhang, L., Ruan, X., Yang, M.H.:  Saliency detection via dense and sparse reconstruction.  In: ICCV. (2013) 2976–2983
  * [11] Wang, L., Wang, L., Lu, H., Zhang, P., Ruan, X.:  Saliency detection with recurrent fully convolutional networks.  In: ECCV. (2016) 825–841
  * [12] Li, X., Zhao, L., Wei, L., Yang, M.H., Wu, F., Zhuang, Y., Ling, H., Wang, J.:  Deepsaliency: Multi-task deep neural network model for salient object detection.  TIP 25(8) (2016) 3919–3930
  * [13] Liu, N., Han, J.:  Dhsnet: Deep hierarchical saliency network for salient object detection.  In: CVPR. (2016) 678–686
  * [14] Zhang, P., Wang, D., Lu, H., Wang, H., Ruan, X.:  Amulet: Aggregating multi-level convolutional features for salient object detection.  In: ICCV. (2017) 202–211
  * [15] Hou, Q., Cheng, M.M., Hu, X., Tu, Z., Borji, A.:  Deeply supervised salient object detection with short connections.  In: CVPR. (2017) 3203–3212
  * [16] Simonyan, K., Zisserman, A.:  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556 (2014)
  * [17] Wang, L., Lu, H., Ruan, X., Yang, M.H.:  Deep networks for saliency detection via local estimation and global search.  In: CVPR. (2015) 3183–3192
  * [18] Zhao, R., Ouyang, W., Li, H., Wang, X.:  Saliency detection by multi-context deep learning.  In: CVPR. (2015) 1265–1274
  * [19] Li, G., Yu, Y.:  Visual saliency based on multiscale deep features.  In: CVPR. (2015) 5455–5463
  * [20] Li, G., Yu, Y.:  Deep contrast learning for salient object detection.  In: CVPR. (2016) 478–487
  * [21] Long, J., Shelhamer, E., Darrell, T.:  Fully convolutional networks for semantic segmentation.  In: CVPR. (2015) 3431–3440
  * [22] Xie, S., Tu, Z.:  Holistically-nested edge detection.  In: ICCV. (2015) 1395–1403
  * [23] Zhang, P., Wang, D., Lu, H., Wang, H., Yin, B.:  Learning uncertain convolutional features for accurate saliency detection.  In: ICCV. (2017) 212–221
  * [24] Wang, T., Borji, A., Zhang, L., Zhang, P., Lu, H.:  A stagewise refinement model for detecting salient objects in images.  In: ICCV. (2017) 4019–4028
  * [25] Lee, G., Tai, Y.W., Kim, J.:  Deep saliency with encoded low level distance map and high level features.  In: CVPR. (2016) 660–668
  * [26] Allman, J., Miezin, F., McGuinness, E.:  Stimulus specific responses from beyond the classical receptive field: neurophysiological mechanisms for local-global comparisons in visual neurons.  Annual review of neuroscience 8(1) (1985) 407–430
  * [27] Hanson, A.:  Computer vision systems.  Elsevier (1978)
  * [28] Levin, A., Weiss, Y.:  User assisted separation of reflections from a single image using a sparsity prior.  IEEE TPAMI 29(9) (2007)
  * [29] Li, Y., Brown, M.S.:  Single image layer separation using relative smoothness.  In: CVPR. (2014) 2752–2759
  * [30] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.:  Imagenet: A large-scale hierarchical image database.  In: CVPR. (2009)
  * [31] Shen, H.L., Zhang, H.G., Shao, S.J., Xin, J.H.:  Chromaticity-based separation of reflection components in a single image.  PR 41(8) (2008) 2461–2469
  * [32] Li, Y., Wang, N., Shi, J., Liu, J., Hou, X.:  Revisiting batch normalization for practical domain adaptation.  arXiv preprint arXiv:1603.04779 (2016)
  * [33] Saihui Hou, X.L., Wang, Z.:  Dualnet: Learn complementary features for image recognition.  In: ICCV. (2017) 1097–1105
  * [34] Huang, G., Liu, Z., van der Maaten, L., Weinberger, K.Q.:  Densely connected convolutional networks.  In: CVPR. (2017) 4700–4708
  * [35] Johnson, J., Alahi, A., Fei-Fei, L.:  Perceptual losses for real-time style transfer and super-resolution.  In: ECCV. (2016) 694–711
  * [36] Yang, C., Zhang, L., Lu, H., Ruan, X., Yang, M.H.:  Saliency detection via graph-based manifold ranking.  In: CVPR. (2013) 3166–3173
  * [37] Wang, L., Lu, H., Wang, Y., Feng, M., Wang, D., Yin, B., Ruan, X.:  Learning to detect salient objects with image-level supervision.  In: CVPR. (2017) 136–145
  * [38] Shi, J., Yan, Q., Xu, L., Jia, J.:  Hierarchical image saliency detection on extended cssd.  IEEE TPAMI 38(4) (2016) 717–729
  * [39] Li, Y., Hou, X., Koch, C., Rehg, J., Yuille, A.:  The secrets of salient object segmentation.  In: CVPR. (2014) 280–287
  * [40] Everingham, M., Gool, L.V., Williams, C.K.I., Winn, J.M., Zisserman, A.:  The pascal visual object classes (voc) challenge.  IJCV 88 (2010) 303–338
  * [41] Borji, A.:  What is a salient object? a dataset and a baseline model for salient object detection.  IEEE TIP 24(2) (2015) 742–756
  * [42] Jiang, H., Wang, J., Yuan, Z., Wu, Y., Zheng, N., Li, S.:  Salient object detection: A discriminative regional feature integration approach.  In: CVPR. (2013) 2083–2090
  * [43] Fan, D.P., Cheng, M.M., Liu, Y., Li, T., Borji, A.:  Structure-measure: A new way to evaluate foreground maps.  In: ICCV. (2017) 4548–4557
  * [44] Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.:  Caffe: Convolutional architecture for fast feature embedding.  In: ACMMM. (2014) 675–678
  * [45] Tong, N., Lu, H., Ruan, X., Yang, M.H.:  Salient object detection via bootstrap learning.  In: CVPR. (2015) 1884–1892
  * [46] Badrinarayanan, V., Handa, A., Cipolla, R.:  Segnet: A deep convolutional encoder-decoder architecture for robust semantic pixel-wise labelling.  arXiv preprint arXiv:1505.07293 (2015)