the algorithm must have stopped due to either $\hat{\eta}_{\hat{r}}^{+}\leq
1/2$ or $\hat{\eta}_{\hat{r}}^{-}\geq 1/2$. In either case
$\hat{\eta}_{\hat{r}}(x)$ is on the same side of $1/2$ as $\eta(x)$, so
$\hat{f}(x)=f^{*}(x)$ and (24) holds since the left hand side is equal to
zero.

Suppose on the other hand that $\hat{r}>r^{*}$. Let

$\bar{\sigma}\vcentcolon=\min_{r\in\hat{R}}\left\\{\max\left(\hat{\sigma}_{r}(x),C_{\alpha}r^{\alpha}\right)\right\\}.$
(36)

Again, since $r\mapsto\hat{\sigma}_{r}$ is non-increasing while $r\mapsto
C_{\alpha}r^{\alpha}$ is increasing, we see that the minimum in the above
expression must be attained at either $r^{*}$ or $2r^{*}$. If the minimum
occurs at $r^{*}$, we have
$\bar{\sigma}=\hat{\sigma}_{r^{*}}\geq\hat{\sigma}_{2r^{*}}$, and in the
latter case we have $\hat{\sigma}_{2r^{*}}\leq
C_{\alpha}(2r^{*})^{\alpha}=\bar{\sigma}$; it follows that we have
$[\hat{\eta}_{2r^{*}}^{-}(x),\hat{\eta}_{2r^{*}}^{+}(x)]\subset[\hat{\eta}_{2r^{*}}-2\bar{\sigma},\hat{\eta}_{2r^{*}}+2\bar{\sigma}]$.
Now, $\hat{r}>r^{*}$ means $\hat{r}\geq 2r^{*}$, and so by construction we
have

$[\hat{\eta}_{\hat{r}}^{-}(x),\hat{\eta}_{\hat{r}}^{+}(x)]\subset[\hat{\eta}_{2r^{*}}^{-}(x),\hat{\eta}_{2r^{*}}^{+}(x)],$

and and
$\eta(x)\in[\hat{\eta}_{2r^{*}}-2\bar{\sigma},\hat{\eta}_{2r^{*}}+2\bar{\sigma}]$
holds by (35), and so it follows that

$\absolutevalue{\hat{\eta}_{\hat{r}}(x)-\eta(x)}\leq 2\bar{\sigma}.$ (37)

Now, since obviously $\epsilon<r_{n}$, there exists an $r\in\hat{R}$ such that
$r\leq r_{n}<2r$, and so (36) and (37) imply

$\absolutevalue{\hat{\eta}_{\hat{r}}(x)-\eta(x)}\leq
4\left(\hat{\sigma}_{r_{n}}(x)+C_{\alpha}r_{n}^{\alpha}\right),$

and the final claim follows immediately because

$\\{\hat{f}_{\hat{r}}(X)\neq
f^{*}(X)\\}\subset\left\\{\absolutevalue{\eta(X)-\tfrac{1}{2}}\leq\absolutevalue{\hat{\eta}_{\hat{r}}(X)-\eta(X)}\right\\},$

and $x\in\mathcal{X}_{Q}$ was arbitrary. ∎

###### Proof.

(Theorem 3.) We proceed in the same way as for Theorem 1. Let $\hat{f}$ denote
the classifier given by Algorithm 1. Recall that we have set
$\hat{V}_{r}(x)\vcentcolon=|\tilde{A}_{r}(x)\cap\mathbf{X}|^{-1}$, and
$V_{r}(x)=\left(n_{P}P_{X}(\tilde{A}_{r}(x))+n_{Q}Q_{X}(\tilde{A}_{r}(x))\right)^{-1}$,
and we let
$\Phi\vcentcolon=\Phi_{r_{n}}(X)=\\{\hat{V}_{r_{n}}(X)<2V_{r_{n}}(X)\\}.$
Further, let

$\Omega_{\lambda}\vcentcolon=\\{V_{r_{n}}(X)\leq\lambda\\}.$

Applying Lemma 26 with expectations taken on the event $\Phi$, we see that
(recalling that we write $M(x)=\absolutevalue{\eta(x)-\tfrac{1}{2}}$)

$\displaystyle\mathbb{E}[\mathcal{E}(\hat{f})]$ $\displaystyle\leq
2\mathbb{E}\left[M(X)\cdot{\mathbbold 1}\left\\{M(X)\leq
2\frac{C_{n,\delta}}{\sqrt{|\tilde{A}_{r_{n}}(X)\cap\mathbf{X}|}}\right\\}\cdot{\mathbbold
1}\\{\Phi\\}\right]$ $\displaystyle\quad+2\mathbb{E}\left[M(X)\cdot{\mathbbold
1}\left\\{M(X)\leq 2C_{\alpha}r_{n}^{\alpha}\right\\}\cdot{\mathbbold
1}\\{\Phi\\}\right]+\mathbb{E}\left[{\mathbbold
1}\\{\Phi^{\mathsf{c}}\\}{\mathbbold 1}\\{\Omega_{\lambda}\\}+{\mathbbold
1}\\{\Omega_{\lambda}^{\mathsf{c}}\\}\right].$

By Lemmas 15,17 and 22, we can bound the four terms in the above display,
yielding

$\displaystyle\mathbb{E}[\mathcal{E}(\hat{f})]\leq
C\left(r_{n}^{\alpha(\beta+1)}+\left[C_{n,\delta}^{2}\min\left(\frac{r_{n}^{-\gamma}}{n_{P}},\frac{r_{n}^{-d}}{n_{Q}}\right)\right]^{\frac{\beta+1}{\beta+2}}+\exp\\{-\lambda^{-1}/8\\}+\frac{1}{\lambda}\min\left(\frac{r_{n}^{-\gamma}}{n_{P}},\frac{r_{n}^{-d}}{n_{Q}}\right)\right)+\delta.$

Setting $\delta\vcentcolon=\tfrac{1}{n_{P}+n_{Q}}$ gives
$C_{n,\delta}\leq\sqrt{d\log(n_{P}+n_{Q})}$, and now again we can set
$\lambda=r_{n}^{\alpha}$ and ignore the third term, which gives a final bound
of

$\displaystyle\mathbb{E}\left[\mathcal{E}(\hat{f})\right]\leq
C\min\left\\{\left(\frac{\log
n_{P}}{n_{P}}\right)^{\tfrac{\alpha(\beta+1)}{2\alpha+\alpha\beta+\gamma}},\left(\frac{\log
n_{Q}}{n_{Q}}\right)^{\tfrac{\alpha(\beta+1)}{2\alpha+\alpha\beta+d}}\right\\},$

for some $C>0$ free of $n_{P},n_{Q}$, and this completes the proof.

∎

As noted in Section 6, the procedure which we have implemented in the
experiments, in which the nodes of the dyadic trees are split successively
into two children (rather than $2^{d}$) comes with the same guarantees. We
formalize this here. Let $\Pi_{\epsilon}$ denote an arbitrary cyclical dyadic
partition of $[0,1]^{D}$ down to $\epsilon$-scale; that is, beginning with the
node $[0,1]^{D}$, each hyper-rectangular cell is split by choosing one of the
dimensions along which the node is largest and bisecting along that dimension,
and this procedure is repeated until all the cells have been collapsed until
the side lengths $r$ satisfy $r<\epsilon$. The regression estimates and
classifier are defined as in (12) (with envelopes of cells taken by enlarging
according to the width of the smallest side). Set
$\epsilon=(n_{P}+n_{Q})^{-1/2}$, and let $\hat{f}^{\prime}$ be the classifier
that results from applying Algorithm 1 to this tree.

######  Corollary 35.

Let $\hat{f}^{\prime}$ be as above. Then there is a $C^{\prime}>0,$ free of
$n_{P},n_{Q}$, such that

$\displaystyle\mathbb{E}\left[\mathcal{E}(\hat{f}^{\prime})\right]\leq
C^{\prime}\min\left\\{\left(\frac{\log
n_{P}}{n_{P}}\right)^{\tfrac{\alpha(\beta+1)}{2\alpha+\alpha\beta+\gamma}},\left(\frac{\log
n_{Q}}{n_{Q}}\right)^{\tfrac{\alpha(\beta+1)}{2\alpha+\alpha\beta+d}}\right\\},$

###### Proof.

The argument is identical to that used for the regular dyadic tree. Of course,
the union bound used to prove Lemma 23 will need to be taken over a larger set
of cells, but this has no effect on the result other than changing the value
of the leading constant in the rate. ∎

###  D.4 Localizing to the Decision Boundary

###### Lemma 29.

First, note that we have (recall that $\\{\hat{f}_{\epsilon}\neq
f^{*}\\}\subset\\{|\eta(x)-\tfrac{1}{2}|\leq|\hat{\eta}_{r^{+}}(x)-\eta(x)|\\}$)

$\displaystyle\mathbb{E}\,\mathcal{E}(\hat{f}_{\epsilon};\mathcal{G}_{\epsilon}^{+})$
$\displaystyle=2\,\mathbb{E}\left[M(X){\mathbbold
1}\left\\{\hat{f}_{\epsilon}\neq f^{*}\right\\}{\mathbbold
1}\\{X\in\mathcal{G}_{\epsilon}^{+}\\}\right]\leq
2\,\mathbb{E}\left[M(X)\cdot{\mathbbold
1}\left\\{M(X)\leq\absolutevalue{\hat{\eta}_{r_{+}}(X)-\eta(X)}\right\\}\cdot{\mathbbold
1}\\{X\in\mathcal{G}_{\epsilon}^{+}\\}\right]$ $\displaystyle\leq
2\,\mathbb{E}\left[M(X)\cdot{\mathbbold 1}\left\\{M(X)\leq
2\absolutevalue{\hat{\eta}_{r_{+}}(X)-\tilde{\eta}_{r_{+}}(X)}\right\\}\cdot{\mathbbold
1}\\{X\in\mathcal{G}_{\epsilon}^{+}\\}\right]$
$\displaystyle\quad+2\,\mathbb{E}\left[M(X)\cdot{\mathbbold 1}\left\\{M(X)\leq
2\absolutevalue{\tilde{\eta}_{r_{+}}(X)-\eta(X)}\right\\}\cdot{\mathbbold
1}\\{X\in\mathcal{G}_{\epsilon}^{+}\\}\right].$

Now, let $\Phi_{r}(X)$ be the event introduced in Notation 16. On
$\Phi_{r}(X)$, we have (see Lemma 13)
$\absolutevalue{\tilde{\eta}_{r_{+}}(X)-\eta(X)}\leq
C_{\alpha}r_{+}^{\alpha}$, and so $r_{+}<(\epsilon/2C_{\alpha})^{1/\alpha}$
implies that

$\displaystyle{\mathbbold 1}\left\\{M(X)\leq
2\absolutevalue{\tilde{\eta}_{r_{+}}(X)-\eta(X)}\right\\}{\mathbbold
1}\\{X\in\mathcal{G}_{\epsilon}^{+}\\}{\mathbbold 1}\\{\Phi_{r}(X)\\}=0.$

We then have

$\displaystyle\mathbb{E}\
\mathcal{E}(\hat{f}_{\epsilon};\mathcal{G}_{\epsilon}^{+})\leq
2\,\mathbb{E}\left[M(X)\cdot{\mathbbold 1}\left\\{M(X)\leq
2\absolutevalue{\hat{\eta}_{r_{+}}(X)-\tilde{\eta}_{r_{+}}(X)}\right\\}{\mathbbold
1}\\{X\in\mathcal{G}_{\epsilon}^{+}\\}\right]+\mathbb{P}(\Phi_{r}(X)^{\mathsf{c}}).$

By Lemma 22, we have

$\displaystyle\mathbb{P}(\Phi_{r}(X)^{\mathsf{c}})\leq
C\,\min\left(\frac{1}{n_{P}}r_{+}^{-\gamma^{*}},\frac{1}{n_{Q}}r_{+}^{-d}\right)\leq\left(\frac{1}{\epsilon}\right)\,C\,\min\left(\frac{1}{n_{P}}r_{+}^{-\gamma^{*}},\frac{1}{n_{Q}}r_{+}^{-d}\right)$

since $\epsilon<1$. The remaining term is bounded using the same steps as in
the proof of Lemma 17 with $t=\epsilon$, where the first term vanishes because
we integrate over $\mathcal{G}_{\epsilon}^{+}$. ∎

###### Proof.

[Theorem 32] Fix $\epsilon>0$. By Theorem 3, there is a universal
$C=C(\mathcal{T})$ such that

$\displaystyle\mathbb{E}\ \mathcal{E}(\hat{f};\mathcal{G}_{\epsilon}^{-})\leq
C\min\left(\left(\frac{\log(n_{P})}{n_{P}}\right)^{\tfrac{\alpha(\beta+1)}{2\alpha+\alpha\beta+\gamma(\epsilon)}},\left(\frac{\log(n_{Q})}{n_{Q}}\right)^{\tfrac{\alpha(\beta+1)}{2\alpha+\alpha\beta+d}}\right).$

Now, set $r_{n}^{+}=(\epsilon/4C_{\alpha})^{1/\alpha}$. Assume for now that
$\epsilon\geq 4C_{\alpha}(n_{P}+n_{Q})^{-\alpha}$. Then we have
$\tfrac{1}{n_{P}+n_{Q}}\leq r_{n}^{+}$, and so an application of Lemma 25 at
level $r_{n}^{+}$ (with $\delta=(n_{P}+n_{Q})^{-1}$ and $C_{n}=C_{n,\delta}$)
yields that, for $x\in\mathcal{G}_{\epsilon}^{+}$, we have with probability at
least $1-(n_{P}+n_{Q})^{-1}$ that

$\displaystyle{\mathbbold 1}\\{\hat{f}(x)\neq f^{*}(x)\\}\leq{\mathbbold
1}\left\\{M(x)\leq
2\left(C_{n}|\tilde{A}_{r_{n}^{+}}(x)\cap\mathbf{X}|^{-1/2}+C_{\alpha}(r_{n}^{+})^{\alpha}\right)\right\\},$

and then splitting the indicator on the right-hand side gives

$\displaystyle{\mathbbold 1}\\{\hat{f}(x)\neq f^{*}(x)\\}\leq{\mathbbold
1}\left\\{\absolutevalue{\eta(x)-\tfrac{1}{2}}\leq
4C_{n}|\tilde{A}_{r_{n}^{+}}(x)\cap\mathbf{X}|^{-1/2}\right\\}+{\mathbbold
1}\left\\{\absolutevalue{\eta(x)-\tfrac{1}{2}}\leq
4C_{\alpha}(r_{n}^{+})^{\alpha}\right\\}.$

Now, by construction we have $4C_{\alpha}(r_{n}^{+})^{\alpha}=\epsilon$, and
therefore

${\mathbbold 1}\\{x\in\mathcal{G}_{\epsilon}^{+}\\}\,{\mathbbold
1}\\{\absolutevalue{\eta(x)-\tfrac{1}{2}}\leq
4C_{\alpha}(r_{n}^{+})^{\alpha}\\}=0,$

and so by Proposition 16 it remains only to bound

$\displaystyle\mathbb{E}\left[M(X)\cdot{\mathbbold 1}\left\\{M(X)\leq
4C_{n}|\tilde{A}_{r_{n}^{+}}(X)\cap\mathbf{X}|^{-1/2}\right\\}\right],$

which we can achieve by following the exact steps used to prove Theorem 3. It
remains only to consider the case
$\epsilon<4C_{\alpha}(n_{P}+n_{Q})^{-\alpha}$. Assume $n_{P}\geq n_{Q}$. In
this case, we have

$\displaystyle\min\left(\frac{1}{n_{P}}\epsilon^{-(1+\gamma^{*}/\alpha)},\frac{1}{n_{Q}}\epsilon^{-(1+d/\alpha)}\right)\geq\frac{1}{n_{P}}\epsilon^{-(1+d/\alpha)}\geq(4C_{\alpha})^{-(1+d/\alpha)}\
n_{P}^{-1+\alpha+d},$

where we use $\gamma^{*}\geq d$. If $\alpha+d>1$, this bound exceeds $1$
provided that $n_{P}\geq(4C_{\alpha})^{\tfrac{\alpha+d}{\alpha(\alpha+d-1)}}$.
If $n_{Q}\geq n_{P}$, then using the same argument we get a bound that exceeds
$1$ when $n_{Q}\geq(4C_{\alpha})^{\tfrac{\alpha+d}{\alpha(\alpha+d-1)}}$, and
so the result holds provided

$\max(n_{P},n_{Q})\geq(4C_{\alpha})^{\tfrac{\alpha+d}{\alpha(\alpha+d-1)}}.$

∎

##  Appendix E Supplemental Experiments

In this section we present the results of some supplemental experiments. First
we carry out a comparison of our method against the dyadic-tree pruning
algorithm introduced in [35]. We then investigate the performance of our
method against the baselines considered in Section 6, except we use Algorithm
1 and the baselines to prune trees grown using the CART algorithm [9], which
are non-dyadic. Our theoretical results do not apply to this setting, but the
results nonetheless demonstrate that Algorithm 1 can improve over these
baselines in the context of non-dyadic trees.

###  E.1 Comparison to Cost-Complexity Pruning

Decision trees have long been studied in the context of classification, and
there exist numerous methods for pruning which employ techniques from
structural risk minimization. A noteworthy contribution in this direction is
that of [35], who devised an ingenious penalty term for selecting dyadic trees
that provably attain minimax optimal learning rates for the expected excess
risk under conditions similar to ours in the one-sample case.

Their method consists of penalizing a tree $T$ with leaves $\pi(T)$ according
to the following penalty:

$\displaystyle\Phi_{n}(T)=\sum_{A\in\pi(T)}\sqrt{2p_{n}(A)\frac{[[A]]\log
2+\log(2/\delta)}{n}},$ (38)

where $p_{n}(A)$ is the mass of the leaf under the sampling distribution,
$\delta\in(0,1)$ is the usual confidence parameter, and $[[A]]$ is chosen such
that $\sum_{A\in\pi(T)}2^{-[[A]]}\leq 1$, which they show can be done via the
Kraft inequality for prefix codes [13]. The penalty (38) induces _unbalanced_
trees, which can produce better approximations of the decision boundary than
balanced trees [35]. The local depth selection of Algorithm 1 is similar in
that it mimics unbalanced trees by choosing different levels of the tree for
different regions of feature space when making predictions, so we provide a
comparison against this baseline in order to verify that the advantage of our
method is not simply that it yields unbalanced trees, but rather that it does
so in a way that correctly aggregates the information added by the source data
under covariate shift.

Note that the main result of [35] relies on an application of a relative
Chernoff bound to derive (c.f. [35], Theorem 2) that with probability at least
$1-\delta$, one has

$\displaystyle\absolutevalue{R(T)-\hat{R}_{n}(T)}\leq\Phi_{n}(T)$ (39)

for all $T$ in a suitable class of dyadic decision trees, where
$R,\hat{R}_{n}$ are respectively the risk and empirical risk of the tree $T$.
Unfortunately, in the transfer learning setting in which we have access to
samples from a source $P$ and a target $Q$, extending the penalty (38) in the
obvious way does not lead to a bound of the form (39) with the target risk
$R_{Q}(T)$, but rather would feature the risk under a convex combination of
source and target $R_{\alpha P+(1-\alpha)Q}(T)$, where
$\alpha=n_{P}/(n_{P}+n_{Q})$. It may be possible to derive optimal rates for
transfer using a penalty akin to (38) by employing the risk minimization
strategy outlined in [21], although the latter studies a different notion of
discrepancy between $P$ and $Q$, and so resulting rates are not directly
comparable. We do not pursue this further here.

We consider two approaches based on the penalty (38) to further serve as
baselines.

##### Baseline 1 (SN)

We consider directly implementing the penalty of [35], with the combined
sample used to compute the penalty:

$\Phi^{(SN)}_{n}(T)=\sum_{A\in\pi(T)}\sqrt{2p_{n}(A)\frac{[[A]]\log
2+\log(2n)}{n}},$

where $n=n_{P}+n_{Q}$ and
$p_{n}(A)=\tfrac{1}{n}\sum_{i=1}^{n_{P}+n_{Q}}{\mathbbold 1}\\{X_{i}\in A\\}$.

##### Baseline 2 (SN-Q)

We implement the same penalty, using only the target samples:

$\Phi^{(SNQ)}_{n_{Q}}(T)=\sum_{A\in\pi(T)}\sqrt{2p_{n_{Q}}(A)\frac{[[A]]\log
2+\log(2n_{Q})}{n_{Q}}};$

source samples are used to build the regression estimates in each leaf once
the tree has been pruned.

In the former case we optimize $R_{n}(T)+c\,\Phi_{n}^{(SN)}(T)$ for
$T\subset\mathbf{T}$ where $R_{n}(T)$ is the empirical risk of $T$ under the
full sample and $\mathbf{T}$ is an initial cyclical dyadic tree; in the
latter, we optimize $R_{n_{Q}}(T)+c\,\Phi_{n_{Q}}^{(SNQ)}(T)$, where
$R_{n_{Q}}(T)$ is the empirical risk under the target; the constant $c$ is
tuned by cross-validation over the target sample.

Figure 5 shows the result of using the SN penalty to prune a cyclical dyadic
tree, against the uniform-depth CV baseline, and against Algorithm 1. As we
see in Figure 5, the cost-complexity pruning approaches (SN and SN-Q) do not
achieve the performance of choosing the depth via 2-fold cross-validation over
the target sample. We note that, as pointed out in [35], this method is quite
sensitive to the choice of dampening constant for the penalty, a problem which
is heavily exacerbated in the transfer settings in which there are relatively
few target samples (such as those considered here).

(a)  (b)

Figure 5: Risk estimates for our adaptive method (AD), compared to the
baseline cross-validation and the pruning method of [35] (SN, SN-Q), applied
to the Algerian Forest Fire data (left) and the MAGIC data (right). Error bars
show standard errors over 50 and 20 iterations, respectively.

###  E.2 On the Case $n_{P}=0$.

Methods such as our Algorithm 1, based on the ICI approach indebted to
Lepski’s method, are to the best of our knowledge currently the only
classifiers that are able to achieve the (log-spoiled) minimax optimal rates
that we have presented here. However, there is nothing particular to the two-
sample transfer learning problem about ICI in general or Algorithm 1 in
particular, and the procedure yields minimax optimal rates in the case
$n_{P}=0$ as well, leading us to expect that it should be at least competitive
with other methods in this case. Figure 6 demonstrates that this is true for
the Crop Mapping and MAGIC data, where we see that AD slightly outperforms the
CV-based baselines. We leave a more thorough empirical investigation of this
question to further work.

(a)  (b)

Figure 6:  Target risk estimates for three tree-depth selection methods
applied the Crop Mapping (d = 174) and MAGIC (d = 10) datasets when $n_{P}=0$.
Error bars show the standard errors over 10 iterations.

###  E.3 Beyond Dyadic Trees

Here we repeat the experiments shown in Section 6, but we implement the
various depth-selection methods on trees grown using the vanilla CART
algorithm of [9] (that is, splits are chosen to minimize the so-called Gini
impurity at each node). For the sake of variety, we consider two further
datasets: Wine Quality, and Steel Plates Faults, described in Table 1. Note
that the Wine Quality dataset has 10 classes indicating various quality
levels. We label wines with quality at least $6$ as $1$ and the rest as $0$ to
get a binary classification problem.

Although we can no longer claim the rates of Theorem 1 when Algorithm 1 is
applied to these trees, as we see in Figure 7, the adaptive (AD) performs at
least as well and sometimes significantly better than the cross-validation
based depth-selection methods, suggesting that it could be useful as a pruning
procedure even when trees are grown using more elaborate splitting criteria.
We leave further exploration of this possibility for future work.

(a)  (b)

Figure 7:  Target risk estimates for four tree-depth selection methods applied
to CART trees for the Wine Quality dataset (d = 11; left), and the Steel
Plates Faults dataset (d = 27; right). Error bars show the standard errors
over 10 iterations.

##  Appendix F On the Multiple Source Problem.

A simple extension of Lemma 20 allows us to re-express our results when there
are a multiplicity of source distributions. Suppose that we are now in a
setting in which there are $k$ source distributions $P_{1},\dots,P_{k}$, such
that each $(P_{i},Q)\in\mathcal{T}$ with transfer exponents $\gamma_{i}$ (with
the same constant $C_{\gamma}$, c.f. (8)); without loss of generality, suppose
that the target distribution is $P_{1}$. Assuming a sample of size
$n=n_{1}+\dots+n_{k}$ with $n_{i}$ samples from $P_{i}$, the natural extension
of the local variance term to multiple samples is

$V_{r}(x)=\frac{1}{\sum_{i=1}^{k}n_{i}P_{i}(\tilde{A}_{r}(x))}.$

In this case, we have

######  Lemma 36.

Consider the multiple source setting with $n_{i}$ samples from source $i$,
$P_{i}$, where $P_{i}$ has aggregate exponent $\gamma_{i}$ w.r.t. the source,
with constant $C_{\gamma}$. Then we have

$\mathbb{E}_{Q_{X}}[V_{r}(X)]\leq C_{\gamma}\,\frac{1}{n}r^{-\bar{\gamma}},$

where $\bar{\gamma}=\sum_{i=1}^{k}w_{i}\gamma_{i}$, with weights
$w_{i}=n_{i}/n$.

###### Proof.

Let $I$ denote a single draw from the multinomial distribution with
probabilities $w=(w_{1},\dots,w_{k})$. As usual, $X$ is drawn from $Q_{X}$,
$\tilde{A}_{r}(x)$ denotes the envelope of the cell containing $X$ at level
$r$. We let $P_{I}(A)$ and $\gamma_{I}$ be random variables equal to
$P_{i}(A)$ and $\gamma_{i}$ when $I=i$. We have, for $0<r<1$,

$\displaystyle\mathbb{E}_{Q_{X}}[V_{r}(X)]$
$\displaystyle=\mathbb{E}_{Q_{X}}\left[\left\\{\sum_{i=1}^{k}n_{i}P_{i}(\tilde{A}_{r}(X))\right\\}^{-1}\right]=\frac{1}{n}\,\mathbb{E}_{Q_{X}}\left[\left\\{\sum_{i=1}^{k}w_{i}P_{i}(\tilde{A}_{r}(X))\right\\}^{-1}\right]$
$\displaystyle=\frac{1}{n}\,\mathbb{E}_{Q_{X}}\left\\{\mathbb{E}_{I}P_{I}(\tilde{A}_{r}(X))\right\\}^{-1}\leq\frac{1}{n}\,\mathbb{E}_{Q_{X}}\left\\{\mathbb{E}_{I}P_{I}(\tilde{A}_{r}(X))^{-1}\right\\}$
$\displaystyle=\frac{1}{n}\,\mathbb{E}_{I}\mathbb{E}_{Q_{X}}P_{I}(\tilde{A}_{r}(X))^{-1}$
$\displaystyle=\frac{1}{n}\,\mathbb{E}_{I}\left(\sum_{A\in\pi(T_{r})}\frac{Q_{X}(A_{r}(X))}{P_{I}(\tilde{A}_{r}(X))}\right)\leq\frac{1}{n}\,\mathbb{E}_{I}\left(\sum_{A\in\pi(T_{r})}\frac{Q_{X}(\tilde{A}(X))}{P_{I}(\tilde{A}(X))}\right)$
$\displaystyle\leq\frac{1}{n}\,\mathbb{E}_{I}\,C_{\gamma}r^{-\gamma_{I}}\leq
C_{\gamma}\,\frac{1}{n}\,r^{-\mathbb{E}\gamma_{I}}=C_{\gamma}\,\frac{1}{n}\,r^{-\sum
w_{i}\gamma_{i}},$

where the inequality on the second line and the second inequality on the fifth
line are applications of Jensen’s inequality, and the first inequality on the
fifth line is an application of Lemma 19. ∎

Using this Lemma, the proofs of our upper bounds can be repeated with no
change (since the aggregate transfer exponent works its way into the rates
only via this variance bound) giving results in terms of the full sample size
$n$ and the sample-weighted average of the aggregate transfer exponents,
$\bar{\gamma}$. Unlike the one-source case bounds we have presented however,
it remains unclear whether the risk bounds for multiple sources $k\geq 2$
expressed in terms of $\bar{\gamma}$ are tight in any meaningful sense.

##  Appendix G A Pathological Example.

In this section we provide an example of non-doubling measures $P,Q$ for which
the aggregate and integrated transfer exponents are equal -
$\gamma^{*}=\rho^{*}$ \- but that these do not agree with the value that we
would obtain if we defined the exponent by taking the sum of mass ratios over
the dyadic partition of the ambient space. Suppose that we have measures $P,Q$
supported on $[0,1]^{d}$, and define

$\displaystyle\Lambda(P,Q,n)=\sum_{C\in\mathcal{D}^{n}}\frac{Q(C)}{P(C)},$

where $\mathcal{D}^{n}$ is the regular dyadic partition of $[0,1]^{d}$ of
order $n$. In analogy with how we have defined the aggregate exponent between
$P$ and $Q$, let $\gamma_{D}(P,Q)=\gamma_{D}>0$ be any constant for which
there exists a $C>0$ such that for all $n=0,1,2,\dots$ we have

$\displaystyle\Lambda(P,Q,n)\leq C\,2^{n\gamma_{D}},$

and let $\gamma^{*}_{D}$ be the minimal $\gamma_{D}$ such that the above
holds. Let $d=2$, and consider the following construction: consider two
countable sequence of boxes $\\{B_{i};\>i=1,2,\dots\\}$ and
$\\{B_{i}^{\prime};\>i=1,2,\dots\\}$, where $B_{i}$ is a square with side
length $2^{-i}$ with top-right corner at
$x_{i}=(\sum_{k=0}^{i-1}2^{-i},2^{-i})$, and $B_{i}^{\prime}$ is a square of
side-length $2^{-2i}$ with bottom-left corner at $x_{i}$; see Figure 8 below.
We take $Q=\sum_{i=1}^{\infty}Q_{i}$ and $P=P_{0}+\sum_{i=1}^{\infty}P_{i}$,
where $Q_{i},P_{i}$ are supported on $B_{i}\cup B_{i}^{\prime}$ for each
$i\geq 1$. Let $Q_{i}$ uniformly assign mass $3\cdot(1/4)^{i}$ to $B_{i}\cup
B_{i}^{\prime}$, yielding density $q_{i}$. Let $P_{i}$ be equal to $Q_{i}$ on
$B_{i}$. On $B_{i}^{\prime}$, let $P_{i}$ have density
$q_{i}\norm{x-x_{i}}^{\nu}$ for some $\nu>0$. Finally, let $P$ have an atom at
$x_{0}=(0,1)$ with all remaining mass. Note that, by calculations identical to
those from Example 2, we have $\gamma^{*}(P,Q)=\max(2,\nu)$. We proceed to
estimate $\Lambda(P,Q,n)$ for $n\geq 1$. Observe that by construction, there
is a cube $C\in\mathcal{D}^{n}$ such that $Q(C)=Q(B_{n}^{\prime})$, and
$P(C)=P(B_{n}^{\prime})$. Further, there is a constant $c_{0}$ such that
$\cup_{i=1}^{n}B_{i}$ is exactly covered by $c_{0}2^{2n}$ cubes in
$\mathcal{D}^{n}$, and by construction for each of these cubes we have
$Q(C)/P(C)=1$. Therefore we have

$\displaystyle\Lambda(P,Q,n)=\sum_{C\in\mathcal{D}^{n}}\frac{Q(C)}{P(C)}\geq
c_{o}2^{2n}+\frac{Q(B_{n}^{\prime})}{P(B_{n}^{\prime})}=c_{0}2^{-2n}+\frac{q_{n}2^{4n}}{q_{n}c(\nu)2^{2n(2+\nu)}}=c_{0}2^{2n}+c(\nu)^{-1}\,2^{n\,2\nu},$

where $c(\nu)>0$ is some constant depending only on $\nu$. It follows
immediately that $\gamma^{*}_{D}(P,Q)\geq\max(2,2\nu)$, and so for $\nu>1$ we
have $\gamma^{*}_{D}(P,Q)>\gamma^{*}(P,Q)$. In particular, this shows that if
we have used $\gamma_{D}$ to derive our results, the induced rates would not
be as sharp as those using the aggregate exponent $\gamma$. Observe also that
since $P$ is not a doubling measure, this example also serves to demonstrate
that doubling is not necessary for the equivalence in Proposition 9 to hold,
since one can straightforwardly verify that $\gamma^{*}=\rho^{*}$ in this
case.

$B_{1}$$B_{1}^{\prime}$$B_{2}$$B_{2}^{\prime}$$B_{3}$$B_{3}^{\prime}$…$x_{1}$$x_{0}$$x_{2}$$x_{3}$

Figure 8: Illustration of the support of the measures $P,Q$ described above on
the unit square $[0,1]^{2}$.


# Parameter-Free Channel Attention for Image Classification and Super-
Resolution

Yuxuan Shi, Lingxiao Yang, Wangpeng An, Xiantong Zhen, Liuqing Wang  This work
was supported in part by The National Natural Science Foundation of China (No.
62002176, 62176068, and 62171309). Corresponding author: Liuqing Wang
(wlq@nankai.edu.cn)Yuxuan Shi is with the School of Statistics and Data
Science, Nankai University, Tianjin 300071, China. (E-mail:
<EMAIL_ADDRESS>Lingxiao Yang is with the School of Computer Science and
Engineering, Sun Yat-sen University, Guangzhou 510006, China. (E-mail:
<EMAIL_ADDRESS>Wangpeng An is with Tiktok Inc, Mountain View, CA,
USA, 94041. (E-mail<EMAIL_ADDRESS>. Xiantong Zhen is with United
Imaging Healthcare, Co., Ltd., China (e-mail: zhenxt@gmail.com). Liuqing Wang
is with Nankai University, Tianjin 300071, China. (E-mail<EMAIL_ADDRESS>
###### Abstract

The channel attention mechanism is a useful technique widely employed in deep
convolutional neural networks to boost the performance for image processing
tasks, e.g., image classification and image super-resolution. It is usually
designed as a parameterized sub-network and embedded into the convolutional
layers of the network to learn more powerful feature representations. However,
current channel attention induces more parameters and therefore leads to
higher computational costs. To deal with this issue, in this work, we propose
a Parameter-Free Channel Attention (PFCA) module to boost the performance of
popular image classification and image super-resolution networks, but
completely sweep out the parameter growth of channel attention. Experiments on
CIFAR-100, ImageNet, and DIV2K validate that our PFCA module improves the
performance of ResNet on image classification and improves the performance of
MSRResNet on image super-resolution tasks, respectively, while bringing little
growth of parameters and FLOPs.

###### Index Terms:

Parameter-free channel attention, image classification, image super-resolution

##  I Introduction

Attention mechanisms [1, 2] have been validated to be very effective on
boosting the performance of deep neural networks [1, 2, 3]. Among them,
channel attention [1] is a commonly used attention mechanism that assign
adaptive weights over different feature maps to improve the performance of
image classification networks. Therefore, it has become a key component of
deep network architectures. Besides, the effectiveness of channel attention
[1] on image super-resolution has also been illustrated in the work of deep
residual channel attention network [4] and cascading residual network [5].

The parameter-free attention mechanism has been studied in [6] to assign
position importance over the feature maps by spatial attention without
parameter growth. The parameter-free attention mechanism was also studied from
the neuroscience perspective in [7]. Different from current attention
mechanism that needs to be trained from data, this work utilizes a fixed
perception mechanism to pixel-wise importance over the feature maps without
additional parameters. Compared with popular parameterized channel or spatial
attention mechanisms [1, 8], parameter free attention mechanisms could achieve
better effect [6, 7] while avoiding additional parameter growth, or reduce the
model complexity and computational costs while preserving the network
performance.

In this paper, we develop a Parameter-Free Channel Attention (PFCA) module to
replace the standard channel attention module [1] in popular image
classification and image super-resolution networks. Our PFCA module exploits
useful statistic information in different channels of a feature maps, and is a
plug-and-play module can be embedded into a network to enhance its feature
representation capability. By embedding our PFCA into ResNets and MSRResNet,
experiments on CIFAR-100, ImageNet, and DIV2K demonstrate that our PFCA module
is effective on boosting the performance of ResNet-18, ResNet-50, ResNet-101
and MSRResNet on image classification and super-resolution, respectively,
while bringing little computational costs and parameter growth.

The rest of this paper is organized as follows. In Section II, we briefly
introduce the related works. In Section III, we present our PFCA module for
image classification and super-resolution tasks. Extensive experiments are
conducted in Section IV to evaluate the performance of our PFCA module.
Section V concludes this work.

##  II Related Works

###  II-A Image Classification

Image classification has been widely tackled by convolutional neural networks
(CNNs) ever since AlexNet [9]. Then VGG [10] extends the network’s depth and
width. GoogleNet [11] provides an inception module to reduce the parameters of
convolutional networks. The design of CNN architecture could affect the
feature extraction function. In this work, we use deep residual network
(ResNet) [12], the most frequently used CNN architecture, as the backbone for
image classification tasks. It has been validated by squeeze and excitation
network (SE-Net) [1] that attention mechanism helps improve the effectiveness
of network to concentrate on special channels of feature maps. Improvements in
optimization take effect to boost the network training process [13]. The study
of feature representation has also been testified to be useful for
classification [14].

Figure 1: Illustration of our Parameter-Free Channel Attention (PFCA) module.
The module gets the feature map $\mathbf{X}$ as the input and outputs feature
map $\mathbf{Y}$. $\mathbf{U}$ and $\mathbf{V}$ are the channel level vector
generated after the global pooling layer and sigmoid activation respectively.
$\mu$ and $\sigma^{2}$ are the channel-level mean and variance of $\mathbf{U}$
and $\lambda$ is used to control variance.

###  II-B Image Super-Resolution Task

Using convolutional neural network to realize image super-resolution (SR) was
firstly proposed in SRCNN [15]. The direct use of original LR image in the
following ESPCNN [16] as input of the network instead of the upsampled image
in SRCNN is to reduce the computation load. After that, VDSR [17] introduces
the concept of residual image super-resolution , which has been proved to be
effective in SR tasks. Since then, the residual block was provided in [18] to
help design the network architecture of SR network, which was generally used
in many following works. However, these methods could hardly be applied to
practical scenarios due to their heavy computations, thus pushing the
efficient SR network coming out. Some methods are presented including depth-
wise convolutions [19] and group convolutions [20, 21, 22] in order to save
computation. The attention mechanism is also used to improve SR performance.
RCAN [4] uses channel attention module to make its network concentrate on
important channels. PAN [3] introduces a pixel attention module to adaptively
weight different pixels of feature maps.

###  II-C Attention Mechanisms

Attention mechanisms have attracted great attention since it was successfully
applied in natural language processing tasks [23] , which is mainly used to
help networks concentrating on partial information of data flow and generating
the corresponding dimensional weight maps. When it comes to the image
processing task, there are mainly three types of attention mechanisms: channel
attention (CA) [1], spatial attention (SA) [2] and pixel attention (PA) [3],
depending on their generated weight maps’ dimension level. Self-Attention [24]
is also very popular apart from the three. Among the above mechanisms, channel
attention is the first to propose in SE-Net [1] and then polished in SK-Net
[25]. After that, attempts to incorporate both channel attention and spatial
attention for stronger attention capability in Convolutional Block Attention
Module (CBAM) [8] are tried several times. Bottleneck Attention module (BAM)
[26] also uses double attention modules to add their attention matrices to get
the final attention map. SA-Net [27] splits the channel feature maps
separately into the channel attention module and spatial attention module.
Apart from these, there also exists the temporal attention, i.e. the attention
in time level [28].

##  III Parameter-Free Channel Attention

###  III-A Preliminary on Channel Attention Mechanism

To demonstrate the Parameter-Free Channel Attention module (PFCA), we should
first review the channel attention mechanism. Consider the input feature map
$\mathbf{X}\in\mathbb{R}^{N\times C\times H\times W}$, where $N$ is batch, $C$
is channel and $H\times W$ is feature map size, the channel attention module
is to compute the weighted map from the channel-level vector obtained by a
pooling layer on $\mathbf{X}$, such as the average pooling layer. The squeeze
and excitation process [1], including a well-known module which consists of
two parameterized Multilayer Perceptrons (MLP), helps to integrate the channel
level information and outputs the weighted map. The weighted attention map is
multiplied by input $\mathbf{X}$ to get the final output $\mathbf{Y}$. This
process can be described as

$\mathbf{Y}=\mathbf{X}\cdot\text{Attention}(\text{Pooling}(\mathbf{X}))$ (1)

Channel attention is like a gate that controls the flow of feature maps in the
channel level, making some of these feature maps more important to be
processed by the inter network module. In other words, this two MLP parts
could function as an “importance” interpreter for feature maps. In this work,
we try to provide a method that is similar to channel attention but does not
include additional parameters.

###  III-B Parameter-Free Channel Attention

The Parameter-Free Channel Attention (PFCA) is similar to the channel
attention of the channel attention (CA) mechanism. PFCA is inspired by simple
attention module [7] which aims to provide a parameter-free pixel-level
attention. We apply this idea to the channel-level attention. The attention
map $\mathbf{V}=(\mathbf{V}_{1},\mathbf{V}_{2},...,\mathbf{V}_{C})$ $\in$
$\mathbb{R}^{N\times C\times 1\times 1}$ is attained from its channel-level
vector $\mathbf{U}=(\mathbf{U}_{1},\mathbf{U}_{2},...,\mathbf{U}_{C})$ $\in$
$\mathbb{R}^{N\times C\times 1\times 1}$ which is also provided by a pooling
layer, e.g. average pooling layer. Then the channel-level vector $\mathbf{U}$
is computed by its element-wise operation below:

$\mathbf{V}_{j}=\frac{(\mathbf{U}_{j}-\mu)^{2}+2(\sigma^{2}+\lambda)}{4(\sigma^{2}+\lambda)}~{},~{}j=1,2,...,C$
(2)

Here $\mu$ and $\sigma^{2}$ are respectively the mean value and variance of
channel vector $\mathbf{U}$ computed in channel dimension. $\lambda$ is a
small value to control the variance and is set to 1e-4 in the following tasks.
After that, the weighted attention map $\mathbf{V}$ is generated by the
sigmoid activation of the above part and then is multiplied by input
$\mathbf{X}$ to get the output result.

$\mathbf{Y}=\mathbf{X}\cdot\text{sigmoid}(\mathbf{V})$ (3)

The overall PFCA module is illustrated in Figure 1. The pooling layer and the
activation part is the same with channel attention. We perform a fixed
perception to analyze the channel importance.

Figure 2: Illustration of the basic block (a) and the bottleneck block (b)
with PFCA module used in our image classification model. Both PFCA are
inserted before the addition operation.

Figure 3: Illustration of the modified compact residual block (a), and the
proposed residual block with PFCA module (b) in our image super-resolution
model. PFCA is inserted after the second convolution layer and before the
addition operation in each compact residual block.

###  III-C Image Classification and Super-Resolution Networks

Our PFCA module is designed to make a difference in raising the current
network performance, so we could insert the PFCA module into commonly used CNN
models. However, the network structures would be different when inserting the
proposed module in the object models since they vary in different tasks. To
make the PFCA module more applicable, here we choose the widely used residual
blocks [12] to build our image classification and image super-resolution
models.

Image classification. Here, we use the ResNet-18/50/101 [12] as our image
classification backbones. We first embed our PFCA module into the basic
residual block in ResNet-18, and into the bottleneck residual block in
ResNet-50 and ResNet-101 [12]. The corresponding revised residual block and
bottleneck residual block are illustrated in Figure 2. The PFCA module is put
before the addition operation in the basic residual block and the bottleneck
residual block [12], similar to [1]. Then we replace the basic residual block
in ResNet-18 by our revised residual block, and replace the bottleneck
residual block in ResNet-50 and ResNet-101 by our revised bottleneck residual
block.

Image super-resolution. Here, we employ the MSRResNet [29] as the baseline
super-resolution model, which uses the SRResNet [18] as the backbone but
remove all the Batch Normalization blocks [30]. The baseline model MSRResNet
utilizes a compact residual block instead of the original one in [18]. Then we
put the PFCA module between the second convolution layer and the addition
operation in the compact residual block, as shown in Figure 3. We replace the
compact residual block by our revised one in MSRResNet.

TABLE I: Classification accuracy (%) of different methods on ImageNet [9] valid set and CIFAR-100 [31] test set. The number of parameters (M) and FLOPs (G) are reported. Model | Params | FLOPs | CIFAR-100 | ImageNet
---|---|---|---|---
TOP 1 | TOP 5
ResNet18 | 11.15 | 1.69 | 75.71 | 70.49 | 89.55
\+ CA | 11.23 | 1.70 | 76.22 | 71.05 | 90.15
\+ PFCA | 11.15 | 1.70 | 75.93 | 70.65 | 89.76
ResNet50 | 24.37 | 3.83 | 78.66 | 75.75 | 92.57
\+ CA | 26.79 | 3.84 | 78.85 | 76.97 | 93.55
\+ PFCA | 24.37 | 3.83 | 79.00 | 76.82 | 93.44
ResNet101 | 42.49 | 7.30 | 78.79 | 77.52 | 93.73
\+ CA | 47.04 | 7.31 | 79.56 | 78.02 | 94.12
\+ PFCA | 42.49 | 7.30 | 79.11 | 77.98 | 93.91
TABLE II: PSNR/SSIM results of comparison image super-resolution methods. “PA”, “CA”, and “PFCA” means Pixel Attention, Channel Attention, and Parameter-Free Channel Attention, respectively. FLOPs are computed on a $256\times 256$ image. Model | Params (K) | Flops (G) | Set5 | Set14 | B100 | Urban100 | Manga109
---|---|---|---|---|---|---|---
SRCNN [15] | 20.1 | 21.07 | 30.48/0.8628 | 27.49/0.7503 | 26.90/0.7101 | 24.52/0.7221 | 27.66/0.8505
EDSR [32] | 43089.9 | 3293.90 | 32.48/0.8988 | 28.81/0.7879 | 27.72/0.7420 | 26.65/0.8036 | 31.04/0.9158
MSRResNet [29] | 1517.6 | 166.36 | 32.19/0.8954 | 28.65/0.7835 | 27.60/0.7374 | 26.20/0.7900 | 30.55/0.9101
\+ PA | 1584.1 | 170.66 | 32.24/0.8956 | 28.64/0.7834 | 27.61/0.7375 | 26.19/0.7900 | 30.55/0.9099
\+ CA | 1526.9 | 166.36 | 32.26/0.8963 | 28.65/0.7829 | 27.60/0.7373 | 26.26/0.7914 | 30.70/0.9115
\+ PFCA | 1517.6 | 166.36 | 32.26/0.8961 | 28.69/0.7838 | 27.61/0.7375 | 26.26/0.7917 | 30.69/0.9115

PSNR/SSIM

22.32 dB/0.7379

25.03 dB/0.8494

29.10 dB/0.9306

28.61 dB/0.9257

28.72 dB/0.9267

PSNR/SSIM

24.34 dB/0.6946

26.02 dB/0.7504

27.95 dB/0.7901

27.52 dB/0.7830

27.61 dB/0.7854

PSNR/SSIM
(a) HR

16.61 dB/0.4398
(b) LR bicubic

17.15 dB/0.5061
(c) SRCNN [15]

19.15 dB/0.6785
(d) EDSR [32]

19.08 dB/0.6682
(e) MSRResNet [29]

19.39 dB/0.6830
(f) MSRResNet+PFCA

Figure 4: Visual quality and PSNR/SSIM results by different image super-
resolution methods on Set5 (1st row), Set14 (2nd row) and Urban100 (3rd row).

##  IV Experiments

###  IV-A Experiment Details

In our experiments, image classification task is performed on ImageNet 2012
dataset [33] and CIFAR-100 dataset [31]. ImageNet 2012 dataset contains 1.28
million train images and 50,000 valid images with total 1000 classes.
CIFAR-100 dataset contains 50,000 train images and 10,000 test images with 100
classes. Image super-resolution task is performed on DIV2K dataset [34] with
contains 800 images to train. Experiments are performed by using Pytorch
framework with one NVIDIA RTX 3090 GPU.

###  IV-B Image Classification

When training classification networks on ImageNet dataset, input images are
adjusted to size $224\times 224$. We use the same Data augmentation method in
[12, 1]. For optimization process, SGD optimizer is used with momentum 0.9 and
a mini-batch size 128. The learning rate is initially set to 0.1 and reduces
to $0.1$ of the previous value for every 30 epochs. Weight decay is used and
set to $10^{-4}$. The network models are trained totally 100 epochs from
scratch. Classification test result is presented on the Table I. Flops are
computed with a single image with size $224\times 224$. It is indicated that
adding PFCA module could effectively increase the classification accuracy on
both top-1 and top-5.

We follow the same optimization strategy in [31] to train ResNet on CIFAR
dataset where batch size is 128 and the model trained for 200 epochs. The
initial learning rate here is set to 0.05 and divided by 5 at epoch 60, 120,
and 160 and weight decay is $5\times 10^{-4}$. L1 loss function is used to
train the network and SGD optimization strategy is used with momentum set to
0.9. We present top-1 and top-5 accuracy as test metrics on both datasets.
Classification test result is presented in the Table I. It is indicated that
adding PFCA module could effectively increase the classification accuracy. One
can see that our PFCA module also helps increase the classification accuracy
on CIFAR-100 dataset.

###  IV-C Image Super-Resolution

We further testify the effectiveness of our proposed Parameter-Free Channel
Attention module on image super-resolution task. To effectively compare
different attention, we provide four candidate models to train: the baseline
model MSRResNet [29] with no attention, with PA [3], with CA and with PFCA.
The number of residual blocks is 16. We train models on DIV2K train dataset
[34] which contains 800 high resolution images. The HR images are randomly
cropped with patch size $256\times 256$. Mini-batch size is set to 32. The
cosine annealing learning scheme is used when training model, and learning
rate is set with $2\times 10^{-4}$ initially and the minimum value to
$10^{-7}$. We set weight decay to $5\times 10^{-8}$. Restart period is set
after each 250k iterations. All models are trained for total 1000k iterations.
The corresponding LR images are obtained by downsampling HR images by 4
downscale factor using bicubic method. After the training stage, we use five
standard benchmark datasets, Set5 [20], Set14 [35], B100 [36], Urban100 [19]
and Manga109 [21], to test the performance. Evaluation metric is set with
average peak signal to noise ratio (PSNR) and average structural similarity
index (SSIM) on each test set. All test metrics are calculated on the Y
channel of images. In addition to the comparison of MSRResNet [29] and our
model, we also compare with SRCNN [15] and EDSR [32] results in the paper,
since the former is the first CNN model for super-resolution task and the
latter one has similar architecture as ours but with much more parameters.

Objective results are provided in Table II. We observe that adding PFCA in the
residual block helps improve the PSNR and SSIM values compared to the baseline
MSRResNet. Adding different attentions could bring effect improvement but with
different parameters. Note that the PSNR/SSIM values of PFCA are similar to
those of CA. That means PFCA could achieve comparable performance of CA with
little parameter growth.

Comparisons on visual quality are presented in Figure 4. Images are chosen
from the test set. The visual result and their corresponding test metrics are
listed below the image. It reveals that the restored quality of image of
adding PFCA part is better in the local texture part. Thus the PFCA module
could effectively help improve the visual quality.

##  V Conclusion

In this paper, we proposed a Parameter-Free Channel Attention (PFCA) that
could serve the function of channel importance estimation, while not
introducing additional parameters. We put this PFCA module into the image
classification model as well as the image super-resolution model and then
performed experiments on image classification and image super-resolution. The
experiment results validate that our PFCA module could function as an
effective attention module in deep networks. According to some experiment
results, the fixed structure of PFCA needs to be further improved to achieve
better performance compared to CA.

## References

  * [1] Jie Hu, Li Shen, and Gang Sun,  “Squeeze-and-excitation networks,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7132–7141.
  * [2] Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang,  “Second-order attention network for single image super-resolution,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 11065–11074.
  * [3] Hengyuan Zhao, Xiangtao Kong, Jingwen He, Yu Qiao, and Chao Dong,  “Efficient image super-resolution using pixel attention,”  in European Conference on Computer Vision. Springer, 2020, pp. 56–72.
  * [4] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu,  “Image super-resolution using very deep residual channel attention networks,”  in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 286–301.
  * [5] Namhyuk Ahn, Byungkon Kang, and Kyung-Ah Sohn,  “Fast, accurate, and lightweight super-resolution with cascading residual network,”  in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 252–268.
  * [6] Haoran Wang, Yue Fan, Zexin Wang, Licheng Jiao, and Bernt Schiele,  “Parameter-free spatial attention network for person re-identification,”  arXiv preprint arXiv:1811.12150, 2018.
  * [7] Lingxiao Yang, Ru-Yuan Zhang, Lida Li, and Xiaohua Xie,  “Simam: A simple, parameter-free attention module for convolutional neural networks,”  in Proceedings of the 38th International Conference on Machine Learning, Marina Meila and Tong Zhang, Eds. 18–24 Jul 2021, vol. 139 of Proceedings of Machine Learning Research, pp. 11863–11874, PMLR.
  * [8] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon,  “Cbam: Convolutional block attention module,”  in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 3–19.
  * [9] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton,  “Imagenet classification with deep convolutional neural networks,”  Advances in neural information processing systems, vol. 25, pp. 1097–1105, 2012.
  * [10] Karen Simonyan and Andrew Zisserman,  “Very deep convolutional networks for large-scale image recognition,”  arXiv preprint arXiv:1409.1556, 2014.
  * [11] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich,  “Going deeper with convolutions,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 1–9.
  * [12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [13] Wangpeng An, Haoqian Wang, Qingyun Sun, Jun Xu, Qionghai Dai, and Lei Zhang,  “A pid controller approach for stochastic optimization of deep networks,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 8522–8531.
  * [14] Jun Xu, Wangpeng An, Lei Zhang, and David Zhang,  “Sparse, collaborative, or nonnegative representation: which helps pattern classification?,”  Pattern Recognition, vol. 88, pp. 679–688, 2019.
  * [15] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang,  “Image super-resolution using deep convolutional networks,”  IEEE transactions on pattern analysis and machine intelligence, vol. 38, no. 2, pp. 295–307, 2015.
  * [16] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang,  “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 1874–1883.
  * [17] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee,  “Accurate image super-resolution using very deep convolutional networks,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 1646–1654.
  * [18] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al.,  “Photo-realistic single image super-resolution using a generative adversarial network,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4681–4690.
  * [19] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja,  “Single image super-resolution from transformed self-exemplars,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 5197–5206.
  * [20] Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie Line Alberi-Morel,  “Low-complexity single-image super-resolution based on nonnegative neighbor embedding,”  2012\.
  * [21] Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa,  “Sketch-based manga retrieval using manga109 dataset,”  Multimedia Tools and Applications, vol. 76, no. 20, pp. 21811–21838, 2017.
  * [22] Jingwen He, Chao Dong, and Yu Qiao,  “Modulating image restoration with continual levels via adaptive feature modification layers,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 11056–11064.
  * [23] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio,  “Neural machine translation by jointly learning to align and translate,”  arXiv preprint arXiv:1409.0473, 2014.
  * [24] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  in Advances in neural information processing systems, 2017, pp. 5998–6008.
  * [25] Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang,  “Selective kernel networks,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 510–519.
  * [26] Jongchan Park, Sanghyun Woo, Joon-Young Lee, and In So Kweon,  “Bam: Bottleneck attention module,”  arXiv preprint arXiv:1807.06514, 2018.
  * [27] Qing-Long Zhang and Yu-Bin Yang,  “Sa-net: Shuffle attention for deep convolutional neural networks,”  in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 2235–2239.
  * [28] Gang Xu, Jun Xu, Zhen Li, Liang Wang, Xing Sun, and Ming-Ming Cheng,  “Temporal modulation network for controllable space-time video super-resolution,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 6388–6397.
  * [29] Kai Zhang, Martin Danelljan, Yawei Li, Radu Timofte, Jie Liu, Jie Tang, et al.,  “AIM 2020 challenge on efficient super-resolution: Methods and results,”  in Computer Vision - ECCV 2020 Workshops - Glasgow, UK, August 23-28, 2020, Proceedings, Part III, Adrien Bartoli and Andrea Fusiello, Eds. 2020, vol. 12537 of Lecture Notes in Computer Science, pp. 5–40, Springer.
  * [30] Sergey Ioffe and Christian Szegedy,  “Batch normalization: Accelerating deep network training by reducing internal covariate shift,”  in International conference on machine learning. PMLR, 2015, pp. 448–456.
  * [31] Alex Krizhevsky, Geoffrey Hinton, et al.,  “Learning multiple layers of features from tiny images,”  2009\.
  * [32] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee,  “Enhanced deep residual networks for single image super-resolution,”  in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2017, pp. 136–144.
  * [33] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei,  “Imagenet: A large-scale hierarchical image database,”  in 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009, pp. 248–255.
  * [34] Eirikur Agustsson and Radu Timofte,  “Ntire 2017 challenge on single image super-resolution: Dataset and study,”  in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
  * [35] Roman Zeyde, Michael Elad, and Matan Protter,  “On single image scale-up using sparse-representations,”  in International conference on curves and surfaces. Springer, 2010, pp. 711–730.
  * [36] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik,  “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,”  in Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001. IEEE, 2001, vol. 2, pp. 416–423.