# Learning Class Unique Features in Fine-Grained Visual Classification

Runkai Zheng  Zhijia Yu  Yinqi Zhang  Chris Ding  Hei Victor Cheng  Li Liu

###### Abstract

A major challenge in Fine-Grained Visual Classification (FGVC) is
distinguishing various categories with high inter-class similarity by learning
the feature that differentiate the details. Conventional cross entropy trained
Convolutional Neural Network (CNN) fails this challenge as it may suffer from
producing inter-class invariant features in FGVC. In this work, we
innovatively propose to regularize the training of CNN by enforcing the
uniqueness of the features to each category from an information theoretic
perspective. To achieve this goal, we formulate a minimax loss based on a game
theoretic framework, where a Nash equilibria is proved to be consistent with
this regularization objective. Besides, to prevent from a feasible solution of
minimax loss that may produce redundant features, we present a Feature
Redundancy Loss (FRL) based on normalized inner product between each selected
feature map pair to complement the proposed minimax loss. Superior
experimental results on several influential benchmarks along with
visualization show that our method gives full play to the performance of the
baseline model without additional computation and achieves comparable results
with state-of-the-art models.

Machine Learning, ICML

Figure 1: Three images of similar bird species from CUB-200-2011 dataset are
selected to present the major challenge in FGVC. The first row presents the
original images and the second row presents the activation maps. The
activation maps are obtained from forward propagating the image of Red Winged
Blackbird to the CE trained CNN and extracting the penultimate layer feature
maps. Then the channel that has the largest mean activation value is selected.
An observation is that the same channel also has high activation value when we
input the other two categories of images (Brewer Blackbird and Rusty
Blackbird).

Convolutional Neural Networks (CNN) achieves a great success in the computer
vision domain. The large diversity in standard visual recognition tasks make
it possible for CNN to well learn discriminative features by minimizing the
cross entropy (CE) loss. When it comes to Fine-Grained Visual Classification
(FGVC), different categories with highly similar appearance leads to inter-
class invariants and intra-class variants, which limit the performance of
standard CE trained CNN.

Over the past few years, FGVC has attracted lots of attention in research
community. Early works (Zhang et al., 2014; Branson et al., ; Wei et al.,
2016) utilize multi-stage architecture that consists of a localization network
and a classification network. The localization network is responsible for
detecting discriminative regions, which requires bounding box or part
annotations for training. Then the classification network works on the cropped
regions given by the localization network. However, these approaches depend on
annotations and cannot be trained end-to-end, thus lead to extra training
cost. To solve the above mentioned challenges, recent approaches manage to
develop end-to-end networks that only require weak supervision. Commonly these
approaches outperform baseline models by mimicking human actions like
attention mechanism and part localization (Zheng et al., 2017; Fu et al.,
2017; Sun et al., 2018; Wang et al., 2018; Chen et al., 2019; Ding et al.,
2019; Wang et al., 2020). However, these works focus on discriminative parts
on spatial domain. Even if the regions of interest are correctly cropped or
detected, CNN will inevitably encode unnecessary information that may mix up
with other categories. As shown in Fig. 1, the problem is, although the model
can well localize the discriminative parts, the channel that responsible for
detecting the feature will also be activated when encounter a similar texture.
That means localization is not sufficient for learning a good feature if the
filters can not precisely encode the unique feature. It is a challenge to
extract the features that contain unique information about the categories.

In this work, we propose an explicit regularization objective for encoding
unique features with a theoretical guarantee. Our motivation is based on an
assumption that a unique feature should contain only the information of a
specific category and not any other categories. In other words, for a given
image, we expect the extracted features to be highly correlated with the
target class without extra information about the non-target classes. We call
this kind of features Class Unique Features (CUFs). To achieve this goal, we
formulate CUF using the Mutual Information (MI), from which we deduce an
explicit regularization objective, i.e., Maximum Non-Target distribution
Entropy (MaxNTE). To efficiently optimize the objective, we propose a game
theoretic framework to simplify the problem formulation. Under this game
theoretic framework, the existence of Nash equilibria and the consistency
between the outcome and our objective are proved rigorously.

In summary, our contribution includes:

  1. 1.

We innovatively formulate our assumption to an ideal CUF learner from an
information theoretic perspective and deduce an explicit regularization
objective.

  2. 2.

We construct a game-theoretic framework between the model and the adversary.
On this basis, we arrive at a simple yet efficient minimax (MM) loss to
achieve the regularization goal. To reduce the feature redundancy brought by
the minimax loss, we further propose a Feature Redundancy Loss (FRL),
encouraging the model to focus on multiple discriminative parts, as a
complement to the minimax loss.

  3. 3.

Experimental results on influential benchmarks of both FGVC and standard
visual classification show that our method outperforms the baseline models by
a large margin and achieves state-of-the-art (SOTA) results on FGVC-Aircraft
and Standard Cars dataset.

##  1 Related Work

###  1.1 Fine-Grained Visual Classification

Recently, FGVC is a research hotspot in the field of computer vision. We
mainly discuss related works according to the following three research
branches.

Attention mechanism and part localization were explored to settle this
problem, as the model is able to learn to pay attention to the region or
features that contain inter-class variations. Benefited from the interaction
of part learning and feature learning, Multi-Attention CNN (MA-CNN) was
proposed in (Zheng et al., 2017) to extract part-based fine-grained features.
(Fu et al., 2017) utilized recurrent architecture to repeatedly crop and scale
the regions of interest by attention mechanism. (Sun et al., 2018) proposed
One-Squeeze Multi-Excitation that generate multiple attention map based on
Multi-Attention Multi Class Constraint to efficiently obtain the highly
discriminative part. (Wang et al., 2018) used $1\times 1$ convolution kernel
as a discriminative patch detector and designed an asymmetric, multi-channel
structure to enhance the learning of discriminative mid-level patches. (Chen
et al., 2019) shuffled the local regions to enforce the network to focus on
the most discriminative patches. (Ding et al., 2019) used sparse attention for
feature sampling to capture detailed visual evidence without losing the
context information.

High-order statistics were explored for aggregating features to improve the
first-order statistics such as max pooling and average pooling because they
were difficult to capture the diversity of features among different
categories. (Lin et al., 2015) produced an image descriptor via pooling the
outer product from two CNN feature extractor sub-branches, which was able to
model local pairwise feature interactions in a translational invariant manner.
(Gao et al., 2016) approximated bilinear pooling operation by applying low-
dimensional approximation of the polynomial kernel to speed up the
computation. (Wang et al., 2019a) inserted Matrix Power Normalized COVariance
(MPN-COV) block into the final layer of convolutions to obtain a global
representation by second order statics.

Regularization based methods usually do not need extra computation and thus
are much light weight compared with the above mentioned methods. They
developed efficient training manner that can boost the performance of simple
baseline models. (Dubey et al., 2018b) formulated the relation between model
selection and feature diversity, and utilizing the idea of maximum entropy to
minimize the lower bound of Frobenius norm of the weights and thus improved
the performance of models in fine-grained visual tasks. (Dubey et al., 2018a)
minimized the L2 distance between the prediction probability distribution of
the random sample pairs of the training set to confuse the network and prevent
from overfitting. Our method is also based on regularization of output
distribution, and thus can be a simple and lightweight tool to be used among
similar tasks.

###  1.2 Label smoothing

Label smoothing (Szegedy et al., 2016) was first proposed to prevent deep
learning model from overconfident in classification problem. The
characteristic of softmax function makes it impossible for a model to
convergent to the hard 0 and 1 targets (Goodfellow et al., 2016). Thus the
model may keep seeking for extreme prediction and become overfitting. Label
smoothing introduces uniform noise distribution $u$ to the ground truth labels
by replacing 0 and 1 targets with $\frac{\epsilon}{n-1}$ and $1-\epsilon$,
where $\epsilon$ is a hyperparameter determines the amount of smoothing. Our
method produces uniformly distributed probabilities on non-target class, which
is similar to the ground truth of label smoothing. However, our method is
different from label smoothing in terms of motivation, training manner and
resulting outputs. More precisely, 1) our motivation is to achieve an
assumption on MI between extracted features and output distribution, while LS
is proposed to inject noise to the labels to prevent from extreme logits and
overfitting. 2) We use a minimax loss to achieve our objective while LS
directly takes the designed target to supervise the model. 3) In terms of the
regularization results, our method leads to uniform distribution on non-target
classes, but LS can not.

###  1.3 Mutual Information

Mutual Information (MI) is a measure of information in information theory,
which indicates the mutual dependencies between two random variables. More
specifically, it quantifies the amount of information about another random
variable when one of the variables is observed. MI has been used in the field
of deep learning. (Tishby & Zaslavsky, 2015) firstly showed that Deep Neural
Networks (DNN) can be quantified by the mutual information between the layers
and the input and output variables. They provide a novel perspective that the
goal of DNN is to optimize Information Bottleneck (IB) trade off between
compression and prediction. (Hjelm et al., 2018) wielded the rich knowledge
about mutual information into the construction of encoder, called Deep
InforMax (DIM), which maximized the mutual information between the inputs and
the high-level representation. Belghazi et al. (Belghazi et al., 2018)
proposed Mutual Information Neural Estimator (MINE) and applied it to
Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) to improve
the reconstruction quality and alleviate mode-drop in GANs.

###  1.4 Game Theory

Game Theory provides mathematical models of strategic interaction among
intelligent decision makers (Myerson, 1991). It is a mathematical theory and
method to study the phenomenon of competition, and was studied in (Osborne et
al., 2004) the interaction between the formulaic incentive structures. With
the increasing popularity of Artificial Intelligent, game theory has been
applied to different fields including Multi-Agent Reinforcement Learning
(Bowling & Veloso, 2000) and GANs (Goodfellow et al., 2014; Goodfellow, 2016).
In this work, we apply the game theory to the object recognition task.

##  2 Method

Our goal is to extract features that do not contain information from non-
target classes, i.e., CUF. This can be formulated by minimizing the MI between
extracted feature and the non-target output distribution. By the formulation,
we further deduce a regularization objective, where a key finding is derived
(i.e., all output probabilities over non-target classes should be uniformly
distributed). To efficiently optimize this objective function, we propose a
minimax loss to simplify the reformulated regularization objective.

###  2.1 Mutual Information based Problem Formulation

We denote the input space by $\mathcal{X}$, the label space by
$C=\\{1,2,\dots,n\\}$, where $n>2$ is the number of classes. The training data
are all i.i.d sample pairs $(x,y)$, where $y$ is in the form of a
$n$-dimensional one-hot vector.

Let $\Phi$ be a parametric function mapping from the input space to the
feature space. Overall parameter set of the model is $\theta$. Consider one of
the fix target categories $t\in C$, and the corresponding input $X^{t}$ (i.e.,
images that belong to the class $t$). $\hat{Y}_{C\setminus t}$ is the
predicted output of non-target classes with distribution $q_{C\setminus
t}=softmax(z_{C\setminus t})$, where $z_{C\setminus
t}=[z_{1},z_{2},\dots,z_{t-1},z_{t+1},\dots,z_{n}]$. Note that all the random
variables $y,z,q,t$ are dependent on input $x$, here we omit the dependence
for brevity.

The problem formulation can be written as minimizing the MI between predicted
output of non-target classes $\hat{Y}_{C\setminus t}$ and extracted features
$\Phi(X^{t})$, i.e. $I_{\theta}(\hat{Y}_{C\setminus t};\Phi(X^{t}))$. However,
this MI cannot be computed in practice as the distribution of $\Phi(X^{t})$ is
intractable. To resolve the intractability issue, we apply the data processing
inequality (Cover & Thomas, 2012) and use the obtained upper bound, the MI
between input $X^{t}$ of class $t$ and predicted output of non-target classes
$\hat{Y}_{C\setminus t}$, as our objective:

$I_{\theta}(\hat{Y}_{C\setminus t};X^{t}),$ (1)

where $I_{\theta}(\hat{Y}_{C\setminus t};X^{t})$ represents the mutual
information under model parameter $\theta$.

According to the property of MI, i.e., $I(A;B)=H(A)-H(A|B)$, where $H$ is the
entropy, and $A,B$ are two random variables, Eq. 1 can be decomposed into the
difference between entropy and conditional entropy:

$\displaystyle I_{\theta}(\hat{Y}_{C\setminus
t};X^{t})=H_{\theta}(\hat{Y}_{C\setminus t})-H_{\theta}(\hat{Y}_{C\setminus
t}|X^{t}),$

where $H_{\theta}(\cdot)$ denotes the entropy under model parameter $\theta$.
Computing $H_{\theta}(\hat{Y}_{C\setminus t})$ involves the marginalization
over $X^{t}$ which is computationally intractable in practice. Thus, instead
of directly optimizing Eq. 1, we consider optimizing its upper bound. From the
fact that entropy reaches its upper bound when all the probabilities are
equal, we have:

$\displaystyle H_{\theta}(\hat{Y}_{C\setminus
t})\leq-(n-1)\frac{1}{n-1}\log\frac{1}{n-1}=\log{(n-1)},$

The following lemma gives a theoretic justification for using the upper bound
to replace Eq. 1.

######  Lemma 1.

When the conditional probability distribution over non-target classes is
uniform, the MI in Eq. 1 is 0, and hence $I_{\theta}(\hat{Y}_{C\setminus
t};\Phi(X^{t}))=0$.

Lemma 1 shows that making the conditional distribution of the non-target
classes uniform is desired. Thus we formulate the problem using the upper
bound as maximizing $H_{\theta}(\hat{Y}_{C\setminus t}|X^{t})$, which promotes
the distribution to be uniform. When the mutual information is
$I_{\theta}(\hat{Y}_{C\setminus t};\Phi(X^{t}))=0$, this suggests that the
extracted features contains no information about the non-target classes.

In practice, we maximize the empirical conditional entropy for each class $t$:

$\displaystyle\frac{1}{N^{t}}\sum_{i=1}^{N_{t}}H_{\theta}(\hat{Y}_{C\setminus
t}|x^{t}_{i}),$

where $N_{t}$ is the number of training samples in class $t$.

We use the empirical conditional entropy as a regularization term with a
weight parameter $\lambda$ into the CE objective function to form the overall
objective function (i.e., MaxNTE):

$\displaystyle\mathcal{L}_{MaxNTE}={E}_{x\sim\mathcal{X}}[D_{CE}(y||q(x;\theta))-\lambda
H_{\theta}(\hat{Y}_{C\setminus t}|X^{t})].$ (2)

Here, $H_{\theta}(\hat{Y}_{C\setminus t}|X^{t})$ has a reachable upper bound.
However, directly taking this as the objective function may not be the best
choice for our goal, since the gradients become extremely small when closing
to the upper bound. From Lemma 1, we know a sufficient condition for
$I_{\theta}(\hat{Y}_{C\setminus t};\Phi(X^{t}))=0$ is to enforce the
conditional distribution of non-target classes to be uniform. In the
following, we propose an efficient minimax loss based on game theory to
achieve this target, which is lightweight and also insensitive to the choice
of hyper-parameter. The comparison of MaxNTE and the new proposed loss will be
provided in the experiments.

Figure 2: (a) During the training phase (before convergence), the model keep
promoting the smallest probability in the non-target model output by assign
$1-p_{t}$ to the index of the minimum value in $q_{C\setminus t}$, i.e..,
$q_{3}$ in (a). (b) After iterations of training, the model output
distribution will finally be uniformly distributed over non-target classes.
When reaching a convergence, a Nash equilibrium exists between the optimal
solution of the model and the adversary.

###  2.2 Game Theoretic Framework

In this section, we introduce the game theoretic framework in detail. The
resulting loss function will be shown in Eq. 2.2.3, which is used as the major
part of our main method in the experiments.

####  2.2.1 Preliminaries

Here we introduce some basic game-theoretic definitions (Myerson, 1991) that
we will use later.

######  Definition 1.

A strategic game is a tuple $G=\left\langle I,(A_{i})_{i\in I},(u_{i})_{i\in
I}\right\rangle$, where $I$ is a nonempty set of players, $A_{i}$ is the set
of actions available to each player $i\in I$, $A=\prod_{i\in I}A_{i}$ is the
profiles of actions and $u_{i}:A\rightarrow R$ defines the payoff function for
each player $i\in I$. A two player strictly competitive game or zero-sum game
is the strategic game $G$ with $I=\\{1,2\\}$ and for all $a\in A$:

$u_{1}(a)=-u_{2}(a).$

######  Definition 2.

A mixed strategy set $S_{i}$ is the set of all probability distributions over
$A_{i}$. $s_{i}\in S_{i}$ defines a mixed strategy for each player $i\in I$,
and $s_{i}(a_{i}^{k})$ is the probability that player $i$ plays $a_{i}^{k}\in
A_{i}$. In a two player game, the expected payoff of player $i$ playing a
mixed strategy against pure strategy $a_{-i}^{*}$ can be calculated as:

$U_{i}(s)=\sum_{a_{i}^{k}\in
A_{i}}s_{i}(a_{i}^{k})u_{i}(a_{i}^{k},a_{-i}^{*}).$

Likewise, the expected payoff of playing a pure strategy $a_{i}^{*}$ against
mixed strategy can be calculated as:

$U_{i}(s)=\sum_{a_{-i}^{k}\in
A_{-i}}s_{-i}(a_{-i}^{k})u_{i}(a_{i}^{*},a_{-i}^{k}),$

where $-i$ denotes the player other than $i$.

######  Definition 3.

Let $G$ be the strategic game, $i\in I$ be a player, and $s_{-i}\in S_{-i}$ be
a strategy profile of players other than $i$. Then a strategy $s^{*}_{i}\in
S_{i}$ is a best response of player $i$ to $s_{-i}$ if:

$\forall s_{i}\in S_{i},\ \ U_{i}(s_{i}^{*},s_{-i})\geq U_{i}(s_{i},s_{-i}).$

######  Definition 4.

A Nash equilibrium of the strategic game $G$ is a action profile $s^{*}\in S$
such that for every player $i$, $s^{*}_{i}$ is the best response to
$s^{*}_{-i}$.

####  2.2.2 $p-q$ zero-sum strategic game

We define a zero-sum strategic game played between the _model_ and a designed
_adversary_ with loss of the model defined as $D_{CE}(p||q)$. We let $p$ be
the ground truth label vector, normally one-hot encoding, while in our work we
specifically design it for the objective. Here, $p$ is the strategy of the
adversary and $q$ is the strategy of the model. We assume that the model is a
classifier with confidence $q_{t}$ on the target class. The model aims to
assign the rest of the probability $1-q_{t}$ to non-target classes to minimize
the loss. The adversary is the controller of the ground truth with fixed
$p_{t}$ for the target class, and it aims to maximize the loss via adjusting
the distribution on non-target class of $p$. This is a dynamic game that the
two players play in order, in which the model goes first, and the adversary
can adjust the strategy according to the previous action of the model. Fig. 2
gives an overview of the proposed game theoretic framework.

######  Definition 5.

For $p_{t},q_{t}\in(0,1)$, the defined strategic game is a tuple
$G=\left\langle(P,Q),(A_{P},A_{Q}),(u_{P},u_{Q})\right\rangle$ with:

$\displaystyle A_{P}$ $\displaystyle=\\{(p_{i})_{1\leq i\leq
n},p_{i}\in(0,1),\sum_{i\neq t}p_{i}=1-p_{t}\\}.$ $\displaystyle A_{Q}$
$\displaystyle=\\{(q_{i})_{1\leq i\leq n},q_{i}\in(0,1),\sum_{i\neq
t}q_{i}=1-q_{t}\\}.$ $\displaystyle u_{P}$
$\displaystyle=D_{CE}(p||q)=-u_{Q}.$

By Definition 1, $G$ is a two-player zero-sum game. In the following, we will
prove the existence of Nash equilibrium.

######  Theorem 1.

For $p_{t},q_{t}\in(0,1)$, we have:

$\forall q\in A_{Q},\quad p^{*}=\mathop{\arg\max}_{p}D_{CE}(p||q),$

where $p^{*}=(p_{i}^{*})_{1\leq i\leq n}$:

$p_{i}^{*}=\begin{cases}p_{t},&i=t;\\\ 1-p_{t},&i=k;\\\
0,&otherwise\end{cases},$

for any $k=\mathop{\arg\min}_{z}(q_{C\setminus t})_{z}$.

Note that in the rest of the paper, if there exists more than one minimum
value in $q_{C\setminus t}$, $k=\mathop{\arg\min}_{z}(q_{C\setminus t})_{z}$
refers to randomly taking one of them.

######  Theorem 2.

For $p_{t},q_{t}\in(0,1)$, we have:

$\quad q^{*}=\mathop{\arg\min}_{q\in A_{Q}}D_{CE}(p^{*}||q),$

where

$q_{i}^{*}=\left\\{\begin{matrix}q_{t},&i=t;\\\
\frac{1-q_{t}}{n-1},&otherwise.\end{matrix}\right.$

Theorem 1 gives the worst case payoff for the model $q$. However, since
$p^{*}$ depends on the index of the minimum value in $q$, they are not the
best responses to each other. For example, when $q=q^{*}$, the adversary
chooses one of the indexes of the minimum values in $q$ to determine $p^{*}$.
Once $p^{*}$ is fixed, $q$ is no longer the best responses to $p^{*}$, since
there exist a better $q$ to get a higher payoff (e.g., change the position of
the minimum value). Thus, we need randomize $p$ to avoid this situation.
Specifically, when $q=q^{*}$, the model uniformly distributes the
probabilities over non-target classes. The adversary can randomly choose one
of them since they are all the smallest value and the adversary’s strategy
becomes a mixed strategy. In this case, the two strategies form a Nash
equilibrium. To show this mathematically, we give the following theorem.

######  Theorem 3.

Define an action subset for the adversary:

$\displaystyle a_{P}^{*}\subset
A_{P}=\left\\{(p_{i})|p_{i}=\begin{cases}p_{t},&i=t;\\\ 1-p_{t},&i=k;\\\
0,&otherwise\end{cases}\right\\},$

where $k=\\{1,2,\dots,t-1,t+1,\dots,n$}.

For the model:

$a_{Q}^{*}=\left\\{(q_{i})|q_{i}=\left\\{\begin{matrix}q_{t},&i=t;\\\
\frac{1-q_{t}}{n-1},&otherwise\end{matrix}\right.\\\ \right\\}.$

Then we have the following strategies:

$s_{P}^{*}(p)=\left\\{\begin{matrix}\frac{1}{n-1},&p\in a_{P}^{*};\\\
0,&otherwise.\end{matrix}\right.$ $s_{Q}^{*}(q)=\left\\{\begin{matrix}1,&q\in
a_{Q}^{*};\\\ 0,&otherwise,\end{matrix}\right.$

such that $s^{*}=(s_{P}^{*},s_{Q}^{*})$ forms a Nash equilibrium.

The detailed proofs of the three theorems are in Appendix.

####  2.2.3 Minimax Loss

The Nash equilibrium in a two player zero-sum game is equivalent to a minimax
solution (Ferreira et al., 2012). Thus, by training with the worst-case payoff
$D_{CE}(p^{*}||q)$, we expect that the model output ultimately converges to
the best response $q^{*}$. Finally our proposed minimax loss (MM) is defined
as:

$\displaystyle\mathcal{L}_{MM}$
$\displaystyle=\mathbb{E}_{x\sim\mathcal{X}}[D_{CE}(p^{*}||q)]$
$\displaystyle=\mathbb{E}_{x\sim\mathcal{X}}[-p_{t}\log
q(x;\theta)_{t}-(1-p_{t})\log q(x;\theta)_{k}],$ (3)

where $k={\mathop{\arg\min}_{a}}(q_{C\setminus t})_{a}$. Here, we leave
$p_{t}$ as a hyper-parameter to weight the regularizer of the objective
corresponding to the class $t$. When $p_{t}$ is set to 1 for all classes, the
loss function is equivalent to the standard CE loss. It is worth noting that,
in regularization methods such as label smoothing and confidence penalty, the
number of log operations increases with the number of categories. MM has only
one more log operation than the cross entropy loss, while gains more
performance in many tasks.

Figure 3: The feature redundancy problem in MM trained model. As shown in the
first row, the three channels with the largest activation values detect the
same region of the bird. That means the model’s prediction is over dependent
on the single feature. Once the region is cropped, as shown in the image
below, the model cannot correctly predict the object category. The introduced
FRL is shown to be able to eliminate this problem.

###  2.3 Feature Redundancy Loss

MM promotes the feature uniqueness of each category. There may exist more than
one solutions that having this property. In some cases, the obtained features
can be redundant, different feature maps of a specific category are almost the
same, as shown in the first row of Fig. 3. We want the extracted features to
be more diverse, because single feature can be unreliable especially when the
training set is small. Combining multiple features for decision can avoid
wrong prediction under unexpected cases such as occlusion and make the model
more robust. The second row of Fig. 3 shows the case that one of the important
regions is blocked, in which MM trained model that rely on single feature
fails to make a correct prediction. Therefore we add an additional
regularization term to choose more diverse features while maintaining the
class uniqueness.

To enforce the difference of feature maps, we use normalized inner product to
measure the similarity among feature maps of top activation as a loss
function, named Feature Redundancy Loss. Specifically, in each forward sample,
we select the feature maps that has top $K$ activation values before global
average pooling. Let the shape of the selected feature maps $\phi=\Phi(x)$
from a sample $x$ be $(K,H,W)$, we calculate the normalized inner product
between each pairs.

$\mathcal{L}_{FRL}=\sum_{i=1}^{K-1}\sum_{j=i+1}^{K}\frac{\langle\phi_{i}\phi_{j}\rangle}{\|{\phi_{i}}\|\|{\phi_{j}}\|}$
(4)

The loss can be calculated parallelly using tensor operation, thus a simple
yet efficient trick. Our final loss is the weighted sum of MM and FRL:

$\mathcal{L}=\mathcal{L}_{MM}+\lambda\mathcal{L}_{FRL}$ (5)

##  3 Experiments

###  3.1 Experimental Setup

For evaluating our method, we use the following three benchmarks: CUB-200-2011
(Wah et al., 2011), FGVC-Aircraft (Maji et al., 2013), Stanford Cars (Krause
et al., 2013). Further more, we assesses the effect of our method on standard
visual classification benchmarks: CIFAR-10 (Krizhevsky et al., 2009),
CIFAR-100 (Krizhevsky et al., 2009), STL-10 (Coates et al., 2011). Different
methods are compared using ResNet18 (He et al., 2016), VGGNet11 (Simonyan &
Zisserman, 2014), DenseNet161 (Huang et al., 2017) as the backbone models. The
statistics of six datasets and the implementation details are introduced in
Appendix.

Figure 4: Comparison between MM trained and MaxNTE loss trained models. Non-
target classes entropy (scale using left vertical axis) and test accuracy
(scale using right vertical axis) are shown. The entropy upper bound 2.197 is
shown as the blue dotted line. We see that MM trained model reaches this upper
bound, while MaxNTE trained model does not.

We first compare our proposed MM with MaxNTE. Then we quantitatively compare
our proposed method with different methods on FGVC tasks as well as the
standard visual classification tasks. Finally we conduct visualization to
further show the effect of our method.

###  3.2 Comparison between MaxNTE and MM

We compare MM with the MaxNTE that directly minimize Eq. 2 on CIFAR-10
dataset. Note that when $p_{t}$ set to 1 for all classes (i.e. $1-p_{t}$ set
to 0) and $\lambda$ set to 0, both of the losses are equal to the cross
entropy loss. In MaxNTE, as shown in Fig. 4, the left figure, the test
accuracy is decreasing when $\lambda$ is increasing, the maximum entropy
reaches its bottleneck at about 2.12. The right figure shows that as $1-p_{t}$
(hyperparameter) increases, the entropy gradually approaches its upper bound
($\log(n-1)=\log 9\approx 2.19$) with stable test accuracy. The results show
that MM can achieve our goal in a more efficient way, thus we apply it to the
following experiments.

###  3.3 Quantitative results

##### Fine-Grained Visual Classification

From Table 1, our proposed method improves the performance of three baseline
models (i.e., ResNet-18, VGGNet-11 and DenseNet-161) across all three datasets
(i.e., CUB-200-2011, FGVC-Aircraft and Stanford Cars). For example, training
VGGNet-11 with MM obtained significant improvements of 2.50% on average across
three datasets compared with CE. LS performs better than CE since it also
encourages the model to produce an output close to our objective. Besides,
DenseNet-161 with MM achieves best results compared with other baselines.

Table 1: Comparison with three baseline models. Backbone | Method | CUB | Aircraft | Cars
---|---|---|---|---
ResNet-18 | CE |  81.32 $\pm$ 0.31 |  89.89 $\pm$ 0.14 |  88.50 $\pm$ 0.21
LS |  81.83 $\pm$ 0.22 |  89.77 $\pm$ 0.27 |  91.06 $\pm$ 0.18
MM | 83.14 $\pm$ 0.18 | 90.37 $\pm$ 0.14 | 91.74 $\pm$ 0.11
VGGNet-11 | CE |  77.76 $\pm$ 0.28 |  85.38 s$\pm$ 0.66 |  87.32 $\pm$ 0.47
LS |  77.94 $\pm$ 0.23 |  87.57 $\pm$ 0.19 |  89.46 $\pm$ 0.32
MM | 80.41 $\pm$ 0.15 | 87.72 $\pm$ 0.22 | 89.83 $\pm$ 0.28
DenseNet-161 | CE |  86.69 $\pm$ 0.32 |  90.94 $\pm$ 0.15 |  94.21 $\pm$ 0.12
LS |  87.63 $\pm$ 0.15 |  92.65 $\pm$ 0.21 |  94.27 $\pm$ 0.16
MM | 87.98 $\pm$ 0.14 | 93.34 $\pm$ 0.19 | 94.72 $\pm$ 0.11

The overall experimental results compared with recent works including SOTA are
shown in Table 2. Our methods outperforms regularization-based methods (e.g.
MaxEnt and PC) across all three datasets. While SOTA models achieve excellent
results, they rely on extra structure or computational cost. Our proposed
methods can fully bring out the potential of the baseline models and achieve
SOTA in FGVC-Aircraft and Stanford Cars by only regularization.

Table 2: Comparison with SOTA methods. * means the best performance among regularization-based methods. Method | CUB | Aircraft | Cars
---|---|---|---
B-CNN ((Lin et al., 2015)) | 84.1 | 84.1 | 91.3
CBP ((Gao et al., 2016)) | 84.3 | 84.1 | 91.2
KP ((Cui et al., 2017)) | 86.2 | 86.9 | 92.4
iSQRT-COV ((Li et al., 2018)) | 88.7 | 91.4 | 93.3
MA-CNN ((Zheng et al., 2017)) | 86.5 | 91.8 | 92.8
RA-CNN ((Fu et al., 2017)) | 85.3 | 92.5 | 93.0
MAMC ((Sun et al., 2018)) | 86.5 | — | 93.0
DFL-CNN ((Wang et al., 2018)) | 87.4 | — | 93.8
NTS-Net ((Yang et al., 2018)) | 87.5 | 91.4 | 93.9
MaxEnt ((Dubey et al., 2018b)) | 86.5 | 89.2 | 92.9
PC ((Dubey et al., 2018a)) | 86.9 | 89.8 | 93.0
DCL ((Chen et al., 2019)) | 87.8 | 93.0 | 94.5
S3N ((Ding et al., 2019) | 88.5 | 92.8 | 94.7
DF-GMM ((Wang et al., 2019b) | 88.8 | 93.8 | 94.8
MGE-CNN ((Zhang et al., 2019) | 89.4 | — | 93.9
GCL ((Wang et al., 2020) | 88.3 | 93.2 | 94.0
API-Net ((Zhuang et al., 2020)) | 90.0 | 93.9 | 95.3
ELoPE ((Hanselmann & Ney, 2020) | 88.5 | 93.5 | 95.0
DFL ((Liu et al., 2020) | 89.1 | 93.4 | 94.3
CIN ((Gao et al., 2020)) | 88.1 | 92.8 | 94.5
ACNet ((Ji et al., 2020) | 88.1 | 92.4 | 94.6
DenseNet161+MM(Ours) | 88.0 | 93.3 | 94.7
DenseNet161+MM+FRL(Ours) | 88.5* | 94.0 | 95.2*
Table 3: Comparison with three baseline models on standard visual classification tasks. Backbone | Method | CIFAR-10 | CIFAR-100 | STL-10
---|---|---|---|---
VGGNet-11 | CE |  92.26$\pm$0.08 |  70.37$\pm$0.33 |  79.80$\pm$0.31
CP | 92.62$\pm$0.05 |  70.30$\pm$0.19 |  80.17$\pm$0.14
LS |  92.28$\pm$0.06 |  71.34$\pm$0.07 |  80.41$\pm$0.08
MM |  92.43$\pm$0.06 | 71.62$\pm$0.18 | 82.26$\pm$0.09
ResNet-18 | CE |  94.94$\pm$0.12 |  75.79$\pm$0.03 |  83.44$\pm$0.23
CP |  95.11$\pm$0.01 |  76.01$\pm$0.31 |  83.75$\pm$0.02
LS |  95.08$\pm$0.11 |  76.24$\pm$0.21 |  84.03$\pm$0.01
MM | 95.33$\pm$0.12 | 76.64$\pm$0.07 | 85.42$\pm$0.04

##### Standard Visual Classification

We compare our method with two output regularization based methods: Confidence
Penalty (CP)(Pereyra et al., 2017) and Label Smoothing (LS)(Szegedy et al.,
2016). As shown in Table 3. our method outperforms several output
regularization based methods across almost all the datasets and architectures.
In CIFAR-10 and CIFAR-100, the improvements are not significant, and the test
accuracy of VGGNet-11 with confidence penalty is slightly higher than that of
our method in CIFAR-10. In STL-10, our method outperforms three baselines by a
large margin.

##### Ablation Study

We perform ablation experiments to show how different parts of our method
work. As shown in Table 4, both MM and FRL greatly improves the performance of
the baseline model. MM brings greater improvement overall, because CE can not
precisely encode the unique feature even with FRL. Moreover, as FRL works by
reducing the similarity of different feature maps, the effect of FRL on
baseline model shows that feature redundancy also exists in regular training
using CE. It is worth noting that hyper-parameters $\lambda$, $K$ and $p_{t}$
have little affect on the proposed MM and FRL.

Table 4: Ablation study using DenseNet-161 as the baseline model. Minimax | FRL | CUB | Aircraft | Cars
---|---|---|---|---
|  |  86.69 $\pm$ 0.32 |  91.26 $\pm$ 0.15 |  94.21 $\pm$ 0.12
$\checkmark$ |  |  87.98 $\pm$ 0.14 |  93.34 $\pm$ 0.19 |  94.74 $\pm$ 0.11
| $\checkmark$ |  87.14 $\pm$ 0.16 |  92.71 $\pm$ 0.48 |  94.71 $\pm$ 0.04
$\checkmark$ | $\checkmark$ | 88.48 $\pm$ 0.24 | 93.96 $\pm$ 0.11 | 95.18 $\pm$ 0.14

###  3.4 Qualitative Result

Figure 5: The comparison of the learned filters of models trained with CE and
our MM. We select the three images of similar bird species to demonstrate the
effect of our method. The two channels with the largest activation values
according to the forward pass of Red Winged Blackbird (the images in the first
row) are presented. Although CE trained filters are able to capture the
important features, they are confused with other categories. MM trained
filters can extract precisely the unique features which will not be activated
when encounter similar objects. Figure 6: The comparison of the filters with
largest activation values of different models. The left side present the
original image. On the right side, the first row is the model trained with CE
as the baseline. The second row is the model trained with MM and the third row
is the model trained with MM and FRL. Compared with baseline, MM minimizes the
activation of irrelevant regions, so as to produce a clear and focused
activation map. On the basis of the effect of MM, FRL encourages the model to
focus on different discriminative regions.

To show in detail how our approach works, we visualize the penultimate layer
feature maps with top activation values. We up-sample the feature maps to
match the original image by bi-linear interpolation. As shown in Fig. 5, CE
trained model (in the first row, column 2, 3) correctly localize the important
parts of the bird, but it will cover irrelevant areas, which may lead to false
triggering of other features (see the second and the third row). The filters
trained by MM that responsible for detecting the unique features of Red Winged
Blackbird do not response to other species as CE trained filters do.

Fig. 6 shows the second row demonstrate the different effects of our proposed
MM and FRL. With our regularization, the model become more concentration so as
to avoid introducing information about irrelevant categories. However, the
regularization objective will lead to redundancy features, i.e. all the
feature maps point on the same area, which means that the model become
overdependent on single feature, and may be harmful for generalization. FRL
well eliminates this problem by forcing the model to distract its attention.
The final results are shown in the last row of Fig. 6, the model attention
become both focused and diversified. With a clearer activation maps, we can
better see how the model works.

##  4 Conclusion

In this work, we provide an information theoretic point of view, to address
the major challenge in FGVC, i.e., learning the features unique to categories.
We formulate the aim to minimizing the MI between the learned features and
non-target classes, based on which we deduce an explicit regularization
objective. To efficiently achieve our objective, we construct a game-theory
based framework to derive a stable minimax loss, which is proved to converge
to a Nash equilibrium. Furthermore, FRL is proposed to avoid over depending on
single feature as a complement of MM. As a result, the model is able to
extract the most distinctive parts of the object and reduce the influence of
background noise. By only regularization, our proposed methods bring the
potential of the baseline models into full play and achieves competitive
results with SOTA models without extra computational cost.

## References

  * Belghazi et al. (2018) Belghazi, M. I., Baratin, A., Rajeshwar, S., Ozair, S., Bengio, Y., Courville, A., and Hjelm, D.  Mutual information neural estimation.  In _International Conference on Machine Learning_ , pp. 531–540, 2018.
  * Bowling & Veloso (2000) Bowling, M. and Veloso, M.  An analysis of stochastic game theory for multiagent reinforcement learning.  2000\.
  * (3) Branson, S., Van Horn, G., Belongie, S., Perona, P., and Tech, C.  Bird species categorization using pose normalized deep convolutional nets.
  * Chen et al. (2019) Chen, Y., Bai, Y., Zhang, W., and Mei, T.  Destruction and construction learning for fine-grained image recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 5157–5166, 2019.
  * Coates et al. (2011) Coates, A., Ng, A., and Lee, H.  An analysis of single-layer networks in unsupervised feature learning.  In _Proceedings of the fourteenth international conference on artificial intelligence and statistics_ , pp. 215–223, 2011.
  * Cover & Thomas (2012) Cover, T. M. and Thomas, J. A.  _Elements of Information Theory_.  John Wiley & Sons, 2012.
  * Cui et al. (2017) Cui, Y., Zhou, F., Wang, J., Liu, X., Lin, Y., and Belongie, S.  Kernel pooling for convolutional neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 2921–2930, 2017.
  * Ding et al. (2019) Ding, Y., Zhou, Y., Zhu, Y., Ye, Q., and Jiao, J.  Selective sparse sampling for fine-grained image recognition.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 6599–6608, 2019.
  * Dubey et al. (2018a) Dubey, A., Gupta, O., Guo, P., Raskar, R., Farrell, R., and Naik, N.  Pairwise confusion for fine-grained visual classification.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pp. 70–86, 2018a.
  * Dubey et al. (2018b) Dubey, A., Gupta, O., Raskar, R., and Naik, N.  Maximum-entropy fine grained classification.  In _Advances in Neural Information Processing Systems_ , pp. 637–647, 2018b.
  * Ferreira et al. (2012) Ferreira, M. A. M., Andrade, M., Matos, M. C. P., Filipe, J. A., and Coelho, M. P.  Minimax theorem and nash equilibrium.  2012\.
  * Fu et al. (2017) Fu, J., Zheng, H., and Mei, T.  Look closer to see better: Recurrent attention convolutional neural network for fine-grained image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 4438–4446, 2017.
  * Gao et al. (2016) Gao, Y., Beijbom, O., Zhang, N., and Darrell, T.  Compact bilinear pooling.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 317–326, 2016.
  * Gao et al. (2020) Gao, Y., Han, X., Wang, X., Huang, W., and Scott, M.  Channel interaction networks for fine-grained image categorization.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pp. 10818–10825, 2020.
  * Goodfellow (2016) Goodfellow, I.  Nips 2016 tutorial: Generative adversarial networks.  _arXiv preprint arXiv:1701.00160_ , 2016.
  * Goodfellow et al. (2014) Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y.  Generative adversarial nets.  In _Advances in neural information processing systems_ , pp. 2672–2680, 2014.
  * Goodfellow et al. (2016) Goodfellow, I., Bengio, Y., and Courville, A.  _Deep Learning_.  MIT Press, 2016.  http://www.deeplearningbook.org.
  * Hanselmann & Ney (2020) Hanselmann, H. and Ney, H.  Elope: Fine-grained visual classification with efficient localization, pooling and embedding.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pp. 1247–1256, 2020.
  * He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * Hjelm et al. (2018) Hjelm, R. D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., and Bengio, Y.  Learning deep representations by mutual information estimation and maximization.  _arXiv preprint arXiv:1808.06670_ , 2018.
  * Huang et al. (2017) Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q.  Densely connected convolutional networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 4700–4708, 2017.
  * Ji et al. (2020) Ji, R., Wen, L., Zhang, L., Du, D., Wu, Y., Zhao, C., Liu, X., and Huang, F.  Attention convolutional binary neural tree for fine-grained visual categorization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10468–10477, 2020.
  * Krause et al. (2013) Krause, J., Stark, M., Deng, J., and Fei-Fei, L.  3d object representations for fine-grained categorization.  In _4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13)_ , Sydney, Australia, 2013.
  * Krizhevsky et al. (2009) Krizhevsky, A., Hinton, G., et al.  Learning multiple layers of features from tiny images.  2009\.
  * Li et al. (2018) Li, P., Xie, J., Wang, Q., and Gao, Z.  Towards faster training of global covariance pooling networks by iterative matrix square root normalization.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 947–955, 2018.
  * Lin et al. (2015) Lin, T.-Y., RoyChowdhury, A., and Maji, S.  Bilinear cnn models for fine-grained visual recognition.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 1449–1457, 2015.
  * Liu et al. (2020) Liu, C., Xie, H., Zha, Z.-J., Ma, L., Yu, L., and Zhang, Y.  Filtration and distillation: Enhancing region attention for fine-grained visual categorization.  volume 34, pp. 11555–11562, 2020.
  * Maji et al. (2013) Maji, S., Rahtu, E., Kannala, J., Blaschko, M., and Vedaldi, A.  Fine-grained visual classification of aircraft.  _arXiv preprint arXiv:1306.5151_ , 2013.
  * Myerson (1991) Myerson, R.  Game theory: Analysis of conflict harvard univ.  _Press, Cambridge_ , 1991.
  * Osborne et al. (2004) Osborne, M. J. et al.  _An introduction to game theory_ , volume 3.  Oxford university press New York, 2004.
  * Pereyra et al. (2017) Pereyra, G., Tucker, G., Chorowski, J., Kaiser, Ł., and Hinton, G.  Regularizing neural networks by penalizing confident output distributions.  _arXiv preprint arXiv:1701.06548_ , 2017.
  * Simonyan & Zisserman (2014) Simonyan, K. and Zisserman, A.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * Sun et al. (2018) Sun, M., Yuan, Y., Zhou, F., and Ding, E.  Multi-attention multi-class constraint for fine-grained image recognition.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pp. 805–821, 2018.
  * Szegedy et al. (2016) Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z.  Rethinking the inception architecture for computer vision.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 2818–2826, 2016.
  * Tishby & Zaslavsky (2015) Tishby, N. and Zaslavsky, N.  Deep learning and the information bottleneck principle.  In _2015 IEEE Information Theory Workshop (ITW)_ , pp. 1–5. IEEE, 2015.
  * Wah et al. (2011) Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S.  The caltech-ucsd birds-200-2011 dataset.  2011\.
  * Wang et al. (2019a) Wang, Q., Xie, J., Zuo, W., Zhang, L., and Li, P.  Deep cnns meet global covariance pooling: Better representation and generalization.  _arXiv preprint arXiv:1904.06836_ , 2019a.
  * Wang et al. (2018) Wang, Y., Morariu, V. I., and Davis, L. S.  Learning a discriminative filter bank within a cnn for fine-grained recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 4148–4157, 2018.
  * Wang et al. (2019b) Wang, Z., Wang, S., Zhang, P., Li, H., Zhong, W., and Li, J.  Weakly supervised fine-grained image classification via correlation-guided discriminative learning.  In _Proceedings of the 27th ACM International Conference on Multimedia_ , pp. 1851–1860, 2019b.
  * Wang et al. (2020) Wang, Z., Wang, S., Li, H., Dou, Z., and Li, J.  Graph-propagation based correlation learning for weakly supervised fine-grained image classification.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pp. 12289–12296, 2020.
  * Wei et al. (2016) Wei, X.-S., Xie, C.-W., and Wu, J.  Mask-cnn: Localizing parts and selecting descriptors for fine-grained image recognition.  _arXiv preprint arXiv:1605.06878_ , 2016.
  * Yang et al. (2018) Yang, Z., Luo, T., Wang, D., Hu, Z., Gao, J., and Wang, L.  Learning to navigate for fine-grained classification.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pp. 420–435, 2018.
  * Zhang et al. (2019) Zhang, L., Huang, S., Liu, W., and Tao, D.  Learning a mixture of granularity-specific experts for fine-grained categorization.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 8331–8340, 2019.
  * Zhang et al. (2014) Zhang, N., Donahue, J., Girshick, R., and Darrell, T.  Part-based r-cnns for fine-grained category detection.  In _European conference on computer vision_ , pp. 834–849. Springer, 2014.
  * Zheng et al. (2017) Zheng, H., Fu, J., Mei, T., and Luo, J.  Learning multi-attention convolutional neural network for fine-grained image recognition.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 5209–5217, 2017.
  * Zhuang et al. (2020) Zhuang, P., Wang, Y., and Qiao, Y.  Learning attentive pairwise interaction for fine-grained classification.  _arXiv_ , pp. arXiv–2002, 2020.