# Multi-View Active Fine-Grained Recognition

Ruoyi Du‡, Wenqing Yu‡, Heqing Wang‡, Dongliang Chang‡,
Ting-En Lin, Yongbin Li, and Zhanyu Ma‡
‡Pattern Recognition and Intelligent System Laboratory, School of Artificial
Intelligence
Beijing University of Posts and Telecommunications, Beijing $100876$
{duruoyi, yuwenqing, wangheqing, changdongliang<EMAIL_ADDRESS>

###### Abstract

As fine-grained visual classification (FGVC) being developed for decades,
great works related have exposed a key direction – finding discriminative
local regions and revealing subtle differences. However, unlike identifying
visual contents within static images, for recognizing objects in the real
physical world, discriminative information is not only present within seen
local regions but also hides in other unseen perspectives. In other words, in
addition to focusing on the distinguishable part from the whole, for efficient
and accurate recognition, it is required to infer the key perspective with a
few glances, _e.g._ , people may recognize a “Benz AMG GT” with a glance of
its front and then know that taking a look at its exhaust pipe can help to
tell which year’s model it is. In this paper, back to reality, we put forward
the problem of active fine-grained recognition (AFGR) and complete this study
in three steps: (i) a hierarchical, multi-view, fine-grained vehicle dataset
is collected as the testbed, (ii) a simple experiment is designed to verify
that different perspectives contribute differently for FGVC and different
categories own different discriminative perspective, (iii) a policy-gradient-
based framework is adopted to achieve efficient recognition with active view
selection. Comprehensive experiments demonstrate that the proposed method
delivers a better performance-efficient trade-off than previous FGVC methods
and advanced neural networks. Codes are available at: https://github.com/PRIS-
CV/AFGR.

##  1 Introduction

Aiming at recognizing the sub-categories of objects belong to the same class,
in the past two decades, research on fine-grained visual classification (FGVC)
has yielded extensive outstanding arts lin2015bilinear ; xiao2015application ;
fu2017look ; wang2018learning ; chen2019destruction ; du2020fine ;
chang2021your ; du2021progressive that surpass human experts in many
application scenarios, _e.g.,_ recognizing cars krause20133d ; yang2015large ,
aircraft maji2013fine , birds wah2011caltech ; van2018inaturalist , and foods
min2021large . Despite the great success, the previous efforts on FGVC largely
remain limited to a single-view-based paradigm, _i.e._ , identifying the
visual content within one single static image. This paradigm may be sufficient
for coarse-grained classification where the saturated inter-class differences
are easy to capture (_e.g._ , one can distinguish a coupe from other vehicles
by its streamlined body, seductive engine, or headlamps). However, things are
different for the fine-grained classification scenario where discriminative
clues are rare – one can only dig the subtle structural differences of exhaust
pipes to distinguish between different years’ models of “Benz AMG GT”, and
there is no other way. Predictably, for single-view-based approaches, an image
(view) without discriminative clues existing is completely indistinguishable
at the fine-grained level, which fundamentally limits the model’s theoretical
performance.

Figure 1: Process of the proposed active fine-grained recognition (AFGR).
Instead of visiting all possible views, the model is able to infer the next
discriminative view according to existing visual information. Here we take two
steps for a brief illustration.

Factually, visual recognition is never limited to observing $2$D environments
and processing static images. Vision algorithms equipped by portal devices
(_e.g._ , smartphone, smart glasses, etc.) or embodied AI agents
franklin1997autonomous (_e.g._ , intelligent robots) play the core roles
during machine-environment interaction and have become one of the focuses of
computer vision research. Therefore, to embrace the new trend, a natural
extension of ordinary FGVC follows – in addition to locating discriminative
parts within an image, we aim to infer the unseen distinguishable perspective
within the physical world ($3$D environment). As shown in Figure 1, with a
single glance from the front, the algorithm may be confused about which year’s
model the “Benz AMG GT” is but can infer that looking at its back will help.

Specifically, we re-propose the concept of active vision aloimonos1988active
in the context of FGVC termed active fine-grained recognition (AFGR) with two
essential hypotheses. Firstly, the discriminative information hides in various
object views for different fine-grained categories, which determines that
discriminative perspective inference is non-trivial and worth studying.
Secondly, indistinguishable views also contain visual clues leading to the
discriminative perspective, which ensures the solvability of the problem.

To start with, due to the absence of qualified datasets, we first collect a
hierarchical, fine-grained, multi-view vehicle dataset named Multi-view Cars
(MvCars) as the testbed. MvCars contains $20$ models of cars from $4$ brands
and covers more than one car type (_e.g._ , coupe, SUV, etc.) for each brand.
Furthermore, to ensure the difficulty of MvCars, we include each brand with
two similar categories, _e.g._ , different years’ models of the same series.
There are $7$ aligned views for each car and about five thousand images
included in the dataset. Right after that, our first hypothesis is
verified(see Section 4), which indicates that MvCars is sound for the problem
raised.

Secondly, our next contribution following is an efficient multi-view fine-
grained recognition framework via active next-view selection. In particular,
following the general idea of view-based $3$D object understanding su2015multi
, an extraction-aggregation architecture is designed as the feature encoder,
where a convolutional neural network (_e.g._ , ResNet$50$ he2016deep ) is
first applied to extract single-view features independently, and then a
recurrent neural network (_e.g._ , GRU chung2014empirical ) is adopted to
aggregate multi-view features and form global descriptions. Afterward, we
formulate the next-view selection as a sequential decision process, where the
model is demanded to decide the next discriminative view (action) according to
previously observed views (state). Thus, a proximal policy optimization (PPO)
schulman2017proximal is implemented and revised for training. Note that the
proposed framework does not rely on specific neural network architectures. It
can extend any visual recognition network to an FGVC expert in the 3D
environment.

Finally, several carefully designed baselines are re-produced on MvCars as
benchmark results, including general neural networks in a multi-view
recognition setting, and popular FGVC methods. Instead of time
costs/computation budgets, we adopt the required step numbers for reliable
prediction to measure the model efficiency. This is because the time cost for
acquiring one more view far outweighs the inference cost, and it may need
users’ efforts (for applications on portal devices). The experimental results
demonstrate that the proposed method delivers a better performance-efficient
trade-off than all competitors. After that, an analysis of the upper bound of
the proposed method reveals the FGVC characteristic inherited by AFGR. In
addition, comprehensive ablation studies are carried out to verify the
necessities of each model component.

##  2 Related Work

###  2.1 Fine-Grained Visual Classification

Due to the inherent subtle inter-class variance and the relatively large
intra-class variance, fine-grained visual classification is much more
challenging than ordinary coarse-grained classification. With vigorous efforts
made by researchers, great progress has been made in many directions.
Localization based approaches zhang2014part ; krause2015fine ;
xiao2015application ; fu2017look ; wang2018learning that explicitly locate
discriminative parts for feature extraction to alleviate the intra-class
variance. High-order encoding methods lin2015bilinear ; gao2016compact ;
fu2017look ; yu2018hierarchical ; zheng2019learning that adopt high-order
feature interactions for better representation ability that can capture subtle
difference. Chen _et al._ chen2019destruction and Du _et al._ du2020fine train
the model with jigsaw patches to implicitly encourage knowledge mining from
local regions. Recently, Chang _et al._ chang2021your leverage the underlying
hierarchical structure of fine-grained categories to achieve user-friendly
outputs and better performance.

Except for good performances being brought, these aforementioned works also
reveal that FGVC is never just a harder classification problem but a stand-
alone field that requires well-directed research. In this paper, to further
broaden the horizon of FGVC, we propose the active fine-grained recognition
(AFGR) task aiming at effective recognition of fine-grained categories in the
$3$D environment along with a targeted dataset. It is worth noting that the
CompCars yang2015large dataset also provides a car dataset with view
annotations. However, its multi-view images are taken from different objects,
making it less suitable for the raised problem.

###  2.2 Multi-View Recognition

Elsewhere for ordinary object recognition problems, certain progress has been
made to recognizing $3$D objects with three streams can be summarized
chen2021mvt : point-based methods qi2017pointnet ; qi2017pointnet++ ;
aoki2019pointnetlk ; ni2020pointnet++ , volume-based methods
maturana2015voxnet ; wu20153d ; qi2016volumetric ; meng2019vv , and view-based
methods chiu2007virtual ; su2015multi ; johns2016pairwise ;
kanezaki2018rotationnet ; yu2018multi ; yang2019learning . Among them, point-
based and volume-based approaches demand to perceive the $3$D structure of
objects via lidar, depth sensor or something else, which makes them less
practicable in daliy applications, _e.g._ , recognizing an unfamiliar car for
detailed information simply with a mobile phone. On the contrary, view-based
methods that leverage multiple surrounding $2$D views as descriptors for $3$D
objects tend to be an optimal choice.

Specifically, view-based methods share the core idea that encoding single-view
features through vision neural network and then aggregating multi-view
features. Su _et al._ su2015multi first approaches the multi-view recognition
problem with CNN for feature extraction and sum-pooling for aggregation. Then,
Johns _et al._ johns2016pairwise decomposes image sequences into image pair
sets, and then aggregates the pair-based classification in a weighted manner.
After that, feature concatenation wang2019dominant , hierarchical attention
han20193d2seqviews , and weighted fusion feng2018gvcnn are also adopted for
better aggregating sequence features. In addition, sequences models (_e.g._ ,
LSTM hochreiter1997long , GRU chung2014empirical , Transformer blocks
vaswani2017attention , etc) are also widely considered jayaraman2016look ;
han2018seqviews2seqlabels ; chen2021mvt and demonstrate their effectiveness.

In this paper, specifically towards the active fine-grained recognition (AFGR)
task we raised, traditional multi-view recognition dataset (_e.g._ , RGB-D
lai2011large , ModelNet10, ModelNet40 wu20153d ) is not sufficient any more.
Thus, we first collect a fine-grained, multi-view vehicle dataset named MvCars
as our testbed. Then, an active fine-grained recognition framework is built
upon the general extraction-aggregation scheme. Note that, similar to ours,
some approaches also take recognition efficiency into consideration
jayaraman2016look ; johns2016pairwise by actively controlling the agent motion
within a viewing sphere. While a strict viewing sphere is not readily
available in daily applications, especially for recognition with portable
devices, hence we consider the view selection as a discontinuous
classification problem here.

##  3 Methodology

Figure 2: Illustration of the proposed AFGR framework where three training
stages are included: Stage I for training a multi-view recognition model with
smooth predictions, Stage II for optimizing the next-view selection component
based on the behavior of the classifier, and Stage III for fine-tuning the
recognition model along with the trajectory decided by the actor. Here we use
three training steps for brief illustration.

###  3.1 Overview

Here we first give an overview of data flow during inference along with the
setting of active fine-grained recognition (AFGR).

Data structure. For AFGR, a dataset consists of $N$ samples can be expressed
as $\\{X_{i},y_{i}\\}_{i=1}^{N}$, where
$X_{i}=\\{x_{i}^{1},\dots,x_{i}^{v},\dots,x_{i}^{V}\\}$ is a sequence of
images depict a specific sample from $V$ perspectives and $y_{i}$ is their
common ground-truth label. Note that, for arbitrary two samples $X_{i}$ and
$X_{j}$, $x_{i}^{v}$ and $x_{j}^{v}$ are taken from the same perspective,
which means the annotations of views $\\{1,\dots,V\\}$ are aligned.

Inference process. For sample $X_{i}$, the model will take an image
$x_{i}^{v_{1}}$ from arbitrary view $v_{1}$ as the initial visual input, which
simulates the situation that the model may start recognition while facing any
views of the target object. After that, the recognition process will carry on
step-by-step. In particular, at step $t$ with input $x_{i}^{v_{t}}$ from view
$v_{t}$, the model will utilize all currently perceived information
$\\{x_{i}^{v_{1}},\dots,x_{i}^{v_{t-1}},x_{i}^{v_{t}}\\}$ to deliver the
category prediction $\hat{y}_{i}^{t}$ and the next-view proposal $v_{t+1}$.
Then, a inference cycle is closed, and the process can keep going with
$x_{i}^{v_{t+1}}$ as the next input.

Framework component. To process a sequence of correlated visual inputs, an
extraction-aggregation structure tends to be an intuitive choice.
Specifically, for any image $x_{i}^{v_{t}}$ input the system, a CNN-based
feature extractor $\mathcal{F}(\cdot)$ is first applied to extract single-view
feature as $f_{i}^{v_{t}}=\mathcal{F}(x_{i}^{v_{t}})$. It is worth reminding
that the feature extractors for different views share their weights and this
design will not leads to additional parameters. After that, an ideal model
should take all previously acquired information into consider. Thus, a
recurrent neural network is introduced as the aggregator $\mathcal{R}(\cdot)$
that aggregates features from all seen perspectives. In particular, here we
adopts two aggregator with same structure but individual weights
$\mathcal{R}_{e}(\cdot)$ and $\mathcal{R}_{s}(\cdot)$ that,
$\mathcal{R}_{e}(\cdot)$ form global embeddings
$e_{i}^{t}=\mathcal{R}_{e}(f_{i}^{v_{1}},\dots,f_{i}^{v_{t}})$ for category
prediction, while $\mathcal{R}_{s}(\cdot)$ depicts the current states
$s_{i}^{t}=\mathcal{R}_{s}(f_{i}^{v_{1}},\dots,f_{i}^{v_{t}})$ for next-view
selection. Finally, a classifier $\mathcal{P}(\cdot)$ and an actor
$\mathcal{A}(\cdot)$ are equipped in parallel with outputs
$\hat{y}_{i}^{t}=\mathcal{P}(e_{i}^{t})$ and $v_{t}=\mathcal{A}(s_{i}^{t})$,
respectively.

###  3.2 Model Training

According to the aforementioned inference process, we can tell that the
recognition component and the next-view selection component work in a separate
but not independent manner. The mission of the recognition component is quite
straightforward – conducting category prediction based on acquired information
as well as possible. While the optimization goal of the next-view selection
component largely depends on the behavior of recognition – basically, the
actor should try to select the next-view that can maximize the prediction
probability of the target category. Therefore, a three stages training
framework is intuitively designed: Stage I aims to train a good recognition
model (including $\mathcal{F}(\cdot)$, $\mathcal{R}_{e}(\cdot)$, and
$\mathcal{P}(\cdot)$) that can handle sequence input, Stage II aims to
optimize next-view selection (where $\mathcal{R}_{s}(\cdot)$ and
$\mathcal{A}(\cdot)$ participate) according to the behavior of the trained
recognition model, and Stages III aims to refine the recognition model under
the trajectories decided by the actor. The whole framework is illustrated in
Figure 2, and introductions about the three stages are as follows.

Stage I. We first train a recognition model that can handle a sequence of
inputs with dynamic length. Each training iteration is divided into $T$ steps
with input sequence lengths from $1$ to $T$. For the $t$-th step, a new image
$x_{i}^{v_{t}}$ is randomly selected from unseen views and appended to the
input sequence at the $(t-1)$-th step. Here we set $T=V$ to ensure the
sequence is no-duplicated. Thus, with cross-entropy for optimization, the loss
function for a batch of $B$ samples can be formulated as:

$L_{CE}(\hat{y}_{i}^{t},y_{i})=\frac{-1}{BT}\sum_{i=1}^{B}\sum_{t=1}^{T}y_{i}\times
log(\hat{y}_{i}^{t}).$ (1)

Note that the inductive bias behind training the recognition component in the
first place is that its behavior can reveal view discrimination – a more
discriminative view will greatly reduce the entropy of category prediction.
However, a well-convergent classification model often tends to deliver high
confidence predictions, especially for the small-scale datasets in the FGVC
scenario, which will cause little changes in prediction probabilities and
limit the information being revealed. Therefore, we further introduce an
entropy maximization constraint to encourage smooth predictions. Specifically,
let $p_{i}^{t}$ be the output of the classifier before the softmax function. A
softer version of the prediction can be obtained by introducing a pre-defined
temperature $h$, which is expressed as:

$\hat{y}_{i,j}^{{}^{\prime}t}=\frac{exp(p_{i,j}^{t})/h}{\sum_{k}exp(p_{i,k}^{t})/h},$
(2)

where $j$ and $k$ indicate channel index of $p_{i}^{t}$. Then, we minimize the
Euclidean Distance between $\hat{y}_{i}^{t}$ and $\hat{y}_{i}^{{}^{\prime}t}$
to achieve entropy maximization as:

$L_{EM}(\hat{y}_{i}^{t},\hat{y}_{i}^{{}^{\prime}t})=\frac{-1}{BT}\sum_{i=1}^{B}\sum_{t=1}^{T}|\hat{y}_{i}^{t}-\hat{y}_{i}^{{}^{\prime}t}|^{2}.$
(3)

The total loss of Stgae I is
$L_{Stage1}=L_{CE}(\hat{y}_{i}^{t},y_{i})+L_{EM}(\hat{y}_{i}^{t},\hat{y}_{i}^{{}^{\prime}t})$,
and the degree of entropy maximization constrain can be control by different
temperature $h$.

Stage II. Here, the recognition components ($\mathcal{F}(\cdot)$,
$\mathcal{R}_{e}(\cdot)$, and $\mathcal{P}(\cdot)$) are frozen, and we only
optimize $\mathcal{R}_{s}(\cdot)$ and $\mathcal{A}(\cdot)$ for next-view
selection. As a sequential decision problem, we adopt policy gradient method
for optimization instead of directly optimizing with the classification loss,
since the view selection process is non-differentiable. At the $t$-th ($t\geq
2$) training step, the model will receive the input $x_{i}^{v_{t}}$ with the
perspective $v_{t}$ decide by the actor at the $t-1$-th step. Then the view
selection components can be updated according to the change of target category
prediction probability, _i.e._ , the rewards is set as
$r_{i}^{t}=\hat{y}_{iy_{i}}^{t}-\hat{y}_{iy_{i}}^{t-1}$. And the $t$-th
($t\geq 2$) step’s loss function of Stage II can be simply expressed as:
$L_{stage2}=L_{PG}(v_{t},\hat{y}_{i}^{t-1},\hat{y}_{i}^{t})$.

It is worth noting that, for popular policy gradient algorithms
schulman2016high ; schulman2017proximal , the total reward for the current
step’s optimization is a (weighted) sum of all feature rewards from now on.
This is because these methods are designed for scenarios where an agent is
required to achieve an ultimate goal through a series of actions. However, on
the contrary, AFGR aims at using as few steps as possible to achieve as high
accuracy as possible, _i.e._ , we care more about how to achieve the best
performance at the current step rather than in the future. Therefore, we
slightly modify the policy gradient algorithm by utilizing only $r_{i}^{t}$
for the $t$-th step’s optimization.

Stage III. There is nothing new in this stage, all settings are the same as
Stage I except for (i) the selected view $v_{t}$ when $t\geq 2$ is given by
the actor, and (ii) the entropy maximization constraint is removed (_i.e._ ,
$L_{Stage1}=L_{CE}(\hat{y}_{i}^{t},y_{i})$). We hope the model can be refined
under standard classification supervision (_i.e._ , purely with the cross-
entropy loss) to especially adjust the trajectories decided by the actor.

###  3.3 Design Details

Feature extractor $\mathcal{F}(\cdot)$. The feature extractor can be any
backbone network for vision tasks, including various CNN architectures and
Transformers. Besides, by replacing $\mathcal{F}(\cdot)$ with other FGVC
models, the proposed method can also extend them to work in $3$D environments.

Feature aggregator $\mathcal{R}_{e}(\cdot)$ and $\mathcal{R}_{s}(\cdot)$. The
two feature aggregators should be able to aggregate information from sequences
with variable lengths. Here we adopt GRU chung2014empirical for best
performance. There are also alternatives like LSTM hochreiter1997long , self-
attention block vaswani2017attention , etc, which we will discuss in Appendix
B.

Classifier $\mathcal{P}(\cdot)$ and actor $\mathcal{A}(\cdot)$. Both the
classifier and the actor are formed by one fully connected layer. For the
cases that equip the proposed framework with other FGVC approaches, the
structure of the classifier can be modified accordingly.

Policy gradient algorithm. We adopt the proximal policy optimization (PPO)
schulman2017proximal for the training of next-view selection with the reward
of the current step only. Details can be found in Appendix A.

##  4 Dataset

Figure 3: The quantitative analysis of the collected MvCars dataset. The
broken-lines show model accuracy based on $7$ individual views. And the bars
represent the differences between the maximum and minimum accuracy for each
category.

Data collection and statistic. The Multi-view Cars (MvCars) dataset is
collected from $4$ automobile sale sites1111.www.autohome.com
2.www.yiche.com
3.www.dongchedi.com
4.www.pcauto.com.cn where cars are displayed from different perspectives. To
ensure the diversity and representativeness of MvCars, we choose $20$ models
of cars from $4$ popular brands (Mercedes-Benz, Volkswagen, Toyota, and
Nissan) where each brand contains cars of at least $2$ types (_e.g._ , coupe,
SUV, etc). For each car, we annotated $7$ aligned perspectives – front-left,
front-right, side-front, side-middle, side-back, back-left, and back-right,
and samples with missing perspectives are discarded. In total, there are
$4669$ images collected and then split into $2450$/$2219$ for train/test set,
respectively.

Quality verification. With the collected MvCars, here we first experimentally
validate our first hypothesis mentioned in Section 1 – the discriminative
information hides in various object views for different fine-grained
categories. Factually, it is two-fold: (i) different perspectives contribute
differently to FGVC, otherwise, actively selecting object view is meaningless,
and (ii) different categories own different discriminative perspectives,
otherwise, there is a trivial solution existing – consistently seeking the
fixed distinguishable view.

In particular, for each perspective, we train a ResNet50 he2016deep for
classification and obtain its accuracy in each category. Therefore, for any
specific category, we can tell which perspective is more distinguishable by
comparing the performances of $7$ models based on different views. The
experimental results are shown in Figure 3. Bars in the graph indicate the
differences between the maximum and minimum accuracy of each category, where
we can observe that the differences are about $50.5\%$ on average and at least
more than $30.3\%$. It powerfully proves that different perspectives
contribute differently in the context of FGVC. On the other hand, broken-lines
in the graph represent view accuracy changes along with different categories.
The interaction of lines indicates that the ranking of view discrimination is
not consistent, demonstrating that different categories have different
discriminative perspectives.

In one word, in MvCars, different perspectives provide significantly various
meanings for FGVC, which is also hard to pre-defined via prior knowledge.
Thus, an active recognition method is called for, and the collected MvCars
dataset can serve as an eligible testbed.

##  5 Experiment

Table 1: Results of the proposed method against different baselines. The table
is divided into five sections according to different backbones, and the best
results of each section are marked in bold.

Method Backbone mAcc. ($\%$) w-mAcc. ($\%$) Step$2$-Acc. ($\%$) Hierarchical
BCNN ResNet$50$ $81.37\pm 0.2$ $81.57\pm 0.3$ $80.90\pm 0.6$ Pairwise
Confusion ResNet$50$ $82.35\pm 0.5$ $82.25\pm 0.6$ $81.48\pm 0.5$ CrossX
ResNet$50$ $84.92\pm 0.4$ $85.13\pm 0.3$ $84.79\pm 0.3$ PMG ResNet$50$
$84.68\pm 0.5$ $84.72\pm 0.6$ $84.10\pm 0.6$ CAL ResNet$50$ $84.23\pm 0.1$
$84.54\pm 0.3$ $84.39\pm 0.5$ Sequence Baseline ResNet$50$ $84.18\pm 1.2$
$84.56\pm 1.3$ $83.05\pm 1.7$ Ours ResNet$50$ $\mathbf{86.07\pm 0.5}$
$\mathbf{87.66\pm 0.5}$ $\mathbf{87.20\pm 0.4}$ Sequence Baseline
DenseNet$169$ $85.43\pm 0.7$ $85.70\pm 0.7$ $85.07\pm 1.0$ Ours DenseNet$169$
$\mathbf{85.92\pm 0.9}$ $\mathbf{86.53\pm 0.8}$ $\mathbf{86.03\pm 0.6}$
Sequence Baseline EfficientNet_b$3$ $83.70\pm 0.6$ $83.85\pm 0.5$ $81.98\pm
0.4$ Ours EfficientNet_b$3$ $\mathbf{84.67\pm 0.3}$ $\mathbf{85.27\pm 0.6}$
$\mathbf{83.91\pm 0.6}$ Sequence Baseline RegNetY_$1.6$GF $84.38\pm 0.1$
$84.75\pm 0.5$ $83.91\pm 1.0$ Ours RegNetY_$1.6$GF $\mathbf{84.75\pm 0.1}$
$\mathbf{85.22\pm 0.1}$ $\mathbf{84.44\pm 0.2}$ TransFG ViT-B_$16$ $81.18\pm
0.4$ $81.09\pm 0.4$ $80.33\pm 0.2$ Sequence Baseline ViT-B_$16$ $80.61\pm 0.7$
$81.26\pm 0.9$ $79.96\pm 0.9$ Ours ViT-B_$16$ $\mathbf{81.44\pm 0.8}$
$\mathbf{82.23\pm 1.0}$ $\mathbf{81.30\pm 0.9}$

In this section, first, we introduce the baseline models for comparison and
the metrics for evaluation. Then we discuss the comparison results in Section
5.1. After that, we discuss the performance upper bound of our model in
Section 5.2. Finally, ablation studies are carried out in Section 5.3 to
verify our design choices. In addition, the implementation details can be
found in Appendix A., and additional ablation studies about hyper-parameters
and network architectures can be found in Appendix B.

Baseline models. For extensively evaluation, two groups of baseline methods
are designed and implemented. The first is state-of-the-art FGVC methods,
including Hierarchical BCNN yu2018hierarchical , Pairwise Confusion
dubey2018pairwise , CrossX luo2019cross , PMG du2020fine , CAL
rao2021counterfactual , and TransFG he2021transfg . To extend these approaches
to the multi-view recognition scenario, we employ a naive model ensemble
scheme, _i.e._ , at the $t$-th step, the average of $t$ inputs’ predictions is
adopted as the current result. The second group is advanced vision neural
networks, including ResNet he2016deep , DenseNet huang2017densely ,
EfficientNet tan2019efficientnet , RegNet-Y radosavovic2020designing , and ViT
dosovitskiy2020image . Due to their conciseness (no complicated training
strategies or carefully designed structures), we can easily implement them in
the extraction-aggregation form (more general for multi-view recognition
jayaraman2016look ; han2018seqviews2seqlabels ; chen2021mvt ) with GRU
chung2014empirical for feature aggregation. The second sequence-based baseline
group is also used to demonstrate the generalization ability of the proposed
framework by serving as the recognition model trained in Stage I. Note that,
for these baseline methods, the input of each step is randomly selected with
no duplicate view.

Evaluation Metrics. For quantitative evaluation, results based on $3$ metrics
are reported: (i) Mean Accuracy (mAcc) that takes the mean value of all $T$
steps’ accuracy, which can be regarded as the area under the accuracy-step
line that represents the general performances of models, (ii) Weighted Mean
Accuracy (w-mAcc) that weights different steps with exponentially decreased
weights, since the performance of the first few steps should be more important
in the consideration of efficiency222Here we take
$[0.0000,0.5079,0.2540,0.1270,0.0635,0.0317,0.0159]$ for w-mAcc when $T=7$.
The accuracy of the first step is weighted by $0.0$ because it is randomly
selected and does not relate to the performance of active selection., and
(iii) Step$2$ Accuracy (Step$2$-Acc) that takes the $2$-nd step’s accuracy to
highlight the profit of the first view selection333Step$2$-Acc can be regarded
as w-mAcc with weight set $[0.0,1.0,0.0,0.0,0.0,0.0,0.0]$.. In addition,
following wang2020glance , we introduce a dynamic exit strategy to further
reveal the model potential under given step expectations – given the
expectation of step number, confidence thresholds for exiting inference at
each step are dynamically defined according to the training data, which
enables better resource allocation among all test data (details can be found
in appendix A.).

###  5.1 Main Results

Figure 4: Accuracy-step lines/curve of the proposed method against
competitors.

Figure 5: The performance upper bound from the perspective of trajectory
decision.

The results of the proposed method against all mentioned baselines are
reported in Table 1. The table is organized into $5$ sections according to
different backbone networks, and we mainly focus on the comparison within each
section for fairness. For ResNet$50$ he2016deep as the base model, we can
observe that the sequence-based model (sequence baseline in the table), aiming
at multi-view recognition, delivers quite competitive results that even
consistently surpass FGVC methods like Hierarchical BCNN yu2018hierarchical
and Pairwise Confusion dubey2018pairwise . On the contrary, the proposed
method outperforms it by $\sim 2.\%$, $\sim 3.\%$, $\sim 4.\%$ for mAcc,
w-mAcc, and Step$2$-Acc, respectively. The larger margins on w-mAcc and
Step$2$-Acc also demonstrate its superiority in efficiency that benefits from
the active next-view selection scheme. Besides, there is no doubt that our
framework obtains state-of-the-art performance with any backbone networks,
which indicates its robustness and generalization ability.

To better illustrate the change of model accuracy over inference steps, we
show the accuracy-step lines of all models with ResNet$50$ as the backbone in
Figure 5. In addition, we also include the curve formed by the dynamic exit
strategy wang2020glance for our model. Firstly, we can observe that when the
step number $t=1$, _i.e._ , the prediction is conducted based on a single
image, FGVC approaches demonstrate their professionalism by outperforming both
the proposed method and the sequence baseline. This is reasonable since the
proposed one is just a ResNet$50$-based classification model with random
inputs when $t=1$. However, for $t\geq 2$, our model immediately dominated the
game – specifically, it surpasses all competitors with significant margins
when $t=2$, echoing the results of Step$2$-Acc in Table 1. We attribute this
to the effectiveness of our next-view selection mechanism. Last but not least,
with a better resource allocation brought by the exit strategy via dynamic
sequence length arrangement, a significant further improvement can be observed
in the first few steps – we can obtain the best performance with $\sim 2.$
steps less.

At this point, the audiences may question why our model’s performance does not
consistently increase. With the same question, we study the upper bound of our
model in the next subsection.

###  5.2 Upper Bound Analysis

Due to the finite total view numbers, we are able to visit all possible
trajectories for each sample. Therefore, a performance upper bound can be
obtained from the perspective of trajectory decision. In particular, given a
sequence length, any sample can be regarded as a correctly classified sample
long as there exists one trajectory that can yield the correct prediction. As
shown in Figure 5, the degradation in the last few steps is also observed on
our upper bound. We attribute this to the inherent feature of fine-grained
recognition in the $3$D environment – the discriminative clues only hide in a
few views, and the noises caused by intra-class variance will be more likely
to be introduced when full visual information (_i.e._ , all views) is
included. This particularly echoes the essential insight in the 2$D$ fine-
grained recognition where subtle differences of local regions are
discriminative, and the global structures are more likely disturbed.

###  5.3 Ablation Study

Table 2: Results of ablation studies. The best results are marked in bold.

Method Backbone mAcc. ($\%$) w-mAcc. ($\%$) Step$2$-Acc. ($\%$) Random
Selection ResNet$50$ $85.43\pm 0.5$ $86.14\pm 0.7$ $84.87\pm 0.8$ Allow
Duplicate View ResNet$50$ $85.33\pm 0.3$ $87.04\pm 0.3$ $86.72\pm 0.4$ w/o
Entropy Maximization ResNet$50$ $85.91\pm 0.9$ $87.07\pm 0.9$ $86.07\pm 0.9$
w/ Future Rewards ResNet$50$ $86.01\pm 0.6$ $87.10\pm 0.6$ $86.17\pm 0.6$ w/o
Stage III ResNet$50$ $85.31\pm 1.6$ $86.57\pm 1.6$ $85.63\pm 1.7$ Ours
ResNet$50$ $\mathbf{86.07\pm 0.5}$ $\mathbf{87.66\pm 0.5}$ $\mathbf{87.20\pm
0.4}$

In this section, we evaluate several variants of the proposed method based on
ResNet$50$ to demonstrate the necessities of our designs. First, to directly
verify the effectiveness of the active next-view selection mechanism, we study
our model trained via $3$ stages with randomly selected inputs. Fortunately,
the proposed method passes the test with significant margins of $\sim 0.6\%$,
$\sim 1.5\%$, and $\sim 2.3\%$. Additionally, for all evaluations before, an
artificial restriction is added to ensure new views selected are unseen. It is
intuitive since unseen views can offer complementary information, and the
information about which views have been selected is easily acquired. Here we
also evaluate by allowing duplicate views, and the model performance degrades
with no surprise. After that, our designs for model training are also
demonstrated to be effective. It is worth noting that when we include the
future rewards for policy optimization, mAcc is not significantly affected (
with a slight degradation of $0.06\%$), but w-mAcc and Step2-Acc decrease by
$\sim 0.6\%$ and $\sim 1.0\%$. This indicates that future rewards may be
meaningful for traditional sequential decision problems but not for AFGR which
highly requires efficiency.

##  6 Conclusion

In this paper, we extend the fine-grained visual classification to $3$D
environments and put forward the active fine-grained recognition (AFGR)
problem. A multi-view car dataset (MvCars) is collected as a qualified
benchmark. We re-implement several FGVC approaches and several vision neural
networks under a general multi-view recognition scheme as baseline methods. A
policy-gradient-based framework is introduced for the problem raised. The
proposed method yields the best performance on MvCars. We also discuss the
upper bound of our framework from the perspective of trajectory decision.

## Acknowledgments and Disclosure of Funding

## References

  * [1] John Aloimonos, Isaac Weiss, and Amit Bandyopadhyay.  Active vision.  International Journal of Computer Vision, 1988.
  * [2] Yasuhiro Aoki, Hunter Goforth, Rangaprasad Arun Srivatsan, and Simon Lucey.  Pointnetlk: Robust & efficient point cloud registration using pointnet.  In CVPR, 2019.
  * [3] Dongliang Chang, Kaiyue Pang, Yixiao Zheng, Zhanyu Ma, Yi-Zhe Song, and Jun Guo.  Your" flamingo" is my" bird": Fine-grained, or not.  In CVPR, 2021.
  * [4] Shuo Chen, Tan Yu, and Ping Li.  Mvt: Multi-view vision transformer for 3d object recognition.  In BMVC, 2021.
  * [5] Yue Chen, Yalong Bai, Wei Zhang, and Tao Mei.  Destruction and construction learning for fine-grained image recognition.  In CVPR, 2019.
  * [6] Han-Pang Chiu, Leslie Pack Kaelbling, and Tomás Lozano-Pérez.  Virtual training for multi-view object class recognition.  In CVPR, 2007.
  * [7] Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  In NeurIPS Workshops, 2014.
  * [8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR, 2020.
  * [9] Ruoyi Du, Dongliang Chang, Ayan Kumar Bhunia, Jiyang Xie, Zhanyu Ma, Yi-Zhe Song, and Jun Guo.  Fine-grained visual classification via progressive multi-granularity training of jigsaw patches.  In ECCV, 2020.
  * [10] Ruoyi Du, Jiyang Xie, Zhanyu Ma, Dongliang Chang, Yi-Zhe Song, and Jun Guo.  Progressive learning of category-consistent multi-granularity features for fine-grained visual classification.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021\.
  * [11] Abhimanyu Dubey, Otkrist Gupta, Pei Guo, Ramesh Raskar, Ryan Farrell, and Nikhil Naik.  Pairwise confusion for fine-grained visual classification.  In ECCV, 2018.
  * [12] Yifan Feng, Zizhao Zhang, Xibin Zhao, Rongrong Ji, and Yue Gao.  Gvcnn: Group-view convolutional neural networks for 3d shape recognition.  In CVPR, 2018.
  * [13] Stan Franklin.  Autonomous agents as embodied ai.  Cybernetics & Systems, 1997.
  * [14] Jianlong Fu, Heliang Zheng, and Tao Mei.  Look closer to see better: Recurrent attention convolutional neural network for fine-grained image recognition.  In CVPR, 2017.
  * [15] Yang Gao, Oscar Beijbom, Ning Zhang, and Trevor Darrell.  Compact bilinear pooling.  In CVPR, 2016.
  * [16] Zhizhong Han, Honglei Lu, Zhenbao Liu, Chi-Man Vong, Yu-Shen Liu, Matthias Zwicker, Junwei Han, and CL Philip Chen.  3d2seqviews: Aggregating sequential views for 3d global feature learning by cnn with hierarchical attention aggregation.  IEEE Transactions on Image Processing, 2019.
  * [17] Zhizhong Han, Mingyang Shang, Zhenbao Liu, Chi-Man Vong, Yu-Shen Liu, Matthias Zwicker, Junwei Han, and CL Philip Chen.  Seqviews2seqlabels: Learning 3d global features via aggregating sequential views by rnn with attention.  IEEE Transactions on Image Processing, 2018.
  * [18] Ju He, Jie-Neng Chen, Shuai Liu, Adam Kortylewski, Cheng Yang, Yutong Bai, Changhu Wang, and Alan Yuille.  Transfg: A transformer architecture for fine-grained recognition.  In AAAI, 2022.
  * [19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [20] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural Computation, 1997.
  * [21] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In CVPR, 2017.
  * [22] Dinesh Jayaraman and Kristen Grauman.  Look-ahead before you leap: end-to-end active recognition by forecasting the effect of motion.  In ECCV, 2016.
  * [23] Edward Johns, Stefan Leutenegger, and Andrew J Davison.  Pairwise decomposition of image sequences for active multi-view recognition.  In CVPR, 2016.
  * [24] Asako Kanezaki, Yasuyuki Matsushita, and Yoshifumi Nishida.  Rotationnet: Joint object categorization and pose estimation using multiviews from unsupervised viewpoints.  In CVPR, 2018.
  * [25] Jonathan Krause, Hailin Jin, Jianchao Yang, and Li Fei-Fei.  Fine-grained recognition without part annotations.  In CVPR, 2015.
  * [26] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In ICCV workshops, 2013.
  * [27] Kevin Lai, Liefeng Bo, Xiaofeng Ren, and Dieter Fox.  A large-scale hierarchical multi-view rgb-d object dataset.  In ICRA. IEEE, 2011.
  * [28] Tsung-Yu Lin, Aruni RoyChowdhury, and Subhransu Maji.  Bilinear cnn models for fine-grained visual recognition.  In ICCV, 2015.
  * [29] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  arXiv preprint arXiv:1608.03983, 2016.
  * [30] Wei Luo, Xitong Yang, Xianjie Mo, Yuheng Lu, Larry S Davis, Jun Li, Jian Yang, and Ser-Nam Lim.  Cross-x learning for fine-grained visual categorization.  In ICCV, 2019.
  * [31] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi.  Fine-grained visual classification of aircraft.  arXiv preprint arXiv:1306.5151, 2013.
  * [32] Daniel Maturana and Sebastian Scherer.  Voxnet: A 3d convolutional neural network for real-time object recognition.  In IROS, 2015.
  * [33] Hsien-Yu Meng, Lin Gao, Yu-Kun Lai, and Dinesh Manocha.  Vv-net: Voxel vae net with group convolutions for point cloud segmentation.  In ICCV, 2019.
  * [34] Weiqing Min, Zhiling Wang, Yuxin Liu, Mengjiang Luo, Liping Kang, Xiaoming Wei, Xiaolin Wei, and Shuqiang Jiang.  Large scale visual food recognition.  arXiv preprint arXiv:2103.16107, 2021.
  * [35] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu.  Asynchronous methods for deep reinforcement learning.  In ICML, 2016.
  * [36] Peiyuan Ni, Wenguang Zhang, Xiaoxiao Zhu, and Qixin Cao.  Pointnet++ grasping: learning an end-to-end spatial grasp generation algorithm from sparse point clouds.  In ICRA, 2020.
  * [37] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In CVPR, 2017.
  * [38] Charles R Qi, Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, and Leonidas J Guibas.  Volumetric and multi-view cnns for object classification on 3d data.  In CVPR, 2016.
  * [39] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  NeurIPS, 2017.
  * [40] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár.  Designing network design spaces.  In CVPR, 2020.
  * [41] Yongming Rao, Guangyi Chen, Jiwen Lu, and Jie Zhou.  Counterfactual attention learning for fine-grained visual categorization and re-identification.  In CVPR, 2021.
  * [42] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel.  High-dimensional continuous control using generalized advantage estimation.  In ICLR, 2016.
  * [43] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347, 2017.
  * [44] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller.  Multi-view convolutional neural networks for 3d shape recognition.  In ICCV, 2015.
  * [45] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In ICML, 2019.
  * [46] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie.  The inaturalist species classification and detection dataset.  In CVPR, 2018.
  * [47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  NeurIPS, 2017.
  * [48] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie.  The caltech-ucsd birds-200-2011 dataset.  Technical Report, California Institute of Technology, 2011.
  * [49] Chu Wang, Marcello Pelillo, and Kaleem Siddiqi.  Dominant set clustering and pooling for multi-view 3d object recognition.  In BMVC, 2017.
  * [50] Yaming Wang, Vlad I Morariu, and Larry S Davis.  Learning a discriminative filter bank within a cnn for fine-grained recognition.  In CVPR, 2018.
  * [51] Yulin Wang, Kangchen Lv, Rui Huang, Shiji Song, Le Yang, and Gao Huang.  Glance and focus: a dynamic approach to reducing spatial redundancy in image classification.  In NeurIPS, 2020.
  * [52] Xiu-Shen Wei, Yi-Zhe Song, Oisin Mac Aodha, Jianxin Wu, Yuxin Peng, Jinhui Tang, Jian Yang, and Serge Belongie.  Fine-grained image analysis with deep learning: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021\.
  * [53] Ronald J Williams.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  Machine Learning, 1992.
  * [54] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao.  3d shapenets: A deep representation for volumetric shapes.  In CVPR, 2015.
  * [55] Tianjun Xiao, Yichong Xu, Kuiyuan Yang, Jiaxing Zhang, Yuxin Peng, and Zheng Zhang.  The application of two-level attention models in deep convolutional neural network for fine-grained image classification.  In CVPR, 2015.
  * [56] Linjie Yang, Ping Luo, Chen Change Loy, and Xiaoou Tang.  A large-scale car dataset for fine-grained categorization and verification.  In CVPR, 2015.
  * [57] Ze Yang and Liwei Wang.  Learning relationships for multi-view 3d object recognition.  In ICCV, 2019.
  * [58] Chaojian Yu, Xinyi Zhao, Qi Zheng, Peng Zhang, and Xinge You.  Hierarchical bilinear pooling for fine-grained visual recognition.  In ECCV, 2018.
  * [59] Tan Yu, Jingjing Meng, and Junsong Yuan.  Multi-view harmonized bilinear network for 3d object recognition.  In CVPR, 2018.
  * [60] Ning Zhang, Jeff Donahue, Ross Girshick, and Trevor Darrell.  Part-based r-cnns for fine-grained category detection.  In ECCV. Springer, 2014.
  * [61] Heliang Zheng, Jianlong Fu, Zheng-Jun Zha, and Jiebo Luo.  Learning deep bilinear transformation for fine-grained image representation.  In NeurIPS, 2019.

## Appendix

##  Appendix A Implementation Details

###  A.1 Implementation of the Policy Gradient Algorithm

For training the actor $\mathcal{A}(\cdot)$ (the next-view selection module)
at Stgae II, we adopt the proximal policy optimization (PPO) algorithm
schulman2017proximal with a slight modification. Specifically, given a series
of inputs ${x_{i}^{v_{1}},\dots,x_{i}^{v_{t}}}$ at the $t$-th step, the
extractor $\mathcal{F}(\cdot)$ and the aggregator $\mathcal{R}_{s}(\cdot)$ are
first applied to form the current state:

$s_{i}^{t}=\mathcal{R}_{s}(\mathcal{F}(x_{i}^{v_{1}}),\dots,\mathcal{F}(x_{i}^{v_{t}})).$
(4)

And then, the actor take the state $s_{i}^{t}$ as input and decide the next
view proposal $v_{t+1}$ as the action (_i.e._ ,
$v_{t+1}=\mathcal{A}(s_{i}^{t})$). For the general PPO algorithm with the
reward $r_{i}^{t}$ for $t$-th step, the advantage estimator $\hat{A}_{i}^{t}$
can be expressed as:

$\hat{A}_{t}=-V(s_{i}^{t})+r_{i}^{t}+\gamma
r_{i}^{t+1}+\dots+\gamma^{T-t}r_{i}^{T},$ (5)

where $V(s_{i}^{t})$ is the learned state-value function, $\gamma\in(0,1)$ is
a pre-defined discount factor, $T$ is the maximum length of the input
sequence. The principle behind it is straightforward – the current action
should not only benefit the next step but also contribute to the overall goal.
However, in this work, aiming at achieving reliable prediction with the least
number of steps, we only focus on the profit at the very next step, _i.e._ ,
we set $\gamma=0$. The advantage estimator we use can be formulated by:

$\hat{A}_{i}^{t}=-V(s_{i}^{t})+r_{i}^{t}.$ (6)

After that, we denote the prediction probability of $v_{t}$ by
$\mathcal{A}(v_{t}|s_{i}^{t})$. Then the clipped surrogate objective is:

$L_{CLIP}=\frac{1}{B}\sum_{i=1}^{B}\sum_{t=2}^{T}min\left\\{\frac{\mathcal{A}(v_{t}|s_{i}^{t})}{\mathcal{A}_{old}(v_{t}|s_{i}^{t})}\hat{A}_{i}^{t},clip(\frac{\mathcal{A}(v_{t}|s_{i}^{t})}{\mathcal{A}_{old}(v_{t}|s_{i}^{t})},1-\epsilon,1+\epsilon)\hat{A}_{i}^{t}\right\\},$
(7)

where $\mathcal{A}_{old}(\cdot)$ stands for the actor before update, and
$\epsilon\in(0,1)$ is a hyper-parameter. Note that $t$ starts from $t=2$ since
the first view is randomly selected. Finally, the overall objective of Stage
II can be expressed as:

$L_{Stage2}=L_{CLIP}-c_{1}L_{VF}+c_{2}L_{E},$ (8)

where
$L_{VF}=\frac{1}{B}\sum_{i=1}^{B}\sum_{t=2}^{T}(V(s_{i}^{t})-V^{target}(s_{i}^{t}))^{2}$
is the squared-error loss suggested by schulman2016high , and
$L_{E}=\frac{1}{B}\sum_{i=1}^{B}\sum_{t=2}^{T}S_{\mathcal{A}}(s_{i}^{t})$ is
the entropy bonus following williams1992simple ; mnih2016asynchronous .
$c_{1}$ and $c_{2}$ is hyper-parameters to balance the three loss components.

###  A.2 Training and Inference Details

Stage I. Similar to the training of most FGVC models, the backbones (ResNet
he2016deep , DenseNet huang2017densely , EfficientNet tan2019efficientnet ,
RegNet-Y radosavovic2020designing , and ViT dosovitskiy2020image ) are all
first initialized with ImageNet pre-trained weights. We use SGD optimizor with
a momentum of $0.9$ and the cosine learning rate schedule loshchilov2016sgdr
for optimization. The start learning rate is set to be $0.005$ for the
backbone and $0.05$ for the other components. The input images are random-
resize-cropped to $224\times 224$. The model is trained for $60$ epochs. The
temperature $h$ for entropy maximization is set to be $2$.

Stage II. We use the Adam optimizor with $\beta_{1}=0.9$, $\beta_{2}=0.999$,
and the cosine learning rate schedule loshchilov2016sgdr for optimization. The
start learning rate is set to be $0.00005$ for the backbone and $0.0005$ for
the other components. The input images are random-resize-cropped to $224\times
224$. The model is trained for $15$ epochs. The hyper-parameters $\epsilon$,
$c_{1}$, and $c_{2}$ are set to be $0.2$, $0.5$, and $0.01$, respectively.

Stage III. Similar to $StageI$, the SGD optimizer with a momentum of $0.9$ and
the cosine learning rate schedule loshchilov2016sgdr is adopted for
optimization. The start learning rate is set to be $0.005$ for both the
backbone and other components. The input images are random-resize-cropped to
$224\times 224$. The model is trained for $60$ epochs.

Inference. The input images are first resized to a fixed size $256\times 256$
and then center-cropped to $224\times 224$.

##  Appendix B Additional Experimental Results

###  B.1 Aggregator Architecture

Here we conduct ablation studies to select the best aggregator architecture.
There are four options being evaluated: multiple fully connected layers, LSTM
hochreiter1997long , GRU chung2014empirical , and self-attention
vaswani2017attention . Specifically, we train $T$ fully connected layers for
each step with different channel numbers for the multiple fully connected
layer scheme. The feature sequence
$\\{x_{i}^{1},\dots,x_{i}^{T},\dots,x_{i}^{T}\\}$ is concatenated and
processed by the corresponding fully connected layer. As for the self-
attention architecture, we adopt $4$ multi-head attention layers with $8$
attention heads. We experiment with only Stage I which is enough to reveal the
option with the best feature aggregation ability. The experimental results in
Table 3 suggest that GRU can deliver the best performance.

Table 3: Ablation studies about different aggregator architectures. The best
results are marked in bold.

Architecture mAcc. ($\%$) Multiple FC Layer $81.97\pm 0.8$ LSTM $83.76\pm 0.5$
GRU $\mathbf{84.18\pm 1.1}$ Self-Attention $82.82\pm 1.8$

###  B.2 Learning Rate

Here we carry out ablation studies about learning rates at each training
stage. The experiments are conducted in a stage-by-stage manner, _i.e._ , the
optimal learning rate is selected for each stage according to the model
performance at the current stage, and once we finish the current stage, we
will move to the next stage with the best model at the current stage as
initialization. The experimental results are reported in Table 4, 5, and 6 for
three stages respectively. Note that we only use mAcc for evaluation in Stage
I since there is no active view selection yet. Finally, the optimal learning
rates for the three stages are $0.05$, $0.0005$, and $0.005$, respectively.

Table 4: Ablation studies about the learning rate of Stage I. The best results
are marked in bold.

Learning Rate mAcc. ($\%$) $0.1$ $19.45\pm 8.7$ $0.05$ $\mathbf{84.18\pm 1.1}$
$0.02$ $80.81\pm 0.5$ $0.01$ $68.16\pm 0.3$ $0.005$ $37.54\pm 0.6$

Table 5: Ablation studies about the learning rate of Stage II. The best
results are marked in bold.

Learning Rate mAcc. ($\%$) w-mAcc. ($\%$) Step$2$-Acc. ($\%$) $0.001$
$84.71\pm 1.1$ $86.10\pm 1.0$ $85.19\pm 0.9$ $0.0005$ $\mathbf{85.31\pm 1.6}$
$\mathbf{86.57\pm 1.6}$ $\mathbf{85.63\pm 1.7}$ $0.0002$ $84.66\pm 1.0$
$85.79\pm 1.0$ $84.69\pm 1.0$ $0.0001$ $84.49\pm 1.2$ $85.29\pm 1.5$ $84.16\pm
1.5$ $0.00005$ $84.53\pm 1.2$ $85.22\pm 1.6$ $83.85\pm 1.6$

Table 6: Ablation studies about the learning rate of Stage III. The best
results are marked in bold.

Learning Rate mAcc. ($\%$) w-mAcc. ($\%$) Step$2$-Acc. ($\%$) $0.01$ $85.49\pm
0.5$ $87.12\pm 0.7$ $86.24\pm 0.8$ $0.005$ $\mathbf{86.07\pm 0.5}$
$\mathbf{87.66\pm 0.5}$ $\mathbf{87.20\pm 0.4}$ $0.002$ $85.51\pm 0.7$
$86.69\pm 0.9$ $85.70\pm 0.8$ $0.001$ $85.34\pm 0.8$ $86.54\pm 0.9$ $85.47\pm
0.6$ $0.0005$ $85.20\pm 0.7$ $86.48\pm 0.7$ $85.53\pm 0.4$

###  B.3 Temperature for Entropy Maximization

Here we discuss the effect of the temperature $h$ for entropy maximization.
Instead of directly maximizing the entropy of model prediction, we apply a
temperature $h>1$ to smooth the prediction distribution as the optimization
target. In this way, we are able to explicitly control the degree of entropy
maximization constraint. Note that $h=1$ is equivalent to the entropy
maximization being disabled. In addition to applying a consistent $h$, we also
experiment with a series of exponentially decreased $h$ starting from $5$ –
$[5,3,2,1.5,1.25,1.125,1.0625]$, which follows our intuitive conjecture that
the model should yield more confident predictions with more visual inputs.
Finally, according to Table 7, we choose $h=2$ since it leads to two of the
three best results.

Table 7: Ablation studies about different temperature $h$ for the entropy
maximization constraint. The best results are marked in bold.

Temperature $h$ mAcc. ($\%$) w-mAcc. ($\%$) Step$2$-Acc. ($\%$) $1$ $85.91\pm
0.9$ $87.07\pm 0.9$ $86.07\pm 0.9$ $2$ $86.07\pm 0.5$ $\mathbf{87.66\pm 0.5}$
$\mathbf{87.20\pm 0.4}$ $5$ $\mathbf{86.18\pm 0.1}$ $87.30\pm 0.3$ $86.36\pm
0.2$ $10$ $85.54\pm 0.1$ $86.64\pm 0.7$ $85.70\pm 0.8$ $20$ $85.52\pm 0.5$
$86.77\pm 0.7$ $86.05\pm 0.7$ $[5.0,3.0,2.0,1.5,1.25,1.125,1.0625]$ $85.92\pm
0.6$ $87.24\pm 0.6$ $86.54\pm 0.6$

###  B.4 Training Scheme

In this paper, we adopt a multi-stage training scheme for best performance.
However, an end-to-end training strategy is also practicable for the proposed
framework. Therefore, a comparison of these two schemes is carried out.
Specifically, we merge the three training stages into one, _i.e._ , the model
is optimized via $L_{CE}$ for recognition, $L_{EM}$ for smooth prediction, and
$L_{PG}$ for next-view selection together at each iteration. The model is
trained for $60$ epochs. The experimental results are reported in Table 8. The
stage-by-stage scheme outperforms the end-to-end scheme with significant
margins, which indicates the necessity of adopting three training stages
separately for their different objectives.

Table 8: Ablation studies about different training schemes. The best results
are marked in bold.

Training Scheme mAcc. ($\%$) w-mAcc. ($\%$) Step$2$-Acc. ($\%$) End-to-End
$81.41\pm 0.7$ $85.81\pm 0.7$ $85.73\pm 0.9$ Stage-by-Stage $\mathbf{86.07\pm
0.5}$ $\mathbf{87.66\pm 0.5}$ $\mathbf{87.20\pm 0.4}$

##  Appendix C Further Discussion

###  C.1 Limitation

The limitations of this work are mainly two-fold. Firstly, for academic
purposes only, the collected MvCars is relatively small-scale under the
current trend of developing large-scale datasets, making it insufficient to
support mature commercial applications. Here we only try to break the ice,
hoping to arouse the attention of the FGVC community so as to emerge more and
deeper research achievements beyond the $2$D scenario. Secondly, the proposed
method adopts GRU chung2014empirical for feature aggregation, which makes it
order-sensitive – different input orders of the same contents may change the
prediction results. A significant further impact is that the model performance
still lower than the upper bound with a margin of $\sim 4.\%$ at the last step
(_i.e._ , all visual information is acquired). However, an ideal recognition
model based on sequence inputs should be order-invariant rather than
forgetting early inputs. Therefore, developing better feature aggregation
techniques may be a meaningful future direction.

###  C.2 Broader Impact

Fine-grained visual classification has demonstrated its application value in
many fields, _e.g._ , intelligent retail, intelligent transportation,
automatic biodiversity monitoring, and many more wei2021survey . Recently,
with the development of hardware equipment, portal devices and embodied AI
agents tend to be the carrier of computer vision algorithms, which put forward
requirements to the vision algorithms for the dynamic information processing
ability in $3$D environments. However, the advanced FGVC techniques are still
limited to processing $2$D static images despite the great success. In this
work, with a newly collected testbed and a viable approach, we may motivate
other researchers to develop more effective/efficient algorithms or contribute
more challenging datasets to the problem raised. Embracing the coming
approaching trend, we believe this could be a new stage for fine-grained
recognition research and potentially boost other related tasks, _e.g._ ,
active fine-grained retrieval, fine-grained $3$D object generation, _etc._

On the other hand, as the common negative impact for all FGVC tasks, it may be
used for military purposes or facilitate criminal behaviours. Besides, the
proposed method also suffers the risk of potential adversarial attacks due to
the inherent characteristics of deep-neural-network-based models. However, we
believe the consequent benefits outweigh the potential negative effects.