(etoolbox) Package etoolbox Error: “uline not a macroSee the etoolbox package
documentation for explanation.

#  : A Vision Foundation Model for the Tree of Life

Samuel Stevens  Jiaman Wu  Matthew J Thompson  The Ohio State University
Elizabeth G Campolongo  The Ohio State University  Chan Hee Song  The Ohio
State University  David Edward Carlyn  The Ohio State University  Li Dong
Microsoft Research  Wasila M Dahdul  University of California, Irvine  Charles
Stewart  Rensselaer Polytechnic Institute
Tanya Berger-Wolf  The Ohio State University  Wei-Lun Chao  The Ohio State
University  Yu Su

###### Abstract

Images of the natural world, collected by a variety of cameras, from drones to
individual phones, are increasingly abundant sources of biological
information. There is an explosion of computational methods and tools,
particularly computer vision, for extracting biologically relevant information
from images for science and conservation. Yet most of these are bespoke
approaches designed for a specific task and are not easily adaptable or
extendable to new questions, contexts, and datasets. _A vision model for
general organismal biology questions on images is of timely need._ To approach
this, we curate and release TreeOfLife-10M, the largest and most diverse ML-
ready dataset of biology images. We then develop , a foundation model for the
tree of life, leveraging the unique properties of biology captured by
TreeOfLife-10M, namely the abundance and variety of images of plants, animals,
and fungi, together with the availability of rich structured biological
knowledge. We rigorously benchmark our approach on diverse fine-grained
biology classification tasks, and find that consistently and substantially
outperforms existing baselines (by 17% to 20% absolute). Intrinsic evaluation
reveals that has learned a hierarchical representation conforming to the tree
of life, shedding light on its strong generalizability.111See
github.com/Imageomics/bioclip for models, data and code.

11footnotetext: Equal contribution<EMAIL_ADDRESS>
(d) Onoclea sensibilis

(e) Onoclea hintonii

Figure 1:  (a) Two taxa, or taxonomic labels, for two different plants,
Onoclea sensibilis (d) and Onoclea hintonii (e). These taxa are identical
except for the species. (b) The autoregressive text encoder naturally encodes
the hierarchical structure of the taxonomy. See how the Order token(s)
(Polypodiales) can incorporate information from the Kingdom, Phylum and Class
tokens, but nothing later in the hierarchy. This helps align the visual
representations to this same hierarchical structure (see Sec. 4.6). (c) These
hierarchical representations of taxonomic labels are fed into the standard
contrastive pre-training objective and are matched with image representations
(d) and (e).

##  1 Introduction

Digital images and computer vision are quickly becoming pervasively used tools
to study the natural world, from evolutionary biology [13, 48] to ecology and
biodiversity [79, 5, 74]. The capability to rapidly convert vast quantities of
images from museums [61], camera traps [6, 7, 74, 56, 1], and citizen science
platforms [37, 57, 76, 2, 51, 75, 77, 59, 72, 83, 85, 55] into actionable
information (e.g., species classification, individual identification, and
trait detection) has accelerated and enabled new advances in tasks such as
species delineation [30], understanding mechanisms of adaptation [36, 21],
abundance and population structure estimation [37, 78, 55, 3], and
biodiversity monitoring and conservation [79].

However, applying computer vision to answer a biological question is still a
laborious task requiring substantial machine learning expertise and
effort—biologists must manually label sufficient data for the specific taxa
and task of interest, and find and train a suitable model for the task.
Meanwhile, foundation models [12] such as CLIP [66] and GPT-3 [14] are
extraordinarily valuable by enabling zero-shot or few-shot learning for a wide
range of tasks. An analogous vision foundation model for biology should be
useful for tasks spanning the entire tree of life [34, 50] instead of just the
taxa it has been trained on. That would significantly lower the barrier to
apply AI to biology.

In this work, we aim to develop such a vision foundation model for the tree of
life. To be broadly useful for real-world biology tasks, this model should
meet the following criteria. First, it should generalize to the entire tree of
life, where possible, to ensure it supports researchers studying many
different clades rather than a niche. Furthermore, it is infeasible to collect
training data that covers the millions of known taxa [35, 41], so the model
must generalize to taxa not present in training data. Second, it should learn
fine-grained representations of images of organisms as biology frequently
engages with organisms that are visually similar, like closely related species
within the same genus [64] or species mimicking others’ appearances for a
fitness advantage [36]. This fine-grained granularity is crucial because the
tree of life organizes living things into both broad categories (animal,
fungus, and plant) and very fine-grained ones (see Fig. 1). Finally, due to
the high cost of data collection and labeling in biology, strong performance
in the low-data regime (i.e., zero-shot or few-shot) is critical.

While the goals of generalization, fine-grained classification, and data
efficiency are not new in computer vision, existing general-domain vision
models [66, 92, 58] trained on hundreds of millions of images fall short when
applied to evolutionary biology and ecology. Specifically, existing vision
models produce general fine-grained representations, useful for comparing
common organisms like dogs and wolves, but not for more fine-grained
comparisons, e.g., Onoclea sensibilis L. and Onoclea hintonii (see Fig. 1).

We identify two major barriers to developing a vision foundation model for
biology. First, there is a need for suitable pre-training datasets: existing
datasets [86, 84, 82, 26] lack either scale, diversity, or fine-grained
labels. Second, there is a need to investigate suitable pre-training
strategies that leverage special properties of the biology domain to better
achieve the three pivotal goals, e.g., the tree of life taxonomy, which is
insufficiently considered in mainstream pre-training algorithms [45, 58, 66].

In light of these goals and challenges in achieving them, we introduce 1)
TreeOfLife-10M, a large-scale ML-ready biology image dataset, and 2) , a
vision foundation model for the tree of life, trained with suitable use of
taxa in TreeOfLife-10M. We outline the contributions, conceptual framework,
and design decisions below:

1\. TreeOfLife-10M: a large-scale, diverse ML-ready biology image dataset. We
curate and release the largest-to-date ML-ready dataset of biology images with
associated taxonomic labels, containing over $10$ million images covering
$454$ thousand taxa in the tree of life.222 By ML-ready, we mean the data is
standardized in a format suitable for training ML models and is readily
available for downloading.  In comparison, the current largest ML-ready
biology image dataset, iNat21 [82], contains only $2.7$ million images
covering $10$ thousand taxa. TreeOfLife-10M integrates existing high-quality
datasets like iNat21 and Bioscan-1M [26]. More importantly, it includes newly
curated images from the Encyclopedia of Life (eol.org), which supplies most of
TreeOfLife-10M’s data diversity. Every image in TreeOfLife-10M is labeled with
its taxonomic hierarchy to the finest level possible, as well as higher
taxonomic ranks in the tree of life (see Figs. 1 and 3 for examples of
taxonomic ranks and labels). TreeOfLife-10M enables training and future
biology foundation models.

2\. : a vision foundation model for the tree of life. With a large-scale
labeled dataset like TreeOfLife-10M, a standard, intuitive training strategy
(as adopted by other vision models like ResNet50 [31] and Swin Transformer
[45]) is to use a supervised classification objective and learn to predict the
taxonomic indices from an image. However, this fails to recognize and leverage
the rich structure of taxonomic labels—taxa do not exist in isolation but are
interconnected in a comprehensive taxonomy. Consequently, a model trained via
plain supervised classification may not generalize well to taxa unseen in
training, nor could it support zero-shot classification of unseen taxa.

Instead, we propose a novel strategy combining CLIP-style multimodal
contrastive learning [66] with the rich biological taxonomy for . We “flatten”
the taxonomy from Kingdom to the distal-most taxon rank into a string called
taxonomic name, and use the CLIP contrastive learning objective to learn to
match images with their corresponding taxonomic names. Intuitively, this helps
the model generalize to unseen taxa—even if the model has not seen a species,
it has likely learned a reasonable representation for that species’ genus or
family (see Fig. 1). also supports zero-shot classification with taxonomic
names of unseen taxa. We further propose, and demonstrate the effectiveness
of, a mixed text type training strategy; by mixing different text types (e.g.,
taxonomic vs. scientific vs. common names) during training, we retain the
generalization from taxonomic names while being more flexibility at test time.
For example, still excels even if only common species names are offered by
downstream users.

3\. Comprehensive benchmarking. We comprehensively evaluate on $10$ fine-
grained image classification datasets covering animals, plants, and fungi,
including a newly curated Rare Species dataset unseen in training. achieves
strong performance in both zero-shot and few-shot settings and substantially
outperforms both CLIP [66] and OpenCLIP [39], leading to an average absolute
improvement of 18% (zero-shot) and 17% (few-shot). Intrinsic analysis further
reveals that has learned a more fine-grained hierarchical representation
conforming to the tree of life, explaining its superior generalization.

Dataset |  Description | Images | Unique Classes
---|---|---|---
iNat21 |  Citizen scientist labeled image dataset from iNaturalist for fine-grained classification. | 2.7M | $10,000$
Bioscan-1M |  Expert labeled image dataset of insects for classification. | 1.1M | $7,831$
EOL |  A new dataset with citizen scientist images sourced from Encyclopedia of Life and taxonomic labels standardized by us. | 6.6M | $448,910$
TreeOfLife-10M |  Largest-to-date ML-ready dataset of biology images with taxonomic labels. | 10.4M | $454,103$

Table 1:  Training data sources used in TreeOfLife-10M. We integrate and
canonicalize taxonomic labels across the sources (Sec. 2.2).

##  2 TreeOfLife-10M

Recent work has shown that data quality and diversity is critical when
training CLIP models [22, 54, 24]. We curate TreeOfLife-10M, the most diverse
large-scale public ML-ready dataset for computer vision models in biology.

###  2.1 Images

The largest ML-ready biology image dataset is iNat21 [82], which contains
$2.7$M images of $10$K species. Despite this class breadth compared to popular
general-domain datasets like ImageNet-1K [67], $10$K species are rather
limited in the context of biology. The International Union for Conservation of
Nature (IUCN) reported over $2$M total described species in 2022, with over
$10$K bird species and over $10$K reptile species alone [41]. iNat21’s species
diversity limits its potential for pre-training a foundation model for the
entire tree of life.

Motivated to find high-quality biology images with a focus on species
diversity, we turn to the Encyclopedia of Life project (EOL; eol.org). EOL
collaborates with a variety of institutions to gather and label millions of
images. We download $6.6$M images from EOL and expand our dataset to cover an
additional $\mathbf{440}$K taxa.

Species are not evenly distributed among the different subtrees in the tree of
life; insects (of the class Insecta with $1$M+ species), birds (of the class
Aves with $10$K+ species) and reptiles (of the class Reptilia with $10$K+
species) are examples of highly diverse subtrees with many more species. To
help a foundation model learn extremely fine-grained visual representations
for insects, we also incorporate Bioscan-1M [26], a recent dataset of $1$M lab
images of insects, covering $494$ different families.333 We note that
Bioscan-1M’s label granularity may still be limited for insects. $98.6$% of
Bioscan-1M’s images are labeled to the family level but only $22.5$% and
$7.5$% of the images have genus or species indicated, respectively. Lack of
label granularity is an inherent challenge.  Furthermore, Bioscan-1M contains
lab images, rather than in situ images like iNat21, diversifying the image
distribution.

Figure 2:  Treemap of the 108 phyla in TreeOfLife-10M. Different colors are
different phyla; nested boxes represent classes, orders, and families. Box
size, not number of inner boxes, represents relative number of samples.

###  2.2 Metadata & Aggregation

The TreeOfLife-10M dataset integrates iNat21 (training split), our curated EOL
dataset, and Bioscan-1M by aggregating the images and canonicalizing the
labels. _This is a highly non-trivial task because taxonomic hierarchies are
notoriously noisy and rarely consistent between sources_ [33, 29, 49, 60, 4],
likely contributing to the prior lack of image datasets large enough to train
a foundation-scale vision model for the entire tree of life. We carefully
unify and backfill taxonomic hierarchies from EOL, the Integrated Taxonomic
Information System (ITIS) [40] and iNaturalist with special consideration for
the existence of homonyms (genus-species labels shared among higher-order
taxa). For more information on this process, the challenges, our solutions,
and remaining issues, see Appendix C.

###  2.3 Release & Statistics

Tab. 1 presents dataset statistics: TreeOfLife-10M has over $10$M images
across more than $450$K unique taxonomic names. Fig. 2 shows the distribution
of images by phyla and the respective lower-rank taxa (order through family).

We will release our curated training and test data (TreeOfLife-10M and Rare
Species, described in Sec. 4.2) on Hugging Face (with DOIs) under a public
domain waiver, to the extent primary source licenses allow. This will include
a CSV with image metadata and links to the primary sources, accompanied by a
GitHub repository with the scripts to generate the datasets.444 We encourage
future work to cite iNat21 [82], Bioscan-1M [26] and to appropriately
attribute images from EOL based on their licenses if citing TreeOfLife-10M.

|  Name |  Description | Examples | Classes | Labels
---|---|---|---|---|---
Animals | Birds 525 |  Scraped dataset of bird images from web search. [65] | $89,885$ | $525$ | Taxonomic
Plankton |  Expert-labeled in situ images of plankton [32]. | $4,080$ | $102$ | Mixed
Insects |  Expert and volunteer-labeled in-the-wild citizen science images of insects [71]. | $4,680$ | $117$ | Scientific
Insects 2 |  Mixed common and scientific name classification for insect pests [88]. | $4,080$ | $102$ | Mixed
Plants & Fungi | PlantNet |  Citizen science species-labeled plant images, some drawings [25]. | $1,000$ | $25$ | Scientific
Fungi |  Expert-labeled images of Danish fungi [63]. | $1,000$ | $25$ | Scientific
PlantVillage |  Museum-style leaf specimens labeled with common names [23]. | $1,520$ | $38$ | Common
Medicinal Leaf |  Species classification of leaves from mature, healthy medicinal plants [68]. | $1,040$ | $26$ | Scientific
PlantDoc |  17 diseases for 13 plant species [73]. | $1,080$ | $27$ | Common
| Rare Species |  Subset of species in the IUCN Red List categories: Near Threatened through Extinct in the Wild (iucnredlist.org). | $12,000$ | $400$ | Taxonomic

Table 2:  Datasets used for evaluation. All tasks are classification evaluated
with Top-1 accuracy.

##  3 Modeling

is initialized from OpenAI’s public CLIP checkpoint and continually pre-
trained on TreeOfLife-10M with CLIP’s multimodal contrastive learning
objective.

###  3.1 Why CLIP?

Compared with general domain computer vision tasks, one of the most salient
differences for the biology domain is its rich label space. Not only are the
taxon labels large in quantity (there are $2$M+ recorded species as of 2022
[41]), but they are also connected with each other in a hierarchical taxonomy.
This is a challenge for training a good foundation model that can achieve
satisfactory coverage and generalization. Despite this, the intricate
structure in the label space, accumulated through centuries of biology
research, provides very rich signal for learning better generalization.
Intuitively, if the label space’s structure is successfully encoded in a
foundation model, even if the model has not seen a certain species, it will
likely have learned a good representation for that species’ corresponding
genus or family. Such a hierarchical representation serves as a strong prior
to enable few-shot or even zero-shot learning of new taxa.

Many vision foundation models, such as ResNet [31] and Swin Transformer [45],
adopt a supervised classification objective and directly learn the mapping
from input images to class indices. As a result, each class label is treated
as a distinct symbol, and their relationships are neglected. _A key
realization of our work is that the multimodal contrastive learning objective
used in CLIP can be repurposed for leveraging the hierarchical structure of
the label space._ This is not an obvious choice; after all, TreeOfLife-10M is
largely labeled with class labels and not with free-form text like image
captions. The autoregressive text encoder naturally embeds the taxonomic
hierarchy into a dense label space by conditioning later taxonomic rank
representations on higher ranks (Fig. 1). While hierarchical classification
[9, 93, 11] can also leverage taxonomy, we empirically show that CLIP-style
contrastive learning significantly improves generalization (Sec. 4.4). We note
that repurposing CLIP’s multimodal contrastive learning objective for learning
hierarchical representations conforming to a taxonomy is a novel and non-
trivial technical contribution.

CLIP trains two uni-modal embedding models, a vision encoder and a text
encoder, to (1) maximize feature similarity between positive (image, text)
pairs and (2) minimize feature similarity between negative (image, text)
pairs, where positive pairs from the training data and negative pairs are all
other possible (image, text) pairings in a batch. After training, CLIP’s
encoder models embed individual instances of their respective modalities into
a shared feature space. Next, we discuss formatting the text input to CLIP to
incorporate the taxonomic structure.

###  3.2 Text Types

Text Type |  Example
---|---
Common |  black-billed magpie
Scientific |  Pica hudsonia
Taxonomic |  Animalia Chordata Aves Passeriformes Corvidae Pica hudsonia
Scientific + Common |  Pica hudsonia with common name black-billed magpie
Taxonomic + Common |  Animalia Chordata Aves Passeriformes Corvidae Pica hudsonia with common name black-billed magpie
Table 3:  Text types considered in the training of .

An advantage of CLIP is that the text encoder accepts free-form text. In
biology, unlike other classification tasks, class names are diversely
formatted. We consider the following:

Taxonomic name. A standard seven-level biology taxonomy from higher to lower
level is kingdom, phylum, class, order, family, genus and species. For each
species, we “flatten” the taxonomy by concatenating all labels from root to
leaf into a single string, which we call the taxonomic name.

Scientific name. Scientific names are composed of genus and species (e.g.,
Pica hudsonia).

Common name. Taxonomy categories are usually Latin, which is not often seen in
generalist image-text pre-training datasets. Instead, the common name, such as
“black-billed magpie,” is more widespread. Note that common names may not have
a 1-to-1 mapping to taxa: A single species may have multiple common names, or
the same common name may refer to multiple species.

For certain downstream use cases of , it might be the case that only one type
of label, e.g., scientific names, is available. To improve the flexibility at
inference time, we propose a mixed text type training strategy: at each
training step, we pair each input image with a text randomly sampled from all
of its available text types (shown in Tab. 3). We empirically show that this
simple strategy retains the generalization benefits of taxonomic names while
providing more flexibility in using other names at inference time (Sec. 4.3).
The final text input to CLIP is the name in the standard CLIP template, e.g.,
“a photo of Pica hudsonia”.

##  4 Experiments

We train on TreeOfLife-10M, compare to general vision models and investigate
how our modeling choices affect ’s performance.

###  4.1 Training and Evaluation Details

To train , we initialize from OpenAI’s CLIP weights [66] with a ViT-B/16
vision transformer [20] image encoder and a $77$-token causal autoregressive
transformer text encoder. We continue pre-training on TreeOfLife-10M for 100
epochs with a cosine learning rate schedule [46]. We train on $8$ NVIDIA
A100-80GB GPUs over $2$ nodes with a global batch size of $32,768$. We also
train a baseline model on only the iNat21 dataset and multiple ablation models
on 1M examples randomly sampled from TreeOfLife-10M (Secs. 4.3 and 4.4),
following the same procedure for except with a smaller global batch size of
$16,384$ on $4$ NVIDIA A100 GPUs on $1$ node. All hyperparameters and training
details are in Appendix D; the training and evaluation code will be released.

We evaluate on 10 different classification tasks: the 8 biologically-relevant
tasks from Meta-Album [80], Birds 525 [65] and our new Rare Species task
(described in Sec. 4.2). Meta-Album is a dataset collection for meta-learning,
encompassing various subjects. Specifically, we use the Plankton, Insects,
Insects 2, PlantNet, Fungi, PlantVillage, Medicinal Leaf, and PlantDoc
datasets. Our classification tasks cover all four multi-celled kingdoms in the
tree of life (animals, plants, fungi, and protists) and have a diverse image
distribution (photographs, microscope images, drawings, and museum specimens).
Tab. 2 provides an overview of the datasets; they comprise a variety of label
types from full taxonomic names to only scientific or common name, or a mix of
the latter.

For zero-shot learning, we follow the same procedure as CLIP. For few-shot
learning, we follow SimpleShot [87] to use a nearest-centroid classifier. For
$k$-shot learning, we first randomly sample $k$ examples for each class and
obtain the image embedding from the visual encoder of the pre-trained models.
We then compute the average feature vector of the $k$ embeddings as the
centroid for each class. All the examples left in the dataset are used for
testing. After applying mean subtraction and L2-normalization to each centroid
and test feature vector, we choose the class with the nearest centroid to the
test vector as the prediction. We repeat each few-shot experiment $5$ times
with different random seeds and report the mean accuracy in Tab. 4. Results
with standard deviations are reported in Appendix E.

We compare with the original OpenAI CLIP [66] as well as OpenCLIP [39] trained
on LAION-400M [70]. Intuitively, common names of organisms are most pervasive
in the training data of CLIP and OpenCLIP and these models work best with
common names. This is also confirmed in our preliminary tests. Therefore, we
use common names as class labels for CLIP and OpenCLIP by default unless
unavailable for a dataset. can leverage taxonomic names, so we use taxonomic +
common names by default. However, as noted in Tab. 2, the test datasets come
in a variety of labels. Whenever the preferred label type is not available, we
use labels that come with the dataset.

|  Animals |  Plants & Fungi |  |  |
---|---|---|---|---|---
Model |  Birds 525 |  Plankton |  Insects |  Insects 2 |  PlantNet |  Fungi |  PlantVillage |  Med. Leaf |  PlantDoc |  Rare Species |  Mean ($\Delta$)
Random Guessing | $0.2$ | $1.2$ | $1.0$ | $1.0$ | $4.0$ | $4.0$ | $2.6$ | $4.0$ | $3.7$ | $0.3$ | $2.2$ |
Zero-Shot Classification
CLIP | $49.9$ | $3.2$ | $9.1$ | $9.8$ | $58.5$ | $10.2$ | $5.4$ | $15.9$ | $26.1$ | $26.6$ | $21.4$ | –
OpenCLIP | $54.7$ | $2.2$ | $6.5$ | $9.6$ | $50.2$ | $5.7$ | $8.0$ | $12.4$ | $25.8$ | $31.0$ | $20.6$ | $-0.8$
| $72.1$ | $6.1$ | $34.8$ | $20.4$ | $91.4$ | $40.7$ | $24.4$ | $38.6$ | $28.4$ | $37.8$ | $39.4$ | $+18.0$
– iNat21 Only | $56.1$ | $2.6$ | $30.7$ | $11.5$ | $88.2$ | $43.0$ | $18.4$ | $25.6$ | $20.5$ | $19.4$ | $31.6$ | $+10.2$
One-Shot Classification
CLIP | $43.7$ | $25.1$ | $21.6$ | $13.7$ | $42.1$ | $17.2$ | $49.7$ | $70.1$ | $24.8$ | $28.4$ | $33.6$ | –
OpenCLIP | $53.7$ | $32.3$ | $23.2$ | $14.3$ | $45.1$ | $18.4$ | $53.6$ | $71.2$ | $26.8$ | $29.3$ | $36.7$ | $+3.1$
| $71.8$ | $30.6$ | $57.4$ | $20.4$ | $64.5$ | $40.3$ | $58.8$ | $84.3$ | $30.7$ | $45.3$ | $50.4$ | $+16.8$
– iNat21 Only | $74.8$ | $29.6$ | $53.9$ | $19.7$ | $67.4$ | $35.5$ | $55.2$ | $75.1$ | $27.8$ | $37.6$ | $47.6$ | $+14.0$
Five-Shot Classification
CLIP | $73.5$ | $41.2$ | $39.9$ | $24.6$ | $65.2$ | $27.9$ | $71.8$ | $89.7$ | $35.2$ | $46.9$ | $51.5$ | –
OpenCLIP | $81.9$ | $52.5$ | $42.6$ | $25.0$ | $68.0$ | $30.6$ | $77.8$ | $91.3$ | $42.0$ | $48.4$ | $56.0$ | $+4.5$
| $90.0$ | $49.3$ | $77.8$ | $33.6$ | $85.6$ | $62.3$ | $80.9$ | $95.9$ | $47.5$ | $66.4$ | $68.9$ | $+17.4$
– iNat21 Only | $90.1$ | $48.2$ | $73.7$ | $32.1$ | $84.7$ | $55.6$ | $77.2$ | $93.5$ | $41.0$ | $56.0$ | $65.1$ | $+13.6$

Table 4:  Zero-, one- and five-shot classification top-1 accuracy for different CLIP models. Bold indicates best accuracy. All models use the same architecture: ViT-B/16 vision encoders, 77-token text encoder. “iNat21 Only” follows the same procedure as but uses iNat21 instead of TreeOfLife-10M. $\Delta$ denotes the difference in mean accuracy with CLIP.  Dataset |  Train$\downarrow$Test$\rightarrow$ | Com | Sci | Tax | Sci+Com | Tax+Com
---|---|---|---|---|---|---
ToL-1M | Com | $25.5$ | $9.5$ | $10.6$ | $22.6$ | $23.1$
Sci | $11.1$ | $22.2$ | $4.4$ | $20.9$ | $7.7$
Tax | $13.5$ | $10.5$ | $26.5$ | $15.9$ | $24.1$
Sci+Com | $25.2$ | $12.8$ | $12.5$ | $27.2$ | $25.3$
Tax+Com | $22.1$ | $8.0$ | $19.5$ | $25.1$ | $30.0$
Mixture | $26.8$ | $24.7$ | $26.4$ | $30.0$ | $30.5$
iNat21-2.7M | Mixture | $20.9$ | $14.7$ | $15.6$ | $21.7$ | $21.8$
ToL-10M | Mixture | $31.1$ | $30.3$ | $33.4$ | $36.6$ | $37.8$
Table 5:  Zero-shot accuracy on species not seen during training (Rare Species
task). Different rows and columns indicate different text types during
training and test time, respectively. Blue indicates best accuracy and Orange
indicates second-best. Using the taxonomic name over the scientific name
always improves accuracy ($22.2$$\rightarrow$$26.5$ and
$27.2$$\rightarrow$$30.0$). The final rows use the full iNat21 dataset and
TreeOfLife-10M for reference.

###  4.2 Can Generalize to Unseen Taxa?

Taxonomic names are added, removed, and changed as biologists discover and
classify new and existing species. should generalize to unseen taxonomic names
so that it doesn’t need to be re-trained for every new species. To empirically
answer whether generalizes well to unseen taxa, we introduce a new evaluation
task that is both biologically and empirically motivated: Rare Species.

Classifying “rare” species is an important and challenging computer vision
application in biology, particularly in the context of conservation efforts
around the world [79]. To the best of our knowledge, there is no diverse,
publicly available rare species classification dataset. Recently published
examples [53, 44] lack species diversity with only a dozen classes, compared
to our 400. We aim to fill this gap and collect all the species on the IUCN
Red List (iucnredlist.org) classified555 IUCN has classified $150,388$ species
and generally updates their list twice per year (IUCN Update Schedule). The
classifications used for this dataset are current as of July 13, 2023.  as
Near Threatened, Vulnerable, Endangered, Critically Endangered, or Extinct in
the Wild. There are approximately $25,000$ species in these categories, and we
select $400$ species from this list where each is represented by at least $30$
images in our EOL dataset. These species are then completely removed from
TreeOfLife-10M, creating an unseen Rare Species test set with $30$ images per
species. This dataset demonstrates (1) ’s out-of-distribution generalization
to unseen taxa, (2) ’s potential applications, and (3) a crucial large Rare
Species dataset for the community to address the ongoing biodiversity crisis.

Results. Tab. 4 shows that substantially outperforms both baseline CLIP
models, as well as the iNat21-trained CLIP model, at zero-shot classification,
especially on unseen taxa (see the column labeled “Rare Species”).

We attribute ’s strong zero-shot performance on this broad and diverse set of
tasks to the broad and diverse classes present in TreeOfLife-10M. We further
explore how data diversity leads to broadly useful image representations in
Sec. 4.4. With multimodal contrastive learning (Sec. 3.1) and the
incorporation of taxonomic structure (Sec. 3.2), capitalizes on this data
diversity for strong zero-shot generalization.

Objective | Mean 1-Shot | Mean 5-shot
---|---|---
Cross-entropy | $16.7$ | $26.3$
Hier. cross-entropy | $19.3$ | $30.7$
CLIP | $45.1$ | $64.2$

Table 6:  One- and five-shot classification top-1 accuracy for different pre-
training objectives on TreeOfLife-1M. Results are macro-averaged over all the
test sets in Tab. 4.

###  4.3 How Do Text Types Affect Generalization?

We investigate how different text types affect zero-shot generalization by
training on a 10% subset of TreeOfLife-10M (10% due to computational
constraints). We use our Rare Species dataset because the test classes have
every text type, and all species are excluded from training, making it ideal
for testing generalization to unseen taxa. Nguyen et al. [54] find that the
diversity of captions makes stronger vision models and [69] randomly use one
of five different captions for each image during training rather than a single
fixed caption. Similarly, we use a mixed text type strategy (Sec. 3.2). How
does that affect performance?

Results. The zero-shot ablation results are in Tab. 5; there are several
salient observations. First, using taxonomic + common names yields the
strongest performance, showing the importance of incorporating the taxonomic
structure for generalization. Second, when only using a single text type for
training, performance degrades substantially when a different text type is
used at test time. Using mixed text types for training, while not the
strongest, yields consistently strong performance across the board. These
results indicate that mixed text type pre-training largely retains the
generalization benefits of using taxonomic names while also providing
flexibility of different text types for inference, an important property for a
foundation model that may be used for diverse downstream tasks. Finally, using
1M examples from TreeOfLife-10M outperforms using 2.7M examples from iNat21,
further confirming the importance of the added data diversity from
TreeOfLife-10M.

Figure 3:  T-SNE visualization of image features, colored by taxonomic labels.
(B) is visualized in the first and third row and OpenAI’s CLIP (O) is
visualized in the second and fourth rows. ’s features better preserve the
hierarchical structure: while both and CLIP cleanly separate the phyla in the
Animalia Kingdom (top left), only successfully separates the orders in the
Insecta Class (top right) and the families in the Lepidoptera Order (bottom
left).

###  4.4 Is the CLIP Objective Necessary?

Using the CLIP objective to pre-train on a labeled image dataset is an
unintuitive decision (Goyal et al. [27] fine-tune using the CLIP objective,
but do not pretrain). We justify our choice by training two ViT-B/16 models on
TreeOfLife-1M using a cross-entropy classification loss and a multitask
hierarchical variant, then compare them against the CLIP objective under the
few-shot setting. The multitask hierarchical training objective is to predict
the labels for kingdom, phylum, etc., down to species, using cross entropy for
each level of the taxonomy, then summing those losses [11]. Pseudo-code is
provided in Fig. D1.

Results. We evaluate each model on the same set of $10$ tasks but only in the
one-shot and five-shot settings because non-CLIP models cannot do zero-shot
classification. We report mean accuracy in Tab. 6. The hierarchical
classification model outperforms simple classification and is comparable to
the CLIP baseline (see Tab. 4). However, the CLIP objective massively
outperforms both baselines and strongly justifies our repurposing of the CLIP
objective.

###  4.5 Can Classify More Than Species?

is trained on a (contrastive) species-classification objective, which might
limit its downstream use beyond species classification. We consider plant
diagnosis with the PlantVillage and PlantDoc datasets, which require
classifying both species and the disease present (if any).

Large-scale data labeling is expensive in biology, but biologists always label
several instances for use in field guides or museum collections. Few-shot
classification is thus an ideal setting for this sort of task transfer.

Results. We find that outperforms baselines at classifying plant diseases
based on visual symptoms, in both zero-shot and few-shot settings (see the
PlantVillage and PlantDoc columns in Tab. 4).

Notably, while Radford et al. [66] find that CLIP one-shot and two-shot
classification is often worse than zero-shot (because few-shot settings cannot
use the semantic information in the class name), has learned useful visual
representations that are useful even with only one labeled example: ’s mean
one-shot accuracy is 9.1% higher than zero-shot accuracy.

###  4.6 Does Learn the Hierarchy?

demonstrates strong performance in the low-data regime on our extrinsic
evaluation, but why? We further conduct an intrinsic evaluation and directly
visualize the image feature representations has learned to shed light on this
question (Fig. 3). We embed image representations from iNat21’s validation set
(which is not see during training) in two dimensions using t-SNE [81], then
color the points by the image’s taxonomic label. For each plot, we run t-SNE
independently on the subset of examples under the labeled taxonomical rank.
Each plot visualizes one rank of the taxonomic hierarchy and the top six
categories of the following rank, e.g., the top left plot visualizes the six
most common phyla of the Animalia Kingdom. At higher ranks like kingdom
(omitted for space) and phylum, both CLIP and have good separation, but ’s
representation is more fine-grained and contains a richer clustering
structure. At lower ranks, produces evidently more separable features, while
CLIP’s features are cluttered and lack a clear clustering structure. Appendix
F has more qualitative results and visuals.

##  5 Related Work

Multimodal foundation model training data. CLIP [66] trained state-of-the-art
vision models from noisy, web-scale ($100$M+) image-text datasets using a
contrastive objective that is optimized for image retrieval. ALIGN [42] and
BASIC [62] further scaled the number of training examples from 400M to 6.6B,
improving vision representation quality. However, further work [22, 54, 24,
90, 91] all find that _dataset diversity and better alignment between the
image and caption semantics_ are more important than dataset size and lead to
stronger performance on downstream tasks with less compute. _TreeOfLife-10M
emphasizes the importance of diversity_, adding over $440$K classes to
iNat21’s $10$K and leading to stronger zero-shot performance even when
training on fewer images (TreeOfLife-1M).

Domain-specific CLIPs. Domain-specific training often beats general training
[28, 17]. Labeling domain-specific datasets can be prohibitively expensive
because annotators must be subject-matter experts. Image-text training is thus
particularly potent because annotations are optional; models can learn from
noisy image-text pairs. Ikezogwo et al. [38], Lu et al. [47] gathered $1$M+
image-text pairs for use in computational pathology, where expert-labeled
examples are difficult to gather due to both time and privacy. We gather
$10\times$ the images, emphasizing class diversity.

Hierarchy in computer vision. Hierarchy in computer vision is a well-studied
topic, in part because ImageNet [67] concepts are scraped from the
hierarchical WordNet [52]. Bilal et al. [10] develop a visual interface to
study vision model predictions on ImageNet and find that model confusion
patterns follow the same hierarchical structure present in the classes. They
leverage this knowledge to incorporate hierarchy into AlexNet’s architecture
[43] and improve the top-1 error on ImageNet by $8$% absolute. Bertinetto et
al. [9] measure image classifiers’ mistakes’ severity and propose alternative
training objectives that incorporate class hierarchy. This dramatically
reduces mistake severity at the expense of worsening top-1 accuracy. Zhang et
al. [93] propose a contrastive learning objective, HiMulCon, where the
hierarchical distance between labels corresponds to the desired distance in
the embedding space. They use HiMulCon to fine-tune a ResNet that was pre-
trained with cross-entropy on ImageNet and outperform cross-entropy on both
ImageNet and iNat17 [84].

We repurpose the CLIP objective and feed taxonomic names to an autoregressive
text encoder, which forms a dense hierarchical label space into which a vision
encoder learns to embed images. This dense label space is better suited to the
$454$K unique labels, while prior work applied hierarchies to smaller label
spaces.

Computer vision for biology. Fine-grained classification is a classic
challenge in computer vision, and biological images are often used to
benchmark models. Wah et al. [86], Berg et al. [8], Piosenka [65] all use bird
species classification to evaluate fine-grained classification ability. Xiao
et al. [89], Cole et al. [19] use biology tasks in contrastive learning
frameworks, and Cole et al. [18] uses a biology task in weakly supervised
object detection.

##  6 Conclusion

We introduce TreeOfLife-10M and , a large-scale diverse biology image dataset
and a foundation model for the tree of life, respectively. Through extensive
evaluation, we show that is a strong fine-grained classifier for biology in
both zero- and few-shot settings. We corroborate our hypothesis that using the
entire taxonomic name leads to stronger generalization than other caption
types through an ablation on unseen species and by visualizing ’s
representations, finding that -embedded images better match the taxonomic
hierarchy.

Although we leverage the CLIP objective to efficiently learn visual
representations over hundreds of thousands of taxa, is fundamentally still
trained with a classification objective. In future work, we will further scale
up the data, e.g., incorporating research-grade images from inaturalist.org
with $100$M+ images, and collect richer textual descriptions of species’
appearances such that can extract fine-grained trait-level representations.

## Acknowledgements

The authors would like to thank Josef Uyeda, Jim Balhoff, Dan Rubenstein, Hank
Bart, Hilmar Lapp, Sara Beery, and colleagues from the Imageomics Institute
and the OSU NLP group for their valuable feedback. We also thank the
Bioscan-1M team and the iNaturalist team for making their data available and
easy to use, and Jennifer Hammack at EOL for her invaluable help in accessing
EOL’s images.

This research was supported by NSF OAC 2118240 and used computing resources
from the Ohio Supercomputer Center [15].

## References

  * Ahumada et al. [2020] Jorge A Ahumada, Eric Fegraus, Tanya Birch, Nicole Flores, Roland Kays, Timothy G O’Brien, Jonathan Palmer, Stephanie Schuttler, Jennifer Y Zhao, Walter Jetz, Margaret Kinnaird, Sayali Kulkarni, Arnaud Lyet, David Thau, Michelle Duong, Ruth Oliver, and Anthony Dancer.  Wildlife Insights: A Platform to Maximize the Potential of Camera Trap and Other Passive Sensor Wildlife Data for the Planet.  _Environmental Conservation_ , 47(1):1–6, 2020\.  Edition: 2019/09/26 ISBN: 0376-8929 Publisher: Cambridge University Press.
  * Antonelli et al. [2023] Alexandre Antonelli, Kiran L. Dhanjal‐Adams, and Daniele Silvestro.  Integrating machine learning, remote sensing and citizen science to create an early warning system for biodiversity.  _PLANTS, PEOPLE, PLANET_ , 5(3):307–316, 2023\.
  * Araujo et al. [2022] Gonzalo Araujo, Ariana Agustines, Steffen S. Bach, Jesse E. M. Cochran, Emilio De La Parra-Galván, Rafael De La Parra-Venegas, Stella Diamant, Alistair Dove, Steve Fox, Rachel T. Graham, Sofia M. Green, Jonathan R. Green, Royale S. Hardenstine, Alex Hearn, Mahardika R. Himawan, Rhys Hobbs, Jason Holmberg, Ibrahim Shameel, Mohammed Y. Jaidah, Jessica Labaja, Savi Leblond, Christine G. Legaspi, Rossana Maguiño, Kirsty Magson, Stacia D. Marcoux, Travis M. Marcoux, Sarah Anne Marley, Meynard Matalobos, Alejandra Mendoza, Joni A. Miranda, Brad M. Norman, Cameron T. Perry, Simon J. Pierce, Alessandro Ponzo, Clare E. M. Prebble, Dení Ramírez-Macías, Richard Rees, Katie E. Reeve-Arnold, Samantha D. Reynolds, David P. Robinson, Christoph A. Rohner, David Rowat, Sally Snow, Abraham Vázquez-Haikin, and Alex M. Watts.  Improving sightings-derived residency estimation for whale shark aggregations: A novel metric applied to a global data set.  _Frontiers in Marine Science_ , 9:775691, 2022.
  * A. Rees and Cranston [2017] Jonathan A. Rees and Karen Cranston.  Automated assembly of a reference taxonomy for phylogenetic data synthesis.  _Biodiversity Data Journal_ , 5:e12581, 2017.
  * Beery [2021] Sara Beery.  Scaling Biodiversity Monitoring for the Data Age.  _XRDS: Crossroads, The ACM Magazine for Students_ , 27(4):14–18, 2021.
  * Beery et al. [2020] Sara Beery, Elijah Cole, and Arvi Gjoka.  The iwildcam 2020 competition dataset.  _arXiv preprint arXiv:2004.10340_ , 2020.
  * Beery et al. [2021] Sara Beery, Arushi Agarwal, Elijah Cole, and Vighnesh Birodkar.  The iWildCam 2021 Competition Dataset, 2021.  arXiv:2105.03494 [cs].
  * Berg et al. [2014] Thomas Berg, Jiongxin Liu, Seung Woo Lee, Michelle L. Alexander, David W. Jacobs, and Peter N. Belhumeur.  Birdsnap: Large-scale fine-grained visual categorization of birds.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2014.
  * Bertinetto et al. [2020] Luca Bertinetto, Romain Mueller, Konstantinos Tertikas, Sina Samangooei, and Nicholas A Lord.  Making better mistakes: Leveraging class hierarchies with deep networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 12506–12515, 2020.
  * Bilal et al. [2018] Alsallakh Bilal, Amin Jourabloo, Mao Ye, Xiaoming Liu, and Liu Ren.  Do convolutional neural networks learn class hierarchy?  _IEEE Transactions on Visualization and Computer Graphics_ , 24(1):152–162, 2018.
  * Bjerge et al. [2023] Kim Bjerge, Quentin Geissmann, Jamie Alison, Hjalte MR Mann, Toke T Høye, Mads Dyrmann, and Henrik Karstoft.  Hierarchical classification of insects with multitask learning and anomaly detection.  _Ecological Informatics_ , 77:102278, 2023.
  * Bommasani et al. [2021] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ , 2021.
  * Borowiec et al. [2022] Marek L Borowiec, Rebecca B Dikow, Paul B Frandsen, Alexander McKeeken, Gabriele Valentini, and Alexander E White.  Deep learning as a tool for ecology and evolution.  _Methods in Ecology and Evolution_ , 13(8):1640–1660, 2022.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Center [1987] Ohio Supercomputer Center.  Ohio supercomputer center, 1987.
  * Chao et al. [2016] Wei-Lun Chao, Soravit Changpinyo, Boqing Gong, and Fei Sha.  An empirical study and analysis of generalized zero-shot learning for object recognition in the wild.  In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14_ , pages 52–68. Springer, 2016.
  * Chia et al. [2022] Patrick John Chia, Giuseppe Attanasio, Federico Bianchi, Silvia Terragni, Ana Rita Magalhães, Diogo Goncalves, Ciro Greco, and Jacopo Tagliabue.  Contrastive language and vision learning of general fashion concepts.  _Scientific Reports_ , 12(1):18958, 2022.
  * Cole et al. [2022a] Elijah Cole, Kimberly Wilber, Grant Van Horn, Xuan Yang, Marco Fornoni, Pietro Perona, Serge Belongie, Andrew Howard, and Oisin Mac Aodha.  On label granularity and object localization.  In _European Conference on Computer Vision_ , pages 604–620. Springer, 2022a.
  * Cole et al. [2022b] Elijah Cole, Xuan Yang, Kimberly Wilber, Oisin Mac Aodha, and Serge Belongie.  When does contrastive visual representation learning work?  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14755–14764, 2022b.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Ezray et al. [2019] Briana D. Ezray, Drew C. Wham, Carrie E. Hill, and Heather M. Hines.  Unsupervised machine learning reveals mimicry complexes in bumblebees occur along a perceptual continuum.  _Proceedings of the Royal Society B: Biological Sciences_ , 286(1910):20191501, 2019.
  * Fang et al. [2022] Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt.  Data determines distributional robustness in contrastive language image pre-training (clip).  In _International Conference on Machine Learning_ , pages 6216–6234. PMLR, 2022.
  * G. and J. [2019] Geetharamani G. and Arun Pandian J.  Identification of plant leaf diseases using a nine-layer deep convolutional neural network.  _Computers & Electrical Engineering_, 76:323–338, 2019\.
  * Gadre et al. [2023] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al.  Datacomp: In search of the next generation of multimodal datasets.  _arXiv preprint arXiv:2304.14108_ , 2023.
  * Garcin et al. [2021] Camille Garcin, alexis joly, Pierre Bonnet, Antoine Affouard, Jean-Christophe Lombardo, Mathias Chouet, Maximilien Servajean, Titouan Lorieul, and Joseph Salmon.  Pl@ntnet-300k: a plant image dataset with high label ambiguity and a long-tailed distribution.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_ , 2021.
  * Gharaee et al. [2023] Zahra Gharaee, ZeMing Gong, Nicholas Pellegrino, Iuliia Zarubiieva, Joakim Bruslund Haurum, Scott C Lowe, Jaclyn TA McKeown, Chris CY Ho, Joschka McLeod, Yi-Yun C Wei, et al.  A step towards worldwide biodiversity assessment: The bioscan-1m insect dataset.  _arXiv preprint arXiv:2307.10455_ , 2023.
  * Goyal et al. [2023] Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan.  Finetune like you pretrain: Improved finetuning of zero-shot vision models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 19338–19347, 2023.
  * Gu et al. [2021] Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon.  Domain-specific language model pretraining for biomedical natural language processing.  _ACM Transactions on Computing for Healthcare (HEALTH)_ , 3(1):1–23, 2021.
  * Guralnick et al. [2015] Robert P. Guralnick, Nico Cellinese, John Deck, Richard L. Pyle, John Kunze, Lyubomir Penev, Ramona Walls, Gregor Hagedorn, Donat Agosti, John Wieczorek, Terry Catapano, and Roderic D. M. Page.  Community next steps for making globally unique identifiers work for biocollections data.  _ZooKeys_ , 494:133–154, 2015.
  * Hansen et al. [2020] Oskar L. P. Hansen, Jens-Christian Svenning, Kent Olsen, Steen Dupont, Beulah H. Garner, Alexandros Iosifidis, Benjamin W. Price, and Toke T. Høye.  Species-level image classification with convolutional neural network enables insect identification from habitus images.  _Ecology and Evolution_ , 10(2):737–747, 2020\.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Heidi M. Sosik [2015] Emily F. Brownlee Heidi M. Sosik, Emily E. Peacock.  Annotated plankton images - data set for developing and evaluating classification methods., 2015.
  * Hinchliff et al. [2015a] Cody E. Hinchliff, Stephen A. Smith, James F. Allman, J. Gordon Burleigh, Ruchi Chaudhary, Lyndon M. Coghill, Keith A. Crandall, Jiabin Deng, Bryan T. Drew, Romina Gazis, Karl Gude, David S. Hibbett, Laura A. Katz, H. Dail Laughinghouse, Emily Jane McTavish, Peter E. Midford, Christopher L. Owen, Richard H. Ree, Jonathan A. Rees, Douglas E. Soltis, Tiffani Williams, and Karen A. Cranston.  Synthesis of phylogeny and taxonomy into a comprehensive tree of life.  _Proceedings of the National Academy of Sciences_ , 112(41):12764–12769, 2015a.
  * Hinchliff et al. [2015b] Cody E Hinchliff, Stephen A Smith, James F Allman, J Gordon Burleigh, Ruchi Chaudhary, Lyndon M Coghill, Keith A Crandall, Jiabin Deng, Bryan T Drew, Romina Gazis, et al.  Synthesis of phylogeny and taxonomy into a comprehensive tree of life.  _Proceedings of the National Academy of Sciences_ , 112(41):12764–12769, 2015b.
  * Hobern et al. [2021] Donald Hobern, Saroj K Barik, Les Christidis, Stephen T. Garnett, Paul Kirk, Thomas M Orrell, Thomas Pape, Richard L Pyle, Kevin R Thiele, Frank E Zachos, et al.  Towards a global list of accepted species vi: The catalogue of life checklist.  _Organisms Diversity & Evolution_, 21(4):677–690, 2021.
  * Hoyal Cuthill et al. [2019] Jennifer F Hoyal Cuthill, Nicholas Guttenberg, Sophie Ledger, Robyn Crowther, and Blanca Huertas.  Deep learning on butterfly phenotypes tests evolution’s oldest mathematical model.  _Science advances_ , 5(8):eaaw4967, 2019.
  * Høye et al. [2021] Toke T Høye, Johanna Ärje, Kim Bjerge, Oskar LP Hansen, Alexandros Iosifidis, Florian Leese, Hjalte MR Mann, Kristian Meissner, Claus Melvad, and Jenni Raitoharju.  Deep learning and computer vision will transform entomology.  _Proceedings of the National Academy of Sciences_ , 118(2):e2002545117, 2021.
  * Ikezogwo et al. [2023] Wisdom Oluchi Ikezogwo, Mehmet Saygin Seyfioglu, Fatemeh Ghezloo, Dylan Stefan Chan Geva, Fatwir Sheikh Mohammed, Pavan Kumar Anand, Ranjay Krishna, and Linda Shapiro.  Quilt-1m: One million image-text pairs for histopathology.  _arXiv preprint arXiv:2306.11207_ , 2023.
  * Ilharco et al. [2021] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt.  Openclip, 2021.  If you use this software, please cite it as below.
  * [40] ITIS.  Integrated taxonomic information system (itis) on-line database.  www.itis.gov.  Retrieved July 21, 2023.
  * IUCN [2022] IUCN.  Iucn red list summary table 1a, 2022.
  * Jia et al. [2021] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International conference on machine learning_ , pages 4904–4916. PMLR, 2021.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2012.
  * [44] Dan Liu, Jin Hou, Shaoli Huang, Jing Liu, Yuxin He, Bochuan Zheng, Jifeng Ning, and Jingdong Zhang.  Lote-animal: A long time-span dataset for endangered animal behavior understanding.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 10012–10022, 2021.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  SGDR: stochastic gradient descent with warm restarts.  In _5th International Conference on Learning Representations_ , 2017\.
  * Lu et al. [2023] Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Ivy Liang, Tong Ding, Guillaume Jaume, Igor Odintsov, Andrew Zhang, Long Phi Le, et al.  Towards a visual-language foundation model for computational pathology.  _arXiv preprint arXiv:2307.12914_ , 2023.
  * Lürig et al. [2021] Moritz D Lürig, Seth Donoughe, Erik I Svensson, Arthur Porto, and Masahito Tsuboi.  Computer vision, machine learning, and the promise of phenomics in ecology and evolutionary biology.  _Frontiers in Ecology and Evolution_ , 9:642774, 2021.
  * L. Pyle [2016] Richard L. Pyle.  Towards a global names architecture: The future of indexing scientific names.  _ZooKeys_ , 550:261–281, 2016.
  * Maddison and Schultz [2007] David R. Maddison and K.-S. Schultz.  The Tree of Life Web Project.  2007\.
  * McKinley et al. [2017] Duncan C. McKinley, Abe J. Miller-Rushing, Heidi L. Ballard, Rick Bonney, Hutch Brown, Susan C. Cook-Patton, Daniel M. Evans, Rebecca A. French, Julia K. Parrish, Tina B. Phillips, Sean F. Ryan, Lea A. Shanley, Jennifer L. Shirk, Kristine F. Stepenuck, Jake F. Weltzin, Andrea Wiggins, Owen D. Boyle, Russell D. Briggs, Stuart F. Chapin, David A. Hewitt, Peter W. Preuss, and Michael A. Soukup.  Citizen science can improve conservation science, natural resource management, and environmental protection.  _Biological Conservation_ , 208:15–28, 2017.
  * Miller [1995] George A. Miller.  Wordnet: A lexical database for english.  _Commun. ACM_ , 38(11):39–41, 1995.
  * Mou et al. [2023] Chao Mou, Aokang Liang, Chunying Hu, Fanyu Meng, Baixun Han, and Fu Xu.  Monitoring endangered and rare wildlife in the field: A foundation deep learning model integrating human knowledge for incremental recognition with few data and low cost.  _Animals_ , 13(20):3168, 2023.
  * Nguyen et al. [2022] Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt.  Quality not quantity: On the interaction between dataset design and robustness of clip.  _Advances in Neural Information Processing Systems_ , 35:21455–21469, 2022.
  * Norman et al. [2017] Bradley M. Norman, Jason A. Holmberg, Zaven Arzoumanian, Samantha D. Reynolds, Rory P. Wilson, Dani Rob, Simon J. Pierce, Adrian C. Gleiss, Rafael De La Parra, Beatriz Galvan, Deni Ramirez-Macias, David Robinson, Steve Fox, Rachel Graham, David Rowat, Matthew Potenski, Marie Levine, Jennifer A. Mckinney, Eric Hoffmayer, Alistair D. M. Dove, Robert Hueter, Alessandro Ponzo, Gonzalo Araujo, Elson Aca, David David, Richard Rees, Alan Duncan, Christoph A. Rohner, Clare E. M. Prebble, Alex Hearn, David Acuna, Michael L. Berumen, Abraham Vázquez, Jonathan Green, Steffen S. Bach, Jennifer V. Schmidt, Stephen J. Beatty, and David L. Morgan.  Undersea Constellations: The Global Biology of an Endangered Marine Megavertebrate Further Informed through Citizen Science.  _BioScience_ , 67(12):1029–1043, 2017.
  * Norouzzadeh et al. [2021] Mohammad Sadegh Norouzzadeh, Dan Morris, Sara Beery, Neel Joshi, Nebojsa Jojic, and Jeff Clune.  A deep active learning system for species identification and counting in camera trap images.  _Methods in Ecology and Evolution_ , 12(1):150–161, 2021.
  * Nugent [2018] Jill Nugent.  Inaturalist.  _Science Scope_ , 41(7):12–13, 2018.
  * Oquab et al. [2023] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski.  Dinov2: Learning robust visual features without supervision, 2023.
  * Parham et al. [2017] Jason Parham, Jonathan Crall, Charles Stewart, Tanya Berger-Wolf, and Daniel Rubenstein.  Animal Population Censusing at Scale with Citizen Science and Photographic Identification.  In _AAAI 2017 Spring Symposium on AISOC_ , 2017.
  * Patterson et al. [2016] David Patterson, Dmitry Mozzherin, David Peter Shorthouse, and Anne Thessen.  Challenges with using names to link digital biodiversity information.  _Biodiversity Data Journal_ , 4:e8080, 2016.
  * Pearson et al. [2020] Katelin D Pearson, Gil Nelson, Myla FJ Aronson, Pierre Bonnet, Laura Brenskelle, Charles C Davis, Ellen G Denny, Elizabeth R Ellwood, Hervé Goëau, J Mason Heberling, et al.  Machine learning using digitized herbarium specimens to advance phenological research.  _BioScience_ , 70(7):610–620, 2020.
  * Pham et al. [2023] Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, et al.  Combined scaling for zero-shot transfer learning.  _Neurocomputing_ , 555:126658, 2023.
  * Picek et al. [2021] Lukas Picek, Milan Sulc, Jiri Matas, Jacob Heilmann-Clausen, Thomas S. Jeppesen, Thomas Laessoe, and Tobias Froslev.  Danish fungi 2020 - not just another image recognition dataset.  2021\.
  * Pinho et al. [2022] Catarina Pinho, Antigoni Kaliontzopoulou, Carlos A Ferreira, and João Gama.  Identification of morphologically cryptic species with computer vision models: wall lizards (Squamata: Lacertidae: Podarcis) as a case study.  _Zoological Journal of the Linnean Society_ , 198(1):184–201, 2022.
  * Piosenka [2023] Gerald Piosenka.  Birds 525 species - image classification.  2023\.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Russakovsky et al. [2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ , 115:211–252, 2015.
  * S and J [2020] Roopashree S and Anitha J.  Medicinal leaf dataset.  2020\.
  * Santurkar et al. [2022] Shibani Santurkar, Yann Dubois, Rohan Taori, Percy Liang, and Tatsunori Hashimoto.  Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learning, 2022.
  * Schuhmann et al. [2021] Cristoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Jenia Jitsev, and Aran Komatsuzaki.  LAION-400M: Open dataset of CLIP-filtered 400 million image-text pairs.  In _Proceedings of Neurips Data-Centric AI Workshop_ , 2021.
  * Serret et al. [2019] Hortense Serret, Nicolas Deguines, Yikweon Jang, Gregoire Lois, and Romain Julliard.  Data quality and participant engagement in citizen science: comparing two approaches for monitoring pollinators in france and south korea.  _Citizen Science: Theory and Practice_ , 4(1):22, 2019.
  * Simpson et al. [2014] Robert Simpson, Kevin R. Page, and David De Roure.  Zooniverse: observing the world’s largest citizen science platform.  In _Proceedings of the 23rd International Conference on World Wide Web_ , pages 1049–1054, Seoul, Korea, 2014. Association for Computing Machinery.  Type: 10.1145/2567948.2579215.
  * Singh et al. [2020] Davinder Singh, Naman Jain, Pranjali Jain, Pratik Kayal, Sudhakar Kumawat, and Nipun Batra.  Plantdoc: A dataset for visual plant disease detection.  In _Proceedings of the 7th ACM IKDD CoDS and 25th COMAD_ , pages 249–253, New York, NY, USA, 2020. Association for Computing Machinery.
  * Steenweg et al. [2017] Robin Steenweg, Mark Hebblewhite, Roland Kays, Jorge Ahumada, Jason T Fisher, Cole Burton, Susan E Townsend, Chris Carbone, J Marcus Rowcliffe, Jesse Whittington, Jedediah Brodie, J Andrew Royle, Adam Switalski, Anthony P Clevenger, Nicole Heim, and Lindsey N Rich.  Scaling-up camera traps: monitoring the planet’s biodiversity with networks of remote sensors.  _Frontiers in Ecology and the Environment_ , 15(1):26–34, 2017.  ISBN: 1540-9295 Publisher: John Wiley & Sons, Ltd Type: https://doi.org/10.1002/fee.1448.
  * Sullivan et al. [2014a] Brian L. Sullivan, Jocelyn L. Aycrigg, Jessie H. Barry, Rick E. Bonney, Nicholas Bruns, Caren B. Cooper, Theo Damoulas, André A. Dhondt, Tom Dietterich, Andrew Farnsworth, Daniel Fink, John W. Fitzpatrick, Thomas Fredericks, Jeff Gerbracht, Carla Gomes, Wesley M. Hochachka, Marshall J. Iliff, Carl Lagoze, Frank A. La Sorte, Matthew Merrifield, Will Morris, Tina B. Phillips, Mark Reynolds, Amanda D. Rodewald, Kenneth V. Rosenberg, Nancy M. Trautmann, Andrea Wiggins, David W. Winkler, Weng-Keen Wong, Christopher L. Wood, Jun Yu, and Steve Kelling.  The eBird enterprise: An integrated approach to development and application of citizen science.  _Biological Conservation_ , 169:31–40, 2014a.
  * Sullivan et al. [2014b] Brian L Sullivan, Jocelyn L Aycrigg, Jessie H Barry, Rick E Bonney, Nicholas Bruns, Caren B Cooper, Theo Damoulas, André A Dhondt, Tom Dietterich, Andrew Farnsworth, et al.  The ebird enterprise: An integrated approach to development and application of citizen science.  _Biological conservation_ , 169:31–40, 2014b.
  * Swanson et al. [2015] Alexandra Swanson, Margaret Kosmala, Chris Lintott, Robert Simpson, Arfon Smith, and Craig Packer.  Snapshot Serengeti, high-frequency annotated camera trap images of 40 mammalian species in an African savanna.  _Scientific Data_ , 2(150026):1–14, 2015.
  * Teng et al. [2023] Mélisande Teng, Amna Elmustafa, Benjamin Akera, Hugo Larochelle, and David Rolnick.  Bird Distribution Modelling using Remote Sensing and Citizen Science data, 2023.  arXiv:2305.01079 [cs].
  * Tuia et al. [2022] Devis Tuia, Benjamin Kellenberger, Sara Beery, Blair R Costelloe, Silvia Zuffi, Benjamin Risse, Alexander Mathis, Mackenzie W Mathis, Frank van Langevelde, Tilo Burghardt, et al.  Perspectives in machine learning for wildlife conservation.  _Nature communications_ , 13(1):792, 2022.
  * Ullah et al. [2022] Ihsan Ullah, Dustin Carrion, Sergio Escalera, Isabelle M Guyon, Mike Huisman, Felix Mohr, Jan N van Rijn, Haozhe Sun, Joaquin Vanschoren, and Phan Anh Vu.  Meta-album: Multi-domain meta-dataset for few-shot image classification.  In _Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2022.
  * Van der Maaten and Hinton [2008] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  _Journal of machine learning research_ , 9(11), 2008.
  * Van Horn and Mac Aodha [2021] Grant Van Horn and Oisin Mac Aodha.  inat challenge 2021 - fgvc8, 2021.
  * Van Horn et al. [2015] Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie.  Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection.  In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 595–604, 2015.  ISSN: 1063-6919.
  * Van Horn et al. [2018a] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie.  The inaturalist species classification and detection dataset.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018a.
  * Van Horn et al. [2018b] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie.  The iNaturalist Species Classification and Detection Dataset, 2018b.  arXiv:1707.06642 [cs].
  * Wah et al. [2011] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.  The caltech-ucsd birds-200-2011 dataset.  Technical Report CNS-TR-2011-001, California Institute of Technology, 2011\.
  * Wang et al. [2019] Yan Wang, Wei-Lun Chao, Kilian Q. Weinberger, and Laurens van der Maaten.  Simpleshot: Revisiting nearest-neighbor classification for few-shot learning.  _CoRR_ , 2019.
  * Wu et al. [2019] Xiaoping Wu, Chi Zhan, Yukun Lai, Ming-Ming Cheng, and Jufeng Yang.  Ip102: A large-scale benchmark dataset for insect pest recognition.  In _IEEE CVPR_ , pages 8787–8796, 2019.
  * Xiao et al. [2021] Tete Xiao, Xiaolong Wang, Alexei A Efros, and Trevor Darrell.  What should not be contrastive in contrastive learning.  In _International Conference on Learning Representations_ , 2021.
  * Xu et al. [2023a] Hu Xu, Saining Xie, Po-Yao Huang, Licheng Yu, Russell Howes, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer.  CiT: Curation in Training for Effective Vision-Language Data, 2023a.
  * Xu et al. [2023b] Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer.  Demystifying clip data, 2023b.
  * Yuan et al. [2021] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al.  Florence: A new foundation model for computer vision.  _arXiv preprint arXiv:2111.11432_ , 2021.
  * Zhang et al. [2022] Shu Zhang, Ran Xu, Caiming Xiong, and Chetan Ramaiah.  Use all the labels: A hierarchical multi-label contrastive learning framework.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16660–16669, 2022.

## Appendices

Many details are omitted in the main text because of space concerns; we
present relevant details here.

  1. 1.

Appendix A: Reproducibility statement

  2. 2.

Appendix B: Ethics statement

  3. 3.

Appendix C: Details of training data aggregation

  4. 4.

Appendix D: Training details and hyperparameters

  5. 5.

Appendix E: Standard deviations for few-shot results

  6. 6.

Appendix F: Example zero-shot predictions on our evaluation tasks.

  7. 7.

Appendix G: Additional text-type results

##  Appendix A Reproducibility Statement

We will ensure reproducibility of our results by releasing our datasets
(TreeOfLife-10M and Rare Species), data pre-processing code, training code,
evaluation code, code to generate all figures (Figs. 2 and 3), and pre-trained
model weights. With these resources, anyone with sufficient compute resources
can download the original data, then reproduce the pre-processing, training,
and evaluation. For those with limited compute, the pre-trained model weights
will enable full reproducibility of our evaluation results (Sec. 4).

##  Appendix B Ethics Statement

We are not aware of any major ethical issues that arise from our work. is
further pre-trained from the original CLIP model; many of the same concerns
(class design, surveillance, etc.) apply; however, these concerns are
discussed in great detail in Radford et al. [66], so we will focus on
addressing these concerns as they relate to the biological addition provided
in .

How could affect endangered species–does or TreeOfLife-10M pose a threat by
aiding poachers? Though leads to improved automatic species classification, it
does not include specific geographic information such as GPS coordinates.
Furthermore, animal conservation status is not included during training.

Could have a negative impact on biologists? is designed to combine visual cues
with an established taxonomic hierarchy to aid in scientific discovery.
Concerns regarding over-reliance on model predictions is a warning that
accompanies many–if not all–contemporary models and is not unique to . The
goal is for to aid biologists in their work, not to replace them. As such, it
is important for users to retain that understanding/context when applying to
downstream tasks.

##  Appendix C Training Data Aggregation

We aggregate images and labels from the iNat21 training data, Bioscan-1M’s,
and data downloaded from EOL. While every image has at least one taxonomic
rank labeled, full taxonomic hierarchies and common names are scraped on a
best-effort basis from metadata providers, including iNaturalist (iNaturalist
Taxonomy DarwinCore Archive), Encyclopedia of Life (eol.org) and Integrated
Taxonomic Information System (ITIS) (itis.gov).

We create a lookup between scientific name and taxonomic hierarchy and a
lookup between scientific name and common name. We populate these lookups
using the following sources in order of descending prioritization, as earlier
sources are considered more authoritative. That is, if a duplicate appears in
a later source, it is superseded by the higher priority source: Bioscan-1M
metadata, EOL aggregate datasets: information retrieved using EOL page IDs
with the pages API, which checks for a match in the ITIS hierarchy for higher-
level taxa standardization (setting aside homonyms for proper linkage). The
full list of taxa and vernacular names provided by iNaturalist and the iNat21
training set class names were maintained. From here, any taxa that could not
be resolved using these sources were fed through the Global Names Resolver
(GNR) API. Overall we were able to achieve 84% full taxa labeling for images
in TreeOfLife-10M, for context, 10% of TreeOfLife-10M is only labeled down to
the family rank (Bioscan-1M), thus, genus-species information is not
available.

##  Appendix D Hyperparameters & Training Details

Tabs. D1 and D2 contain our training hyperparameters for the different models.
Tab. D2 notes the different epochs at which we had the lowest validation loss,
as evaluated using the CLIP objective on the validation split of
TreeOfLife-10M (even for the TreeOfLife-1M models). We will release our
training code upon acceptance.

Hyperparameter | Value
---|---
Architecture | ViT-B/16
Max learning rate | $1\times 10^{-4}$
Warm-up steps | $1,000$
Weight Decay | $0.2$
Input Res. | $224\times 224$
Table D1: Common hyperparameters among all models we train. Dataset | Text Type | Batch Size | Epoch
---|---|---|---
TreeOfLife-10M | Mixture | 32K | 100
iNat21 Only | Mixture | 16K | 65
TreeOfLife-1M | Common | 16K | 86
Scientific | 87
Taxonomy | 87
Sci+Com | 87
Tax+Com | 86
Mixture | 91
Table D2:  Hyperparameters that differ between the various models we train. We
use a smaller batch size and only 1M examples for our text-type ablation
because of limited compute.

We trained a hierarchical classification model in Sec. 4.4. Python pseudocode
for the training objective is in Fig. D1. We will publicly release full
training code upon acceptance.

⬇

import torch.nn.functional as F

\pardef forward(vit, heads, images, h_labels):

”””

vit: vision transformer.

heads: linear layers, one for each taxonomic

rank.

images: batch of input images

h_labels: hierarchical labels; each image has

7 labels

”””

img_feats = vit(images)

h_logits = [head(img_feats) for head in heads]

losses = [F.cross_entropy(logits, label)

for logits, labels in zip(h_logits, h_labels)]

return sum(losses)

Figure D1:  Python code to calculate the hierarchical multitask objective.
Each image has 7 class labels: one for each taxonomic rank. The ViT calculates
dense features for each image, then each taxonomic rank has its own linear
layer that produces logits. By summing the losses, the ViT learns to produce
image features that are useful for classifying images at multiple taxonomic
ranks.

##  Appendix E Standard Deviation of Main Results

Tabs. E3 and E4 show the accuracy with standard deviation over five runs on
the test sets presented in Tab. 2. Since we randomly select the training
examples from the datasets for few-shot, accuracies vary based on which
examples are train examples and which are test examples. However, the
variation is small enough that our conclusions in Sec. 4.5 still hold. Zero-
shot results are deterministic and have no variation.

Model |  Birds 525 |  Plankton |  Insects |  Insects 2 |  Rare Species
---|---|---|---|---|---
One-Shot Classification
CLIP | $43.7\pm 0.26$ | $25.1\pm 0.71$ | $21.6\pm 1.05$ | $13.7\pm 1.09$ | $28.4\pm 0.92$
OpenCLIP | $53.7\pm 0.52$ | $\mathbf{32.3\pm 0.63}$ | $23.2\pm 1.58$ | $14.3\pm 0.67$ | $29.3\pm 0.68$
| $71.8\pm 0.47$ | $30.6\pm 0.77$ | $\mathbf{57.4\pm 2.4}$ | $\mathbf{20.4\pm 1.28}$ | $\mathbf{45.3\pm 1.16}$
– iNat21 Only | $\mathbf{74.8\pm 0.89}$ | $29.6\pm 0.82$ | $53.9\pm 0.97$ | $19.7\pm 0.80$ | $37.6\pm 0.63$
Five-Shot Classification
CLIP | $73.5\pm 0.37$ | $41.2\pm 1.01$ | $39.9\pm 0.86$ | $24.6\pm 0.90$ | $46.9\pm 0.21$
OpenCLIP | $81.9\pm 0.25$ | $\mathbf{52.5\pm 0.83}$ | $42.6\pm 0.82$ | $25.0\pm 0.83$ | $48.4\pm 0.62$
| $90.0\pm 0.12$ | $49.3\pm 1.14$ | $\mathbf{77.8\pm 0.81}$ | $\mathbf{33.6\pm 0.74}$ | $\mathbf{66.4\pm 0.16}$
– iNat21 Only | $\mathbf{90.1\pm 0.08}$ | $48.2\pm 1.24$ | $73.7\pm 0.65$ | $32.1\pm 1.97$ | $56.0\pm 0.36$
Table E3: Accuracy with standard deviation of five runs on animals and rare species in Tab. 4 Model |  PlantNet |  Fungi |  PlantVillage |  Med. Leaf |  PlantDoc
---|---|---|---|---|---
One-Shot Classification
CLIP | $42.1\pm 3.40$ | $17.2\pm 0.78$ | $49.7\pm 2.53$ | $70.1\pm 2.83$ | $24.8\pm 1.61$
OpenCLIP | $45.1\pm 3.40$ | $18.4\pm 1.26$ | $53.6\pm 0.79$ | $71.2\pm 3.58$ | $26.8\pm 1.45$
| $64.5\pm 2.15$ | $\mathbf{40.3\pm 3.00}$ | $\mathbf{58.8\pm 2.83}$ | $\mathbf{84.3\pm 1.90}$ | $\mathbf{30.7\pm 1.75}$
– iNat21 Only | $\mathbf{67.4\pm 4.54}$ | $35.5\pm 2.93$ | $55.2\pm 1.58$ | $75.1\pm 1.16$ | $27.8\pm 1.31$
Five-Shot Classification
CLIP | $65.2\pm 1.25$ | $27.9\pm 2.54$ | $71.8\pm 1.46$ | $89.7\pm 1.45$ | $35.2\pm 1.59$
OpenCLIP | $68.0\pm 0.86$ | $30.6\pm 1.26$ | $77.8\pm 1.28$ | $91.3\pm 0.85$ | $42.0\pm 1.32$
| $\mathbf{85.6\pm 1.79}$ | $\mathbf{62.3\pm 1.82}$ | $\mathbf{80.9\pm 1.04}$ | $\mathbf{95.9\pm 1.07}$ | $\mathbf{47.5\pm 1.35}$
– iNat21 Only | $84.7\pm 1.24$ | $55.6\pm 2.61$ | $77.2\pm 0.68$ | $93.5\pm 1.13$ | $41.0\pm 1.75$
Table E4: Accuracy with standard deviation of five runs on plants and fungi in
Tab. 4

##  Appendix F Example Predictions

Figure F2:  Example predictions for and CLIP on Birds 525, Plankton, Insects,
Insects2, PlantNet and Fungi tasks. Ground truth labels are green; incorrect
predictions are red. Left: Correct predictions. Center, Right: Images that
CLIP incorrectly labels, but correctly labels.

Figure F3:  Example predictions for and CLIP on PlantVillage, Medicinal Leaf,
PlantDoc and Rare Species. Ground truth labels are green; incorrect
predictions are red. Left: Correct predictions. Center, Right: Images that
CLIP incorrectly labels, but correctly labels.

Figs. F2 and F3 show and CLIP zero-shot predictions on all ten evaluation
tasks. We randomly pick examples from each dataset that correctly labels and
example that CLIP incorrect labels but correctly labels. performs well on a
variety of tasks, including out-of-distribution images (Plankton, Medicinal
Leaf) and mixes of scientific and common names (PlantVillage, PlantDoc).

##  Appendix G More Results of Text-Type

We investigated the effects of text-type during training and testing in Sec.
4.3 using the Rare Species task. We present zero-shot results for all text-
types on all tasks using the same procedure as in Sec. 4.2, where we use
whatever taxonomic+common if available, otherwise whatever text-type is
available.

|  Animals |  Plants & Fungi |  |  |
---|---|---|---|---|---
Training Text Type |  Birds 525 |  Plankton |  Insects |  Insects 2 |  PlantNet |  Fungi |  PlantVillage |  Med. Leaf |  PlantDoc |  Rare Species |  Mean ($\Delta$)
Random Guessing | $0.2$ | $1.2$ | $1.0$ | $1.0$ | $4.0$ | $4.0$ | $2.6$ | $4.0$ | $3.7$ | $0.3$ | $2.2$ |
Common | $42.4$ | $4.4$ | $15.8$ | $13.3$ | $45.2$ | $20.7$ | $10.7$ | $15.4$ | $19.6$ | $23.1$ | $21.0$ | $-12.0$
Scientific | $11.3$ | $3.8$ | $18.7$ | $11.0$ | $84.8$ | $35.3$ | $12.5$ | $20.3$ | $13.9$ | $7.7$ | $21.9$ | $-11.1$
Taxonomic | $46.6$ | $2.2$ | $25.1$ | $8.7$ | $70.4$ | $29.0$ | $8.8$ | $18.4$ | $12.8$ | $24.1$ | $24.6$ | $-8.4$
Sci+Com | $55.0$ | $2.2$ | $19.2$ | $12.6$ | $71.5$ | $24.8$ | $17.6$ | $21.5$ | $20.0$ | $25.3$ | $26.9$ | $-6.1$
Tax+Com | $61.9$ | $2.0$ | $27.4$ | $11.6$ | $68.4$ | $19.2$ | $10.4$ | $19.5$ | $15.8$ | $30.0$ | $26.6$ | $-6.4$
Mixture | $66.6$ | $3.5$ | $30.6$ | $17.3$ | $86.3$ | $32.8$ | $19.9$ | $18.7$ | $24.5$ | $30.5$ | $33.0$ | –

Table G5:  Zero–shot classification top-1 accuracy for different text-types
used during training. Bold indicates best accuracy. All models use the same
architecture (ViT-B/16 vision encoders, 77-token text encoder) and are trained
on the same dataset (TreeOfLife-1M). $\Delta$ denotes the difference in mean
accuracy with “Mixture”, which is the text-type we used for .

##  Appendix H Generalized Zero-Shot Learning

Chao et al. [16] introduced generalized zero-shot learning, where a model must
label images of unseen classes from a set of both seen and unseen labels. We
pick out a set of 400 seen species from TreeOfLife-10M using the same
methodology as we used for the Rare Species task. We classify the same images
from the Rare Species task using this set of 800 labels (a mix of seen and
unseen). CLIP and OpenCLIP achieve $24.1$% and $21.7$% top-1 accuracy, while
achieves $26.4$% top-1 accuracy in this challenging GZSL setting.