# Pre-training Language Models for Comparative Reasoning

Mengxia Yu, Zhihan Zhang, Wenhao Yu, Meng Jiang
Department of Computer Science and Engineering
University of Notre Dame
<EMAIL_ADDRESS>
###### Abstract

In this paper, we propose a novel framework to pre-train language models for
enhancing their abilities of comparative reasoning over texts. While recent
research has developed models for NLP tasks that require comparative
reasoning, they suffer from costly manual data labeling and limited
generalizability to different tasks. Our approach involves a scalable method
for collecting data for text-based entity comparison, which leverages both
structured and unstructured data, and the design of three novel pre-training
tasks. Evaluation on a range of downstream tasks including comparative
question answering, question generation, and summarization shows that our pre-
training framework significantly improves the comparative reasoning abilities
of language models, especially under low-resource conditions. This work also
releases the first integrated benchmark for comparative reasoning over texts.

##  1 Introduction

Figure 1: The framework of pre-training language models (LMs) for comparative
reasoning abilities. In Step 1, we collect quintuples for entity comparison by
combining structured knowledge base (i.e., Wikidata) and unstructured corpora
(i.e., Gigawords, CC-News, Wikipedia). Details are in $\S$ 3.2.2. In Step 2,
to obtain text-based pre-training data, we convert the quintuples into
synthetic QA pairs and summaries with a set of QA templates and a fine-tuned
data-to-text model, respectively. We gather Wikipedia documents as text
descriptions of entities. Details are in $\S$ 3.2.3. In Step 3, we design
novel seq-to-seq pre-training tasks for the LMs. Details are described in $\S$
3.3.

Comparative reasoning constitutes a fundamental cognitive ability that plays a
crucial role in human decision-making processes. It involves comparing and
contrasting various objects, concepts, or entities to draw conclusions or make
informed decisions. For example, consumers often compare products based on
features such as price, quality, and user reviews before making a purchase
decision. Similarly, policymakers weigh the advantages and disadvantages of
different policy proposals to address pressing issues. In the context of
textual documents, comparative reasoning is crucial for tasks such as
identifying differences between research papers, contrasting news articles
from different sources, or synthesizing the arguments of opposing viewpoints
in a debate.

In recent years, there have been studies developing natural language
processing (NLP) models capable of mining (Jindal and Liu, 2006; Li et al.,
2011), understanding (Bondarenko et al., 2022; Bista et al., 2019), and
generating (Iso et al., 2022; Beloucif et al., 2022) comparative contents over
texts. Yet, a substantial barrier persists: these models often require labor-
intensive manual data labeling, rendering them costly and unfeasible for
large-scale applications. Moreover, these models are designed for a particular
task, which limits their generalizability on different or emerging tasks.
Meanwhile, pre-trained language models (PLM) such as BART Lewis et al. (2020)
and T5 Raffel et al. (2020) exhibit good generalizability on several NLP
tasks. However, existing pre-training methods (e.g., masked language modeling
and span in-filling) could not grant the language models strong comparative
reasoning abilities, especially in few-shot and zero-shot settings (see
results in Table 4).

In response to these challenges, in this paper, we present a novel pre-
training framework to enhance the comparison reasoning abilities of PLMs,
specifically by capturing comparable information within paired documents more
effectively. Our approach pilots around a scalable, labor-free data collection
method that can gather a wealth of facts for entity comparison by combining
structured (e.g., Wikidata) and unstructured data (e.g., news sources and
Wikipedia). We represent these facts as quintuples, which consist of a pair of
entities and the corresponding values of their shared property. To enable the
pre-training in a text-to-text manner, we convert the quintuples into textual
components such as question-answer pairs and brief summaries of the two
entities. We further design three pre-training tasks, including generating
synthetic comparative answers, questions, and summaries, given the documents
of two entities as contexts. Subsequently, we unify the pre-training tasks by
multi-task learning. To our best knowledge, we are the first to pre-train
language models for comparative reasoning.

We assess the effectiveness of our approach by benchmarking comparative
reasoning on a suite of tasks, including comparative question answering,
comparative question generation, and comparative summarization. Our
experimental results demonstrate a notable improvement in the performance of
conventional PLMs including BART and T5 under limited-resource scenarios.

Our contributions are three-fold:

  * •

We propose a scalable method for synthesizing data for entity comparison,
leveraging both structured and unstructured data sources.

  * •

We present a novel framework for pre-training PLMs to enhance their
comparative reasoning abilities on multiple related tasks.

  * •

We provide the first benchmark for entity comparison over texts, serving as a
foundation for future research in this domain.

##  2 Related Work

###  2.1 Comparative Reasoning Tasks

Early studies focused on mining explicit comparative information from massive
corpora, such as identifying comparative sentences Jindal and Liu (2006),
mining comparable entities (Li et al., 2011), and classifying components of
comparison Beloucif et al. (2022). Recent work focused more on natural
language generation tasks such as generating arguments to answer comparative
questions Chekalina et al. (2021), generating comparable questions for news
articles Beloucif et al. (2022), and summarize comparative opinions (Lerman
and McDonald, 2009; Iso et al., 2022). However, the existing techniques were
designed for specific tasks, and they are limited by the scarcity of
supervised data, which poses a challenge due to the labor-intensive nature of
data collection.

###  2.2 Language Models Pre-training

The combination of structured and unstructured data in language model pre-
training has garnered considerable attention in recent research. Early work
proposed to fuse KG information and text by encoding the graph structure and
take entity embeddings as a part of the input, such as ERNIE Zhang et al.
(2019), K-Adapter Wang et al. (2020), KEPLER Wang et al. (2021), K-BERT Liu et
al. (2020), and JAKET Yu et al. (2022). Another branch of work proposed to
integrate entity information (Xiong et al., 2020; Qin et al., 2021; Zhang et
al., 2022) or relation information (Qin et al., 2021; Hu et al., 2021) without
modifying language model’s structure. Notably, RGPT-QA Hu et al. (2021)
combines Wikidata and Wikipedia to generate synthetic QA pairs for pre-
training. Although their pre-trained data was constructed by statements of
single entity, making it not suitable for multi-hop QA such as comparative
questions, it demonstrated the effectiveness of combining Wikidata and
Wikipedia to generate synthetic QA data for pre-training. However, the above
pre-training methods are designed for LMs for natural language understanding
(NLU), which are not suitable for NLG tasks.

Recent work such as KG-BART Liu et al. (2021) and OREOLM Hu et al. (2022)
brought structured knowledge into generative LMs by integrating graph-based
knowledge augmented modules. However, they require knowledge graph as a part
of their inputs and process them by the graph neural network-based modules.

##  3 Pre-training Framework

Task | Source $\rightarrow$ Target
---|---
Comparative Answer Generation | Answer the comparative question. Question: {Q} Context: {D1} [SEP] {D2} $\rightarrow$ A
Comparative QA Pairs Generation | Generate a comparative question-answer pair. Context: {D1} [SEP] {D2} $\rightarrow$ Q; A
Comparative Summary Generation | Generate a comparative summary. Context: {D1} [SEP] {D2} $\rightarrow$ S
Text Infilling (BART pre-training) | {corrupted D1} [SEP] {corrupted D2} $\rightarrow$ {D1} [SEP] {D2}

Table 1: Task names and the format of source-target sequence format in each of
our multi-task pre-training tasks.

Our framework explicitly teaches language models at the pre-training stage
about comparative reasoning. Specifically, they are given a pair of documents,
each describing an entity, and are trained to generate a piece of text
pertaining to the comparison between these two entities. Regarding the types
of output texts, we design three sequence-to-sequence pre-training tasks that
require the model to simultaneously attend to both documents and extract
information for pairwise comparison. The pre-training tasks include
comparative answer generation ($\S$3.3.1), question-answer generation
($\S$3.3.2), and summarization ($\S$3.3.3).

To enable large-scale pre-training with a data collection, we utilized
structured data (e.g., Wikidata) and unstructured corpora (e.g., Gigawords,
CC-News and Wikipedia) to obtain quintuples, a novel structural unit on entity
comparison. Then we convert the quintuples to textual components for the
sequence-to-sequence pre-training.

###  3.1 Notations

Given a (head) entity $e$, we denote a Wikidata statement as $(e,p,v)$, where
$p$ is a property or relation, and $v$ is a value or (tail) entity. Given
entities $e_{1}$ and $e_{2}$, we obtain their Wikidata statements. From the
two sets of statements, we first extract _quintuples_ for entity comparison. A
quintuple is represented as $(e_{1},e_{2},p,v_{1},v_{2})$, where $p$ is a
common property of $e_{1}$ and $e_{2}$, and $v_{1}$ and $v_{2}$ are the
corresponding values in the Wikidata statements. Such quintuples enable the
comparison on shared properties, reflecting the similarity or difference
between the corresponding property values or tail entities.

Then, we convert quintuples into textual forms, such as question-answer pairs,
which are denoted by $(Q,A)$, and summaries denoted by $S$. Additionally, we
extract text descriptions $D_{1}$ and $D_{2}$ from Wikipedia for $e_{1}$ and
$e_{2}$, respectively.

###  3.2 Pre-training Data Preparation

####  3.2.1 Data sources

Our approach uses large-scale data with free access, as needed in most pre-
training frameworks. We design novel pre-training tasks using both
unstructured and structured data sources, compared to the existing frameworks
that use mainly the text corpora. Because the structured data help define the
tasks for comparative reasoning.

Wikidata111For Wikidata, we use the dump wikidata-20220103-all. is a
collaborative knowledge base that stores data in a structured format. Wikidata
contains a set of statements that describe entities, where each statement
includes a property and a value, as denoted in $\S$3.1. Values can be object
entities which have unique identifiers, or literal values including date
values, numerical values or strings. Each entity and property is associated
with a set of aliases.

Wikipedia222For Wikipedia, we use the data dump enwiki-20220101-pages-
articles-multistream. is a encyclopedia that contains a vast collection of
articles covering a wide range of entities. A Wikidata entity can be linked to
a Wikipedia article through a property named “sitelink”.

Text corpora, encompassing news sources (e.g.,
Gigawords333https://catalog.ldc.upenn.edu/LDC2011T07, CC-News444
https://huggingface.co/datasets/cc_news) and Wikipedia, offer an abundance of
information for determining the comparability of entities and the properties
for the comparisons. For example, a sentence in a piece of news from New York
Times like _“The show, with a book by the screenwriter _Diablo Cody_ (‘Juno’)
and staging by director _Diane Paulus_ (‘Waitress’), takes on the good work
…,”_ indicating that Diablo Cody and Diane Paulus could be compared on the
property of work (values: screenwriter vs. director).

####  3.2.2 Quintuple collection

In this section, we introduce the process of collecting quintuples by
combining Wikidata and text corpora. The underlying hypothesis guiding our
efforts is that when a pair of statements concerning the same property of
related entities co-occur in a textual context, there is a high probability
that these statements are indeed comparable.

To extract this comparability information, we first sample a document from the
text corpora. Then, we link Wikidata statements to the sentences in corpora by
identifying the mentions of entity $e$, property $p$, and value $v$ using
string matching based on their aliases provided in Wikidata. Specifically, a
statement $(e,p,v)$ is linked to a sentence if the aliases of $e$, $p$, and
$v$ all appear in the sentence. To increase the linking accuracy, we properly
tokenize the sentences, convert all text to lowercase, and remove stop words.
Next, we pair $(e_{1},p_{1},v_{1})$ and $(e_{2},p_{2},v_{2})$ if they satisfy
the following criteria:

  1. 1.

$e_{1}$ and $e_{2}$ belong to the same category, e.g., they both have the
value human for property instance of. This ensures the entities are analogous
to each other.

  2. 2.

$p_{1}$ and $p_{2}$ are equal. This follows the principle that comparisons are
made on a common property of two entities.

  3. 3.

The sentences linked to $(e_{1},p_{1},v_{1})$ and $(e_{2},p_{2},v_{2})$ co-
occur within the same context (e.g., a short paragraph of a news article).
Being mentioned together indicates a high probability of implicit comparison.

We denote such a statement pair as a quintuple $(e_{1},e_{2},p,v_{1},v_{2})$.
Such quintuples store information for entity comparison.

####  3.2.3 Quintuple textualization

In order to pre-train the language model in a text-to-text manner, it is
crucial to represent the comparative information inherent in the quintuples in
a textual form. We aim to explicitly train the model to capture comparable
information from a pair of documents and make comparisons. To this end, we
propose to input a pair of documents, each containing a text description of a
single entity, and train the model to generate texts involving comparison.

As part of this process, we extract documents for each pair of entities
$e_{1}$ and $e_{2}$. First, we find Wikipedia articles of $e_{1}$ and $e_{2}$
by the links provided in Wikidata. Next, we split the articles into
10-sentence segments. To ensure the information within the quintuple can be
inferred from the documents, we link $(e_{1},p,v_{1})$ and $(e_{2},p,v_{2})$
to sentences in the articles of $e_{1}$ and $e_{2}$, respectively. We link the
statements based on two assumptions: (1) Within an article pertaining to
entity $e$, sentences are highly probable to discuss $e$ as their subject; (2)
If a sentence in a Wikipedia article of $e$ mentions both $e$ and $v$ from a
Wikipedia statement $(e,p,v)$ , then it is highly likely that the sentence
describes the fact of $(e,p,v)$. Thus, we link the statements to sentences
whenever $(e,v)$ or $(p,v)$ can be matched. To assess the linking quality, we
randomly sampled 100 statement-sentence links and manually evaluate the the
accuracy. The linking accuracy exceeds 95%, indicating the Wikidata statements
are effectively linked to the sentences. Finally, We choose text segments from
the articles of $e_{1}$ that can be linked to the $(e_{1},p,v_{1})$ as
document $D_{1}$. Similarly, we obtain $D_{2}$ for $e_{2}$.

To enable the model to make comparisons in a text-to-text manner, the
comparison knowledge encapsulated within the quintuples is converted into
natural language forms, namely, question-answer pairs and summaries. The
conversion to question-answer pairs $(Q,A)$ are synthesized using predefined
templates shown in Table 2.

To generate synthetic comparative summaries $S$, we utilize a data-to-text
model Ribeiro et al. (2021) that has been fine-tuned with DART Nan et al.
(2021) dataset. This allows us to transform quintuples into concise
declarative sentences.

All:
---
Q: Do $e_{1}$ and $e_{2}$ have the same/different value of $p$?
A: Yes/No
Q: Do $e_{1}$ and $e_{2}$ both have the value of v1 in terms of $p$?
A: Yes/No
Q: What are the $p$ of $e_{1}$ and $e_{2}$?
A: $v_{1}$, $v_{2}$
When $v_{1}$ $\neq$ $v_{2}$:
Q: Which one of the following entity’s $p$ is $v_{1}$? $e_{1}$ or $e_{2}$?
A: $e_{1}$
Q: Is $e_{1}$’s $p$ $v_{1}$ or $v_{2}$?
A: $v_{1}$
When $v_{1}$ = $v_{2}$:
Q: Which entity has the same value as $e_{1}$ in terms of $p$?
A: $e_{2}$
Q: $e_{1}$ and $e_{2}$ are known for what (value) of $p$?
A: $v_{1}$/$v_{2}$

Table 2: Synthetic QA templates. All indicates the following templates is
applicable to all quintuples. The templates under When $v_{1}$ $\neq$ $v_{2}$:
or When $v_{1}$ $\neq$ $v_{2}$: will be applied to quintuples whose $v_{1}$
and $v_{2}$ are different or the same, respectively.

###  3.3 Pre-training Tasks and Objectives

We propose three comparative pre-training tasks, including comparative answer
generation, QA pairs generation, and summary generation. They are all text
generation tasks, which fit the architectures of popular language models such
as BART and T5 very well. We unify the pre-training tasks with task specific
prompts, as shown in Table 1.

####  3.3.1 Comparative answer generation

We concatenate the synthesized comparative questions and the two documents as
input. The model is trained to generate the corresponding answer. This task
not only activates the attention mechanism between the question and relevant
contexts in each document, more importantly, it encourages the interaction
between both documents, which is essential to make the comparison. We define
the loss function for a training sample as

$\mathcal{L}_{\rm{QA}}=-\sum_{(Q_{i},A_{i})\in\mathcal{T}}\log
P(A_{i}|Q_{i},D_{1},D_{2}),$ (1)

in which $\mathcal{T}$ is a set of QA pairs derived from the templates. and
$P(\cdot)$ is the predicted probability.

####  3.3.2 Comparative QA pairs generation

Given two documents, the model is required to generate comparative questions
and answers. By learning to generate comparative questions, the model learns
to attend to both documents and reason about common and valuable properties of
the two entities:

$\mathcal{L}_{\rm{QAG}}=-\sum_{(Q_{i},A_{i})\in\mathcal{T}}\log
P(Q_{i},A_{i}|D_{1},D_{2}).$ (2)

####  3.3.3 Comparative summary generation

Given two documents, the model is tasked with generating short comparative
summaries that represent the comparable statements:

$\mathcal{L}_{\rm{SUM}}=-\sum_{S\in\mathcal{S}}\log P(S|D_{1},D_{2}),$ (3)

where $\mathcal{S}$ is the set of summaries from quintuple textualization.

####  3.3.4 Pre-training objectives

Inspired by the multi-task prompted training Sanh et al. (2022), we unify the
aforementioned pre-training tasks with natural language prompts. The detailed
format of source and target sequences are shown in Table 1. The model is
jointly optimized for all tasks using a shared loss function, which encourages
the model to learn generalizable representations that are beneficial across
tasks. For BART, we train the proposed pre-training tasks together with the
text infilling (TI) task, where the model requires to reconstruct the text
corrupted with randomly masked spans, as described in Lewis et al. 2020. The
detailed We denote the loss function for text infilling as as
$\mathcal{L}_{\rm{TI}}$. Hence, the overall objective $\mathcal{L}$ is
formulated as follows:

$\mathcal{L}=\mathcal{L}_{\rm{QA}}+\mathcal{L}_{\rm{QAG}}+\mathcal{L}_{\rm{SUM}}+\mathcal{L}_{\rm{TI}}.$
(4)

##  4 Experiments

| # Train | # Validation | # Test
---|---|---|---
HotpotCMP | 17,456 | 1,487 | -
2WikiCMP | 51,963 | 3,040 | -
CocoTrip | 20 | 10 | 18
Diffen | 20 | 100 | -

Table 3: Statistics of our downstream datasets.

| Comparative QA | Comparative QG | Comparative Summarization
---|---|---|---
HotpotQACMP | 2WikiQACMP | HotpotQGCMP | 2WikiQGCMP | CocoTrip | Diffen
EM | F1 | EM | F1 | BLEU | RL | BLEU | RL | R2 | RL | R2 | RL
| ChatGPT | 73.45 | 62.68 | 82.95 | 74.33 | 7.36 | 29.16 | 10.89 | 33.02 | 6.80 | 18.59 | 9.29 | 21.69
Full-data | BART | 69.27 | 75.70 | 91.87 | 92.43 | 16.29 | 43.41 | 35.28 | 61.94 | - | - | - | -
\+ CMP | 69.26 | 75.43 | 91.81 | 92.30 | 17.18 | 43.66 | 35.82 | 62.13 | - | - | - | -
T5 | 73.16 | 79.20 | 87.40 | 89.67 | 17.57 | 44.70 | 36.02 | 62.70 | - | - | - | -
\+ CMP | 72.69 | 78.83 | 88.75 | 91.08 | 17.26 | 44.65 | 36.12 | 63.18 | - | - | - | -
Few-shot | BART | 33.82 | 39.70 | 37.65 | 39.67 | 11.38 | 39.04 | 30.02 | 57.14 | 23.63 | 44.99 | 10.04 | 24.39
\+ CMP | 44.31 | 52.15 | 57.58 | 58.49 | 12.75 | 39.33 | 29.34 | 55.61 | 27.60 | 47.90 | 12.11 | 26.69
T5 | 48.89 | 54.71 | 43.85 | 45.63 | 6.48 | 30.95 | 6.71 | 28.44 | 29.18 | 45.57 | 9.21 | 24.03
\+ CMP | 50.50 | 58.29 | 56.51 | 58.33 | 8.18 | 33.88 | 12.12 | 37.95 | 30.48 | 49.19 | 8.12 | 23.04
Zero-shot | BART | 0.00 | 11.93 | 0.00 | 19.3 | 1.70 | 18.53 | 3.45 | 20.34 | 4.09 | 18.32 | 6.32 | 17.90
\+ CMP | 31.47 | 39.04 | 40.55 | 42.47 | 6.86 | 29.13 | 9.21 | 32.23 | 6.11 | 24.43 | 8.02 | 20.22
T5 | 20.44 | 28.87 | 20.88 | 26.92 | 1.21 | 18.7 | 2.38 | 18.53 | 8.94 | 25.28 | 5.61 | 17.65
\+ CMP | 44.25 | 52.62 | 54.34 | 56.30 | 7.24 | 28.99 | 5.83 | 32.31 | 8.63 | 28.19 | 5.72 | 18.43

Table 4: Main results. Our pre-trained models denoted by +CMP, bring
_significant_ performance gain to BART and T5 in zero-shot (e.g., relatively
+$82\%$ and +$220\%$ of F1 on HotpotQACMP) and few-shot (e.g., relatively
+$29\%$ and +$52\%$ of F1 on 2WikiQACMP) settings across all tasks. In full-
data settings that assume a huge number of labeled examples are available, our
approach makes smaller improvements on the two models.

###  4.1 Datasets and Evaluation Metrics

To evaluate our proposed method, we consider downstream tasks involving
comparative reasoning, including comparative question answering (QA),
comparative question generation (QG) and comparative summarization. In this
section, we introduce the downstream datasets and evaluation metrics. The
statistics of the datasets are shown in Table 3.

####  4.1.1 Comparative question answering

Comparative question answering (QA) requires to compare two or more entities
on their shared properties. Since our focus on comparison over documents
instead of knowledge retrieval, we do not include distractor passages but
directly use the gold evidence passages as the context for question answering.
For evaluation, we calculate the exact match (EM) score between the predicted
answer and the ground-truth answer, after necessary normalization Chen et al.
(2017). Besides, unigram F-1 scores are also calculated as a complementary
metric to compare the similarity beteween the predicted answer and the ground-
truth.

##### HotpotQACMP and 2WikiQACMP

HotpotQA Yang et al. (2018) and 2WikiMultihopQA (2Wiki) Ho et al. (2020) are
factual question answering datasets collected from English Wikipedia. These
datasets require multi-hop reasoning on different entities before reaching the
correct answer. To focus on comparative questions, we obtain the subset of
comparison questions based on their question type annotations and denote them
as HotpotQACMP and 2WikiQACMP, repspectively.

| Comparative QA | Comparative QG | Comparative Summarization
---|---|---|---
Hotpot | 2Wiki | Hotpot | 2Wiki | CocoTrip | Diffen
EM | F1 | EM | F1 | BLEU | RL | BLEU | RL | R2 | RL | R2 | RL
Few-shot | BART | 33.82 | 39.70 | 37.65 | 39.67 | 11.38 | 39.04 | 30.02 | 57.14 | 23.63 | 44.99 | 10.04 | 24.39
\+ CMP | 44.31 | 52.15 | 57.58 | 58.49 | 12.75 | 39.33 | 29.34 | 55.61 | 27.60 | 47.90 | 12.11 | 26.69
\+ CMP_QA | 45.25 | 52.46 | 55.96 | 57.22 | 11.76 | 39.51 | 25.73 | 52.43 | 24.91 | 45.36 | 12.36 | 25.92
\+ CMP_QAG | 32.21 | 38.17 | 45.13 | 46.63 | 12.57 | 39.29 | 26.89 | 52.74 | 27.22 | 42.55 | 12.28 | 26.85
\+ CMP_SUM | 32.41 | 37.81 | 32.34 | 34.33 | 12.50 | 39.40 | 33.04 | 59.29 | 30.54 | 47.84 | 12.10 | 26.69
Zero-shot | BART | 0.00 | 11.93 | 0.00 | 19.3 | 1.70 | 18.53 | 3.45 | 20.34 | 4.09 | 18.32 | 6.32 | 17.90
\+ CMP | 31.47 | 39.04 | 40.55 | 42.47 | 6.86 | 29.13 | 9.21 | 32.23 | 6.11 | 24.43 | 8.02 | 20.22
\+ CMP_QA | 34.50 | 42.41 | 43.85 | 45.83 | 1.57 | 19.39 | 3.31 | 20.27 | 0.00 | 5.14 | 1.61 | 6.72
\+ CMP_QAG | 0.00 | 13.79 | 0.00 | 20.85 | 6.16 | 27.89 | 8.00 | 29.44 | 1.39 | 16.62 | 6.13 | 18.22
\+ CMP_SUM | 0.00 | 16.56 | 0.00 | 20.02 | 5.33 | 32.78 | 6.76 | 32.78 | 5.53 | 25.79 | 8.34 | 20.00

Table 5: Few-shot and Zero-shot results of models with multi-task pre-training
(denoted by +CMP) vs. single-task pre-training (denote by +CMP_QA, +CMP_QAG,
and +CMP_SUM). On each task, the (multi-task) unified model shows performance
gain over BART, which is comparable to task-specific pre-trained models. In
the meanwhile, it also improve on other tasks, showing the effectiveness of
our unified multi-task pre-training.

####  4.1.2 Comparative question generation

Comparative question generation (QG) aims at generating questions that draw
comparisons between the shared properties of two entities, given their textual
descriptions. The task poses the challenge of identifying and inquiring about
properties that humans would deem interesting and worthy of comparison. For
comparative QG, we adopt well-established evaluation metrics, including BLEU
Papineni et al. (2002) and ROUGE Lin (2004).

For comparative QG, we perform answer-unaware QG and use datasets converted
from comparative QA, denoted by HotpotQGCMP and 2WikiQGCMP respectively.

####  4.1.3 Comparative summarization

Comparative summarization aims at generating summaries that highlight the
similarities or differences between two entities given their descriptions.
Following the convention in text summarization Zhang et al. (2020) , we
evaluate the generated summaries with ROUGE scores.

##### CocoTrip

For the CocoTrip dataset Iso et al. (2022), we employ the common opinion
summarization setting. The task involves summarizing the shared opinions from
two sets of reviews about two hotels. We concatenate the reviews as input.

##### Diffen

To address the absence of available datasets for the comparative summarization
of two entities, we have taken the initiative to curate a unique dataset. The
dataset is collected from Diffen.com, a website recognized for offering high-
quality, human-authored comparisons between different people or objects to
help people make informed decisions. Comparison articles on Diffen.com
typically include a brief introduction summarizing the similarities and
differences. We manually collect these introductory paragraphs as comparative
summaries. To gather input sources, we obtain Wikipedia articles as entity
descriptions. The task aims at generating a comparative summary based on the
given text descriptions of two entities. The input sequence consists of
concatenated entity descriptions, with each description truncated to the first
512 tokens due to the text length restriction.

###  4.2 Experimental Setup

As a pilot study on pre-training for comparative reasoning, we adopt the pre-
trained BART Lewis et al. (2020) and T5 Raffel et al. (2020) as baselines.
Models that are further trained on our comparative objectives are denoted as
BART+CMP and T5+CMP, respectively.

For each downstream dataset, we implemented three distinct settings - full-
data fine-tuning, few-shot learning, and zero-shot learning - to experiment
with both the baseline and our proposed method. In the context of the few-shot
learning scenario, we randomly selected 100 instances from the original
training set. However, given the limited number of training instances
available in CocoTrip and Diffen (specifically, only 20 instances), we merge
the full-data and few-shot settings for these particular datasets. For
Cocotrip, where the test set is available, we select the best model based on
the validation set, and report the results on the test set. For other dataset,
we report the results on the validation sets.

###  4.3 Experimental Results

####  4.3.1 Effects of comparative pre-training

In the comprehensive evaluation across six datasets on three tasks, we compare
the performance of the proposed method (in the form of “+CMP”) against BART
and T5. Main results are listed in Table 4.

In the full-data setting, both our proposed models, BART+CMP and T5+CMP,
demonstrate performances on par with their baselines across all tasks.
Specifically, on HotpotQA, the BART+CMP model achieved an EM of 69, while BART
achieved an EM of 69.27, and similarly for T5, the metrics are 72.69 and
73.16, respectively. Similar patterns are observed for other metrics and
datasets, emphasizing the competitive performance of our approach in data-
abundant scenarios.

However, in low-resource scenarios, as represented by the few-shot and zero-
shot settings, the superiority of our method over the baselines became clearly
evident. For few-shot learning, our models outperform the baselines on most
datasets. Among three tasks, our models show the most significant improvement
on the comparative QA task, demonstrating the effectiveness of our synthetic
QA pre-training. In zero-shot setting, BART+CMP and T5+CMP consistently
surpass their baselines by a large margin. For instance, on HotpotQG, BART+CMP
improves over BART by a substantial margin in BLEU (6.86 vs. 1.70) and ROUGE-L
(29.13 vs. 18.53). Likewise, T5+CMP surpasses T5, with BLEU and ROUGE-L scores
of 7.24 and 28.99 against 1.21 and 18.7, respectively.

Therefore, these results illustrate that our proposed pre-training method
greatly enhances LM’s performance in low resource scenarios, while retaining
competitive performance in scenarios with abundant training data.

####  4.3.2 Multi-task v.s. single-task pre-training

To further explore the benefits of multi-task pre-training, we compare the
performance of our models pre-trained on single task (i.e., QA, QAG or SUM)
with the unified models pre-trained on all proposed tasks. Results are shown
in Table 5. When the model is pre-trained on a single task, we observed a
significant improvement in performance on the downstream task that closely
resembled the pre-training task. However, the model did not exhibit similar
improvements on other tasks that were unrelated or less similar in nature.
This finding suggests that pre-training on a single task enhances the model’s
ability to transfer knowledge specifically to tasks with similar
characteristics. For the unified model, we observed substantial improvements
in performance across all downstream tasks. The improvement brought by the
multi-task pretrained model on each task is comparable to the gains achieved
through the corresponding task-specific pre-training. This outcome suggests
that multitask pre-training enables the model to learn more generalized
representations and effectively leverage the shared knowledge across different
tasks.

####  4.3.3 Case Study

To intuitively show the comparative reasoning ability of our pre-trained
model, we present an example of comparative summarization in Table 6. Given
documents describing airsoft and paintball, models are expected to generate a
summary comparing the commonalities and differences of these two games.
However, without exhaustive fine-tuning, the generated summary of BART fails
to describe the correct relationship between these two entities. On the
contrary, after pre-trained on various comparative reasoning objectives, our
model generates high quality comparative summaries based on the provided
documents under the few-shot setting. The generated summary includes that both
games are popular shooting sports while also comparing their differences in
their equipment.

$\mathbf{D_{1}}$: Airsoft is a team game in which participants eliminate
opposing players by tagging them out of play with spherical plastic
projectiles shot with mock air weapons called airsoft guns. … (446 words left)
---
$\mathbf{D_{2}}$: Paintball is a competitive team shooting sport in which
players eliminate opponents from play by hitting them with spherical dye-
filled gelatin capsules called paintballs that break upon impact. … (472 words
left)
Gold: Airsoft is a popular combat simulation game where participants are
eliminated when hit by pellets launched from guns that resemble real firearms.
In paintball participants try to hit each other with paintballs launched from
a special paintball marker/gun. While airsoft is cheaper and provides a more
realistic warfare experience, paintball is more popular, more organized and
has larger events.
BART (R-L: 18.66, R-2: 4.39)
Airsoft is a team shooting sport in which participants eliminate opponents by
hitting them with airsoft guns. Airsoft guns are shaped like basketballs or
baseball bats and are equipped with a series of round-shaped projectiles
called paintballs.
BART+CMP (R-L: 19.17, R-2: 8.62)
Airsoft and Paintball are two of the most popular shooting sports of all time.
Airsoft is a shooting sport that involves hitting opponents with airsoft guns,
while Paintball is a more aggressive game that uses a softer, more aggressive,
ball-shaped paintball.

Table 6: A test example of Diffen dataset. BART and BART+CMP refer to the
model predictions after few-shot fine-tuning.

##  5 Conclusion

In this paper, we presented a novel framework for pre-training language models
for comparative reasoning. It obtained quintuples for entity comparison by
combining structured and unstructured data, converted the quintuples into
textual components, and employed them in three novel sequence-to-sequence pre-
training tasks. We demonstrated the effects of the pre-training tasks on six
downstream datasets, especially in limited-resource scenarios. To facilitate
the assessment of models’ capability of entity comparison over texts, we
release the benchmark for future research.

## Limitations

In our pre-training framework, we generate synthetic data for comparative
answer generation pre-training with templates, which can cause some synthetic
questions not fluent. Such noises in the pre-training data might limit the
downstream performance. Similarly, the language of the synthetic summaries
generated by a trained data-to-text model are rigid and lack of diversity and
flexibility. Future work can adopt more advanced approaches to convert
quintuples into more fluent and diverse texts for pre-training.

## Acknowledgements

This work was supported by NSF IIS-2119531, IIS-2137396, IIS-2142827,
CCF-1901059, and ONR N00014-22-1-2507. Wenhao Yu is also supported by
Bloomberg Data Science Ph.D Fellowship.

## References

  * Beloucif et al. (2022) Meriem Beloucif, Seid Muhie Yimam, Steffen Stahlhacke, and Chris Biemann. 2022.  Elvis vs. M. Jackson: Who has more albums? classification and identification of elements in comparative questions.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 3771–3779, Marseille, France. European Language Resources Association.
  * Bista et al. (2019) Umanga Bista, Alexander Mathews, Minjeong Shin, Aditya Krishna Menon, and Lexing Xie. 2019.  Comparative document summarisation via classification.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 20–28.
  * Bondarenko et al. (2022) Alexander Bondarenko, Yamen Ajjour, Valentin Dittmar, Niklas Homann, Pavel Braslavski, and Matthias Hagen. 2022.  Towards understanding and answering comparative questions.  In _Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining_ , pages 66–74.
  * Chekalina et al. (2021) Viktoriia Chekalina, Alexander Bondarenko, Chris Biemann, Meriem Beloucif, Varvara Logacheva, and Alexander Panchenko. 2021.  Which is better for deep learning: Python or MATLAB? answering comparative questions in natural language.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations_ , pages 302–311, Online. Association for Computational Linguistics.
  * Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017.  Reading wikipedia to answer open-domain questions.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017_ , pages 1870–1879. Association for Computational Linguistics.
  * Ho et al. (2020) Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020.  Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 6609–6625, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Hu et al. (2021) Ziniu Hu, Yizhou Sun, and Kai-Wei Chang. 2021.  Relation-guided pre-training for open-domain question answering.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 3431–3448, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Hu et al. (2022) Ziniu Hu, Yichong Xu, Wenhao Yu, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Kai-Wei Chang, and Yizhou Sun. 2022.  Empowering language models with knowledge graph reasoning for open-domain question answering.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9562–9581, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Iso et al. (2022) Hayate Iso, Xiaolan Wang, Stefanos Angelidis, and Yoshihiko Suhara. 2022.  Comparative opinion summarization via collaborative decoding.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 3307–3324, Dublin, Ireland. Association for Computational Linguistics.
  * Jindal and Liu (2006) Nitin Jindal and Bing Liu. 2006.  Identifying comparative sentences in text documents.  In _Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval_ , pages 244–251.
  * Lerman and McDonald (2009) Kevin Lerman and Ryan McDonald. 2009.  Contrastive summarization: An experiment with consumer reviews.  In _Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers_ , pages 113–116, Boulder, Colorado. Association for Computational Linguistics.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Li et al. (2011) Shasha Li, Chin-Yew Lin, Young-In Song, and Zhoujun Li. 2011.  Comparable entity mining from comparative questions.  _IEEE transactions on knowledge and data engineering_ , 25(7):1498–1509.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu et al. (2020) Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. 2020.  K-bert: Enabling language representation with knowledge graph.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 2901–2908.
  * Liu et al. (2021) Y Liu, Y Wan, L He, H Peng, and PS Yu. 2021.  Kg-bart: Knowledge graph-augmented bart for generative commonsense reasoning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_.
  * Nan et al. (2021) Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, Yangxiaokang Liu, Nadia Irwanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Mutethia Mutuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Chern Tan, Xi Victoria Lin, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. 2021\.  DART: Open-domain structured data record to text generation.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 432–447, Online. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Qin et al. (2021) Yujia Qin, Yankai Lin, Ryuichi Takanobu, Zhiyuan Liu, Peng Li, Heng Ji, Minlie Huang, Maosong Sun, and Jie Zhou. 2021.  ERICA: Improving entity and relation understanding for pre-trained language models via contrastive learning.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3350–3363, Online. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551.
  * Ribeiro et al. (2021) Leonardo F. R. Ribeiro, Martin Schmitt, Hinrich Schütze, and Iryna Gurevych. 2021.  Investigating pretrained language models for graph-to-text generation.  In _Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI_ , pages 211–227, Online. Association for Computational Linguistics.
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022.  Multitask prompted training enables zero-shot task generalization.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.
  * Wang et al. (2020) Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Guihong Cao, Daxin Jiang, Ming Zhou, et al. 2020.  K-adapter: Infusing knowledge into pre-trained models with adapters.  _arXiv preprint arXiv:2002.01808_.
  * Wang et al. (2021) Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021.  KEPLER: A unified model for knowledge embedding and pre-trained language representation.  _Transactions of the Association for Computational Linguistics_ , 9:176–194.
  * Xiong et al. (2020) Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov. 2020.  Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model.  In _International Conference on Learning Representations_.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018.  HotpotQA: A dataset for diverse, explainable multi-hop question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics.
  * Yu et al. (2022) Donghan Yu, Chenguang Zhu, Yiming Yang, and Michael Zeng. 2022.  Jaket: Joint pre-training of knowledge graph and language understanding.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 11630–11638.
  * Zhang et al. (2020) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.  In _International Conference on Machine Learning_ , pages 11328–11339. PMLR.
  * Zhang et al. (2019) Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019.  ERNIE: Enhanced language representation with informative entities.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1441–1451, Florence, Italy. Association for Computational Linguistics.
  * Zhang et al. (2021) Zhihan Zhang, Xiubo Geng, Tao Qin, Yunfang Wu, and Daxin Jiang. 2021.  Knowledge-aware procedural text understanding with multi-stage training.  In _WWW ’21: The Web Conference 2021_.
  * Zhang et al. (2023) Zhihan Zhang, Wenhao Yu, Mengxia Yu, Zhichun Guo, and Meng Jiang. 2023.  A survey of multi-task learning in natural language processing: Regarding task relatedness and training methods.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023_.
  * Zhang et al. (2022) Zhihan Zhang, Wenhao Yu, Chenguang Zhu, and Meng Jiang. 2022.  A unified encoder-decoder framework with entity memory.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP_.


# Rethinking binary hyperparameters for deep transfer learning for image
classification

Jo Plested
Department of Computer Science
University of New South Wales
Northcott Dr, Campbell ACT 2612
<EMAIL_ADDRESS>
Xuyang Shen
School of Computer Science
Australian National University
<EMAIL_ADDRESS>
Tom Gedeon
School of Computer Science
Australian National University
<EMAIL_ADDRESS>

###### Abstract

The current standard for a variety of computer vision tasks using smaller
numbers of labelled training examples is to fine-tune from weights pre-trained
on a large image classification dataset such as ImageNet. The application of
transfer learning and transfer learning methods tends to be rigidly binary. A
model is either pre-trained or not pre-trained. Pre-training a model either
increases performance or decreases it, the latter being defined as negative
transfer. Application of L2-SP regularisation that decays the weights towards
their pre-trained values is either applied or all weights are decayed towards
0. This paper re-examines these assumptions. Our recommendations are based on
extensive empirical evaluation that demonstrate the application of a non-
binary approach to achieve optimal results. (1) Achieving best performance on
each individual dataset requires careful adjustment of various transfer
learning hyperparameters not usually considered, including number of layers to
transfer, different learning rates for different layers and different
combinations of L2SP and L2 regularization. (2) Best practice can be achieved
using a number of measures of how well the pre-trained weights fit the target
dataset to guide optimal hyperparameters. We present methods for non-binary
transfer learning including combining L2SP and L2 regularization and
performing non-traditional fine-tuning hyperparameter searches. Finally we
suggest heuristics for determining the optimal transfer learning
hyperparameters. The benefits of using a non-binary approach are supported by
final results that come close to or exceed state of the art performance on a
variety of tasks that have traditionally been more difficult for transfer
learning.

##  1 Introduction

Convolutional neural networks (CNNs) have achieved many successes in image
classification in recent years [17, 9, 19, 24, 25]. It has been consistently
demonstrated that CNNs work best when there is abundant labelled data
available for the task and very deep models can be trained [28, 22, 14].
However, there are many real world scenarios where the large amounts of
training data required to obtain the best performance cannot be met or are
prohibitively expensive. Transfer learning has been shown to improve
performance in a wide variety of computer vision tasks, particularly when the
source and target tasks are closely related and the target task is small [28,
22, 6, 8, 33, 27, 21, 20, 37]. It has become standard practice to pre-train on
Imagenet 1K for many different tasks where the available labeled datasets are
orders of magnitude smaller than Imagenet 1K [21, 20, 37, 24, 19, 27, 25, 26,
9]. Several papers published in recent years have questioned this established
paradigm. They have shown that when the target dataset is very different from
Imagenet 1K and a reasonable amount of data is available, training from
scratch can match or even out perform pre-trained and fine-tuned models [12,
35, 34, 40, 22].

The standard transfer learning strategy is to transfer all layers apart from
the final classification layer, and either use a single initial learning rate
and other hyperparameters for fine-tuning all layers, or freeze some layers.
Given that lower layers in a deep neural network are known to be more general
and higher layers more specialised [39], we argue that the binary way of
approaching transfer learning and fine-tuning is counter intuitive. There is
no intuitive reason to think that all layers should be treated the same when
fine-tuning, or that pre-trained weights from all layers will be applicable to
the new task. If transferring all layers results in negative transfer, could
transferring some number of lower more general layers improve performance? If
using an L2SP weight decay on all transferred layers for regularisation
decreases performance over decaying towards 0, might applying the L2SP
regularisation to some number of lower layers that are more applicable to the
target dataset result in improved performance?

We performed extensive experiments across four different datasets to:

  * •

re-examine the assumptions that transfer learning hyperparameters should be
binary, and

  * •

find the optimal settings for number of layers to transfer, initial learning
rates for different layers, and number of layers to apply L2SP regularisation
vs decaying towards 0.

We developed methods for non-binary transfer learning including combining L2SP
and L2 regularization and performing non-traditional fine-tuning
hyperparameter searches. We show that the optimal settings result in
significantly better performance than binary settings for all datasets except
the most closely related. Finally we suggest heuristics for determining the
optimal transfer learning hyperparameters.

##  2 Related work

The standard transfer learning strategy is to transfer all layers apart from
the final classification layer, then use a search strategy to find the best
single initial learning rate and other hyperparameters. Several studies
include extensive hyperparameter searches over learning rate and weight decay
[15, 22], momentum [18], and L2SP [21]. This commonly used strategy originates
from various works showing that performance on the target task increases as
the number of layers transferred increases [39, 4, 1, 2]. All these works were
completed prior to advances in residual networks [13], and other advances to
improve learning for deep CNN architectures, and searches for optimal
combinations of learning rates and number of layers to transfer were not
performed. Additionally, in two of the works layers that were not transferred
were discarded completely rather than reinitialising and training them from
scratch. This resulted in smaller models with less layers transferred and a
strong bias towards better results when more layers were transferred [1, 2].
Further studies have shown the combination of a lower learning rate and fewer
layers transferred may be optimal [30]. However, again modern residual
networks were not used and only very similar source and target datasets were
selected.

It has been shown that the similarity between source and target datasets has a
strong impact on performance for transfer learning:

  1. 1.

More closely related datasets can be better than more source data for pre-
training [28, 22, 6, 8, 33].

  2. 2.

A multi-step pre-training process where the interim dataset is smaller and
more closely related to the target dataset can outperform a single step pre-
training process when originating from a very different, large source dataset
[27].

  3. 3.

Self-supervised pre-training on a closely related source dataset can be better
than supervised training on a less closely related dataset [42].

  4. 4.

L2SP regularization, where the weights are decayed towards their pre-trained
values rather than 0 during fine-tuning, improves performance when the source
and target dataset are closely related, but hinders it when they are less
related [21, 20, 37]

  5. 5.

Momentum should be lower for more closely related source and target datasets
[18].

These five factors demonstrate the importance of the relationship between the
source and target dataset in transfer learning.

##  3 Methodology

###  3.1 Datasets

We performed evaluations on four different small target datasets, each being
less than 10,000 training images. We chose one that is very similar to
Imagenet 1K that transfer learning and sub methods have traditionally
performed best on. This is used as a baseline for comparison to show that the
default methods that perform badly on the other datasets chosen do perform
well on this one. For each of the other target datasets it has been shown that
traditional transfer learning strategies:

  * •

do not perform well on them and/or

  * •

they are very different to the source dataset used for pre-training.

We used the standard available train, validation and test splits for the three
datasets for which they were available. For Caltech256-30 we used the first 30
items from each class for the train split, the next 20 for the validation
split and the remainder for the test split.

####  3.1.1 Source dataset

We used Imagenet 1K as the source dataset as it is most commonly used source
dataset and therefore the most suitable to demonstrate our approach.

####  3.1.2 Target datasets

These are the final datasets that we transferred the models to and measured
performance on. We used a standard 299 $\times$ 299 image size for all
datasets.

##### Most similar to Imagenet 1K

We chose Caltech 256-30 (Caltech) [10] as the most similar to Imagenet. It
contains 256 different general subordinate and superordinate classes. As our
focus is on small target datasets, we chose the smallest commonly used split
with 30 examples per class.

##### Fine-grained

Fine-grained object classification datasets contain subordinate classes from
one particular superordinate class. We chose two where standard transfer
learning has performed badly [11, 15].

  * •

Stanford Cars (Cars): Contains 196 different makes and models of cars with
8,144 training examples and 8,041 test examples [16].

  * •

FGVC Aircraft (Aircraft): Contains 100 different makes and models of aircraft
with 6,667 training examples and 3,333 test examples [23].

##### Very different to Imagenet 1K

We evaluated performance on another dataset that is intuitively very different
to Imagenet 1K as it consists of images depicting adjectives instead of nouns.

  * •

Describable Textures (DTD) [5]: consists of 3,760 training examples of texture
images jointly annotated with 47 attributes. The texture images are collected
“in the wild” by searching for texture adjectives on the web.

###  3.2 Model

Inception v4 [36] was selected for evaluation as it has been shown to have
state of the art performance for transferring from Imagenet 1K to a broad
range of target tasks [15]. Our code is adapted from [38] and we used the
publicly available pre-trained Inception v4 model. Our code is available at
https://anonymous.4open.science/r/Non-binary-deep-transfer-learning-for-image-
classification-872D. We did not experiment with pre-training settings, for
example removing regularization settings as suggested by [15], as it was
beyond the scope of this work and the capacity of our compute resources.

###  3.3 Evaluation metric

We used top-1 accuracy for all results for easiest comparison with existing
results on the chosen datasets. Graphs showing ablation studies with different
hyperparameters show results on one run. Final reported results are averages
and standard deviations over four runs. For all experiments we used single
crop and for our final comparison to state of the art for each dataset we used
an ensemble of all four runs and 10-crop.

###  3.4 Compute resources

The experiments were run on compute nodes containing four Nvidia V100 GPUs and
two 24-core Intel Xeon Scalable ’Cascade Lake’ processors.

Each individual experiment in the ablation studies ran on one GPU and 12 cores
from one processor. Running time varied between approximately six hours for
DTD to one day for Caltech.

Each final experiment was run on a full compute node with four GPUs and two
24-core processors. Runnng time varied between approximately one day for DTD
and Aircraft to two days for Caltech.

##  4 Results

###  4.1 Assessing the suitability of the fixed features

We first examined performance using the pre-trained Inception v4 model as a
fixed feature extractor for comparison and insight into the suitability of the
pre-trained features. We trained a simple fully connected neural network
classification layer for each dataset and compared it to training the full
model from random initialization. The comparisons for all datasets are shown
in Table 1.

The class normalized Fisher score on target datasets using the pre-trained but
not fine-tuned weights, was used as a measure of how well the pre-trained
weights separate the classes in the target dataset [7]. A larger normalized
Fisher score shows more separation between classes in the feature space.

$F\left(\mathbf{W}\right)=Tr\left\\{s_{w}^{-1}s_{B}\right\\}/N_{c}$

where $s_{w}^{-1}$ is the inverse of the within class covariance, $s_{B}$ is
the between class covariance and $N_{c}$ is the number of classes for the
target dataset being measured. See [7] for further details.

We also calculated domain similarity between the fixed features of the source
and target domain using the earth mover distance (EMD) [32, 29] and applying
the procedure defined in [6].

The domain similarity calculated using the EMD has been shown to correlate
well with the improvement in performance on the target task from pre-training
with a particular source dataset [6].

Table 1: Domain measures |  trained from random initialization |  fixed features classification |  EMD domain similarity |  normalised Fisher score |  state of the art
---|---|---|---|---|---
Caltech |  67.2 |  83.4 |  0.568 |  1.35 |  84.9 [20]
Cars |  92.7 |  64.2 |  0.536 |  1.18 |  96.0 (@448) [31]
Aircraft |  88.8 |  59.9 |  0.557 |  0.83 |  93.9 (@448) [41]
DTD |  66.8 |  74.6 |  0.540 |  3.47 |  78.9 [3]

###  4.2 Default settings

For the first experiment we transferred all but the final classification layer
and trained all layers at the same learning rate as per standard transfer
learning practice. We performed a search using the validation set to find the
optimal single fine-tuning learning rate for each dataset. The results in
Figure 1 show the accuracy for each learning rate for each dataset.

The optimal single learning rate for Caltech, the most similar dataset to
ImageNet, is an order of magnitude lower than the optimal learning rate for
the fine-grained classification tasks Stanford Cars and Aircraft. This shows
that the optimal weights for Caltech are much closer to the pre-trained
values. The surprising result was that the optimal learning rate for DTD was
very similar to Caltech and also an order of magnitude lower than Stanford
Cars and Aircraft. Final results on the test set for each dataset are shown in
Table 2.

Figure 1: Optimal learning rates for each dataset. Left to right Caltech, Cars, Aircraft, DTD Table 2: Final results default settings | Caltech | Stanford Cars | Aircraft | DTD
---|---|---|---|---
Learning rate | 0.0025 | 0.025 | 0.025 | 0.002
Accuracy | 83.69$\pm$0.0784 | 94.59$\pm$0.110 | 93.78$\pm$0.137 | 77.30 $\pm$0.726

###  4.3 Optimal learning rate decreases as more layers are reinitialized and
transferring all possible layers often reduces performance on the target task

To the best of our knowledge the combination of learning rate and number of
layers reinitialized when performing transfer learning has not been examined
in modern residual models. To examine the relationship between learning rate
and number of layers reinitialized, we searched for the optimal learning rate
for each of 1-3 blocks of layers reinitialized across each of the four
datasets. One layer is the default with the final classification layer only
being reinitialized. Two and three involve reinitializing an additional one or
two Inception C blocks of layers respectively as well as the final layer. For
consistency we refer to both the final classification layer and the Inception
C blocks as layers. Figure 2 shows the performance of fine-tuning with various
combinations of learning rate and layers reinitialized. The optimal learning
rate when reinitializing more than one layer is always lower than when
reinitializing only the final classification layer. Also the optimal number of
layers to reinitialize is more than one for all datasets except Cars. Table 3
shows the final results for the optimal learning rate and accuracy for each
number of layers reinitialized.

##### Reinitializing more layers has a more significant effect when the
optimal learning rate for reinitializing just one layer is higher

Caltech and DTD have an order of magnitude lower optimal learning rates than
Cars and Aircraft, but reinitializing more layers for the former results in a
significant increase in accuracy whereas there is none for the latter. This
result initially seems counter intuitive as Cars and Aircraft are less similar
to Imagenet than Caltech and DTD. However, a lower learning rate tempers the
ability of the upper layers of the model to specialise to the new domain and
even very closely related source and target domains have different final
classes and thus optimal feature spaces. The combination of a lower learning
rate and reinitializing more layers likely allows the models to keep the
closely related lower and middle layers and create new specialist upper
layers.

Figure 2: Number of layers reinitialized versus learning rate. Left to right Caltech, Cars, Aircraft, DTD  Table 3: Final optimal learning rate for each number of layers reinitialized |  Caltech |  Cars |  Aircraft |  DTD
---|---|---|---|---
1 layer lr |  0.0025 |  0.025 |  0.025 |  0.002
Accuracy |  83.69$\pm$0.078 |  94.59$\pm$0.110 |  93.78$\pm$0.137 |  77.30 $\pm$0.726
2 layers lr |  0.015 |  0.015 |  0.02 |  0.001
Accuracy |  84.31$\pm$0.114 |  94.59$\pm$ 0.205 |  93.56$\pm$0.942 |  78.92 $\pm$0.191
3 layers lr |  0.015 |  0.015 |  0.02 |  0.0015
Accuracy |  83.57$\pm$0.104 |  94.46$\pm$0.348 |  92.96$\pm$1.588 |  78.90$\pm$0.243

###  4.4 Lower learning rates for lower layers works better when optimal
learning rate is higher

Conventionally learning rates for different layers are set so that either all
layers are fine-tuned with the same learning rate or some are frozen. The
thinking is that as lower layers are more general and higher layers more task
specific [39] the lower layers do not need to be fine-tuned. Recent work has
shown that fine-tuning tends to work better in most cases [30]. However,
setting different learning rates for different layers is not generally
considered. We examined the effects of applying lower learning rates to lower
layers that are likely to generalise better to the target task. Figure 4 shows
that for the Stanford Cars and Aircraft where the optimal initial learning
rate is higher, setting lower learning rates for lower layers significantly
improves performance whereas for Caltech and DTD it does not.

Figure 3: Lower learning rates for lower layers. Left to right Caltech, Cars, Aircraft, DTD. Legend shows number of layers reinitialized, high learning rate, low learning rate. X axis is number layers trained starting with the low learning rate Table 4: Optimal learning rates for each number of layers reinitialized with different learning rates for lower layers. Learning rates are in the format (high layers learning rate, low layers learning rate, number of low layers)  |  Caltech |  Cars |  Aircraft |  DTD
---|---|---|---|---
Default accuracy |  83.69$\pm$0.0784 |  94.59$\pm$0.110 |  93.78$\pm$0.137 |  77.30$\pm$0.726
1 layer lrs |  0.002 0.001 12 |  0.025 0.01 8 |  0.025 0.015 10 |  0.0025 0.0015 10
Accuracy |  83.95$\pm$0.196 |  94.86$\pm$0.0460 |  94.32 $\pm$0.144 |  77.753$\pm$0.456
2 layers lrs |  0.002 0.001 12 |  0.025 0.01 8 |  0.025 0.015 10 |  0.0015 0.001 14
Accuracy |  84.15$\pm$0.553 |  94.78 $\pm$0.132 |  94.17 $\pm$0.311 |  78.59$\pm$0.127
3 layers lrs |  0.002 0.001 12 |  0.025 0.01 10 |  0.01 0.025 0.01 8 |  0.0015 0.001 12
Accuracy |  83.46$\pm$0.143 |  94.83 $\pm$0.0832. |  94.50 $\pm$0.192 |  78.84 $\pm$0.464

###  4.5 L2SP

The L2SP regularizer decays pre-trained weights towards their pre-trained
values rather than towards zero during fine-tuning. In the standard transfer
learning paradigm with only the final classification reinitialized, when the
L2SP regularizer is applied the final classification layer only is decayed
towards 0.

The values $\alpha$ and $\beta$ are tuneable hyperparameters to control the
amount of regularization applied to the pre-trained and randomly initialized
layers respectively.

The original experiments showing the effectiveness of the L2SP regularizer
[21] were done on target datasets that are extremely similar to the source
datasets used. They showed that a high level of regularization decaying
towards the pre-trained weights is beneficial on these datasets. It has since
been shown that the L2SP regularizer can result in minimal improvement or even
worse performance when the source and target datasets are less related [18,
37].

Our results shown in Figure 4 align with the original paper [21] for the
dataset Caltech showing that standard L2SP regularization does result in an
improvement in performance over L2 regularization. Our results also align with
[18, 37] in showing that for the datasets we have chosen to be different from
Imagenet, and known to be more difficult to transfer to, L2SP regularization
performs worse than L2 regularization. In general the lower the setting for
alpha (the L2SP regularization hyperparameter) the better the performance.

Figure 4: L2SP with default settings. Left to right Caltech, Cars, Aircraft,
DTD

We relaxed the binary assumption that L2SP regularization must be applied to
all pre-trained layers. We used L2SP regularization for lower layers that we
expected to be more similar to the source dataset and L2 regularization for
upper layers to allow them to specialise to the target dataset. We searched
for the optimal combinations of L2SP and L2 weight regularization along with
the $\alpha$ and $\beta$ hyperparameters for each dataset. We show the best
settings for number of layers and amount of L2SP regularization in Table 5 and
graph the optimal settings compared to the best default settings in Figure 5.

We make the following observations:

  1. 1.

A combination of L2SP and L2 regularization is optimal for most settings of
the L2SP regularization hyperparameter ($\alpha$) for all datasets except
Caltech.

  2. 2.

When more layers are trained with L2 rather than L2SP regularization the
optimal L2 regularization hyperparameter ($\beta$) is lower as the squared sum
of the weights in these layers will be larger for the same model.

Further experiments were performed to search for the combination of optimal
L2SP vs L2 regularization settings with optimal number of layers transferred
and different learning rates for different layers. These results are also
shown in Table 5.

Table 5: Best default and optimal L2SP settings and results |  Caltech |  Cars |  Aircraft |  DTD
---|---|---|---|---
L2 |  83.694 |  94.59 |  93.78 |  77.30
Default L2SP 1 new layer $\alpha$, $\beta$ |  0.01 0.01 |  0.0001 0.01 |  0.0001 0.01 |  0.0001 0.01
result |  84.52 |  94.42 |  93.29 |  78.32
Optimal L2SP 1 new layer $\alpha$, $\beta$ |  0.01 0.01 L2SP layers all |  0.01 0.001 L2SP layers 10 |  0.0001 0.001 L2SP layers 10 |  0.001 0.001 L2SP layers 10
result |  84.52 |  94.57 |  93.86 |  78.32
Optimal L2SP overall $\alpha$, $\beta$ |  0.01 0.01 L2SP layers all |  0.01 0.001 L2SP layers 10 |  0.0001 0.001 L2SP layers 10 |  0.001 0.001 1.0 L2SP layers 14
result |  84.52 |  94.65 |  94.22 |  78.90

Figure 5: Best default L2SP optimal L2SP settings. Left to right Caltech,
Cars, Aircraft, DTD

###  4.6 Final optimal settings and how to predict them

The final results and optimal settings are shown in Table 6 and Figure 6.
Table 7 shows a clear distinction between target datasets that are more
similar to the source dataset and for which the pre-trained weights are better
able to separate the classes in feature space and those that are less similar
with pre-trained weights that fit the target task poorly. The best measure for
determining whether the pre-trained weights are well suited is a comparison of
the performance achieved with fixed pre-trained features and that achieved
through training from random initialization. As this method is computationally
intensive a reasonable alternative may be the normalized Fisher Ratio, but as
the differences are not as pronounced in all cases it should be further
investigated on more datasets to see how reliable it is as a heuristic. The
EMD measure of domain similarity is a poor predictor of the suitability of the
pre-trained weights.

Targets datasets for which the pre-trained weights are well suited need:

  * •

a much lower learning rate for fine-tuning,

  * •

more than one layer to be reinitialized from random weights, and

  * •

some or all layers trained with L2SP regularization.

Target datasets where the pre-trained weights are not well suited need:

  * •

a much higher learning rate for fine-tuning with a lower learning rate for
lower layers,

  * •

only the final classification layer reinitialized, and

  * •

L2 rather than L2SP regularization.

Using the above best practice, non-binary transfer learning procedures we
achieved state of the art or close to, on three out of the four datasets. We
used publicly available pre-trained weights and no additional methods for
either pre-training or fine-tuning.

Table 6: Optimal settings versus best default settings |  Caltech |  Cars |  Aircraft |  DTD
---|---|---|---|---
Default settings |  0.0025 |  0.025 |  0.025 |  0.002
Default result |  83.69 |  94.59 |  93.78 |  77.30
Default L2SP |  84.52 |  94.42 |  93.29 |  78.32
Optimal settings |  1 new layer lr 0.0025 L2SP 0.01 0.01 |  1 new layer high lr 0.025 low lr 0.01 low layers 8 no L2SP |  3 new layers FC lr 0.01 high lr 0.025 low lr 0.01 low layers 8 no L2SP |  new layers 2 lr 0.002 L2SP 0.001 0.001 L2SP layers 14
Optimal result |  84.52 |  94.86 |  94.50 |  78.90
Ensemble |  85.94 |  95.35 |  95.11 |  79.79
State of the art |  84.9 [20] |  96.0 (@448) [31] |  93.9 (@448) [41] |  78.9 [3]

Figure 6: Optimal settings versus best default settings. Left to right Caltech, Cars, Aircraft, DTD Table 7: Predicting optimal settings based on pre-trained features | Caltech | DTD | Cars | Aircraft
---|---|---|---|---
Fisher ratio | 1.35 | 3.47 | 1.18 | 0.83
EMD similarity | 0.568 | 0.540 | 0.536 | 0.557
Random initialisation minus fixed features | -16.2 | -7.8 | 28.5 | 28.9
Optimal learning rate | Low | Low | High | High
L2SP | Yes | Yes | No | No
More layers reinitiailized | Yes | Yes | No | No
Low layers at low learning rate | No | No | Yes | Yes

##  5 Discussion

Traditional binary assumptions about transfer learning hyperparameters should
be discarded in favour of a tailored approach to each individual dataset.
These assumptions include transferring all possible layers or none, training
all layers at the same learning rate or freezing some, and using L2SP
regularization or L2 regularization for all layers. Our work demonstrates that
optimal transfer learning non-binary hyperparameters are dataset dependent and
strongly influenced by how well the pre-trained weights fit the target task.
For a particular dataset, optimal non-binary transfer learning hyperparameters
can be determined based on the difference between model performance when fixed
features are used and when the full model is trained from random
initialization as shown in Table 7. We recommend using the settings shown on
the left in this table for target datasets where the difference is negative
and settings shown on the right for positive differences. Target datasets for
which the pre-trained weights are well suited and target datasets for which
they are not result in large differences in this value. These differences
should still be pronounced even if suboptimal learning hyperparameters are
used for this initial test due to limited resources for hyperparameter search.
This heuristic for determining optimal hyperparameters should be useful in
most transfer learning for image classification cases. The normalized Fisher
Ratio may be useful in some cases, however, care should be taken because the
differences are not as pronounced. The EMD domain similarity measure should
not be used to determine transfer learning hyperparameters.

##  6 Broader Impact

This research is focused on improving transfer learning for small target image
classification datasets. The positive impacts are easy to observe in improved
performance of models utilising transfer learning in a wide range of real
world applications. For example, faster and more accurate medical diagnostics
where there is limited data available.

However, improvements to image classification models can have positive or
negative impacts. More accurate models could reduce errors but result in
overconfidence and more reliance on results in areas where the full
limitations are not well understood and taken into account. Improved image
classification models could also be used in areas with potential negative
impacts, for example:

  * •

use of intrusive facial recognition systems

  * •

discrimination based on subtle items such as religious symbols

##  7 Limitations and future work

The main aim of this work is to highlight the need for a non-binary approach
to achieving optimal performance in transfer learning applied to image
classification on small target datasets. The relationship between the source
and the target dataset and how well the pre-trained weights fit the target
task has been shown to be important to transfer learning performance numerous
times [28, 22, 6, 8, 33, 21, 20, 37, 27, 42, 18]. However, the guidance as to
how to set transfer learning hyperparameters based on the heuristics presented
in this paper is based only on the experiments on the four datasets outlined
in this paper. Care should be taken if applying this guidance to new datasets,
particularly outside of image classification.

## References

  * [1] Pulkit Agrawal, Ross Girshick, and Jitendra Malik.  Analyzing the performance of multilayer neural networks for object recognition.  In European conference on computer vision, pages 329–344. Springer, 2014.
  * [2] Hossein Azizpour, Ali Sharif Razavian, Josephine Sullivan, Atsuto Maki, and Stefan Carlsson.  Factors of transferability for a generic convnet representation.  IEEE transactions on pattern analysis and machine intelligence, 38(9):1790–1802, 2015.
  * [3] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [4] Brian Chu, Vashisht Madhavan, Oscar Beijbom, Judy Hoffman, and Trevor Darrell.  Best practices for fine-tuning visual classifiers to new domains.  In European conference on computer vision, pages 435–442. Springer, 2016.
  * [5] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi.  Describing textures in the wild.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3606–3613, 2014.
  * [6] Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Serge Belongie.  Large scale fine-grained categorization and domain-specific transfer learning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4109–4118, 2018.
  * [7] K Fukenaga.  Introduction to statistical pattern recognition.  2ed, Academic Press, San Diego, 1990.
  * [8] Weifeng Ge and Yizhou Yu.  Borrowing treasures from the wealthy: Deep transfer learning through selective joint fine-tuning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1086–1095, 2017.
  * [9] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580–587, 2014.
  * [10] Gregory Griffin, Alex Holub, and Pietro Perona.  Caltech256 object category dataset.  California Institute of Technology, 2007.
  * [11] Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris.  Spottune: transfer learning through adaptive fine-tuning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4805–4814, 2019.
  * [12] Kaiming He, Ross Girshick, and Piotr Dollár.  Rethinking imagenet pre-training.  arXiv preprint arXiv:1811.08883, 2018.
  * [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [14] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby.  Big transfer (bit): General visual representation learning.  arXiv preprint arXiv:1912.11370, 2019.
  * [15] Simon Kornblith, Jonathon Shlens, and Quoc V Le.  Do better imagenet models transfer better?  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2661–2671, 2019.
  * [16] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
  * [17] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In Advances in neural information processing systems, pages 1097–1105, 2012.
  * [18] Hao Li, Pratik Chaudhari, Hao Yang, Michael Lam, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto.  Rethinking the hyperparameters for fine-tuning.  arXiv preprint arXiv:2002.11770, 2020.
  * [19] Shan Li and Weihong Deng.  Deep facial expression recognition: A survey.  IEEE Transactions on Affective Computing, 2020.
  * [20] Xingjian Li, Haoyi Xiong, Hanchao Wang, Yuxuan Rao, Liping Liu, and Jun Huan.  Delta: Deep learning transfer using feature map with attention for convolutional networks.  arXiv preprint arXiv:1901.09229, 2019.
  * [21] Xuhong Li, Yves Grandvalet, and Franck Davoine.  Explicit inductive bias for transfer learning with convolutional networks.  arXiv preprint arXiv:1802.01483, 2018.
  * [22] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten.  Exploring the limits of weakly supervised pretraining.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 181–196, 2018.
  * [23] S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi.  Fine-grained visual classification of aircraft.  Technical report, Toyota Technological Institute at Chicago, 2013.
  * [24] Iacopo Masi, Yue Wu, Tal Hassner, and Prem Natarajan.  Deep face recognition: A survey.  In 2018 31st SIBGRAPI conference on graphics, patterns and images (SIBGRAPI), pages 471–478. IEEE, 2018.
  * [25] Maciej A Mazurowski, Mateusz Buda, Ashirbani Saha, and Mustafa R Bashir.  Deep learning in radiology: An overview of the concepts and a survey of the state of the art with focus on mri.  Journal of magnetic resonance imaging, 49(4):939–954, 2019.
  * [26] Romain Mormont, Pierre Geurts, and Raphaël Marée.  Comparison of deep transfer learning strategies for digital pathology.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 2262–2271, 2018.
  * [27] Hong-Wei Ng, Viet Dung Nguyen, Vassilios Vonikakis, and Stefan Winkler.  Deep learning for emotion recognition on small datasets using transfer learning.  In Proceedings of the 2015 ACM on international conference on multimodal interaction, pages 443–449, 2015.
  * [28] Jiquan Ngiam, Daiyi Peng, Vijay Vasudevan, Simon Kornblith, Quoc V Le, and Ruoming Pang.  Domain adaptive transfer learning with specialist models.  arXiv preprint arXiv:1811.07056, 2018.
  * [29] Shmuel Peleg, Michael Werman, and Hillel Rom.  A unified approach to the change of resolution: Space and gray-level.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(7):739–742, 1989.
  * [30] Jo Plested and Tom Gedeon.  An analysis of the interaction between transfer learning protocols in deep neural networks.  In International Conference on Neural Information Processing, pages 312–323. Springer, 2019.
  * [31] Tal Ridnik, Hussam Lawen, Asaf Noy, and Itamar Friedman.  Tresnet: High performance gpu-dedicated architecture.  arXiv preprint arXiv:2003.13630, 2020.
  * [32] Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas.  The earth mover’s distance as a metric for image retrieval.  International journal of computer vision, 40(2):99–121, 2000.
  * [33] Matthia Sabatelli, Mike Kestemont, Walter Daelemans, and Pierre Geurts.  Deep transfer learning for art classification problems.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 0–0, 2018.
  * [34] Zhiqiang Shen, Zhuang Liu, Jianguo Li, Yu-Gang Jiang, Yurong Chen, and Xiangyang Xue.  Dsod: Learning deeply supervised object detectors from scratch.  In Proceedings of the IEEE international conference on computer vision, pages 1919–1927, 2017.
  * [35] Zhiqiang Shen, Honghui Shi, Rogerio Feris, Liangliang Cao, Shuicheng Yan, Ding Liu, Xinchao Wang, Xiangyang Xue, and Thomas S Huang.  Learning object detectors from scratch with gated recurrent feature pyramids.  arXiv preprint arXiv:1712.00886, 1, 2017.
  * [36] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi.  Inception-v4, inception-resnet and the impact of residual connections on learning.  In Thirty-First AAAI Conference on Artificial Intelligence, 2017\.
  * [37] Ruosi Wan, Haoyi Xiong, Xingjian Li, Zhanxing Zhu, and Jun Huan.  Towards making deep transfer learning never hurt.  In 2019 IEEE International Conference on Data Mining (ICDM), pages 578–587. IEEE, 2019.
  * [38] Ross Wightman.  Pytorch image models.  https://github.com/rwightman/pytorch-image-models, 2019.
  * [39] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson.  How transferable are features in deep neural networks?  In Advances in neural information processing systems, pages 3320–3328, 2014.
  * [40] Rui Zhu, Shifeng Zhang, Xiaobo Wang, Longyin Wen, Hailin Shi, Liefeng Bo, and Tao Mei.  Scratchdet: Training single-shot object detectors from scratch.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2268–2277, 2019.
  * [41] Peiqin Zhuang, Yali Wang, and Yu Qiao.  Learning attentive pairwise interaction for fine-grained classification.  In Proceedings of the AAAI Conference on Artificial Intelligence, number 07 in 34, pages 13130–13137, 2020.
  * [42] Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin D Cubuk, and Quoc V Le.  Rethinking pre-training and self-training.  arXiv preprint arXiv:2006.06882, 2020.