# Few-shot Knowledge Graph Relational Reasoning
via Subgraph Adaptation

Haochen Liu
University of Virginia
<EMAIL_ADDRESS>
Chen Chen
University of Virginia
<EMAIL_ADDRESS>
&Song Wang
University of Virginia
<EMAIL_ADDRESS>
Jundong Li
University of Virginia
<EMAIL_ADDRESS>
###### Abstract

Few-shot Knowledge Graph (KG) Relational Reasoning aims to predict unseen
triplets (i.e., query triplets) for rare relations in KGs, given only several
triplets of these relations as references (i.e., support triplets). This task
has gained significant traction due to the widespread use of knowledge graphs
in various natural language processing applications. Previous approaches have
utilized meta-training methods and manually constructed meta-relation sets to
tackle this task. Recent efforts have focused on edge-mask-based methods,
which exploit the structure of the contextualized graphs of target triplets
(i.e., a subgraph containing relevant triplets in the KG). However, existing
edge-mask-based methods have limitations in extracting insufficient
information from KG and are highly influenced by spurious information in KG.
To overcome these challenges, we propose SAFER (Subgraph Adaptation for FEw-
shot Relational Reasoning), a novel approach that effectively adapts the
information in contextualized graphs to various subgraphs generated from
support and query triplets to perform the prediction. Specifically, SAFER
enables the extraction of more comprehensive information from support triplets
while minimizing the impact of spurious information when predicting query
triplets. Experimental results on three prevalent datasets demonstrate the
superiority of our proposed framework SAFER.111Our code is available at
https://github.com/HaochenLiu2000/SAFER.

Few-shot Knowledge Graph Relational Reasoning
via Subgraph Adaptation

Haochen Liu University of Virginia<EMAIL_ADDRESS>Chen Chen University of
Virginia<EMAIL_ADDRESS>Song Wang University of Virginia
<EMAIL_ADDRESS>Jundong Li University of Virginia<EMAIL_ADDRESS>

##  1 Introduction

Figure 1: We provide an instance for the two limitations of edge-mask-based
methods. In this example, there are two support triplets (music, created_by,
musican) and (news article, created_by, reporter). When extracting support
information by finding the common subgraph, the extraction of edges with
similar meanings but in different graphs will fail, and some spurious
information will be extracted, which cannot correctly represent the logical
pattern of the relation created_by.

Knowledge Graphs (KGs) consist of many triplets, i.e., (head, relation, tail),
which represent specific relationships between real-world entities Wang et al.
(2017); Ji et al. (2022). These triplets form directed graphs that store
knowledge information and can be applied to various knowledge-based tasks
Liang et al. (2022); Wang et al. (2023) such as question answering Huang et
al. (2019); Saxena et al. (2020), information extraction Hoffmann et al.
(2011); Daiber et al. (2013), program analysis Liang et al. (2023), and
language model enhancement Zhang et al. (2020b); Yasunaga et al. (2021); Xie
et al. (2022). However, KGs generally cannot encompass all the necessary
knowledge triplets required by downstream tasks, as most KGs are severely
incomplete Xiong et al. (2018). Therefore, it becomes crucial to complete KGs
by inferring potential missing relations between entities. In particular,
existing works for KG completion Bordes et al. (2013); Zhu et al. (2021);
Zhang et al. (2022) often assume the availability of sufficient instances
(i.e., triplets) for each relation to be predicted. However, in real-world
scenarios, it is common to encounter _few-shot relations_ , where only limited
instances of triplets with these relations, called _support_ triplets, are
available. KGs are constantly being updated, for example, by including
knowledge from social networks. This often results in new relations with a
relatively scarce number of discovered triplets, as the labeling process can
be laborious. These new relations are generally known as few-shot relations.
Consequently, predicting new relations with only limited triplets becomes a
significant task Ma and Wang (2023). Therefore, it is crucial to perform the
Few-shot KG Relational Reasoning (Few-shot KGR) task Xiong et al. (2018),
which aims to predict the existence of (unseen) query triplets of a relation,
given a background KG and a set of a limited number of _support triplets_ of
the relation as the _support set_.

Currently, there exist two types of approaches for solving the Few-shot KGR
task. The first type is _meta-learning-based_ methods Chen et al. (2019);
Zhang et al. (2020a); Sun et al. (2022), which utilize the meta-learning
framework Finn et al. (2017) to transfer useful knowledge to new KGR tasks
Hospedales et al. (2021) with a limited number of support triplets, to tackle
the issue of data scarcity in the target few-shot tasks. Nevertheless, the
distribution of the manually selected target relations plays an important role
in these methods, which will result in suboptimal performance if the meta-
training sets are not well-designed. To address this limitation, more recent
studies have explored _edge-mask-based_ approaches Huang et al. (2022); Meng
et al. (2023), providing an alternative solution to Few-shot KGR tasks. Edge-
mask-based methods analyze each support (or query) triplet by first retrieving
its contextualized graph, i.e., the subgraph that consists of the head and
tail entities of a triplet, and the most relevant entities and relations of
the triplet. The subgraph is referred to as the support (or query) graph. Then
they extract common subgraphs across support graphs in the form of masks that
identify edges with shared meanings for predictions on query triplets.

Despite the effectiveness of these works, we argue that there are still two
major limitations of edge-mask-based methods. (1) Existing edge-mask-based
approaches assume that the largest common subgraph (masks) shared across all
support graphs is sufficient to represent the unseen target relation. However,
this assumption is difficult to satisfy in certain cases, e.g., when dealing
with triplets that involve different but similar relations across other
support graphs. As shown in Figure 1, on the support graphs of the target
relation created_by, the relations produced_by and published_in preserve
similar meanings. However, the strategy of learning edge masks fails to
harness the valuable insights from these different yet similar relations,
resulting in the insufficient extraction of information from created_by. (2)
The extracted common subgraph (masks) often contains unrelated spurious
information that can negatively impact prediction performance. For example,
during the extraction process in Figure 1 regarding the target relation
created_by, the support graphs may include spurious relations like
related_job, as it can be unhelpful or even misleading when predicting query
triplets of relation created_by.

To overcome the aforementioned challenges, we propose SAFER (Subgraph
Adaptation for FEw-shot Relational Reasoning), a novel subgraph-based approach
that effectively utilizes useful information from support graphs while
excluding spurious information. In SAFER, we first generate the contextualized
graphs of support and query triplets with edge weights representing the
importance of each relation for performing relational reasoning. Subsequently,
we perform Subgraph Adaptation comprising two crucial modules: _Support
Adaptation_ and _Query Adaptation_ , which aim to extract valuable information
from support graphs and exclude spurious information, respectively. In our
_Support Adaptation_ module, we incorporate information from each support
graph into others to enable the adaptation to support graphs with different
structures to extract and utilize useful information, e.g., similar relations.
In our _Query Adaptation_ module, we adapt the support information to the
structure of the query graph so that spurious information among support graphs
can be filtered out in a query-adaptive manner. As a result, we can
effectively alleviate the adverse impact of spurious information. In summary,
our contributions in this paper are as follows:

  1. 1.

We scrutinize the challenges of few-shot knowledge graph relational reasoning
(Few-shot KGR) from the perspective of extracting informative common
subgraphs. We also discuss the necessity of tackling the challenges.

  2. 2.

We develop a novel Few-shot KGR framework consisting of Subgraph Generation
and Subgraph Adaptation. Subgraph Adaptation includes (1) a Support Adaptation
(SA) module that enables a more comprehensive extraction of information from
the support graphs; (2) a Query Adaptation (QA) module that allows for
excluding the influence of spurious information in the extracted information.

  3. 3.

We conduct experiments on three prevalent real-world KG datasets of different
scales. The results further demonstrate the superiority of SAFER over other
state-of-the-art approaches.

##  2 Related Work

###  2.1 Meta-learning-based Few-shot KGR

Meta-learning Finn et al. (2017); Hospedales et al. (2021) is an effective
learning paradigm that transfers generalizable knowledge learned from training
tasks to new test tasks. Meta-learning necessitates a meta-training set that
comprises multiple Few-shot KGR tasks for training purposes and then
generalizes learned knowledge to tasks in the meta-test set. For example,
GMatching Xiong et al. (2018) and FSRL Zhang et al. (2020a), acquire a
universal metric to match query triplets with support triplets Wang et al.
(2021b). The performance of meta-learning is significantly influenced by the
quality of the manually created meta-training set. Moreover, the meta-training
set is sampled from the same distribution as the meta-test set, which is
impractical in practice Huang et al. (2022). To overcome these problems, some
alternative studies based on subgraph structures are proposed to tackle the
Few-shot KGR task.

###  2.2 Edge-mask-based Few-shot KGR

Edge-mask-based methods, such as CSR Huang et al. (2022) and SARF Meng et al.
(2023), consider the few-shot relational reasoning task as an inductive
reasoning problem Spelda (2020); Teru et al. (2020), which relies on the
relevant relations(i.e., edges) of the triplet Galárraga et al. (2013); Lin et
al. (2018); Qu et al. (2021) in KG to perform the prediction. These methods
employ an encoder-decoder model to encode the shared subgraphs of support
samples (masks), i.e., common subgraphs in KG that connect the two entities of
the triplets, into an embedding representing the target relation. The decoder
uses the embedding to reconstruct the edge masks in a query graph showing the
shared edges. These approaches take advantage of the edge structure to perform
reasoning. However, these methods have the limitation that the largest common
subgraph among support graphs may lose some of the relation’s logical
patterns, and the spurious information extracted will detrimentally affect the
prediction. In this paper, our approach uses a novel adaptation process to
address the shortcomings of incomplete utilization of structure information in
edge-mask-based methods.

##  3 Problem Formulation

We study the problem of _Few-shot Knowledge Graph Relational Reasoning_ ,
i.e., Few-shot KGR Xiong et al. (2018); Chen et al. (2019). We first denote
the background KG as $\mathcal{G}=(\mathcal{E},\mathcal{R},\mathcal{T})$,
where $\mathcal{E}$ and $\mathcal{R}$ are sets of entities and relations.
$\mathcal{T}=\\{(h,r,t)|h,t\in\mathcal{E},r\in\mathcal{R}\\}$ represents the
facts as triplets, each of which contains a head entity, a tail entity, and a
relation. For a new target relation $r^{\prime}\notin\mathcal{R}$, we are
given a support set $S_{r^{\prime}}$ with $K$ triplets
$\\{(h_{i},r^{\prime},t_{i})\\}_{i=1}^{K}$ of $r^{\prime}$, where
$h_{i},t_{i}\in\mathcal{E}$. The number of triplets in the support set $K$ is
relatively small ($K\leq 5$). With $S_{r^{\prime}}$ as the reference, we aim
to predict tail entities, given a head entity $h_{q}$, i.e.,
$(h_{q},r^{\prime},?)$. There are usually multiple candidates of the tail
entity that need to be scored and ranked. Then the candidate with the highest
score is considered as the prediction result. So we will consider the query
triplet $(h_{q},r^{\prime},c)$ ($c$ is a candidate) as a full triplet to
score.

##  4 Methodology

In this section, we introduce details of our proposed framework SAFER. As
illustrated in Figure 2, for each support (or query) triplet, we first extract
a support (or query) graph from the background KG and assign weights for each
edge on the graph. Then we conduct Subgraph Adaptation on the generated
support and query graphs and finally achieve the prediction score for a query
triplet.

Figure 2: The framework of SAFER, which shows the scoring pipeline for a query
tail candidate $c$ of target relation $r^{\prime}$. We represent the same
relations in colors, while the gray relations are all different. We first
extract the contextualized graph of each support and query triplet and assign
weights to all edges using an aggregation process $P_{w}$ (the width of edges
represents weights). Then we apply another aggregation process $P_{a}$ and two
adaptation operations to perform support information extraction and query
candidate scoring.

###  4.1 Retrieving Contextualized Graphs

To obtain structural information for the unseen target relation, we utilize
the contextualized graphs of support and query triplets, i.e., _support
graphs_ and _query graphs_. Contextualized graphs are generated based on the
enclosing subgraph strategy proposed by Zhang and Chen (2018); Teru et al.
(2020). We introduce how to construct contextualized graphs in Appendix A.1.

###  4.2 Edge Weight Assignment

After acquiring the contextualized graph, we propose to assign weights to all
edges on the contextualized graphs based on their importance to the target
relation. We assign the weight $w_{e}$ for each edge $e$ by incorporating
information from all support graphs to determine the importance, such that we
can effectively leverage the information within all relations.

Specifically, we leverage the PathCon Wang et al. (2021a) model to extract
structural information and calculate the edge weights, as it can measure graph
isomorphism. While edge-mask-based methods apply the model repeatedly between
any two graphs to get the masks, we only apply it to get an overall embedding
$g_{all}$ of all support graphs.

We define an aggregation process $P_{w}$ with $L$ iterations as follows:

$b_{v}^{i}=\frac{1}{1+|\\{e|e\in N(v)\\}|}\sum_{e\in N(v)}b_{e}^{i},$ (1)
$r_{v}^{i}=b_{v}^{i}\|\mathbbm{1}(v=h)\|\mathbbm{1}(v=t),$ (2)
$b_{e}^{i+1}=f(r_{u}^{i}\|r_{v}^{i}\|b_{e}^{i}),u,v\in N(e),$ (3)

where $b_{e}^{i}$ (or $b_{v}^{i}$) is the learned edge (or node) embedding in
iteration $i$. $N(v)$ is the set of all neighboring edges of $v$. $f$ is a
neural network (NN) consisting of both non-linear and linear layers. $\|$
denotes the concatenation of two vectors (or scalars). In particular, Eq. (1)
aggregates the embeddings of neighboring edges of each node. Then Eq. (2) adds
the label of head and tail so that the information of a node’s relative
position to head and tail can be considered. Eq. (3) updates all edge
embeddings based on the current embedding of the edge and its two end nodes.

In the first step, we initilize $b_{e}^{0}$ with the pretrained relation
embedding $v_{e}$ of the relation on edge $e$. We define the embedding of $G$
as follows:

$g(G)=\text{MaxPool}(b_{v}^{L})\|b_{h}^{L}\|b_{t}^{L},$ (4)

where $\text{MaxPool}(b_{v}^{L})$ is the max-pooling of all node embeddings in
$G$.

In the second step, similarly, we apply $P_{w}$ again to acquire the weights
of edges in both the support graphs and the query graphs. Additionally, we use
the average of the embeddings of all support graphs $g_{all}$ from the first
step as an input to incorporate the overall information in the support set and
initialize $b_{e}^{0}$ as $v_{e}\|g_{all}$. Here $g_{all}$ is defined as
follows:

$g_{all}=\frac{1}{K}\sum_{k}g(G_{s}^{k}).$ (5)

Here $G_{s}^{k}$ is the $k$-th support graph. We use another $f$ in this step.
Then we perform $P_{w}$ on the target graph $G$. Finally, we calculate the
weight $w_{e}$ of edge $e$:

$w_{e}=\frac{1}{1+\exp(-\text{Linear}(b_{e}^{L}))},$ (6)

where $\text{Linear}(\cdot)$ is a linear layer, and $w_{e}$ will serve as the
edge weight of $e$ in the subsequential adaptation modules.

Note that weight assignment does not rely on specific loss functions or
ground-truth definitions for edge weights. Instead, it is trained in an end-
to-end manner along with other modules in the subsequent sections. All edges
in the support graphs can contribute to the subsequential adaptation modules
based on the weight.

###  4.3 Subgraph Adaptation

In this subsection, we introduce the process of our Subgraph Adaptation
module, including _Support Adaptation_ (SA) and _Query Adaptation_ (QA).

After obtaining the edge-weighted support graphs and query graphs, we achieve
embeddings that contain the information from different subgraphs by
aggregations. While performing the aggregations, we further adapt graph
information to all support and query graphs to perform SA and QA. We first
define an $L$-iteration aggregation process $P_{a}$, which is utilized in both
SA and QA:

$a_{v}^{i}(k)=\frac{1}{1+\sum_{e\in N(v)}w_{e}(k)}\sum_{e\in
N(v)}b_{e}^{i}(k)\cdot w_{e}(k),$ (7) $\displaystyle
b_{v}^{i}(k)=\left\\{\begin{array}[]{ll}T_{SA}(\\{a^{i}_{v}(m)\\}_{m=1}^{K}),&\text{if}\
\text{}\text{SA},\\\
T_{QA}(a^{i}_{v}(k),\\{b^{i}_{t}(m)\\}_{m=1}^{K};\lambda),&\text{if}\
\text{}\text{QA},\end{array}\right.$ (8)
$r_{v}^{i}(k)=b_{v}^{i}(k)\|\mathbbm{1}(v=h)\|\mathbbm{1}(v=t),$ (9)
$b_{e}^{i+1}(k)=f(r_{u}^{i}(k)\|r_{v}^{i}(k)\|b_{e}^{i}(k)),u,v\in N(e),$ (10)

where $k$ indicates that a term is calculated on the $k$-th support graph, and
it can be replaced by $q$ to represent the value on a query graph in _Query
Adaptation_ (e.g., $a^{i}_{v}(q)$ and $b^{i}_{v}(q)$). $N(v)$ is the set of
all neighboring edges of node $v$. $w_{e}$ is the weight of edge $e$.
$a_{v}^{i}$ is the aggregation output of node $v$ at iteration $i$. Here Eq.
(7) aggregates the embeddings of all neighboring edges of each node based on
edge weights. $b_{v}^{i}$ (or $b_{e}^{i}$) is the learned node (or edge)
embedding in iteration $i$. The adaptation steps are $T_{SA}(\cdot)$ (for SA)
and $T_{QA}(\cdot)$ (for QA), and the details will be introduced in the
following subsections. $f$ is a neural network (NN) consisting of non-linear
and linear layers acting in both SA and QA. $\lambda$ is a hyperparameter used
in QA to be introduced. Note that we initialize $b_{e}^{0}(k)$ with the
pretrained embedding of the relation on edge $e$ to incorporate more
information.

####  4.3.1 Support Adaptation

To extract valuable information from all support graphs and reduce the
omissions of information, we propose the _Support Adaptation_ (SA) strategy
that enables the incorporation of information from all support graphs when
learning the embedding for each support graph. During aggregation on each
graph, we average the learned embeddings of the tail entities in all support
graphs after each iteration to absorb beneficial information from all other
support graphs. In particular, we choose to average the embeddings of tail
entities (instead of other entities), because the tail entity preserves the
most crucial information for the prediction of the target relation. The
averaged embedding will be used to update embeddings of all edges connected to
tail entities in all support graphs. This strategy ensures the transfer of
relational information from one support graph to various others, thereby
enabling adaptation to structures of different support graphs during
subsequent aggregation steps. In this way, all edges in the support graph can
contribute to SA based on their weights.

In SA, we apply $P_{a}$ to all $K$ support graphs for $L$ iterations.
$T_{SA}(\cdot)$ is defined as

$\displaystyle T_{SA}(\\{a^{i}_{v}(m)\\}_{m=1}^{K})=$ (11)
$\displaystyle\left\\{\begin{array}[]{ll}\frac{1}{K}\sum_{m=1}^{K}a^{i}_{t}(m),&\text{if}\
v=t,\\\\[6.0pt] a_{v}^{i}(k),&\text{otherwise}.\end{array}\right.$

Via Eq. (11), we manage to incorporate information from other support graphs
when performing aggregation on each support graph. Generally, if the
information from a specific relation in a support graph can be easily
propagated on another support graph with a different relation, we can infer
that these two relations maintain similar meanings. Therefore, our SA strategy
allows for extracting relevant relations (e.g., different yet similar
relations) among support graphs.

####  4.3.2 Query Adaptation

_Query Adaptation_ (QA) is the subsequent module that can exclude the
influence of spurious information extracted by the SA module. Generally, we
predict the score of a query triplet by comparing the similarity between
information learned from the query graph and the support graphs. To deal with
the presence of spurious information across query and support graphs, our QA
module adapts the tail node embeddings in support graphs to the structure of
the query graph. In this manner, the support information unhelpful for query
scoring will be filtered out, due to different structures between support
graphs and query graphs. Then we calculate the score of a query triplet by
comparing the filtered support embedding with the embedding of the query
graph.

To perform QA, we apply the aggregation process $P_{a}$ to the query graph of
the query triplet candidate. $T_{QA}(\cdot)$ is defined as follows:

$\displaystyle T_{QA}(a_{v}^{i}(q),\\{b^{i}_{t}(m)\\}_{m=1}^{K};\lambda)=$
(12) $\displaystyle\left\\{\begin{array}[]{ll}(1-\lambda)\cdot
a_{t}^{i}(q)+\frac{\lambda}{K}\sum_{m=1}^{K}b^{i}_{t}(m),&\text{if}\
v=t,\\\\[6.0pt] a_{v}^{i}(q),&\text{otherwise}.\end{array}\right.$

Here $\lambda\geq 0$ is a hyperparameter of QA, which shows the ratio of
incorporation of extracted support information and the information from the
query graph. In this manner, we perform aggregation for support information on
the query graph. As a result, our QA module can exclude the influence of
spurious information in support graphs, thus achieving more precise prediction
results.

To perform prediction for a query triplet, we compare two embeddings, $E_{s}$
and $E_{q}$, which involve (filtered) support information and query
information, respectively. Specifically, we define

$E_{s}=T_{QA}(a_{t}^{L}(q),\\{b^{L}_{t}(m)\\}_{m=1}^{K};\lambda)$ (13)

as the result of the filtered support information with $\lambda>0$ obtained
from Eq. (12). For $E_{q}$, we perform $P_{a}$ with $\lambda=0$ to ensure that
there is no incorporation of support information. We define $E_{q}$ as
follows:

$E_{q}=T_{QA}(a_{t}^{L}(q),\\{b^{L}_{t}(m)\\}_{m=1}^{K};0).$ (14)

As the calculation of $E_{q}$ does not involve information from support
graphs, $E_{q}$ only contains the query information. Additionally, we
concatenate the average of pretrained embeddings of all support and query tail
entities to $E_{s}$ and $E_{q}$, respectively, so that the pretrained entity
embedding can also contribute to the scoring. In particular, we use the cosine
similarity between $E_{s}$ and $E_{q}$ to measure the score of a query
candidate, denoted as

$s(t_{q})=\text{cos}(E_{s}\|\frac{1}{K}\sum_{k=1}^{K}v_{t_{s,k}},E_{q}\|v_{t_{q}}),$
(15)

where $s(t_{q})$ is the score for $t_{q}$, i.e., the tail entity of the query
triplet. $t_{s,k}$ is the tail entity of the $k$-th support triplet. We use
$v_{t_{s,k}}$ (or $v_{t_{q}}$) to denote the pretrained node embedding of
$t_{s,k}$ (or $t_{q}$). Note that both $E_{s}$ and $E_{q}$ are solely acquired
via aggregation on the query graph. This ensures exclusion of spurious
information in support graphs, thus achieving more precise scoring results.

###  4.4 Training Objective

To train the overall SAFER framework, we leverage contrastive learning with
positive samples (i.e., same relation in support and query triplets) and
negative samples (i.e., different relations in support and query triplets).
Specifically, we use the Margin Ranking Loss:

$\mathcal{L}=\max(s_{neg}-s_{pos}+\gamma,0),$ (16)

where $s_{pos}$ and $s_{neg}$ are scores of the positive sample and the
negative sample, respectively. $\gamma\in\mathbbm{R}$ is a hyperparameter
utilized to control the margin that separates positive and negative samples.

##  5 Experiments

In this section, we elaborate on the experiments for evaluating our proposed
framework.

###  5.1 Experimental Settings

####  5.1.1 Datasets

We evaluate our framework and other baselines on three real-world Few-shot KGR
datasets, generated based on NELL Mitchell et al. (2018), FB15K-237 Toutanova
et al. (2015), and ConceptNet Speer et al. (2017), respectively. The NELL
dataset is a subset of NELL-One Chen et al. (2019) by selecting the relations
that have between 50 and 500 triples as few-shot tasks. For FB15K-237 and
ConceptNet, we select the fewest 30 and 2 appearing relations as test few-shot
tasks, respectively, following Lv et al. (2019) and Chen et al. (2019). Table
1 lists the statistics of all three datasets.

####  5.1.2 Evaluation Metrics

We perform the evaluation for our framework and all baselines by calculating
the scores for query candidates of each test instance using the standard
ranking metrics. In particular, we utilize the Mean Reciprocal Ranking (MRR)
and Hits@h. The MRR measures the average reciprocal rank of the correct
candidate in the ranking of all candidates, where a higher value indicates
better performance. We also compute the Hits@h value, which measures the
percentage of the correct candidates ranked within the top $h=\\{1,5,10\\}$
positions. In evaluation, each correct candidate in the test set is paired
with 50 other candidate negative triplets.

####  5.1.3 Baselines

We compare our framework with existing Few-shot KGR methods, including MetaR
Chen et al. (2019), FSRL Zhang et al. (2020a), CSR-OPT Huang et al. (2022),
CSR-GNN Huang et al. (2022), SARF+Learn Meng et al. (2023), and SARF+Summat
Meng et al. (2023). For meta-learning-based methods, the training is achieved
by randomly sampling tasks from the KG rather than the meta-training split
that is originally provided, to avoid the influence of manually constructed
meta-training sets.

Table 1:  Statistics of three Few-shot KGR datasets.

Dataset | # Entities | # Relations | # Edges | # Tasks
---|---|---|---|---
NELL | 68,544 | 291 | 181,109 | 11
FB15K-237 | 14,543 | 200 | 268,039 | 30
ConceptNet | 790,703 | 14 | 2,541,996 | 2

Table 2:  Performance comparison of different KG datasets. The best and
second-best results are shown in bold and underlined, respectively.

Dataset | Method | MRR | Hits@1 | Hits@5 | Hits@10
---|---|---|---|---|---
NELL | MetaR | 0.471 | 0.322 | 0.647 | 0.763
FSRL | 0.490 | 0.327 | 0.695 | 0.853
CSR-OPT | 0.463 | 0.321 | 0.629 | 0.760
CSR-GNN | 0.577 | 0.442 | 0.746 | 0.858
SARF+Learn | 0.627 | 0.493 | 0.798 | 0.877
SARF+Summat | 0.626 | 0.493 | 0.797 | 0.875
SAFER (ours) | 0.674 | 0.560 | 0.812 | 0.887
FB15K-237 | MetaR | 0.805 | 0.740 | 0.881 | 0.937
FSRL | 0.684 | 0.573 | 0.817 | 0.912
CSR-OPT | 0.619 | 0.512 | 0.747 | 0.824
CSR-GNN | 0.781 | 0.718 | 0.851 | 0.907
SARF+Learn | 0.779 | 0.718 | 0.846 | 0.905
SARF+Summat | 0.753 | 0.688 | 0.814 | 0.884
SAFER (ours) | 0.793 | 0.728 | 0.860 | 0.914
ConceptNet | MetaR | 0.318 | 0.226 | 0.390 | 0.496
FSRL | 0.577 | 0.469 | 0.695 | 0.753
CSR-OPT | 0.559 | 0.450 | 0.692 | 0.736
CSR-GNN | 0.606 | 0.496 | 0.735 | 0.777
SARF+Learn | 0.613 | 0.511 | 0.731 | 0.771
SARF+Summat | 0.624 | 0.527 | 0.729 | 0.768
SAFER (ours) | 0.638 | 0.564 | 0.721 | 0.743

###  5.2 Performance Comparison

The detailed settings of our experiments are in Appendix A.2. We evaluate
SAFER along with other methods on the three datasets. For baseline
performance, we use the experimental results from Huang et al. (2022) and Meng
et al. (2023). Table 2 shows that our method outperforms baselines in most
cases. In NELL and ConceptNet, the improvement of SAFER on the testing MRR is
$7.67\%$ and $2.24\%$. The improvement of Hit@1 is $13.59\%$ and $7.02\%$. On
FB15K-237, our method is the second best, while being very close to MetaR. The
reason is that FB15K-237 contains a large number of relations whose
contextualized graphs contain only one triplet, and thus the methods based on
the subgraphs’ structure (i.e., CSR, SARF, SAFER) are limited in performance.

Compared to baselines, SAFER shows more significant advantages in MRR and
Hits@1. This is because, for the query candidates with high scores, the
information provided by the support and query graphs will be similar. Thus,
the spurious information in support graphs will more seriously impact the
scoring. Nevertheless, our process avoids spurious information in support
graphs, which contributes more to the detailed comparison between high-score
samples. Thus, SAFER achieves a more precise scoring result.

Figure 3: The performance of our proposed method SAFER with different
$\lambda$.

###  5.3 Hyperparameter Study

The value of $\lambda$ balances the removal of spurious information and the
prevention of over-filtering in QA. To study the impact of $\lambda$, we
conduct experiments with different values of $\lambda$, ranging from 0.001 to
1. The experimental results are presented in Figure 3. In general, these
results indicate that different datasets have different optimal values of
$\lambda$. For both MRR and Hits@1, the optimal $\lambda$ is 0.1 for NELL and
0.5 for FB15K-237 and ConceptNet. When $\lambda=1$, the scoring process is
actually a direct comparison between the outputs $b_{t}^{L}$ of support graphs
and the query graph in $P_{a}$ without any adaptation. In this case, the
results are much worse than the optimal results, which demonstrates the
strength of our QA module. For the NELL dataset, the optimal value of
$\lambda$ is much smaller because the candidates in NELL have more complex
subgraphs and thus require a more precise comparison of the detailed local
features.

###  5.4 Ablation Study

Table 3:  Ablation study on three datasets. The best results are shown in
bold.

Dataset | Method | MRR | Hits@1 | Hits@5 | Hits@10
---|---|---|---|---|---
NELL | SAFER | 0.674 | 0.560 | 0.812 | 0.887
SAFER$\backslash$W | 0.546 | 0.428 | 0.683 | 0.752
SAFER$\backslash$S | 0.575 | 0.434 | 0.753 | 0.832
SAFER$\backslash$Q | 0.533 | 0.422 | 0.659 | 0.715
FB15K-237 | SAFER | 0.793 | 0.728 | 0.860 | 0.914
SAFER$\backslash$W | 0.761 | 0.689 | 0.840 | 0.901
SAFER$\backslash$S | 0.761 | 0.688 | 0.841 | 0.901
SAFER$\backslash$Q | 0.778 | 0.713 | 0.846 | 0.905
ConceptNet | SAFER | 0.638 | 0.564 | 0.721 | 0.743
SAFER$\backslash$W | 0.474 | 0.331 | 0.632 | 0.729
SAFER$\backslash$S | 0.510 | 0.399 | 0.629 | 0.728
SAFER$\backslash$Q | 0.533 | 0.404 | 0.710 | 0.742

In this subsection, we conduct an ablation study to evaluate the contributions
of the three modules in SAFER: Weight Assignment, Support Adaptation, and
Query Adaptation. In particular, we remove one module in SAFER each time and
report the performance of the revised model on all three datasets. For
SAFER$\backslash$W, we directly set the weight $w_{e}=1$ for all edges to
remove the Weight Assignment module. For SAFER$\backslash$S, we remove the SA
module by removing the averaging in each iteration of $P_{a}$ and only using
the average of its final outputs as the support embedding. For
SAFER$\backslash$Q, we set $\lambda=1$ to change the scoring into a direct
comparison between the outputs $b_{t}^{L}$ of support graphs and the query
graph in $P_{a}$ without QA.

The results of the ablation study, presented in Table 3, validate the
effectiveness of all modules in SAFER. Removing the Weight Assignment module
significantly decreases the MRR metric. This demonstrates the importance of
the weights in the data preparation. Furthermore, removing the SA module leads
to a decrease in all evaluation metrics. This is because, at each iteration of
the $P_{a}$, the aggregations of embeddings from other graphs can emphasize
relevant relations in the support graphs. Without this module, the adaptation
process becomes a simple average of the final outputs of $P_{a}$ of all
support graphs, resulting in a loss of emphasis on critical information.
Furthermore, the results highlight the importance of the QA module,
particularly in terms of MRR and Hit@1 that reflect the similarity between
high-score candidates and support samples. By filtering the support
information, QA ensures that only relevant, and useful information from the
support graph is retained. This prevents the inclusion of spurious information
within the predefined limits (e.g. common subgraph), thus ultimately
contributing to improved performance.

###  5.5 Case Study

Figure 4: An instance on dataset ConceptNet using the edge-mask-based method
CSR and our method SAFER. The figure shows part of support and query graphs
and the scores of the 3-top candidates of the two methods. The shown edges
prove the limitation of the extraction of common subgraphs in edge-mask-based
methods.

In this section, we study the case that, in existing edge-mask-based methods,
the extracted masks (common subgraph) could not correctly represent the target
relation all the time. We use a real example in the ConceptNet test set to
demonstrate the limitations of extracting common subgraphs to represent the
logical pattern of the target relation.

We consider the 2-shot relational reasoning task with two support triplets
(art, created_by, artist) and (babies, created_by, humans), along with a query
triplet (article, created_by, writer). Here we use an example with both two
cases of extracted spurious relations and unextracted relevant relations in
the edge-mask-based methods to showcase the two limitations of edge-mask-based
methods, as shown in Figure 4. In the observed support graphs, we can identify
two edges of relations at_location and related_to as similar but unshared
information, and edges of relation action as spurious information.

Regarding the prediction results, our approach SAFER ranks the true answer of
the correct tail entity writer as first of all candidates, whereas the CSR
model ranks it as third of all candidates. In the scoring result of CSR,
incorrect candidates guideline and autism both receive higher scores than
writer. This study shows that our SAFER can actually solve the two limitations
of existing edge-mask-based methods in information extraction and processing.

##  6 Conclusion

In this paper, we introduce SAFER, a novel approach designed to address the
challenges in Few-shot Knowledge Graph Relational Reasoning (Few-shot KGR).
SAFER overcomes the limitations of existing methods by extracting useful
information while excluding spurious information. We first generate edge-
weighted subgraphs of triplets to retrieve useful information from the
knowledge graph. With the generated subgraphs, we perform Support Adaptation,
which enables the incorporation of useful information that is difficult to
extract (e.g., different yet similar relations). Subsequently, our Query
Adaptation module filters out spurious information that is easily extracted
(e.g., unhelpful relations that are shared across support graphs).
Experimental evaluations on three datasets demonstrate the superiority of
SAFER over other state-of-the-art baselines under different evaluation
metrics. In summary, our work provides valuable insights into the potential of
subgraph adaptation to improve performance on Few-shot KGR tasks.

##  7 Acknowledgement

This work is supported in part by the National Science Foundation under grants
(IIS-2006844, IIS-2144209, IIS-2223769, CNS2154962, and BCS-2228534), the
Commonwealth Cyber Initiative Awards under grants (VV-1Q23-007, HV2Q23-003,
and VV-1Q24-011), the JP Morgan Chase Faculty Research Award, and the Cisco
Faculty Research Award.

## References

  * Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013.  Translating embeddings for modeling multi-relational data.  _NeurIPS_.
  * Chen et al. (2019) Mingyang Chen, Wen Zhang, Wei Zhang, Qiang Chen, and Huajun Chen. 2019.  Meta relational learning for few-shot link prediction in knowledge graphs.  In _EMNLP-IJCNLP_.
  * Daiber et al. (2013) Joachim Daiber, Max Jakob, Chris Hokamp, and Pablo N. Mendes. 2013.  Improving efficiency and accuracy in multilingual entity extraction.  In _SEMANTICS_.
  * Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _ICML_.
  * Galárraga et al. (2013) Luis Antonio Galárraga, Christina Teflioudi, Katja Hose, and Fabian M. Suchanek. 2013.  AMIE: association rule mining under incomplete evidence in ontological knowledge bases.  In _WWW_.
  * Hoffmann et al. (2011) Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S. Weld. 2011.  Knowledge-based weak supervision for information extraction of overlapping relations.  In _ACL_.
  * Hospedales et al. (2021) Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. 2021.  Meta-learning in neural networks: A survey.  _TPAMI_.
  * Huang et al. (2022) Qian Huang, Hongyu Ren, and Jure Leskovec. 2022.  Few-shot relational reasoning via connection subgraph pretraining.  In _NeurIPS_.
  * Huang et al. (2019) Xiao Huang, Jingyuan Zhang, Dingcheng Li, and Ping Li. 2019.  Knowledge graph embedding based question answering.  In _WSDM_.
  * Ji et al. (2022) Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and Philip S. Yu. 2022.  A survey on knowledge graphs: Representation, acquisition, and applications.  _TNNLS_.
  * Liang et al. (2022) Ke Liang, Lingyuan Meng, Meng Liu, Yue Liu, Wenxuan Tu, Siwei Wang, Sihang Zhou, Xinwang Liu, and Fuchun Sun. 2022.  Reasoning over different types of knowledge graphs: Static, temporal and multi-modal.  _arXiv preprint arXiv:2212.05767_.
  * Liang et al. (2023) Ke Liang, Jim Tan, Dongrui Zeng, Yongzhe Huang, Xiaolei Huang, and Gang Tan. 2023.  Abslearn: a gnn-based framework for aliasing and buffer-size information retrieval.  _PAA_.
  * Lin et al. (2018) Xi Victoria Lin, Richard Socher, and Caiming Xiong. 2018.  Multi-hop knowledge graph reasoning with reward shaping.  In _EMNLP_.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _ICRL_.
  * Lv et al. (2019) Xin Lv, Yuxian Gu, Xu Han, Lei Hou, Juanzi Li, and Zhiyuan Liu. 2019.  Adapting meta knowledge graph information for multi-hop reasoning over few-shot relations.  In _EMNLP-IJCNLP_.
  * Ma and Wang (2023) Haodi Ma and Daisy Zhe Wang. 2023.  A survey on few-shot knowledge graph completion with structural and commonsense knowledge.  _arXiv preprint arXiv:2301.01172_.
  * Meng et al. (2023) Lingyuan Meng, Ke Liang, Bin Xiao, Sihang Zhou, Yue Liu, Meng Liu, Xihong Yang, and Xinwang Liu. 2023.  Sarf: Aliasing relation assisted self-supervised learning for few-shot relation reasoning.  _arXiv preprint arXiv:2304.10297_.
  * Mitchell et al. (2018) T. Mitchell, W. Cohen, E. Hruschka, P. Talukdar, B. Yang, J. Betteridge, A. Carlson, B. Dalvi, M. Gardner, B. Kisiel, J. Krishnamurthy, N. Lao, K. Mazaitis, T. Mohamed, N. Nakashole, E. Platanios, A. Ritter, M. Samadi, B. Settles, R. Wang, D. Wijaya, A. Gupta, X. Chen, A. Saparov, M. Greaves, and J. Welling. 2018.  Never-ending learning.  _Commun. ACM_.
  * Qu et al. (2021) Meng Qu, Junkun Chen, Louis-Pascal A. C. Xhonneux, Yoshua Bengio, and Jian Tang. 2021.  Rnnlogic: Learning logic rules for reasoning on knowledge graphs.  In _ICLR_.
  * Saxena et al. (2020) Apoorv Saxena, Aditay Tripathi, and Partha Talukdar. 2020.  Improving multi-hop question answering over knowledge graphs using knowledge base embeddings.  In _ACL_.
  * Speer et al. (2017) Robyn Speer, Joshua Chin, and Catherine Havasi. 2017.  Conceptnet 5.5: An open multilingual graph of general knowledge.  In _AAAI_.
  * Spelda (2020) Petr Spelda. 2020.  Machine learning, inductive reasoning, and reliability of generalisations.  _AI_.
  * Sun et al. (2022) Jian Sun, Yu Zhou, and Chengqing Zong. 2022.  One-shot relation learning for knowledge graphs via neighborhood aggregation and paths encoding.  _TALLIP_.
  * Teru et al. (2020) Komal K. Teru, Etienne G. Denis, and William L. Hamilton. 2020.  Inductive relation prediction by subgraph reasoning.  In _ICML_.
  * Toutanova et al. (2015) Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choudhury, and Michael Gamon. 2015.  Representing text for joint embedding of text and knowledge bases.  In _EMNLP_.
  * Trouillon et al. (2016) Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016.  Complex embeddings for simple link prediction.  In _ICML_.
  * Wang et al. (2021a) Hongwei Wang, Hongyu Ren, and Jure Leskovec. 2021a.  Relational message passing for knowledge graph completion.  In _SIGKDD_.
  * Wang et al. (2017) Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. 2017.  Knowledge graph embedding: A survey of approaches and applications.  _TKDE_.
  * Wang et al. (2021b) Song Wang, Xiao Huang, Chen Chen, Liang Wu, and Jundong Li. 2021b.  Reform: Error-aware few-shot knowledge graph completion.  In _CIKM_.
  * Wang et al. (2023) Song Wang, Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, et al. 2023.  Knowledge editing for large language models: A survey.  _arXiv preprint arXiv:2310.16218_.
  * Xie et al. (2022) Qianqian Xie, Jennifer Bishop, Prayag Tiwari, and Sophia Ananiadou. 2022.  Pre-trained language models with domain knowledge for biomedical extractive summarization.  _KBS_.
  * Xiong et al. (2018) Wenhan Xiong, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. 2018.  One-shot relational learning for knowledge graphs.  In _EMNLP_.
  * Yasunaga et al. (2021) Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021.  QA-GNN: reasoning with language models and knowledge graphs for question answering.  In _NAACL-HLT_.
  * Zhang et al. (2020a) Chuxu Zhang, Huaxiu Yao, Chao Huang, Meng Jiang, Zhenhui Li, and Nitesh V. Chawla. 2020a.  Few-shot knowledge graph completion.  In _AAAI_.
  * Zhang et al. (2022) Denghui Zhang, Zixuan Yuan, Hao Liu, Xiaodong Lin, and Hui Xiong. 2022.  Learning to walk with dual agents for knowledge graph reasoning.  In _AAAI_.
  * Zhang and Chen (2018) Muhan Zhang and Yixin Chen. 2018.  Link prediction based on graph neural networks.  In _NeurIPS_.
  * Zhang et al. (2020b) Yice Zhang, Jiaxuan Lin, Yang Fan, Peng Jin, Yuanchao Liu, and Bingquan Liu. 2020b.  CN-HIT-IT.NLP at semeval-2020 task 4: Enhanced language representation with multiple knowledge triples.  In _SemEval_.
  * Zhu et al. (2021) Zhaocheng Zhu, Zuobai Zhang, Louis-Pascal A. C. Xhonneux, and Jian Tang. 2021.  Neural bellman-ford networks: A general graph neural network framework for link prediction.  In _NeurIPS_.

##  Appendix A Appendix

###  A.1 Retrieving Contextualized Graphs

In this section, we introduce how we retrieve contextualized graphs from a
triplet.

Contextualized graphs are generated based on the enclosing subgraph strategy
proposed by Zhang and Chen (2018); Teru et al. (2020). Specifically, for a
given triplet $(h,r,t)$, we first sample the nodes within $n$-hop undirected
neighbors of both the head entity $h$ and the tail entity $t$ from the
background KG. To include sufficient nodes for logic extraction, we also
perform random sampling from all neighbors of $h$ and $t$. The resulting
contextualized graph is induced by all selected nodes and their connections.
It should be noted that the specific value of $n$ is determined based on the
density of the KG. In particular, these contextualized graphs can capture the
local structure and relevant entities surrounding the support and query
triplets, thus allowing us to extract valuable information for the relational
reasoning task.

###  A.2 Experimental Settings

In this section, we delve into a more comprehensive exposition of our
experimental setups, including detailed parameter settings, as applied to the
three distinct real KG datasets.

In our experiments, we have employed 3-shot relational reasoning tasks across
all three datasets. For the NELL dataset, we set $n=2$ hops, whereas, for both
the FB15K-237 and ConceptNet datasets, we use $n=1$ hop when generating the
contextualized graphs of their respective triplets.

Regarding the neural network $f$, we have incorporated three distinct neural
networks for the first and second steps of weight assignment and the
adaptation module. The overall iteration of all modules is set to four, and
the hidden dimension of all embeddings (excluding the initialization) has been
standardized to 128. For the standard model, we choose the hyperparameter
$\lambda$ in Query Adaptation as $\lambda=0.1$ for NELL and $\lambda=0.5$ for
FB15K-237 and ConceptNet. All methods have utilized 100-dimensional relation
and entity embeddings.

For pretrained embeddings, we have employed TransE Bordes et al. (2013) for
the NELL and FB15K-237 datasets, while ComplEx Trouillon et al. (2016) has
been utilized for ConceptNet. In the context of the NELL dataset, the TransE
embeddings have been integrated by concatenating $v_{head}-v_{tail}$ to
$E_{s}$ and $E_{q}$ within the _Query Adaptation_ phase. Here, $v_{head}$ and
$v_{tail}$ signify the pretrained embeddings of the head and tail entities,
and an optional neural network ($NN(v_{head}-v_{tail})$) can also be added.
For the FB15K-237 dataset, a $BatchNorm$ Layer has been introduced within the
$Linear$ layer in Eq. (6).

Regarding optimization, we have employed AdamW Loshchilov and Hutter (2019)
with the learning rate $10^{-5}$, utilizing a linear schedule with 2,000 warm-
up steps and a total of 20,000 steps.

To ensure robustness and reliability, each reported experimental result is
derived from the average value obtained through conducting three independent
experiments.

###  A.3 Experimental Details

We conduct all our SAFER training and testing procedures using NVIDIA RTX
A6000 GPUs with a memory capacity of 48GB. Each training and testing instance
was executed on a single GPU, and conducted using Python 3.10.10. We implement
our framework with PyTorch.

###  A.4 Limitations

In this section, we introduce the limitations of our work in detail. Our SAFER
model incorporates the Query Adaptation (QA) module to mitigate the inclusion
of spurious information derived from the Support Adaptation (SA) module. For
tail candidates with notably high scores, indicating substantial similarity
between query and support graphs, the presence of extracted spurious
information can severely impact the scoring process. In this way, the model
tends to compare the most important and detailed information between support
and query. Consequently, this has resulted in a remarkable enhancement in Mean
Reciprocal Rank (MRR) and Hits@1 metrics.

However, this adaptation process inadvertently can still lead to the omission
of certain global information from the support graph. This is a consequence of
transferring all support information for processing onto the query graph.
Consequently, the improvements of SAFER in Hits@5 and Hits@10 metrics are not
as pronounced as those observed in MRR and Hits@1.

At present, we have yet to devise a solution to effectively integrate global
information into predictions. Balancing the incorporation of detailed and
global information concurrently presents a challenge that necessitates further
investigation and future research endeavors.


# Disentangled Feature Representation for Few-shot Image Classification

Hao Cheng1, Yufei Wang1, Haoliang Li2, Alex C. Kot1, Bihan Wen1
Bihan Wen is the corresponding author.

###### Abstract

Learning the generalizable feature representation is critical for few-shot
image classification. While recent works exploited task-specific feature
embedding using meta-tasks for few-shot learning, they are limited in many
challenging tasks as being distracted by the excursive features such as the
background, domain and style of the image samples. In this work, we propose a
novel Disentangled Feature Representation framework, dubbed DFR, for few-shot
learning applications. DFR can adaptively decouple the discriminative features
that are modeled by the classification branch, from the class-irrelevant
component of the variation branch. In general, most of the popular deep few-
shot learning methods can be plugged in as the classification branch, thus DFR
can boost their performance on various few-shot tasks. Furthermore, we propose
a novel FS-DomainNet dataset based on DomainNet, for benchmarking the few-shot
domain generalization tasks. We conducted extensive experiments to evaluate
the proposed DFR on general and fine-grained few-shot classification, as well
as few-shot domain generalization, using the corresponding four benchmarks,
i.e., mini-ImageNet, tiered-ImageNet, CUB, as well as the proposed FS-
DomainNet. Thanks to the effective feature disentangling, the DFR-based few-
shot classifiers achieved the state-of-the-art results on all datasets.

## Introduction

While deep neural networks achieved superior results on image classification
via supervised learning from large-scale datasets, it is challenging to
classify a query sample using only few labelled data, which is known as few-
shot classification (Fei-Fei, Fergus, and Perona 2006). How to learn the
discriminative feature representation that can be generalized from the
training set to new classes in testing is critical for few-shot tasks. Popular
few-shot methods applied _meta-learning_ (Vinyals et al. 2016) by episodic
training from a large amount of simulated meta-tasks, to obtain a task-
specific feature embedding associated with a distance metric (e.g., cosine or
Euclidean distance) for classification.

Figure 1: Excursive features (highlighted in boxes) that distract few-shot
classification for fine-grained and multi-domain tasks.

In practice, many excursive features of image data, e.g., style, domain and
background, are typically class-irrelevant. Figure 1 shows two such examples
in fine-grained and multi-domain classification tasks, respectively, which are
challenging for few-shot learning: (1) Only the subtle traits are critical to
characterize and differentiate the objects of fine-grained classes; (2) The
style and domain information dominates the image visual presence, but they are
in fact the excursive and class-irrelevant features. As the subtle traits vary
in different simulated meta-tasks, they can hardly be preserved by the learned
embedding. On the contrary, the excursive features usually distract the
feature embedding (Tokmakov, Wang, and Hebert 2019; Zhang et al. 2020),
leading to the degraded few-shot classification results. To rectify such
limitations, most recent few-shot methods attempted to suppress excursive
features or propose proper metrics, e.g., LCR (Tokmakov, Wang, and Hebert
2019), DeepEMD (Zhang et al. 2020), FEAT (Ye et al. 2020) and CNL (Zhao et al.
2021). However, none of the existing methods explicitly extract the class-
specific representation from the excursive image features.

In this paper, we present a novel approach to incorporate deep disentangling
for few-shot image classification. Such approach can selectively extract the
subtle traits for each task, while maintaining the model generalization.
First, we propose a novel Disentangled Feature Representation (DFR) framework
which can be applied to most few-shot learning methods. DFR contains two
branches: the classification branch extracts the discriminative features of
the image sample, while the variation branch encodes the class-irrelevant
information that complements the image representation. A RelationNet (Sung et
al. 2018) is applied in the variation branch to measure the feature similarity
of each sample pair. A hybrid loss is applied for training DFR, including a
reconstruction loss to ensure image information preservation, as well as the
translation, discriminative and cross-entropy losses for class-specific
feature disentangling. At the inference stage, only the disentangled features
from the classification branch are used for class prediction. Second, we
integrate the proposed DFR framework into representative baselines for few-
shot classification, including the popular ProtoNet (Snell, Swersky, and Zemel
2017) and the state-of-the-art DeepEMD (Zhang et al. 2020) and FEAT (Ye et al.
2020), to carefully investigate the behaviour of DFR with feature
visualization and analysis. Extensive experiments are conducted on a set of
few-shot tasks, i.e., general image classification, fine-grained image
classification, and domain generalization over four benchmarks to demonstrate
the effectiveness of our DFR framework.

Our main contributions are summarized as follows:

  * •

We propose a novel disentangled feature representation (DFR) framework, which
can be easily applied to most of the few-shot learning methods to extract
class-specific features from excessive information.

  * •

We propose a novel benchmark named FS-DomainNet based on DomainNet (Peng et
al. 2019) and fully study the few-shot domain generalization task with two
evaluation settings.

  * •

We evaluate the DFR framework over four few-shot benchmarks, i.e., mini-
ImageNet, tiered-ImageNet, CUB-200-2011, and the proposed FS-DomainNet
dataset. Results show that incorporating DFR into existing few-shot
algorithms, including both baseline and state-of-the-art methods, can generate
consistent gains for multi few-shot classification tasks under both 5-way
1-shot and 5-way 5-shot settings.

Figure 2: DFR for few-shot image classification: Given a few-shot meta-task
$\mathcal{T}_{FS}$ with support ($\mathcal{S}$) and query ($\mathcal{Q}$)
image sets, the encoders ($E_{cls}$ and $E_{var}$) of the classification and
variation branches extract the class-specific and class-irrelevant features,
respectively. $E_{cls}$ is a classic backbone used in the few-shot methods
(e.g., ResNet-12 in this work) which follows the blue stream. The output of
$E_{cls}$ is used for few-shot classification, and the output of $E_{var}$ is
fed to the RelationNet with a gradient reverse layer (GRL) to remove any
class-related information. An MLP block extracts the class information from
the classification branch to guide the image reconstruction and translation.
Specifically, the Decoder D can achieve self-reconstruction, class-
reconstruction, or class-translation according to the different inputs of MLP.

## Related Work

### Few-shot Learning

According to the meta-learning framework (Vinyals et al. 2016), there are
mainly three types of few-shot learning methods. Firstly, the gradient-based
methods utilize a good model initialization (Finn, Abbeel, and Levine 2017;
Nichol, Achiam, and Schulman 2018) or optimization strategy (Ravi and
Larochelle 2017; Rusu et al. 2019; Lee et al. 2019; Liu, Schiele, and Sun
2020) to quickly adapt to novel tasks. Secondly, the data augmentation-based
methods focus on generating (Gidaris and Komodakis 2019; Li et al. 2020a) or
gathering augmented data (Hariharan and Girshick 2017; Wang et al. 2018; Yang,
Liu, and Xu 2021) to enable classification from limited samples. In this work,
we focus on the third type, namely the metric learning-based methods, i.e., to
learn the discriminative feature embedding for distinguishing different image
classes. For example, ProtoNet (Snell, Swersky, and Zemel 2017) considered the
class-mean representation as the prototype of each class and applied the
Euclidean distance metric for classification. LCR (Tokmakov, Wang, and Hebert
2019) applied the subspace-based embedding for each class and DeepEMD (Zhang
et al. 2020) adopted the earth mover’s distance as the metric function to
compare the similarity between two feature maps in a structured way. FEAT (Ye
et al. 2020) defined four kinds of set-to-set transformation including self-
attention transformer (Jaderberg et al. 2015) to learn a task-specific feature
embedding for few-shot learning. Based on prior knowledge, COMET (Cao, Brbic,
and Leskovec 2021) mapped some high-level visual concepts into a semi-
structured metric space and then learned an ensemble classifier by combining
the outputs of independent concept learners. Tang et al. (Tang, Wertheimer,
and Hariharan 2020) also uses a semi-structured feature space based on
independent prior knowledge concepts to do pose normalization for fine-grained
tasks. Our work does not intend to propose new metrics, but focuses on
extracting the class-specific features from the variations distracting the
metric learning, thus to improve few-shot classification.

### Disentangled Feature Representations

Disentangled feature representation aims to learn an interpretable
representation for image variants, which has been widely studied in tasks such
as face generation (Chen et al. 2016), style translation (Lee et al. 2020; Liu
et al. 2019), image restoration (Li et al. 2020b), video prediction (Hsieh et
al. 2018) and image classification (Prabhudesai et al. 2021; Li et al. 2021).
InfoGAN (Chen et al. 2016) applied an unsupervised method to learn
interpretable and disentangled representations by maximized mutual
information. DRIT (Lee et al. 2020) embedded images into a content space and a
domain-specific attribute space and applied a cycle consistency loss for style
translation. FDR (Li et al. 2020b) applied channel-wise feature
disentanglement to reduce the interference between hybrid distortions for
hybrid-distorted image restoration. Li et al. (Li et al. 2021) proposed a
disentangled-VAE to excavate category-distilling information from visual and
semantic features for generalized zero-shot learning.

It is noteworthy that the very recent D3DP (Prabhudesai et al. 2021) also
adopted a feature disentangling scheme for few-shot detection and VQA, by
dividing high-dimensional data (e.g., RGB-D) into individual objects and other
attributes. However, our DFR significantly differs from D3DP in the following
aspects: DFR is a general-purpose feature extractor for image classification,
while D3DP only disentangles individual object to tackle more specific object
detection and VQA in 3D scenes. Besides, our DFR works on real images from
few-shot benchmarks while D3DP only works with synthetic scenes. Moreover, the
DFR framework is much simpler comparing to D3DP with fewer parameters and more
efficient algorithms. Thus DFR can serve an enhanced feature extractor for
classic backbones that are widely used in most of the existing few-shot
methods.

## Proposed Method

In this section, we start with a brief introduction to few-shot learning. Then
the proposed DFR framework is explained in detail, followed by the loss
function of our model and why our DFR works well.

### Problem Definition

Given a training image set with the base classes $\mathcal{C}_{train}$, few-
shot image classification task aims to predict the novel classes
$\mathcal{C}_{test}$ from the testing set, i.e.,
$\mathcal{C}_{train}\cap\mathcal{C}_{test}=\emptyset$. Thus, the trained
classifier from $\mathcal{C}_{train}$ needs to be generalized to
$\mathcal{C}_{test}$ in the testing stage with only few labeled samples. In
this paper, we follow the meta-learning strategy (i.e., the $N$-way $K$-shot
setting) (Vinyals et al. 2016) to simulate meta-tasks in the training set that
are similar to the few-shot setting at the testing stage, i.e., each meta-task
$\mathcal{T}_{FS}$ contains a support set $\mathcal{S}$, and a query set
$\mathcal{Q}$. The support set $\mathcal{S}$ contains $N$ classes with $K$
labeled samples ($N$ and $K$ are both very small) and a query set
$\mathcal{Q}$ with unlabeled query samples from $N$ classes is used to
evaluate the performance.

### Disentangled Feature Representation

Figure 2 is an overview of the proposed DFR framework. With a few-shot task
$\mathcal{T}_{FS}$ of support set $\mathcal{S}$ and query set $\mathcal{Q}$,
the objective is to extract discriminative features for classification from
the excursive information of each image $x_{i}$. The proposed DFR consists of
two branches with two encoders, i.e., $E_{cls}$ and $E_{var}$ for the
classification and variation branches, respectively, and one decoder $D$, as
well as a discriminator with a gradient reverse layer and a relation module.

Classification Branch. In principle, any classic metric-based backbone for
few-shot learning can be applied as $E_{cls}$ in this branch, to extract the
class-specific features of each $x_{i}$. In this work, the commonly-used
ResNet-12 backbone is adopted as $E_{cls}$, and the classifier $f(\cdot)$
varies for different few-shot learning baselines being used (e.g., ProtoNet
(Snell, Swersky, and Zemel 2017), DeepEMD (Zhang et al. 2020) and FEAT (Ye et
al. 2020) are applied in this work, with the corresponding models denoted as
+DFR). Therefore, the query sample $x_{i}^{\mathcal{Q}}$ can be classified
based on the support samples $x^{\mathcal{S}}$ as

$\hat{y}_{i}=f(E_{cls}(x_{i}^{\mathcal{Q}});\\{E_{cls}(x^{\mathcal{S}}),y^{\mathcal{S}}\\}).$
(1)

Variation Branch. The role of the variation branch is to encode the class-
irrelevant information of image samples, which consists of an encoder
$E_{var}$ followed by a discriminator. The feature map dimension (i.e.,
$h\times w$) of $E_{var}(x_{i})$ is set to be higher than that of
$E_{cls}(x_{i})$, to contain more excursive image features. Moreover, we apply
instance normalization (IN) and adaptive instance normalization (AdaIN)
instead of batch normalization to achieve information transfer for variation
encoder $E_{var}$ and decoder $D$, respectively. The discriminator is formed
by a gradient reversal layer (GRL) and a RelationNet $r_{\varphi}$ (Sung et
al. 2018) to measure the variation feature similarity between any two samples.
To be specific, the GRL acts as an identity transform in forward pass, and it
multiples the gradient from the subsequent level by a constant
$\textnormal{-}\lambda$ during back-propagation. In training, we construct
positive and negative pairs from the meta-task by their real labels. The
relation module $r_{\varphi}$ outputs a score $s_{i}\in[0,1]$ indicating the
probability that the pair $x_{i1}$ and $x_{i2}$ are from the same class as

$s_{i}=r_{\varphi}\left(E_{var}(x_{i1}),E_{var}(x_{i2})\right).$ (2)

Decoder Module. To preserve the image information and achieve feature
disentanglement, a decoder module with a MLP module and a decoder network $D$
combines the classification and variation branches for image reconstruction
and translation.

To be specific, the output feature of the classification branch is fed to the
MLP module $g$ to extract class-specific information $(\mu,\sigma)$ of each
sample for scaling the feature of the variation branch in the follow-up
decoder. The decoder can reconstruct or translate the source image based on
different sources of $(\mu,\sigma)$, shown in Figure 2, as

$\hat{x}_{i}=D(E_{var},g(X)),$ (3)

where $X$ can be the feature of the $i$-th sample itself for self-
reconstruction, or the mean feature of class $y_{i}$ for class-reconstruction
or another class $y_{j}$ with $j\neq i$ for class-translation.

### Loss Function

The objective function consists of the discriminative loss $L_{dis}$, cross-
entropy loss $L_{cls}$, reconstruction loss $L_{rec}$ and translation loss
$L_{tran}$.

Discriminative Loss. To remove the class-specific information in the variation
branch, we incorporate the binary cross-entropy loss to optimize the variation
feature maps based on the score of RelationNet as

$L_{dis}=-\sum_{i=1}^{P}\left(l_{i}\cdot log(s_{i})+(1-l_{i})\cdot
log(1-s_{i})\right),$ (4)

where $P$ denotes the number of training pairs, $s_{i}$ is the relation score
of the $i$-th pair which is calculated by (2), and $l_{i}=0$ or $1$ indicates
the ground truth whether the $i$-th training pair is positive. We minimize
$L_{dis}$ in training, and apply GRL to reverse the gradient during back-
propagation to achieve feature disentangling, i.e., to minimize the class-
specific information captured by the variation feature.

Cross-Entropy Loss. To preserve class-related features for few-shot
classification, we minimize the cross-entropy loss $L_{cls}$ for the
classification branch for query samples of all classes as

$L_{cls}=-\sum_{i=1}^{Q}y_{i}\log
P\left(\hat{y}_{i}=y_{i}\mid\mathcal{T}_{FS}\right),$ (5)

where $Q$ is the number of query samples in a meta-task $\mathcal{T}$, $y_{i}$
and $\hat{y_{i}}$ denote the true and predicted class label of each query
sample $x_{i}$, respectively.

Reconstruction and Translation Loss. To ensure that the disentangled
classification and variation features can jointly restore the input image, an
$\ell_{1}$-norm penalty for image reconstruction and a perceptual loss
(Johnson, Alahi, and Fei-Fei 2016) are applied after decoding for self-
reconstruction and class-reconstruction as

$L_{rec}=\frac{1}{M}\sum_{i=1}^{M}\|x_{i}-\hat{x}_{i}\|_{1}+\frac{1}{M}\sum_{i=1}^{M}\|\phi(x_{i})-\phi(\hat{x}_{i}^{c_{i}})\|_{1},$
(6)

where $M$ denotes the number of samples in a meta-task $\mathcal{T}_{FS}$.
$\hat{x}_{i}$ and $\hat{x}_{i}^{c_{i}}$ are the reconstructed images of
$x_{i}$ based on the feature of the $i$-th sample itself and mean feature
$c_{i}$ of class $y_{i}$ using (3), respectively.

Moreover, the perceptual loss is also adapted to measure perceptual
differences between the output image $\hat{x}_{i}^{c_{j}}$ and the support set
of the $j$-th class for class-translation to achieve feature disentanglement
as

$L_{tran}=\frac{1}{N}\sum_{i=1}^{M}\sum_{l=1}^{K}\|\phi(x_{l}^{\mathcal{S}_{j}})-\phi(\hat{x}_{i}^{c_{j}})\|_{1}.$
(7)

The total loss for training DFR can be formulated as

$L_{total}=\lambda_{1}\cdot L_{dis}+\lambda_{2}\cdot L_{rec}+\lambda_{3}\cdot
L_{tran}+L_{cls},$ (8)

where $\lambda_{1}$, $\lambda_{2}$ and $\lambda_{3}$ denote the weights
parameters of $L_{dis}$, $L_{rec}$ and $L_{tran}$ relative to $L_{cls}$,
respectively.

Figure 3: The t-SNE visualization of the feature representations: (a) the learned features of ResNet-12 backbone for methods w/o DFR, (b) the output features of the classification branch, and (c) the output features of the variation branch. Method | mini-ImageNet | tiered-ImageNet
---|---|---
5-way 1-shot | 5-way 5-shot | 5-way 1-shot | 5-way 5-shot
TADAM (Oreshkin, López, and Lacoste 2018) | 58.50 $\pm$ 0.30 | 76.70 $\pm$ 0.30 |  - |  -
AFHN (Li et al. 2020a) | 62.38 $\pm$ 0.72 | 78.16 $\pm$ 0.56 |  - |  -
MetaOptNet (Lee et al. 2019) | 62.64 $\pm$ 0.82 | 78.63 $\pm$ 0.46 | 65.99 $\pm$ 0.72 | 81.56 $\pm$ 0.53
DSN (Simon et al. 2020) | 62.64 $\pm$ 0.66 | 78.83 $\pm$ 0.45 | 66.22 $\pm$ 0.75 | 82.79 $\pm$ 0.48
MatchNet (Vinyals et al. 2016) | 63.08 $\pm$ 0.80 | 75.99 $\pm$ 0.60 | 68.50 $\pm$ 0.92 | 80.60 $\pm$ 0.71
E3BM (Liu, Schiele, and Sun 2020) | 63.80 $\pm$ 0.40 | 80.10 $\pm$ 0.30 | 71.20 $\pm$ 0.40 | 85.30 $\pm$ 0.30
CAN (Hou et al. 2019) | 63.85 $\pm$ 0.48 | 79.44 $\pm$ 0.34 | 69.89 $\pm$ 0.51 | 84.23 $\pm$ 0.37
CTM (Li et al. 2019) | 64.12 $\pm$ 0.82 | 80.51 $\pm$ 0.13 | 68.41 $\pm$ 0.39 | 84.28 $\pm$ 1.73
P-Transfer (Shen et al. 2021) | 64.21 $\pm$ 0.77 | 80.38 $\pm$ 0.59 |  - |  -
RFS (Tian et al. 2020) | 64.82 $\pm$ 0.60 | 82.14 $\pm$ 0.43 | 71.52 $\pm$ 0.69 | 86.03 $\pm$ 0.49
ConstellationNet (Xu et al. 2020) | 64.89 $\pm$ 0.23 | 79.95 $\pm$ 0.17 |  - |  -
FRN (Wertheimer, Tang, and Hariharan 2021) | 66.45 $\pm$ 0.19 | 82.83 $\pm$ 0.13 | 71.16 $\pm$ 0.22 | 86.01 $\pm$ 0.15
infoPatch (Liu et al. 2021) | 67.67 $\pm$ 0.45 | 82.44 $\pm$ 0.31 | 71.51 $\pm$ 0.52 | 85.44 $\pm$ 0.35
ProtoNet (Snell, Swersky, and Zemel 2017) | 61.83 $\pm$ 0.20 | 79.86 $\pm$ 0.14 | 66.84 $\pm$ 0.23 | 84.54 $\pm$ 0.16
ProtoNet + DFR | 64.84 $\pm$ 0.20 | 81.10 $\pm$ 0.14 | 70.22 $\pm$ 0.23 | 84.74 $\pm$ 0.16
DeepEMD (Zhang et al. 2020) | 64.93 $\pm$ 0.29 | 81.73 $\pm$ 0.57 | 70.47 $\pm$ 0.33 | 84.76 $\pm$ 0.61
DeepEMD + DFR | 65.41 $\pm$ 0.28 | 82.18 $\pm$ 0.55 | 71.56 $\pm$ 0.31 | 86.23 $\pm$ 0.58
FEAT (Ye et al. 2020) | 66.52 $\pm$ 0.20 | 81.46 $\pm$ 0.14 | 70.30 $\pm$ 0.23 | 84.55 $\pm$ 0.16
FEAT + DFR | 67.74 $\pm$ 0.86 | 82.49 $\pm$ 0.57 | 71.31 $\pm$ 0.93 | 85.12 $\pm$ 0.64
Table 1: Few-shot classification accuracy ($\%$) averaged over mini-ImageNet
and tiered-ImageNet with the ResNet backbone.

### Why It Works

DFR framework aims to extract only class-related information for
classification. Different from other attempts towards more adaptive embedding
using attention mechanism (Hou et al. 2019; Li et al. 2019; Ye et al. 2020),
our classification and variation branches play as the adversaries by
minimizing $L_{cls}$ and $L_{dis}$ simultaneously. In practice, the
classification and variation features of an image are always complementary,
thus the image reconstruction quality is enforced after fusion by minimizing
$L_{rec}$. It is essential to preserve the image representation in DFR for
few-shot classification: as the class-specific features can be task-varying
thus hard to be generalized, any information loss throughout the inter-state
flow may potentially limit the model performance. Such design is contrast to
the classic feature embedding for few-shot learning, in which image features
are always projected onto the lower-dimensional manifolds (Simon et al. 2020).
Our $E_{cls}$ feature has much lower dimension comparing to the $E_{var}$
feature, as the class-irrelevant information (e.g., image style and
background) are typically excessive. To this end, a more restrictive
classification feature will significantly reduce the model bias, thus to
enhance its generalizability in few-shot tasks.

We visualize the feature distributions w/o and w/ DFR using t-SNE (Van der
Maaten and Hinton 2008) to verify our intuition. Figure 3 (a) shows that the
learned features extracted from the ResNet-12 backbone are less discriminative
without using the DFR framework. While when applying the DFR framework, the
classification branch clusters in Figure 3 (b) are more separable from each
other, and the output features of the variation branch in Figure 3 (c)
contains more class-irrelevant information that meets our expectations.

## Experiment

We conduct extensive experiments on two few-shot benchmarks, i.e., Mini-
ImageNet and Tiered-ImageNet on general few-shot classification tasks to
evaluate the performance of our proposed DFR framework. After that, we
introduce a novel FS-DomainNet dataset with the proposed two evaluation
settings for benchmarking few-shot domain generalization task (FS-DG).
Moreover, we evaluate the performance of DFR on CUB-200-2011 benchmark on
fine-grained few-shot classification task. 111The code of the proposed DFR
model and FS-DomainNet dataset will be available on
https://github.com/chengcv/DFRFS..

### Implementation Details

We use ResNet-12 network (Lee et al. 2019) as our backbone $E_{cls}$ for
classification branch and set the number of channels as $[64,160,320,640]$,
which are similar to the competing methods. The encoder $E_{var}$ consists of
four convolutional blocks and two residual blocks. The decoder contains a two-
layer MLP block and a decoder network $D$ with residual blocks and upscale
convolutional blocks. The I/O channel numbers of variation encoder and decoder
are all set to $128$. The level ratio $\lambda$ of GRL layer is set to $1$.

Data augmentation including resizing, random cropping, color jitter and random
flipping following (Ye et al. 2020) are applied for all methods in training.
Our models are all trained using SGD optimizer, with the weight decay as
$5e{-4}$, and the momentum as $0.9$.

We conduct experiments under both $5$-way $1$-shot and $5$-way $5$-shot
settings with $15$ query images each class, i.e., $5\times(1\,and\,5)+5\times
15$ samples for $1$-shot and $5$-shot tasks, respectively. We report the mean
accuracy of randomly sampled $10k$ tasks as well as the $95\%$ confidence
intervals on the testing set as mentioned in (Ye et al. 2020; Zhang et al.
2020). To verify the effectiveness of our proposed DFR framework, we combined
DFR with three few-shot algorithms: a commonly used baseline ProtoNet (Snell,
Swersky, and Zemel 2017), two state-of-the-art methods DeepEMD (Zhang et al.
2020) and FEAT (Ye et al. 2020).222We utilized the official codes released by
the authors, for implementations of ProtoNet, DeepEMD and FEAT and the
corresponding DFR models. The results are all obtained by following the
unified setting for fair comparison, which may not exactly match with the
results reported in their original papers. Note that we only adopt the FCN
version of DeepEMD for comparison over all datasets.

### General Few-shot Classification

We first conduct experiments on two general few-shot benchmarks: mini-ImageNet
and tiered-ImageNet.

Mini-ImageNet. Mini-ImageNet (Vinyals et al. 2016) is a subset of the
ILSVRC-12 challenge (Krizhevsky, Sutskever, and Hinton 2012) proposed for few-
shot classification. It contains 100 diverse classes with 600 images of size
$84\times 84\times 3$ in each category. Following the class split setting
(Ravi and Larochelle 2017) used in previous works, all 100 classes are divided
into 64, 16 and 20 classes for training, validation and testing, respectively.

Tiered-ImageNet. Similar to mini-ImageNet, Tiered-ImageNet (Ren et al. 2018)
is also a subset of ILSVRC-12, which contains more classes that are organized
in a hierarchical structure, i.e., 608 classes from 34 top categories. We
follow the setups proposed by (Ren et al. 2018), and split 608 categories into
351, 97 and 160 for training, validation and testing, respectively.

The classification results are shown in Table 1. It is clear that FEAT+DFR
achieves state-of-the-art results on mini-ImageNet benchmark, while
DeepEMD+DFR achieves state-of-the-art results on tiered-ImageNet benchmark.
Moreover, we observe that the improvements by DFR remain inconsistent for all
baselines. By adopting the DFR framework, the 5-way 1-shot accuracies by
ProtoNet are increased by $3.0\%$ and $3.4\%$ on mini-ImageNet and tiered-
ImageNet, respectively, which are even comparable to more sophisticated
methods. For the other two methods DeepEMD and FEAT, which are the current
state-of-the-art FS methods, their FS classification results can still be
further boosted by $1\%$ in average, after applying the DFR framework.

Data Split | Class | Domain
---|---|---
Train | Test | Source | Target
Classic DG Setting | $-$ | $-$ | $\triangle$ | $\diamondsuit$
Classic FS Setting | $\triangle$ | $\mathcal{S},\mathcal{Q}$ | $-$ | $-$
FS-DG Setting A | $\triangle$ | $\mathcal{S},\mathcal{Q}$ | $\triangle$ | $\mathcal{S},\mathcal{Q}$
FS-DG Setting B | $\triangle$ | $\mathcal{S},\mathcal{Q}$ |  $\triangle$ $\mathcal{S}$ | $\mathcal{Q}$
Table 2: Comparison of different settings for DG and FS tasks. $\triangle$:
training data selection for DG and FS tasks. $\diamondsuit$: testing data
selection for DG tasks. $\mathcal{S},\mathcal{Q}$: FS support and query data
selection. The general FS and DG tasks do not split the domain and class sets,
respectively.

### Few-shot Domain Generalization

Domain generalization (DG) aims to learn a domain-agnostic model from multiple
sources that can classify data from any target domain. DG tasks become more
challenging when there exists a class gap (besides domain gap) between the
training and testing sets, i.e., DG under the few-shot setting. General few-
shot learning does not consider the influence caused by the domain gap, thus
the DG models can hardly be generalized to unseen domains. In this work, we
consider a more challenging Few-Shot Domain Generalization (FS-DG) problem,
i.e., both domain and class gaps exist between the training (source) and
testing (target) sets. In our FS-DG experiments (under both the Setting A and
B), only the training samples from the source domains are selected in
training. Specifically, an $N$-way $K$-shot FS-DG task contains support and
query samples from $N$ classes on the source domain in the meta-training step,
and then the trained model is to predict the query data label out of the
testing classes on the target domain. Here we propose two FS-DG evaluation
settings based on different domains of support set $\mathcal{S}$ as: (1)
Setting A: Support set is only from the target domain and (2) Setting B:
Support set is only from the source domain. Both settings can evaluate the
generalizability of the model, i.e., ability to extract domain-invariant and
class-specific features. Recent works (Ye et al. 2020; Du et al. 2021) also
attempted simple FS-DG tasks to evaluate their proposed FS models. However,
only preliminary results are reported following the simple setting (i.e.,
Setting B in Table 2) without comprehensive investigation of the effect of
domain gap on novel classes (test class set). We further conduct experiments
with full evaluation settings to validate the proposed DFR for FS-DG tasks
using a novel FS-DomainNet Benchmark.

#### FS-DomainNet Benchmark.

We propose FS-DomainNet for benchmarking few-shot domain generalization.
Different from the few-shot domainnet (Du et al. 2021) which only contains 200
classes with 1000 images each class, FS-DomainNet captures a much larger
subset of DomainNet (Peng et al. 2019), i.e., 569010 images from six distinct
domains (i.e., Sketch, Quickdraw, Real, Painting, Clipart and Infograph) with
345 different categories of objects from 24 divisions. We reorganize it for
few-shot learning and select all categories (i.e., 527156 images of 299
classes) that include at least the number of samples (i.e., 20) required by
the $5$-shot setting on each domain.  Then we split 299 categories into 191,
47 and 61 for training, validation and testing, respectively, while
maintaining the consistency of class split on each domain. More detailed
descriptions and data examples of FS-DomainNet are included in our
Supplementary Materials. Different from existing few-shot benchmarks, FS-
DomainNet additionally includes objects that are collected from multiple
domains considering both the domain and class gaps, and the sample size varies
greatly between different categories to enable more challenging FS-DG task
settings. Additionally, FS-DomainNet can also be utilized for the few-shot
domain adaptation and general few-shot classification tasks.

Method | Setting A | Setting B
---|---|---
1-shot | 5-shot | 1-shot | 5-shot
MatchNet | 45.23 | 54.92 | 40.61 | 49.09
ProtoNet | 47.96 | 66.64 | 48.70 | 67.96
ProtoNet+DFR | 49.29 | 68.73 | 49.76 | 70.34
DeepEMD | 53.20 | 70.59 | 51.97 | 70.62
DeepEMD+DFR | 54.47 | 71.60 | 54.06 | 72.33
FEAT | 51.83 | 69.26 | 52.46 | 71.54
FEAT+DFR | 52.58 | 69.93 | 54.75 | 71.91
Table 3: FS-DG Classification accuracy ($\%$) averaged on FS-DomainNet with
two evaluation settings under the 5-way setting.

#### Experimental Setups.

Following the classic DG setting, we choose five out of six domains from FS-
DomainNet as the source domains and the remaining one as the target domain. We
report the average FS-DG accuracies over the splits with each of the six
domains as the target domain.

For $1$-shot tasks, we randomly select one support sample only from one random
domain for each class; For $5$-shot tasks, we select one labeled sample of
each source domain for each class, i.e., each meta-task contains the same
support samples of each domain. For query samples of each class with multi-
domains under both $1$\- and $5$-shot settings, we select the same number of
query samples from each domain, i.e., $\|Q\|=3\times 5=15$.

#### Results.

Table 3 shows the average accuracy of six target domains on the FS-DomainNet
benchmark for two evaluation settings. It is clear that DFR can provide
consistent improvement on classification accuracies for all FS baselines under
both settings. Besides, DFR provides more significant boosting for FS-DG
performance under the Setting B, thanks to its effective disentanglement of
class-specific features. Comparing to 5-shot tests, DFR provides less help for
1-shot tests, as learning from only one support sample from a random source
domain for each category in meta-task is more challenging.

It is worth noting that both ProtoNet and FEAT perform better under the
Setting B, while DeepEMD generates better results under the Setting A,
comparing to the other setting. It is due to the unique design of DeepEMD by
adapting the channel-wise EMD metric based on the feature maps, which
inexplicitly incorporates the similarity of domain information. In the FS-DG
setting A, the support and query data are from the same domain, which is, in
fact, advantageous for DeepEMD, while the domain gap between support and query
sets, on the contrary, degrades the DeepEMD performance under the Setting B.
After applying the proposed DFR, the feature map in the classification branch
removes the interference information, which can always improve DeepEMD under
both settings. More experiment results and analysis on the FS-DomainNet
dataset can be found in our Supplementary Materials.

Method | CUB
---|---
5-way 1-shot | 5-way 5-shot
RelationNet (Sung et al. 2018) | 66.20 $\pm$ 0.99 | 82.30 $\pm$ 0.58
MAML (Finn, Abbeel, and Levine 2017) | 67.28 $\pm$ 1.08 | 83.47 $\pm$ 0.59
MatchNet (Vinyals et al. 2016) | 71.87 $\pm$ 0.85 | 85.08 $\pm$ 0.57
COMET (Cao, Brbic, and Leskovec 2021) | 72.20 $\pm$ 0.90 | 87.60 $\pm$ 0.50
P-Transfer (Shen et al. 2021) | 73.88 $\pm$ 0.92 | 87.81 $\pm$ 0.48
ProtoNet (Snell, Swersky, and Zemel 2017) | 72.25 $\pm$ 0.21 | 87.47 $\pm$ 0.13
ProtoNet+DFR | 73.52 $\pm$ 0.21 | 87.90 $\pm$ 0.13
DeepEMD (Zhang et al. 2020) | 74.88 $\pm$ 0.30 | 88.52 $\pm$ 0.52
DeepEMD+DFR | 76.78 $\pm$ 0.29 | 89.19 $\pm$ 0.52
FEAT (Ye et al. 2020) | 75.68 $\pm$ 0.20 | 87.91 $\pm$ 0.13
FEAT+DFR | 77.14 $\pm$ 0.21 | 88.97 $\pm$ 0.13

Table 4: Fine-grained few-shot classification accuracy ($\%$) averaged on CUB
with the ResNet backbone.

### Fine-grained Few-shot Classification

We further evaluate DFR on a fine-grained benchmark, i.e., Caltech-UCSD Birds
200-2011 (CUB) (Wah et al. 2011) which was initially proposed for fine-grained
image classification, which contains 200 different birds with 11788 images.
Following the split in (Chen et al. 2019; Hilliard et al. 2018), 200 classes
are divided into 100, 50 and 50 for training, validation and testing,
respectively. We also pre-process the data by cropping each image with the
provided bounding box according to the prior work (Ye et al. 2020; Wertheimer,
Tang, and Hariharan 2021).

Table 4 reports the fine-grained few-shot classification results with both
5-way 1-shot and 5-way 5-shot tests. Comparing to the general and multi-domain
few-shot benchmarks that contain significant differences between the
categories, fine-grained classification only includes minor intra-class
differences. The domain information in the fine-grained dataset may contribute
to the category, making it a challenging task. It is clear that the proposed
DFR can also significantly and consistently boost all FS baselines, with
$0.5\%$ to $1.9\%$ additional improvements on CUB dataset. It demonstrates
that DFR can effectively remove the excursive features, and thus highlight the
subtle traits which are critical for fine-grained FS classification.

### Ablation Study

DFR | $\lambda_{1}$ | $\lambda_{2}$ | $\lambda_{3}$ | 1-shot | 5-shot
---|---|---|---|---|---
✗ | - | - | - | 66.52 | 81.46
✓ | 1.0 | - | 1.0 | 66.75 | 81.98
✓ | 1.0 | 1.0 | - | 66.99 | 82.16
✓ | 1.0 | 1.0 | 1.0 | 67.74 | 82.49
Table 5: Ablation study on mini-ImageNet dataset of FEAT with the proposed DFR
framework.

We investigate the weights in our formulation and incorporate FEAT as the
baseline method. Table 5 shows that FEAT+DFR achieves the best performance
when weighting parameters are all set to $1.0$. Compared with $L_{rec}$ and
$L_{tran}$, the discriminative loss $L_{dis}$ has a more significant impact on
performance as it affects the class-specific information removed from the
variation branch, which is directly related to the classification ability of
the classification branch. Overall, we find that the performance is minimally
affected by loss weight which also shows the robustness of our framework.

## Conclusion

We propose a novel and effective Disentangled Feature Representation (DFR)
framework for few-shot image classification. Unlike the feature embeddings
which may encode the excursive image information, such as background and
domain, the proposed DFR aims to extract the class-specific features which is
essential in most few-shot learning pipelines. Furthermore, to tackle the
challenges of the domain gap in few-shot learning, we propose a novel
benchmarking dataset (FS-DomainNet) for the few-shot domain generalization
task. We have studied the importance of applying DFR in few-shot tasks by
visualizing the t-SNE of the extracted features w/o DFR and disentangled
features from the classification and variation branches. Experimental results
on four datasets, including three tasks (general image classification, fine-
grained classification, and domain generalization) under the few-shot
settings, evaluate the effectiveness of the proposed DFR framework.

## References

  * Cao, Brbic, and Leskovec (2021) Cao, K.; Brbic, M.; and Leskovec, J. 2021.  Concept Learners for Few-Shot Learning.  In _International Conference on Learning Representations_.
  * Chen et al. (2019) Chen, W.-Y.; Liu, Y.-C.; Kira, Z.; Wang, Y.-C. F.; and Huang, J.-B. 2019.  A Closer Look at Few-shot Classification.  In _International Conference on Learning Representations_.
  * Chen et al. (2016) Chen, X.; Duan, Y.; Houthooft, R.; Schulman, J.; Sutskever, I.; and Abbeel, P. 2016\.  InfoGAN: interpretable representation learning by information maximizing generative adversarial nets.  In _Proceedings of the 30th International Conference on Neural Information Processing Systems_ , 2180–2188.
  * Du et al. (2021) Du, Y.; Zhen, X.; Shao, L.; and Snoek, C. G. M. 2021.  MetaNorm: Learning to Normalize Few-Shot Batches Across Domains.  In _International Conference on Learning Representations_.
  * Fei-Fei, Fergus, and Perona (2006) Fei-Fei, L.; Fergus, R.; and Perona, P. 2006.  One-shot learning of object categories.  _IEEE transactions on pattern analysis and machine intelligence_ , 28(4): 594–611.
  * Finn, Abbeel, and Levine (2017) Finn, C.; Abbeel, P.; and Levine, S. 2017.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _International Conference on Machine Learning_ , 1126–1135. PMLR.
  * Gidaris and Komodakis (2019) Gidaris, S.; and Komodakis, N. 2019.  Generating classification weights with gnn denoising autoencoders for few-shot learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 21–30.
  * Hariharan and Girshick (2017) Hariharan, B.; and Girshick, R. 2017.  Low-shot visual recognition by shrinking and hallucinating features.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 3018–3027.
  * Hilliard et al. (2018) Hilliard, N.; Phillips, L.; Howland, S.; Yankov, A.; Corley, C. D.; and Hodas, N. O. 2018.  Few-shot learning with metric-agnostic conditional embeddings.  _arXiv preprint arXiv:1802.04376_.
  * Hou et al. (2019) Hou, R.; Chang, H.; Ma, B.; Shan, S.; and Chen, X. 2019.  Cross attention network for few-shot classification.  In _Proceedings of the 33rd International Conference on Neural Information Processing Systems_ , 4003–4014.
  * Hsieh et al. (2018) Hsieh, J.-T.; Liu, B.; Huang, D.-A.; Fei-Fei, L.; and Niebles, J. C. 2018.  Learning to decompose and disentangle representations for video prediction.  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , 515–524.
  * Jaderberg et al. (2015) Jaderberg, M.; Simonyan, K.; Zisserman, A.; and Kavukcuoglu, K. 2015.  Spatial transformer networks.  In _Proceedings of the 28th International Conference on Neural Information Processing Systems-Volume 2_ , 2017–2025.
  * Johnson, Alahi, and Fei-Fei (2016) Johnson, J.; Alahi, A.; and Fei-Fei, L. 2016.  Perceptual losses for real-time style transfer and super-resolution.  In _European conference on computer vision_ , 694–711. Springer.
  * Krizhevsky, Sutskever, and Hinton (2012) Krizhevsky, A.; Sutskever, I.; and Hinton, G. E. 2012.  Imagenet classification with deep convolutional neural networks.  In _Advances in neural information processing systems_ , 1097–1105.
  * Lee et al. (2020) Lee, H.-Y.; Tseng, H.-Y.; Mao, Q.; Huang, J.-B.; Lu, Y.-D.; Singh, M.; and Yang, M.-H. 2020.  Drit++: Diverse image-to-image translation via disentangled representations.  _International Journal of Computer Vision_ , 128(10): 2402–2417.
  * Lee et al. (2019) Lee, K.; Maji, S.; Ravichandran, A.; and Soatto, S. 2019.  Meta-learning with differentiable convex optimization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 10657–10665.
  * Li et al. (2019) Li, H.; Eigen, D.; Dodge, S.; Zeiler, M.; and Wang, X. 2019.  Finding task-relevant features for few-shot learning by category traversal.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 1–10.
  * Li et al. (2020a) Li, K.; Zhang, Y.; Li, K.; and Fu, Y. 2020a.  Adversarial feature hallucination networks for few-shot learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 13470–13479.
  * Li et al. (2020b) Li, X.; Jin, X.; Lin, J.; Liu, S.; Wu, Y.; Yu, T.; Zhou, W.; and Chen, Z. 2020b.  Learning Disentangled Feature Representation for Hybrid-distorted Image Restoration.  In _European Conference on Computer Vision_ , 313–329. Springer.
  * Li et al. (2021) Li, X.; Xu, Z.; Wei, K.; and Deng, C. 2021.  Generalized Zero-Shot Learning via Disentangled Representation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, 1966–1974.
  * Liu et al. (2021) Liu, C.; Fu, Y.; Xu, C.; Yang, S.; Li, J.; Wang, C.; and Zhang, L. 2021.  Learning a Few-shot Embedding Model with Contrastive Learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, 8635–8643.
  * Liu et al. (2019) Liu, M.-Y.; Huang, X.; Mallya, A.; Karras, T.; Aila, T.; Lehtinen, J.; and Kautz, J. 2019.  Few-shot unsupervised image-to-image translation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 10551–10560.
  * Liu, Schiele, and Sun (2020) Liu, Y.; Schiele, B.; and Sun, Q. 2020.  An Ensemble of Epoch-wise Empirical Bayes for Few-shot Learning.  In _European Conference on Computer Vision (ECCV)_.
  * Nichol, Achiam, and Schulman (2018) Nichol, A.; Achiam, J.; and Schulman, J. 2018.  On first-order meta-learning algorithms.  _arXiv preprint arXiv:1803.02999_.
  * Oreshkin, López, and Lacoste (2018) Oreshkin, B. N.; López, P. R.; and Lacoste, A. 2018.  TADAM: Task dependent adaptive metric for improved few-shot learning.  In _NeurIPS_.
  * Peng et al. (2019) Peng, X.; Bai, Q.; Xia, X.; Huang, Z.; Saenko, K.; and Wang, B. 2019.  Moment matching for multi-source domain adaptation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 1406–1415.
  * Prabhudesai et al. (2021) Prabhudesai, M.; Lal, S.; Patil, D.; Tung, H.-Y.; Harley, A. W.; and Fragkiadaki, K. 2021.  Disentangling 3D Prototypical Networks for Few-Shot Concept Learning.  In _International Conference on Learning Representations_.
  * Ravi and Larochelle (2017) Ravi, S.; and Larochelle, H. 2017.  Optimization as a model for few-shot learning.  In _International Conference on Learning Representations_.
  * Ren et al. (2018) Ren, M.; Triantafillou, E.; Ravi, S.; Snell, J.; Swersky, K.; Tenenbaum, J. B.; Larochelle, H.; and Zemel, R. S. 2018.  Meta-learning for semi-supervised few-shot classification.  In _ICLR_.
  * Rusu et al. (2019) Rusu, A. A.; Rao, D.; Sygnowski, J.; Vinyals, O.; Pascanu, R.; Osindero, S.; and Hadsell, R. 2019.  Meta-Learning with Latent Embedding Optimization.  In _International Conference on Learning Representations_.
  * Shen et al. (2021) Shen, Z.; Liu, Z.; Qin, J.; Savvides, M.; and Cheng, K.-T. 2021.  Partial Is Better Than All: Revisiting Fine-tuning Strategy for Few-shot Learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, 9594–9602.
  * Simon et al. (2020) Simon, C.; Koniusz, P.; Nock, R.; and Harandi, M. 2020.  Adaptive subspaces for few-shot learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 4136–4145.
  * Snell, Swersky, and Zemel (2017) Snell, J.; Swersky, K.; and Zemel, R. 2017.  Prototypical networks for few-shot learning.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , 4080–4090.
  * Sung et al. (2018) Sung, F.; Yang, Y.; Zhang, L.; Xiang, T.; Torr, P. H.; and Hospedales, T. M. 2018\.  Learning to Compare: Relation Network for Few-Shot Learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_.
  * Tang, Wertheimer, and Hariharan (2020) Tang, L.; Wertheimer, D.; and Hariharan, B. 2020.  Revisiting pose-normalization for fine-grained few-shot recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 14352–14361.
  * Tian et al. (2020) Tian, Y.; Wang, Y.; Krishnan, D.; Tenenbaum, J. B.; and Isola, P. 2020.  Rethinking few-shot image classification: a good embedding is all you need?  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16_ , 266–282. Springer.
  * Tokmakov, Wang, and Hebert (2019) Tokmakov, P.; Wang, Y.-X.; and Hebert, M. 2019.  Learning compositional representations for few-shot recognition.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 6372–6381.
  * Van der Maaten and Hinton (2008) Van der Maaten, L.; and Hinton, G. 2008.  Visualizing data using t-SNE.  _Journal of machine learning research_ , 9(11).
  * Vinyals et al. (2016) Vinyals, O.; Blundell, C.; Lillicrap, T.; Kavukcuoglu, K.; and Wierstra, D. 2016\.  Matching networks for one shot learning.  In _Proceedings of the 30th International Conference on Neural Information Processing Systems_ , 3637–3645.
  * Wah et al. (2011) Wah, C.; Branson, S.; Welinder, P.; Perona, P.; and Belongie, S. 2011.  The caltech-ucsd birds-200-2011 dataset.
  * Wang et al. (2018) Wang, Y.-X.; Girshick, R.; Hebert, M.; and Hariharan, B. 2018.  Low-shot learning from imaginary data.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 7278–7286.
  * Wertheimer, Tang, and Hariharan (2021) Wertheimer, D.; Tang, L.; and Hariharan, B. 2021.  Few-Shot Classification With Feature Map Reconstruction Networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 8012–8021.
  * Xu et al. (2020) Xu, W.; Wang, H.; Tu, Z.; et al. 2020.  Attentional Constellation Nets for Few-Shot Learning.  In _International Conference on Learning Representations_.
  * Yang, Liu, and Xu (2021) Yang, S.; Liu, L.; and Xu, M. 2021.  Free Lunch for Few-shot Learning: Distribution Calibration.  In _International Conference on Learning Representations_.
  * Ye et al. (2020) Ye, H.-J.; Hu, H.; Zhan, D.-C.; and Sha, F. 2020.  Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 8808–8817.
  * Zhang et al. (2020) Zhang, C.; Cai, Y.; Lin, G.; and Shen, C. 2020.  DeepEMD: Few-Shot Image Classification With Differentiable Earth Mover’s Distance and Structured Classifiers.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Zhao et al. (2021) Zhao, J.; Yang, Y.; Lin, X.; Yang, J.; and He, L. 2021.  Looking Wider for Better Adaptive Representation in Few-Shot Learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, 10981–10989.