# HyperShot: Few-Shot Learning by Kernel HyperNetworks

Marcin Sendera  Marcin Przewieźlikowski  Konrad Karanowski  Maciej Zieba
Jacek Tabor  Przemysław Spurek

###### Abstract

Few-shot models aim at making predictions using a minimal number of labeled
examples from a given task. The main challenge in this area is the one-shot
setting where only one element represents each class. We propose HyperShot \-
the fusion of kernels and hypernetwork paradigm. Compared to reference
approaches that apply a gradient-based adjustment of the parameters, our model
aims to switch the classification module parameters depending on the task’s
embedding. In practice, we utilize a hypernetwork, which takes the aggregated
information from support data and returns the classifier’s parameters
handcrafted for the considered problem. Moreover, we introduce the kernel-
based representation of the support examples delivered to hypernetwork to
create the parameters of the classification module. Consequently, we rely on
relations between embeddings of the support examples instead of direct feature
values provided by the backbone models. Thanks to this approach, our model can
adapt to highly different tasks.

few-shot learning, meta-learning, kernels methods, hypernetworks

##  1 Introduction

Current Artificial Intelligence techniques cannot rapidly generalize from a
few examples. This common inability is because most deep neural networks must
be trained on large-scale data. In contrast, humans can learn new tasks
quickly by utilizing what they learned in the past. Few-shot learning models
try to fill this gap by learning how to learn from a limited number of
examples. Few-shot learning is the problem of making predictions based on a
small number of samples. The goal of few-shot learning is not to recognize a
fixed set of labels but to learn how to quickly adapt to new tasks with a
small amount of training data. After training, the model can classify new data
using only a few training samples.

Figure 1: The overview of HyperShot architecture. We create the kernel matrix
for features extracted from support examples, which is further processed by a
hypernetwork. The role of the hypernetwork is to create the set of parameters
for the considered task. The target network further uses the parameters to
classify the query examples.

The most popular approaches for few-shot learning use the meta-learning
framework. We sample few-shot classification tasks from training samples
belonging to the base classes and optimize the model to perform well on these
tasks. We work with $N$-way and $K$-shot tasks, where we have $N$ classes with
$K$ support samples and $Q$ query samples in each category. The goal is to
classify these $N\times Q$ query samples into the $N$ classes based on the
$N\times K$ support samples.

One of the most common approaches to Few-shot learning is Model-Agnostic Meta-
Learning (MAML) (Finn et al., 2017), where the model is trained to adapt to
new Few-shot learning tasks quickly. The main idea is to produce universal
weights which can be rapidly updated to solve new small tasks. The model’s
main limitation is the complicated optimization procedure that uses an inner
and outer loop, which can be seen as second-order optimization. Moreover, such
an optimization procedure only partially implements the _learn how to learn_
paradigm, which assumes the model learns some rules to adapt to the new tasks.
The induction of such rules by MAML is limited by gradient-based optimization
applied to adjust the parameters of the classification module to the new task.
In practice, we can realize learning rules which are based on gradient
optimization. New classification module parameters are obtained by gradient
descent.

This paper introduces a new strategy, which solves the above problem with
restriction only to gradient-based decision rules. Our goal is to mimic the
human learning process. First, we look at the entire support set and extract
the information to distinguish objects in classes. Then, based on the relation
between features, we create the decision rules.

We combine the Hypernetworks paradigm with kernel methods to realize such a
scenario, see Fig. 1. Hypernetworks, introduced in (Ha et al., 2016), are
defined as neural models that generate weights for a separate target network
solving a specific task. In our approach, the Hypernetwork aggregates the
information from the support set and produces weights of the target model,
classifying the query set.

Kernel methods realize the first part of the process. For each of the few-shot
tasks, we extract the features from the support set through the backbone
architecture and calculate kernel values between them. Then we use a
Hypernetwork architecture – a neural network that takes kernel representation
and produces decision rules in the form of a classifier (target network)
dedicated to the query set.

Such a solution realizes the _learn how to learn_ paradigm. Hypernetwork,
which produces the decision rules from kernel representation of the support
set, is trained by gradient-based method and can be seen as a classical
learning process. However, the weights of the target network (decision rules)
can realize different strategies and lay in varying weight space parts. Our
approach allows training a universal architecture model, which can be quickly
updated to new tasks without any second-order procedure (inner and outer
loop).

We perform an extensive experimental study of our approach by benchmarking it
on various one-shot and few-shot image classification tasks. We find that
HyperShot demonstrates high accuracy in all tasks, performing comparably or
better than the other recently proposed methods. Moreover, HyperShot shows a
strong ability to generalize, as evidenced by its performance on cross-domain
classification tasks.

The contributions of our work can be summarized as follows:

  * •

In this paper, we propose a model which realizes the _learn how to learn_
paradigm by modeling learning rules which are not based on gradient
optimization and can produce completely different strategies.

  * •

We propose a new approach to solve the few-shot learning problem by
aggregating information from the support set and directly producing weights
from the neural network dedicated to the query set.

  * •

We propose HyperShot, which uses the Hypernetwork paradigm to produce the
weights dedicated for each task.

##  2 HyperShot: Hypernetwork for few-shot learning

In this section, we present our HyperShot model for few-shot learning.

###  2.1 Background

#### Few-shot learning

The terminology describing the few-shot learning setup is dispersive due to
the colliding definitions used in the literature. For a unified taxonomy, we
refer the reader to (Chen et al., 2019; Wang et al., 2020). Here, we use the
nomenclature derived from the meta-learning literature, which is the most
prevalent at the time of writing. Let:

$\mathcal{S}=\\{(\mathbf{x}_{l},\mathbf{y}_{l})\\}_{l=1}^{L}$ (1)

be a support-set containing input-output pairs, with $L$ examples with the
equal class distribution. In the one-shot scenario, each class is represented
by a single example, and $L=K$, where $K$ is the number of the considered
classes in the given task. Whereas, for few-shot scenarios, each class usually
has from $2$ to $5$ representatives in the support set $\mathcal{S}$. Let:

$\mathcal{Q}=\\{(\mathbf{x}_{m},\mathbf{y}_{m})\\}_{m=1}^{M}$ (2)

be a query-set (sometimes referred to in the literature as a target-set), with
$M$ examples, where $M$ is typically one order of magnitude greater than $K$.
For ease of notation, the support and query sets are grouped in a task
$\mathcal{T}=\\{\mathcal{S},\mathcal{Q}\\}$. During the training stage, the
models for few-shot applications are fed by randomly selected examples from
training set $\mathcal{D}=\\{\mathcal{T}_{n}\\}^{N}_{n=1}$, defined as a
collection of such tasks.

During the inference stage, we consider task
$\mathcal{T}_{*}=\\{\mathcal{S}_{*},\mathcal{X}_{*}\\}$, where
$\mathcal{S}_{*}$ is a support set with the known class values for a given
task, and $\mathcal{X}_{*}$ is a set of query (unlabeled) inputs. The goal is
to predict the class labels for query inputs $\mathbf{x}\in\mathcal{X}_{*}$,
assuming support set $\mathcal{S}_{*}$ and using the model trained on
$\mathcal{D}$.

#### Hypernetwork

In the canonical work (Ha et al., 2016), hyper-networks are defined as neural
models that generate weights for a separate target network solving a specific
task. The authors aim to reduce the number of trainable parameters by
designing a hyper-network with a smaller number of parameters than the target
network. Making an analogy between hyper-networks and generative models, the
authors of (Sheikh et al., 2017) use this mechanism to generate a diverse set
of target networks approximating the same function.

###  2.2 HyperShot - overview

We introduce our HyperShot - model that utilizes hypernetwork for few-shot
problems. The main idea of the proposed approach is to predict the values of
the parameters for a classification network that makes predictions on query
images given the information extracted from support examples for a given task.
Thanks to this approach, we can switch the classifier’s parameters between
completely different tasks based on the support set. The information about the
current task is extracted from the support set using parametrized kernel
function that operates on embedding space. Thanks to this approach, we use
relations among the support examples instead of taking the direct values of
the embedding values as an input to the hypernetwork. Consequently, this
approach is robust to the embedding values for new tasks far from the feature
regions observed during training. The classification of the query image is
also performed using the kernel values calculated with respect to the support
set.

The architecture of the HyperShot is provided in Fig. 2. We aim to predict the
class distribution $p(\mathbf{y}|S,\mathbf{x})$, given a query image
$\mathbf{x}$ and set of support examples
$S=\\{(\mathbf{x}_{l},y_{l})\\}_{l=1}^{K}$. First, all images from the support
set are grouped by their corresponding class values. Next, each of the images
$\mathbf{x}_{l}$ from the support set is transformed using encoding network
$E(\cdot)$, which creates low-dimensional representations of the images,
$E(\mathbf{x}_{l})=\mathbf{z}_{l}$. The constructed embeddings are sorted
according to class labels and stored in the matrix
$\mathbf{Z}_{S}=[\mathbf{z}_{\pi(1)},\dots,\mathbf{z}_{\pi(K)}]^{\mathrm{T}}$,
where $\pi(\cdot)$ is the bijective function, that satisfies $y_{\pi(l)}\leq
y_{\pi(k)}$ for $l\leq k$.

Figure 2: The general architecture of HyperShot. First, the examples from a
support set are sorted according to the corresponding class labels and
transformed by encoding network $E(\cdot)$ to obtain the matrix of ordered
embeddings of the support examples, $\mathbf{Z}_{S}$. The low-dimensional
representations stored in $\mathbf{Z}_{S}$ are further used to compute kernel
matrix $\mathbf{K}_{S,S}$. The values of the kernel matrix are passed to the
hypernetwork $H(\cdot)$ that creates the parameters $\boldsymbol{\theta}_{T}$
for the target classification module $T(\cdot)$. The query image $\mathbf{x}$
is processed by encoder $E(\cdot)$, and the vector of kernel values
$\mathbf{k}_{\mathbf{x},S}$ is calculated between query embedding
$\mathbf{z}_{\mathbf{x}}$ and the corresponding representations of support
examples, $\mathbf{Z}_{S}$. The kernel vector $\mathbf{k}_{\mathbf{x},S}$ is
further passed to target model $T(\cdot)$ to obtain the probability
distribution for the considered classes.

In the next step we calculate the kernel matrix $\mathbf{K}_{S,S}$, for vector
pairs stored in rows of $\mathbf{Z}_{S}$. To achieve this, we use the
parametrized kernel function $k(\cdot,\cdot)$, and calculate $k_{i,j}$ element
of matrix $\mathbf{K}_{S,S}$ in the following way:

$k_{i,j}=k(\mathbf{z}_{\pi(i)},\mathbf{z}_{\pi(j)}).$ (3)

The kernel matrix $\mathbf{K}_{S,S}$ represents the extracted information
about the relations between support examples for a given task. The matrix
$\mathbf{K}_{S,S}$ is further reshaped to the vector format and delivered to
the input of the hypernetwork $H(\cdot)$. The role of the hypernetwork is to
provide the parameters $\boldsymbol{\theta}_{T}$ of target model $T(\cdot)$
responsible for the classification of the query object. Thanks to that
approach, we can switch between the parameters for entirely different tasks
without moving via the gradient-controlled trajectory, like in some reference
approaches like MAML.

The query image $\mathbf{x}$ is classified in the following manner. First, the
input image is transformed to low-dimensional feature representation
$z_{\mathbf{x}}$ by encoder $E(\mathbf{x})$. Further, the kernel vector
$\mathbf{k}_{\mathbf{x},S}$ between the query embedding and sorted support
vectors $\mathbf{Z}_{S}$ is calculated in the following way:

$\mathbf{k}_{\mathbf{x},S}=[k(\mathbf{z}_{\mathbf{x}},\mathbf{z}_{\pi(1)}),\dots,k(\mathbf{z}_{\mathbf{x}},\mathbf{z}_{\pi(K)})]^{\mathrm{T}}.$
(4)

The vector $\mathbf{k}_{\mathbf{x},S}$ is further provided on the input of
target model $T(\cdot)$ that is using the parameters $\boldsymbol{\theta}_{T}$
returned by hypernetwork $H(\cdot)$. The target model returns the probability
distribution $p(\mathbf{y}|S,\mathbf{x})$ for each class considered in the
task.

Figure 3: Simple 2D example illustrating the application of cosine kernel for
HyperShot. We consider the two support examples from different classes
represented by vectors $\mathbf{f_{1}}$ and $\mathbf{f_{2}}$. For this simple
scenario, the input of hypernetwork is represented simply by the cosine of
$\alpha$, which is an angle between vectors $\mathbf{f}_{1}$ and
$\mathbf{f}_{2}$. We aim at classifying the query example $\mathbf{x}$
represented by a vector $\mathbf{f}_{\mathbf{x}}$. Considering our approach,
we deliver to the target network $T(\cdot)$ the cosine values of angles
between first ($\alpha_{\mathbf{x},1}$) and second ($\alpha_{\mathbf{x},2}$)
support vectors and classify the query example using the weights
$\boldsymbol{\theta}_{T}$ created by hypernetwork $H(\cdot)$ from
$\cos{\alpha}$ (remaining components on the diagonal of $\mathbf{K}_{S,S}$ are
constant for cosine kernel).

###  2.3 Kernel function

One of the key components of our approach is a kernel function
$k(\cdot,\cdot)$. In this work we consider the dot product of the transformed
vectors given by:

$k(\mathbf{z}_{1},\mathbf{z}_{2})=\mathbf{f}(\mathbf{z}_{1})^{\mathrm{T}}\mathbf{f}(\mathbf{z}_{2}),$
(5)

where $\mathbf{f}(\cdot)$ can be a parametrized transformation function,
represented by MLP model, or simply an identity operation,
$\mathbf{f}(\mathbf{z})=\mathbf{z}$. In Euclidean space this criterion can be
expressed as
$k(\mathbf{z}_{1},\mathbf{z}_{2})=||\mathbf{f}(\mathbf{z}_{1})||\cdot||\mathbf{f}(\mathbf{z}_{2})||\cos{\alpha}$,
where $\alpha$ is an angle between vectors $\mathbf{f}(\mathbf{z}_{1})$ and
$\mathbf{f}(\mathbf{z}_{2})$. The main feature of this function is that it
considers the vectors’ norms, which can be problematic for some tasks that are
outliers regarding the representations created by $\mathbf{f}(\cdot)$.
Therefore, we consider in our experiments also the cosine kernel function
given by:

$k_{c}(\mathbf{z}_{1},\mathbf{z}_{2})=\frac{\mathbf{f}(\mathbf{z}_{1})^{\mathrm{T}}\mathbf{f}(\mathbf{z}_{2})}{||\mathbf{f}(\mathbf{z}_{1})||\cdot||\mathbf{f}(\mathbf{z}_{2})||},$
(6)

that represents the normalized version dot product. Considering the
geometrical representation, $k_{c}(\mathbf{z}_{1},\mathbf{z}_{2})$ can be
expressed as $\cos{\alpha}$ (see the example given by Fig. 3). The support set
is represented by two examples from different classes, $\mathbf{f}_{1}$ and
$\mathbf{f}_{2}$. The target model parameters $\boldsymbol{\theta}_{T}$ are
created based only on the cosine value of the angle between vectors
$\mathbf{f}_{1}$ and $\mathbf{f}_{2}$. During the classification stage, the
query example is represented by $\mathbf{f}_{\mathbf{x}}$, and the
classification is applied on the cosine values of angles between
$\mathbf{f}_{\mathbf{x}}$ and $\mathbf{f}_{1}$, and $\mathbf{f}_{\mathbf{x}}$
and $\mathbf{f}_{2}$, respectively.

Algorithm 1 HyperShot - training and prediction functions

Require: Training set $\mathcal{D}=\\{\mathcal{T}_{n}\\}^{N}_{n=1}$, and
$\mathcal{T}_{*}=\\{\mathcal{S}_{*},\mathcal{X}_{*}\\}$ test task.
Parameters: $\boldsymbol{\theta}_{H}$ \- parameters ,
$\boldsymbol{\theta}_{k}$ \- kernel parameters, and $\boldsymbol{\theta}_{E}$
\- encoder parameters
Hyperparameters: $N_{train}$ \- number of training iterations, $N_{tune}$
number of tuning iterations, $\alpha$\- step size.

1:function Train($\mathcal{D}$, $\alpha$, $N_{train}$,
$\boldsymbol{\theta}_{H}$, $\boldsymbol{\theta}_{k}$,
$\boldsymbol{\theta}_{E}$)

2: while $n\leq N_{train}$ do

3: Sample task $\mathcal{T}=\\{\mathcal{S},\mathcal{Q}\\}\sim\mathcal{D}$

4: Assign support
$\mathcal{S}=\\{(\mathbf{x}_{m},\mathbf{y}_{m})\\}_{m=1}^{M}$

5: $L=-\sum_{m=1}^{M}\sum_{k=1}^{K}y_{m}^{k}\log
p(y_{m}^{k}|\mathcal{S}_{i},\mathbf{x}_{m},\boldsymbol{\theta}_{H},\boldsymbol{\theta}_{k},\boldsymbol{\theta}_{E})$

6: Update:
$\boldsymbol{\theta}_{E}\leftarrow\boldsymbol{\theta}_{E}-\alpha\nabla_{\theta_{E}}\mathcal{L}$,

7:
$\boldsymbol{\theta}_{H}\leftarrow\boldsymbol{\theta}_{H}-\alpha\nabla_{\theta_{H}}\mathcal{L}$,

8:
$\boldsymbol{\theta}_{k}\leftarrow\boldsymbol{\theta}_{k}-\alpha\nabla_{\theta_{k}}\mathcal{L}$

9: $n=n+1$

10: end while

11: return $\boldsymbol{\theta}_{H}$, $\boldsymbol{\theta}_{k}$,
$\boldsymbol{\theta}_{E}$

12:end function

13:function Predict($\mathcal{T}_{*}$, $\alpha$, $N_{tune}$,
$\boldsymbol{\theta}_{H}$, $\boldsymbol{\theta}_{k}$,
$\boldsymbol{\theta}_{E}$)

14: Create tuning task:
$\mathcal{T}_{t}=\\{\mathcal{S}_{*},\mathcal{S}_{*}\\}$

15: Tune $\boldsymbol{\hat{\theta}}_{H}$, $\boldsymbol{\hat{\theta}}_{k}$,
$\boldsymbol{\hat{\theta}}_{E}$ = Train($\mathcal{T}_{t}$, $\alpha$,
$N_{tune}$, $\boldsymbol{\theta}_{H}$, $\boldsymbol{\theta}_{k}$,
$\boldsymbol{\theta}_{E}$)

16: for each $\mathbf{x}\in\mathcal{X}_{*}$ do

17: return
$\arg\max_{\mathbf{y}}p(\mathbf{y}|\mathcal{S}_{*},\mathbf{x},\boldsymbol{\hat{\theta}}_{H},\boldsymbol{\hat{\theta}}_{k},\boldsymbol{\hat{\theta}}_{E})$

18: end for

19:end function

###  2.4 Training and prediction

The training procedure assumes the following parametrization of the model
components. The encoder $E:=E_{\boldsymbol{\theta}_{E}}$ is parametrized by
$\boldsymbol{\theta_{E}}$, the hypernetwork $H=H_{\boldsymbol{\theta}_{H}}$ by
$\boldsymbol{\theta}_{H}$, and the kernel function $k$ by
$\boldsymbol{\theta}_{k}$. We assume that training set $\mathcal{D}$ is
represented by tasks $\mathcal{T}_{i}$ composed of support $\mathcal{S}_{i}$
and query $\mathcal{Q}_{i}$ examples. The training is performed by optimizing
the cross-entropy criterion:

$L=-\sum_{\mathcal{T}_{i}\in\mathcal{D}}\sum_{m=1}^{M}\sum_{k=1}^{K}y_{i,m}^{k}\log
p(y_{i,m}^{k}|\mathcal{S}_{i},\mathbf{x}_{i,m}),$ (7)

where $(\mathbf{x}_{i,n},\mathbf{y}_{i,n})$ are examples from query set
$\mathcal{Q}_{i}$, where
$Q_{i}=\\{(\mathbf{x}_{i,m},\mathbf{y}_{i,m})\\}_{m=1}^{M}$. The distribution
for currently considered classes $p(\mathbf{y}|\mathcal{S},\mathbf{x})$ is
returned by target network T of HyperShot. During the training, we jointly
optimize the parameters $\boldsymbol{\theta}_{H}$, $\boldsymbol{\theta}_{k}$,
and $\boldsymbol{\theta}_{E}$, minimizing the $L$ loss.

During the inference stage, we consider the task $\mathcal{T}_{*}$, composed
of a set of labeled support examples $\mathcal{S}_{*}$ and a set of unlabelled
query examples represented by input values $\mathcal{X}_{*}$ that the model
should classify. We can simply take the probability values
$p(\mathbf{y}|\mathcal{S}_{*},\mathbf{x})$ assuming the given support set
$\mathcal{S}_{*}$ and single query observation $\mathbf{x}$ from
$\mathcal{X}_{*}$, using the model with trained parameters
$\boldsymbol{\theta}_{H}$, $\boldsymbol{\theta}_{k}$, and
$\boldsymbol{\theta}_{E}$. However, we observe that slightly better results
are obtained while tuning the model’s parameters on the considered task. We do
not have access to labels for query examples. Therefore we imitate the query
set for this task simply by taking support examples and creating the tuning
task $\mathcal{T}_{i}=\\{\mathcal{S}_{*},\mathcal{S}_{*}\\}$ and updating the
parameters of the model using several gradient iterations. The detailed
presentation o training and prediction procedures are provided by Algorithm 1.

###  2.5 Adaptation to few-shot scenarios

The proposed approach uses the ordering function $\pi(\cdot)$ that keeps the
consistency between support kernel matrix $\mathbf{K}_{S,S}$ and the vector of
kernel values $\mathbf{k}_{\mathbf{x},S}$ for query example $\mathbf{x}$. For
few-shot scenarios, each class has more than one representative in the support
set. As a consequence, there are various possibilities to order the feature
vectors in the support set inside the considered class. To eliminate this
issue, we propose to apply the aggregation function to the embeddings
$\mathbf{z}$ considering the support examples from the same class. Thanks to
this approach, the kernel matrix is calculated based on the aggregated values
of the latent space of encoding network $E$, making our approach independent
of the ordering among the embeddings from the same class. In experimental
studies, we examine the quality of _mean_ aggregation operation (averaged)
against simple class-wise concatenation of the embeddings (fine-grained) in
ablation studies.

##  3 Related Work

Feature transfer (Zhuang et al., 2020) is a baseline procedure for few-shot
learning and consists of pre-training the neural network and a classifier.
During meta-validation, the classifier is fine-tuned to the novel tasks. (Chen
et al., 2019) extend this idea by using cosine distance between the examples.

In recent years, a variety of meta-learning methods (Hospedales et al., 2020;
Schmidhuber, 1992; Bengio et al., 1992) have been proposed to tackle the
problem of few-shot learning. The various meta-learning architectures for few-
shot learning can be roughly categorized into several groups:

Memory-based methods (Ravi & Larochelle, 2017; Munkhdalai et al., 2018;
Santoro et al., 2016; Mishra et al., 2018; Munkhdalai & Yu, 2017; Zhen et al.,
2020) are based on the idea to train a meta-learner with memory to learn novel
concepts.

Metric-based methods meta-learn a deep representation with a metric in feature
space, such that distance between examples from the support and query set with
the same class have a small distance in such space. Some of the earliest works
exploring this notion are Matching Networks (Vinyals et al., 2016) and
Prototypical Networks (Snell et al., 2017), which form prototypes based on
embeddings of the examples from the support set in the learned feature space
and classify the query set based on the distance to those prototypes. Numerous
subsequent works aim to improve the expressiveness of the prototypes through
various techniques. (Oreshkin et al., 2018) achieve this by conditioning the
network on specific tasks, thus making the learned space task-dependent. (Hu
et al., 2021) transform embeddings of support and query examples in the
feature space to make their distributions closer to Gaussian. (Sung et al.,
2018) propose Relation Nets, which learn the metric function instead of using
a fixed one, such as Euclidean or cosine distance.

Similar to the above methods, HyperShot uses a kernel function that predicts
the relations between the examples in a given task. The key difference is that
instead of performing a nearest-neighbor classification based on the kernel
values, in HyperShot, the kernel matrix is classified by a task-specific
classifier generated by the hypernetwork.

Optimization-based methods follow the idea of an optimization process over
support set within the meta-learning framework like MetaOptNet (Lee et al.,
2019), Model-Agnostic Meta-Learning (MAML), and its extensions (Finn et al.,
2017; Nichol et al., 2018; Raghu et al., 2019; Rajeswaran et al., 2019; Finn
et al., 2018; Nichol et al., 2018). Those techniques aim to train general
models, which can adapt their parameters to the support set at hand in a small
number of gradient steps. Similar to such techniques, HyperShot also aims to
produce task-specific models but utilizes a hypernetwork instead of
optimization to achieve that goal.

Gaussian processes (Rasmussen, 2003) possess many properties useful in few-
shot learning, such as natural robustness to the limited amounts of data and
the ability to estimate uncertainty. When combined with meta-learned deep
kernels, (Patacchiola et al., 2020), Gaussian processes were demonstrated to
be a suitable tool for few-shot regression and classification, dubbed Deep
Kernel Transfer (DKT). The assumption that such a universal deep kernel has
enough data to generalize well to unseen tasks has been challenged in
subsequent works. (Wang et al., 2021) introduced a technique of learning dense
Gaussian processes by inducing variables. This approach achieves substantial
performance improvement over the alternative methods. Similarly, HyperShot
also depends on learning a model that estimates task-specific functions’
parameters. However, HyperShot employs a hypernetwork instead of a Gaussian
process to achieve that goal.

Hypernetworks (Ha et al., 2016) have been proposed as a solution to few-shot
learning problems in a number of works but have not been researched as widely
as the approaches mentioned above. Multiple works proposed various variations
of hyper-networks that predict a shallow classifier’s parameters given the
support examples (Bauer et al., 2017; Qiao et al., 2017). Subsequent works
have extended those models by calculating cosine similarity between the query
examples and the generated classifier weights (Gidaris & Komodakis, 2018) and
utilizing a probabilistic model that predicts a distribution over the
parameters suitable for the given task (Gordon et al., 2018). More recently,
(Zhmoginov et al., 2022) explored generating all of the parameters of the
target network with a transformer-based hypernetwork, but found that for
larger target networks, it is sufficient to generate only the parameters of
the final classification layer. A key characteristic of the above approaches
is that during inference, the hypernetwork predicts weights responsible for
classifying each class independently, based solely on the examples of that
class from the support set. This property makes such solutions agnostic to the
number of classes in a task, useful in practical applications. However, it
also means that the hypernetwork does not take advantage of the inter-class
differences in the task at hand.

In contrast, HyperShot exploits those differences by utilizing kernels, which
helps improve its performance.

##  4 Experiments

In the typical few-shot learning setting, making a valuable and fair
comparison between proposed models is often complicated because of the
existence of the significant differences in architectures and implementations
of known methods. In order to limit the influence of the deeper backbone
(feature extractor) architectures, we follow the unified procedure proposed by
(Chen et al., 2019).

In this section, we describe the experimental analysis and performance of the
HyperShot in the large variety of few-shot benchmarks. Specifically, we
consider both classification (see Section 4.1) and cross-domain adaptation
(see Section 4.2) tasks. Whereas the classification problems are focused on
the most typical few-shot applications, the latter cross-domain benchmarks
check the ability of the models to adapt to out-of-distribution tasks.
Additionally, we perform an ablation study of the possible adaptation
procedures of HyperShot to few-shot scenarios - presented in Section 4.3.

In all of the reported experiments, the tasks consist of 5 classes (5-way) and
1 or 5 support examples (1 or 5-shot). Unless indicated otherwise, all
compared models use a known and widely utilized backbone consisting of four
convolutional layers (each consisting of a 2D convolution, a batch-norm layer,
and a ReLU non-linearity; each layer consists of 64 channels) and have been
trained from scratch.

We report the performance of two variants of HyperShot:

  * •

HyperShot \- models generated by the hypernetworks for each task.

  * •

HyperShot + finetuning \- models generated by hypernetworks finetuned to the
support examples of each task for 10 training steps111In the case of the
finetuned hypernetworks, we tune a copy of the hypernetwork separately for
each validation task. This way, we ensure that our model does not take unfair
advantage of the validation tasks..

In all cases, we observe a modest performance boost thanks to finetuning the
hypernetwork.

Comprehensive details for each training procedure are reported in the
Appendix.

###  4.1 Classification

Table 1: The classification accuracy results for the inference tasks on CUB
and mini-ImageNet datasets in the $1$-shot setting. The highest results are
bold and second-highest in italic (the larger, the better).

Method | CUB | mini-ImageNet
---|---|---
ML-LSTM (Ravi & Larochelle, 2017) | – | $43.44\pm 0.77$
SNAIL (Mishra et al., 2018) | – | $45.10$
iMAML-HF (Rajeswaran et al., 2019) | – | $49.30\pm 1.88$
LLAMA (Grant et al., 2018) | – | $49.40\pm 1.83$
VERSA (Gordon et al., 2018) | – | $48.53\pm 1.84$
Amortized VI (Gordon et al., 2018) | – | $44.13\pm 1.78$
Meta-Mixture (Jerfel et al., 2019) | – | $49.60\pm 1.50$
SimpleShot (Wang et al., 2019) | – | $49.69\pm 0.19$
Feature Transfer (Zhuang et al., 2020) | $46.19\pm 0.64$ | $39.51\pm 0.23$
Baseline++ (Chen et al., 2019) | $61.75\pm 0.95$ | $47.15\pm 0.49$
MatchingNet (Vinyals et al., 2016) | $60.19\pm 1.02$ | $48.25\pm 0.65$
ProtoNet (Snell et al., 2017) | $52.52\pm 1.90$ | $44.19\pm 1.30$
MAML (Finn et al., 2017) | $56.11\pm 0.69$ | $45.39\pm 0.49$
RelationNet (Sung et al., 2018) | $62.52\pm 0.34$ | $48.76\pm 0.17$
DKT + CosSim (Patacchiola et al., 2020) | $63.37\pm 0.19$ | $48.64\pm 0.45$
DKT + BNCosSim (Patacchiola et al., 2020) | $62.96\pm 0.62$ | $49.73\pm 0.07$
GPLDLA (Kim & Hospedales, 2021) | $63.40\pm 0.14$ | $52.58\pm 0.19$
amortized Bayesian prototype meta-learning (Sun et al., 2021) | $63.46\pm 0.98$ | $53.28\pm 0.91$
VAMPIRE (Nguyen et al., 2020) | – | $51.54\pm 0.74$
PLATIPUS (Finn et al., 2018) | – | $50.13\pm 1.86$
ABML (Ravi & Beatson, 2018) | $49.57\pm 0.42$ | $45.00\pm 0.60$
Bayesian MAML (Yoon et al., 2018) | $55.93\pm 0.71$ | $53.80\pm 1.46$
OVE PG GP + Cosine (ML) (Snell & Zemel, 2020) | $63.98\pm 0.43$ | $50.02\pm 0.35$
OVE PG GP + Cosine (PL) (Snell & Zemel, 2020) | $60.11\pm 0.26$ | $48.00\pm 0.24$
Reptile (Nichol et al., 2018) | – | $49.97\pm 0.32$
R2-D2 (Bertinetto et al., 2018) | – | $48.70\pm 0.60$
VSM (Zhen et al., 2020) | – | $\mathit{54.73\pm 1.60}$
PPA (Qiao et al., 2017) | – | $54.53\pm 0.40$
DFSVLwF (Gidaris & Komodakis, 2018) | – | $\mathbf{56.20\pm 0.86}$
HyperShot (ours) | $\mathit{65.27\pm 0.24}$ | $52.42\pm 0.46$
HyperShot + finetuning (ours) | $\mathbf{66.13\pm 0.26}$ | $53.18\pm 0.45$

Table 2: The classification accuracy results for the inference tasks on CUB
and mini-ImageNet datasets in the $5$-shot setting. The highest results are
bold and second-highest in italic (the larger, the better).

Method | CUB | mini-ImageNet
---|---|---
ML-LSTM (Ravi & Larochelle, 2017) | – | $60.60\pm 0.71$
SNAIL (Mishra et al., 2018) | – | $55.20$
VERSA (Gordon et al., 2018) | – | $67.37\pm 0.86$
Amortized VI (Gordon et al., 2018) | – | $55.68\pm 0.91$
Meta-Mixture (Jerfel et al., 2019) | – | $64.60\pm 0.92$
SimpleShot (Wang et al., 2019) | – | $66.92\pm 0.17$
Feature Transfer | $68.40\pm 0.79$ | $60.51\pm 0.55$
Baseline++ (Chen et al., 2019) | $78.51\pm 0.59$ | $66.18\pm 0.18$
MatchingNet (Vinyals et al., 2016) | $75.11\pm 0.35$ | $62.71\pm 0.44$
ProtoNet (Snell et al., 2017) | $75.93\pm 0.46$ | $64.07\pm 0.65$
MAML (Finn et al., 2017) | $74.84\pm 0.62$ | $61.58\pm 0.53$
RelationNet (Sung et al., 2018) | $78.22\pm 0.07$ | $64.20\pm 0.28$
DKT + CosSim (Patacchiola et al., 2020) | $77.73\pm 0.26$ | $62.85\pm 0.37$
DKT + BNCosSim (Patacchiola et al., 2020) | $77.76\pm 0.62$ | $64.00\pm 0.09$
GPLDLA (Kim & Hospedales, 2021) | $78.86\pm 0.35$ | –
amortized Bayesian prototype meta-learning (Sun et al., 2021) | $\mathbf{80.94\pm 0.62}$ | $\mathbf{70.44\pm 0.72}$
VAMPIRE (Nguyen et al., 2020) | – | $64.31\pm 0.74$
ABML (Ravi & Beatson, 2018) | $68.94\pm 0.16$ | –
Bayesian MAML (Yoon et al., 2018) | – | $64.23\pm 0.69$
OVE PG GP + Cosine (ML) (Snell & Zemel, 2020) | $77.44\pm 0.18$ | $64.58\pm 0.31$
OVE PG GP + Cosine (PL) (Snell & Zemel, 2020) | $79.07\pm 0.05$ | $67.14\pm 0.23$
Reptile (Nichol et al., 2018) | – | $65.99\pm 0.58$
R2-D2 (Bertinetto et al., 2018) | – | $65.50\pm 0.60$
VSM (Zhen et al., 2020) | – | $68.01\pm 0.90$
HyperShot | $79.80\pm 0.16$ | $68.78\pm 0.29$
HyperShot + finetuning | $\mathit{80.07\pm 0.22}$ | $\mathit{69.62\pm 0.28}$

Firstly, we consider a classical few-shot learning scenario, where all the
classification tasks (both training and inference) come from the same dataset.
The main aim of the proposed classification experiments is to find the ability
of the few-shot models to adapt to never-seen tasks from the same data
distribution.

We benchmark the performance of the HyperShot and other methods on two
challenging and widely considered datasets: Caltech-USCD Birds (CUB) (Wah et
al., 2011) and mini-ImageNet (Ravi & Larochelle, 2017). The following
experiments are in the most popular setting, 5-way, consisting of 5 random
classes. In all experiments, the query set of each task consists of 16 samples
for each class (80 in total). We provide the additional training details in
the Appendix. We compare HyperShot to a vast pool of the state-of-the-art
algorithms, including the canonical methods (like Matching Networks (Vinyals
et al., 2016), Prototypical Networks (Snell et al., 2017), MAML (Finn et al.,
2017), and its extensions) as well as the recently popular Bayesian methods
mostly build upon the Gaussian Processes framework (like DKT (Patacchiola et
al., 2020)).

We first consider the more challenging 1-shot task and report the results in
Table 1. We further consider the 5-shot setting and report the results in
Table 2. The additional results comparing methods on larger backbones are
included in Appendix.

In the $1$-shot scenario, HyperShot achieves the highest accuracies in the CUB
dataset with and without utilizing a finetuning procedure ($66.13\%$ with a
finetuning, $65.27\%$ without) and performs better than any other model.
However, in the mini-ImageNet dataset, our approach is among the best models
($53.18\%$), slightly losing with DFSVLwF (Gidaris & Komodakis, 2018)
($56.20\%$).

Considering the $5$-shot scenario, HyperShot is the second-best model
achieving $80.07\%$ in the CUB dataset and $69.62\%$ in the mini-ImageNet,
whereas the best model, amortized Bayesian prototype meta-learning, is
insignificantly better and achieves $80.94\%$ and $70.44\%$ on the mentioned
datasets, respectively.

The obtained results clearly show that HyperShot achieves state-of-the-art or
comparable results to the best models on the intensive set of a standard
classification few-shot learning setting.

###  4.2 Cross-domain adaptation

Table 3: The classification accuracy results for the inference tasks on cross-
domain tasks (Omniglot$\rightarrow$EMNIST and mini-ImageNet$\rightarrow$CUB)
datasets in the $1$-shot setting. The highest results are bold and second-
highest in italic (the larger, the better).

| Omni$\rightarrow$EMNIST | mini-ImageNet$\rightarrow$CUB
---|---|---
Method | 1-shot | 5-shot | 1-shot | 5-shot
Feature Transfer | 64.22 $\pm$ 1.24 | 86.10 $\pm$ 0.84 | 32.77 $\pm$ 0.35 | 50.34 $\pm$ 0.27
Baseline$++$ (Chen et al., 2019) | 56.84 $\pm$ 0.91 | 80.01 $\pm$ 0.92 | 39.19 $\pm$ 0.12 | $57.31\pm 0.11$
MatchingNet (Vinyals et al., 2016) | 75.01 $\pm$ 2.09 | 87.41 $\pm$ 1.79 | 36.98 $\pm$ 0.06 | 50.72 $\pm$ 0.36
ProtoNet (Snell et al., 2017) | 72.04 $\pm$ 0.82 | 87.22 $\pm$ 1.01 | 33.27 $\pm$ 1.09 | 52.16 $\pm$ 0.17
MAML (Finn et al., 2017) | 72.68 $\pm$ 1.85 | 83.54 $\pm$ 1.79 | 34.01 $\pm$ 1.25 | 48.83 $\pm$ 0.62
RelationNet (Sung et al., 2018) | 75.62 $\pm$ 1.00 | 87.84 $\pm$ 0.27 | 37.13 $\pm$ 0.20 | 51.76 $\pm$ 1.48
DKT (Patacchiola et al., 2020) | 75.40 $\pm$ 1.10 | $\mathit{90.30\pm 0.49}$ | $\mathbf{40.14\pm{0.18}}$ | 56.40 $\pm$ 1.34
Bayesian MAML (Yoon et al., 2018) | $63.94\pm 0.47$ | $65.26\pm 0.30$ | $33.52\pm 0.36$ | $51.35\pm 0.16$
OVE PG GP + Cosine (ML) (Snell & Zemel, 2020) | $68.43\pm 0.67$ | $86.22\pm 0.20$ | $39.66\pm 0.18$ | $55.71\pm 0.31$
OVE PG GP + Cosine (PL) (Snell & Zemel, 2020) | $77.00\pm 0.50$ | $87.52\pm 0.19$ | $37.49\pm 0.11$ | $57.23\pm 0.31$
HyperShot | $\mathit{78.06\pm 0.24}$ | $89.04\pm 0.18$ | $39.09\pm 0.28$ | $\mathit{57.77\pm 0.33}$
HyperShot + finetuning | $\mathbf{80.65\pm 0.30}$ | $\mathbf{90.81\pm 0.16}$ | $\mathit{40.03\pm 0.41}$ | $\mathbf{58.86\pm 0.38}$

In the cross-domain adaptation setting, the model is evaluated on tasks coming
from a different distribution than the one it had been trained on. Therefore,
such a task is more challenging than standard classification and is a
plausible indicator of a model’s ability to generalize. In order to benchmark
the performance of HyperShot in cross-domain adaptation, we merge data from
two datasets so that the training fold is drawn from the first dataset and
validation and testing fold – from another one. Specifically, we test
HyperShot on two cross-domain classification tasks:
mini-ImageNet $\rightarrow$ CUB (model trained on mini-ImageNet and evaluated
on CUB) and Omniglot $\rightarrow$ EMNIST in the $1$-shot and $5$-shot
settings. We report the results in Table 3. In most settings, HyperShot
achieves the highest accuracy, except for $1$-shot mini-ImageNet $\rightarrow$
CUB classification, where its accuracy is on par with the accuracy achieved by
DKT (Patacchiola et al., 2020) ($40.14\%$ and $40.03\%$ achieved by DKT and
HyperShot, respectively). We note that just in the case of regular
classification, finetuning the hypernetwork on the individual tasks
consistently improves its performance.

###  4.3 Aggregating support examples in the 5-shot setting

In HyperShot, the hypernetwork generates the weights of the information about
the support examples, expressed through the support-support kernel matrix. In
the case of $5$-way $1$-shot classification, each task consists of $5$ support
examples, and therefore, the size of the kernel matrix is $(5\times 5)$, and
the input size of the hypernetwork is $25$. However, when the number of
support examples grows, increasing the size of the kernel matrix may be
impractical and lead to overparametrization of the hypernetwork.

Since hypernetworks are known to be sensitive to large input sizes (Ha et al.,
2016), we consider a way to maintain a constant input size of HyperShot,
independent of the number of support examples of each class by using means of
support embeddings of each class for kernel calculation, instead of individual
embeddings. Prior works suggest that when there are multiple examples of a
class, an averaged embedding of such class represents it sufficiently in the
embedding space (Snell et al., 2017).

To verify this approach, in the $5$-shot setting, we train HyperShot with two
variants of calculating the inputs to the kernel matrix:

  * •

fine-grained – utilizing a hypernetwork that takes as an input a kernel matrix
between each of the embeddings of the individual support examples. This kernel
matrix has a shape of $(25\times 25)$.

  * •

averaged – utilizing a hypernetwork where the kernel matrix is calculated
between the means of embeddings of each class. The kernel matrix in this
approach has a shape of $(5\times 5)$.

We benchmark both variants of HyperShot on the $5$-shot classification task on
CUB and mini-ImageNet datasets. Moreover, we compare these approaches also on
cross-domain classification between the Omniglot and EMNIST datasets. We
report the accuracies in Table 4. It is evident that averaging the embeddings
before calculating the kernel matrix yields superior results.

Table 4: The classification accuracy results for HyperShot in the $5$-shot
setting with two variants of the support embeddings aggregation. The
performance measured on Omniglot$\rightarrow$EMNIST, CUB, and mini-
ImageNet$\rightarrow$CUB tasks. The larger, the better.

| Omni$\rightarrow$EMNIST | CUB | mini-ImageNet
---|---|---|---
HyperShot (fine-grained) | $87.55\pm 0.19$ | $78.05\pm 0.20$ | $67.07\pm 0.47$
HyperShot (averaged) | $89.04\pm 0.18$ | $79.80\pm 0.16$ | $69.62\pm 0.28$

##  5 Conclusion

In this work, we introduced HyperShot –- a new framework that uses kernel
methods combined with hypernetworks. Our method uses the kernel-based
representation of the support examples and a hypernetwork paradigm to create
the query set’s classification module. We concentrate on relations between
embeddings of the support examples instead of direct feature values. Thanks to
this approach, our model can adapt to highly different tasks.

We evaluate the HyperShot model on various one-shot and few-shot image
classification tasks. HyperShot demonstrates high accuracy in all tasks,
performing comparably or better to state-of-the-art solutions. Furthermore,
the model has a strong ability to generalize, as evidenced by its performance
on cross-domain classification tasks.

## References

  * Bauer et al. (2017) Bauer, M., Rojas-Carulla, M., Światkowski, J. B., Schölkopf, B., and Turner, R. E.  Discriminative k-shot learning using probabilistic models, 2017.
  * Bengio et al. (1992) Bengio, S., Bengio, Y., Cloutier, J., and Gecsei, J.  On the optimization of a synaptic learning rule.  1992\.
  * Bertinetto et al. (2018) Bertinetto, L., Henriques, J. F., Torr, P., and Vedaldi, A.  Meta-learning with differentiable closed-form solvers.  In _International Conference on Learning Representations_ , 2018.
  * Chen et al. (2019) Chen, W.-Y., Liu, Y.-C., Kira, Z., Wang, Y.-C. F., and Huang, J.-B.  A closer look at few-shot classification.  _arXiv preprint arXiv:1904.04232_ , 2019.
  * Cohen et al. (2017) Cohen, G., Afshar, S., Tapson, J., and van Schaik, A.  Emnist: an extension of mnist to handwritten letters (2017).  _arXiv preprint arXiv:1702.05373_ , 2017.
  * Finn et al. (2017) Finn, C., Abbeel, P., and Levine, S.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _International Conference on Machine Learning_ , pp. 1126–1135. PMLR, 2017.
  * Finn et al. (2018) Finn, C., Xu, K., and Levine, S.  Probabilistic model-agnostic meta-learning.  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , pp. 9537–9548, 2018.
  * Gidaris & Komodakis (2018) Gidaris, S. and Komodakis, N.  Dynamic few-shot visual learning without forgetting.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 4367–4375, 2018.
  * Gordon et al. (2018) Gordon, J., Bronskill, J., Bauer, M., Nowozin, S., and Turner, R.  Meta-learning probabilistic inference for prediction.  In _International Conference on Learning Representations_ , 2018.
  * Grant et al. (2018) Grant, E., Finn, C., Levine, S., Darrell, T., and Griffiths, T.  Recasting gradient-based meta-learning as hierarchical bayes.  In _International Conference on Learning Representations_ , 2018.
  * Ha et al. (2016) Ha, D., Dai, A., and Le, Q. V.  Hypernetworks.  _arXiv preprint arXiv:1609.09106_ , 2016.
  * He et al. (2015) He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition.  _arXiv preprint arXiv:1512.03385_ , 2015.
  * Hospedales et al. (2020) Hospedales, T., Antoniou, A., Micaelli, P., and Storkey, A.  Meta-learning in neural networks: A survey, 2020.
  * Hu et al. (2021) Hu, Y., Gripon, V., and Pateux, S.  Leveraging the feature distribution in transfer-based few-shot learning, 2021.
  * Jerfel et al. (2019) Jerfel, G., Grant, E., Griffiths, T. L., and Heller, K.  Reconciling meta-learning and continual learning with online mixtures of tasks.  In _Proceedings of the 33rd International Conference on Neural Information Processing Systems_ , pp. 9122–9133, 2019.
  * Kim & Hospedales (2021) Kim, M. and Hospedales, T.  Gaussian process meta few-shot classifier learning via linear discriminant laplace approximation.  _arXiv preprint arXiv:2111.05392_ , 2021.
  * Kingma & Ba (2014) Kingma, D. and Ba, J.  Adam: A method for stochastic optimization.  _International Conference on Learning Representations_ , 12 2014.
  * Lake et al. (2011) Lake, B., Salakhutdinov, R., Gross, J., and Tenenbaum, J.  One shot learning of simple visual concepts.  In _Proceedings of the annual meeting of the cognitive science society_ , volume 33, 2011.
  * Lee et al. (2019) Lee, K., Maji, S., Ravichandran, A., and Soatto, S.  Meta-learning with differentiable convex optimization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10657–10665, 2019.
  * Mishra et al. (2018) Mishra, N., Rohaninejad, M., Chen, X., and Abbeel, P.  A simple neural attentive meta-learner.  In _International Conference on Learning Representations_ , 2018.
  * Munkhdalai & Yu (2017) Munkhdalai, T. and Yu, H.  Meta networks.  In _International Conference on Machine Learning_ , pp. 2554–2563. PMLR, 2017.
  * Munkhdalai et al. (2018) Munkhdalai, T., Yuan, X., Mehri, S., and Trischler, A.  Rapid adaptation with conditionally shifted neurons.  In _International Conference on Machine Learning_ , pp. 3664–3673. PMLR, 2018.
  * Nguyen et al. (2020) Nguyen, C., Do, T.-T., and Carneiro, G.  Uncertainty in model-agnostic meta-learning using variational inference.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pp. 3090–3100, 2020.
  * Nichol et al. (2018) Nichol, A., Achiam, J., and Schulman, J.  On first-order meta-learning algorithms.  _arXiv preprint arXiv:1803.02999_ , 2018.
  * Oreshkin et al. (2018) Oreshkin, B. N., Rodriguez, P., and Lacoste, A.  Tadam: Task dependent adaptive metric for improved few-shot learning.  _arXiv preprint arXiv:1805.10123_ , 2018.
  * Patacchiola et al. (2020) Patacchiola, M., Turner, J., Crowley, E. J., O’Boyle, M., and Storkey, A. J.  Bayesian meta-learning for the few-shot setting via deep kernels.  _Advances in Neural Information Processing Systems_ , 33, 2020.
  * Qiao et al. (2017) Qiao, S., Liu, C., Shen, W., and Yuille, A.  Few-shot image recognition by predicting parameters from activations, 2017\.
  * Raghu et al. (2019) Raghu, A., Raghu, M., Bengio, S., and Vinyals, O.  Rapid learning or feature reuse? towards understanding the effectiveness of maml.  _arXiv preprint arXiv:1909.09157_ , 2019.
  * Rajeswaran et al. (2019) Rajeswaran, A., Finn, C., Kakade, S. M., and Levine, S.  Meta-learning with implicit gradients.  _Advances in Neural Information Processing Systems_ , 32:113–124, 2019.
  * Rasmussen (2003) Rasmussen, C. E.  Gaussian processes in machine learning.  In _Summer school on machine learning_ , pp. 63–71. Springer, 2003\.
  * Ravi & Beatson (2018) Ravi, S. and Beatson, A.  Amortized bayesian meta-learning.  In _International Conference on Learning Representations_ , 2018.
  * Ravi & Larochelle (2017) Ravi, S. and Larochelle, H.  Optimization as a model for few-shot learning.  In _ICLR_ , 2017.
  * Russakovsky et al. (2015) Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ , 115(3):211–252, 2015.
  * Santoro et al. (2016) Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., and Lillicrap, T.  Meta-learning with memory-augmented neural networks.  In _International conference on machine learning_ , pp. 1842–1850. PMLR, 2016.
  * Schmidhuber (1992) Schmidhuber, J.  Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Networks.  _Neural Computation_ , 4(1):131–139, 01 1992\.  ISSN 0899-7667.  doi: 10.1162/neco.1992.4.1.131.  URL https://doi.org/10.1162/neco.1992.4.1.131.
  * Sheikh et al. (2017) Sheikh, A.-S., Rasul, K., Merentitis, A., and Bergmann, U.  Stochastic maximum likelihood optimization via hypernetworks.  _arXiv preprint arXiv:1712.01141_ , 2017.
  * Snell & Zemel (2020) Snell, J. and Zemel, R.  Bayesian few-shot classification with one-vs-each pólya-gamma augmented gaussian processes.  In _International Conference on Learning Representations_ , 2020.
  * Snell et al. (2017) Snell, J., Swersky, K., and Zemel, R. S.  Prototypical networks for few-shot learning.  _arXiv preprint arXiv:1703.05175_ , 2017.
  * Sun et al. (2021) Sun, Z., Wu, J., Li, X., Yang, W., and Xue, J.-H.  Amortized bayesian prototype meta-learning: A new probabilistic meta-learning approach to few-shot image classification.  In _International Conference on Artificial Intelligence and Statistics_ , pp. 1414–1422. PMLR, 2021.
  * Sung et al. (2018) Sung, F., Yang, Y., Zhang, L., Xiang, T., Torr, P. H., and Hospedales, T. M.  Learning to compare: Relation network for few-shot learning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 1199–1208, 2018.
  * Vinyals et al. (2016) Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., et al.  Matching networks for one shot learning.  _Advances in neural information processing systems_ , 29:3630–3638, 2016.
  * Wah et al. (2011) Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S.  The Caltech-UCSD Birds-200-2011 Dataset.  Technical Report CNS-TR-2011-001, California Institute of Technology, 2011\.
  * Wang et al. (2019) Wang, Y., Chao, W.-L., Weinberger, K. Q., and van der Maaten, L.  Simpleshot: Revisiting nearest-neighbor classification for few-shot learning.  _arXiv preprint arXiv:1911.04623_ , 2019.
  * Wang et al. (2020) Wang, Y., Yao, Q., Kwok, J., and Ni, L. M.  Generalizing from a few examples: A survey on few-shot learning, 2020\.
  * Wang et al. (2021) Wang, Z., Miao, Z., Zhen, X., and Qiu, Q.  Learning to learn dense gaussian processes for few-shot learning.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Yoon et al. (2018) Yoon, J., Kim, T., Dia, O., Kim, S., Bengio, Y., and Ahn, S.  Bayesian model-agnostic meta-learning.  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , pp. 7343–7353, 2018.
  * Zhen et al. (2020) Zhen, X., Du, Y.-J., Xiong, H., Qiu, Q., Snoek, C., and Shao, L.  Learning to learn variational semantic memory.  In _NeurIPS_ , 2020.
  * Zhmoginov et al. (2022) Zhmoginov, A., Sandler, M., and Vladymyrov, M.  Hypertransformer: Model generation for supervised and semi-supervised few-shot learning, 2022.
  * Zhuang et al. (2020) Zhuang, F., Qi, Z., Duan, K., Xi, D., Zhu, Y., Zhu, H., Xiong, H., and He, Q.  A comprehensive survey on transfer learning, 2020.

##  Appendix A Additional results - ResNet10

This section provides additional results in the $5$-way ($1$-shot and
$5$-shot) classification tasks for models using a larger backbone, namely
ResNet-10 (He et al., 2015). We provide the results for CUB and mini-ImageNet
datasets in Tables 5 and 6.

In the CUB dataset classification tasks (see Table 5), HyperShot is amongst
the state-of-the-art models achieving classification accuracy often equal
within the variance to the best models. Considering the $5$-shot scenario, the
highest classification result across the evaluated methods ($86.38\%\pm 0.15$)
obtained the GPLDLA model based on the Gaussian Processes framework. However,
the HyperShot performance, $86.28\%\pm 0.29$, is the second-best but even lies
within the variance of the best model. In the $1$-shot setting, ProtoNet
obtains the highest result ($73.22\%\pm 0.92$), whereas HyperShot is the third
one ($71.99\%\pm 0.70$) but still equal according to the variances.

In the mini-ImageNet classification task, HyperShot achieves the second-best
accuracy in both 1-shot and 5-shot settings222In the case of the mini-ImageNet
classification with ResNet10, we benchmarked all of the listed models
ourselves. To our best knowledge, previously, there were no reported
benchmarks on this dataset with the ResNet-10 backbone.. In the $1$-shot
setting, the DKT model (Patacchiola et al., 2020) achieved the best result,
with HyperShot being a close second, with only $0.04$ pp difference. In the
$5$-shot setting, the baseline++ approach outperforms all others by a large
margin (Chen et al., 2019), whereas HyperShot and ProtoNet (Snell et al.,
2017) achieve similar, second-best results. We observe that apart from
HyperShot, which achieves second-best results in both settings, models which
perform well in one setting are outperformed by others in the second and vice
versa.

It is worth noticing that HyperShot without finetuning steps performances
sometimes slightly better than the same with finetuning. We even observe that
a few first steps of finetuning procedure result in an unnoticeable increase
of accuracy of the basic model. However, the usual $10$ steps result in this
setting in slightly worse performance, so one should use it cautiously. We
decided to report the results after the standard finetuning procedure only.

Table 5:  The classification accuracy results for the inference tasks in the CUB dataset in the 5-way (1-shot and 5-shot) scenarios. We consider models using the ResNet-10 backbone. The highest results are bold and second-highest in italic (the larger, the better). Method | 1-shot | 5-shot
---|---|---
Feature Transfer | $63.64\pm 0.91$ | $81.27\pm 0.57$
Baseline++ (Chen et al., 2019) | $69.55\pm 0.89$ | $85.17\pm 0.50$
MatchingNet (Vinyals et al., 2016) | $71.29\pm 0.87$ | $83.47\pm 0.58$
ProtoNet (Snell et al., 2017) | $\mathbf{73.22\pm 0.92}$ | $85.01\pm 0.52$
MAML (Finn et al., 2017) | $70.32\pm 0.99$ | $80.93\pm 0.71$
RelationNet (Sung et al., 2018) | $70.47\pm 0.99$ | $83.70\pm 0.55$
DKT + CosSim (Patacchiola et al., 2020) | $70.81\pm 0.52$ | $83.26\pm 0.50$
DKT + BNCosSim (Patacchiola et al., 2020) | $\mathit{72.27\pm 0.30}$ | $85.64\pm 0.29$
SimpleShot (Wang et al., 2019) | $53.78\pm 0.21$ | $71.41\pm 0.17$
GPLDLA (Kim & Hospedales, 2021) | $71.30\pm 0.16$ | $\mathbf{86.38\pm 0.15}$
HyperShot | $71.99\pm 0.70$ | $86.28\pm 0.29$
HyperShot + finetuning | $71.60\pm 0.59$ | $\mathit{86.22\pm 0.30}$
Table 6:  The classification accuracy results for the inference tasks in the mini-ImageNet dataset in the 5-way (1-shot and 5-shot) scenarios. We consider models using the ResNet-10 backbone. The highest results are bold and second-highest in italic (the larger, the better). Method | 1-shot | 5-shot
---|---|---
Baseline++ (Chen et al., 2019) | $54.35\pm 0.34$ | $\mathbf{75.26\pm 0.16}$
MatchingNet (Vinyals et al., 2016) | $54.18\pm 0.09$ | $67.71\pm 0.20$
ProtoNet (Snell et al., 2017) | $53.28\pm 0.17$ | $73.04\pm 0.15$
RelationNet (Sung et al., 2018) | $51.88\pm 0.45$ | $67.21\pm 0.16$
DKT + BNCosSim (Patacchiola et al., 2020) | $\mathbf{56.03\pm 0.50}$ | $71.28\pm 0.12$
HyperShot | $55.36\pm 0.64$ | $\mathit{73.06\pm 0.30}$
HyperShot + finetuning | $\mathit{55.99\pm 0.63}$ | $72.87\pm 0.33$

##  Appendix B Training details

In this section, we present in detail the architecture and hyperparameters of
HyperShot.

#### Architecture overview

From a high-level perspective, the architecture of HyperShot consists of three
parts:

  * •

backbone - a convolutional feature extractor.

  * •

neck - a sequence of zero or more fully-connected layers with ReLU
nonlinearities in between.

  * •

heads - for each parameter of the target network, a sequence of one or more
linear layers, which predicts the values of that parameter. All heads of
HyperShot have identical lengths, hidden sizes, and input sizes that depend on
the generated parameter’s size.

The target network generated by HyperShot re-uses its backbone. We outline
this architecture in Figure 4.

Figure 4: A detailed outline of the architecture of HyperShot, with the
denoted flow of parameters generated by the hypernetwork heads.

#### Backbone

For each experiment described in the main body of this work, we follow
(Patacchiola et al., 2020) in using a shallow backbone (feature extractor) for
HyperShot as well as referential models. This backbone consists of four
convolutional layers, each consisting of a convolution, batch normalization,
and ReLU nonlinearity. Apart from the first convolution, which has the number
of input size equal to the number of image channels, each convolution has an
input and output size of 64. We apply max-pooling between each convolution,
which decreases by half the resolution of the processed feature maps. The
output of the backbone is flattened so that the further layers can process it.

We perform additional experiments described in Appendix A where instead of the
above backbone, we utilize ResNet-10 (He et al., 2015).

#### Datasets

For the purpose of making a fair comparison, we follow the procedure presented
in, e.g., (Patacchiola et al., 2020; Chen et al., 2019). In the case of the
CUB dataset (Wah et al., 2011), we split the whole amount of $200$ classes
($11788$ images) across train, validation, and test consisting of $100$, $50$,
and $50$ classes, respectively (Chen et al., 2019). The mini-ImageNet dataset
(Ravi & Larochelle, 2017) is created as the subset of ImageNet (Russakovsky et
al., 2015), which consists of $100$ different classes represented by $600$
images for each one. We followed the standard procedure and divided the mini-
ImageNet into $64$ classes for the train, $16$ for the validation set, and the
remaining $20$ classes for the test. The well-known Omniglot dataset (Lake et
al., 2011) is a collection of characters from $50$ different languages. The
Omniglot contains $1623$ white and black characters in total. We utilize the
standard procedure to include the examples rotated by $90^{\circ}$ and
increase the size of the dataset to $6492$, from which $4114$ were further
used in training. Finally, the EMNIST dataset (Cohen et al., 2017) collects
the characters and digits coming from the English alphabet, which we split
into $31$ classes for the test and $31$ for validation.

#### Data augmentation

We apply data augmentation during model training in all experiments, except
Omniglot $\rightarrow$ EMNIST cross-domain classification. The augmentation
pipeline is identical to the one used by (Patacchiola et al., 2020) and
consists of the random crop, horizontal flip, and color jitter steps.

###  B.1 Hyperparameters

Below, we outline the hyperparameters of architecture and training procedures
used in each experiment.

We use cosine similarity as a kernel function and averaged support embeddings
aggregation in all experiments. HyperShot is trained with the learning rate of
$0.001$ with the Adam optimizer (Kingma & Ba, 2014) and no learning rate
scheduler. Task-specific finetuning is also performed with the Adam optimizer
and the learning rate of $0.0001$.

For the natural image tasks (CUB, mini-ImageNet, mini-ImageNet $\rightarrow$
CUB classification), we use a hypernetwork with the neck length o $2$, head
lengths of $3$, and a hidden size of $4096$, which produce a target network
with a single fully-connected layer. We perform training for $10000$ epochs.

For the simpler Omniglot $\rightarrow$ EMNIST character classification task,
we train a smaller hypernetwork with the neck length of $1$, head lengths of
$2$, and the hidden size of $512$, which produces a target network with two
fully-connected layers and a hidden size of $128$. We train this hypernetwork
for a shorter number of epochs, namely $2000$.

We summarize all the above hyperparameters in Table 7.

Table 7: Hyperparameters

hyperparameter | CUB | mini-ImageNet |  mini-ImageNet $\rightarrow$ CUB |  Omniglot $\rightarrow$ EMNIST
---|---|---|---|---
kernel function | cosine similarity | cosine similarity | cosine similarity | cosine similarity
learning rate | $0.001$ | $0.001$ | $0.001$ | $0.001$
hypernetwork’s head layers no. | $3$ | $3$ | $3$ | $2$
hypernetwork’s neck layers no. | $2$ | $2$ | $2$ | $1$
hypernetwork layers’ hidden dim | $4096$ | $4096$ | $4096$ | $512$
support embeddings aggregation | averaged | averaged | averaged | averaged
taskset size | $1$ | $1$ | $1$ | $1$
target network layers no. | $1$ | $1$ | $1$ | $2$
target network activation | ReLU | ReLU | ReLU | ReLU
finetuning epochs (if used) | $10$ | $10$ | $10$ | $10$
finetuning learning rate | $0.0001$ | $0.0001$ | $0.0001$ | $0.0001$
optimizer | Adam | Adam | Adam | Adam
epochs no. | $10000$ | $10000$ | $10000$ | $2000$


Uzbek Mathematical
---
Journal, 2021, No XX, pp.1-1

A discrete-time epidemic SISI model

S.K.Shoyimardonov

Abstract. We consider a discrete-time epidemic SISI model in case when the
population size is a constant, so the per capita death rate is equal to per
capita birth rate. The evolution operator of this model is a non-linear
operator which depends on seven parameters. Reducing it to a quadratic
stochastic operator, we prove uniqueness of interior fixed point of the
operator and study the limit behavior of the trajectory under some conditions
to parameters.

Keywords: epidemic, discrete-time, fixed point, simplex, trajectory, limit.

MSC (2010): 37N25, 92D30

##  1 Introduction

Classical epidemic models usually assume that either immunity does not exist
(the SIS model) or that experiencing the infection provides permanent or
temporary protection against it (the SIR and SIRS models). In the SIS model a
typical individual starts off susceptible, at some stage catches the infection
and after an infectious period becomes completely susceptible again. But, in
SISI model a member of a population after recovering can infected second time
also. In [2] the main example of SISI epidemic model was discussed as bovine
respiratory syncytial virus (BRSV) amongst cattle in continuous time. In this
paper it was assumed that the population size under consideration is a
constant, so the per capita death rate is equal to per capita birth rate. In
epidemiology, a _susceptible_ individual is a member of a population who is at
risk of becoming infected by a disease. A _susceptibility_ only refers to the
fact that the virus is able to get into the cell. _Infectivity_ is the ability
of a pathogen to establish an infection. A _Pathogen_ is any organism that can
produce disease.

Continuous time SISI model is as follows [1]:

$\begin{cases}\frac{dS}{dt}&=b(S+I+S_{1}+I_{1})-\mu
S-\beta_{1}A(I,I_{1})S\\\\[5.69054pt] \frac{dI}{dt}&=-\mu I-\alpha
I+\beta_{1}A(I,I_{1})S\\\\[5.69054pt] \frac{dS_{1}}{dt}&=-\mu S_{1}+\alpha
I-\beta_{2}A(I,I_{1})S_{1}\\\\[5.69054pt] \frac{dI_{1}}{dt}&=-\mu
I_{1}+\beta_{2}A(I,I_{1})S_{1}\end{cases}$ (1.1)

where $S-$ density of susceptibles who did not have the disease before, $I-$
density of first time infected persons, $S_{1}-$ density of recovereds,
$I_{1}-$ density of second time infected persons, $b-$ birth rate, $\mu-$
death rate, $\alpha-$ recovery rate, $\beta_{1}-$ susceptibility of persons in
$S$, $\beta_{2}-$ susceptibility of persons in $S_{1},$ $k_{1}-$ infectivity
of persons in $I$, $k_{2}-$ infectivity of persons in $I_{1}.$ Moreover,
$A(I,I_{1})$ denotes the so-called force of infection,

$A(I,I_{1})=\frac{k_{1}I+k_{2}I_{1}}{P}$

and $P=S+I+S_{1}+I_{1}$ denotes the total population size.

In [3] it was assumed that birth rate is a same with death rate, i.e., $b=\mu$
and using some replacements it was obtained a new system:

$x=\frac{S}{P},\ \ u=\frac{I}{P},\ \ y=\frac{S_{1}}{P},\ \ v=\frac{I_{1}}{P},$

$\begin{cases}\frac{dx}{dt}&=b-bx-\beta_{1}A(u,v)x\\\\[5.69054pt]
\frac{du}{dt}&=-bu-\alpha u+\beta_{1}A(u,v)x\\\\[5.69054pt]
\frac{dy}{dt}&=-by+\alpha u-\beta_{2}A(u,v)y\\\\[5.69054pt]
\frac{dv}{dt}&=-bv+\beta_{2}A(u,v)y\end{cases}$ (1.2)

where all parameters are non-negative and $x+u+y+v=1$.

_The quadratic stochastic operator_ (QSO) [5], [6], [7] is a mapping of the
standard simplex.

$S^{m-1}=\\{x=(x_{1},...,x_{m})\in\mathbb{R}^{m}:x_{i}\geq
0,\sum\limits_{i=1}^{m}x_{i}=1\\}$ (1.3)

into itself, of the form

$V:x^{\prime}_{k}=\sum\limits_{i=1}^{m}\sum\limits_{j=1}^{m}P_{ij,k}x_{i}x_{j},\qquad
k=1,...,m,$ (1.4)

where the coefficients $P_{ij,k}$ satisfy the following conditions

$P_{ij,k}\geq 0,\quad
P_{ij,k}=P_{ji,k},\quad\sum\limits_{k=1}^{m}P_{ij,k}=1,\qquad(i,j,k=1,...,m).$
(1.5)

For a given $\lambda^{(0)}\in S^{m-1}$ the _trajectory_ (orbit)
$\\{\lambda^{(n)};n\geq 0\\}$ of $\lambda^{(0)}$ under the action of QSO (1.4)
is defined by

$\lambda^{(n+1)}=V(\lambda^{(n)}),\;n=0,1,2,...$

The main problem in mathematical biology consists in the study of the
asymptotical behaviour of the trajectories.

##  2 Main Part

In [3] it was studied the following discrete-time version of the system (1.2).

$V:\left\\{\begin{aligned} x^{(1)}&=x+b-bx-\beta_{1}A(u,v)x\\\
u^{(1)}&=u-bu-\alpha u+\beta_{1}A(u,v)x\\\ y^{(1)}&=y-by+\alpha
u-\beta_{2}A(u,v)y\\\ v^{(1)}&=v-bv+\beta_{2}A(u,v)y\end{aligned}\right.$
(2.1)

where $A(u,v)=k_{1}u+k_{2}v.$

######  Proposition 2.1.

[3] We have $V\left(S^{3}\right)\subset S^{3}$ if and only if the non-negative
parameters $b,\alpha,\beta_{1},\beta_{2},k_{1},k_{2}$ verify the following
conditions

$\begin{array}[]{cccc}\alpha+b\leq 1,&\beta_{1}k_{2}\leq 2,&\beta_{2}k_{1}\leq
2,\vskip 6.0pt plus 2.0pt minus 2.0pt\\\ b+\beta_{2}k_{2}\leq
1,&\left|b-\beta_{1}k_{1}\right|\leq 1,&\left|b-\beta_{2}k_{2}\right|\leq
1,\vskip 6.0pt plus 2.0pt minus 2.0pt\\\ \left|b-\beta_{1}k_{2}\right|\leq
1,&\left|\alpha+b-\beta_{1}k_{1}\right|\leq
1,&\left|\alpha-b-\beta_{2}k_{1}\right|\leq 1.\end{array}$ (2.2)

Moreover, under conditions (2.2) the operator $V$ is a QSO.

Recall that fixed point of the operator $F$ is a solution of equation
$F(x)=x.$

Let us denote

  * $\lambda_{1}=\left(1,0,0,0\right),\ \ \lambda_{2}=\left(0,1,0,0\right),\ \ \lambda_{3}=\left(0,0,1,0\right),\ \ \lambda_{4}=\left(0,0,0,1\right),$

  * $\Lambda_{5}=\\{\lambda=(x,u,y,v)\in S^{3}:x=u=0\\},$

  * $\Lambda_{6}=\\{\lambda=(x,u,y,v)\in S^{3}:x=y=0\\},$

  * $\Lambda_{7}=\\{\lambda=(x,u,y,v)\in S^{3}:x=v=0\\},$

  * $\Lambda_{8}=\\{\lambda=(x,u,y,v)\in S^{3}:u=y=0\\},$

  * $\Lambda_{9}=\\{\lambda=(x,u,y,v)\in S^{3}:u=v=0\\},$

  * $\Lambda_{10}=\\{\lambda=(x,u,y,v)\in S^{3}:y=v=0\\},$

  * $\Lambda_{11}=\\{\lambda=(x,u,y,v)\in S^{3}:x=0\\},$

  * $\Lambda_{12}=\\{\lambda=(x,u,y,v)\in S^{3}:u=0\\},$

  * $\Lambda_{13}=\\{\lambda=(x,u,y,v)\in S^{3}:y=0\\},$

  * $\Lambda_{14}=\\{\lambda=(x,u,y,v)\in S^{3}:v=0\\},$

  * $\lambda_{15}=\left(\frac{b}{\beta_{1}k_{1}},\frac{\beta_{1}k_{1}-b}{\beta_{1}k_{1}},0,0\right),\ \ \lambda_{16}=\left(\frac{b+\alpha}{\beta_{1}k_{1}},\frac{b(\beta_{1}k_{1}-b-\alpha)}{\beta_{1}k_{1}(b+\alpha)},\frac{\alpha(\beta_{1}k_{1}-b-\alpha)}{\beta_{1}k_{1}(b+\alpha)},0\right),$

  * $\lambda_{17}=\left(\frac{b}{b+\beta_{1}A},\frac{b\beta_{1}A}{(b+\beta_{1}A)(b+\alpha)},\frac{\alpha b\beta_{1}A}{(b+\beta_{1}A)(b+\beta_{2}A)(b+\alpha)},\frac{\alpha\beta_{1}\beta_{2}A^{2}}{(b+\beta_{1}A)(b+\beta_{2}A)(b+\alpha)}\right),$

where $A$ is a positive solution of the equation

$1=\frac{b\beta_{1}k_{1}}{(b+\beta_{1}A)(b+\alpha)}+\frac{\alpha\beta_{1}\beta_{2}k_{2}A}{(b+\beta_{1}A)(b+\beta_{2}A)(b+\alpha)}$
(2.3)

######  Remark 2.2.

If $b=\alpha=k_{1}=k_{2}=0$ or $b=\alpha=\beta_{1}=\beta_{2}=0$ then the
operator (2.1) is an identity operator.

By the following proposition we give all possible fixed points of the operator
$V.$

######  Proposition 2.3.

[3] Let $Fix(V)$ be set of fixed points of the operator (2.1). Then

$Fix(V)=\left\\{\begin{array}[]{lll}\\{\lambda_{1}\\}\\\\[5.69054pt]
\\{\lambda_{4}\\}\bigcup\Lambda_{9},\ \ {\rm if}\ \ b=0\\\\[5.69054pt]
\Lambda_{6}\bigcup\Lambda_{9},\ \ \ \ {\rm if}\ \ b=\alpha=0\\\\[5.69054pt]
\Lambda_{8}\bigcup\Lambda_{9},\ \ \ \ {\rm if}\ \ b=\beta_{1}=0\\\\[5.69054pt]
\Lambda_{5}\bigcup\Lambda_{9},\ \ \ \ {\rm if}\ \ b=\beta_{2}=0\\\\[5.69054pt]
\\{\lambda_{4}\\}\bigcup\Lambda_{9},\ \ \ \ {\rm if}\ \
b=k_{1}=0\\\\[5.69054pt] \Lambda_{12},\ \ \ \ {\rm if}\ \
b=k_{2}=0\\\\[5.69054pt] \Lambda_{9}\bigcup\Lambda_{13},\ \ \ \ {\rm if}\ \
b=\alpha=\beta_{1}=0\\\\[5.69054pt] \Lambda_{9}\bigcup\Lambda_{11},\ \ \ \
{\rm if}\ \ b=\alpha=\beta_{2}=0\\\\[5.69054pt]
\Lambda_{6}\bigcup\Lambda_{14},\ \ \ \ {\rm if}\ \
b=\alpha=k_{1}=0\\\\[5.69054pt] \Lambda_{6}\bigcup\Lambda_{12},\ \ \ \ {\rm
if}\ \ b=\alpha=k_{2}=0\\\\[5.69054pt] \Lambda_{12},\ \ \ \ \ \ \ \ \ \ \ {\rm
if}\ \ b=\beta_{1}=\beta_{2}=0\\\\[5.69054pt] \Lambda_{8}\bigcup\Lambda_{9},\
\ \ \ {\rm if}\ \ b=\beta_{1}=k_{1}=0\\\\[5.69054pt] \Lambda_{12},\ \ \ \ {\rm
if}\ \ b=\beta_{1}=k_{2}=0\\\\[5.69054pt] \Lambda_{5}\bigcup\Lambda_{9},\ \ \
\ {\rm if}\ \ b=\beta_{2}=k_{1}=0\\\\[5.69054pt] \Lambda_{12},\ \ \ \ {\rm
if}\ \ b=\beta_{2}=k_{2}=0\\\\[5.69054pt] \\{\lambda_{1},\lambda_{15}\\},\ \ \
\ {\rm if}\ \ b>0,\alpha=0\ \ {\rm and}\ \ \beta_{1}k_{1}>b\\\\[5.69054pt]
\\{\lambda_{1},\lambda_{16}\\},\ \ \ \ {\rm if}\ \
b>0,\,\alpha>0,\,\beta_{2}=0,\,\beta_{1}k_{1}>b+\alpha\\\\[5.69054pt]
\\{\lambda_{1},\lambda_{17}\\},\ \ \ \ {\rm if}\ \ \alpha
b\beta_{1}\beta_{2}k_{1}>0\end{array}\right.$

We interested in the positive solutions of the equation (2.3).

######  Theorem 2.4.

For the equation (2.3) the following cases hold:
(i) If $\beta_{1}k_{1}=b+\alpha$ and $\alpha\beta_{2}k_{2}>b\beta_{1}k_{1}$
then the equation (2.3) has unique positive solution;
(ii) If $\beta_{1}k_{1}>b+\alpha$ the the equation (2.3) has unique positive
solution;
(iii) If $\beta_{1}k_{1}<b+\alpha$ then the equation (2.3) does not have
positive solution.

###### Proof.

First of all, we denote by $f(x),g(x):$

$f(x)=b+\beta_{1}x,\ \ \ \ \ \
g(x)=\frac{b\beta_{1}k_{1}}{b+\alpha}+\frac{\alpha\beta_{1}\beta_{2}k_{2}x}{(b+\beta_{2}x)(b+\alpha)}.$
(2.4)

Then the roots of the equation (2.3) are roots of the equation $f(x)=g(x).$

_(i)_ If $\beta_{1}k_{1}=b+\alpha$ then by (2.4) it follows that

$\beta_{1}x=\frac{\alpha\beta_{2}k_{2}x}{k_{1}(b+\beta_{2}x)}\ \ \Rightarrow
x_{1}=0,\ \
x_{2}=\frac{\alpha\beta_{2}k_{2}-b\beta_{1}k_{1}}{\beta_{1}\beta_{2}k_{1}}.$

By condition of the theorem $x_{2}>0.$

_(ii)_ First, we find the horizontal asymptote of the graph of the function
$g(x):$

$y=\frac{\beta_{1}(bk_{1}+\alpha k_{2})}{b+\alpha}=const.$

Moreover, right part (with respect to vertical asymptote) of the graphic of
$g(x)$ is increasing and convex. By the condition of theorem
$\beta_{1}k_{1}>b+\alpha,$ so $\frac{b\beta_{1}k_{1}}{b+\alpha}>b,$ i.e.,
$f(0)<g(0).$ Therefore, the equation (2.4) has one positive solution (Fig. 2).

_(iii)_ Let $\beta_{1}k_{1}<b+\alpha$. Then $f(0)>g(0).$ Assume that
$f(x)=g(x)$ has positive solution. Then there are two solutions $x_{1},x_{2}$
(suppose that $x_{1}<x_{2}$) and for any $x\in(x_{1};x_{2}),$ $f(x)<g(x).$
Note that $g(x)$ is convex and increasing function, so there exists
$\bar{x}\in(x_{1};x_{2})$ such that at this point $f(x)$ and $g(x)$ have same
slope. Let’s equate these slopes at the point $\bar{x}$:

$\beta_{1}=\frac{\alpha
b\beta_{1}\beta_{2}k_{2}}{(b+\alpha)(b+\beta_{2}\bar{x})^{2}}\Rightarrow\beta_{2}\bar{x}=-b+\sqrt{\frac{\alpha
b\beta_{2}k_{2}}{b+\alpha}}.$

Since $\bar{x}>0$ one obtains that
$k_{2}>\frac{b(b+\alpha)}{\alpha\beta_{2}}.$ In addition,
$f(\bar{x})<g(\bar{x}),$ from this after some calculations we get the
condition for parameters:

$k_{2}>\frac{b(b+\alpha)}{\alpha\beta_{2}}\left(\frac{\beta_{2}}{\beta_{1}}\left(1-\frac{\beta_{1}k_{1}}{b+\alpha}\right)+\frac{1}{b}\right)^{2}$

But, from $\beta_{1}k_{1}<b+\alpha$ and using positiveness all parameters we
have

$k_{2}>\frac{b(b+\alpha)}{\alpha\beta_{2}}\left(\frac{\beta_{2}}{\beta_{1}}\left(1-\frac{\beta_{1}k_{1}}{b+\alpha}\right)+\frac{1}{b}\right)^{2}>\frac{b(b+\alpha)}{\alpha\beta_{2}}\left(\frac{1}{b}\right)^{2}=\frac{b+\alpha}{b\alpha\beta_{2}}>\frac{1}{\alpha\beta_{2}}.$

From this we get $\beta_{2}k_{2}>\frac{1}{\alpha}>1$ which is contradiction to
the conditions (2.2). Hence, there is no any positive solution in this case
(Fig. 2). Theorem is proved. ∎

Figure 1:
$\beta_{1}k_{1}>b+\alpha:\alpha=0.3,b=0.2,\beta_{1}=0.7,\beta_{2}=0.6,k_{1}=1,k_{2}=0.3$

Figure 2:
$\beta_{1}k_{1}<b+\alpha:\alpha=0.3,b=0.2,\beta_{1}=0.7,\beta_{2}=0.6,k_{1}=0.5,k_{2}=0.3$

######  Definition 2.5.

[4]. A fixed point $p$ for $F:\mathbb{R}^{m}\rightarrow\mathbb{R}^{m}$ is
called _hyperbolic_ if the Jacobian matrix $\textbf{J}=\textbf{J}_{F}$ of the
map $F$ at the point $p$ has no eigenvalues on the unit circle.There are three
types of hyperbolic fixed points:

(1) $p$ is an attracting fixed point if all of the eigenvalues of
$\textbf{J}(p)$ are less than one in absolute value.

(2) $p$ is an repelling fixed point if all of the eigenvalues of
$\textbf{J}(p)$ are greater than one in absolute value.

(3) $p$ is a saddle point otherwise.

Conjecture 1. [3] If $\beta_{2}=0$ then for an initial point
$\lambda^{0}=\left(x^{0},u^{0},y^{0},v^{0}\right)\in S^{3}$ (except fixed
points) the trajectory has the following limit

$\lim_{n\to\infty}V^{(n)}(\lambda^{0})=\begin{cases}\lambda_{1}&\text{if }\ \
\ \ \beta_{1}k_{1}\leq b+\alpha,b\alpha>0\\\ \lambda_{16}&\text{if }\ \
u^{0}+v^{0}>0\ \ \text{and}\ \ \beta_{1}k_{1}>b+\alpha,b\alpha>0\\\
\end{cases}$

######  Theorem 2.6.

If $\beta_{2}=0$ and $\beta_{1}k_{1}>b+\alpha,b\alpha>0$ then there exists a
neighborhood $U(\lambda_{16})$ such that

$\lim_{n\to\infty}V^{(n)}=\lambda_{16}.$

###### Proof.

If we prove the attractiveness of the fixed point $\lambda_{16}$ then based on
general theory (see [4], [8]) we can say that there exists required
neighbourhood. Assume that $\beta_{2}=0$ then the Jacobian matrix of the
operator $V$ is

$J=\left[\begin{array}[]{cccccc}1-b-\beta_{1}A&-\beta_{1}k_{1}x&0&-\beta_{1}k_{2}x\\\
\beta_{1}A&1-b-\alpha+\beta_{1}k_{1}x&0&\beta_{1}k_{2}x\\\ 0&\alpha&1-b&0\\\
0&0&0&1-b\end{array}\right]$

Recall that
$\lambda_{16}=\left(\frac{b+\alpha}{\beta_{1}k_{1}},\frac{b(\beta_{1}k_{1}-b-\alpha)}{\beta_{1}k_{1}(b+\alpha)},\frac{\alpha(\beta_{1}k_{1}-b-\alpha)}{\beta_{1}k_{1}(b+\alpha)},0\right),$
then

$J(\lambda_{16})=\left[\begin{array}[]{cccccc}1-b-\beta_{1}A&-(b+\alpha)&0&-\frac{k_{2}}{k_{1}}(b+\alpha)\\\
\beta_{1}A&1&0&\frac{k_{2}}{k_{1}}(b+\alpha)\\\ 0&\alpha&1-b&0\\\
0&0&0&1-b\end{array}\right]$

From this to find eigenvalues we get

$|J(\lambda_{16})-\mu
E|=\left|\begin{array}[]{cccccc}1-b-\beta_{1}A-\mu&-(b+\alpha)&0&-\frac{k_{2}}{k_{1}}(b+\alpha)\\\
\beta_{1}A&1-\mu&0&\frac{k_{2}}{k_{1}}(b+\alpha)\\\ 0&\alpha&1-b-\mu&0\\\
0&0&0&1-b-\mu\end{array}\right|=0$

where $E$ is an identity matrix, solving this determinant we obtain the
following equation:

$(1-b-\mu)^{2}((1-b-\beta_{1}A-\mu)(1-\mu)+\beta_{1}A(b+\alpha))=0$ (2.5)

Thus, two coinciding solutions of the (2.5) are $\mu_{1}=\mu_{2}=1-b$ which
are always between 0 and 1. Next, we find other solutions, from (2.5) we have

$\mu^{2}-(2-b-\beta_{1}A)\mu+\beta_{1}A(b+\alpha)+1-b-\beta_{1}A=0\ \ \ \
\Rightarrow$

$\mu^{2}-(1-b+1-\beta_{1}A)\mu+\beta_{1}A\alpha+(1-b)(1-\beta_{1}A)=0$

then the discriminant is

$D=(1-b+1-\beta_{1}A)^{2}-4(1-b)(1-\beta_{1}A)-4\beta_{1}A\alpha=(b-\beta_{1}A)^{2}-4\beta_{1}A\alpha$

thus,

$\mu_{3}=\frac{2-b-\beta_{1}A-\sqrt{(b-\beta_{1}A)^{2}-4\beta_{1}A\alpha}}{2},$
$\mu_{4}=\frac{2-b-\beta_{1}A+\sqrt{(b-\beta_{1}A)^{2}-4\beta_{1}A\alpha}}{2}.$

where $A=k_{1}u=\frac{b(\beta_{1}k_{1}-b-\alpha)}{\beta_{1}(b+\alpha)}.$

Case: $(b-\beta_{1}A)^{2}-4\beta_{1}A\alpha\geq 0.$ Let us show that
$|\mu_{3}|<1,|\mu_{4}|<1.$ Since $\beta_{1}k_{1}>b+\alpha,b\alpha>0$ we have

$b+\beta_{1}A+\sqrt{(b-\beta_{1}A)^{2}-4\beta_{1}A\alpha}>0$

and

$b+\beta_{1}A-\sqrt{(b-\beta_{1}A)^{2}-4\beta_{1}A\alpha}>b+\beta_{1}A-\sqrt{(b-\beta_{1}A)^{2}}>b+\beta_{1}A-|b-\beta_{1}A|>0.$

From this we get $\mu_{3}<1,\mu_{4}<1.$ We consider the conditions
$\mu_{3}>-1,\mu_{4}>-1.$ It is enough to show that

$\frac{2-b-\beta_{1}A-\sqrt{(b-\beta_{1}A)^{2}-4\beta_{1}A\alpha}}{2}>-1,$

i.e., $b+\beta_{1}A+\sqrt{(b-\beta_{1}A)^{2}-4\beta_{1}A\alpha}<4.$ Since

$b+\beta_{1}A+\sqrt{(b-\beta_{1}A)^{2}-4\beta_{1}A\alpha}<b+\beta_{1}A+|b-\beta_{1}A|,$

it follows that if $b>\beta_{1}A$ then $2b<2<4,$ if $b<\beta_{1}A$ then
$2\beta_{1}A=2\beta_{1}k_{1}u<2\beta_{1}k_{1}\leq 2(1+b)<4,$ because, by the
conditions (2.2) it obtains $|b-\beta_{1}k_{1}|\leq
1\Rightarrow\beta_{1}k_{1}\leq 1+b.$

Case: $(b-\beta_{1}A)^{2}-4\beta_{1}A\alpha<0.$ First, we show that
$\beta_{1}A\leq 1.$ By the conditions (2.2) we have
$|\alpha+b-\beta_{1}k_{1}|\leq 1$ and from this $\beta_{1}k_{1}-b-\alpha\leq
1,$ so $\beta_{1}A=\frac{b(\beta_{1}k_{1}-b-\alpha)}{b+\alpha}<1.$ In
addition,
$\mu_{3,4}=1-\frac{b+\beta_{1}A}{2}\mp\frac{\sqrt{4\beta_{1}A\alpha-(b-\beta_{1}A)^{2}}}{2}i,$
and from this

$|\mu_{3,4}|=\sqrt{\left(1-\frac{b+\beta_{1}A}{2}\right)^{2}+\frac{4\beta_{1}A\alpha-(b-\beta_{1}A)^{2}}{4}}=$

$=\sqrt{1-\beta_{1}A(1-\alpha)-b(1-\beta_{1}A)}<1.$

The proof of the Theorem is completed. ∎

Conjecture 2.[3] If $\alpha b\beta_{1}\beta_{2}k_{1}>0$ then for an initial
point $\lambda^{0}=\left(x^{0},u^{0},y^{0},v^{0}\right)\in S^{3}$ (except
fixed points) the trajectory has the following limit

$\lim_{n\to\infty}V^{(n)}(\lambda^{0})=\begin{cases}\lambda_{1}&\text{if }\ \
u^{0}=v^{0}=0\ \ \text{or}\ \ \beta_{1}k_{1}\leq b+\alpha\\\
\lambda_{17}&\text{if }\ \ u^{0}+v^{0}>0\ \ \text{and}\ \
\beta_{1}k_{1}>b+\alpha\\\ \end{cases}$

######  Theorem 2.7.

If $k_{2}=0$ then for an initial point
$\lambda^{0}=\left(x^{0},u^{0},y^{0},v^{0}\right)\in S^{3}$ (except fixed
points) the trajectory has the following limit

$\lim_{n\to\infty}V^{(n)}(\lambda^{0})=\lambda_{1}=(1;0;0;0)\ \ if\ \ u^{0}=0\
\ or\ \ \beta_{1}k_{1}\leq b+\alpha$

###### Proof.

Suppose that $k_{2}=0$ then the operator (2.1) has the following
representation:

$V:\left\\{\begin{aligned} x^{(1)}&=x+b-bx-\beta_{1}k_{1}ux\\\
u^{(1)}&=u-bu-\alpha u+\beta_{1}k_{1}ux\\\ y^{(1)}&=y-by+\alpha
u-\beta_{2}k_{1}uy\\\ v^{(1)}&=v-bv+\beta_{2}k_{1}uy\end{aligned}\right.$
(2.6)

1) Assume $u^{0}=0$ then $u^{(n)}=0,$ and so from first equation of (2.6) we
have

$\lim_{n\to\infty}x^{(n+1)}=x\lim_{n\to\infty}(1-b)^{n}+b\sum_{k=0}^{\infty}(1-b)^{k}=1\
\ \text{for any}\ \ \lambda^{0}\in S^{3}$

and a limit of the operator is $\lambda_{1}.$

2) For $\beta_{1}k_{1}\leq b+\alpha$ from second equation of (2.6) one has
$u^{(1)}=u-(b+\alpha-\beta_{1}k_{1}x)u\leq u,$ so the sequence $u^{(n)}$ has a
limit $\bar{u}.$ Assume $\bar{u}>0,$ then by taking a limit we get

$\lim_{n\to\infty}u^{(n+1)}=\lim_{n\to\infty}u^{(n)}-\lim_{n\to\infty}(b+\alpha-\beta_{1}k_{1}x^{(n)})u^{(n)}\Rightarrow$
$\Rightarrow\bar{u}=\bar{u}-(b+\alpha)\bar{u}+\beta_{1}k_{1}\lim_{n\to\infty}x^{(n)}u^{(n)}\Rightarrow\lim_{n\to\infty}x^{(n)}\left(\frac{u^{(n)}}{\bar{u}}\right)=\frac{b+\alpha}{\beta_{1}k_{1}}\geq
1$

Note that $u^{(n)}$ non-increasing sequence and for any $n\in N,$ $x^{(n)}\leq
1,$ so from $\frac{b+\alpha}{\beta_{1}k_{1}}\geq 1$ we have a contradiction.
Thus, $\bar{u}=0.$ Next, by adding last two equations of (2.6) we have
$y^{(1)}+v^{(1)}=(1-b)(y+v)+\alpha u.$ By denoting $y^{(n)}+v^{(n)}=z^{(n)}$
one has $z^{(1)}=(1-b)z+\alpha u.$ We formulate a new operator with respect to
$u,z:$

$W:\left\\{\begin{aligned} u^{(1)}&=u-bu-\alpha u+\beta_{1}k_{1}ux\\\
z^{(1)}&=z-bz+\alpha u\\\ \end{aligned}\right.$ (2.7)

Here we have a useful lemma.

######  Lemma 2.8.

If $\beta_{1}k_{1}\leq b+\alpha$ then the set

$M=\\{(u;y)\in S^{1}:bz-\alpha u\geq 0\\}$

is an invariant with respect to operator (2.7).

###### Proof.

Let $(u,z)\in M,$ i.e., $bz-\alpha u\geq 0.$ We check the condition
$bz^{(1)}-\alpha u^{(1)}\geq 0$:

$bz^{(1)}-\alpha u^{(1)}=b(z-bz+\alpha u)-\alpha(u-bu-\alpha
u+\beta_{1}k_{1}ux)=$ $=bz-\alpha u+b\alpha u-b^{2}z+b\alpha
u+\alpha^{2}u-\alpha\beta_{1}k_{1}ux=$ $=bz-\alpha u-b(bz-\alpha u)+\alpha
u(b+\alpha-\beta_{1}k_{1}x)=$ $=(bz-\alpha u)(1-b)+\alpha
u(b+\alpha-\beta_{1}k_{1}x)\geq 0.$

Thus, the Lemma is proved. ∎

By the Lemma (2.8) we show the existence of limit $z^{(n)}.$ Assume that
$(u^{0},z^{0})\in M$ then $z^{(n)}$ is decreasing. Let be $(u^{0},z^{0})\notin
M$ then there are two possible cases:
(a) If after some finite step $k,$ $(u^{(k)},z^{(k)})\in M$ then
$(u^{(n)},z^{(n)})\in M$ for all $n>k,$ so $z^{(n)}$ is decreasing for all
$n>k.$
(b) If $(u^{(n)},z^{(n)})\notin M$ for any $n\in N$ then $bz^{(n)}-\alpha
u^{(n)}<0,$ $\forall n\in N,$ and so the sequence $z^{(n)}$ is increasing.
Therefore, the sequences $z^{(n)}$ has a limit $\bar{z}.$ If we take a limit
from $z^{(n+1)}=(1-b)z^{(n)}+\alpha u^{(n)},$ from positiveness of $b$ and
from $\bar{u}=0$ we get $\bar{z}=0.$ Consequently,

$\lim_{n\to\infty}(u^{(n)}+y^{(n)}+v^{(n)})=0$

and so $\lim_{n\to\infty}x^{(n)}=0.$ The proof of the theorem is finished. ∎

##  3 Conclusion

The crucial point of the SISI model is that an individual can be infected
twice. We proved uniqueness of the positive solution of the equation (2.3) and
it provides uniqueness of fixed point of the operator inside of the simplex
$S^{3}.$ Biological meaning of this result is that if susceptibility of
persons in $S$ multiplying infectivity of persons in $I$ greater than sum of
birth rate and recovery rate then the disease becomes endemic at the limit
(assume that operator has a limit). In addition, there is no susceptibility of
persons in $S_{1}$ such that for any initial state near $\lambda_{16}$ number
of second time infected persons disappears at the limit. If susceptibility of
persons in $S$ multiplying infectivity of persons in $I$ less than sum of
birth rate and recovery rate and either there is no infectivity of persons in
$I_{1}$ or susceptibility of persons in $S_{1}$ then for any initial states
stays only susceptible persons who did not have the disease before. $\Box$

References

## References

  * [1] Johannes M$\ddot{u}$ller, Christina Kuttler. Methods and models in mathematical biology. Springer, 2015, 721 p.
  * [2] D. Greenhalgh, O. Diekmann, M. de Jong. Subcritical endemic steady states in mathematical models for animal infections with incomplete immunity. Math.Biosc. 2000, 165, 25 p.
  * [3] S.K. Shoyimardonov. A non-linear discrete-time dynamical system related to epidemic SISI model. In arXiv:2008.12677v1 [math.DS], 2020, 17 p.
  * [4] R.L. Devaney. An Introduction to Chaotic Dynamical System. Westview Press. 2003, 336 p.
  * [5] R.N. Ganikhodzhaev, F.M. Mukhamedov, U.A. Rozikov. Quadratic stochastic operators and processes: results and open problems, Inf. Dim. Anal. Quant. Prob. Rel. Fields. 14(2). 2011, pp. 279-335.
  * [6] Y.I. Lyubich. Mathematical structures in population genetics, Springer-Verlag, Berlin. 1992, 383 p.
  * [7] U.A. Rozikov, Population dynamics: algebraic and probabilistic approach. World Sci. Publ. Singapore. 2020, 460 pp.
  * [8] U.A. Rozikov, An introduction to mathematical billiards. World Sci. Publ. Singapore. 2019, 224 pp.

Shoyimardonov S.K.,
---
V.I.Romanovskiy Institute of Mathematics, 4, University str., Tashkent,
Uzbekistan.
e-mail<EMAIL_ADDRESS>